There is a natural (programmatic) order to the processing of a conditional expression like the one you are passing to the (if:) macro, the actual order can depend slightly on the programming language being used but mostly it is based on BOMDAS.
This order can also be influenced by the implementation of the parser that is converting the TwineScript you enter into a Passage into something that the story format's engine can execute.
Each sub-clause of a multi part conditional expression is evaluated separately to determine if it result (Boolean) is either true or false , those sub-clauses results are then combined based on the joiners (and, or, etc...) used to determine if the overall result is true or false.
If we break down your original example (based on the error message) with get :
a. (history:) contains "passage1"
b. not (history:)
c. ??? contains "passage2"
Which results in:
a. Boolean: true or false depending on if the Passage name was found or not.
b. Error: because (history:) returns an Array and the not operator doesn't know what to do with it.
c. Potential Error: I am not sure what is being evaluated in this case.
If you break down the example supplied by @geekdragon you get.
a. (history:) contains "passage1")
b. (history:) contains "passage2")
c. not B
Which results in:
a. Boolean: true or false.
b. Boolean: true or false.
c. Boolean: true or false.