The challenge here is that the natural thing to do is to use "IF" in the report filter just like you used in your question. Remember this: NEVER use "IF" in a filter.
The filter you want is this:
(?prompt? = 1 and [City] in ('A', 'B', 'C')) or
(?prompt? = 2 and [City] in ('D', 'E', 'F'))