There seems to be a contradiction between what these individuals are telling Paul “through the Spirit” and what Paul felt constrained to do by the Spirit (20:22-23).
1. Paul is disobedient to the Spirit.
The Spirit is telling Paul not to go, and he is simply being stubborn, disobedient, and determined to go to Jerusalem regardless.
However, the language in ch.20 is that the Spirit is constraining him to go (vv.22-23).
Paul’s attitude is not one of rebellion, but utter submission (v.24).
Furthermore, the rest of the book (chs. 21-28) outlines Paul’s path from Jerusalem to Rome as something positive and fitting the very programmatic design of 1:8.
2. “Soft prophecy.”
We have a case here of what might be called “soft prophecy,” i.e., it comes from the Spirit’s influence generally speaking, but is open to error and misconstrual. Therefore, what the disciples here are telling Paul to do in 21:4, as well as 21:10ff, is generally but not perfectly accurate (Wayne Grudem’s view, popular among many continuationists and charismatics).
It is troubling to surmise that prophetic revelation is not reliable or entirely accurate, or that God would fail to convey this information without it being intercepted by human fallibility. If it is true that prophecy can error, what does this mean for our ability to trust other prophecy / the Bible as a whole?
It is odd to expect a qualitative difference between OT prophecy (proper) and NT “soft” prophecy, especially in terms of it moving from infallible (better) to fallible (worse), especially when everything else about the New Covenant is better. Unless given really good reason, we should assume both forms of prophecy to be the same. And if there is a reasonable way to understand this passage along those lines (i.e., prophecy remains the same) that is to be preferred.
Many appeal to Agabus’ prophecy in 2:10ff as an example of such “soft” potentially somewhat erroneous prophecy, since Agabus predicts that the Jews will bind Paul when in fact it ends of being the Romans. However, this is surely to press the details too far. This is not how prophecy works elsewhere (consider Acts 2:36 where Peter accuses his Jewish hearers of killing Jesus, although it was in fact the Romans who in fact carried out the execution). So here, although the Romans in fact arrest Paul, the Jewish crowds certainly play a role in his arrest (ch. 21).