So you can assume one of the vanishing points to always be at the center.

That's not true. The vanishing points don't stay at the center. Nihmar's initial statement - that the vanishing points slide along the horizon line - is correct.

The thing is, we're not just rotating the boxes around their own origin - we're rotating them relative to some point farther back, so the boxes themselves can be treated as rotating around their origin and physically moved in space. So the vanishing point is sliding, and then the whole thing is being moved. That doesn't have anything to do with them not being boxes though (which you're right, they're not). Them not being boxes just means that those depth lines exist in pairs, with two going to one vanishing point, and the other two going off to a slightly different VP.