Even though it appears that you’ll have limits in the movements of the pieces:
I think the swivel casters are best be in the trailing position, near the pushers so they can push and steer from the same end [that is, opposite of the steering in a car.]. But if you have to move the piece back in the direction that you just came from (or if the swivels are in the lead), then you need someone in front to steer and someone behind to push. [I’m assuming that a fixed pair is on, say, the left side, and the swivel pair are on the right, as opposed to front-back pairing.]
All of the pieces do not necessarily need to have the wheels configured the same. That is, some may have the fixed pair on the left and the other with the fixed pair on the right (or front or back, if it makes sense to use that arrangement.) You may want to consider when you want or need the smoothest movement, not to mention room to maneuver in the wings, or if there is some other critical timing. These considerations may determine what is optimum, considering your budget limitations. [For example, if you need to clear the stage quickly, then you’ll want to have the casters set up so the pieces can be pushed off smoothly, with the swivels trailing. But there may be a couple situations where you will just be stuck and you’ll have to make the best of it. On the other hand, once you get the set piece movement details worked out, there won’t be many problems.]
[Probably won’t hurt to label which end has the swivels and which end has the fixed.)
Also, after you get the piece in position, twist the swivel casters to the direction that they are to go next.
Inspect the wheels (and swivels) for threads, fibers, and other crud that can jam the wheel, axle, and swivel.
Joe