The inner tube should be just a few thousand's longer than the bronze bushings. The shim was used if the inner tube was too long.
If you fit the swing arm and it has some side play, slide sideways on the inner tube then the shim will fill up the sideplay.
I would install the swing arm with out the shim. Torque the bolt to about 50 ft/lbs. check for side play. Also check to see if the swing arm moves up and down.
On mine at 50 ft/lbs the swing arm had no play and would just barely hold the swing arm where you leave it and very easy to move. Add a few ft/lbs till the swing arm holds where you put it and has a bit of tension.
If the bolt is too loose then the inner tube isn't clamped between the frame sides. You will think the bushings are bad even when ok because the inner tube is turning around the bolt, not the swing arm bushings turning around the inner tube.
If with proper torque and you can't move the swing arm the inner tube is a bit short. Remove the swing arm and be sure the bushings are fully seated. If they are you might try sanding the bushings a bit, reinstall the S/A then recheck the fit.
Sand till you get proper fit.
Leo