Washers between the grease seals and frame wont work.
Your pivot tube should be a bit longer than the swing arm. A few thousandths inch. In the stock set up if there was excess side play, the pivot tube too long, they added shims between the grease seals and swing arm. The shims had a center hole large enough so the pivot tube could pass through.
This filled the gap betwe the length of the swing arm and length of the pivot tube.
Finding any of these shims might be tough. To determine the thickness you need to measure the length of you pivot tube and swing arm. Subtract the swing arm from the pivot tube. subtract a few thousandths from that. Devide by two, you want the same on both sides. Find some shim material that thick ness and cut them to fit.
Leo