Welcome to the board.
It sounds like the shifter is out of adjustment inside somewhere or possibly bent shift forks. Do you have a repair manual. If so it should explain how the shifter works and how to check and adjust the mechanism.
It will also explain how to pull the engine and split the cases to get to the transmission to inspect the shift forks. It may also be the dogs and the slots in the gears are worn. They have a slight taper on the dogs and slots so they lock together when engaged. This taper can be worn so they don't lock together.
Only inspection can tell for sure.
With enough money any thing can be done. The XS650 engine may fit with enough mods to the XS500 frame.
You might try to find a forum that deals with the XS500, others with them might know some tricks to fix your problems with out as much work as I described.
Best of luck.
Leo
