If the spacer's that tight then (at least) one of the two bearing's outer races isn't seated, either all the way or it's cocked, guess I'd get a socket and do the tap all the way around the race til you hear the tone change. Mark the bore on both sides before you start see if moving one bearing in also moves the other out.
Sorry to say the bearings are often sacrificed in removal. With an under sized rod inserted part way hammer sideways on the end sticking out, the spacer WILL move sideways, then you can get the bearings out.