When looking at the pics, I wonder if it wouldn't be possible to have the brake arm pointing backwards, below the footpeg. This would then of course require the arm on the pedal to run horizontal as well, and be longer. Then a short, near vertical rod with uniballs between the arms.
I see you are using a stock brake pedal, cut off for a brake arm. I am pretty sure "Fast from the Past" has individual Tarozzi parts for sale, both for gear shaft and brake shaft. I got some from them a few years ago, saved me from cutting the stock parts.
Another solution for "mild" rearsets like yours, is something I saw on an XS at Phillip Island racetrack in Australia. He had simply welded on a new brake pivot bushing behind the frame tube, and moved the brake arm/shaft and unmodified stock brake pedal to the new pivot bushing. Then cut the brake stay accordingly. It may not look as good as an alloy pedal, but on the other hand, you avoid any linkage.
The optimal solution for any kind of rearsets would be a rear disc brake, or what about a hydraulic cylinder for the drum? Basically replacing the brake stay with some sort of hydraulic cylinder. Maybe a clutch slave from a car could be adapted somehow? I guess the drum brake works equally well if the drum brake arm is pushed backwards instead of forward.