Update - Solved
The carb barb rubber cap was messed up on the right carb only. When the bike would get hot it would really start to seep air. It wasn't something obvious upon visual inspection. The air cap I use on the left carb is homemade, a piece of hose with a plug in it. The ones that come with the carbs do not seem to be durable (My left one just disappeared one day while driving)
I should have checked this right away, but the way these carbs are mounted, the barb on the left carb faces to the outside of the bike, and is really obvious. The barb on the right carb faces to the inside, so when you are looking at it you don't see it. It is kind of "out of sight, out of mind"