These can always be grey-area judgement calls. Kinda like how some folks down here try to stretch their tire mileage by running them until the cords show-thru and they go flat.
You could view this from three angles:
Immediate functional impact - Putting it back together as-is may not adversely affect the idling and performance.
Irritation factor - Yep, it'll clatter, and valve adjustments will be inconsistent.
Long term reliability - It may limp along for a thousand, or couple hundred, or 1 mile. Continued operation will likely produce a lot of iron filings. Rough bearing surfaces that no longer provide the requirements for thin-film boundary layer lubrication usually chew themselves up.
Depends on your plans, tolerance levels, and risk management. The choice is always yours.
I'm reminded of interesting solutions that folks have come up with in the past to 'band-aid' mechanical problems. Like that old trick of putting sawdust in a differential to quiet the howling noise. It actually works! Just long enuff to pawn-off the car onto an unsuspecting buyer. Used to be a rather common practice down here. And my other two favorites: Motor Honey and STP. There's no shortage of "put this magic stuff in your vehicle" products to reinforce the "ignore it and hope it goes away" philosophy of life.
*whew* Seems that the older I get, the longer the stories...