Any of those fine grinding polishing abrasives can be harmfull if inhaled. most are based on some form very fine dust-like silicon dioxide (glass) crystals.
In the lungs they create a medical condition
Silicosis which has killed off generations of metal finishers, stone workers and any number of related trades. "Black Lung" from coal dust is a somewhat similar disease. As I recall at times
Silicosis was called "white lung" by stone cutters and quarrymen
In industry now OSHA requires workers who risk exposure to wear very special dust filter masks or even positive pressure hoods and suits.
I doubt that our personal reloading exposure to a little bit of jewelers rouge in the polishing medium will have that drastic effect but once ANY gets in the lungs it stays there permanently.
That residual glass dust probably melts in the heat-pressure of combustion in the cartridge and along with glass particles from the primer could be part of the fouling in your bore. It could possibly be a contributor to bore wear.
If t'were me I'd clean it, probably by washing rather than blowing it into the shop air---sure don't want to dilute all those good lead particles from my bullet casting