A trick an old timer showed me, I won't elaborate how long ago that was - if the part can take it a light tap or rap - preferably on a vise or anvil- knocks the small slivers off. Like screw drivers and pliers and such, the impact will diminish the residual magnetism.
From a blog somewhere...
"Impact
Significant force can still throw a magnet out of magnetization. Hitting a magnet with enough force to destroy its north and south pole alignment will force the magnet out of alignment. Any violent trauma can cause this to happen including drilling and hitting with a tool. If the grains inside the magnet no longer line up properly, the magnet will no longer function properly."
I know most don't care to smack their precision instruments
but a lot of other items will take it. I find using files as much as I do, a light rap on the vise, clears the chalk, filings and mag bugs that gather.
Joe, I use duct tape more often for the slivers in my hands the glue works a little better on it and if I can see the direction the sliver is going in, I use 320 grit emery paper, place it on the affected area and swipe it in the direction to pull a sliver out- if I have the wrong direction though, it runs it in deeper.
Greg