I would stuff a rag or paper towel down on top of the bearing to keep stuff out of it.
rig up a 1/2" drill motor to the shaft some how,
with someone else running it,
and then use emery cloth or some other abrasive with your fingers putting pressure on it to clean up the sleeve