That brown stuff? Do you have a stainless steel prop? If so, that is likely rust. Remember a lot of SS alloys have enough steel to get rust flakes.
If you grind if off, you could unbalance the prop. I would leave it alone, or rub it with some very fine sandpaper.