If there are really deep scars on the ball, I would be concerned and check the coupler. The scratches on the "car side" are there because during towing all the force is exerted on that side of the ball. Only during breaking does it exert force on the back of the ball. After many many miles, the scratches will cover the ball entirely until it becomes a dull grey in color. When you touch the ball at this point, it will leave a residue on your finger, as well as the inside of the coupler. It is my belief that this residue acts like a dry lubricant, kinda like the dry graphite lubricant we used to use on pinewood derby cars as kids. This is the story of heavily used balls. Recreational ball/coupler setups often develop surface rust, but that is harmless. I think the rust that smokeonthewater is talking about is heavy pitting in the coupler. This will cause the heavy deep scars I mentioned. If that's the case, and I doubt it is, you may need to replace the coupler on the trailer. I agree with the %99.9999 evaluation of your situation being just fine. As always, a picture is worth a 1,000 words.
Here we are, talking about our greasy scratched balls... :doh: