If you aren't seeing fibers, the gelcoat is still there. Is it a metal flake bass boat? I've heard if you get down to the flake layer, it'll turn grey as you sand/polish.
First step is to clean the dirt off. Next, wet sand. If it's bad, you may have to start down around 400 grit or lower, then...