If you live in a climate that gets below freezing don't chance it. If for any reason water has gotten into the oil, it can freeze and damage your drive.
I know my drive oil looks used even after a moderate summer of usage. This summer rained so much that we probably only went out maybe 10 times. Maybe 10 hours of run time and the oil looked bad.
For what it takes to change the outdrive oil, it's a no brainer. A couple quarts of the proper oil, a drain pan (I use a throw away aluminum foil pan), two replacement gasket washers for the drain plugs and the cheapo oil pump tool that screws onto the oil bottle.