You could have already worn out plastic bushings on steering shaft due to use and/or using that OB with too loose stearing load. Move middle OB back and forth and check if with too much side play on steering shaft. If don't plan changing them as it's a time demanding job, adjust swivel bracket's steering load to a point than when tiller grip ir released boat will follow a straight water course. Steering shaft must be well internally greased, will need a grease gun for that job.
Happy Boating