Is that first picture showing the tilt position of the engine when you are having the problem? If so, that part looks fine for what you have.
Where did you get the boat? Do you know if that engine came with it? Reason why I ask these questions is this:
1. That engine was not mounted in that scored hole on your boat. No deformation of the transom where the clamp bracket would have bitten into it.
2. To go up 3 holes at 3/4" per hole would put your antiventilation plate...(that horizontal plate just below your level in your first pic.) above engine running water level and the prop would ventilate.......suck in surface air....speed reduction, engine rpms excessively high.....racing.
3. No doubt the clamp bracket is scored. If you know for a fact that the rig is as it is when new then I'd say that's marks from the bolts holding the engine in the shipping container before Tracker put the rig together.
-------------------------------
Need 3 things:
1. Get down on one knee about 6-8 ft directly behind the boat and snap a picture showing the engine in the normal run tilt....as shown in pic #1 I assume, and shows the shape of the hull...dead rise and location of the AV plate with respect to the bottom. If you can't get the shot on one knee, lie down and take it. You want the camera to shoot approximately parallel to the hull from transom to mid ship.
2. While your camera is out, take a couple of pictures of the interior of your boat with equipment shown as it is on the water when you are having the problem.
3. How much do you weigh, and do you take others along when you have the problem.
-------------------------------
Possible causes of the problem:
1. Engine hp not high enough to power through your problem. A 50 horse 4 stroker on a 17' boat, even if it is a Tracker alum. isn't all that much if loaded.
2. Thrust angle too high (lower unit at the propeller end is too far from the transom) pushing the transom down during the hole shot. If your setup is as I mentioned above that isn't your problem.
3. Too much weight aft. The pictures I need to see will help with that.
That about sums it up.
When we finish, I might suggest that you install a "Whales Tail" (brand name....others similar out there with different names) or similar stern lifting device. Costs around a hundred bucks, give or take, and mounts to the anti ventilation plate. Gets it's name from the fact that it looks like it's name sake. What it does is act like a big board attached to your engine and the force of the incoming water against it causes it to rise taking the stern up with it and decreases your nose high attitude and duration of the hole shot. For your problem and your engine, boat, performance envelope, it probably will be just the thing
Will check back in the am and other times the next couple of days or so.
Mark