This is all a question of degree and repetition. Higher degree of severity, and higher number of occurrences will absolutely add up to be bad. Depending on the severity, you may not be hurting the hull, but you are making all hardware joints, and all glassed joints, stress. So even though the hull may stay together (one piece) all of the joints; top to the hull, deck to the hull, engine mounts, drive to the transom, windshield, etc etc. etc, are being stressed. All of us encounter situations where we misjudge things, cross wakes, wakes, swells with wakes, etc. etc., but the best thing to do for the comfort and safety of you and your passengers is to anticipate issues and minimize them. Maybe a better way to think about it is that constant pounding can't be good.
Learn what types of wakes cause the worst bang and learn how to deal with them. I work my throttle and steering and trim CONSTANTLY unless it is dead flat. There is always a way to soften the blow. Just keep practicing.
Edit: BTW, I carry a decent combo screwdriver and a 1/2 inch wrench at the helm. Whenever I think about it I tighten hardware. You will be amazed how much, and how often, they come loose