Let me give you my 2 cents on this. I have a 2008 RAV4, 4 cyl. Tow rating is 1500. The V6 version is rated 3500. (why so much more???) My first boat was a 17 ft boat and trailer, weighed about 2600 loaded, NO trailer brakes. Pulled it all over the place, always felt fine however, I was more concerned about stopping the boat that getting it going (especially down hills). Second boat was 18 ft, weight of about 3300 loaded. It HAD surge brakes. I used to routinely pull it 20 miles, no issues. With the surge brakes and a better suspension, it felt safer than the smaller boat. I would say that any more than 20 miles, I would rent a truck. Also I preferred to keep my speed under 60 mph even though I crept up to 70 a few times.
Now as other posters mentioned, at some point with certain roads, it can feel like the tail wagging the dog. I felt that once and don't care to feel that again. That was the first time I towed the larger boat and my tongue was too light. Keeping your speed down under 60 helps a ton too. But to be truly safe, a larger vehicle is the way to go.