It will do no harm with it off, but will be noisier than it was with it on…
But…my main concern and thoughts when I see this happen, is around two things…
1 it shouldn’t happen really and often a sign of being installed wrongly. If it’s not done right and the visual inspection shows it’s still in tact and not burst and just came off from the tilting of the drive…it’s not been fitted properly.
2 would be either way you look at it…it’s either came off from not being properly fitted, or it’s as a result of wear or time gone by…my worry would be around the other bellows on the drive. I won’t need to tell you…that if the shift cable bellow has had the same ignorance or time gone by applied…it won’t just result in a noisy exhaust note…worst case it will be a whole lot worse or even the unthinkable, with the family onboard!! Best case is the drive shaft below giving up next and only a bearing or outdrive failing with water Ingress.
Anyway. Hate to be the voice of doom…but of all the things associated with an outdrive that scare me the most…it’s things like this. You’ve been lucky and had fore warning with the best of the three of the interfaces failing first. The one that does no real damage../but it’s certainly a sign of things much worse to come, of ignored. All the best.