You need to call in when you cross that invisible dotted line even if you don't "Land" in Canada. This is to advise them you are in their waters. If you "land", which means you've officially entered their country, then you have to check in at one of their "marine stations". Basically it's just a land-line phone located at various marinas around the border. You stop there and call in and give them all your info (boat reg #, you and your pax name and id #'s, etc) and they give you a number to verify you checked in. Keep this number handy if you ever get stopped. Once you've done this, then you can anchor or travel in the country.
If you have a NEXUS card (and everyone on board has one) then you can just call in.
IF YOU HAVE "LANDED" in Canada, then you need to re-enter the USA and contact our border folks. This is usually much more of a hassle than the Canadian side, because we don't have all the land-line locations like they do. You have to call into their #, then wait at your boat if they want to come out and inspect the boat...which can take HOURS, or they direct you to another location to checkin.
btw, the definition of "landing" or entering Canada is if you stop at a port, anchor, set foot on land (even if it's below water), meet up with another vessel. There is a grey area if you 'loiter' in their waters... it's OK to pass through, but if you're just going around in circles or drifting and swimming it's up for interpretation.