Depends on how you plan to 'finish' the rear area and exactly what you are asking if it's ok to use epoxy 1st on.
If you want to glue the 2 pieces of 3/4" ply together w/ epoxy then use poly for everything else, yes you can. BUT you need to be diligent and not get epoxy on either of the exposed faces of the finished 1.5" laminated transom or any of the edges.
If you want to glue the transom together, cover the transom w/ epoxy to waterproof it, bed it to the fiberglass transom skin, then tab it to the hull, ALL with epoxy and then PAINT the exposed epoxy, then YES you can. BUT where the stringers intersect the transom will need to also be done w/ epoxy, rather then poly. The rest of the stringer, bedding the stringers and tabbing them to the hull may be done w/ poly. But again you will need to be diligent to make sure you do the stringers 1st w/ poly then attach them to the transom w/ epoxy and tabbing.
Epoxy will adhere to polyester resin and gelcoat.
Gelcoat and poly resin WON'T adhere to epoxy, or at least not well and not without significant prep.
Any epoxy coated repairs will need to be covered or painted, it isn't UV stable.
rather then risk any problems w/ the adhesion between poly and epoxy, I'd use poly resin for everything.