For a faster swim just swim with the shorts. The jersey will likely slow you down with the mesh pockets that are probably on the back. That being said, It's can be tricky putting on a tight jersey when you're all wet from the water, so practicing will help you from being caught off guard in the transition area. Try unrolling it as you put it on. Also, not sure what distance you're looking at, but with Sprint/Oly, just grab your race belt and go topless

What does USAT say about body paint? If Sports Illustrated can show topless women in body paint for their swimsuit issue, then you should be able to get away with drawing a singlet with a Sharpie.

Classy.

Pretty sure it's a WTC rule, but not a USAT rule.

I think that's right Jeff, WTC rule certainly and I'm pretty sure I've gone shirtless in USAT sanctioned races.
BUT, if I wanted to wear a top, I'd wear it during the swim. As Jeff points out it's probably going to cost you a few seconds on a short swim, but I try and keep things simple and would rather avoid getting tangled up in a tight wet top in T-1.

HIMFL - I asked a ref, "Can I go shirtless for the swim" He checked with the head ref and said "absolutely not."
waiting at the start, I looked around at the sea of shirtless men, and kicked myself for asking. Next time i shall draw one on with a sharpie.