You will find that folks go without stuff in shorter races, but at their peril. Never assume your tires are good and you won't puncture.
Two schools of thought:1. If you puncture in a sprint, the race is over because by the time you fix it, you'll have lost alot of time.
2. Carry what you need because you finish everything you start. And you prefer not to walk.
Your call.
In long races...If you flatten you could be in for a long wait for a ride back ,or a long walk.
It sounds as though your system works good for you,why change?...that's basicaly what I do, although sometimes I stuff the bag in my jersey pocket in T1 and drop it at T2.

I've an x-lab setup I use for races, and a bike bag that I use for training.

Not finishing a race to a flat, even a sprint, just seems silly to me. I've known friends who flatted and still managed to pull back to place in their age group.

The minor amount of weight (that won't make a difference) doesn't seem like a reason to me, and the walk back if I flat is more than reason enough.