When I turn the averaging mode on, the fader jumps up to +10db. I find it annoying, is there a way around it?

Yes, this is a known little issue. But fortunately very easy to get over. Upon playback, the faders will go to 0.
Also, putting “park on silence” on and setting a slow release will get over this issue and give you better tracking for material that is made of broken regions (checker-boarded).