As said first off it is all about the length of the inlet tract AFTER the throttle body. When you get off the accelerator the TB snaps shut but the engine can still use all the air that is in your intercooler, piping and intake manifold, and the engine is unloaded since you are on the clutch, the revs rise.
Happens to all of them, especially with a front mount as that increases the post-TB volume. And yeah, you do sound a bit like an L-plater, but I can live with that.