My only guess is, the shield kept the heat around the manifold which means less energy was lost to heating the manifold up since it stayed hot with the shield..

and now with out the shield you are loosing energy that was used to spool the turbo to heating the manifold...

I dont think that just taking off the shield would create any noticible delays in spool though... unless a bolt was keeping a hole closed and now you have an exhuast leak...

Did you just take off the shield, or did you not put it back on after doin something else?