Altitude while flying

This is kinda like a yes and no thing... please be the judge here.

You know when you fly, that SL automatically corrects you're altitude to a specific hight from the ground? It's cool when flying over not-too-steep mountains and stuff. You'll fly over them and back down. This is nice.

Now, what I don't like is while I'm on the ground and maybe hit the FLY to move a little faster or get to a second story of a building with least-fuss-intended, once you go horizontal SL starts pushing you up up up up untill you are "that certain height". Annoying when you just want to move sideways a bit not too far. You have to move go down, lag, go back up, correct again.. you know.

Is there a way to work this around?