I see it on almost every bike that has that pump, and mine is wobbly as well. Could maybe also have something to do with how far the handle is pushed in when installing it on the frame. That one looks doesn't look pushed in at all. Anyway, count yourself lucky, because it bothers me personally.
Handle on mine is solid thank you very much