You'd sort that by tightening the grub screws that are located on the bottom of the shell.
Definitely check the chain tension first though and if you've been riding with it loose then you might want to strip it all down and check that the insert (or the shell) hasn't worn.
Think play is between eccentric insert and shell. In that when I move the crank the whole silver bit moves in the shell.