assuming the hub bearings are very smooth, which they should be for a wheel to spin freely, then there's very little resistance to any lateral movement. so, especially with weight on it, the trailer would move left and right all the time, probably knocking you off balance, no?
is that dampened in any way? otherwise wouldn't it be twitchy as hell?