I had this happen a few weeks ago, but only when I was fetching something from rear jersey pockets while descending so a lot of my weight was over the rear wheel. I had (over?)tightened my (threaded) headset prior to the ride, so maybe that had something to do with it too?
I checked wheel, all fine. Rear rack not loaded. So I guess I get bike shop to check headset.