Depends on how heavy they are when they're filled, really. If it's a light load you won't really notice it much using one pannier, but if it's a heavy load you'll be glad of the distribution to the other side. I've never actually fallen off due to being loaded on one side but I've had a few bike-drop moments when trying to get stopped or started with higher-weight unbalanced loading.
^this, it makes standing your bike up a massive pain, but it's not a big deal to ride. If you're going to be on your bike for a while, I'd balance the weight across 2 panniers. For short journeys one will be fine (which side to have it is another debate). Packing heavy stuff (laptop, toolkit, water etc.) as far inboard and low as possible can also help.
^this, it makes standing your bike up a massive pain, but it's not a big deal to ride. If you're going to be on your bike for a while, I'd balance the weight across 2 panniers. For short journeys one will be fine (which side to have it is another debate). Packing heavy stuff (laptop, toolkit, water etc.) as far inboard and low as possible can also help.