I can't see why you couldn't pair ST-5700 with RD-7800, but that combination doesn't appear in any of the official charts because RD-7800 was discontinued before ST-5700 was released. Front compatibility might be an issue, as I have a feeling that 5700/6700/7900 have a different pull ratio compared with 5600/6600/7800
Somebody has done a really good job of trying to answer that question:
http://www.celebrazio.net/bicycling/shimano_compatibility.html
I can't see why you couldn't pair ST-5700 with RD-7800, but that combination doesn't appear in any of the official charts because RD-7800 was discontinued before ST-5700 was released. Front compatibility might be an issue, as I have a feeling that 5700/6700/7900 have a different pull ratio compared with 5600/6600/7800