If you have 11-32t on your wheel already with the 105 GS, how far have the b-screw gone in?
If it managed to clear a 32t cassette on the small ring with little adjusting on the b-screw, then try and tighten it further and see how much gap you can obtain between the upper jockey wheel and 32t sprocket.
AFAIK it can only take a 28. Its first gen sram road.
The problem is without a washer the mech moves side to side 4/5mm on the knuckle.
I'm sorry if my layman's grasp is a bit weak.
Edit; in hindsight I should read who you replied to. As you were.
There's a possibility it may work.
If you have 11-32t on your wheel already with the 105 GS, how far have the b-screw gone in?
If it managed to clear a 32t cassette on the small ring with little adjusting on the b-screw, then try and tighten it further and see how much gap you can obtain between the upper jockey wheel and 32t sprocket.