-
Get bolsters with rubber guards
Problem with these is they encourage you to hold the bolster incorrectly. You should hold the bolster like a pencil so that if you miss your hand only receives a glancing blow which will still make you turn the air blue shouting but you won't break anything. By holding the bolster in your fist as most people instinctively do if you miss your hand still takes most of the force and as you pointed out the small bit of rubber between your hand and the hammer doesn't actually offer that much protection.
Get bolsters with rubber guards if you are doing it by hand. You WILL miss and smash your hand otherwise. I managed to do so, even with the guard, when taking out the chimney and fire place.
https://www.screwfix.com/p/magnusson-guarded-bolster-cold-chisel-set-3-pieces/5429V
An SDS with a chisel bit would be easier though if you can source one.