-
Party wall agreements don't really work like that - ultimately, they cannot refuse consent (or access) based on whether they like the work or not, providing the appropriate procedures and timescales have been followed.
As @EstelleGetty mentions, it can be used as an opportunity for them to protect their future interests, viz. not wanting to build a shitty flashed in box as an example, by having a wall built up (and potentially shouldering some of the costs).
Having an independent surveyor prevents problem down the line - a shared surveyor is a nice idea, but that independence is a coin toss.
There seems to be a first mover advantage here - at our old place, the neighbours went into the loft and their dormer goes to the center line.
At one stage we wanted to go into the loft too, and would have needed to either build up the whatever-the-upstand-bit-is-called, or have some awful bodge with flashing over the top to stop crap getting into the gap.
My wife, who was the owner of the property, naively signed the party wall agreement. What she should absolutely have done was to have insisted on her own surveyor. Nothing but aggro because of not doing so.