We had Party Wall letters to our house when next door were building a home office in their garden. Not even touching the property border; never mind any walls.
Doesn't have to touch the border or walls to fall under the Party Wall Act...
"
2. What does the Act cover?
...
Excavation within 3 or 6 metres of a neighbouring building(s) or structure(s), depending on the depth of the hole or proposed foundations (see paragraphs 28 to 30).
Apparently that's only if they're digging down further than the foundations of the neighbouring property or it requires underpinning - would have to be a whopping great garden office for that to be relevant.
Doesn't have to touch the border or walls to fall under the Party Wall Act...
"
2. What does the Act cover?
...
"
Source: https://www.gov.uk/government/publications/preventing-and-resolving-disputes-in-relation-to-party-walls/the-party-wall-etc-act-1996-explanatory-booklet