Your issue lies in the fact that the "-
" sign is used by Gmail as a search operator.
Please have a look at this link.
Search operators you can use with Gmail
+-----------------------------------+---------------------------+
| What you can search by | Search operator & example |
+-----------------------------------+---------------------------+
| Remove messages from your results | - |
| | Example: dinner -movie |
+-----------------------------------+---------------------------+
So, using your syntax, what you are actually asking, is for all messages under the sub-label Z
to be to excluded from your search.
When it comes to labels the recommended way is:
+------------------------------------+---------------------------+
| What you can search by | Search operator & example |
+------------------------------------+---------------------------+
| Messages that have a certain label | label: |
| | Example: label:friends |
+------------------------------------+---------------------------+

Please have a look at the following images as well and notice the difference.
Top search bar while in the inbox

And now top search bar while in the label

Note: Labels are only added to a message, and not an entire conversation.
Furthermore, I am a bit confused by your provided example:
(a) John (0) label:XY label:XY/Z
(b) Mary (0) label:XY label:XY/Z
(c) Zack, me (5) label:XY label:XY/Z
where (0)
indicates that I have not yet answered emails (a) and (b),
while (5)
indicates that 'Zack' and myself have already exchanged a
bunch of emails.
Depending on where the numbers appear -most likely under Settings /Labels
- (0)
indicates that you have (0)
conversations, while (5)
indicates that there are (5)
conversations that you have exchanged with 'Zack' (within a bunch of emails under that label).
On the other hand, if the (5)
appears on the left sidebar next to a label it indicates that there are (5)
unread emails among the ones you have exchanged with 'Zack' (within a bunch of conversations under that label). But then you wouldn't see any (0)'s
on the sidebar.
EDIT (following your comments)
" You are right, ...labels are apparently assigned to messages, not (as suggested by the Gmail UI) entire conversations/threads..."
Please go to your Sent
emails and check whether you have assigned to your specific sent emails ("emails to which I have already answered") XY
or Z
labels.
In addition, as you can see the above screenshots, rather than typing -Z
or Z
, type XY
instead. This brings up label XY/Z
as well.
Also. As you can see in the image bellow, when typing just -Z
Gmail presents a few thousand results. So it treats it as the letter Z.

Final words. If nothing works, save your sanity by changing Z to ZAA or something. After all they are just labels. Your labels. They are there to make your life easier.