Alas, the "mute" feature only keeps new messages from putting that thread back into the inbox. It doesn't keep them out of a label even when using that label as a secondary inbox. See https://support.google.com/mail/answer/47787?hl=en&ref_topic=3394654
AFAICT filter criteria apply to incoming messages before spam filtering and other filters and such, while the filter actions apply to the message or its thread (add label, skip inbox).
-is:mute worked in a filter (ie if the test applied to the new message's thread instead of to the new message, which isn't yet muted), you could use one filter to skip the inbox and a second filter with the same criteria plus
-is:mute to apply the label.
Or similarly if the filter criteria could test labels on the new message's thread, you could move a thread to a different label as a hack way to mute it, then use a pair of filters with
-label:muted as part of the second filter's criteria.
But currently gmail doesn't work that way. You can use the Send Feedback feature (in the gear menu) to request one of these tweaks.
As s desperate way to handle an occasional annoying thread, you could add a filter just for that thread which just removes it from the Inbox.
One more idea: Write a Greasemonkey script or a Chrome extension to extend Gmail.