There's no way to to this out of the box. I added custom search engine with keyword "yes" and search string
javascript:[...document.querySelectorAll('div[role="button"]')].filter(i => i.textContent === 'Yes')[0].click()
and named it 'Say "yes" to Google Calendar invitation'.
And the other one with keyword "no" and search string
javascript:[...document.querySelectorAll('div[role="button"]')].filter(i => i.textContent === 'No')[0].click()
and named it 'Say "no" to Google Calendar invitation'.
You can also create "Maybe" if you want to.
To run it you should focus URL panel (Ctrl + L or Cmd + L) and they type "yes" or "no" depending on your decision.
How does it work?
Instead of loading search engine URL, this custom engine will execute a small piece of JavaScript. This piece of JavaScript first searches for a button with the appropriate text on the page and then clicks it.