Is there a way to disable just the popup that appears at the bottom right when receiving a message from someone, without disabling all notifications?
Right now, the only two states available are 1.)all notifications (audio, window flashing, and popups) turned on, or 2.)all notifications turned off.
You have likely already found it, but the toggle is found in the arrow drop-down menu. You can disable notifications for an hour, eight hours, or two days.
You may be able to learn more and/or inqure at the Google Product Forum for Hangouts: http://productforums.google.com/forum/#!forum/hangouts
An existing notifications-related thread can be found in the forums here: http://productforums.google.com/forum/#!topic/hangouts/P8RNjXTEvuc
The following link specifically mentions muting the sound, but it also took care of popups for me https://support.google.com/hangouts/answer/3111923?hl=en:
This setting applies to all Hangouts notifications from hangouts.google.com, Gmail, and the Chrome extension on the same desktop computer.
The mute setting worked for Hangouts in Yosemite (toolbar), in GMail and at hangouts.google.com, displaying a button that enabled me to cancel the mute from all three locations.