Youtube use Google accounts to authenticate users and they aren't intended to be used by several users so the "managers" concept doesn't apply.
By the other hand, Youtube channels can be linked to Google+ pages and be managed by the Google+ pages managers. At this time, Google+ pages are managed from Google My Business dashboard.
From Add and remove page managers - Google My Business Help
Removing managers
To remove a manager, or remove yourself as a manager of a page:
- Log in to Google My Business and choose the page you’d like to manage.
- Click the gear icon next to your business name and choose Settings.
- Click the Managers tab in top navigation menu.
- Click the X associated with the person you'd like to remove.
When a manager is removed, both the former manager and the page owner
will be notified and told who removed them.
When you remove a manager, they’ll no longer be able to act as that
page, or take any kind of administrative actions. However, all of that
manager’s past posts, comments, and other actions will remain intact.
Learn how to transfer ownership of a Google+
page.