I have Facebook open in one tab, and I open YouTube in another. YouTube asks me if I want to sign in with my Facebook account. Can YouTube tell if I have Facebook open in a different tab or window?
Yes, it is possible to tell that you are logged in (or at least used the service). In at least two ways:
1) Explicit connection. If YouTube uses the Facebook API to connect and check whether Facebook knows who you are in that session.
2) Using CSS tricks, where a website can check whether you have specific (well-known) images or files in your browser cache. They cannot tell who you are, but they can figure out which services you use.
The second way is how bookmarking services or OpenID providers can show you your top five services out of several hundreds they support.
Yes, it is possible, using an invisible 1x1 GIF web bug to access your Facebook cookies.
This was the source of the infamous Facebook Beacon fiasco. As a result of the ensuing lawsuit, Facebook has ceased to do this, and most likely will not try to again.
It appears that Facebook DOES know you are logged-in on a number of websites - that's probably because you are seeing the new "Like" social widgets ("plugins") that are all over the web.
To the average user, it appears that FB knows you are logged in, because you see which of your friends have "liked" that site or piece of content. However, technically, the FB social plugins are small iframes that are still hosted on facebook.com: