I have no inside information on how Twitter calculates 'user affinity'...but it seems likely to me that it's a 'two-way street' problem.
By that I mean, even if you haven't used Twitter's Find My Friends feature, a number of people who have you in their address book may have used that same feature to import their Gmail/Yahoo/Hotmail contacts. With the info from those people, Twitter knows to not only suggest you to them - but also them to you.
When you add in other 'tells' (e.g., mutual people you both interact with, geographic proximity, etc), chances are Twitter can make a good guess as to who's a connection and who's not. And then you get a somewhat-creepy email that seems to know more about you than it should.
LinkedIn does a similar thing - which is why you'll often receive a People You May Know suggestion on LinkedIn of someone you haven't spoken to for years. Perhaps you were automatically added to their email contact list years ago. When that person then imports their contacts into LinkedIn, LinkedIn makes the two-way connection in the background - and recommends they connect with you as well as you connect with them.
This is how I wound up getting a 'LinkedIn connection' suggestion from someone who emailed me about renting a room in my apartment off of a Craigslist ad almost 10 years ago (& 1000 miles away).
Which, not gonna lie, was a little creepy.