It might look a bit crazy, but I'd like to do some stats on all tweets from the beginning.

I heard a team of scientists got them by asking Twitter directly, but I wondered if it was possible to get this data in another way.

At the end I will filter them, so I could also search in an archive file, but I need data fron a large time scale. Actually I don't need them all, the last year would be good (the more, the better).

This question Is there an app with a historical archive of twitter messages for a specific hashtag? suggests searching on Google, but I don't think I can "crawl" Google, so I'm not sure it's possible to fetch the data, or is it ? Plus I would like the tweets not to be added twice, and I'd like to get all retweets as well, which might not be included in Google search.

Another source could be the Library of Congress which will archive twitter history, but I've looked for this data, but didn't find anything, apparently they don't have it yet.

So do you guys know another source which keeps track of all tweets, or how I could use one of the above ? I'd appreciate any suggestion.

  • 3
    As far as I know, the Library of Congress isn't even archiving protected tweets (i.e., those from users with a non-public twitter feed). I doubt Twitter will open those up to you. Commented Dec 16, 2010 at 19:04
  • 1
    You would need many hundreds of terabytes of local storage to put all those tweets in, and then in which to create your relational database to store filtered data…
    – msanford
    Commented Dec 17, 2010 at 2:41
  • 1
    I don't need protected tweets, and no the Library of Congress isn't archiving them.
    – Julien
    Commented Dec 17, 2010 at 10:32

3 Answers 3


I'm not sure it's possible to get ALL tweets without providing a search term. That would be a hell of a lot of data - even just for this year.

There are a list of services here however all them require you to provide a search term.

Your best bet may be Archivist as they actually allow you to download the source code of the site. You may be able to customise this to do what you want.

If all that fails, your only other option would be to contact Twitter directly and see if they can provide you with a years worth of data. I wouldn't hold out much hope for that though!

Good luck!

  • 1
    Except: "Can I export an archive and get the tweets in the archive? No. The Twitter Terms of Service (dev.twitter.com/pages/api_terms) does not allow this." This may cause problems to the OP.
    – msanford
    Commented Dec 17, 2010 at 2:30
  • Thanks, the list you provided has good options for what I'd like to do.
    – Julien
    Commented Dec 20, 2010 at 12:47

There are two places you may want to start from. They may not provide what you want, but at least you will be much more familiar with the issues.

  1. Gnip, which has a special agreement with Twitter to resell their twits in bulk
  2. DataSift that are providing an SQL-like layer on top of Twitter stream
  1. Open gMail
  2. Compose New Message
  3. To:[email protected]
  4. Subject: A copy of your database please?
  5. Content: Mr. Stone, would you please email me a copy of your twitter database. Regards,
  6. Voila!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.