Tell me more ×
Web Applications Stack Exchange is a question and answer site for power users of web applications. It's 100% free, no registration required.
up vote 2 down vote favorite

It might look a bit crazy, but I'd like to do some stats on all tweets from the beginning.

I heard a team of scientists got them by asking Twitter directly, but I wondered if it was possible to get this data in another way.

At the end I will filter them, so I could also search in an archive file, but I need data fron a large time scale. Actually I don't need them all, the last year would be good (the more, the better).

This question Is there an app with a historical archive of twitter messages for a specific hashtag? suggests searching on Google, but I don't think I can "crawl" Google, so I'm not sure it's possible to fetch the data, or is it ? Plus I would like the tweets not to be added twice, and I'd like to get all retweets as well, which might not be included in Google search.

Another source could be the Library of Congress which will archive twitter history, but I've looked for this data, but didn't find anything, apparently they don't have it yet.

So do you guys know another source which keeps track of all tweets, or how I could use one of the above ? I'd appreciate any suggestion.

share|improve this question
2  
As far as I know, the Library of Congress isn't even archiving protected tweets (i.e., those from users with a non-public twitter feed). I doubt Twitter will open those up to you. – Firefeather Dec 16 '10 at 19:04
You would need many hundreds of terabytes of local storage to put all those tweets in, and then in which to create your relational database to store filtered data… – msanford Dec 17 '10 at 2:41
I don't need protected tweets, and no the Library of Congress isn't archiving them. – Julien Dec 17 '10 at 10:32

3 Answers

up vote 2 down vote accepted

I'm not sure it's possible to get ALL tweets without providing a search term. That would be a hell of a lot of data - even just for this year.

There are a list of services here however all them require you to provide a search term.

Your best bet may be Archivist as they actually allow you to download the source code of the site. You may be able to customise this to do what you want.

If all that fails, your only other option would be to contact Twitter directly and see if they can provide you with a years worth of data. I wouldn't hold out much hope for that though!

Good luck!

share|improve this answer
1  
Except: "Can I export an archive and get the tweets in the archive? No. The Twitter Terms of Service (dev.twitter.com/pages/api_terms) does not allow this." This may cause problems to the OP. – msanford Dec 17 '10 at 2:30
Thanks, the list you provided has good options for what I'd like to do. – Julien Dec 20 '10 at 12:47
up vote 4 down vote

There are two places you may want to start from. They may not provide what you want, but at least you will be much more familiar with the issues.

  1. Gnip, which has a special agreement with Twitter to resell their twits in bulk
  2. DataSift that are providing an SQL-like layer on top of Twitter stream
share|improve this answer
up vote 1 down vote
  1. Open gMail
  2. Compose New Message
  3. To:biz.stone@twitter.com
  4. Subject: A copy of your database please?
  5. Content: Mr. Stone, would you please email me a copy of your twitter database. Regards,
  6. Voila!
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.