Retrieve old hashtagged tweets on Twitter from various users

Question

I need to retrieve an archive or a list of tweets up to a year old, that were posted by various users, and included specific hashtags. A long search on Google created the notion this is impossible even with using the Twitter API. We have a programmer on the team that can create anything needed with the API, I just need a lead or a tip on whether this is impossible.

Thanks,

Answer 1

It actually depends on the amount of tweets with that hashtag. If the amount is really less, it might be possible. Twitter documentation mentions that -

Clients may access a theoretical maximum of 3,200 statuses via the page and count parameters for the user_timeline REST API methods. Other timeline methods have a theoretical maximum of 800 statuses. Requests for more than the limit will result in a reply with a status code of 200 and an empty result in the format requested. Twitter still maintains a database of all the tweets sent by a user. However, to ensure performance of the site, this artificial limit is temporarily in place.

Add to that, the search API's page parameters says -

The page number (starting at 1) to return, up to a max of roughly 1500 results (based on rpp * page).

So that would mean that Twitter doesn't intend to give back All the data it consumed. It'll hold back some. In that case, you cannot do anything, unless you find a third party service that indexes tweets.

The best bet if to try searching combining the since_id, rpp and until parameters and see how far you can go.

Answer 2

Bibhas is correct that Twitter offers limited access to historical tweets; however, there are other tools that archive the Twitter firehose for historical searching and referencing.

I have personally used Radian6 to do this. Radian6 is not cheap, but it has been a reliable tool for searching historical tweets back at least a few years. You may be able to contact them and get a free trial, or if you work for a non-profit, there are discounts.

Sysomos MAP is another company that offers the ability to search Twitter archives. I have not used this, but in my opinion it's the "#2" option.

Answer 3

This isn't possible, the search API is currently not built for searching old tweets. Twitter has made it known to its developers that they are concentrating on real time efforts.

Your developer will have to use a combination of search engines, Twitter search API (to the extent of it's limit) and Twitter clients that have attempted this before.

For example, I tried to make a localized hashtag to represent days I was sick with #sickphwdday

Twitter realtime web search sickphwday from:phwd or #sickphwdday - None
SnapBird sickphwdday - 2
Twitter CLI t search user @phwd "sickphwdday" - 2
Google site:twitter.com/phwd "sickphwday" - None
Bing Twitter sickphwdday - None
Topsy Twitter sickphwdday - None
Downloading 2000 most recent tweets and grepping (grep "sickphwdday" *.json) - 2
TweetNest http://philippeharewood.com/twitter/search?q=sickphwdday - 2

_{*Of course when I submit this post sickphwdday will show up for this site.}