Is there a tool out there that will crawl one of your friend's status updates and download them to a file? I'm doing some research and it's a pain to keep hitting "older posts.."
-
1"I'm doing some stalking" FTFY. Jokes aside, I don't know of any way to do this now that they stopped piping this stuff to RSS but maybe there's some backdoor functionality I'm unaware of.– ShinraiFeb 8, 2011 at 23:14
-
I suggest you elaborate a bit about what you're specifically trying to do and since Super User isn't about websites, it should be more general regarding crawling any site.– Ivo FlipseFeb 8, 2011 at 23:22
-
I would like to collect a friend's status updates for a period of time. Specifically, from the present back one year, I want to collect the text of the status updates from his profile.– CaveatrobFeb 8, 2011 at 23:47
-
PS - Stalking would seem to be easier done with some other method. This is data collection.– CaveatrobFeb 8, 2011 at 23:48
-
1Facebook has an official API for retrieving user data. Any other scraping would be considered a violation of their terms of use. See developers.facebook.com for more info.– nhinkleFeb 9, 2011 at 9:17
|
Show 1 more comment
2 Answers
You may want to look at the questions on StackOverflow about the Facebook Graph API. Your solution for instance, may involve something like wget and a URL of the form (or similar): https://graph.facebook.com/USERNAME_HERE/feed.
E.g.: https://graph.facebook.com/cocacola/feed
In fact by examining the bottom previous/next links from this page, I can see that adding a query similar to ?limit=25 and perhaps even &since=2011-02-09T04 may be just what you need.
Try fb-crawl.
It's a Perl script that will insert wall posts, user information, and friends into a MySQL database. From there you can sift through the data using MySQL SELECT statements. It doesn't make any API calls, so if you can see it on Facebook, fb-crawl can get it.