1

Over on Stack Overflow I asked a question regarding accessing Google Spreadsheets with CURL, which doesn't seem to be possible any more, even when setting a valid user agent string. My question arising from that finding is the following:

How does Google know that my request sent with CURL via a valid user agent string does not originate from a browser?

Improve this question
2
  • @pnuts: I am asking a slightly different questions over at stackoverflow. There I am interested in using google's API. Here and with this question I am curious as to how google can distinguish between a CURL request and a request performed by a human being, when the user agent strings sent by both methods are the same. – DudeOnRock Nov 3 '14 at 23:29
  • I have not been able to confirm that the answer provided at stackoverflow actually solves my problem, which is why I haven't marked it as answered. Once I go back to that project from April, I will try that approach. Disregarding whether or not that answer solves that problem, when I was working on that project I was able to access the spreadsheet through the browser and not with CURL, using the same link, which suggested to me that google can distinguish between humans and CURL requests. – DudeOnRock Nov 3 '14 at 23:38
4

Without asking Google, there's really no way to know for sure. However, aside from the user agent, they could be looking for certain commonly used HTTP headers that are routinely sent by web browsers but normally omitted in curl requests. This may include, but not be limited to:

  • cookies - saved cookies, which may include session or login credentials
  • referer - the page you came from
  • accept - the type of content you can handle
  • accept-language - the language(s) you can read
  • accept-encoding - the data encodings you support
  • caching-related headers
  • security-related headers
  • others...

These headers are optional in most circumstances, but sending them helps a server give the best possible response, so browsers normally send them. A clever site could use the lack of common headers as a way to tell the difference between a client using a normal browser and using a downloader like curl. Especially a site like Google, which gets enough traffic to have a pretty good idea of what kind of behavior to expect from most popular browsers.

Having said that, I personally doubt Google Sheets is trying to block curl; rather, there's probably a necessary header that is being left out on the curl request. You can always use the --header option of curl to add any necessary headers to the curl request in order to better mimic real-world browser behavior.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.