18

Google claims to be fair, and it is in the company's interest (most of the time) to scour the Internet for anything and everything that its spiders can access. I want to know:

  • What type of (publicly accessible) content does Google fail to deliver?
  • Is there a specific type of content that Google cannot retrieve?

References, especially to Google's own documentation, would be especially awesome.

Improve this question
1

7 Answers 7

Reset to default
21

A few ideas on the type of things:

  1. Content explicitly disallowed by a domain's robots.txt file is excluded from the Google index.
  2. Websites that are not linked from other websites that Google already knows. That is, there are probably a lot of websites that do not get linked from visible pages, those websites are never going to be found by the Google spider unless they're manually submitted to Google via the Webmaster Tools.
  3. Websites that are behind web forms that you need to fill out.
  4. Census images. Since the content are images that are often manually index, they usually found on paid-for sites like ancestry.com.

Learn more about the Deep Web

Improve this answer
4
  • 2
    Point 2 it’s not true. You can submit a site to Google via the Webmaster Tools, and it will get indexed even if it’s not linked from other web sites.
    – Alex
    Apr 18, 2012 at 5:09
  • That is true. Good clarification.
    – amh
    Apr 18, 2012 at 5:16
  • 3
    Actually I will disagree with the second point. I had a test web server on my PC and it was indexed. I found this by checking the access log.
    – Bakudan
    May 17, 2012 at 18:22
  • 2
    I've registered new domains, started developing them and found Googlebot crawling them just a few days later. I thought this was sort of creepy until I realized - I might not be the first person to own this domain :) They don't disclose specifics, but I'm sure they keep track of domains people thought were important enough to register and at least a few visited at some point, at least for some period of time.
    – Tim Post
    Apr 8, 2013 at 20:43
6

Aside from Twitter, Google does not index Tumblr all that well. Blog posts on Tumblr are easier to find using Tumblr search. Also everything on Google Sites isn't (or is hardly) indexed. If you start a Google site, get your own domain.

Smaller blogs that aren't regularly updated are often dumped from search results. Plus anything that they think is a splog.

Improve this answer
5

Well, most of the Twitter content is not indexed by Google, even if it’s public. It used to be available to Google, but that’s no longer the case since their agreement expired.

Source.

Improve this answer
1
  • Whilst true, the question asks what "high-quality content" isn't avaliable in Google. Most of Twitter wouldn't meet that criteria :) I do see the odd tweet show up in Google search though.
    – HappyTimeGopher
    Apr 25, 2012 at 13:01
4

It depends in which country you are. In Germany it does not show thousands of sites that the government thinks are not good for you, and the list increases by the thousands every year.

Google is the motor of Internet censorship. If you want a free Internet, use some non-evil companies, like DuckDuckGo or others.

Improve this answer
4

You cannot search for a keyword with special characters in Google Search:

Generally, punctuation is ignored, including @#$%^&*()=+[]\ and other special characters

This is especially annoying when Google some code.

Improve this answer
3

Google removes search results deemed to infringe intellectual property rights following DMCA take-down and similar requests. See Google's search result removal request form (it may have an additional URL btw).

Improve this answer
1

Sites with so much content that google simply hasn't had time (or the inclination) to index it all.

Sites that don't have a crawlable site map and require google to provide search terms to access the results available on the site might not be fully indexed.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.