Take the 2-minute tour ×
Web Applications Stack Exchange is a question and answer site for power users of web applications. It's 100% free, no registration required.

Google claims to be fair, and it is in the company's interest (most of the time) to scour the Internet for anything and everything that its spiders can access. I want to know:

  • What type of (publicly accessible) content does Google fail to deliver?
  • Is there a specific type of content that Google cannot retrieve?

References, especially to Google's own documentation, would be especially awesome.

share|improve this question

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

    
Now that the EU has enacted a "Right to be Forgotten," there's quite a bit we will no longer be able to find. Here are some things that have been "forgotten." –  SamtheBrand Jul 15 at 16:05
add comment

7 Answers 7

A few ideas on the type of things:

  1. Content explicitly disallowed by a domain's robots.txt file is excluded from the Google index.
  2. Websites that are not linked from other websites that Google already knows. That is, there are probably a lot of websites that do not get linked from visible pages, those websites are never going to be found by the Google spider unless they're manually submitted to Google via the Webmaster Tools.
  3. Websites that are behind web forms that you need to fill out.
  4. Census images. Since the content are images that are often manually index, they usually found on paid-for sites like ancestry.com.

Learn more about the Deep Web

share|improve this answer
2  
Point 2 it’s not true. You can submit a site to Google via the Webmaster Tools, and it will get indexed even if it’s not linked from other web sites. –  Alex Apr 18 '12 at 5:09
    
That is true. Good clarification. –  amh Apr 18 '12 at 5:16
3  
Actually I will disagree with the second point. I had a test web server on my PC and it was indexed. I found this by checking the access log. –  Bakudan May 17 '12 at 18:22
2  
I've registered new domains, started developing them and found Googlebot crawling them just a few days later. I thought this was sort of creepy until I realized - I might not be the first person to own this domain :) They don't disclose specifics, but I'm sure they keep track of domains people thought were important enough to register and at least a few visited at some point, at least for some period of time. –  Tim Post Apr 8 '13 at 20:43
add comment

Aside from Twitter, Google does not index Tumblr all that well. Blog posts on Tumblr are easier to find using Tumblr search. Also everything on Google Sites isn't (or is hardly) indexed. If you start a Google site, get your own domain.

Smaller blogs that aren't regularly updated are often dumped from search results. Plus anything that they think is a splog.

share|improve this answer
add comment

Well, most of the Twitter content is not indexed by Google, even if it’s public. It used to be available to Google, but that’s no longer the case since their agreement expired.

Source.

share|improve this answer
    
Whilst true, the question asks what "high-quality content" isn't avaliable in Google. Most of Twitter wouldn't meet that criteria :) I do see the odd tweet show up in Google search though. –  HappyTimeGopher Apr 25 '12 at 13:01
add comment

It depends in which country you are. In Germany it does not show thousands of sites that the government thinks are not good for you, and the list increases by the thousands every year.

Google is the motor of Internet censorship. If you want a free Internet, use some non-evil companies, like DuckDuckGo or others.

share|improve this answer
add comment

Google removes search results deemed to infringe intellectual property rights following DMCA take-down and similar requests. See Google's search result removal request form (it may have an additional URL btw).

share|improve this answer
add comment

You cannot search for a keyword with special characters in Google Search:

Generally, punctuation is ignored, including @#$%^&*()=+[]\ and other special characters

This is especially annoying when Google some code.

share|improve this answer
add comment

Sites with so much content that google simply hasn't had time (or the inclination) to index it all.

Sites that don't have a crawlable site map and require google to provide search terms to access the results available on the site might not be fully indexed.

share|improve this answer
add comment

protected by jonsca Apr 25 at 6:27

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.