Take the 2-minute tour ×
Web Applications Stack Exchange is a question and answer site for power users of web applications. It's 100% free, no registration required.

I used Github search which is good though sometimes I like to use Google's parameters for better search handling of what I want.

site:github.com foreach

Unfortunately for the first few pages, I am just finding repositories and not the actual code.

share|improve this question

2 Answers 2

up vote 2 down vote accepted

If the search engine crawlers honour the robots.txt then no, you won't be able to use external search engines to search for code hosted on GitHub.

Here's the line from their robots.txt file disallowing that:

Disallow: /*/*/blob/*

If you go jumping about repos you'll see the path to the source code is something like this:

https://github.com/asciidoctor/asciidoctor/blob/master/lib/asciidoctor.rb

This matches the rule and thus, you won't get anything back when using compliant search engines.

share|improve this answer

The accepted answer (as of 18 Sep 2013) is incorrect. GitHub's robots.txt does include a rule disallowing /*/*/blob/*. However, it also includes a rule allowing /*/*/blob/master. Contents of the master branch will be indexed by Google.

User-agent: Googlebot
Allow: /*/*/tree/master
Allow: /*/*/blob/master
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /*/*/pulse
Disallow: /*/*/tree/*
Disallow: /*/*/blob/*

Google follows the rule for the most specific rule, so /*/*/blob/master will be allowed. See "Order of precedence for group-member records" on their Robots.txt Specifications page.

If you want to keep your content hidden from Google, the best option is to use a private repository, so nobody can see it. Second best would be to use a branch that is not master.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.