4

I used Github search which is good though sometimes I like to use Google's parameters for better search handling of what I want.

site:github.com foreach

Unfortunately for the first few pages, I am just finding repositories and not the actual code.

2 Answers 2

1

If the search engine crawlers honour the robots.txt then no, you won't be able to use external search engines to search for code hosted on GitHub.

Here's the line from their robots.txt file disallowing that:

Disallow: /*/*/blob/*

If you go jumping about repos you'll see the path to the source code is something like this:

https://github.com/asciidoctor/asciidoctor/blob/master/lib/asciidoctor.rb

This matches the rule and thus, you won't get anything back when using compliant search engines.

4

The accepted answer (as of 18 Sep 2013) is incorrect. GitHub's robots.txt does include a rule disallowing /*/*/blob/*. However, it also includes a rule allowing /*/*/blob/master. Contents of the master branch will be indexed by Google.

User-agent: Googlebot
Allow: /*/*/tree/master
Allow: /*/*/blob/master
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /*/*/pulse
Disallow: /*/*/tree/*
Disallow: /*/*/blob/*

Google follows the rule for the most specific rule, so /*/*/blob/master will be allowed. See "Order of precedence for group-member records" on their Robots.txt Specifications page.

If you want to keep your content hidden from Google, the best option is to use a private repository, so nobody can see it. Second best would be to use a branch that is not master.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.