9

While using the W3C link checker, I found out that my Github Wiki can't be crawled:

https://github.com/aegif/CmisSync/wiki/Getting-started-with-CmisSync-development
Status: (N/A) Forbidden by robots.txt

This is unfortunate, as I would like people to easily find this Wiki on search engines.

QUESTION: How can I make my Github Wiki crawlable by search engines?
Or am I mistaken and Github's robots.txt is actually OK?

9

The GitHub robots.txt does explicitly disallow crawling of the wiki pages, for example in the Googlebot section:

User-agent: Googlebot
Allow: /*/*/tree/master
Allow: /*/*/blob/master
...
Disallow: /*/*/wiki/*/*

As this is the site-wide robots file, there isn't any getting around it.

It is an interesting choice, since GitHub describes wikis as a place to "share long-form content about your project". Since by default public wikis are editable by any user, perhaps it is a heavy handed protection from spammers.

0

GitHub wikis are searchable by engines that support it. See the first two lines of https://github.com/robots.txt :

# If you would like to crawl GitHub contact us at support@github.com.
# We also provide an extensive API: https://developer.github.com/

That's probably for parsing various wiki formats etc.

For example search in Google for "openrefine broker protocol" and the first hit is a page under a Github project wiki.

  • Hmm, so why is Google indexing the Broker-Protocol wiki, and not the OP's wiki? – Vidar S. Ramdal Apr 7 '15 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.