Web Applications Stack Exchange is a question and answer site for power users of web applications. It only takes a minute to sign up.
Sign up to join this community
While using the W3C link checker, I found out that my Github Wiki can't be crawled:
https://github.com/aegif/CmisSync/wiki/Getting-started-with-CmisSync-development
Status: (N/A) Forbidden by robots.txt
This is unfortunate, as I would like people to easily find this Wiki on search engines.
QUESTION: How can I make my Github Wiki crawlable by search engines?
Or am I mistaken and Github's robots.txt is actually OK?
The GitHub robots.txt does explicitly disallow crawling of the wiki pages, for example in the Googlebot section:
User-agent: Googlebot
Allow: /*/*/tree/master
Allow: /*/*/blob/master
...
Disallow: /*/*/wiki/*/*
As this is the site-wide robots file, there isn't any getting around it.
It is an interesting choice, since GitHub describes wikis as a place to "share long-form content about your project". Since by default public wikis are editable by any user, perhaps it is a heavy handed protection from spammers.
GitHub wikis are searchable by engines that support it. See the first two lines of https://github.com/robots.txt :
# If you would like to crawl GitHub contact us at support@github.com.
# We also provide an extensive API: https://developer.github.com/
That's probably for parsing various wiki formats etc.
For example search in Google for "openrefine broker protocol" and the first hit is a page under a Github project wiki.
