11

While using the W3C link checker, I found out that my Github Wiki can't be crawled:

https://github.com/aegif/CmisSync/wiki/Getting-started-with-CmisSync-development
Status: (N/A) Forbidden by robots.txt

This is unfortunate, as I would like people to easily find this Wiki on search engines.

QUESTION: How can I make my Github Wiki crawlable by search engines?
Or am I mistaken and Github's robots.txt is actually OK?

1

3 Answers 3

10

The GitHub robots.txt does explicitly disallow crawling of the wiki pages, for example in the Googlebot section:

User-agent: Googlebot
Allow: /*/*/tree/master
Allow: /*/*/blob/master
...
Disallow: /*/*/wiki/*/*

As this is the site-wide robots file, there isn't any getting around it.

It is an interesting choice, since GitHub describes wikis as a place to "share long-form content about your project". Since by default public wikis are editable by any user, perhaps it is a heavy handed protection from spammers.

0

GitHub wikis are searchable by engines that support it. See the first two lines of https://github.com/robots.txt :

# If you would like to crawl GitHub contact us at [email protected].
# We also provide an extensive API: https://developer.github.com/

That's probably for parsing various wiki formats etc.

For example search in Google for "openrefine broker protocol" and the first hit is a page under a Github project wiki.

1
  • 1
    Hmm, so why is Google indexing the Broker-Protocol wiki, and not the OP's wiki? Apr 7, 2015 at 12:20
0

I produce a service called https://github-wiki-see.page that tries to re-expose GitHub Wiki content in a manner that search engines can index them and give users who come across the exposed content a large button to get to the content they want. It's quite a hack. I got inspired by sites "stealing" Stack Overflow content and reposting it with ads.

I have a sitemap generator that scans GitHub events for new wiki content and generates sitemaps to hint search engines. The sitemaps were also initially seeded with a through query of GH Archive. This isn't perfect and search engines may not and do not thoroughly crawl this. As a case study, your wiki actually doesn't show up in Google's search console as being indexed on my end but it is still in the sitemaps; I'll probably keep monitoring the situation there. If you want, you can throw a link somewhere on GitHub or anywhere else search engines can actually index linking to my service's view of your wiki and that will also help get it indexed. This is the part of the answer where you can use my service to get it indexed.

As for GitHub itself, unfortunately, as of 2021, GitHub Wiki content is still uncrawlable directly on GitHub. The current official suggestion is to use GitHub Pages and my addon to that is to add an "Edit on GitHub" link to the content to make it a bit more Wiki-like. Maybe with GitHub Actions being available nowadays you can do this sort of mirroring yourself. This is the part of the answer where you can make something DIY and get it indexed without my service.

Additionally, GitHub no longer blocks by robots.txt. They block GitHub Wiki content from being indexed with headers now. My guess is that their robots.txt blocking was a little leaky and letting Wiki URLs to be indexed and maybe more optimistically, they may allow Wikis to selectively indexed in the future.

GitHub has also clarified why they block GitHub wiki content. Apparently it has something to do with spammy wiki edits causing lower rankings or something. You can find more information and stance straight from GitHub staff here: https://github.com/github/feedback/discussions/4992. The good news is that they are aware of the bad situation of search engines not indexing "the great content on GitHub wikis" but the bad news is that they are non-commital to solving the issue.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.