0

If there is a page, that isn't archived in the Waybackmachine how can I add this site to the archive? Or is there another service like that where you can add custom sites to be archived "forever"?

I would like to be able to proove that the site existed like I took a screenshot today.

Does this also work for https-sites?

I found this in the FAQ: My site’s not archived! How can I add it?

but it doesen't work for that page: https://github.com/ZachWick/TableCSVExport

Some other pages have stated, that

Page cannot be crawled or displayed due to robots.txt.

like http://web.archive.org/web/*/https://github.com/gilbitron/WordPress-Settings-Framework

But http://web.archive.org/liveweb/https://github.com/ZachWick/TableCSVExport just shows the live page

Improve this question
3
  • Did you read the Wayback Machine FAQ?
    – ale
    Commented Jul 1, 2013 at 18:28
  • sure, I tried to add it there, but it doesen't work. that's why I ask here and don't deserve a downvote without comment
    – rubo77
    Commented Jul 2, 2013 at 6:26
  • 1
    Do you see other similar Github pages in the Archive? github.com/robots.txt has a lot of exclusions. In fact, except for some well-known bots, they exclude everything. It's possible that Alexa's bot can't crawl Github at all.
    – ale
    Commented Jul 2, 2013 at 12:41

1 Answer 1

Reset to default
2

As their FAQ's state:

How can I get my site included in the Wayback Machine?

Much of our archived web data comes from our own crawls or from Alexa Internet's crawls. Neither organization has a "crawl my site now!" submission process. Internet Archive's crawls tend to find sites that are well linked from other sites. The best way to ensure that we find your web site is to make sure it is included in online directories and that similar/related sites link to you.

Alexa Internet uses its own methods to discover sites to crawl. It may be helpful to install the free Alexa toolbar and visit the site you want crawled to make sure they know about it.

Regardless of who is crawling the site, you should ensure that your site's 'robots.txt' rules and in-page META robots directives do not tell crawlers to avoid your site.

When a site is crawled, there is usually at least a 6-month lag, and sometimes as much as a 24-month lag, between the date that web pages are crawled and when they appear in the Wayback Machine.

In some cases, crawled content from certain projects may appear in a much shorter timeframe — as little as a few weeks from when it was crawled. Older material for the same pages and sites may still appear separately, months later.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.