20

To save a single page via Wayback Machine I can go to:

How can I archive the website recursively by Wayback Machine?


There is some project such as wayback-machine-downloader, but I'm looking for a feature which allows me the upload the website recursively.

  • Are you having any difficulty using the URL technique you mention in your question? I've had it work hundreds of times, but I've also had it fail many times too. I get about an 80% success rate, even for the same pages on different dates. – RockPaperLz- Mask it or Casket 2 days ago
17

Since Wayback Machine doesn't provide such feature, I've found some workaround.

  1. First, mirror the website using wget, e.g.

    wget -m https://example.com/
    
  2. Then use curl to archive all pages one by one that you've downloaded.

    find . -name "*.html" -exec curl -v "https://web.archive.org/save/https://{}" ';'
    

    Note: You can change .html to .php, or include certain type of files.

  • If the site doesn't use extensions (such as html or php - like SE is setup), how do you do adapt your command? – d-b Jul 22 '18 at 8:41
  • 2
    You can change -name "*.html" to -type f to include all files. – kenorb Jul 22 '18 at 12:55
  • How does this work with query parameters? – Mithical Mar 9 '19 at 18:29
  • It doesn't seem to work anymore: it prints 429 Too Many Requests – simplegamer May 30 '20 at 23:40
  • 1
    The --convert-links arguments can be useful as well - the former makes the links relative; this does require adding the domain URL into the "save/https:///domain" but will help when the site doesn't use absolute URLs. – Modelmat Aug 1 '20 at 9:37
7

If you wish to archive a small website, the Archive Team maintains the ArchiveBot, an IRC bot where you can request to crawl websites. The Archive Team will then submit the crawled pages to the Internet Archive's Wayback Machine.

  • This is incredibly helpful. – Guy Oct 20 '19 at 19:39
2

The Wayback Machine doesn't offer a way to submit an entire site, only a single page as you've already found. This is touched on in a couple of points of their Wayback Machine FAQ:

Can I add pages to the Wayback Machine?

On https://archive.org/web you can use the "Save Page Now" feature to save a specific page one time. This does not currently add the URL to any future crawls nor does it save more than that one page. It does not save multiple pages, directories or entire sites.

and

How can I get my site included in the Wayback Machine?

Much of our archived web data comes from our own crawls or from Alexa Internet's crawls. Neither organization has a "crawl my site now!" submission process. Internet Archive's crawls tend to find sites that are well linked from other sites. The best way to ensure that we find your web site is to make sure it is included in online directories and that similar/related sites link to you.

  • 1
    This is not an answer to the question. Just because there is no official way to do it, the task is not impossible to perform. In fact, it should be pretty easy to whip up a script that adds links recursively. – d-b Apr 12 '18 at 13:04
  • @d-b, kenorb's answer appears to be what you were asking for. Incidentally, this answer is more useful to me in the moment, as I just wanted The WaybackMachine to capture one page for me now. – cp.engr May 6 '18 at 17:21
2

It's now possible to archive a page and all the linked pages. Depending on the setup of your website it can do the trick

A quote from the announcement:

You can now save all the “outlinks” of a web page with a single click. By selecting the “save outlinks” checkbox you can save the requested page and also all linked pages

New contributor
iaeiou is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
  • From where is this option available? – RockPaperLz- Mask it or Casket 2 days ago
  • 1
    I've edited the answer (hope they'll accept) that Save Page Now requires a Sign In in order to use this option (which was added back in late 2019). @kenorb please consider changing the accepted answer into this one! – LWC 2 days ago
1

This article on archive.org also suggests a paid service which will do the crawling for you as often as you'd like:

  1. Sign up for an Archive-It Account

Archive-It is a subscription service provided by Internet Archive that allows you to run your own crawling projects without any technical expertise. Tell us what to crawl and how often to crawl it, and we execute the crawl and put the results in the Wayback Machine.

This is probably not what you're after, but for some businesses this service might be useful. I assume it helps to fund archive.org, which otherwise is free.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.