20

To save a single page via Wayback Machine I can go to:

How can I archive the website recursively by Wayback Machine?

There is some project such as wayback-machine-downloader, but I'm looking for a feature which allows me the upload the website recursively.

Improve this question
17

Since Wayback Machine doesn't provide such feature, I've found some workaround.

  1. First, mirror the website using wget, e.g.

    wget -m https://example.com/

  2. Then use curl to archive all pages one by one that you've downloaded.

    find . -name "*.html" -exec curl -v "https://web.archive.org/save/https://{}" ';'

    Note: You can change .html to .php, or include certain type of files.

Improve this answer
  • If the site doesn't use extensions (such as html or php - like SE is setup), how do you do adapt your command? – d-b Jul 22 '18 at 8:41
  • 2
    You can change -name "*.html" to -type f to include all files. – kenorb Jul 22 '18 at 12:55
  • How does this work with query parameters? – Mithical Mar 9 '19 at 18:29
  • It doesn't seem to work anymore: it prints 429 Too Many Requests – simplegamer May 30 '20 at 23:40
  • 1
    The --convert-links arguments can be useful as well - the former makes the links relative; this does require adding the domain URL into the "save/https:///domain" but will help when the site doesn't use absolute URLs. – Modelmat Aug 1 '20 at 9:37
7

If you wish to archive a small website, the Archive Team maintains the ArchiveBot, an IRC bot where you can request to crawl websites. The Archive Team will then submit the crawled pages to the Internet Archive's Wayback Machine.

Improve this answer
  • This is incredibly helpful. – Guy Oct 20 '19 at 19:39
2

The Wayback Machine doesn't offer a way to submit an entire site, only a single page as you've already found. This is touched on in a couple of points of their Wayback Machine FAQ:

Can I add pages to the Wayback Machine?

On https://archive.org/web you can use the "Save Page Now" feature to save a specific page one time. This does not currently add the URL to any future crawls nor does it save more than that one page. It does not save multiple pages, directories or entire sites.

and

How can I get my site included in the Wayback Machine?

Much of our archived web data comes from our own crawls or from Alexa Internet's crawls. Neither organization has a "crawl my site now!" submission process. Internet Archive's crawls tend to find sites that are well linked from other sites. The best way to ensure that we find your web site is to make sure it is included in online directories and that similar/related sites link to you.

Improve this answer
  • 1
    This is not an answer to the question. Just because there is no official way to do it, the task is not impossible to perform. In fact, it should be pretty easy to whip up a script that adds links recursively. – d-b Apr 12 '18 at 13:04
  • @d-b, kenorb's answer appears to be what you were asking for. Incidentally, this answer is more useful to me in the moment, as I just wanted The WaybackMachine to capture one page for me now. – cp.engr May 6 '18 at 17:21
1

This article on archive.org also suggests a paid service which will do the crawling for you as often as you'd like:

  1. Sign up for an Archive-It Account

Archive-It is a subscription service provided by Internet Archive that allows you to run your own crawling projects without any technical expertise. Tell us what to crawl and how often to crawl it, and we execute the crawl and put the results in the Wayback Machine.

This is probably not what you're after, but for some businesses this service might be useful. I assume it helps to fund archive.org, which otherwise is free.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.