34

To save a single page via Wayback Machine I can go to:

How can I archive the website recursively by Wayback Machine?

There is some project such as wayback-machine-downloader, but I'm looking for a feature which allows me the upload the website recursively.

Improve this question
2
  • Are you having any difficulty using the URL technique you mention in your question? I've had it work hundreds of times, but I've also had it fail many times too. I get about an 80% success rate, even for the same pages on different dates.
    – Amazon Dies In Darkness
    Commented Jan 23, 2021 at 4:32
  • @RockPaperLz-MaskitorCasket Copy and pasting each URL one-by-one manually into a web form is fairly impractical and human error prone for savings hundreds or thousands of pages or more.
    – Taylor D. Edmiston
    Commented Apr 15, 2023 at 22:45

5 Answers 5

Reset to default
25

Since Wayback Machine doesn't provide such feature, I've found some workaround.

  1. First, mirror the website using wget, e.g.

    wget -m https://example.com/

  2. Then use curl to archive all pages one by one that you've downloaded.

    find . -name "*.html" -exec curl -v "https://web.archive.org/save/https://{}" ';'

    Note: You can change .html to .php, or include certain type of files.

Improve this answer
9
  • If the site doesn't use extensions (such as html or php - like SE is setup), how do you do adapt your command?
    – d-b
    Commented Jul 22, 2018 at 8:41
  • 4
    You can change -name "*.html" to -type f to include all files.
    – kenorb
    Commented Jul 22, 2018 at 12:55
  • How does this work with query parameters?
    – Mithical
    Commented Mar 9, 2019 at 18:29
  • It doesn't seem to work anymore: it prints 429 Too Many Requests
    – simplegamer
    Commented May 30, 2020 at 23:40
  • 3
    The --convert-links arguments can be useful as well - the former makes the links relative; this does require adding the domain URL into the "save/https:///domain" but will help when the site doesn't use absolute URLs.
    – IllustriousMagenta
    Commented Aug 1, 2020 at 9:37
14

If you wish to archive a small website, the Archive Team maintains the ArchiveBot, an IRC bot where you can request to crawl websites. The Archive Team will then submit the crawled pages to the Internet Archive's Wayback Machine.

Improve this answer
2
7

Using the Wayback Machine's Save Page Now interface, you can now also save outlinks (linked pages).

A quote from the original 2019 announcement:

You can now save all the “outlinks” of a web page with a single click. By selecting the “save outlinks” checkbox you can save the requested page and also all linked pages

The Wayback Machine's Save Page Now interface can be used without an account however extra features like saving outlinks, saving screenshots, and saving pages to your my web archive all require signing in.

Improve this answer
2
  • 1
    From where is this option available?
    – Amazon Dies In Darkness
    Commented Jan 23, 2021 at 4:33
  • 3
    I've edited the answer (hope they'll accept) that Save Page Now requires a Sign In in order to use this option (which was added back in late 2019). @kenorb please consider changing the accepted answer into this one!
    – LWC
    Commented Jan 23, 2021 at 8:28
4

The Wayback Machine doesn't offer a way to submit an entire site, only a single page as you've already found. This is touched on in a couple of points of their Wayback Machine FAQ:

Can I add pages to the Wayback Machine?

On https://archive.org/web you can use the "Save Page Now" feature to save a specific page one time. This does not currently add the URL to any future crawls nor does it save more than that one page. It does not save multiple pages, directories or entire sites.

and

How can I get my site included in the Wayback Machine?

Much of our archived web data comes from our own crawls or from Alexa Internet's crawls. Neither organization has a "crawl my site now!" submission process. Internet Archive's crawls tend to find sites that are well linked from other sites. The best way to ensure that we find your web site is to make sure it is included in online directories and that similar/related sites link to you.

Improve this answer
2
  • 1
    This is not an answer to the question. Just because there is no official way to do it, the task is not impossible to perform. In fact, it should be pretty easy to whip up a script that adds links recursively.
    – d-b
    Commented Apr 12, 2018 at 13:04
  • 1
    @d-b, kenorb's answer appears to be what you were asking for. Incidentally, this answer is more useful to me in the moment, as I just wanted The WaybackMachine to capture one page for me now.
    – cp.engr
    Commented May 6, 2018 at 17:21
1

This article on archive.org also suggests a paid service which will do the crawling for you as often as you'd like:

  1. Sign up for an Archive-It Account

Archive-It is a subscription service provided by Internet Archive that allows you to run your own crawling projects without any technical expertise. Tell us what to crawl and how often to crawl it, and we execute the crawl and put the results in the Wayback Machine.

This is probably not what you're after, but for some businesses this service might be useful. I assume it helps to fund archive.org, which otherwise is free.

Improve this answer
4
  • I can login to archive-it, but I can not sign up to archive-it, suggesting this service is no longer available.
    – MastaBaba
    Commented Jan 5, 2022 at 10:53
  • Hmm, perhaps it's only open to partners? They advertise the services here. Try to contact the support perhaps? I'm not a partner/user of that service so only shared the info I found online.
    – stason
    Commented Jan 5, 2022 at 23:08
  • 1
    Possible. I put in a request on the ArchiveBot (mentioned in this answer, above, webapps.stackexchange.com/a/129886/29284), and that seems to have done the trick, too.
    – MastaBaba
    Commented Jan 5, 2022 at 23:33
  • There is no self-registration availability. It seems they must create the accounts internally after you fill out the contact form and subscribe to a paid plan.
    – Taylor D. Edmiston
    Commented Apr 15, 2023 at 22:53

Not the answer you're looking for? Browse other questions tagged or ask your own question.