5

To archive a webpage in the Internet Archive's Wayback Machine, I usually do:

wget --spider 'https://web.archive.org/save/https://example.com'

Is there a similar method that I can use to archive web pages to archive.today?

Improve this question
1

2 Answers 2

Reset to default
1

I've analyzed the request of manually saving a file (Firefox' developer tools have a handy 'Copy as cURL' function for this - see the bottom of the post for the actual request). It includes a lot of fluff (user agent, cookies, origin, etc.) which can be omitted, and escaping the slashes in the URL also isn't necessary. Simply executing

curl -v 'https://archive.vn/submit/' \
  --data-raw 'url=https://webapps.stackexchange.com/users/218839/flux'

is already sufficient to archive your profile page. Initially, the response was some HTML containing a 'work in progress' link: https://archive.vn/wip/dk2xB which you can use to monitor the progress and/or as a final link.

<html><body><script>setInterval(function(){document.location.replace("https://archive.vn/wip/dk2xB")},1000)</script><div>
      <img width="48" height="48" style="vertical-align:middle" src="https://archive.vn/loading.gif"/>
      <span style="vertical-align:middle;font-size:48px;padding-left:5px">Loading</span>
      <hr/>
    </div></body></html>

Now that I try it again, a couple of hours later, I don't get HTML as response but a HTTP 302 (Found) with the final URL in the Location header: https://archive.vn/dk2xB.

This is how the archived page looks like:

enter image description here

The original cURL request is

curl 'https://archive.vn/submit/'\
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:81.0) Gecko/20100101 Firefox/81.0'\
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'\
  -H 'Accept-Language: en-US,en;q=0.5'\
  --compressed\
  -H 'Content-Type: application/x-www-form-urlencoded'\
  -H 'Origin: https://archive.vn'\
  -H 'Connection: keep-alive'\
  -H 'Referer: https://archive.vn/'\
  -H 'Cookie: _ga=GA1.2.661111166.1603535444'\
  -H 'Upgrade-Insecure-Requests: 1'\
  -H 'TE: Trailers'\
  --data-raw 'submitid=1Z%2FjKja%2BtkGo%2BmykS2%2BrMYgTje4YZV9xk8OIlwY4NT2mLExajP7ZRmnTbJku2aMX&url=https%3A%2F%2Fwebapps.stackexchange.com%2Fquestions%2F148066%2Fhow-do-i-archive-a-webpage-to-archive-today-using-wget-or-curl'
Improve this answer
5
  • 1
    Unfortunately it doesn't work anymore since now it asks for a captcha
    – arturn
    Commented Apr 14, 2021 at 9:03
  • @arturn it still works for me: I just tried your profile and it worked: archive.vn/yLgzH
    – Glorfindel
    Commented Apr 14, 2021 at 10:24
  • Please could you give-me more details? I've tried some times but it seems it doesn't work.. I'm executing the following command "curl -v 'archive.vn/submit' --data-raw 'url=webapps.stackexchange.com/users/270171/arturn'". Thanks!
    – arturn
    Commented Apr 16, 2021 at 11:19
  • @arturn I guess it depends on your location or something ... You could try opening a question on Super User (as it's not really about using a web application through a browser anymore) and include full verbose output of your command.
    – Glorfindel
    Commented Apr 16, 2021 at 11:28
  • @arturn BTW I asked and archive.is said there's supported API for it, so there's no reliable way to do it: webapps.stackexchange.com/a/163494/51862 Any workaround could break at any time in the future as the increase CAPTCHA security as needed.
    – Ciro Santilli OurBigBook.com
    Commented Mar 16, 2022 at 22:19
0

https://blog.archive.today/post/678411898279067648/hello-i-am-developing-an-application-that says that there is no officially supported way. That is likely an edited version of a question I asked around that time on their Ask me form. The edited question:

Hello. I am developing an application that programmatically loads archived webpages from your service, but the captcha has destroyed this ability.

The answer:

No, I can’t afford automated saving. The current hardware can barely handle manual. Of course, it is possible to create a paid API service to buy new servers. But… the current crisis of supply, payment, and trust has shown that limiting growth was the right thing to do. If the archive had gone that way, it would have to be shut down now.

Another thing that seems to almost work is a GET request to:

https://archive.ph?run=1&url=URL

This is mentioned at: https://en.wikipedia.org/wiki/Help:Using_archive.today

However, when I've tried it now as of March 2022, it was always giving me a Cloudfare captcha, so it didn't really work. Visiting that same URL on the browser does work however, so they must be using some very weak bot protection.

E.g. to archive https://stackoverflow.com/questions/71483165/deadlock-occuring-after-multiple-iterations-of-queries-in-threads-multithreadin I tried:

curl -v 'https://archive.ph?run=1&url=https://stackoverflow.com/questions/71483165/deadlock-occuring-after-multiple-iterations-of-queries-in-threads-multithreadin'

but redirecting the output to a file and viewing it we see that it is a captcha page.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.