7

Is there any way to download a snapshot of an entire website from the Wayback Machine?

I am aware that we can use wget to recover old website off Wayback Machine (e.g. wget -np -e robots=off --mirror --domains=staticweb.archive.org,web.archive.org http://web.archive.org/web/19970708161549/http://www.slackworks.com/) but I wonder whether it's possible to do so directly from the Wayback Machine website.

  • You can always do the good ol' File -> Save – Vidar S. Ramdal Nov 11 '14 at 9:08
  • Sounds a bit tedious if you has to be done for each page of the website :-) – Franck Dernoncourt Nov 11 '14 at 16:11
4

It is not possible to download a snapshot from the Wayback Machine directly as it is not allowed by their Terms of Use.

From their FAQs:

Can people download sites from the Wayback?

Our terms of use specify that users of the Wayback Machine are not to copy data from the collection.

  • That doesn't mean that people don't want to do so and aren't going to do so. – Micheal Johnson Oct 16 '15 at 14:30
  • It may be worth noting that this section was removed from their FAQ and was replaced with this one: archive.org/about/faqs.php#26 . Quite simply put, the Wayback Machine doesn't offer copies of their own snapshots. Their language seems to be a bit vague though on whether other tools that download from the wayback are allowed or not. – noirscape Jul 10 '17 at 18:37
7

I've came across the same issue and I've ended up coding a Ruby gem.

To Install:

$ gem install wayback_machine_downloader

Then, run wayback_machine_downloader with the base url of the website you want to retrieve as a parameter (e.g, example.com):

$ wayback_machine_downloader http://example.com

More information: https://github.com/hartator/wayback-machine-downloader

1

There is a service that can download from web archive http://waybackdownloader.com

  • 2
    Please disclose whether you are affiliated with the site. – jonsca Jan 20 '15 at 0:59
  • 1
    It's one thing helping people write a script to download a website from the Wayback Machine, it's another thing charging for a service that goes against the terms of use. – Micheal Johnson Oct 16 '15 at 14:35
1

I made script for downloading whole site:

waybackmachine.sh
#!/usr/bin/env bash
# Wayback machine downloader
#TODO: Remove redundancy (download only newest files in given time period - not all of them and then write over them)
############################
clear

#Enter domain without http:// and www.
domain="google.com"
#Set matchType to "prefix" if you have multiple subdomains, or "exact" if you want only one page 
matchType="domain"

#Set datefilter to 1 if you want to download data from specific time period
datefilter=0
from="19700101120001" #yyyyMMddhhmmss
to="20000101120001" #yyyyMMddhhmmss

#Set this to 1 if your page has lots of captured pages with ? in url (experimental)
swapurlarguments=0
usersign='&' #sign to replace ? with

##############################################################
# Do not edit after this point
##############################################################
#Getting snapshot list
full="http://web.archive.org/cdx/search/cdx?url="
full+="$domain"
full+="&matchType=$matchType"
    if [ $datefilter = 1 ]
        then
            full+="&from=$from&to=$to"
        fi
full+="&output=json&fl=timestamp,original&fastLatest=true&filter=statuscode:200&collapse=original"  #Form request url

wget $full -O rawlist.json #Get snapshot list to file rawlist.json


#Do parsing and downloading stuff
sed 's/\"//g' rawlist.json  > list.json #Remove " from file for easier processing
rm rawlist.json #Remove unnecessary file
i=0; #Set file counter to 0
numoflines=$(cat list.json | wc -l ) #Fill numoflines with number of files to download
while read line;do # For every file
        rawcurrent="${line:1:${#line}-3}" #Remove brackets from JSON line
    IFS=', ' read -a current <<< "$rawcurrent" #Separate timestamp and url
    timestamp="${current[0]}"
    originalurl="${current[1]}"
    waybackurl="http://web.archive.org/web/$timestamp" 
    waybackurl+="id_/$originalurl" #Form request url
    file_path="$domain/"
    sufix="$(echo $originalurl | grep / | cut -d/ -f2- | cut -d/ -f3-)"
     [[ $sufix = "" ]] && file_path+="index.html" || file_path+="$sufix" #Determine local filename
clear
echo " $i out of $numoflines" #Show progress
echo "$file_path"
mkdir -p -- "${file_path%/*}" && touch -- "$file_path" #Make local file for data to be written
    wget -N $waybackurl -O $file_path #Download actual file
    ((i++))
done < list.json

#If user chose, replace ? with usersign
    if [ $swapurlarguments = 1 ]
        then
            cd $domain
            for i in *; do mv "$i" "`echo $i | sed "s/\?/\$usersign/g"`"; done #Replace ? in filenames with usersign
            find ./ -type f -exec sed -i "s/\?/\$usersign/g" {} \; #Replace ? in files with usersign
        fi
  • 1
    Better to add important contents of the link in answer . – serenesat Sep 17 '15 at 19:45
  • I believe that whole script is the answer – lazanet Oct 7 '15 at 10:44
-1

Maybe you can try a wayback machine downloader like http://www.waybackmachinedownloader.com or http://www.waybackmachinedownloads.com

Both of those 2 websites have free demo to scrape few pages from the archive URL to be sure that the script works.

There is other tools but I didn't test them so I can't give a feedback.

  • Please add essential contents of the link into answer to make it more useful, as link only answer will become useless if link has changed. – serenesat Oct 13 '17 at 20:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.