Downloading a snapshot from the Wayback Machine?

Question

Is there any way to download a snapshot of an entire website from the Wayback Machine?

I am aware that we can use wget to recover old website off Wayback Machine (e.g. wget -np -e robots=off --mirror --domains=staticweb.archive.org,web.archive.org http://web.archive.org/web/19970708161549/http://www.slackworks.com/) but I wonder whether it's possible to do so directly from the Wayback Machine website.

Answer 1

It is not possible to download a snapshot from the Wayback Machine directly as it is not allowed by their Terms of Use.

From their FAQs:

Can people download sites from the Wayback?

Our terms of use specify that users of the Wayback Machine are not to copy data from the collection.

Answer 2

I've came across the same issue and I've ended up coding a Ruby gem.

To Install:

$ gem install wayback_machine_downloader

Then, run wayback_machine_downloader with the base url of the website you want to retrieve as a parameter (e.g, example.com):

$ wayback_machine_downloader http://example.com

More information: https://github.com/hartator/wayback-machine-downloader

Answer 3

There is a service that can download from web archive http://waybackdownloader.com

Answer 4

I made script for downloading whole site:

waybackmachine.sh
#!/usr/bin/env bash
# Wayback machine downloader
#TODO: Remove redundancy (download only newest files in given time period - not all of them and then write over them)
############################
clear

#Enter domain without http:// and www.
domain="google.com"
#Set matchType to "prefix" if you have multiple subdomains, or "exact" if you want only one page 
matchType="domain"

#Set datefilter to 1 if you want to download data from specific time period
datefilter=0
from="19700101120001" #yyyyMMddhhmmss
to="20000101120001" #yyyyMMddhhmmss

#Set this to 1 if your page has lots of captured pages with ? in url (experimental)
swapurlarguments=0
usersign='&' #sign to replace ? with

##############################################################
# Do not edit after this point
##############################################################
#Getting snapshot list
full="http://web.archive.org/cdx/search/cdx?url="
full+="$domain"
full+="&matchType=$matchType"
    if [ $datefilter = 1 ]
        then
            full+="&from=$from&to=$to"
        fi
full+="&output=json&fl=timestamp,original&fastLatest=true&filter=statuscode:200&collapse=original"  #Form request url

wget $full -O rawlist.json #Get snapshot list to file rawlist.json


#Do parsing and downloading stuff
sed 's/\"//g' rawlist.json  > list.json #Remove " from file for easier processing
rm rawlist.json #Remove unnecessary file
i=0; #Set file counter to 0
numoflines=$(cat list.json | wc -l ) #Fill numoflines with number of files to download
while read line;do # For every file
        rawcurrent="${line:1:${#line}-3}" #Remove brackets from JSON line
    IFS=', ' read -a current <<< "$rawcurrent" #Separate timestamp and url
    timestamp="${current[0]}"
    originalurl="${current[1]}"
    waybackurl="http://web.archive.org/web/$timestamp" 
    waybackurl+="id_/$originalurl" #Form request url
    file_path="$domain/"
    sufix="$(echo $originalurl | grep / | cut -d/ -f2- | cut -d/ -f3-)"
     [[ $sufix = "" ]] && file_path+="index.html" || file_path+="$sufix" #Determine local filename
clear
echo " $i out of $numoflines" #Show progress
echo "$file_path"
mkdir -p -- "${file_path%/*}" && touch -- "$file_path" #Make local file for data to be written
    wget -N $waybackurl -O $file_path #Download actual file
    ((i++))
done < list.json

#If user chose, replace ? with usersign
    if [ $swapurlarguments = 1 ]
        then
            cd $domain
            for i in *; do mv "$i" "`echo $i | sed "s/\?/\$usersign/g"`"; done #Replace ? in filenames with usersign
            find ./ -type f -exec sed -i "s/\?/\$usersign/g" {} \; #Replace ? in files with usersign
        fi

Answer 5

Maybe you can try a wayback machine downloader like http://www.waybackmachinedownloader.com or http://www.waybackmachinedownloads.com

Both of those 2 websites have free demo to scrape few pages from the archive URL to be sure that the script works.

There is other tools but I didn't test them so I can't give a feedback.