I need to download all Gutenberg ebooks, in plain text format (not html) and only in English language.
Anyone has suggestions how to download them all from the Gutenberg server?
I need them to make a linguistic research.
Web Applications Stack Exchange is a question and answer site for power users of web applications. It only takes a minute to sign up.
Sign up to join this communityI need to download all Gutenberg ebooks, in plain text format (not html) and only in English language.
Anyone has suggestions how to download them all from the Gutenberg server?
I need them to make a linguistic research.
According to Information About Robot Access to our Pages:
Robot access to our site should be left as last resource, when everything else has failed. Also, remember that the Project Gutenberg web site is copyrighted.
However, there is hope:
Better Alternatives
- Get an offline version of the Project Gutenberg web site.
- Get all Project Gutenberg ebook files.
- Get the Project Gutenberg catalog data.
And:
[...] You can get all our eBooks in zipped files by pointing your robot at http://www.gutenberg.org/robot/harvest
[...] Unpacking the zip files will produce another 70,000 files.
This is an example of how to get all files using
wget
:wget -w 2 -m http://www.gutenberg.org/robot/harvest
[...] If you want only some types of files say:
wget -w 2 -m http://www.gutenberg.org/robot/harvest?filetypes[]=txt
[...] If you want only files in a given language say:
wget -w 2 -m http://www.gutenberg.org/robot/harvest?langs[]=de
So, I'd quess:
wget -w 2 -m http://www.gutenberg.org/robot/harvest?filetypes[]=txt&langs[]=en
--level=0
. But I guess you better allow to abort and restart: try --level 9999 --no-clobber
, which will skip files you already have (assuming you're still in the same folder on disk).
-c
option, but still. I gave offset=xxx
in the URL to be mirrored but still it is downloading from first page.
Sep 22, 2013 at 7:07
While the selected answer is correct, it will potentially cause two problems:
wget
command will fail it's recursive checks on downloading the files from an external mirror.The below solution fixes these problems:
wget -H -w 2 -m http://www.gutenberg.org/robot/harvest?filetypes[]=txt&langs[]=en \
--referer="http://www.google.com" \
--user-agent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6" \
--header="Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5" \
--header="Accept-Language: en-us,en;q=0.5" \
--header="Accept-Encoding: gzip,deflate" \
--header="Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7" \
--header="Keep-Alive: 300"
You may want to change the referer and user-agent strings to provide a bit of randomness.
You can download the entire Gutenberg collection of English books and of other languages in a single ZIM file, which is highly compressed and can then be opened with Kiwix both on desktop and Android. The English books are 40 GB.
Command line ZIM tools for Linux can be downloaded from:
ftp://mirrors.pglaf.org/mirrors/gutenberg-iso has a couple of good options.
ftp://mirrors.pglaf.org/mirrors/gutenberg-iso/pgdvd042010.iso is an 8GB file that should be sufficient for your needs.
There is more information here:
https://www.gutenberg.org/wiki/Gutenberg:The_CD_and_DVD_Project#Downloading_Via_FTP, it gives all the options of downloading the archive, including FTP and BitTorrent.
Another option is the great tool at http://pgiso.pglaf.org/.
I wrote a script which will download all (not just English) the ePub books files from the PG FTP server. It will take about 12 hours as the server connection is slow, and it'll be a ton of data. You can find it here: https://gist.github.com/mxplusb/78363da5fe1e68a4e89d9fa8c720e034
For access to all the plain text English books, I setup a private mirror as described in Mirroring How-To. The ftp.ibiblio.org mirror worked with rsync:
rsync -av --del ftp.ibiblio.org::gutenberg /var/www/gutenberg
The recommended wget solution at Information About Robot Access to our Pages did not work for me.
Why not use all your over-complicated and shown above programming skills and knowledge to make a simple button that links all those actions and says "Download All Current Books" -with a language option tab when you click it.
I am sure most of the users that come to the site are e-book collectors, and manual downloading certain books on topics that interested them is OK for 1 or 2 books. But making a bigger collection manually is a drag. Nevertheless if they need it for research or would simply like to own a huge digital library of books at their own PC. Most people are turned off and scatter away from the site when they realize that they have to be a Computer Wizard to do this.So a "Download All Current Books" button would benefit the site the project and its users, and it will certainly produce even more visitors to the site. So that way everyone is happy.