How to download all English books from Gutenberg?

Question

I need to download all Gutenberg ebooks, in plain text format (not html) and only in English language.

Anyone has suggestions how to download them all from the Gutenberg server?

I need them to make a linguistic research.

Answer 1

According to Information About Robot Access to our Pages:

Robot access to our site should be left as last resource, when everything else has failed. Also, remember that the Project Gutenberg web site is copyrighted.

However, there is hope:

Better Alternatives

• Get an offline version of the Project Gutenberg web site.
• Get all Project Gutenberg ebook files.
• Get the Project Gutenberg catalog data.

And:

[...] You can get all our eBooks in zipped files by pointing your robot at http://www.gutenberg.org/robot/harvest

[...] Unpacking the zip files will produce another 70,000 files.

This is an example of how to get all files using wget:

wget -w 2 -m http://www.gutenberg.org/robot/harvest

[...] If you want only some types of files say:

wget -w 2 -m http://www.gutenberg.org/robot/harvest?filetypes[]=txt

[...] If you want only files in a given language say:

wget -w 2 -m http://www.gutenberg.org/robot/harvest?langs[]=de

So, I'd quess:

wget -w 2 -m http://www.gutenberg.org/robot/harvest?filetypes[]=txt&langs[]=en

Answer 2

While the selected answer is correct, it will potentially cause two problems:

1. You may receive a 403 error denying access to the pagination under the presumption you are downloading as a bot
2. There's the potential that you'll be sent to an external mirror, meaning the wget command will fail it's recursive checks on downloading the files from an external mirror.

The below solution fixes these problems:

wget -H -w 2 -m http://www.gutenberg.org/robot/harvest?filetypes[]=txt&langs[]=en \
--referer="http://www.google.com" \
--user-agent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6" \
--header="Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5" \
--header="Accept-Language: en-us,en;q=0.5" \
--header="Accept-Encoding: gzip,deflate" \
--header="Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7" \
--header="Keep-Alive: 300"

You may want to change the referer and user-agent strings to provide a bit of randomness.