Web Applications Stack Exchange is a question and answer site for power users of web applications. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I need to download all Gutenberg ebooks, in plain text format (not html) and only in English language.

Anyone has suggestions how to download them all from the Gutenberg server?

I need them to make a linguistic research.

share|improve this question

migrated from superuser.com Feb 11 '11 at 14:55

This question came from our site for computer enthusiasts and power users.

up vote 18 down vote accepted

According to Information About Robot Access to our Pages:

Robot access to our site should be left as last resource, when everything else has failed. Also, remember that the Project Gutenberg web site is copyrighted.

However, there is hope:

Better Alternatives

  • Get an offline version of the Project Gutenberg web site.
  • Get all Project Gutenberg ebook files.
  • Get the Project Gutenberg catalog data.

And:

[...] You can get all our eBooks in zipped files by pointing your robot at http://www.gutenberg.org/robot/harvest

[...] Unpacking the zip files will produce another 70,000 files.

This is an example of how to get all files using wget:

wget -w 2 -m http://www.gutenberg.org/robot/harvest

[...] If you want only some types of files say:

wget -w 2 -m http://www.gutenberg.org/robot/harvest?filetypes[]=txt

[...] If you want only files in a given language say:

wget -w 2 -m http://www.gutenberg.org/robot/harvest?langs[]=de

So, I'd quess:

wget -w 2 -m http://www.gutenberg.org/robot/harvest?filetypes[]=txt&langs[]=en
share|improve this answer
    
Is there a way to tell wget to limit the number of files that it downloads while crawling (e.g. first 100 text files it encounters)? – rohanbk Feb 11 '11 at 20:38
    
Also, when we have a number of links in a text file (absolute uri, say "gutenberg.org/files/1.zip, gutenberg.org/files/2.zip";, what parameter is used to provide such a text file as a number of download links to WGET? – EugeneP Feb 12 '11 at 9:22
    
@rohanbk, you can see what will be downloaded by browsing the URL itself, like gutenberg.org/robot/harvest?filetypes[]=txt&langs[]=en This shows it's actually paginated, but the number of files per page is not constant. (Maybe based on size?) So, to NOT recurse, according to the wget manual, you might try --level=0. But I guess you better allow to abort and restart: try --level 9999 --no-clobber, which will skip files you already have (assuming you're still in the same folder on disk). – Arjan Feb 12 '11 at 13:20
    
@EugeneP, see --input-file in the manual. – Arjan Feb 12 '11 at 13:24
    
@Arjan Is there a way to specify offset at the start of download? My downloading interrupted due to some reasons and now wget has started checking files from the first page. I had used -c option, but still. I gave offset=xxx in the URL to be mirrored but still it is downloading from first page. – user13107 Sep 22 '13 at 7:07

While the selected answer is correct, it will potentially cause two problems:

  1. You may receive a 403 error denying access to the pagination under the presumption you are downloading as a bot
  2. There's the potential that you'll be sent to an external mirror, meaning the wget command will fail it's recursive checks on downloading the files from an external mirror.

The below solution fixes these problems:

wget -H -w 2 -m http://www.gutenberg.org/robot/harvest?filetypes[]=txt&langs[]=en \
--referer="http://www.google.com" \
--user-agent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6" \
--header="Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5" \
--header="Accept-Language: en-us,en;q=0.5" \
--header="Accept-Encoding: gzip,deflate" \
--header="Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7" \
--header="Keep-Alive: 300"

You may want to change the referer and user-agent strings to provide a bit of randomness.

share|improve this answer

Another option is the great tool at http://pgiso.pglaf.org/.

  1. Enter an id range (e.g. 1-10000)
  2. Select the desired file types
  3. Choose the languages you want to include
  4. Wait for the notification
  5. Download
share|improve this answer

You can download the entire Gutenberg collection of English books and of other languages in a single ZIM file, which is highly compressed and can then be opened with Kiwix both on desktop and Android. The English books are 40 GB.

share|improve this answer
    
no linux client for kiwix – aquagremlin Feb 14 at 17:24
    
@aquagremlin Uh? kiwix.org/wiki/Software#GNU.2FLinux It's even packaged in some distros. – Nemo Feb 14 at 17:28
1  
sorry. no sleep and didnt see the 'other systems' below the big button. – aquagremlin Feb 14 at 17:31

Why not use all your over-complicated and shown above programming skills and knowledge to make a simple button that links all those actions and says "Download All Current Books" -with a language option tab when you click it.

I am sure most of the users that come to the site are e-book collectors, and manual downloading certain books on topics that interested them is OK for 1 or 2 books. But making a bigger collection manually is a drag. Nevertheless if they need it for research or would simply like to own a huge digital library of books at their own PC. Most people are turned off and scatter away from the site when they realize that they have to be a Computer Wizard to do this.So a "Download All Current Books" button would benefit the site the project and its users, and it will certainly produce even more visitors to the site. So that way everyone is happy.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.