I need to download all Gutenberg ebooks, in plain text format (not html) and only in English language.
Anyone has suggestions how to download them all from the Gutenberg server?
I need them to make a linguistic research.
Add a comment
|
9 Answers
According to Information About Robot Access to our Pages:
The Project Gutenberg website is intended for human users only. Any perceived use of automated tools to access the Project Gutenberg website will result in a temporary or permanent block of your IP address. The only exceptions to this rule are below.
The page in the links above give you examples to use wget. So, I'd quess:
wget -w 2 -m -H http://www.gutenberg.org/robot/harvest?filetypes[]=txt&langs[]=en
Aside, back in February 2011 when this answer was posted, and up to October 2011, its wiki said:
Robot access to our site should be left as last resource, when everything else has failed. Also, remember that the Project Gutenberg web site is copyrighted.
[...] You can get all our eBooks in zipped files by pointing your robot at http://www.gutenberg.org/robot/harvest
[...] Unpacking the zip files will produce another 70,000 files.
This is an example of how to get all files using
wget:wget -w 2 -m http://www.gutenberg.org/robot/harvest[...] If you want only some types of files say:
wget -w 2 -m http://www.gutenberg.org/robot/harvest?filetypes[]=txt[...] If you want only files in a given language say:
wget -w 2 -m http://www.gutenberg.org/robot/harvest?langs[]=de
-
Is there a way to tell wget to limit the number of files that it downloads while crawling (e.g. first 100 text files it encounters)?– rohanbkCommented Feb 11, 2011 at 20:38
-
Also, when we have a number of links in a text file (absolute uri, say "gutenberg.org/files/1.zip, gutenberg.org/files/2.zip", what parameter is used to provide such a text file as a number of download links to WGET?– EugenePCommented Feb 12, 2011 at 9:22
-
@rohanbk, you can see what will be downloaded by browsing the URL itself, like gutenberg.org/robot/harvest?filetypes[]=txt&langs[]=en This shows it's actually paginated, but the number of files per page is not constant. (Maybe based on size?) So, to NOT recurse, according to the wget manual, you might try
--level=0. But I guess you better allow to abort and restart: try--level 9999 --no-clobber, which will skip files you already have (assuming you're still in the same folder on disk).– ArjanCommented Feb 12, 2011 at 13:20 -
1
-
@Arjan Is there a way to specify offset at the start of download? My downloading interrupted due to some reasons and now wget has started checking files from the first page. I had used
-coption, but still. I gaveoffset=xxxin the URL to be mirrored but still it is downloading from first page. Commented Sep 22, 2013 at 7:07
You can download the entire Gutenberg collection of English books and of other languages in a single ZIM file, which is highly compressed and can then be opened with Kiwix both on desktop and Android. The English books are 40 GB.
ZIM tools
Command line ZIM tools for Linux can be downloaded from:
-
@aquagremlin Uh? kiwix.org/wiki/Software#GNU.2FLinux It's even packaged in some distros.– NemoCommented Feb 14, 2016 at 17:28
-
1
-
@A.D I'm not sure what you mean. It's text + images in HTML, packaged in ZIM rather than EPUB or other. You can get plain text out of it if you want, but my answer is mostly for people who prefer formatted books etc.– NemoCommented Nov 28, 2017 at 12:25
-
@Nemo How would I get the plain text out of a Zim file? Commented Jan 11, 2021 at 5:18
-
@cbracketdash it depends what you mean by plain text; you can use openzim to extract the HTML, which might have more formatting than you want.– NemoCommented Jan 11, 2021 at 14:59
While the selected answer is correct, it will potentially cause two problems:
- You may receive a 403 error denying access to the pagination under the presumption you are downloading as a bot
- There's the potential that you'll be sent to an external mirror, meaning the
wgetcommand will fail it's recursive checks on downloading the files from an external mirror.
The below solution fixes these problems:
wget -H -w 2 -m http://www.gutenberg.org/robot/harvest?filetypes[]=txt&langs[]=en \
--referer="http://www.google.com" \
--user-agent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6" \
--header="Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5" \
--header="Accept-Language: en-us,en;q=0.5" \
--header="Accept-Encoding: gzip,deflate" \
--header="Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7" \
--header="Keep-Alive: 300"
You may want to change the referer and user-agent strings to provide a bit of randomness.
ftp://mirrors.pglaf.org/mirrors/gutenberg-iso has a couple of good options.
ftp://mirrors.pglaf.org/mirrors/gutenberg-iso/pgdvd042010.iso is an 8GB file that should be sufficient for your needs.
There is more information here:
https://www.gutenberg.org/wiki/Gutenberg:The_CD_and_DVD_Project#Downloading_Via_FTP, it gives all the options of downloading the archive, including FTP and BitTorrent.
-
1I recommend using a torrent as a more ethical option (on the same page as that third link): gutenberg.org/wiki/… It's more ethical because it puts less strain on Project Gutenberg's servers. It's probably faster and more effective, too. Plus, they recommend it over ftp. If you use Linux, Transmission is a good bit torrent client for the purpose. This will only give you books that existed on the site in 2010, however, but it seems the best option when shadowhorst's answer isn't functional. Commented Dec 24, 2016 at 11:39
Another option is the great tool at http://pgiso.pglaf.org/.
- Enter an id range (e.g. 1-10000)
- Select the desired file types
- Choose the languages you want to include
- Wait for the notification
- Download
-
2
For access to all the plain text English books, I setup a private mirror as described in Project Gutenberg's Mirroring Guide. The ftp.ibiblio.org mirror worked with rsync:
mkdir $HOME/gutenberg
rsync -av --del ftp.ibiblio.org::gutenberg $HOME/gutenberg
To download only text files:
rsync -avm --include '*/' --include '*.txt' --exclude '*' --del ftp.ibiblio.org::gutenberg $HOME/gutenberg
Retrieving the file name listing will take anywhere from a few minutes to hours depending on Internet connection speed. After the file names have been retrieved, rsync will proceed to download all plain text files.
I wrote a script which will download all (not just English) the ePub books files from the PG FTP server. It will take about 12 hours as the server connection is slow, and it'll be a ton of data. You can find it here: https://gist.github.com/mxplusb/78363da5fe1e68a4e89d9fa8c720e034
I tried all the above answers, but none of them are resilient or fast. I would suggest using httrack.
httrack http://www.gutenberg.org/robot/harvest\?filetypes\[\]\=txt\&langs\[\]\=en -O "./guten" --mirrorlinks -%v "+aleph.gutenberg.org/*.zip" --continue
Why not use all your over-complicated and shown above programming skills and knowledge to make a simple button that links all those actions and says "Download All Current Books" -with a language option tab when you click it.
I am sure most of the users that come to the site are e-book collectors, and manual downloading certain books on topics that interested them is OK for 1 or 2 books. But making a bigger collection manually is a drag. Nevertheless if they need it for research or would simply like to own a huge digital library of books at their own PC. Most people are turned off and scatter away from the site when they realize that they have to be a Computer Wizard to do this.So a "Download All Current Books" button would benefit the site the project and its users, and it will certainly produce even more visitors to the site. So that way everyone is happy.
