Ref: http://blog.incognitech.in/download-files-from-apache-server-listing-directory/
You can use following command:
wget --execute="robots = off" --mirror --convert-links --no-parent --wait=5 <website-url>
Explanation with each options
wget
: Simple Command to make CURL request and download remote files to our local machine.
--execute="robots = off"
: This will ignore robots.txt file while crawling through pages. It is helpful if you're not getting all of the files.
--mirror
: This option will basically mirror the directory structure for the given URL. It's a shortcut for -N -r -l inf --no-remove-listing
which means:
-N
: don't re-retrieve files unless newer than local
-r
: specify recursive download
-l inf
: maximum recursion depth (inf or 0 for infinite)
--no-remove-listing
: don't remove '.listing' files
--convert-links
: make links in downloaded HTML or CSS point to local files
--no-parent
: don't ascend to the parent directory
--wait=5
: wait 5 seconds between retrievals. So that we don't thrash the server.
<website-url>
: This is the website url from where to download the files.
Happy Downloading :smiley: