I know how it generates the site map and all. What I need is a complete sitemap of all topics covered in wikipedia.org. Anyone know where I could get it? I have been crawling it but keep running out of space with my crawler program.
There is a download available for all pages in the article namespace of the English language wikipedia which is located at:
However they do not generate this kind of list for every language, so you need to create it yourself from the all content dump. See the full list of all available dumps (just change the
en-part of the URL to the language you want to see).