Take the 2-minute tour ×
Web Applications Stack Exchange is a question and answer site for power users of web applications. It's 100% free, no registration required.

I know how it generates the site map and all. What I need is a complete sitemap of all topics covered in wikipedia.org. Anyone know where I could get it? I have been crawling it but keep running out of space with my crawler program.

share|improve this question

migrated from stackoverflow.com Aug 7 '11 at 8:54

This question came from our site for professional and enthusiast programmers.

2  
Why scrape Wikipedia when you can download the content? meta.wikimedia.org/wiki/Data_dumps –  icktoofay Aug 7 '11 at 5:41
    
Wikipedia (English) has 3,696,583 articles. Think about how big that sitemap page would be ... (a few hundred MB in size) –  freedompeace Aug 7 '11 at 5:44
1  
Maybe modify your crawler program to store data to disk instead of trying to keep it all in memory? –  aroth Aug 7 '11 at 5:53
add comment

1 Answer 1

There is a download available for all pages in the article namespace of the English language wikipedia which is located at:

http://dumps.wikimedia.org/enwiki/latest/enwiki-latest-all-titles-in-ns0.gz

However they do not generate this kind of list for every language, so you need to create it yourself from the all content dump. See the full list of all available dumps (just change the en-part of the URL to the language you want to see).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.