14

Is there a way to download a list of all Wikipedia categories?

1

4 Answers 4

12

You can download the Wikpedia categories from here. Here is the latest categories in SQL format.

This is a duplicate of this SO question.

2
4

This is really a general MediaWiki question, not a Wikipedia question, since there's nothing special about Wikipedia in this regard.

There is a stable, documented, way to obtain a list of all categories from a MediaWiki server. It is documented in the MediaWiki API documentation. You simply need to write, or obtain, a tool that makes use of this protocol.

2
  • This is not "general MediaWiki question", because: 1) MediaWik does not contains Wikipedia's content; 2) Wikipedia users has no access to DB, only to dumps.
    – AndriuZ
    Jul 13, 2018 at 13:16
  • @AndriuZ You don't understand. MediaWiki is software, not content. The MediaWiki API is exposed by sites like Wikipedia, Wiktionary, and others. Mar 30, 2019 at 11:49
2

Use the API. A list of API clients is here: https://www.mediawiki.org/wiki/API:Client_code

For example, using the command-line wikiget:

./wikiget.awk -z wiktionary -c "Spanish proper nouns" > output.txt
1
1

dumps + mysqldump-to-csv

Here are some minimal working commands to extract it from the dumps.

As mentioned at: https://stackoverflow.com/questions/24474288/how-to-obtain-a-list-of-titles-of-all-wikipedia-articles/77248954#77248954 importing the MySQL dumps is very slow party due to indices we don't need. Therefore, it can be much faster to just parse the dumps directly. The sanest approach I found so far is with: https://github.com/jamesmishra/mysqldump-to-csv from https://stackoverflow.com/questions/12040816/dump-all-tables-in-csv-format-using-mysqldump/28168617#28168617

git clone https://github.com/jamesmishra/mysqldump-to-csv
cd mysqldump-to-csv
git checkout 24301dfa739c13025844ed3ff5a8abe093ced6cc
patch <<'EOF'
diff --git a/mysqldump_to_csv.py b/mysqldump_to_csv.py
index b49cfe7..8d5bb2a 100644
--- a/mysqldump_to_csv.py
+++ b/mysqldump_to_csv.py
@@ -101,7 +101,8 @@ def main():
     # listed in sys.argv[1:]
     # or stdin if no args given.
     try:
-        for line in fileinput.input():
+        sys.stdin.reconfigure(errors='ignore')
+        for line in fileinput.input(encoding="utf-8", errors="ignore"):
             # Look for an INSERT statement and parse it.
             if is_insert(line):
                 values = get_values(line)
EOF
cd ..

and then we can just:

wget https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-category.sql.gz
zcat enwiki-latest-category.sql.gz | python mysqldump-to-csv/mysqldump_to_csv.py | csvtool format '%(2)\n' - > categories.txt

This extracted the current ~2.3 million categories in just 9s after the download on my Lenovo ThinkPad P51 SSD.

This information is also present in the pages table, every page with a namespace of 14 is a category as per https://en.wikipedia.org/wiki/Wikipedia:Namespace The category table likely exists to contain extra information about category pages via title JOIN that does not apply to article pages. So we could also:

wget https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-page.sql.gz
time zcat enwiki-latest-page.sql.gz | python mysqldump-to-csv/mysqldump_to_csv.py | csvtool format '%(2) %(4) %(3)\n' - | awk '$1==14 && $2==0 {print $3}' > titles-csv.txt

but this one is slower as it has to process more data.

Tested on Ubuntu 23.04.

Not the answer you're looking for? Browse other questions tagged or ask your own question.