Is there a way to download a list of all Wikipedia categories? [closed]

Question

Is there a way to download a list of all Wikipedia categories?

Answer 1

You can download the Wikpedia categories from here. Here is the latest categories in SQL format.

This is a duplicate of this SO question.

Answer 2

This is really a general MediaWiki question, not a Wikipedia question, since there's nothing special about Wikipedia in this regard.

There is a stable, documented, way to obtain a list of all categories from a MediaWiki server. It is documented in the MediaWiki API documentation. You simply need to write, or obtain, a tool that makes use of this protocol.

Answer 3

Use the API. A list of API clients is here: https://www.mediawiki.org/wiki/API:Client_code

For example, using the command-line wikiget:

./wikiget.awk -z wiktionary -c "Spanish proper nouns" > output.txt

Answer 4

dumps + mysqldump-to-csv

Here are some minimal working commands to extract it from the dumps.

As mentioned at: https://stackoverflow.com/questions/24474288/how-to-obtain-a-list-of-titles-of-all-wikipedia-articles/77248954#77248954 importing the MySQL dumps is very slow party due to indices we don't need. Therefore, it can be much faster to just parse the dumps directly. The sanest approach I found so far is with: https://github.com/jamesmishra/mysqldump-to-csv from https://stackoverflow.com/questions/12040816/dump-all-tables-in-csv-format-using-mysqldump/28168617#28168617

git clone https://github.com/jamesmishra/mysqldump-to-csv
cd mysqldump-to-csv
git checkout 24301dfa739c13025844ed3ff5a8abe093ced6cc
patch <<'EOF'
diff --git a/mysqldump_to_csv.py b/mysqldump_to_csv.py
index b49cfe7..8d5bb2a 100644
--- a/mysqldump_to_csv.py
+++ b/mysqldump_to_csv.py
@@ -101,7 +101,8 @@ def main():
     # listed in sys.argv[1:]
     # or stdin if no args given.
     try:
-        for line in fileinput.input():
+        sys.stdin.reconfigure(errors='ignore')
+        for line in fileinput.input(encoding="utf-8", errors="ignore"):
             # Look for an INSERT statement and parse it.
             if is_insert(line):
                 values = get_values(line)
EOF
cd ..

and then we can just:

wget https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-category.sql.gz
zcat enwiki-latest-category.sql.gz | python mysqldump-to-csv/mysqldump_to_csv.py | csvtool format '%(2)\n' - > categories.txt

This extracted the current ~2.3 million categories in just 9s after the download on my Lenovo ThinkPad P51 SSD.

This information is also present in the pages table, every page with a namespace of 14 is a category as per https://en.wikipedia.org/wiki/Wikipedia:Namespace The category table likely exists to contain extra information about category pages via title JOIN that does not apply to article pages. So we could also:

wget https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-page.sql.gz
time zcat enwiki-latest-page.sql.gz | python mysqldump-to-csv/mysqldump_to_csv.py | csvtool format '%(2) %(4) %(3)\n' - | awk '$1==14 && $2==0 {print $3}' > titles-csv.txt

but this one is slower as it has to process more data.

Tested on Ubuntu 23.04.