1

I need this understanding for legally scaping Wikipedia/Wikimedia from their so-called pagecounts dumps. While I am writing software, this is not a programming question. It is a question about the file format. I believe that my question does belong here but in case you believe it does not, please direct me to the right Stack Exchange site. Or to a mailing list or sommething similar.

The files are available here: https://dumps.wikimedia.org/other/pagecounts-ez/merged/

sample file that I looked at is: https://dumps.wikimedia.org/other/pagecounts-ez/merged/2019/2019-01/pagecounts-2019-01-01.bz2

It is pretty big - around 400 Mb when archived. If you were to unarchive (un-bz2) it, you would see this comment at the top of the file:

# Wikimedia page request counts for 01/01/2019 (dd/mm/yyyy)
#
# Each line shows 'project page daily-total hourly-counts'
#
# Project is 'language-code project-code'
#
# Project-code is
#
# b:wikibooks,
# k:wiktionary,
# n:wikinews,
# q:wikiquote,
# s:wikisource,
# v:wikiversity,
# wo:wikivoyage,
# z:wikipedia (z added by merge script: ...

For example the line 28 aa.b MediaWiki:Ipb_already_blocked 3 B1M1X1 is meant to convey the following (unless I made a mistake): language_code=aa, project_code=b:wikibooks, page=MediaWiki:Ipb_already_blocked, daily_total=3, 1-2AM=1, noon-1PM=1, 11PM-midnight=1

However, the above description of the file format is incomplete. I counted all the project codes in this file and in addition to b, n, q, s, v, wo and z I encountered: m, m.d, d, wd, m.m, m.s, m.q, m.b, voy, w, m.v, y and zero. Full list:

Counter({'z': 20230021, 'm': 18184241, 'm.d': 1123132, 'd': 1032908, 'wd': 667656, 'm.m': 468812, 's': 213008, 'm.s': 190193, 'm.q': 111030, 'b': 100118, 'q': 95519, 'm.b': 58849, 'n': 51576, 'voy': 36258, 'm.voy': 25983, 'w': 25506, 'v': 25045, 'm.v': 12886, 'm.n': 9851, 'y': 92, 'zero': 1})

I imagine that the m. prefix stands for mobile and voy stands for wikivoyage and perhaps zero is to be ignored, but I still do not know what these stand for: m, d, wd, w, y <= This is the crux of my question. I suspect that I would enncounter other project codes in other files but that is outside the scope of this question.

6
  • 1
    zero is for zero-rated Wikipedia, a type of mobile view (which doesn't exist anymore).
    – Tgr
    Commented Apr 11, 2020 at 8:09
  • 1
    Reported as T249984.
    – Tgr
    Commented Apr 11, 2020 at 8:40
  • @Tgr, many thanks.
    – Leonid
    Commented Apr 11, 2020 at 18:27
  • @Tgr, I made a small edit to the post. I never found a k in that file. Actually it looks like I never found a wo either. The string representation of the Conter object (essentually a str->int dict) is posted in its entirety. This is in reference to a comment in phabricator T249984 "The StackExchange question says they found occurrences of k but I haven't found any. d is the correct code for Wiktionary.". Just wanted to make my bug report less confusing.
    – Leonid
    Commented May 2, 2020 at 3:55
  • 1
    Not sure there was a real resolution, other than the pointer to meta.wikimedia.org/wiki/Learning_patterns/… which seems pretty detailed.
    – Tgr
    Commented Feb 12, 2021 at 23:18

1 Answer 1

1

m, d, wd, w are all shortcuts to Wikimedia sister sites

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.