When I run a query for "hers" in Google Ngram Viewer, I get back the word's frequency of occurrence as a percentage. We know the outcome percentage; what's the denominator on the other side? Is it 100 million words? More?
Add a comment
|
1 Answer
Currently the total number of words is 468,491,999,592. (Approaching half a trillion)
You can obtain that figure by downloading the latest dataset.
In that dataset, the total_counts file(txt) lists the number of unigrams, pages, and books for each year.
Summing up the unigram counts yields the 468,491,999,592 figure.
Important notes:
Currently (Nov 2015), the latest Ngram data is the Version 20120701 set. It was compiled in 2012, but covers books from 1505 to 2008.
A unigram is mostly the same as a word. Details of Google's parsing may yield differences in (hopefully) rare cases.
Only words within sentences are counted. Reference: Syntactic Annotations for the Google Books Ngram Corpus(PDF), section 3.2.
The percent displayed on the graph is normalized per year. This means that there is no one "denominator" if you are trying to figure the real counts of a given word. (But you can get that data from the downloaded datasets.) See below.
How the graph is calculated:
Consider the word "circumvallate" (all lower case). From the data download, we look up the 1978 and 1979 figures and get:
| Word | Count in 1978 books | Count in 1979 books |
|---|---|---|
| circumvallate | 335 | 261 |
| All words | 4,365,839,878 | 4,528,331,460 |
Which calculates out as:
- 0.0000076732% in 1978
- 0.0000057637% in 1979
If we then look at the corresponding graph, and hover over the respective points, we get:
Which agrees with what we calculate.
