Angus Grieve-Smith

In <a href="https://www.grieve-smith.com/blog/2013/12/existentials-and-universals/">my previous post</a>, I discussed the differences between existential and universal statements. In particular, the standard of evidence is different: to be sure that an existential statement is correct we only need to see one example, but to be sure a universal is correct we have to have examined everything.
But what if we don&#8217;t have the time to examine everything, and we don&#8217;t have to be absolutely sure? As it turns out, a lot of times we can be pretty sure. We just need a representative sample of everything. It&#8217;s quicker than examining every member of your population, and it may even be more accurate, since there are always measurement errors, and measuring a lot of things increases the chance of an error.
Pierre-Simon Laplace figured that out for the French Empire. In the early nineteenth century, Napoleon had conquered half Europe, but he didn&#8217;t have a good idea how many subjects he had. Based on the work of Thomas Bayes, Laplace knew that a relatively small sample of data would give him a good estimate. He also figured out that he needed the sample to be representative to get a good estimate.
“The most precise method consists of (1) choosing districts distributed in a roughly uniform manner throughout the Empire, in order to generalize the result, independently of local circumstances,” wrote Laplace in 1814. If you didn&#8217;t have a uniform distribution, you might wind up getting all your data from mountainous districts and underestimating the population, or getting data from urban districts and overestimating. Another way to avoid basing your generalizations on unrepresentative data is random sampling.
<a href="https://www.grieve-smith.com/blog/wp-content/uploads/2014/01/266669-MBPT_Spotlight_1031_The_Literary_Digest.jpg"><img loading="lazy" decoding="async" src="https://www.grieve-smith.com/blog/wp-content/uploads/2014/01/266669-MBPT_Spotlight_1031_The_Literary_Digest.jpg" alt="&quot;Is Our Face Red!&quot; says the Literary Digest" width="225" height="297" class="alignright size-full wp-image-307" /></a>A lot of social scientists, including linguists, understand the value of sampling. But many of them don&#8217;t understand that it&#8217;s representative sampling that has value. Unrepresentative samples are worse than no samples, because they can give you a false sense of certainty.
A famous example <a href="http://en.wikipedia.org/w/index.php?title=Bias_%28statistics%29&amp;oldid=55730748#Biased_sample">mentioned in Wikipedia</a> is when a Literary Digest poll forecast that Alfred M. Landon would defeat Franklin Delano Roosevelt in the 1936 Presidential election. That poll was biased because the sample was taken from lists of people who owned telephones and automobiles, and those people were not representative of the voters overall. The editors of the Literary Digest were not justified in generalizing those universal statements to the electorate as a whole, and thus failed to predict Roosevelt&#8217;s re-election.
<div id="attachment_311" style="width: 298px" class="wp-caption alignleft"><a href="https://www.grieve-smith.com/blog/wp-content/uploads/2014/01/averageitalianfemale.jpg"><img loading="lazy" decoding="async" aria-describedby="caption-attachment-311" src="https://www.grieve-smith.com/blog/wp-content/uploads/2014/01/averageitalianfemale-288x500.jpg" alt="&quot;Average Italian Female&quot; by Colin Spears" width="288" height="500" class="size-medium wp-image-311" /></a>&#8220;Average Italian Female&#8221; by Colin Spears</div>What can be deceiving is that you get things that look like averages and percentages. And they are averages and percentages! But they&#8217;re not necessarily averages of the things you want an average of. A striking example comes from a blogger named Colin Spears, who was intrigued by <a href="http://faceresearch.org/demos/average" target="_blank">a &#8220;facial averaging&#8221; site</a> set up by some researchers at the University of Aberdeen (they&#8217;ve since moved to Glasgow). Spears uploaded pictures from 41 groups, including &#8220;Chad and Cameroonian&#8221; and created &#8220;<a href="https://pmsol3.wordpress.com/category/editors-comment/" target="_blank">averages</a>.&#8221; These pictures were picked up by a number of websites, stripped of their credits, and bundled with all kinds of misleading and inaccurate information, <a href="http://researchtheheadlines.org/2013/10/04/no-science-behind-the-story/" target="_blank">as detailed by Lisa De Bruine</a>, one of the creators of the software used by Spears.
Some bloggers, like <a href="http://jezebel.com/5758201/see-the-average-face-of-women-from-40-different-countries/" target="_blank">Jezebel&#8217;s Margaret Hartmann</a>, noted that the &#8220;averages&#8221; all looked to be around twenty years old, which is not <a href="http://en.wikipedia.org/wiki/List_of_countries_by_median_age" target="_blank">the median age for most countries</a> according to the CIA World Fact Book (which presumably relies on better samples). In fact, the median age for Italian women (see image) is 45.6. The average look of the image is in the twenties, because that&#8217;s the age of the images that Spears uploaded to the Aberdeen site. So we got averages of some Italian women, but nothing that actually represents the average (of all) Italian women. (Some blog posts about this even showed a very light-skinned face for &#8220;Average South African Woman,&#8221; but that was just a mislabeled &#8220;Average Argentine Woman.&#8221;)
Keep this in mind the next time you see an average or a percentage. What was their sampling method? If it wasn&#8217;t uniform or random, it&#8217;s not an average or percentage of anything meaningful. If you trust it, you may wind up spreading inaccuracies, like a prediction for President Landon or a twentysomething average Italian woman. And won&#8217;t your face be red!

In [my previous
post](https://www.grieve-smith.com/blog/2013/12/existentials-and-universals/),
I discussed the differences between existential and universal
statements. In particular, the standard of evidence is different: to be
sure that an existential statement is correct we only need to see one
example, but to be sure a universal is correct we have to have examined
everything.

But what if we don't have the time to examine everything, and we don't
have to be absolutely sure? As it turns out, a lot of times we can be
pretty sure. We just need a representative sample of everything. It's
quicker than examining every member of your population, and it may even
be more accurate, since there are always measurement errors, and
measuring a lot of things increases the chance of an error.

Pierre-Simon Laplace figured that out for the French Empire. In the
early nineteenth century, Napoleon had conquered half Europe, but he
didn't have a good idea how many subjects he had. Based on the work of
Thomas Bayes, Laplace knew that a relatively small sample of data would
give him a good estimate. He also figured out that he needed the sample
to be representative to get a good estimate.

"The most precise method consists of (1) choosing districts distributed
in a roughly uniform manner throughout the Empire, in order to
generalize the result, independently of local circumstances," wrote
Laplace in 1814. If you didn't have a uniform distribution, you might
wind up getting all your data from mountainous districts and
underestimating the population, or getting data from urban districts and
overestimating. Another way to avoid basing your generalizations on
unrepresentative data is random sampling.

[![\"Is Our Face Red!\" says the Literary
Digest](https://www.grieve-smith.com/blog/wp-content/uploads/2014/01/266669-MBPT_Spotlight_1031_The_Literary_Digest.jpg){.alignright
.size-full .wp-image-307 loading="lazy" decoding="async" width="225"
height="297"}](https://www.grieve-smith.com/blog/wp-content/uploads/2014/01/266669-MBPT_Spotlight_1031_The_Literary_Digest.jpg)A
lot of social scientists, including linguists, understand the value of
sampling. But many of them don't understand that it's *representative*
sampling that has value. Unrepresentative samples are worse than no
samples, because they can give you a false sense of certainty.

A famous example [mentioned in
Wikipedia](http://en.wikipedia.org/w/index.php?title=Bias_%28statistics%29&oldid=55730748#Biased_sample)
is when a *Literary Digest* poll forecast that Alfred M. Landon would
defeat Franklin Delano Roosevelt in the 1936 Presidential election. That
poll was biased because the sample was taken from lists of people who
owned telephones and automobiles, and those people were not
representative of the voters overall. The editors of the *Literary
Digest* were not justified in generalizing those universal statements to
the electorate as a whole, and thus failed to predict Roosevelt's
re-election.

::: {#attachment_311 .wp-caption .alignleft style="width: 298px"}
[![\"Average Italian Female\" by Colin
Spears](https://www.grieve-smith.com/blog/wp-content/uploads/2014/01/averageitalianfemale-288x500.jpg){.size-medium
.wp-image-311 loading="lazy" decoding="async"
aria-describedby="caption-attachment-311" width="288"
height="500"}](https://www.grieve-smith.com/blog/wp-content/uploads/2014/01/averageitalianfemale.jpg)

"Average Italian Female" by Colin Spears
:::

What can be deceiving is that you get things that look like averages and
percentages. And they are averages and percentages! But they're not
necessarily averages of the things you want an average of. A striking
example comes from a blogger named Colin Spears, who was intrigued by [a
"facial averaging"
site](http://faceresearch.org/demos/average){target="_blank"} set up by
some researchers at the University of Aberdeen (they've since moved to
Glasgow). Spears uploaded pictures from 41 groups, including "Chad and
Cameroonian" and created
"[averages](https://pmsol3.wordpress.com/category/editors-comment/){target="_blank"}."
These pictures were picked up by a number of websites, stripped of their
credits, and bundled with all kinds of misleading and inaccurate
information, [as detailed by Lisa De
Bruine](http://researchtheheadlines.org/2013/10/04/no-science-behind-the-story/){target="_blank"},
one of the creators of the software used by Spears.

Some bloggers, like [Jezebel's Margaret
Hartmann](http://jezebel.com/5758201/see-the-average-face-of-women-from-40-different-countries/){target="_blank"},
noted that the "averages" all looked to be around twenty years old,
which is not [the median age for most
countries](http://en.wikipedia.org/wiki/List_of_countries_by_median_age){target="_blank"}
according to the CIA World Fact Book (which presumably relies on better
samples). In fact, the median age for Italian women (see image) is 45.6.
The average look of the image is in the twenties, because that's the age
of the images that Spears uploaded to the Aberdeen site. So we got
*averages of some* Italian women, but nothing that actually represents
*the average* (of all) Italian women. (Some blog posts about this even
showed a very light-skinned face for "Average South African Woman," but
that was just a mislabeled "Average Argentine Woman.")

Keep this in mind the next time you see an average or a percentage. What
was their sampling method? If it wasn't uniform or random, it's not an
average or percentage of anything meaningful. If you trust it, you may
wind up spreading inaccuracies, like a prediction for President Landon
or a twentysomething average Italian woman. And won't your face be red!


In my previous post, I discussed the differences between existential and universal statements. In particular, the standard of evidence is different: to be sure that an existential statement is correct we only need to see one example, but to be sure a universal is correct we have to have examined everything. But what if we don’t have the time to examine everything, and we don’t have to be absolutely sure?


Publicaciones de Rogue Scholar

Estimating universals, averages and percentages