Mathematics is to science what seasoning is to cooking. It
is impossible to imagine modern cooking without spices. However, and this is an
extremely important qualification, not all spices lead to good cooking. Some combination
of spices may ruin an otherwise good dish, and other combination of spices may
even cover up the taste of a foul, putrefying, or poisonous dish.
The application of mathematics to science follows the same
rule. While modern science would not exist without mathematics, some
applications of mathematics either ruin what otherwise would be good science or
cover up trivial claims or outright quackery. The IQ testing and much of the neoclassical
economics are examples of the latter. Unfortunately, fraudulent use of
mathematics has spread to other social sciences as well. This evidenced, among other, by the fact that
most scientists cannot replicate results reported by their peers .
There are many reasons behind this trend, but I would like
to focus on two of them: (i) the sudden
availability of exotic spices in the hands of chefs, and (ii) the symbiotic
relationship between chefs and restaurateurs who employ them.
A. The availability of exotic spices.
As we all know, the quest for spices was the main reason
behind early European explorations. Spices had to be imported to the medieval
Europe from the far-away lands through trade routes fraught with many
dangers. This made them rare and very
expensive. As a result, they were available only to those select few who could
afford them. However, the improvements in
transportation and warfare technologies made these exotic spices more and more
available, which gave birth to fine European cuisine. The next push came with globalization that
spread formerly “exotic” cuisine – Chinese, Indian, Middle Eastern, etc. all
over the world, creating the new type of cooking known as "fast food". Every shopping mall in the US and Europe has
“ethnic” (Chinese, Japanese, Vietnamese, Mediterranean, etc.) eateries offering
hastily prepared simple and rather crappy dishes whose taste is virtually
indistinguishable from one another.
The same process occurred in science. The medieval science – limited for the most
part to theology – used logic and sophistry as its main methods. However, the development and application of mathematics
gave birth to the modern physics, astronomy, and other empirical sciences that dethroned
theology. The next push came from the
development of computers that made mathematical calculations easy and
effortless. Today the availability of
cheap computer numerical data manipulation programs made these tools available
to anyone involved in any kind of scientific inquiry, including those called “idiographic sciences” in the continental European tradition and concerned
with various manifestations of human behavior (history, anthropology,
psychology, sociology, economics etc.). The
distinctive characteristic of the idiographic sciences, which distinguished
them form nomothetic sciences, was their focus on understanding of unique human
phenomena rather than general rules of nature, which in turn called for
qualitative observations rather than quantitative methods.
The availability of cheap computers and computer programs fundamentally
altered not only the direction of research in the idiographic sciences, but
also what kinds of data are being collected. Since qualitative data are more difficult
to process by computer software, their collection often takes the back seat in
favor of quantitative – or rather pseudo-quantitative - data collected by
opinion surveys. They are
pseudo-quantitative, because they use numerical scales representing intensity
(e.g. strongly agree, somewhat agree, neither agree not disagree, etc.), but
they cannot be processed as “real” numbers.
For “real” numbers, such as 1,2, 3, 4 etc. we can say that
the difference between 1 and 2 is the same as that between 3 and 4, and that 2
is twice as big as 1 just as 4 is twice as big as 2. However, when those numbers are being used as
mere symbols representing multiple choices in an opinion survey, they cease to
be “real” numbers. They can be replaced
with letters a,b,c,d, etc. or even pictograms representing different choices
cooked up by survey designers. The reason why they are not “real” numbers but
pictograms is that we cannot say that a distance between choice A and choice B
(e.g. strongly agree and moderately agree) is the same as between b and c
(moderately agree and neither agree nor disagree).
Research shows that subjective perceptions of quantities
themselves differ from their numerical properties. For example, a 5 percent change in
probability is perceived differently depending on the overall probability of an
outcome (i.e. whether it is 10%, 50% or 90%).
When it comes to opinions and perceptions, that level of subjectivity is
even higher. For example, if I only
“moderately agree” with an opinion on, say, capital punishment, it may not take
much to persuade me to be an agnostic (neither agree nor disagree). However, if I have a strong feeling (strongly
agree or strongly disagree), it typically takes much more to move me into the
“moderate agreement/disagreement” direction.
There are other cognitive biases distorting these
measurements as well. If I ask
respondents to measure, say, the length of sticks by using a measuring tape
calibrated in centimeters or inches, most of the reported results will be reasonably
accurate, even if some respondents may not be familiar with a particular scale
of measurement. It is so, because the
measurement requires simple application of a well calibrated tool that does not change
when applied to the object being measured.
That is, measuring tapes are typically made out of steel and do not
expand or shrink to “fit” the particular object being measured. If the measurement tape were made out of
rubber, however, the measurements it produces would be rubbish, because the
length of the tape would change each time it is applied to the object.
Using subjective scales of the type strongly/moderately
agree or disagree is like using a rubber measuring tape. The scale itself changes each time the
measurement is applied. If I am
currently in pain, my perception of it is affected by my current experience, so
I will report my pain as severe on a rating scale. However, if I am asked to evaluate my past
pain experience on the same scale, I would report it as less severe, because
humans have the tendency to forget and minimize past negative experiences. Likewise, if I am asked to record my emotional
state or my views on a policy proposal, my rating would be affected by two
things – how this question is worded, and what I am told or asked to do
before being asked that question. If
previous line of inquiry evoked negative experiences my answers would differ
from those if the previous line of inquiry evoked positive experiences (the so
called “anchoring bias”).
It is therefore clear that answers solicited using such
measurement scales are nothing more than subjective opinions that will likely
change if the circumstances under which the measurement is takes changes. Assigning numbers to these measurements creates
a false illusion that they represent numerical quantities, similar to the
measurements of physical objects taken with a measuring tape made of steel. In reality, it is assigning numbers to the
shapes of clouds in the sky.
Transforming these shapes into numbers does not changer the idiographic
nature of these observations into nomothetic science leading to the general
laws, because such laws simply do not exist in this particular case. While each cloud was certainly caused by
objective and measurable atmospheric conditions, its particular shape is
idiosyncratic and impossible to predict from these measurable atmospheric
conditions.
More conscientious researchers may refrain from treating such
subjective responses like “real” numbers and limit their analysis to reporting
frequency counts of broad categories of responses (e.g. how many people
agreed and how many disagreed), but the availability of cheap data processing
software make such analysis look “pedestrian” and pressure is applied to use
more “advanced” techniques. I am
speaking from experience here. Some time
ago, an anonymous peer reviewer of my paper using frequency-based contingency
tables showing distributions of opinions collected in a survey called this
technique “pedestrian” and suggested one based on regression. In other words, let’s treat them as “real”
numbers. This advice reminds me of the old economist joke – he could not find a
can opener on an uninhabited island, so he assumed he had one.
The problem is not limited to the assumptions about quantitative
properties of the data, but the kind of research that gains dominance in social
sciences with the advent of cheap computational tools. This new research paradigm favors questions
that can be answered by numerical or quasi-numerical data, because such data
are easy to collect and process. Hence
the proliferation of various opinion surveys.
The idiocy of this approach lies not only in the misinterpretation of
numerical data, but more importantly, in intellectual laziness is fosters. Researchers abandon the difficult
intellectual task of trying to understand how people think and under what
conditions in favor of giving them simplistic multiple choice tests involving
prefabricated opinion statements, because such simplistic multiple choice tests
are easy to score and process by computers.
If this is not the proverbial drunkard’s search, I do not know what is.
B. The symbiotic relationship between chefs and
restaurateurs
Imagine, if you will, a renowned chef coming to the
restaurateur who employs her and saying “I tested several recipes, but they do
not meet the high standards of this fine dining establishment.” “So what are we going to serve our guests
tonight?” asks the restaurateur. “I do
not have anything ready yet” answers the chef, but with more testing I should
be able to come up with a good recipe in a week or two.” It is likely that this chef would be
looking for another job after this conversation.
The same applies to scientific knowledge. As Bruno Latour (“Science in Action”) aptly
observed, the production of science differs from the presentation of scientific
facts after the have been produced, that is, accepted as facts by the
scientific community. Whereas the
presentation of agreed upon facts is subjected to only one requirement – truth (i.e.
concordance with the reality) – the production of science is a much messier process. For one thing, it involves a search in the
dark, before “the truth” has been discovered.
All that is known during this process involves various, often
conflicting claims posed by people representing different schools of thought to
what they believe is factually true.
Testing these claims requires enormous resources, teams of researchers
and support personnel, laboratories, expensive instruments, and channels of
communication with the scientific community.
The procurement of these enormous resources requires the involvement of
a much larger number of people than the scientists who do the actual
research. It requires institutions –
universities and research laboratories – run by an army of administrative staff.
It involves designers and manufacturers of specialized
equipment without which research would not be possible. It also requires funding, which in turn is procured
by linking the prospective results of scientific inquiries to the interest of
people who control financial resources, such as government officials, corporate
bosses, or investors.
All those people – the administrators, the government
officials, the corporate bosses, and the investors – want to see the results of
their efforts and investments. In that
sense, they act like the restaurateur in our story – they expect their chefs to
produce meal of certain quality, but they will not settle for a chef who tells
them that she tested several recipes and found none of them satisfactory. Yet, when we look at science as a set of
agreed upon facts, all that messy production process disappears from the eyes,
and we are served with an equivalent of a meal served to us on a sliver plate
that need to pass only our taste test.
This is why the actual production of science, like cooking,
is very different form tasting the already prepared product. In the idealized world, chefs and scientists
look for recipes high and low, test them and select only those that pass the
rigorous test – the ultimate consistency between the object (a dish or natural phenomenon) and human perception of it (excellent flavor or truth). This is how
scientific research appears to Karl Popper – as a long series of attempts to
falsify different scientific hypotheses to see which one will withstand this
test. This may be a logical way to
proceed if one’s goal is to find THE ultimate truth in some unspecified but
likely long period of time, but it is a waste of effort if one’s goal is to
find a truth that is good enough to pass some rudimentary plausibility test and
satisfy the stakeholders in the scientific production process – the administrators,
the government bureaucrats, the funders, the investors and, last but not least,
the researchers themselves whose academic careers and salaries depend on producing
tangible results. Falsifying hypotheses may
produce results in a long run, but the stakeholders, like our restaurateur,
need to put the food on the table now, not in some distant future. They will not settle for a response “we have
eliminated some hypotheses, but we have nothing positive yet.” People providing such responses would be soon
looking for other jobs, whether they are chefs or scientists.
Here is where the ubiquity of computers and software
utilizing statistical analysis comes handy – they can produce minimally
satisfactory results in a rather short time period and with relatively little
effort. They can do it for two
reasons. First, their use requires
numbers, which in turn, leads to the substitution of qualitative phenomena with
something that looks like numbers, but it really is not. I covered this process in section A of this
essay. Second, it substitutes causal
models with statistical correlations and
inductive methods of reasoning with statistical significance tests.
We all learn in Research Methods 101 that correlation is not
causation and there is no need to further repeat this rather hackneyed truth. Suffice it to say that correlation may – but does
not have to – imply a causal connection, so finding a correlation is a useful
first step into an inquiry into what causes the phenomenon we want
explain. Unfortunately, for many research
projects, especially in social and behavioral sciences, this is the only
step. These researchers often shout
Eureka as soon as they discover a meaningful correlation. And how do they know if the correlation is
meaningful? By looking at the results of
the statistical significance test or the p-statistic, which by convention has
to be lower than 0.05 (or 5%) to be called “significant.” However, yelling Eureka after finding a “significant”
statistical correlation is like coming to a five star restaurant and being
served macaroni and cheese out of a can.
It is barely edible, all right, but hardly worth the five star restaurant
price. Here is why.
The p-statistic simply means that if we were to repeatedly draw a representative
sample of the same size as one at hand from the same population, the value of the
correlation we are looking at would be zero (meaning no correlation) in 5 (or
fewer, depending on the actual value of the p-statistic) in 100 such trials. In other words, there is only 5% (or less)
chance that the correlation that our analysis uncovered in the data is due to
random chance. This is all that there is
to it. Does this mean that there is a
cause-effect relation behind this correlation?
The answer is that we really have no way of knowing with
only this information at hand. Imagine
the following situation. I want to know
if it rained at night when I was asleep, so I look at the grass in my front
yard. If it is dry, I can safely conclude
that it did not rain. But what if it is
wet? Can I conclude that it rained? I can, but I could be dead wrong because rain
is not the only thing that makes grass wet.
It could have been my automatic sprinkler, or it could have been mist on
a chilly night. The same reasoning
applies to statistical significance testing.
What we are really testing here is whether the observed correlation is
an artifact of the statistical procedure – random selection of samples whose
composition slightly varies due to chance.
Our p-statistic tells us the probability of that, or in the
methodological parlance, the probability that the null-hypothesis (i.e. the
hypothesis of no correlation) is true.
If that probability is less than 5%, we by convention conclude that the
null hypothesis is false. The grass is
dry, therefore it did not rain that night.
But the falsification of the null hypothesis is not a sufficient reason to
claim that the variables in question are actually correlated, just as there is
insufficient reason to conclude that it rained when the grass is wet.
What is more, statistical correlations change depending on
the presence of other variables in our model.
Suppose for example that when I look at prices of cars in different
countries I find that in the US cars cost on average $25k or more, while in the
Global South countries cars cost in the vicinity of $15k. I also discover that there are far more cars
in the US than in the Global South. If I
entered these findings into a computer, I would learn that there is a positive
correlation between the price of cars and the number of cars sold, which would
make most economists scratch their heads as this contradicts everything they
know about prices and the size of the demand.
These economists would tell me that I would need to consider other
factors that affect prices and demand for cars, such as earnings, cost of other
goods, availability of roads etc. So if following
the economists’ advice, I enter them into my model, the value of the initially
observed correlation will almost certainly change, and it may even become
negative as the economists expected. However,
if I believe that my initial findings were correct and the correlation between
prices and quantity of cars is actually positive due to monopolization of
transportation by car companies, I will measure the availability of alternative
means of transportation, enter it into my model in the hope that the relationship
between price and the number of cars sold moves back into the positive
territory.
If this looks like a wild goose chase, there is certainly
some truth in it. Most statistical
models are highly dependent on analytic procedures used to construct them. These include the nature and the quality of the
data, the data cleaning and preparation procedures, the type and number of variables
in the model, and in more complex models, such as factor analysis,
researcher-supplied assumptions about the nature of the expected outcome that
are necessary for the computerized analysis to run. As Paul Ormerod (“Death of Economics”)
argued, complex econometric models can produce a virtually infinite number of solutions,
depending on more or less arbitrary assumptions that analysts build into
them. A similar conclusion was reached
by Stephen Jay Gould (“The Mismeasure of Man”) in regard to complex psychometric
models.
So if these complex statistical tools indeed put the
researchers on a wild goose chase, why is there such a strong pressure in the
research community to use these tools? The
answer is that, unlike in the wild nature where the wild goose typically gets
away after leading her pursuer sufficiently far astray, in the academic research
world the pursuers almost always get something from such a chase. It may not be the goose they were after, but
a few feathers, an insect or two, or a mouse if they are lucky. In other words, they will find statistically significant
correlations that computers will invariably produce if run long enough, which
then they can claim as their big catch in their research papers and use as a
bait to attract further research grants and support. There is a tangible product of the chase
after all, which makes everyone happy, the chef and the restaurateur and his
guest, the researches, the university administrators, the government
bureaucrats, and the funders and investors.
None of it would be possible without computer tools capable
of cranking out correlations in the matter of seconds from the garbage that
passes for the quantitative data. This has
transformed the science as Popper claimed it to be - falsification of scientific
hypotheses. Indeed there is little
economic value in such a pursuit, just as there is little value in a chef
deconstructing the food served by other restaurants. What pays out is spending most of the energy on
cranking out food for thought that the administrators and paying clients would
buy. There is little wonder that so few
of these results are replicable, since their construction involves a fair
amount of chance and data preparation.
We have achieved a lofty goal.
As the proliferation of computers and quantitative analytic tools reaches
unprecedented proportions, the social science research resembles, more and more
closely, an elaborate chase after a wild goose in the thicket of numbers that
pass for the data. On some great and
glorious day social scientists will reach the object of their physics envy and
turn their discipline into what theology was in the Middle Ages, an impressive
logically coherent intellectual edifice whose empirical relevance and
predictive power is on a par with that of a chimp randomly pushing computer
buttons.