Mathematics is to science what ketchup is to food - it improves
the taste of otherwise unpalatable dishes, but it kills more subtle flavors of
everything else. This is particularly true of social sciences, where the
availability of cheap computer numerical data manipulation programs
fundamentally altered not only the direction of research, but also what kinds
of data are being collected.
Since qualitative data are more difficult
to process by computer software, their collection often takes the back seat in
favor of quantitative – or rather pseudo-quantitative - data collected by
opinion surveys. They are pseudo-quantitative,
because they use numerical scales representing intensity (e.g. strongly agree,
somewhat agree, neither agree not disagree, etc.), but they cannot be processed
as “real” numbers.
For “real” numbers, such as 1,2, 3, 4 etc. we can say that the
difference between 1 and 2 is the same as that between 3 and 4, and that 4 is
twice as big as 2. However, when those
numbers are being used as mere symbols representing multiple choices in an
opinion survey, they cease to be “real” numbers. They can be replaced with letters a,b,c,d,
etc. or even pictograms representing different choices cooked up by survey
designers. The reason why they are not “real” numbers but pictograms is that we
cannot say that a distance between choice a and choice b (e.g. strongly agree
and moderately agree) is the same as between b and c (moderately agree and
neither agree nor disagree).
Research shows that subjective perceptions of quantities
themselves differ from their numerical properties. For example, a 5 percent change in
probability is perceived differently depending on the overall probability of an
outcome (i.e. whether it is 10%, 50% or 90%).
When it comes to opinions and perceptions, that level of subjectivity is
even higher. For example, if I only “moderately
agree” with an opinion on, say, capital punishment, it may not take much to
persuade me to be an agnostic (neither agree nor disagree). However, if I have a strong feeling (strongly
agree or strongly disagree), it typically takes much more to move me into the “moderate
agreement/disagreement” direction.
Yet, assigning numbers to these options creates a false illusion
that they represent numerical quantities.
More conscientious researchers may refrain from treating them like “real”
numbers and limit their analysis to reporting frequency counts, but the
availability of cheap data processing software make such analysis look “pedestrian”
and a pressure is applied to use more “advanced” techniques. I am speaking from experience here. Some time ago, an anonymous peer reviewer of
my paper using frequency-based contingency tables showing distributions of opinions
collected in a survey called this technique “pedestrian” and suggested one
based on regression. In other words, let’s
treat them as “real” numbers. This advice reminds me of the old economist joke –
he could not find a can opener on an uninhabited island, so he assumed he had one.
The problem is not limited to the assumptions about quantitative
properties of the data, but the kind of research that gains dominance in social
sciences with the advent of cheap computational tools. This new research paradigm favors questions
that can be answered by numerical or quasi-numerical data, because such data
are easy to collect and process. Hence
the proliferation of various opinion surveys.
The idiocy of this approach lies not only in the misinterpretation of
numerical data, but more importantly, in intellectual laziness is fosters. Researchers abandon the difficult
intellectual task of trying to understand how people think and under what
conditions in favor of giving them simplistic multiple choice tests involving
pre-fabricated opinion statements, because such simplistic multiple choice
tests are easy to score and process by computers. If this is not the proverbial drunkard’s
search, I do not know what is.
Another implication of this observation is that science, or at least social science, is not
progress achieved by systematic testing of scientific theories as Karl Popper
believed, but rather movements between what Imre Lakatos called “scientific
research programmes.” The purpose of a
scientific research programme is not theory testing, as Popper believed, but ‘problem
shift” – that is, the construction of auxiliary hypotheses that render
contradicting evidence irrelevant to save core assumptions of a favored theory
from empirical refutation. Problem
shifts may take the form of crude “gate keeping” of the orthodoxy, for example
in economics, as decried by John Kenneth Galbraith, or more subtle forms,
such as changes in academic fads or the availability of new instruments of
scientific research.
The use of computer software utilizing mathematical analysis in
social science represents such a problem shift due to new tools. The problems researched and theories proposed
to explain them tend to be limited to those that lend themselves to being
processed by computerized tools. This
puts social science on the trajectory to become what theology was in the Middle
Ages, an impressive logically coherent intellectual edifice whose empirical relevance
and predictive power is on a par with that of a chimp randomly pushing computer
buttons.
2 quick comments:
ReplyDeletethe first sentence struck me as (among) the most absurd thing(s) i've ever read; but then i originate in the hard and semi-hard sciences (physics and theorectical biology). In those cases math is more like DNA, or technology (organized matter, from elementary particles to particle detectors to spectroscopy and microscopes...if you want to see a double helix or microbe) than ketchup.
You can have science without math just as one can have organized evolving matter without DNA (eg before life emerged as in Turing patterns) ---- people did all kinds of experiments and darwin came up with evolution with little math. But that has changed. Modern life requires DNA and modern science requires things like computers which rely on quantum theory which relies on advanced math.
You can live for awjhile on burgers, but less on ketchup (though i think its surpirising what people can live on ).
2. the rest of your comment I agree with to a large extent. I read a fair amount of social science from behavioral econ, psychology, etc. Computers today have made it very easy to write papers that look very rigorous but are the modern equivalent of art by people who make it by throwing paint on a wall because that is a well known and respected technique. ( i make no judgement about the guy who is famous for doing that----there are papers by symkin however which show that many people have a hard time distinguishing 'masters' of this form with ones created by kindergartners).
In other words its GIGO. Alot of the people do know how to do regression analyses on large data sets etc but in fact today with matematica you can almost get a paper published in a good physics journal without knowing how to solve any equations---you just type the variables into the program and you'll come up with something almosty indistinguishable from something that might even be of Noble prize quality.
Behavior genetics and behavioral economics use high quality analyses on poorly designed experiments (eg you only study WEIRD (henrich of ubc) people, or infer universal results by studying a sample of 3rd year PhD economics students in your 11am monday seminar).
Large scale studies have slightly different issues. One of my favorites is 'Whorfian Economics'---you can google it (the posts on the blog replicated typo and language log i find on point)
I am not against math, of course, I am against misusing it to hide poverty of thought or logical weakness of the argument under impressively looking equations and procedures. In other words, using them as 'problem shifts' to avoid critique and empirical refutation - to quote from Lakatos.
DeleteThe opening sentence is, of course, a journalistic hyperbole, but a good, attention drawing one. Would you read this piece if the first sentence read 'Let's us now discuss the misuses of data analysis techniques in social science research'? I doubt.
Another point, math is indispensable for theory building, which is basically summarizing and systematizing what is already known (which is also the meaning of the Greek root of the word, according to Heidegger's "Modern Science, Metaphysics and Mathematics). But it is a different story when it comes to discoveries and breakthroughs. A lot of these came about through the use of new and better instruments or simply by luck rather than by refining the theory. Of course, the instruments could not be produced without math. Again, I am not disputing the importance of numerical data analysis in hypothesis testing, but that the hypotheses themselves do not need to and often are not mathematically derived.
Yeah---I wouldn't have commented if you had used that other first sentence.
ReplyDeleteThere are quite a few papers of that sort around---eg Ioannidis in PLOS medicine 2005-'why most published scientific research findings are wrong', the "perspectives in psychological science' group which aims to replicate psychology studies, etc. Simkin's stuff is on arxiv e.g. www.arxiv.org/abs/0909.2479. The main thing I see is there is so much published its difficult to know what is logically correct (e.g. the recent results showing reinhardt-rogoff 'austerity economics' results were due to data processing errors, or arxiv.org/abs/0906.0950) and what is meaningful (e.g. results on polling which are due to the way a question is phrased).
i see the Heidegger essay is on-line---it looks coherent and good. (The little I have glanced at Heidegger was it looked like Finnegan's Wake or Wittgenstein or Derrida, so this is surprising.)
I think I do agree with you on 'problem shifts'---this looks a bit like Kuhn's 'paradigm shifts'. What is interesting to me is how changing a formalism or notation (or technology) can lead to new results. Newton's 'fluxions' are not taught in calculus 1 anymore, and studies over time will show a caterpillar is the same as a butterfly (kafka). Alot of people these days suggest computers may make alot of mathematical analyses unnecesary---it will just be in the software, and so can be ignored just as humans don't have to think about metabolism in detail.
I read at times 'real world economics review' which is filled with 'heterodox economics' half of which seems to consist of diatribes against standard economics because it is mathematical. While there is no dought much of standard economics is wrong or imperfect, some of this view seems to come from a pure anti-math bias (people want to take over tenured faculty positions by waving hands and writing prose about economics rather than doing econometrics). I see this as throwing the baby out with the bathwater so you get the bathtub. Its like saying because Boltzmann was wrong---the world is not an ideal gas---or because some of Newtonian physics describes systems which are much more predictable than what is observable in the world---'new approaches' are needed. But their new bottles just add some nonlinear corrections to old wine (eg newtonian dynamics are often chaotic).
I think the anti-math attitude of heterodox critics of neoclassical economics is akin to certain atheist critiques of religion. They focus on the doctrine i.e. theology whereas their real beef is with social and political power these camps have. Likewise, academic Marxists like to criticize the "logic of capitalism" - which is supposed to be contradictory - even though their real beef is with power of capitalists in countries like the US. It is obvious when you realize that there is not much Marxist critique of capitalist logic in Sweden.
ReplyDeleteNeoclassical econ is a school of thought, or more accurately, a guild of professionals aiming to protect its market niche. They do it, inter alia, by rigid formalism of their discourse and strict gate keeping, which keeps the challengers at bay. It is no different than, say, a union hall giving jobs to members only. Most challengers fail to see this organizational dimension and instead focus on the doctrinal aspects i.e. "theology." The low predictive power of that doctrine is attributed to its excessive formalism i.e. "math" behind it.
The real problem, however, is that math is simply used by the economic guild as a gate keeper - just like Latin was the gate keeper for medieval theology. There is nothing wrong with Latin or math - the problem is in the guild-like organization of science and professions. If you happen to be in a profession that lacks such guild-like organization - e.g. sociology, psychology, business administration - those gate keeping mechanisms can be really annoying. They allow members of those guild-like disciplines to encroach on the turf of other disciplines, but limit the ability of those other disciplines to reciprocate.
In short, the critics should focus on the guild organization of the economic profession rather than the tools of their trade. Otherwise, it is like a dog biting a stick that hits him instead of biting the man who wields the stick.
BTW, I downloaded the paper on "Whorfian Economics" by Paul Chen and quickly perused it. It looks really interesting and I forwarded it to someone who specializes in linguistics. I cannot decide, at this point, whether it is linguistics informed by utility maximization model or the other way around. If the latter, this would make it behavioral economics rather than the neoclassical orthodoxy. I will think more about it when I have a chance to read the paper more closely. Thanks for the reference anyway.
thanks for the reply (attention---i usually get ignored, partly because of my style/typing, and i realize people can't read everything----as was my point (and i gather yours) about trying to figure out what is a good scientific paper ).
ReplyDeletei would read the blog entries on the 'whorfian economics' paper i mentioned---just google whorfian economics and 'language log' (a linguistics blog from u pa. with plenty of 'famous' people) and even better the 'replicated typo' blog.
i will say i do find chen's approach very interesting. However, if you look at the critiques---including by the authors of a hardcore linguistic classification scheme he used---he got it all wrong. he doesn't know his linguistics. also, to make his caae he had to come up with his own linguistic categories to 'fill in the dots'. It would be like if i was trying to correlate properties of elementary particles (eg some symmetry group like the 8-fold way of gelman) with Levi-Strauss' classification of kinship categories in cultures, and noticed there was a good fit in many places (a top quark corresponds to the shaman, the bottom quark to the warrior, etc.) and then since there were some missing links, i decided to supplement the physics with some new symmetries (particles) so i could have categories for other kinship classes.
my reading of the critiques is that this is what Chen has done. As noted the the linguists whose work he uses say he's off the wall. (just as if I called up Marray Gell-Man and told him i was using his 8-fold way to classify society, and he looked at my work, and noticed i had to make it a 16.74432 *** (-)7.333 way.
But then maybe Chen and I would have shown the originals were wrong.
Also, the replicated typo post shows the kinds of correlations Chen finds may be (or likely are) spurious. You can make many (possibly arbitrary) choices for linguistic categories and then correlate them with various social phenomena. "If you smoke menthol, you are more likely to go to jail".
That said, I think these excercizes in math/science are fine---but they are that. I'm all in favor of studying spherical cows to see how far you get. (Its amazing sometimes what you get with simple models, though they are deceptive because of non-isomorphism---you can get the same result from many approaches. (eg the income distribution is modeled pretty well using an ideal gas approximation from statistical mechanics---see Yakovenko on arxiv.org (or google statistical mechanics of money), but that doesnt mean that exponential type distribution arises that way; marx might not agree since in his delerium epiphenomena like 'classes' ,'struggle', capital, exploitation and profit motive arise in his scheme as opposed to perfectly rational identical particles which are hard spheres.
Similarily, i dont have a big issue with all the various GIGO yet rigorous studies in the literature as excercizes----except they are funded, create careers, use up funds which might better be used beyond making someone's careerfor and creating a guild/caste.
I might disagree that business admin, soc, psychol, etc. lack 'guild structure'. It may be less explicit but I think it exists. The guild structure of neoclassical econ is pretty strict (confuc(s)ion ritual) as is physics and math, and alot of biology (especially 'systems biology'---computational). As noted all that stuff i find as interesting as spherical cows, which you can gradually deform into reality, as rituals can morph into communication again (eg birds of paradise). The problem is too often they atrophy into rigor mortis.
My interests are mostly in empirical studies of human behavior and social relations - which have a great deal of randomness and unpredictability. I think this is what makes them interesting - it fuses art with science.
ReplyDeleteI am greatly influenced by cognitivists (such as Kahneman or Lakoff) and their focus on pre-rational determinants of how we think. It is how we frame or shape the black box and what goes into it and what is left out rather than how the information is processed after it gets inside of it. Also by Weberian sociological theory in which formal rationality is an ideal type (or ideal gas if you will), but the actual focus is on how empirical behavior differs from that ideal form and why.
As to Chen's paper - my understanding of it, which is based on a very quick reading, is that he used categories devised by linguists (EUROTYP) which he merely extended to languages not included in that taxonomy. I do not have sufficient knowledge of linguistics to pass any judgment on the merits of using this classification. Closer to my interest is the measurement of his dependent variable i.e. attitudes toward the future. Here is where the rubber really meets the road and the outcome depends how much traction or slack is in this connection.
Finally, regarding the guild structure of modern professions -
I agree that all professions are guilds or one kind or another, as you say. The purpose of any profession is to monopolize the market on their services and keep competitors at bay. However,
I distinguish between those professions that create guild and gate keeping structures for the most part themselves (e.g. lawyers or doctors), and those that depend on other institutions (e.g. academia or government) to create it. I would put business admin, soc, psyc, poli sci etc. in the second category. Econ, otoh, is closer to the former - they are more like doctors and lawyers, they do a great deal of gate keeping themselves, while the academia plays a relatively small part in it.
apparently u blocked me. i got a nice study from jhu sociology sortuh like whorfian ecs on my blog. enshallah and i aint no muslim or atheist..
ReplyDeletei changed my mind. i guess the power was out.
ReplyDeletelook at language log and replicated typo. (blogs) the linguists go through whorfian econ; chen has a ted talk too. still that spherical ciow is interesting as an excercize. (one can follow d'arcy thompson or norbert weiner/von neumann, and with a few parameters a kangaroo will dance on one ear. qed