Monday, April 20, 2020

The Malthusian problem for scientific research

It seems that there is a kind of inverse Malthusian structure to scientific research and knowledge. Topics for research and investigation multiply geometrically, while actual research and the creation of knowledge can only proceed in a selective and linear way. This is true in every field -- natural science, biology, social science, poetry. Take Darwin. He specialized in finches for a good while. But he could easily have taken up worms, beetles, or lizards, or he could have turned to conifers, oak trees, or cactuses. The evidence of speciation lies everywhere in the living world, and it is literally impossible for a generation of scientists of natural history to study them all.

Or consider a topic of current interest to me, the features that lead to dysfunctional performance in organizations large and small. Once we notice that the specific workings of an organization lead to harmful patterns that we care about a great deal, it makes sense to consider case studies of an unbounded number of organizations in every sector. How did the UAW work such that rampant corruption emerged? What features of the Chinese Communist Party led it to the profound secrecy tactics routinely practiced by its officials? What features of the Xerox Corporation made it unable to turn the mouse-based computer interface system into a commercial blockbuster? Each of these questions suggests the value of an organized case study, and surely we would learn a lot from each study. But each such study takes a person-year to complete, and a given scholar is unlikely to want to spend the rest of her career doing case studies like these. So the vast majority of such studies will never be undertaken. 

This observation has very intriguing implications for the nature of our knowledge about the world -- natural, biological, and social. It seems to imply that our knowledge of the world will always be radically incomplete, with vast volumes of research questions unaddressed and sources of empirical phenomena unexamined. We might take it as a premise that there is nothing in the world that cannot be understood if investigated scientifically; but these reflections suggest that we are still forced to conclude that there is a limitless range of phenomena that have not been investigated, and will never be.

It is possible that philosophers of physics would argue that this "incompleteness" result does not apply to the realm of physical phenomena, because physics is concerned to discover a small number of fundamental principles and laws about how the micro- and macro-worlds of physical phenomena work. The diversity of the physical world is then untroubling, because every domain of physics can be subsumed under these basic principles and theories. Theories of gravitation, subatomic particles and forces, space-time relativity, and the quantum nature of the world are obscure but general and simple, and there is at least the hope that we might arrive at a comprehensive physics with the resources needed to explain all physical phenomena, from black-hole pairs to the nature of dark matter.

Whatever the case with physics, the phenomena of the social world are plainly not regulated by a simple set of fundamental principles and laws. Rather, heterogeneity, exception, diversity, and human creativity are fundamental characteristics of the social world. And this implies the inherent incompleteness of social knowledge. Variation and heterogeneity are the rule; so novel cases are always available, and studying them always leads to new insights and knowledge. Therefore there are always domains of phenomena that have not yet been examined, understood, or explained. This conclusion is a bit like the diagonal proof of the existence of irrational numbers that drove Cantor mad: every number can be represented as an infinite decimal, and yet for every list of infinite decimals it is simple to generate another infinite decimal that is not on the list.

Further, in this respect it may seem that the biological realm resembles the social realm in these respects, so that biological science is inherently incomplete as well. Even granting that the theories of evolution and natural selection are fundamental and universal in biological systems, the principles specified in these theories guarantee diversification and variation in biological outcomes. As a result we might argue that the science of living systems too is inherently incomplete, with new areas of inquiry outstripping the ability of the scientific enterprise to investigate them. In a surprising way the uncertainties we confront in the Covid-19 crisis seem to illustrate this situation. We don't know whether this particular virus will stimulate an enduring immunity in individuals who have experienced the infection, and "first principles" in virology do not seem to afford a determinate answer to the question.

Consider these two patterns. The first is woven linen; the second is the pattern of habitat for invasive species across the United States. The weave of the linen is mechanical and regular; it covers all parts of the space with a grid of fiber. The second is the path-dependent result of invasion of habitat by multiple invasive species. Certain areas are intensively inhabited, while other areas are essentially free of invasive species. The regularity of the first image is a design feature of the process that created the fabric; the irregularity and variation of the second image is the consequence of multiple independent and somewhat stochastic yet opportunistic exploratory movements of the various species. Is scientific research more similar to the first pattern or the second?

I would suggest that scientific research more resembles the second process than the first. Researchers are guided by their scientific curiosity, the availability of research funding, and the assumptions about the importance of various topics embodied in their professions; and the result is a set of investigations and findings that are very intensive in some areas, while completely absent in other areas of the potential "knowledge space".

Is this a troubling finding? Only if one thought that the goal of science is to eventually provide an answer to every possible empirical question, and to provide a general basis for explaining everything. If, on the other hand, we believe that science is an open-ended process, and that the selection of research topics is subject to a great deal of social and personal contingency, then the incompleteness of science comes as no surprise. Science is always exploratory, and there is much to explore in human experience.

(Several earlier posts have addressed the question of defining the scope of the social sciences; link, link, link, link, link.)

No comments: