This is the inaugural post in my blog, in which I argue that we must completely rethink the foundations of statistics and experimental design.

Since the middle of the twentieth century, philosophers of science have vehemently criticized what remains a very popular ldeal of science: the so-called "Value-Free Ideal." According to this ideal, scientists' decisions about which hypotheses to endorse should be dictated by the evidence alone, and importantly, should not be influenced by moral, political, and social values. Philosophers of science -- especially those in the feminist tradition (e.g, Longino and Wylie) -- have argued that the influence of moral and political values in scientific decision-making is not only inevitable; it is often better for science and more just.

What motivates rejecting the value-free ideal? No existing scientific theory or hypothesis is an exceptionless, perfectly accurate, and complete description of reality. Scientific "theories" are more accurately called "models", and different models serve many different purposes. For example, different theories might predict different phenomena to different degrees of accuracy. Thus, if different people have different needs and different desires, they might prefer different scientific models for their purposes. When scientists endorse a hypothesis deserves further investigation, or when they endorse a hypothesis be taken as granted in policy-making, their decision will affect many people whose interests might not be perfectly aligned. Critics of the value-free ideal argue, therefore, that scientists should incorporate moral and political values when endorsing hypotheses. Importantly, some such critics (e.g., Kitcher in Science, Truth, and Democracy) have argued that scientific decision-making should roughly adhere to the same democratic norms as other political decisions, as scientists' decisions (like policy decisions) may have wide-sweeping consequences for many members of society.

I believe that philosophers of science have failed to recognize an important consequence of rejecting the value-free ideal. Namely, rejection requires retrofitting the foundations of statistics and experimental design.

There are two common ways of justifying statistical procedures. The first is broadly what epistemologists would call "reliabilist": statistical estimators and hypothesis tests are justified to the extent that they maximize our chance of endorsing true hypotheses and minimize the frequency with we endorse false ones. The rejection of the value-free ideal shows us reliabilist foundations will not do: reliabilism mistakenly treat scientific models as if they were intended to be true descriptions of reality, and it fails to balance the different purposes that models might serve. The second foundation is decision-theoretic: statistical estimators and hypothesis tests are justified to the extent that they minimize some loss function that captures the researcher's interests. Typically, loss functions quantify something akin to "approximate truth" or "approximate predictive accuracy." In other words, a hypothesis or estimate is good to the extent that it is close to the true (or measured) value of some unknown parameter or quantity. A decision-theoretic foundation for statistics likewise will not do: it either mistakenly assumes that all parties affected by scientists' decisions share a loss function (i.e., share interest) or it treats a single researcher's loss function as if it could fairly guide scientific inquiry without regard for the interests of others.

The correct foundations for statistical must involve frameworks for collective choice, like social choice theory, bargaining theory, and game theory more broadly. Those are the mathematical frameworks that allow us to evaluate how to promote the diverse interests of all those affected by scientific research and to reconcile those interests when they conflict.

In my talk today at Arizona State University, I will explain how random sampling and randomized experiments can be justified within this collective choice framework. This part of the presentation is joint work with Johannes van Vliet. I will also discuss joint work with Kade Cicchella, in which we argue that, from the traditional decision-theoretic standpoint, it is often irrational to allow only evidence to guide your decisions. I then provide an alternative interpretation of what scientists and statisticians are doing when they publish statistical estimates that are intended to summarize what the "evidence supports".

Slides from my talk are here.