May 15, 2019

President’s Column: From Statistically Significant to Significantly Statistical

IMS President Xiao-Li Meng writes another President’s Column:

We statisticians have successfully—perhaps too successfully—taught everyone that the larger the size, the higher the power to lend credence to an alternative. This is evident from the 2017 Nature Human Behaviour’s “Redefine Statistical Significance,” which has over 70 authors, and from the 2019 Nature’s “Retire Statistical Significance,” with its more than 800 signatories. The statistical community’s organized responses regarding the troubled p-value have been led most visibly by American Statistical Association (ASA), via the 2016 ASA’s Statement on p-Values, the 2017 ASA Symposium on A World Beyond p<0.05, and the post-symposium special issue in The American Statistician (TAS 2019), with its 43 articles on what do to in a world in which p-value has been de-valued.

Given the increased attention to the issue of replicability, what can IMS contribute to the larger conversation? Inspired by a predecessor, I have a somewhat unusual idea, which requires your thoughtfulness in order to be consummated. So please, read on.

If the number 43 is too large for you (because you have taught many that n=30 is a good approximation for n=∞ under normal circumstances), the editorial of TAS 2019 by Wasserstein, Schirm, and Lazar is a gentle and humble tour guide. It summarizes the key recommendations by an ATOM: “Accept uncertainty. Be thoughtful, open, and modest.” Indeed, the thoughtfulness and modesty of our profession are well-reflected by the very fact that many statisticians endorse the call to abandon the term “statistical significance.” I have yet to identify another discipline with quite so many members who endorse the idea of abandoning its publicly most-recognized concept.

To a layperson, saying something is “statistically significant” is analogous to saying it is “mathematically proven” or “scientifically valid.” Such colloquial associations are in fact what motivated the call to abandon the term “statistical significance,” because the methods behind it are far less rigorous than mathematical proofs, and far too simplistic for establishing scientific validity. Yet we should not overlook the epistemological effectiveness of such confidence- inducing terms in promoting and sustaining the public awareness and appreciation of the societal relevance of a discipline (e.g., mathematics) or a collection of them (e.g., science). As Aristotle reminds us, our expectations of absolute exactitude should be qualified when it comes to matters of human opinion and action.

The question, then, is what alternative statistical concept could conceivably maintain the virtues of “statistical significance” without much of its vice? How about we simply drop the word “significance” ?Just as we question if a finding is scientific, a study is ethical, a project is economical, an action is legal, or a policy is moral, we can—and should—ask of any study, “Is it statistical?” While the concepts of being scientific, ethical, economical, legal, and moral are endlessly contested, they have considerable use as yardsticks in both common and specialized parlance. Experts and laypersons alike may ask “Is it X?” with the term “X” signifying what something is or is not. The point is not to lay down incontrovertible definitions but rather to open up questions about what “X” is. Indeed, the lack of such routine questioning would itself be a troubling sign for a society or a historical period.

I dare to suggest that in the light of the dramatically increased societal attention to data science, we should promulgate the use of “statistical” as a yardstick. “Unstatistical” studies can do much harm to our societies in both the short and long term, just as unethical studies or uneconomical projects can. The concept of being statistical will not be any more perplexing than any of the concepts mentioned above, and its pithiness will enhance its effectiveness in public discourse and research communications, as well as in private conversations. IMS, as the world’s leading learned society in foundational thinking and the building-up of statistics and probability, can play a vital role in framing its core rhetorical components. Indeed, to the best of my knowledge, “Is it statistical?” was first posed by Bernard Silverman, 2000–2001 IMS President (in a private conversation years ago), as a parallel to the question “Is it legal?” or “Is it ethical?”

In the spirit of “casting stones to attract jades” (테漏多圖 in Chinese), I list below my proposal on the virtues of being statistical, the practice of which should help to reduce the prevalence of irreplicable research findings. I purposefully set the bar high in order to provoke, and hence, I would be happy to praise a study as being “significantly statistical” if it demonstrates—with due diligence—all of the following virtues, as called for by the purposes and design of the study:

Discuss the collection, pre-processing, quality and limitations of the data, and the implications of these;

Elucidate, assess, and discuss data analysis and modeling assumptions, as well as their consequences;

Investigate and evince a good understanding of selection biases, confounding factors, and when/whether causal conclusions can be drawn;

Exhibit coherent probabilistic thinking and treatments of multivariate relationships and distributions;

Apply statistical methods with reasonable justifications and acknowledge their shortcomings;

Conduct appropriate uncertainty propagation, quantification, and representations;

Show good understanding of statistical principles, such as conditioning and the bias–variance trade-off.

A list of virtues can never be exhaustive. There are also other virtues that are critical for data science, but they are not purely or primarily statistical considerations. For example, it is a virtue to understand trade-offs between statistical and computational efficiency, to ensure computational stability and scalability, to consider carefully policy implications, and to describe the essential scientific background, etc.

An invitation to you

My list here is only an invitation for IMS members to contemplate what should be the core considerations of “statistical” or “significantly statistical”. I would greatly appreciate hearing from you. Please either comment (below) or send your thoughts to meng@stat.harvard.edu as I prepare for my IMS Presidential Address at JSM 2019.

Of course, I’d appreciate it most if we all can practice what we preach, by constantly asking ourselves, “Is my study statistical?

Share

Leave a comment

*

Share

Welcome!

Welcome to the IMS Bulletin website! We are developing the way we communicate news and information more effectively with members. The print Bulletin is still with us (free with IMS membership), and still available as a PDF to download, but in addition, we are placing some of the news, columns and articles on this blog site, which will allow you the opportunity to interact more. We are always keen to hear from IMS members, and encourage you to write articles and reports that other IMS members would find interesting. Contact the IMS Bulletin at bulletin@imstat.org

What is “Open Forum”?

In the Open Forum, any IMS member can propose a topic for discussion. Email your subject and an opening paragraph (to bulletin@imstat.org) and we'll post it to start off the discussion. Other readers can join in the debate by commenting on the post. Search other Open Forum posts by using the Open Forum category link below. Start a discussion today!

About IMS

The Institute of Mathematical Statistics is an international scholarly society devoted to the development and dissemination of the theory and applications of statistics and probability. We have about 4,500 members around the world. Visit IMS at http://imstat.org
Latest Issue