May 17, 2015

Terence’s Stuff: Ideas

Today, from Melbourne, I watched my friend, colleague and former (50%) PhD student Bin Yu being inducted into the US National Academy of Sciences in Washington DC. As a former President of the IMS, Bin Yu will be familiar to all of you, and I am sure you all shared my pleasure and satisfaction in this recognition of her research achievements. The time we worked together 28 years ago ranks among the happiest of my professional career. I wondered what it was that brought us together, and concluded that it was ideas.

Not long after I joined the Berkeley statistics faculty in 1987, I gave a talk about Jorma Rissanen’s notion of minimum description length (MDL) and its use in model selection. At the time Bin was working with Lucien Le Cam on a topic in empirical processes, and doing very well with it, but she showed an interest in MDL. Together we began reading Rissanen’s Annals of Statistics paper on stochastic complexity, which had appeared the year before. We also met up with Jorma, who lived and worked in nearby San José. Soon Bin was hooked on information theory in general and MDL in particular, beginning her many contributions to these topics.

My awareness of information theory went back to my first year at the University of Melbourne. I used to spend time between classes and labs in the library, mainly people-watching, but also dipping into books. One which fascinated me, to which I returned again and again, was Norbert Wiener’s Cybernetics.


There were so many ideas in that book, I’d put it back on the shelf with my head spinning. Fourier analysis, group theory, ergodic theory, statistical mechanics, Brownian motion, prediction, computers, communication, information, language, psychopathology… The list went on and on, all in one book. This was heady stuff for a freshman science student. I used to wonder whether the time would ever come when I could understand the mathematics, and perhaps properly appreciate the book.

Most of this went into cold storage for next four years, as I completed a fairly standard degree in math and stat. We did some Fourier analysis, and also some group theory, but didn’t connect the two as Wiener had done. In 1965 I went to teach at Monash University, and began a part-time PhD supervised by Peter Finch. He too was interested in ideas, including information theory, and in due course he got me to teach two undergraduate courses on that topic. I used Wolfowitz’s Coding Theorems of Information Theory, spiced up by Shannon’s classic for the introductory course, and Billingsley’s Ergodic Theory and Information for the advanced course. I loved the ideas, I enjoyed the theory, and it was fun (and quite crazy) trying to teach it to undergrads. I was usually just one lecture ahead of the best students. (My explanation: it was the 1960s.) Also at that time I became exposed to ideas of computability and (un)solvability—themes discussed by Gödel, Turing and others—and learned some Hilbert space theory. Bit by bit, I was getting the background to understanding Wiener’s book.

Fast-forward over 20 years, to when Jorma Rissanen visited Canberra. He talked about MDL and the circle of closely related ideas: the work of Kolmogorov on random numbers, of Chaitin on algorithmic complexity, and of Solomonoff on inductive inference. Statistical theory hadn’t seemed as interesting before, though I never dreamed that I would do research in the area. But soon after meeting Jorma I went to Berkeley, met up with Bin Yu, and they were able to help me contribute a little to the field. So thank you Norbert Wiener for inspiring me with your ideas, and similar thanks to Jorma Rissanen. It was my pleasure to share some of these ideas with Bin Yu, and get an enormous amount back in return.

I am hardly alone in being in love with ideas. But one thing I knew back in 1961, in that library during the breaks between my people watching, was that ideas are just ideas until you have the technical skills to do something with them. At that time I didn’t understand Fourier analysis on groups, the ergodic theorem, linear prediction and so on, and so my appreciation of Wiener’s message was necessarily very superficial. I’m not sure I’m ready even now, as I still know little about psychopathology, Bergsonian time or the firing of neurons. But whether it’s cybernetics or consciousness, evolution or the Big Bang, it is always a long journey to a deeper understanding of the fascinating ideas in science. If we take part of that journey with a brilliant person like Bin Yu, we are extraordinarily lucky. Thanks Bin, and congratulations!

Leave a comment



Welcome to the IMS Bulletin website! We are developing the way we communicate news and information more effectively with members. The print Bulletin is still with us (free with IMS membership), and still available as a PDF to download, but in addition, we are placing some of the news, columns and articles on this blog site, which will allow you the opportunity to interact more. We are always keen to hear from IMS members, and encourage you to write articles and reports that other IMS members would find interesting. Contact the IMS Bulletin at

What is “Open Forum”?

In the Open Forum, any IMS member can propose a topic for discussion. Email your subject and an opening paragraph (to and we'll post it to start off the discussion. Other readers can join in the debate by commenting on the post. Search other Open Forum posts by using the Open Forum category link below. Start a discussion today!

About IMS

The Institute of Mathematical Statistics is an international scholarly society devoted to the development and dissemination of the theory and applications of statistics and probability. We have about 4,500 members around the world. Visit IMS at
Latest Issue