Now that the regular season has finished in the NBA, our Wisdom of Crowds predictions (see the mid-season post below, linked here) can be evaluated. We had predictions of the orders the 15 teams would finish in the Eastern and Western conferences, collected from 100s of people. Using these, we formed a standard aggregate prediction using the Borda count method, and a psychologically motivated prediction using what we call the Thurstone method. The actual predictions were detailed in our earlier post.

To measure the accuracy of the rankings, we used the standard Kendall Tau measure. This is basically a count of the number of pair-wise swaps of teams you need to turn to convert a prediction to the true final order. Lower taus are better, with zero being perfect prediction. The figure below shows a histogram of all the taus for the individual people’s predictions, and for the Borda and Thurstone aggregate predictions, for both conferences.

Out of the 172 people predicting the East, the Borda aggregate does as well or better than 126 (73%), while the Thurstone aggregate does as well or better than 156 (91%). Out of the 156 people predicting the West, the Borda aggregate does as well or better than 135 (87%), and the Thurstone aggregate does as well or better than 144 (92%). So, both Wisdom of Crowds methods did well, relative to people’s performance, and the Thurstone approach was better than the Borda approach.

We’ve just submitted a paper:

Zhang, S., & Lee, M.D. (submitted). Optimal experimental design for a class of bandit problems,

that applies a nice statistical framework, developed by Jay Myung and colleagues, to the problem of designing two-armed finite-horizon bandit problems.

The lab has been interested in the “Wisdom of the Crowds” phenomenon lately. This is the idea that forming “group” answers to difficult questions, by aggregating over the answers given by individuals, can lead to performance that is better than most or even the best individual in the group.

We are especially on how models of human cognition and decision-making can contribute to Wisdom of the Crowds research. In the end, the individual decisions that form the raw data are generated by people, and it seems understanding how the decisions are made should help.

A lot of people in the lab are also interested in sport. So, we just finished collecting the predictions of a bunch of people for the 15-team NBA Eastern and Western Conference ladders at the end of the regular season. We had 156 people for the West, and 172 for the East. Using the standard Borda count method for aggregation, here’s what came out as the group prediction: Read more »

This paper

• Lee, M.D., & Wetzels, R. (submitted). Individual differences in attention during category learning.

which we recently submitted to the Cognitive Science conference has an interesting result. It takes a standard condensation category learning task — in which it is usually assumed people pay attention to both stimulus dimensions, because they are both relevant to learning the category structure — and applies Nosofsky’s Generalized Context Model.

A standard analysis, assuming no individual differences, gives the standard divided attention result. But a re-analysis that allows for individual differences finds two groups of subjects, half of whom attend to one dimension, half of whom attend to the other, and basically none of whom divide their attention.

The posterior distributions over attention and generalization parameters for the two groups are shown in the top panels of the picture (click the thumbnail at the top of the post to get a good view), and their posterior predictives below that show their good fit to the qualitatively different category learning behavior of the two groups. Food for thought …

Welcome to the MADLAB Blog. We are the Memory and Decision-Making Laboratory, in the Department of Cognitive Sciences, at the University of California, Irvine.

This is the place our faculty, students and visitors post updates on their current research, papers, presentations, code, and everything else.