Now that the regular season has finished in the NBA, our Wisdom of Crowds predictions (see the mid-season post below, linked here) can be evaluated. We had predictions of the orders the 15 teams would finish in the Eastern and Western conferences, collected from 100s of people. Using these, we formed a standard aggregate prediction using the Borda count method, and a psychologically motivated prediction using what we call the Thurstone method. The actual predictions were detailed in our earlier post.

To measure the accuracy of the rankings, we used the standard Kendall Tau measure. This is basically a count of the number of pair-wise swaps of teams you need to turn to convert a prediction to the true final order. Lower taus are better, with zero being perfect prediction. The figure below shows a histogram of all the taus for the individual people’s predictions, and for the Borda and Thurstone aggregate predictions, for both conferences.

Out of the 172 people predicting the East, the Borda aggregate does as well or better than 126 (73%), while the Thurstone aggregate does as well or better than 156 (91%). Out of the 156 people predicting the West, the Borda aggregate does as well or better than 135 (87%), and the Thurstone aggregate does as well or better than 144 (92%). So, both Wisdom of Crowds methods did well, relative to people’s performance, and the Thurstone approach was better than the Borda approach.

We’ve just submitted a paper:

Zhang, S., & Lee, M.D. (submitted). Optimal experimental design for a class of bandit problems,

that applies a nice statistical framework, developed by Jay Myung and colleagues, to the problem of designing two-armed finite-horizon bandit problems.

We’ve just submitted a paper:

Lee, M.D., Zhang, S., & Shi, J. (submitted). The wisdom of the crowd playing the Price is Right.

looking at whether there it is possible to combine the four bids contestants make in the Price is Right, to give a more accurate estimate of the true value of the prize.

The nice finding is that using models of decision-making helps, because it lets you aggregate over what people know about the price, rather than what they say when they bid.

As the snippet below shows, sometimes people bid $1, not because they think that’s the right price, but because it’s a clever strategy to maximize their chance of winning. Also, apparently, people sometimes bid $420. As one of us originally said:

I think it’s rare to have a “lucky number” so big as 420. And I can’t think up a second reason why he was doing that!

Hemmer, P., Shi, J., & Steyvers, M. (submitted). The influence of prior knowledge on recall for height.

In this paper we explore how having general knowledge about heights of people, as well as specific knowledge about the height of men and women can influence recall for the height of a person.

We focus on naturalistic stimuli pertaining to the height of males and females because  there is a prominent difference in mean height between genders. In addition,  the stereotype of correlation between certain height ranges and gender is both accurate and universal and is drawn from real-world distributions and is consistent in nature.

Panel A shows how recall might be biased toward the overall mean population height. People at heights below the mean population height are overestimated and people at heights above the mean population height are underestimated. Read more »

The lab has been interested in the “Wisdom of the Crowds” phenomenon lately. This is the idea that forming “group” answers to difficult questions, by aggregating over the answers given by individuals, can lead to performance that is better than most or even the best individual in the group.

We are especially on how models of human cognition and decision-making can contribute to Wisdom of the Crowds research. In the end, the individual decisions that form the raw data are generated by people, and it seems understanding how the decisions are made should help.

A lot of people in the lab are also interested in sport. So, we just finished collecting the predictions of a bunch of people for the 15-team NBA Eastern and Western Conference ladders at the end of the regular season. We had 156 people for the West, and 172 for the East. Using the standard Borda count method for aggregation, here’s what came out as the group prediction: Read more »

This paper

• Lee, M.D., & Wetzels, R. (submitted). Individual differences in attention during category learning.

which we recently submitted to the Cognitive Science conference has an interesting result. It takes a standard condensation category learning task — in which it is usually assumed people pay attention to both stimulus dimensions, because they are both relevant to learning the category structure — and applies Nosofsky’s Generalized Context Model.

A standard analysis, assuming no individual differences, gives the standard divided attention result. But a re-analysis that allows for individual differences finds two groups of subjects, half of whom attend to one dimension, half of whom attend to the other, and basically none of whom divide their attention.

The posterior distributions over attention and generalization parameters for the two groups are shown in the top panels of the picture (click the thumbnail at the top of the post to get a good view), and their posterior predictives below that show their good fit to the qualitatively different category learning behavior of the two groups. Food for thought …