What’s Trending After Lunch? Geniuses Predict The Near Future On Twitter

Researchers at the genius factory that is the Massachusetts Institute of Technology have created a new algorithm they say that can predict which Twitter topics will start trending hours before they actually do. Developed by MIT Associate Professor Devavrat Shah and his student Stanislav Nikolov, the algorithm will be presented this month at MIT’s Interdisciplinary Workshop on Information and Decision in Social Networks.

Twitter’s famous “Trending Topics” sidebar provides a kind of snapshot of what people are talking about on the site at any given moment -- anything from a natural disaster like Hurricane Sandy to Miley Cyrus’s new haircut. The topics are determined by Twitter’s own internal algorithm, which factors in both the number of tweets about a topic and recent increases in that number.

In turn, Shah and Nikolov’s algorithm combs through data about Twitter topics that previously did, and did not, trend. Then it tries to find meaningful patterns within those topics. According to an article posted Thursday by Larry Hardesty of the MIT news office, the algorithm can predict which topics will trend with 95 percent accuracy. On average, the algorithm can predict trending topics an hour and a half before Twitter’s algorithm puts them on the list, and sometimes as long as four or five hours before.

Researchers believe the technology will be particularly useful for ad-supported websites, which rely on real-time analytics to tap into the mass consciousness of web surfers and maximize their traffic.

“People go to social-media sites to find out what’s happening now,” Ashish Goel, an associate professor at Stanford University, told MIT. “So in that sense, speeding up the process is something that is very useful.”  

Hardesty writes that the Twitter-predicting algorithm represents a new approach to statistical research, one that employs a non-parametric technique in which the algorithm does not assume that the shape of a pattern is fixed. In theory, the researchers say, this approach could be used to predict not just Twitter topics but any quantity that varies over time: from stock prices, to morning commute times, to the duration of the common cold.

Building a machine that can accurately predict future events has become one of the holy grails of science. For the last two years, European scientists have been working on Living Earth Simulator, a massive super-computer project that they believe will simulate all activity on planet earth, thereby giving us a window into the variables that lead to everything from droughts to pandemic outbreaks.

By comparison, predicting the latest hashtag meme and shout-outs to Justin Bieber might seem a little less ambitious. But it’s a start.

As part of MIT’s “News at Noon” program, Devavrat Shah will discuss his new algorithm on Friday, Nov. 9, at the MIT campus in Cambridge, Mass. You can find more information here.