December 18, 2012 By Tanya Roscorla
An elective class arms students with big data analysis skills that are in high demand.
In general, people with big data analysis skills and real experience are hard to find, said Gilad Mishne, engineering manager of search at Twitter. And competition between companies for these skilled people is hot, especially in the Silicon Valley.
Because users send 400 million tweets a day, Twitter needs complex algorithms to analyze all that data. And it needs skilled engineers to do the job. So it partnered with UC Berkeley on a class that teaches students how to analyze data using real tweets.
The "Analyzing Big Data with Twitter" class included lectures from UC Berkeley professors and 15 volunteers from Twitter on the technology behind the social network. These lectures were video taped and are available online for anyone to watch.
Along with the lecturers, 12 volunteer mentors from Twitter worked with 40 undergraduate and graduate students as they built data-driven applications. For example, one team used tweets to find funky restaurants around campus. This application was something they could use the next day, and both the engineer supervising the team and the students had fun working on it.
"This is priceless," Mishne said. "The first thing I actually look at when I see a CV (curriculum vitae) is, 'Did this student do something real? Did he build something or did she build something with real data?' And this is exactly what I would look for — this kind of experience."
In one project, students analyzed Twitter interest graphs (who links to whom) and conversation graphs (who refers to whom), said Marti A. Hearst, professor in the UC Berkeley School of Information. Students made interesting visualizations for this assignment through simple graphing algorithms that showed hundreds of thousands of interests that people discussed online.
"To me, what was interesting was how much you can see about the different topics that do arise from who links to whom, especially if you're looking at more well-known people, not necessarily celebrities, but people who have a lot of influence in the twitter sphere," Hearst said.
These students could use their newly acquired skills in applications including public health, business and city planning. As more types of data and real time data are collected, analysts can see accurate trends in the spread of disease. They can understand what customers think about a product and trigger a response. And they can see where new fire stations or social services should be located based on where people are living and moving to.
But more students need to be trained to do this kind of work. And to help train these students, Mishne said he would like to repeat a class like this with UC Berkeley and other places that express interest.
This story was originally published at the Center for Digital Education website.
You may use or reference this story with attribution and a link to