Erick Schonfeld (2009)

From EmergiaWiki

Erick Schonfeld (2009, February 15). Mining The Thought Stream. TechCrunch. Retrieved February 16 from http://www.techcrunch.com/2009/02/15/mining-the-thought-stream/

[edit] Main ideas

• Peer into the thoughts of millions of people as they were thinking
• These thoughts immediately available in a database that could be mined easily
  • to tell you what people both individually and in aggregate are thinking right nowabout any imaginable subject or event?
• That would be a different kind of search engine altogether. A real-time search engine. A what’s-happening-right-now search engine.

[edit] Excerpts

In fact, the crude beginnings of this “now” search engine already exists. It is called Twitter, and it is a big reason why new investors poured another $35 million into the two-year-old startup on Friday.

Twitter is not the only company trying to solve this problem. Facebook, FriendFeed, and even Google are trying to crack it, but Twitter has a decided advantage in that it is capturing the vast majority of the real-time thought stream on the Web (because more people enter their thoughts directly into Twitter’s database than any other, and are doing so at an increasing rate).

What makes Google and other search engines so valuable is that they capture people’s intent—what they are looking for, what they desire, what they want to learn about. But they don’t do a great job at capturing what people are doing or what they are thinking about. For thoughts and events that are happening right now, searching Twitter increasingly brings up better results than searching Google.

Whether you want to know how people are mentally gearing up, [...] Twitter will give you a pretty good smattering of sentiment and opinion. It is also a lot faster at getting out the essential details about breaking news.

Twitter’s search engine is powered by Summize [...] but it also developed a feature called Track, that allowed people to follow the mention of specified keywords. John Borthwick, an investor in Summize (and thus now an investor in Twitter), explained in a blog post earlier this month ago why he thinks that “Twitter search changes everything.” Excerpt:

Imagine you are in line waiting for coffee and you hear people chattering about a plane landing on the Hudson. You go back to your desk and search Google for plane on the Hudson — today — weeks after the event, Google is replete with results — but the DAY of the incident there was nothing on the topic to be found on Google. Yet at http://search.twitter.com the conversations are right there in front of you. The same holds for any topical issues — lipstick on pig? — for real time questions, real time branding analysis, tracking a new product launch — on pretty much any subject if you want to know whats happening now, search.twitter.com will come up with a superior result set.

. . . How is real time search different? History isn’t that relevant — relevancy is driven mostly by time. . . . This reformulation of search as navigation is, I think, a step into a very new and different future. Google.com has suddenly become the source for pages — not conversations, not the real time web. What comes next? I think context is the next hurdle. Social context and page based context. . . . Twitter search today is crude — but so was Google.com once upon a not so long time ago.

In aggregate, what you get is a direct view into consumer sentiment, political sentiment, any kind of sentiment. For companies trying to figure out what people are thinking about their brands, searching Twitter is a good place to start. To get a sense of what I’m talking about, try searching for “iPhone,” “Zune,” or “Volvo wagon”.

[...] IVP partner, and Twitter investor, Todd Chaffee, suggests:

If they were really smart they could partner with Twitter and make Twitter their real-time feed.

Doing that would require Google to “affirm Twitter’s dominance in this category and the importance of the Twitter data stream,” contends Borthwick. [...]

An undifferentiated thought stream of the masses at some point becomes unwieldy. In order to truly mine that data, Twitter needs to figure out how to extract the common sentiments from the noise (something which Summize was originally designed to do, by the way, but it was putting the cart before the horse—you need to be able to do simple searches before you start looking for patterns). But what is the best way to rank real-time search results—by number of followers, retweets, some other variable? It is not exactly clear. But if Twitter doesn’t solve this problem, someone else will and they will make a lot of money if they do it right.