This will be the first of a series of posts I plan to publish over the next few weeks discussing the topic of Collective Intelligence. This series of posts will draw heavily from Santam Alag’s excellent book Collective Intelligence in Action.
These posts will present a conceptual overview of key strategies for programming CI, and will not delve into code examples. For that, I recommend picking up Alag’s book. You won’t be disappointed!
At a bare minimum, I’d like to discuss the following topics:
- Basic algorithms for applying Collective Intelligence
- Gathering Intelligence from User Interaction
- Calculating Similarity
- Extracting Intelligence from Tags
- Extracting Intelligence from Content
However, time allowing, I’d like to address some specific CI algorithms such as Content Filtering, Collaborative Filtering, Search, and Recommendation Engines.
This initial post will serve as an introduction to the world of Collective Intelligence (or CI). Future posts will focus more on programming Collective Intelligence in social web applications. These posts will be primarily of interest to web developers, but will also provide insight into anyone interested in how collective intelligence works in web applications.
What is Collective Intelligence?
To my knowledge, the term Collective Intelligence was first coined – in the sense we mean it here – in a seminal paper published by Tim O’Reilly titled What is Web 2.0, published in September 2005. In this paper, O’Reilly states the following:
The central principle behind the success of the giants born in the Web 1.0 era who have survived to lead the Web 2.0 era appears to be this, that they have embraced the power of the web to harness collective intelligence
I rather like Joshua Porter‘s comments which come close to capturing, IMO, the essence of Collective Intelligence. In his book Designing for the Social Web, Porter states that Collective Intelligence is all about:
[Aggregating] the individual actions of many people in order to surface the best or most relevant content. … Collective Intelligence is based on the idea that by aggregating the behavior of many people, we can gain novel insights.
Satnam Alag in his excellent book Collective Intelligence in Action, comments that the Collective Intelligence of Users in essence is:
- The intelligence that’s extracted out from the collective set of interactions and contributions made by your users.
- The use of this intelligence to act as a filter for what’s valuable in your application for a user.
The common thread is “aggregated opinion”. Quoting Porter:
Digg and other aggregation systems rely on the fact that while no individual is right all the time, in the collective a large number of users can be amazingly accurate in their decisions and behavior. Amazon, Digg, Google, Netflix, and many other sites base their recommendations of products, news, sites, movies, etc. on aggregated opinion.
One result of Web 2.0-style applications that use Collective Intelligence is that, to quote Tim O’Reilly, the applications get better the more people use them.
What has changed about the Web to make CI so important?
Once again, I’ll defer to Satnam Alag to set the context for the increasing importance of CI:
Web applications are undergoing a revolution.
In this post-dot-com era, the web is transforming. Newer web applications trust their users, invite them to interact, connect them with others, gain early feedback from them, and then use the collected information to constantly improve the application. Web applications that take this approach develop deeper relationships with their users, provide more value to users who return more often, and ultimately offer more targeted experiences for each user according to her personal need.
Web users are undergoing a transformation.
Users are expressing themselves. This expression may be in the form of sharing their opinions on a product or a service through reviews or comments; through sharing and tagging content; through participation in an online community; or by contributing new content.
This increased user interaction and participation gives rise to data that can be converted into intelligence in your application. The use of collective intelligence to personalize a site for a user, to aid him in searching and making decisions, and to make the application more sticky are cherished goals that web applications try to fulfill.
In a nutshell, the Web has become social.
How is Collective Intelligence used in Social Web Application?
The insights and patterns gleaned from Collective Intelligence are the product of algorithms of various degress of sophistication. Alag lists the following ways to harness Collective Intelligence in your application:
- Aggregate information lists
- Ratings, reviews, and recommendations
- User-generated content: blogs, wikis, message boards
- Tagging, bookmarking, voting, saving
- Tag Cloud navigation
- Analyze content to build user profiles
- Clustering and predictive models
- Recommendation engines
- Search
- Harness external content – provide relevant information from the blogosphere and external sites
Alag comments that:
Web applications that leverage Collective Intelligence develop deeper relationships with their users, provide more value to users who return more often, and ultimately offer more targeted experiences for each user according to her personal need.
Amazon, Yelp, Netflix, Google Search, Google News, Del.iciou.us, and Digg are just some of the more popular sites that leverage Collective Intelligence to target relevant content to their users.
Applying Collective Intelligence in your application
Alag states that there are three things that need to happen to apply collective intelligence in your application.
You need to:
- Allow users to interact with your site and with each other, learning about each user through their interactions and contributions.
- Aggregate what you learn about your users and their contributions using some useful models.
- Leverage those models to recommend relevant content to your users.
Joshua Porter refers to these three steps as:
- Initial Action
- Display
- Feedback
He provides the following table to illustrate the different forms these three steps take at various popular social websites:
Why should I care about Collective Intelligence?
Harnessing collective intelligence is critical to web-based business strategies in the Web 2.0 world. In Tim O’Reilly’s seminal paper defining the core characteristics of a Web 2.0 application, Collective Intelligence is positioned as a critical element. Dion Hinchcliffe also views Collective Intelligence as a core pillar of Web 2.0-based business strategies, as illustrated by the slide below (from a presentation he gave at Web2.0 Expo 2009):
What are the key technologies underpinning Collective Intelligence?
The key technologies underpinning Collective Intelligence significantly derive from two streams of research: Information Retrieval and Machine Learning. I won’t delve into these research areas in this post, but will try to briefly explore these topics as they pertain to CI in later posts.
In Summary
Well, that should serve as enough of an apetizer to introduce the topic. Much more to come.
glenn
Also in this series
- Collective Intelligence – Part 2: Basic Algorithms
- Collective Intelligence – Part 3: Gathering Intelligence from User Interaction
- Collective Intelligence – Part 4: Calculating Similarity
- Collective Intelligence – Part 5: Extracting Intelligence from Tags
