Whilst the broader commercial world is obsessed with innovation and its potential to solve everything from global warming to America’s trade deficit with China, law firms are seen as ‘sleepy hollows,’ immune from, or ‘above’ such ‘fads’. Mention innovation to a lawyer and most will default to the debate about how best to charge their clients or whether they should incorporate or not – imagine if that were the primary driver for those charged with developing truly innovative products and services in the ‘real world’!

Why should we be interested?

Ultimately because clients have choices and firms seek to influence those choices by differentiating their services from their competitors through innovation. This can take the form of a new service as such, or changing any of the key elements which comprise your current ‘business model’ –more about business models later. Importantly most of us are content to view competition by simply driving the ‘efficiencies’ in our current model rather than considering any fundamental change in the model itself i.e. more of the same but doing it better/faster/cheaper than our competitors. Nothing wrong with that but that is not the sort of innovation that I am primarily concerned with. True innovation can be an industry destroyer, a game changer and economic history is replete with examples – more recently the decimation of the print media based newspaper business by the internet is a clear example. Such transformations have been occurring ever since the industrial revolution began when canals were replaced by railways and then by cars, when the whaling industry (then one of the biggest employers in the world) was destroyed by first gas and then the electric light.

We instinctively have difficulty in relating such upheavals to our own practices. The law firm model is steeped in history and little in fact has changed in its basic elements. I’m sure the investors in horse transport and silent films thought very similarly. The economy however is now primarily built around services and legal services is one of many such services and wherever there is value to be captured there are eager opportunists and ‘innovators’ not to mention legislators ready to play a similar role.

Innovation, particularly disruptive innovation is generally sourced from outside the industry itself but as IBM, GE and many others have demonstrated innovation can be driven from within. All innovation is not of course necessarily ‘good’ as the ‘innovative financial products’ developed by the finance industry have recently so clearly demonstrated but it is a ‘real and present danger’ to every sector of the economy – nothing and no-body is immune. The lessons to be learnt from our manufacturing brethren is that it is folly to assume that your current business model is the only way to operate – you too can innovate and be part of creating the next wave rather than simply being overwhelmed by it.

The question is not only one of whether law firms in their current form will survive but will their successors need/want lawyers as we currently understand that role, and even if they do, how much of the value they currently capture will find its way into their pockets in the future?

What is the current model?
If capturing maximum value is the primary driver it relies on:
• Limited equity partners
• High leverage (the ratio between the equity partners and other authors)
• High legislative/educational barriers to entry and competition
• ‘High touch’ management of clients and authors/potential partners
• High levels of ‘informational asymmetry’ between lawyers and clients.
• Lack of credible alternatives

Innovation should be considered in two broad contexts. The first is the ‘business model’ which the firm intends to use which encompasses all the key elements of how the firm proposes to create value for its client and capture some return for so doing. The second (which is really a sub-set of the first), is the development of new services which the firm intends to offer. The second is the more common focus for innovation, particularly in the world of ‘product creation’. But you don’t have to look far to see a whole range of services being offered, particularly over the net, that we didn’t know or think we needed – think Google, YouTube, EBay.

A useful diagram which helps you consider the broader context of innovation can be found in the work of Alex Osterwalder which is reproduced below.

The exercise then becomes one of analysing what you currently do as a practice across all of the nine key elements found in that model. In every one of those key elements there may exist opportunity for innovation and the development of a new model even if the key services you are offering remains the same. The current focus is repeatedly upon ‘revenue streams’ (pricing), but open your thinking up to consider the potential impact of outsourcing (dramatically reducing your cost structure), use of the internet, (changing both your conventional ‘distribution channels’, target customer etc).

You may of course develop a whole new service e.g. ‘carbon trading swaps’ or a new model that others or perhaps you can use e.g. franchising. The important point is to critically consider every aspect of that model and think deeply about what you could do and perhaps experiment with. Our legislation for instance has for many years permitted ‘multi-disciplinary’ practices (the partner network’), mentioned above but there are very few examples on the ground.

In short the probability of the emergence major disruptive business models and innovative services is increasing rapidly. The legal sector despite recent battering remains an attractive and profitable sector and generates $20 billion plus in revenue in Australia alone per year. There are many interests who would dearly like a slice of that action and law firms should be conscious of the fragility of their current models and the ease with which legislative changes alone might expose them to the ‘gales of creative destruction’ which Schumpeter so frequently spoke of that are the very basis and strength of our capitalist system.

Oh yes, and I almost forgot to mention that Google has announced it is getting into the business of providing ‘legal information’ on-line….now there’s some food for thought!

¹Duncan Hart is the principal of Duncan Hart Consulting with over 30 years of consulting and legal experience in Australia, the US, UK and China. He can be contacted at dh@duncanhartconsulting.com or 0414 562 267.

A few days ago, Timothy Geithner was in a Q & A session with the Joint Economic Committee on Capitol Hill.  Everyone had a single-use plastic water bottle at their seat.  Everyone.  This scene normally would seem very, well, normal.  But once you understand the perils of single-use plastic, once you’ve been exposed to the crazy reality of a product that lasts 500-1,000 years in the environment but which will be used for 5 minutes, you never see such scenes the same way again.  The illusion begins to become transparent as you visualize the life cycle of most plastic. If you’ve been following the Trash Island project, you likely aren’t able to see a water bottle cap in the street without imagining its journey into the sewer with the next rain, out to sea, floating inexorably toward an ocean gyre, ending in the stomach of a dead baby albatross who was fed the bottle cap lovingly by its mother.

Earlier this month, NextNow Collaboratory joined collaborator Plastic Pollution Coalition in San Francisco for dinner and discussion about the road ahead.  Those present included San Francisco Mayor Gavin Newsom, Fifth District Supervisor Ross Mirkarimi, who legislated the first-in-the-Nation ban on plastic bags in chain grocery stores and drug stores (which sparked similar legislation around the world from Oakland to Canada, Paris to Beijing), and Deborah Santana, supporter of peace and social justice. All were treated to a wonderful meal and a heavy dose of reality.  The Plastic Pollution website is a real education.  From Plastic Pollution 101:

Plastic is forever. Plastic, which is made from petroleum, is a material that the Earth cannot digest. Every bit of plastic that has ever been created still exists, except for a small amount that has been incinerated, releasing toxic chemicals.

Plastic has become a plague. In the environment, plastic breaks down into smaller and smaller particles that absorb toxic chemicals, are ingested by wildlife on land and in the ocean, and enter the food chain that we depend upon.

Plastic affects human health. Harmful chemicals leached by plastics are already present in the bloodstream and tissues of almost every one of us.

Recycling is not a sustainable solution. The reality is that most of our plastic waste is land filled, down cycled or exported to other countries. And tragically, millions of tons of plastic are poisoning our environment.

The Pacific Garbage Patch. This accumulation of plastic fragments in the ocean is just a symptom. The real garbage patch is here with us: in our stores and homes, and increasingly inside of our own bodies.

PPC is not the only organization reporting on the problem.  On November 17, CNN reported that:

• during a recent survey off the coast of California, researchers were shocked by density and pervasiveness of small plastic particles floating just below the surface;
• out of more than a hundred samples in a recent expedition, not a single net returned without little plastic particles.
• scientists say that much of the debris comes from land, rather than from ships. Litter from up to hundreds of miles inland can come from places like cars or storm drains and end up in the ocean.

Plastic Pollution is also about global warming.  Earlier this year, the Stop Trashing the Climate report, written by GAIA, the Institute for Local Self Reliance and Eco-Cycle, outlined compelling evidence that preventing waste is one of the fastest, cheapest, and most effective strategies available for combating climate change.

Please become informed and sign the Single Use Plastic Emergency Response (SUPER) Hero Pledge:

Refuse, Reduce, Reuse, Recycle

]]> http://emergentbydesign.com/2009/11/26/twitters-intelligent-welcome-to-web-3-0/ Thu, 26 Nov 2009 17:20:44 +0000 Venessa Miemis http://emergentbydesign.com/2009/11/26/twitters-intelligent-welcome-to-web-3-0/

“Collective Intelligence (CI) is the capacity of human collectives to engage in intellectual cooperation in order to create, innovate, and invent.”
- Pierre Levy + James Surowiecki + Mark Tovey

I wrote a post a few days ago, Is Twitter a Complex Adaptive System?, that proposed the idea that Twitter may be evolving into an entity of sorts, a collective intelligence. I’ve come across some new posts that are amplifying that meme, and I just want to keep the thoughtstream going.

Insight #1

I was reading an article by Nova Spivack from 2006 over on Ray Kurzweil’s site, titled The Third-Generation Web is Coming. In it, he lays out the evolution from Web 1.0 –> Web 2.0 –> Web 3.0, a more intelligent web “which emphasizes machine-facilitated understanding of  information in order to provide a more productive and intuitive user experience.”

He also lays out the key technology trends driving the evolution. Among them are Ubiquitous Connectivity (broadband, mobile internet), Network Computing (SaaS, P2P, cloud computing), and Open Information (open APIs, open-source software, OpenID).

Sound familiar?

Insight #2

Then this article from ReadWriteWeb passed through my tweetstream, The Future is all about Context: The Pragmatic Web. The author, Alisa Leonard-Hansen, paints a picture for “a highly relevant and individualized Web experience based on the ubiquity of our identity data.”

However, with the rise of the social Web, we see that what truly makes our online experiences meaningful is not necessarily the Web’s ability to approximate human language or to return search results with syntactical exactness….Rather, meaningful and relevant experiences now are born out of the context of our identities and social graph: the pragmatics, or contextual meaning, of our online identities. My Web experience becomes more meaningful and relevant to me when it is layered with contextual social data based on my identity. This is the pragmatic Web.

Then she goes into a different direction, talking about business models and Facebook. She finishes by just touching on the implications of the centralization of identity data, which is such an extremely important concept that I need to save that discussion for an entirely separate post. But to begin building your framework for understanding the importance of open technology, bookmark the Peer to Peer Foundation website and the P2P blog.

Dave Winer echoed the sentiment of context in his recent post, How (slowly) we add metadata to tweets, in reference to the new retweet feature on Twitter and issues about attribution. He links to an article by Alex Bowyer, A better design for twitter retweets, which is a very well thought out post that deserves to be read.

Insight #3

I then found this article by Dean Pomerleau, titled Twitter and the Global Brain. He starts by explaining that some recent evidence in neuroscience has suggested a new model for understanding how the brain is altered during learning. He then makes a correlation between that process and the emerging structure and function of the real-time web, i.e. Twitter.

Imagine a twitter user as a neuron. He/she makes the equivalent of a synapse with each of his/her followers. When a twitter user sends out a tweet, it is the equivalent of a neuron firing. Followers who receive the tweet decide whether to propagate the activity by retweeting the message, in a sense by deciding whether they too should fire in response to the tweet.

He acknowledges that this is not exactly how Twitter works yet, but he goes through some interesting examples of how it could work, and if that system became automated, it could signal the emergence of a type of Global Brain.

I think he’s done a very good job of distilling this concept into terms that are understandable to the layman.

Insight #4

Then, just this very morning, I discovered this site: information economy meta language. It’s mission is as follows:

The main mission of the Collective Intelligence Lab is to pursue theoretical, empirical and applied researches related to ieml. This general mission can be decomposed in three sub-tasks:
to develop the vocabulary and the grammar of the information economy metalanguage,
to design and build the technical and methodological tools that will encourage and spread ieml uses,
to exploit ieml-related tools and methods for the study of information economy, the improvement of knowledge management and the growth of collective intelligence.

On the site, I found this document by Pierre Levy , the team’s Director, titled From social computing to reflexive collective intelligence: The IEML research program (PDF), which appears to be due for publication in the Information Sciences journal in 2010. It’s a dense 25 page paper, but I see a lot of correlation between what he’s proposing and what I’m proposing with the Metathinking concept.

Essentially, he’s saying that we need a new language infrastructure for the web in order for it to evolve to the sematic web stage. I’ve been echoing that thought by proposing we need a new thought architecture in order to process the new types of information that we’re encountering.

This ‘new language’ Levy describes seems to be the necessary structure that would have to be created in order for Pomerleau’s ‘Global Brain’ to emerge.

Synthesis

This all seems to be brought together beautifully by the Guardian article, After social networks, what’s next?

In it, venture capitalist Peter Theil answers the question by asking us to evaluate what stage we’re at with social networks, and reminding us that we often don’t realize the implications of what we’re seeing even when it’s staring us right in the face.

So, based on the above insights, the pieces seem to be in place. Allow me to answer the question with this question:

Have we transitioned to Web 3.0?

_____

I’m so glad I was able to write this post on Thanksgiving. I’m thankful to my network for giving me the information that inspired this post. Thank you all!

This post made possible by:
@tonnet – tweeted Dave Winer article
@futurescape – RT @brainpicker to ReadWriteWeb article
@ideahive – tweeted Guardian article
@novaspivack – tweeted link to ieml site & Levy paper

People mentioned in this post:
Dave Winer @davewiner (Scripting News blog
Alex Bowyer @alexbfree
Nova Spivack @novaspivack (creator of Twine)
Michel Bauwens @mbauwens (creator of P2P Foundation)
Pierre Levy @plevy (his wikipedia page)
Ray Kurzweil – not actively on twitter yet, but here’s his site

Some contextual quotes to chew on:

To change something, build a new model that makes the existing model obsolete” – Buckminster Fuller

The future has already arrived. It’s just not evenly distributed. – William Gibson

The mark of a well educated person is not necessarily in knowing all the answers, but in knowing where to find them. -Douglas Everett

- @venessamiemis

Our victors' trophy - now with the Brixton Hive

Whew, it’s done. Hive Mind Challenge 1.0 has come and gone like a whirlwind.

You’ll remember that the Hive Mind Challenge is a game that Adrian Hon and I developed. It’s a cheating pub quiz – you’re allowed to use any means necessary to answer the questions, including the internet and calling friends. In other words, the game’s a test of collective intelligence.

I arrived early at our venue, the Coach & Horses in Soho – only to discover a case of EPIC FAIL: the WiFi was down.

For a pub quiz based entirely on the idea that people are supposed to use google to find their answers, this was a bad thing, but I was very impressed by the resilience of the crowd. People came in and settled with their 3G dongles and smartphones, ready to tackle the task.

We started with some easy questions, to light the fire:

Round 1 – Easy

10 easy questions – Trivia stuff, easily Googleable

1. What is the height of the Eiffel Tower? (324 metres)
2. Niagara Falls is on an outlet from which lake? (Lake Erie)
3. Where is the world’s tallest structure? (The Burj Dubai)
4. Which player has the most Premier League winners medals? (Ryan Giggs)
5. What is the full name of the chemical family banned by the UN’s Montreal Protocol? (CFC’s – chlorofluorocarbons)
6. Who is the richest person living in England? (Lakshmi Mittal, as noted on the Times Rich List 2009)
7. Which flag is the world’s oldest? (Denmark)
8. Who designed the video games Doom and Quake? (John Carmack)
9. What were the names of Christopher Columbus’ three ships? (Nina, Pinta, Santa Maria)
10. What is the world’s third-tallest mountain? (Kangchenjunga, Nepal/Tibet)

Round two became a little harder, as people had to engage a little more creative thinking:

Round 2 – Medium

10 harder questions – Not easy to Google

1. What is the minimum amount of time it would take to send a message from Earth to Mars, right now, to the nearest 10 seconds? (A: 8 min 15 sec)
2. What animal is depicted on page 7 of the November 7th Economist (A: A bull)
3. What’s this Song?  (A: New World, Devotchka – For this question, Adrian played a song on his laptop, and most people used Shazam to figure it out.)
4. What states in US border lake Michigan? (A: Illinois, Indiana, Michigan, Wisconsin)
5. If you stood at the corner of Calle San Martin and Calle de la Duena in San Martin de Valdeglesias, Spain, and dug a vertical shaft straight down through the Earth’s core, where would you surface? (Answer: Main Street/Pioneer Highway in Palmerston North, New Zealand)
6. What event was taken as a bad omen by Constantine XI, shortly before his death? (A: Moon rising in eclipse, 22 May 1463)
7. If you took the Trans-Siberian from Yaroslavski station at 21:35 on Tuesday, when would you arrive in Ulan Bator? (A: 7:35 on Sunday)
8. What’s this song 2? (A: She Drives Me Crazy – Fine Young Cannibals Though Adrian played this one as a MIDI file, so it was un-Shazamable and had to be figured out the old fashioned way.)
9. For this question, Adrian and I made the British Sign Language sign for ‘Coca Cola’ and had people translate it into spoken/written English.
10. For this question, we informed the teams that we were going to ring two payphones in ten minutes: one in Oxford Circus, one in Clapham. If someone answered them with a team’s name, they got the points. One team got to the Oxford Circus phone.

We stopped for an intermission and tallied up the points. Round 3 was even harder – we honestly didn’t know how many of these people would be able to answer. For this round, we gave them a sheet with the questions all on it at once, and half an hour to solve as many as they could.

Round 3 – Difficult (but slow)

10 questions, all posed simultaneously – Effectively unGoogleable

1. agtaq gufnx mbvrp eselx vurnm xsmqc aqzxa gakro altam yrvtn tpqzy vgnbx nofqw gonov? (A: this is a Vigniere cypher, and the answer is ’stationx’)
2. What is the total age of President Obama’s cabinet in days – excluding Cabinet-level officers? (A: 312698)
3. What is this location? (A: Krak des Chevaliers, in Syria)
   
4. When David Cameron returned to London from this year’s Conservative Party Conference, what platform did his train depart from? (A: Platform 7, though I can’t find the link to the video where this was apparent)
5. Whose name is this? (A: Amenirdis)
   
6. Why did the engineers of the longest continuous-span suspension bridge in the world change the blueprints after construction started? (A: The Kobe Earthquake happened during the construction of the Akashi-Kaikyop bridge in Japan, moving the pillars 1 metre apart but leaving the structure otherwise unharmed. The length of the main span was extended 1 metre to  1991 metres to accommodate the earth’s new position.)
   
7. What is this constellation’s latin name? (A: Cetus)
   
8. How far apart were Adrian and Philip on October 30th, 2009 at 3:30pm, to the nearest 100m? (A: 320.06km. You can get this one from our Flickr feeds.)
9. Identify this Beatles song? (Revolution)
   
10. When is the next visible pass of the International Space Station from the birthplace of ‘The Chief Designer’, in local time (16:25:53, November 26)

The last round was quick questions, with people calling out the answers:

Round 4 – Difficult (but quick)

10 questions, posed consecutively – Effectively unGoogleable, but experts would know

1. How do you say ‘Good Morning’ in Arabic? (A: Saba al-kheer)
2. What’s the 100th digit in pi? (A: 9)
3. What plant do these leaves come from?: (A: Acer Palmatum, common name Japanese Maple – the picture contains several varieties of the species)
   
4. If you looked West from the walls of Constantinople on April 6th, 1463, what would you have seen? (A: Mohammed the conqueror and his army of turks.)
5. How many incorporated cities in the US have a population over 2 million? (4: NY, LA, Chicago, Houston)
6. What does this mean? 諜 (Spying/intelligence)
7. How many people have more foreign policy experience than Sarah Palin on Facebook, to the nearest thousand? (A: about 223,000)
8. What’s the score in the Fulham vs. Blackburn game? (A: this was an annoying question, as Fulham had the impertinence to score just as we were asking it. So 2-0 and 1-0 were both correct.
   
9. Who has the most Twitter followers in this room? (A: @jemimah_knight)

Overall, it was a fun evening and I had a great time. I think our players were magnificent, especially considering the bandwidth issues with the wonky WiFi. But as one of the Brixton Hive put it, “Usually messing about trying to get on the internet is annoying – but this time it just made it feel all MacGyver.”

We sure had fun, and we’ll be putting lots of improvements into Hive Mind Challenge 2.0.

See you there . . .

Figure – Book cover of Toby Segaran’s, “Programming Collective Intelligence – Building Smart Web 2.0 Applications“, O’Reilly Media, 368 pp., August 2007.

{scopus online description} Want to tap the power behind search rankings, product recommendations, social bookmarking, and online matchmaking? This fascinating book demonstrates how you can build Web 2.0 applications to mine the enormous amount of data created by people on the Internet. With the sophisticated algorithms in this book, you can write smart programs to access interesting data-sets from other web sites, collect data from users of your own applications, and analyze and understand the data once you’ve found it. Programming Collective Intelligence takes you into the world of machine learning and statistics, and explains how to draw conclusions about user experience, marketing, personal tastes, and human behavior in general — all from information that you and others collect every day. Each algorithm is described clearly and concisely with code that can immediately be used on your web site, blog, Wiki, or specialized application.

{even if I don’t totally agree, here’s a “over-rated” description – specially on the scientific side, by someone “dwa” – link above} Programming Collective Intelligence is a new book from O’Reilly, which was written by Toby Segaran. The author graduated from MIT and is currently working at Metaweb Technologies. He develops ways to put large public data-sets into Freebase, a free online semantic database. You can find more information about him on his blog:  http://blog.kiwitobes.com/. Web 2.0 cannot exist without Collective Intelligence. The “giants” use it everywhere, YouTube recommends similar movies, Last.fm knows what would you like to listen and Flickr which photos are your favorites etc. This technology empowers intelligent search, clustering, building price models and ranking on the web. I cannot imagine modern service without data analysis. That is the reason why it is worth to start read about it. There are many titles about collective intelligence but recently I have read two, this one and “Collective Intelligence in Action“. Both are very pragmatic, but the O’Reilly’s one is more focused on the merit of the CI. The code listings are much shorter (but examples are written in Python, so that was easy). In general these books comparison is like Java vs. Python. If you would like to build recommendation engine “in Action”/Java way, you would have to read whole book, attach extra jar-s and design dozens of classes. The rapid Python way requires reading only 15 pages and voila, you have got the first recommendations. It is awesome!

So how about rest of the book, there are still 319 pages! Further chapters say about: discovering groups, searching, ranking, optimization, document filtering, decision trees, price models or genetic algorithms. The book explains how to implement Simulated Annealing, k-Nearest Neighbors, Bayesian Classifier and many more. Take a look at the table of contents (here: http://oreilly.com/catalog/9780596529321/preview.html), it does not list all the algorithms but you can find more information there. Each chapter has about 20-30 pages. You do not have to read them all, you can choose the most important and still know what is going on. Every chapter contains minimum amount of theoretical introduction, for total beginners it might be not enough. I recommend this book for students who had statistics course (not only IT or computing science), this book will show you how to use your knowledge in practice _ there are many inspiring examples. For those who do not know Python - do not be afraid _ at the beginning you will find short introduction to language syntax. All listings are very short and well described by the author _ sometimes line by line. The book also contains necessary information about basic standard libraries responsible for xml processing or web pages downloading. If you would like to start learn about collective intelligence I would strongly recommend reading “Programming Collective Intelligence” first, then “Collective Intelligence in Action”. The first one shows how easy it is to implement basic algorithms, the second one would show you how to use existing open source projects related to machine learning.

“I think this participative technology, social software as people call it can transform individual lives, firms, government and it is not all about sort of broad capitalist attitudes. I think it can affect some of the things people really care such as health education education, welfare.”, JP Rangaswami.

Directed by Ivo Gormley, Us Now is a 60 min. documentary about the power of mass collaboration, government and the internet. Question is:  In a world in which information is like air, what happens to power? New technologies and a closely related culture of collaboration present radical new models of social organisation. This project brings together leading practitioners and thinkers in this field and asks them to determine the opportunity for government.

Who else can we thank?

Another way of saying it:

God is not on the Left, Right, or Center.

I know that is blasphemy to millions. But they seem to be worshiping tribal gods. Tribal gods are OK, but brittle.

Thank God, or the creative nature of the universe, that we are not all on the right, or all on the left, or all in the center. We are only all human, and only all earthlings.

If we were all on the right, or left, we would certainly hurtle over the cliff of our choosing, even faster than we’re now approaching the cliff that no one is choosing.

If we were all in the center, we would still be getting closer and closer to the non-chosen cliff, because there is too much investment in the status quo, and too little appetite for change.

But we’re not all in one place, one ideology, one proclivity, one style, one perspective.

That’s not a design flaw; that IS the design.

Even though that’s the design, we rail against it.

Even though, by our nature, we’re not all of one perspective, we’re in huge trouble now because enough of us haven’t waked up to realize that all our perspectives and peculiarities are needed to survive and thrive.  Or, that is, all are needed to be part of our on-going conversations, deliberations, adaptations, and innovations.

My heroes (dual gender) are those who have waked up to that, and who are able to speak well, and model well, in ways that are convincing to large segments of center, right, and left. Not just talking and inspiring (though those are important) but demonstrating – not as in protesting, but as in showing how.

At Amazon.com, you click ‘more info’ for the book you have been looking for. Even before you can read the reviews, there is a bar that says ‘Customers who bought this product also bought’, followed by a list of examples of books in the same genre. This list is an aggregation of hundreds, maybe thousands of people clicking on any kind of links. Eventually, companies organize this so-called collective intelligence and applies it wherever it is appropriate.

This whole process is a smart way of advertising other, yet similar products and it is probably rather successful. However, by aggregating every single click, people are being trivialized. You might simply be buying a gift for a friend online, something you would never consider buying. The next time when you are shopping online at this particular webpage, a list of products will be offered to you, most of which you do not care for¹.

Even though in some cases these reference points might be helpful for you to spend more money, in other cases it can be rather annoying. It becomes even more annoying once companies start sending you newsletters with special offers for those uninteresting products mentioned before. Simply because companies store you in a specific group; once again they trivialize you.

As stated before, collective intelligence will probably lead to higher profits for companies one way or another. Therefore, in today’s capitalist world, it is not likely to disappear. As a result, we should probably just become temporarily blind and ignorant when we are being trivialized in the media.

¹ On the entire homepage of http://www.amazon.com, there are a total of three subtle suggestion options consistent with your previous clicks (‘More Items to Consider, Customers with Similar Searches Purchased and Your Recent History: Continue Shopping).

Breakthrough Ideas and Gadgets Range From Simple Designs to Hi-tech Product Designs.

For the past few months, I have been writing mostly on urban vertical farming start-up , urban underground farming business concept and Japan  social enterprise start-up model. Well, let’s take a break this time.

This time round we will just focus on winning innovative product design concepts, ideas, solutions in a wide ranging field of modern innovations and technologies. For some of us, it’s surprising to know that sometimes a simple idea and concept can make a lot of difference to our modern day lives.

      Source: PicApp - An Earth Hour event held in Sydney Harbour, Australia. 

Sometimes simple Green ideas and Green inventions is useful in  reducing global warming, carbon footprint,  household energy consumption reduction,  household clutter organization and implementing global food security crisis solutions. Yes, this is just some simple everyday living examples.

Lots of today’s modern day convenience is brought by simple idea and great innovation product breakthrough. TIME’s 50 Best Inventions of 2009 online innovation contest features a wide ranging innovation product concept, designs and inventions. 

Peer learning request

James Ang, the Gen Y blogger of The omniGens Blog, acknowledges that there is other online innovation poll and innovation product design competition like International Design Excellence Award Winners 2009, IDEA 2009, that he may not know of. Please share with us if you know any of these innovation online polls and product design award contest. Thanks.

[James Ang is a Gen Y Blogger (or Gen Y Pro blogger - stands for Professional Blogger) who blogs mainly on innovation, peer learning,  personal development and well beings.]

About The omniGenerations Blog’s Concept

The omniGenerations Blog (or The omniGens blog in short) is a peer learning community blog or a peer learning micro wiki blog  that focus on the holistic development of our lives. The omniGens Blog leverages on the collective wisdom of our proactive community in helping us to achieve simple work – life balance and happiness. Make Life Simple.

PLEASE TAKE NOTE: All comments need to be approved before appearing in the contribution box / comment. Time and space constraints may prevent all comments from appearing. We will only approve comments directly related to this blog article and this blog. Please use appropriate language, exercise restrain,  responsible, sensitivity, tolerance, care and courtesy in your contribution. For more details, please refer to The omniGens Blog’s purpose and objective.

Copyright 2009 James Ang. All rights reserved.

What skills will journalists need in 2020?

This was one of the questions we discussed on Monday morning at Open ‘09. Will journalists need to know how to code HTML? To shoot and edit video for the web? To write stories in 140 characters? The more we talked, the more one simple answer coalesced in my mind.

They’ll need to understand collective intelligence.

Collective intelligence is the number one skill journalists need today, let alone 2020. It’s the ability to exploit networks effectively, to take advantage of the skills, knowledge and expertise distributed through the loose networks of people every one of us is plugged into.

Connections

Journalism has a lot of crafts – writing, shooting pictures, and so on – but the core craft that underlies all of this has always been about connections. Being a good journalist means, above all, creating and tending a network of contacts who trust you and who will be interested in talking to you when something important is going on. The writing skills, the photography or whatever aren’t really of much use unless you’ve got the nous and the network in place to pick the stories up in the first place.

And that’s the one part of journalism that is being changed more profoundly than any other – because of social media.

One of the reasons that social media is so popular is because it allows us to manage relationships with a lot of people efficiently. For journalists, this is of supreme importance. But this isn’t just another way of communicating and socializing (though that is very important in its own right).

Because it is so good a communications medium, social media allows us to do something quite different, and that’s to enact a sort of collective intelligence. Social media allows us to communicate widely and interactively with large groups of people that we know only tenuously, or even not at all. It is the most effective and least costly means of group formation and coordination – a subject explored in depth in Clay Shirky’s Here Comes Everybody. This technology makes communication so efficient that it actually makes possible entirely new behaviors and ways of working. Wikipedia is the classic example of this, of course. No one would have thought it possible before Wikipedia was published, and yet it is now the first reference of choice for many people.

In news, social media means that we can magnify our capabilities in information gathering and dissemination. For example, if I sent you a two-minute phone camera video shot in Arabic, how long would it take you to find someone to translate it for you?

If you’re working without social media, it could take hours. Depends if you personally know someone who speaks Arabic. But if you don’t? You need to start calling around contacts, and their contacts, and . . . this takes time.

A journalist adept in collective intelligence should be able to get a two-minute video translated from any language to any language inside an hour. I’d say fifteen minutes for common languages. I don’t have that many twitter followers, and even I can get a message out to a few hundred people instantly. With re-tweets, a message I send could reach thousands of people in minutes. That’s a lot of minds to tap.

This network of expertise and knowledge is immensely valuable. This is the hive mind, the collective intelligence. When it is accessed and focused in the appropriate way, it can solve almost any problem. Players of alternate reality games have proven this, by doing things like writing entire books and cracking military codes without payment, without command structures, or for any other purpose than their own enjoyment.

Journalists gain a powerful skill if they learn how to be a part of this collective intelligence, to work with it and within it. It is an ability and an attitude as much as a skill, but it can be learned (that’s one part of the thinking behind the Hive Mind Challenge).

Writing, photography, video and all the other skills of journalism production can work with the material gleaned from collective intelligence, but it remains the base.

And above all, this is what journalists in 2020 will need to be: masters of the hive mind, the focal points of collective intelligence. They will need to be constantly permeating the network, collecting disparate scraps of wisdom and collating them into powerful insights. They will need to be expert network tenders and community managers.

Heck, they need to be this way now.

Jimmy Wales just passed through my twitterstream earlier with this:

jimmy_wales New hashtag meme? #wikiwednesday – tweet about your favorite wiki, your favorite wikipedia page, crazy stuff others may not have seen.

and it made me think about how people are using Twitter Chat and hashtags to have conversations about topics. Let’s call these “tweetstorms”……. brainstorming sessions that happen on Twitter.

And what happens in a tweetstorm? Wikithinking! Ideas are being thrown out, chewed on, edited, refined, distilled…. and, dare I say, innovated?

___

Referenced:
Jimmy Wales @jimmy_wales

Just saw this in the twitterstream

Scobleizer Lots of good stuff on Twitter today: http://twitter.com/scobleiz… are my favorite tweets out of thousands.

so I clicked through and had to admit, there was some really good stuff in there. So I clicked around on the favorites lists of a few of the people that tweet awesome content from my twitterstream, and found the results quite mixed. Many of them are not added to regularly, and I was surprised to see that one of my favorite info sources has never favorited a single tweet.

I wonder if this is a habit we could develop to help us find even better info… if we set up Lists to organize people into certain interest groups, and if those people are consistently adding the content they scour into their favorites category, pretty soon we have a really great set of resources.

___

referenced:

Robert Scobe @scobleizer

thoughtstream update:

notthisbody @VenessaMiemis I use favorites religiously. i probably have many more than tweets. haven’t yet tried to publish an rss feed of them…hmmm

This is the fifth of a series of posts on the topic of programming Collective Intelligence in web applications. This series of posts will draw heavily from Santam Alag’s excellent book Collective Intelligence in Action.

These posts will present a conceptual overview of key strategies for programming CI, and will not delve into code examples. For that, I recommend picking up Alag’s book. You won’t be disappointed!

Click on the following links to access previous posts in this series:

• Part 1: Introduction
• Part 2: Basic Algorithms
• Part 3: Gathering Intelligence from User Interaction
• Part 4: Calculating Similarity

Introduction

So far in this series of posts, we’ve been introduced to some basic algorithms in CI, looked at various forms of user interaction, and explored how we used term vectors and similarity matrices to calcuate the similarity between users, items, and items and users. In this post, we’ll explore how to gather intelligence from tags.

Alag introduces the topic of gathering intelligence from tags as follows:

Users tagging items—adding keywords or phrases to items—is now ubiquitous on the web. This simple process of a user adding labels or tags to items, bookmarking items, sharing items, or simply viewing items provides a rich dataset that can translate into intelligence, for both the user and the items. This intelligence can be in the form of finding items related to the one tagged; connecting with other users who have similarly tagged items; or drawing the user to discover alternate tags that have been associated with an item of interest and through that finding other related items.

With that introduction, let’s begin.

Introduction to Tagging

Quoting Alag:

Tagging is the process of adding freeform text, either words or small phrases, to items. These keywords or tags can be attached to anything in your application—users, photos, articles, bookmarks, products, blog entries, podcasts, videos, and more.

[Previously] we looked at using term vectors to associate metadata with text. Each term or tag in the term vector represents a dimension. The collective set of terms or tags in your application defines the vocabulary for your application. When this same vocabulary is used to describe both the user and the items, we can compute the similarity of items with other items and the similarity of the item to the user’s metadata to find content that’s relevant to the user.

In this case, tags can be used to represent metadata. Using the context in which they appear and to whom they appear, they can serve as dynamic navigation links.

In essence, tags enable us to:

1. Build a metadata model (term vector) for our users and items. The common terminology between users and items enables us to compute the similarity of an item to another item or to a user.
2. Build dynamic navigation links in our application, for example, a tag cloud or hyperlinked phrases in the text displayed to the user.
3. Use metadata to personalize and connect users with other users.
4. Build a vocabulary for our application.
5. Bookmark items, which can be shared with other users.

Content-based vs. Collaborative-based Metadata

Alag emphasizes the distinction between content-based and collaborative-based sources of metadata. Quoting Alag:

In the content-based approach, metadata associated with the item is developed by analyzing the item’s content. This is represented by a term vector, a set of tags with their relative weights. Similarly, metadata can be associated with the user by aggregating the metadata of all the items visited by the user
within a window of time.

In the collaborative approach, user actions are used for deriving metadata. User tagging is an example of such an approach. Basically, the metadata associated with the item can be computed by computing the term vector from the tags—taking the relative frequency of the tags associated with the item and normalizing the counts.

When you think about metadata for a user and item using tags, think about a term vector with tags and their related weights.

Categorizing Tags based on how they are generated

We can categorize tags based on who generated them. There are three main types of tags: professionally generated, user-generated, and machine-generated.

Professionally generated Tags

Again quoting Alag:

There are a number of applications that are content rich and provide different kinds of content—articles, videos, photos, blogs—to their users. Vertical-centric medical sites, news sites, topic-focused group sites, or any site that has a professional editor generating content are examples of such sites.

In these kinds of sites, the professional editors are typically domain experts, familiar with content domain, and are usually
paid for their services. The first type of tags we cover is tags generated by such domain experts, which we call professionally generated tags.

Tags that are generated by domain experts have the following characteristics:

• They bring out the concepts related to the text.
• They capture the associated semantic value, using words that may not be found in the text.
• They can be authored to be displayed on the user interface.
• They can provide a view that isn’t centered around just the content of interest, but provides a more global overview.
• They can leverage synonyms—similar words.
• They can be multi-term phrases.
• The set of words used can be controlled, with a controlled vocabulary.

Professionally generated tags require a lot of manpower and can be expensive, especially if a large amount of new content is being generated, perhaps by the users. These characteristics can be challenging for an automated algorithm.

User-generated Tags

Back to Alag:

It’s now common to allow users to tag items. Tags generated by the users fall into the category of user-generated tags, and the process of adding tags to items is commonly known as tagging.

Tagging enables a user to associate freeform text to an item, in a way that makes sense to him, rather than using a fixed terminology that may have been developed by the content owner or created professionally.

[For example, considering the tagging processes] at del.icio.us. Here, a user can associate any tag or keyword with a URL. The system displays a list of recommended and popular tags to guide the user.

The use of users to create tags in your application is a great example of leveraging the collective power of your users. Items that are popular will tend to be frequently tagged. From an intelligence point of view, for a user, what matters most is which items people similar to the user are tagging.

User-generated tags have the following characteristics:

• They use terms that are familiar to the user.
• They bring out the concepts related to the text.
• They capture the associated semantic value, using words that may not be found in the text.
• They can be multi-term phrases.
• They provide valuable collaborative information about the user and the item.
• They may include a wide variety of terms that are close in meaning.

User-generated tags will need to be stemmed to take care of plurals and filtered for obscenity. Since tags are freeform, variants of the same tag may appear. For example, collective intelligence and collectiveintelligence may appear as two tags.

[Additionally,] you may want to offer recommended tags to the user based on the dictionary of tags created in your application and the first few characters typed by the user.

Machine-generated Tags

Tags or terms generated through an automated algorithm are known as machine-generated tags. Alag provides several examples in his book of extracting tags using an automated algorithm – for example, generating tags by analyzing the textual content of a document.

Again from Alag:

An algorithm generates tags by parsing through text and detecting terms and phrases.

Machine-generated tags have the following characteristics:

• They use terms that are contained in the text, with the exception of injected synonyms.
• They’re usually single terms—Multi-term phrases are more difficult to extract and are usually done using a set of predefined phrases. These predefined phrases can be built using either professional or user-generated tags.
• They can generate a lot of noisy tags—tags that can have multiple meanings based on the context, including polysemy and homonyms.—For example, the word gain can have a number of meanings—height gain, weight gain, stock price gain, capital gain, amplifier gain, and so on. Again, detecting multiple-term phrases, which are a
  lot more specific than single terms, can help solve this problem.

In the absence of user-generated and professionally generated tags, machine-generated tags are the only alternative. This is especially true for analyzing user-generated content.

How to leverage Tags in your application

Alag leads off this section of his book with the following:

It’s useful to build metadata by analyzing the tags associated with an item and placed by a user. This metadata can then be used to find items and users of interest for the user. In addition to this, tagging can be useful to build dynamic navigation in your
application, to target search, and to build folksonomies. In this section, we briefly review these three use cases.

I’m not going to explore the specific use cases that Alag covers in his book. Again, you know where to find the details.

Other topics

Alag concludes his chapter on extracting intelligence from tagging with:

1. An example that illustrates the process of extracting intelligence from user tagging, and
2. Thoughts on building a scalable persistence architecture for tagging

Exploring the tagging example and Alag’s thoughts on a persistence architecture for tagging is beyond the introductory scope of this post. Please see Alag’s book for more information.

In Summary

Hopefully this post has given you a bit of a flavor of how Tags are used to surface collective intelligence in a social web application. In the final post in this series, I’ll be exploring extracting intelligence from textual content.

Also in this series

• Collective Intelligence – Part 1: Introduction
• Collective Intelligence – Part 2: Basic Algorithms
• Collective Intelligence – Part 3: Gathering Intelligence from User Interaction
• Collective Intelligence – Part 4: Calculating Similarity
]]> http://emergentbydesign.com/2009/11/17/is-twitter-a-complex-adaptive-system/ Tue, 17 Nov 2009 18:55:39 +0000 Venessa Miemis http://emergentbydesign.com/2009/11/17/is-twitter-a-complex-adaptive-system/



I’ve seen a bunch of posts bubble up over the past few days that are really sparking my curiousity about what is really going on with Twitter, so I need to do a little brain dump. Bear with me.

Insight #1

An article by Rosabeth Moss Kanter was just published today on the Harvard Business Review website, titled On Twitter and in the Workplace, It’s Power to the Connectors. In it, she highlights the fact that there is an organizational trend moving away from the hierarchical networks of the 20th century, and towards complex, distributed, non-hierarchical structures of business organization and leadership.

She also points out that success today is based on a person’s ability to leverage power and influence within their social networks, to act as “connectors” between people and information, and in turn build social capital.

She leaves the evaluation of the significance of Twitter open-ended, but she lays out a few characteristics of Twitter that I found most interesting:

In the World According to Twitter, giving away access to information rewards the giver by building followers. The more followers, the more information comes to the giver to distribute, which in turn builds more followers. The process cannot be commanded or controlled; followers opt in and out as they choose. The results are transparent and purely quantitative; network size is all that matters. Networks of this sort are self-organizing and democratic but without any collective interaction.

(just keep those points in mind, I’m going to come back to it)

Insight #2

Also published today over on Stowe Boyd’s blog, /Message, was a post titled The Rise Of Networks, The End Of Process. He makes a case for the abandonment of worn out systems of industrial management thinking, and a move towards a social way of structuring work.

He points out that the explosion of the social web is allowing us to connect with others in a previously impossible way, and the ability it’s giving us to share information and ideas is actually reforming our learning process and the way we think:

People are thronging on social sites like Facebook and Twitter because they are a straightforward way to stay connected with others, and this in turn shapes our worldview.

This same sentiment was also hit upon by JR Johnson on mashable in the post Social Media can Change the World through Common Ground.

He also points out that as we are awakening to the power of this interaction on the web, the most progressive companies and individuals are the ones actively creating new business models around this information, hybrids that combine existing frameworks with new social models.

From a social viewpoint, the architecture of business seems all wrong.

It’s becoming clear that to constrict a person’s capabilities into rigid, set roles that limit creativity and innovation just doesn’t make sense. Diving talent into silos is an outdated paradigm. Rather, we should be encouraging the facilitation of diverse groups of people working together on common problems. I touched on the potential power of this in a previous post, “The Future of Collaboration Begins with Visualizing Human Capital.“

I think his points completely validate the need for a new approach to thinking in general, which is exactly what I’m outlining in my ‘metathinking manifesto‘.

Insight #3

Wim Rampen is also noticing a trend, with yesterday’s post, Connecting the Dots, referencing Graham Hill’s recent post, A Manifesto for Social Business, and Mitch Lieberman’s post Social Just is…, both acknowledging the power of customer networks, looked at through the lens of Social Business. Hill laid out fifteen trends shaping the future of business, which clearly outline the fundamental shift underway:

I would almost go as far to say that we are fast approaching a period of ‘Business Enlightenment’, based not so much on the linear thinking that drove the Enlightenment in the 18th Century, as on networked, emergent thinking which is driving so much new thinking in the 21st.

Everyone is catching on – Lieberman’s post also references Esteban Kolsky’s new 5 part series on the Roadmap to Social CRM, an in-depth series of blog posts that outlines how to develop a Social Business strategy.

Insight #4

Here’s where things get interesting. From a learning standpoint, there is proof emerging that using Twitter builds intelligence. A study revealed these benefits:

All of the study participants were new to Twitter and had not previously used it or any similar microblogging service…..In a relatively short period of time, the participants formed quite sophisticated peer networks…..Peer support became a key feature of this student network, with activity rising just prior to assessment deadlines or during revision for exams. Content analysis of the messages indicated clear evidence of the emergence of personal learning networks…..Twitter is also very attractive as a data collection tool for assessing and recording the student experience, with a wide range of free and increasingly sophisticated online analysis tools available.

Synthesis

At the surface level, one could look at this information and agree that yes, social networks, and specifically the real-time network of Twitter, enable people to communicate and collaborate on new levels. I think there’s something deeper happening.

I’ve been reading about complex adaptive systems lately, and many of its key properties seem strikingly similar to what’s occurring on Twitter:

• Emergence: Rather than being planned or controlled the agents in the system interact in apparently random ways. From all these interactions patterns emerge which informs the behaviour of the agents within the system and the behaviour of the system itself.
• Co-evolution: All systems exist within their own environment and they are also part of that environment. Therefore, as their environment changes they need to change to ensure best fit.
• Requisite Variety: The greater the variety within the system the stronger it is. In fact ambiguity and paradox abound in complex adaptive systems which use contradictions to create new possibilities to co-evolve with their environment.
• Connectivity: The ways in which the agents in a system connect and relate to one another is critical to the survival of the system, because it is from these connections that the patterns are formed and the feedback disseminated. The relationships between the agents are generally more important than the agents themselves.
• Simple Rules: Complex adaptive systems are not complicated. The emerging patterns may have a rich variety, but like a kaleidoscope the rules governing the function of the system are quite simple
• Iteration: Small changes in the initial conditions of the system can have significant effects after they have passed through the emergence – feedback loop a few times (often referred to as the butterfly effect)
• Self Organising: There is no hierarchy of command and control in a complex adaptive system. There is no planning or managing, but there is a constant re-organising to find the best fit with the environment.
• Edge of Chaos: Complexity theory is not the same as chaos theory, which is derived from mathematics. But chaos does have a place in complexity theory in that systems exist on a spectrum ranging from equilibrium to chaos. A system in equilibrium does not have the internal dynamics to enable it to respond to its environment and will slowly (or quickly) die. A system in chaos ceases to function as a system. The most productive state to be in is at the edge of chaos where there is maximum variety and creativity, leading to new possibilities.
• Nested Systems: Most systems are nested within other systems and many systems are systems of smaller systems.

Complex adaptive systems are all around us. Most things we take for granted are complex adaptive systems, and the agents in every system exist and behave in total ignorance of the concept but that does not impede their contribution to the system. Complex Adaptive Systems are a model for thinking about the world around us not a model for predicting what will happen. I have found that in nearly all situations I can view what is happening in Complex Adaptive Systems terms and that this opens up a variety of new options which give me more choice and more freedom.

Is this perhaps the framework that we’ve all been hitting upon without realizing it? Many people have been sensing there is something special about the way we’re able to access and exchange information and ideas on Twitter, organize into Twibes and niche groups to tackle problems together, and develop strategies (like using lists and separate accounts) to filter out the content that matters most to us.

Final question: Is Twitter not a social media platform, but an actual entity, an intelligence made up of all of us?

___

further thoughts: If you have room for one more idea to provide another context, consider yesterday’s post by Tim O’Reilly on The War For the Web. If we start to experience real, measurable collective benefits from our ability to leverage the intelligence of the real-time web, will it be exploited, or will we ensure a system that keep our information and knowledge flows open source?

sources of the thoughtstream:

I would also highly suggest taking a look through Pierre Levy’s slideshare on Collective Intelligence & Cyberspace, which I found on Victor Godot’s site.

Insights from the Twittersphere

@SmartStorming Innovation is really a game of connect-the-dots. Try combining two or more seemingly unrelated things in a new way that creates value.

@spikenlilli Halpern: “How does one learn to see?” “Make associations between data points” – relational, generative, gestalt, anticipatory design #IPF09

@Innovation360 Can innovation be systematized? http://is.gd/4VCpm

@acarvin Hargadon: social media can unleash our latent creativity. #ncti2009

@WebStudio13 RT @craignewmark – RT @AlecJRoss: “The more freely information flows, the stronger the society becomes.” via @ariannahuff

People referenced in this post

Rosabeth Kanter @RosabethKanter
Stowe Boyd @stoweboyd
mashable @mashable
Wim Rampen @wimrampen
Graham Hill @grahamhill
Mitch Lieberman @mjayliebs
Estaban Kolsky @ekolsky
Tim O’Reilly @timoreilly
Victor Godot @victorgodot

This post made possible by:

@SameerPatel – RT’d @stoweboyd’s article
@SocialNetDaily – RT’d @AnneDGallager @HarvardBiz @KellySpors to @RosabethKanter’s article
@Wildcat2030 – RT’d @UniofLeics @TheHistoryWoman @timeshighered to Twitter in academia study
@emahlee – RT’d @anildash to @timoreilly’s article
@phaloo – tweeted @mashable article
@ekolsky – tweeted Roadmap to Social CRM article

note: I’m going to try as often as possible to reference posts in this way, because I think it’s a good illustration of how thoughts and ideas are developing as a result of distributed knowledge, and it’s easier for me to follow my own train of thought.

I saw all of these posts within the last 48 hours in my twitterstream…. I don’t know that I would have come up with this by reading RSS feeds or by using other news sites.

]]> http://glennas.wordpress.com/2009/11/17/collective-intelligence-part-4-quantifying-similarity/ Tue, 17 Nov 2009 16:42:16 +0000 glennas http://glennas.wordpress.com/2009/11/17/collective-intelligence-part-4-quantifying-similarity/

This is the fourth of a series of posts on the topic of programming Collective Intelligence in web applications. This series of posts will draw heavily from Santam Alag’s excellent book Collective Intelligence in Action.

These posts will present a conceptual overview of key strategies for programming CI, and will not delve into code examples. For that, I recommend picking up Alag’s book. You won’t be disappointed!

Click on the following links to access previous posts in this series:

• Part 1: Introduction
• Part 2: Basic Algorithms
• Part 3: Gathering Intelligence from User Interaction

Determining Similarity using a Similarity Matrix

The essential task in developing collective intelligence is determining similarity between things – between users and items, between different items, and between groups of users.

In Collective Intelligence, this typically involves computing similarities in the form of a similarity matrix (or similarity table). A similarity matrix compares the values in two Term Vectors, and computes the relative similarity between comparable entries in each term vector. Please refer to this previous post, for a brief introduction to terms and term vectors.

In chapter 2 of his book, Alag calculates similarity tables using 3 basic approaches:

1. Cosine-based similarity
2. Correlation-based similarity
3. Adjusted-cosine-based similarity

I’m not going to get into the specific differences between the different methods, but I will provide a general example (from Alag’s book) to illustrate the approach.

User similarity in rating Photos

The example Alag gives involves 3 different users rating 3 different photos. They express their ranking of a photo as a number between 1 and 5. These ratings are displayed in the table below:



If we were to calculate a similiarity matrix (using the cosine-based approach) comparing how similar the photos are to each other, we’d get the following table:



This table tells us that Photo1 and Photo2 are very similar. The closer to 1 a value in the similarity table is, the more similar the items are to each other.

You can use the same approach to calculating similarity between users’ preferences for the photos. If we do the calculations, we get the following results:



Here we see that Jane and Doe are very similar.

In Alag’s book, he details the specific algorithm for caclulating each of the above similarity tables, and shows the different results obtained using the 3 methods listed above (i.e. cosine-based, correlation-based, and adjusted cosine-based methods). He also provides examples based on user ratings for photos, as well as user ranking of articles based on which articles they bookmarked. However, the basic approach is the same as illustrated above.

In Summary

In this post we looked at the basic task of calculating similarity between items and users. In the next post, we’ll look at the specific scenario of extracting intelligence from tags.

Also in this series

• Collective Intelligence – Part 1: Introduction
• Collective Intelligence – Part 2: Basic Algorithms
• Collective Intelligence – Part 3: Gathering Intelligence from User Interaction
• Collective Intelligence – Part 5: Extracting Intelligence from Tags
]]> http://glennas.wordpress.com/2009/11/17/collective-intelligence-part-3-quantifying-user-interaction/ Tue, 17 Nov 2009 13:56:34 +0000 glennas http://glennas.wordpress.com/2009/11/17/collective-intelligence-part-3-quantifying-user-interaction/

This is the third of a series of posts on the topic of programming Collective Intelligence in web applications. This series of posts will draw heavily from Santam Alag’s excellent book Collective Intelligence in Action.

These posts will present a conceptual overview of key strategies for programming CI, and will not delve into code examples. For that, I recommend picking up Alag’s book. You won’t be disappointed!

Click on the following links to access previous posts in this series:

• Part 1: Introduction
• Part 2: Basic Algorithms

Introduction – Applying CI in your Application

Alag states that there are three things that need to happen to apply collective intelligence in your application.

You need to:

1. Allow users to interact with your site and with each other, learning about each user through their interactions and contributions.
2. Aggregate what you learn about your users and their contributions using some useful models.
3. Leverage those models to recommend relevant content to your users.

This post will focus on the first of these steps: specifically the different forms of user interaction that capture the raw data used to derive collective intelligence in social web applications.

In Alag’s book, he provides persistence models for capturing this user interaction data. In this post, however, I will not be discussing the specific persistence models that model these user interactions. Please pick up a copy of Alag’s book if you are interested in the details of how the data collected from these user interactions are captured in underlying persistence models.

Gathering Intelligence from User Interaction

Quoting Alag:

To extract intelligence from a user’s interaction in your application, it isn’t enough to know what content the user looked at or visited. You need to quantify the quality of the interaction. A user may like the article or may dislike it, these being two extremes. What one need is a quantification of how the user liked the item relative to other items.

Remember, we’re trying to ascertain what kind of information is of interest to the user. The user may provide this directly by rating or voting for an article, or it may need to be derived, for example, by looking at the content the user has consumed. We can also learn about the item that the user is interacting with in the process.

In this section, we look at how users provide quantifiable information through their interactions. … Some of the interactions such as ratings and voting are explicit in the user’s intent, while other interactions such as using clicks are noisy – the intent of the user isn’t perfectly known and is implicit.

Alag discusses 6 examples of user interaction from which collective intelligence data might be extracted. These are:

1. Rating and Voting
2. E-mailing of Forwarding a Link
3. Bookmarking and Saving
4. Purchasing Items
5. Click-stream
6. Reviews

I would generalize “e-mailing and forwarding a link” to “forwarding and sharing content” generally, of which “e-mailing and forwarding a link” is variation.

This post will provide a very light treatment of some of the forms of user interaction from which collective intelligence is derived.As mentioned above, I will not be exploring the persistence models that capture the user data from these interactions.

So, first up, rating and voting.

Rating and Voting

Quoting Alag:

Asking the user to rate an item of interest is an explicit way of getting feedback o how well the user liked the item. The advantage with a user rating content is that the information provided is quantifiable and can be used directly.

Alag has a very nice section on the specific data and persistence models that underlie the rating and voting data captured from user intereaction. Please refer to his book for this additional detail.

Forwarding and Sharing Content

Forwarding and sharing is another activity that can be considered a positive vote for an item. Alag briefly discusses a variation of this activity in the form of a user e-mailing or forwarding a link

Bookmarking and Saving

A few quick comments from Alag:

Online bookmarking services such as del.icio.us allow users to store and retrieve URLs, also known as bookmarks. Users can discover interesting links that other users have bookmarked through recommendations, hot lists, and other such features. By bookmarking URLs, a user is explicitly expressing interest in the material associated with the bookmark. URLs that are commonly bookmarked bubble up higher in the site.

The process of saving an item or adding it to a list is similar to bookmarking and provides similar information.

Bookmarking and saving is another user interaction activity for which Alag explores the underlying persistence model.

Purchasing Items

In an e-commerce site, when users purchase items, they’re casting an explicit vote of confidence in the item – unless the item is returned after purchase, in which case it’s a negative vote. Recommendation engines, for example the one used by Amazon, can be built from analyzing the procurement history of users. Users that buy similar items can be correlated and items that have been bought by other users can be recommended to a user.

Click-stream

Quoting Alag:

So far we’ve looked at fairly explict was of determining whether a user liked or disliked a particular item, through ratings, voting, forwarding, and purchasing items. When a list of items is presented to a user, there’s a good chance that the user will click on one of them based on the title and description. But after quickly scanning the item, the user may find the item to be not relevant and may browse back or search for other items.

A simply way to quantify an article’s relevance is to record a positive vote for any item clicked. This approach is used by Google News to personalize the site. To furthre filter out the noise, such as items the user didn’t really like, you could look at the amount of time the user spent on the article. Of course, this isn’t fail proof. For example, the user could have left the room to get some coffee or been interrupted when looking at the article. But on average, simply looking at whether an item was visited and the time spent on it provides useful information that can be mined later.

You can also gather useful statistics from this data:

• What is the average time a user spends on a particular item?
• For a user, what is the average time spent on any given article?

Reviews

Web 2.0 is all about connecting people with similar people. This similarity may be based on similar tastes, positions, opinions, or geographic location. Tastes and opinions are often expressed through reviews and recommendations. These have the greatest impact on other users when:

• They’re unbiased
• The reviews are from similar users
• They’re from a person of influence

Depending on the application, the information provided by a user may be available to the entire population of users, or may be privately available only to a select group of users.

Perhaps the biggest reasons why people review items and share their experiences are to be discovered by others and for boasting rights. Reviewers enjoy the recognition, and typically like the site and want to contribute to it. Most of them enjoy doing it. A number of applications highlight the contributions made by users, by having a Top Reviewers list. Reviews from top reviewers are also typically placed toward the top and featured more prominently. Sites may also feature one of their top reviewers on the site as an incentive to contribute.

Here again, Alag provides additional commentary around the persistence model underlying Reviews. See the book for details.

In Summary

In this post, we (very) briefly explored forms of user interaction that provide the raw data that applications use to derive collection intelligence to provide useful and relevant content to their users. In future posts in this series, we’ll explore how collective intelligence algorithms are used to aggregate this content, and provide useful insight and information to the users or a social web application.

Also in this series

• Collective Intelligence – Part 1: Introduction
• Collective Intelligence – Part 2: Basic Algorithms
• Collective Intelligence – Part 4: Calculating Similarity
• Collective Intelligence – Part 5: Extracting Intelligence from Tags
]]> http://glennas.wordpress.com/2009/11/16/collective-intelligence-part-2-basic-algorithms/ Mon, 16 Nov 2009 22:46:02 +0000 glennas http://glennas.wordpress.com/2009/11/16/collective-intelligence-part-2-basic-algorithms/

This is the second of a series of posts on the topic of programming Collective Intelligence in web applications. This series of posts will draw heavily from Santam Alag’s excellent book Collective Intelligence in Action.

These posts will present a conceptual overview of key strategies for programming CI, and will not delve into code examples. For that, I recommend picking up Alag’s book. You won’t be disappointed!

Click on the following links to access previous posts in this series:

• Part 1: Introduction

Introduction

Quoting Alag (which I’ll be doing a lot of!):

In order to correlate users with content and with each other, we need a common language to compute relevance between items [or Social Objects], between users, and between users and items. Content-based relevance is achored in the content itself, as is done by information retrieval systems. Collaborative-based relevance leverages the user interaction to discern meaningful relationships. Also, since a lot of content is in the form of unstructured text, it’s helpful to understand how metadata can be developed from unstructured text. In this section, we cover these three fundamental concepts of learning algorithms.

We begin by abstracting the various types of content, so that the concepts and algorithms can be applied to all of them.

Users and Items

Quoting Alag:

As shown in [the figure below], most applications generally consist of users and items. Items may be articles, both user-generated and professionally developed: videos, photos, blog entries, questions and answers posted on message boards, or products and services sold in your application. If your application is a social-networking application, or if you’re looking to connect one user with another, then a user is also a type of item.



Alag continues:

Associated with each item is metadata, which may be in the form of professionally-developed keywords, user-generated tags, keywords extracted by an algorithm after analyzing the text, ratings, popularity ranking, or just about anything that provides a higher level of information about the item and can be used to correlate items together.

When an item is a user, in most applications there’s no content associated with a user (unless your application has a text-based descriptive profile of the user). In this case, metadata for a user will consist of profile-based data and user-action based data.

There are three main sources of developing metadata for an item: (i) attribute-based, (ii) content-based, and (iii) user-action based. Alag discusses these next.

Attribute-based

Quoting Alag:

Metadata can be generated by looking at the attributes of the user or the item. The user attribute information is typically dependent on the nature of the domain of the application. It may contain information such as age, sex, geographical location, profession, annual income, or education level. Similarily, most nonuser items have attributes associated with them. For example, a product may have a price, the name of the author or manufacturer, the geographical location where it’s available, and so on.

Content-based

Metadata can be generated by analyzing the contents of a document. As we see in the following sections, there’s been a lot of work done in the area of information retrieval and text mining to extra metadata associated with unstructured text. The title, subtitles, keywords, frequency counts of words in a document and across all documents of interest, and other data provide useful information that can then be converted into metadata for that item.

User-action based

Metadata can be generated by analyzing the interactions of users with items. User interactions provide valuable insight into preferences and interests. Some of the interactions are fairly explicit in terms of their intentions, such as purchasing and item, contributing content, rating an item, or voting. Other interactions are a lot more difficult to discern, such as a user clicking on an article and the system determining whether the user liked that item or not. This interaction can be used to build metadata about the user and the item.

Alag advises thinking about users and items having an associated vector of metadata attributes. The similarity or relevance between two users or two items or a user and item can be measured by looking at the similarity between the two vectors.

Content-based Analysis and Collaborative Filtering

Alag explains that User-centric applications aim to make the application more valuable for users by applying CI to personalize the site. There are two basic approaches to personalization: content-based and collaboration-based.

Content-based Analysis

Again, quoting Alag:

Content-based approaches analyze the content to build a representation for the content. Terms and phrases (multiple terms in a row) appearing in the document are typically used to build this representation. Terms are converted into their basic form by a process known as stemming. Terms with their associated weights, commonly known as term vectors, then represent the metadata associated with the text. Similarity between two content items is measured by measuring the similiarity associated with their term vectors.

A user’s profile can also be developed by analyzing the set of content the user interacted with. In this case, the user’s profile will have the same set of terms as the items, enabling you to compute the similarities between a user and an item. Content-based recommendation systems do a good job of finding related items, but they can’t predict the quality of the item – how popular an item is or how a user will like the items. This is where collaborative-based methods come in.

Collaborative Filtering

A collaborative-based approach aims to use the information provided by the interactions of users to predict items of interest to a user. For example, in a system where users rate items, a collaborative-based approach will find patterns in the way items have been rated by the user and other users to find additional items of interest for a user. This approach aims to match a user’s metadata to that of other similar users and recommend items liked by them. Items that are liked by or popular with a certain segment of your user population will appear often in their interaction history – viewed often, purchased often, and so forth. The frequency or occurrence of ratings provided by users are indicative of the quality of the item or the appropriate segment of your user population. Sites that user collaborative filtering include Amazon, Google, and Netflix.

Continuing:

There are two main approaches in collaborative filtering: memory-based and model-based. In memory-based systems, a similarity measure is used to find similar users and then make a prediction using a weighted average of the ratings of the similar users. This approach can have scalability issues and is sensitive to data sparseness. A model-based approach aims to build a model for prediction using a variety of approaches: linear algebra, probabilistic methods, neural networks, clustering, latent classes, and so on. They normally have fast runtime predicting abilities.

Since a lot of information that we deal with is in the form of unstructured text, Alag proceeds to review basic concepts about how intelligence is extracted from unstructured text.

Representing Intelligence from Unstructured Text

Alag begins this section as follows:

This section deals with developing a representation for unstructured text by using the content of the text. Fortunately, we can leverage a lot of work that’s been done in the area of information retrieval. This section introduces you to terms and term vectors, used to represent metadata associated with text.

Continuing:

Let’s consider an example where the text being analyzed is the phrase “Collective Intelligence in Action.”

In its most basic form, a text document consists of terms—words that appear in the text. In our example, there are four terms: Collective, Intelligence, in, and Action. When terms are joined together, they form phrases. Collective Intelligence and Collective Intelligence in Action are two useful phrases in our document.

The Vector Space Model representation is one of the most commonly used methods for representing a document. A document is represented by a term vector, which consists of terms appearing in the document and a relative weight
for each of the terms. The term vector is one representation of metadata associated with an item. The weight associated with each term is a product of two computations: term frequency and inverse document frequency.

Term frequency (TF) is a count of how often a term appears. Words that appear often may be more relevant to the topic of interest. Given a particular domain, some words
appear more often than others. For example, in a set of books about Java, the word Java will appear often. We have to be more discriminating to find items that have these lesscommon terms: Spring, Hibernate, and Intelligence. This is the motivation behind inverse document frequency (IDF). IDF aims to boost terms that are less frequent.

Commonly occurring terms such as a, the, and in
don’t add much value in representing the document. These are commonly known as stop words and are removed from the term vector. Terms are also converted to lowercase. Further, words are stemmed—brought to their root form—to
handle plurals. For example, toy and toys will be stemmed to toi. The position of words, for example whether they appear in the title, keywords, abstract, or the body, can also influence the relative weights of the terms used to represent the document. Further, synonyms may be used to inject terms into the representation.

To recap, here are the four steps Alag presents for analyzing text:

1. Tokenization – Parse the text to generate terms. Sophisticated analyzers can also extract phrases from text.
2. Normalize – Convert them into a normalized form such as converting text into lower case.
3. Eliminate stop words – Eliminate terms that appear very often.
4. Stemming – Convert the terms into their stemmed form to handle plurals.

Computing Similarities

Quoting Alag:

So far we’ve looked at what a term vector is and have some basic knowledge of how they’re computed. Let’s next look at how to compute similarities between them. An item that’s very similar to another item will have a high value for the computed similarity metric. An item whose term vector has a high computed similarity to that of a user’s will be very relevant to a user—chances are that if we can build a term vector to capture the likes of a user, then the user will like items that have a similar term vector.

A term vector is a vector where the direction is the magnitude of the weights for each of the terms. The term vector has multiple dimensions—thousands to possibly millions, depending on your application.

Multidimensional vectors are difficult to visualize, but the principles used can be illustrated by using a two-dimensional vector, as shown below.



Alag, again:

Given a vector representation, we normalize the vector such that its length is of size 1 and compare vectors by computing the similarity between them. Chapter 8 develops the Java classes for doing this computation. For now, just think of vectors as a
means to represent information with a well-developed math to compute similarities between them.

Types of Datasets

In this section of the book, Alag discusses the difference between densely- and sparsely-populated datasets. The difference?

• A densely-populated dataset has more rows that columns, with a value for each cell. The classic example of a densely-populated dataset is a database table, where every record has an entry for every, or nearly-every field.
• A sparsely-populated dataset is a dataset where each row has very few entries per column. For example, an Amazon customer may potentially be associated with any book in Amazon’s inventory. In this example, each book in Amazon’s universe would potentially be a field in the customer’s record (or vector). However, a record representing the books that a customers had viewed or bought would only contain entries for a very few of these many books. Thus, the table that associated all Amazon users with potentially all of Amazon’s books would be a “sparse” dataset.

Well, that about wraps it up for this blog post. In the next blog post in this series, we’ll look at the many forms of user interaction in an social application, and how they are converted into collective intelligence.

glenn

Also in this series

• Collective Intelligence – Part 1: Introduction
• Collective Intelligence – Part 3: Gathering Intelligence from User Interaction
• Collective Intelligence – Part 4: Calculating Similarity
• Collective Intelligence – Part 5: Extracting Intelligence from Tags
]]> http://glennas.wordpress.com/2009/11/15/defining-requirements-for-social-web-applications-%e2%80%93-part-6-collective-intelligence/ Sun, 15 Nov 2009 22:35:03 +0000 glennas http://glennas.wordpress.com/2009/11/15/defining-requirements-for-social-web-applications-%e2%80%93-part-6-collective-intelligence/

This is the 6th post in a series on Defining Requirements for Social Web Applications. As with previous posts in this series, the content is largely borrowed from Joshua Porter’s book Designing for the Social Web. Porter’s book is a gem, and if the topic of social web design is of interest to you, I highly recommend you pick up a copy.

This post also borrows significantly from Satnam Alag’s book Collective Intelligence in Action.

Click on the following links to access previous posts in this series:

• Part 1: An Introduction
• Part 2: The Framework
• Part 3: Social Objects
• Part 4: Defining Core User Actions
• Part 5: Motivations for User Participation

Introduction

To my knowledge, the term Collective Intelligence was first coined – in the sense we mean it here – in a seminal paper published by Tim O’Reilly titled What is Web 2.0, published in September 2005. In this paper, O’Reilly states the following:

The central principle behind the success of the giants born in the Web 1.0 era who have survived to lead the Web 2.0 era appears to be this, that they have embraced the power of the web to harness collective intelligence

I rather like Joshua Porter’s comments which come close to capturing, IMO, the essence of Collective Intelligence. Porter states that Collective Intelligence is all about:

[Aggregating] the individual actions of many people in order to surface the best or most relevant content. … Collective Intelligence is based on the idea that by aggregating the behavior of many people, we can gain novel insights.

Satnam Alag in his excellent book Collective Intelligence in Action, comments that the Collective Intelligence of Users in essence is:

• The intelligence that’s extracted out from the collective set of interactions and contributions made by your users.
• The use of this intelligence to act as a filter for what’s valuable in your application for a user.

The common thread is “aggregated opinion”. Quoting Porter:

Digg and other aggregation systems rely on the fact that while no individual is right all the time, in the collective a large number of users can be amazingly accurate in their decisions and behavior. Amazon, Digg, Google, Netflix, and many other sites base their recommendations of products, news, sites, movies, etc. on aggregated opinion.

One result of Web 2.0-style applications that use Collective Intelligence is that, to quote Tim O’Reilly, the applications get better the more people use them.

The insights and patterns gleaned from Collective Intelligence are the product of algorithms of various degress of sophistication. Alag lists the following ways to harness Collective Intelligence in your application:

• Aggregate information lists
• Ratings, reviews, and recommendations
• User-generated content: blogs, wikis, message boards
• Tagging, bookmarking, voting, saving
• Tag Cloud navigation
• Analyze content to build user profiles
• Clustering and predictive models
• Recommendation engines
• Search
• Harness external content – provide relevant information from the blogosphere and external sites

Alag comments that:

Web applications that leverage Collective Intelligence develop deeper relationships with their users, provide more value to users who return more often, and ultimately offer more targeted experiences for each user according to her personal need.

Amazon, Yelp, Netflix, Google Search, Google News, Del.iciou.us, and Digg are just some of the more popular sites that leverage Collective Intelligence to target relevant content to their users.

Applying Collective Intelligence in your application

Alag states that there are three things that need to happen to apply collective intelligence in your application.

You need to:

1. Allow users to interact with your site and with each other, learning about each user through their interactions and contributions.
2. Aggregate what you learn about your users and their contributions using some useful models.
3. Leverage those models to recommend relevant content to your users.

Joshua Porter refers to these three steps as:

1. Initial Action
2. Display
3. Feedback

He provides the following table to illustrate the different forms these three steps take at various popular social websites:



Let’s see what Josha Porter has to say about these 3 steps.

Initial Action

The first step is for users to add content. Porter takes Digg as his case study.

On Digg, like on many social sites, you need an account to submit stories. Then, the process of submitting stories has two steps.

The first step is to enter the link you’re submitting. This is a normal URL. You also choose the type of content it is: a news story, image, or video. Digg helps people by providing a nice set of guidelines.

After you click “Continue” in step 1, Digg takes a moment to analyze the line to see if it’s a duplicate. This helps keep the system clean. When Digg thinks you’ve submitted duplicate content, it notifies you that the story has already been submitted.

Porter continues:

If the submission is not a duplicate, Digg analyzes the page and grabs any relevant content from it, including the page title, a description, and any images on the page. It then allows you to choose which elements are appropriate as part of your submission. This step makes it much easier to digg content, as you don’t have to do any heavy lifting of grabbing the content yourself.

Finally, Digg checks to make sure that the submitter of content is indeed a human being.

The initial action on Digg is a crucial step in the system. It determines what content is allowed, makes sure the content is unique, adds data that supports the story, and determines how can and cannot submit content. These decisions act as a barrier of entry to the system. The quality of content Digg that receives entry into the Digg system depends on the checks at this stage.

Adding Tags

Some services allow people to tag content, which allows aggregation of the content in additional, helpful ways. Porter uses the example of Del.iciou.us, which lets you add tags to bookmarks as you enter them into the system.

Aggregate Display

Quoting Porter:

The display of content is crucial to how people will interact with it. If content is displayed prominently then people will consider it more important. Content displayed less prominently will be considered less important.

In general, content is deemed more important when it is displayed:

• On a home page. The home page is visited the most of any page, and therefore it garners the most attention from both site owners and readers.
• More often. The more content is displayed and repeated, the more it is considered valuable.
• At the top of a page. Just like on the front of a newspaper, above the fold is the prime real estate. The top of a web page is where the most important content is placed.
• Higher in ranked displays. When content is ranked, such as in a “most emailed” list, the content at the top is deemed most valuable.

Porter continues:

When content first gets added to an adapative system, it is usually displayed in an appropriately less prominent location. Digg, for example, has what they call an Upcoming page, which displays all new submissions into the system in reverse-chronological order. These freshly-submitted stories stay on the upcoming page a short period of time, getting pushed off in favor of even fresher content. The Upcoming page is crucial to the functioning of the Digg site because it forces each story to gain its own popularity.

All of these stories aspire to reach the Digg home page, the ultimate place for grabbing attention, where they will be seen by thousands of people in a very short period of time. In fact, the burst of attention resulting from being on the Digg homepage often makes the site unreachable. So many people visit the site from Digg that the web server is overwhelmed and either slows to a crawl or breaks outright.

Types of Aggregation Order

Porter goes on to list some of the more popular ways that applications built for collective intelligence display content to their users to ensure that it is relevant and compelling to their audience. These are:

• Chronological order
• Popularity within a time range
• Participant ranking
• Collaborative filtering – filtering content based on your preferences and the recommendations of others
• Relevance
• Social

– displaying content based on who it’s from

• User-based views – so the user can see their own content

Feedback

Types of Feedback

Finally, social applications that leverage Collective Intelligence are dependent on feedback to provide value. Porter highlights some different types of feedback: Implicit vs. Explicit, and Positive vs. Negative.

I’ll quote Porter’s comments of Implicit vs. Explicit feedback:

Typically, a combination of implicit and explicit feedback is used to create a picture of popularity. For example, Amazon’s bestseller list (based on implicit feedback) also show ratings (based on explicit feedback).

Implicit feedback is based on user behavior that is captured while someone moves through a site. Examples include downloading, bookmarking, and purchases.

Explicit feedback comes from someone’s explicitly-declared preferences, including ratings, reviews, and comments. While this sort of feedback tends to be more accurate in reflecting user taste, it also requires more work from the user and so less data can be collected.

Make Feedback easy

Finally, Porter has a few words to say about the importance of making feedback an easy, simple task for the user.

In Summary

Wow, that was a decent-sized post as well. So that’s a brief journey into how some of the more popular sites on the web leverage collective intelligence to keep their users engaged, and deliver interesting and relevant content.

In the next post, we’ll look at one more chapter from Porter’s book, that being devoted to application functionality designed to make it easy to share content with your friends and the world.

glenn

Also in this series

• Defining Requirements for Social Web Applications – Part 1: Introduction
• Defining Requirements for Social Web Applications – Part 2: The Framework
• Defining Requirements for Social Web Applications – Part 3: Social Objects
• Defining Requirements for Social Web Applications – Part 4: Defining Core User Actions
• Defining Requirements for Social Web Applications – Part 5: Motivations for User Participation
• Defining Requirements for Social Web Applications – Part 7: Enabling Sharing
]]> http://justseventhings.com/2009/11/15/collective-intelligence-as-a-killer-app-goldman-sachs/ Sun, 15 Nov 2009 15:26:05 +0000 siconroy http://justseventhings.com/2009/11/15/collective-intelligence-as-a-killer-app-goldman-sachs/

A few years back I read an unauthorised article on Goldman Sachs that highlighted one of their core strengths as their management information systems. I’ve been a little obsessed by them ever since, and was delighted to read the Sunday Times magazine 8/11/9 feature article ‘Inside the goldmine’. Of particular interest was the journalist, John Arlidge’s, observations on how the people inside Goldmans work together.

The importance of all specialist and management information systems to the firm is undoubted from a review of the Technology section of their corporate site.  ’We think we make better decisions’, Liz Beshel Goldman’s Global Treasurer is quoted as saying ….. with $1 trillion a day flowing across the balance sheet. The “mark to market” pricing of the bank’s assets on a daily basis highlighted a trend that led to the decision by the bank to reduce its exposure to the housing and mortgage markets in advance of most other players. Its losses from the mortgage sector following the credit crunch were $1.7bn – lower than any other investment bank.

Arlidge notes however that Goldman is not about individual benefit: “it’s a team effort”. ‘When Goldman gets behind something, everyone in the giant hive wants a piece of the action’.

This reference to giant hive led me to Dušan Teodorović and Mauro Dell’ Orco’s paper on ‘Bee colony optimization – a cooperative learning approach…’ where they note that ’various natural systems teach us that very simple individual organisms can create systems able to perform highly complex tasks by dynamically interacting with each other’, contrary to ‘a great number of traditional engineering models and algorithms used to solve complex problems [that] are based on control and centralisation.’

Obviously not composed of simple individual organisms, the conclusion that ’these communication systems between individual insects contribute to the formation of the “collective intelligence” of the social insect colonies’, seemed to tie in to Arlidge’s observations.

The time and effort commitments of the staff are as you would expect. As well as a 24/7 culture and few people taking their full holiday allowance,  Chief Accountant Sarah Smith is quoted, ‘When you’re needed, you’re here. And if you’re needed and you’re not answering your phone, you won’t be needed very long’. The use of language from someone quoted as a veteran Goldman banker is also revealing: ‘You are programmed at an early stage to go out more than the other guy, to see more people – clients, hedge funds or private equity guys’

The collective intelligence of the colony is well articulated in the article: ‘Goldman staffers are… trained to “brain pick” contacts and clients harder than the other guy…”you offer something in return , but you always come back with something. Then you feed it to colleagues who go to work on trying to use the information to make money”. The underpinning culture and structures support this: ’The bosses work hard to foster a “we’re in this together”, family-style approach. Others say it feels more like a cult, but they mean it as a compliment’. The bonuses are based on the performance of the firm as a whole, rather than the individual effort recognition seen in other areas of banking.

That this information and communication are the ultimate drivers of GS’ success is perhaps sealed by the nature of their ‘dynamic interaction’: ’Goldmanites are forced to check their secure voicemail morning, noon and night… Goldman is the biggest user of voicemail in the world…”mind bullets” consist of anything from the latest profit and loss figures, to reports of what the chief executives of key clients have told Blankfein and his top team over lunch’

The importance of the whole, and the enduring protection of the hive – and its inhabitants ongoing – is brought home by two of the article’s final observations: ‘Goldmanites, curiously, question their ability. “there is a deep and constant paranoia about everything we do”…it applies to an individual’s performance and the prospects for the firm as a whole. Insecurity is hard-wired into the system..once hired, each staff member is constantly and confidentially reviewed by those they work with… the bottom 3-5% every year [about 1,500 people]‘ are let go. Its survival and compound growth at a global level, its ‘killer app. ….. [is] its extraordinary networking ability. The firm is the greatest talent network in the world’. ‘Top performers are encouraged to get on, make all the money they will ever need in their thirties, then get out….. getting top jobs in treasuries, central banks and stock exchanges around the world’

That this “collective intelligence” of the colony is also ‘the front-end of our client franchise‘, with their M&A advisory capacity having #1 rankings 2004-2009 across all major deal levels, completes the system. I said I was a little obsessed before. Now I’m fascinated.

Add to: Facebook | Digg | Del.icio.us | Stumbleupon | Reddit | Blinklist | Twitter | Technorati | Yahoo Buzz | Newsvine

]]> http://100trillion.wordpress.com/2009/11/14/memories-immortality-tulkus/ Sat, 14 Nov 2009 20:06:09 +0000 duncanwork http://100trillion.wordpress.com/2009/11/14/memories-immortality-tulkus/

Bodies are memories, constantly remembering how to recreate and repair themselves, with slight deviations, gradually noticed as age.

Bodies – appendages, organs, cells, and genes – also retain a memory of millions of years of evolution, passed offspring to offspring.

Even rocks retain memories of ancient sediments deep in long-gone oceans or in churning fiery depths inside the earth.

Organizations are memories, also remembering how to constantly recreate, repair and maintain, all while learning to adapt.   There are restaurants in China that are thousands of years old, governments and their agencies, corporations and their offspring and mutations, all persistent memories.

Without memories, no continuity, no underlying stability, no identity.

During sleep I forget, forget who I am, and dream of strange new identities and settings, shifting from one to the next.  I lapse into deep sleep and forget even my dreams.   I wake.  My body is still here.  All my memories are still here – reminding me of my identity, my ambitions and desires, my plans, my worries, my friends and enemies, and what’s in the refrigerator.   From  nothingness during the night, each day “I” am reincarnated.

And via the Internet we can find how many times and how many people have recorded variations of these thoughts.   Together we are the memory of our species.

Now to the most recent reason I started thinking about this:

Tulkus – Passing Memory and Identity from Life to Life




I recently read a book that explained in more detail than I had known before, about tulkus, reincarnations of Tibetan lamas.   The most famous tulku is the Dalai Lama, who is considered to be the 14^th reincarnation in a line that began with the first Dalai Lama, born in 1391.

The book I recently read (and quote from below), The Dance of 17 Lives, by Mick Brown, 2004, is about the most ancient line of tulkus, the Karmapa lamas, and especially about the 16^th and 17^th Karmapas.   The first Karmapa was born in 1110 AD; the 17^th was born in Tibet in 1985, and also recognized and installed there as a child.  He escaped, under the noses of the Chinese, in the last few days of 1999 and fled to Dharamsala, in India.

From reading about tulkus, it is clear that much more than myth and untestable belief are involved in making this amazing system work.   Whether one believes or not in reincarnation of old souls into new bodies is not of great importance.   What is most interesting is how the system is able to transmit a huge body of personal memories, teachings and understanding,  relationships, and, in effect, entire identities from one incarnation of body and mind to another.

As stated by one lama interviewed by the author, “The Dalai Lama says that being a tulku is like a raw diamond – not worth much. You have to cut and polish it.  You have to study and practice.  Being recognized as a tulku is the beginning of the process, not the end of it.’”

The description of the training of newly recognized tulkus is especially interesting (p. 29):  “Like a young king, the tulku was therefore at the centre of an elaborate court of attendants, teachers and administrators, the object of enormous care and attention, and of an investment in manpower and money which would last for many years. … such attention would, in normal circumstances, be expected to turn any child’s head; but the teachings the young tulku received were specifically designed to counteract this and to develop the qualities of humility, self-effacement and a detachment from the worldly wealth and power which surrounded him.  …Only after years of such training would they be considered ready to assume their role as a teacher.”

In the case of the Karmapas, the book describes how there is a very large core of knowledge and teachings, called the “Golden Rosary” and dating back 900 years, that have been transmitted by the Karmapa to a few of his closest followers.  These closest followers are themselves high tulku lamas that serve the Karmapa, life after life, making the relationships especially meaningful and close. When a Karmapa dies, these core followers then assume responsibility for teaching the tradition to the next Karmapa, so that  the link is never broken. As stated by one of the Karmapa’s closest followers, “’The teachings of the Golden Rosary comprised innumerable different practices, and there was a lineage for every one of them – for each initiation, every ritual, for a multitude of meditation practices.’”

As indicated, this type of reincarnation requires an immense, interconnected system of traditions, teachings, devoted followers, and accumulation of wealth.   It is a system that created and maintained one of the most remarkable and inspiring cultures in the world, lasting many centuries, and still surviving.  Tulkus are obviously not just people, but great social and spiritual institutions that are maintained in unbroken lines.   Unlike kings and popes, a tulku does not just inherit a title, teachings, and devoted followers, but an actual personal identity and a set of close personal relationships.

In modern societies, we have evolved similar yet secular institutions:  Governments with their constitutions and offices, corporations with their charters and offices, agencies with their mandates described in legal codes.    Even though our governments and cultural institutions are largely secular, great principles are often involved, passed down, fought over, and fought for.

Corporations, with their continually reincarnating identities, are recognized officially as citizens with full rights, and often enormous power.    Yet there seems to be no adequate system for also ensuring that these institutions will also continually reacquire the wisdom and concern for the common good that should accompany such power.   Though recognized as citizens, they seem instead to be expected, and even legally required to behave in self-interest above all.    But this seems to be getting into another subject.

Let us build good memories to pass on.

]]> http://shakarian.wordpress.com/2009/11/12/parallel-thought-dachis-group/ Fri, 13 Nov 2009 01:29:41 +0000 Chris Shakarian http://shakarian.wordpress.com/2009/11/12/parallel-thought-dachis-group/

It’s not often that you stumble upon a group of people who’s thinking about the future of business so closely parallels your own.. a social business design consultancy called Dachis Group. The Dachis Group helps businesses make sense of how to effectively apply a social business design strategy internally, to bring about transformative social progress, emergent innovation and hive mentality.

Incidentally, if you have the time (20 min), here is David Armano (principal at Dachis) talking about personal branding (which I found shortly after I writing about it – wierd):

]]> http://omnigens.wordpress.com/2009/11/11/000019/ Wed, 11 Nov 2009 15:59:46 +0000 James Ang http://omnigens.wordpress.com/2009/11/11/000019/

How Hi-tech Urban Farming R&D and Social Enterprise Become A Winning Approach. 

 

Whenever we think of Research and Development (R&D), we tend to think only post-graduate scientists and researchers working together. Rarely do we thinks of any non-university school leavers involvement. Well, the Japan first hi-tech urban underground farming project is an exception example. There can be other examples that I may not know of.

 

Let’s make life simple. For this article, we just focus on the  Japan urban underground farming concept example. In this Japanese business cum social enterprise case study, the  Pasona Group which is a Japanese temporary staffing recruitment agency creates an agricultural sector employment awareness. Why there is a need for such awareness? In Japan alone, the country is facing an ageing working population. Japanese agricultural workforce population is mostly make up of an ageing population.

     

         

 Source: Japundit, Tomatoes growing in Pasona O2, Hi-tech urban underground.  

    

In addition, there is a need to create employment for unemployed youth and non-degree school leavers. Before the current global subprime mortgage crisis , it used to be a general global trend that not many youths want to work in plantation fields. At the same time, we are facing a global food security crisis. Beside improving traditional soil-based farming output, we need to research for better alternative urban farming solutions.

‘By working proactively and collectively, we can find win -win solutions to many issues.’  

     James Ang, The omniGens Blog  - Make Life Simple   

 

While researching for better hi-tech urban underground farming techniques, Pasona O₂ (pronounced as “oh-tsoo”) creates jobs for unemployed youths. (I not sure exactly how people are directly hired in Pasona O₂ urban underground farm project.)  Some refer this business model as social enterprise model. It is a win-win situation for parties involve in this advanced urban underground farming research.

  

Based on News report, the crops production is not significant at the moment. Pasona O₂ urban underground farm project best annual rice production is around 60 kg. This is normal for any major innovation and technology breakthrough. After all, both advanced urban vertical farming and urban underground farming  concept model is still in its infancy development stage . 

 

Let’s look at automobile development history. There is a huge technology development improvement from the first steam-powered vehicle built around 1672 to first petrol-powered car in 1894 to other alternative powered autos. Automobile technology development takes a long history to reach current advanced level. Even now, automobile technologies still constantly revolve. This creates jobs for many people in the process.

 

By working proactively and collectively, we can find win -win solutions to many issues. For example, we can achieve reduction in global warming, carbon footprint while maintaining global economic growth. Yes, it’s definitely not an easy task. Perhaps, we can find some win – win solutions to these global issues. Let’s start one step at a time. Just like what we have known about our automobile development history.   

 

 

About The omniGenerations Blog’s Concept

The omniGenerations Blog (or The omniGens blog in short) is a peer learning community blog or a peer learning micro wiki blog  that focus on the holistic development of our lives. The omniGens Blog leverages on the collective wisdom of our proactive community in helping us to achieve simple work – life balance and happiness. Make Life Simple.

 

PLEASE TAKE NOTE: All comments need to be approved before appearing in the contribution box / comment. Time and space constraints may prevent all comments from appearing. We will only approve comments directly related to this blog article and this blog. Please use appropriate language, exercise restrain,  responsible, sensitivity, tolerance, care and courtesy in your contribution. For more details, please refer to The omniGens Blog’s purpose and objective.

 

 

]]> http://jstromsk.wordpress.com/2009/11/10/pierre-levys-collective-intelligence-mankinds-emerging-world-in-cyberspace-in-brief-and-use/ Tue, 10 Nov 2009 18:40:40 +0000 mfstromski http://jstromsk.wordpress.com/2009/11/10/pierre-levys-collective-intelligence-mankinds-emerging-world-in-cyberspace-in-brief-and-use/

Pierre Levy is a professor in the department of hyperspace media at the University of Paris and is a firm believer that technology will eventual create “living cities” where physical location takes a back seat to interaction between its members.

A few insights from his work “Collective Intelligence: Mankind’s Emerging World in Cyberspace,” include the importance of listening, thoughts on real-time democracy and new forms of writing.

A decent example of Levy’s work in action is Journo Student: Thoughts and Exploits. In the blog, the author discusses the future of various mediums and also gives examples of where online writing is heading *cough* *cough* “twitter” *cough* and how to use them properly.

It’s been a while since the author has posted, the last updated being in April 2009, but as time has shown, her posts, especially about twitter, have come true.

On a more personal level, I find Levy to be right. The future definitely seems to be moving toward a cyber-community, with citizens hailing from all parts of the world.

Now, I wonder, how long will it be until we establish a single, world-wide government, just like those movies and television shows about the future have established.

]]> http://molliewanie.wordpress.com/2009/11/11/pierre-levys-collective-intelligence-mankinds-emeging-world-in-cyberspace/ Wed, 11 Nov 2009 00:39:21 +0000 Mollie Dobersek http://molliewanie.wordpress.com/2009/11/11/pierre-levys-collective-intelligence-mankinds-emeging-world-in-cyberspace/

Levy stresses the importance of the role of listening for journalists. A role that is commonly over looked in the media. We may not be psychologists, but we must use listening skills in a different way. In order to understand the expression of a problem, to assume and take a position on an issue, and to formulate arguments based on positions, we must take listening into consideration.

Christopher Harper describes the roles of the digital journalist rather well in his article Journalism in a Digital Age.

Harper states;

“Online journalism places far more power in the hands of the user, allowing the reader to challenge the traditional role of the publication as the gatekeeper of news and information. The user can depend on the gatekeeper to select and filter the news in the tradition manner, or the user can drill down to the basic documents of a story. In short, the user can look over the shoulder of the reporter by researching the original documents and easily comparing one reporter’s story with those of others by scanning news publications throughout the country. Archives also become easily accessible.

Second, online journalism opens up new ways of storytelling, primarily through the technical components of the new medium. Simply put, online journalists can provide a variety of media–text, audio, video, and photographs–unlike any other media. Data searching provides a means to acess information unable in other media.

Third, online journalism can provide outlets for nontraditional means of news and information. The internet enable everyone who owns a computer to have his or her own printing press. “

The digital news environment will constantly continue to evolve throughout our lifetime.

There was an interesting article on one of “the best journalism blogs”, Teaching Online Journalism which was describing how journalism students of today should have keen knowledge in;

1. Capturing audio and editing it
2. Video recording, production and editing
3. Web skills (which could be production, design and coding, Web journalism and blogging).

All of which are going to become required knowledge in the near future as we enter the digital age.

]]>