How do some  webpages on the Internet become so ubiquitous that we are rarely ever less than two short links away from viewing that page? How have some webpages become so popular among the hundreds of billions of webpages that are on the Internet currently, not to mention the new ones that pop up almost every second? More importantly how do certain pages become the centre for web activity and lead us to other pages within their network? Albert-Lázló Barabási explored these questions about networks and the organization/disorganization of interconnectivity within his Chapter titled Hubs and Connectors .

Barabási begins the piece by explaining both the power and importance of links within the web. Quite simply the larger number of incoming links that direct visitors to your webpage then the higher the number of visitors you will have, which in turn increases your overall profile on the web. This sets your page apart from others, allowing for greater visibility and secures your place within the hiearchy as a destination site online. The perception of networks, its growth and mechanisms has traditionally followed the random worldview theory of networks as defined by Erdos-Rényi. Within this theory the interior of networks are connected at random, by nodes (this can be thought of as webpages online) and each node within the network is not distinguishable from another, meaning that they uniformly connect to the other nodes within the network the same number of times. (This example is presented in the previous post.) Hence the distribution of the links is equal between all the nodes thus abolishing any distinguishable character that one node may have over another. This distinguishable character within a network typically infers that there is “a few highly connected nodes, or hubs.” (58) According to the random network theory hubs could not exist because all nodes share the same amount of links to other nodes. Additionally the total number of nodes is static over time, leaving no room for growth in terms of the number of nodes existing within the network. Typically this theory has been represented by bell curves; in which various nodes are dispersed randomly and all connect to each other an equal amount of times, thus creating a peak within the middle of the graph. Accordingly within this random network theory all webpages would have an equal opportunity to be seen and heard. Barabási observes that this theory of networks is quite simply not true, and holds no ‘real world’ application and/or example.

He contests that the Erdos-Rényi theory of networks, which consist of nodes that are randomly connected with “roughly the same number of incoming links”, is not applicable at all to the hierarchal pattern of popular webpages prevalent within the web today. Barabási states that some webpages such as Amazon.com, Yahoo! etc. have an exceptional amount of incoming links and thus dominate the web through the sheer enormity and range of their incoming links from various networks across the web. Webpages like Amazon inherently become the hubs of the Internet. Additionally Barabási notes that just like actors within Hollywood (as observed within the Kevin Bacon phenomena, in which Bacon is only 2.79 degrees removed from everyone within the Hollywood network) the strength of these Internet hubs do not rely on the sheer size of the network but the range. This means that the links these nodes make to other various distant networks adds range to the network that the hubs possess. This adds variety and brings dissimilar things together. In contrast to hubs, Barabási points out that his own webpage is so insignificant within the web, due to the near infinite range of the Internet network, that the chance of a person viewing it is about forty in a billion. Within this contrast lies the main concerning question of the Internet and it’s network; how do some pages become these hubs of the Internet, in comparison to the majority of pages that exists in near obscurity?

Barabási claims that the theory and research of networks as represented by Erdos-Rényi random network theory fails to answer this question because of it’s fundamental underlying assumptions; ¹ Static: the amount of nodes in a Erdos-Rényi network are set, meaning there is no room for growth, which is in exact opposition to the nature of the web itself and ² Equality (Random Interconnectivity): each node within the network contains an equal amount of links to the other nodes thus making each node indistinguishable from the last, hence the nodes are linked randomly and are equal (there can be no room for hub webpages such as Google!). Within the random network’s nodes are considered a “characteristic scale[,] embodied by the average node and fixed by the peak degree of distribution.” (pg70)

 Barabási dismisses this entire method of perceiving networks, and states that the actual distribution, connection and dominance (or lack thereof) certain webpages within the Internet is determined by inverting the two random network assumptions. The new foundations of network theory are ruled by the following;

“¹Growth: For each give period of time we add a new node to the network. This step underscores the fact that networks are assembled one node at a time

² Preferential Attachment: We assume that each new node connects to the existing nodes with two links. The probability that it will choose a given node is proportional to the number of links the chosen node has. That is, given the choice between two nodes, one with twice as may link as the other; it is twice as likely that the new node will connect to the more connected node”.  (pg 86)

These foundations are further represented not in bell curves but by the Italian Economist Pareto’s 80/20 Rule and the power law phenomena it produces. Within a power law ruled network the majority of nodes have only a few links (incoming and outgoing) and these tiny nodes co-exist with a few big hubs that are highly connected with many nodes that traverse various differing networks. The power law exemplifies Pareto’s Rule, which states that 80% of effects are derived only from 20% of causes. The approximate 80/20 rule applied to the web means that “80% of the links on the web point to only 15% of total webpges”. (66)

Pareto’s 80/20 Rule and the power law graph that is supports provides us with a very different perspective on networks, especially according to how Barabási perceives networks in the Internet. The power law graph shows a slope that rises and then declines running horizontally parallel out to infinity.  The nodes located with this horizontal section account for the 80% of wepages and the peak of the slope which is located on the left hand side of the graph closest to the vertical line is the 15% of webpages that account for hubs. This unequal distribution of incoming links as presented within the power law graph is a more accurate description of the current state of the Internet. 

 As Barabási describes;

“The power law distribution thus forces us to abandon the idea of scale, or a characteristic node. In a continuous hierarchy there is no single node which we could pick out and claim to be characteristic of all the nodes. There is no intrinsic scale in these networks. This is the reason my research group started to describe networks with power-law degree distribution as scale-free.” (pg70)

This principle that describes the web and its interconnectivity is not a phenomenon distinctive to the web but applies to a variety of situations within human life.

The effect of power laws and the study of its presence is not a new phenomenon at all. In the last few years the existence of the Internet has dramatically changed how some view the various power laws at work within out world. The Long Tail as it was coined by Chris Anderson has become a staple of what some believe will be the future for commerce, specifically within the music industry. Anderson felt that the hubs within the music industry were an effect of the old 20^th century system of hits culture which encompassed production, manufacturing and the distribution of music. These hubs within music could be seen as various major record labels, major publishing companies, big name acts, and their hit repertoire. For many years the music industry was ruled by power laws, with only 20% (termed by Anderson as the ‘head’ of the power law) of the products (the term products primarily refers to music recordings like CD’s. LP vinyl etc) accounted for 80%( termed by Anderson as the ‘tail’ of the power law) of the sales.  The 20%, of  the ‘head’, are comprised of the goliaths of the music industry such as Michael Jackson’s classic ‘Thriller’ or by more recent releases like Rihanna’s album ‘A Good Girl Gone Bad’. The other 80% of niche genre music (such as instead of the main genre of Rock, you want Acid Rock) was virtually inaccessible, because the justification of producing, manufacturing, distributing and stocking this niche music  would not be meet via mass profits…until now.

Anderson proposed that with the democratization of distribution via Internet retail sites like Amazon, anyone could find and purchase a piece of music that they enjoyed. By riding the tail of the power law to the outer reaches music lovers could find music that was deeper within the niche of music that they always wanted to own, and/or discover new music. Additionally if we are to take a look at Barabási’s notion of networks within the Internet, the hubs of music such as Jay-Z could refer music lovers to other musicians within that rap/hip-hop genre of music like Nas, and the links from the Nas node could lead to other lesser known acts.

The sheer enormity of information and access that the Internet provides would allow for music lovers (or a consumer of any product/service) to be able to find exactly what they want, and be able to obtain that item. Through the Long Tail Anderson argues that we could begin to focus more on the niche products (or in this case music genres) that for so long have been relegated to the sidelines as the hit parade came into town. In fact Anderson argues that in terms of commerce businesses could focus on selling ‘alot of a little’. For example Anderson starts with just exploring the ‘A’ section of music genres and discovers a niche of Afro-Cuban Jazz. Although this isn’t a big genre within the ‘A’ section, in it there are hubs  like artist Tito Puente. You could focus as a retailer on having Tito Puente in addition to other artists within the same genre, like the Buena Vista Social Club. You may only sell a few copies of each artist, but the total of that entire sale of those artists in combination would equate to a profit either equal or similar to that amount if you might have sold if you were just focusing on the ‘head. Accordingly this would mark the end of hits, or is it?

Kanye, Lil’Wayne, Miley Cyrus, Pink, Kings of Leon etc, are all still selling millions of copies of their albums (or at the very least a solid half a million). These are mass marketed and mass produced artists, and more importantly as hubs of the music industry they still influence alot of other artists that are developing. Although in our preferences we might become very niche, the power and influence of hubs still remains. Hence although the playing field has been leveled, and as consumers we can have access to music like never before, we still need some sort of guidance. Anderson calls this ‘filtering’, which sites like Amazon with its Recommendations technology and Google through its search engine provide by offering a means of shifting through all of the products available.

 However, once again he is missing the bigger point of networks and the position of hubs within that network. For many music lovers finding new music is a journey that starts with a hub like John Mayer and then through the connections that artist makes as a hub whether it’s through genre, featured artists on his album or label mates, music lovers had to start at the ‘head’ of the power law and then ride to the ‘tail’ to find that music. If those hubs didn’t exist then discovery would be highly difficult. The  Erdos-Rényi random network theory was disproved because of this very thing;  if all the nodes have scale and there are no distinguishable characteristics making any one node different from another than the same information is passed through that type of network. Within the music industry this would mean that no new music would be able to be found, and by only focusing on the niches the bigger picture of music trends and tastes would be ignored. Hubs need niches and niches need hubs. Anderson’s continued crusade against hubs or ‘hits’ destabilizes the entire network theory; and its applications within music, Internet and life in general. Anderson fails to realize that as within any network; when a node has a choice between linking between two other nodes it will most likely choose the node that has the higher amount of links. The same can be said for music fans and webpages alike; hits are still hits.

The reading for this week, Linked by Albert-Laszlo Barabasi disproves the idea of randomness in networks and champions the idea of hubs and connectors that function to bring together society. The idea behind linked and being connected to others is not new nor is it phenomenal: it is in our blood. This is our history and thus, inevitable. The Bible clears demonstrates this connection as in the beginning, there was Adam, then Eve, then Cain and Able. After that, humanity arises and creates towns to settle and we develop. Thus, we are all connected or linked in some way. If one is agnostic or believes in science, then think of evolution. People evolved as running from animals to hunting them in groups. Once in groups, they became nomads, and then quite surely, someone decided to settle down in a location, therefore forming a town. Thus a hub is created and with it, streets, other towns, etc…

It is divided into 3 chapters. I will discuss each chapter separately by its part and tie it together at the end. They are all connected; rather, they are components of a larger truth.

The Fifth Link: Malcolm Gladwell’s The Tipping Point notices an interesting phenomenon: “sprinkled among every walk of life…are a handful of people with a truly extraordinary knack of making friends and acquaintances. They are connectors” (55). This brings in the idea of connectors as highly relevant components of our social lives. Another way to define connectors is by calling them hubs. These hubs have a large number of links. One can create the analogy of a large city, such as Los Angeles and the suburbs that surrounds it. Many highways enter and go out of Los Angeles, connecting many different cities along the way. This discovery has basically disproven Erdos-Renyi’s theory of random worldview and Watts and Strogatz’s simple circle network model. Social networks exhibit clusters and hubs, both points that demonstrate a worldly rule that applies.

We can also apply the theory of links and connectors/hubs to cyberspace. The World Wide Web is the ultimate space for freedom, an environment that represents and defines the meaning of space without limits, boundaries, nor rules. Anyone can publish their work online and allow anyone to see it. Problem is, there are billions of websites. The question that arises is visibility. On the Web, the “measure of visibility is the number of links. The more incoming links pointing to your Webpage, the more visible it is” (57). As such, only websites that become hubs are highly visible (Amazon, Google). These are hubs: connectors with enormous links to other nodes.

I can create a website, thus it becomes a node. Google has grown, thus it has become a hub. Without a link, both websites exist and move in different worlds, or just space. The internet is as big as the universe, each node moves on its own, until it makes contact. This is the link. My website can function in its own galaxy as networks tend to form a cluster. Clusters are “nodes that are linked only to nodes in their subculture or genre” (61).  As a cluster, it is easier to find a common connection to a hub. For example, if my website is about knee pains, then I would like to create some link to a popular website such as WebMD. This in turn, can allow me to link with Google. In two links, my website has escaped randomness and can be visible.

The Sixth Link: Vilfredo Pareto may be a well known Italian economist, but I feel that his thought process behind the 80/20 Rule is sheer brilliance. It is all around us, and yet, he is the first to notice that it applies to the world. This is Simple Genius: understanding a law within the world that is so visible, and yet, invisible to the eye. 80/20 Rule applies to many things, but is generally regarded as 20 own rest of the 80. This also applies to network as well. It can be proven through a mathematical expression called a power law. In contrast to a bell curve, which is a “distribution rather similar to the peaked distribution characterizing random networks,” (67) the power law is by definition a special kind of mathematical relationship between two quantities. The exact definition is as follows: if one quantity is the frequency of an event, the other is the size of the event, then the relationship has a power law distribution when the frequency of the event decreases at a greater rate than the size increases. In layman terms, think of 20% of the population ruling 80% of the world.

Power laws basically functions to prove mathematically the fact that in most “real networks the majority of nodes have only a few links and that these numerous tiny nodes coexist with a few big hubs, nodes with an anomalously high number of links” (70). Again, this relates to Gladwell’s idea of connectors, but proving quantitatively that it exists within the realm of the World Wide Web. It demonstrates that real networks are not random at all, but exist under the power law. The interesting idea that Barabasi proposes is that in the beginning, nodes tend to be chaotic and without any form or order. However, through time, this disorder turned into order through self-organization. Under the theory of phase transitions, real networks demonstrate self creation from disorder into the 80/20 Rule. Barabasi’s idea is highly compelling, as he demonstrates quantitative analysis and by using the Web, a blank space which is like a universe unto itself, to explain a simple law that applies to the world. Everyone is intricately linked, because we are social beings. Networking is only a function of humanity, which always existed, but has now been proven through the power law. A question that arises is why do hubs and links form and how does it form in such a manner that to the naked eye is so random and chaotic, and yet, has order.

The Seventh Link: Erdos and Renyi’s model of networks rely on two principles. First is the idea that all nodes are fixed and remains unchanged throughout the network’s life. Second is that all nodes are equivalent. However, this is obviously not the case as there exists hubs/connectors, links, and change from disorder to order through self-organization. To understand how it does this, we must understand that the web is constantly growing. It is changing and growing. This is quite self-explanatory. There has been exponential growth of websites on the World Wide Web. As there is growth, we can safely disprove the static nature of Erdos-Renyi’s model of networks. We also do not randomly decide on which websites to link. We choose based upon our knowledge and social upbringing. We prefer certain websites over others because we are comfortable and familiar with that certain product. Thus, rises the concept of hubs. Barabasi brings up the idea of preferential attachment: “when choosing between two pages, one with twice as many links as the other, about twice as many people link to the more connected page. While our individual choices are highly unpredictable, as a group we follow strict patterns” (85). In many sense, this is true. We are all sheep and follow the leader.

In truth, randomness does not really exist unless it is the role of a die. Linking between networks is not random. Though unmentioned as third criteria, I believe that popularity and attractiveness plays an important role in the addition of links and creation of hubs. Webpages that have more links are more likely to be “linked to again” (86). Thus, there exists first person advantage. Older nodes have greater chances to become bigger and eventually rise as a hub. As a senior member within a link, this node has greater links and more nodes want to be linked to that certain node. Barabasi has given compelling evidence that real networks are not random, but constantly evolving and growing, attracting more links through the legitimacy of the power law.

We are living in a complex world and yet, guided by the invisible hand or law that is, in my opinion, inexplicable. The idea of real networks applies not only to websites, but to humanity in large. I believe that Barabasi’s point is that humans are social beings and we are followers. There are different people: some are leaders and some are followers. This difference guides the principle of 80/20 Rule as well as the power law. Barabasi has done something quite remarkable: take a simple, obvious, and yet invisible rule, and proved it scientifically and gave it a name. Kudos.

Roopnarine, P. D. 2009. Ecological modeling of paleocommunity food webs. in G. Dietl and K. Flessa, eds., Conservation Paleobiology, The Paleontological Society Papers, 15: 195-220.

Find the paper here:
http://zeus.calacademy.org/roopnarine/Selected_Publications/Roopnarine_09.pdf
or here
http://zeus.calacademy.org/publications/

Guild-level trophic link distribution

Spent a great week at the Annual Meeting of the Geological Society of America. The Paleontology Society session on Conservation Paleobiology was a lot of fun, and my students also presented great posters. Now back to the coral reef.

I’ve been cleaning up the data, because with some much data, errors are bound to creep in. I believe that the current data are now accurate, and the metanetwork statistics are 265 guilds (including primary producers) and 4,651 links. That yields a metanetwork connectance of 0.066. The link distribution should therefore also be different, and indeed it is. The figure shows the no. of links per guild, and the regression plot demonstrates that the distribution is still a power law distribution. The exponent is smaller than previously calculated, (), but this is the guild-level network and does not reflect species richnesses (yet).

Trophic level vs. no. of links

The next question that I’m looking at is the distribution of trophic levels among guilds and species. I therefore calculated trophic level for all guilds. The first figure (scatter plot) plots trophic level against the number of prey or incoming links to each guild. There are two things to notice: First, the variance of trophic levels decreases as the number of links, or diet generality of the guild increases. Second, the decrease in the variance is asymmetric, in that there is a bias against being a generalist of low trophic level. This is obvious if you look at all the empty space being vacated below the data points as no. of links increases. I can think of two non-exclusive explanations for this. If you think about a food chain, consumers toward the top of the chain simply have more prey to select from (on an evolutionary timescale), and therefore there should be a natural increase in the number of generalists as trophic level increases. Also, note that there are also many specialists of high trophic level. Perhaps the ability to exert power over other species, as a predator, combined with the previous statement, explains this observation. Finally, what is the distribution of trophic levels within the community? The second figure is a simple histogram plot of all non-primary consumer guilds (i.e. omnivores and carnivores). The distribution is approximately normal, with a definite central tendency. On average, most guilds in the reef are of similar trophic level! That’s very interesting. And referring to the previous scatter plot, we know that there is a biased composition in the tails of the distribution, in that the upper tail (higher trophic level) is a mixed composition of specialist to generalist guilds, but the lower tail is basically restricted to low trophic level specialists.

Guild trophic level distribution

Some of you may have noticed that our trophic levels are non-integer numbers. Primary producers all occupy trophic level 1, and primary consumers are trophic level 2. “Above” that, trophic level is calculated on the basis of the trophic levels of your prey. Exactly how we do that will remain a secret for now.

Species-level trophic link distribution.

The data presented in the previous post examined in-link or in-degree distribution at the guild level, i.e. species are aggregated into ecological guilds. A comment on the previous post asked whether we’ve used any grouping algorithms for guild recognition, and the answer is no, at least not yet (and thanks again for the comment). The current guilds are based primarily on trophic habits and habitat, and other features such as the presence of photo- or chemosymbionts. Guild derived algorithmically would be based on species-level network topology, and ideally, the two would be very similar. Anyway, I noticed the comment when I logged on to post the current results. What I’ve done is to expand the guild-level network (metanetwork) to the species-level, and then re-examine the trophic link distribution. There is no guarantee that the two distributions should agree. For example, it is quite possible that guilds of high in-degree (lots of prey), though few in number, are very species rich, and hence one would lose the decay distribution at the species level. Conversely, guilds of low in-degree could be tremendously more species rich, and would expand disproportionately, when compared to high in-degree guilds, when expanded into member species. Nevertheless, for this dataset, when guilds are actually expanded from 255 consumer guilds to 704 consumer species, the scale-free nature of the distribution is reinforced. The new function is y=11158x^-1.981, implying a power law exponent very close to 2. Neat.

Trophic link distribution

What sort of network is the coral reef food web? In other words, how are the links or interactions between nodes in a food web distributed? Food webs have been modelled variously as everything from random (Poisson) networks to networks based on exponential, power law or mixed distributions, with or without hierarchical structure. Empirical measures suggest that link distributions in real world food webs follow exponential or power law distributions, perhaps a mixture of both (differentiated by scale). One of my worries with those measures is that they are based on food webs of varying sizes, and more importantly, levels of taxonomic and ecological resolution. So, for example, how much does it matter if your food web covers only a small part of the community’s taxonomic diversity, or only part of the trophic diversity? What about the level of aggregation of species into more inclusive groups? The high resolution of the coral food web presents an opportunity to address some of these questions, and here’s the first one: How are trophic in-links distributed at the guild level? Recall that guilds here are groups of species with potentially the same prey and predators. I say potentially, for while we have very specific trophic data for some species, e.g. heavily studied fish, data are less certain for many smaller or less well known species. Still, there are 265 guilds in this dataset, and 4,756 links (see previous post). The histogram is a basic frequency histogram of the number of links per guild. As predicted on the basis of previously studied food webs, the distribution is a (right-skewed) decay distribution, with a greater number of species possessing fewer prey, i.e. being relative specialists, and a few species having a broad repetoire of prey, i.e. relative generalists. The extreme generalists (to the right or tail of the distribution) are all large sharks, the most extreme being the tiger shark, Galeocerdo cuvier. These species range from microscopic, single-celled dinoflagellates to large carcharhinid sharks!

What type of distribution is this? A simple logarithmic transform of the data is shown in the second figure, and regression of the data yields the following function: y = 17238x^-1.9496 (r-squared=0.95). The significant and extremely good fit of a linear function to the transformed data suggests that the underlying link distribution is a power law distribution of the form , where is the link probability, is the number of prey available, and is the power law exponent. An exponent of ~1.95 is tantalizingly close to other empirical measures. Even more exciting, for me at least, is the fact that we have predicted on the basis of previous work that power law exponents that promote resistance or robustness to secondary extinctions should lie in the range 2-2.5. That work was based on terrestrial food webs from the Late Permian, 250+ million years ago!

EDIT: As has been pointed out, I made a rather embarrassing miscalculation in the original post, which made me seriously underestimate the CTR. I evidently need to evaluate my quantitative credentials

My previous post on conversation monitoring was tweeted and retweeted by several individuals. Firstly, I’m grateful that people both read this blog and are motivated to share something I’ve written.

However, the additional traffic that this Twitter activity generated has left me wondering how valuable this social activity is to individuals or organisations that look to spread their message through this sphere.

What follows are some rough numbers given that:

1. WordPress.com stats are pretty basic
2. I’ve left it two weeks to do the maths, and so follower numbers will have changed
3. Follower overlap and actual exposures are unknown

Nevertheless:

• To my knowledge, the post was tweeted/retweeted 10 times
• Combined number of people following those who linked the post is 10,354 as of today
• The post probably got 100 additional hits as a result of Twitter activity

A couple of guesstimated calculations:

• At an absolute level, this represents a click through rate of 1%
• If I made the assumption that 5,000 followers are unduplicated (the largest follower count for a retweeter is over 3,000), the CTR changes to 2%
• How many of the followers would have seen the tweet? A fifth? That changes the CTR to 10%

10% is OK for a CTR, but it isn’t spectacular. The best ad campaigns with a strong call to action (e.g. competition entry) would achieve that.

The argument is that these 10% are going to be of a much higher quality than random visitors – they have acted upon a social recommendation and are likely to be engaged and interested in the content.

But that argument should work for the click through itself. If someone you follow and trust is recommending something, shouldn’t you be more likely to click through than if it were a random link or ad?

There a few issues at play here, which are causing this level of CTR

• Noise – Twitter is popular; there are a lot of tweets and links to browse and skim
• Ambient intimacy – often, it is enough for me to know that person X has linked to a post on conversation monitoring by @curiouslyp. I may prefer to browse the remaining tweets rather than click through to this post
• Power laws – if the post on conversation monitoring was by @jowyang or @chrisbrogan I may click through since they are renowned experts. Who is @curiouslyp and what would he know about this topic?
• Nature of followers – my prior post was relevant to the PR community – very active on Twitter. I suspect posts of a different subject matter are unlikely to be spread and consumed to the same degree

It is nice to think that the future is social, and that these networks will power traffic in future. But those perpetuating this – in my opinion – myth are those for whom power laws benefit, and who spend an inordinate amount of time on social networks (most likely because it is there job to do so). The average person does not have the time nor inclination to follow through on many, let alone all, posts or links.

So, in my opinion, the return on conversation is pretty minimal. Nevertheless, I did find it interesting to map how my post spread through Twitter via social graphs and, to repeat, I am grateful to the few that took the time to read and pass on my post.

sk

Image credit: http://www.flickr.com/photos/ironmonkey480/

Tanya Berger-Wolf sent me a brave little paper recently: Mathematics and the Internet: A Source of Enormous Confusion and Great Potential by Willinger, Alderson and Doyle.  This paper does a post mortem on the popular belief that empirical studies show that the Internet is scale-free.  Spoiler: They don’t.

Willinger et al. trace this erroneous belief starting with a paper by Pansiot and Grad (who are by no means responsible), that collected data on node degrees “to get some experimental data on the shape of multicast trees one can actually obtain in [the real] Internet”.  This data was collected using trace routes, which was reasonable given that its purpose was to determine the shape of multicast trees.  In 1999, Faloutsous, Faloutsous and Faloutsous used the data from Pansiot and Grad for a purpose for which it was not intended: they used it to infer information about the degree distribution for the Internet’s router-level topology.  Next, Barabasi et al., who were studying the WWW graph, and observing scale-free distributions there, picked up the reports of scale-free distributions for the Internet and added this to their list of networks with such properties.

A paper soon followed by Barabasi and Albert that described the preferential attachment model of network growth, wherein nodes are added one at a time to a network; each node has a constant number of out links; and the destinations of these out links are selected from the current nodes with probability proportional to each current node’s number of incoming links. Barabasi and Albert showed that this model generates a network with a scale-free distribution, and posited the preferential attachment as a model of the growth of the Internet and the WWW network.  The preferential attachment model had actually been studied over about 75 years by Yule, Luria and Delbruck, and Simon, but the rediscovery by Albert and Barabasi, and the possible connection to modern networks generated a huge amount of excitement and a large number of follow-up papers.  In particular, a slew of follow-up papers specifically studied properties of preferential attachment (PA) networks, i.e. networks generated via the preferential attachment model of growth. It was shown that PA networks are very robust to random deletions, but very fragile to adversarial deletions.  One commonly accepted conclusion was that the Internet, like PA networks, had high degree nodes, that were very centrally located, and whose removal would easily disconnect the network.  This idea was actually celebrated as a nice implication from theoretical network models to the real world.

So what’s the problem?  Basically the research community made two huge mistakes; mistakes that some people in the community still have not recovered from (in the sense that “recover from” means  “become aware of”)!

Mistake 1: Empirical studies do not support the claim that the Internet is scale-free

Willinger et al. go through several problems about why traceroutes do not create accurate maps of the Internet.  There are many technical details here that are hard for a theoretician like me to understand.  However, one detail that caught my eye was the fact that when a traceroute encounters an “opaque layer-2″ cloud, “it falsely ‘discovers’ a high-degree node that is really a logical entity  … rather than a physical node”.  This causes traceroutes to report high-degree nodes in the core of the router-level Internet, even when such nodes don’t exist.  This type of bias is claimed to be even worse than the well-known “over-sampling of high-degree nodes” bias that traceroutes also have.

Mistake 2: Scale-free networks are not PA networks

PA Networks are scale-free, but the converse is not true.  In particular, there are scale-free networks that are robust to adversarial deletions.  Intuitively, it’s possible to have a scale-free network where the high degree nodes are not at the “core” of the network.  More generally, it’s possible to have a scale free network with good expansion properties.  Such networks are robust to adversarial deletions.  In fact, Wilinger et al., suggest that the Internet is more likely to have the high degree nodes at the “periphery” of the network since each router can only handle a certain amount of traffic, and more edges are only possible if each edge is handling less traffic.

What Next?

Here the Willinger paper looses steam.  They suggest a first principles approach to Internet modeling, where the network formed is one that tries to optimize a given algorithmic problem over the network.  This is great.  What is not so great is the model they propose, which assumes that the network creation problem is solved in a completely centralized manner.  Much better would be if they had used game theory as a starting point.  After all, the Internet is inherently a massive, distributed enterprise. There is actually some really cool work done in algorithmic game theory on network creation games.  For example, see Chapter 19 of The Book, or this paper by Fabrikant et al. (shout out to Elitza Maneva in the et al. for this paper, who was visiting UPC while I was there on sabbatical).  Yes, it’s true that most of these game theory result have shown that network creation games have small price of anarchy.  However, that does not imply that the network topology you get in a Nash equilibria will be like the topology you get in the socially optimum solution.  I’d like to see some results on the types of topologies one might expect in an Nash equilibria for network creation games.

Black Swans swimming in Reading, UK

Nassim Nicholas Taleb’s (the author of “The Black Swan“) writing style is very provocative, brash and presumptuous, however the argument he presents about “Black Swans“, very unlikely events with deep consequences, is one of the most brilliant I’ve read so far.

Beyond all his attacks to “empty suits” (experts and consultants of a certain field), Nobel prizes and “fawning MBA students” (hem hem…), Taleb clearly shows our limits in understanding the causes which triggered an event and our poor skills in forecasting. Very interesting at this regard are his examples about how we are biased in explaining past events (narrative fallacy) and how we systematically consider evidences backing our theories but ignore facts which disprove our beliefs (we often forget that 1000 evidences don’t prove a theory but just 1 counter-evidence is enough to disprove it).

The first 2 parts of the book are a sad enumeration of how poor is our critical thought and how we always underestimate our ignorance (great the example of Umberto Eco’s anti-library) yet these parts are very useful in highlighting how we tend to commit the turkey’s error (the farmer has fed me for the last 100o days and so I’m sure he will do again tomorrow).

The last part deals instead with what we can do when facing risks and having to take decisions; most of this section is labeled “optional” by the author but I believe is a necessary read as well. Having demolished (convincingly) the Bell’s curve as a universal tool for risk evaluation, Taleb talks about Power Law distributions, with interesting overlaps with Chris Anderson’s Long Tail and Mandelbrot’s Geometry.

A book absolutely brilliant, which teaches that what we don’t know is as important as what we do know, regardless the field: financial risks’ evaluation, historical events, everyday events…

Related articles by Zemanta

• Antilibraries(kottke.org)
• Can You See Any Black Swan in Your Rear-view Mirror?(socialhallucinations.com)
• Ten Against the Swan: Nassim Nicholas Taleb’s 10 Theses (omnivoracious.com)

Whether you like Nassim Taleb’s writing style or not, his books and the ideas within are worth careful study.

For example, a key theme of Nassim Taleb’s “The Black Swan” is to pay close attention to the possibility of the rare event, the outlier that brings devastating consequence. Taleb forcefully and intelligently argues that these outliers are much more common than we think (1 in 100 year floods happen every 3-5 years) especially when we base our statistical analysis on the assumptions of a normal distribution of data and independence.

However, some interesting research by Didier Sornette at the Swiss Federal Institute of Technology shows that while power laws can explain some of the extreme events that have taken place in history, there are some extreme outliers that even defy classic power laws.

A post in MIT’s Technology Review blog cites the following:

“Sornette gives as an example the distribution of city sizes in France, which follows a classic power law, meaning that there are many small cities and only a few large ones. On a log-to-log scale, this distribution gives a straight line–except for Paris, which is an outlier and many times larger than it ought to be if it were to follow the power law.”

The article also mentions that the city of London also follows this same example.

Sornette calls these extreme outliers “Dragon Kings.” A sobering commentary from the article ensues; “(The) seemingly ubiquitous presence of these dragon kings in all kinds of data sets means that extreme events are significantly more likely than power laws suggest.”

This in turn suggests that the Mother of all Black Swans might be unaccounted for in your data set. And if this is the case, does this mean that we cannot predict these extreme events,  and/or that preparation is futile?

I would love to hear your thoughts!

I have just uploaded a paper to the arXiv, entitled “Scl, sails and surgery”. The paper discusses a connection between stable commutator length in free groups and the geometry of sails. This is an interesting example of what sometimes happens in geometry, where a complicated topological problem in low dimensions can be translated into a “simple” geometric problem in high dimensions. Other examples include the Veronese embedding in Algebraic geometry (i.e. the embedding of one projective space into another taking a point with homogeneous co-ordinates to the point whose homogeneous co-ordinates are the monomials of some fixed degree in the ), which lets one exhibit any projective variety as an intersection of a Veronese variety (whose geometry is understood very well) with a linear subspace.

In my paper, the fundamental problem is to compute stable commutator length in free groups, and more generally in free products of Abelian groups. Let’s focus on the case of a group where are free abelian of finite rank. A is just a wedge of tori of dimension equal to the ranks of . Let be a free homotopy class of -manifold in , which is homologically trivial. Formally, we can think of as a chain in , the vector space of group -boundaries, modulo homogenization; i.e. quotiented by the subspace spanned by chains of the form and . One wants to find the simplest surface mapping to that rationally bounds . I.e. we want to find a map such that factors through , and so that the boundary wraps homologically times around each loop of , in such a way as to infimize . This infimum, over all maps of all surfaces of all possible genus, is the stable commutator length of the chain . Computing this quantity for all such finite chains is tantamount to understanding the bounded cohomology of a free group in dimension .

Given such a surface , one can cut it up into simpler pieces, along the preimage of the basepoint . Since is a surface with boundary, these simpler pieces are surfaces with corners. In general, understanding how a surface can be assembled from an abstract collection of surfaces with corners is a hopeless task. When one tries to glue the pieces back together, one runs into trouble at the corners — how does one decide when a collection of surfaces “closes up” around a corner? The wrong decision leads to branch points; moreover, a decision made at one corner will propogate along an edge and lead to constraints on the choices one can make at other corners. This problem arises again and again in low-dimensional topology, and has several different (and not always equivalent) formulations and guises, including -

• Given an abstract branched surface and a weight on that surface, when is there an unbranched surface carried by the abstract branched surface and realizing the weight?
• Given a triangulation of a -manifold and a collection of normal surface types in each simplex satisfying the gluing constraints but *not*  necessarily satisfying the quadrilateral condition (i.e. there might be more than one quadrilateral type per simplex), when is there an immersed unbranched normal surface in the manifold realizing the weight?
• Given an immersed curve in the plane, when is there an immersion from the disk to the plane whose boundary is the given curve?
• Given a polyhedral surface (arising e.g. in computer graphics), how can one choose smooth approximations of the polygonal faces that mesh smoothly at the vertices?

I think of all these problems as examples of what I like to call the holonomy problem, since all of them can be reduced, in one way or another, to studying representations of fundamental groups of punctured surfaces into finite groups. The fortunate “accident” in this case is that every corner arises by intersecting a cut with a boundary edge of . Consequently, one never wants to glue more than two pieces up at any corner, and the holonomy problem does not arise. Hence in principle, to understand the surface one just needs to understand the pieces of that can arise by cutting, and the ways in which they can be reassembled.

This is still not a complete solution of the problem, since infinitely many kinds of pieces can arise by cutting complicated surfaces . The -manifold decomposes into a collection of arcs in the tori and which we denote respectively, and the surface (hereafter abbreviated to ) has edges that alternate between elements of , and edges mapping to . Since is a torus, handles of mapping to can be compressed, reducing the complexity of , and thereby , so one need only consider planar surfaces .

Let denote the real vector space with basis the set of ordered pairs of elements of (not necessarily distinct), and the real vector space with basis the elements of . A surface determines a non-negative integral vector , by counting the number of times a given pair of edges appear in succession on one of the (oriented) boundary components of . The vector satisfies two linear constraints. First, there is a map defined on a basis vector by . The vector satisfies . Second, each element is a based loop in , and therefore corresponds to an element in the free abelian group . Define on a basis vector by (warning: the notation obscures the fact that and map to quite different vector spaces). Then ; moreover, a non-negative rational vector satisfying has a multiple of the form for some as above. Denote the subspace of consisting of non-negative vectors in the kernel of and by . This is a rational polyhedral cone — i.e. a cone with finitely many extremal rays, each spanned by a rational vector.

Although every integral is equal to for some , many different correspond to a given . Moreover, if we are allowed to consider formal weighted sums of surfaces, then even more possibilities. In order to compute stable commutator length, we must determine, for a given vector , an expression where the are positive real numbers, which minimizes . Here denotes orbifold Euler characteristic of a surface with corners; each corner contributes to . The reason one counts complexity using this modified definition is that the result is additive: . The contribution to from corners is a linear function on . Moreover, a component with can be covered by a surface of high genus and compressed (increasing ); so such a term can always be replaced by a formal sum for which . Thus the only nonlinear contribution to comes from the surfaces whose underlying topological surface is a disk.

Call a vector a disk vector if where is topologically a disk (with corners). It turns out that the set of disk vectors has the following simple form: it is equal to the union of the integer lattice points contained in certain of the open faces of (those satisfying a combinatorial criterion). Define the sail of to be equal to the boundary of the convex hull of the polyhedron (where here denotes Minkowski sum). The Klein function is the unique continuous function on , linear on rays, that is equal to exactly on the sail. Then over expressions satisfies where denotes norm. To calculate stable commutator length, one minimizes over contained in a certain rational polyhedron in .

Sails are considered elsewhere by several authors; usually, people take to be the set of all integer vectors except the vertex of the cone, and the sail is therefore the boundary of the convex hull of this (simpler) set. Klein introduced sails as a higher-dimensional generalization of continued fractions: if is a polyhedral cone in two dimensions (i.e. a sector in the plane, normalized so that one edge is the horizontal axis, say), the vertices of the sail are the continued fraction approximations of the boundary slope. Arnold has revived the study of such objects in recent years. They arise in many different interesting contexts, such as numerical analysis (especially diophantine approximation) and algebraic number theory. For example, let be a matrix with irreducible characteristic equation, and all eigenvalues real and positive. There is a basis for consisting of eigenvalues, spanning a convex cone . The cone — and therefore its sail — is invariant under ; moreover, there is a subgroup of consisting of matrices with the same set of eigenvectors; this observation follows from Dirichlet’s theorem on the units in a number field, and is due to Tsuchihashi. This abelian group acts freely on the sail with quotient a (topological) torus of dimension , together with a “canonical” cell decomposition. This connection between number theory and combinatorics is quite mysterious; for example, Arnold asks: which cell decompositions can arise? This is unknown even in the case .

The most interesting aspect of this correspondence, between stable commutator length and sails, is that it allows one to introduce parameters. An element in a free group can be expressed as a word in letters , e.g. , which is usually abbreviated with exponential notation, e.g. . Having introduced this notation, one can think of the exponents as parameters, and study stable commutator length in families of words, e.g. . Under the correspondence above, the parameters only affect the coefficients of the linear map , and therefore one obtains families of polyhedral cones  whose extremal rays depend linearly on the exponent parameters. This lets one prove many facts about the stable commutator length spectrum in a free group, including:

Theorem: The image of a nonabelian free group of rank at least under scl in is precisely .

and

Theorem: For each , the image of the free group under scl contains a well-ordered sequence of values with ordinal type . The image of contains a well-ordered sequence of values with ordinal type .

One can also say things about the precise dependence of scl on parameters in particular families. More conjecturally, one would like to use this correspondence to say something about the statistical distribution of scl in free groups. Experimentally, this distribution appears to obey power laws, in the sense that a given (reduced) fraction appears in certain infinite families of elements with frequency proportional to for some power (which unfortunately depends in a rather opaque way on the family). Such power laws are reminiscent of Arnold tongues in dynamics, one of the best-known examples of phase locking of coupled nonlinear oscillators. Heuristically one expects such power laws to appear in the geometry of “random” sails — this is explained by the fact that the (affine) geometry of a sail depends only on its orbit, and the existence of invariant measures on a natural moduli space; see e.g. Kontsevich and Suhov. The simplest example concerns the (-dimensional) cone spanned by a random integral vector in . The orbit of such a vector depends only on the gcd of the two co-ordinates. As is easy to see, the probability distribution of the gcd of a random pair of integers obeys a power law: with probability . The rigorous justification of the power laws observed in the scl spectrum of free groups remains the focus of current research by myself and my students.

Piere Fraigniaud and George Giakkoupis from the University of Paris at Diderot have a really nice paper in this upcoming Principles of Distributed Computing (PODC). Their paper titled “The Effect of Power-Law Degrees on the Navigability of Small Worlds” builds on the classic paper by Jon Kleinberg on navigating small world networks.

Jon Kleinberg’s classic paper concerns navigation in a grid network where each node, in addition to its local edges, has one additional long range edge. What Kleinberg showed is that provided that, for each node, this long range link covers a distance d with probability proportional to d^2, then a greedy routing algorithm will ensure that any node can reach any other node in the network within no more than about log^2 n hops where n is the number of nodes in the network[1]. Moreover, the exponent in this probability is pretty important, even a slight deviation from an exponent of 2 results in networks that can not be efficiently navigated by greedy algorithms. In this way, Kleinberg was thus one of the first people to describe a type of network that might mimic the social network that allowed quick routing in Stanley Milgram’s famous six degrees of separation experiments.

So what about this new paper by Piere and George? Well for many years the exponent of 2 in log^2 has bothered people. Piere and George show that it is possible to get rid of it with power laws. In particular, they show that if instead of each node having exactly 1 long distance link, the number of long distance links per node follows a certain powerlaw distribution, then greedy routing works in about log n hops. A powerlaw distribution means that the number of nodes with a number of long distance links x is proportional to x^k for some fixed constant k. This is a so called heavy tail distribution which occurs in many natural complex systems. Surprisingly, Piere and George show that the type of powerlaw distribution for which greedy routing works is when k is in the range between 2 and 3, which is very similar to the exponent one observes for degree distributions in many naturally occuring social networks.

As far as I know this is one of the first papers that suggests a nice functional property of powerlaw distributions. In particular, it shows that powerlaw distributions are more powerful than other distributions in achieving a specific mathematical goal. Are there other algorithmic or mathematical problems that powerlaw distributions are “good” for. It looks like a very nice paper.

[1] I’m using the term about to meet O(log^2 n) or roughly a function that grows like C log^2 n for some fixed constant C.

Las leyes de potencia están de moda a la hora de interpretar datos experimentales. Pero las leyes de potencia son un arma de doble filo. No es fácil estimar sus parámetros utilizando un estimador de máxima verosimilitud. Códigos en Matlab y en R que te permiten estimar los parámetros de una ley de potencias así como calcular la bondad de dicho ajuste los puedes encontrar en esta página web, resultado de un artículo que se publicará en la prestigiosa SIAM Review, “Power-law distributions in empirical data,” de Aaron Clauset, Cosma Rohilla Shalizi, y M. E. J. Newman, disponible en ArXiv desde junio de 2007 (last revised 2 Feb 2009).

El código más interesante plfit.m no funciona en la versión 6 de Matlab, requiriendo al menos la versión 7 (los cambios para adaptarlo a la versión 6 no son difíciles de hacer pero hay que hacerlos). Es un código lento pero ni mucho menos tan lento como plvar.m que estima el error en los parámetros del ajuste y plpva.m que determina el valor p del ajuste mediante un test de Kolmogorov-Smirnov (si p<0.1 la ley de potencias es un pésimo ajuste a los datos). Estos últimos comandos repiten 1000 veces un cálculo que tarda decenas de segundos. Así que hay que tomárselo con mucha tranquilidad.

Mucha gente afirma que las citas de artículos científicos de un investigador siguen una ley de potencia. Uno de los ejemplos del artículo de Clauset et al. para las citas totales en el ISI WOS (Web of Science) a todos los artículos publicados en una serie de años encuentra un valor de p=0,2 que indica cierta evidencia, pero afirma que una ley de potencias truncada (con un corte) obtiene un p=0,87 (un valor muy bueno). ¿Qué pasará con un autor individual? He buscado en el ISI WOS las publicaciones y su número de citas (a día de hoy) de varios investigadores para comprobar si con los programas de Clauset et al. es válida la hipótesis de que siguen una ley de potencias. Los resultados hasta el año 2008 (inclusive) son los siguientes:

- Edward Witten que tiene 271 artículos y un índice h de 120; obtenemos alpha =1,89 +/- 0,38, con xmin =102 +/- 133, lo que claramente nos hace dudar de la validez de la ley de la potencia, de hecho lo confirma un p=0,001.

- Jorge E. Hirsch, inventor del índice h, que tiene 215 artículos y un índice h de 52; obtenemos un alpha=2,11 +/- 0,29, con xmin=36 +/- 20, un un valor p=0,028. Tampoco sigue una ley de potencia.

- Un investigador español senior con 204 artículos y un índice h de 12; obtenemos alpha=2,33 +/- 0,59, con xmin=4 +/-3, y un p=0,001.

- Un investigador español joven con 48 artículos y un índice h de 8; obtenemos alpha=1,83 +/- 0,50, con xmin=3 +/-4, y un p=0,25.

Son solo 4 ejemplos, pero parece claro que las leyes de potencia no describen bien las citas de investigadores.

Last year, at the beginning of October I decided to dedicate my second post on financial markets (I, II) to Black Swans. Swans are beautiful animals, but while white swans are vulgar and omnipresent at every pond, black swans are rare! Meanwhile, 2 days ago (June 1) the Wall Street Journal comes with this very awkward - and by all means for that precise reason - interesting article written by journalist Scott Patterson, where Mr. Taleb’s name pops-up again (image below).

Well, … let’s face it: you could put your money in the bank and have – let’s say – a 3% revenue at the end of your fiscal year. Or you could apply it to raise a new fancy gourmet restaurant at your local vicinity. Restaurants and local food stores are known to have 5-7% revenues in one year, not to speak on the immense burden they represent as well as for some associated risks – specially these days. But then you may think – better than banks, right? Right! Or, just to give you another example on this increasing scale - raising a little bit the risk -, on the other hand you could apply your money in stock markets. Main financial indexes (Dow Jones, NASDAQ, etc) are known to have an annual average revenue of 10-12% (since 1918). Not these days of course, where high volatility and entropy in the markets are installed. Well,  emergent countries like China are raising themselves at 12%/year also. We could go on and on with so many other examples. Some say that Eolic parks could achieve 40%. Normally the cost of one eolic tower is around 1 million euros, which could be paid back after one year producing energy trough wind at normal operating conditions. The rest are maintenance costs, as well as initial investment in terrains, etc. So, what’s new? Consider this. For moments imagine yourself having 100% in revenues, just last year, at this precise dramatic context. That’s 10 times what the market does in regular years, 20 times what your favorite restaurant does. Moreover, there is a substantial difference between all these examples. If you keep dropping money at the restaurant (for instance the revenue you have earned in the last year), still liquid revenues will be the same in the next year (unless you open a new dinner room next to the first one, while the awful burden keeps increasing). Some business are static and linear in time while others are exponential. As Alice in the wonderland, you will need to keep running twice as faster in order to be at the same place. Amazing those differences, no? Well, not for those “lovely” animal creatures known as Black Swans. According to Patterson, … Funds run by Universa, which is managed and owned by Mr. Taleb’s long-time collaborator Mark Spitznagel, last year gained more than 100% thanks to its bearish bets. Universa now runs about $6 billion, up from the $300 million it began with in January 2007. Excerpts from the Wall Street Journal article (Black Swan Fund Makes a Big Bet on Inflation) follow below. So, why the hell I do not feel at all surprised by this?! Really, I am not. Let me just say, I do have my own reasons:

  [...] A hedge fund firm that reaped huge rewards betting against the market last year is about to open a fund premised on another wager: that the massive stimulus efforts of global governments will lead to hyperinflation. The firm, Universa Investments L.P., is known for its ties to gloomy investor Nassim Nicholas Taleb, author of the 2007 bestseller “The Black Swan,” which describes the impact of extreme events on the world and financial markets.

Funds run by Universa, which is managed and owned by Mr. Taleb’s long-time collaborator Mark Spitznagel, last year gained more than 100% thanks to its bearish bets. Universa now runs about $6 billion, up from the $300 million it began with in January 2007. Earlier this year, Mr. Spitznagel closed several funds to new investors….

Mr. Taleb doesn’t have an ownership interest in the Santa Monica, Calif., firm, but he has significant investments in it and helps shape its strategies. The term “black swan,” which has become a market catchphrase in the last few years, alludes to the once-widespread belief in the West that all swans are white. The notion was proven false when European explorers discovered black swans in Australia. A black-swan event, according to Mr. Taleb, is something that is extreme and highly unexpected. … [...]

An interesting article in the Wall Street Journal by the father of modern portfolio theory, Harry Markowitz, discussing the current financial crisis. He emphasises the need for diversification across uncorrelated risk as the key to minimising balanced portfolio risk, and highlights its lack as a major factor behind our present predicament. However to my mind this still begs a question of whether any risk can be truly uncorrelated in complex systems. Power law environments such as financial markets are defined by their interconnectedness, and the presence of positive and negative feedback loops which undermine their predictability. That interconnectedness makes identifying uncorrelated risk exceptionally problematic, especially when such correlations have been hidden inside repackaged derivatives and insurance products.

In systems that are intrinsically unpredictable, no risk management framework can ever tell us which decisions to make: that is essentially unknowable. Instead, good risk strategies should direct us towards minimising our liability, in order to minimise the cost of failure. If we consider “current liability” within the field of software product development as our investment in features that have yet to ship and prove their business value generation capabilities, then this gives us a clear, objective steer towards frequent iterative releases of minimum marketable featuresets and trying to incur as much cost as possible at the point of value generation (i.e. through Cloud platforms, open source software, etc). I think that is the real reason why agile methodologies have been so much more successful that traditional upfront-planning approaches: they allow organisations to be much more efficient at limiting their technology investment liability.

In the last post we argued for a more rigourous, quantitative approach to featureset valuation over the conventional, implicit and overly blunt mechanisms of product backlog prioritisation. We borrowed a simple valuation equation from decision tree analysis to give us a more powerful tool for both managing risk and determining the optimal exercise point for any MMF:

Value = (Estimated Generated Value * Estimated Risk) - Estimated Cost

where 

Estimated Risk = Estimated Project Risk * Estimated Market Risk

A few comments are worth noting about this equation:
1.) It contains no time-dependent variable. The equation simply assumes a standard amortisation period to be agreed with stakeholders (typically 12 to 24 months). Market payback functions and similar are ignored as they introduce complexity and hence risk (as described in more detail below). We are not seeking accuracy per se, but simply enough accuracy to enable us to make the correct implementation decisions.
2.) It is very simplistic. Risk management must be reflexive: at the most basic level, project risk can be divided into two fundamental groupings:

• Model-independent risks
• Model-related risk

The former includes typical factors such as new technologies, staff quality and training, external project dependecies, etc. The latter includes two components: the inaccuracy of the risk model and the incomprehensibility of the risk model. We start incurring inaccuracy risk as soon as the simplicity of our model is so great that it leads us to make bad decisions or else provides no guidance. MoSCoW prioritisation is a good example of this. On the other hand we start incurring incomprehensibility risk as soon as the risk model is so complex that it is no longer comprehensible by everyone in the delivery team (which will clearly be relative across different teams). The current financial crisis is a large-scale example of a collapse in incomprehensibility risk management. If financial risk models had been reflexive and taken their own complexity into account as a risk factor, then there is no way we would have ended up with situations where cumulative liabilities were only even vaguely understood by financial maths PhDs: if we take a team of twenty people, it is clear that a sophisticated and accurate model that is only understood by one person entails vastly more risk than a simplistic, less accurate model that everyone can follow. We can generalise this in our estimation process as follows:

Total Risk = Project Risk * Market Risk * Model Incomprehensibility Risk * Model Inaccuracy Risk

or as functions:

Total Risk = Risk(Project) * Risk(Market) * Risk(Incomprehensibility(Model)) * Risk(Inaccuracy(Model))

Furthermore, given our general human tendency towards overcomplexity for most situations this can be approximated to

Total Risk = Risk(Project) * Risk(Market) * Risk(Incomprehensibility(Model))

3.) All risk is assigned as a multiplier against Generated Value, rather than treating delivery risk as an inverse multiplier of Cost. I have had very interesting conversations about this recently with both Chris Matts and some of the product managers I am working with. They have suggested a more accurate valuation might be some variation of:

Value = (Estimated Generated Value * Estimated Realisation Risk) - (Estimated Cost / Estimated Delivery Risk)

In other words, risks affecting technical delivery should result in a greater risk-adjusted cost rather than a lesser risk-adjusted revenue. This is probably more accurate. However is that level of accuracy necessary? In my opinion at least, no. Firstly it creates a degree of confusion as regards how to differentiate revenue realisation risk and delivery risk: is your marketing campaign launch really manifestly different in risk terms from your software release? If either fails it is going to blow the return on investment model, so I would say fundamentally no. Secondly, I might be wrong but I got the feeling that part of the reticence to accept the simpler equation from our product management was a preference against their revenue forecasts being infected by a thing over which they had no control: namely delivery risk (perhaps a reflection of our general psychological tendency to perceive greater risk in situations where we have no control). However that is a major added benefit in my opinion: it helps break down the traditional divides between “the business” and “IT”. As the technology staff of Lehman Brothers will now no doubt attest, the only people who aren’t part of “the business” are the people who work for someone else.

For me, this approach creates the missing link between high-level project business cases and the MMF backlog. We start with a high-level return on investment model in the business case, that then gets factored down into componentised return on investement models as part of the MMF valuation process. These ROI components effectively comprise the business level acceptance tests for the business case. The componentised ROI models then drive out the MMF acceptance tests, from which we define our unit tests and then start development. In this way, we complete the chain of red-gree-refactor cycles from the highest level of commercial strategy down to unit testing a few lines of code. The scale invariance of this approach I find particularly aestheticly pleasing: it is red-green-refactor for complex systems…

A really awesome slide

When I first stated this law I noted that might add commentary on the notion of Power Law Training, specifically the application from Art De Vany. I like the idea, but disagree with the application.To generalize, De Vany is a fan of varied frequency, working out only when you really want to, when you feel good about doing so. He is also a fan of pyramiding weight up while dropping repetitions (15, 8, 4) as a means to fatigue all the fibers in sequence. It all sounds good in theory.

Here are the troubles with this:

1. 15 reps isn’t going to adequately fatigue the Type Ia fibers at all. Slow twitch fibers are most active during aerobic exercise, responding extremely well to work done in the oxidative energy pathway and aren’t going to be fatigued in 15 reps. To quote Lyle McDonald, ” It takes minutes to fatigue Type I fibers, probably longer, like hours.”
2. Intensity, specifically momentary muscular effort, isn’t the only variable for which the power law can apply.

So how can you apply the power law in another way? Here are a few examples:

1. Decrease “effort” but increase volume. If my max deadlift is 385, I could do single rep warmup sets until my main workset at 385. The Westside gurus have pointed out that adding more volume to your lower sets might reduce max force output, but you’re not maxing out. You’ll move more tonnage for more stimulation and this can be cycled in randomly…
2. If you use Chaos Training. The notion that your first “work” set becomes the benchmark in your routines and you perform whatever else you want for a given muscle group. As long as the benchmark is improving, you are doing well with the rest of the workout you’re making up as you go along.
3. A full Blitz cycle would be taking whatever your normal workload would be and, for a period of no more than 2 weeks, doubling or tripling the workload (both sets and work days). The intent is to push yourself to the brink of overtraining, back off and don’t train for a week, and you end up overcompensating with more muscle.
4. A variation for competition would be Pendelay’s Hormone Manipulation Cycle. Pyramiding up in intensity each week (low, medium, brutally high, brutally high, medium, low, competition) depresses hormones, setting them up for a big rebound for a competition in week 7 of that sequence.

The paleo folks (in which I find myself partaking in) are as dogmatic as any other fitness subculture. I might seem like I’m slaughtering a sacred cow, but there’s more than one word written about this. Give it a try.

Um aus der unglaublichen Menge von Blogs hervorzustechen, braucht es Verlinkungen, die regelmässig viele Leser auf die Website eines Blogs bringen.
Die wichtigsten Grundelemente von Blogs sind Blogrolls, Permalinks und Trackback.
Eine Blogroll findet sich auf jedem Blog. In dieser Liste werden Blogs aufgeführt, die vom Autor regelmässig gelesen werden. Blogrolls sind einerseits ein Navigations-Tool und andererseits eine Art der sozialen Anerkennung (Marlow, 2004, S. 3).
Jeder Post erhält eine feste Internetadresse, einen sogenannten Permalink. Damit ist jeder Blog-Beitrag und auch jeder Kommentar eines Blog-Lesers referenzierbar, was das Verlinken auf fremde Blog-Beiträge deutlich vereinfacht (Seeber, 2008, S. 19).
Trackbacks sind Links zwischen zwei Blog-Posts. Der Autor des zweiten Posts bezieht sich in seinem Text auf den ersten Post. Er kann nun ein Signal an diesen ersten Post senden, welches dort als Kommentar mit Link erscheint. Dadurch können die Leser des Ursprungs-Posts gleich in den Kommentaren sehen, dass andere Blogger über das selbe Thema schrieben.

Diese drei Elemente erklären, wie Blogs vernetzt sind. Welche Blogs nun häufig verlinkt werden und dadurch Aufmerksamkeit erhalten, kann durch die sogenannte Power Law-Verteilung erklärt werden. „We know that power law distributions tend to arise in social systems where many people express their preferences among many options. We also know that as the number of options rise, the curve becomes more extreme“ (Shirky, 2003).
In der Blogosphäre hat es nun viele Blogs, wie im letzten Post thematisiert, und es gibt viele Leser, die sich aus diesen Blogs Favoriten auswählen und auf sie verlinken. Man könnte nun eigentlich davon ausgehen, dass die Leser ihre Favoriten frei auswählen und damit die meisten Blogs etwa gleich populär wären. Dem ist jedoch nicht so, denn „people’s choices do affect one another. If we assume that any blog chosen by one user is more likely, by even a fractional amount, to be chosen by another user, the system changes dramatically. Alice, the first user, chooses her blogs unaffected by anyone else, but Bob has a slightly higher chance of liking Alice’s blogs than the others. When Bob is done, any blog that both he and Alice like has a higher chance of being picked by Carmen, and so on, with a small number of blogs becoming increasingly likely to be chosen in the future because they were chosen in the past“ (Shirky, 2003). Dadurch werden Blogs, die schon von vielen Personen als Favoriten ausgewählt wurden, immer populärer und erhalten viel Aufmerksamkeit. Es kommt zur oben erwähnten Power Law-Verteilung und das heisst, dass ein grosser Anteil der Links auf einen kleinen Anteil der existierenden Blogs eingeht. Diese Blogs werden auch A-Blogs oder A-List Blogs genannt.

Die Blogsuchmaschine Technorati berechnet einen Autoritätswert, der ebenfalls auf der Verlinkung unter den Blogs beruht. Ausschlaggebend für eine hohe Technorati Authority ist, wie viele Blogs in den letzten sechs Monaten auf den eigenen Blog verlinkten. Die Top 100 Blogs können bei Technorati angesehen werden. Zur Zeit hat der Blog The Huffington Post den höchsten Authority-Wert. Dies ist übrigens ein weiteres Beispiel für einen Blog, der dem Bereich Citizen Journalism zugeordnet werden kann.

Uno degli aspetti più interessanti della folksonomy, la categorizzazione condivisa  e distribuita tramite tag, è che riesce a creare piuttosto rapidamente un set di etichette stabili per ogni contenuto, evidenziato dall’apparire di una legge di potenza: pochi tag sono utilizzati da moltissimi utenti, moltissimi tag sono utilizzati solo da qualche utente. Questo accade a dispetto di diversi  problemi che tale tipo di tecnica comporta: differenze culturali e linguistiche, ambiguità dei termini, acronimi, parole multiple, difficoltà di etichettare dati strutturati come le date, solo per citare i più rilevanti.

Le cause che determinano la stabilizzazione dell’insieme dei tag e l’apparire di una power law non sono del tutto chiare: i sospetti principali ricadono su un possibile background culturale comune agli utenti e soprattutto sul meccanismo di imitazione, che spinge gli utenti a confermare etichette già presenti. Alcune ricerche si spingono ad ipotizzare che la curva di potenza non apparirebbe se non fosse data agli utenti la possibilità di vedere l’ insieme di tag già definito da altri.

Una ricerca appena pubblicata sembra però smentire tale ipotesi: la legge di potenza emerge anche nel cosiddetto blind tagging, il tagging senza suggerimenti dando più credito alla spiegazione sul background comune degli utenti. Ciò che comunque ancora manca, secondo gli autori della ricerca, è la comprensione dei modelli cognitivi che guidano la capacità di categorizzazione umana, ed in particolare quella tramite il tagging.

Hi everybody,

Well I finally finished Shirky and I have to say there were several reasons I enjoyed this book immensely.

1. As Jason said to me during a class break, Shirky had no agenda. At least if he did he kept it pretty much to himself and just informed us about the power of organizing.
2. He relegated Keen to one paragraph on page 209 with an example of Keen complaining about big time professional ad agencies losing money.
3. He explained the Power law. This is different from Pareto’s 80/20 law that says, for instance that 20% of the people do 80% of the work. It is more like, in Wikipedia, or another social network, maybe 3 people do 1000 or 10,000 times the other contributors, and yet it all works out. Most people do a below average amount in contributions whilst a few do a huge amount…but all are needed to make social online groups work. I think many studies could look at this power law.
4. The cost of organizing groups has dropped to nil with the tools on the net
5. I like how he explains that the price of failure has dropped to nil as well. It’s the fact that you can afford to fail that groups can form, or even try to form. Groups like the picture-sharing group from NYC would have had no audience in the past, but now can be seen around the world.
6. I love the amount of time he spent with meet-up and the fact that meet-up (and groups like it) can actually allow people to connect around common interests (often extremely nichy), geographically in real-life. Hence, the internet can bring virtual groups into actual real-life groups. I experienced this myself locally in the primary election cycle. I actually met up with some folks online and participated in a political parade in Knoxville with people that I met for the first time in real life that day. So, contrary to doomers and gloomers, the internet can foster local geographical community in ways that would have been imposible before the new tools for organization on the net got cheap and attractive.
7. Shirky is realistic in seeing that the genie is out of the bottle for good and bad and that both good and bad will occur and must be dealt with accordingly.
8. In the final chapter, Shirky explains the three main components needed to make a group work: the promise, the tool, and the bargain and how these tools must be lined up perfectly in order for an organization or group to “work”. It is just good to know!
9. Oh and also, I don’t know if Shirky ever said Web 2.0… and that is a positive at this point, IMNSHO.

I thought this book was extremely well written and I will look for more Shirky.  Will you?

John Cummins

As reported in my previous post on Debian Dependency Maps we started to study the properties of dependency relation and the kind of networks the relation can generate.  One preliminary study we (me along with Arnab K. Ray and Rajiv Nair) posit a  nonlinear model for the global analysis of data pertaining to the semantic network of a complex operating system (free and open-source software). While the distribution of links in the dependency network of this system is scale-free for the intermediate nodes, we found that the richest nodes deviate from this trend, and exhibit a nonlinearity-induced saturation effect. This also distinguishes the two directed networks of incoming and outgoing links from each other. The initial condition for a dynamic model, evolving towards the steady dependency distribution, determines the saturation properties of the mature scale-free network.

Here I give some of the motivations on conducting this study and some conclusions.
The full paper with all the technical details is uploaded at arxiv.org.

Scale-free distributions in complex networks have been very well studied by now. The ubiquity of scale-free properties is quite noteworthy, and spans across vastly  diverse domains like (to name a few) the World Wide Web and the Internet, the social, ecological, biological and linguistic networks, income and wealth distributions, trade and business networks, and semantic networks.

It should occasion no surprise, therefore, that further developments have led to the discovery of scale-free features in the architecture of computer software as well. A recent
work  has shown that the structure of object-oriented software is a heterogeneous network characterised by a power-law distribution.  More in keeping with the purpose of this present paper, an earlier work on complex networks in software engineering had found evidence of power-law behaviour in the inter-package dependency networks in free and open-source software (FOSS).  It is a matter of common knowledge that when it comes to installing a software package from the  Debian GNU/Linux repository, many other packages — the “dependencies” — are also called for as prerequisites. This leads to a network of these dependencies, and every such package may be treated as a node in a network of dependency relationships. Each dependency relationship connecting any two packages (nodes) is treated as a link (an edge), and every link establishes a relation between a prior package and a posterior package, whereby the functions defined in the
prior package are called in the posterior package. This enables reuse (economy) of functions and eliminates duplicate development. As a result the whole operating system emerges as a coherent and stable semantic network. However, unlike other semantic networks, the network of nodes in the Debian repository is founded on a single relation spanning across all its nodes: Y depends on X; its inverse, X is required for Y .
So, given any particular node, its links (the relations with other nodes) can be of two types — incoming links and outgoing links — as a result of which, there will arise two distinct kinds of directed network.  For the network of incoming links, a newly-reported work  has empirically established the relevance of Zipf’s law and the conditions attendant on it in Debian GNU/Linux distribution. Carrying further along these very lines, the present study purports to analyse and model the finite-size effects in a FOSS network. There is a general appreciation that for any system with a finite size, the power-law trend is not manifested indefinitely, and in the context of the FOSS network, this is a matter that is recognised as one worthy of a more thorough investigation. Deviations from the power-law trend appear for both the heavily-linked and the sparsely-linked nodes. The former case corresponds to the distribution of a disproportionately high number of links connected to a very few special nodes — the so-called “top nodes” (or rich nodes).  The importance of these nodes is, therefore, a self-evident fact.

The data needed for the modelling pertain to the current stable Debian release, Etch (Debian GNU/Linux 4.0).  The respective networks of both the incoming links and the outgoing links span 18630 nodes (software packages).
The study argues for the significance of non-linearity and saturation, as regards a quantitative characterisation of the incoming and outgoing distribution in the Debian GNU/Linux network.  One might rightly expect to encounter similar features in other networks.  And indeed, given the possibility that the entire network of software packages in an operating system can be construed to be a semantic (albeit non-autonomous) system, its characteristics can furnish a model that can shed light on much more complex but realistic autonomous semantic and cognitive systems, such as the human society, or
even the human mind.

In the road ahead, the gnowledge lab will conduct a similar study for the dependencies between concepts and activities as and when we obtain sufficient number of nodes at gnowledge.org.  Currently we have only about 1000 dependency relations.  As more people get to know about the need of establishing dependency relation between concepts, and as and when the portal itself matures with features to attract more users we can study the properties of the resulting knowledge network.

The full paper with all the technical details is uploaded at arxiv.org.