I came across this flowchart that helps you determine the best chart to use based on the kind of data you are trying to get across.   I know that for me I have a tendency to rely on a few “go-to” charts that I’ve come to rely on when in fact there may be a better option out there that more elegantly expresses exactly what I’m trying to communicate.  This chart looks like a great resource to use to check your instinct against best practice.

Click on the image for a larger and easier to read version.

Source: http://extremepresentation.typepad.com/blog/2006/09/choosing_a_good.html

1. Healthcare .  In this visualization, GE took 500,000 records from the millions in its electronic medical record database, and calculated the out of pocket and insurer cost of a handful of chronic conditions by age.  One of the few both uses of and displays/designs of a radar graph, the visual changes as one increases and decreases age.  This also has a predictive analytics component in that it answers a user’s question “What if I develop a case of hypertension, what will that cost me when I’m 65?”  Like any good analytic, being able to see the data brings up more actionable and specific questions that this analytics doesn’t answer but the data set could.
2. Digital Media.  This visualization is part of The New York Talk Exchange, a visualization project developed by the Senseable City Lab at MIT.  Perhaps the potential applications of the analytics are as explosive or more than the specific data they used in this case.  The analytic shows starting or sourcing neighborhood within NYC and to where their communications via the AT&T network were destined.  Users can see the frequency distribution of endpoints, and by comparison who was talking to whom geographically across sister boroughs.  Imagine creating a site map of your web site, application, team or workflow, and seeing the frequency of where the user, function, business process or capital goes next.  Predictive analytics can say: “If we invest in area x where is that capital, profit opportunity, or waste most likely to go next, and how does that change if I make another investment?”  Perhaps only this specific visualization is best for that type of comparative predictive analytic.
3. Retail.  According to Well Formed Data, Sankey Diagrams and stacked bar charts informed this (4MB .pdf download) time series visualization of how medical journals in related fields merged into a cohesive ‘basket’ of journals in the emergent field of neuroscience.  While on the surface not retail related, it points to a very compelling—and as yet a visualization I’ve never seen produced— which would explain how specific products that drive volume and profit affinitize into specific types of market baskets.  Replace a) each journal with a specific retail product, which the user can color code at run-time for visualization, perhaps color-coded for on/off promotion, b) the eigenfactor of each journal which is represented by the weight/width of the specific line with the amount of profit or volume the item produces—again the user can choose profit or volume or some other measure at run time, c) the ten or so portfolio of ending blocks of journals as specific types of market baskets, and d) the breadth of starting lines moves from medical disciplines to aisles or departments in a store.  In effect what you have here as a decade long time series one has compressed into a single, specific shopping trip.  Data can include many trips, a single store, one day or years of data.  The predictive angle is to be able to answer questions like “If I promote this item, does it move away from its core ‘7 items per basket, quick trip’ basket into a ‘destination item weekly stock-up’ basket?”  One can also look historically how different store consumers shop categories, volume vs. profit items, and more.

These predictive analytic visualizations start with healthcare as the least complex and become more complex.  Depending on how frequently the user needs the data updated, the amount of data (in the digital media/telco example clearly tens or hundreds of gigabytes), the processing (in the retail example, calculating eigenvalues in an analytic that could be analyzed hourly, especially for promotional out-of-stocks, for example), and the speed required means a predictive analytic visualization is not something to try at home with an off-the-shelf database platform and hardware.

http://www.salescentric.com/relationship_charts.asp

Here is Information visualization et. al., Pedro M Cruz’ master thesis blog, including a project visualizing Os Lusíadas.

Whew. I can relax.

For the past 2-3 months, I’ve been working on my first real Objective-C project (my iphone app is still going, it just took a back seat to this): An application that will read tcpdump output and animate the packets over time using their inherent byte / packet structure

And now…it’s up and in beta-ish quality. (Meaning it works, though some error checking and minor features arent quite where I want them.)

You can download it here for free: http://sintixerr.wordpress.com/pkviz-packet-visualizer-and-animator/

See it in motion here:

This project was important to me and has been a long time coming. I’ve wanted to write a packet visualizer since I first started working with data viz 5 or so years ago at NetSec and was using Advizor. That tool cost thousands of dollars per seat, didnt really animate (at least the way I needed), and only parsed CSV or databases. The free tools – like GnuPlot, just weren’t up to the task at all.

I also wanted something that could plot out data in interesting, pretty ways for some art projects I have in mind.

So, I originally started this time around on a quest to write a short python parser for tcpdump ascii hex output to put into <some generic viz tool> just to get started…but somehow I ended up writing a full-fledged visualizer (my first GUI project ever, I might add!). The learning process was a blast – I feel like I’m a much better coder for it – and I’ll be able to extend/expand on this to use for other art and security projects that are on my plate or are coming up.

I’m pretty excited about it. To see this finished through after years of whining to myself about it, procrastinating, and genuinely not having enough time, is pretty awesome. I’ve even already created a couple of cool shots that I’m happy to call “art” (granted, there is some photoshop processing here, but they’re both true to their originals!):

Anyway, Mac Users, check out the tool and let me know what you think!

The Economist just published a fascinating article on fluid dynamics called “The Skeleton of Water” which may have important implications for Crisis Mapping Analytics. We often speak of conflict in terms of waves, e.g., “preventing the next wave of conflict,” but there may be more to the analogy than meets the eye.

Source: The Economist

One of the main challenges that Crisis Mapping Analytics faces is the complexity involved in modeling conflict dynamics over space and time. But we’re hardly the first to face this challenge. Take the field of fluid dynamics, which is the subject of the article in question. Breakthrough research now reveals that hidden structures within liquids and gasses guide the movement of everything from pollution to airplanes.

The atmosphere and the ocean are, it seems, dominated by invisible barriers that have come to be known as Lagrangian Coherent Structures.

To understand what a Lagrangian Coherent Structure [LCS] is, it helps to imagine a crowd at a railway terminus [...]. Some people will be arriving. Some will be leaving. And, whichever they are doing, they will be going to and from numerous different platforms.

The result is chaos, but structured chaos. What emerges is a shifting pattern of borders between groups of people with different goals. These borders are Lagrangian coherent structures. They are intangible, immaterial and would be undetectable if the passengers stopped moving. But they are also real enough to be treated mathematically.

And it turns out to be easier to study the behaviour of such fluids by looking at the barriers than at the bodies of fluid which those barriers separate.

Scientists across numerous fields are now jumping on the Lagrangian bandwagon to explore the applicability of LCS to their work.

For example, a professor at CalTech recently “used the technique to study the hunting behaviour of jellyfish. He has shown how parts of the ocean are temporarily protected from the depredations of these creatures because they cannot cross the invisible barriers imposed by Lagrange.” Meanwhile, another professor notes that “the coherent-structure approach might also be used to help predictions of the passage of hurricanes; they, too, are constrained by the invisible barriers that Lagrange’s theory describes.”

There is also reason to believe that conflict follows a power law (like earthquakes, forest fires, etc.). In other words, conflict may be scale invariant. This may imply that there is no “average size” of conflict regardless of scale, i.e., from interpersonal to international conflict. As it turns out,  “Lagrangian coherent structures can appear at all sorts of scales. What goes for an airport or a bay can be scaled up to an ocean, or the air above it, and down to the flow off an aircraft’s wingtip, or a ship’s hull.”

One of the exciting conclusions from this latest research in fluid dynamics is that “boundaries between things are often as important as the things themselves.” In other words, we need to think more about what artists call negative space, which is “ the space around and between the subject(s) of an image. Negative space may be most evident when the space around a subject, and not the subject itself, forms an interesting or artistically relevant shape, and such space is occasionally used to artistic effect as the “real” subject of an image.”

Take the following picture, for example.

Source: Wikipedia

There is an intimate relationship between the object and the space around it. Negative spaces are important because they may reveal otherwise hidden patterns. In crisis mapping, like fluid dynamics, we need to delineate this negative space over time to identify patterns in three dimensional space. This is what the mathematics of Lagrangian Coherent Structures may allow us to do.

And so, it is very tempting to start thinking about conflict (and peace) in terms of fluid dynamics given the context of crisis mapping. Have we been too preoccupied with crisis data instead of modeling the boundaries and event trajectories? Perhaps conflict flows like a river, with turbulent eddies producing violent pathways while other areas are clear, still pools of peace. So why not collaborate with mathematicians to find out whether LCS’s can shed light on why “islands of peace” sometimes exist in a sea of conflict?

Patrick Philippe Meier with Guest Blogger Jen Ziemke

This word is the way i learn, but also one of my biggest downfalls–both in the organization as well as now–why??? i learn mainly by visualization-memorization. If i see it in writing it is so much more real then hearing it–sitting in a lecture in college–YUCK–for me, doing it online–and actually reading the information, seeing it, etc—so much easier for me to understand, remember, etc–and that was exactly my downfall in the organization i was in—the things i signed (Contract, as well as my evaluation)–people could call me any kinds of names, bad “christian”, bad mom, fat, ugly, etc–i have probably been called worse–hearing it–honestly would go in one ear and out the other–screw them, if they don’t like me–their problem—the difference came when i visually SAW what was wrong with me—it is now in writing–i could see it—and in some cases was actually shown it in the “bible”–not only was i hearing my faults, i was seeing it, and NOW was agreeing to it–then i started questioning so much about me—

and it hinders me now–i can not visually read, look at the “bible” (for a variety of reasons)–if someone tells me the answers, truth is in the “bible” at this point does not help me much–i know that, i want that, but………… right now cant. some have mentioned hear/listen to it on DVD/CD, etc–and another thing i struggle with anyways-but learning in such a visual way–hearing it really does not help me much at this time—or hearing over and over “god” loves you i think, yeah whatever–i want to visually see the proof–not the “god” loves you and nothing else, but the proof, the writing proof if you know what i mean–i try to research different things on line here–sometimes it is good–but most of the time it is again a hinderous–because i visually see something–visually see the proof they put (right or wrong) then there goes my questioning again–i was just told this, now i read this, what is true, what is right, hence there goes the confusion.
Even when i ask questions (which i do ALOT) there are very few people i really “believe”(unfortunately until i find something online again LOL)–still want the writing-and maybe that is why i do not talk much on the phone–i try to save everything that is written to me here on line–so i can go back and read, and or visualize it later (if that makes sense)–and it.

That i guess to is a big reason why i read so many books on the subject of spiritual abuse–and write poetry.

I hope all that even makes sense, it does to me.

It doesn’t cost you anything to think.  Your thoughts are as free as free can get.  And those thoughts will bring you ideas of what you desire and those thoughts breed feelings and focus of what you want and that is all free as well.  So why can’t the life you want, that comes from your thoughts and feelings, come free and clear right to you?  The start of it all is free and so is the end result.  Everything else will fill in between the lines. Free thought, free end result, and free in between as long as you envision just that.

We had such a wonderful day yesterday.  Loads of family, loads of food, and loads of laughter…just as it should be.  Mom is doing very well.  She is still feeling good.  Yesterday kind of took it out of her but I can hear the coffee grinder going this morning so she’s got her energy back.  I have so much to be thankful for and you are one of thankful thoughts.

www.bethandleemccain.com

I may have had an insight into my own “persistent episodic visual amnesia” (PEVA). This is what I call the problem talked about in the Visualisation FAQs page. “PEVA” is my own name for the problem, which others have shared here too. None of us have found any facts about a known medical condition.

One of the things that I notice is that I have vivid, full-colour dreams. But these dreams are almost always set in the distant past. They almost never take place in the period after my early twenties when PEVA first started. But in the early hours I awoke with a migraine. The dreams I had after I fell asleep were in the present. The setting was not the usual stereotype of a past family home. The characters in my dream were my current partner and one of her boys.

I am diagnosed with persistent migraine aura, and frequent and severe migraine. About half those who have commented on this site have a migraine diagnosis. This is a high number, but it is a statistically small sample. One of the ideas my family doctor has is that PEVA is due to my persistent aura. This is a constant electrical disturbance in the brain. In me, one way it shows up is through visual snow. For more information, please see the PMA & Visual Snow FAQs page.

So, it is interesting that I should have rare dreams of the present during a migraine. The attack was still there when I awoke after the dreams. I don’t know how that may work, as the theory is it is permanent aura that disrupt my memory. Also, it is not scientific. One example is not statistically significant. And one person’s anecdotal report is also suspect. But without facts we can find, it could be a start.

If anyone else reading this has PEVA, please could you let me know:

• If you have been diagnosed with migraine or not,
• If you have persistent aura such as visual snow or not,
• If you have vivid, full-colour dreams or not, and
• If your dreams are usually set in the past?

Take care.

Sometimes it can be confusing as to the right kind of action for a loved one.  In our feelings of love for someone else we can become confused what to do.  In order to make an ‘inner’ decision it is important to detach from the personality/physical self of another. When we think of love on the physical self level instead of the higher plane level it can form a very limiting bond between you and the person you are confused about.  The only responsibility that you have is to yourself because if you take responsibility for another it becomes limiting for everyone involved.   You can release the situation to the Universe knowing an answer of ease will come in return.  Once you release it,  see it as taken care and keep moving forward in your own life.  Another will only have release in their own life according to where they are in their own thoughts. They are choosing their experience…not you.  You can continue to create your own life and release any feelings that may hinder you.  They are responsible for changing their circumstances, not you.  Just be loving and accepting of who they are.  They don’t need fixing.  They are creating their life and even though it may not be what they (or you) want, accept the ‘being’ they are within.

A wonderful day full of turkey and all the trimmings!  We have more family coming in today so we have a houseful!  Guess I better get into the kitchen and cook breakfast.  Have a wonderful day!

HuCo 500 – Weekly questions

An affordance, it is said, points two ways, to the environment and to the observer.  …It says only that the information to specify the utilities of the environment is accompanied by information to specify the observer himself, his body, legs, hands, and mouth.  This is wholly inconsistent with dualism of any form…  The awareness of the world and of one’s complementary relations to the world are not separable.(Gibson, 141)

Maybe I am misunderstanding Gibson’s point, but the claim that the theory of affordances dispels the notion of dualism of any kind seems to be quite a leap, making assumptions about what reality and perception are.  How do you separate subjective sense-making (i.e. perception of one’s environment/reality) from objective fact (what Gibson calls “invariants”)?  It is certainly true that the affordances of an object as evaluated by a biped, for instance, will be different than those evaluated by a quadruped, and that either evaluation says something about the evaluator (i.e. that they are a two-legged or a four-legged).  That doesn’t mean that the act of evaluating—the interpretive or perceptive act—is not subject to the context and experience of the ego.  Gibson makes allowances for “misinformation” and “misperception” (142); I’m not sure how he can reconcile this allowance with his claim that there can be no mind-body/abstract-concrete/mental-physical dualism.    

Is there still such a large gap in the affordances of paper compared to computers, as Gaver describes (115-117)?  Do computers today offer all the same affordances of paper?  Is there anything that paper affords that computers/cell phones/electronic devices today do not?

Readings:

Gaver, William W. “Situating Action II: Affordances for Interaction: The Social Is Material for Design.” Ecological Psychology 8(2), 1996. 111-129

Gibson, J.J. “The Theory of Affordances.” The Ecological Approach to Visual Perception. Hillsdale, NJ: Lawrence Erlbaum Associates, 1986.

Take a look at Kurt’s weekend hack to make a visualization of the Echo Nest artist similarity space.  Very nice.  Can’t wait for Kurt to make it interactive and show artist info. Neat!

Wired Magazine always publishes some interesting videos dedicated to science and, let’s say, fun. Here are the entries and a few sample videos.

Best Science Visualization Videos of 2009

Volume rendering is a technique used to display two-dimensional data in three-dimensional space. ImageVis3D is a new volume-rendering program developed by the NIH/NCRR Center for Integrative Biomedical Computing, designed to be simpler, faster and more interactive than standard volume-rendering programs. This video demonstrates some of the key features of ImageVis3D and gives examples of the type of data it can render in three dimensions.

Top 10 Scientific Music Videos

Today’s ‘Live Better With Willie Jolley’ Tip: Fly Like a Bumblebee

No fear. No doubt. This is it, look out! Success is an amazing adventure. It is the process of having a dream and then pursuing that dream whole-heartedly. It is getting to the point where you believe it is impossible to fail.

I had the opportunity to meet Mary Kay Ash, the founder and president of Mary Kay Cosmetics. Mary Kay Ash is a remarkable woman who had a dream that others called impossible. She refused to listen to the naysayers, and went on to build a business that is now one of the largest cosmetic companies in the world. The symbol for her company is a bumblebee and it represents the ability to do the impossible.

Why? Because scientists and animal experts have emphatically stated that it is impossible for the bumblebee to fly because it has small wings and a big body. But the bumblebee doesn’t know that it is impossible for it to fly, so it flies anyway!

The only one that ultimately keeps you from being great is YOU, and your limiting beliefs. Stop saying, “I can’t”, and start saying, ‘‘it’s impossible for me to fail,’’ and you, too, will do unbelievable things!

Visit my website at www.williejolley.com for free motivation and make this a great day!

Think about the difference between positive and negative thinking and how it makes you feel.  When positive, it feels as if some kind of magic is coming in When you think negative, what is there?  Nothing but fear, doubt, and feelings that make you feel worse.  When you think positive about your life and all that you can accomplish there is a sense of excitement and joy in having the life you desire. With negative thoughts you feel deflated as if life is hurtful.  Do your best to see the positive in your life.  This doesn’t mean that you don’t take care of what is in front of you or that you ignore what life seems to be bringing,  just start retraining your mind to shift focus to something better.  Do things a bit differently that day.  Give a smile to someone on the street and start focusing on positive things you can do with your life right now.  As you feel more positive and do more positive things, the Universe can feel from you what you want and it will come right to you through the magic of positive thought.

Lee is holding down the fort today while Mom and I go to the ‘big’ city for her radiation therapy.  She’s feeling wonderful and is meeting so many new people.  She’s funny.  She is such a private person but when she gets out in public, she becomes Ethel Merman…over the top happy, outgoing, and friend to all!  She even sings!! Love that Mom!

Nathan at Flowing Data explained how to create a US County thematic map using free tools and it immensely helped me doing the same for Russian regions.

The Result

Inward FDI Performance Index 2005-2007

Step 1: Get the blank map

Unfortunately, we can’t take the blank Russia SVG map from Wikimedia Commons. First, it doesn’t take into account the latest mergers in 2008 (Irkutsk Oblast + Ust-Orda Buryat Autonomnous Okrug = Irkutsk Oblast, Chita Oblast + Agin-Buryat Autonomous Okrug = Zabaykalsky Krai). Second, the regions aren’t identified in the SVG file by their ISO_3166-2:RU code, but by their concatenated transliterated names. Third, the Federal Service of State Statistics doesn’t publish figures of several regions (like Yamalo-Nenets Autonomnous Okrug) because they fall under the administrative jurisdiction of other federal subjects of Russia (in this case Tyumen Oblast). It means that we have to group these regions together so that they can “inherit” the data visualization from their respective jurisdiction.

Get the new map I just uploaded to Wikimedia Commons here.

Step 2: Run python script

You can simply follow Nathan’s blog post and write it yourself, or use mine instead. I wrote it because I am too lazy to rewrite the script each time I want to colorize a map with different parameters.

Get it here (don’t forget to get BeautifulSoup first),: http://pastie.textmate.org/713719

Sample usage: python chloropleth.py -d Inward_FDI_Performance_Index_2005_2007.csv -c RdYlBu6.dat -l IFPI.dat -i Russia2009blank.svg -o IFPI.svg

Parameters

• -d : Region specific data. Remove the headers and save it as a CSV file. Put the ISO3166-2:RU code into the first column, and the value into the second column
• -c: Color scheme. It’s a simple text file containing the hexadecimal colors in reverse order, e.g. #D73027 #FC8D59 #FEE090 #E0F3F8 #91BFDB #4575B4
Use ColorBrewer to help you select the colors to use.
• -l: Legend file, e.g. 1 0.5 0.25 0.10 0 -0.1 The script will assign the last color in your list if the data point is above the first value in your legend file, and so on
• -i: Input SVG file, i.e. the blank map
• -o: Output SVG file

Enjoy

By now you know I love data visualization and that’s why I feel compelled to show this dynamic map of unemployment statistics from January 2007 to September 2009. The choice of colors from pale yellow to black  make a dramatic impact as the map turns darker and darker. For some reason this reminded me of Dagney Taggart’s view as she flew over the world at the end of Ayn Rand’s  incendiary novel, Atlas Shrugged.

Magnetic resonance imaging opened a new chapter in the history of medical diagnostics, but it still cannot answer all the questions. Researchers at the Iowa State University came up with a wonderful solution. They developed a software, BodyViz, that can convert common 2D MRI and CAT scans into 3D visualizations, enabling physicians to navigate inside the body using an Xbox controller.

Two-dimensional imaging technologies have been used in medicine for a long time, said  (BodyViz co-founder) Eliot Winer, an Iowa State associate professor of mechanical engineering and an associate director of Iowa State’s  Virtual Reality Applications Center. But those flat images aren’t easily read and understood by anybody but specialists.

“If I’m a surgeon or an oncologist or a primary care physician, I deal with patients in 3-D,” Winer said.

(The creators) like to quote a doctor who told a reporter that when preparing for complex procedures, “2-D is guessing and 3-D is knowing.”

Hans Rosling's Data Graphics at TED Conference

Edward Tufte is fantastic. If I had a log cabin I would go and read his books for days. After purchasing a film projector last week, I have immersed myself in watching TED talks projected large scale in my home. Also fantastic. Hans Rosling’s 20 minute lecture on being able to visualize statistics is simple phenomenal. Countries are represented in circles of different sizes relating to population and are placed on an X,Y axis of health and wealth and Rosling animates the graph moving the placement of those countries on the graph as time progresses. We see countries evolving or failing. Reassigning the X,Y values, we see life expectancy rise up in most countries but literally see sub Saharan countries fall back with HIV-related deaths. Who know that watching moving circles could be an emotional experience – but it is. The bottom line is that so many types organizations have access to this data, but Rosling makes the point that this data is sitting in databases not being comprehended.
While so many major global organizations can benefit from this instigating initiatives, museums can simply play the role of projecting this information for the public.
Thanks TED.

HC Smith

Reef fish food web, Greater Antilles

Here are a couple of renderings of the vertebrate-only component of the coral reef food web. Reminder: the food web is what we expect to see for a reef in the Greater Antilles of the Caribbean, based on data collected around the mid-20th century. The vertebrate component comprises all fish and sea turtle species. The upper figure is the expected food web, and includes 196 species and 995 trophic interactions. Species are arranged on the periphery of the diagram, with interaction represented by the lines crossing the interior. The very busy, or hub species are higher trophic level predators, mostly carcharhinid sharks.

Jamaica coral reef fish food web