**********

NO BOUNDARIES, NO LIMITS, TAXING IDEOLOGY

Peaks, Troughs and Averages – independent of Application and Scale -
in new Depth, Detail and Perspective

Vertical Data Layers along a ‘visual z-axis’

Profiling Data is based on layering complex data[3] into vertical slices along a ‘visual z-axis’. This disruptive software method processes multivariate, univariate, sequential and time series data alike. It is now apparent that its algorithms can be usefully applied to bridging knowledge gaps between finance, economics, business and science.

In particular, market data, business intelligence, medical and climate data, measurements, sensor data, experimental results, clinical and experiential data can all be visualised with more depth, detail and other advantages:

• ‘Extreme data profiles’ show minima, maxima, averages and other statistical measures and derived metadata:
  • This allows for sorting, ranking, weighting and prioritising visually.
• A new forecasting mechanism can be applied for short-, medium and long-term trends:
  • Overlaying these trends allows for new predictive time windows.
  • Parameters are derived from historic data to fine tune the generic approach.

In business, time is money. In life and science, knowledge is power. It is therefore a promising perspective to examine business and science related data with the same tool!

In the spirit of Humboldtian Science[4], ‘software lenses’[5] are state of the art measuring instruments to compare hitherto incomparable data and images from finance, business and science. Visual assessments by data experts, combined with numerical processing power, allow for creating ‘Smart Knowledge Portals’, e.g.:

• Smart Particle Physics: meta views of individual as well as grouped experiments;
• Smart Science: the data source and expertise determine domain, scale and insights;
• Smart Energy Consumption: systems of control and communication are fine tuned and tweaked as peaks, troughs and other numerical metadata are aligned over different time spans.

It is hoped that collaboration can be created between CERN and 3D Metrics[6] to accelerate science:

• roughly 50 Years after Computer Operations were established[7]
• 25-20 years after the world-wide web was born[8]
• hopefully appreciating the potential of ‘software lenses’ as ‘inter-data processors’ and new instruments of investigation[9] – for meta views and in-depth analysis – of data- and measurement-based science.

PS. I took the phrase ’from the impossible to the imaginable’ from Ben Segal‘s review of the book on How the Web was Born.

Related articles

• SMART Knowledge Portals as New Instruments of Investigation – for lifelong learning and education (3dmetrics.me.uk)
• Why Open Source Is Essential To Big Data (10gen.com)
• Visualisation – the key that unlocks data’s value? (cloudave.com)
• Big Data Infographic | How Big is Big Data? | Domo | Blog (domo.com)
• How Are You Managing Big Data? Data, Data Everywhere | Domo | Blog (domo.com)
• The Rise of Big Data (nationalresearchcenter.wordpress.com)
• Keeping an eye on big data (irishtimes.com)
• Lessons From A Crash Course In Data Science (fastcolabs.com)

Wine Me, Dine Me:

Does wine speak make you feel super inadequate?? Find out what a tannin is other useful stuff to make you feel smarter around those wine snobs.

Luscious Locks:

Never have a bad hair day again.

Nesting:

Save money and avoid toxins.

DIY:

Got some old furniture kicking around?? Polish it up and give it a new look.

Techie Love:

Block websites from collecting your data.

Climbing the Ladder:

How to win over any interviewer.

Body Lovin’:

Trouble falling asleep? Try listening to the most.relaxing.song.ever.

Inspirational:

Being alone doesn’t mean your lonely.

Giggles and Such:

Punch yourself in the face. Right now.

Have a great weekend!

In the age of information, big data is getting bigger. The rate of data growth is so fast in fact that 90% of the data existing in the world today has been created only over the past two years. This wealth of information comes streaming in from a multitude of channels across many industries. The question for educators and learners is how to use big data to improve the educational experience.

Before we can start leveraging such information, we should understand the realities of big data and common myths surrounding it.

Myth 1: The more data, the better.

Two factors are involved in terms of quantity. First, there’s the sheer volume. Second, there’s the breadth of data; the range of types of information available. When it comes to both, more isn’t necessarily better.

At best, an overabundance or unbalanced variety of data can lack any entirely clear results and at worst, can actually produce conflicting outcomes. Having a heavy volume of one certain kind of information might lead analysts in the wrong direction. For example, a large quantity of transaction data might leave out a great deal of insight provided by social interaction data.

Given the rate at which our society produces new stats and figures, big data can easily become out of date. Gathering mass amounts of information for the sake of volume may not always prove helpful. At some point it runs the risk of losing relevance. Therefore determining which pieces of data to use is of high importance.

Myth 2: Data Scientists are the only ones who can effectively extract insight from big data.

Described by some as “part analyst, part artist,” the big data scientist is often thought of as the only key to unlocking the solutions found in mass quantities of information. While that might have held true at one point, new data tools are making it easier for all education leaders to engage and utilize big data as well. For example, by presenting information in a visual format, such as a heat map, these tools allow more people to grab hold of high level directional indicators and recognize what to focus on within their field.

Myth 3: Big data alone will provide solutions to all our problems.  

There’s no arguing that big data has the potential to deliver incredible insight into human behavior and lead to successful solutions based on that insight. But in order to arrive at truly meaningful results, big data most often needs to be placed in context.

Take information derived from Facebook as an example. A post might receive a myriad of “likes,” but this action could have a variety of reasons behind it. A user could “like” a comment because his or her thoughts align with that of the specific post or it could simply be because the user has an emotional bias towards the person who posted. Without an understanding of the context behind data points, results and the solutions that are derived from them run the risk of overlooking important variables.

In the education realm, successfully leveraging big data can help teachers identify the kinds of learning materials that will be most helpful to students on an individual level as well as provide valuable insight into educational trends across an entire school system. With more options at our fingertips, we can start embracing the big data movement from all positions throughout the industry; tapping into its potential in order to facilitate improvement within the education system.

If a church or government were doing these things, it would feel authoritarian, but when technologists are the culprits, we seem hip, fresh, and inventive. People will accept ideas presented in technological form that would be abhorrent in any other form. It is utterly strange to hear my many old friends in the world of digital culture claim to be the true sons of the Renaissance without realizing that using computers to reduce individual expression is a primitive, retrograde activity, no matter how sophisticated your tools are.” ~ Jaron Lanier, You are Not a Gadget

facebook (Photo credit: sitmonkeysupreme)

I think Facebook might, on some deeply hidden level, actually be a bit dodgy. Thinking about human nature, there is at least the possibility of some truly slimy values driving the development and changes to this social media behemoth.

Nevertheless…

Facebook for me, fulfils exactly what it’s purported purpose is ~ to keep me connected with my family half a world away. That’s all I expect it to do. So on the whole, FB does an exceptional job in this role.

I don’t expect Fb or it’s platform, ideals, or agenda to become my social life. I don’t expect Fb to replace my face-to-face friendships, my Friday Social Circle, or the opportunities I have to be in the actual presence of my daughters, grandsons, family, or friends. I don’t “friend” people I don’t know because my definition of friendship is much deeper than simply having common likes or dislikes. I don’t post things I wouldn’t want my grandsons to see in 30 years. I don’t talk about things I shouldn’t, I keep a boundary between my personal and professional lives, and I don’t forget that what people post/write/picture of themselves is generally their idea of the “good parts version.”

Lastly, I stay on top of privacy settings, simply because to me, keeping my personal life personal is a no-brainer.  ’Personal’ happens along a continuum ~ my idea of personal is much, much broader than Bill’s, and our respective pages reflect this difference. This aspect (privacy) is no different than turning my back to the rude eavesdropper at the next table in the local coffee shop. I’m not going to leave, but I’m not going to share willingly, either.

If Fb is data mining (and why wouldn’t they? It’s not a charity endeavour) then whatever information they get from my likes and dislikes, web surfing, etc., is going into the pool with the millions of others who have similar tastes as me. If I’m then susceptible to the annoying ads that have now become part of my news feed, that would be my problem – not Mark Zuckerberg‘s.

In all likelihood, Fb will become more and more commercial, and if/when it gets to the point where the annoyance far outweighs the value gained in maintaining contact with distant loved ones, I’ll look for a smaller, less hyped app. There will definitely be some, and when I find what I’m looking for, me and my friends will migrate en masse and wave goodbye to Fb.

In the meantime, I take the suggestions about security seriously. Take a moment to read through the article below and make sure you’ve done what you need to do to protect yourself.

How Facebook Exploits Your Private Information | LinkedIn.

Related articles

• LinkedIn Just Became My Go-To Social Network (fool.com)
• 3 Things You Should Never Do On Facebook (prweb.com)
• Fav: “Jaron Lanier: Why Facebook Isn’t Free” #dbpvids (dbpxhaust.wordpress.com)
• Stalked On LinkedIn: Victim Is Not Alone in Her Fight to Block Abusive Members (socialtimes.com)

But the new guidelines also featured added restrictions on an especially sensitive area of FBI counterterrorism work: mosques. Under the new rules, agents could no longer enter a religious organization without special new approval—in some cases directly from FBI headquarters.

SOURCE @ swampland.time.com

Reblogged from Indonesia Proud:

Tidak dapat dipungkiri bahwa jumlah publikasi penelitian Indonesia memang rendah. Namun, ternyata kualitas riset Indonesia tergolong unggul.

Hal itu terlihat dari hasil analisis Thomson Reuters, sumber informasi intelijen terkemuka di dunia untuk perusahaan dan para profesional, yang disampaikan dalam konferensi pers di Jakarta, pada 26 April 2012.

Diketahui, secara ranking, jumlah publikasi riset Indonesia tergolong kedua terendah se-Asia Tenggara. Berdasarkan jumlah publikasi, negara yang paling produktif dalam riset berturut-turut adalah Singapura, Malaysia, dan Thailand.

Read more… 205 more words

Posted on May 1, 2013by ilpatriot

Algorithms Every Data Scientist Should Know: Reservoir Sampling

In the last few years we have produced more data than in all of human history. We live our lives constantly producing a stream of data, it controls our lives, not in a Matrix or Skynet kind of way, but every time we interact (text, call, tweet), conduct a transaction, perform an internet search, complete a national census or even simply give birth or die, you are creating data and contributing that in the right hands is valuable and powerful tool. A recent EMC study claims that less than 1% of global data is actually analysed.

With smarter data mining algorithms and more powerful computers, data mining on a mega scale is becoming a reality, an era of Big Data. This is a dataset so large that it holds many potential secrets about us and the universe beyond. We like to think that as individuals we are in control of our own destinies, and on an individual scale I believe we are. But on a larger scale, human behaviour has shown to become regular and predictable. Discovering patterns in a chaotic dataset of human behavior is becoming a reality and its potential is beginning to be exploited. It is being used everywhere from the world of finance, in determining what commodities and stocks will be effected by world events, to the routes driven by police on the beat to increase their chances of encountering crime. Google is already able to predict who you are, targeting specific adverts to you based on your search results, interests, email keywords, check ins etc.

John Graunt’s publication – Bills of Mortality

Big Data is hardly a new idea; it is essentially searching for patterns in sets of data and turning it into useful information. This has been done since the days of the Bubonic Plague when in 1662 John Graunt analysed masses of public data on death records to track and thus predict the spread of the plague. Tracking the spread of a modern epidemic in real time is now well within the realms of possibility, add to this other data such as flight records, seasonal tourism, bird migratory patterns and you have a huge potential untapped database that is ready to be exploited.

Big data is an integral part astronomical research too. Astronomy has traditionally been a science founded on data hunting, ever since the Babylonians recorded star positions and predicted their patterns, distinguishing stars from planets. Vast amounts of data are now gathered from telescopes and satellites worldwide. For example, the Radio Jove project is adding to all the data recorded on Jupiter and waiting to be utilised. But this is just a single planet in our solar system, a planet we have been observing for centuries. Looking deeper into space (and there is a lot of it!), requires a lot more searching, the Alma telescope array in Chile, for example, will record 30 Tb of data per second! The challenge will be to organise and make use of this data and hopefully gain a lot more detail about each one of those stars and beyond what the Babylonians tracked across the skies millennia ago.

Scrambled data anyone? Humpty-dumpty before the great fall was low entropy, If only all the kings horses and all the kings men could piece humpty-dumpty together again

But what is information? Information is real. It is a subtle concept; inherently it is order from disordered data and subject to the laws of physics as anything else. What does this mean? It means that it is an inseparable part of the physical world. Here’s where the science comes in. Science says that over time the disorder of the universe increases, its entropy. Entropy is a measure of chaos, the degree of disorder. The entropy of the universe is always increasing since the moment of the big bang. It’s the reason why a hot cup of coffee doesn’t get hotter. It would be absorbing heat from its surrounding, making its surroundings colder and decreasing the entropy. This would make the molecules more ordered, intuitively we know this is ludicrous. Now picture a carton of eggs falling and smashing, if I were to say that it is possible to put the eggs back together given nothing but information on the eggs, their spatial coordinates in relation to a fixed point, it would be difficult, but given enough data, certainly possible. With large enough dataset, perhaps.

This is similar to a scenario first dreamt up by James Maxwell in a famous thought experiment that hypothetically violated the second law of thermodynamics. He thought about a container full of gas molecules at equilibrium, divided into two sections by an insulated partition. There is a small window that can be open and closed (by Maxwell’s Demon) to allow the faster than average molecules to the right side of the partition and let the slower ones move to the left. This will over time mean one side of the chamber heats up while the other cools down, since average molecular speed is proportional to temperature. Ordering the system – decreasing its entropy, done with nothing more than the information about the velocity of the molecules and seemingly violating the second law of thermodynamics.

Maxwell’s thought experiment, sadly he didn’t live long enough to see his demon exorcised

Now back to the carton of eggs. Think of the enormous amount of information, how many bits, you need to describe the smashing of a carton of eggs. To describe every interaction between the trillions of atoms requires an unimaginable amount of data. This has to be stored somewhere, be it a piece of paper, hard drive or sequence of knots in a rope. Crucially, we only have a finite storage capacity and to continue recording our process, memory has to be deleted and replaced. Similarly with Maxwell’s Demon, it would need to ‘forget’ information it had on the moving molecules after each operation. The erasure of information carries a thermodynamic penalty, it actually expends energy. Thus Maxwell’s Demon was proven to be impossible, the expending of energy creates more entropy than the ordering process of the molecules. NOTHING can violate the 2nd law of Thermodynamics.

1 bit of data, the smallest fundamental unit of data

Even forgetting something takes a little energy – the Landauer limit. It is extremely small, 2.85×10-25 Joules! This is the amount of energy that is required to delete or reset 1 bit of data from the universe, disorder from the order of information. This Erasure of information is an irreversible process that increases the entropy of the universe. It explains why we cannot reverse the cooling of a cup of coffee on a table, which would amount to obtaining free energy as simply unlimited infinite memory storage does not exist.

-Salman

Further reading:

Nature: The unavoidable cost of computing revealed 

When a Good Theory meets a Bad Idealization: The Failure of the Thermodynamics of Computation

MUST SEE VIDEO:  http://video.pbs.org/video/2365004424

Kathy Buffington, a retired teacher from Rochester, N.Y., said she was rattled by the images of a locked-down Boston, even if it was warranted in this case. But she said that in a country dealing with the threat of terrorism since the September 2001 attacks, the fight against it should not be a pretext for more pervasive forms of surveillance. “wiretapping without a warrant goes too far, now that the immediate 9/11 crisis is over.”

Enigma: http://www.enigma.io @enigma_io is a search and discovery platform for public data.  It makes it easy to analyze data from more than 100,000 publicly available data sources, which while out there is often tough for those who need it to locate it.  It’s already being used by journalists from the New York Times to break stories, financial firms, educational institutions and large companies in their business intelligence efforts.  Enigma puts this data into an easily sliceable system to glean insights from assuming you have the knowledge workers in place that know how to manipulate and extrapolate from it.  The company also has a browser plug that when clicked highlights these sources when viewing a web page.  For instance, efforts in the CSR space from a brand page.

This tool has tremendous potential for companies to identify what their competitors are doing in various areas assuming the data is public.  It’s significantly disrupting issues of infrastructure and content.  From a brand perspective this could be a huge help in gleaning insights in that up front situation analysis process.

HealthyOut: http://www.healthyout.com @healthyout provides users with personalized menus of food from local restaurants that’s set up as a subscription delivery service to help them lose weight or live healthier lifestyles. For a $28 monthly fee, the company taps into local restaurant kitchens to your home or office as if you had a personal nutritionist.  The cost of the food is an additional cost dependent on the local restaurant.  Their differentiator is that they go beyond the boxed or prepared meal service instead tapping in pre-existing menus but providing personalized instructions to the restaurant based on the user’s pre-set criteria.  They company has already raised a $1MM A round and is operating at 200K users in their NYC only beta.

I really like this company for brands with small business efforts, local chains who might be able to integrate healthy menu options into the product or those activating in the health vertical.

Talkz: http://www.talkz.com With Kik Voxxer, What Now and others, the messaging space is an admittedly crowded category but Talkz has a few tricks up their sleeves to differentiate themselves.  What the messaging service combines text, voice, image capture, doodle and other features to enhance the messaging experience.  Probably the most novel is what they call “voice cloning.” Over time the technology uses iSpeech technology to learn your voice so the next time you text, the recipient can hear the message in you own voice.  They’re even cloning celebrities (with a number on-board TBA) to make the experience more fun.  During their pitch they used Barack Obama’s voice to demonstrate a call.  The great thing is that recipients don’t need to have the app downloaded to hear the message but the company is hoping some of the viral factors innate to the product will create service switchers.  The app is currently only available iOS.

From a brand perspective this is a pretty interesting way to integrate celebrity endorsements into messaging as well as branded backgrounds for photos but all of these ideas are just conceptual at the moment.

In this electronic goldfish bowl environment, we need to advise our client managements that proactive steps have to be taken to minimize the possibility that an employee will do something to put the company at risk, whether or not the risk has a direct business consequence, such as a major regulatory investigation, a major financial loss or the possibility of a lawsuit.  Second, and related, organizations must make sure that the effect of any lawsuit is minimized in the event that one occurs.

A key factor in managing the risk from employee misbehavior, and underlined by the SEC’s board member investigation, is to have in place an effective financial and general compliance system to monitor the behavior of employees to ensure that their behavior complies with company policy.  Eying this opportunity, technology vendors have flooded the market with software in recent years to help firms store, manage and retrieve the necessary files to identify potential and actual misbehavior.  While many of these solutions allow the buyer to check the compliance box on the compliance score sheet, the stored data is, in reality, a treasure trove for regulators and adversary attorneys building a case while it does little to actually identify and control employee misbehavior.

To truly mitigate this type of risk, it’s necessary to be able to analyze patterns in communication and organizational behavior in ways that go beyond basic keyword filtering and associated counting and statistics.  Advanced analysis and understanding of behavior patterns form the basis for identifying the type of anomalous behavior characteristic of insider trading and other fraud scenarios that merits further investigation.

These same technologies can be used for retrospective investigation (internal or external).  They can also be applied to records retention, where a smarter approach can better identify materials that can be legally and safely purged.  As a result of storing fewer records, storage costs are decreased and, in the event of a lawsuit or regulatory investigation, there are fewer items for the opposing attorneys or regulators to review, resulting in lower costs and decreased risk of untoward information coming to light.

Remember that your client’s existing data constitutes a baseline for evaluating “normal” corporate behavior.  As such it addresses one of the most challenging issues for enterprise monitoring: how to keep an eye on a vast volume of data.  Knowing what is normal makes it possible to identify and focus on behaviors, like that of the board members under investigation, that are outside the norm.  Analysis of such data in defending against an investigation or lawsuit is also useful since the client can prove that the alleged offenses perpetrated by a limited group of employees are not typical of its usual practice or ethical culture.

Despite the current slow economic recovery, now is not the time to place investment in employee compliance monitoring on the back burner.  Such a response can put an organization in an even more vulnerable position.

Launching today and updated every 24 hours with faster refresh rates to come, Hedonometer.org uses English-language tweets to create a happiness index. The system is based on a 10,000-word strong “emotional temperature” database, where words are ranked on a scale of 1-9 by volunteers using Amazon’s Mechanical Turk. Words like “laughter,” “happiness,” and “love” top the list, while “loneliness,” “bad,” “inflation,” and “surgery,” along with assorted expletives, round out the bottom, with rankings close to 1. The emoticon “:(“ has a rating of 2.36.

Users can zoom in on any day all the way back to September 10, 2008, check out the balance of positive and negative words, and see how these compare to the week before and after. Saturdays, for example, tend to be happier than Tuesdays. Christmas Day stands out as being the happiest day of the year, every year. The hedonometer developers, mathematicians from the University of Vermont along with scientists from the MITRE Corporation, found that April 15, the day of the Boston bombings, was the unhappiest day on record, with an average happiness index of 5.88. Other recent sad days include December 14 last year (Newtown school shooting) and June 25, 2009 (death of Michael Jackson).

Indeed, eyeballing the global happiness index suggests a slight downward slope since 2008. Whether or not this effect is real depends on establishing a normal background happiness level, and comparison with geographic, socioeconomic, and political metrics. What’s interesting is that the hedonometer is turning more than 50 million daily micro-statements into a “quantitative macro-story,” as UV’s Chris Danforth put it. Individually insignificant words and tweets swell into a collective emotional response, the blips and dips of which stand out and correlate with major events.

The research from Danforth and his colleagues got some press earlier this month, when they reported that happiness went up the further Twitter users were from home. Other insights from the same team included the fact that obesity and happiness were inversely correlated, and that cities’ happiness scores were related to swear words, suggesting that “geoprofanity” could be a good marker for regional happiness differences.

The hedonometer is set to draw on more data streams soon, including blogs, news transcripts, and Bit.ly shortened links, and will be data mining in a dozen languages. Nonetheless, the happiness index will remain an aggregate measure, like a nation’s GDP, and may not have much impact in and of itself. The underlying methodology, however, is the real driver, with broad applicability to big data, whether social media-generated or not.

Image via Chris Danforth, University of Vermont

I can certainly understand why data mining scares the average consumer. Supermarkets get you to sign up for their loyalty programs so that they know everything you buy from them. And I know I get a spooky feeling when I express an interest about some product in one place on the Internet and then see ads for that product pop up on Facebook or my Google search.

But data mining is a valuable tool and every ISP should be using it – just not in the same way that the supermarkets and Facebook do it. In fact, we probably need to come up with a better terminology for doing the things I am suggesting below.

There are a number of tools around that let you look at data about customer usage and these tools allow an ISP to do the following:

• Spambots. There is a wide array of spambots and other malware on the web that can infect customers’ computers. The worst of these, from a network perspective are spambots, which take over your customer’s computers and use it to send out spam. Most ISPs monitor email usage from their own domain and can spot when one of their users has been taken over by a spambot. But most customers these days do not use the email names and domains assigned by their ISP. Instead they web email addresses such as gmail or even the older AOL. And some spambots create new email addresses that the customer doesn’t even know about. And so data mining can be used to look for customers with unusual upload traffic. No customer is going to be offended if you ask them if they are uploading traffic 24 hours per day if in the process you help eliminate Trojan horses and spambots from their computer.

118 – Another File Sharing Session (Photo credit: erickespinosa)

• Web servers. Most ISPs do not want a customer to be using a residential ISP account to run a commercial web server. A web server is a device that is being used to run a website or service that drives a large amount of download traffic. Such a website might be used for e-commerce for example. But far too often web servers are used to run porn sites. ISPs are not against web servers, but they do expect people who operate them to buy the proper business level service. A web server can be full 24-hours per day, and that is generally not the level of service that is intended for a shared residential product. Data mining can be used to identify web servers and the customer can be directed to a more appropriate (and appropriately priced) service.

• Data Caps. Most ISPs have set some cap on the amount of usage that a customer can download in a month. And these caps do not have to be small. I have one client that has a 2 terabyte cap each month for residential downloads. But there is no sense in having a data cap if you can’t actually measure how much bandwidth each customer is using. Data mining tools are the way to measure customers’ usage.

• File sharing. Most ISPs have terms of service that prohibit customers from sharing copyrighted materials with others. But realistically an ISP is not going to know what customers are sharing with each other unless you get a complaint from a copyright holder. But many ISPs still like to get a handle on file-sharing because such traffic can eat up a lot of system bandwidth. Data mining can help you identify customers who are probably involved in one of the common file sharing programs.  An awful lot of file sharing is done by teenagers. I have clients who send out friendly reminders to customers who they think are file sharing that say something like: “We notice by your internet usage that you are probably running a file sharing program. We would just like to remind you that it is illegal to share copyrighted material and that there have been cases where copyright owners have gotten significant settlements by suing people who were sharing their property.” Such notices cut down on a lot of file sharing traffic as parent pressure kids into doing the right thing.

So you should be data mining. But perhaps the things I have described could all better be classified as network management, a term that would not dismay your customers.

Justice Alito, writing for a unanimous court, essentially found that a state could distinguish between residents and nonresidents for purposes of determining who is entitled to receive governmental information from that state. These non-Virginia-resident petitioners sought Virginia records, and their requests were denied. One petitioner was an ex-resident and sought information about a child support matter. Another petitioner sought records for the purpose of reselling the data.

The Privileges and Immunities Clause provides, “The Citizens of each State [are] entitled to all Privileges and Immunities of Citizens in the several States.” U. S. Const., Art. IV, §2, cl. 1. The Court noted that the state law was designed to give residents access to their own government and that those citizens were supporting that effort with their taxes.

The “dormant Commerce Clause jurisprudence ‘significantly limits the ability of States and localities to regulate or otherwise burden the flow of interstate commerce.” (Citation omitted.) The Court rejected out of hand the idea that a state law allowing citizens access to records had any regulatory or burdensome effect on interstate commerce.

This will be seen as a victory for state and local governments that provide records for their citizens and want to protect against the commercialization and data-mining of public records.

[ShahzadHL10]
Shahzad, R.; Haider, S. & Lavesson, N.
Detection of Spyware by Mining Executable Files
Availability, Reliability, and Security, 2010. ARES ’10 International Conference on, 2010, 295-302

Source: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5438080&tag=1

That’s not to say that she didn’t know – I’m sure she did. But when I told her that it was Open Source INTelligence, and further explained the difference, say, between that and traditional HUMINT, she cut me off to explain that she’s a veteran investigator.

And elicitation specialist.

Now she runs the site chartgirl.com. Click the image on the left to see the full sized edition of the one that kicked off this post, or click here.

We’d started talking because I asked her permission to run here on PLI the graphic above, that she released today. It tracks quite clearly the who-knew-what-and-when-they-knew-it of the Tsarnaev investigation. I’d seen it online, tweeted by Th3J35t3r , and it was to me a fantastic vindication of what I was talking about the other day when I posted my looooooooooooong post about why the Boston Marathon Bombings was not evidence of an intelligence failure.

I asked Hilary what brought her to do this kind of work – I’m not the first to do so; see a great piece in Reuters about her here.

That’s when she asked me the question about OSINT. “I’ve heard that word a few times today,” she said.

Sargent says that she has not yet turned her charting abilities into a business, but she keeps getting stuck reading newspaper articles that really need a chart and don’t have one.

So at this point, her startup costs seem to be her weekends.

Sargent said that one of the reasons she wanted to make this chart was listening to Lindsay Graham saying that, but for a spellcheck, the FBI woulda had him.

Sargent’s understanding of the intricacies of data sharing have led her to the conclusion that when it comes to interagency sharing, offering access is not the same thing as data sharing.

“I was trying to show the idea that, ‘they should have caught this guy,’ was wrong,” she told me.  If you look at her chart, which she compiled from open news sources and scholarly descriptions of the organizations and systems it describes, you can see that no agency was even remotely close to knowing enough to catch him. “Each agency knew a different thing.”

Sargent says she made the chart to clarify some of the inconsistencies and flat out wrong statements in the press about even basic things, like a “Watch List”.

Mainly, though, she said, she did it because it needed to be done and no one else had done it.

“I start with a similar pencil and paper version,” she told me. “I know kind of the rough basics, like that TSA and Customs are part of DHS; I didn’t know about TIDE or a lot of these other more arcane databases.

“I also didn’t know the way to structure to describe it visually. It seemed clear there wer four things that, had everyone who could have possibly known and focused on, they would have caught this guy.

So, if, for example, there was one senior guy at a JTTF who had spoken with the guys in Russia and with the family, and known about the son’s activity and his overseas travel, then they would have known.”

Sargent started mainly by reading media articles – there haven’t been a lot of documents released on this yet, anyway. She spent a ton of hours reading about the differences between several of the federal lists.

“Most articles did a bad job delineating between, for example, TIDE and the No Fly List and TECS. That was the most important distinction. For the media to say he was “On a watch list” is really misleading.

“Then I just read a couple of hundred articles, and a bunch of stuff written over the ast ten years about how all this stuff works.”

“I didn’t include description of 90-day window,” she said. “I try to walk a line between only charting what I completely understand; I don’t put things I don’t understand in the chart. I don’t want people from the intelligence community to look at my chart and roll there eyes; that is my goal anyway, and it is a hard thing to do.

“I am sure a lot of people in the IC have been reading the New York Times and rolling their eyes,” she said.

The real reason I am writing about her here is because of how she synthesized all these data and then produced the analysis, and basically it was straightforward OSINT work. She Googles a lot, and she has the kind of personality that lends itself to wanting to find the answer and being willing to spend the time to get it.

One good tip she gives, that we agree with and do, is that for things like TIDE and TECS and No Fly that you start at Wikipedia to get a basic idea, but never quote it – rather, look at the link collection at the bottom of the listing and go directly to the scholarly works and articles quoted therein. She warns that this is for researching things more than people  - unless you’re a B-list celebrity or higher, the Wikipedia pages and the links tend to be self-serving paff for people.

But for things? Great stuff.

Her one statement about cops and research is very important for analysts to remember when they speak with detectives or cops who say that they just Don’t Understand The Google.

“Oh yeah? Well tell him to get me a score from some game on ESPN and I guarantee you he can find it in seconds on his phone. These are the same skills.

“They can do it.”

This is an excellent write-up for the differences between:

• Statistics
• Machine Learning
• Data Mining
• Informatics
• Big Data
• Predictive Analytics
• Data Science

Databases are built to store data. But, as Jeff Jones has most vocally pointed out, simply keeping the data is not enough in adversarial settings. You also need to keep the questions, and treat them as part of the ongoing data. The reason is obvious once you think about it — intelligence analysts need not only to know the known facts; they also need to know that someone else has asked the same question they just asked. Questions are part of the mental model of analysts, part of their situational awareness, but current systems don’t capture this part and preserve it so that others can build on it. In other words, we don’t just need to connect the dots; we need to connect the edges!

Another part of this is that, once questions are kept, they can be re-asked automatically. This is immensely powerful. At present, an analyst can pose a question (“has X ever communicated with Y?”), get a negative answer, only for information about such a communication to arrive a microsecond later and not be noticed. In fast changing environments, this can happen frequently, but it’s implausible to expect analysts to remember and re-pose their questions at intervals, just in case.

We still have some way to go with the tools and techniques available for intelligence analysis.

But I’m here to tell you that big data can also be used as an excuse NOT to act. There are always more numbers to get. There are always more scenarios to evaluate. We will know more in a week. A month. A year. An hour. Five minutes.

Stop already!

Sometimes you have to just do something. I remember picking up a very useful expression from someone, “that client is wearing a belt and suspenders.” What’s it mean? It means they are beyond cautious. They want too much validation. And they look nerdy and ugly and dumb too.

(Now, let me very very clear on this — that was for a long lost client from a million years ago who is out of business, no relation to anyone living!!!)

At any rate, as someone else said, paralysis by analysis.

Hey, sometimes you just gotta throw the pitch. See if the other guy swings. See what happens.

But, alas, the blanket of numbers is too often a security blanket we won’t let go of. And in the meantime, a world of opportunity could be passing us by. Oh, and we look real nerdy too.

I’m just sayin…