By Ed Sperling

Just because you have an abundance of data and expensive tools doesn’t mean you’re making good use of them.

The amount of data inside corporations is exploding. In fact, there has never been such a wealth of information available in the history of business, and there has never been the vast array of tools to dissect it.

CIOs are generating reports for business leaders that slice the data horizontally and vertically, project growth, calculate productivity and profitability, and compare all of this historically and competitively. They are even pulling out tidbits of data that may appear randomly and building models based upon recurrences that escaped notice by even the most astute teams of experts. But is all this stuff right?

Forbes caught up with Marcus Schaper, principal at McKinsey & Co.’s Business Technology Office in Hamburg, Germany, to talk about how data is being used and where the pitfalls are.

Forbes: How do we keep up with an explosion of data and turn it into something truly valuable?

Schaper: This is a real challenge for many CIOs and many business units. We are all swamped by data. The availability of data is not the issue. It’s how we can extract information from all of that data and make good business decisions from it. That’s where many companies fail, despite the fact that many of them have very expensive and relatively good tools.

How do they solve that?

First of all, they have to ask the right questions. Then they need to derive the right structure to bring the data into. And finally they have to present it in a way that people can understand and trust. Being able to trust it is even more difficult because the data has to be in sync with the world as we know it. If the reports you get on a daily or weekly basis are different than what you know from your daily life, then you don’t trust them and you don’t trust the system.

You’re talking about creating a data model, but aren’t models inherently flawed? Look at what happened to Wall Street in 2007.

The problem with models is not so much that they’re difficult to create. It’s whether everyone involved agrees with the same semantics. If you want a revenue report or a profitability report, you need to figure out what should be included. Once you have clarity on that, the other steps are much easier.

Do these skills exist?

They’re not very common. We need a completely new skill set for the CIO and the IT department. They have to speak the language of requirements for the application as well as the business language of reports.

But you’re also now dealing with organizations that change quickly. How do you program that into the models?

It’s a question of two things. One is the underlying data architecture, which is an IT topic. You have to be able to find the modularity in the data so you can access all different cuts and views. If the company wants productivity data this year that will be different from profitability data next year, it all may be different from the growth program in year three. You need a data architecture that allows all the different cuts and views and which enables quick access. That’s one piece, and it’s a key piece. The more difficult piece is the interface between IT and the business side. If you really want to have views and cuts of the different pieces, then the business side needs to sit down with IT and agree on the structure and the semantics and the business structure. The notion of sitting together every now and then and reviewing where the business is going and what that means in terms of reports–that’s something that people don’t do on the business side. That needs to change.

Is there a risk of relying too much on data and not enough on people’s observations and insights?

People already rely on data quite a lot. All my clients get reports and they draw conclusions based upon those reports. In discussions with other people they are either validated or not. Numbers have a tendency to anchor in your head more than beliefs or opinions, but there is also a need to correct the data if it is wrong to make sure the number of anchors you have is correct as possible.

Here’s an example: One logistics company that’s a client of ours was trying to understand the profitability of its products. They get bulk part numbers, but when you try to boil it down, the average profitability does not apply to a large customer because they get different rates. The products are not comparable over different regions either, because there are different service levels and costs. So looking at the same products in different parts of the world, the overall data you have is probably wrong. Too often we use incomplete data or average data and you come to the wrong conclusions.

As companies outsource applications into clouds, does it become more difficult to gather data?

Data gathering isn’t the issue. Data is lying around everywhere and it’s not technically a problem to get together. But it does get more difficult to agree on the right indicators and the definitions behind them. If you have a small board of five people, you can sit down and agree on everything in one board meeting. If you have a global company, it’s much tougher. If it’s in a cloud it’s even harder. You might not even know your counterparts, and the risk is higher that they do not have the same interpretation of certain terms you are using. If you are using lots of different data and content providers and they give you pre-process information, it’s nearly impossible in the cloud environment to agree on the same semantics.

So where do you start?

You basically have to re-invent your language. You have to start with your own definitions and language. And you have to get everyone involved so they understand it the same way. This is a pure business issue. It has nothing to do with IT. After that you can establish your data structures.

How do you add in the necessary flexibility, such as when you acquire a new company?

You don’t have to change it as much when you buy a company as when your own company changes direction a little bit. When companies switch from cost-cutting to growth, all of your indicators for cost-cutting are no longer relevant for growth. You need to build intelligence into your glossary so you have different missions or directions. After that it depends on where your data is coming from. You might find different companies have completely different data schemes.

Is all the data necessary?

No, and some companies might do better if they throw away some data they don’t need. Sometimes the data they have is incompatible and it can make people misunderstand connections between different items. Most of the data is very valuable, but there is an art to defining what you should trust and what you should ignore.

Do people understand what they’re getting in the data, even if it’s done right?

In my experience they understand half of the key performance indicators. The business guys need to really understand the business from a number of perspectives. You may need a data translator who can translate the data into business intelligence. There are very, very few people who can do this.

Ed Sperling is the editor of several technology trade publications and has covered technology for more than 20 years. Contact him at esperlin@yahoo.com.

Definition: Business Intelligence(BI) is the delivery of accurate, useful information to the appropriate decision makers within the necessary time frame to support effective decision making.

Or

Business Intelligence (BI) refers to skills, processes, technologies, applications and practices used to support decision making.(Source: wiki)

BI uses technologies, processes, and applications to analyze mostly internal, structured data and business processes.

There are bunch of other videos online explaining the definition of Business Intelligence.  However I felt this video to be more simple and yet clear in understanding.

To facilitate effective decision making there are different BI technologies available to process and analyse different Business processes and transactions to present future desired business outcomes. 

All the BI technologies uses BI Tools that are like software applications designed to report, analyse and present data.

The common functions of BI technologies can be listed as below:

1. Reporting
2. Online analytical processing(OLAP)
3. Analytics
4. Data mining
5. Business performance management
6. Benchmarking
7. Text mining 
8. Predictive analytics

 The famous proprietory BI tools are listed below how ever a comprehensive list can be found( here)

1. Cognos(IBM)
2.  Hyperion(ORACLE)
3. SQL SERVER Analysis and Reporting services(MICROSOFT)
4. Sybase
5. Teradata
6. Business Objects(SAP)

 BI tools use information stored in Data Warehouses and Data Marts but all tools do not require them.

Ssrs 2005 Reporting Services

This session will be a cornucopia of three sub-sessions. The first part will be to convince the skeptics. Why does every organization should consider SQL Server Reporting as part of its front-end solution? What will SSRS do better than a typical web application/site or a client-server application? The second portion will be a quick demo of the possibility and will be the shortest. The final part will talk about the best practices, tips from the field and will cover the implementation techniques.

Sql Server Performance Tuning

This session is for you if you want to learn tips and techniques that are used to optimize database development with special emphasis on SQL Server 2005. If you write lot of stored procedures and want to learn the tools of a DBA, this is the session for you. If you are new to SQL Server development environment, you will learn how the various constructs compare to each other and better performance can be produced every time with a brief introduction to understanding Execution Plans.

Building technology to suss out bad guys is the easy part. Getting agencies to collaborate? Not so much.

By Stephen Brobst, chief technology officer, Teradata Corporation

Brobst: Don't blame volume of data for security lapses. Photo: Teradata.

When President Barack Obama or then-national security advisor Condoleezza Rice observed that intelligence agencies failed “to connect the dots” for either the botched Christmas bombing of Northwest flight 253 or the tragedy of 9/11, they evoked a simple children’s game of drawing lines on a page to complete a picture. Connecting dots about known terrorists’ plans, then, sounds simple.

Let’s be realistic. It’s not.

The biggest challenge our security organizations face involve overcoming cultural clashes and power struggles within and between them – and only strong leadership in Washington can address those issues.  There also are complex technology issues involved in deploying a comprehensive, responsive, and collaborative analytics system across multiple agencies.  This may be the easier problem to address.

After all, the commercial sector has been successfully collaborating across technology organizations (think suppliers and purchasers or merged companies) for years.  Intelligence agencies are awash in data from various legacy systems running multiple databases that are seen as barriers to implementing a practical, centralized repository of data.However, I know of one major healthcare provider that was able to consolidate operational data from hundreds of facilities into a single data warehouse. The company’s supply chain operation alone depends on 16 discrete databases, which only can retain data for 30 days. Now, all those databases feed into the central data warehouse and overall historical trends are visible to procurement analysts, resulting in savings amounting to millions of dollars through such things as tighter inventory management and improved contract negotiations.

Although saving lives rather than money is the primary goal of intelligence analysts, among the benefits prized most by the healthcare provider I mentioned is the “ability to analyze, understand, and respond effectively to complex market developments” as they are happening. Swap “terrorists” for “market” and you have exactly the kind of system that will help connect dots.

Data volume is not the problem

Some people expressed concern because Umar Farouk Abdulmutallab’s name was in the sprawling Terrorist Identities Datamart Environment or TIDE database of 550,000 people suspected of having connections to terrorist groups. It’s been said that the size of TIDE was the problem, allowing the failed Christmas bomber to board his flight.

That’s nonsense. For a well-designed data warehouse, getting actionable information from data on that many individuals is easy.

Many thousands of companies have hundreds of terabytes of information in their data warehouses. And there are increasing numbers of companies with petabytes.

There’s a major U.S. Internet site, for example, that maintains many petabytes of information on its 89 million active users worldwide and lists more than 200 million items online. Daunting, yes, but with the right tools, an information treasure chest. For example, the company not only knows that in 2009 the site moved more than a half million items related to the Twilight series, but also that 31,871 of them pertained to teenage heartthrob Robert Pattinson, while only 4,183 were about his co-star Kristen Stewart. This kind of granular detail is critical for detecting and reacting to market trends.

Modern data warehouse and business intelligence technologies are definitely up to the task to deliver real-time information to cross-agency intelligence analysts. These tools can be installed and would begin providing information within months of deployment. So, what is standing in the way?

Culture is the weak link

Historically, an enterprise’s culture is the culprit. Organizations are slow to change. Studies by The Data Warehousing Institute (TDWI) and others have shown that data warehouse deployments fail primarily for one or more reasons involving weak management support, lack of user involvement, staff turnover, organizational politics, or an inadequate grasp or ever-changing nature of the project’s scope. Low on the list of stumbling blocks is technology.

In discussing the intelligence failures of 9/11 in 2005, Sunil Desai wrote in Policy Review that “the essence of the problem is that the entire interagency community is dominated by individual agency cultures rather than a common interagency culture.” As evidenced by the recent Christmas bombing attempt, the interagency culture problem persists.

The Obama Administration is trying to encourage greater collaboration among intelligence agencies. While it is relatively easy to integrate data from disparate sources into a single repository, it is much more difficult to achieve interagency collaboration. The Project on National Security Reform has identified numerous structural problems among the various intelligence agencies. Managers and staff, for instance, are not evaluated on how well they collaborate outside their agency. Also, there are few, if any, mechanisms to work horizontally with other agencies. The group’s 2009 report repeated Desai’s insight that no interagency culture exists.

Intelligence professionals need to understand that their careers and people’s lives depend on interagency collaboration. Leadership from the highest levels must instill that change into their organizations and create mechanisms, including interagency data warehouses, for an interagency culture to flourish. When that happens, and with the right analytical tools in hand, intelligence analysts will have a fuller picture of terrorists’ plots to work with, not just the dots.

Brobst is chief technology officer for Teradata (TDC), a Dayton, Ohio-based company focused on data warehousing and enterprise analytics.

Hello!  This is my first main post to my blog and it’s about time!  I will do my best to make posts on a regular basis.  There just never seems to be enough time during the day (at least non-kid time).  A little about myself and Web Presence…

I am an Information Technology professional with over 14 years of experience (11 in the “working world” and 3 as a “stay-at-home mom doing freelancing work”) in many industries (finance, insurance, telecommunications, retail, online marketing).  I started out as a programmer, moved up to an analyst, did some consulting for a while and my last job was as a Product Manager for reporting.  The last 5 years of “working world” I was in Data Warehousing and Reporting and LOVED IT.  In 2005 I became pregant with twins (still a shocker!) and as a result, my husband and I felt it would be best to make the move from the city to the burbs’.  Unfortunately, a communte from New Lenox, IL to Evanston, IL wasn’t realistic and the company would not allow me to work out a telecommuting package (which was most unfortunate for me and even moreso, I think, for them).  As a result, we decided I would be a “SAHM”.  Well, I couldn’t sit still long… even with newborn twins. I never realized I would miss my career as much as I did until I no longer had it.  After about a year or so, I so thankfully came across a posting on Mamasource.com for a Technical Assistant.  Now, almost two years later, I am still working for Meredith Bromfield at Crossing Your Bridge (www.CrossingYourBridge.com).  I have done everything from communications, creation of newsletters, flyers and brochures to re-designing her website and then re-designing it and developing the new website myself to editing  anything and everything (hand-outs to books) and marketing (online and off).  It has truly been a blessing.  Through Meredith I started to perform some work for Smart Women’s Coaching doing odds and end technical support as well. 

During my work with Meredith, I found that SO many companies still do not have a web presence (ha ha, yes – hence the name) …. clubs, organizations, churches (a big one), networks and then looked into daycares, people selling their home, real estate agents, direct sales, schools and the list goes on and on.  As a result… Web Presence CG!!!!  I want to help YOU improve your BOTTOM LINE and you WILL do that by having a WEB PRESENCE!!!!  Our focus is on small companies/organizations…

There is a lot more work involved in really trying to start-up a company and really trying to make a go at it… all new to me and I am learning so much.  Plus, my strengths always were in analysis, project management and design rather than development so I have been needing to focus on a development tool and learn it… inside and out.  And oh yea…I still have my other freelance work and… my kids! 

So please bare with me… if you know of anyone who still does not have a website… please send them my way. 

More to come…

p.s.  Check out www.WebPresenceCG.com…. have no fear, this was just created VERY FAST to get a page holder up…. MUCH work needs to be done!

Following a tremendous response to FREE email subscription services, where in walkin detail being sent to your email id every day, JumboWalkin.com team is very pleased to let you know that it has launched the FREE SMS service also for instant alerts in you mobile on latest walkin interviews across india.

This service enables you to get info on new walkin interviews in india as and when posted on Jumbowalkin.com. So why wait?

Please take s minute to Subscribe to FREE Walkin SMS alert service from JumboWalkin.com, here is how you can do it.

Are you a gmail user? then Please follow this link – http://labs.google.co.in/smschannels/subscribe/JumboWalkin

If you are a non gmail user – Please SMS ‘ON JumboWalkin’ to 9870807070

Good luck!

This is a new kind of blog post for me – it’s specifically designed to elicit ideas, suggestions and examples for my Big Data Summit (click to register) kickoff on Feb 18th. Consider it an experiment in crowdsourcing from a very smart community (my readers.) No, I’m not a shameless suckup – you know who you are, and you’re all pretty darn smart.

We have some great speakers – Amazon, Mobclix and Aster Data - and added sponsors like Informatica, Dell and impetus. So there will be plenty to talk about, including storage, analytics, programming paradigms, etc.

Here’s my dilemma: I imagine we’ll have newbies as well as more advanced attendees. So I cannot skip the basics like:

• Store it yourself? Host it elsewhere? Move it and process, then return it?
• DB or not DB? (subcategory: SQL or NoSQL?) and, a corollary:
• Where should analytics be done? Inside or outside a DBMS?
• Where are the resources to learn more?
• Who are the providers of key technologies?

This may take my 30 minutes and I’d hardly scratch the surface. But maybe I can blast through it fast and get to some other issues. What would you want to hear if you were there? Or if you’re involved in projects and have learned some lessons you think the world should know, what would they be? I’d be grateful for any input my readers have to offer. Feel free to send me links, experiences, interesting use cases you’ve seen or participated in. Put them in the comments below.

I ran into ‘time’ difficulties on populating fact tables with timeIds and dateIds.  Even a relatively simple query was taking too long to fetch correct values into table.  I guess doing 10 million row joins between two tables (10m_ratings and fact_rating) joined to dime_time/dime_date is a bit too much for my laptop.  Maybe I stared at it too much.

The only thing that I can think of is regenerating both fact tables and including the original timeStamp in both of them this time.  This will save a join and, hopefully, make populating these fact tables speedy enough not to drive me nuts.  I can remove these columns when they are not needed anymore.

Properly indexing fields used in the UPDATEs from prior posting will help tremendously as well.

This is not even the cool part of this project but more of a foundation so that I know how things work.  Commercial products do these things for you anyways I think.

This revised schema better illustrates this change.

Informatica has announced another, long-rumored acquisition: Siperian, thus continuing a steady march toward a comprehensive portfolio play. In 2009, its strong growth path made it the clear independent leader in data integration.  With Release 9, its vision of a data integration platform grew to providing a comprehensive approach to everything from data discovery services to data quality. While growth slowed during a tough year for the economy overall, Informatica grew revenue in every quarter, and made key acquisitions in 3 successive quarters (Applimation, AddressDoctor and AgentLogic) and began to make significant moves into the cloud via partnerships with Amazon, Salesforce and others. Agent Logic added event detection and processing to support real-time alerting and response. As 2010 begins, this latest move is synergistic from the outset; Rob Karel points out in his excellent blog post that “Siperian MDM technology…already is deeply integrated with Informatica’s identity resolution and postal address technology. In addition…Siperian MDM customers [are] using Informatica for data integration and data quality, meaning there is a lot of existing experience and know-how on integrating Informatica’s portfolio with Siperian.”

Oracle’s purchase of MDM player Silver Creek systems, widely perceived as a pre-emptive shot across the bow at Informatica, takes on new resonance now. Informatica has hitherto avoided direct competition for certain parts of its portfolio, but one can almost hear the gloves coming off.  Oracle licenses Informatica’s data quality technology as part of its own MDM offerings (of which there are several, now compounded by the Sun acquisition). We’ll see how that plays out in the months ahead. IBM, SAP and smaller specialists like Kalido and Initiate Systems play here as well, and one can almost hear the checkbooks opening as the impact of Informatica’s competitive stature is felt. Perhaps Informatica will get a little wiggle room to move while Oracle’s massive projects of Sun and mySQL digestion and the Fusion apps launch play out, but the future looks very active.

Meanwhile, Informatica Cloud has been a source of steady, incremental growth. There are many use cases where Informatica had heard that traditional tools were too expensive, and unnecessarily full-featured.  It’s identified different buyers outside of IT in administrative and sales, marketing and support operations roles, and since  Q107 when its first cloud-based offering was brought to market, it’s learned how to find them and speak their language. There is a separate sales team, separate marketing group – even a separate web site. Even the internal accounting is different: because of the different nature of subscription pricing and how it changes the flows of funds, to Informatica’s corporate balance sheet the Cloud take is “other revenue.” And of course, the execution is different – a subscription model and a different price point, high volume, inside sales. Self-service would simply not have worked without this flexibility, and the team was given the flexibility to work it all out.

The effort began with the salesforce.com market. It was a natural fit, and adoption has been steady, so much so that integration jobs now that run in Informatica’s cloud already number 20,000 per day. Takeup has been broad; 500 Salesforce.com customers were using the service in December 2009. Informatica Cloud won Salesforce’s “AppExchange product of the year” for the second year in a row. And this is mostly new business: 74% of all deals in 2009 were new logos, the company asserts.

Master Data synchronization was a clear use case. Another is to replicate data out of the system to operate on it locally for compliance, reporting, backup. Informatica’s core engine has been turned into a plugin – its secure agent requires only 80 Mb and is self-updating and self-upgrading. Another 120 Salesforce customers are using Informatica’s PowerExchange connector for on-premise PowerCenter access to Salesforce. Some are using both the Informatica Cloud and the connector. For the Informatica Cloud Services, a simple visual GUI runs in the cloud, showing how local data connects directly to the data within Salesforce and other cloud application environments. There is a scheduler, and for more sophisticated uses, it’s possible to export content into XML files and bring it into a local PowerCenter environment. Data quality assessment is available now– leveraging specific profiling services for Salesforce – enhancing a user’s account contact opportunity or lead, and providing a scoring mechanism.

Overall, Informatica presents a portfolio that continues to grow, increasing experience with alternative delivery and deployment architectures, and steady financial results. As spending rebounds, the attractiveness of its rich and ever more complete portfolio positions Informatica well for another strong performace in 2010.

Disclosure: Informatica is not a client of IT Market Strategy

Last Saturday was the Alabama Code Camp, held in Mobile AL. For those unfamiliar with the Alabama Code Camps, we hold on average two a year, and they shift from city to city with different user groups acting as the host group. Other cities include Huntsville, Birmingham, and Montgomery. This time though the Lower Alabama Dot Net User Group under the leadership of Ryan Duclos hosted, and what a great event it was. Everything ran smoothly, there was plenty of drinks and pizza to go around, and some good swag to boot. A big congrats to Ryan and his team of volunteers for a great event, also thanks to Microsoft for sponsoring and the University of South Alabama for the venue.

I was kept busy at this code camp, doing three sessions. The first session was “Introduction to Microsoft PowerPivot”. The slide deck can be found at http://arcanecode.files.wordpress.com/2010/01/powerpivot_long.pdf. To see all my PowerPivot posts, simply pick it in the categories to the right or use this link: http://arcanecode.com/category/powerpivot/.

My second session was on Full Text Searching. You can find code samples and the PDF for the presentation at my code gallery site, http://code.msdn.microsoft.com/SqlServerFTS.

The final presentation was an introduction to Business Intelligence and Data Warehousing. Here is the link to the presentations slides in PDF format. As promised in the session I added the additional information for the Kimball Group book.

A quick apology for my delay in posting, a nasty head cold has had me in Zombie land since I got back. Thanks to all who attended, I appreciate you being very interactive, lots of questions, and very attentive. I look forward to the next time Mobile hosts the Alabama Code Camp.

With the previous schema changes in place, it is now a matter of running queries in order to complete our fact_rating and fact_tag tables.  Both these tables are missing timeId and dateId.

For the table fact_rating, the following potentially long time queries need be executed. For timeId:

UPDATE fact_rating SET timeId = (
SELECT a.timeId FROM dim_time a
WHERE a.military = hour(from_unixtime(fact_rating.timestamp))
AND a.minute = minute(from_unixtime(fact_rating.timestamp))
)

For dateId:

UPDATE fact_rating SET dateId = (
SELECT a.dateId FROM dim_date a
WHERE a.year = year(from_unixtime(fact_rating.timestamp))
AND a.month = month(from_unixtime(fact_rating.timestamp))
AND a.day = day(from_unixtime(fact_rating.timestamp))
)

Similarly, for the table fact_tag, we run the following for timeId:

UPDATE fact_tag SET timeId = (
SELECT a.timeId FROM dim_time a
WHERE a.military = hour(from_unixtime(fact_tag.timestamp))
AND a.minute = minute(from_unixtime(fact_tag.timestamp))
)

For dateId, again, the snippet is similar to the fact_rating one above:

UPDATE fact_tag SET dateId = (
SELECT a.dateId FROM dim_date a
WHERE a.year = year(from_unixtime(fact_tag.timestamp))
AND a.month = month(from_unixtime(fact_tag.timestamp))
AND a.day = day(from_unixtime(fact_tag.timestamp))
)

It is worth adding a where clause to each of these and exclude items with id being updated equals null. (I.E. – for dateId,  append ‘WHERE dateId is null’ at the end of update query.  Some of these may (just may) take days so this is a no brainer although I didn’t do in all cases.

If anyone is thinking about how long these take, I recall the timeId population for table fact_rating took my Core 2 Duo 2.26, 4GB of ram laptop 18 hours and change.  Its an ‘execute and go to bed’ operation for sure for me.  Looking forward to having a completed star-schema to play with!

Last week I wrote a post about the differences between direct and staging of data from SaaS/cloud applications. I received a very thorough comment on the Informatica Perspectives blog about the benefits of staging data that I think is worth summarizing here:

You can read the full comment here. Any other experiences or best practices to share? Do you agree / disagree? And what if you don’t have the IT resources for this type of architecture? Is direct for SMB and short-term tactical requirements only?

The 10 million ratings set from Movielens allows us to create two fact tables (linked?!).  We can create a fact table for ratings and another one for tags.

Worth noting that a userIds between these two schemas (one from ratings.dat and the other from tags.dat) do match across sets.  I.E. – userId 1234 in tags dataset is user 1234 (if existing) in ratings dataset.  So we could link these but, for now, its simpler not to.

Information provided in the 10 million ratings set allows us to create similar star schemas as follows:

Ratings

Tags

I’ll be running these proposed schemas by my peers at work and post back any insight I am sure to be missing from these.  A lot of design here is based on my perception of what matters and its worth seeking advice before I trail off too much.

It is odd that we have no more information for the users in our set.  The smaller, 1 million set does provide move interesting user information but no tags from said users.  Perhaps it is worth considering both datasets at the same time and treating them as two different sets of study.

After 27 years of steady growth, Austin, Texas-based Pervasive (PVSW) has become a $47M annual run rate software provider. Its portfolio includes a “zero admin, light footprint database” (the former BTrieve, now PervasiveSQL), data integration software (for SaaS and on premises applications), and data synchronization products for such apps as salesforce.com, Quickbooks and Microsoft Dynamics CRM. In 2009, it began leveraging its DataRush processing engine as a product, providing a solution for companies that want to take advantage of multicore architectures to drive dramatically enhanced performance on much smaller footprints, for programming data services tasks such as aggregation, de-duplication, cleansing, integration, matching and sorting, as well as data mining and predictive analytics.

Pervasive has racked up 36 consecutive profitable quarters over the decade, and through the last couple of years has seen a significant ramp up in revenue. Claiming over 4500 customers and over 1000 SaaS implementations, it projects a sense of being in the right place as the market catches up to it. It’s moving its portfolio steadily into the cloud – the Pervasive DataCloud, running on Amazon today. But it remains little-known to the general IT community. The database side of the firm represents 2/3 of its revenue and has been steady, but its database brand is rarely visible.

Why the stealth model? “We do sell our data products to large corporations, as well as through SIs and SaaS vendors, but we designed them to be embedded,” Jim Falgout, Pervasive DataRush chief technologist, told me in a recent briefing. “We can compete with majors like Informatica, IBM and Oracle on product, but that’s not our primary focus. The database, for example, is used almost entirely for embedded applications – that’s a different market and a different selling model.” Today, although the other Pervasive product teams emphasize brand, the majority of Pervasive’s sales are through the channel, and growing faster there than through direct sales. Over 150 Pervasive Integration ISVs including McKesson, Sage, Epicor, Intuit, Metavante, Eloqua, Daptiv, and Xactly are using the firm’s software. System Integrators such as Ceridian and CSC in vertical markets like healthcare and telecom are also an area of focus. Pervasive focuses aggressively on making the connection to its partners when inquiries come in, both in North America and internationally.

The flagship Data Integrator offering (which began many years ago with the acquisition of Data Junction) provides connections to a huge number of data sources. Pervasive’s ISVs and SaaS vendors put it through its paces for connections to databases and applications for B2B integration using a variety of industry standards including  HL7, HIPAA, EDI, NCPDP, FIX, UCCnet, ACORD, as well as various XML and flat formats.  In the past few years Pervasive has been increasingly used for connecting SaaS applications at the periphery, as within the SaaS stack. The company asserts that it has been a partner of Salesforce.com longer, and has more in-production customers, than any other vendor offering Salesforce integration.  DataCloud offers the ability to do specific integration and synchronization jobs without any on-premise software. It also includes pre-processing capabilities, such as DataProfiler, from Pervasive’s broader portfolio of data integration products.

The new news is DataRush, and the equation is simple and persuasive: more cores = more speed. And no rewrites needed when moving to similar processor architectures – even across different operating systems. For programming organizations that are using R for statistical projects, or coding ETL and data analysis work with MapReduce, the performance benefits claimed are very attractive. Nena Marín, Ph.D. Pervasive DataRush chief scientist, described the software platform as java-based, relying on a message passing architecture. Difficult problems like deadlock detection, threading, and resource management can be roadblocks that prevent programmers from getting the full value of parallel architectures.  DataRush makes it easier to code than using Java primitives. Pervasive also provides profiling and debugging tools, tackling an often daunting challenge for many-threaded app architectures.

Our conversation veered into discussions of AMD’s Code Analyst, which allows programmers to take info from the JVM and combine it with processor information to look at such useful indicators as cache misses and overflows. By this time, I was deeper into the internals than I wanted to be, so we returned to a discussion of use cases – and things became even more interesting. In today’s market, where running code against files is often a viable alternative to building a permanent data warehouse, Pervasive’s performance benefits offer sizable advantages, and the claims are compelling. Parallelizing specific algorithms outperformed R and MapReduce jobs in dramatic fashion in some cases Pervasive showed me.

A note of caution: at this stage, Pervasive has few attributable customer stories to show, so much of the material presented was internal tests and benchmarks. I’m told Pervasive has now completed implementations for some unnamed customers (and have others in process) using Pervasive DataMatcher, a solution built on Pervasive DataRush to provide fuzzy matching on large datasets with rapid throughput.  The usual caveats I offer with no named references apply here: nothing substitutes for your own data and your own business problems. Pervasive offers a free download; get your hands on it and run your own tests.

DataRush is a new business for Pervasive, and is being treated like one: the team is relatively standalone. The division has its own sales and field organizations and its own R&D. Pervasive is clearly not yet funding dramatic growth; it seems content to expect DataRush to be “significant within 2-3 years,” recognizing the long sales cycle in its channel and ISV model. This is likely to be a self-fulfilling prophecy unless Pervasive makes significant additions to its current marketing. A few case studies, some focus on reaching the programmer community through social media, events and publications could go a long way. It’s unlikely that Pervasive DataRush will remain the only provider of such services; it needs to capitalize on its lead now and gain a branded foothold.

Disclosures: Pervasive Software is not a client.

Today I am at the Steel City SQL Users Group, presenting on Microsoft’s newest BI Tool, PowerPivot.

The slide deck for this meeting can be found at:

http://arcanecode.files.wordpress.com/2010/01/powerpivot_long.pdf

To see all my posts so far on PowerPivot, you can use the link below to filter.

http://arcanecode.com/category/powerpivot/

Thanks for coming!

What a great opportunity to put the MovieLens data to good use (again).  Previously, I had used this dataset to build a movie recommendation engine based on the book Collective Intelligence.  I will be spending lots of time at work creating OLAP cubes for business reporting.

Immediately, I thought the Movielens dataset would be as good as it comes to practice concepts and practices I’ll be exposed to at work.  I am going to need all the help and practice since the world of Business Intelligence is new to me.

A little history about the dataset I am referring to can be found at the GroupLens Research.  The dataset I am planning on using can be found here. This time, I’ll be using the 10 million ratings set since it seems everything at work is measured in millions.  There are a few other variations of this data. Look around; you may find something you like.

After downloading the files, I proceeded to import all the data as is into our ‘original’ database.  I am using some late flavor of mySQL and was able to import the data without much fuss.  The only thing I had to do was replace the delimiters (double colons ::) in the movies data file for something else.  Movie titles with colons where being inadvertently ’split’ across more than one column on import.  Doing this produces a database with three initial tables.

One table holds information on the users rating at least 20 movies, another one holds all the movie information and the last one holds the 10 million ratings for these movies looking like this:

10m_ratings(userID, movieID, rating, timestamp)
10m_tags(userID, movieID, tag, timestamp)
10m_movies(movieID, title, genres)

Table naming prefix aids infer purpose since I am restricting myself for one database for this project.  More so, sql styles and the like suit me fine

In total, you end up with a database a bit less than 400 MBs.  Relevant information for data I will be using can be found here.  If there is any interest and licensing allows it, I’ll post a sql dump or db backup to share and same someone some time.

ADBMS vendor SAND Technology’s report on its 2009 fiscal year seemed to offer little reason to change my earlier skeptical position on the firm. Its 2009 revenue was essentially flat at $7 million (Canadian dollars throughout). Cost of sales, R&D, and SG&A – and the firm’s net loss – were also nearly unchanged. And yet, there are changes going on, and they are positive signs, especially for a year in which the IT market will rebound. Net income for SAND’s fiscal 2010 first quarter was $553,253 on revenues of $2,485,464 – a substantial turnaround from a net loss of $989,850 on revenues of $1,223,928 for fiscal Q1 2009. One quarter is not a trend, but it is a good sign.

In 2009, SAND’s North American revenue was up and Europe down – both for the second successive year, reflecting traction where it is focusing. Both were up in fiscal Q1, VP Linda Arens tells me. A partner in Korea has begun to produce very strong results and may expand its vertical orientation beyond its foothold in banking.  SAND has a little more cash in the bank, especially after its Q1, and is talking about using some to add more direct sales staff. This would also be a transition; the direct sales team has been tiny thus far, with partners like Accenture and others driving much of the success. The company is focusing its field efforts on SAP sites – there are many of them – and this targeting is likely to help what they tell me is already a better pipeline than they have seen in some time. SAP has done several joint customer installs with SAND for its business warehouse (SAP BW.)

SAND’s technology – its “100% Indexed” tokenized data format, columnar orientation, and removal of redundant data – make it attractive to organizations that are seeking to reduce the footprint of data that doesn’t need to be “online” in the traditional sense, but still must be accessible via standard SQL, ODBC, JDBC or OLE/DB. The company has tightened its messaging somewhat, honing in on the nearline repository idea and associating it with the notion of Corporate Memory that permits rapid response to unpredictable needs.  PCI Data Security Standard (PCI DSS) certification is in test with some customers who requested it. (PCI DSS specifies requirements for security management, policies, procedures, network architecture, software design and other protective measures for customer account data.) There’s lots of competition for the “exploration warehouse” business, but focusing on ease of use, and integration with commonly used tools, is helping to break down the doors.

By comparison to many of its large competitors, SAND remains a tiny player. This manifests itself in its tiny sales force, a website that has no current press releases (even though many have been issued), and its relatively low profile compared to newer firms in the same space.  But it’s not alone in finding that many firms are receptive to alternative approaches to nearline analytics, and as consolidation at the top continues, SAND may well be starting a growth spurt in 2010. It belongs on my shortlist as I watch the market – and maybe on yours too.

=============

With this post, I add a new disclosure section to my blog. Transparency has been a frequent topic in the independent analyst circles I travel in, and I hope my readers find this useful.  I own no stock  in any technology firms. Vendors will be listed as clients if I have done paid projects for them within the past year and/or if I have a retained consultant relationship with them. Under no circumstances does my blog writing depend on these relationships – the blog is not, and never will be, written for pay.

Disclosures: SAND Technology is not a client.

There were many predictions in the Software industry for 2010. One of Industry thought leaders Nenshad Bardoliwalla  had his predictions in the area of “Trends in Analytics, BI and Performance Management.” His predictions about how vendors will have packaged/strategy driven execution applications, slice and dice capabilities from the BI vendors returning to its decision centric roots and advance visualization capabilities got me thinking about my favorite topic about purpose built Applications.

What is a purpose driven/built analytic application (PDAA) after all? It is an analytic application which addresses a much focused business area or process and provides insight into the opportunities (for improvements), challenges (performance). In order for such analytic application to provide insight…

1. It needs to be designed for a specific purpose (or problem in mind), and that purpose or focus really needs to be narrow (to be able to provide holistic insight)
2. It needs to rely on purpose built visualization and needs to use Web 2.0 style technologies to make analytic insight pervasive (some examples to follow)
3. It needs to provide descriptive, prescriptive and predictive capabilities to provide holistic insight
   1. Descriptive capabilities will provide view into state of current affairs
   2. Prescriptive capabilities will provide what users need to focus on as a follow up, it also helps in guiding users as to what questions they should ask next to build the holistic insight
   3. Predictive capabilities will facilitate what if analysis and provide insight into what situation business might expect should the current situation continue.
   4. It implicitly provides users with what questions users should ask in a given situation and provides either complete answers or data points leading up to those answers…

Many a times, because of the very specific purpose and narrow focus, most of the insights provided by purpose built analytic applications can be manifested right into the operational application via purpose built gadgets or even purpose built controls. Single dashboard with a interactivity around the widgets/gadgets in the dashboard will typically provide complete insight into the focus/purpose of the analytic application.

Let us discuss an example of what a purpose built analytic application could be…Every organization which has sales force actively selling products/services of the organization has a weekly call to review the pipeline. This is typically done region by region basis and the data is then rolled up at a global level.  A purpose driven analytic application in this situation would be “Weekly Pipeline Review” application. In this application rather than providing free form slicing dicing/reporting capabilities around pipeline data (which will be traditional way), this type of application will focus on:

1. Current Pipeline
2. Changes to the pipeline from last week (positive, negative: As this is what is really watched closed in this call to make sure forecast numbers can be achieved)
3. Indicate impact of the changes to the pipeline on achieving goals/forecast. Based on these changes, extrapolate the impact on Sales Organizations plans…. (what-if)
4. Provide visibility into deals which might be problematic based on the past performance and heuristics (this is what I call prescriptive)
5. Provide visibility into deals which are likely to move faster and close faster, again based on past performance. (Again prescriptive)
6. Provide account names in which incremental up sell can be done (again based on past performance in similar accounts) but there are no active deals/opportunities etc…
7. Provide visibility into individuals and regions which are at risk of missing their forecast based on their past and current performance.

There are different visualizations which can be used to build such type of application. Focus of this analytic application is to help Sales VP’s and Sales Operations to get through weekly pipeline review call quickly by focusing on exceptions (both on the positive and negative side) and provide full insight into the impact of the changes, areas which they should focus into etc…. Hopefully this explains in detail the difference between purposes built analytic application vs. traditional data warehouse or traditional analytic application.

Let us now briefly look at how purpose built UI supports some of the important aspects (holistic insight) of the purpose built applications. Many of you have used Google portal and have uploaded iGoogle gadgets. One can look at iGoogle gadgets as purpose built applications which focus on one specific area of interest to you.  Take a look at one of the samples put together by Pallav Nadhani to demonstrate Fusion charts visualizations. This gadget is a perfect example of how purpose built UI helps in creating the focus and holistic insight of the analytic Application. This gadget provides complete Weather picture for a location for today or for future.

 There is a company out of New Zealand, Sonar6 which provides product solution around performance management (much focused, purpose driven)/Talent Management. They have done fantastic job of building purpose built application and delivered that application through purpose built UI. I especially like the way they have provided analytic and reporting capabilities (helicopter view) around performance management. You can register for their demo or can look at their brochure/PowerPoint presentations.

There are several other vendors who have made purpose built analytics pervasive in our day to day lives. Recommendation engine built by Amazon is a perfect example of “Purpose built Analytics”

In the end, I truly believe that purpose built analytic applications can and will maximize the value/insight delivered to the end users/customers while keeping the focus of the analytics narrow.

I would love to know your thought around purpose built applications. What has been your experience?