The Wizard of Oz has just turned seventy years old and is one of the most watched films ever! So just what exactly has this got to do with good database design?

Well in the film Dorothy and her trusty dog Toto have to follow the yellow brick road to finish their journey. Along the way when they left the road they met all sorts of problems but eventually with the help of some new found friends Dorothy got back onto the road and eventually reached her destination.

No matter what business you are in you will have set procedures to complete a specific task. You might have to follow a specific sales process. Maybe there are legal restrictions and requirements to consider. Perhaps you are using set scripts when staff speak to customers. Are you sending out catalogues and brochures?

So when you or your staff step off the “yellow brick road” are you running the very real risk that something might go wrong? Is that vital piece of information you need to complete that sale missing? Ask yourself how many times YOU have cut corners and ended up going back to re-do things. I know I have many a time and almost every time I have ended up spending extra time back tracking and starting over!

How much time does it take to sort out problems caused when people don’t stick to the script or the laid down procedures. In business we all know time is money! Perhaps you use scripts, check boxes and tick lists to help, but are they always followed or used?

So here’s the thing! If in your business you have set procedures, what safety net do you have in place to keep everything on task? It might be that a database system can do exactly that for your business. A good system can make sure that every “i” is dotted and every “t” crossed. Never again forget that vital piece of information to complete a sale!

I strongly recommend any business to take time out and consider exactly what is in place to make sure that procedures are followed?

How much quicker would Dorothy arrive at the Emerald City if she had not ended up in the castle with the wicked witch and instead stayed on the yellow brick road?

When kicking off data management initiatives a large and key component is establishing the data stewards that represent the data that is collected, managed, and leveraged in business intelligence.  By having these data stewards, and subsequently a data management committee, companies feel safe that the proper data governance practices are going to be put in place.  Not so.  Ownership (=Stewardship)  does not equate to governance.

Many factors contribute to governance and business boundaries can quickly be broken down if you approach governance in business silos.  As you walk through your process of data collection you’ll quickly find that what is considered the preferred source of data may not be generated by the team that determines what should stay, what should be modified, and what should go.  In fact, depending on how you view the data, conflicts arise as to what is considered accurate, appropriate, of the contributing factor in decision and business point of view.

This is something I’ve run into recently when building a business intelligence solution for web analytics.  Even within my own department of advertising executives, views of what transactional data should be considered the record of source is up for grabs depending on who is the recipient of the information and how it is used.  Levels of accuracy vary depending on when data is needed, how it may be used for marketing optimizations, or if it will be used to actualize spending for billing.  Throw into the mix that data feeds coming from vendors are constantly changing as they actualize transactions over the course of days, weeks, and even months, and finding the truth in the data becomes a challenge that defies religious opinion on the subject.

Sorting through the challenges of governance to determine what makes data reliable requires looking at a variety of factors and allowing for multiple views and uses.

• Reliability of source
• Time of collection
• Actualization
• Business process affected/use of data in decisions
• Degree of accuracy required

If you will notice, I do not include ownership.  This is the artificial governance.  Ownership in establishing governance only serves to create a framework around the above factors that creates credibility.  Ownership, and then the transformation to stewardship, serves to continuously monitor, enforce, and improve governance around data needs.

Start your data management off on the right foot, don’t confuse ownership with governance.

By Amirth Ramakrishnan

In these tough economic times, companies are looking to leverage their existing assets to cut costs and grow their businesses. These assets include physical, human and informational assets. Physical assets include plants, machinery and buildings. Human assets (people) use these physical assets to deliver products and services to their customers.

In recent years, companies have generated huge amounts of data in the process of creating, developing and delivering products and services to customers. The data is resident in companies’ information systems, often in enterprise resource planning systems. As companies mature, they face formidable challenges in managing this growing mountain of data. Over time, many corporate strategies evolve as a result of changing business climate and issues, resulting in strategies that are often reactive and suboptimal.

Businesses can adopt a comprehensive approach to effectively leverage information in ERP systems for competitive advantage. The approach outlined in this article serves as a useful starting point for adopting an effective EIM strategy or to validate an existing EIM strategy.

Strategy Overview

Companies need to pursue the following elements of a four-dimensional strategy to ensure a comprehensive approach to EIM. This strategy takes a “lifecycle” approach to EIM, i.e., it is based on how data is acquired, managed, secured and disposed of in an organization. Most of the other approaches are based on classifying data based on its type (for example, into master data, transaction data, metadata, etc.) but they do not provide insight into how an EIM strategy should be initiated and deployed within an organization. 

EIM Governance: The core of the strategy is based on the right governance structure for managing EIM initiatives within the company. First, the governance team should establish the policies and procedures to launch EIM projects based on long-term business strategy and return on investment. ROI places a quantitative value on the business benefit of the program or project that leverages the information resident within the company. The governance team should consist of both providers (e.g., the order management team which is the source of transactional data) and consumers (e.g., the business intelligence team which uses transactional data to provide reports or inputs to senior management).

The governance team plays a pivotal role in ensuring the right projects are accepted. In the initial stages of the company’s EIM roadmap, focus should be placed on projects that help companies acquire information.

Acquire: Companies acquire information through internal (transactional as well as other operational systems) and external systems during the course of running their business. The process for validating, accepting and managing this information is an important part of the EIM strategy. Typically, enterprise-wide standards have to be implemented to ensure the correct level of information is captured during these interactions to enable a cohesive EIM strategy.

Share: One of the major problems that enterprises face is that organizational information is in silos and there is not a single view of customers, suppliers, partners or even employees. These problems multiply when companies acquire, merge, disinvest or spin off divisions or geographical units. Projects and initiatives that enable enterprise information to be transparent and readily available to business leadership and other stakeholders fall under this category. These initiatives (for example, data warehouse, reporting, etc.) thus enable quick decision-making and help senior leadership to understand how different businesses and geographies have performed. BI and data warehouse tools are typically used to enable these initiatives.

Improve: The quality of information available within a company’s systems is dependent on innumerable factors, some of which are beyond enterprise control. Companies receive feeds and data from suppliers, partners and customers, and there is high likelihood of inconsistent data quality. These problems become apparent during software upgrades and migrations or ERP implementations. IT leadership should measure the information quality through various metrics (such as error rates in order or service fulfillment, invoicing and billing errors, etc.) and target specific improvements through focused projects that improve information quality.

Secure: Every enterprise has to zealously guard their informational assets against willful or accidental breach of security. The potential hit on company’s business makes it imperative for IT leadership to ensure security of information. Projects and processes to ensure the right level of access and security need to be implemented to ensure a strong and secure informational infrastructure that business leadership can rely on to make tactical and strategic decisions. Projects that help ensure security of information are typically run in parallel to other programs that acquire, share and improve the enterprise information infrastructure.

Evolution of EIM Strategy

Tailor the EIM strategy based on specific company needs. Startups and young companies typically focuses on acquisition of information programs and projects that will help them generate leads and opportunities to sell their products and services. Companies that are more established tend to focus on improving the quality and transparency of the available information. Lastly, companies in highly regulated and competitive industries (for example, government, health care and financial services) invest a significant portion of their IT budgets in information security initiatives.

EIM Building Blocks

Some of the technology building blocks supporting a successful EIM strategy involve a service-oriented architecture that will enable improvements to ensure companies can acquire, share, improve and secure their information assets. In addition, companies need to standardize on tools, processes and standards to ensure their EIM strategy is understood and implemented at all levels of the organization. Further, given the global nature of the business, IT leadership should evaluate their systems, processes and tools that can adapt to global and virtual teams.In the coming years, enterprises will increasingly face challenges relating to unstructured information like emails, text messages, tweets, blogs as well as internal portals and other content management tools. The strategic elements to manage these streams of information remain the same. However, software tools and applications will continue to evolve and help achieve a unified lifecycle-based EIM strategy.

About the author: Amirth Ramakrishnan is a senior account manager with Sierra Atlantic Inc. with more than 14 years of experience in the global IT services industry. He works with customers to define and implement IT strategies and solutions for the extended enterprise by using the Global Delivery Model. He is based out of Austin, Texas and can be reached at Amirth.Ramakrishnan@sierraatlantic.com

After receiving a comment on last month’s post I decided to do a follow-up and detail a little further how Microsoft Dynamics CRM manages the merging of duplicate records.  For the purposes of this post I’ll stick to using Contacts as the example.  However, the same is true for Accounts and many other tables. 

For our sample records let’s say we have just two contacts that are duplicates.  Contact A has four service calls associated with it.  Each of these service calls have relevant data that you want to retain.  Contact B has three service calls associated with it and each service call has data that needs to be retained.

Upon merging Contact A with Contact B (so in this case B is the keeper record), there will be seven service calls associated with Contact B.  This is accomplished through the use of three data elements in each Contact transaction. These fields are MasterID, Merged, and Statecode. 

Merged is an indicator field where 1 indicates that the transaction is indeed merged.  Statecode is another indicator field indicating active and inactive transactions.  In Dynamics a statecode of value of 1 is inactive and 0 is active.  Yes, you read that right.  Zero is active. 

 The magic of the duplicate consolidation lies in the MasterID field.  For consolidated records the MasterID is equal to the unique identifier of the keeper record.  So in our example if the unique identifier of Contact B was 1234, the MasterID of  Contact A will be 1234.  The Merged field would be 1 and Statecode of 1.  This allows all associated transactions of Contact A to be “realigned” to Contact B without having to perform massive updates to many tables.  The MasterID functions as a re-pointer for all related transactions associated to the merged record.  So there are no changes to the related transactions.  The re-pointer directs all transactions related to the non-keeper record to the new keeper record.

For those transactions which are not merged the MasterID is NULL (no need to store the unique identifier twice).

Service interruptions of digital systems can inconvenience millions of people and have a significant financial impact on the provider. If the Amazon web site, or Google’s Gmail, or the Visa payments network goes down even for a few minutes, it can make front page news.

As digital systems grow larger and more complex, it can become very challenging to monitor their health and status, which is the first step in detecting potential problems, identifying the root causes, and taking appropriate preventive actions. These types of systems can contain thousands of different data feeds, data flows and processes. A problem with just one of them can interrupt payments, ads, and status updates, respectively. Often there are hourly, daily, weekly and seasonal variations in the data that complicates the detection of problems.

One way to gain some insight into this problem is to look at the origins in the 1920’s of statistical quality control. Walter Andrew Shewhart (1891 – 1967) was an engineer at the Western Electric Company, which manufactured hardware for the Bell Telephone Company, from 1918-1924. From 1925 to 1956 he was a member of the Technical Staff of Bell Telephone Company [ASQ].

One of the problems that concerned him was identifying potential problems in factory assembly lines. For example, the dimensions and weight of metal parts that are sampled from an assembly can be recorded. He distinguished between two types of variations in these measurements:

• Common cause of variation (or noise) occurs as a normal part of the manufacturing process.
• A special cause of variation is not part of the normal manufacturing process, but represents a problem.

One of the goals of statistical quality control is to distinguish between these two types of variation and to quickly identify special causes of variation.

Shewhart introduction control charts as a tool for distinguishing between common and special causes of variation. A control chart had a central line and upper and lower control limits. When the measurement exceeded either the upper or lower control limits, it was considered a potential special cause of variation and investigated. Usually, the upper and lower control limits were three standard deviations above and and below the mean.

As anyone who has investigated potential data quality problems knows, identifying roots causes of potential problems is not easy and Shewhart also introduced a four step approach to these types of investigations that became known as the Shewhart Cycle, the Deming Cycle or the Plan-Do-Check-Act Cycle:

• Plan. Identify an opportunity or potential problem and make a plan for improving it or changing it.
• Do. Implement the change on a small scale and collect the appropriate data.
• Check. Use data to analyze statistically the results of the change and determine whether it made a difference.
• Act. If the change was successful, implement it on a wider scale and continuously monitor and improve your results. If the change did not work, begin the cycle again.

These same ideas are still used today as the basis for health and monitoring systems. Well designed digital systems these days are designed from the ground up so that appropriate log data is produced. Instead of a single assembly line producing physical items, there are thousands or millions of digital processes producing (nearly) continuous digital data. Often this data is available through an http interface and is continually collected.

This is dashboard from the open source Augustus system for health and status monitoring.

Instead of a control chart, a change detection model is used, such as a CUSUM or GLR statistical model [Poor]. Instead of building a single model, a model for each cell in a multi-dimensional cube of models is built [Bugajski]. Instead of looking at the charts each day, an online dash board is used that is at the hub of an operations center.

Baseline and change detection models for each cell in a multi-dimensional data cube of models can be built easily using the open source Augustus system.

References

[ASQ] ASQ, The History of Quality – Overview, retrieved from www.asq.org.

[Bugajski] Joseph Bugajski, Chris Curry, Robert L. Grossman, David Locke and Steve Vejcik, Data Quality Models for High Volume Transaction Streams: A Case Study, Proceedings of the Second Workshop on Data Mining Case Studies and Success Stories, ACM 2007

[Poor] H. Vincent Poor and Olympia Hadjiliadi, Quickest Detection, Cambridge University Press, 2008.

Let me give you a real world example about how a well constructed database system can take the strain of your business and improve your productivity at the same time!

Every Friday, one business I know used to have to generate specific personalised reports for around 40/50 clients. Even though they were using a database system this took on average four hours every week.

They would create a report for the first client, save it to a folder with a unique name. They would then, using Outlook, create a new email for this client, locating the email address and backup email address from within Outlook. The body of the text would be modified to personalise it for the client.

After clicking “send” they would repeat this process for each other client!

It did not take long to create a system where the operator opened a form listing all clients and then clicked a box next to each specific client needing a report that week. That took around sixty seconds to do. They then clicked on one more button and the system took over automatically creating each personalised report and emailing it directly to those who needed it using a personalised email. The system logged the email and saved copies of the reports.

This upgrade also helped in other ways as the original system meant that email addresses were being used from two different sources and I have already explained in a previous blog why that is not good business practice.

This one simple update to an existing system saves around TWO HUNDRED HOURS each year. Put another way, that is more than a month’s salary for one person who could now be put to work elsewhere in the business.

So when was the last time you reviewed your work processes? Have you ever reviewed them? How many repetitive jobs do you undertake every day, week, year? Sometimes it can be very difficult to see what is right in front of you!

So you undertake a review and discover several processes that you just know you can streamline but you say to yourself that it is going to cost too much to sort out.. or is it? Many will fall into the trap of just looking at the cost of developing such enhancements and ignore the full benefits both in financial and productivity terms.

Then there are the intangible benefits! Think back to my earlier example, two hundred extra hours to work on getting more sales or adding value to existing sales. How much extra revenue could that produce for your business? You could soon find that the cost of development has been more than covered by the extra revenue!

So if you find in your business processes that you continually have to repeat it might just be worth considering ways you can boost your efficiency and in doing so…. your bottom line profit!

“One of the biggest technical obstacles in the world of cloud computing is integrating cloud applications with each other and with on-premise systems.”

This quote comes from a ChannelWeb article today focused on cloud integration. Based on the number of Dreamforce sessions on this topic, it’s clearly top of mind for both line of business and IT executives seeking to do more in the cloud heading into 2010.

Earlier today I spent a bit of time going through some of the sessions on integration (simply login to Dreamforce.com and go to the Content tab). Here are some of the primary take-aways I gathered when it comes to cloud integration best practices:

Considerations:

• Effective integration has benefits before, during and after implementation.
• Successful implementation needs the right software, engagement by all stakeholders and the right processes.
• Don’t run before you can walk – agile development, bite-sized chunks.
• Expect incremental changes – you can’t anticipate everything.
• Use a phase approach when integrating salesforce.com with other applications.
• Plan adequate testing cycles – “plan before building, test before launching.”
• Project management is critical.
• Training is critical.
• Involving end users early and often is critical.
• Salesforce.com and their partners are always available to assist.

Approach:

• Real-time vs. Batch Integration?
• Hub vs. Point to Point?

Customer Master

• Integrate with all of your applications, which will allow you to centralize all of your customer data into an one system, which will make it easier for the entire company.
• One view of customer essential for effective customer management and asset/resource maximization.
• Choose your system of record.
• Treat salesforce.com as your golden source of information.

Data Quality

• Start data cleanse early, especially if quality is poor. Ongoing data ownership is key.
• Stop errors at the source.
• Be sure the input data is structured, formatted as needed.
• Maintain data clean from duplicates by having a process for duplicate records.

Got some more? Which of these best practices are/were most important to you? And when it comes to a cloud integration solution, what’s most important to you?

Quick Poll:

I said in my previous blog that I would cover “why” you should use a database. “Why” is a small word with a big meaning and as there are so many reasons why a business should consider using a database, I am going to break this down into bite size chunks!

First of all I am going to cover the need to have good quality, accurate information recorded no matter what system you use. I would argue that for any business the quality of your contact with your clients/customers is one of the most important aspects that needs to be considered.

Let’s just take a moment to consider our own personal experiences with other companies and organisations. How do you feel when you receive correspondence that has your name spelt wrongly? It happens to me all the time and yes I know my surname is not that common but the companies that take the time to get it right move up in my estimation and are more likely to get my business now and in the future!

Okay, so mistakes happen but then how much more annoying is it when you tell them about the mistake yet it is still not corrected! If companies cannot achieve something as basic as spelling your name correctly, what else are they doing wrong!

Most people will shop around and attention to detail can help give you the edge against your competitors.

No matter what client information you need for your business you can guarantee it is like storing shifting sand. Details constantly change. People move house, change their mobile phone numbers, change email details, move jobs, have children, get married, get divorced! The list is never ending!

Maybe you have client details recorded on a spreadsheet. Perhaps on more than one spreadsheet! Maybe you also have details recorded in a paper file and in an accounting package? So you phone a client and find out they have moved. You note this in the paper file but forget to update the spreadsheet and the accounts package.

You don’t realise that there is a problem until that invoice you sent out 90 days ago has not been paid and you have not been getting responses to reminder letters. By the time you have established what has happened you have not only wasted valuable business hours chasing payments but also your cash flow has been adversely affected possibly costing you more money by way of interest charged or not gained!

A well written database system will store client information ONCE and used it throughout your business. This makes it far easier to keep up to date which saves you time AND money.

My next blog will consider the importance of NOT wasting time in business!

Dataqualitypro présente les 7 compétences essentielles d’un responsable de la qualité des données dans une petite, moyenne ou grande structure.

Chaque initiative de mise en place de projet de qualité de données requiert un leadership efficace pour aider à créer le bon environnement pour l’ensemble des acteurs du projet.

Efficacité dans la sélection des projets

Les problématiques et interrogations sur la qualité des données dans toute organisation ne sont pas difficiles à trouver. Il existe d’innombrables «bonnes causes» attendent de bénéficier d’une campagne de la qualité des données dans toute organisation qui est guidée par les données. Sélection du projet est primordial.

Le chef de la qualité des données doit exercer des mesures de contrainte, l’égalité ainsi que l’engagement, la patience ainsi que la passion.

Sélectionner le bon projet pour le lancement est l’endroit où de nombreux dirigeants échouent finalement dans leur quête pour mettre la qualité des données sur la feuille de route de l’entreprise.

Choisir le bon projet et vous avez des unités commerciales qui réclament pour votre aide. Obtenez-le mal et vous serez considéré comme un centre de coûts et de trouver qu’il est extrêmement difficile d’obtenir des fonds supplémentaires.

Voici quelques conseils pour choisir le bon projet :

• Vérifiez que votre projet est lié aux priorités stratégiques: Si vous avez créé une analyse de rentabilisation (conseillé), alors cela devrait être évident, mais il est essentiel que les avantages de votre initiative ne peut être clairement attribuée à des stratégies d’entreprise.
• Soyez réaliste quant à la portée: La gestion des données de qualité ne peuvent être résolus en quelques semaines ou quelques mois. Cela peut prendre des années à transformer la culture d’une organisation afin que des améliorations durables peuvent avoir lieu. Cela dit, de nombreux projets échouent parce qu’ils sont trop ambitieux. L’essentiel est de réaliser des projets qui créent des avantages tangibles à intervalles réguliers. C’est là laser tels l’objectif et la hiérarchisation sont essentiels.
• Créer des résultats tangibles: Alors qu’il est souvent important de lancer des projets qui mettent l’accent sur la “plus doux” prestations d’entreprise telles que la satisfaction des clients et réduire le risque, il n’y a rien d’aussi fort que les résultats finaux. Un choix parfait projet a souvent un mélange de prestations matérielles et immatérielles, mais comme chef de projet, vous serez tenus pour responsables d’assurer ainsi que votre projet sera capable de démontrer des améliorations mesurables qui ne récupère pas seulement les coûts du projet, mais créer des recettes pour de nouvelles initiatives.
• Plan d’un pipeline durable des futurs projets: Un dirigeant efficace la qualité des données doivent planifier 2 ou 3 projets à venir. Si vous avez défini un objectif d’offrir le retour sur investissement positif sur vos projets (et vous devriez) alors vous devriez être constamment évaluer quels sont les projets futurs de lancement qui aura réussi à étendre la gestion des données de qualité au sein de votre organisation.

Efficacité dans la sélection des acteurs du projet

Des données à long terme d’amélioration de la qualité ne dépend pas d’une fissure sur le déploiement de l’équipe de “gourous” qui peuvent être déposés dans n’importe quelle situation et de créer instantanément des conditions de réussite.

Une équipe efficace la qualité des données est une force de collaboration, alliant connaissance de l’entreprise en profondeur avec la capacité technique. Plus important encore, les membres de l’équipe doit apprendre à se livrer, pas dicter. Pour mettre en œuvre une amélioration de la qualité des données est relativement simple, mais peut être un exercice inutile si soi-disant experts apporter des améliorations qui ne peut être soutenue avancer. La gestion du changement est essentielle et tout le monde dans l’équipe devrait comprendre l’impact que leurs efforts ont sur la population active. Trop nombreux dirigeants tentent de forcer le changement sur une communauté involontaire. Les résultats immédiats suivie d’un déclin progressif est souvent le cas.

Efficacité dans la sélection de la méthode à mettre en place

En tant que leader qualité des données, il est responsable de la ratification de la stratégie ou la méthodologie à adopter. La question essentielle que le leader devra garder en tête est la suivante : « l’organisation est vraiment assez mûre pour la mise en place d’une stratégie d’entreprise de qualité? »

Il s’agit donc d’adapter les méthodologies de référence au cadre unique de l’environnement.

Attribution des rôles et responsabilités

Un dirigeant efficace la qualité des données doit être en mesure de répartir les responsabilités appropriés afin que les décisions importantes puissent être prises dans les meilleures conditions. Il est fréquent que des chefs d’entreprises soient investis dans la mise en place de projets mais rechignent à prendre des décisions difficiles lorsque les résultats de l’évaluation de la qualité des données entrent en jeu.

Demandez à participer à ces discussions difficiles au départ du projet, et assurez-vous que chacun des intervenants soit en accord avec le niveau d’engagement nécessaire. Créez un diagramme RACI et expliquer comment chaque personne impliquée dans votre projet sera tenu de rendre compte.

Communication efficace

Sans doute la compétence la plus importante du chef de la qualité des données est la communication.

Un jour, le leader se devra de présenter les résultats de l’étude sur la qualité des données à une équipe de direction et par la suite à un groupe d’utilisateur. La réussite d’un projet de qualité des données repose entièrement sur l’efficacité de la communication, car elle est le fondement de la gestion du changement.
L’interaction efficace entre l’ensemble des communautés du projet permettra de lever un nombre considérable de peur, incertitude ou blocage.

Efficacité dans la formation, le soutien et l’assistance

Dans votre rôle de leader, vous serez responsable de fournir à votre équipe la formation la plus appropriée et un soutien tout au long du projet. Les projets les plus réussis de qualité des données résultent d’une formation spécialisée pour tous les membres de l’équipe.

Gestion du changement

La gestion du changement est un sujet très vaste sujet qui intervient pleinement dans le succès de votre projet de qualité des données. Voici donc quelques-unes des phases de gestion du changement sur lesquelles il faut être vigilant pour devenir un leader de qualité des données accompli :

• Créer une urgence pour le changement
• Former une coalition puissante
• Créer une vision commune du changement
• Communiquer la vision
• Supprimer les obstacles
• Créer un objectif commun à court terme
• S’appuyer sur les changements identifiés
• Ancrer les changements dans la culture d’entreprise

In this article Martin Fowler discusses technical debt.

He says: “In this metaphor, doing things the quick and dirty way sets us up with a technical debt, which is similar to a financial debt. Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design. Although it costs to pay down the principal, we gain by reduced interest payments in the future.”

He also explains the use of the ‘Technical debt quadrant’. Is the debt Inadvertant or deliberate? Is it a reckless or prudent debt – can you afford to pay the debt back in the long run?

As well as ‘technical debt’ maybe we should start thinking about ‘data quality debt’ in the enterprise?

Design decissions taken now might mean having to continually support the information system to ‘fix the data issues’ or even carry out really expensive data cleansing exercises in a few years time.

It could be a really useful metaphor for discussing data quality and wider data management issues with project teams within IT and more importantly the business.

So a few, quick examples of what might contribute to ‘data quality debt’ in an enterprise:

• When setting up a system/business process not identitying clear ‘owners/guardians’ of the data. Who is going to maintain the data and keep it up to date?
• No clear definitions of the data or what it’s purpose is. If there aren’t any then overtime it is highly likely that the data will be misused and its quality will diminish.
• Columns not being set as compulsory to make it easier to carry out initial data imports – as some records in the initial data load do not have that data item.

I enjoyed the good-natured contest (i.e., a blog-bout) between Henrik Liliendahl Sørensen, Charles Blyth and Jim Harris. The contest was a Blogging Olympics of sorts, with the Great Britain, United States and Denmark competing for the Gold, Silver, and Bronze medals in an event called “Three Single Versions of a Shared Version of the Truth.”

I read all three posts, and the excellent comments on them, and I then voted for Charles Blyth.  Here’s why:

I worked as a Systems Programmer in the ’80s in the airline industry. Remarkable as it sounds, system programmers in each airline used to modify the IBM supplied Operating System, which was known as the Airline Control Program (ACP), later renamed as Transaction Processing Facility (TPF).  Could you imagine each business in the world modifying Windows today? I’m not talking about a configuration change, I’m talking about an Assembler code change to the internals of the operating system. The late ‘80s saw the development of Global “Computer Reservations Systems” (CRS systems) including AMADEUS and GALILEO.  I moved from Aer Lingus, a small Irish airline, to work in London on the British Airways systems, to enable the British Airways systems share information and communicate with the new Global CRS systems. I learnt very important lessons during those years.

1. The criticality of standards
2. The drive for interoperability of systems
3. The drive towards information sharing
4. The drive away from bespoke development

What has the above got to do with MDM and the single version of the truth?

In the 70’s and 80’s, each airline was “re-inventing” the wheel by taking the base IBM Operating System, and then changing it.  Each airline started with the same base, but then Darwin’s theory of evolution kicked in (as it always does in Bespoke development environments).  This worked fine as long as each airline effectively worked in a standalone manner, and connecting flights required passengers to re-check in with a different airline etc.  This model was blown away with the arrival of Global CRS systems. Interconnectivity of airline reservation systems became critical, and this required all airlines to adhere to common standards.

We are still in the bespoke era of Master Data Management.  This will continue for some time.  Breaking out of this mode will require a major breakthrough.  The human genome project, in which DNA is being ‘deciphered’ is one of the finest examples of how the “open source” model can bring greater benefit to all.   The equivalent within the Data world could be the opening up of proprietary data models.  IBM developed the Financial Services Data Model (FSDM).   The FSDM became an ‘overnight success’ when BASEL II arrived.  Those Financial Institutions that had adopted the FSDM were in a position to find the data required by the Regulators relatively easily.

Imagine a world in which the Financial Regulator(s) used the same Data Model as the Financial Organisations?

Naturally, such a model would not be set in stone.  There would be incremental improvements – with new versions published on a regular (perhaps  yearly, biannually, or maybe every 5 years).

Back to the great “Truth” debate.  Charle’s view most closely aligns with mine, and I particularly liked his reference to granularity – keep going till one reaches the lowest granularity required – remember, one can always summarise up, but one cannot take apart what has been summarised up.

Most importantly, I would like to thank Henrik Liliendahl Sørensen, Charles Blyth and Jim Harris for holding this debate.  Debates like this signal the beginning of the move towards a “Single Version of the Truth” – and has already led to a major step forward.  Dean Groves suggested on Henrik’s blog that the word “version” be changed to “vision” – Suddenly, we had agreement – We all aspire to a “Single VISION of the Truth”.

I look forward to many more debates of this nature.

You can see the results of the vote here

-BETTER INFORMATION RESULTS IN BETTER DECISIONS

By Philip Teplitzky, HPSquared, LLC

phil.teplitzky@hpsquaredllc.com

This analysis offers a description of an Information Quality & Maturity assessment model and methodology.  The objective is to identify the current state of Information Quality within an organization, or part of an organization, and to use it as a way of developing a quality improvement plan.  It is a component of our overall Information Maturity and Information Governance Business Process Review and Improvement offering and one may consider it an Information Quality Process Redesign (IQBPR).  We have found, based on our implementation experience, that the methodology provides great insight to the root causes of poor Information Quality supports the development of qualified remedial action programs.  It is a good example of the transition from theory into pragmatic and operational methodology.  The transformation from where rubber meets the sky to where the rubber meets the road..

In response to the requests and demands of many organizations we have developed an Information Quality Business Process Redesign (IQBPR) methodology.  The objective is to provide operational Information and Data Management groups with a mechanism to:

1. Determine, based on a set of objective metrics and structured categories, the state of an organization’s Data Quality, Information Maturity, and issues that have a detrimental impact on either and by inference Business Decision making capabilities.
2. Identify the Root Causes of Information and Data Quality issues.
3. Create a Plan for upgrading and improving Information Quality, Information Maturity and Business Decision making capabilities.
4. Provide an ongoing method for monitoring and measuring Information Quality and determining Information Maturity.

Our experience has not come across a widely available and well formulated methodology that provides a means of diagnosing and an approach for addressing root cause deficiencies in data and information quality.  What does exist are:

• Several excellent research efforts underway, which focus on creating taxonomies of data but not Information Quality deficiencies and issues.
• Firms who will administer diagnostic reviews and document their observations and identify Data but not Information Quality issues.
• Consultants who will provide advice on how to improve data quality and integrity but not overall Information Quality.

Our evolving approach is to establish a well-formulated, customizable and operationally available Information Maturity assessment methodology which defines and assesses business behavior patterns across key Information Management functional categories.  There are Data Maturity Models (DAMA, EDM Council), Data Quality Models but no Information Maturity or Quality models.  However and perhaps most importantly, we are not aware of a comprehensive model,  and methodology that links Data Quality and Data Maturity to Information Maturity and in turn relates it all to increased quality of business decisions!  It is our intent to create and deploy this methodology and to make it as easy to use and as widely accepted as a Systems Development Life Cycle (SDLC).

Information Quality, like good software engineering practices and a taste for fine Scotch whiskey, is an acquired taste.  To that end, we have created the Information Quality Business Process Redesign Methodology.  It is our basic lemma that what are required are better business decisions!  To make better decisions you need better Information and to have better Information you need a more mature information environment.  Data Quality and Data Maturity are components of better information but only one of many components, albeit important and necessary.

It is our intention to extend our approach and methodology to include other areas of Information to include, but not be limited to;

• Information Maturity Business Process Redesign (IMBPR).
• Information Systems Life Cycle.
• Information related Audit, Risk and Control.

We are basing our approach and related methodologies on:

• Industry standards such as the Information Maturity Model being developed by the EDM Council
• The great work done by DAMA (Data Administration Management Association) with their DMABOK
• Our credentials  of developing information-centric applications and methodologies as well as the work we have done on Information related Audit and Control.

It is our belief that Information Quality cannot be improved, and by inference better business decisions made, at least on a long-term basis, in isolation.  They must be part of, albeit an important part, of a larger information improvement process.  Processes, must be monitored, measured and constantly improved as one cannot manage what cannot be measured.  Like any machine, the Information Quality machine will develop errors and will deviate from spec.  This is attributable to:

• Changes occur in the operational environment, (e.g. in the MDM (Master Data Management) space the Meta Data definitions from the data aggregators change and if the machine is not re-calibrated there will be errors).
• Deficiencies in the original design, only one designer was omnipotent and omniscient and we are not sure he/she got it right!
• Changes to the downstream applications, new requirements result in functional changes, applications are retired and new ones implemented.  Applications are in a constant state of change.
• Entropy, overtime things just break, just ask Murphy (of Law fame).

This is a work in progress, we have to-date implemented some of the parts of the methodology.  These components have been deployed as part of our Information Resource Management and Information Quality practices and we are confident that over time we will be able to create appropriate sets of components and integrate them into a comprehensive Information Quality and Maturity Methodology and Life Cycle.  Our approach’s goal is to help organizations make better decisions.

Phil Teplitzky is a Managing Director with HPSquared LLC.  He has been a thought leader in the data management field for over three decades as a consultant, educator and corporate executive.  An active member in many industry associations Phil is a trusted adviser to many executives with overall responsiblity for enabling quality data management

“All NDAs are lifted” were the magic words uttered by Steve Miranda from Oracle at the Fusion Inner Circle Event at Oracle OpenWorld on October 15^th.

Just to make sure, I asked Steve explicitly during the Q&A section of the program if it was okay under the non-disclosure agreement we had all signed to write about Fusion on my blog, and he said “Yes.”

Hub Designs was invited back in February to help Oracle’s Fusion MDM team with some design review, validation, and testing activities. In return for our assistance, we’ve gotten to see Fusion MDM inside and out, and we can proudly say that we are one of the very few trusted partners who helped Oracle to design and develop the application.

We participated in a lot of conference calls with Haidong Song, Oracle’s Product Strategy Director for Customer MDM, and other members of his team. And we attended a week-long “hands-on validation” event at Oracle headquarters in August, looking specifically at the customer data management aspects of the Fusion MDM hub.

My first impressions of Fusion MDM during that hands-on session were very favorable. I remember thinking to myself, “Oracle could almost start selling this into the MDM hub market right now!”

Of course, Fusion isn’t scheduled to ship until sometime in 2010, and there’s still plenty of work to be done between now and then. But the core functionality needed for master data management was there, and the Oracle Fusion MDM team had a room full of customers and partners banging on it for a week without any significant crashes or issues.

There was plenty to like in Fusion that didn’t relate specifically to master data management – the new and improved user interface, the embedded analytics, the modern, standards-based architecture, the usability research that Oracle has done, the improved business processes, the built-in collaboration capabilities …

But the fundamentals of MDM were strong as well. Haidong and his team demonstrated how to import and consolidate customer data from outside sources, and we did our first hands-on lab session bringing in a small customer data load from a desktop file, such as a list of trade show leads.

We also tested a larger volume of customer data being brought into Fusion MDM through the Bulk Import process.

We did another exercise simulating how a typical customer data steward would identify potential duplicate customers, and then resolve those duplicates by merging the duplicate parties.

We also got a good look at the Informatica components that Oracle is bundling into Fusion on an OEM basis: the former Identity Systems matching engine and the former Address Doctor address cleansing tool. Previous Oracle MDM products like Customer Data Hub have had loose integration with Trillium and Firstlogic for address cleansing, but it’s refreshing to see Oracle investing in deep integration with industry leading solutions.

I think there are going to be a lot of Oracle customers who will move to Fusion MDM as the first wave of their overall migration to Fusion, who will see Fusion MDM as a good way to get some early experience with the Fusion applications family, before committing their mission critical Enterprise Resource Planning (ERP) applications to the Fusion platform.

And in 2010 and beyond, I think will be a lot of potential customers who evaluate Fusion MDM positively on its own merits against competitive MDM hubs. Oracle brings a robust data model, open architecture, and a next-generation approach to master data management, with state-of-the-art matching, data quality, middleware, and business process management.

Please let me know by commenting here what your thoughts and expectations are for Oracle’s Fusion MDM hub.

I had a great time at the Oracle OpenWorld conference this year.

Oracle did a great job organizing the MDM track. There were a lot of great presentations, and a good balance of speakers between Oracle people, outside consultants and experts, and end users with success stories to share.

David Butler, Senior Director of MDM Marketing at Oracle, was kind enough to convert my presentation titled “Best Practices in Master Data Management and Data Governance” to PDF format and to post it on the Oracle.com MDM web page.

You can find it in the ‘Partners’ portlet on the right hand side of the page, or just click here.

The excellent IAIDQ World Quality Day webinar looked at what the Data Quality landscape might be like in 5 years time, in 2014.  This got me thinking.  Dylan Jones excellent article on The perils of procrastination made me think some more…

Plug and Play Data

I believe that we data quality professionals need a paradigm shift in the way we think about data.  We need to make “Get data right first time” and  ”Data Quality By Design” such no brainers that procrastination is not an option.   We need to promote a vision of the future in which all data is reusable and interchangeable – a world of “Plug and Play Data”.

Everybody, even senior business management, understand the concepts of “plug and play” and reusable play blocks.  For “plug and play” to succeed, interconnecting parts must be complete, fully moulded, and conform to clearly defined standards.  Hence “plug and play data” must be complete, fully populated, and conform to clearly defined standards (business rules).

How can organisations “get it right first time” and create “plug and play data”?
It is now relatively simple to invoke cloud based verification from any part of a system through which data enters.

For example, when opening a new “Student” bank account, cloud based verification might prompt the bank assistant with a message like “Mr. Jones’ date of birth suggests he is 48 years old.  Is his date of birth correct?  Is a “Student Account” appropriate for Mr. Jones”?

In conclusion:

We Data Quality Professionals need to educate both Business and IT on the need for, and the benefits of “plug and play data”.   We need to explain to senior management that data is no longer needed or used by only one application.  We need to explain that even tactical solutions within Lines of Business need to consider Enterprise demands for data such as:

1. Data feed into regulatory systems (e.g Anti Money Laundering, BASEL II, Solvency II)
2. Access from or data feed into CRM system
3. Access from or data feed into Business Intelligence system
4. Ad hoc provision of data to satisfy regulatory requests
5. Increasingly – feeds to and from other organisations in the supply chain
6. Ultimate replacement of application with newer generation system

We must educate the business on the increasingly dynamic information requirements of the Enterprise – which can only be satisfied by getting data “right first time” and by creating “plug and play data” that can be easily reused and interconnected.

What do you think?

Seattle plumber Evan Conklin is very angry at HelpHive. He made his feelings plain in a comment on one of my posts about the company. Now the Seattle blog TechFlash posts that part of Conklin’s anger is about call tracking:

That was until last Friday when Conklin stumbled upon HelpHive, a Seattle online directory of local service providers. Conklin couldn’t believe what he saw. His business listing on HelpHive included a phone number, but it wasn’t the one he’d used for the past 30 years. It was a new number generated and controlled by HelpHive, a proxy number of sorts that the Internet upstart had set up to track calls it was passing on to the plumber.

Conklin was appalled with the idea that a third-party Web site could create a new phone number for his business, thinking that it was simply a way to get between him and his customer and to eventually start charging him for leads. “They have no right to do that,” said Conklin.”These guys are like vampires, sitting behind laptops siphoning off business.” 

The questions this dispute raises (again) are:

• Who owns SMB data? (facts are public in the US and cannot be copyrighted)
• Can companies swap in things like call tracking numbers (very widespread) or other similar lead capture devices without the consent of the business itself? 

We’ll see if Conklin goes so far as to file a suit against HelpHive. If he were to, does anyone know of cases in this area that may have resolved the question of whether call-tracking or similar devices can be used without an SMB’s prior consent?

Cliente: Instituto EcoFuturo

Ano: 2009

Palavras-Chave: Data Quality, Business Intelligence, Marketing Direto

Situação: O Instituto EcoFuturo possuía uma série de planilhas contendo informações espalhadas e em vários layouts sobre escolas de todo Brasil, corpo diretivo e docente de cada escola e nomes de alunos que participaram de seis edições anuais de um campeonato de redação com projeção internacional. Toda nova campanha de marketing direcionada a esse público exigia um dispendioso trabalho de coleta de informações e organização, e mesmo assim ocorriam muitas devoluções de correspondências do correio em razão de dados desatualizados. As quantidades para produção de peças de marketing sempre foram estimadas por não existir um relatório correto e consolidado da situação do banco de dados.

Necessidade: A principal necessidade do Instituto EcoFuturo era ter num primeiro momento uma base de dados única e higienizada (sem duplicidades com informações padronizadas), cujo acesso fosse feito através de uma ferramenta com amplas e flexíveis visões das informações das escolas, contatos (diretores e professores) e alunos que participaram dos campeonatos. Esse banco de dados deveria permitir a manutenção dos dados existentes, exclusão e inclusão de novos registros, sem perder as características de um banco de dados único e higienizado.

Solução: O projeto consistiu em uma análise da situação atual, desenho de um modelo de banco de dados aderente às regras de negócio desejadas, idealização de estratégia para realização de datacleansing e consolidação dos dados (tratamento de banco de dados através de ferramenta específica), elaboração de relatórios capazes de demonstrar a situação antes e depois do datacleansing, importação dos dados tratados para a base única modelada, desenvolvimento e publicação de visões em um sistema de Business Intelligence e apresentação do sistema sugerido para administração dos dados tratados.

This post is one of a series dealing with common Enterprise Wide Data Governance Issues.  Assess the status of this issue in your Enterprise by clicking here:  Data Governance Issue Assessment Process

I'm right, he's wrong!

Different Business Units sometimes use different business rules to perform the same task.

Withing retail banking for example, Business Unit A might use “Account Type” to distinguish personal accounts from business accounts, while Business Unit B might use “Account Fee Rate”.

Impact(s) can include:

1. Undercharging of Business Accounts mistakenly identified as Personal Accounts, resulting in loss of revenue.
2. Overcharging of Personal Accounts mistakenly identified as Business Accounts, which could lead to a fine or other sanctions from the Financial Regulator.
3. Anti Money Laundering (AML) system generates false alerts on Business Accounts mistakenly identified as Personal Accounts.
4. AML system fails to generate alert on suspicious activity (e.g. large cash lodgements) on a personal account misidentified as a Business Account, which could lead to a regulatory fine.
5. Projects dependent on existing data (e.g. AML, CRM, BI) discover that the business rules they require are inconsistent.

Solution:
Agree and implement the following Policy:  (in addition to the policies listed for Data Governance Issue #10)

• Responsibility for resolving cross business unit business rule discrepancies lies with the Enterprise Data Architect.

For further details on Business rules – see Business Rules Case Study.

Your experience:
Have you faced a situation in which different business units use different business rules?   Please share your experience by posting a comment – Thank you – Ken.

Southwark Council has launched a £700,000 legal battle against IBM in a row over a new computer system.

The council said it has lost confidence in IBMs credibility and integrity, and does not believe it will ever be able to have a workable business relationship in future.

Now the council is demanding damages of £717,061 from IBM after branding the new system defective. In 2005, Southwark wanted to install a new computer system allowing different departments to share information through a central repository of key data, called a master data management system, according to a High Court writ.

The council considered a rival system before going ahead with a contract with IBM, running to 19 pages, in which it agreed to pay £259,112 for the IBM Websphere software licence.   The council also agreed to pay £99,124 for skilled staff to implement the system, and £67,000 for the final piece of the system, Orchards ArcIndex, as well as consultancy services and support and maintenance, it is alleged.

But a preliminary review of the system in July 2007 found problems, including a serious deficiency with a vital system component, Orchard ArcIndex, the writ says.  There were also problems with the user interface, messaging integration, matching strategy, and a lack of reporting capabilities, the writ claims.

The council brands the system unfit for purpose and of unsatisfactory quality, and accuses IBM of failing to use reasonable skill and care when designing and testing the system. IBM also negligently claimed during negotiations that the system would meet the councils requirements, but these claims were false, and the system could not deliver its requirements, the writ claims. IBM failed to research solutions after a workshop to resolve problems, the writ says.

The council argues that although it tried to mitigate its loss by trying to find a solution with IBM that would make the system work, IBM tried to absolve itself from any responsibility for the software even though the contract for ArcIndex was an essential part of the system, it is alleged.   The writ was issued by Deborah Collins of the councils legal department.  Computing has requested a comment from IBM.

Contract – Business Objects BOXI 3.1 – Data Services/Data Quality ETL Developer – SC Cleared – Excellent rates available

I have an urgent requirement for a Business Objects Data Quality/Data Services ETL Consultant for a 6 month contract based in London. The ideal candidate should have experience with Business Objects Data Quality this tool is bundled with Data Services the BOXI 3.1 edition of Business objects Data Integrator (BODI). The candidate will be purely working with the ETL tool so experience with Universe Design or Report Development in unnecessary.

The ideal candidate should hold current CTC, SC or DV Clearance (Home Office, Police or MOD) however I may consider candidates willing to go through this process.

Excellent rates Available

Skills must have:
Business Objects XIR3 Data Quality experience

Other skills that would benefit:
Oracle 10g/11g or SQL 2005

Please send CV’s ASAP and I will contact you with more details.
Regards
Adam Sparey
Senior Consultant
adamsparey@peoplesource.co.uk
0117 922 7000