In the Sept/Oct 2009 Crosstalk issue, Dr. Lui Sha writes about complex cyber-physical systems with mixed criticality–such as defense systems, avionics systems, and medical devices–that need to be resilient against faults, failures, and hazards that are under software control. In his article, Sha reviews some architectural patterns for building resilient systems and points out that patterns are most often captured in architectural models. Sha also asserts that patterns must be adapted for “new application requirements.” In order to provide computer-aided verification of the adaptation of those patterns, formal verification in software modeling languages is recommended. He includes an example of a medical system modeled using the SAE International Architecture Analysis and Design Language (AADL)

Certification programs for software architects abound. How can you tell which programs best measure an architect’s capabilities? Or which ones are right for you? SATURN 2010 plans a track on architecture certification to pull together details on certification programs.

Last month I attended the Joint Working IEEE/IFIP Conference on Software Architecture and European Conference on Software Architecture (WICSA/ECSA 2009).

This was the eighth WICSA offering in a series of conferences beginning in 1999.  A good summary of the journey the work in architecture has taken over the years is contained in a paper by Paul Clements and Mary Shaw on The Golden Age of Software Architecture. Paul initially put forth these ideas during a panel at WICSA 2005 that marked the 10 year anniversary of the first International Software Architecture Workshop (ISAW-1) and the 10th anniversaries of the IEEE TSE and IEEE Software special issues on software architectures.  Another good summary can be found in a paper that came out at the same time on The Past, Present, and Future for Software Architecture by members of the IFIP Working Group on Software Architecture, who also support the WICSA conference.

I have been involved with WICSA since its inception.  It is one of my favorites due to its unique nature as a working conference, the mix of research and practice, and the sense of community among the 100 plus attendees.  At WICSA this year, the emphasis was on the relationship between software architecture and “higher” level concerns such as systems of systems, enterprise architecture, and value of architecture.

The program reflected this emphasis in its series of keynotes.  Jaap Schekkerman kicked off the conference with a keynote on Why Business & IT Professionals do not understand each other and how Enterprise Architects can Bridge that Gap. Schekkerman started with the premise that the human brain has two different thinking styles (right brain/left brain).  He expanded on the idea to include the thinking styles of software engineers/architects and business people and reflected on how enterprise architects can bridge the gap between both groups. I thought this was an interesting way to approach the problem and complemented the keynote that Eoin Woods gave at SATURN 2008 on Putting Software in its Place: Classifying the Architectural Species where he approached the challenge of bridging from the roles of the different types of architects.

Eoin Woods gave a keynote of Using Design Principles to Unify Architecture and Design.  He focused his remarks on the question of how can we avoid conflict and allow enterprise architects, software architects and development teams to work together cooperatively.  He explained how the use of design principles are a unifying concept that help these different groups to work together.

Alex Wolf asked the question, Where is software architecture? in his keynote that concluded the conference.  This included questions of the sort, What is the current state of ideas in software architecture? What is the right direction to look to find promising new ideas? And, if software architecture is more than just an intellectual exercise, then where, quite literally, is it to be found in modern software systems? He offered three areas for exploration: (1) moving from the notion of a system having a single architecture to having multiple, simultaneous, intersecting, interacting, and maybe even conflicting architectures; (2) shifting the emphasis on architecture as components and connectors to understanding emergent system behaviors and better utilizing techniques such as simulation (based on models) and experimentation (based on implementation); and (3) understanding the incentives to document and discovering architecture in the ways engineers already document using deployment descriptors, manifests, configuration files, and service-level agreements, and so on.

More details of the conference and discussion on the working session topics are available at the conference wiki.

- Robert Nord, SEI

In this tough economical situation, leading a multi-million, multi-year project to a failure is not an option for any corporation.  Leaders or managers who lead project for years without successful implementation must be watched closely by the executive management for their performance during this economical situation. I totally understand an organization can learn from their mistakes, but successful organization learns from others mistakes but not on their own cost.

The project managers and leaders are seeking the best possible ways to ensure successful implementation in spite of any unforeseen hindrance in the execution. Even for the multi-million dollar projects utilizing the cost effectively solution are demanded. The project managers and leaders are seeking the best enterprise or system architect to provide solution to exceed customer expectation, meet requirements and at the same time provide most innovative and cost effective solution.  Enterprise architects or system architects are required to have a set of tools in their arsenal to provide a rapid cost effective solution with highest possible success rate.

Rapid Solution Delivery Framework is an abstract concept used to rapidly define a cost effective and innovative solution. The key components of the framework are:

1. Cloud computing
2. Open source
3. Social network tools

Cloud computing: It addresses the rapid infrastructure setup or applicaton setup or business rules for any solution delivery. The cloud platform may come from external public cloud provider like Amazon or internal cloud provider or external private cloud provider (like secure24, rackspace for infrastructure) . It is rapid way of setting up the infrastructure for the solution delivery. There may be cases where cloud platform may also provide business solutions.

Open Source: There are lots of commercial software product companies leveraging the open source or supporting or sponsoring the open source to improve their commercial product quality tremendously. For instance, IBM supports Apache product development and leverages the Apache web server and bundles it as a IBM HTTP Server (I H S). The open source concepts are not pertained to system software. The open source concept proliferated to various segments and even in the chemical bonding structural analysis there are lots of open source tools available where the researchers and students easily learn and collaborate. Open source has become an integral of any IT solution delivery model.  Open source provides wide range of solutions from operating system, system software (webserver, database server, application server), application development frameworks (spring, hibernate, struts), security (OpenSSO, Crowd, OpenLDAP) , application development IDEs (like eclipse, Galileo ) and etc. These open source products not only provide quality solutions but also provide cost effective solutions.

Social network tools: Social networking concepts and its implementation matured over the years and it is extensively being used beyond for personal use. There are major corporations like Cisco, Deloitte, IBM uses the social network tools for better collaboration among the employees to provide effective solutions to their clients. The tools landscape in the social network media is vast and deserves a separate blog. To name a few major categories like Video, Video Aggregation, documents, events, Wiki, LiveCasting, Pictures, social bookmarks, Crowdsourced content, blog and etc. There are at least 3-4 popular tools in each of those categories. The social networking technology landscape plays a vital role in leveraging existing solutions, collaborate better among various stakeholder of the projects.

Vivek Kundra implemented few projects in DC state government as a CTO are few practical examples how social networking platform provides a foundation for a rapid cost effective solution.

Along with these key components, depends on the business requirements, there may be other commerical products like SAP, or Flex (for UI) will be part of the over all IT solution.

Cloud computing concept continue to receive acceptance and its adaption increases exponentially.

I heard today from one of vendor representative that he grew up in the farm and growing up in the farm means he had to learn whatever required in no time to do farming. He confirmed my thoughts on how farm boys grow up in the farm. Cloud computing basically took the farm boy approach. Do whatever is required to do the job!

Fifteen years ago, the PC support team use to install the operating system (windows 3.11,MS-DOS ) in each and every PCs and PC support cost use to be a significant cost in the IT shop. As repetitive mundane tasks mandates  innovation, solution like Norton Ghost solution were emerged and hence deployment made easy and drove the PC support cost down.

Cloud computing took the Norton Ghost solution to the server environment.  Like me, I can imagine most of the IT members spent significant time in  building the development, testing, staging and production environment for each application and spent  time to ensure that all the environment were build alike and still had environment behaved differently for unknown reasons. Cloud computing solves all theses problems.

Key components of cloud computing:

Utilizing the external storage in the server environment has been in use for quite a while. In some cases, like manufacturing production databases, selecting the apt external storage platform including hardware, storage operating system, connectivity and raid types plays a vital role in the system architecture.  Likewise, in the cloud computing, the storage and processing unit are physically separated and logically connected as and when required. It includes the boot volume too.  To manage both processing units and storage units, an external operating system called cloud computing operating system is used. There are vendors like vmware sells cloud computing operating system like vmware vSphere and there are open source cloud computing operating system like eyeOS

• Processing Units
• Storage units
• Cloud computing OS
• Network units

Architecture of cloud computing:

For external users

Have a OC3 or OC12 or OC48 internet link based on your requirement to connect to the internet for your data center. Have a cisco 7200 series and cisco ACE series for your first and second layers of your network. Have netcahce or ssl accelerator or intrusion prevention devices based on your requirements. Connect the internal vlan with blade server (HP) and storage units like EMC. Slice the EMC luns to have a preload images like Redhat, Windows with preloaded system software like application server, web server, domain name server, database server, director server and etc. Install the cloud computing operating system in the external boxes and link both processing and storage units. The cloud computing operating system manages both processing and storage units.

If there are requirements to connect stand alone servers, it can be connected in the internal vlan.

The architecture is straight forward and purposefully I’m not drawing it.

For internal users:

The same architecture expect OC3 internet link.

Cloud Computing Future:

The concept is very cost effective and efficient. Instead of building images for production servers with application server, web server, database server and etc.. the image will be elevated to more business unit image. Loan processing image, collection and customer service image, delinquency image, skip tracing image,  bankruptcy, treasury, securitization, point of sale and etc.  The technology cloud is in the phase of reaching maturity and business cloud is emerging. There is a huge potential for the players who get quickly into the business cloud.

Think about for a minute, if you are bank, why do you want to spend time on building IT infrastructure to process loan. Banks JUST want to focus on getting good papers, good customers and not worry about how the assets are stored, retrieved and processed. Whoever get into business cloud will lead that market segment for a while!!

Back in the day you had to actually know how a computer worked to program the thing; today… not so much. That’s not meant negatively. It’s just that we’re about half way between punch cards and simply telling a computer what to do like on Star Trek. I mention this because I’ve encountered a lot of “senior that still use the terms tier and layer interchangeably when referring to system architecture. The difference is significant yet simple to distinguish.

A tier is the “physical” separation of code, where a layer refers to the logical segregation of functionality. Physical separation of tiers does not necessarily mean separate servers.

Take an architecture that segregates functionality into User Interface (UI), Business Logic (BL) and Data Access (DAL).

In a small 3 Layer system, one could place logic for each layer in separate methods within the same DLL. For that matter the same class.

In a 3-tier system code for each of these types of logic would be placed in separate DLL’s. They could even be placed on separate servers.

If you have you UI and BL in one project (i.e. DLL) and your DAL in a separate project all running on the same machine you’ve got yourself a 3-layer, 2-tier architecture. I use this example because it is a very common real world scenario today.

That’s all there is to distinguishing between a tier and a layer.

This is significant because while using layers is almost always a good idea, using tiers can be quite costly. It is generally around 1000 times slower to make a call across a process boundary on the same machine than to make the same call within the same process. Things get exponentially worse if it’s on a separate machine, even on the same network.

Additionally, the phrase too much of a good thing applies with regard to tiers because of the impact on the complexity of the application and infrastructure required. Tiers should be considered an evil necessary to accommodate some levels of extensibility, security and fault tolerance. However, anyone that says you should always use 3-tier architecture should be placed in timeout with the other silly children.

Inversely, you can almost never go wrong implementing distinct layers. Don’t get carried away with it, but well thought out and separated layers simplify maintenance. Not to mention keeping open the option of breaking out into tiers should the need ever arise.

As with any decision regarding architecture one must weigh all of the benefits against the true long term costs when deciding what layers and tiers are right for their unique situation.

Q. As part of writing your academic essay Part B of assignment 2 you will need to draw on some literature – textbooks, journal articles and websites that define traditional and object-oriented approaches to information systems development and describe their similarities and differences. Any textbooks, articles and websites that provide information cited in your essay should be included in the reference list of your essay. This week you should begin an annotated bibliography in your post to the blog for this week. The resources page for assignment 3 contains a description of what an annotated bibliography is and what it should contain. It is expected that you will list two or three textbooks, articles and/or websites. For each textbook, article or website you should write a paragraph or two briefly summarising it and explaining how it will support the arguments that you are making in your academic essay. Your annotated bibliography must be started by the end of this week. It does not need to be completed this week. I suggest that you add information to your annotated bibliography as you add references to the reference list of your essay.

Answer

Hatley , Derek J. and Pirbhai, Imtiaz A., Strategies for Real-Time System Specification, Dorset House Publishing, 1988.

It has done good description on the structured and object-oriented approach. Firstly, a nice advantages and disadvantages of both approach is given making the concept clearer.  Then, there is process for combining both approach and is stated that performing both approach provides a better design. Then the difference and similarities are shown. It is stated that the OO approach and structured approach is fundamentally different in that how the control is handled. It is stated that a good structure design has a single “smart” controller whereas in the OO design, the control is partitioned into each object. The author emphasize that we need to be able to blend these design techniques to address the challenges of designing systems today.

Sircar , S., Nerur, S.P., and Mahapatra, R. (2001), Revolution or Evolution? A comparision of Object –Oriented Structured Systems Development Methods, MIS Quarterly, Vol. 25, No. 4, pp. 457-471.

The information is given about how the systems development life cycle (SDLC) can be defined as a process of understanding how an information system can support business needs or requirements of an organization.  The dilemma between the two approaches is shown and clears the common misunderstanding that the OO approach is not the latest methodology of SAD. It describes as while the structured approach provides clean-cut steps from the beginning to the end of SDLC, the object –oriented has no clean-cut steps in moving from analysis to design. The iterative and incremental approach of OO approach refers to continuous testing and refinement of the system throughout the project lifecycle.

Shelly, G, Cashman, T and Rosenblatt, H 2008, System Analysis and Design, 7th edn, Thomson Course Technology, Boston

It is described here that structured and object-oriented analysis both are dependent upon data, processes and models and both follow approaches. The System Development Life Cycle (SDLC) approach that the structured analysis uses for the process development is also followed by the Object-oriented analysis. Both approaches are not the best. The traditional analysis treats process and data as separate components whereas object-oriented analysis combines data and the processes into things called objects. The traditional analysis is “process-centered” whereas the object-oriented analysis is “object-centered”. The traditional analysis is based on detail plan so it is called predictive approach so it divides the processes into series of phases called the Waterfall model (SDLC) whereas the object-oriented phases tend to be less sequential and more interactive and represents an interactive model.

Facilitator: Paul Clements, SEI

Panelists: Mark W. Maier, Rebecca Wirfs-Brock, John A. Zachman, and Randy Case (Raytheon)

Panel: Exploring Boundaries Among Enterprise, Systems, and Software Architecture

Wednesday, May 6, 2009, 1:00 pm

View Panel abstract >

Panel explored the boundaries between genres of architectures. What do genres have in common, are there commonalities we can leverage, and what are the differences between the genres. Each panelist gave brief remarks, which were followed by questions from the audience and discussion among the panelists.

Zachman: This is a language problem. We have a lot of words we use, but not enough semantic clarity. For example, “enterprise architecture” could mean a lot of things. Without a framework or ontology, use of the word “enterprise” isn’t definitive and can be interpreted in many different ways.

Similarly with software and systems architecture, without a crisp way to allocate responsibilities and roles, we have difficulty communicating with each other. Need a way to objectively assign responsibility and allocate responsibilities.

Wirfs-Brock: Thinking about responsibilities and interactions between them. How to communicate between these models. For one client, I taught a class to mixed enterprise and application architects. Enterprise architects want to make sure governance models are relevant to application architects. But lining up their principles and values is challenging. We were able to make linkages between enterprise and application perspectives. They operate on different timescales.  As a designer in software architecture, I feel comfortable going to people who model business processes, but my colleagues don’t. They don’t see it as real work. They see enterprise guys as looking out too much into the future. Need collaborative projects. Both sides need to get rewarded for this kind of collaboration.

Case: If you’re not doing architecture at many levels, it’s not going to work. John talked about architecture. Mark talks about architecting. We talk about the architect, who has to think at a lot of different levels. This is a framework for the architect—different from what John portrayed. This framework portrays a continuum of enterprise, systems, products, components. Standards are relevant at different levels of the hierarchy.

• Enterprise
• System
• Products (SW, HW)

How to deal with standards at different levels? We can talk about attributes and boundaries: at enterprise level, you’re talking about objectives, whereas at product level, you’re talking about implementation. Scope of architecture is broader at the enterprise level, and hence hard to bound. But at implementation level, if scope isn’t narrowed, I don’t know what to build.Tools are a cottage industry for architecture. I have to span lots of tools and standards. Use of tools changes based on underlying point of view.

Q: What should an architect do with these tools, because they don’t play together?

A: The question is, Who’s the audience for the tool? I have to pick the tool for the audience I’m talking to. Primitives underlie what an architect has to do.

Maier: Genres. Genre of objects: What do we architect? Alternative is a genre of role: buyer, owner, builder, user. The notion of genre is broader. As a system architect, I’m interested in building well-scoped collections of HW and SW, acting as buyer’s agent, not a builder’s agent. Define mission, someone else builds system in response to that. Linkage to strategy – architecture as the technical embodiment of strategy. “Architecture is decisions, not documents.” It reflects basic decisions about the fundamental structure of the thing, regardless of documentation.

Relationship to other genres: systems architecting exists in the context of a human enterprise. We derive architecture out from the basic direction of strategy. Bad and unfit strategy leads to bad and unfit systems. Buyer/user architect vs. builder architect: it matters which side you sit on.

In general, we pay too much attention to documentation and not enough to decisions. Poor documentation leads to many ills, but not necessarily bad decisions.

Some systems have emerged without documentation or architecting, e.g., the dabbawala system.

To evaluate fitness for purpose, we have to understand purpose. Too much focus on documentation leads to three-ring-binder architecture, without much decision quality. Better to focus on who makes decisions and how.

Hierarchy  models = systems engineering catechism. It’s both true and false. Abandon notion of strict hierarchies. Hierarchy isn’t only way to look at a system. Think about it as a mission thread. What must occur to solve particular problem?

Zachman: Have to deal with things one variable at a time. I don’t like hierarchies. It’s a good implementation model, but not a framework. The world isn’t a hierarchy. Not good for engineering or normalizing things. OK for implementation only.

Q: Architecture is a communication problem. These levels have different cultures. Tools have semantic representations and assumptions built in.

Zachman: Tools respond to their view of the marketplace. The marketplace drives the tool community at short end of spectrum. They help us do implementation. Architecture work is engineering, not manufacturing. A lot of problems today are a result of a lack of architecture. We need engineering. There’s been a dearth of this.

Wirfs-Brock: We learn from implementation. Building architects did things from experience. Challenge is that we need to work at multiple levels at same time. Toolmakers need to make different representations of same concept for different purposes. The focus on execution and implementation as a designer of systems is constraining. I have to think down too early. There are multiple purposes for representation.

Maier: Cynical about grand unified processing tools. Rather have rich set of tools and make things up as we go along. One grand, glorious tool is not a likely solution.

Wirfs-Brock: We need diversity of tools.

Case: Architect must address multiple stakeholders and consumers.

Maier: The most useful tools are program-specific.

Wirfs-Brock: Why not just do things the way we do them now?

Case: We’re translating a lot between tools. Have to keep architecture books twice. Problem is that it doubles effort with no appreciable value for that effort.

Maier: Constraints or opportunities from software may influence the system. Being able to carry constraints and opportunities doesn’t depend on tool A or tool B. Need understanding of overall quality attributes of the system.

Yesterday, I attended the tutorial on Pattern-Oriented Software Architecture: A Pattern Language for Distributed Computing by Doug Schmidt. While explaining a pattern sequence, he used an analogy of ordering hamburgers to explain two different ways of communicating in distributed systems. At McDonald’s, when a customer orders, the employee enters the request at the register, reaches back to the “cache” of precooked hamburgers kept warm under a heat lamp, and delivers it to the customer; the interaction is an example of “synchronous” communication. At Wendy’s, when a customer orders, the employee enters the request at the register for a hamburger cooked to order, gives the customer a token and continues to serve the next customer in line; the interaction is an example of “asynchronous” communication. After a certain amount of time the customer might go back to the counter and ask if the order is ready (“polling”) or wait at the table for the order to be brought out when it is ready (“published”).

During lunch I mentioned the use of analogies during the tutorial. Another person at the table said she was attending the Software Architecture: Principles and Practices course. She liked the analogy used by the instructor of relating documenting software architecture views to a doctor ordering diagnostic tests to see different views of the patient in terms of the different structures of the human body.

We were both intrigued by the use of analogy and are interested in hearing people’s experiences using analogies to communicate architecture concepts.

What analogies have you used or heard to explain architecture concepts? Did you think it was helpful? Or, did you think it was inappropriate (e.g., misleading or oversimplifying)?

– Robert Nord, SEI

As we kick-off the fifth annual SATURN conference, we’re also excited to announce a new effort to engage architecture experts and practitioners: the SATURN Network Blog.

The SEI Architecture Technology User Network (SATURN) is a professional network of software, systems, and enterprise architects from around the world, representing industry, academia, and government.  This blog will be dedicated to architecture-centric engineering topics that are imperative to our field.

The theme for SATURN 2009, “architecture at all scales,” encompasses furthering our understanding of enterprise, systems and software architectures including what they are, how they relate to and influence each other, and which architectural best practices they involve.

To address architecture at all scales, we are honored to have world-renowned keynote speakers John A. Zachman and Rebecca Wirfs-Brock, representatives and speakers from more than 60 organizations, and a distinguished panel of experts in a wide range of architecture genres participating in the 2 SEI courses, 5 tutorials, 21 presentations, and several birds-of-a-feather sessions being offered this year.

Since last year’s conference, we have sponsored several events and established a number of ways for the architecture community to stay better connected including webinars, a SATURN LinkedIn group, and now this blog.

Please check back often over the next two days for a look at what’s happening at SATURN 2009 – and also after the conference to continue sharing and learning about architecture-centric engineering.

SATURN Co-Chairs

Ipek Ozkaya and Rob Wojcik

The Issue

Isaac Newton had it right: “…systems have a tendency to increase their entropy [over time].”

Newton’s second law of thermodynamics basically states that, without the application of energy, all things move toward a state of chaos.  The same can be said of IT.  Unchecked, technology systems move toward chaos.

Solutions are added to solutions.  Over time, data must be shared between systems to reduce duplicate data entry and lower errors…so more systems are added to “orchestrate” the flow of data.  Users begin to experience the pain of having to access multiple disconnected systems, so we throw a portal over top to knit the systems together (at least superficially) and reduce the number of paths to access systems and hide all those duplicated logins.

Hopefully Thought-Provoking Questions

• Is the use of tools for data orchestration and portal services actually reducing the overall complexity of IT?
• Are we actually solving the problem, or are we using these solutions like band-aids?

My Two Cents’ Worth…

Don’t get me wrong – these solutions have their use.  Data orchestration and portals, when properly implemented into an environment where the underlying architecture is reasonably well understood (and in the best case scenario, documented) can be extremely useful tools.  However, observation and experience with these solutons tells me that they’re being implemented before the “architecture spaghetti” is untangled.

Instead of simplifying IT, they add to the complexity, and worse still, to the end user (including the folks to set the budget), it appears that you have solved the problem.  (A gold star for you…for now.)

The good news is that these solutions tend to be loosely coupled – meaning that you can eventually change the systems that are using the data orchestration and portal services without the entire thing coming to a grinding halt.  To an extent, these systems can help reduce the fragility of your architecture.  However, how are you going to get the executives to loosen the purse strings and fund your systems architecture project when they’re no longer experiencing the obvious pain associated with complexity?

That quick fix has just come back to bite you in the…

I can’t count the number of times I’ve heard IT staff say that one of the biggest problems associated with getting projects funded and scheduled appropriately is the fact that the underlying complexity of these systems is hidden from the view of users.  You’re not helping yourself every time you implement a “simple” quick fix.

In fifteen years, the single best piece of advice I’ve ever received is, “you are NOT going to be a hero – get over it.”  Real solutions take time – and we have to stop acting like these solutions are simple, especially around senior executives.  The gold star that you receive today for being a hero and solving a problem in the short term tarnishes pretty fast when the long term problem of poorly-understood architecture rears its big, ugly head.

In my next post, we’ll discuss ways of looking at underlying causes of complexity and starting the process of getting senior executives on side to help you address the issue.

I was looking to establish why data quality problems arise. My experience was that data quality issues arise due to factors such as the following:

• Human Errors – A major source of incorrect data in systems. A large portion of these errors can be attributed to a system’s inability to validate data, but some errors are logical in nature. For instance, if a telephone operator is an inputter (s)he may mishear a respondent’s town and input a town that may be valid but can be wrong from a business perspective.
• Deliberate Manipulations – A user may be forced by the legacy system to enter data that is prima-facie incorrect but is inevitable because the legacy system would reject the data otherwise.
• Conformance to a target system model definition – This can be a factor when a legacy system data conversion and migration is taking place. The target system model may dictate the data to be in a certain format, and the need for conversion and migration may make it compulsive for changes to be made to the data.

but wanted to triangulate my experience with the thoughts of experts within industry sectors as disparate as investment management, call centre management and market research. I asked sixteen experts what they thought the main causes of data quality problems were. Semi-structured interviews were conducted with each participant.

English (1999) suggests the main data quality problems arise from poor data architecture, inconsistently defined departmental data, inability to relate data from different data sources, missing and inaccurate data values, inconsistent use of data fields, unacceptable query performance (timeliness of information) and lack of business sponsor, and it is specific examples such as these I was looking to elicit from the answers, while acknowledging that over time, even in the best maintained systems, deficiencies in stored data will develop (Ballou & Tayi, 1989).

43.75% of respondents attributed major data quality issues to organisations having distributed systems (15.56% of responses). Often there exist disparate systems within an enterprise that use different business rules, and store data with different definitions and possibly conflicting values. Customer data exists in silos – there may be a marketing database, a transactions or operational database and a customer service database, all holding different information on or ‘views’ of the same customers. As Respondent 6 recognised that “database design recommendations say you should not store and update the same data in multiple places because it prevents consistency, but this is not always enforced, especially when systems have been unified in a chaotic way, or phased in.” Without a proper integration model to eliminate inconsistent definitions, formats and values, the organisation often finds it is unable to aggregate and harmonise new data as it is introduced from various sources.

37.5% of respondents cited a lack of naming conventions and standards causing a diminution in data quality (13.33% of responses). The symptoms of this are also discussed in the research paper Data Quality Issues In Marketing Databases.

A quarter of respondents suggested a failure of organisations to harness the maximum potential of technological advances to ensure erroneous or duplicated address data isn’t stored in the marketing database (8.89% of responses). Specifically, 12.5% respondents suggested a lack of address verification at point of entry was a significant contributor to poor data quality (4.44% of responses). Address verification modules are an increasingly common feature of systems, their presence allowing for instant address verification at point of entry, immediate identification of a potential duplicate entry and significantly reducing the number of keystrokes at point-of-entry. Where systems are used in countries where street-level post coding exists, such as the United Kingdom, these modules can facilitate the collecting of a full address simply by entering a house number and postcode. Even in a more reactive manner, software advances can aid in improving data quality through address hygiene software.

Poor stewardship of data is cited as a major contributory factor to poor data quality by 25% of respondents (8.89% of coded responses). Respondent 3 suggests that “databases full of junk data happen most often when there is no business owner of the data.” Also, failure to implement processes that properly capture the information needed by the enterprise can be seen as another prominent example of poor data stewardship that tangibly affects the enterprise’s marketing data quality, and this aspect was underlined by two of those respondents.

Input rules that are too restrictive and bypassed completely are a significant contributor to poor quality data according to a quarter of respondents (8.89% of responses for this question). This is a major cause of too little data being captured – “Imposing superfluous controls on data input and editing might cause important data to be lost entirely,” as Respondent 1 notes. It is also closely related to inaccuracy in data stored, or a lack of granularity in the data stored, as Respondent 8 noted – “data entry forms often have a single structure, causing users to force complex inputs into a one size fits all form.”

12.5% of respondents identified inappropriate access levels as having a detrimental effect on the quality of data in systems (4.44% of responses). This can take the form of easy access to data, which may conflict with good security practice or privacy requirements. Conversely, as Respondent 1 suggests, “if the balance is too great in favour of barriers to access… it will obviously have an impact on the quality of data held in the system.”

12.5% of respondents suggested an increasingly overwhelming amount of data (‘information overload’) is adversely affecting quality of data held in information systems, not least marketing databases and customer relationship management systems (4.44% of responses). As Respondent 6 noted, “In larger organisations there is often simply too much data, duplicated across the organisation.” This can make it “difficult to access required data in a reasonable time, negatively impacting on information systems quality. Customers on a telephone call, for instance, expect an operator to retrieve their account details in a reasonable amount of time, and may be put off using the services if it takes an eternity to retrieve a billing history,” as Respondent 1 points out.

Overburdened or time-strapped staff was cited as a key contributor to data quality issues by 12.5% of respondents (4.44% of responses). Respondent 9 commented that “organisational focus on speed and productivity alone is often a fairly significant cause of data quality problems. For example, something like a record store or bookshop that takes orders by phone might make it a priority to reduce time for each phone call to get greater volume of orders with fewer staff, but this will guarantee a lot of duplicate customer records and unverified addresses being captured. If pay incentives or bonuses are based on number of orders taken… this causes problems like this to happen.”

An inability to identify duplicates or instance of customer records already stored on system at the point of entry is a key contributor to poor data quality, specifically the issue of duplicate data, and this was raised as a key issue by 12.5% of respondents (4.44% of responses).

12.5% of respondents underlined their concerns regarding turnover of information (churn) and the impact this can have on an enterprise’s data and decision-making if they are unable to keep up with it (4.44% of responses). Respondent 16 cited figures from their previous organisation, Dun & Bradstreet, who suggest that in a single year, 20.7% of the business postal addresses in an enterprise’s data store will have changed (of new business,  the rate of change is higher, at 27.3%). Telephone numbers change at the rate of 18%. Company names are unstable, changing at the rate of 12.4% a year, making it difficult to sustain effective customer relationships if data isn’t constantly updated. Data quality inherently degrades at a rapid rate, and steps must be taken to ensure this is counteracted.

12.5% of respondents were concerned that many organisations treat information as a by-product rather than a product and as a result they often address data quality issues in a reactive manner (4.44% of responses). They assert that information is not a by-product but a direct product of processes that capture knowledge about the actors, locations and events discovered while conducting business transactions. A corollary of this approach is that there is often no plan for information lifecycle management. Information captured by an enterprise goes through various stages of significance. Initially, it may have to be available for several days then over time will become needed less frequently. Later, it may be needed in order for litigation or regulatory reasons. Therefore, as information has a lifecycle it must be managed appropriately, using a combination of products and services used to manage it based on criteria such as reducing storage costs, ensuring easy access when needed and speeding up business critical applications. Failure to manage it appropriately can have a degradative impact on data quality.

A related issue is that of limited resources or infrastructure. If there doesn’t exist the will or budget to computerise areas of the business, not all communications or transactions may be captured. That this was raised by only one respondent (forming 6.25% of respondents, 2.22% of responses) is perhaps indicative of the strides many enterprises have made in ensuring a great deal of business information is properly captured, facilitated in part by decreasing storage costs and effective off-the-shelf database management software.

One respondent suggested selective judgment in data production, resulting in biased or skewed information, can have an adverse effect on data quality, particularly where that is used for decisioning. Although this is more prevalent in softer systems, such as medical systems, it can still arise in marketing databases, particularly where descriptive text fields are used to store notes on suppliers or customers.

The cost of deciphering and inputting coded or jargonised information can have an adverse effect on the quality and retrievability of an enterprise’s data. Similarly, this issue will be most prevalent in systems such as those in specialised areas such as the medical profession. This was contributed by a single respondent.

Another data quality issue is caused by difficulties in representing non-numeric and non-textual information, for instances images. There is a large storage overhead for storage of such information, and although when stored in systems such data may be retrievable, systems have so far struggled with indexing, aggregating, manipulating such data and identifying trends. This issue was raised by one respondent.

One respondent cited changing data needs in an organisation as a primary source of data quality problems. Data consumers’ needs and the organisation environment are often in a state of flux, meaning systems will need to be redesigned often to maintain the quality and reliability of the data. If the organisation’s processes and systems fail to keep pace with the changing environment and needs of data consumers, the corollary can be lost data or analysis and decisioning based on incomplete or inaccurate information.

Lack of training for staff employed with entering data can have a profound effect on data quality. The accurate and efficient capture of data requires attention to detail and typing speed. Spelling, punctuation and numeracy skills of staff are also important if the enterprise is to accurately and efficiently handle information. However, students increasingly acquire word processing, spreadsheet, and database management skills using computer software in secondary schools, colleges, universities, meaning organisations can rely on a pool of workers who have already acquired many of the skills necessary to capture data accurately and efficiently. That only one respondent underlined this as a salient issue may reflect that it is no longer such a concern for the reasons explained above.

Lack of effective cross checks and validation beyond the point of entry to assure the validity and veracity of collected data was cited as a source of data quality issues by one respondent. Often entry-point validation will be implemented but this will not necessarily eliminate the need for further validation and cross checks against existing data sources, as the user interface is often bypassed by operators performing bulk inserts of records or mass updates to the database for valid business reasons. Errors that can occur in these instances also need to be identified and protected against or corrected in order to maintain data quality and integrity. Further applications that handle validation logic can be introduced (through triggers, or stored procedures that search for business rule violations or anomalies in expected trends), or, less reactively, it can be made a requirement that all transactional updates are run through the component to check validity.

References

Ballou, D. P., & Tayi, G. K. (1989). Methodology For Allocating Resources For Data Quality Enhancement. Communications Of The ACM, 32 (3).
English, L. (1999). Improving Data Warehouse And Business Information Quality: Methods For Reducing Costs And Increasing Profits. Indianapolis: John Wiley & Sons.

I had a dream . The dream was to map out all business processes in an organization, develop the run time from the process map, validate the function of the new run time and sun down all existing systems. It was a dream few years ago. I challenge myself, it is not a dream any more and we, as an industry,  are moving towards it. We may not be there today to make it real but we are in the path of it.

Information technology is slowing becoming business technology and technology will become a black box managed by set of providers. Business technology management team will manage only set of  providers but not technology in near future.

Phase #1 of the dream:

• Select a cost effective BPMS software
• Utilize internal & external resources to map out the processes using the BPMS
• Host it in the cost effective platform as a service (Cloud computing)
• Leverage the centralized business process map to simulate/emulate any new business ideas
• Validate the new business ideas for strong business case
• Develop the application from the process map and launch it in PaaS (Platform as a service) like Google apps.

 The first step of the phase #1 is to select the cost effective BPMS software. I do not need to amplify why it needs to be cost effective and economical option. Due to the global economical crisis, most of the organization is on the lights on mode, and minimum budget allocated for any discretionary spend for this year. Economical and cost effective alternatives are not differentiators in the competitive environment, it is a survival ingredient.

When I explored the cost effective BPMS software, Intalio, the open source BPMS is in top of the list. Intalio also integrates with Google Apps and salesforce.com.

The question is, what is the total cost of ownership for Intalio installation in an enterprise. More to come on the findings..

As the recent outage at Google demonstrates, including people in the design of a system is often overlooked in technology focused organizations. In that particular example, Google left the ability for a single user to take then entire system down by adding “/” to the blocked site list. Why would this ever need to happen?

More generally, IT organizations need to recognize the importance of people in our architectures. What good is a password if people will write it down? What good is a workflow if people are still printing and signing forms? What good is an ERP system if the numbers are still manipulated by hand?

Every project should include not only a technical understanding of what is being implemented, but also an understanding of the business processes and people that will operate in the system — both users and administrators.

Email misuse?

It seems as though all my recent blog posts are preoccupied with email. Perhaps, as a “solution architect”, I see that email misuse is an impediment to efficiency, productivity, and technology platform evolution. Reliance on email slows development and adoption of systems that are better suited for information delivery and processing. Email is the duct tape of system integration and monitoring/alerting infrastructure. When email is misused, system reporting is crippled and process analysis is next-to-impossible. Worse than that, the opportunity to build an efficient technology foundation, the benefits of which can be reaped over and over again, is missed as effort is directed towards an inferior architectural approach. I can put email misuse into 3 major categories:

1. Email as a monitoring solution
2. Email as a reporting solution
3. Email as a document management solution

I sort of covered the third one in my previous blog (Email Attachments are Evil), so I’ll focus on the first two in the sections below.

Email as a monitoring solution

Instead of a proper monitoring infrastructure, many systems rely on email as a vehicle for communicating alerts, status information, notifications, et cetera. It’s a “quick and dirty” way of putting something in place when a ‘real’ monitoring solution is missing or inadequate. It seems to be a very common scenario to write scripts that, as part of, their workflow, send an email to indicate the completion (or failure) of a particular step or execution milestone. What ends up happening, especially in a heterogeneous environment with many systems, is that you get an unmanageable collection of scripts (written by different authors) with point-to-point emails (addresses embedded in scripts) being sent to an unknown number of individuals or distribution lists for undocumented reasons. Visibility into issues is greatly reduced, ability to react quickly and appropriately is compromised, ability to report accurately to customers is also compromised (which becomes ever more important if you have any kind of an SLA to maintain). Data collection becomes a manual process – expensive, not scalable, and error-prone. A “master” view into the state of the systems is near impossible.

It’s as classic an argument for a “service bus” approach as one could think of. Wouldn’t it be better to have a central “interceptor” for system events generated by any system that could then provide an appropriate alert delivery channel (which would, in fact, be email – or maybe instant messaging, or maybe a web service call, or…)? Build it once, and all other monitoring “subscribers” would “plug” into this infrastructure – perhaps even using email as a delivery channel of alert information to the monitoring system.

Email as a reporting solution

Another common misuse of email is the delivery of source data for analysis. Say, you need to provide previous day’s usage statistics to your customer. A common approach would be to have your DBA write a query that runs at 12:01am on the new day, and emails the results to your technology team or a distribution list. An analyst on the team takes the data (usually in CSV format), copies them to a spreadsheet (possibly an existing one), performs whatever analysis magic that needs to happen, and then sends out a chart, pivot table, whatever, to the customer. This last step is performed daily, and if any changes are needed to the source data, date range, whatever – the DBA again is involved and (at best) modifies the original query or (most likely) creates yet another query, which follows the same process. Let’s say you also have to do the same thing monthly, and for more than one customer, and across multiple systems – and you can probably guess (or maybe don’t even have to guess) that it quickly becomes quite unsustainable and expensive.

In other words, as soon as you have to scale, all hell breaks loose. After a while, the management of the copy/paste process takes up more and more time, and as the system and reporting requirements grow, errors are introduced, the number of queries becomes unmanageable, and someone gets yelled at because nothing reconciles with anything else.

Again, a service-bus approach is the way out of this reporting spaghetti. Create a data warehouse, or a pseudo-data warehouse if OLAP isn’t your thing yet, then make all your data dumps go into a separate reporting database. At the very least, you’ll reduce the number of copy/paste operations if your analysts can connect their spreadsheets directly to the reporting database. You also reduce some complexity, as even through such a simple step you could reduce the number of queries run on the system (run only dailies, for example, and sum things up by date – instead of running dailies plus the monthly reports). The addition of dimensional modeling (conformed dimensions, fact tables, et cetera) and OLAP will take this yet another step towards a pan-enterprise, cross-system, ad hoc, reliable reporting.

In conclusion…

Hopefully you didn’t recognize your organization when you read the description of email misuse in the above two sections. If you nodded your head and thought “yep, that’s us” – at least you’ve either found a confirmation of what you already suspected, or now have an idea of an alternative to the world of hurt in which you or someone in your organization probably find yourselves.

What is a pattern? (in software engineering world)

A predefined solution for a problem for the given context.

Key words in the above definition is problem, solution and the context.

What are the different categories of the patterns ? (in software engineering)

• Analysis patterns
• Design patterns
• Architectural patterns
• Technology patterns
• Integration patterns

Why do we use the patterns?

Objective of patterns is to enhance

• Reusabililty
• Repeatability

in return will fetch

• Cost efficiency
• Agility (time to market)

Where the pattern concepts started in general ?

The pattern concept has been in use for thousands of years. In the building architecture and design, we have been using the patterns quite extensively. Ancient buildings like pyramids, roman collegium and lot more used the patterns in the implementation. For instance, the pillars used in ancient temples built in Thailand, India and etc are patterns. They did not design the pillars every time. The pillars are to hold the load of the building. We can notice the reusability and repeatability of the pillars in multiple places in those ancient building architecture which in returned improved the ancients to be effective and quick.

What are the types of patterns in each category?

There are conceptual, logical and physical patterns in each category. Conceptual patterns are very high level solution for the given problem at the given context. Physical patterns spells out the implementation solution for the given problem at the given context. Logical patterns are between physical and conceptual.

Software system architecture is collection of viewpoints. The types of viewpoints depends on the type of systems. Software system architecture for a typical web application contains information and functional views primarily focuses on the structural (or static) representation of the system and the deployment, concurrency and operational views are the behavioral (or run-time/dynamic) representation of the system and the development view is guidelines. Views are instance of viewpoints. Viewpoint is a set of standards, guidelines, templates, patterns for constructing a view.

Functional viewpoint:
The functional viewpoint represents the business processes and function elements of the system. It is mandatory for all web development projects since all web projects are executed to meet a set of functional requirement.
Information viewpoint:
It represents the data and flow of the data within the system. The data or information is integral of the web system.
Deployment viewpoint:
The typical web environment contains minimum a web server with internal disk, domain name server, web browser, inter network or intra network. The environment could be complex for a transaction application. It contains quite few infrastructure components like web server, router, switches, firewall, application server, domain name server, intrusion detection devices, messaging services, storage management systems, load balancer, ssl accelerator, external cache devices etc. It is essential to understand the deployment view of the system upfront during the architecture definition phase of the software development life cycle.
Development viewpoint
In an enterprise environment, there are set of standards, policies, patterns, anti-patterns, reference models, lessons learned, domain knowledge, reference architecture and etc. The development team is expected to reuse the existing references. These artifacts are provided to the application developed team by the centralized architecture team. With recent success of the outsourcing development model, the development viewpoint is required to be part of the architecture.
Concurrency viewpoint:
The functional elements of the system are to be served simultaneously for concurrent users in the web environment. It is required to study the behavior of the concurrency structure of functional elements. It enables to create the necessary environment setup like thread configuration for the operational viewpoint.
Operation viewpoint
The gap between the application development process and the infrastructure management process is quite significant in the industry in spite of the well known IT infrastructure management frameworks like ITIL being in place and adapted widely. The longevity, sustainability, maintainability of the system are determined how the operation views are analyzed and designed during the architecture phase of the project. It may be optional if the web application is developed for a limited time.

Consistency and repeatability in the architecture definition process is a must have requirement in an organization. It is like parents teaching some moral lessons to the kids. Kids will watch the parents if parents do practice what they teach them. If parents does not practice what they teach the kids, the kids are not going to buy-in to the parent’s teaching. Same situation with the architects. Architects emphasis and promote consistency, repeatability, efficiency and etc to the project team and if they do not practice those imperatives in their process and work product, it is going to be a hard sell for them.

Picture speaks more than thousand words. The above diagram illustrates a model, software system architects would follow to define the architecture blue print. The architect’s tool box are the knowledge gained through the internal process, individual experience, continuous improvement. The reference model and pattern catalog provides a framework for repeatability in the architecture definition process and in return implements those imperatives in the end product of the process. ie. software system architecture blueprint.