Many applications have the primary concern of storing and retrieving data. The raw data by itself is often not very useful, so an additional process is put into place to turn that data into useful information. Many applications generate these reports from the data quickly at the user’s request through either a narrow SQL select statement, in application data processing, or both. However, in larger applications where the data is too large to handle in memory at a time, processing is too heavy, and report customization is too varied, information generation needs to be pushed to its own scalable system.

A lot of data has various pieces of metadata associated with it naturally. Data such as what user added the record, the date it was added or modified (or both), maybe the size of the record, categories, tags, keywords, or other pieces of associate data that is used to break it up into more manageable chunks. This meta data is useful, but only if we can use it as an aggregate to generate specific information relating to the grouping of items based on this data.

Sometimes generating these more specific reports is as easy as adding additional WHERE or GROUP BY clauses in SQL. However, when more advanced business rules are taking place where there isn’t an easy or succinct way of extracting this information via a query, or if the query returns such a large amount of data as to cause memory issues, a different approach can be taken.

For instance; in an application I am currently working on we need to generate reports based on a table with about 5 million rows. The SQL queries we use can limit the amount of rows returned to perhaps a few hundred thousand for some of our larger reports. However, a lot of the data needs to be processed in application code rather than by the database itself due to some special business rules. Because of this, we end up creating a large number of objects in Java to hold these result rows. If multiple users are generating different reports we might end up holding too many of these objects in memory at a time and receive an OOM error. Also, the processing on this data can be intense enough that if the server is slammed with report requests that the entire system slows down, causing difficulties for people wanting to insert or modify data. This is the case I am in while I contemplate offline report generation.

The basic idea is that the main application should be concerned purely with manipulating the data of the system. That is basic CRUD stuff such as creating new records, updating them, the rare deletions, and showing single records to the user (so they can edit or delete it). We want that part of the application to remain fast, and not be effected by the purely read-only needs imposed by report generation. In order to nullify the impact, we move reporting to its own system that reads from a separate read-only replication of our production database.

When a report request comes in to our application, we send a request to the separate reporting system. This can be done either as a web service or maybe an RPC call. The reporting system uses its read-only copy of the data to generate the report numbers and send it back, causing no speed delay for insertion or regular operation of the main application.

This doesn’t solve our OOM issues however, as many drivers for our database (MySQL) return ResultSet objects with the entire contents of the results which might be too large to fit into memory. However, since we’re using a read-only list anyway we can convert the table or tables we use to process our results into flat files that can be read in on a line by line basis, perform some intermediate result processing, deallocate those lines and work on additional lines. Since our reports are mostly generating statistical data over a large data set, we can process results on that data set in parallel using multiple threads or possibly multiple computers using a Hadoop cluster.

By making report generation asynchronous to our applications general work flow we will free up the processing power and the connection pool that’s used to handle requests by asking users to either poll for when the result is finished or to notify the system when a report is finished and thereby avoid the instances where we use all of our available connections or resources processing reports. There is also the added possibility of continuously generating all possible report data on a separate machine or cluster to decrease access time by increasing storage requirements.

I’m currently researching the use of MapReduce in Hadoop for processing this flat file of all data for reports. In addition, I’m researching a few languages that are reported to be good at concurrent processing so that I can benefit from multiple cores when generating reports from our raw data. My current focus is on Scala, Erlang, and Clojure, but I do not have much to report on those areas yet. If anyone has any information those languages as far as report generation based on a largish data set (currently 5 million, but the rate of growth is fairly alarming), let me know.

It stands on a high cliff, and saw a young woman on the beach below. She walks along the surf, smooth, rhythmic, spontaneous movement. One can not help looking at the grace and beauty as she dances on the line to navigate with accuracy, efficiency and, of course, continue the waves. Even with a tank on her back and fins in hand, ready to enter the water is pretty, is in tune, you are under sensual.

After he is a man who came out, but in very different waysway. Exaggerates and accentuates her every move. (almost ridiculous), as if to say: "I am the greatest of all time." "I am under the water, hear me roar." I float like a butterfly and stings like a bee …." (The problem is that you float and Sting you ….) The fact is that it is not the best, is a script of an old image that must disappear. Good divers, experienced divers, see this as "a
A preconceived formula for an inefficient, out-of-sync, an accident waiting tohappen. " A cowboy "" At least buddy-Moron ….."

The sex of the divers above has nothing to do with the problem. What you see as you look down, the situation is effectively a diver underwater shooting.

"Effective" Diving is a functional interaction between the divers, equipment, and water. The movements are carried out effectively. Are oriented tasks, but fluid. The diver sensual – at any skill level or gender – is in harmony with its environment.Other Other Sensuous are peaceful. What is sensitive about their dives is gentleness and ability he has, how they move through the water. The creatures of the sea and everything, you know what type of diving and reward him accordingly. An eagle ray gracefully, dancing dolphins, seals and harbor of love are some of the prices of the ocean.

"Interested" dives on the other hand, can be recognized by its unnecessary movements. Attempt to bring the strike spectacular underwaterimmersed in their gestures and movements independent. This type of diving is the most pleasant and inviting to be around like a worm in the apple!. Often, they are impressionable, and then try to imitate the macho style of diving. The insensitive, blind diver will take only the most obvious movements of his idol immersion and then try to exaggerate in order to be even better. Although in the case of diving hero who aspire to be would never witnessed their show, the idol could only think that as a diverdorsal perverse form of pond scum. "

Thus, we have the "I am the greatest diver" syndrome. I saw a lot of beginners try to copy the style of their elders diving. But, unfortunately, the novice does not see that the sharp angle and Perhaps a technical input into the burning boat. What I do not see or understand is that the hero
chosen lifestyle and skills of diving, as his trademark. The label in May to develop a much earlier period. The diver was struck infocus on trying to emulate that the eccentricity of the great diversity fails to play with the dynamics that allowed the hero to become the hero of the
the first place. The plunger moves affected by the water with a poorly developed sense of the interaction between body, equipment, and water. Consequently, the sea and all creatures, is to understand what type of diving and rewarded accordingly. "Hows come an'it Thar never any fish your shot?" Orworse …. "I remember when it was divine 'off the coast hun-ERT and whether the em-dy-five meters." "I curse runt beside me, out of the air … I had your swim as a sum of a bitch …." Needless to say that this kind of sub are generally not able to progress to higher levels of capacity to dive. Still believe that the diver "master" is better than a dive master. And what a dive master is much more advanced than an instructor. …. Go Figure

Imagine for a moment, you dive through a narrowlooking at the canyon walls, as we slide down the road. Fire coral, sponges, plants, and growth on both sides of the valley. A turtle gracefully and effortlessly, slipping only 4 yards away. Want to attack as a boost or take lightly as a feather and the talent of an artist balanced? As we accept this challenge will show whether you are a sensual, experienced diver and if you have the unity of movement and spirit, as creatures of the sea – or if you are aidiot who likes to destroy everything for everyone around you.

Diving is no longer limited to a little muscle men who can run five miles a tank on the back. Now, more and more women, old men and boys are attracted to the sport. They are not interested in a side of fish killed not for food, but only to impress. Are more interested
The beauty of the recreational value and the adventure of the underwater world. I am impressed by the spectacle.Collecting Shell, underwater photography, or simply exploring a new dive site has its own charm. What is spectacular about these ideas is that divers have come to promote the beauty, softness, movement and weight of "the underwater environment. Effectiveness is determined by a mixture of aggression, the capacity and, especially,
review.

My Links : http://astore.amazon.com/asportsblog-20 http://mxm-story.blogspot.com/ http://joystick-de.co.cc/

Java scalability specialist Terracotta has acquired the intellectual property associated with Quartz, a popular open-source job scheduler, part of Terracotta’s mission to integrate common open-source Java application components into its middleware solution. Terracotta has already integrated SQL-query service Hibernate, and it acquired popular distributed caching solution Ehcache in August. These integrations make Terracotta a more formidable competitor in the quest to manage data in cloud or scale-out infrastructures, where it battles relational databases, proprietary caching solutions like Oracle Coherence and, increasingly, flash-based solutions.

As more Java applications make their ways into large-scale environments and private clouds, Terracotta wants to make it easier for users to scale these critical components. The company bases its business model around the “scale continuum” – easing the transition from one node to several nodes to, ultimately, a virtualized cloud environment. Potentially important on this front is the fact that Quartz is embedded into application framework SpringSource, which VMware recently acquired. Terracotta and VMware (s vmw) have partnered on customer implementations in the past, and the synergies between these two vendors and their new assets should make running Java applications in the cloud easier than ever thought possible.

Despite the fact that Quartz is a job scheduler, the database market might see the biggest effects from this acquisition. Terracotta’s bread and butter is in-memory data management, and bringing Quartz’s large user base into the fold exposes Terracotta’s method to an even greater audience. Quartz has tens of thousands of deployments already, including within companies like Vodafone (s vod), Cisco (c csco) and Adobe (s adbe), and it is embedded in popular open-source application frameworks such as JBoss and the aforementioned SpringSource.

One of the most difficult things I find I have to do is coming up with an elegant solution for a difficult problem. Many problems can be solved in a rough and tumble sort of way by tackling it with brute force. However, when you really want an application that will scale well and perform at its peak you seek out those elegant solutions that look so obvious to someone who reads it, but are so difficult to come up with initially.

My basic theory when faced with coming up with a solution, or looking at improvements to a previous solution, is that if it looks more complicated than you think it should be, it probably is. But how do you reduce the complexity that seems so inherent in the problem at hand? There really isn’t a single answer to that question, else I would be selling it to the highest bidder and retiring by the time I was 30. However, there often exists a few good ways to shift your mind in a way that makes an elegant solution easier to emerge.

First, sometimes it is good to just step back for a moment. When you’re looking at the details of a problem intensely, you miss out on some of the larger picture things that can help you find a better solution. Recently I had a problem that I was pounding my head against for days. The problem was partially due to old requirements restricting my progress. I found that if you have the ability to look at some business requirements and discover that perhaps they aren’t really required, you can change the entire scope of your problem so that an elegant solution can emerge. Sometimes it is not possible to change a business requirement. Even when you can, changing the requirements can cause new difficulties to arise, such as migrating data. However, a confusing or complicated business rule should be changed if it makes it both difficult for you to design around and difficult for the users to understand.

Another method that I used on the same problem is exactly the opposite. Instead of looking back at a high level, dive down deeper. When looking at a problem and trying to come up with a solution you might have several ideas floating around in your head. You jot them down and think a little bit about how they might solve your problem, but you’re still unsure. If you have these possible solutions but are not sure if they would work, pick one and dive further into it. Try to draw it out more fully. Maybe create a prototype or fully design a database table or series of tables. Perhaps you can get a better idea of how something will interact by drawing up the interfaces between pieces of a program or writing out an algorithm is pseudocode. Instead of just viewing a problem from a high level, coming up with a solution, and immediately coding it, intermediate steps such as these can help you find flaws you wouldn’t encounter until implementation time when they’re more expensive to fix.

The best feeling in the world is taking an old design that took thousands of lines of code, or dozens of lines of generated and complicated SQL and reducing it to a few hundred lines of code and a line or two of SQL. Sometimes those types of changes can’t be made unless you change your schema or base objects, but if you’re able to change them to reduce complexity from something monolithic and unreadable to something small and concise, then you’ve discovered elegance.

Your work will be smoother, and your confidence higher if you find that elegant design.

By Eric Clarke, President of Orion Advisor Services

As investment advisors become more tech savvy, they increasingly rely on a wider variety of desktop and web-based applications to run their businesses. But what happens when two software packages at one business collide? And does the need to operate efficiently in an uncertain market make this issue even more critical?

Imagine this: In an effort to make life better for you and your staff, you upgrade to an enterprise-level CRM system. But just as your team is wondering how they ever lived without the latest and greatest CRM, you realize it doesn’t share data with your portfolio accounting software. Your critical systems will now require duplication of data entry and will likely never be in synch—not a great example of efficiency.

Efficiency is More Important Than Ever
In today’s economy and utter uncertainty in the market, efficiency is more important than ever. So when it comes to investing in technology, look past the bullet-point promises of boxed software and seek tight integration among your most crucial applications to become more efficient.

“As more consolidation occurs in our industry, small and mid-sized advisors need to find ways to utilize integration in order to compete,” says Sondra Purcell, managing member of Purcell Advisory Services LLC in Tacoma, Washington. Purcell Advisory Services recently solved their CRM and portfolio accounting needs by finding independent providers whose systems were built on the concept of integration.

Advisors need to find technology that fits their business models. At Orion, we use the strength of our technology, and our experience serving advisors to continually develop a flexible product that will fit an advisor’s way of doing business. We were built directly from the needs of an RIA, so we understand the challenges advisors face. And when you put our scale behind your business, you can free up your resources to spend time and effort where it counts. That’s a model of efficiency we strive to fit into for advisors.

“No advisor should have to change their business around someone else’s software,” adds Purcell. “And one-size-fits-all promises are not something you can build your business on.”

We know advisors feel much more comfortable in their CRM, than any other system, but they rely heavily on the reports and data provided by their Portfolio Accounting systems. With web-based integration between Orion’s system and E*Assist/Salesforce.com, for example, an advisor can pull critical data and reports for a client all through the E*Assist (Salesforce.com) CRM. That type of integration makes advisors much more efficient.

How Integration Works
Integration works when two systems use open technologies, heavily leveraging web services, allowing them to independently develop their own products, but use common platforms that will allow each other to hook into one another’s core functionalities. A simple benefit like eliminating duplicate entry of data can provide real efficiencies across an advisor’s firm.

This model gives advisors enormous agility for their business to seek best-of-breed solutions instead of being tied to massive, proprietary or lethargic one-size-fits-all systems,” says Ted Tsung, chief executive officer of E*Assist, a compliance and operations solutions provider.

Sondra Purcell adds, “We needed more than a box of software to run our business. Expensive servers and hardware are not an investment in my business, they are sunk costs that slow you down. On top of that, computer hardware is not a good investment. It starts becoming less valuable as soon as you plug it in.”

While you cannot run a business without some of your own hardware, you can certainly seek out partners who build web-based platforms that bring the heavy-lifting onto their own systems, allowing you to run more efficiently and more light-weight. That is important because growth doesn’t just mean getting bigger, it means getting better. So, if you are spending too much time managing something other than client relationships or money, you are making yourself less profitable.

Find Solutions That Integrate
Whenever possible, Orion urges advisors to stop focusing on the software, and figure out their wants and needs before shopping for solutions that can integrate with each other. Orion is building a strong network of integrations with like-minded firms. Through technologies like web-services, and industry-standard practices, we can remain independent while building solutions that will work with other systems, even as they improve their products.

Orion strives to develop integration points between your core applications, like CRM and Financial Planning software. As members of the AdvisorConnect platform, Orion, E*Assist/Salesforce.com and MoneyGuidePro all build best of breed solutions that work together.

Eric Clarke is President of Omaha, NE-based Orion Advisor Services, LLC. Orion is the Ultimate Portfolio Accounting Service Bureau, designed specifically for registered investment advisors.

The previous six posts suggest accelerating social innovation is an essential notion for Canada. The Public Policy Forum created a useful definition for social innovation:

“Social innovation is the successful application of new ideas in way that produce positive social, environmental, and cultural outcomes. Social innovation can be an initiative, product, process or program that profoundly changes the basic routines, resource and authority flows or beliefs of any system. Successful social innovations have durability and broad impact.”

Recognizing it is important for a deeper societal collaboration and sound societal initiatives. And, we know and respect that we are also part of a global family and we can care as deeply about Africa and Asia as we do our Canada, taking steps now to ensure sustainability.

Social innovations to which we collectively aspire are addressing big issues for our time. These social innovations also require much: innovation thinking, sound initiatives well-implemented and societal changes. These social innovations also require champions and personal participation from the rich diversity of our multi-cultural nation in ways that take these ideas and guide them improving life for all Canadians, and our global family.

It is recognized that the need exists for greater capacity building – in government, the not for profit and profit worlds – and collaboration will always be a central consideration.

Just as there are risks from doing nothing, risks too apply to innovation. By its nature innovation embodies risk, and on occasion, creative failure. Yet, which is the greater risk for Canada, and the global family?

Convergence coincident with ever-greater technological advancements make imperative that we tell our story better – our dialogue and actions demand better communications for a wider audience, including non specialized groups and communities.

Social innovators across Canada are and can stand upon the shoulders of those who go before, building upon the success of smaller applications. In these efforts scalability plays an important role. Enabling mechanisms such as collaborative networks, innovation hubs and processes designed to deepen cross-sector research and development cry out for successful implementation for enabling the breath of humanity in our larger society. And concurrently is a profound need to have solid metrics that identify and evaluate the quantitative and qualitative results relating to initiatives.

This requires a fundamental change of the landscape, a culture engaged in the search for sustainable solutions across broad spectrums.

Practice makes perfect.

“All our knowledge has its origins in our perceptions.” Leonardo da Vinci

“…Innovation is the specific instrument of entrepreneurship. It is the act that endows resources with a new capacity to create wealth.” Peter F. Drucker

We all strive toward this beautiful reality in social innovation:

“Eureka” Archimedes

Be Inspired Today!

The New Currency SDM “Change…At the Speed of Thought”

This session reflected on: What is successful social innovation? What is required for those successes?

Visionaries often exist in a state of loneliness and isolation and this is no less true for social innovators. A new collaborative culture must be created to replace the old vacuum that opens the door to vision and actions.

The speakers agreed that one of the key elements in successful social innovation was a powerful vision. Vision recognizing the processes that need to occur and are able to accommodate a step-by-step approach.

Innovations developed, whether local or national, need to be identified that can be easily replicated and scalable. At present this is not the case. Often, what works locally, may not work nationally and vice versa.

Therefore, it is necessary for innovators to identify patterns of change. To do so requires a deeper integration across all sectors to address the ever changing needs of communities.

Suggesting the necessity of innovators to “help the government do the right thing,” it was noted that a paralysis exists in government that is not the result of dutiful bureaucrats and to combat this paradigm required a reinvigoration and harnessing of creativity.

The ‘wicked’ problems facing the world at present: global warming, poverty and homelessness, to name three, have amazingly creative people responding to them. With their vision and the drive for excellence a retreat from the helping “them” has led to a reckoning of helping “us.”

In that regard, though difficult, across sectors, it is necessary for better engagement. Engaging government, business, not for profits, researchers and academia alike will create massive amounts of new information that can be collected and distributed once through collaborative networks. This will result in new efficiencies that serve to improve output while at same time encouraging innovation.

From increased demand and lower capital at present it is necessary to identify where those networks can be deployed, quickly, easily and simply in efforts to enable: “squeezing water from rocks.” Increasingly as the full effects of the Global Financial Crisis are felt it is becoming more important to identify water releasing rocks.

Key decision makers need to realize the validity of such integration and collaborative efforts so that an innovation policy framework results that fundamentally changes the landscape. In effect partnership ecology that can and will adapt to the ever changing needs of society and concurrently with the advance of technology.

The costs associated with this framework and innovation on the societal level are not costs at all rather an opportunity to limit unnecessary expenditures across all sectors which results in a net increase in societal benefit and the bottom line.

This strategy was best advocated by the following metaphor: We, as a nation, need to enable thousands of flowers to bloom so that big ideas can happen. In order for those flowers to bloom seeds need to be planted and tended to. The seeds are the collaborative efforts, while tending the innovation.

Social innovation requires a common language to build a community of collaborators. This will produce a ground swell of activity, ideas and innovation that lead to successes.

The next post explores ways to tackle ‘wicked problems’ in the context of social innovation in the 21rst century.

“Where there is no vision, the people perish” Proverbs 29:18

Be Inspired Today!

The New Currency SDM “Change…At the Speed of Thought”

Most applications involve storing and retrieving some form of data. It can be posts on a blog, financial information, status updates of friends, or any other large number of diverse topics. Some data is fairly simple and straight forward. A blog for instance is not a very complicated structure to model. There’s a title, the post body, an author, a creation date, maybe even a few fancy things like tags or categories. Overall though it isn’t that complicated, and the interaction between that piece of data and other data isn’t overly complex.

However, some data is complicated, and the interaction between that data and other pieces of data can get quite complicated. That complication needs to be pondered quite heavily when creating your data model. You might think you know all of the data fields of each piece of functionality and write up a quick data model based on the assumptions of what you’re storing. However, in large projects where the usage is high, choosing the correct data model can often be more about reading the data than writing.

Because let’s face it, most data is read, not written. Every time you write some data into your data store, that same piece of data is probably read multiple times by different users or used multiple times in different reports. Writing data is often infrequent compared to how often it is read. In highly scalable designs, you should account for the unbalanced nature of your reads to your writes. This might involve spending a little more time to make your writes by computing additional metadata in advance so that each read will already have that metadata rather than computing it at read time.

One structure you can use for increased read times and reduced computation times is a summary table. A summary table is basically a pre-computed set of data based on your real data table that can be pulled from during read requests rather than deriving the results from your real data on each request. For example, perhaps you have a one to many relationship between Foo and Bar. A single Foo can contain zero to many Bars. Each Bar has a Baz property that can be set to either 1, 2 or 3. Now imagine you want to list the number of Foo items that have Bars with a Baz of 2, and you want to display those results like so:

Foo  | #Bars with Baz of 2
==========================
Foo1 | 2
Foo2 | 4
Foo3 | 1

Your original data might look like:

Foo1 { Bar1 { Baz : 1 }, Bar2 { Baz : 2 }, Bar3 { Baz : 2 } },
Foo2 { Bar1 { Baz : 2 }, Bar2 { Baz : 2 }, Bar3 { Baz : 2 }, Bar4 {Baz : 2 } },
Foo3 { Bar1 { Baz : 3 }, Bar2 { Baz : 2 }, Bar3 { Baz : 1} },
Foo4 { Bar1 { Baz : 1 }, Bar2 { Baz : 1 }, Bar3 { Baz : 3 } }

So to create that above table you might need to do something like the following psudocode:

Map<Foo, BazNum> dataMap;
for(Foo f in getAllFoos()) {
   List<Bar> bars = f.getAllBars();
   for(Bar b in bars) {
      if (b.getBaz() == 2) {
         dataMap.set(f, dataMap.get(f)++);
      }
   }
}

for (Foo f in dataMap.keySet()) {
   print (f.name() + " | " + dataMap.get(f));
}

To generate that table your program has to go through 3 for loops. Ok you can probably remove the last for loop if you put the print statement in the inner loop above it, however most of the time you’re going to be separating generating the data from displaying the data into different functions so the second iteration will be required. This is an O(n²) operation due to the nested loop.

Imagine if you have 200 Foo objects, each of which has on average 10 million Bar objects. Just storing the objects as tuples in your data store and compiling the report on each request is going to bog down your processing machine very quickly, especially if you receive loads ranging in the hundreds per second. However, if you pre-determine this information during your write, you can easily retrieve the result at any time with a very small processor cost.

insert parentFoo, name, baz into Bar values ("foo1", "bar9000", 2);
update FooBaz set bar2s = bar2s + 1 where foo = "foo1";

Now instead of selecting a count(), joining on Bar, and grouping by Foo, you can merely select from FooBaz with a possible where clause.

This is a rather contrived example that you might not see as great of a speed increase as in the real world, however if you imagine a much more complex data model with many nested objects that would require multiple joins to generate the proper data; or even worse, multiple joins followed by application level processing followed by additional selects followed by more processing; you will see where pre-calculating results upon writes can be a large time saver.

Sometimes it isn’t possible to update a result simply upon a write however. One is forced to do a full set of computations in order to get a result based on all of the data available after the write has happened. This does not mean that you can’t use summary tables with pre-computed results however. You might need to run a separate task to update the summary table on a scheduled basis with a cron job, allowing reports requested to be slightly lagging real-time data. If this is a trade-off that you can afford, scheduled report generation can make impossible request-time level reports turn into possible time-delayed reports.

Today we announced hyperguard SaaS—the industry’s first dWAF as a SaaS through Amazon Web Services (AWS).  AWS customers or solution providers can protect applications by applying hyperguard SaaS either as software plug-in to an existing web server Amazon Machine Image (AMI), or by using AoD’s custom AMI.  The solution solves the limitations of traditional WAFs being forced to secure cloud applications, which they weren’t specifically designed for.

It is highly scalable and ideal for virtualized resources—AoD hosts the resource-heavy pieces of the dWAF on Amazon EC2 and leaves just a small footprint on the customer’s AMI. Therefore, hyperguard scales simply with the number of web server AMIs that run the customer’s application being protected without a need to purchase additional AMIs.  This allows customers to pay on a use-case basis and avoid investing in intensive solutions.

hyperguard SaaS provides web application security monitoring, detection-only and protection modes. For additional information or to test the service for free go to http://aws.artofdefence.com

Image Source: http://www.aturfingertipspa.com

Managed File Transfer is a hot topic right now.  So hot that John Redfern, MD of Inovis UK & EMEA, has written a very interesting piece in International Clinical Trials on the importance of transferring critical and sensitive data safely and securely using MFT. John is very wise, so you should check the article out - esp those who are driven by compliance, security, scalability, and guaranteed delivery of their data.

When working on a new project one of the first things you should do is gather a list of requirements of what the goals of the project actually are. It’s hard to create something if you don’t have any solid idea of what it is that you’re creating. This list of requirements doesn’t have to be hard fast – there are always changes and new requirements that pop up. But you should have a basic idea of what you’re trying to do before you try to do it at least.

The process of defining your requirements however does not stop at just the initial analysis or design phases. Requirement gathering should be a continuous process throughout development and maintenance. The agile approach that I champion requires that all parts of development be flexible, including requirement gathering. This does not mean that adding new requirements should be a process you throw in at the whim of a user however. Balancing what goes into a project and what stays out, along with prioritizing between requirements is the key to avoiding scope creep and actually releasing a successful project.

There are dozens of tools you can use to keep your project data together. For the initial requirement phase I tend to like just paper. Graph paper to be specific as I find that it is useful for keeping lines neat and drawing diagrams on. However some people use a visualization tool like viso or OmniGraffle, or maybe a spreadsheet, word processing document, or wiki. Maybe  just a plain text file or a bunch of sticky notes. Whatever you use, just make sure you can keep it together to reference later or find a way digitize it so that you have it in one easy and accessible place.

The way I like to start with is make a quick list of all of the tasks that your project should do (or does) and rank them by importance. When starting a new project you should focus on those features that are most important first. This gives you a nice launching point for the users to see what the project does, along with the ability to use it in a useful fashion early on. If you’re doing maintenance, this list gives you the ability to focus your maintenance efforts on those areas that are most important to your customers first.

Try to make sure that there is one main thing that your program does. A goal that it helps the user achieve. Generally I like a project that can be summed up in a single sentence, even if the overall use of the application can be expanded. Something like “A program for storing test results and aggregating those results into useful information.”  That definition describes generically what our program does, but you can expand it to include other goals that support that main goal such as “Store a description and process of the test itself so that you have context for the result you’re storing.” and “Organize tests into interrelated groups that relate to a common business process.”

One task that people either forget or perform incorrectly is to solicit feedback from the customer/user. Forgetting to ask the user for what they want means that you’re making assumptions that could very well turn out to be incorrect. Working for weeks or months to churn out a product that the customer doesn’t even want isn’t a very successful project. Knowing what the customer wants will make it easier to create a product that they are willing to use and that will improve their ability to achieve their goals.

However, one important thing to note is that user’s don’t know what they want. I know that seems to contradict what I said earlier about asking the users what they want. Why ask them if they don’t even know? The user’s think they know, but often they can only tell you what they can imagine. It’s up to you to not take what the user says at face value and instead think about the underlying problem they are describing, ignoring any implementation details they might have provided, and determine a way for them to achieve the real goals that they desire. Asking them how a menu should look or what buttons they want where isn’t very helpful. The user isn’t going to want to spend the time it takes or the research necessary to tell you exactly what would really be best for them. That will take research on your part and probably some demos or prototypes that you can show to the user and gather feedback.

Once you have a list of your prioritized goals you should sketch up a rough plan on how you can achieve them. If your in maintenance mode, maybe some of your goals are too difficult to fit into the current architecture. You need to come up with a plan for modifying the structure of the project so that your goals are achievable. You should try to remain flexible during this process as most of those goals are still loose and subject to change as business requirements or new data comes to light.

Its often difficult to keep in mind your base goal for a product when all of the nagging implementation details start to bog you down. If you ever find yourself in a situation where you’re working on fixing some problem and wondering how even got to this problem, or you think it’s too difficult and there must be an easier way, stop. There probably is, and you might have lost focus on your scope.

This evening I imported my 50,000th picture to my Lightroom 2 Catalog (database). That’s a year’s photography.

Lightroom 50,000 Photos

Much to my surprise Lightroom didn’t burst into flames or otherwise have a meltdown. There’s an old joke about an undocumented IBM S/360 opcode: HCF – halt and catch fire.

I was heartened by the fact that Jeff Friedl told me he has over 77,000 photos in a single catalog.

I’d love to know what database tweaking Adobe did / plan on Lightroom 3 to improve its scalability.

]]> http://jonnyrainey.wordpress.com/2009/11/05/week-7/ Thu, 05 Nov 2009 00:02:00 +0000 jonnyrainey http://jonnyrainey.wordpress.com/2009/11/05/week-7/

Cloud Computing and E-Commerce

Cloud computing is an internet based use of computing technology. It is seen as a paradigm shift in that details are hidden away from users who do not have knowledge or control over the infrastructure that supports them. It typically involves the use of dynamically scalable infrastructures and makes use of virtualization technologies for economical reasons. The term came about when one looks at computer network diagrams and is an abstraction of the underlying infrastructure it conceals.

Cloud computing has several benefits and drawbacks, some are listed below;

Benefits

Save on electricity as only the server end need to be powerful. Client machines do not require much processing power as work is done on the server.

Suits small and medium companies who may not have finance to have a dedicated server farm in house.

Scalability – As the company doesn’t own the hardware, they don’t have to pay as much for expansion, the cloud computing business will buy more resources when required.

Consumers need only pay for the resources they use. Users do not have to pay for potential one time use scenarios.

Drawbacks

Issues of the Data Protection Act appear seeing how the user does not have physical access to his/her data. By using cloud services such as those provided by Google may be in breach of council, or local government policies.

If the users lose connection to the server then there is downtime. Many offerings provide offline capabilities but services such as email would still be affected. As downtime is out of your control, this makes it harder to know when normal service would resume in case of a problem.

Current broadband bandwidths do not cater for high speed large data access. If many users were to try to use a cloud computing service then speeds would drop, especially for any business moves around large files.

In ecommerce today, cloud computing is already in high use, such services such as Amazons Flexible Payments Service and Google Checkout use the cloud based model in order to allow customers to make quick and easy transactions with online services. With companies being able to outsource these core functions of their online business, they in turn will not require to pay people who would normally do this job and as such they would be able to lower prices for the consumer.

Introduction:

The IP Multimedia subsystem provides a well defined and streamlined architecture for charging multimedia calls and services. The IMS network elements interface to the charging platform over the DIAMETER protocol to enable both pre-paid and post-paid charging.

Architectural Overview of IMS Charging:

The IMS charging platform is sub-divided into two major components:

1. The Charging Data Function (CDF).

2. The Online Charging System (OCS).

The CDF is responsible for receiving triggers for offline (post-paid) charging, while the OCS  is responsible for receiving pre-paid charging triggers. The CDF supports the Rf DIAMETER application, while the OCS supports the Ro DIAMETER application interface.

The Charging Detail Records (CDRs) are collected and co-related at the Charging Gateway Function (CGF). The CGF acts as a gateway to the Billing System, which performs mediation duties.

Scalability Challenges:

In the IMS architecture, the P-CSCF, S-CSCF, SIP-application servers, MRF ,MGCF,BGCF and the I-BCF all need to support the offline charging application interface (Rf interface). This means, that they all act as charging clients (CTF) in case of post-paid scenarios and send triggers to the CDF.

The S-CSCF, the MRF and SIP Application servers support the online charging interface towards the OCS. The IMS Gateway function facilitates the online charging functionality in the IMS architecture by acting as a SIP AS.

A quick look at this architectural challenge, presents us with a nearly all connected architecture for the CDF (it interfaces with almost all network elements for postpaid charging). This means, that the CDF acts as a multiplexer of incoming DIAMETER commands.



Even for a simple IMS call, the CDF will receive triggers from the P-CSCF and the S-CSCF. If there is an application server involvement (Eg: Supplementary services) in the call flow, it will receive a trigger from that AS as well. If an I-BCF is present in the network, and the call needs to be terminated in another IMS domain, the I-BCF will also send a trigger to the CDF. This is a very practical scenario, as almost all IMS customers will subscribe to at least one supplementary service and every IMS core network is expected to have border control and peering functions for security (I-BCF acting as an entry and exit point to the network).

This translates into four DIAMETER transactions for the CDF for a single IMS transaction. A single IMS call initiated by an INVITE may have multiple chargeable transactions involved (those for UPDATEs, RE-INVITEs and finally for BYE).

The OCS interfaces with only the S-CSCF, the MRFC and SIP application servers. Hence it is expected to be less loaded as compared to the CDF as shown below:

Scalability Requirements Quantified:

Let us take an example of one IMS call for the sake of calculation. A reasonable load of 100 calls per second is assumed in the calculations.

We will only consider “chargeable” transactions i.e. transactions for whom a DIAMETER trigger will be sent to the CDF. A single call can have an INVITE transaction, an UPDATE transaction, a RE-INVITE transaction (assuming there was only 1 re-invite in the call) and finally a BYE transaction. Thus, we have 4 “chargeable” transactions. For each transaction, a DIAMETER ACR/ACA exchange takes place between each network node and the CDF.

Even for a reasonable load of 100 calls per second on the IMS core, the DIAMETER transactions for the CDF will need to scale up to 1600 TPS. This provides us with a 1:16 scalability requirement for the Charging Data Function.

For the Online Charging System, only the S-CSCF (through the IMS GWF), SIP application servers (if any in the call path) and the MRFC send charging triggers.

For the sake of calculation, if we have one application server in the path of an IMS call, then a single IMS transaction will result in 2 DIAMETER transactions on the Ro interface. Taking the above assumptions, where a single call has 4 “chargeable” transactions, and a moderate load of 100 cps, the OCS requires to support 800 TPS. This provides us with a 1:8 scalability requirement for the Online Charging System.

The scalability factor requirement for both the CDF and OCS is considerable. In case of the OCS, the response time of the CCR will also affect the call setup latency, as the calls to CCR are synchronous. Unless a CCA is received, the SIP signaling is not put through.

Other considerations to plan for scalability:

In view of the above scalability requirements, we can now plan further on the nature of the deployment architecture of the CDF and the OCS.  Even though the scalability requirements for the OCS may seem to be half (800 TPS) of what is projected for the CDF (1600 TPS) , but the complexity of the OCS is much more as compared to the CDF. The online charging system has many internal modules responsible for real-time rate determination, calculation of the units to be debited and account balance management. Moreover, these modules need to be invoked for each IMS call. The OCS also needs to support time based and content based charging paradigms. This means, that the OCS node is a mission-critical and real-time charging engine. Apart from real-time traffic, non-real time traffic such as pre-paid balance inquiries, pre-paid recharging etc also need to be handled at the OCS (either through an IVR or a SMS based mechanism).

On the other hand, the CDF is responsible for creating and dumping the CDR files. There is no real time rate determination or account balance management involved. The raw CDRs are transported to the billing system (BSS) over FTP, where the itemized billing and mediation takes place.

Deployment planning and possible scenarios:

In view of these architectural discrepancies and varying levels of complexities between the CDF and the OCS, it is clear that there needs to be an architectural separation between the OCS and CDF for deployment. This means, that we have to deploy both nodes independently on dedicated machines and possibly in their own independent clusters.

Let us consider a single IMS domain. A single IMS domain will consist of its own S-CSCF, P-CSCF, SIP application servers (as needed), MRF, I-BCF(or a comparable SBC) and the charging platform (consisting of the OCS and the CDF).

For greater scalability, it is proposed to have a dedicated cluster for each charging entity. The OCS application should have its own cluster and so should the CDF. Both the OCS and CDF may share the same gateway (CGF) to interface to the Billing domain. The clusters should have a DIAMETER load balancer (for the Ro and Rf applications) installed to distribute load amongst the cluster instances.

For hardware configuration, the cluster members may reside on the same machine (for smaller deployments) and on a hardware pair (active-standby) to achieve hardware redundancy. Usually, for carrier grade deployments, hardware redundancy is a necessity.

Discussed below are certain deployment configurations of the OCS and the CDF for varying load requirements. The deployment architectural choices shown below also consider hardware redundancy.

NOTE: “Servers” may vary from case to case. You may go for a T-1000, T-2000 or a higher end SUN server. You may also go for a “cheaper” option by using HP servers (8 cores and 16 GB RAM). For large-scale carrier grade deployments, an ATCA is a must (12 blades and above).

Deployment for up to 1 million BHCA (approx 270 CPS):

270 CPS is assumed to be the call rate of SIP signaling during peak load. This is the most common deployment scenario considered in telecom and usually serves as a benchmark.

For IMS deployments for up to 270 CPS, we can have multiple options. We may go for a HP server pair configured as follows:

a) Server-1 has CDF active and OCS as stand-by.

b) Server-2 has CDF stand-by and OCS as active.

This simple deployment will provide us with software and hardware redundancy, while also providing dedicated 8 core servers for each charging application (OCS and CDF).

Software redundancy can be performed by providing 2 independent processes of the CDF and OCS on each server (active) and 2 more processes of the CDF and OCS (for stand-by). A software load balancer may be used for distributing this load amongst the OCS and CDF processes.

This is called a 1+1 active-standby configuration. Each active instance of the CDF is expected to handle up to 4320 TPS at peak load. Each active instance of the OCS is expected to handle up to 2160 TPS at peak load.

This deployment scenario is shown below. The red-arrows depict change-over in case of hardware failure. Software fault tolerance is taken care of by switching between the software instances of the OCS and the CDF by using a software load balancer.

Each server may host multiple software processes for the CDF and the OCS. Each process can be a full CDF/OCS application in its own right. The load balancer may also be deployed in an active-standby configuration similar to the CDF and the OCS. The CDF and OCS active processes may be more in number, based on the scalability requirements. Each new process of the OCS or the CDF may be instantiated using a CLI or over SNMP, when the element management systems get an alarm of possible overload or traffic peaks.

This  kind of a deployment architecture is shown below:

Other interfaces for this deployment can be a command line interface (CLI) for polling the system and performing administrative tasks. Other interfaces will be over SNMP to the northbound management systems for monitoring the system health and servicing alarms.

Other possibilities:

In case of higher loads, the system can switch to a 1+1 Active-Active configuration, where all software processes of the CDF and the OCS are accepting DIAMETER requests. The load balancer will handle the distribution of the requests amongst the processes. In case the capacity is still not sufficient, new application processes of the CDF and the OCS can be started by firing the appropriate CLI command to scale horizontally.

For deployments for over 1 million BHCA, as shown earlier, the load on the charging platform will be even higher. For catering to such traffic, a time will come when the system needs to scale out. We may require a server pair for the CDF and the OCS dedicatedly.

Another interim option is to have all active CDF processes on one machine and all active OCS process on another machine and horizontally scale by increasing the number of processes.

Conclusion:

This post was an attempt to quantify the challenges at hand for scaling the Charging platform in the conetxt of IMS. As seen, the requirements for scalability for the charging platform is more challenging than the other core network nodes. The scalability factors increase almost exponentially as the load on the IMS core increases. Hence, a robust scalability architecture needs to be devised for catering to the same.

Penelope Trunk, the Brazen Careerist, wants employers to view potential hires in a non-traditional way. Penelope has built quite a following over the past decade writing career advice to generation Y. She has written a book (Brazen Careerist, The New Rules for Success), she has a syndicated column and her blog has over 40,000 subscribers. She firmly believes that traditional career path fundamentals are outdated and instructs her Generation Y followers to pursue a different path. She believes that traditional career sites are out of touch with the new web 2.0 reality and are a waste of time for young career minded individuals. Given that the older job seekers have longer resumes and deeper backgrounds, she has come up with an alternative way for Generation Y job seekers present themselves to future employers using Web 2.0 networking.

Since the younger Generation Y workers are less experienced than their older Generation X and Baby Boomer counterparts, they have fewer opportunities to secure jobs or advance their careers using the old paradigm of wisdom and expertise gained from years of past employment. Penelope believes that your value to a company should be based more on your ideas and potential than your background and experience. She also believes that the best way to present your self is through social networking. In the past, a lot of young job seekers have been burned by their own content residing on social networking sites like Myspace, Facebook and Twitter. Inappropriate content was their undoing in most cases.

Before the web came about, Baby Boomers relied on newspapers and personal networking to advance their careers. In the early days of the internet, Generation X used Monster and Linkedin to network and find the job they wanted. In every case, the preceding generations have embraced new technologies if they didn’t want to be left behind. Since traditional career sites focus on experience, younger candidates don’t fare that well. Generation Y feels more comfortable with conversation based networking, so social networking is more appealing to them as a way to connect with employers. Older careerists had better begin to embrace Web 2.0 if they expect to remain competitive in the job market.

In a recent press release, the Brazen Careerist articulates its vision: “Brazen Careerist is a Generation Y-targeted career site that uses social networking to showcase job seekers’ ideas and potential over their background and experience. Brazen Careerist “levels the playing field” for Generation Y by reversing the disadvantage these candidates have faced on other online career sites where experienced Gen Xers and Baby Boomers appear more employable. For recruiters and hiring managers, Brazen Careerist will offer unique deep insight into the critical thinking skills, ideas, knowledge, and peer influence of high performing Generation Y candidates whose potential would not be discoverable from an online resume.” You can learn more at www.brazencareerist.com.

So, will the Generation Y kids ultimately beat out the older more seasoned Baby Boomer and Generation X job seekers in securing the best jobs with the best companies? It is way too early to speculate on that, but my advice to the older career minded individuals is this: you better embrace the new technologies and Web 2.0 sites or you will be left out in the cold for sure. If you are not on www.linkedin.com, then get cracking and start networking. If you consider yourself an expert in your field or profession, then start writing a blog and build a following. Most companies are looking for thought leaders and if you can turn a phrase and have some really good ideas about your profession, share them and you will benefit in more ways than you know. You can go to www.wordpress.com  and set up a blog for free.

Another major consideration for older job seekers is compensation. More and more companies that have been hit hard by the recession are looking at ways to reduce their budgets and younger, less experienced employees are much less expensive to hire than their older counterparts. So my second piece of advice is to be more flexible when it comes to salary and benefits. Once you demonstrate your value to the company, the compensation will take care of itself. Your potential employer will appreciate your contribution to his bottom line and won’t forget you when the economy turns around.

A final note about Penelope Trunk. She was born in 1966, thus making her a member of Generation X and proving that you don’t have to be a member of Generation Y to leverage social networking and Web 2.0 for your own financial gain. Clever individuals can achieve significant success regardless of age. Go Penelope!

Got a opinion? We would love to hear your thoughts on this topic. If you are a Baby Boomer, and haven’t yet joined my web community, www.boomeropinion.com, please do. It is free and only takes a minute to sign up. You can participate in our polls on major issues facing America and Baby Boomers. You can also join in our discussion forums or even set up your own blog at our site.

Last week, I began this series on the secrets to building a billion dollar web property by giving a brief analysis of the early development of Facebook and Myspace. I discussed how their visions and value propositions morphed over time. I stated that the number one secret of their success was user engagement. And yes, your vision and value proposition will determine the level and frequency of user engagement, but not always on the first implementation of your concept. You have to continue fine tuning that vision until your growth becomes viral and exponential.

Neither Facebook or Myspace became successful based on the founders’ original target market, value proposition and vision. And they are two of the most successful web sites in the history of the internet. So lesson Number One is: If your site design does not truly engage your users day after day, month after month, and year after year, then regardless of how brilliant your original vision or concept might be, you’ve got nothing.

Neither Facebook or Myspace would have been able to accommodate the exponential growth they experienced had they not considered scalability in the overall design of their sites. There are those who still think that building a successful web property is only about offering your visitors content, media or engagement they can’t get from another web site. Without scalability, your site will crash often and die an early death once exponential viral growth overwhelms your computing resources. Hopefully, if you are in the process of starting a web community, scalability will be a major consideration.

For you non technical readers, scalability covers four critical areas. These critical areas include development platform and language, development talent, hardware and hosting, and last but not least, caching. Skimp or minimize resources in any one of these areas and you will never make it to the next level. Over time visitors don’t return to sites that can’t deliver on the promise of their functionality and content availability. It even happens to the major players…I just logged into Twitter and got this message: Twitter is over capacity…Too many tweets…Please wait a moment and try again. Twitter can get away with this now, but for how long?

Let’s start with the software platform that was or will be used to develop your site. This includes the language and operating system platform on which your web site is to be built. Bottom line: selecting the right development language and platform is critical to building a scalable startup. If you choose a platform that allows you to get the web site up quickly, but is inherently un-scalable, your gamble will come back to bite you in the butt. If your site engages the users, you will also need a scalable database backend to maintain all the information about your membership and their profiles. If you don’t have the technical expertise to understand scalability, then hire a web developer or partner with someone who has that expertise.

That brings me to the second critical factor in achieving scalability. Scalable web startups are willing to make the necessary investment when it comes to securing talent. Hiring and/or seeking advice from a web developers or web architects in the early stages of your site development is absolutely essential to your success down the road. You might counter by telling me that you can’t afford such high level talent. My suggestion would be to seek out a web developer or developers with the required expertise and offer them equity in your startup. If you don’t have the funding to pay them as regular employees, then maybe getting in on the ground floor will entice them.  If your vision and value proposition have merit in their eyes, then they will be satisfied just to work for a piece of the company.

Since my primary target audience is non-technical, I don’t intend to get bogged down in the technical details of hardware and hosting scalability, so I will give the briefest of summaries on those issues. We have already discussed the development platform and talent required, but there are some additional areas that haven’t been covered. It is essential to understand bandwidth requirements with respect to your server connections to the internet.

Suffice it to say, through the use of web hosting services, you can scale your bandwidth up over time as needed to accommodate the traffic on your site. Whether you use the hosting site’s servers or your own, that should not be an issue. The primary difference has to do with costs and your web developers will help you make those determinations.  If your site really takes off, then you will have to transition to your own in-house server farms.

The last critical factor I will mention has to do with caching. Your development guys will understand the importance of caching and should have it as a very high priority from the beginning. Basically, caching takes the load off the disk drives and processes the routines and requests through temporary memory (mem-caches) which will speed up those processes. Speed and performance is critical to a successful web site and poor performance will have a very negative impact on your traffic. Users tend to stay away from sites that are difficult to log into or navigate.

So there you have it. Secret Number Two is: Scalabilitiy. From concept to launch, scalability has to be a major consideration when developing your site. The best web concept in the world won’t make your site successful if you can’t provide your users 100% availability and functionality.

I posted the following to Terry White’s Tech Blog about Lightroom 3:

http://terrywhite.com/techblog/archives/3747

I am downloading the beta now in Bangkok, Thailand. My internet connection is very slow but I’ll get there.

My biggest question is about database capacity and performance. I’m approaching 50,000 images in LR 2 now and my catalog is about 1GB not counting the previews cache. That’s about one year’s work.

I want to be able to use one LR catalog for every photo I take for the rest of my life. If I have a powerful machine and sufficient disk, is this a reasonable expectation that LR will be able to support many hundreds of thousands of images?

The way I work I often need to go back to pictures I took years ago so splitting into multiple catalogs (e.g. 1 per year) would be very inconvenient.

Clearly database technology can handle that kind of task – e.g. MySQL – but I am not so sure about SQLite.

Thanks! I am looking forward to trying the new features.

I did not get an immediate response, although Terry is good about responding to other questions.

Adobe has been vague about performance and capacity in everything I have seen. The LR 3 beta is no exception. They say:

Lightroom has been stripped down to the “engine block” in order to rebuild a performance architecture that meets the needs of photographers with growing image collections and increasing megapixels.

But what does that mean? Did they change the SQLite database engine? What are their performance and scalability objectives for LR3?

Adobe has this appearance of openness with their blogs but I feel they don’t share much beyond the standard marketing-approved talking points.

As always I see both sides. I was nervous about talking publicly about performance and scalability when I worked for HP. There are so many variables and it’s easy for one’s words to come back to haunt you (and the company). But that is a long time ago and we should be better at it now.

Are my scalability expectations reasonable or not? I cannot believe I am unique.

I just recently found out that James Robertson had posted a video of my “Share Everything” talk from ESUG 2008 several months ago. I must have missed the announcement:)

An MP4 version of the talk was posted here a month ago.

And here’s a link to the slides.

Microsoft has released their Remote Desktop Load Simulation Tools which have nothing to do with Remote Desktop in the RDP sense.  Instead, the tools are designed for 32-bit and 64-bit server capacity planning and performance/scalability analysis.  According to Microsoft:

In a server-based computing environment, all application execution and data processing occur on the server. Therefore it is extremely interesting to test the scalability and capacity of servers to determine how many client sessions a server can typically support under a variety of different scenarios. One of the most reliable ways to find out the number or users a server can support for a particular scenario is to log on a large number of users on the server simultaneously. The Remote Desktop Load Simulation tools provide the functionality which makes it possible to generate the required user load on the server.

Supported operating systems are:

• Windows Server 2008
• Windows Server 2008 Datacenter
• Windows Server 2008 Datacenter without Hyper-V
• Windows Server 2008 Enterprise
• Windows Server 2008 Enterprise without Hyper-V
• Windows Server 2008 for Itanium-based Systems
• Windows Server 2008 R2
• Windows Server 2008 R2 for Itanium-based Systems
• Windows Server 2008 Service Pack 2
• Windows Server 2008 Standard
• Windows Server 2008 Standard without Hyper-V

(Notice the lack of Windows 2003 support?)

A minimal test environment requires:

1. Target Remote Desktop Server
2. Client Workstations
3. Test Controller Host

A giant hurdle for buying a system/solution as a software is the need to buy hardware, install it, configure and manage it. You need to train people on the products’ operational aspects and retain that skill within the company.

(Free) Open Source Software (FOSS) is great to spread, to get adoption and support for a product. You enable the developers and architects to play around with the stuff! The real challenge for FOSS (and other software) products is to go beyond the happy and content developer and also provide a painless path for the adopters to provide business value without a huge investment hurdle in terms of hardware, software, traning or services.

I think the reason why something like Google Analytics or Salesforce.com is successful is that it is extremely painless to start using it. You can focus on the business problem rather than the IT stuff. Obviously this is nothing new, and the examples I gave has been around for years. Software as a Service is great.

Then, you have all the talk about the real-time web and putting information quickly, as it happens – “real time” – on the users’ desktops. This is what Twitter and Facebook is about, but real-time web is also needed for e-commerce and gaming and a lot of other areas. There are even conferences about it, so it must be happening

Lastly, the final piece of the puzzle are Service Level Agreements. In order to provide “real time web” messaging as a service there is a clear advantage of being close to the information consumers, both in terms of scaling out and in terms of guaranteed latency. I think it is going to be hard to commit meaningful SLA:s without being in the edge.

If you remove the need to invest in infrastructure, the need to train people on the operational aspects and then get excellent scalability and low latency guaranteed by contract, I’d buy it in a second. Who will provide me with the Real Time Web as a service?

Source: http://www.earley.com/blog/sharepoint-content-structure-let-thousand-content-types-bloom

“How many content types should you have?”

This is the question that came up in a conference call last week on SharePoint architecture. This organization had implemented their corporate portal on SharePoint 2007 and was interested in going forward with more portal sites but had some concerns about the approach to information architecture they had undertaken.

I answered what I would answer no matter what technology it was – “Only as many as you really need to implement the appropriate level of metadata, workflow and templates.” Which is of course vague, as most good consultant-ese is. So I followed up with the notion that when we work on web content management implementations, we typically end up with 10-15 content types for a site of medium complexity. We always try to keep the structure simple and number of content types few for many good reasons, ranging from ease of content structure management to content publisher user experience.

The folks on the phone were quiet for a minute… You see, the previous consultant they had worked with had a bit of a different (read opposite) approach. The philosophy they described was that SharePoint content types should be created to the maximum degree of granularity (e.g. one content type per library) so as to reduce the need for content publishers to select a content type and tag metadata values. For example, if you had a site for human resources forms, you would have one library and content type for medical forms, one library and content type for dental forms, etc. Each content type would be extremely specific and require little tagging. “If you need 30,000 content types, then so be it” is the idea. (insert eye twitch.)

The intent behind this – to reduce uncertainty and effort for content publishers – is noble and good, but the overly-granular content types seems to be in the realm of sledgehammer to kill a fly. To help explain why, I thought I’d enlist the help of a couple of friends and colleagues.

First, I emailed content management guru Bob Boiko, author of the Content Management Bible, to see if he agreed. His response?

“How many content types is the right number? The fewest possible to squeeze the most value out of the info you possess. If it were my system, I would create a generic type and put all the info that I could not find a business justification for into that bucket. It’s not worth naming if you can’t say clearly why you are managing it. Then I would start with the info we have decided is most valuable and put real energy into naming the type and fleshing out the metadata behind it. Then on to the next most valuable and so on till I ran out of resources. In that way, the effort of typing is spent on the stuff that is most likely to repay the effort.“

Amen to that! But I also wanted to get a tool-specific view from my colleague and SharePoint expert friend Shawn Shell. So I skyped him…

So, what do you think?

Well, having a content type for every document library is certainly an interesting approach, though I think your SharePoint administrators, as well as your users, will go quite mad.

So, I think the argument is that having this many content types is supposed to make it easier on the users by presetting all choices and removing the potential for error. If you never have to choose a content type because each library has a very specific default that matches the content you are creating, then there’s no confusion, the idea seems to be… From a general content management perspective, this is flawed. But what about from a SharePoint-specific standpoint?

I can understand why this might make sense on the surface.  Unfortunately, I think you end up exchanging one kind of confusion for another.  Further, there’s a huge maintenance implication as well. For example, if you have a content type for each library, you are, for all practical purposes requiring the user to decide where to physically store a document.  This physical storage then implies your classification — regardless of whether a default content type is applied.

So, you’re basically recreating all the ills of a fileshare folder structure!

In essence yes. To make matters worse, more complex SharePoint environments will necessarily include multiple applications and multiple site collections. Because content types are site collection bound, administrators will have lots more administration to create, maintain and ensure consistency across the applications and site collection. This would normally be true, but when you have such an overload of content types and libraries, the complexities of management are compounded.

So, if you have 50 content types, and you need to use them in 2 or 3 site collections, you’d have to create 150 content types. Good argument to keep your use of content types judicious. Is there a hard limit to the number of content types one can manage in a site collection?

The answer is “sort of.”  There’s no specific hard limit to the number of content types in a site collection, but there are some general “soft limits” in the product around numbers of objects (generally 2000). This particular limit is an interface limit where users will see slower performance if you’re trying to display more than 2000 items.  The condition won’t typically manifest itself for normal users, but it will for administration. The other real limit is the content type schema can’t exceed 2 Gb.  While this seems like a pretty high limit, if you have a content type for each library, loads of libraries in a site collection and robust content types, there’s certainly a chance to hit this limit.

What about search? I assume that a plethora of content types would have adverse effects on search.

It absolutely does.  Like everything we’ve discussed here, the impact is primarily two fold: 1) administration and 2) user experience. Content types, as well as columns, can be used as facets for search.  If you have an overwhelming number of facets in results, the value facets bring is reduced.  Plus, as I mentioned before, having large numbers of content types could also produce performance problems when trying to enumerate all of the type included in the search result.

From an administrative standpoint, we’re back to managing all of these content types across site collections, ensuring that the columns in those content types are mapped to managed columns (a requirement for surfacing the metadata in search results) and, if you have multiple Shared Services providers, that this work is done across all SSPs.

I expect there will also be a usability issue for those trying to create content outside of the SharePoint interface. Wouldn’t users have to choose from the plethora of content types if they started in Word for?

This is another excellent point.  Often, when discussing solutions within SharePoint, we think only of the web interface. When developing any solution, however, you need to keep both the Office and Windows Explorer interface in mind as well. Interestingly, using multiple document libraries, with a content type for each library, makes a little more sense from the end users perspective, since it’s similar to physical file shares and folders.
However, the same challenges that many organizations are facing related to management of file shares can manifest themselves when using the multiple library and matching content type approach as well — putting these organizations back in the same unmanageable place they started.

Great, thanks Shawn for your insights! I’ll be sure to spread the word to avoid a content type pandemic.

So there you have it folks. Less is more. Standardize, simplify and don’t let your content types multiply needlessly. Your content contributors and SharePoint administrators will thank you.