Nov.24 – The company launched "Amazon EC2 Adoption Services" which consist of  planning, design, build and management for EC2 environments. In addition to those standard services, the mail magazine for mobile and the access analysis services will also be delivered.

J <http://www.asahi.com/digital/bcnnews/BCN200911240002.html>

Azure. EC2. App Engine.

Do any of those names sound familiar? If so then join us for the first meeting of the McHenry Cloud Developers Group in Crystal Lake, Illinois!

Our first meeting will be an informal discussion of the goal of the group, ideas for projects and presentations. Also, we want to know what you are looking to get out of the group.

The meeting is Monday, December 14th from 7:00 PM until 8:00 PM. We’ll be meeting in the downstairs Ames-1 meeting room.

Please join us in Crystal Lake and join the projects or hang back and soak up some info!

Our first meeting will be held at the Crystal Lake Public Library.

Register here!

Directions:
Crystal Lake Public Library
126 W Paddock St
Crystal Lake, IL 60014-6194

View Larger Map

Also, we have a brand new site up. http://www.clouddevelopersgroup.com. It’s a very early draft but feel free to check it out.

I would like to announce the launch of a new Cloud Computing community website!

www.AllYourClouds.com

This is a new community website which focuses on having the answers for all your clouds.

Got a question about Amazon EC2, Azure, Google app engine, Go-grid, rackspace, etc? Need to know how to modify your code? Wondering how to migrate?
Just post the question and someone in the community will answer it for you.

The best part about it, is that the site uses OpenID, so there is no need to sign up. Just click to log in with your existing credentials (Google, wordpress, blogger, etc. etc.)

I don’t have much server background, and probably its a lot easier to set up your servers with amazon EC2 than setting them up yourself or at other hosting solutions. I don’t care, it really annoys me that everybody claims how easy it is, showing you how to launch an instance in a minute. Yes, thats really easy, but you quit too abruptly. Nobody wants to set up an instance, then terminate it and lose everything you did on that instance. I say, never press terminate, you will lose every change from the last ami save. You are warned, be prepared or waste a lot of time.

Read on for links an tips for the whole cycle of launching, changing, saving and registering AMI. Everybody deploying to EC2 really have to do all this as a part of setting everything up, being prepared to launch more instances of the same kind.

If you launch an instance without knowing how to save and register AMIs, save yourself a lot of time (and a few pennies), by not doing a lot of work you might lose. If you do a lot of work, don’t press terminate, it’s not the same as a shutdown of your computer. You have to look at it as an ISO image of files, if you read it, work with the content, you’ll have to create a new ISO image to burn on your CD to keep the changes. At page 26 in the ElasticFox it says “Bundling a Linux/UNIX instance requires the use of the AMI tools”. OK, as you probably understand, I’m burn.

I am now successfully using both ec2-api-command-line-tools, elasticfox and amazons web console.

Ok, so what is the minimum need to get a small server running ubuntu and a mysql database?

• EC2 for your server and database
• S3 for storing your AMIs and register them
• EBS is optional, but you should set one up as a safety-net, replicating your data.
• both the ec2 and ami command line tools.

My links, tips and references.

The ec2 api command line tool:

• http://paulstamatiou.com/how-to-getting-started-with-amazon-ec2 EU? Important addition, put this in you bashrc or bash_profile too

export EC2_URL='https://eu-west-1.ec2.amazonaws.com/'

• AWSEC2 Command Line Reference

ElasticFox

• Amazon ElasticFox pdf tutorial
• If you’re using the alestic ubuntu images below, remember to change “SSH user” to ‘ubuntu’, from the Tools menu in ElasticFox (icon top-right)

EBS to replicate your data

• MySQL database server, including snapshot backup and restore.

Finding ubuntu EC2 AMIs:

• http://alestic.com/

SAVING AMIs

First, make sure you have a bucket in S3 for storing the your new ami. I use S3 Firefox Organizer

• I suppose you have launched an instance
• then partly follow this http://robrohan.com/2009/01/30/saving-a-customised-linux-amazon-instance-ec2-and-s3/
• with the exception if your using alestic ubuntu AMIs (they don’t use root, but user=ubuntu). Run this on your EC2 instance:

mirror=http://us.ec2.archive.ubuntu.com/ubuntu/
printf "%s\n%s\n" "deb ${mirror} karmic multiverse" \
"deb-src ${mirror} karmic main" |
sudo tee /etc/apt/sources.list.d/multiverse.list
sudo apt-get update && sudo apt-get install ec2-ami-tools

I struggled a lot with sudo, but found the solution mentioned above, at here at google groups

Now you should be all set to launch a new instance based on that new AMI of yours! Break a leg!

*Update* Microsoft have updated their FAQ with instance pricing

In my previous post on the newly released Azure SDK I touched on the ability to set a size for your VM instance.

Lets delve down into what size virtual machines are available (values from http://msdn.microsoft.com/en-us/library/ee814754.aspx)

The sizes are easy to follow, they are all just multiples of the base VM size. Microsoft have said in their FAQ that the pricing is based on multiples of the small VM size. It is based on “CPU cores / hour”, so $0.12 per hour for the small VM, $0.24 for medium, $0.48 for large, etc.

Lets draw up a matrix to compare the Microsoft Azure and Amazon EC2 pricing side by side:

Azure						Amazon EC2
CPU cores	Memory	Platform	OS	cost / hour		"Compute units"	Memory	Platform	OS	cost / hour
1	1.7 GB	64 Bit	Windows Server 2008 R2	$0.12		1	1.7 GB	32 Bit	Windows Server 2003	$0.12
2	3.5 GB	64 Bit	Windows Server 2008 R2	$0.24		N/A	N/A	N/A	N/A	N/A
4	7 GB	64 Bit	Windows Server 2008 R2	$0.48		4	7.5 GB	64 Bit	Windows Server 2003	$0.48
8	15 GB	64 Bit	Windows Server 2008 R2	$0.96		8	15 GB	64 Bit	Windows Server 2003	$0.96

(Values taken from http://msdn.microsoft.com/en-us/library/ee814754.aspx and http://aws.amazon.com/ec2/instance-types/ )

The Azure and EC2 packages are roughly equivalent to each other and the “CPU” speeds are roughly the same (Azure: 1.5-1.7GHz, EC2: 1.0-1.2GHz Xeon). Where they still do differ though is Amazon is still only offering Windows Server 2003, and the low end instance is only 32bit compared.

Conclusion

From a .Net developer’s perspective I see Azure winning here. For the same price I can get Windows server 2008 R2 over a 2003 instance, which gives me IIS7. If you are developing cutting edge .Net code (like ASP.Net MVC), IIS7 is going to be a lot easier to work with than IIS6. Of course if you need to control more in your instance then Amazon is going to offer you that over Azure.

Footnote: Amazon does have a few more instance types available that offer higher CPU processing or increased memory. Here I have just used the basic instance types for easier comparison.

I attended the Business Technology Summit in Bangalore last week – 3^rd and 4^th November. There were 3 tracks on cloud computing, Service Oriented Architecture and Business Intelligence, and I chose a mix of sessions across each.

Overall impression: The BT Summit was heavily focused on cloud computing with half of second day having a deep dive into Amazon’s EC2 cloud offering, and several keynotes. SOA and web services, REST and similar architectural sessions were interspersed but definitely not a first-class citizen. BI came a poor third with a poor choice of sessions, and more of a rehash of what is out there for everyone, rather than something on the cutting-edge including use of appliances and columnar databases, as also in-memory databases and use of Flash and AJAX for interactive BI front-ends.

Session-wise review: (Speaker profiles available here). I was able to speak to and ask questions of Vinod Kumar, Vijay Doddavaram, Abhinav Agarwal and Dr. Bob Marcus.

Keynotes:

• Smart services, bright future by Howard Charney

Probably the highlight of the keynotes, this was a pep-talk about the inevitable interconnected future with smart products and services and for good measure Charney threw out some statistics on broadband growth and bandwidth usage and India’s readiness and potential in the scheme of things.

• Give cloud a chance by Ramkumar Kothandaraman

The worst of the lot – this started by comparing the spectrum of offerings in the cloud from Amazon’s DIY EC2 and AWS, Google appengine and apps to Microsoft’s Azure and ended up as a promo touting Azure as the best buy among all.

• Cloud computing: State of the union address by Shouvick Mukherjee

A very good keynote, focusing on what makes sense to migrate to the cloud and what doesn’t, what are the hidden costs, the myth of unlimited elasticity in the cloud and what Yahoo is doing to use open source software like Hadoop and Hive for cloud computing. In the short time span, Shouvick also tried to address some of the other considerations – including re-architecting existing applications, availability, data storage and movement considerations.

• Managing IT in turbulent times by Som Sharma

This post-lunch keynote by Sharma was a rambling talk on how technology keeps redefining our lives, and why it is important to think outside-the-box. He used the example of the iPhone to illustrate how such thinking has the potential to alter the established rules of the industry and redefine it as we know it.

• A New world to protect by Nils Puhlmann

Puhlmann provided the security perspective on how easy it to break/hack enterprise systems and how anti-virus and anti-spyware are always playing catch-up, the entire economy that is spawned by the “bad-guys” in technology and why our systems need to be smart and be built from the ground-up for security rather than as an afterthought. He provided valuable insights into what questions we should ask ourselves as we embrace cloud computing, the changing technology landscape making it easy for consuming information but easier still for the security breachers. Puhlmann concluded by suggesting it may be worthwhile including a level of risk assessment and mitigation, and collaboration with ethical hackers, rather than trying to do the impossible of removing all security threats.

• Moving towards a virtual enterprise by Srinivas Varadarajan

Barely managed to sleep through it – this one talked about moving towards a virtual enterprise – with a focus on virtualized architecture, including cloud computing. As boring as they can get.

Other sessions:

• SOA, Composite Applications, and Cloud Computing: Three pillars of a modern technology solution by Robert Schneider

Robert  Schneider presented the different facets of SOA, Composite applications (superset of mash-ups) and Cloud computing and contrasted them regarding the time to yield benefits, the maturity of the vision, involvement and buy-in from business and where they lie in the tactical-strategic plane. There wasn’t anything regarding why we are stuck with these three for a modern technology solution, or what other paradigms are out there beyond the old-world enterprise computing framework, possibly due to time constraints.

• Self-service analysis and the future of Business Intelligence by Vinod Kumar

A lot of the BI folks were waiting for this, as Vinod performed the Project Gemini (Office 2010 Excel and PowerPivot) demo live for the first time in India, with several folks, including yours truly, sitting on the stairs. [We have had to rely on Youtube videos and MS Office 2010 preview videos earlier]. The demo was impressive fetching over 13 million records into Excel using a standard DDR laptop, using compression and in-memory technologies. The bigger question around unleashing another round of Excel hell went unanswered due to time constraints, however the presentation probably hinted at Microsoft’s vision of “self-service BI” or so-called “underground-BI” as the power-users of Excel (estimated at 2M worldwide, at 4% of the Excel user base) have been doing. Microsoft’s strategy around pushing SharePoint adoption in the Enterprise was made clear tacitly with SharePoint being the only “portal” to publish and share BI analysis (typical size of these Excel spreadsheets is upwards of 200MB) with other users in the enterprise.

• Designing and Implementing RESTful web services by Eben Hewitt

Eben Hewitt started off with a very brief comparison between SOAP (Simple Object Access Protocol) modeled more on the lines of RPC (Remote Procedure Call) and REST (Representational State Transfer) and clarified that REST is more an architectural style rather than specifications. The remainder of the talk delved into details of implementation of REST – usage of simple ‘verbs’ and complexity in ‘nouns’, uniform interface, using named resources, java REST frameworks like Jersey, MIME types – JSON, XML, YAML and HTTP operations supported – POST, GET, PUT and DELETE.

• Business Intelligence project execution on a shoestring by Rajesh Ramaswamy

I attended with some expectations on how a BI project can be executed possibly with open-source or free software like MySQL/Postgres, Pentaho/Talend, Jaspersoft/MicroStrategy reporting suite etc., but was highly disappointed by the presentation. Ramaswamy spoke on BI usage, barriers to BI adoption, costs of BI implementation and spewed statistics like m&m’s with cursory references to Forrester, Gartner and “research studies”, but there wasn’t anything tangible on how to go about a project execution except for some common-sense talk on “evaluating options” between open-source and licensing costs, offshoring and outsourcing, RDBMS vs. analytica databases and appliances etc.

• Business Intelligence – Leveraging and Navigating during current challenging times by Vijay Doddavaram

Vijay spoke of the current global economic downturn and how it had taken everyone unawares during the downturn as well as when the current quarter the tide seems to have returned. With the example of a fictitious company in China, he illustrated the importance of trade-off between tactical and strategic decision making and whether and how business intelligence can make a difference in either a downturn and the upswing (whether it is a U, V, or a W curve). Thought-provoking, one couldn’t help feel that BI software has not yet eliminated the “intelligence” that people bring to the table, and made a distinct point about the “human analysis/intelligence” against the out-of-the-box actionable-intelligence marketed by the BI vendors. It would have been interesting to prolong the discussion, with a focus on the “predictive-analytics” offerings in the market (from SAP, WPC, SPSS and the open-source R etc.), we had once again run out of time, and it was the last session of the day as well.

• Towards a unified Business Intelligence and Enterprise Performance Management Strategy by Abhinav Agarwal

Abhinav is from Oracle and he used this session to basically present the BI and EPM strategy of Oracle. Refreshing when contrasted with the usual Oracle marketing hype, Abhinav made it a point to stress the difficulty of delivering best-in-breed products due to numerous acquisitions and the inevitable integrations compared to the disruptive start-ups which could be one-trick ponies but nevertheless manage to push the technology envelope. Most of the session focused on Oracle BI server offering and the roadmap of integrating with the Fusion middleware, and brief touchpoints on the capabilities of the Oracle BI server: federated queries (acquired from nQuire, which Siebel systems had acquired, prior to being bought by Oracle), and real-time updates, including Oracle RTD (Real-time Decisions) and the segregation of the BI and EPM software offerings.

• 10 Things software architects should know by Eben Hewitt

I was able to attend part of it, but for the most part- the bottomline of this talk was the trade-offs architects need to make and understanding there may not be a “solution” to a problem, it may just be “moving the problem” – the idea that each “solution” brings its own issues and tradeoffs into the picture. Being more focused on java APIs and cloud computing frameworks, it could have done better with something related to networks and database architecture in general for audience to relate better (for most of my time, I couldn’t relate to a BI applications and data-warehousing infrastructure).

• Cloud initiatives and standards roadmaps by Dr. Bob Marcus

Being late from an overcrowded dining hall, I was able to attend part of this. Bob spoke of the various public and private initiatives including those from the federal government, NASA Nebula and made the distinction early on between the types of offerings on the cloud: SaaS (Software as a service), IaaS (Infrastructure as a service) and PaaS (Platform as a Service). He mentioned in passing the data.gov and apps.gov initiatives of the Obama administration as also about RACE (Rapid Access Computing Environment) from the Dept. of Defense – Defense Information Systems Agency.

• Building Enterprise Dashboards by Vivek Khurana

Vivek Khurana did a very short presentation to an overflowing hall on clichéd but nevertheless important aspects of information visualization while designing dashboards: clutter vs. simplicity, proper designing of KPIs, importance of delivery to mobile devices, and learning from news aggregation sites and portals on presentation.

• Implementing Enterprise 2.0 using Open Source products by Udayan Banerjee

Banerjee did a great job of presenting what his vision of implementing Enterprise 2.0 in NIIT was – implementing SLATES (coined by Andrew McAfee) – Search, Links, Authoring, Tags, Extensions and Signals. Within half-an-hour he navigated us through using open-source products for collaboration using blogs and wiki (MediaWiki), using single-sign-on with enterprise databases, using links and tag clouds and integrating Search as well as implementing a text-based instant messenger.

• ECM-CMIS and the emergence of standards by Alan Pelz-Sharpe

I had missed the earlier session of Alan on lessons learnt using SharePoint, so I made it a point to attend the last of this at the summit – even though it meant I had no clue sometimes of what was being talked about! Alan spoke of the emergence of the multi-vendor CMIS standard for Enterprise Content Management – the various facets of ECM – from digital and media assets, email archiving, Internet content, web analytics, document types, rich media and the problems with the earlier Java standards like JSR 170 – most notably the absence of support from Microsoft. He also spoke about the vendor landscape and a 9-block rating similar to Gartner’s magic quadrant, plus various other important standards, including XAM – eXtensible Access Method – a storage standard developed by SNIA (Storage and Networking Industry Association)

Presentation files: Most presentation files are available here. You’ll need to register though to download.

- Maloy

Amazon Web Services released it’s  CloudWatch API and many people are using it to gather metrics/information about their servers. This article describes the process used to write a nagios compatible plugin that gathers data from the Amazon CloudWatch API. We decided to write this plugin in a nagios compatible mode because nagios is basically the defacto standard in monitoring return code.

Background

This plugin was written for DartWare’s InterMapper as a probe pack for it’s Free 5 Server Cloud Monitoring solution. More information about DartWare’s monitoring solutions in and out of the cloud can be found at: http://intermapper.com

Almost all network monitoring solutions support nagios compatible scripts including InterMapper. InterMapper is an extremely versatile network and system monitoring solution. This plugin returns all nagios compatible messages and furthermore uses InterMapper formatting when nagios is not specifically set to true.

What do I need to make this plugin work?

The CloudWatch monitoring plugin is written in Python and relies on the BOTO framework. The BOTO framework is included with the download at the bottom of the page.

Examples

The following execution of the CloudWatch Script will give you CPU utilization for a specific InstanceID. The options -w and -c represent the warning and critical thresholds within nagios.

CPUUtilization

The warning and critical thresholds are measured in percent.

./awscloudwatch.py -a <access-key> -s <secrect-key> -i <instance-id> -m CPUUtilization -w 10 -c 70

NetworkIn

The warning and critical thresholds are measured in bytes/sec.

./awscloudwatch.py -a <access-key> -s <secrect-key> -i <instance-id> -m CPUUtilization -w 10000 -c 900000

Results Look Like?

By default all info messages are displayed with InterMapper’s  probe formatting, this allows InterMapper users to track detailed information and make it graphable.

\{ $CPUUtilization := “2.78″ }

To get standard nagios INFO messages just pass “–nagios True”  and the response will look more like:

OK: CPUUtilization is 2.73

What other metrics are supported?

Currently all of CloudWatch metrics can be monitored. Please see the following API document for all available metrics and their data types.

http://docs.amazonwebservices.com/AmazonCloudWatch/latest/DeveloperGuide/arch-AmazonCloudWatch-metricscollected.html

Download The Plugin Now

The CloudWatch plugin can be downloaded by clicking here . To learn more about how InterMapper’s free custom AWS AMI visit here

I’ve noticed there were some searches coming through with people asking how much it costs to host servers on Amazon’s EC2. Without going into detail, last month I ran 3 Linux instances (m1.small) with no EBS and minor S3 usage, which cost me about $200 a month.

Compared to my previous hosting company – hostmonster.com – Amazon is way more expensive. However, I now find that taking care of my server administration needs is much easier and (gasp!) fun.

I recently came across MOA (Massive Online Analysis), an environment for what its developers call massive data mining, or data stream mining. This New Zealand-based project is related to Weka, a Java-based framework for machine learning which I’ve used quite a bit over the years. Data stream mining differs from plain old data mining in that the data is assumed to arrive quickly and continuously, as in a stream, and in an unpredictable order. Therefore the full data set will typically be many times larger than your computer’s memory (which already rules out some commonly used algorithms), and each example can only be briefly examined once, after which it is discarded. Therefore the statistical model has to be updated incrementally, and often must be ready to be applied at any point between training examples.

I also came across a press release describing version 2.0 of KnowledgeMiner for Excel, a data mining software apparently used by customers like Pfizer, NASA and Boeing, and which is based on GMDH (Group Method of Data Handling), a paradigm I hadn’t heard about before. I failed to install KnowledgeMiner for Excel for my Mac due to some obscure install error, but from what I gather, the GMDH framework involves a kind of automatic model selection, making it easier to use for non-experts in data mining. (Of course I haven’t tried it, so it’s hard to evaluate the claim.) The example data set provided with the software package has to do with climate data and modeling, so it should be fun to try as soon as I get it working:

The new KnowledgeMiner is now capable of high-dimensional modeling and prediction of climate and has an included example using air and sea surface temperature data. This is a first for a data-mining software package: to offer anyone the ability to see for themselves that global temperatures are rising steadily, using publicly available data. The biggest surprise is seeing that the changes are greatest and accelerating in the northern latitudes. By using data from the past, KnowledgeMiner (yX) can show predictions for future years. Go to this link to see the climate change data displayed graphically in a slideshow through the year 2020:

There’s also an interesting new toolkit for web mining from BixoLabs. They’ve built what they call an elastic web mining platform in Amazon’s Elastic Compute Cloud (on top of Hadoop, Cascading and a web mining framework called Bixo, for those of you who care). The whole thing is pre-configured and scalable, and from the tutorials on the site, it seems pretty easy to set it up to crawl the web to your heart’s content.

In the 9.10 release of Ubuntu Server Edition, we introduced something new for people who are exploring cloud computing using Amazon’s Elastic Compute Cloud (EC2).

If you haven’t tried it yet, EC2 is essentially an API for managing virtual servers. Using a command line tool or script, you request a new VM, and moments later, it is ready for you to ssh in, preconfigured with your ssh public key. When you’re finished, you shut it down, and receive a bill for only the time and Internet bandwidth you used (about $0.10 per hour and $0.10-$0.17 per gigabyte). There is no downloading, no installing, and no hardware required for you to set up a server. The first step is to boot it up.

Starting with release 9.10, every release of Ubuntu Server Edition is simultaneously available on EC2. This means you can have a new Ubuntu server up and running using your EC2 account with a single command. Ready-to-run Ubuntu machine images are published on EC2 whenever a new Ubuntu release or milestone is available. All you need to know is the AMI, a short string which uniquely identifies the image you want. The AMIs for the 9.10 release are on the download page, in 32- and 64-bit versions for US and Europe zones.

Similarly, all Server Edition development builds are available on EC2 as well. When the first builds of Lucid (Ubuntu 10.04) are created, there will be AMIs for those as well. If you want to test drive a new feature, or check compatibility with your application, just fire up a new instance on EC2, do your work, and then terminate it. The whole process can be completed in less than a minute. If you find a problem in our development builds, just run ubuntu-bug on the virtual machine as you normally would, and apport will automatically attach the relevant EC2 details to your bug report.

As I mentioned above, EC2 does charge for Internet bandwidth. It does not charge for local bandwidth within your EC2 zone. For this reason, Canonical has set up Ubuntu archive mirrors within EC2, so that you can download all Ubuntu packages and updates for free. Ubuntu virtual machines inside EC2 are automatically configured to use the appropriate mirror, so you don’t need to think about it.

This is an exciting step forward in making Ubuntu more convenient and powerful to use on EC2, and I encourage you to give it a try. If you’ve never used EC2 before, just follow our Starter’s Guide to get set up.

Researchers from MIT and UC San Diego recently demonstrated an attack against Amazon’s EC2 where an attack virtual machine can launch attacks against a victim virtual machine that is located on the same physical server.

Does this mean that there is a security vulnerability within EC2? Yes.

Should you be concerned? Not really.

Read more .. ..

En Voota creemos que cada uno debe dedicarse a lo que mejor sabe hacer. Por eso, desde el primer momento, no quisimos encargarnos de comprar, instalar y configurar servidores. Amazon Web Services (AWS) ofrece un sistema de virtualización que nos abstrae de gran parte de este trabajo de sistemas. De momento empezaremos utilizando 2 de sus servicios:

• EC2: Amazon Elastic Cloud Computing
• S3: Amazon Simple Storage Service

Así utilizamos EC2

Este es, sin duda, el núcleo de los servicios de virtualización. Permite crear servidores web, servidores de bases de datos y … servidores en general. El concepto es simple: Permite crear todos los servidores que necesite el proyecto desde una interfaz web. Por supuesto son servidores virtuales, pero a todos los efectos es igual que tener un servidor en el despacho de abajo, con la ventaja de que puedes incrementar el número de servidores con un clic de ratón.

Lista de instancias en Amazon EC2

Las instancias en EC2 son servidores en ejecución. Se paga por el uso de instancias, lo cual permite tener más o menos servidores sirviendo páginas en función del tráfico del sitio en cada momento.

Otro concepto muy interesante es del AMI (Amazon Machine Image) que no es otra cosa que un servidor “congelado” o, más técnicamente, una imagen de servidor. Cuando se necesita arrancar una nueva instancia, se abre la “nevera” y se le dice a Amazon: “Arráncame uno de estos” y se crea una instancia nueva.

Elegir un AMI

Cualquiera en la comunidad puede crear sus propias imágenes al igual que cualquier distribuidor de software. En el caso de Voota, utilizamos Ubuntu server. Canonical nos lo pone fácil y pone a disposición de cualquier usuario de EC2 su “servidor congelado”. Ya está disponible en Karmic Koala (la nueva versión de ubuntu).

Pero esto es sólo para la primera vez. En Voota, una vez que tuvimos el servidor instalado con el software necesario y listo para servir páginas, le hicimos una foto. Es decir, lo congelamos a un AMI. De esta forma, la próxima vez que sea necesario arrancar una nueva instancia, lo haremos desde nuestro propio AMI, que ya está listo para funcionar.

AMI del servidor web de Voota

Así utilizamos S3

S3 es un servicio de almacenamiento. Permite almacenar cualquier tipo de información mediante ficheros (imágenes, vídeos, AMIs, backups, etc.). Para nosotros es muy útil a la hora de almacenar las imágenes y los vídeos. Un error muy común en el desarrollo web es almacenar estos ficheros multimedia en un disco del servidor. Todo funciona bien hasta que se necesitan más servidores y por tanto compartir estos ficheros de imágenes. Hay varias soluciones para este problema:

• Sincronizar continuamente los ficheros entre todos los servidores. Bastante poco práctico sobre todo a medida que va creciendo la estructura.
• Utilizar una unidad compartida de modo que todos los servidores lean y escriban en el mismo disco. Más cómodo pero se corre el peligro de saturar el disco con lo  cual el balanceo de los servidores no servirá para nada.
• Crear un servidor de ficheros al que todos los servidores web envíen los ficheros cuando hay que “escribir”. Y que sea este servidor de ficheros el encargado de servirlas directamente al navegador del usuario del sitio web.

Esto último es lo que nos ofrece S3. Cuando algún usuario de Voota carga una fotografía, ésta se envía directamente a S3, quien la almacena. Cuando esta fotografía ha de aparecer en alguna página web, será S3 el encargado de servirla.

Foto de Zapatero almacenada en S3

Esta foto está almacenada en http://imagesvoota.s3.amazonaws.com/politicos/bw_p_238.jpg

Otra utilidad de S3 es el almacenaje de los AMIs o “servidores congelados” que mencionaba antes. Para poder utilizar un AMI en EC2 es obligatorio tenerlo almacenado en S3.

We’re very excited to announce the Public Terabyte Dataset project.

This is a high quality crawl of top web sites, using AWS’s Elastic Map Reduce, Concurrent’s Cascading workflow API, and Bixolab’s elastic web mining platform.

Hosting for the resulting dataset will be provided by Amazon in S3, and freely available to all EC2 users.

In addition, the code used to create and process the dataset will be available for download from http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=263

Questions and input on the project can be submitted at http://bixolabs.com/PTD/

Everyone knows that the longevity and ultimate success of a platform lies in the developers willing to create on that platform. Creating a healthy developer ecosystem in which both the developer and the platform vendor can grow and prosper is very important. No one knows this better than Microsoft. It has long been their mantra that “developers, developers, developers” are the key to success. With most of the new large scale websites and platforms being launched on open source platforms and now the cloud, this is becoming even more important.

If you just do some casual research, it is easy to see that the largest and most heavily trafficked sites on the web are written in languages such as PHP, Ruby on Rails or even Java, and these are all deployed on open source platforms. Which version of Unix or Linux they reside on isn’t important. What is important is that developers are migrating in droves to these languages and frameworks, and new developers are learning these languages and frameworks instead of the typical Microsoft .NET stuff. So how does this relate to cloud computing?

Well, whether it was by virtue of luck or actual planning, most of the cloud computing vendors (if not all but Microsoft shops) chose to deploy their infrastructures on open source platforms. Google’s App Engine, long the bastion of true could computing is mainly a Python development platform. The underpinnings are all open source, and Google does contribute significantly back to the cause. Amazon’s Web Services are also built upon solid open source foundations, and the majority of platforms that can be deployed in EC2 instances are Linux distributions. These open source platforms were chosen when language/framework pairs were developed and launched, such as Ruby on Rails, Python on Django, and even Groovy on Grails. These frameworks have provided a solid base upon which to build cloud computing applications and services.

So why can it be a good idea to become a cloud software developer? I would say that there are several reasons to do so, not the least of which is marketability. The hype around cloud computing is at fever pitch, and there are many reasons for it. The root cause is the promise of increased efficiency at a lower price point. One often downplayed benefit that I feel is the most important benefit is flexibility. With cloud computing, the promise of true flexibility and scalability has finally been reached, and the multiple layers of abstraction from the hardware and operating system have granted developers the freedom to focus on their code, not the underlying stuff. By understanding the concepts of developing in the cloud, you have opened yourself up to a new market that is akin to a dam about to burst.

Businesses are moving to the cloud (whether private, public or hybrid) at a break-neck pace, and positioning yourself as a competent developer in this space can only pay off. It may even pay off huge. Just remember that tying yourself to a particular vendor or platform is not the best way to approach. You should focus on a language and framework that is common to most or all cloud infrastructures and leverage your knowledge of development as a whole to pitch your skills as wide as possible. Understanding the benefits and limitations of the cloud will allow you to avoid costly mistakes, and your superiors will see you as the ‘go to person’ for the next wave of IT.

As soon as read the AWS newsletter about Amazon RDS, I started looking for a Megaphone to start shouting at folks – keep away! Amazon RDS or Relational Database Service places Amazon into the mire of shared hosting and AW users into a position of false confidence. Harsh words considering, overall, I feel Amazon’s service offerings are best-in-class. AWS offerings have historically pushed the envelope with regard to practical usage-based computing, something which ancient providers such as Sun and IBM have attempted to accomplish for decades; in this case I define practical as both usable and cost effective for small and large tasks. Up until now such systems weren’t trivialized to x86 hardware and required special programming considerations, access to academic institutions and/or a large budget. By combining SLA-supported x86 virtualization alongside application services such as S3, SQS, and SimpleDB, AWS has provided a usage-based on-demand computing solution which is simpler than task-based computing and as secure and reliable as virtualized or shared hosting. With it’s on-demand nature AWS is a cost effective for everything from small tasks to those requiring a datacenter of processors.

So why is Amazon RDS so bad, so much that you shouldn’t use it?

Well, there’s not an easy answer, the better question is to ask yourself: Why do you think AWS will be better than your own MySQL deployment? There is no right answer because almost any answer will probably, one day, bite you in the ass. Hard. I mean data loss, and it won’t be Amazon’s fault.

RDBMS systems and applications which depend on them are built from the ground up to rely on persistence, integrity, and static data models (schema). In contrast AWS has been built for distribution, decentralization, and the “cloud”. For Amazon, this service is somewhat of a U-turn from their original direction and has also placed a stamp on their forehead which says “That MySQL Guy” which is not good — I have nothing against mysql, however, as a de facto entry-level (free open source) software, it has accrued a strong following of immature software. Such software has nothing to do with the basic purposes of AWS or MySQL but has everything to do with how Amazon’s support and engineering staff will be spending their time which is supporting users and software which aren’t built for the cloud.

I hope that RDS won’t be a situation of butterflies & hurricanes but here’s a quick list of why the relative cost of RDS is high both for Amazon (the company) and all of it’s AWS users:

• Cost for Amazon (operations, engineers, and products)
  • MySQL, like most open source systems, has been historically buggy software with a trailing release+testing+production schedule which requires continuous testing between production releases for large deployments (such as RDS).
  • MySQL has a large set of features which vary across releases and which share equal presence in production; in other words, Amazon will need to cater to providing production support for multiple versions, not just the latest stable version.
  • Amazon has no control over features and capabilities of MySQL and is thus limited to what MySQL provides; while MySQL provides many “good things”, Amazon will still be obligated to maintain through the bad. This is a shared disadvantage of AWS Map Reduce via Hadoop however, those are mostly mitigated because Map Reduce is such a low-level distributed system.
  • MySQL is very flexible and itself scales very well however it doesn’t do so by itself and requires a significant effort to be properly configured for the data being managed. All the folks who don’t know this will default into thinking Amazon will do this for them and will be disappointed when it doesn’t “just work”. Whether they ditch RDS or bug Amazon’s support, either way, it’s not a positive situation.

• Cost for AWS (primarily EC2) users
  • Potential degradation of service and support for EC2 instances
    • With RDS available Amazon can defer issues with regard to running MySQL on EC2 instances to a recommendation for RDS — this will be a terrible waste of time for both parties.
    • MySQL is a very centralized system and by transitioning the decision of where MySQL resides in the AWS cloud from the user to Amazon, Amazon will be further centralizing the impact of MySQL on the cloud. Whereas users will randomly have MySQL deployed across any EC2 instance, Amazon will be appointing MySQL to specific hardware; this is based on the assumption that Amazon is clustering RDS deployments onto local hardware and not randomly deploying instances in the cloud. This is somewhat of a compromise for security and adds significant SLA risks (read: cost) to Amazon. In short, when a MySQL cluster dies – a LOT of folks are going to be VERY unhappy – their support tickets will be a burden to staff and their requests for credits will be a financial cost. Moreover, support staff will be yielding priority to these customers over other services because of the implicit severity.
  • Increased cost
    • RDS instances cost >10% more than regular instances and only come with the added benefit of backups — something which every system should already have in place. If you do choose to delegate the task of backups to RDS, you’re paying extra for a task you’ve already thought about doing yourself.
    • Cost of keeping your database, it’s backups, and it’s history all within AWS is multiplicative and if you grow to the point where you’re ready to move off you’ll be charged to transfer all the data to an external system. While this is a subjective cost it’s still worth pointing out; if folks aren’t already doing backups right, they’ll likely not know that cost effective database backups make use of binary logging facilities, not filesystem snapshots, and use significantly less disk space (and thus I/O).
  • False confidence
    • As I’ve mentioned before, letting other folks control your backups for you is a mistake. Failure is a matter of when, not if, and you’ll be in better control of responding if you understand what you’re dealing with. Just because RDS is doing you’re backups doesn’t mean you’re safe.
    • RDS users will expect MySQL to scale on-demand as everything else works that way with AWS and it’s just not that simple. Scaling a database requires analysis and a balanced combination of server settings, data normalization, and indexes; all of these things will still be the user’s responsibility and Amazon’s solution of “throw hardware at it” is a haunted path to send it’s users down.

Overall, I feel that Amazon could quickly cannibalize the value and quality of AWS if they (continue to) introduce trivial services. Supporting open source software they have no control over is a significant increase in relative support and operations cost. Amazon seems to be approaching this by making the cost of RDS instances more than EC2 which is a mistake because the real cost is the lost opportunity of engineers spending their time on systems which are more efficient for cloud computing – Amazon could charge 3 times an EC2 instance and their engineers would still be better off building technologies for cloud-based systems and not centralized RDBMS-dependent web applications.

Where I feel Amazon has fallen short the most, is that RDS only provides single-instance MySQL support and nothing more. No load balancing, replication, Hadoop integration, or any other form of data abstraction which could make it functional in a cloud computing context. Not implementing these features is a very clear indicator that AWS is focused more on short term revenue generating feature rather than cost effective cloud computing systems or improving the shortfalls of legacy centralized system.

With all this said, I have to consider the possibility of this being a good move for Amazon. I present the potential issues with RDS simply to warn folks from relying on it as a crutch, and, to point out the new direction AWS has veered is into choppy waters. There are several aspects of RDS which will give Amazon insight into correlations among and between the varying systems of data storage and processing – comparing SimpleDB, MapReduce, MySQL, and general resource consumption could shed light onto how their cloud is being used at a higher level than processors and bandwidth. Last, Amazon might be aware that MySQL is a crutch and is putting the service out there as a way to wean folks off of centralized systems.

Yesterday’s release of the RightScale platform introduced two new features that I’m really excited about: the ServerTemplate Library and the use of Machine Tags on servers. (Ooops, I shouldn’t forget the new features for RackSpace, but I’ll talk about those next week.)

We’ve had rather sophisticated sharing of ServerTemplates in RightScale for over a year now allowing certain users to share ServerTemplates, RightScripts and other design artifacts with other RightScale users. This enables us to publish free ServerTemplates to all our users, premium ones to our customers and it also lets ISVs on our platform publish ServerTemplates for free or for pay to their users and customers. In addition, each of the design artifacts is versioned such that users who have launched servers with a ServerTemplate last year can still launch new servers with exactly the same version of that ServerTemplate.

A result of all this publishing, sharing and versioning is that there’s a lot to choose from. So much that drop-down menus have become really unwieldy and this is where the new library comes into play. In the past, when adding a server to a deployment one had to find the correct ServerTemplate from the list of all available templates in the RightScale system. Now this has become a two-step process where you first import the ServerTemplates of interest from the library into your account and then only the imported templates are shown in all the drop-down selection menus. Separating the library import/export step will also allow us to significantly upgrade the experience browsing all the design artifacts in the library over the coming releases, stay tuned…

We introduced Flickr style machine tags recently and we’re expanding their use with this release. One of the really exciting new features is that servers now have tags and we’ve integrated the tags with the routing of messages between servers, with Chef (via the RightLink agents) and with the UI. All this is still in alpha but it’s starting to take shape. Our first real use-case is the registration of application servers with load balancers. The way it works is that when a load balancer comes up and is ready for operation it adds a “loadbalancer:lb=www” tag to say “I’m a load balancer for the www vhost”. When an app server starts up, it requests all servers in the deployment with a “loadbalancer:lb=www” tag to run a Chef recipe that adds the app server to the load balancer rotation. This way, the app server doesn’t need to know which or how many load balancers there are. The tag matching, communication, and running of the Chef recipe are all done by the RightLink agents.

In order to let new load balancers come up when app servers are already running we can do the same tag-location in reverse: app servers announce “loadbalancer:app=www” to say “I’m an app server serving vhost www” and load balancers on start-up can add all app servers to their config by querying for all servers with that tag. For overall resiliency it’s a good idea for load balancers to re-query the set of app servers and to update their config accordingly. This catches race conditions as well as issues where portions of the app servers may be temporarily invisible due to network partitions. The theme here is “eventual consistency” and we’re still evaluating what the best primitives are to support high availability.

You may wonder why the examples above use such long tags and that’s really where machine tags come in. The “loadbalancer:” prefix helps isolate the tags to coordinate the load balancer registration from other tags. Think of “loadbalancer” as being the name of the application or feature that uses these tags, e.g. the load balancer registration. The “lb=www” and “app=www” tag predicate and value can be used to support multiple vhosts. So a load balancer could announce “loadbalancer:lb=www” and “loadbalancer:lb=api” to indicate that it’s load balancing the www and api vhosts. And an api app server then would only query for the “lb=api” tag and it would only announce the “app=api” counterpart.

While all this is happening amongst the servers, the RightScale UI provides access to all the tags, so one can see the servers announce the various tags and one can even intervene and manually modify these tags. We might provide a “don’t touch” notion for some tags, but right now it’s much more important to us to be able to expose all this machinery. As an ops guy there are few things I loathe more than hidden automation that I can’t inspect and override when I need to.

Of course there’s more in the new release than just these two features: more support for RackSpace (monitoring in particular), improved support for Chef, support for new AWS features, and more…

I am sitting at Bangalore airport, having spent the day at nasscom and the previous week in Japan. A little homesick and exhausted.

One thing which hit me in Japan and here is the number of people talking about cloud. At first I dismissed it as people following the buzz. But it seemed to be deeper than that.

One thing very different about India and Japan is that they are phone centric. When people start a company in India they think mobile, not PC. In this world cloud means VAS or value added services on the mobile network.

Those services have always lived in the cloud. Imagine a headline like “Your phone will backup your computer to the cloud”. It is the complete opposite of what we would think in the US “your computer will backup your phone”.

Phone based services have always lived in the cloud. As netbooks and things like kindle which connect to the phone network by default OR if the connectivity is provided by the same companies the notion of cloud based services changes.

So we may find in a copuple of years that the cloud is a telco or mobile operator thing with countries like Japan and India way ahead in consumer adoption while the US is ahead in enterprise adoption.

Wonder which one will be bigger?