Yup, I’m moving to the Mac.  It’s really cool – it has this terminal thingy that you can just type commands into – what an elegant interface!  Simple black and white and it’s real clear that all I need to do is type! 

A windows user asked me how to log into an Amazon EC2 instance using SSH.  I said, “um, well , you, wait, I was just doing this … um.  I’ll look it up and send you instructions.”  Yes, I had already done it, but was confused because I was actually in the middle of setting up secure access for GitHub as well.   I’m going to host some demos for the EventBus (particularly for Apache Pivot) on GitHub since my project on java.net will forever be stuck in CVS even though new projects can use SVN or Mercurial.

So let’s configure this Mac to talk to a new EC2 instance, then allow it to hook up to GitHub too (in a second post).  That might be mildly interesting since there may be a cat of keys involved – OK, maybe not interesting anyway.

To use ssh on an EC2 instance, ensure that when you create the instance, you also create a security group with port 22 open, since that’s what SSH runs over.   You will also create a keypair.  (You don’t need the Access Key Identifiers that you use for command line tools.  The ssh keys for an instance are different from the Access Key Identifiers and are created when you create (or reuse) a keypair for the instance. )

When you create the keypair for the instance you are prompted to download the key file (pem).  This is the private part of the keypair.  If you lose it, you are out of luck, AFAIK.  By default, on my Mac using Firefox, the download, the private part of the pair, gets downloaded to /Users/mbushe/Downloads/<name-of-key-pair>.pem.  The public part of the keypair is held on the server.  You will be forced by ssh to protect your private key, so run:

chmod 700 /Users/mbushe/Downloads/<name-of-key-pair>.pem

..so that only you can access it on your local box.  I’m going to assume you are connecting as root, but a good image will have you connect as another user, set up special.  (OTOH, sometimes you have to log in as root.  For example, if you are using a Content Management System that opens the CIFS port, you need to log in as root since it’s under 1000.)  The supported Alfresco EC2 image will boot you if you log in as root and ask you to in as ubuntu.

To connect using ssh as root, do the following (I’m giving all my info since I’m trashing this instance and key anyway):

ssh -i /Users/mbushe/Downloads/my-key-pair-name.pem root@ec2-67-202-0-107.compute-1.amazonaws.com

You will get this:The authenticity of host ‘ec2-67-202-0-107.compute-1.amazonaws.com (67.202.0.107)’ can’t be established.
RSA key fingerprint is 24:e6:b6:62:fd:f3:54:c0:5f:86:06:97:8d:b7:d4:4f.
Are you sure you want to continue connecting (yes/no)?

At this point, you are connecting with something out there in the world that has a public key with a certain fingerprint that’s displayed.  Some nafarious teenager in your neighbor could be sitting on your wireless network and be running a DNS server for you that points you to his machine instead of yours, or someone scarier anywhere between you and what you are connecting to.  My neighbors likely haven’t hacked into my amazon box and stolen the public key though.  Can you trust everyone on Amazon’s staff though?  If I wanted to cause trouble, that’s the job I’d be applying for. This is why Bruce Schneier consults on human hacks now.  But to limit the risk, you should connect to your AWS console, click on the instance (not the keypairs), and show the system log.  Likely somewhere near the bottom, you’ll see the keys for the fingerprints:

ec2: #############################################################
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: 2048 66:27:32:9f:05:f7:84:97:53:f4:53:46:67:36:5f:6d /etc/ssh/ssh_host_rsa_key.pub (RSA)
ec2: 1024 00:84:16:fd:4e:5f:7d:8d:07:20:40:1a:e3:36:0b:94 /etc/ssh/ssh_host_dsa_key.pub (DSA)
ec2: -----END SSH HOST KEY FINGERPRINTS-----
ec2: #############################################################

Look at the RSA fingerprint from the system log and see if it matches the output of ssh.  If it does, type yes, get warm fuzzy feelings and continue on.  If it doesn’t type no and trash your instance!  Hopefully you set up your instance via a script you can run again on a new instance.  Isn’t virtualization great?

Between snow, getting some prerequisite scripts and docs a bit too late and various other stuff getting in the way, there hasn’t been too much progress today. I think I have everything set up to launch a complete cluster of Atlas agents in the sky, and get them to attach to the overlay VPN and call home, but the mechanics aren’t quite working yet (I’m drowning in Ruby dependency issues).

Today was supposed to be my last working day of the year (though I’ll be in the office on Monday for a client meeting that couldn’t be moved), so this saga will draw to a halt for the time being. I may get some time to hack away over Christmas, but no promises.

Two small victories:

1. I figured out the command line I need to kill the SSL Elastic Server manager (which conflicts with Atlas in wanting port 4433):
   ps -ef | grep ssl_server | awk ‘{ print $2; }’ | xargs kill -9
2. I also figured out why my security groups between vpncubed-client and vpncubed-mgr weren’t working – I was using public IPs for wget and in vpncubed.conf rather than private IPs – doh!

Hopefully this story will have a quick and happy ending in the New Year.

The multicast woes are now behind me (thanks Dimitriy), and I now have a fabric that spans my home network and EC2. The problem with multicast turned out to be firewall related, and the simple fix was:

/sbin/iptables -I OUTPUT -o tun0 -j ACCEPT
/sbin/iptables -I INPUT -i tun0 -j ACCEPT

Tomorrow I’ll try to get something running on the fabric, and will also take a look at automating the deployment process for members of the fabric.

I didn’t get to spend my full attention on this over the last couple of days, and somewhat as expected I’ve run into trouble with multicast. Right now it seems that whenever I put a node into the VPN overlay network it stops being capable of doing multicast.

I’ll report back once these issues are resolved, and hopefully getting a fabric up and running that spans my home network and the cloud will be a few simple steps beyond.

Follow the below steps to share your owned Amazon AMI with other user.

AMIs can be listed with ec2-describe-images command of ec2 api.

The flag “-o self” is used to list only images owned by us.

Prompt> ec2-describe-images -o self
IMAGE ami-4ac2529b my-bucket/ubuntu-base.manifest.xml 495219955132 available private
IMAGE ami-5ad964c3 my-bucket/windows-server.manifest.xml 495219955132 available private

Launch permissions for an image are set by modifying an AMIs launch permission attribute.

To make the Ubuntu image public, we add all to the launch permission attribute.

Prompt> ec2-modify-image-attribute ami-4ac2529b --launch-permission --add all
launchPermission ami-4ac2529b ADD group all

Now we want to give a specific user access to windows server.

We do this by adding only his AWS account ID to the launch permission attribute.

In this example, we use the short hand forms of –launch-permission and –add.

Prompt> ec2-modify-image-attribute ami-5ad964c3 -l -a 349782532074
launchPermission ami-5ad964c3 ADD userId 349782352074

NOTE: Instead of “-a” we can use “-r” to remove / revoke launch permissions from the user.

Launch permissions on an AMI can be checked at any time by running ec2-describe-image-attribute.

$ ec2-describe-image-attribute ami-4ca2529b -l
launchPermission ami-4ca2529b group all

Just a simpler graphic to explain the pricing from the AWS page.

I’ve known the chaps at Paremus since shortly after they set up shop, and I’ve watched the evolution of ServiceFabric since its earliest days. Since it has all the makings of a killer PaaS offering I thought I’d sharpen up my practical cloud skills by getting it running on EC2.

The first challenge is that ServiceFabric uses multicast to communicate between nodes in the fabric, and this isn’t something supported by EC2 (or any other IaaS that I’m aware of). This isn’t a problem though, as I set up CohesiveFT’s VPNcubed, which supports multicast. It also has the side benefit of allowing me to create a network topology that spans cloud and non cloud machines, so I can throw in some boxes from my home network to try out hybrid configurations. I kept things simple, and set up a single manager for the VPN-Cubed for EC2 Free Edition, which went pretty much as described in the step by step guide.

The next stage was to create some workload, so I used Elastic Server to create an AMI that had Ubuntu 9.04 as the base, along with the VPN-Cubed client, Sun Java 6 and Paremus’s Nimble. Nimble wasn’t there already, but it was a few minutes work to upload the package and enrol it into the build system, which then created and provisioned an EC2 instance for me automatically.

Once the Nimble enabled AMI was up and running I got it connected into the VPN overlay, and started up Nimble with:

./posh -sc "repos -l springdm;add
org.springframework.osgi.samples.simplewebapp@active"

I recommend giving this a go yourself if you have 5 minutes to spare – it’s a wonderful demo of dynamic provisioning.

Once Nimble had done it’s stuff it was then just a question of browsing to the http://nimble-machine-vpn-addr:8080/simple-web-app and I could see that the plumbing was working.

Snags along the way:

• Firewalls – maybe stating the obvious, but it really is crucial to get the right end points defined as being able to talk to each other, and security groups didn’t quite seem to cut it as expected.
• OpenVPN throwing its toys out of the pram over an SSL verification error because the date was wrong on one of my home VMs. This stuff is much easier to diagnose when using OpenVPN straight from the command line (openvpn vpncubed.conf) rather than via it’s daemon.

So, that’s it for day one, a working dynamically provisioned web application running within a VPN overlay network.

For day two I’m moving on to full fat ServiceFabric, and will join battle properly with multicast and VPN binding issues. Wish me luck.

I first came across the concept of a virtual resource market at the Grid Today conference in 2004 when Steve Yatko (who later became my boss) made his keynote presentation on ‘Service Oriented Computing’. Apparently we were not alone in thinking about this convergence of grid computing (as we called it then) and economics, with JP Rangaswami and Sean Park discussing similar concepts back in 2002. A few years later I found myself presenting on the same topic at OGF20, and a short while later Steve, Vlad and I applied for a patent (which still remains pending).

It was only a couple of weeks ago that I was discussing how soon this would come to pass with a friendly cloud startup CEO at IGT2009, and he thought it was some way off. He can join Simon Wardley in the camp that thought this would happen but take longer. I personally was always optimistic that this would happen sooner than later, though I must confess to some recent confusion over Amazon’s capacity management for AWS – just what is ’spare’ capacity, and what happens when Amazon itself is busy (in the run up to Christmas)?

Today saw those questions answered as Amazon announced the introduction of spot instances, which allows Amazon to auction EC2 capacity (in addition to the existing on demand and reserved instance prices). Right now the market is closed[1], and people can’t sell their reserved instances to others, but it’s reasonable to expect that these things will come to pass. I’m not sure at this stage whether the VRM will become a reality first in the Amazon public cloud (and I guess I’m with Simon on thinking that there’s a great opportunity for them there) or in the private data centres of large enterprises (which is the problem we were looking at 5 years or more ago); but as of today we’re a whole lot closer to that vision.

[1] and there’s some speculation about transparency – do different people see different prices?

AWS recently announced a new service: Spot Instances.

Today we launched a new option for acquiring Amazon EC2 Compute resources: Spot Instances. Using this option, customers bid any price they like on unused Amazon EC2 capacity and run those instances for as long their bid exceeds the current “Spot Price.” Spot Instances are ideal for tasks that can be flexible as to when they start and stop. This gives our customers an exciting new approach to IT cost management.

The central concept in this new option is that of the Spot Price, which we determine based on current supply and demand and will fluctuate periodically. If the maximum price a customer has bid exceeds the current Spot Price then their instances will be run, priced at the current Spot Price. If the Spot Price rises above the customer’s bid, their instances will be terminated and restarted (if the customer wants it restarted at all) when the Spot Price falls below the customer’s bid. This gives customers exact control over the maximum cost they are incurring for their workloads, and often will provide them with substantial savings. It is important to note that customers will pay only the existing Spot Price; the maximum price just specifies how much a customer is willing to pay for capacity as the Spot Price changes.

Interestingly, this isn’t a technological innovation but is a major business innovation. The instances they are offering are the same instances offered in the tried and true AWS EC2 system. However, now they can offer these instances at a (presumed) lower price with the caveat that you may lose your instance if the market price for that compute power goes above what you are willing to pay for it.

What strikes me about this is the amazing efficiency of the system. Amazon could (in theory) rent out 100% of their aalable computing power through the EC2/spot instance system. If Amazon needs the computer power back, such as during the Christmas shopping season, they can raise the spot price and reclaim many of the resources. If a third party needs more compute power than is available, they increase their bid and drive up the price.

It should be interesting to see applications built around this model. Protein folding is the obvios example, but I can also see this as very useful for graphics rendering or even mundane tasks such as sending out newsletters.

Following are the 3 easy steps to bundle an Amazon EC2 Instance to an AMI (Amazon Image)

These steps require Amazon EC2 API tools to be installed on your system.

<and private key, certificate and access keys for your AWS account>

Step 1.

Login to the Unix / Linux instance and run the following command to create a bundle

Prompt> ec2-bundle-vol --destination /mnt/bundle-image/ 

 --prefix <target image prefix> 

 --cert <Path to X509 Certificate> 

 --privatekey <Path to X509 private key > 

 --user <AWS Account Number> 

 --exclude <exclude dir if any>

Step 2.

Execute following command to upload the Bundled Image to Amazon S3 Bucket -

Prompt>ec2-upload-bundle 

-b $bucket

-m /mnt/bundle-image/$prefix.manifest.xml

-a $AWS_ACCESS_KEY_ID

-s $AWS_SECRET_ACCESS_KEY

Step 3.

Now that the AMI files have been uploaded to S3, you register the image as a new AMI.

This is done back with the EC2 API tools installed:

Prompt> ec2-register $bucket/$prefix.manifest.xml

Robert McMillan of the IDG News Service is reporting that cyber criminals gained access to an Amazon Web Services (AWS) account, and used Amazon’s cloud infrastructure to manage and run its botnet. Expect more cloud-based attacks such as this one in the future.

The botnet was a Zeus bot (Zbot) variant. The Zeus trojan is a  program that criminals use to gather personal and financial data from its victims.

Hackers that create trojans such as Zeus are becoming increasingly organized and function like corporations, according to a security recent report published by Microsoft. That structure enables regular malware release schedules, and gives criminals the ability to exploit complex vulnerabilities in software–even as operating systems become more secure.

Law enforcement has made some progress toward shutting down the data centers that criminals use to host their infrastructure, but the crooks are seemingly one step ahead, and have now migrated to Web-based services. IDG reports that unnamed law enforcement officials have begun to worry that stolen credit cards could be used to purchase cloud computing services such as AWS.

That’s a given. I hope that cloud providers take action to discover malware on their server, and have the capacity to shut it down before serious damage can be done. They have a responsibility to do so.

Cloud computing seems to be one of the big industry buzzwords.  But what’s all the buzz about?

Well, for one thing, $50/application and a potential $15K in prize money, thanks to Elance and Microsoft… but more on that later.  In addition to crafty gimmicks in which yours truly hopes to take part, cloud computing seems to be catching on in organizations and among professionals seeking to reduce time to market, equipment costs and service outages.  Several major players are on board, including Microsoft (Windows Azure), Amazon.com (Amazon Web Services), Google (Google Apps, AppEngine) and Salesforce.com (Force.com).  There are many other players and I’m not intentionally leaving any off the list – for the detail-oriented (and marketing managers), feel free to add a comment promoting your service of choice.  I don’t mind… really.

I’ve personally dabbled with Windows Azure and Amazon Web Services.  It’s worth discussing both, as they represent fairly different service models.  Amazon Web Services encompasses a number of offerings within the AWS brand (like Mechanical Turk, my personal favorite), but is most well-known for EC2, the elastic compute cloud.  To put it as simply as possible, imagine an unlimited number of servers that you had access to whenever you needed to scale up or down your applications and services.  Essentially, that’s what Amazon offers with EC2.  It’s akin to an infinite supply of ready-to-use servers in a remote data center – you can access the machines remotely, set up software and services and be up and running.  This is often referred to as “infrastructure as a service.”  On the other end of the spectrum, you have Windows Azure, often described as “platform as a service.”  To be frank, I hear that most often from people who are not fond of Microsoft, so I’m not sure how far removed from Amazon’s service Azure truly is.  Many of the services available to users are mirrored; if Amazon has a data storage service, Microsoft’s is comparable, both in terms of functionality and price.  However, I have heard that the encapsulation of some functionality – you don’t create “from scratch” images in Windows Azure – makes it easier to get up and running, while limiting your overall flexibility.

For the past few months, I have been attending the recently organized Amazon Web Services Chicago Meet-up to learn more about AWS and cloud computing.  For those of you in the area, I encourage dropping in for the January meeting.  Everyone is very friendly, the breakfast hits the spot and lots of knowledge is served up.  On the other hand, if you are into Windows Azure, you might check out this promotion that Microsoft and Elance are offering (which I eluded to above… you were patient enough to read everything first, right?).  Yesterday night, I got an email from Elance inviting me to participate in a special project to create a Windows Azure application.  Every successful proposal would receive $50 on completion of a live project – it’s brilliant and everyone wins.  Microsoft has several new developers using and promoting Azure, Elance has a bunch of new professionals seeking projects (not to mention the publicity from this event) and I get $50 – sweet!

Stay tuned – once I get approved, I will share details of my Windows Azure application, which everyone will be welcome to beta test.

Amazon recently announced an important new feature for their Elastic Compute Cloud. Previously, each instance was based on an image that could be a maximum of 10 GB in size. So, each machine you brought up could have a root partition up to 10 GB in size and additional storage would need to be added in other ways. The size restriction alone is somewhat limiting. Amazon has not only addressed that, but given users some other very powerful abilities.

Now, you can define an image in an EBS snapshot. That means the size of your root partition can be as large as 1 TB. Yes, that’s 100 times larger than the old 10 GB limit. Beyond the obvious benefit of having larger images, you can also stop instances. Stopping an instance is different than terminating an instance. The distinction is important because stopping an instance is very much like hitting the “pause” button. It doesn’t take a lot to realize that pausing a running instance and being able to start it up again later is very powerful! Instances tend to boot faster off EBS. As  you might expect, if you create a really large volume for a root partition (like 100s of GBs), it will take longer to come up. That’s just because it takes longer to create larger volumes than smaller ones.

Let’s go further and look at how powerful it is to have snapshots as the basis for images. By having a snapshot that you can create EBS volumes from, that means you can mount a volume, based on your snapshot (which represents your image) and make modifications to it! This is immensely helpful when trying to make changes to an image. Previously, it was somewhat more awkward to modify an image. You actually had to boot it up and run it. But now, even if there is an error that prevents proper running, you can access the image storage and make changes. Very useful!

Of course judging by the number of public AMIs out there, there are a great number of images backed by S3 that people will want to convert. Towards this end, I came up with a script to convert AMIs from the old to the new style. Here’s the cliff’s notes version.

Use an instance in the same region as your image to do the following,

• download the image bundle to the ephemeral store
• unbundle the image (resulting in a single file)
• create a temporary EBS volume in the same availability zone as the instance
• attach the volume to your instance
• copy the unbundled image onto the raw EBS volume
• mount the EBS volume
• edit /etc/fstab on the volume to remove the ephemeral store mount line
• unmount and detach the volume
• create a snapshot of the EBS volume
• register the snapshot as an image, and you’re done!

During the private beta for this feature, I created an AMI to handle all of this, so you boot the AMI with a set of parameters and it does the dirty work. The script uses the standard API and AMI tools that Amazon supplies. I’ll roll that out on the public cloud shortly.

Here’s the interesting portion of the script (parsing arguments and setting up environment variable for the tools has been omitted) :

AMI_DESC=`$EC2_HOME/bin/ec2dim |grep $AMI_ID`
MANIFEST=`echo $AMI_DESC | awk '{ print $3 }'`
ARCH=`echo $AMI_DESC | awk '{ print $7 }'`
MANIFEST_PATH=`dirname $MANIFEST`/
MANIFEST_PREFIX=`basename $MANIFEST |awk -F. '{ print $1 }'`

Download the bundle to /mnt

echo grabbing bundle $MANIFEST_PATH $MANIFEST_PREFIX
/usr/local/bin/ec2-download-bundle -b $MANIFEST_PATH -a $ACCESS_ID -s $SECRET_KEY -k pk.pem -p $MANIFEST_PREFIX -d /mnt

Unbundle the image into a single (rather large) file.

echo unbundling, this will take a while
/usr/local/bin/ec2-unbundle -k pk.pem -m /mnt/$MANIFEST_PREFIX.manifest.xml  -s /mnt -d /mnt

Create an EBS volume, 10 GB. This size is used because that is the largest size for an S3 based AMI. Using launch options I show at the end of this article, you can increase that at run time. Notice, the availability zone comes from instance metadata. We must wait till the volume is created before moving on.

ZONE=`curl http://169.254.169.254/latest/meta-data/placement/availability-zone`
VOL_ID=`$EC2_HOME/bin/ec2addvol -s 50 -z $ZONE | awk '{ print $2 }'`
STATUS=creating
while [ $STATUS != "available" ]
do
echo volume $STATUS, waiting for volume create...
sleep 3
STATUS=`$EC2_HOME/bin/ec2dvol $VOL_ID | awk '{ print $5 }'`
done

Attach the volume

INST_ID=`curl http://169.254.169.254/latest/meta-data/instance-id`
$EC2_HOME/bin/ec2attvol $VOL_ID -i $INST_ID -d $EBS_DEV

Here’s where we turn the image into a real volume, using our old friend “dd”

echo copying image to volume, this will also take a while
dd if=/mnt/$MANIFEST_PREFIX of=$EBS_DEV

Mount the volume and remove ephemeral store entry from /etc/fstab. This is required because “Boot from EBS” doesn’t use the ephemeral store by default.

mount $EBS_DEV /perm
cat /perm/etc/fstab |grep -v mnt >/tmp/fstab
mv /perm/etc/fstab /perm/etc/fstab.bak
mv /tmp/fstab /perm/etc/

Then, unmount and detach the volume. We’re nearly there.

umount /perm
$EC2_HOME/bin/ec2detvol $VOL_ID -i $INST_ID

Create a snapshot and wait for it to complete.

SNAP_ID=`$EC2_HOME/bin/ec2addsnap $VOL_ID -d "created by createAMI.sh" | awk '{ print $2 }'`
# now, wait for it
STATUS=pending
while [ $STATUS != "completed" ]
do
echo volume $STATUS, waiting for snap complete...
sleep 3
STATUS=`$EC2_HOME/bin/ec2dsnap $SNAP_ID | awk '{ print $4 }'`
done

Finally, delete the volume and register the snapshot

$EC2_HOME/bin/ec2delvol $VOL_ID
$EC2_HOME/bin/ec2reg -s $SNAP_ID -a $ARCH -d $DESCR -n $MANIFEST_PREFIX

To run your AMI with a larger root partition, use a command like this (which specifies 100GB);
  ec2-run-instances –key <KEYPAIR> –block-device-mapping /dev/sda1=:100 <AMI_ID>
]]> http://ec2lab.com/2009/12/04/aws-raid-0-performance-issue/ Fri, 04 Dec 2009 12:55:06 +0000 webtoprint http://ec2lab.com/2009/12/04/aws-raid-0-performance-issue/

Not all EBS were created equal. Some are slower than the others. Having 2 or 3 EBS as disks in a RAID 0 array will give you performance of the slowest.

Test your AWS RAID0 before deploying real data onto it. You may want to try different EBS drives to arrive to a more or less equal configuration.

Tonight Amazon made a milestone release introducing the ability to boot instances from an EBS volume and stop & start instances. In addition, just a few weeks after announcing their plans to expand AWS to the far east, today they’ve moved west and made a US west coast cloud available. (Do they need a compass?) For the AWS view on all this see Werner’s Blog as well as Jeff Barr’s postings. But one thing at a time…

Amazon introduces US west coast cloud

Almost exactly a year after the first geographical expansion of EC2 to Europe today is the second big step to the west coast. What is notable about the EC2 architecture is that each one of these expansions constitutes a new cloud or “region” in EC2 speak. This means that now in addition to the US-EAST-1 and EU-WEST-1 regions we have a new US-WEST-1 region. Each region operates autonomously from the others in order to provide failure isolation, which has benefits as well as downsides. A major benefit is obviously the redundancy one can get by operating in more than one region or placing DR in a region other than the one used for one’s primary service. The downside is that sharing across regions is not as easy as one might imagine. For example, machine images (AMIs) are not shared, so for each image you’re using in one region you have to copy and re-register the image in the other, and then it has a different id you need to keep track of and reference. We didn’t plan it this way, but our multi-cloud support turns out to be very helpful in managing operations in multiple EC2 regions. For example, in RightScale you can define ServerTemplates that use different images in different clouds, this means that as you update your ServerTemplate it automatically works across clouds and thus EC2 regions.

For redundant operations the comparison between the cloud and DIY datacenters is becoming ever more lopsided. Who can really afford to lose the man-hours, the cap-ex, the time-to-market, and incur the headaches it takes to set-up a datacenter from scratch, even if it’s in a traditional colo? And who can afford to go through all that again to set-up a second or DR site? The ease with which it is now possible to set-up a DR site in the cloud that is a faithful replica of the primary site is really remarkable. And the best is that the second site can be extremely low cost because very little needs to be running there: most of it can be fired up on-demand in the case something happens. If you already have your own datacenter/colo set-up then all hope is not lost. Setting up DR in the cloud is one of the common use-cases we see.

Amazon Instances Boot from EBS

The real sea change about to occur in EC2 is booting from EBS. Tonight’s release includes a ton of new features which build on the recently introduced ability to publish EBS snapshots. Here’s a quick summary:

• instances can boot from an EBS snapshot instead of a traditional AMI, EC2 creates an EBS volume from the snapshot and makes it the root partition
• instances can also boot from an EBS volume, which means that a “boot from EBS” instance can effectively be stopped and restarted later by keeping the volume around and launching a fresh instance from the same volume
• instances can now be stopped and restarted later, which works almost exactly as described in the bullet above except for the fact that the instance id (the i-12345678 number) remains the same
• almost all attributes of an instance can change while stopped, including the instance size (naturally the availability zone is one thing that can’t change)
• EBS snapshots can be registered and published as images, so now we have “traditional images” as well as “EBS images” (I wonder what AWS will call these)
• images can specify snapshots and volumes to be automatically mounted at boot, and they can specify EIPs to be attached at boot, the run-instances API call can add/override these “image defaults”
• instances can be “locked”, which prevents their accidental termination
• instances can be bundled into images using an API call (with shutdown or optionally without)

That’s a long list of features to digest! What’s going on here is that AWS is responding to the needs of enterprise customers who have many ‘legacy’ applications that are not designed to scale out or to play nice with the operations agility enabled by the cloud. It’s for the apps that sysadmins spend weeks setting up and then do their utmost not to touch again. Now they can be installed on an EBS root volume and servers can be launched and relaunched as needed without having to touch the config. Basically this enables the old-school way of managing servers to be applied to EC2.

But these new features are also of great benefit to those operating scalable arrays of servers or web 2.0 web sites. It is now much easier to make changes to a clean server image: mount the image as a volume onto an extra server, edit the software/config on the image (e.g. using chroot and the native packaging system), when happy create an image from the volume and boot a server. Test it out and fix any problems in the original volume. Repeat until happy. If done correctly, this results in clean images that are not polluted by repeated boots and other operations, which is one goal we’ve always pursued with the RightImages we publish.

The stopping and starting of servers can also make development more cost effective. Developers that use dev & test servers can stop them at the end of the day and start them back up when they next need them. In fact, many servers could be set-up to stop by themselves if there has been no activity for a while. (This reminds me that I saw that the three longest running instances visible by RightScale have been running for over 1000 days and that the account they run in has seen no activity since then, except for credit card charges I assume, impressive and scary at the same time!)

Stopping and starting servers can also be abused. For example, it can be used to implement “dumb auto-scaling”: simply stop some servers when the load drops and start them back up later. The good thing is that you don’t end up with fresh servers on start, so they don’t have to self-configure, the bad thing is, well, that you don’t end up with fresh servers, servers come up believing the world hasn’t changed since they were last stopped. I think of this as abuse because it’s easy to forget to update one of the stopped servers when making changes to the system, whether these are changes to the software installed on each server or changes to the rest of the system each server needs to communicate with. In other words, the danger of having a zombie come back to life and create mayhem is high. Better keep a basic amount of hygiene and start with fresh servers.

The Cloud Marches On…

It will be interesting to see how EC2 and its user base continue to evolve. With each release Amazon offers more options. That’s more ways to do interesting stuff, but also more ways to shoot oneself in the foot and more stuff to ‘grok’ to get started. Maybe the most important, though, is that the Boot from EBS features rank very high on the “remove sales objections” scale: not every application is ready for the former EC2 cloud, not every sysadmin is ready for it either, by far not. I have to admit that all this leaves me with mixed feelings. EC2 used to have a simple & clean model, it required some rethinking but that was for the better. It was clear how to deploy highly scalable, highly redundant applications with a high degree of automation. Now that there are 10 ways to skin the proverbial cat it’s much harder to stay on track and to leverage automation. Where early customers needed help figuring out how to operate in the world of EC2’s disposable servers today’s customers need help just navigating through all the options available in EC2 and which to apply to each application or use-case.

Support for the new features and the new US-WEST region in RightScale will become available with our next release, currently scheduled to go live just before xmas. Full support for booting from EBS will take a little longer as it has far-reaching implications. I’m sure that many of our customers will be operating in the new west coast region and that  it may even have some appeal to those in the far east and south pacific as “one step closer” to a local presence.  As always, we’d love to hear your thoughts on the new features, how you’re planning to use them, and how you’d like to see us support them.

Updates:

• AWS now gives each region a little local character: US-WEST-1 is listed as “N. California”, US-EAST-1 as “N. Virginia”, and “EU-WEST-1″ as “Ireland”.
• Nice blog post on some of the mechanics of using Boot from EBS by Shlomo Swidler (but see comment below)
• Some things you can’t do with traditional AMIs: start & stop instance, create image (new way of bundling)
• Some things you can’t do with EBS-based AMIs: dev pay, protect the content of public AMIs (someone can mount the content as a data volume and pull files off it)
• If you plan to create a public EBS-based AMI beware of deleted files: don’t just “delete” files with sensitive data on the volume because they can be “undeleted”, you have to erase the blocks, or better, not put anything sensitive there in the first place

The discussions surrounding the new Architectural Woodwork Standards are heating up. While it’s an imperfect solution, we’ve started a set of wiki pages to capture these discussions, share ideas, and offer support to both the construction and design community.

The pages have a ‘notify me’ tab which allows users to set certain pages or the whole wiki to broadcast an email anytime there is a change made. It’s a charming way to stay up to date about an on-going discussion or technical point.

The wiki also includes a hot link to an online area to submit suggestions for improvement to the Architectural Woodwork Standards directly to the AWI Technical Committee.

To be part of this virtual office, go to this URL: http://tinyurl.com/AWstandards

Hi all,

this post is around AWS, my server farm.

November 25th will be the first anniversary of using Amazon Web Services. After an year, I can say that I changed my mind, but this is the end of the story, so we back to the start.

I’m a computer programmer, I love all things about that, and what I found really interesting is the low level. For example, my favorite language is C (not ++), I like to assemble pcs, create myself the boards, cables  and whatelse. In few words: hardware is my life.

One year ago, I changed my job and I started to work for Reflab, as sysadmin, and they ask me about AWS. The first reaction was not so very good: virtualization of server, no physical contact with the server,  mmmm, too much… I didn’t like it. With a package full of doubts, I made the registration and I started my first server.

The documentation was not so good in really, but merging different sources it was not so hard. I’d like to say that my first server is running today, but is not the true: I had to shutdown that server cos was a small instance for testing. What I can say is that start a server takes 5 minutes.

Now we have 11 server online who are running happy and healthy. The counter of fault is less than 5 who required to restart the server. I thing it is a good result: more the 60% of server runs 100% of time, and the other 40% runs for 99.99% of time. After that, we do not have any problem about connection or disk fault.

The conclusion: we are very enjoy about this choice.

As I say, i’m the one who likes to see his hardware, but this solution is very great.

So… Happy birthday to you, happy birthday to you

The attachment_fu plugin written by technoweenie takes care of uploading and retrieving files using S3 and CloudFront, it has other options too such has saving to the database, local file system, rack space. Anyways in this post I describe how to use the plugin with S3 and CloudFront

1. Create a sample rails project.
2. Install aws-s3 gem, ‘sudo gem install aws-s3‘.
3. Install attachment_fu plugin “./script/plugin install git://github.com/technoweenie/attachment_fu.git“.
4. Assuming that you already have an Amazon S3 account, create a new S3 bucket.
5. Next step is to sign up for the Amazon CloudFront, this is Amazon’s CDN and like any other AWS you pay for what you use, in addition it integrates with your S3 buckets.
6. Now log into the CloudFront’s console and create a new distribution channel that is backed with the S3 bucket created in above steps.
7. Rename amazon_s3.yml.tpl to amazon_s3.yml (it will be in your config folder)
8. Edit the amazon_s3.yml file and fill in the appropriate information.
9. Make a model ‘./script/generate model file_meta_data size:integer content_type:string filename:string’ .
10. Edit the newly created model and add the following lines to it.( ‘has_attachment’ and ‘validated_as_attachment’ are provided by the plugin, there are many other options that you can specify to read more on the options refer to this page).

has_attachment   :storage => :s3,
:cloudfront => true
validates_as_attachment

11. Generate controller ‘./script/generate controller Upload index show new edit create update destroy’.
12. Edit the Upload controller and add the following

def index
  @fileMetaDatas = FileMetaData.all
end
def new
  @fileMetaData = FileMetaData.new
end
def create
  @fileMetaData = FileMetaData.new(params[:fileMetaData])
  if @fileMetaData.save
    flash[:notice] = 'File was successfully created.'
    redirect_to :controller => :upload, :action => :index
  else
    render :action => :new
  end
end

13. Edit the new.erb.html file under the upload controller folder and add the following

<form_for(:fileMetaData, :url => upload_file_path, :html => { :multipart => true }) do |f| >

Upload A File:
<%= f.file_field :uploaded_data %>
<%= submit_tag 'Create' %>
<% end -%>

14. Edit the index.html.erb file under the Uploads controller and add the following

File List
<% for fileMetaData in @fileMetaDatas -%>
<%= link_to fileMetaData.public_filename,fileMetaData.public_filename %>
<% end %>

15. Edit the Routes and add this new route “map.upload_file ‘/new’, :controller => ‘upload’, :action => ‘create’“
16. Run migrations
17. And run the server, that’s it now you browse to http://localhost:3000/upload/new to upload a file.