After a lengthy review process, I finally received the approval to open source Cloud MapReduce – an implementation of MapReduce on top of the Amazon cloud Operating System (OS). It was developed as part of a  research project we have done at Accenture Technology Labs. This shows that Accenture is not only committed to using open source technology, but we are also committed to continue our contribution to the community.

MapReduce was first invented by Google in 2003 to cope with the challenge of processing an exponentially growing amount of data. In the same year the technology was invented, Google’s production index system was converted to MapReduce. Since then, it is quickly proven to be applicable to a wide range of problems. For example, there are roughly 10,000 MapReduce jobs written in Google by June 2007, and there are 2,217,000 MapReduce job runs in the month of September 2007.

MapReduce enjoyed wide adoption outside of Google too. Many enterprises are increasingly facing the same challenges of dealing with a large amount of data. They want to analyze and act on their data quickly to gain competitive advantages, but their existing technology could not keep up with the workload. MapReduce could be the perfect answer to address the challenge.

There are already several open source implementations of MapReduce. The most popular one is Hadoop. Recently, it has gained a lot of tractions in the market. Even Amazon is offering an Elastic MapReduce service which is providing Hadoop on-demand. However, even after 3 years of many engineer’s dedication, Hadoop still has many limitations. For example, Hadoop is still based on a master/slave architecture, where the master node is not only the scalability bottleneck, but it is also a single point of failure. The reason is that implementing a fully distributed system is very difficult.

Cloud MapReduce is not just another implementation — it is not a clone of Hadoop. Instead, it is based on a totally different concept. Hadoop is complex and inefficient because it is designed to run on bare-bone hardware; therefore, Hadoop has to implement many functionalities to make a cluster of servers appear as a single big server. In  comparison, Cloud MapReduce is built on top of the Amazon cloud Operating System(OS), using cloud services such as S3/SQS/SimpleDB. Even though a cloud service could be running on many servers behind the scene, Amazon presents a single big server abstraction, which greatly simplifies a MapReduce implementation.

By building on the Amazon cloud OS, Cloud MapReduce achieves three key advantages over Hadoop.

• It is faster. In one case, it is 60 times faster than Hadoop (Actual speedup depends on the application and the input data).
• It is more scalable and failure resistant. It is fully distributed and there is not a single point of bottleneck or a single point of failure.
• It is dramatically simpler. It has only 3,000 lines of code, two orders of magnitude smaller than Hadoop.

All these advantages directly translate into lower cost, higher reliability and faster turn-around for enterprises to gain competitive advantages.

On the surface, it looks surprising that a simple implementation like Cloud MapReduce could outperform Hadoop. However, if you count in the efforts from hundreds of Amazon engineers, it is natural that we are able to develop a more scalable and higher performance system. Cloud MapReduce demonstrates the power of leveraging cloud services for application design.

Cloud MapReduce has an ambitious vision, so there are many areas that we are looking for help on from the community. Even though Cloud MapReduce was only developed on Amazon OS initially, we envision it will run on many cloud services in the future. For example, it could be ported to Windows Azure, filling a missing capability in Azure that there is no large-scale processing framework at all (Hadoop does not run in Azure). The ultimate goal is to run Cloud MapReduce inside a private cloud. We envision an enterprise would deploy similar cloud services behind the firewall, so that Cloud MapReduce can just build on top. There are already open source projects filling that vision, such as project Voldemort for storage and ActiveMQ for queuing.

Check out the Cloud MapReduce project. We welcome your contributions. Please join the Cloud MapReduce discussions to share you thoughts.

The analysis leaves a lot to be desired, but the conclusions of CloudScaling’s Randy Bias cannot be ignored. And while simple, the math is grounded in reasonable assumptions and backed by independent sources.

His figure puts AWS revenue for EC2 alone at almost $220 million annually. This is real revenue. And we think he estimate for S3 revenues is way low – the bandwidth costs alone are enormous. There are some interesting points raised in the comments. Care to raise any here?

There’s this new show on A&E TV called Hoarders. From the show’s website, each episode “is a fascinating look inside the lives of two different people whose inability to part with their belongings is so out of control that they are on the verge of a personal crisis”. It’s an interesting show about people, who quite simply, have too much stuff. I’ve watched a few episodes, it’s somewhat repetitive, and strangely addictive in the way that only these things can be. Though I never gave the show much thought after I finished watching an episode, a few days ago I had a strange epiphany: I might be a data hoarder.

Here’s the gist of it: I’m afraid of losing data. It’s not that I have a ton of important stuff which I use regularly, in fact much of what I have on hard drive (besides my music and pictures) are things I will probably never actively use again. What I’m actually afraid of is that someday I’m going to want some file (or some specific version of some file) and I won’t be able to find it. Now even if I do have the file, I might not find it due to poor organization and data retrieval systems, but that’s a matter for another blog post. What I’m afraid of is pure, simple data loss: I start working on a project, which I only have one copy of, and something happens to that one copy, whether it be a hard drive crash, or just human error and accidental deletion. And then I have to start all over again, with no real idea of what I did the first time.

Now, thanks to technology I’ve been able to deal with my hoarding instincts, without having dozens of different versions littering my hard drive and doing manual backups every week. At the heart of my system is Git, which lets me keep everything that’s important to me in strict version control. It also lets me easily keep files in sync between different machines, which is a problem I still haven’t completely solved (especially for public machines). By keeping things in sync between three different machines, I have backups in three completely different (as in physically separate) places.

The second thing that keeps my data in control is Amazon’s S3, with JungleDisk. Once a week, this ships all my Git repositories, music, pictures and various software installers to Amazon’s massively distributed storage servers for less than $5 a month. The choice was either this, paying as I go, or buying a terabyte hard disk. Personally I think made the right choice, since my backups are not only safe and secure in a far away place, and I didn’t have to shell out a lumpsum in one go.

Now all this was fine, but lately I’ve been having this urge to record everything. And I mean everything. There are all my tweets and dents which go out into the ether of cyberspace, which I might someday want to have on record. There are all the websites I visit and most recently all the music I listen to and the movies I see. In a perfect world, I would have all my tweets saved to a Git repository and all the DVDs I watch and music I listen to would be instantly ripped and placed in cold storage in an Amazon bucket (or a terabyte disk). And this may not be a good thing, for the reason that I wouldn’t see most of the DVDs for the second time and I have no idea why I would want to save my tweets (or ever look them up).

In the past week I’ve been sorely tempted to actually buy a terabyte hard drive and start manually ripping all the DVDs that I watch. I even went so far as to install Handbrake on my Mac Mini. I’ve been trying very hard to override the temptation with my logic (and laziness). It’s been hard but I’ve been successful so far. Underneath this is perhaps a more important issue: how much data is enough and how safe is safe enough? Keeping my own created data completely backed up in multiple places I think is perfectly acceptable, but I think that ripping all the DVDs is borderline obsessive. It would be an interesting thing too, and might be worth something in terms of street geek cred, but it’s not something that I can seriously see myself doing (and it’s possibly illegal too).

So there you have it, I’m a data hoarder, or at least I have data hoarding tendencies. No, I don’t need an intervention yet, and I don’t need treatment. In fact, I think I’m at the point where I’m reliably saving and backing up everything that I create (that’s more than 140 characters) but not randomly saving everything that I come into contact with. Maybe in another place and time I will actually be saving all my movies as well, but that will probably in terms of actually buying DVDs and having a properly organized collection, instead of borrowing them from the library. For the time being, I trust my digital jewels to Git, three computers and Amazon S3.

[NOTE: This was for Leopard but should work fine in Snow Leopard...]

I absolutely love Time Machine. The simplicity of such a powerful backup system is fantastic; I’ve really grown accustomed to the pretty UI and using Quick Look to find the files I want. I currently have a 500 GB external drive that I use for Time Machine on my Macbook Pro. This is a great solution, but only for the very short periods of time that I actually spend at my desk. I find myself going upwards of 4 days without a backup sometimes, which begins to defeat the purpose of backing up my files in the first place. Every time I look at how many days it has been since my last backup I cringe and wish I had a way to have an Time Machine backup my files online (I could use a Time Capsule, but online is available everywhere not just my network and is cheaper).

There are currently lots of online backup solutions. Two popular options are Carbonite (though their Mac support isn’t quite finished yet) and Mozy, both of which provide unlimited backup storage and bandwith for $49.95 a year and $54.45 (non-commercial) a year respectively. Neither of these options, however, really matched what I wanted. I wanted multiple backups of the same file over time so that I could “go back in time” like I was used to with Time Machine to find the file(s) I wanted, I didn’t just want the most recent version of my file backed up and that’s it. I also really wanted to be able to use Time Machine with the service if at all possible, but I knew that would be asking too much. Or is it…?

After some research I have finally found a way for me to backup my files online using Time Machine (though this feature is technically not supported by Time Machine and by using it you run an increased risk of destroying your data at any time)! The process is pretty simple, we setup Time Machine to allow it use a network drive, setup an Amazon S3 account, get JungleDisk (which also has the ability to do backups itself like Mozy or Carbonite, but we won’t be using it for this purpose) so we can use our S3 account as a network drive, and then tell Time Machine to use our S3 network drive. (Side note: if you’d rather just use a network drive instead online storage, you don’t have to read on.)

Before we continue, I need to re-iterate that doing this is not supported by Time Machine! Though I and others so far have yet to have any problems, you do run the increased chance of loosing all your data that you backup. So with this said, follow this guide at your own risk. If something goes wrong it’s your fault, not mine!

As I said, the first step we have to take is enabling network drives in Time Machine. To do this, open up the Terminal application (found in the Utilities folder within your Applications folder) and type the following EXACTLY, press enter, then quit the Terminal:

 defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

The next step is to setup an Amazon S3 account. Amazon S3 is a pay-as-you-go web storage service offered by Amazon, which means we only have to pay for the storage and bandwith that we actually use. Rather than explaining how to do that here, we will jump to the next step (you’ll see why). Download and install JungleDisk for the Macintosh and follow the simple onscreen instructions, including setting up an S3 account.

If all goes according to plan you should now have a network drive mounted on your desktop, congratulations! The last step is to open up the Time Machine settings under System Preferences and tell Time Machine to use our S3 network drive (pictured below). Voila, we now have Time Machine working with online backups!

Stephen Foskett has been posting some interesting commentary over the last week relating to cloud standards and today discussed the Zend API for PHP.  In previous posts, he’s mentioned the SNIA initiative amongst others.  Have a look at this great post he wrote on why standards aren’t needed.

Now, it’s no secret that I’m keen on the idea of Cloud Computing (and to be more precise, cloud storage in particular), so the concept of evolving standardisation is extremely exciting.  Last year, I discussed RAIC (here and here) – the concept of using multiple clouds to form a redundant repository.

Too Early?

Whilst on the one hand I agree with Stephen that it may be too early for standards to be set, I would also contend that for certain pieces of the cloud storage infrastructure we do need standards, for example security and authentication.  It would be useful to have a consistent authentication model to be applied across cloud storage infrastructures, especially if in the future the ultimate evolution of cloud storage (and for that matter cloud computing) is the ability to dynamically switch workloads and data locations based on service quality (i.e. cost and performance) and availability.

Of course, there will always be the eternal tradeoff between standardisation (which suits the customer) and proprietary interfaces and functionality (which suits the vendor).  Get a customer hooked into proprietary technology and the inertia to change becomes much increased, so even if another vendor does offer a better solution, the cost and effort of change is too great to make the savings/benefits worthwhile.

New Business

Does this mean initiatives like Zend and Cloudloop offer an insight into new business opportunities?  I think they do.  Most, if not all of us will not interface directly with Amazon S3, Nirvanix, Atmos, Rackspace and the others that will spring up.  These companies are infrastructure, not application, providers.  Taking the UK as an example, do I care where or how my electricity is generated or where my gas comes from, as long as it is available when I need it?  No.  Whether the cloud storage infrastructure providers (CSIP) choose to standardise isn’t important.  The future is how easily we can interface into the cloud, and how services such as the following can be easily delivered:

• Create/Retrieve/Update/Delete (CRUD)
• Search
• Index
• Migrate

and of course the influencing factors will be:

• Cost
• Availability
• Performance

Today there are plenty of companies offering services based on cloud storage – exclusively targeting the consumer market or limited business features such as backup.  As things evolve, we’ll see opportunities to move into the Enterprise space.  These will take advantage of extending the data space into the cloud, giving us new and interesting ways of managing data.  Here are some ideas I want to explore in upcoming posts:

• Extending the Global Name Space into the Cloud
• Block-based array tiering and the Storage Cloud
• Using the Storage Cloud for data migration
• Archive, Backup and the storage cloud

Anyone have their own ideas they want to share?

A presentation we prepared recently for a client in the Communications Industry.

We formed an introduction on large Data sets processing with R statistical and data mining software on Amazon EC2 with S3 support. Data distribution and job fragmentation with the use of Hadoop as the MapReduce implementation.

Comments and questions are welcome.

There is now a larger maximum image size of 450×300 for the profiles gallery view. Check out kyle81364 and click on his main profile picture to see the larger image sizes and the new lightbox gallery view. I’ve moved all new image hosting to Amazon’s S3 service to improved the websites performance and enable these larger picture sizes. This should allow everyone a better view of some of the more shy members of DiveIntoThePool.

You have asked for it, and we have listened. We are excited to announce that we have just introduced the pay-as-you-go service option. Instead of paying for a license fee upfront and having to sign up for a cloud service (e.g., Amazon S3, Microsoft Azure), you can now sign up for a single email attachment service with no upfront license fee. You get free software and free upgrade for life and you only need to pay for the service as you go.

Sign up for the pay-as-you-go email attachment service and download the Outlook Attachment software to use the service.

Thanks to Amazon S3, we are able to price the storage cost aggressively at $0.15/GB/month. Compared to the typical enterprise storage cost of $12/GB/month, you are saving two orders of magnitude.

Learn more about our complete cloud email solution.

Gladinet Cloud Desktop mounts cloud storage as local folders* and integrates online applications (Amazon S3, Google Docs, Google Picasa, Box.net, EMC Atmos Online, FTP, WebDav, and more …) with the local desktop.
It is a personal cloud agent, an open online backup platform, and a desktop delivery agent for cloud service providers.

http://www.gladinet.com/

I’ve been using Dropbox for four or five months now and I think it’s great as it is, but it’s going to get even better, I found out yesterday.

What is Dropbox and why is it so great?

On the Dropbox website you can create a user account and download a small client for your computer, whether it’s running Windows, MacOS or Linux. This client creates a folder on your file system under which you can save files, and these files are automatically and securely backed up to the cloud via Amazon’s Simple Storage Service(S3). If you log on to the dropbox website you can browse through your files, and restore old versions of files or deleted files – your ‘undo history’.

If you install the Dropbox client on any other computer and sign in with your user account then it downloads any new or changed files from S3, and any changes you make get saved back to S3 (it only uploads binary changes so it’s pretty efficient as well). So you can have your favourite files synchronised between multiple computers; and you can share files with other dropbox users, too, or even provide public web links.

It’s just so easy to use and I think it’s a great example of what cloud computing should be all about for the end user: a nebulous IT service that just works.

Did I mention that it’s free for your first 2GB of data?

How is the Dropbox service improving?

I’m an iphone user and I’m almost always online, but I don’t want to be downloading files all the time so I’ve been longing for Dropbox to develop a native iphone application that synchronises local copies of my favourite files so I always have them to hand, as it where. And according to an email I received yesterday from Dropbox, that day is coming very soon! They’re also making file sharing more efficient on LANs, but that’s not quite as exciting to me.

On a slightly less positive note they also announced that free users’ undo history will be restricted to 30 days rather than the forever it was before, but that’s good enough for most users and it means more free space for new files.

Hopefully Dropbox will continue to develop their service, which, as a died-in-the-wool Linux lover, I much prefer to Windows Live. In my opinion it’s cloud computing at its simplest and best!

The best and worst thing about working in the technology industry is that you constantly build custom solutions to problems, sometimes quite expensively, and then years later see the same problems get solved through affordable (or free) off-the-shelf products.

Recently I’ve been thinking about solutions that we could have really used during my stint at FOXSports.com, but didn’t exist at the time (2001-2003).

Amazon Web Services (EC2 and S3): It was exciting to support interactive polls during major FOX broadcasts like Super Bowl and World Series, but a huge challenge for the technology organization, particularly in the areas of capacity planning and scaling.  We literally had our hosting provider bring in additional servers for these events, and then decommission them after the events ended.  If we had EC2 we might have been able to scale more flexibly during these events.  Also, we had loads of static content stored in our Oracle database, and served up by our web servers.  S3 would have allowed us to serve this content more effectively while reducing our reliance on a homegrown caching system.

Cloud integration (a la Boomi, CastIron Systems): As a sports website, we had a whole bunch of data and content feeds that we’d get from third parties.  Each feed was a custom integration using different protocols, authentication methods, and required specialized operations support.   If we had solutions like Boomi or CastIron available to us, we could have saved ourselves and our partners a whole lot of development time, and the end result would have been a more operationally supportable set of systems, with more flexibility to onboard new business partners quickly.

Application caching layer (e.g. memcached): We built our own caching platform within our app so that we wouldn’t hit our Oracle database so often with reads.  The cache logic was built in our app and the storage for the cache was an NFS shared volume sitting on a Netapp NAS device.  If we built the site today, we could have leveraged memached (or one of its commercial derivatives) and saved a bunch of dev, testing and debugging time.

Google Analytics: We spent a ton of money on web analytics solutions back in the day.  Google Analytics would have given us much of the same functionality, for free.  Enough said

All of these solutions would have addressed big pain points for our tech team, and consequently for our business as a whole.

Would love to hear any of your war stories related to this topic in the comments.

CloudBerry S3 Backup is a Windows application that automates backup and restore
processes to Amazon S3 secure cloud storage. CloudBerry S3
Backup provides a powerful Backup and Restore program designed
to leverage Amazon S3storage to make your disaster recovery plan simple,
reliable, and affordable.

Disaster recovery planning is often times an afterthought that comes to
light when disaster strikes.

Very seldom do companies fully recover from loss of critical data which could
lead to loss of business.

What is Amazon S3

To use CloudBerry S3 Backup you should create your Amazon S3 account. Amazon S3 is unlimited
secure internet storage that leverages Amazon infrastructure. Amazon S3 is
primarily designed for software developers who want to take advantage of
scalable internet storage in their applications. However, the products like
CloudBerry S3 Backup unleash the power of Amazon S3 and make it available for
less sophisticated consumers.

There are no sign-up or minimum fees. You only pay
for what you use. In many cases your monthly checks will be less than a dollar!
The costs start at $0.15 per GB of data storage a month and ONLY (!) $0.03 per
GB of data transfer until June, 30 2009. The data transfer costs will return
to the normal $0.1 per GB as of July, 1 2009. So hurry up to do the initial
backup!

Check here
for more info on Amazon S3 pricing.

With CloudBerry S3 Backup you can

• Setup your backup process in a couple of minutes
• Strong data encryption protects your data
  from unauthorized access
• Data compression allows you to lower your Amazon S3
  storage and transfer costs
• Scheduling capabilities automate the backup
  process
• Backups are available for recovery 24/7
• Intuitive interface provides easy file and folder
  selection capabilities
• No proprietary storage format. You can access your
  files using other Amazon S3 tools.
• Control bandwidth utilization to do other tasks
  online when backup is running
• Support multiple storage clouds in the future such
  as Microsoft Azure, Sun Cloud Storage Service

This is a list of CloudBerry S3 Backup features.

• Easy installation and configuration
• Scheduling capabilities
• Data encryption
• Data retention schedule
• Secure online storage
• Backup verification
• Alerting notifications
• Indication for successful/failed backup status
• The ability to restore to a particular date
• Backup open files
• Bandwidth throttling
• Support SSL for secure data connection
• Minimize to tray
• Built-in feedback form

Conclusion

CloudBerry S3 Backup is a fully functional backup product that can help computer users ranging from private individuals to small and mid size business leverage Amazon S3 secure and reliable Cloud storage to automate data backup and restore processes. If you are considering using Amazon S3 for data backup purposes it is a good reason to start now and download CloudBerry S3 Backup from CloudBerry Lab website at http://cloudberrydrive.com/

I registered for the beta of EMC’s Atmos Cloud storage offering, Atmos onLine right after EMC World and this week I received confirmation that my registration had been accepted.

So tonight I’ve been looking through the documentation before I start putting together some code to test the service out.

The interesting thing, is (not surprisingly) how initially similar Atmos onLine appears to be to Nirvanix SDN and Amazon S3.  Clearly the devil will be in the detail.

With three major services now available, how will they all compare?  Look out for a post on that soon.  

Finally, I might eventually have the ability to put together RAIC…

This one should probably be filed under “someone should’ve told me this”. I have been using Mozy for about 2 weeks, and I’ve had some hiccups along the way. A couple of weeks I decided my computer files are worth more than $5 a month. With a 1 year old being photographed every day since he was born my pictures are priceless. At that point I decided to do some research about cloud backup solutions. My options where: Carbonite, Amazon S3, Mobile me, and Mozy. At least these were the ones I researched. I went for the price with Mozy, also because I wasn’t sure I could actually back up my attached Fire wire hard drive, which is where I keep all my data. Mozy, I actually signed up when I was still working for Apple, thanks to a customer that mentioned it to me when I offered the option of Mobile me with his computer. I backed up some folders on my computer just to test it out and forgot about them until now. I had the free version so my stuff was sitting there for almost 2 years. First I downloaded the client, and set it up for backing up my pictures, music and documents, this took no time and the intuitive interface was easy to use. I didn’t read any instructions but learned over the period of the last two weeks that for the initial upload of the backup you’ll have to go to the energy saving settings and disable it. Make sure the computer doesn’t go to sleep while uploading. Or the backup will not happen. On a Mac, go to the System preferences, energy settings, then move the slider on “sleep computer after” to “never” and very important, uncheck the “put hard drives to sleep whenever possible” this will stop the transfer over night if your computer is idle. I will give it a full month before I give it my final review but I know it will take several days to upload before I see the real work of this. Especially when I keep adding 1 GB of pictures to my hard drive a week. So far the only problems I’ve had, are things I overlooked. If you want to sign up for a free 2GB account. Requirements:
Mozy currently requires Windows Vista, XP, 2000, or Mac OS X (10.4 and 10.5). If you use it come back and leave your comments, I would like to know how this works for you.

This is our iPhone app with 7 recipes so far. It will eventually have over 50. Comments or suggestions?
AddThis
Posted by Michael Gaylord at 5:06 AM 0 comments
Programming Lessons: iPhone Style

I have been really busy, as many of you have noticed, the last few months juggling my job at ChessCube and building an iPhone application. I have even gone so far as to only work for ChessCube part-time in order to make time to build my application.

The application I have busy building is called 42 Restaurants and it was a concept my girlfriend, Margarét, and I came up with one day while sitting in a sushi restaurant. It goes along the lines of producing a really beautiful iPhone application that showcases recipes from some of the best restaurants in the world. You can read more on the 42 Restaurants website here.

The concept started out simple. My knowledge at the time of the iPhone SDK was very basic and we didn’t want to be too ambitious. The main features would be amazing pictures, simple functionality and great recipes; all the while providing good marketing for the restaurants involved.

Now that we are nearing the end of developing the application, I can let everyone in on some tips to hopefully help their progress with their own application.

1. Read, Read, Read

The main thing that stood me in good stead for learning the SDK was reading. I read everything, from the Apple developer documentation to blogs and online tutorials. The two books that really helped me the most were – The iPhone Developers Cookbook By Erica Sadun and Cocoa Programming for Mac OS X by Aaron Hillegass. The best thing about these two books is that the former teaches you how to build and use components in the SDK without needing to worry about things like code structure and development tools. The latter teaches you the fundamentals of Cocoa programming – even though it contains no coverage of the iPhone SDK.

Understanding the basics is key to using the iPhone SDK.

2. Get To Know XCode

Coming from a development environment from Eclipse, XCode seems at first very different and at times a bit clunky. Now that I have figured out how to use a large portion of it, I can honestly say that I love it. Its simple, fast and it isn’t intrusive while you are coding. One tip which I think most Eclipse & Netbeans programmers will appreciate is to change the Layout Mode. Go to XCode->Preferences->General and change the Layout to All-In-One. All your XCode windows will then be grouped into one place and the interface will alter itself depending on whether your are debugging or coding.

Another important thing to do is download all the SDK documentation to XCode. Take a look at this great blog post to see how to do it without having to use XCode’s built-in documentation downloading mechanisms.

3. Understand How Interface Builder Works

From reading numerous forums on the topic, programmers seem to find it difficult to understand Interface Builder (IB). I was one of them. My distrust of WYSIWYG interface tools led me to building all my view code manually. This, as you can imagine, took me forever until I spent some time trying to understand IB. Now, I pretty much build all my components in IB.

The main concept I had difficulty grasping with IB was what piece of code fitted with what. Each XIB file has two parts, a File’s Owner and a View. The File’s Owner is a class that inherits from UIViewController and the View can be any class that inherits from UIView.

If you want to initialize a XIB file all you need to do is instantiate your view controller with the method initWithNibNamed and then add the view of that view controller as a subview of whatever view you are currently displaying. If you want to do any post instantiation initialization, such as setting text for a UILabel or applying an image to a UIImageView then you override the method viewDidLoad in your view controller.

Don’t forget of course to wire up your IBOutlets and your IBActions in your File’s Owner to your View and to set your File’s Owner to be the custom view controller that you have created in XCode. Otherwise, your view will load and nothing will happen.

4. Don’t Do Cut-and-Paste Coding

This seems to be the number one problem in the development community. There are too many programmers that try to code by searching the Internet looking for solutions to their particular problem. Cutting some sample code from a blog or forum post. Pasting it into their project only to find A) it doesn’t work, B) it is full of bugs or C) badly written.

Understand the principles of what you are trying to do before you go on the web to find a solution. Often you will save time writing the component/algorithm from the ground up and even if you don’t, you will have learnt a lot in the process. The next time you have to build something similar you can use that knowledge.

5. Don’t Be Scared to Rewrite Code

My application was about 80% complete when I realized I was coding around my solution. The architecture that I had designed was firstly, very inefficient memory-wise and secondly, limiting me in terms of the functionality I needed to add. When I started the project I had a very basic understanding of Objective-C and the iPhone SDK, and my solution thus far had evolved into a big, code beast. So I spent 2 days rewriting the core of the application. I lost 2 days but I gained a whole bunch of functionality, improved memory usage and reduced my code base considerably.

6. Lessons

42 Restaurants will hopefully get into the App Store at the end of April. Although the idea seemed simple to start off with, it has grown into a fully fledged recipe application. The sheer amount of design work that Margarét has put into it, makes it in my mind, a masterpiece. Each individual recipe has been custom designed and the application is looking amazing because of it. We both wanted to the application first and foremost to blow the user away with how great it looks and I think we are there.

I hope the code-work on my part will live up to the task. Keep watching this blog and the 42 Restaurants website for news on the launch. We are also on the threshold of starting a round of user testing, so if you have an iPhone and would like to check what we have built so far, post a comment and I will add you to my list of testers.

When putting my head into The Cloud for the first time, I found it very confusing to understand what Cloud Computing means as a term and what it means to me as a developer, specifically

• What are the core concepts of Cloud Computing?
• What are the different types of Cloud Computing?

A short while ago, I happened to dig into this topic because I was giving a talk on Cloud Computing. I’ll post a quick roundup in case anybody is interested.

I like to compare the term Cloud Computing to AJAX. Just like with AJAX, the technology behind Cloud Computing is not completely new or even revolutionary. And just like with AJAX, it actually makes a fair bit of sense to give a bunch of known technologies a new name once you herd them together for a specific purpose, because now whenever someone uses this new term everybody else will know what the heck he is talking about, including

• What technology is being used
• What kind of problems are likely being addressed
• That he is indeed very cool

Or so it should be. I have to say, since Cloud Computing became such a hype and everybody is doing Cloud right now, it doesn’t help in understanding what actually is going on.

Enough talk, you might say, and you’d be right. Let’s get started.

Core Concepts

The most important idea behind Cloud Computing is <Anything> as a Service. Again, the idea of aaS is not brand new. As Rob Conery once pointed out (can’t quite remember where it was), by taking a taxi you’re already leveraging the idea of aaS. Rob called this Car as a Service, and we will quickly look at what aaS means and how we can apply it to cars.

• No aquisition costs, no overhead costs:  You did not have to buy a car to get you to the train station and you do not have to pay a fixed amount of money per month to get the right to call a taxi whenever you need it.
• Variable costs: You only pay for what you consume. In case of a taxi, this is being calculated by distance and time.
• Hidden complexity and maintenance: Some of the major headaches in using a system are being abstracted so you don’t have to care about them. In this case, you don’t care how difficult it is to start that old engine or how often, why and when they have to take it to a workshop to keep it from falling apart as long as you still catch your train.
• Economies of scale: Okay, maybe our example doesn’t work too well for that one, but image that they’re buying lots of taxis and therefore get an amazingly low price and then you as a customer will benefit from that. Right?
• Scalability: In some scenarios, most of the time you need very little resources and suddenly there is a peak and you need lots of it. For example, one day you need to get your whole family (really the whole family: children, parents, relatives) to the train station. You certainly don’t want to buy 4 cars and have them standing around the whole time just to be prepared for this rare occasion? Just get 4 cabs instead of one and you’ll be fine. That’s what I call scalability.

That CaaS comparison worked pretty well for me to show what aaS is about. Then I started to wonder what it means when applied to computers instead of cars.

Different Types

As mentioned above, we can apply this Service principle to virtually anything by just sticking one letter in front of aaS and you’re done. Let anyone else figure out what you meant by it. There are a lot of services in the area of Cloud Computing, that’s one reason why it is such a big mess when you look at it first. I found it helpful to distinguish between 3 different types of Cloud Computing, although there are more and one size doesn’t fit all.

Infrastructure as a Service (IaaS)

In this simplified 3-tier view of Cloud Computing, IaaS is at the bottom of our stack and the most basic layer. It basically provides abstraction over hardware (in fact, IaaS was formerly called Hardware as a Service) by applying the principles of aaS to bare metal. I like to see Amazon Web Services (AWS) as a good example for IaaS. Their services include:

• Elastic Compute Cloud (EC2):  provides you with virtual machines (they are using XEN virtualisation) that are running in Amazon’s datacenter once you fired them up (the VMs, not Amazon). Their EC2 pricing model says you pay them on an hourly basis for that. That’s it! Here is our aaS principle again: No registration fees, no monthly fees, no overhead costs.
• Simple Storage Service (S3): This is a persistent storage service (see, we could already call that Storage as a Service). It goes like this: Once you’ve uploaded your file, they will replicate it so you don’t have to think about backup. You will pay them based on your usage which is, in this case, measured by 1) storage size, 2) time and 3) data transfer (in and out)

They have a lot of other services and are constantly addind new ones, however these two are what I believe to be the most commonly used ones. All of their services are directly accessible as web services (useful for programmatic control) as well as by their management console.

In IaaS, you are admin for every virtual machine you create. You can install every software you need (web server, database and so on), you can also use the programming language of your choice to implement your application you want to host. This gives you great flexibility, but also great responsibility because, at this point, it’s only the hardware you do not have to care about. You still have to patch your OS and all the software you installed, for example. In theory, this gives you infinite scalability in matters of minutes since all resources are virtualized and at every given point in time you could step up and say “give me two more of these”.

Now let’s move on in the stack.

Platform as a Service (PaaS)

PaaS provides more abstraction than IaaS because in addition to hardware, it also provides an abstraction layer for the OS as well as commonly used components like web server, database, load balancer and the likes. One example for PaaS is Microsoft’s Azure platform. Although we’ve been told that Azure is an OS for the cloud, we never actually see this OS because we’re using PaaS.

We’re not admins, we don’t get access to the OS which means we can’t choose what to install and we don’t have to. The responsibility for patching and maintenance for all necessary components is now up to the PaaS provider. For every programming language (Azure currently supports .NET and PHP) a PaaS provider will need another set of tools to run your app. You as developer take your application, load it up to the PaaS provider and say ‘run it’.

See the difference to IaaS? PaaS is a more specialised stack which takes away work as well as decisions. If this doesn’t limit you too much (do they support your programming language?), PaaS will get you started more quickly.

Time for the next step.

Software as a Service (SaaS)

With SaaS, we’re leaving the Cloud Computing domain for developers and enter that for end users. What can Cloud Computing do for end users? All the principles of taxis should apply here too.

With SaaS, you don’t buy the software. You only pay for it when you use it. You don’t have to install it on your machine, it’s running somewhere else and someone else takes care of it. You just use it. Sometimes it’s even free until you need more advanced functionality. Picnik, for example, lets you edit your pictures online, right in your browser. It’s free and gives you enough options for the occasional picture manipulation session, removing the need to install a full-blown image manipulation solution on your machine.

To summarize …

There are 3 manifestations of Cloud Computing:

• Infrastructure as a Service (IaaS)
• Platform as a Service (PaaS)
• Software as a Service (SaaS)

With CC, you get different levels of SEP (Somebody Else’s Problem). Depending on which layer of Cloud Computing you’re using, someone else will take care of necessary hardware, operating system or even a complete application.

The goal is to reduce upfront investments, complexity and time to market and to allow for great scalability and economies of scale using virtualized resources.

Usage in Cloud Computing typically is measured as

• Time (CPU time or real time)
• Data transfer (inbound and outbound)
• Storage (measured in MB or GB)
• Number of transactions (like GET and PUT)

I hope that helped anybody to see Cloud Computing in a less foggy way.

Building on the success of their initial offering, ExpanDrive has released version 2.0 of their Mac client for accessing a wide array of online storage systems.

ExpanDrive is built on MacFUSE, an open-source project that provides the base functionality and SDK for connecting to remote and alternative filesystems. The ExpanDrive developer-alchemists have mixed in a bit of their own Python and Objective-C components to significantly enhance the potency of these MacFUSE-managed filesystems and definitely improved the formula for this major update. Read on to see what’s new in 2.0.

SFTP Overhaul

Not fully satisfied with the status quo, the developers brought the SFTP engine back to the lab and have completely re-engineered it. My tests confirmed that large file copies now transfer in approximately half the time. Traipsing through my cluttered web directories in the Finder is also a less painful experience due to a fully re-worked caching system.

An even more pleasant surprise is how quickly remote edits are detected. The video below shows a text file being edited in both vi on my very remote Linux box and also open locally in TextWrangler. Edits made on the server side show up almost instantaneously now.

FTP/FTPS Support

ExpanDrive has had experimental support for FTP/FTPS filesystems for quite a while and has now made it a first class filesystem in version 2.0. As a daily user of the product, I regularly peruse both their Twitter activity and Get Satisfaction support forums and can attest that they really do listen to users. It’s very obvious that they wanted to make sure that FTP/FTPS support worked under as many conditions as possible before bringing it to their entire user base. If you must use FTP, you should definitely take advantage of the enhanced security in the FTPS protocol if at all possible. Standard FTP transmits both your credentials and all data completely in the clear, making it very easy for malicious users to eavesdrop on your sessions.

GUI Enhancements

Continuing the theme of listening to their users, ExpanDrive has tweaked their GUI to provide more information about the types of drives available and provides the ability to eject volumes from their Drive Manager as well as open volumes in the Finder.

You can now also change the drive icon and use your own icons for ExpanDrive volumes by clicking on the drive icon in the edit dialog:

Live! New! Filesystem! (S3 Support)

Perhaps the most significant enhancement is the inclusion of support for Amazon S3 (s amzn). While inherently not as robust as SFTP volumes, S3-mounted volumes let you use the Finder to manage your buckets (i.e. copy/move/delete files), including setting object permissions via contextual menu. I created a test bucket via S3Fox, used ExpanDrive to move files into it and verified the results back in S3Fox.

Should You Buy/Upgrade?

If you’ve never used ExpanDrive and regularly have to FTP/SFTP files or work with content in S3 buckets, then you really should consider giving ExpanDrive a try (list price is $39.95 for new users). Existing users should also consider upgrading just for the speed and integrity enhancements alone. Couple that with good S3 support and it’s well worth the $19.95 upgrade price (free if you’ve purchased the previous version within 60 days of release).

what are popular services?

Amazon S3, Mosso, Mozy, Carbonite, IDrive, Backblaze, Dropbox, SugarSync, etc.

Which one should i use?

Depends. Encrypting the files before storage is the key. If you want to backup some sensitive files, it will be wise to privately encrypt the file before storing it online. Some of the backup services allow you to have a custom private key using which your data is encrypted. This key (or password) is known only to you and even company backing up your file can’t see the file without knowing the private key.

What is encryption?

Excerpt from wikipedia:

In cryptography, encryption is the process of transforming information (referred to as plaintext) using an algorithm (called cipher) to make it unreadable to anyone except those possessing special knowledge, usually referred to as a key. The result of the process is encrypted information (in cryptography, referred to as ciphertext).

Here are some good links to get you started: wiki, truecrypt, EncFS

I want to share my data across several computers. Is there any service which works better than other?

I like Dropbox and Sugarsync. These services let you install their product on your desktop and sync files placed in certain folders across all computers. Nice thing is that you can access your files from web browser as well. I have heard some good feedback about dropbox. Only thing i didn’t like is that it won’t let you provide private custom key for encryption.

How about Amazon S3 ?

Amazon S3 is cool. It lets you store data on their cloud storage system at very cheap rate. Downside is that it hasn’t introduced any tool for consumers to upload files (but i think it is in works and should be out soon). So you kind of depend on third party tools to manage your files. There are some good ones out there like JungleDisk but you need to pay for them separately. Dropbox and ElephantDrive also use Amazon S3 behind the scenes.

What would be recommend for storing photos and videos?

I will recommend using services like Flickr and SmugMug for photos and videos. But in case you believe that you just want backup facility i will say go with Amazon S3. BTW, smugmug stores photos and videos on amazon s3 infrastructure.

I would like to know more about Online file storage and Encryption. Can you point to some good links?

Sure. Here you go:

http://www.brainonfire.net/blog/remote-encrypted-backup-duplicity-amazon-s3/

http://www.niquille.com/2008/06/15/amazon-s3-tripple-encrypted-true-rsync-backup/

http://jets3t.s3.amazonaws.com/downloads.html