At My Command Unleash Version!

Click on the post title to read the full post and the comments.
 
As you may know I'm a big fun of Continuous Integration, a.k.a CI, (running every trunk commit with some unit tests and integration tests, finding that you ruined the build and earning the blame T-shirt is such a good way to start the morning :-).

Cool things you can do with CI
However, today's CI is much more from just running several tests. These days you can do almost everything with it:

1. Get your latest version from trunk.
2. Build the version.
3. Launch a new virtual machine based on a predefined image or by installing it almost from scratch.
4. Install the version on the new machine.
5. Run smoke tests to provide an instant validation of the code stability.
6. Run nightly builds to carefully check that the version is clean.
7. Run stress tests to validate capacity and long multiple days tests to validate memory leaks.
8. Notify key players with the results.

Cloud Related Cool Scenarios
Cloud computing is basically about obtaining on demand resources through API (the GUI is a fancy stuff that can be useful but is not really a must). Therefore, you can use CI to automate every step in your testing, upload to production and rollback when needed. Some of the scenarios are:

1. Performing Software Upgrade: Launching a new instance, installing the latest software version and creating an Image (AMI in AWS).
2. Launching an entire new integration environment that is identical to the production environment and check its functionality as well as its correspondence with the non functional requirements.
3. Launching a new environment for marketing needs.
4. Managing your new version upload, by creating a script that will prepare new servers, upgrade your database servers (or RDS), add new application and web servers to the load balancer farms, and finally will remove the old server and shut them down.
5. Implement Failover scenarios (DRP) 

Industry Best Practices
According to Amazon (see Jinesh Varia, Technology Evangelist @Amazon Web Services document), CI and automation is the best practice to manage your cloud environment:

1. Create a management server (like CruiseControl or regular CRON jobs).
2. Create a build-test-deploy cycle script using the chosen management server that will generate:
1. New AMI
2. Deploy new servers based on these AMI
3. Add these servers to existing load balancer
4. Attach elastic IPs
5. Generate static files to S3 and CloudFront
6. Generate new queues and notifications using SQS and SNS
7. Remove old servers and unneeded AMIs

Using this method can save you a lot of time in configuration and optimization, as well as many production issues due to human errors.

What CI to choose?
There are many CI products in the market, some of them are open source and other are commercial. Here are some of leading tools, thanks to Alon Nativ advice:

1. CruiseControl: a leading open source product with dozens of plugins and 3rd parties.
2. Apache Continuum: an open source product with many features.
3. Final Builder: a commercial and major tool in the market with great visual interface and many built in scripts and API
4. JetBrains TeamCity: a commercial tool with a built in integration to Amazon AWS.

You can also take a look at RightScale product and services. RightScale provides semi CI in a SaaS manner with many features we discussed before and more. I included here a RightScale video, which is one of the best industry resources regarding this issue:


Bottom Line
Now after going over the details, I recommend you to:

1. Design your cloud environment.
2. Design your baseline AMIs/Images.
3. Design your upgrade, rollback and test-build-deploy scripts based on these basic AMIs.
4. Implement them using your chosen Continuous Integration tool.
5. Roll it out to production.

Keep Performing,
Moshe Kaplan

How Much Money did Amazon Make from that Post?

Click on the title to view the full post
My mate, Dima Shestak, just noticed me regarding a new phenomena in Amazon AWS spot prices in the last few days. The prices are going up...

Is there any connection to my last post: "Solving the Spot Instances Pricing Paradox"? Only time will tell...

Update: a recent graph (June 19, 2010) shows that the trend continues and prices are even touching the on demand prices:

Keep Performing,
Moshe Kaplan