At My Command Unleash Version!

Click on the post title to read the full post and the comments.
 
As you may know I'm a big fun of Continuous Integration, a.k.a CI, (running every trunk commit with some unit tests and integration tests, finding that you ruined the build and earning the blame T-shirt is such a good way to start the morning :-).

Cool things you can do with CI
However, today's CI is much more from just running several tests. These days you can do almost everything with it:

1. Get your latest version from trunk.
2. Build the version.
3. Launch a new virtual machine based on a predefined image or by installing it almost from scratch.
4. Install the version on the new machine.
5. Run smoke tests to provide an instant validation of the code stability.
6. Run nightly builds to carefully check that the version is clean.
7. Run stress tests to validate capacity and long multiple days tests to validate memory leaks.
8. Notify key players with the results.

Cloud Related Cool Scenarios
Cloud computing is basically about obtaining on demand resources through API (the GUI is a fancy stuff that can be useful but is not really a must). Therefore, you can use CI to automate every step in your testing, upload to production and rollback when needed. Some of the scenarios are:

1. Performing Software Upgrade: Launching a new instance, installing the latest software version and creating an Image (AMI in AWS).
2. Launching an entire new integration environment that is identical to the production environment and check its functionality as well as its correspondence with the non functional requirements.
3. Launching a new environment for marketing needs.
4. Managing your new version upload, by creating a script that will prepare new servers, upgrade your database servers (or RDS), add new application and web servers to the load balancer farms, and finally will remove the old server and shut them down.
5. Implement Failover scenarios (DRP) 

Industry Best Practices
According to Amazon (see Jinesh Varia, Technology Evangelist @Amazon Web Services document), CI and automation is the best practice to manage your cloud environment:

1. Create a management server (like CruiseControl or regular CRON jobs).
2. Create a build-test-deploy cycle script using the chosen management server that will generate:
1. New AMI
2. Deploy new servers based on these AMI
3. Add these servers to existing load balancer
4. Attach elastic IPs
5. Generate static files to S3 and CloudFront
6. Generate new queues and notifications using SQS and SNS
7. Remove old servers and unneeded AMIs

Using this method can save you a lot of time in configuration and optimization, as well as many production issues due to human errors.

What CI to choose?
There are many CI products in the market, some of them are open source and other are commercial. Here are some of leading tools, thanks to Alon Nativ advice:

1. CruiseControl: a leading open source product with dozens of plugins and 3rd parties.
2. Apache Continuum: an open source product with many features.
3. Final Builder: a commercial and major tool in the market with great visual interface and many built in scripts and API
4. JetBrains TeamCity: a commercial tool with a built in integration to Amazon AWS.

You can also take a look at RightScale product and services. RightScale provides semi CI in a SaaS manner with many features we discussed before and more. I included here a RightScale video, which is one of the best industry resources regarding this issue:


Bottom Line
Now after going over the details, I recommend you to:

1. Design your cloud environment.
2. Design your baseline AMIs/Images.
3. Design your upgrade, rollback and test-build-deploy scripts based on these basic AMIs.
4. Implement them using your chosen Continuous Integration tool.
5. Roll it out to production.

Keep Performing,
Moshe Kaplan

How Much Money did Amazon Make from that Post?

Click on the title to view the full post
My mate, Dima Shestak, just noticed me regarding a new phenomena in Amazon AWS spot prices in the last few days. The prices are going up...

Is there any connection to my last post: "Solving the Spot Instances Pricing Paradox"? Only time will tell...

Update: a recent graph (June 19, 2010) shows that the trend continues and prices are even touching the on demand prices:

Keep Performing,
Moshe Kaplan

Solving the Spot Instances Pricing Paradox

Click on the title to view the full post
Geva Perry, from Thinking out Cloud, had a great post last December regarding AWS spot instances prices (bidding prices where charge is done based on market current pricing) and why the spot instances pricing should not be higher than on demand instances. The post had a comment thread by James Watters, Guy Rosen, Shlomo Swidler and me discussing why spot instances pricing should not surpass the on demand instances pricing, and why this assumption was broken (for relatively short spikes) in the reality.

NOTE: Please find my previous post regarding spot instances for more information.

Why NoSQL is (not) Really Bad for You?

A few days ago Zeev  Rubinstein, a friend of mine, sent me cool link of Mu Dynamics Research Labs post, that described why they chose CouchDB, a leading Erlang based NoSQL product and what are their major benefits from these products:

1. Low CPU and memory utilization.
2. Map/Reduce support and usage of non normalized data model that fasten their time to market.
3. Avoiding SQL Injections. Since NoSQL systems are not widely spread, there is relatively small number of known attacks against these products. However, please notice that as NoSQL products will turn into mainstream, there will be probably new types of attacks against these products. Yet, if you are an ethical hacker, it can be a new and existing topic that maybe can turn your name to the hottest one in the industry.

Keep Performing,
Moshe Kaplan

When Agile Met Cloud Computing

My mate Alon Nativ just referred me to JetBrains newly introduced TeamCity feature: integration of their Continues Integration (CI) platform with Amazon EC2. This new feature enables on demand instances during the CI process.

What is it good for?
CI tends to have non linear usage behavior: Minor use in the beginning of the project, and massive use in the end of the day and just before releasing the new version.
During these peak hours TeamCity can launch on demand instances, running the build and the tests and shut down the instances when it ends.

 
Bottom Line
It is a fine example of how development teams can keep their infrastructure costs as low as possible, while maintaining maximum flexibility.

Keep Performing,
Moshe Kaplan

Boosting Your Blog Traffic and Performance

Caution: This post does not include technical stuff.
This time I would like to share with you some inside stuff from the other side of boosting traffic performance. This time I would like to share with you several SMO, SEO and user satisfaction boosting methods:

1. Make sure your blog is really in the right language. For example, blogger.com default language is your own one. If you tend to blog in other language, please make sure you changed the blogger.com settings to the appropriate language.
2. Turn your site into a top news site using related links with images widget. Take a look the cool outbrain widget.
   
   Outbrain Thumbnail Widget from Outbrain on Vimeo.
   
3. Place a popular posts widget at your blog's side. If other people thought these posts were good, there is high chance that your next visitor will be interested in them as well. See example and PostRank. Do not forget to register PageRank and subscribe to your blog in order make sure the widget is correctly initialized.
4. Place a recent comments widget. Users content is free traffic generator, and if your visitor is interested in the comment, he may leave another one. See example.
5. Avoid connecting your blog to Google Buzz, Facebook notes, Plaxo and Linkedin. These sites crawl your posts from the blog and prevent users from getting into your original post. According to my experience it can be up to 50% of your daily traffic.
6. Do use posting of links to Linkedin, Twitter (the Linkedin connection is really useful), Facebook, Google Buzz and Plaxo. It will generate high quality traffic to your site.
7. Add social sharing toolbar. Surfers my like your post and retweet/like/etc and will generate more traffic and more credibility. See sociofluid.com solution and WidgetBox and ShareThis.
8. Facebook badges are great to generate returning traffic and credibility. See profile and like.  
9. Add retweets to make it twitter oriented. Add Stumbleupon as well and don't forget digg
10. Place your blog feeder in other sites in order to generate more traffic. See Google example.
11. Use Google Analytics and ClickTale to analyze your users behavior. 
12. Take a look at other additional cool hacks.
13. Write a good stuff :-)

Few more tips

1. June 2010: Update and refresh your Twitter profile. Use twitbacks.com for that. See my Twitter profile as an example.
2. July 2010: Add tools to enlarge your Twitter followers group, that may help you get more traffic later. TwitterCounter is a great example.

Now it's time to start working :-)

Note: There is a great comment thread to this post with interesting insights.

Keep Performing,
Moshe Kaplan

Memories from David Chappell's Lecture or Why does MS Azure still in its Alpha?

Clarification: This post is based on my understanding of the cloud computing market. I may not fully understood the full Microsoft offer, and probably in some aspects my colleague David Chappell and I have different perspectives and opinions.
UPDATE 1: (May 16, 2010) Based on conversation with David Chappell.

I attended yesterday David Chappell's great lecture regarding the Microsoft Azure platform. Except for letting you know that you cannot find a decent WiFi Access in the Microsoft R&D offices, I would like to share with you several technical and product details regarding Azure and my opinion regarding Microsoft Cloud Computing GTM:

1. Both auto scale and getting back to the original state are supported through the API and not automatically . You should do it manually. (UPDATE 1)
2. There is no admin mode: you can install (almost) nothing, no SQL Server, no PHP, no MySQL in service mode and no RDP... or in other words total vendor lock-in. Since HA system requires service mode you must use the Azure stack for HA systems. (Update 1)
3. No imaging or "stopped suspended staged server" mode where you pay only for the storage. You must pay for suspended staged machines (like in GoGrid. unlike Amazon EBS based machines)
4. Bandwidth - few people complain regarding it, because few people use it "there is a lot of room right now...". Please notice that having a lot of room is good. However, only time will tell how Microsoft will
5. Storage - S3, SimpleDB and SQS like storage mechanism that is accessed using REST (no SOAP in the system):
1. Blobs: 
1. EBS like binary data storage
2. Can be mapped to "Azure Drives" that can be used as EBS: NTFS files
3. CDN mechanism Support
2. Tables: 
1. Key value storage (NoSQL)
2. Table>Entiity>Property {@Name;@Type;@Value}. 
3. Don't expect for SQL and it really scalable. You are right, it is sounds just like SimpleDB.
3. Queues: SQS like
1. Used for task distribution between workers
2. Please notice that you should delete messages from the Q, in order to make sure that the message does not appear again (there is a configurable 30 seconds timeout till messages will reappear)
4. Access: grouping data into storage accounts
6. Fabric:
1. Fabric Controller: controls the VMs, networking and etc. A fabric agent is installed on every machine. It is interesting that Microsoft exposes this element and tells us about it, while Amazon keeps it hidden... (UPDATE 1): Please notice that this is a plus for MS, as turns out that Amazon for example does not tell us everything about its infrastructure, and maybe it is time to expose this information.
2. Azure Development Fabric: a downloadable framework that can be installed for development purposes on your premises. Why should you have this mess of incompatibility between development and production environments for the aim of saving few dozens of USD a month? Maybe MS knows... (UPDATE 1): Please notice that saving few bucks is a great idea. However, based on my experience WAN effects and real hosting environment can present different behavior than the one you had on your development premises. Therefore, I highly recommend you to install both your integration and production instances in the same environment.
7. SQL Azure Database:
1. Up to 10GB database. Sharding is a must if you really have data. But hey, if you really had data, you were not here... (UPDATE 1): If you have critical systems with a lot of data, I don't recommend using Azure yet. However, you can find a previous post of mine regarding SQL Server Sharding implementation.
8. Windows Azure Platform AppFabric
1. Service Bus: a SOA like solution that is aimed to connect between intranet web services and the cloud. This a major component that does not exist in other players offering, due to the MS Go To Market (GTM) strategy (see below). If you use other cloud computing environment, you may use this service while integrating it with  your internal IT environment.
2. Access Control

Pricing
Pricing is the #1 reason for using the cloud, and it seems that Microsoft is lagging behind in this

1.    Computing: $0.12-$0.96 per hour. that is X3 AWS spot prices. (UPDATE 1): Please notice that spot prices are not the official Amazon price list. However, it is common method to use it and it is well explained in a previous post of mine.
2.    Storage: $0.15/GB per month + $0.01/10K operations
3.    SQL Azure: $10/GB per month. Yes it is limited to 10GB. (UPDATE 1): Please notice that the Amazon offering is extremely different both in options and in pricing model, so decisions should be taken based on detailed calculation..
4.    Traffic:
1. America/Europe: $0.1/GB in, $0.15/GB out
2. APAC: $0.3/GB in, $0.45/GB out

Target Applications
The Azure plarform best fits the following case scenarios according to Microsoft (based on Amazon experience) David Chappell (David does not work for Microsoft, nor does he speak for them):

1. Massive scale web applications
2. HA apps
3. Apps with variable loads
4. Short/Unpredictable life time applications: marketing campaigns, pre sale...
5. Parallel processing applications: finance
6. Startups
7. Applications that do not fit well to the current organizations:
1. No data center cases
2. Joint venture with other parties
3. When biz guys don't want to see the IT

Microsoft Go To Market Strategy
(Update 1): Please notice that is my analysis regarding Microsoft Go To Market strategy and is based on my market view, and from conversations with market leaders and may not reflect David Chappel's opinion:

1. Microsoft is building on their partnerships and marketing force rather than providing the best technical/pricing solution. They build on these ISVs to expand their existing platform to Azure as part of their online/cloud/SaaS strategy and portfolio. (UPDATE 1): I believe that currently the Azure solution is less flexible than IaaS offers in the market, and with higher cost.
2. MS currently focus in establishing itself as a valid player in the cloud computing market, by comparing itself to Amazon, Google, Force.com and VMware. (UPDATE 1): Microsoft, of course, do not say it by themselves. However, they sponsor respected industry analysts to pass this message
3. It seems that Microsoft was trying to focus in solving enterprises requirements, rather than approaching the Cloud native players: web 2.0 and internet players that are native early adapters... Is it a correct marketing move? Are enterprises really head to this platform, when there are better and more stable platforms? Will they head at all?
4. My 2 Cents: we probably gonna see here another Java/.Net round. However, this time is not just operating system and development environment, but the whole stack from hardware to applications: Microsoft oriented players will try the Azure platform, while others will probably use other solutions. However, this time Microsoft is not in a good position, since (1) major part of the new players in the market prefer Java/Python/PHP/Erlang stacks instead of the .Net stack; and (2) other players already offer Microsoft offering as part of their stack.

(UPDATE 1): Personally I think that Microsoft has a great development force, and it shown in the past that it can get into existing markets with strong leaders (Netscape in the browsers, Oracle in the database, Oracle and SAP in ERP and CRM) and turn in a leader in the market. I believe that this is their target in the cloud computing arena and I'm sure that in the future Azure will be a competent player. However, this is still a first version, and I would not recommend you to run your critical system based on it, as you probably did not run your critical system on the first MS SQL Server version.

Keep Performing,

Moshe Kaplan