Cloud Computing Design and Best Practices

Today I presented "Cloud Computing Design and Best Practices" in CloudCon, one of the largest cloud events in Israel.

It was a great lecture and I would like to share with you some of the lecture's insights:

1. Assumptions: Don't refer cloud computing as a brave new world. It is, but when you design your cloud system, don’t forget to keep care of the basic rules you had before.
2. Create A Road Map: You probably will be able to turn your system into a 1 Billion users system in day one. Therefore, design your road map and understand how you will reach your target. In order to get there, you should define the various system parts and you should understand how each of them will be scaled out/removed/replaced in the future to meet your road map.
3. Start Fast: People love success. Your investors love it, Your Marketing guys Love it, Your Customers love it and even Your Development guys do. Therefore, start safe and fast. If your development is great in C#, start with it. If you have an existing software product, start with it as well. It is always better to start with a model based on technologies and products that you are good at, then getting into an adventure that you cannot control its risks.
4. Minimize Costs: After you decided to take the fast track, you should control your costs, and most important: your growth costs. You should go over your business plan and turn it into technical requirements plan and find your bottlenecks. Based on these bottlenecks, define solution for each of them: if you are in the online ads market, take care of your impressions module; if you are in the video business, take care of your video processing module. Why? In the viral and online business the growth is exponential. Therefore, cost growth is exponential, and you have to take care of it before your budget will run away.
5. Best Strategies:
1. Scale out: think of sharing nothing, understand how to take each server and split it to infinite number of servers. If a larger server procurement is your best solution, you probably in the wrong direction.
2. Sharding: Data is usually the largest obstacle for scaling out, as conservative designs concentrate the data in a single place. If you have a similar case, take the path the giants already taken: Shard your database either if it is MySQL or SQL Server (read my Best Seller Sharding Post) 
3. In Memory Database: In Memory is X5-X10 faster than on disk. Therefore, analyze your system, and understand what you can do without going to disk. In case you may be able avoid neglected number of transactions, use this technique to cut your costs.
6. Refactor on the Run: as a player in a growing business, you don't have the option to rest. A 100 users system is different than a 100M users system, and as the system grows, smaller modules that were neglected in the first phase will become more important in terms of bottleneck, cost or business sensitivity. Your way to handle it should be refactor the system, step after step to meet the business goals.
7. Define the Exit Strategy: You should always remember that your cloud operator is still a vendor and your best partner in the early days may become an obstacle when you become a giant. Therefore, choose carefully your cloud provider tools. I would recommend you to think twice before you choose propriety data stores like SimpleDB, and if you do: Create your own interface and have an exit strategy when needed.
8. Everybody is using Open Source. What if your organization is not an expert in this field? As written before, you should start fast. There are plenty of cloud providers that support Windows and .Net, and you can get a head start if you will use the technology you are familiar with. When you grow, you may refactor your product and add technologies to remove your bottlenecks such as Erlang for Push or LAMP to handle your most common processes that result in 90% of your costs.
9. Looking for more strategies?
1. CDN: extract your static and streaming content to CDN provider. This move will cut your server and network utilization and will improve your end user experience.
2. Smart Clients: turn your end user client to be sensitive to network failures. If you are using Gmail and seen the loading... label instead of 404 page, you probably understand what I'm talking about (otherwise Google for JQuery).
3. Elastic Growth: if your system pattern usage is not uniform, consider turning on and off some of your instances to keep costs down and meet the spikes.
4. Replication: don't forget to keep your data safe and meet failures. Make sure you do it using commodity hardware and software.
5. Prepare for downtime & upgrades: make sure that you can always go on. Downtimes will come, since in large scale everything happens, even what should have not. Make sure you never really shut down you whole system even when you upgrade it.
6. NoSQL and SQL: choose SQL as a start if you are great at it. But don't neglect NoSQL when you will get larger.
10. Risk Management: You are going to do a bold move ahead, and you should prepare yourself as I mentioned before:
1. Choose your vendors carefully and know your exit strategy. Your Cloud Operator is a provider as well.
2. Hedge costs and take care of your bottlenecks.
3. Stress your system all the way to guarantee you can get to the next level.
4. Think one move ahead and keep aligned to your strategy.
5. Listen to your users feedback. Your business depends on your clients; take care of them.

Cloud Computing Design Best Practices

View more presentations from Moshe Kaplan.

Bottom Line
Working according to these rules can help you reach the 1 Bill users system you were looking for


Keep Performing,
Moshe Kaplan

The Path to the Cloud

You decided to make the big decision and migrate to the cloud. 

What Should You Do Now?

Well, migrating an active system between two hosting providers (and even from your small desktop under your desk), can be risky and may affect the business due to downtime or user frustration. Migration to the cloud that enables a new paradigm can be even more complex since it requires changing concepts and deal with a new brave world..

So How Should We Start?

The first rule is to not think of the cloud as a cloud. You should prepare for a simple migration (think of the cloud provider just as another hosting provider). This way you can minimize the major risk of incompatibility. Follow these steps to accomplish the first step:

1. Prepare your integration environment. If you don't have one, it's time for you to have one. If you do have, it's time to make sure that it's 100% compatible with your production environment. We'll use this environment in order to stimulate the migration.
2. Arrange your DNS records. Migrating to another location will probably require modifying your DNS records to reflect the new location. Some hosting providers do not allow naming for external locations (for example GoGrid is an example for such a provider). So you should verify that you know how to change your DNS records, or migrate the records to another register.
3. Implement a DRP environment. You should have one first in your current provider environment in order to eliminate issues such as access lists and network latency. Why is it so important?
1. Hosting migration is risky. You would like to have easy to roll back in case of non successful migration
2. Data Migration take a lot of time. If your databases (or NoSQL) are large, it will take a long time to migrate them between locations (even using a 100Mb channel, it will take a 3 hours downtime to migrate a 100GB database). The most efficient way to shorten this time will be by using Log shipping between the databases, or implement a replication between 2 NoSQL sites. When you'll choose to perform the migration to the new site, all you'll need to do is stopping the primary instance and turn the passive instance to primary one. Since the data is being replicated between sites all the time, the data migration time slot will be minimized to at most few minutes. 
4. Migrate your DRP environment to the new cloud provider. Start with the integration environment in order to verify there are no network issues in implementing it.
5. Verify the new provider stability. Now that you have servers in the new hosting provider, it's time to verify it stability, its network performance and any other issues that may arise and you were not expected to.
6. Implement a Reverse Proxy Solution. Some of your client will be ready for the migration. They may have old stale DNS records or they might use your IP address for some kind of reason. Using reverse proxy can help your routing traffic from your old location to the new one, minimizing lost traffic and downtime.
7. Perform migration test in the integration environment. You should turn the passive site into active one, and check that user can continue work after the downtime. You should document the process, and if times are not good enough, you should exercise the process. After you are satisfied with results, it's time for prime time.
8. Migrate your production DRP site the new hosting provider. Perform the process in a similar manner to the integration DRP migration.
9. Prepare for The big day. Transform your DRP (or the new hosting environment) into a production one.
10. Verify. Wait a few days and verify the system was stabilized, if so make few steps before closing your old hosting facilities and cut costs:
1. Implement a new DRP site both for the integration and the production environments.
2. Verify that the routed traffic from the old site is neglected.
11. The big day. it's time to shut down your old server, close the site and say bye-bye to your old provider.

Just Scale It
After doing so much work, it's time to decide what cloud items do you want to you. It may depend on the cloud provider offering, and your decision how and if you want to avoid vendor lock in:

1. CDN: if you have major static files traffic, or if you are able to turn some of the traffic into static one (think of Linkedin profiles for example), you could store it in CDN facilities that offer can reduce your web servers traffic and better prepare your for peaks. It also can help you 
2. Queues: Avoid taking care of queues availability and let this internet giants take care of it.
3. NoSQL stores: you could implement open source solutions such as Cassandra or chose the provider key value store.
4. Rational Databases: Some cloud providers offer these days a database as a Service that let you get rid of these log shipping, clustering and care of daily backups. If you are interested, drop me a note and I refer you to some interesting companies.
5. Load Balancing: You could choose using software based load balancers such as HAProxy, or choosing the cloud solution.
6. Monitoring: Some cloud provider offer you a system monitoring, saving you major effort to monitor your servers.

Take It One Step Further

Now that you chosen the tools you just have to make sure your system can scale out:

1. Separate back office services from web interfaces, creating pools of small machines that each can take care of their tasks.
2. Parallelize your back end services, making sure that no single service turns into a future bottleneck.
3. In case of push systems and communication between parties, create a directory system that can route events between different servers.

Bottom Line

If you felt that you are bounded to your current hosting provider, it's time to think again. Using this simple, yet not so short method, you could reduce risks and accomplish the change.

Keep Performing,

Moshe Kaplan

Sharding Again

Click on the post title to read the full post and the comments.


It was a long time since we last discussed Sharding. 
Yesterday the Twitter sharding case study was presented at the Metacafe Knowledge sharing meetup by Gidi Meir Morris. Therefore, I think it is a good reason to refresh our minds.


Hey! Where Are All These Whales?
Twitter turned its web common architecture (that caused it many problems) based on Ruby on Rails web layer and MySQL into a 3 layer system that includes:

1. Web application server called Flapps.
2. Sharding middleware called Gizzard. This layer takes care of database requests and sends them to the target shard. It takes care of replication as well.
3. Old good (not so scalable) MySQL in a horizontal sharding mode.

But hey, it is better taking a look in this great Prezi presentation (use the More>Full Screen to better watch the presentation):

A bird's nest:A primer on FlockDB & Gizzard on Prezi

Bottom Line
Highly scalable databases are becoming commodity. In the Era of Cassandra and Gizzard, Google is starting to lose its competitive edge and high priced Oracle, IBM and Microsoft database are no longer a must for a startup. Will it affect these companies' bottom lines? Only time will tell.


Keep Performing,
Moshe Kaplan

How to Monitor Your Cloud Service Provider

Click on the post title to read the full post and the comments.
The emergence of cloud computing and Everything as a Service in last few years, raised many issues in enterprises that started to outsource part of their services and infrastructure into the cloud.
If in the pre-migration IT managers knew the availability of every piece of their infrastructure, now things are a little bit more complex.


But are Things Really Different?
If you had a COTS ERP or CRM system that you bought from SAP, Oracle or Microsoft, did you really know that happening in the system internals? Or did you focus in measuring the end user experience as well as system metrics like CPU and disk utilization?


What Should We Measure?
In order to decide what we should measure, we should ask first several questions:

1. What will cause me troubles if I'll not measure?
2. Where the money is coming out of?
3. What are you paying for?
4. What is the service interface?

All these questions have specific answers in a common agreement between the parties: the SLA (Service Level Agreement). If it is important enough, you should provide metrics for that, and if you have metrics for that you probably can monitor that using your monitoring system. The importance of SLA monitoring in the Cloud world is highly resembled by Oblicore acquisition by CA in Jan 2010,

What Should We Not Measure?

Usually I try to avoid specific devices monitoring of in the cloud provider since it will not provide me any information and will break the interface of the service (sounds like object oriented design). 

However, if you pay for things that are virtual like redundancy or extra capacity, the service provider should provide measurements for that.

Can You Give Us a Small Example?

Usually, I tend to monitor the user experience. For example, if I use a SMS gateway to send SMS through my system, I'll monitor:

1. The submission time of an SMS request
2. The time to get the SMS in a handset that is stimulated using a cellular modem.

I will perform this test once in a while. E.g. if I send dozens of SMS every second, I will perform measurement every few seconds.

The Bottom Line

Choose your metrics carefully according to the defined SLA and monitor them using your monitoring system. Do not be tempted to over measure in order to avoid breaking the interface,

Keep Performing,

Moshe Kaplan

Open is the New Black (hmm.... Standard)

Click on the post title to read the full post and the comments.

Only few years ago when a technological company was lagging behind in a new niche that was turned into a mainstream, it had a magic move: creating a new standard.

Standard: The Magic Touch
Many times the standard technology was less efficient, less robust and with less users than the technology that was advanced by the market leader. Yet, it had a magic charm: it was open, and most the players in the market could use it without royalties or other fees. That was including #2, #3 in the market as well as many other giants that were not focused in that market, but had to integrate and expand their product support to that technology.
It worked for DEC with Ethernet (vs. IBM's Token Ring); it was great for NetApp that chose iSCSI with 3% market share vs. EMC leading Fiber Channel protocol and it was like a charm for a small silicon valley firm named CISCO that chosen IP.

How do you Standardize Software?
Let say you are a software company that is lagging behind. Anyone said Yahoo! vs. Google in the internet market? Yahoo! had a great idea to scale out using a BigTable like product, but had no deep pockets for that.
Another example? Facebook? What if you had the largest image store publicly available, yet you have only few hundred developers and you need a new scalable database. What do you do?

Open: The New Magic Touch
Well, both Yahoo! and Facebook chose Open Source their infrastructure products creating two of the leading infrastructure products in todays world: Apache Hadoop and Apache Cassandra.

And Cloud, What about the Cloud?
Well it seems that last week Rackspace move: establishing OpenStack.org and contributing major parts of its propriety cloud infrastructure code to this project, is a clear say that being #2 in the IaaS market (see Jack of all Clouds latest report) that is being invaded by major players such as VMware, Google, Salesforce.com and Microsoft is not an easy task even to such a major player in the old hosting business. (If you want to know more regarding this move, see Geva Perry's analysis).

Bottom Line
It seems that the cloud market and mostly the IaaS one is getting to their mainstream phase, where scale is a top factor and community is a major factor in survival.

Would you like to share you opinion? Can you add more information? Feel free to comment!

Keep Performing,
Moshe Kaplan

New Blog Announcement: The VP R&D Handbook Blog

Click on the title to view the full post, comments and add your own comment

Last month I opened a new blog

Are You Leaving Us?
No. The focus of the two blogs are totally different, I'll continue to post here new and good stuff more than once in a while.

So, What is the Difference?
Well it is gonna be a major one:

1. Language: Hebrew blog rather that English one.
2. Location: Microsoft Israel rather than Google US.
3. Content: Soft stuff like people management, R&D management issues and other issues from the VP R&D desk rather than cloud, system architecture and performance and other hard core stuff.

Sounds Cool?
If you are interested in this blog, feel free to enter my VP R&D handbook blog that may turn someday into a book.

Last Words
If you feel that you have something to contribute, feel free to send me a draft. I will be glad to have host posts.

Keep Performing,
Moshe Kaplan

Microsoft is Catching Up with the Internet Industry

Click on the title to view the full post, comments and add your own comment
For a long time I blame Microsoft for not really understanding the Internet and cloud Industry basic need: very large systems that can handle billions of daily transactions with a very low revenue per transaction. Now, after admitting its failure in the mobile industry (see Kin's death earlier this week), Microsoft finally reveals AppFabric after a very long time (I would like to thank my colleague Alon Biran for notifying me regarding it).

What is AppFabric?
The roots of AppFabric are located in the "Velocity" project that was developed in the SQL Server group (a year and a half ago I wrote a Knol about). This project was aimed to provide a key-value store (super hash) that will be used for two main objectives:

1. Instant in memory storage data store (database replacement).
2. Shared web session repository that will enable instant failure between web servers (the only Microsoft in house solution till now was using SQL Server as a shared repository)

After being in various CTP and beta versions for almost two years, AppFabric is finally out as part of the Windows platform rather than being a part of the SQL Server product. You may take a look at my colleague David Chappell's AppFabric white paper for complete set of features and scenarios.

Who are the Current Players?
The major players in the market these days are open source (Every each of the products has its own unique features, please feel free to comment on this post regarding it):

1. Memcached: the #1 product in the market. Its roots are in 2003. It is an open source and Linux based product. Yet, Memcached has .Net client API as well as windows porting of the product itself.
2. SharedCache: a native C# open source product that is a Memcached like.
3. StateServer: commercial native windows product by ScaleOut Software.
4. NCache Express: another commercial native windows product by Alachisoft.

Since AppFabric is provided for "free" as part of the Windows platform, the commercial native .Net products may be hurt from the Microsoft move. However, they may gain more users due to better publicity for this kind of solutions in the Microsoft development community,

Why Did It Take So Long?
It seems that Microsoft had a major cannibalization issue: If you provide a good key-value store then you may reduce the databases size and usage intensity. Thus, the number of high cost database licenses will be significant lower and the bottom line will be smaller. Oracle has the same problem with Coherence that is now part of the Fusion platform.
Taking this product into RTM is a true Microsoft understanding that if it really want to take a significant role in the Internet and cloud industry, it will have to sacrifice more than just few cents in its financial reports bottom line.

Bottom Line
These are very good news that Microsoft is finally provide a solution for a basic need of the Internet and cloud industry. However, while Microsoft was catching up with the Industry 2003 news, new requirements were raised including MapReduce, BigTable and other NoSQL advanced features. How much time will it take this time? Probably only Steve Ballmer has the answer.

Keep Performing,
Moshe Kaplan