What if I have been in NYC, London and SF in the same day?

When you analyzing a website performance, it's always a mystery what is the performance issue root cause (well at least till we dive in and reveal it :-).

A common issue in this analysis is what is the network round trip time (RTT) in the play, and who is to blame for. Usually we start with following measures:

1. Analyze the website from our offices. This test includes both Wireshark analysis to analyze the network RTT and FireBug to understand website behavior.
2. Analyze the website performance in the hosting environment by running a terminal services on a server (or the appropriate measure in Linux or UNIX based environment), were RTT should be zero.
3. Finally, we usually connect the site from another site around the world. This is done usually by connecting a server in another hosting environment (booting a server for a hour using Amazon AWS is a great solution for that).

These tests are usually a good method in order to evaluate the communication layer play in the performance issue. However, testing from remote sites is not always accurate and sufficient. In these cases we recommend using a service name browsermob.com, that provides you a FireBug like analysis from several sites around the world.  

Keep Performing!
Moshe Kaplan. 9AFW6C8B3GKN

Boost Your Website Performance (Front End)

This great Yahoo! article is (almost) everything you need in this field.

Keep Performing,
Moshe Kaplan

Should MySQL backup be equal to system downtime?

I had a short conversation with my mate, Romi Kuntsman regarding a common issue in current systems: the database backup.
Since database systems are very active, the database backup process takes relatively long time and the whole system performance is going down (or in some cases the whole system stops responding and going down...)

Should I backup or Should I don't?
Well, first of all, keep backup your system. You never know when your system will corrupt due to hardware failure, when a hacker will decide to "check" your system, or when the regulator will visit your offices.

So how should I keep my system responding to users?
Well... lets take a look at storage systems: when you have a large storage machine (SAN) you usually do not backup the primary site machine, but rather backup the secondary storage machine by splitting it from the main site (or by using a snapshot of the secondary site). This way your primary machine keeps serving clients with interference, while the secondary machine is taking care of the backup. When backup is finished, the sync between the machines is being restored.

So it works for storage systems, how can it work for databases?
Well lets implement a similar design to your database system:

1. Install another MySQL instance
2. Configure this instance to be a slave of your master database
3. Schedule a job to bring down the sync, backup the slave and bring back the sync
   

And what about the hot backup?
Well, if you still want to maintain high availability during backup (and you should), implement a two slaves configuration, where the first slave is used for high availability and the second for backup.

Keep Performing,
Moshe Kaplan. RockeTier. The Performance Experts.

SQL Server NOLOCK: Should I use or should I not?

Well, the simple answer is NO!

What is NOLOCK?
NOLOCKS enables you to make a SELECT statement while avoiding current locks on the tables by other statements such as DELETE and UPDATE.

Why should you use NOLOCK?
Well, the answer is simple, you have locks in the database, users in your website receive exceptions and errors instead of answers, and your boss is getting nervous. The simple way is just place an extra WITH(NOLOCK) and things seems to be OK:

SELECT field_name FROM table_name WITH(NOLOCK)

Why should you avoid NOLOCK?
If your database suffers from locks, avoiding these performance issues now, will result in larger problems in the future. Your database is a key feature in your architecture, and your should take care of him and not avoid the problems.
Moreover, using NOLOCK does not gurrentee that your users will receive updated and current data which may sensitive when financial or sensitive data is getting into place.

Keep Performing,
Moshe Kaplan.

The 2009 IGT annual event - "The World Summit of Cloud Computing" is almost here

Only 5 weeks left to the greatest cloud event of the year with lectures from NYSE, Microsoft, IBM, Amazon, eBay, Mellanox, CloudCamp, Carmel VC and many more.

The event that is taking place at the "Kibutz Shefayim" in Israel at Dec 2nd and 3rd.

Take a look the event website

Keep Performing,
Moshe Kaplan. RockeTier. The Performance Experts.

* Please notice that the event is organized by the IGT that I'm a board member of and that I have internal info that it's going to be at least as great as last year event.

Service: Hoopoe - GPU Cloud

I posted several notes in the last year regarding the niche market of GPU cloud services and why current cloud computing providers such as Amazon AWS that are based on the XEN and VMware virtual instances (hypervisor based) cannot fit them.

Hoopoe is a new service that is focused on this niche market and provides GPU services in the cloud based on NVIDIA TESLA CUDA devices. This service is integrated with Amazon S3.

Hoopoe will be presented by Mordechai Butrashvily in the next IGT GPGPU Meeting at Nov 10, 2009 14:30-17:00.

To reserve your place, send your contact details to info at grid dot org dot il

Keep Performing,
Moshe Kaplan. RockeTier. The Performance Experts.

Load Balancer: Pay less Do More

When we face a buildup of a new mega system that needs to handle dozens of Gb/s of traffic, supporting both HTTPS and HTTP and perform advanced logic such as Comet our main obstacle is how to design it to keep performance high and price low.

Well if you are dealing with such a system (I seen several of these in the last few months) you probably already familiar with hardware load balancers, software load balancers, Acceleration servers (SSL Proxies), caching servers, DNS round robin and CDNs. If you are not sure yet what to do with these components and why, stay with us...
However, please notice that the following is a short list of various technologies available in the market with a bottom line, this is not a complete list and each case should be analyzed according to the case (I think I should hire a lawyer next time...). If you want to learn more feel free to read Willy Tarreau article: "Making applications scalable with Load Balancing" which covers the overall aspects of these technologies.

DNS Round Robin: 
Major Pros: Cheap and simple.
Major Cons: Not dynamic (unless you monitor and manually change it) and if a server fails clients will have major problems acquiring new server address.
Bottom Line: Use it only when servers are for sure up and running (meaning that every DNS node is HA). Very useful to balance several data centers worldwide.

Hardware Load Balancers:

Major Pros: High throughput (10Gbs and counting) low latency and built in HA using VRRP.
Major Cons: Price ($$$).
Bottom Line: Use it for high throughput low latency load balancing or in other words for layer 3/4 load balancing in the data center gateway to balance the traffic between the servers. Cisco, Radware and F5 are good examples.

Accelerators (SSL Proxies and Compression):

Major Pros: Remove overhead from the application servers and keeping traffic as small as possible.
Major Cons: Another layer in the system.
Bottom Line: Use a stack of Apache+ModSSL servers to encrypt/decrypt and compress/decompress behind the HW load balancer keeping cost low and performance high.

Software Load Balancers:

Major Pros: Low cost.
Major Cons: Slow and low throughput.
Bottom Line: Use it when load balancing is almost in the application level (Layer 7) like HTTP Redirect. HAProxy and Apache mod_proxy are good examples.

CDN (Content Delivery Network or Static Files):

Major Pros: Reduce the number of servers in the system
Major Cons: Another layer in the system.
Bottom Line: Use commercial CDN (Amazon S3 and CloudFront are good examples) or dedicated lightweight HTTP like lighttpd to serve static content. It is also recommended to convert dynamic content to static one if possible.

Application Server:

Major Pros: Well we hope at least part of your system is dynamic...
Major Cons: You know, performance boosting is not a joke after all...
Bottom Line: Use optimized system from system to application level, including caching and in memory database to server the dynamic part of the application.

Database:

Major Pros: Well we hope at least part of your system is dynamic... (X2).
Major Cons: You know, performance boosting is not a joke after all... (X2).
Bottom Line: If you plan a very large database, we recommend planning for HA and Sharding from day 1. It's not so difficult and it'll save you a lot work and sleepless nights in the future. And yes, you can use commodity databases such as MySQL.

The bottom line: a wise use of each component can lead you to highly available system while keeping your budget low.

Keep Performing,
Moshe Kaplan. RockeTier. The Performance Experts.