Load Balancer: Pay less Do More

When we face a buildup of a new mega system that needs to handle dozens of Gb/s of traffic, supporting both HTTPS and HTTP and perform advanced logic such as Comet our main obstacle is how to design it to keep performance high and price low.

Well if you are dealing with such a system (I seen several of these in the last few months) you probably already familiar with hardware load balancers, software load balancers, Acceleration servers (SSL Proxies), caching servers, DNS round robin and CDNs. If you are not sure yet what to do with these components and why, stay with us...
However, please notice that the following is a short list of various technologies available in the market with a bottom line, this is not a complete list and each case should be analyzed according to the case (I think I should hire a lawyer next time...). If you want to learn more feel free to read Willy Tarreau article: "Making applications scalable with Load Balancing" which covers the overall aspects of these technologies.

DNS Round Robin: 
Major Pros: Cheap and simple.
Major Cons: Not dynamic (unless you monitor and manually change it) and if a server fails clients will have major problems acquiring new server address.
Bottom Line: Use it only when servers are for sure up and running (meaning that every DNS node is HA). Very useful to balance several data centers worldwide.

Hardware Load Balancers:

Major Pros: High throughput (10Gbs and counting) low latency and built in HA using VRRP.
Major Cons: Price ($$$).
Bottom Line: Use it for high throughput low latency load balancing or in other words for layer 3/4 load balancing in the data center gateway to balance the traffic between the servers. Cisco, Radware and F5 are good examples.

Accelerators (SSL Proxies and Compression):

Major Pros: Remove overhead from the application servers and keeping traffic as small as possible.
Major Cons: Another layer in the system.
Bottom Line: Use a stack of Apache+ModSSL servers to encrypt/decrypt and compress/decompress behind the HW load balancer keeping cost low and performance high.

Software Load Balancers:

Major Pros: Low cost.
Major Cons: Slow and low throughput.
Bottom Line: Use it when load balancing is almost in the application level (Layer 7) like HTTP Redirect. HAProxy and Apache mod_proxy are good examples.

CDN (Content Delivery Network or Static Files):

Major Pros: Reduce the number of servers in the system
Major Cons: Another layer in the system.
Bottom Line: Use commercial CDN (Amazon S3 and CloudFront are good examples) or dedicated lightweight HTTP like lighttpd to serve static content. It is also recommended to convert dynamic content to static one if possible.

Application Server:

Major Pros: Well we hope at least part of your system is dynamic...
Major Cons: You know, performance boosting is not a joke after all...
Bottom Line: Use optimized system from system to application level, including caching and in memory database to server the dynamic part of the application.

Database:

Major Pros: Well we hope at least part of your system is dynamic... (X2).
Major Cons: You know, performance boosting is not a joke after all... (X2).
Bottom Line: If you plan a very large database, we recommend planning for HA and Sharding from day 1. It's not so difficult and it'll save you a lot work and sleepless nights in the future. And yes, you can use commodity databases such as MySQL.

The bottom line: a wise use of each component can lead you to highly available system while keeping your budget low.

Keep Performing,
Moshe Kaplan. RockeTier. The Performance Experts.

OpSource Cloud: New Cloud Provider in the Hood

Just few weeks after Amazon exposed its Virtual Enterprise Cloud, OpSource, a respected hosting provider that is targeted on the SaaS market is exposing its own solution targeted to the same market. OpSource will reveal its product in a Webinar on Wednesday, October 7, 2009 9:00am PDT / Noon EDT.
Stay tuned,

Keep Performing,
Moshe Kaplan. RockeTier. The Performance Experts.

The Basics of SQL Server Performance

When you first hit a SQL Server that cannot server even another single hit you should start with following basic methods:

1. Analyze the application and understand who are the most massive business processes, and analyze their performance. High odds that the problem lies there.
2. Detect open connections using the Activity Monitor
3. Check objects execution time using the SQL Server reports (of course assuming  you are lucky enough and all SQL Server access is done using stored procedures).
4. Implement profiling and check the queries are most frequent and  most time consuming.
5. Detect dead locks (a good sign for that this is the case is low CPU utilization).
   
6. Detect slowdown source by using the following query  (thanks to Henk van der Valk):
   

-- Uncomment the reset or capture the wait stats before you start the batch to investigate:
-- DBCC SQLPERF('sys.dm_os_wait_stats', CLEAR)
-- DBCC SQLPERF('sys.dm_os_latch_stats', CLEAR)

-- show the sql waitstatistics:
SELECT wait_type
, SUM(waiting_tasks_count) AS waiting_tasks_count
, SUM(wait_time_ms) AS wait_time_ms
, SUM(max_wait_time_ms) AS max_wait_time_ms
, SUM(signal_wait_time_ms) AS signal_wait_time_ms
, SUM(wait_time_ms/NULLIF(waiting_tasks_count,0)) as 'Avg_wait_time_ms per waittype req.'

FROM sys.dm_os_wait_stats
WHERE wait_type NOT IN
(
'SQLTRACE_BUFFER_FLUSH'
, 'LAZYWRITER_SLEEP'
, 'CHECKPOINT_QUEUE'
, 'BROKER_TO_FLUSH'
)
AND wait_type NOT LIKE 'XE%'
AND wait_type NOT LIKE 'PREEMPT%'
AND wait_type NOT LIKE 'SLEEP%'
AND wait_type NOT LIKE '%REQ%DEAD%'
AND waiting_tasks_count > 0
GROUP BY wait_type
WITH ROLLUP
ORDER BY wait_time_ms DESC

After doing these initial steps, you finally gathhered enough data to solve the issue...

Keep Performing,
Moshe Kaplan. RockeTier. The Performance Experts.
moshe at rocketier.com

Infinispan: Open Source is getting into the Data Grid Market

I came across Pawel Plaszczak blog post that describes a new product from JBoss named Infinispan (still in beta), which aims the data grid market (seems like things are getting warmer). Pawel also provided a deep comparison between Inifinspan and leading commercial products such as Oracle Coherence and Gigaspaces XAP.

This product is still in beta, and is missing features that exist in the commercial products, but it's worth to take a look on it,

Keep Performing!
Moshe Kaplan. RockeTier. The Performance Experts.

MySQL Slow Conncections

One of our clients had a performance issue with slow connections to its MySQL database.
The system configuration is MySQL 5.1 on Red Hat, and the application Java/Tomcat based and ORM is done using iBatis.

Several things can be done to solve this issue (make the open connection in few ms instead of 9 seconds) :

1. Use skip-name-resolve in my.cnf file in order to avoid DNS queries that slow down the connections (my.cnf example can be found in http://forums.mysql.com/read.php?11,28690). Another work around can be writing the host resolve in the server hosts file.
   
2. Use MySQL Connector/j 5.1.17 that makes iBatis run faster twice than 5.1.16.
   

That's all for now, feel free to add your recommendations

Keep Performing,
Moshe Kaplan. RockeTier. The Performance Experts.