Online Games Industry Software Perfromance Boosting Case Study

Hi,

My colleague, Romi Kuntsman, refered me to Big Fish Games software performance boosting case study that was published not long ago by MySQL (now SUN).

MySQL is a significant player in the internet based systems niche which includes the online advertisement, SaaS, games and gaming and many others. The internet based systems were not long ago considered as not so complex systsms as the enterprise systems.
This assumption is not longer true. Internet based system and mostly the top players in the market are facing challenges that require non conservative solutions. Players such as Facebook, Google, Big Fish and CJ are facing the challenge to handle millions of events per second. This is significantly larger than most enterprise software and n-tier architectures based systems were designed to handle with.

Therefore, MySQL based systems that were considered to be a low-mid range systems, are facing these days the most challenging requirements of traffic and load.

MySQL just released a good example for that (I will present several other cases we dealt and dealing with in future posts): BigFishGames.com is a fast-growing website with over 25 million unique customer accounts and over 2.5 million visitors per month and it well financed (over $80M in the last financial round).

Due to their high growth rate (100%-220% per year in last few years accurding to TechCrunch replys), Big Fish faced performance issues, although they used over 40 different MySQL servers...

What did Big Fish do?

1. Database Profiling and Tuning:
Big Fish DBA team used MySQL Query Analyzer. This product provides a consolidated view of query activities and execution details, and it enables quick identification of poorly running queries and tackle the root causes directly in the SQL code. The DBA team caught a "bad" query running 400K times overnight which never showed up in query logs. Since the Query Analyzer uses a Service Agent listening to application queries and performances metrics, the MySQL servers can always be live and operational when being analyzed. There is no need to switch the servers back and forth between on-line and off-line, which eliminates unnecessary risks to server availability and reliability. Moreover, it enables the DBA team gain a real time statistics, rather than make the analysis based on test environment simulations.
After deploying the MySQL Query Analyzer, Big Fish Games tripled (X3) its database performance within three days.

2. In Memory Database and Distributed Cache Layer
Big Fish Games used Memcached as an in memory database soluton and distributed caching layer. By using Memcached, all online queries are performed in the application server memory, and no online queries are processed in the database. Once in a while the Memcached is being syncronized with the database. This solution improved the end user response time, system scalability and database performance.

3. High Performance Servers
Big Fish Games chosen Sun Fire x64 Servers (not so surprising since MySQL is owned by SUN) in three different architectures:

• Sun Fire X2100: for applications which require lots of local disk space but less I/O or CPU speed
• Sun Fire X4100: for applications which demand fast processors but don't need speedy local disk I/O
  
• Sun Fire X4140: for applications where faster local disk I/O via RAID 10 and battery backed up write cache is essential (Memcached servers)
  

Right selection of the hardware enabled 20x in performance by merely replacing an X4100 server with an X4140 machine (and probably they paided for this selection)

I think that this case study is a great example of how a poor performance system can be boosted in factors of X100 and more. Moreover, these cases are not so rare. Many players in the market gain success, and require better performing systems at the fraction of the cost of legacy enterprise systems, without major redesigning their existing servers, and while keeping their current code base.

Best Regards,
Moshe,
RockeTier, the performance experts

The World Summit of Cloud Computing - Presentations and Impressions

Hi,

The IGT just released the presentations and videos of all the two days lectures taken place in the IGT2008 - World Summit of Cloud Computing. It was a great event with great lecturers and presentations.

You can find my lecture, both slides and video. My focus in the lecture was presenting several aspects from our cloud computing field experience and strategic aspects we have in this fields including: market current status, market niches (IaaS, PaaS and SaaS), how to avoid vendor lock-in and how to establish SLA.
We also discussed why enterprise should get into the market (Who will be the next financial giant after Citi age? maybe Paypal or Google?)
We presented how cloud computing enables enterprises gain equity and Op-Ex reduction, and why software performance boosting is a major requirement for organizations that move into the cloud and want to keep their Op-Ex low.

I would like to recommand also several other facinating lecutres like:
* Stevie Clifton from Animoto with their amazing cloud case study: scaling from 50 CPUs to 3500 CPU to meet the digged effect (we have a client who is going to be the next story in this field, a story for another time)
* Dr Owen O'Malley from Yahoo! and Apache Hadoop with a great lecture on Hadoop and test cases (it is really a great tool, and we already using it)
* Paul Strong with a not so short presentation, but with a great discussion about eBay (did you know that eBay reaches 150 Billion events per day) and the requirements of large enterprise from the cloud.

It was a great event and we hope to see you next year in the IGT 2009,

Best,
Moshe,
RockeTier. The performance experts.

GPU Cloud - VMware moves ahead?

Hi,

Since I wrote about GPU cloud a week ago, I met another two software companies which are interested in this issue.

Is there any change in the foreseen future? The following post by Michael Larabel reveals a possible answer: "Tungsten Graphics has been acquired by VMware. Open source graphics technology development will continue as part of VMware's engineering team". VMware is the leading virtualization player in the market (but weaker in the cloud market) and this move will probably lead to "improving the 3D support within virtualized environments".

Probably, a move of a key technology player will enable cloud service providers provide better GPU support in the future.

Moshe,
RockeTier

RockeTier Annual Brunch

PLEASE NOTICE THAT DUE TO WEATHER CONDITIONS, THE EVENT WAS POSTPONED

Hi,

Performance and Cloud Computing posts will be updated soon, in the meanwhile:

RockeTier is having its annual brunch next week on Friday (26/12), and you are welcome to join us. The event will take place Tel Aviv, Israel on Friday the 26/12 at 11:00. Look forward for good food, good booze and great company, More details at Facebook. The favor of a reply is requested by 21/12/2008

P.S. It's also Moshe's birthday :-)

Yours Truly.

Moshe and the RockeTiers

GPU Cloud

Hi,

We are conducting a survey for one of our clients, that is interested in migrating its highly intensive rich media generation system into the cloud.
Since common cloud computing providers lack of a good GPU support, we looked for a niche player in this field.
However, we did not found one. To be more precise, it seems that we are not the only one who looked for this solution.
Seems like a great opportunity for hosting provider, who wants to find its niche in this growing business,

Ciao,
Moshe,
RockeTier

Top Performance JVM and Cloud Computing

Hi,

In the last three months we at RockeTier help a Java based SaaS company boost their software system performance and achieve their next success story.

Java? Can it be boosted?

Java is a byte code compiled language. Therefore, it usually drags several performance limitations compared to fully compiled code (like unmanaged C++).

One of first recommendations in this project was JVM migration to JRockit. BEA JRockit a JVM optimized for the Intel platform, enabling Java applications to run with increased reliability and performance on lower cost, standard-based platforms. According to industry benchmarks (here and here), BEA JRockit leads in performance over other published RISC-based benchmark results. BEA mentioned a boost ratio of 50% to 100%, These is definitely the numbers we are familiar of.

How JRockit achieve these results?

Well they are using several smart techniques out there:

• Optimized Code Generation - They monitor continuously the code, and adapting the JIT compilation according to the dynamic code, rather than using known general statistics. They are using a background bottleneck detector as well, that collects runtime statistics to detect bottlenecks caused by frequently executed methods. These statistics are being used even on run time to deliver on-the-spot improvements

• Seamless Garbage Collection - They eliminate the pauses and operational disruption that garbage collection often causes. They are doing that in several ways in order to meet different types of applications and environments (more details can be found here).

Why is it relevant to Cloud Computing?

x86 CPUs are the de facto standard in Cloud Computing. Therefore, JVM that better utilize the cloud computer CPU, requires less CPU hours to complete a given task and therefore saves you money every hour. Therefore, using optimized JVM (and software) help you reduce your Op-EX when you choose cloud computing.

And the bottom line?

Surprise, you can download the JRockit here, absolutely free out of charge...

I think there is no doubt in here; it is a call for action.

Start up your engines :-)

Moshe,

RockeTier

Gigaspaces XAP vs Oracle Coherence

Hi,

We analyse several Data Grid solutions these days inc. Oracle Coherence and Gigaspaces XAP. During the literature survey we noticed this interesting debate at The Server Side. Take a look in it if you want to know more, while we prepare our analysis.

Best,
Moshe, RockeTier, the performance experts

The World Summit of Cloud Computing - First Impressions

Hi,

It was a very interesting week in Israel, all fully loaded with cloud computing conferences, sessions and seminars. The central event was the World Summit of Cloud Computing, lasting two days with all the top players in the field including Amazon, Animoto, SUN, eBay, Google, IBM, Cloudera, Yahoo!, Hadoop, and RockeTier. We had a booth as well in the conference. Our performance boosting and cloud computing consulting offers drawn a significant attention from market leaders and it seems that we have pretty much work to catch up now.

In the next post I will summerize several notes from the conference,

Best Regards,
Moshe,
Rocketier, the performance experts
Moshe