Your Storage Probably Effects Your System Performance

One of my clients had started getting some performance alerts in their monitoring systems: "IO is too high".
Probably this not something you will be glad to have.

In a quick analysis we found out that the alerts and the high IO originated from servers that were installed in a new data center.
While the actual CPU utilization devoted to IO wait at the old data center was around 25%, in the new data center it was about 75%.

Who is to Blame?
In the new data center NetApp 2240c was chosen as a storage appliance, while in the old IBM V7000 unified was used. Both systems had SAS disks so we didn't expected a major difference between the two. Yet, it was something worth to explore.

Measurement 
In order to verify the source we made a read/write performance benchmark to both systems by running the following commands:

1. Write: dd if=/dev/zero of=/tmp/outfile count=512 bs=1024k
2. Read: dd if=/tmp/outfile of=/dev/null bs=4096k

UPDATE I: You should also try btest or IOMeter as suggested by Yuval Kashtan 
UPDATE II: when using dd, you should better use dd if=/dev/urandom of=/tmp/outfile.txt bs=2048000 count=100 that actually uses random input and just allocate spaces with nulls

Results
On the NetApp 2240 we got 0.62GB/s write rate and 2.0GB/s read rate (in site #2)
On the IBM V7000 unified we got 0.57GB/s write rate and 2.1GB/s read rate (in site #2)
On the IBM V7000 unified we got 1.1GB/s write rate and 3.4GB/s read rate  (in site #1)
That is almost a 100% boost when we used the IBM system!

Bottom Line
When selecting and migrating between storage appliances pay attention to their performance. Otherwise, you may tackle this differences in production. However, differences should be inspect in the same environment. In our case something that was seemed like a storage issue turned into a VM/OS configuration or network issue (exact cause is still under investigation).

Keep Performing,
Moshe Kaplan

Azure Production Issues, Not a nice thing to share with your friends...

Last night I got a call from a close friend.
"Our production SQL Server VM at Azure is down, and we cannot provide service to clients".
A short analysis issued that we are in a deep trouble:

1. The server status was: Stopped (Failed to start).
2. Repeated tries to start the server resulted in Starting... and back to the fail message.
3. Changing instance configuration as proposed in various forums and blogs was resulted in the same fail message.
4. Microsoft claims that everything is Okay with its data centers.
5. Checking the Azure storage container found out that the specific VHD disk was not updated since the server failure.

Since getting back to cold backup would cause losing too much data, we had to restore somehow the failed server.

The chances were against us. Yet, lucky us, we could do that by restoring the database files from the VHD (VM disk) file that was available at Azure storage. 

How to Recover from the Stopped (Failed to start) VM Machine?

1. Start a new instance in the same availability set (that way you can continue using the same DNS name, instead of also deploying a new version of the app servers).
2. Attach a new large disk to the instance (the failed server disk was 127GB, make sure the allocated disk is larger).
3. Start the new machine.
4. Format the disk as a new drive.
5. Get to your Azure account and download the VHD file from Azure storage. Make sure you download it to the right disk. We found out that the download process takes several hours even when the blob storage is in the same data center as the VM.
6. Mount the VHD file you downloaded as a new disk.
7. Extract the database and log files from the new disk and attach them to the new SQL Server instance.

Other recommendations:

1. Keep your backup files updated and in a safe place.
2. Keep your database data and log files out of the system disk, so you could easily attach them to other servers.

Bottom Line

When the going gets tough, the tough get going

Keep Performing,

Moshe Kaplan

Detecting Performance Bottlenecks in Apache httpd Server

“a problem well put is half solved.” ― John Dewey

One of the most important things to well detect a performance issues in a LAMP system is understand the actual processes that causing the issues.

What should we do?
One way to start with is properly configure the Apache logging in a way that we will get fine details about every request performance.

Key Apache httpd Parameters
%B Size of response in bytes, excluding HTTP headers.
%D The time taken to serve the request, in microseconds.
%{VARNAME}e The contents of the environment variable VARNAME.
%T The time taken to serve the request, in seconds.
%X Connection status when response is completed

How to Configure?
Modify your httpd.conf file according to the following article:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy

How do we Analyze it?
Apache log files can be huge (a common case in systems that have some performance challenges).
A useful tool for analyzing the Apache logs, which Cnaan Aviv introduced to me, is GoAccess.
This tool generates reports, statistics and detects errors.

Bottom Line
When done right, solving performance issues is not a black magic. It just need to be done using a well defined method and the right tools.

Keep Performing,
Moshe Kaplan

Changing MySQL Binary Log Files Location to Another Directory

What is the Default?
Usually in most installations, binary log files are located in the MySQL default directory (/var/lib/mysql) just next to the data files.

Why Should I Move the Binary Logs to Another Directory?
Each data modification (INSERT, UPDATE, DELETE...) and data definition (ALTER, ADD, DROP...) statement that you perform in your server are recorded in the Log files.
Therefore, each time you make any of these statements, you actually update both your data files and your log files. The result is high IO utilization that is focused on a specific disk area.
A common recommendation in the database field is to separate these files to two different disks in order to get a better performance.

How to Perform it?

1. Change the log-bin variable in the my.cnf to log-bin=/path/to/new/directory/mysql-bin
2. Purge as many files as you can (PURGE BINLOG...) in order to minimize the number of moved files (see stop 4).
3. Stop the master (service mysql stop).
4. Move the files to the new directory: mv /var/lib/mysql/mysql-bin.* /path/to/new/directory
5. Start the master again (service mysql start).

Bottom Line
Few steps and your server is ready for more traffic and data.

Keep Performing,
Moshe Kaplan

MongoDB and Java

You can find below some hints for initial Java and MongoDB integration

Take a Look at the Requirements

1. MongoDB
2. MongoDB-Java-Driver
3. JDK. If you want use JDK 1.6 or newer, you will get an error like this one: "DBObject cannot be resolved to a type"

Installing Java

yum -y install java-1.7.0-openjdk

yum -y install java-1.7.0-openjdk-devel

Getting MongoDB driver

Compiling Java

javac -d . *.java

java -cp . com/example/mbeans/Main

Define MongoDB Headers in the Code

import com.mongodb.MongoClient;

import com.mongodb.MongoException;

import com.mongodb.WriteConcern;

import com.mongodb.DB;

import com.mongodb.DBCollection;

import com.mongodb.BasicDBObject;

import com.mongodb.DBObject;

import com.mongodb.DBCursor;

import com.mongodb.ServerAddress;

import com.mongodb.*;

Connect to MongoDB

DB _db;

public void init() {

try {

System.out.println("Connecting to mongo...");

MongoClient mongoClient = new MongoClient("127.0.0.1" , 27017);

_db = mongoClient.getDB("display");

System.out.println("Connected to mongo...");

} catch (Exception e) {

System.out.println("Failed Connecting Mongo...");

}

Query the Database (Get the Number of Connections)

CommandResult stats = _db.command("serverStatus");

return Integer.valueOf((((DBObject)(stats.get("connections"))).get("current")).toString());

Bottom Line

Java and MongoDB integration is not too difficult, you just need to do the right thinks right...

Keep Performing,

Moshe Kaplan

DZone's Definitive Guide to Cloud Providers is here!

I was a part of the team that was contributing to the DZone's  Definitive Guide to Cloud Providers, and it's finally here!
If you consider what cloud solution is best for you, take a look at this guide.

Keep Performing,
Moshe Kaplan

How Much RAM Should You Have for MongoDB?

Why RAM is So Important?
RAM is much faster than hard disks. For example, we gained X7 performance boost when we used MySQL In Memory engine rather than InnoDB engine.

Is It Only For Query Caching?
Most of us are familiar with query caching, when results of a query are saved in cache in case it will be called again.
However, RAM can be used to store a copy of your database in the RAM as well. This enables you getting best performance even when you do queries that were not done before. 

So How Much RAM Do I Need?
My Recommendation that your system RAM should be larger than MongoDB data files + index files: SizeOf(RAM) > SizeOf(Data Files) + SizeOf(Index Files).
That way, your files will always be in memory and you will avoid swapping.

What If My Data is Too Big?
In this case you should choose one of the following strategies:

1. Shard your MongoDB between several servers.
2. Tune your queries (this is a good idea in general).
3. Design your databases and collections to support "Data Hierarchy".
4. Choose SSD as your hard disk solution.
5. Consider vSMP Foundation for Memory Expansion from ScaleMP to increase RAM up to 256TB as our readers suggested (P.S don't forget them to ask for the special price :-)

Bottom Line
Careful design will do miracles to your production environment.

Keep Performing,
Moshe Kaplan