Nov 9, 2013

Azure Monitoring. What to Choose?

You are going live with your Azure system and need to know what is happening there?
Cannot sleep at night as your system crashes from time to time?
Need to know whatever your fast growing system will go out of resources in the near future?
If so, this post is for you!

Decisions

In general you have 3 main options:
  1. Open Source Software that is installed by you and managed by you.
  2. Azure built in monitoring that may require some extra charging and tailoring to your business case.
  3. 3rd Party SaaS offering that provides you an End to End solution.
The Open Source Way
  1. Major OSS monitoring tools are available and widely used in the industry. Some of the leading solutions are Zabbix, Nagios and Ganglia.
  2. Software is available w/ no additional cost
  3. You will need to install the servers by yourself (and of course paying for the instances and manage the data).
  4. Implementing an end to end monitoring (inc. user experience) may require extra servers installation on another data center or using a 3rd party service.
  5. Other downsides are well described by CopperEgg.
The Azure Monitoring Way
  1. If you are already invested in Azure, it's built-in, so why not using it?
  2. The service provides nice utilization graphs, and some minimal alerting system (limited to 10 rules). However, there are no pre configured templates, and some metrics are with additional charging.
  3. Basic website availability is provided using web site monitoring.
  4. Yet, there are several gaps such as VM monitoring, SQL Azure and Azure Storage.
  5. The overall impression is that many features are available, but it's far from being a monitoring product, and you will have to do a lot of integration work by yourself..
3rd Party Monitoring as s Service
  1. CopperEgg: Nice company, the their website could look better. More important, Their features looks pretty similar to the OSS solutions with an integrated website response time and availability service. Pricing: $7-9/server|service/month.
  2. AzureWatch: a very similar product to CopperEgg both in features and pricing (although their landing page looks much better). Pricing: $5-8/server|service/month.
  3. New Relic:Probably the most intersting product around, not only it provides an end to end monitoring, but it also analyzes your performance bottlenecks. Their product is being integrated to your code and provides you with production profiling and alerts you regarding any exception. Pricing: standard version that includes the other providers offers is free. Code level diagnostics and inner level profiling will cost you $25-200/server|service/month
Bottom Line
If you are ready to invest time to get your own solution, the OSS products with Azure integration will probably be your preferred solution. Otherwise, I will recommend you to choose New Relic, that seems to provide the best value in the market for your needs.

Keep Performing

Nov 5, 2013

Apache Session Persistancy Using MongoDB

Your app is booming, you need more web servers and you need to serve users and keep their user experience. When you had a single server you just used session for that, but now, How do you keep sessions across multiple web servers? 
Using a session stickiness load balancer is a good solution, but what happens when a web server needs a restart?

Session Off Loading
Many of you are familiar with the concept of session off loading. The common configuration for it is using Memcached to store the session object, so it is accessible from all web servers. However, this solution lacks the features of high availability by replication and persistence.

The Next Step: High Availablability
Offloading web servers sessions to MongoDB looks like a great solution: key-value store, lazy persistency and built in replication and auto master failover.

Step by Step Solution for LAMP Session Off Loading using MongoDB
  1. Install MongoDB (preferably in a replica set with at least 3 nodes)
  2. Install MongoDB driver for PHP: 
  3. Configure the php.ini: extension=mongo.so
  4. Download and integrate Apache session handler implementation using MongoDB.
  5. Configure your MongoDB setting in the module:
    1. Set the list of nodes in the connectionString setting: 'connectionString' => 'mongodb://SERVER1:27017, mongodb://SERVER2:27017',
    2. Do the same to the list of servers (see at the bottom of this post)
    3. Configure the cookie_domain (or just place a null string there to support all): 'cookie_domain' => ''
    4. Don't forget to enable replica set if you are using one: 'replicaSet' => true,
  6. Add the MongoSession.php to your server and require it: require_once('MongoSession.php');
  7. Replace the session_start() function with $session = new MongoSession();
And that's all. Now you can continue using the session objects as you used to create new great features:
if (!isset($_SESSION['views'])) $_SESSION['views'] = 0;
$_SESSION['views'] = $_SESSION['views'] + 1;
echo $_SESSION['views'];


Bottom Line
Smart work can tackle every scale issue you are facing off...

Keep Performing,

Appendix: 
'servers' => array(
array(
'host' => 'SERVER1',
'port' => Mongo::DEFAULT_PORT,
'username' => null,
'password' => null,
'persistent' => false
),
array(
'host' => 'SERVER2',
'port' => Mongo::DEFAULT_PORT,
'username' => null,
'password' => null,
'persistent' => false

)

Oct 13, 2013

MongoDB Index Tuning and Dex

One of the first things when your system is rolled into production (or better when you have your acceptance tests) is checking the system performance performance and tune it.
When we talk about databases, we usually talk about index tuning, and MongoDB is not different.

How to Detect the Slow Queries?
First enable you MongoDB slow queries log. It's easy, just run db.setProfilingLevel(1, 1) from the command line. 

How to Analyze the Slow Queries?
Well, you can use the built in queries (after all the profiling is saved at a MongoDB collection). However, Dex, a new tool from MongoLab, can help you shorten the time to index...
It's a Python based product that can be easily installed and run and provides a list of recommended indexes based on MongoLab best practices.

How to Install and Run?
  1. Install python: yum -y install python
  2. Install pip
  3. Install Dex using pip: pip install dex
  4. Create a log file: touch mongodb.log
  5. Run it: sudo dex -p -v -f mongodb.log mongodb://user:password@serverDNS:serverPort/database. Don't forget to use the various flags to get errors in case of wrong configuration.

Should We Just Copy and Paste the New Indexes?
As you may know every added index results in a performance degragation when INSERT, DELETE and UPDATE statement are performed. Therefore, we should carefuly select the indexes to be added:

  1. We should check that the new indexes do not overlap existing indexes. If there is an overlap, we should remove/unify the indexes definitions (use the .db.collection.getIndexes() method to retrieve the data.
  2. We should check that the queries are frequently used in order to avoid saving few seconds once a month while doing a SELECT, and pay for that at peak times INSERTs.
Bottom Line
MongoDB queries tuning is easier than ever: Just Tune It!

Keep Performing,
Moshe Kaplan

Sep 15, 2013

Puppet, Configuration Management and How to Get into Production Faster...


Puppet or Chef?
You must take a look at the varios comparissons between the tools, but actually we chosen Puppet since it becomes an industry choice (we verified it by searching Linkedin).

What Puppet Version to choose?
I always like to explore the community editions, but PuppetLabs has a very nice offer for their Enterprise Edition (10 free nodes). Feel free to decide by yourself.

How to Install?
You can compile the source, but repositories are probably the easiest way around.
The CentOS 6.X easiest way is: $ sudo rpm -ivh http://yum.puppetlabs.com/el/6/products/i386/puppetlabs-release-6-7.noarch.rpm

How does It Actually Work?
  1. Central Server/Master Repository: the place where defintions are defined. Please notice that some best practices recommend that you actually define the central server as another node in the system.
  2. Nodes: these are your operational servers that gets configuration procedures from the central server.
  3. Providers: perform the process on each machine, create systems abstraction (different OS distros for example) from the user.
  4. Puppet Modules: these are product configuration definitions that can be installed on your server. For example the definition of Riverbed Stringray that we'll discuss later.
  5. Server Configuration files: these are the configurations of each server (for example your web server). These files includes the selection of production that will be implemented and what configuration decisions were taken. For example if you install a loadbalancer on your server: what is the LB algorithm and what nodes are behind the LB.
How to Add a New Product to your Server?
Just add to the node ( in testServer this case) the product definition (class {'stingray':... in this case) and define the various configuration options (stingray::new_cluster in this case):
node 'testServer' {
    class {'stingray':
        accept_license => 'accept'
    }
    stingray::new_cluster{'new_cluster':
    }
    stingray::pool{'Northern Lights':
        nodes       => ['192.168.22.121:80', '192.168.22.122:80'],
        algorithm   => 'Least Connections',
        persistence => 'NL Persistence',
        monitors    => 'NL Monitor'
    }
}

How to implement Elastic Hosts (and server groups w/ the same function)?
If you want to implement elastic hosts that scale to traffic you should use MCollective to define the server groups

Do you need more information?
I found following PuppetLabs presentation to be very useful:


Bottom Line
Deployment of large systems is easier today, and will be even easier in the near future. You just need to choose the right tools.

Keep Performing,
Moshe Kaplan

Sep 12, 2013

MySQL Multi Master Replication

Why Multi Master Replication?
Master-Slave configuration is a great solution when your system is read bound. But what happens if you want to support high write loads, while keeping the option to select the data using a single SQL statement?

Multi Master replication is the solution for that. However, there are several methods in the market for that, and when choosing the right method, you should carefully define your needs: synchronous method or async one, row based or statement based (traffic between servers) and if time delayed replication is required (for quick recovery from delete for example).

The Multi Master Replication Options
I just kicked the first Israeli MySQL User Group in association with Wix this week, and I wanted to share with you Michael Naumov's presentation from our first event. 

In his talk, Michael presented four methods to support Multi Master replication: MySQL 5.6 Native, NDB, Tungsten and Galera.



Bottom Line
Careful selection of the right MySQL Multi Master replication can save you production issues and can boost your system. Now all left is to define what is needed and what is the matching solution for that.

Keep Performing,
Moshe Kaplan

Aug 28, 2013

Your Storage Probably Effects Your System Performance

One of my clients had started getting some performance alerts in their monitoring systems: "IO is too high".
Probably this not something you will be glad to have.

In a quick analysis we found out that the alerts and the high IO originated from servers that were installed in a new data center.
While the actual CPU utilization devoted to IO wait at the old data center was around 25%, in the new data center it was about 75%.

Who is to Blame?
In the new data center NetApp 2240c was chosen as a storage appliance, while in the old IBM V7000 unified was used. Both systems had SAS disks so we didn't expected a major difference between the two. Yet, it was something worth to explore.

Measurement 
In order to verify the source we made a read/write performance benchmark to both systems by running the following commands:
  1. Write: dd if=/dev/zero of=/tmp/outfile count=512 bs=1024k
  2. Read: dd if=/tmp/outfile of=/dev/null bs=4096k
UPDATE I: You should also try btest or IOMeter as suggested by Yuval Kashtan 
UPDATE II: when using dd, you should better use dd if=/dev/urandom of=/tmp/outfile.txt bs=2048000 count=100 that actually uses random input and just allocate spaces with nulls

Results
On the NetApp 2240 we got 0.62GB/s write rate and 2.0GB/s read rate (in site #2)
On the IBM V7000 unified we got 0.57GB/s write rate and 2.1GB/s read rate (in site #2)
On the IBM V7000 unified we got 1.1GB/s write rate and 3.4GB/s read rate  (in site #1)
That is almost a 100% boost when we used the IBM system!

Bottom Line
When selecting and migrating between storage appliances pay attention to their performance. Otherwise, you may tackle this differences in production. However, differences should be inspect in the same environment. In our case something that was seemed like a storage issue turned into a VM/OS configuration or network issue (exact cause is still under investigation).

Keep Performing,
Moshe Kaplan

Aug 16, 2013

Azure Production Issues, Not a nice thing to share with your friends...

Last night I got a call from a close friend.
"Our production SQL Server VM at Azure is down, and we cannot provide service to clients".
A short analysis issued that we are in a deep trouble:

  1. The server status was: Stopped (Failed to start).
  2. Repeated tries to start the server resulted in Starting... and back to the fail message.
  3. Changing instance configuration as proposed in various forums and blogs was resulted in the same fail message.
  4. Microsoft claims that everything is Okay with its data centers.
  5. Checking the Azure storage container found out that the specific VHD disk was not updated since the server failure.
Since getting back to cold backup would cause losing too much data, we had to restore somehow the failed server.

The chances were against us. Yet, lucky us, we could do that by restoring the database files from the VHD (VM disk) file that was available at Azure storage. 

How to Recover from the Stopped (Failed to start) VM Machine?
  1. Start a new instance in the same availability set (that way you can continue using the same DNS name, instead of also deploying a new version of the app servers).
  2. Attach a new large disk to the instance (the failed server disk was 127GB, make sure the allocated disk is larger).
  3. Start the new machine.
  4. Format the disk as a new drive.
  5. Get to your Azure account and download the VHD file from Azure storage. Make sure you download it to the right disk. We found out that the download process takes several hours even when the blob storage is in the same data center as the VM.
  6. Mount the VHD file you downloaded as a new disk.
  7. Extract the database and log files from the new disk and attach them to the new SQL Server instance.
Other recommendations:
  1. Keep your backup files updated and in a safe place.
  2. Keep your database data and log files out of the system disk, so you could easily attach them to other servers.
Bottom Line
When the going gets tough, the tough get going

Keep Performing,

ShareThis

Intense Debate Comments

Ratings and Recommendations