Showing posts with label Monitoring. Show all posts
Showing posts with label Monitoring. Show all posts

Jan 7, 2014

Monitoring Your SQL Server w/ Monitis

One of the nicest monitoring products around is Monitis.com
If you are using MySQL, you will probably find out that some of the thresholds there should be modified or tuned (or just removed, since they are not critical to your day to day operations):




Key Cache Writes Rate
Actual value: 0
Proposed range: 75-100
What should be done: If you are not using MyISAM, ignore this one. If you do use it, modify the key_buffer_size variable and increase it to accommodate the index in the memory.

Indexes Usage
Actual value: 5
Proposed range: 80-100
What should be done: Indexes usage is a key indicator to understand if you write your queries in a way that utilizes your indexes and if you defined correct indexes to support your questions.
Enable your slow log query and examine your queries. 

Key Cache Hit Rate
Actual value: N/A
Proposed range: 95-100
What should be done: Seems that this is not a critical variable, since you must check it is relevant in your case. If you believe that your system business case should use most data out of cache and value is low, modify the key_buffer (MyISAM) or innodb_buffer_pool_size (InnoDB).

Table Cache Hit Rate
Actual value: N/A
Proposed range: 85-100
What should be done: See the one above

Bottom Line
Monitoring system is critical component of critical systems. However, in most cases you should verify that the default configuration matches your business case (and if it is the case, take care the alerts!).

Keep Performing,
Moshe Kaplan

Nov 9, 2013

Azure Monitoring. What to Choose?

You are going live with your Azure system and need to know what is happening there?
Cannot sleep at night as your system crashes from time to time?
Need to know whatever your fast growing system will go out of resources in the near future?
If so, this post is for you!

Decisions

In general you have 3 main options:
  1. Open Source Software that is installed by you and managed by you.
  2. Azure built in monitoring that may require some extra charging and tailoring to your business case.
  3. 3rd Party SaaS offering that provides you an End to End solution.
The Open Source Way
  1. Major OSS monitoring tools are available and widely used in the industry. Some of the leading solutions are Zabbix, Nagios and Ganglia.
  2. Software is available w/ no additional cost
  3. You will need to install the servers by yourself (and of course paying for the instances and manage the data).
  4. Implementing an end to end monitoring (inc. user experience) may require extra servers installation on another data center or using a 3rd party service.
  5. Other downsides are well described by CopperEgg.
The Azure Monitoring Way
  1. If you are already invested in Azure, it's built-in, so why not using it?
  2. The service provides nice utilization graphs, and some minimal alerting system (limited to 10 rules). However, there are no pre configured templates, and some metrics are with additional charging.
  3. Basic website availability is provided using web site monitoring.
  4. Yet, there are several gaps such as VM monitoring, SQL Azure and Azure Storage.
  5. The overall impression is that many features are available, but it's far from being a monitoring product, and you will have to do a lot of integration work by yourself..
3rd Party Monitoring as s Service
  1. CopperEgg: Nice company, the their website could look better. More important, Their features looks pretty similar to the OSS solutions with an integrated website response time and availability service. Pricing: $7-9/server|service/month.
  2. AzureWatch: a very similar product to CopperEgg both in features and pricing (although their landing page looks much better). Pricing: $5-8/server|service/month.
  3. New Relic:Probably the most intersting product around, not only it provides an end to end monitoring, but it also analyzes your performance bottlenecks. Their product is being integrated to your code and provides you with production profiling and alerts you regarding any exception. Pricing: standard version that includes the other providers offers is free. Code level diagnostics and inner level profiling will cost you $25-200/server|service/month
Bottom Line
If you are ready to invest time to get your own solution, the OSS products with Azure integration will probably be your preferred solution. Otherwise, I will recommend you to choose New Relic, that seems to provide the best value in the market for your needs.

Keep Performing

Jul 16, 2013

Detecting Performance Bottlenecks in Apache httpd Server

“a problem well put is half solved.” ― John Dewey

One of the most important things to well detect a performance issues in a LAMP system is understand the actual processes that causing the issues.

What should we do?
One way to start with is properly configure the Apache logging in a way that we will get fine details about every request performance.

Key Apache httpd Parameters
%B Size of response in bytes, excluding HTTP headers.
%D The time taken to serve the request, in microseconds.
%{VARNAME}e The contents of the environment variable VARNAME.
%T The time taken to serve the request, in seconds.
%X Connection status when response is completed

How to Configure?
Modify your httpd.conf file according to the following article:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy

How do we Analyze it?
Apache log files can be huge (a common case in systems that have some performance challenges).
A useful tool for analyzing the Apache logs, which Cnaan Aviv introduced to me, is GoAccess.
This tool generates reports, statistics and detects errors.


Bottom Line
When done right, solving performance issues is not a black magic. It just need to be done using a well defined method and the right tools.

Keep Performing,
Moshe Kaplan

Jun 25, 2013

MongoDB and Java

You can find below some hints for initial Java and MongoDB integration

Take a Look at the Requirements
  1. MongoDB
  2. MongoDB-Java-Driver
  3. JDK. If you want use JDK 1.6 or newer, you will get an error like this one: "DBObject cannot be resolved to a type"
Installing Java
yum -y install java-1.7.0-openjdk
yum -y install java-1.7.0-openjdk-devel

Getting MongoDB driver

Compiling Java
javac -d . *.java
java -cp . com/example/mbeans/Main

Define MongoDB Headers in the Code
import com.mongodb.MongoClient;
import com.mongodb.MongoException;
import com.mongodb.WriteConcern;
import com.mongodb.DB;
import com.mongodb.DBCollection;
import com.mongodb.BasicDBObject;
import com.mongodb.DBObject;
import com.mongodb.DBCursor;
import com.mongodb.ServerAddress;
import com.mongodb.*;

Connect to MongoDB
DB _db;

public void init() {
try {
System.out.println("Connecting to mongo...");
MongoClient mongoClient = new MongoClient("127.0.0.1" , 27017);
_db = mongoClient.getDB("display");
System.out.println("Connected to mongo...");
} catch (Exception e) {
System.out.println("Failed Connecting Mongo...");
}

Query the Database (Get the Number of Connections)
CommandResult stats = _db.command("serverStatus");
return Integer.valueOf((((DBObject)(stats.get("connections"))).get("current")).toString());

Bottom Line
Java and MongoDB integration is not too difficult, you just need to do the right thinks right...

Keep Performing,

Mar 19, 2013

My MongoDB and PHP Famous 5 Minutes Tutorial

Did you ever wanted to integrate MongoDB monitoring with a third party service?
Did you wanted to integrate MongoDB performance counters with your load stress environment?

This 5 minutes tutorial will help you expose these details to any third party service or application using a simple JSON service, PHP and MongoDB:
  1. Install Apache and PHP and php-devel: sudo yum -y install httpd php php-devel php-pear
  2. Install PHP MongoDB Driver: sudo pecl install mongo
  3. Add extension=mongo.so to you /etc/php.ini
  4. Create a short monitoring code as mongo.php and place it at /var/www/html/:

header('Cache-Control: no-cache, must-revalidate');
header('Expires: Mon, 26 Jul 1997 05:00:00 GMT');
header('Content-type: application/json;charset=UTF-8');

class Monitor {
public $VirtualMemory_MB;
public $PageFaults;
public $CurrentConnections;
public $NetworkIO_In_MB;
public $NetworkIO_Out_MB;
public $SpeedIO_Avg_ms;
public $LockRatio_Per;
public $ReadersLocked;
public $WritersLocked;
public $ObjectsInDatabase;
public $DatabaseDataSize_MB;
public $DatabaseIndexSize_MB;
public $IndexMissRatio_Per;
public $CursorsTimedOut;
public $DeleteSelectRatio_Per;
public $InsertSelectRatio_Per;
public $UpdateSelectRatio_Per;
}

if (!isset($_REQUEST['db'])) {
header($_SERVER['SERVER_PROTOCOL'] . ' 500 Internal Server Error', true, 500);
echo json_encode(array('status'=>0,'message'=>'db is missing'));
exit;
}
$databaseName = $_REQUEST['db'];


// connect
$m = new MongoClient();

// select a database
$db = $m->$databaseName;
$ret = $db->execute('db.stats();');

$result = new Monitor;
$result->ObjectsInDatabase = $ret["retval"]["objects"];
$result->DatabaseDataSize_MB = $ret["retval"]["dataSize"];
$result->DatabaseIndexSize_MB = $ret["retval"]["indexSize"];
//var_dump($ret);

$ret = $db->execute('db.serverStatus();');
$result->VirtualMemory_MB = $ret["retval"]["mem"]["virtual"];
$result->PageFaults = $ret["retval"]["extra_info"]["page_faults"];
$result->CurrentConnections = $ret["retval"]["connections"]["current"];
$result->NetworkIO_In_MB = $ret["retval"]["network"]["bytesIn"]/1000000;
$result->NetworkIO_Out_MB = $ret["retval"]["network"]["bytesOut"]/1000000;
$result->SpeedIO_Avg_ms = $ret["retval"]["backgroundFlushing"]["average_ms"];
$result->LockRatio_Per = $ret["retval"]["globalLock"]["lockTime"]/$ret["retval"]["globalLock"]["totalTime"]*100;
$result->ReadersLocked = $ret["retval"]["globalLock"]["currentQueue"]["readers"];
$result->WritersLocked = $ret["retval"]["globalLock"]["currentQueue"]["writers"];
$result->IndexMissRatio_Per = $ret["retval"]["indexCounters"]["btree"]["missRatio"];
$result->CursorsTimedOut = $ret["retval"]["cursors"]["timedOut"];
$result->DeleteSelectRatio_Per = $ret["retval"]["opcounters"]["delete"]/$ret["retval"]["opcounters"]["query"]*100;
$result->InsertSelectRatio_Per = $ret["retval"]["opcounters"]["insert"]/$ret["retval"]["opcounters"]["query"]*100;
$result->UpdateSelectRatio_Per = $ret["retval"]["opcounters"]["update"]/$ret["retval"]["opcounters"]["query"]*100;

echo json_encode($result);
?>

Bottom Line
You MongoDB is ready for monitoring by an external service: http://localhost/mongo.php

Keep Performing,

May 9, 2011

Monitoring and Taking Care of MySQL Replication

If you get your slave into production soon, your should make sure that you know how take care of it when something goes wrong. 

Knowing that the slave is stuck can monitored using open source tools like Nagios and its MySQL plugins. In general it is wise to have such a tool to monitor all your system and business aspect including the web server and login rates.

How do I know that my slave is stuck or not?
  1. Connect to MySQL on the slave using permitted user.
  2. Run the show slave status\G command.
  3. The result should include the following lines with a Yes value. If any of them is No, there is a problem:
Slave_IO_Running: Yes
Slave_SQL_Running: Yes

It is stuck, What should I do?
  1. Connect to the file system with a permitted user.
  2. Check the values of the replication error log and take care according to it (tail -n 50 /var/lib/mysql/responder3.spd.co.il.err).
Bottom Line
Get into production!


Keep Performing,

Oct 11, 2010

The Path to the Cloud

You decided to make the big decision and migrate to the cloud. 

What Should You Do Now?
Well, migrating an active system between two hosting providers (and even from your small desktop under your desk), can be risky and may affect the business due to downtime or user frustration. Migration to the cloud that enables a new paradigm can be even more complex since it requires changing concepts and deal with a new brave world..

So How Should We Start?
The first rule is to not think of the cloud as a cloud. You should prepare for a simple migration (think of the cloud provider just as another hosting provider). This way you can minimize the major risk of incompatibility. Follow these steps to accomplish the first step:
  1. Prepare your integration environment. If you don't have one, it's time for you to have one. If you do have, it's time to make sure that it's 100% compatible with your production environment. We'll use this environment in order to stimulate the migration.
  2. Arrange your DNS records. Migrating to another location will probably require modifying your DNS records to reflect the new location. Some hosting providers do not allow naming for external locations (for example GoGrid is an example for such a provider). So you should verify that you know how to change your DNS records, or migrate the records to another register.
  3. Implement a DRP environment. You should have one first in your current provider environment in order to eliminate issues such as access lists and network latency. Why is it so important?
    1. Hosting migration is risky. You would like to have easy to roll back in case of non successful migration
    2. Data Migration take a lot of time. If your databases (or NoSQL) are large, it will take a long time to migrate them between locations (even using a 100Mb channel, it will take a 3 hours downtime to migrate a 100GB database). The most efficient way to shorten this time will be by using Log shipping between the databases, or implement a replication between 2 NoSQL sites. When you'll choose to perform the migration to the new site, all you'll need to do is stopping the primary instance and turn the passive instance to primary one. Since the data is being replicated between sites all the time, the data migration time slot will be minimized to at most few minutes. 
  4. Migrate your DRP environment to the new cloud provider. Start with the integration environment in order to verify there are no network issues in implementing it.
  5. Verify the new provider stability. Now that you have servers in the new hosting provider, it's time to verify it stability, its network performance and any other issues that may arise and you were not expected to.
  6. Implement a Reverse Proxy Solution. Some of your client will be ready for the migration. They may have old stale DNS records or they might use your IP address for some kind of reason. Using reverse proxy can help your routing traffic from your old location to the new one, minimizing lost traffic and downtime.
  7. Perform migration test in the integration environment. You should turn the passive site into active one, and check that user can continue work after the downtime. You should document the process, and if times are not good enough, you should exercise the process. After you are satisfied with results, it's time for prime time.
  8. Migrate your production DRP site the new hosting provider. Perform the process in a similar manner to the integration DRP migration.
  9. Prepare for The big day. Transform your DRP (or the new hosting environment) into a production one.
  10. Verify. Wait a few days and verify the system was stabilized, if so make few steps before closing your old hosting facilities and cut costs:
    1. Implement a new DRP site both for the integration and the production environments.
    2. Verify that the routed traffic from the old site is neglected.
  11. The big day. it's time to shut down your old server, close the site and say bye-bye to your old provider.
Just Scale It
After doing so much work, it's time to decide what cloud items do you want to you. It may depend on the cloud provider offering, and your decision how and if you want to avoid vendor lock in:

  1. CDN: if you have major static files traffic, or if you are able to turn some of the traffic into static one (think of Linkedin profiles for example), you could store it in CDN facilities that offer can reduce your web servers traffic and better prepare your for peaks. It also can help you 
  2. Queues: Avoid taking care of queues availability and let this internet giants take care of it.
  3. NoSQL stores: you could implement open source solutions such as Cassandra or chose the provider key value store.
  4. Rational Databases: Some cloud providers offer these days a database as a Service that let you get rid of these log shipping, clustering and care of daily backups. If you are interested, drop me a note and I refer you to some interesting companies.
  5. Load Balancing: You could choose using software based load balancers such as HAProxy, or choosing the cloud solution.
  6. Monitoring: Some cloud provider offer you a system monitoring, saving you major effort to monitor your servers.
Take It One Step Further
Now that you chosen the tools you just have to make sure your system can scale out:
  1. Separate back office services from web interfaces, creating pools of small machines that each can take care of their tasks.
  2. Parallelize your back end services, making sure that no single service turns into a future bottleneck.
  3. In case of push systems and communication between parties, create a directory system that can route events between different servers.
Bottom Line
If you felt that you are bounded to your current hosting provider, it's time to think again. Using this simple, yet not so short method, you could reduce risks and accomplish the change.

Keep Performing,

Aug 28, 2010

How to Monitor Your Cloud Service Provider

Click on the post title to read the full post and the comments.
The emergence of cloud computing and Everything as a Service in last few years, raised many issues in enterprises that started to outsource part of their services and infrastructure into the cloud.
If in the pre-migration IT managers knew the availability of every piece of their infrastructure, now things are a little bit more complex.


But are Things Really Different?
If you had a COTS ERP or CRM system that you bought from SAP, Oracle or Microsoft, did you really know that happening in the system internals? Or did you focus in measuring the end user experience as well as system metrics like CPU and disk utilization?


What Should We Measure?
In order to decide what we should measure, we should ask first several questions:
  1. What will cause me troubles if I'll not measure?
  2. Where the money is coming out of?
  3. What are you paying for?
  4. What is the service interface?
All these questions have specific answers in a common agreement between the parties: the SLA (Service Level Agreement). If it is important enough, you should provide metrics for that, and if you have metrics for that you probably can monitor that using your monitoring system. The importance of SLA monitoring in the Cloud world is highly resembled by Oblicore acquisition by CA in Jan 2010,

What Should We Not Measure?
Usually I try to avoid specific devices monitoring of in the cloud provider since it will not provide me any information and will break the interface of the service (sounds like object oriented design). 
However, if you pay for things that are virtual like redundancy or extra capacity, the service provider should provide measurements for that.

Can You Give Us a Small Example?
Usually, I tend to monitor the user experience. For example, if I use a SMS gateway to send SMS through my system, I'll monitor:
  1. The submission time of an SMS request
  2. The time to get the SMS in a handset that is stimulated using a cellular modem.
I will perform this test once in a while. E.g. if I send dozens of SMS every second, I will perform measurement every few seconds.

The Bottom Line
Choose your metrics carefully according to the defined SLA and monitor them using your monitoring system. Do not be tempted to over measure in order to avoid breaking the interface,

Keep Performing,
Follow MosheKaplan on Twitter

Dec 20, 2009

Expect for the Best. Be Prepared for the Worse.

A major issue in preparing for the disaster is getting clear visibility of your system. Especially if we are talking about distributed multi server and multi data center system.
Paul Venezia from InfoWorld provided a clear and concise description of the major open source tools (yes, free ones), that can give your clear visibility of your system.These tools can help you better know:
  1. When does your CPU utilization get close to 100%?
  2. When should you add more hardware or make an effort to boost your system performance?
  3. What are your high traffic sources?
  4. What is the root cause analysis for system poor performance?
  5. What are the long term treadlines?
 Among the described tools you can find:
  1.  Cacti - System Monitoring Web GUI that provides you graphs to every system metric that can be exposed in SNMP. Most recommended tool.
  2.  Nagios - Network Monitoring tool.
  3. NeDi - Physical network mapping tool (to which port your server is connected). Useful moslty if you are in the on-premise business rather than the cloud business.
  4. Ntop - Traffic analysis tool that can help find what are the traffic sources, and which one keeps you bill so high.
  5. Pancho - Configuring and backup Cisco routers. Again for the on-premise guys.
Using at least some of these tools you can better prepare for the disaster, and many times avoid it.

Keep Performing,
Moshe Kaplan. Performance Expert.

ShareThis

Intense Debate Comments

Ratings and Recommendations