JAVA Production Systems Profiling Done Right!

If you are facing a Java system performance issue in production, and JProfiler is not the right tool for it, probably JMX monitoring using the VisualVM will do the work for you.

Technical
JMX usage from a remote machine can be frustrating. Therefore, please make sure that:

1. Your hostname is included in the /etc/hosts 
1. Get host name using hostname 
2. Add the host name after 127.0.0.1 in /etc/hosts
2. JMX is binded to the external IP:
1. Verify 127.0.0.1 is not presented at: netstat -na | grep 1099
2. If it does presented, add to your java command: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=

If everything is Okay, you will be able to run VisualVM from a remote machine and connect to the remote server.

VisualVM

Now, that you have your VisualVM up and running there are some items you should take a look at:

1. General CPU and memory graphs.
2. Sampler that enables you taking snapshots.
3. Snapshot analysis that enables you a hotspot presentation as well as deep.

Bottom Lin

My recommendation is to have snapshot of the process and then look at the hotspots tab for major calls with actual long CPU time. You should focus on these items.

Keep Performing,

Moshe Kaplan

Azure Monitoring. What to Choose?

You are going live with your Azure system and need to know what is happening there?
Cannot sleep at night as your system crashes from time to time?
Need to know whatever your fast growing system will go out of resources in the near future?
If so, this post is for you!

Decisions
In general you have 3 main options:

1. Open Source Software that is installed by you and managed by you.
2. Azure built in monitoring that may require some extra charging and tailoring to your business case.
3. 3rd Party SaaS offering that provides you an End to End solution.

The Open Source Way

1. Major OSS monitoring tools are available and widely used in the industry. Some of the leading solutions are Zabbix, Nagios and Ganglia.
2. Software is available w/ no additional cost
3. You will need to install the servers by yourself (and of course paying for the instances and manage the data).
4. Implementing an end to end monitoring (inc. user experience) may require extra servers installation on another data center or using a 3rd party service.
5. Other downsides are well described by CopperEgg.

The Azure Monitoring Way

1. If you are already invested in Azure, it's built-in, so why not using it?
2. The service provides nice utilization graphs, and some minimal alerting system (limited to 10 rules). However, there are no pre configured templates, and some metrics are with additional charging.
3. Basic website availability is provided using web site monitoring.
4. Yet, there are several gaps such as VM monitoring, SQL Azure and Azure Storage.
5. The overall impression is that many features are available, but it's far from being a monitoring product, and you will have to do a lot of integration work by yourself..

3rd Party Monitoring as s Service

1. CopperEgg: Nice company, the their website could look better. More important, Their features looks pretty similar to the OSS solutions with an integrated website response time and availability service. Pricing: $7-9/server|service/month.
2. AzureWatch: a very similar product to CopperEgg both in features and pricing (although their landing page looks much better). Pricing: $5-8/server|service/month.
3. New Relic:Probably the most intersting product around, not only it provides an end to end monitoring, but it also analyzes your performance bottlenecks. Their product is being integrated to your code and provides you with production profiling and alerts you regarding any exception. Pricing: standard version that includes the other providers offers is free. Code level diagnostics and inner level profiling will cost you $25-200/server|service/month

Bottom Line

If you are ready to invest time to get your own solution, the OSS products with Azure integration will probably be your preferred solution. Otherwise, I will recommend you to choose New Relic, that seems to provide the best value in the market for your needs.

Keep Performing

Moshe Kaplan

MongoDB Index Tuning and Dex

One of the first things when your system is rolled into production (or better when you have your acceptance tests) is checking the system performance performance and tune it.
When we talk about databases, we usually talk about index tuning, and MongoDB is not different.

How to Detect the Slow Queries?
First enable you MongoDB slow queries log. It's easy, just run db.setProfilingLevel(1, 1) from the command line. 

How to Analyze the Slow Queries?
Well, you can use the built in queries (after all the profiling is saved at a MongoDB collection). However, Dex, a new tool from MongoLab, can help you shorten the time to index...
It's a Python based product that can be easily installed and run and provides a list of recommended indexes based on MongoLab best practices.

How to Install and Run?

1. Install python: yum -y install python
2. Install pip
3. Install Dex using pip: pip install dex
4. Create a log file: touch mongodb.log
5. Run it: sudo dex -p -v -f mongodb.log mongodb://user:password@serverDNS:serverPort/database. Don't forget to use the various flags to get errors in case of wrong configuration.

Should We Just Copy and Paste the New Indexes?
As you may know every added index results in a performance degragation when INSERT, DELETE and UPDATE statement are performed. Therefore, we should carefuly select the indexes to be added:

1. We should check that the new indexes do not overlap existing indexes. If there is an overlap, we should remove/unify the indexes definitions (use the .db.collection.getIndexes() method to retrieve the data.
2. We should check that the queries are frequently used in order to avoid saving few seconds once a month while doing a SELECT, and pay for that at peak times INSERTs.

Bottom Line
MongoDB queries tuning is easier than ever: Just Tune It!

Keep Performing,
Moshe Kaplan

The Basics of SQL Server Performance

When you first hit a SQL Server that cannot server even another single hit you should start with following basic methods:

1. Analyze the application and understand who are the most massive business processes, and analyze their performance. High odds that the problem lies there.
2. Detect open connections using the Activity Monitor
3. Check objects execution time using the SQL Server reports (of course assuming  you are lucky enough and all SQL Server access is done using stored procedures).
4. Implement profiling and check the queries are most frequent and  most time consuming.
5. Detect dead locks (a good sign for that this is the case is low CPU utilization).
   
6. Detect slowdown source by using the following query  (thanks to Henk van der Valk):
   

-- Uncomment the reset or capture the wait stats before you start the batch to investigate:
-- DBCC SQLPERF('sys.dm_os_wait_stats', CLEAR)
-- DBCC SQLPERF('sys.dm_os_latch_stats', CLEAR)

-- show the sql waitstatistics:
SELECT wait_type
, SUM(waiting_tasks_count) AS waiting_tasks_count
, SUM(wait_time_ms) AS wait_time_ms
, SUM(max_wait_time_ms) AS max_wait_time_ms
, SUM(signal_wait_time_ms) AS signal_wait_time_ms
, SUM(wait_time_ms/NULLIF(waiting_tasks_count,0)) as 'Avg_wait_time_ms per waittype req.'

FROM sys.dm_os_wait_stats
WHERE wait_type NOT IN
(
'SQLTRACE_BUFFER_FLUSH'
, 'LAZYWRITER_SLEEP'
, 'CHECKPOINT_QUEUE'
, 'BROKER_TO_FLUSH'
)
AND wait_type NOT LIKE 'XE%'
AND wait_type NOT LIKE 'PREEMPT%'
AND wait_type NOT LIKE 'SLEEP%'
AND wait_type NOT LIKE '%REQ%DEAD%'
AND waiting_tasks_count > 0
GROUP BY wait_type
WITH ROLLUP
ORDER BY wait_time_ms DESC

After doing these initial steps, you finally gathhered enough data to solve the issue...

Keep Performing,
Moshe Kaplan. RockeTier. The Performance Experts.
moshe at rocketier.com

Can you monitor Java performance w/o affecting the system itself?

The Conservative Way
Well, the popular way to monitor code performance and detect bottlenecks is using Profilers. However, profilers insert small portions of code into the code base in order to measure each function processing time. For functions that require relatively long time to complete, this issue can be neglected. In other cases, when a function requires a very short time to complete and its called many times, this behavior can lead to wrong conclusions.

Usually in order to overcome this issue, you should carefully tune your measurement period in order to avoid over measurements.



The Alternative Way
We at RockeTier, are discussing several business opportunities with IBM these days. During these discussions, we noticed a great tool by Jinwoo Hwang from IBM. IBM offers an alternative way. Its tool analyzes Windows (yes, they support only Windows) performance log and Java thread dumps and automatically detects Java threads that consume the majority of system resources.
It is even more interesting since this tool supports both system monitoring and Java thread monitoring. Each can be done separately. Therefore, you gain flexibility, detect the problematic code area and focus on it in your debug environment w/o affecting the production system at all.

You can download this tool from the IBM site.

Keep Performing,
Moshe Kaplan. RockeTier. The Performance and Cloud Computing Experts.