Showing posts with label Logging. Show all posts
Showing posts with label Logging. Show all posts

Jul 16, 2013

Detecting Performance Bottlenecks in Apache httpd Server

“a problem well put is half solved.” ― John Dewey

One of the most important things to well detect a performance issues in a LAMP system is understand the actual processes that causing the issues.

What should we do?
One way to start with is properly configure the Apache logging in a way that we will get fine details about every request performance.

Key Apache httpd Parameters
%B Size of response in bytes, excluding HTTP headers.
%D The time taken to serve the request, in microseconds.
%{VARNAME}e The contents of the environment variable VARNAME.
%T The time taken to serve the request, in seconds.
%X Connection status when response is completed

How to Configure?
Modify your httpd.conf file according to the following article:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy

How do we Analyze it?
Apache log files can be huge (a common case in systems that have some performance challenges).
A useful tool for analyzing the Apache logs, which Cnaan Aviv introduced to me, is GoAccess.
This tool generates reports, statistics and detects errors.


Bottom Line
When done right, solving performance issues is not a black magic. It just need to be done using a well defined method and the right tools.

Keep Performing,
Moshe Kaplan

Dec 21, 2009

Lessons from Facebook:

FB maybe is not the best example for quality design (probably each of us is likely to have at least a single error message a day).

However, since it's the 2nd largest site in the world with contsant exponential growth, it's a good lesson to have. Most interesting that their tech is team is only about 250 people and 30K servers, pretty amazing.


Not long ago a Jeff Rothschild, the Vice President of Technology at Facebook gave a presentation in UC San Diego. You can find a detailed summary by Prof. Amin Vahdat, but I'll make several top comments that I found useful.
  1. FB Software development: mostly PHP (complied), but other languages are used as well.
  2. Common interface between services using an internal development that was turned to open source: Thirft. This interface enables easy connections between the different languages.
  3. Logging Framework that is not dependent on a central repository or its availability. FB is using Hadoop and well as Hive (that was developed there as well). The log size is growing at a 25TB a day.
  4. Operational Monitoring: Separated from the logging mechanism 
  5. LAN can be a bottleneck as well: expect for packet loss and packet drops in the LAN if you stress it too much.
  6. CDN: Facebook is using external CDN to images distribution.
  7. Dedicated file system named Haystack that combines simple storage along with cache directory: file system is accessed only once to get images from it, while directory structure is retrieved from the cache.
  8. Most data is served from Memcached. Database is used mostly for Persistency and data replication between sites (Memcached is being heated by the MySQL itself):
    1. Top challenge: keeping data consistent since Memcached can be messed easily (No search for keys is available).
    2. Mixing information including sizes and types is better - making sure that load on CPU, Memory and etc is distributed equally.
  9. Shared nothing - Keep your system independent - avoid a single bottleneck. Therefore, data is saved in a sharded MySQL from day 1. However, MySQL is used mostly for data persistency and not for conservative database usage pattern:
    1. No Joins in the MySQL
    2. Chosen due to good data consistency + Management Software
    3. 4K servers
    4. Data replication between sites is based on MySQL replication
    5. Memcached is being heated based on the MySQL Replication using a custom API
One last thing, if you are interested in Facebook financials, as well the storage machines (NetApp low end 3070), sizing, traffic, servers, storage procurement and data center costs, take a look at TechCrunch's Michael Arrington post.

Keep Performing,
Moshe Kaplan. Performance Expert.

ShareThis

Intense Debate Comments

Ratings and Recommendations