Apache Session Persistancy Using MongoDB

Your app is booming, you need more web servers and you need to serve users and keep their user experience. When you had a single server you just used session for that, but now, How do you keep sessions across multiple web servers? 

Using a session stickiness load balancer is a good solution, but what happens when a web server needs a restart?

Session Off Loading

Many of you are familiar with the concept of session off loading. The common configuration for it is using Memcached to store the session object, so it is accessible from all web servers. However, this solution lacks the features of high availability by replication and persistence.

The Next Step: High Availablability

Offloading web servers sessions to MongoDB looks like a great solution: key-value store, lazy persistency and built in replication and auto master failover.

Step by Step Solution for LAMP Session Off Loading using MongoDB

1. Install MongoDB (preferably in a replica set with at least 3 nodes)
2. Install MongoDB driver for PHP: 
3. Configure the php.ini: extension=mongo.so
4. Download and integrate Apache session handler implementation using MongoDB.
5. Configure your MongoDB setting in the module:
1. Set the list of nodes in the connectionString setting: 'connectionString' => 'mongodb://SERVER1:27017, mongodb://SERVER2:27017',
2. Do the same to the list of servers (see at the bottom of this post)
3. Configure the cookie_domain (or just place a null string there to support all): 'cookie_domain' => ''
4. Don't forget to enable replica set if you are using one: 'replicaSet' => true,
6. Add the MongoSession.php to your server and require it: require_once('MongoSession.php');
7. Replace the session_start() function with $session = new MongoSession();

And that's all. Now you can continue using the session objects as you used to create new great features:

if (!isset($_SESSION['views'])) $_SESSION['views'] = 0;

$_SESSION['views'] = $_SESSION['views'] + 1;

echo $_SESSION['views'];

Bottom Line

Smart work can tackle every scale issue you are facing off...

Keep Performing,

Moshe Kaplan

Appendix: 

'servers' => array(
array(
'host' => 'SERVER1',
'port' => Mongo::DEFAULT_PORT,
'username' => null,
'password' => null,
'persistent' => false
),
array(
'host' => 'SERVER2',
'port' => Mongo::DEFAULT_PORT,
'username' => null,
'password' => null,
'persistent' => false

)

My MongoDB and PHP Famous 5 Minutes Tutorial

Did you ever wanted to integrate MongoDB monitoring with a third party service?
Did you wanted to integrate MongoDB performance counters with your load stress environment?

This 5 minutes tutorial will help you expose these details to any third party service or application using a simple JSON service, PHP and MongoDB:

1. Install Apache and PHP and php-devel: sudo yum -y install httpd php php-devel php-pear
2. Install PHP MongoDB Driver: sudo pecl install mongo
3. Add extension=mongo.so to you /etc/php.ini
4. Create a short monitoring code as mongo.php and place it at /var/www/html/:

header('Cache-Control: no-cache, must-revalidate');

header('Expires: Mon, 26 Jul 1997 05:00:00 GMT');

header('Content-type: application/json;charset=UTF-8');

class Monitor {

public $VirtualMemory_MB;

public $PageFaults;

public $CurrentConnections;

public $NetworkIO_In_MB;

public $NetworkIO_Out_MB;

public $SpeedIO_Avg_ms;

public $LockRatio_Per;

public $ReadersLocked;

public $WritersLocked;

public $ObjectsInDatabase;

public $DatabaseDataSize_MB;

public $DatabaseIndexSize_MB;

public $IndexMissRatio_Per;

public $CursorsTimedOut;

public $DeleteSelectRatio_Per;

public $InsertSelectRatio_Per;

public $UpdateSelectRatio_Per;

}

if (!isset($_REQUEST['db'])) {

header($_SERVER['SERVER_PROTOCOL'] . ' 500 Internal Server Error', true, 500);

echo json_encode(array('status'=>0,'message'=>'db is missing'));

exit;

}

$databaseName = $_REQUEST['db'];

// connect

$m = new MongoClient();

// select a database

$db = $m->$databaseName;

$ret = $db->execute('db.stats();');

$result = new Monitor;

$result->ObjectsInDatabase = $ret["retval"]["objects"];

$result->DatabaseDataSize_MB = $ret["retval"]["dataSize"];

$result->DatabaseIndexSize_MB = $ret["retval"]["indexSize"];

//var_dump($ret);

$ret = $db->execute('db.serverStatus();');

$result->VirtualMemory_MB = $ret["retval"]["mem"]["virtual"];

$result->PageFaults = $ret["retval"]["extra_info"]["page_faults"];

$result->CurrentConnections = $ret["retval"]["connections"]["current"];

$result->NetworkIO_In_MB = $ret["retval"]["network"]["bytesIn"]/1000000;

$result->NetworkIO_Out_MB = $ret["retval"]["network"]["bytesOut"]/1000000;

$result->SpeedIO_Avg_ms = $ret["retval"]["backgroundFlushing"]["average_ms"];

$result->LockRatio_Per = $ret["retval"]["globalLock"]["lockTime"]/$ret["retval"]["globalLock"]["totalTime"]*100;

$result->ReadersLocked = $ret["retval"]["globalLock"]["currentQueue"]["readers"];

$result->WritersLocked = $ret["retval"]["globalLock"]["currentQueue"]["writers"];

$result->IndexMissRatio_Per = $ret["retval"]["indexCounters"]["btree"]["missRatio"];

$result->CursorsTimedOut = $ret["retval"]["cursors"]["timedOut"];

$result->DeleteSelectRatio_Per = $ret["retval"]["opcounters"]["delete"]/$ret["retval"]["opcounters"]["query"]*100;

$result->InsertSelectRatio_Per = $ret["retval"]["opcounters"]["insert"]/$ret["retval"]["opcounters"]["query"]*100;

$result->UpdateSelectRatio_Per = $ret["retval"]["opcounters"]["update"]/$ret["retval"]["opcounters"]["query"]*100;

echo json_encode($result);

?>

Bottom Line

You MongoDB is ready for monitoring by an external service: http://localhost/mongo.php

Keep Performing,

Moshe Kaplan

Lessons from Facebook:

FB maybe is not the best example for quality design (probably each of us is likely to have at least a single error message a day).

However, since it's the 2nd largest site in the world with contsant exponential growth, it's a good lesson to have. Most interesting that their tech is team is only about 250 people and 30K servers, pretty amazing.


Not long ago a Jeff Rothschild, the Vice President of Technology at Facebook gave a presentation in UC San Diego. You can find a detailed summary by Prof. Amin Vahdat, but I'll make several top comments that I found useful.

1. FB Software development: mostly PHP (complied), but other languages are used as well.
   
2. Common interface between services using an internal development that was turned to open source: Thirft. This interface enables easy connections between the different languages.
   
3. Logging Framework that is not dependent on a central repository or its availability. FB is using Hadoop and well as Hive (that was developed there as well). The log size is growing at a 25TB a day.
   
4. Operational Monitoring: Separated from the logging mechanism 
5. LAN can be a bottleneck as well: expect for packet loss and packet drops in the LAN if you stress it too much.
   
6. CDN: Facebook is using external CDN to images distribution.
   
7. Dedicated file system named Haystack that combines simple storage along with cache directory: file system is accessed only once to get images from it, while directory structure is retrieved from the cache.
   
8. Most data is served from Memcached. Database is used mostly for Persistency and data replication between sites (Memcached is being heated by the MySQL itself):
1. Top challenge: keeping data consistent since Memcached can be messed easily (No search for keys is available).
2. Mixing information including sizes and types is better - making sure that load on CPU, Memory and etc is distributed equally.
   
9. Shared nothing - Keep your system independent - avoid a single bottleneck. Therefore, data is saved in a sharded MySQL from day 1. However, MySQL is used mostly for data persistency and not for conservative database usage pattern:
1. No Joins in the MySQL
2. Chosen due to good data consistency + Management Software
3. 4K servers
   
4. Data replication between sites is based on MySQL replication
5. Memcached is being heated based on the MySQL Replication using a custom API

One last thing, if you are interested in Facebook financials, as well the storage machines (NetApp low end 3070), sizing, traffic, servers, storage procurement and data center costs, take a look at TechCrunch's Michael Arrington post.

Keep Performing,
Moshe Kaplan. Performance Expert.

Tuning LAMP Architecture

It seems that more and more firms are moving to the LAMP architecture these days. Therefore, there is a buzz regarding Apache, Linux, PHP and MySQL tuning
However, since every technology has it own limitations, the need for extreme architectures that overcome their limitations is on the rise as well (I wrote and presented pretty much regarding Sharding, Cloud Computing, In Memory Databases in last few days).

However, it is always good to get back to basic and remember the small advices from smart people how to tune the products themselves. Therefore, I gathered several recommendations to help you get a little bit more from your own application:
LAMP tunning: I would add to this set of articles that you should make sure that you place all your rich content in a CDN, Amazon S3 or any other alternative and not on your own server.
Some information about Apache tunning and even more about HAProxy tuning (RockeTier software based favorite load balancer).

Keep Performing
Moshe Kaplan. RockeTier. The Cloud and Performance Experts.