mongoDB Configuration File Tuning

For those of us who are regular to MySQL, its configuration file (my.cnf) and the many options in it (some may say too many), mongoDB seems to be the simple.
With only few dozens of parameters in the configuration file (mongo.conf) and a dozen that are actually related to performance tuning, it may be a relatively short task to tune a mongoDB configuration file. Probably some of us will pay for that in production...

NOTE: this post does not refer the Sharding configuration.

What Can be Done?

Number of Connections
Like many other products (Apache httpd, MySQL...) the number of user connections can affect performance. mongoDB supports that using maxConns = N. Numbers can reach 20,000, but you should adjust it to your own server resources.

Write to Disk
Disk writing is usually a bottleneck in database systems. Therefore, wrtie to disk frequency and initial storage allocation can highly efhttp://stackoverflow.com/questions/8331099/what-is-the-javascript-engine-that-runs-mongodb-shellfect your system performance. Yet notice that delaying disk writing can effect your system recovery (many of you probably familiar with it from MySQL).
You should notice the following two options:

1. Delay journal (database log) write to disk using journalCommitInterval = 300. This parameter supports intervals between 2ms (slowest but safest for recovery) and 300ms (fastest but prone to recovery options). The default os 100ms, but you may increase it to 300ms to save resources.
2. Preallicate space at mongoDB startup by keeping noprealloc = false.

Disable Services
mongoDB provides many supporting services. Disabling some of them (if you do not use them), may help you save some CPU cycles:

1. BSON validation using objcheck = false.
2. HTTP Interface using nohttpinterface = true. The mongoDB HTTP interface exposes the database statistics and enables simple REST queries through port 28017.
3. Scripting Engine using noscripting = true. mongoDB is using SpiderMonkey JavaScript scripting engine to enable interactive shell. Somehow, I have a feeling that you are not going to disable it.
4. REST service using rest = false. See in the HTTP interface.
5. Profiling service (inc. slow queries logging) using profile = 0.

Bottom Line

mongoDB is a simple, yet very effective tool to solve many business needs. Doing some tuning will help you avoid issues during peak times.

Keep Performing,

Moshe Kaplan

Jenkins? Tomcat? Running Code w/ Permissions? This script will probably help you!

If you need to run some "SUDO" commands from Jenkins, you will probably need to first 1) Add jenkins user to SUDO and 2) Avoid passwords.

The following code will do it like a miracle:

#!/bin/sh
echo 'Providing SUDO script permissions to jenkins'
if ! grep -q 'jenkins' '/etc/sudoers'
then
        echo 'jenkins  ALL= NOPASSWD: ALL' >> /etc/sudoers
        echo 'Defaults:jenkins    !requiretty' >> /etc/sudoers
        echo 'Completed Successfully'
else
        echo 'Already Exists'
fi

Keep Performing,

Moshe Kaplan

How to Find MySQL Locks?

If your MySQL (Master or Slave) seems to be locked, you can always use one of the following methods to detect the locks and take care of them:

1. If it's an InnoDB based database use SHOW ENGINE INNODB STATUS\G
2. Check SHOW FULL PROCESSLIST; to verify if any queries are running now (or just stuck).
3. If you use MyISAM or just need some more information, enable the MySQL debug mode by using > mysqladmin -uuser -ppassword debug and then check for the output at the error log.
4. Consider using tools such as mysqlreport (it has a nice explanation here) or mysqltuner to verify your server configuration.

Bottom Line

Locked database is not a nice thing, but it can be detected and eliminated...

Keep Performing,

Moshe Kaplan

Scaling Your Web Farm and Keep Supporting User Generated Content using File Sharing (NFS)

Have you ever wanted to scale your web operations to multiple web servers and still needed a simple common place to share data between servers?

rsync

A very useful way for that is rsync that is commonly used to distribute the web server static content (HTML, PHP, CSS, and JavaScript files) between different servers.

What to do with User Generated Content?

When we want to store user generated content such images, documents or video that are uploaded by them from time to time we have three options:

1. Push them as BLOBs into the database (or your NoSQL system such as MongoDB).
2. Store them on disk that is accessible by several servers. 
3. Store them on a "File Server as a Service" like Amazon S3.

This time we'll focus on the second option:

DIY with Network Services

The basic solution for DIY is installing a file sharing daemon on one of your servers and share a folder of it to the other web servers.

There are three common file sharing protocols that may be used:

1. SAMBA (that is common and supported by Windows as well).
2. NFS (supported only on  *NIX machines).
3. CIFS that mostly used in Windows, but can be mounted to LINUX machines as well.

In this case I decided to focus on the NFS implementation, so stay tuned:

NFS Configuration

First configure the NFS server

1. Install the NFS RPM:  yum -y install nfs-utils
1. Start service: /etc/init.d/nfs start
2. Open the relevant ports in the iptables FW (2049 and 111):
1. iptables -I INPUT -p tcp -s 192.168.85.0/24 -m state --state NEW,RELATED,ESTABLISHED --dport 2049 -j ACCEPT
2. iptables -I INPUT -p udp -s 192.168.85.0/24 -m state --state NEW,RELATED,ESTABLISHED --dport 2049 -j ACCEPT
3. iptables -I INPUT -p tcp -s 192.168.85.0/24 -m state --state NEW,RELATED,ESTABLISHED --dport 111 -j ACCEPT
4. iptables -I INPUT -p udp -s 192.168.85.0/24 -m state --state NEW,RELATED,ESTABLISHED --dport 111 -j ACCEPT
3. Configure exported locations by editing /etc/exports according to the following examples:
1. Provide every server behind the firewall R/W to this folder: /path/to/directory *(rw)
1. Please notice that in this case you should provide the relevent permissions on disk (since the guest machine will use in default the nobody user to access this disk). For example: chmod -R 777 /path/to/directory
2. Provide a single server read only permission to this folder: /path/to/directory 192.168.2.21(ro)
4. Finally load these exported locations: /usr/sbin/exportfs -a

Finally mount this disk to all other servers:

1. Create local directory: mkdir /local/directory
2. Add line to /etc/fstab
1. SOURCE_SERVER:/path/to/directory /local/directory nfs
3. Mount the folder: mount /local/directory

Bottom Line

Scaling you system is possible with few simple steps, this was one of them

Keep Performing,

Moshe Kaplan

P.S In the DevOps world, scripts is everything. So you may use the following: 
Server:

yum -y install nfs-utils
/etc/init.d/nfs start
mkdir /path/to/directory
chmod -R 777 /path/to/directoryecho '/path/to/directory *(rw)' >> /etc/exports
/usr/sbin/exportfs -a


Client:

#!/bin/sh
mkdir /path/to/directory
echo "$1:/path/to/directory /path/to/directory nfs" >> /etc/fstab
mount /path/to/directory

mongoDB Sharding

If I should have made some safe bets on the near future, I would choose two: Hadoop and mongoDB. 

There is a huge demand for both technologies and many players consider these technologies as a foundation for their future products.

Sharding?
MySQL Sharding was a major issue for large scale installations and it is the same for mongoDB large installations.

Back to Basics
mongo is pretty similar to a regular database, but it has two main advantages: 1) Software engineers love it as it can easily be used for object persist-ency and 2) it support unstructured objects (documents) that can easily store different objects based on the same virtual class.

mongoDB terms

1. Database: database
2. Collections: very similar to tables.
3. Documents: very similar to rows. Yet, a document can be as flexible as a JSON document can be. For example, it may include 1 to many fields in the document itself.
4. mongod: a mongoDB instance or shard.
5. Chunk: a 64MB storage unit that stores documents.
6. Config database: Chunks to mongos mapping directory.

Why use sharding?

1. Support large dataset using commodity servers.
2. Support high IO requirements using commodity disks.

What are mongoDB sharding features?

1. Range-based Data Partitioning: a very similar method to MySQL partitioning. You should choose one or more fields (shard key) that sharding will be based on. You should choose a shard key according to the business logic, like splitting according to account id in a SaaS application.
2. Automatic Data Volume Distribution: mongoDB will take care of the shards balancing by itself according to the chosen shard key.
3. Transparent Query Routing: mongoDB takes care of queries map reduce to multiple shared by itself when a query does not match the shard key (very much like Hadoop).

Key Recommendations for mongoDB Sharding

1. Sufficient Carnality: choose a shard key that can be split later to more shards if a database size is getting too large (exceeds chunk size).
2. Uniform Distribution: choose a sharding key that will spread a in uniform distribution to avoid unbalanced design.
3. Distribute Write Operations: if you have a billing system, prefer to shard according to account id rather than shard according to billing month. Otherwise, in a given day, probably only a single shard will be used.
4. Query according to the shard key: if any of your queries will include the shard key, each of your queries will result in a single shard query. Otherwise, it will generate N queries (one per shard).

Technical Aspects for mongoDB Sharding

1. Every sharded collection must have an index that its first fields are the shard key (use shardCollection for that).
2. Chunk size default limit is 64MB
3. When a chunk reaches this limit, mongoDB will split it to two.
4. If chunks are not distributed uniformly, mongoDB will start migrating chunks between different mongos.
5. Cluster Balancer is taking care of this process.
6. Balancing can cause performance issues and therefore can be restricted to off peak hours (nights and weekends for example) using balancing windows.
7. The shards mapping to mongos is saved at the config database.
8. Replication should be considered as well  a complementary method.

Bottom Line

mongoDB brings to the table an out of the box sharding solution that can scale your operations. Now, you only need to analyze your needs and select the right solution for them.

Keep Performing,

Moshe Kaplan

MySQL Crash Course Presentation

In the last few weeks I lectured a MySQL crash course. The course topics covered almost all what is needed to make an initial ramp up when you get into MySQL: ERD, DDL, DML, installation, security, scaling, backup, Schema design, tuning, master slave and more...

The good news
I got a very good feedback from the students, so I decided to share with you the presentation itself:

MySQL crash course by moshe kaplan from Moshe Kaplan

Keep Performing,
Moshe Kaplan

How to use rsync for high availability environments?

What if...

• What if I have a large number of web servers and I need to deploy the same code on all of them?
• What if  I would like to enable high availability and redundancy for static user content such as images?
• What if I want to to backup files to a central storage?

A Swiss knife for static content replication
rsync was considered for a long time as the best solution for static content and code replication  in environments that consist of large number of servers.

rsync has a simple protocol that replicates a directory (one or more) on a single server to other servers. This can be achieved in two different methods (like SCP that it is based on):

• Push from the master to the slave: rsync [OPTION] … SRC [SRC] … [USER@]

  HOST:DEST
  
  

• Pull from the server by the slave: rsync [OPTION] … [USER@]HOST:SRC [DEST]

Can I perform a change on the destination directory?
Please note that the rsync protocol analyzes differences between two directories, and therefore probably will not match cases when you want to change the content of the destination directory.

How should I authenticate?
Use one of the two options:

1. Static user/pwd using sshpass for non interactive SSH based authentication.
2. PKI authentication using on-the-fly keys generation or pre-generated keys

Master-Master replication

Like in MySQL, Master-Master replication can be achieved by a dual Master-Slave connections setup . Please consider to enable only one of these connections. Then, during a failover, disable the replication. Last, when you bring the master server back enable the other replication.

Note: you may consider using OpenStack Storage for these purposes as well, as it provides an out the box solution for high availability and redundancy that easily supports multi master out of the box

Keep Performing,
Moshe Kaplan