Sep 6, 2016

Ruby is Dead! (You nead to take care of its memory issues)

One of the common problems with Ruby is its memory usage, just like JVM based languages (yes Java, I'm talking about you). Actually Ruby just like Java is based on garbage collector. This GC if not used correctly, can cause "stop the world scenarios" (as we can see in the attached diagram). In this cases, GC actually stops responding while consuming servers whole CPU and in some cases even causes server reboot.
Unicorn and Ruby memory leak causes server downtime
What can be done?

1. Reboot your Ruby Periodically

If you are using the popular Unicorn web server, memory issues may be even more severe, as Unicorn forks. During forking, it's copying the whole memory footprint of the parent (Copy on Write or CoW). Therefore, you may use the "Unicorn Worker Killer" gem that will monitor your server and gracefully will restart specific worker when memory reaches a new high or specific number of requrests were served. Since the gem supports randomization, it assures in a very high probablilty that the server will continue server.

Installing the killer:
gem 'unicorn-worker-killer'

Add to above "require ::File.expand_path('../config/environment', __FILE__)"
# Unicorn self-process killer
require 'unicorn/worker_killer'

And decide your worker graceful reboot method: 
# Max requests per worker
use Unicorn::WorkerKiller::MaxRequests, 3072, 4096
# Max memory size (RSS) per worker
use Unicorn::WorkerKiller::Oom, (192*(1024**2)), (256*(1024**2))
If you are using Ruby 2.X, you can exploit the CoW by better configuring Unicorn.
config/unicorn.rb (and a detailed explenation on setting at sirupsen)
 - worker_processes: 1x your cores
 - timeout: worker request timeout, should be 15 to 30 sec max
 - preload_app: enables CoW, but requires managning connect/disconnect on fork

3. Take care of GC configuration
You can take a look at a detailed discussion at collective idea's post.

Bottom Line
Dynamic languages has their downs, yet w/ the right design you can keep them up and running,

Keep Performing,

Jun 27, 2016

Neo4j Cluster Performance Tuning

Neo4j is one of the leading graph database these days, and it is very popular in recommendation systems, fraud detetion and social networks scenarios.

While the single instance (that is included in the community edition) performs very well (usually w/ under 10ms response time), You may face challenges in cluster mode

Why Should You Expect for Performance Degradation in Neo4j Cluster?
Two simple reasons:
  • Neo4j cluster is a Master-Slave cluster w/ an auto failover method (much like MongoDB). However, unlike MongoDB, primary node deteciton by client is done by a server side load balancer and not by the client's driver. 
  • Cluster replication is syncronious by default, unlike MongoDB async default behviour.
How Much will it Cost us?
  • The various nodes of the cluster should be behind a LB. If you select AWS ELB, it will cost you 7 to 30ms according to our measures below. The ELB latency is increased as request and response become larger (see details on the bottom). Note: impelementing a MongoDB like driver could be a great improvment and will help saving this latency and minimize system cost. and it's a great idea for a side project!
  • The nodes behind the ELB replicate changes.from master to clients. The level of syncronization is controlled by the ha.tx_push_factor parameter w/ a default value of 1. This parameter controls the number of slaves that should recieve the commit before answering the client. By setting it to 0, you avoid syncronization and get a similar result to a single node. Changing the factor wil save 70ms at average (and much more at peak time), and will leave us w/ an average 40ms per query (inc. ELB cost).
You can find the diffrences below, where in the tested environment a community edition instance was replaces by a 3 nodes cluster behind an ELB:
  • In the left section you can see an average of 9ms in the initial state (single community edition instance)
  • In the middle you can see a flactuating response time of 40ms (reads) to 300ms (writes) for a 3 nodes cluster behind ELB w./  ha.tx_push_factor parameter w/ default value 1.
  • In the right section you can see a steady 40ms for both reads and writes for or a 3 nodes cluster behind ELB w./  ha.tx_push_factor parameter w/ value set to 0 (async replication).
Neo4j performance as measured by DataDog client side metrics
Bottom Line
HA have some cost by its side. Better implmementation of the load balancing and right seleciton of syncronization model, can help you gain the needed performance

Keep Performing,

Measure from Inside the Server:
> mytime="$(time ( curl http://localhost:7474/db/manage/server/ha/master ) 2>&1 1>/dev/null )"
> echo "$mytime"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     4    0     4    0     0    581      0 --:--:-- --:--:-- --:--:--   666

real    0m0.006s
user    0m0.005s
sys     0m0.000s

Measure from the Application Server Direct to Master:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     4    0     4    0     0    988      0 --:--:-- --:--:-- --:--:--  1333

real    0m0.006s
user    0m0.000s
sys     0m0.000s

Measure from the Application Server to Master through the ELB:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     4    0     4    0     0    417      0 --:--:-- --:--:-- --:--:--   444

real    0m0.015s
user    0m0.000s
sys     0m0.000s

Some More Measures to Gain Data to Explore the Neo4j Performance:
  • Enable slow log query and filter server time
  • Install monitoring like Datadog in the application level:

Mar 2, 2016

Providing MongoDB User Granular Access to User Cluster

Unlike a single instance MongoDB setup or even a ReplicaSet one, when it gets to a Sharded installation, things may get thougher.

For example, if you gave a user a reading permissions to use MongoChef (a most recommended MongoDB client), when it comes to a clustered intallation, in order to avoid the "not authorized to run inprog" error when running db.currentOp(), you should provide the user with some more permissions (in this case the inprog permissions).

Actually it is pretty simple, but it is also a good example for a secured environment management:

Providing inprog Permissions

1. Get to the admin database
use admin; 

2. Authorize as a permitted user

3. Create a new role that will have permissions to manage the processes

role: "manageOpRole", 
privileges: [ 

resource: { cluster: true }, 
actions: [ "killop", "inprog" ] 

resource: { db: "", collection: "" }, 
actions: [ "killCursors" ] 

roles: [] 


4. Provide the permissions to the user:
      { role: "manageOpRole", db: "admin" }

5. Authenticate as the reading user

6. Verify things actually work! (or doing the definition of done);

Bottom Line
Simple, tested and secured like we always love our environments!

Keep Performing,
Moshe Kaplan

Feb 25, 2016

Spark, Python and Windows

Yetxtit may not be common, but if you want a quick start for a Windows guy on the hotest Big Data platform around, you will find this tutorial relevant for you:

Get the Needed Prerequistes Software
git (for building Spark)
Install and verify git was added to your path

Python 3.5.1
Or even better, try the Anaconda version

Define Java
Get Java 8 (Oracle JDK):

Set JAVA_HOME environment variable to c:\Program Files\Java\jdk1.8.0_73\ (don't use double quotes if you are used to it)

Prepare SBT
Get SBT 0.13.11 (for building spark compilation)

Spark build is huge, therefore we need to increase the memory limit of the sbt from 256MB to something larger (2048MB for example, but saw some cases where 6GB were needed). This can be performed by modify the sbt runner file (C:\Program Files (x86)\sbt\bin\sbt.bat):
"%_JAVACMD%" %_JAVA_OPTS% %SBT_OPTS% -Xmx2048m -Xms2048m -cp "%SBT_HOME%sbt-launch.jar" xsbt.boot.Boot %*

Or better, change these settings at C:\Program Files (x86)\sbt\conf\sbtconfig.txt

Buid Spark
Download Spark 1.6 (No need to intall now, see details below)
  1. Extract the source using 7zip (or anything else that can use tgz files) for example to C:\Spark
  2. Build spark:
    1. cd C:\Spark
    2. "c:\Program Files (x86)\sbt\bin\sbt.bat" package
    3. "c:\Program Files (x86)\sbt\bin\sbt.bat" assembly
Get Winutils
Create Hadoop folder (for example c:\hadoop)
Define HADOOP_HOME environment variable (c:\hadoop)
Download to c:\hadoop\bin the winutils.exe file

Run It Like a Pro
Create a test file c:\spark\verify.txt and enter to it the following text: this is a trial
Run pyspark:
> c:\spark\bin\pyspark
In pyspark enter the following code:
text_file = sc.textFile("file://c:/spark/verify.txt")
The file should be printed

Keep Performing,
Moshe Kaplan

Nov 25, 2015

Apache htaccess Debugging Ugly. This Will Save Your A$$...

If you ever created redirection rules in Apache htaccess or configuration file, you probably know that things can easily turn ugly. Without debugging tools and with long testing cycle, the debugging can be painful.

The htaccess tester tool can solve your issues: just place your requested URL and the actual htaccess that is being used, and you will get the actual result.

Keep Performing,
Moshe Kaplan

Sep 20, 2015

5 Immidiate Steps to Take Care of Your MongoDB Performance

Do you face some performance issues in your MongoDB setup?
In this case use the following steps to provide some first aid to your system and gain some space for a long term architecture (such as Sharding)

Step 1: Enable slow queries
Get intelligence about your system behavior and performance bottlenecks. Usually there is a high correlation between the slow queries and your performance bottleneck, so use the following method to enable your system profiling collection:

db.setProfilingLevel(1, 100);

Step 2: Use explain

Explore the problematic queries using explain. You can also use mtools to analyze the logged queries to find high frequent ones.

Step 3: Create indexes
Your analysis should result with new indexes in order to improve the queries
Don't forget to use index buildup in the background to avoid collections locking and system downtime.

Step 4: Use sparse indexes to reduce the size of the indexes
If you use sparse documents, and heavily using the $exists key words in your queries, using sparse indexes (that includes only documents that includes your field) can minimize your index size the boost your query performance.

Step 5: Use secondary preferred to offload queries to slaves
You probably have a replica set and it's waste of resources not using your slaves for read queries (especially for reporting and search operations).
By changing your connection string to secondary preferred, your application will try to run read queries on the slaves before doing that on your master.
Bottom Line
Using these simple method, you can gain time and space before hitting the wall.

Keep Performing,
Moshe Kaplan

Sep 2, 2015

Prepare for Failure in Your AWS Environment

In the cloud everything can happen. 
Actually everything will happen.

Therefore, in your design, you should be ready for failures: even if you expect your disk mounts to be there for you, they might not be. And you are doing auto scaling, it is most likely that one in a time they won't be there for you.

Therefore, to avoid hanging servers due to failure to mount disks and bad messages as the follow: "The disk drive for /tmp is not ready yet or not present", make sure your servers are not bound by your disks (otherwise you will not be able to contact your servers, or your OpsWorks will notify you that the server is booting forever).

Avoiding Waiting for Your Mount
The secret is a small option: nobootwait that will make sure your server is not waiting for the mount to be ready. You can configure it in your /etc/fstab, or even better in your Chef recipe:
mount "/tmp" do
  device ""
  fstype "nfs"
  options "rw,nobootwait"
  action [:mount, :enable]

Bottom Line
The right design will help you keep you system running in a cloud based environment

Keep Performing,
Moshe Kaplan


Intense Debate Comments

Ratings and Recommendations