Keep Your Website Up and Running During an Upgrade

All of us want to have our website up and running.
However, upgrading a software version requires downtime (even if it minimal).
What should we do?

The answer is simple: follow the following process and gain a highly available system based on load balancing and session off loading, as well as a no downtime procedure to upgrade your system.

Keep Your Site Up and Running @ AWS

1. Place more than a single web instance behind a load balancer. The Elastic Load Balancer (ELB) will be fine for that.
2. Make sure you are using session offloading in your system implementation.
3. Install elbcli on your machine sudo apt-get -y install elbcli
4. Create a credentials file /var/keys/aws_credential_file and secure it sudo chmod 600 /var/keys/aws_credential_file
   AWSAccessKeyId=AKIAblablabla
   AWSSecretKey=blablablablablablablablabla
5. Upgrade the servers one after the another.
6. Before upgrading a server, take it out of the ELB
   sudo elb-deregister-instances-from-lb MyLoadBalancer --instances i-4e05f721  --aws-credential-file=/var/keys/aws_credential_file
7. And after completing the server upgrade take it back in.
   sudo elb-register-instances-with-lb MyLoadBalancer --instances i-4e05f721  --aws-credential-file=/var/keys/aws_credential_file

Bottom Line

A simple process and careful design will keep you system up and running.

Keep Performing,

Moshe Kaplan

Scale Out Patterns for OpenStack (and other Cloud) based Systems

It is a common question these days how one should design their next system to support elastic and growth requirements in the cloud era.

As I got this specific query today, I would like to share with you my answer based on various materials I created in the last few years:

1. Avoid File Storage SPOF: Use AWS S3 or its equivalent open source OpenStack Swift as a central file server repository when needed
How to use OpenStack Swift for your business case

2. Avoid Data Store SPOF: Use clustered data store that recovers automatically w/o a DBA or an operator intervention. Fine examples are AWS MySQL RDS, MongoDB and Cassandra
MongoDB HA concepts

3. Avoid Static Servers: Leverage Autoscale features or customize it for your needs
How to implement your application logic to auto scale your app

4. Avoid Messing with Cache: Use a central sharded cache to avoid cache revocation and reduce data stores load using MongoDB, CouchBase or Redis.

VP R&D Open Seminar: Caching from Moshe Kaplan

5. Offload your servers sessions: in order to avoid users log off and lost transactions:
How to implement a session offloading using a central store

6. Avoid Service Downtime due to Servers Downtime: More issues that can be found in my extensive presentation:
 - Use DNS Load Balancing for Geo LB and DRP (Slide 10)
 - Use CDN to offload network traffic (Slides 16-19)
 - Perform Session Offloading by cookies or a central store (Slides 62-65)

Web systems architecture, Performance and More from Moshe Kaplan

Bottom Line
You can scale your app! just follow the right recommendations

Keep Performing,
Moshe Kaplan

Auto Scaling your Workers based on Queue Length

Having an image or video processing engine?

Having a web crawling solution?

Doing OCR on media?

If so, this post is probably for you.

The Producer-Consumer Auto Scaling Problem

Back at school, you probably learnt how to manage the queue of waiting tasks in a producer-consumer problem. You may learnt how to avoid expired tasks and double processing of same tasks.

However, back then we all assumed the number of workers and producers is fixed.

Well, it is not true any more...

If you system includes a backend processor that its load may vary based on clients demand, you may need to auto scale it. Back then, you needed to purchase new servers. These days it just about launching new instances in your favorite cloud.

Adjusting AWS Auto Scaling to Support this Pattern

AWS Auto Scaling solution is known as the best of its kind in the industry. It supports:

1. Dynamic scaling based on instances load (e.g CPU).
2. Automatically launching and terminating instances.
3. Supporting both on demand (expensive) instances and Spot (marginal cost) instances.

However, it does not support complex decisions such as queue length/instances length ratio out of the box.

Fortunately, AWS provides a complete set of API that let us define a solution that its general flow is described here and here:

1. Create a base AMI to a worker/consumer machine
1. If your code tends to change you may define a method that will update to the latest code when machine is being launched
2. A better solution may be using a DevOps platform such as Puppet or Chef.
2. Define auto scaling initial configuration with an initial size of minimum and maximum number of instances (N = 1).
3. Define a spot instance price: make it reasonable (not too high to avoid extra cost, and not too low to actually launch instances when needed).
4. Create a cron that will run every 1 min and will check the queue length. This cron will terminate or launch instances by adjusting the auto scaling configuration minimum and maximum number of instances (N).

The Algorithm

You need to take several decisions in order to decide how many instances you want on air:

1. What is the ratio of tasks in queue (Q) to running instances (for example R = 2)
2. How quick do you want to reach this ratio. The quicker you do, the higher it will cost (instances that are billed hourly, may be terminated and relaunched again only few minutes after that).

Two examples for a possible algorithms:

1. Keeping the ratio of queue length and running instances constant: for example: N = max[1, Q/R]
2. Soften the ratio by using previous calculated number N': N = (N' + max[1, Q/R])/2

Bottom Line

Auto Scaling is an amazing tool, and by adjusting it we can easily solve complex issues.

Keep Performing,

Moshe Kaplan 

Introduction to MongoDB: The Complete Presentation

I had two great lectures regarding MongoDB and how to best utilize it in the last week. Therefore, I decided to share the presentation with you.

Key Presentation Topics

1. MongoDB Background: Company, Customers and the roots of NoSQL
2. Why more people are choosing MongoDB?
3. Data Design for NoSQL
4. MongoDB Installation
5. Basic DDL and DML syntax
6. MEAN (MongoDB, Express, Angular, node.js)
7. Best Practices for MongoDB migration

P.S Don't miss my Hebrew Big Data Webinar at July 7th, 2014

Keep Performing,
Moshe Kaplan

6 Easy Steps to Configure MongoDB Replication Set

In this tutorial we'll create a 3 nodes cluster, where the first serves as a primary node, and second as a failover node and the third as an Arbiter

1. Setup Mongo and Set a Configuration File
In all the 3 servers adjust the configuration file /etc/mongod.conf:
#Select your replication set name

replSet=[replication_set_name]
#Select the replication log size

oplogSize=1024

replication:
   oplogSizeMB: 
   replSetName: 

Disable the bind_ip parameter to avoid binding to only 127.0.0.1 interface

#bind_ip

2. Restart All 3 mongod Daemons

> sudo service mongod restart

3. Create an Initial Configuration on the Primary
Login to the primary mongo and create an initial configuration. Please notice to use the private IP and not the loopback address (127.0.0.1):
> mongo

Primary> cfg = {"_id" : "[replication_set_name]", "members" : [{"_id" : 0,"host" : "[Primary_Host_IP]:27017"}]}

Primary> rs.initiate(cfg);

4. Add the Failover Instance to the Replication Set

Primary> rs.add("[Failover_Host_IP]:27017")

5. Add the Arbier Instance to the Replication Set

Primary> rs.addArb("[Arbiter_Host_IP]:27017")

6. Verify the Replication Set Status
Primary> rs.status()

Bottom Line
I wish every data cluster setup was as easy as a setup of a MongoDB replication set.

Keep Performing,
Moshe Kaplan

Consider Using SSL? Don't Forget Choosing the Right CDN for that!

Fact #1: More and More Sites are using SSL to Secure Their Users' Transactions
Everybody requires security and privacy these days.
If you don't believe it, take a look at Google, Facebook and Twitter. All them are using HTTPS and SSL to secure all their webpages and API calls including simple feed presentation and search page presentation.
And yes, this fact is still valid even in the post Heartbleed era.

Fact #2: Sites and Widgets are Required for Quicker than Light Loading Time
In the online business, time is money. And faster webpage load times worth a lot of money.

Fact #3: Webpages Secured by SSL have Poor Performance
When you surf for the first time to a website (or take a look at a widget), you are required to have several phases in order to view the website:

1. Resolve the Site DNS.
2. Call for the first Web page.
1. Perform a TCP handshake.
2. Perform a SSL handshake.
3. Retrieve the page itself (+ encryption overhead)
3. Call the embedded resources: images, CSS and JavaScript files (+ encryption overhead).

As you can see, the initial loading of a regular webpage is not short at all. Adding the SSL handshake to this process as well as the encryption and decryption and the overhead on the content, results in even longer times.

What Can be Done?

A common solution is choosing a SSL offloading device such as Radware's Alteon. This device will shorten the encryption and decryption times at the server side. However, it will not reduce the SSL handshake time or shorten the time needed to transfer the page encryption overhead.

The only way to shorten this time is shortening the round trip time between the users and your servers. If this is sounds like a CDN case study, you are right.

CDN is a Key Solution to Managing HTTPS Traffic

Modern CDN solutions support SSL termination at the edge. Therefore, the SSL handshake time can reduced from up to 1 second to several dozens of ms (see in the figures).

This is a major plus to the benefit of shortening the static files serving time by serving a cached copy from the CDN edge.

The good news are that this benefit is valid for both static files and dynamic calls.

Figure 1: Lightweight HTTP REST API Call: Local (Left waterfall) vs Remote (right waterfall), where local call was done from a server located at the same data center as the web server and the remote was done from a remote location with a 200ms round trip to the web server. As we can see most the time is due to the network round trips rather than server processing. Please neglect the initial DNS resolve time.

Figure 2: Same call this time using HTTPS. We can see that the original waiting time was split to two, while SSL connection time doubled it. We can see that in this case as well the SSL processing time at server is neglectable (22ms) while the round trip costs us about 420ms. Please avoid the DNS resolve time in this case as well.

CDN Selection for HTTPS Traffic Cases

While many CDN services support SSL offloading to their own domain (e.g https://your_domain.cdn_provider.com), you probably would like to use your own domain name (e.g https://your_domain.com). Therefore, you should verify that the CDN provider supports custom SSL certificates. Common cloud CDN providers such as AWS and MaxCDN are known to support it, while providers like Microsoft Azure don't. 

Bottom Line

CDN are a corner stone to every web scale deployment these days, and many times you will find they solve issues you were not expecting for them before.

Keep Performing,

Moshe Kaplan

Looking for PostgreSQL Performance Issues

As traffic goes up, even your PostgreSQL may become a bottleneck.
In this cases it is important to analyze the traffic and understand the usage pattern. The way you will be able to tune the system to meet the challenge.

Understand Usage Pattern at Peak Time
Use PostreSQL top project to get key usage patterns in real time:

1. Current active SQL statements running
2. Query plans
3. Locks
4. User tables and indexes statistics

Understand Overall Usage Pattern

To get a broad insight of PostgreSQL behavior use pgFouine. This tool analyzes the PostgreSQL logs and provides detailed usage patterns reports such as leading queries, duration, queries by type and queries patterns.

You can get some of these metrics by querying the pg_catalog schema (such as pg_stat_user_tables and pg_stat_user_indexes), and use log_statement to analyze all queries.

Enable Slow Queries

Probably #1 tool to eliminate performance issues:

1. Add pg_stat_statements  to shared_preload_libraries statement @ postresql.conf
2. Restart PostgreSQL daemon.
3. use the pg_stat_statements view to pinpoint the bottlenecks:

SELECT query, calls, total_time, rows, 100.0 * shared_blks_hit /
nullif(shared_blks_hit + shared_blks_read, 0) AS hit_percent
FROM pg_stat_statements ORDER BY total_time DESC LIMIT 5;


Explain the Execution Plans
Use the Explain statement to analyze slow queries execution paths and eliminate them:

EXPLAIN (FORMAT JSON) SELECT * FROM table_name;

Bottom Line
Using these great tools, you can boost your PostgreSQL and meet the business challenges

Keep Performing,
Moshe Kaplan