Create Your Postgres Cluster in 6 Steps

Setting a Postgres cluster (or even a Master-Slave configuration) was not an easy task before version Postgres 9.0.

In the following lines you will find a short guideline that will help you create you Postgres HA solution:

1. Install postgres on both machines
sudo yum -y install postgresql-server
sudo service postgresql initdb
sudo service postgresql start
sudo sudo -u postgres psql

2. Stop the Postgres instance on both machines, delete the /var/lib/postgresql folder at the slave, and recover it by copy the files from the master.

3. Do initial configuration on both machines to enable logging:
sudo mkdir /var/log/postgresql/
sudo chown -R postgres:postgres /var/log/postgresql/
sudo vi /var/lib/pgsql9/data/postgresql.conf
> log_directory = '/var/log/postgresql'

4. Configure the Master and Slave to enable replication
sudo vi /var/lib/pgsql9/data/postgresql.conf @ Master (enable logging and listening)
> listen_addresses = '*'
> wal_level = hot_standby

> max_wal_senders = 3

sudo vi /var/lib/pgsql9/data/pg_hba.conf @ Master (enable access to the server, tune it, if your cluster is not in a secure environment):
> host  replication   all   192.168.0.2/32      trust
> host    all             all             0.0.0.0/0               trust

sudo vi /var/lib/pgsql9/data/postgresql.conf @ Slave (turn on standby)
> hot_standby = on

sudo vi /var/lib/pgsql9/data/recovery.conf @ Slave  (turn on standby a new file)
> standby_mode = 'on'

> primary_conninfo = 'host=192.168.0.1'

5. Finally, start your slave, and then your master on both sides:
sudo service postgresql start

6. And check replication on by creating database at the Master, and check it at the slave
master> sudo sudo -u postgres psql
CREATE DATABASE a
\l
                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
 a         | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |

slave>  sudo sudo -u postgres psql
\l
                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
 a         | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |

P.S If you are using a CentOS and yum (and not AWS), the following link will be useful for you.

Recovering from Cluster Crash (Thanks for Yariv Rachmani for contribution)
If you find the following errors in the your Postgres error:
> 2015-01-01 19:41:14.419 IST >LOG: started streaming WAL from primary at 0/86000000 on timeline 1
> 2015-01-01 19:41:14.420 IST >FATAL: could not receive data from WAL stream: ERROR: requested WAL segment 000000010000000000000086 has already been removed

In this case the quickest procedure to recover the slave:

1. Stop the service: > service postgresql-9.3 stop
2. Backup the data folder: > cp -prf /var/lib/pgsql/9.3/data /var/lib/pgsql/9.3/data-org
3. Remove the data folder: > rm -rf data
4. Copy the data folder from master: > pg_basebackup -h -D /var/lib/pgsql/9.3/data -U postgres -P -v -x
5. Change permissions: > chown -Rf postgres:postgres data
6. Copy the configuration file from the backup:> cp -p data-org/*.conf data/
7. Start again the slave: > service postgresql-9.3 start
8. Verify the slave log file

Bottom Line
Postgres replication is not so complicated starting with Postgres 9.0. Now, it's your turn to keep your system uptime!

Keep Performing,
Moshe Kaplan

Monitoring Your SQL Server w/ Monitis

One of the nicest monitoring products around is Monitis.com
If you are using MySQL, you will probably find out that some of the thresholds there should be modified or tuned (or just removed, since they are not critical to your day to day operations):

Key Cache Writes Rate
Actual value: 0
Proposed range: 75-100
What should be done: If you are not using MyISAM, ignore this one. If you do use it, modify the key_buffer_size variable and increase it to accommodate the index in the memory.

Indexes Usage
Actual value: 5
Proposed range: 80-100
What should be done: Indexes usage is a key indicator to understand if you write your queries in a way that utilizes your indexes and if you defined correct indexes to support your questions.
Enable your slow log query and examine your queries. 

Key Cache Hit Rate
Actual value: N/A
Proposed range: 95-100
What should be done: Seems that this is not a critical variable, since you must check it is relevant in your case. If you believe that your system business case should use most data out of cache and value is low, modify the key_buffer (MyISAM) or innodb_buffer_pool_size (InnoDB).

Table Cache Hit Rate
Actual value: N/A
Proposed range: 85-100
What should be done: See the one above

Bottom Line
Monitoring system is critical component of critical systems. However, in most cases you should verify that the default configuration matches your business case (and if it is the case, take care the alerts!).

Keep Performing,
Moshe Kaplan

MySQL and Application Servers Connection Pool

As we discussed in the past, MySQL is very lightweight in terms of connections creation and closure overhead, and therefore classic LAMP installations rarely use it.
However, some application frameworks such as Java and Ruby tend to use it and in these cases it is recommended to adjust some configurations in order to avoid long connections cuts.

The Pattern
Since MySQL default is 8 hours timeout, this pattern is usually when connection pools are configured with high number of connections (and low utilization) or when pattern usage is limited to defined hours (daytime for example).

How to Detect this Pattern?
By using MySQLTuner, you will find out that Connections Aborted parameter is too high (I've seen cases with as high as 80%).

What Should We Do?
Adjust the connections timeout, in order to avoid connection pool connections cutting:
#1 day timeout
SET GLOBAL wait_timeout = 86400
SET GLOBAL interactive_timeout = 86400

or in the my.cnf
wait_timeout = 86400
interactive_timeout = 86400

Bottom Line
Connection pools are not a native case for MySQL, but it does not mean your cannot support it wisely.

Keep Performing,
Moshe Kaplan

Some More MySQL Tuning

thread_handling = one-thread-per-connection

MySQL is designed for lightweight connection creation. Therefore, you may not use connection pooling. However, if you are a connection pooling fan (Java and Ruby devs, please raise your hands), don't forget to configure the MySQL for that:

http://www.mysqlperformanceblog.com/2010/10/27/mysql-limitations-part-4-one-thread-per-connection/

http://dev.mysql.com/doc/refman/5.6/en/thread-pool-plugin.html

In this case you should also avoid the thread_cache_size recommendation and use thread_cache_size = 0 (and not 8 for example).

Double Flushing and SSD

Does your data really being written to disk? or does it stuck in the OS caching?

innodb_flush_method = O_DIRECT: bypass the OS caching

innodb_flush_method = O_Sync: Makes sure disk is getting the call

innodb_flush_method = O_DSync: The last two options combined

http://stackoverflow.com/questions/2762697/mysql-innodbinnodb-flush-method

Large RAM configurations

If you have a lot of memory, and many connections (and threads), you will probably find out that your threads are waiting for your memory. In order to avoid it, you may split the InnoDB buffer pool size to smaller sections. Pros: each one manage its section, so data can be served from multiple memory sections (N times faster). Cons: you may find it memory inefficient, as data may be loaded twice to memory.

In any case, make sure innodb_buffer_pool_size/innodb_buffer_pool_instances > 1GB

Please notice that number selection should be based on actual system bottleneck.

Matching IOPS

SATA and SAS disk were providing 100 IOPS (and multipliers of it when using RAIDs). Therefore innodb_io_capacity default was 200.

If you consider migration to SSD based machines (that can provide up to 150K IOPS) you may find it useful to change it to the actual system IOPS capacity.

Please notice that if you use innodb_buffer_pool_instances, you should divide this number by the number of instances.

http://mysqlha.blogspot.com/2013/05/configuring-innodb-for-mysql-56.html

IO Threads

Consider adjusting the number of threads writing to/reading from disk to the number of your system CPU cores

innodb_read_io_threads = N
innodb_write_io_threads = N
http://dba.stackexchange.com/questions/33125/mysql-5-5-determining-correct-writeread-io-threads-on-high-end-system

Bottom Line

With MySQL endless configuration options, you can always find a great options to better tune your system.

Keep Performing,

Moshe Kaplan

Can OpenStack Object Store be a Base for a Video CDN?

Video CDN are amazing from technical aspect as we are talking about high scale systems with some unique business cases.
I would like to share with you some design aspects regarding these systems.

The Video CDN Case Studies
Video CDN includes two main case studies:

1. VOD Case Study: This is a long tail/high throughput scenario where you need a high capacity disks, where only a small portion of it will be used extensively. In order to create a cost effective solution you should have:
1. High capacity storage system with low IOPS needs. Servers with 24-36 2-3TB SATA disks will provide a up to 100TB raw storage with a tag price of $15K.
2. Replication and auto failover mechanism that can distribute content between several servers and can save us from using expensive RAID and Cluster solutions.
3. Caching/Proxy mechanism that will serve the head of the long tail from the memory.
2. Live Broadcast Case Study: This is a No Storage/high throughput scenario where you actually don't need any persistent storage (if a server fails, as soon as its get up again, the data will be no longer relevant). In order to create a cost effective solution you should have:
1. No significant storage.
2. High capacity RAM that should be sized according to:
1. The number of channels you are going to serve.
2. The number of resolutions your going to support (most relevant when you plan to support handhelds and not just widescreens).
3. The amount of time you are going to store (no more that 5 min are needed in case of live, and no more than 4 hours in case of start over).
3. In memory rapid replication (or ramdisk based) mechanism, that will replicate the incoming video to several machines.
4. HTTP interface to serve video chunks to end users.

Serving Static Content rather than Dynamic
Modern Video Encoding systems (such as Google's Widevine) support "Encrypt Once, Use Many", where the content is encrypted once, and decryption keys are distributed to secured clients based on a need to know basis.

Why OpenStack Swift/Object Store?
In short, OpenStack Swift is the OSS equivalent for Amazon propriety AWS S3: "Simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web."

OpenStack Swift/Object Store Benefits

1. Open stack is OSS and therefore easy to evaluate.
2. Active large scale deployments including RackSpace and Comcast.
3. Built in content distribution method that let you distribute the load between multiple servers based on rsync.
4. High availability based on data replication between multiple instances. On a server failure, you just need to take it out of the array and replace with a new one, while other servers keep serving users.
5. This mechanism help you avoid premium hardware such IO controllers and RAID mechanism.
6. Built in HA and DRP based on 5 independent zones.
7. Built in web server, that enables you serving static content as well as HTTP based video streams from the server itself, rather than implementing high end SAN.
8. Built in reverse proxy service that minimizes IO and maximizes throughput, based on Python and Memcache.
9. Built in authentication service
10. Target pricing of $0.4 per 1M servings and $0.055/GB per month if we take AWS as a benchmark.

OpenStack Object Store Architecture
The OpenStack Object Store architecture is well described in two layers:

1. The logical: Accounts (paying customers), Containers (folders) and Objects (blobs).
2. The physical: Zones (Independent clusters), Partitions (of data items), Rings (mapping between URLs and partitions and locations on disks) and Proxies.

Key Concepts

1. Account is actually an independent tenant, as it has its own data store (implemented by SQLite).
2. Replication is done based on large blocks, quorum and MD5.
3. Write to Disk before Expose to Users: When files are uploaded, they are first committed to disk at least at two zones, and then database is updated for availability (so don't expect sub second response for a write).

System Sizing 

Before testing the system you should be familiar with some key sizing metrics obtained by Korea Telecom:

1. Object Store Server Sizing: High capacity storage (36-48 2-3TB SATA drivers that will provide us up to 100TB per server), memory to cache the head of the long tail (24-48GB RAM), 2x1Gbps Eth to support ~500 concurrent requests for long tail request. A single high end CPU should do the work.
2. Proxy Server Sizing: Little storage (500GB SATA disk will do the work), memory to cache the head of the long tail (24 RAM), 2x10Gbps Eth to support ~5000 concurrent requests to support the head of the long tail requests.
3. Switches: you will need a nice backbone for this system. In order to avoid a backbone that is too large, splitting the system to several clusters is recommended.
4. Load Balancing: in order to avoid high end LB, you should use DNS LB, where frequent calls to the DNS are neglect-able relatively to the Media streaming

The Fast Lane: How to Start?
OpenStack Object Store is probably cost effective when looking for large installations as you may need at least 5 physical servers for object and containers store and another 2 for proxies. 
However, you can check the solution based on a single server installation (SAIO):

If you take the fast lane and AWS is the fast lane to your POC, feel free to use the following tips:

1. In the initial installation, some packages will miss in yum so:
1. sudo easy_install eventlet
2. sudo easy_install dnspython
3. sudo easy_install netifaces
4. sudo easy_install pastedeploy
2. No need to start the rsync service (just reboot the machine).
3. Start the service using sudo ./bin/startmain
4. Test the service using  the supload bash script to stimulate a client.

Working with the Web Services
There are 3 main ways to work with Swift web services:

1. AWS tools, as Swift is compliant with AWS S3.
2. HTTP calls as it is based on HTTP.
3. Swift client that streams your major needs.

Working with Swift Client

In the following example we assume a given user test.tester was defined with the password testing, and data is served over the proxy at port 8080

Get statistics:
swift -A http://test.example.com:8080/auth/v1.0 -U test:tester -K testing stat

Upload a file to the videos container:

sudo swift -A http://test.example.com:8080/auth/v1.0 -U test:tester -K testing upload videos ./demo.wvm

Provide read only permissions (Please notice that r: is stands to referral domain, so specifying any other * can help you save bandwidth and minimize content stealing):

sudo swift -A http://test.example.com:8080/auth/v1.0 -U test:tester -K testing post videos  -r '.r:*'

Download the file (where AUTH_test is the user account, videos is the containter and anonymous access was provided as detailed below):
curl http://test.example.com:8080/v1.0/AUTH_test/videos/demo.wvm

Public Access
In order to implement a read only public access your will need to take care of the following items

1. User Management
2. Define anonymous access: delay_auth_decision = 1
3. Configure folder ACL
4. Enable delayed authentication at the proxy configuration (/etc/swift/proxy-server.conf):

[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
# Delaying the auth decision is required to support token-less
# usage for anonymous referrers (‘.r:*’).
delay_auth_decision = 1

Working with Direct HTTP Calls

Get user/pwd

curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://test.example.com:8080/auth/v1.0

> X-Storage-Url: http://127.0.0.1:8080/v1/AUTH_test

> X-Auth-Token: AUTH_tk551e69a150f4439abf6789409f98a047

> Content-Type: text/html; charset=UTF-8

> X-Storage-Token: AUTH_tk551e69a150f4439abf6789409f98a047

Upload file
curl –X PUT -i \

    -H "X-Auth-Token: AUTH_tk26748f1d294343eab28d882a61395f2d" \
    -T /tmp/a.txt \
    https://storage.swiftdrive.com/v1/CF_xer7_343/dogs/JingleRocky.jpg

Bottom Line
OpenStack Object Store (Swift) is exciting tool for anyone who is working with large scale system and especially when talking about CDNs. 

Keep Performing,
Moshe Kaplan

Do You Really Need NoSQL and Big Data Solutions?

Big Data and NoSQL are the biggest buzz around...
Yet, are they the right solutions for your project?

Are You Eligible for Big Data?
As a rule of thumb, if your database is smaller than 100GB and your biggest table is less than 100M rows, you should avoid seeking Big Data solutions. In that case, make sure you well utilize your current RDBMS investments.

When Should You Choose RDBMS?
There are several other reasons to stick with RDBMS (yes, we are talking about MySQL, SQL Server, Oracle and other):

1. You must meet compliance and security procedures (cases: PCI compliance).
2. You need complex reporting based on joins between several tables.
3. You need transactions (cases: financial transactions).
4. You have established data analysts group that cannot be trained to other syntax.

When NoSQL Solutions Should Fit?
There are several high reasons to select NoSQL solution. Check if your case is eligible for it:

1. You are a full stack developer and just look for a persistence storage (cases: blogging system, multi choice exams).
2. You need a quick response from your storage solution based on a key value store (cases: algo trading and online stock exchanges bidding: DSP).
3. You must always return an answer, even if it's not the most updated one (cases: social networks, content management).
4. You need to provide a good enough answer rather the most accurate (cases: search engine).
5. Your data size is too big to be transferred over network (even over a 80Gbps Infiniband). In this cases a better approach is distributing the computation (cases: analytics, statistics).

Bottom Line
NoSQL solutions have expanded your toolbox. Now, you need to focus on selecting the right tool for your business case.

Keep Performing,
Moshe Kaplan

JAVA Production Systems Profiling Done Right!

If you are facing a Java system performance issue in production, and JProfiler is not the right tool for it, probably JMX monitoring using the VisualVM will do the work for you.

Technical
JMX usage from a remote machine can be frustrating. Therefore, please make sure that:

1. Your hostname is included in the /etc/hosts 
1. Get host name using hostname 
2. Add the host name after 127.0.0.1 in /etc/hosts
2. JMX is binded to the external IP:
1. Verify 127.0.0.1 is not presented at: netstat -na | grep 1099
2. If it does presented, add to your java command: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=

If everything is Okay, you will be able to run VisualVM from a remote machine and connect to the remote server.

VisualVM

Now, that you have your VisualVM up and running there are some items you should take a look at:

1. General CPU and memory graphs.
2. Sampler that enables you taking snapshots.
3. Snapshot analysis that enables you a hotspot presentation as well as deep.

Bottom Lin

My recommendation is to have snapshot of the process and then look at the hotspots tab for major calls with actual long CPU time. You should focus on these items.

Keep Performing,

Moshe Kaplan