Sharding Again

Click on the post title to read the full post and the comments.


It was a long time since we last discussed Sharding. 
Yesterday the Twitter sharding case study was presented at the Metacafe Knowledge sharing meetup by Gidi Meir Morris. Therefore, I think it is a good reason to refresh our minds.


Hey! Where Are All These Whales?
Twitter turned its web common architecture (that caused it many problems) based on Ruby on Rails web layer and MySQL into a 3 layer system that includes:

1. Web application server called Flapps.
2. Sharding middleware called Gizzard. This layer takes care of database requests and sends them to the target shard. It takes care of replication as well.
3. Old good (not so scalable) MySQL in a horizontal sharding mode.

But hey, it is better taking a look in this great Prezi presentation (use the More>Full Screen to better watch the presentation):

A bird's nest:A primer on FlockDB & Gizzard on Prezi

Bottom Line
Highly scalable databases are becoming commodity. In the Era of Cassandra and Gizzard, Google is starting to lose its competitive edge and high priced Oracle, IBM and Microsoft database are no longer a must for a startup. Will it affect these companies' bottom lines? Only time will tell.


Keep Performing,
Moshe Kaplan

Secure your Amazon EC2 Database

Hi,

An important issue that must be addressed in every system, and for sure if it is an internet based, is security. When we build high scalable systems, we face this issue time after time.

Therefore, when you install a new system in Amazon EC2, with a MySQL in it, you should consider how to secure and restrict the access to the system and the database.

Amazon provides set of solutions in order to support these needs:
1. Firewall: Amazon EC2 provides a firewall solution. This inbound firewall is configured in a default deny mode and you must explicitly open any ports to allow inbound traffic. The traffic may be restricted by protocol, by service port, as well as by source IP address (individual IP or CIDR block). More can be found here.
2. Security Groups: Security group is the easiest way to aggregate several EC2 servers based on rule, access needs and functionality: Database, application, web, etc (if you are familiar with VLAN segments and FW legs, it's very similar indeed).
Every server is associate at start up with a security group, so access to this server is restricted based on the group definitions. This can be done by defining FW rules between Amazon EC2 security groups and between a security group and the world (for example HTTP may be enabled from the world to the web servers security groups, while the database will enable access in 3306 to the application servers security group and your company static IP)
3. Anti DDoS, IP spoofing and other issues are addressed here

So how should I start?
1. Define you servers roles and the security groups. Tip: usually it should be the same as your AMI
2. Define the FW rules between the groups and the logical rules between each other, the world, your secured office connection (static IP is recommanded) and every other relevant 3rd party.
3. Define the security groups in Amazon (see ec2-...-group RTFM)
4. Define you FW rules. see more here to learn the exact syntax and in the ec2-authorize RTFM

Best,
Moshe. RockeTier, the performance experts