MySQL. An Internet Standard.

As you probably already know, last week I gave a presentation at the Database2011 conference on how MySQL become an Internet Standard?

Database2011 my sql sharding

View more presentations from Moshe Kaplan.

The answer is simple: great community and good support and the proof of the pudding is in the eating: everyone uses it including Facebook, Twitter and Google (if you want to hear other opinions try follow the xaprb.com comments flow).


MySQL Limitiations
Yet, if you will explore the MySQL capabilities you may find out that it lucks real multi threading capabilities. its table size is limited to effective number of 50-100M rows, its SELECT performance is limited to 50 statements on a single table per second and its INSERT performance is limited to 700 INSERT statements per second (based on standard Amazon instance and InnoDB engine).
So, how can the Internet industry with dozens of thousands of SQL statements per second in a medium site use such a limited database?


Answers
The answer is simple: Use Sharding. Sharding is a method for slashing your tables into smaller ones and storing parts of the data in each one of them. There are several common strategies including:

1. Vertical Sharding: Tear a table into two, store only few columns in first table and the rest in the second table.
2. Horizontal Sharding: Place each group of rows in another table. Horizontal Sharding can be implemented using various algorithms that include static hashing, hashing with directory mapping, key based directory mapping and signature based mapping.

The main issue with Sharding is reporting. Since direct grouping cannot be supported when rows are spread in various tables, you should use Map Reduce based solutions to accomplish this task.


Emerging Products
In the last year several new COTS Sharding products were introduced to the market. These products use two mechanisms in order to get over the MySQL limitations:

1. Gizzard and ScaleBase (founded by Industry experts Doron Levari and Liran Zelkha) use the MySQL Proxy mechanism in order to implement a load balancing and Sharding solutions in front of regular MySQL databases.
2. Xeround and Akiban implemented a new type of storage engine based on the MySQL Storage engine API.

Will MySQL remain an Internet standard in the future?
MySQL will keep being an Internet standard, unless unexpected decisions will be taken by Oracle that owns the firm. Unwise marketing decision could harm the community and may cause a major shift to other solutions including NoSQL.


Bottom Line
If you have a great internet idea, you have all the needed tools to start.


Keep Performing,
Moshe Kaplan

Sharding Again

Click on the post title to read the full post and the comments.


It was a long time since we last discussed Sharding. 
Yesterday the Twitter sharding case study was presented at the Metacafe Knowledge sharing meetup by Gidi Meir Morris. Therefore, I think it is a good reason to refresh our minds.


Hey! Where Are All These Whales?
Twitter turned its web common architecture (that caused it many problems) based on Ruby on Rails web layer and MySQL into a 3 layer system that includes:

1. Web application server called Flapps.
2. Sharding middleware called Gizzard. This layer takes care of database requests and sends them to the target shard. It takes care of replication as well.
3. Old good (not so scalable) MySQL in a horizontal sharding mode.

But hey, it is better taking a look in this great Prezi presentation (use the More>Full Screen to better watch the presentation):

A bird's nest:A primer on FlockDB & Gizzard on Prezi

Bottom Line
Highly scalable databases are becoming commodity. In the Era of Cassandra and Gizzard, Google is starting to lose its competitive edge and high priced Oracle, IBM and Microsoft database are no longer a must for a startup. Will it affect these companies' bottom lines? Only time will tell.


Keep Performing,
Moshe Kaplan

Open is the New Black (hmm.... Standard)

Click on the post title to read the full post and the comments.

Only few years ago when a technological company was lagging behind in a new niche that was turned into a mainstream, it had a magic move: creating a new standard.

Standard: The Magic Touch
Many times the standard technology was less efficient, less robust and with less users than the technology that was advanced by the market leader. Yet, it had a magic charm: it was open, and most the players in the market could use it without royalties or other fees. That was including #2, #3 in the market as well as many other giants that were not focused in that market, but had to integrate and expand their product support to that technology.
It worked for DEC with Ethernet (vs. IBM's Token Ring); it was great for NetApp that chose iSCSI with 3% market share vs. EMC leading Fiber Channel protocol and it was like a charm for a small silicon valley firm named CISCO that chosen IP.

How do you Standardize Software?
Let say you are a software company that is lagging behind. Anyone said Yahoo! vs. Google in the internet market? Yahoo! had a great idea to scale out using a BigTable like product, but had no deep pockets for that.
Another example? Facebook? What if you had the largest image store publicly available, yet you have only few hundred developers and you need a new scalable database. What do you do?

Open: The New Magic Touch
Well, both Yahoo! and Facebook chose Open Source their infrastructure products creating two of the leading infrastructure products in todays world: Apache Hadoop and Apache Cassandra.

And Cloud, What about the Cloud?
Well it seems that last week Rackspace move: establishing OpenStack.org and contributing major parts of its propriety cloud infrastructure code to this project, is a clear say that being #2 in the IaaS market (see Jack of all Clouds latest report) that is being invaded by major players such as VMware, Google, Salesforce.com and Microsoft is not an easy task even to such a major player in the old hosting business. (If you want to know more regarding this move, see Geva Perry's analysis).

Bottom Line
It seems that the cloud market and mostly the IaaS one is getting to their mainstream phase, where scale is a top factor and community is a major factor in survival.

Would you like to share you opinion? Can you add more information? Feel free to comment!

Keep Performing,
Moshe Kaplan