Lessons from Facebook:

FB maybe is not the best example for quality design (probably each of us is likely to have at least a single error message a day).

However, since it's the 2nd largest site in the world with contsant exponential growth, it's a good lesson to have. Most interesting that their tech is team is only about 250 people and 30K servers, pretty amazing.


Not long ago a Jeff Rothschild, the Vice President of Technology at Facebook gave a presentation in UC San Diego. You can find a detailed summary by Prof. Amin Vahdat, but I'll make several top comments that I found useful.

1. FB Software development: mostly PHP (complied), but other languages are used as well.
   
2. Common interface between services using an internal development that was turned to open source: Thirft. This interface enables easy connections between the different languages.
   
3. Logging Framework that is not dependent on a central repository or its availability. FB is using Hadoop and well as Hive (that was developed there as well). The log size is growing at a 25TB a day.
   
4. Operational Monitoring: Separated from the logging mechanism 
5. LAN can be a bottleneck as well: expect for packet loss and packet drops in the LAN if you stress it too much.
   
6. CDN: Facebook is using external CDN to images distribution.
   
7. Dedicated file system named Haystack that combines simple storage along with cache directory: file system is accessed only once to get images from it, while directory structure is retrieved from the cache.
   
8. Most data is served from Memcached. Database is used mostly for Persistency and data replication between sites (Memcached is being heated by the MySQL itself):
1. Top challenge: keeping data consistent since Memcached can be messed easily (No search for keys is available).
2. Mixing information including sizes and types is better - making sure that load on CPU, Memory and etc is distributed equally.
   
9. Shared nothing - Keep your system independent - avoid a single bottleneck. Therefore, data is saved in a sharded MySQL from day 1. However, MySQL is used mostly for data persistency and not for conservative database usage pattern:
1. No Joins in the MySQL
2. Chosen due to good data consistency + Management Software
3. 4K servers
   
4. Data replication between sites is based on MySQL replication
5. Memcached is being heated based on the MySQL Replication using a custom API

One last thing, if you are interested in Facebook financials, as well the storage machines (NetApp low end 3070), sizing, traffic, servers, storage procurement and data center costs, take a look at TechCrunch's Michael Arrington post.

Keep Performing,
Moshe Kaplan. Performance Expert.