Showing posts with label Horizontal Sharding. Show all posts
Showing posts with label Horizontal Sharding. Show all posts

Nov 18, 2018

10 Things You Should Know about MongoDB Sharding


MongoDB Sharding is a great way to scale out your writes without changes to application code. However, it requires careful design to enjoy all the scale and performance benefits:
  1. You cannot rename a sharded collection. Think twice before any action!
  2. MongoDB collection cannot be resharded: To reshard a collection, you will have to copy the collection into a new sharded collection (or do it twice if you cannot rename the collection name in the application level). You will need a script like this:
    use app;
    db.c_to_shard.find().forEach(function(doc){
            db.c_to_shard_new.insert(doc);
            db.c_to_shard.remove({"_id":doc._id});
    });
  3. When you run this script, make sure you do it on the cluster (or on a server with minimal latency to the cluster) to accelerate copy.
  4. If collection is already large, you will need to split the script and run multiple processes to fasten the copy.
  5. Avoid creating a unique index on the sharded collection. Sharded collection cannot utilize unique index..
  6. Be careful to avoid selecting a field that does not exist in all the documents. Otherwise, you will find tons of "missing documents".
  7. Follow the previous rule when selecting range based sharding.
  8. Choose the sharding key wise. If you do not sure 100% your keys will be distributed evenly, use a hashed key to ensure it
    db.c_to_shard.ensureIndex({user_id: "hashed"})
    sh.shardCollection("app.c_to_shard", {"user_id": "hashed"})
  9. Verify your shards and chunks are evenly distributed using getShardDistribution(). Verify it once sharded is done, and as data starts to get into the collections. If something is wrong, the smaller the collection, the easier the fix.
    db.c_to_shard.getShardDistribution() or do the same with the following script that will provide you a detailed information on each shard and chunk.
  10. If all chunks reside on a single shard, verify the balancer status, and start if needed:
    sh.getBalancerState()
    sh.startBalancer()
Bottom Line
MongoDB sharding is a great way to scale out your data store write thoughput. However, you should select your sharding key wise, otherwise, it might require a significant effort to fix stuff.

Keep Performing,

Nov 16, 2010

Cloud Computing Design and Best Practices

Today I presented "Cloud Computing Design and Best Practices" in CloudCon, one of the largest cloud events in Israel.

It was a great lecture and I would like to share with you some of the lecture's insights:
  1. Assumptions: Don't refer cloud computing as a brave new world. It is, but when you design your cloud system, don’t forget to keep care of the basic rules you had before.
  2. Create A Road Map: You probably will be able to turn your system into a 1 Billion users system in day one. Therefore, design your road map and understand how you will reach your target. In order to get there, you should define the various system parts and you should understand how each of them will be scaled out/removed/replaced in the future to meet your road map.
  3. Start Fast: People love success. Your investors love it, Your Marketing guys Love it, Your Customers love it and even Your Development guys do. Therefore, start safe and fast. If your development is great in C#, start with it. If you have an existing software product, start with it as well. It is always better to start with a model based on technologies and products that you are good at, then getting into an adventure that you cannot control its risks.
  4. Minimize Costs: After you decided to take the fast track, you should control your costs, and most important: your growth costs. You should go over your business plan and turn it into technical requirements plan and find your bottlenecks. Based on these bottlenecks, define solution for each of them: if you are in the online ads market, take care of your impressions module; if you are in the video business, take care of your video processing module. Why? In the viral and online business the growth is exponential. Therefore, cost growth is exponential, and you have to take care of it before your budget will run away.
  5. Best Strategies:
    1. Scale out: think of sharing nothing, understand how to take each server and split it to infinite number of servers. If a larger server procurement is your best solution, you probably in the wrong direction.
    2. Sharding: Data is usually the largest obstacle for scaling out, as conservative designs concentrate the data in a single place. If you have a similar case, take the path the giants already taken: Shard your database either if it is MySQL or SQL Server (read my Best Seller Sharding Post) 
    3. In Memory Database: In Memory is X5-X10 faster than on disk. Therefore, analyze your system, and understand what you can do without going to disk. In case you may be able avoid neglected number of transactions, use this technique to cut your costs.
  6. Refactor on the Run: as a player in a growing business, you don't have the option to rest. A 100 users system is different than a 100M users system, and as the system grows, smaller modules that were neglected in the first phase will become more important in terms of bottleneck, cost or business sensitivity. Your way to handle it should be refactor the system, step after step to meet the business goals.
  7. Define the Exit Strategy: You should always remember that your cloud operator is still a vendor and your best partner in the early days may become an obstacle when you become a giant. Therefore, choose carefully your cloud provider tools. I would recommend you to think twice before you choose propriety data stores like SimpleDB, and if you do: Create your own interface and have an exit strategy when needed.
  8. Everybody is using Open Source. What if your organization is not an expert in this field? As written before, you should start fast. There are plenty of cloud providers that support Windows and .Net, and you can get a head start if you will use the technology you are familiar with. When you grow, you may refactor your product and add technologies to remove your bottlenecks such as Erlang for Push or LAMP to handle your most common processes that result in 90% of your costs.
  9. Looking for more strategies?
    1. CDN: extract your static and streaming content to CDN provider. This move will cut your server and network utilization and will improve your end user experience.
    2. Smart Clients: turn your end user client to be sensitive to network failures. If you are using Gmail and seen the loading... label instead of 404 page, you probably understand what I'm talking about (otherwise Google for JQuery).
    3. Elastic Growth: if your system pattern usage is not uniform, consider turning on and off some of your instances to keep costs down and meet the spikes.
    4. Replication: don't forget to keep your data safe and meet failures. Make sure you do it using commodity hardware and software.
    5. Prepare for downtime & upgrades: make sure that you can always go on. Downtimes will come, since in large scale everything happens, even what should have not. Make sure you never really shut down you whole system even when you upgrade it.
    6. NoSQL and SQL: choose SQL as a start if you are great at it. But don't neglect NoSQL when you will get larger.
  10. Risk Management: You are going to do a bold move ahead, and you should prepare yourself as I mentioned before:
    1. Choose your vendors carefully and know your exit strategy. Your Cloud Operator is a provider as well.
    2. Hedge costs and take care of your bottlenecks.
    3. Stress your system all the way to guarantee you can get to the next level.
    4. Think one move ahead and keep aligned to your strategy.
    5. Listen to your users feedback. Your business depends on your clients; take care of them.
Bottom Line
Working according to these rules can help you reach the 1 Bill users system you were looking for


Keep Performing,
Moshe Kaplan
Follow MosheKaplan on Twitter

Sep 16, 2010

Sharding Again

Click on the post title to read the full post and the comments.


It was a long time since we last discussed Sharding. 
Yesterday the Twitter sharding case study was presented at the Metacafe Knowledge sharing meetup by Gidi Meir Morris. Therefore, I think it is a good reason to refresh our minds.


Hey! Where Are All These Whales?
Twitter turned its web common architecture (that caused it many problems) based on Ruby on Rails web layer and MySQL into a 3 layer system that includes:

  1. Web application server called Flapps.
  2. Sharding middleware called Gizzard. This layer takes care of database requests and sends them to the target shard. It takes care of replication as well.
  3. Old good (not so scalable) MySQL in a horizontal sharding mode.

But hey, it is better taking a look in this great Prezi presentation (use the More>Full Screen to better watch the presentation):





Bottom Line
Highly scalable databases are becoming commodity. In the Era of Cassandra and Gizzard, Google is starting to lose its competitive edge and high priced Oracle, IBM and Microsoft database are no longer a must for a startup. Will it affect these companies' bottom lines? Only time will tell.


Keep Performing,
Moshe KaplanFollow MosheKaplan on Twitter

Dec 21, 2009

Lessons from Facebook:

FB maybe is not the best example for quality design (probably each of us is likely to have at least a single error message a day).

However, since it's the 2nd largest site in the world with contsant exponential growth, it's a good lesson to have. Most interesting that their tech is team is only about 250 people and 30K servers, pretty amazing.


Not long ago a Jeff Rothschild, the Vice President of Technology at Facebook gave a presentation in UC San Diego. You can find a detailed summary by Prof. Amin Vahdat, but I'll make several top comments that I found useful.
  1. FB Software development: mostly PHP (complied), but other languages are used as well.
  2. Common interface between services using an internal development that was turned to open source: Thirft. This interface enables easy connections between the different languages.
  3. Logging Framework that is not dependent on a central repository or its availability. FB is using Hadoop and well as Hive (that was developed there as well). The log size is growing at a 25TB a day.
  4. Operational Monitoring: Separated from the logging mechanism 
  5. LAN can be a bottleneck as well: expect for packet loss and packet drops in the LAN if you stress it too much.
  6. CDN: Facebook is using external CDN to images distribution.
  7. Dedicated file system named Haystack that combines simple storage along with cache directory: file system is accessed only once to get images from it, while directory structure is retrieved from the cache.
  8. Most data is served from Memcached. Database is used mostly for Persistency and data replication between sites (Memcached is being heated by the MySQL itself):
    1. Top challenge: keeping data consistent since Memcached can be messed easily (No search for keys is available).
    2. Mixing information including sizes and types is better - making sure that load on CPU, Memory and etc is distributed equally.
  9. Shared nothing - Keep your system independent - avoid a single bottleneck. Therefore, data is saved in a sharded MySQL from day 1. However, MySQL is used mostly for data persistency and not for conservative database usage pattern:
    1. No Joins in the MySQL
    2. Chosen due to good data consistency + Management Software
    3. 4K servers
    4. Data replication between sites is based on MySQL replication
    5. Memcached is being heated based on the MySQL Replication using a custom API
One last thing, if you are interested in Facebook financials, as well the storage machines (NetApp low end 3070), sizing, traffic, servers, storage procurement and data center costs, take a look at TechCrunch's Michael Arrington post.

Keep Performing,
Moshe Kaplan. Performance Expert.

Jul 3, 2009

Alphageeks Event and Video

Hi,

A two weeks ago, just a few days before the Java Technology Day, I had a great presentation in a very lovely new geeks meetup group named "AlphaGeeks". I presented there the key issues in developing high load systems based on the Internet giants architectures including Sharding, In memory databases and cloud computing. I presented there key concepts that helped us design and implement a 1 billion events per day ad network system to one of our clients.
This presentation was recorded and is available in blog.tv and also here. Please notice that the lecture is in Hebrew.

Keep Performing,
Moshe Kaplan. RockeTier. The Performance Experts.







May 13, 2009

Cloud Slam 09': The Pareto Illusion

A few weeks ago we presented in Cloud Slam 09' cloud services cost analysis, and how it can be minimized using performance boosting. We also discussed in this presentation databases in the cloud and Sharding based on participants questions.
Feel free to take a look at the recorded presentation as well as the abstract, which are available at Cloud Slam site as well.

The Pareto Illusion - Why we end up paying too much for cloud services and what can we do about it? Moshe Kaplan and Ayal Baron

Video of the session:



Presentation Abstract:
Cloud computing is a whole new game: we are no longer talking about equity and CapEx, but regarding OpEx.

So why are you still paying so much every month for your cloud services? How can it be reduced? What are the industry state of art methodologies to gain more out of your cloud service provider? How to do more with same, and how to gain a better ROI for your project? and last but not least, how to help saving our world.

In this presentation we'll discuss how can you make your software more efficient, and how to focus on core components of the system and achieving 80% boost in 20% effort.

Keep Performing,
Moshe Kaplan. RockeTier. The Cloud and Performance Experts

Feb 23, 2009

SQL Server Partitioning: The bad, the good and the evil

Description
Horizontal Sharding is used to separate rows between several tables based on applicative logic. Microsoft introduced in MS SQL Server 2005 a build in mechanism named Partitioning to support this need without extra code in the business logic itself. This mechanism enables you deciding in which filegroup each row will be placed supporting regular queries to retrieve and update the data, while boosting performance.
Some syntax and code examples are available if you want to master this feature.

What Can be done with this great feature?
  1. You can break large tables into smaller chunks which fits the application logic (e.g partioning client's data according to its id or data according to date).
  2. You can put heavily accessed parts of the table onto fast storage, and less-accessed data onto slower, cheaper storage.
  3. You can boost the backup time in static date partitioning.
  4. You can boost many queries including DELETE, SELECT, UPDATE and so on based on right design.

Pros (Why should I use SQL Server Partitioning rather than Horizontal Sharding)
  1. Horizontal Sharding out of the box.
  2. A lot of thought and effort were invested in this feature to make it working in just few lines of code
Cons
  1. Relatively new (well not so new since it was presented in SQL Server 2005), and many issues were fixed in SQL Server 2008.
  2. Requires Enterprise Edition (10X licensing cost relative to standard edition or in other words: $25K per CPU)
  3. Relatively complex (no support in enterprise manager, and few DBAs will be able to support it) so you probably should master this white paper before taking it into production
  4. Will bound you to SQL Server (Enterprise Edition).
Industry Opinions:
Brent Ozar: "outside of data warehouses, I like to think of partitioning as the nuclear bomb option. When things are going out of control way faster than you can handle with any other strategy, then partitioning works really well. However, it’s expensive to implement (Enterprise Edition plus a SAN) and you don’t want to see it in the hands of people you don’t trust."

Bottom line:
Ask your DBA if they feel safe with this feature, and with your clients DBA. If the answer is no, consider choosing another solution.

I hope now you have all the information to make the decision by yourself. Otherwise, post your comments and we'll be glad to help you,

Keep Performing,
Moshe Kaplan. RockeTier. The Performance Experts.

ShareThis

Intense Debate Comments

Ratings and Recommendations