One of the common problems with Ruby is its memory usage, just like JVM based languages (yes Java, I'm talking about you). Actually Ruby just like Java is based on garbage collector. This GC if not used correctly, can cause "stop the world scenarios" (as we can see in the attached diagram). In this cases, GC actually stops responding while consuming servers whole CPU and in some cases even causes server reboot.
Unicorn and Ruby memory leak causes server downtime
What can be done? 1. Reboot your Ruby Periodically
If you are using the popular Unicorn web server, memory issues may be even more severe, as Unicorn forks. During forking, it's copying the whole memory footprint of the parent (Copy on Write or CoW). Therefore, you may use the "Unicorn Worker Killer" gem that will monitor your server and gracefully will restart specific worker when memory reaches a new high or specific number of requrests were served. Since the gem supports randomization, it assures in a very high probablilty that the server will continue server.
Installing the killer:
gem 'unicorn-worker-killer'
Add to config.ru above "require ::File.expand_path('../config/environment', __FILE__)"
# Max requests per worker use Unicorn::WorkerKiller::MaxRequests, 3072, 4096 # Max memory size (RSS) per worker use Unicorn::WorkerKiller::Oom, (192*(1024**2)), (256*(1024**2))
Click on the post title to read the full post and the comments.
As you may know I'm a big fun of Continuous Integration, a.k.a CI, (running every trunk commit with some unit tests and integration tests, finding that you ruined the build and earning the blame T-shirt is such a good way to start the morning :-).
Cool things you can do with CI
However, today's CI is much more from just running several tests. These days you can do almost everything with it:
Get your latest version from trunk.
Build the version.
Launch a new virtual machine based on a predefined image or by installing it almost from scratch.
Install the version on the new machine.
Run smoke tests to provide an instant validation of the code stability.
Run nightly builds to carefully check that the version is clean.
Run stress tests to validate capacity and long multiple days tests to validate memory leaks.
Notify key players with the results.
Cloud Related Cool Scenarios
Cloud computing is basically about obtaining on demand resources through API (the GUI is a fancy stuff that can be useful but is not really a must). Therefore, you can use CI to automate every step in your testing, upload to production and rollback when needed. Some of the scenarios are:
Performing Software Upgrade: Launching a new instance, installing the latest software version and creating an Image (AMI in AWS).
Launching an entire new integration environment that is identical to the production environment and check its functionality as well as its correspondence with the non functional requirements.
Launching a new environment for marketing needs.
Managing your new version upload, by creating a script that will prepare new servers, upgrade your database servers (or RDS), add new application and web servers to the load balancer farms, and finally will remove the old server and shut them down.
Generate new queues and notifications using SQS and SNS
Remove old servers and unneeded AMIs
Using this method can save you a lot of time in configuration and optimization, as well as many production issues due to human errors.
What CI to choose?
There are many CI products in the market, some of them are open source and other are commercial. Here are some of leading tools, thanks to Alon Nativ advice:
You can also take a look at RightScale product and services. RightScale provides semi CI in a SaaS manner with many features we discussed before and more. I included here a RightScale video, which is one of the best industry resources regarding this issue:
Bottom Line
Now after going over the details, I recommend you to:
Design your cloud environment.
Design your baseline AMIs/Images.
Design your upgrade, rollback and test-build-deploy scripts based on these basic AMIs.
Implement them using your chosen Continuous Integration tool.