Showing posts with label Memory Leaks. Show all posts
Showing posts with label Memory Leaks. Show all posts

Sep 6, 2016

Ruby is Dead! (You nead to take care of its memory issues)

One of the common problems with Ruby is its memory usage, just like JVM based languages (yes Java, I'm talking about you). Actually Ruby just like Java is based on garbage collector. This GC if not used correctly, can cause "stop the world scenarios" (as we can see in the attached diagram). In this cases, GC actually stops responding while consuming servers whole CPU and in some cases even causes server reboot.
Unicorn and Ruby memory leak causes server downtime
What can be done?

1. Reboot your Ruby Periodically

If you are using the popular Unicorn web server, memory issues may be even more severe, as Unicorn forks. During forking, it's copying the whole memory footprint of the parent (Copy on Write or CoW). Therefore, you may use the "Unicorn Worker Killer" gem that will monitor your server and gracefully will restart specific worker when memory reaches a new high or specific number of requrests were served. Since the gem supports randomization, it assures in a very high probablilty that the server will continue server.

Installing the killer:
gem 'unicorn-worker-killer'

Add to config.ru above "require ::File.expand_path('../config/environment', __FILE__)"
# Unicorn self-process killer
require 'unicorn/worker_killer'

And decide your worker graceful reboot method: 
# Max requests per worker
use Unicorn::WorkerKiller::MaxRequests, 3072, 4096
# Max memory size (RSS) per worker
use Unicorn::WorkerKiller::Oom, (192*(1024**2)), (256*(1024**2))
If you are using Ruby 2.X, you can exploit the CoW by better configuring Unicorn.
config/unicorn.rb (and a detailed explenation on setting at sirupsen)
 - worker_processes: 1x your cores
 - timeout: worker request timeout, should be 15 to 30 sec max
 - preload_app: enables CoW, but requires managning connect/disconnect on fork


3. Take care of GC configuration
You can take a look at a detailed discussion at collective idea's post.

Bottom Line
Dynamic languages has their downs, yet w/ the right design you can keep them up and running,

Keep Performing,

Jun 19, 2010

At My Command Unleash Version!

Click on the post title to read the full post and the comments.
 
As you may know I'm a big fun of Continuous Integration, a.k.a CI, (running every trunk commit with some unit tests and integration tests, finding that you ruined the build and earning the blame T-shirt is such a good way to start the morning :-).

Cool things you can do with CI
However, today's CI is much more from just running several tests. These days you can do almost everything with it:
  1. Get your latest version from trunk.
  2. Build the version.
  3. Launch a new virtual machine based on a predefined image or by installing it almost from scratch.
  4. Install the version on the new machine.
  5. Run smoke tests to provide an instant validation of the code stability.
  6. Run nightly builds to carefully check that the version is clean.
  7. Run stress tests to validate capacity and long multiple days tests to validate memory leaks.
  8. Notify key players with the results.
Cloud Related Cool Scenarios
Cloud computing is basically about obtaining on demand resources through API (the GUI is a fancy stuff that can be useful but is not really a must). Therefore, you can use CI to automate every step in your testing, upload to production and rollback when needed. Some of the scenarios are:
  1. Performing Software Upgrade: Launching a new instance, installing the latest software version and creating an Image (AMI in AWS).
  2. Launching an entire new integration environment that is identical to the production environment and check its functionality as well as its correspondence with the non functional requirements.
  3. Launching a new environment for marketing needs.
  4. Managing your new version upload, by creating a script that will prepare new servers, upgrade your database servers (or RDS), add new application and web servers to the load balancer farms, and finally will remove the old server and shut them down.
  5. Implement Failover scenarios (DRP) 

Industry Best Practices
According to Amazon (see Jinesh Varia, Technology Evangelist @Amazon Web Services document), CI and automation is the best practice to manage your cloud environment:
  1. Create a management server (like CruiseControl or regular CRON jobs).
  2. Create a build-test-deploy cycle script using the chosen management server that will generate:
    1. New AMI
    2. Deploy new servers based on these AMI
    3. Add these servers to existing load balancer
    4. Attach elastic IPs
    5. Generate static files to S3 and CloudFront
    6. Generate new queues and notifications using SQS and SNS
    7. Remove old servers and unneeded AMIs
Using this method can save you a lot of time in configuration and optimization, as well as many production issues due to human errors.

What CI to choose?
There are many CI products in the market, some of them are open source and other are commercial. Here are some of leading tools, thanks to Alon Nativ advice:
  1. CruiseControl: a leading open source product with dozens of plugins and 3rd parties.
  2. Apache Continuum: an open source product with many features.
  3. Final Builder: a commercial and major tool in the market with great visual interface and many built in scripts and API
  4. JetBrains TeamCity: a commercial tool with a built in integration to Amazon AWS.
You can also take a look at RightScale product and services. RightScale provides semi CI in a SaaS manner with many features we discussed before and more. I included here a RightScale video, which is one of the best industry resources regarding this issue:


Bottom Line
Now after going over the details, I recommend you to:
  1. Design your cloud environment.
  2. Design your baseline AMIs/Images.
  3. Design your upgrade, rollback and test-build-deploy scripts based on these basic AMIs.
  4. Implement them using your chosen Continuous Integration tool.
  5. Roll it out to production.
Keep Performing,
Moshe Kaplan

    ShareThis

    Intense Debate Comments

    Ratings and Recommendations