Ruby Enterprise Edition 1.8.7-20090928 released

In the not so distant past we launched a Ruby Enterprise Edition sponsorship campaign. We’ve been working hard on the code since that time. And although the campaign target hasn’t been reached, we’ve decided to release it anyway because we believe that this release is of great importance to the community. Just how important is this release? Well, Twitter has beta tested this release and they’re reporting roughly 30% improvement in throughput. Keep reading to learn more and follow us on @phusion_nl for the latest updates. Summary of changes It’s now based on Ruby 1.8.7-p174 Previous REE releases were based on 1.8.6. We were initially reluctant about switching to 1.8.7: many people reported various incompatibility problems introduced by 1.8.7, and it seemed that alternative Ruby implementors were boycotting 1.8.7. However, the tide has now changed. Rails 3.0 will drop support for Ruby versions earlier than 1.8.7, JRuby had recently decided to support 1.8.7 and OS X Snow Leopard as well as all the Linux distros ship 1.8.7. We’ve received a lot of requests from the community for a 1.8.7-based version. In short, everybody is moving towards 1.8.7, and we can’t afford to stay behind. We were initially planning to release a final 1.8.6-based version, but after carefully considering the current state of things, we decided to skip the 1.8.6 release and jump directly to 1.8.7. 1.8.7 is also more compatible with 1.9, allowing a smoother transition to future Ruby versions. The MBARI patches have been integrated Brent Roman’s MBARI patch set uses a variety of techniques to significantly reduce Ruby’s memory usage. It also adds a bunch of performance optimizations, useful features and bug fixes. Ruby’s garbage collector scans the C stack conservatively. This means that any data on the C stack that might look like a pointer pointing to a valid Ruby object will be treated like a real pointer, causing the object to be retained during garbage collection. Because of the way Ruby is written and because of compiler optimizations, there may be a lot of such “pointers” on the stack, even when they’re not really pointers but just garbage data that like look pointers. Brent has refactored and modified many parts of Ruby in order to reduce the possibility of such “garbage pointers” occurring on the stack. As a result, garbage collection will be much more effective. Many objects that would otherwise not have been freed will be freed thanks to the MBARI patches. As a side-effect of the reduced memory usage, applications typically run faster as well. Not only because applications are less likely to be swapped out, but also because modern hardware’s performance depends greatly on the amount of memory traffic. Ruby uses an inefficient, linear time algorithm for extending the stack space when switching context between threads and/or continuations. This is mostly noticeable when you have a lot of threads. Brent replaces this with a constant time algorithm, which is much faster. It adds the methods __file__ and __line__ to the Method, UnboundMethod and Proc classes. Very useful for inspecting large code bases that make heavy use of metaprogramming. These methods are documented in the Ruby Enterprise Edition manual. We’ve ported his patch set to 1.8.7-p174. Especially Twitter benefited a lot from these patches. The zero-copy context switching patch have been integrated as an optional feature Joe Damato and Aman Gupta have created a patch which implements zero-copy context switching in Ruby’s threading system. In Ruby 1.8, whenever Ruby switches context to another thread, it has to copy over the entire thread stack. This is very expensive, and the deeper your function calls are nested, the slower it becomes. The zero-copy context switching patch removes the need to copy any stack data; instead, it keeps all thread stacks at different areas in memory and simply changes the stack pointer. In extreme situations, this patch can increase threading performance by as much as 10 times! Phusion Passenger is not really affected by this patch because its request handling core is not multithreaded, at least not at this moment. But Mongrel is heavily multithreaded and will benefit tremendously from this patch. We’ve made this patch compatible with the MBARI patch set, but there are currently some issues with 64-bit platforms, so we’ve disabled it by default. You can enable it by running the REE installer with --fast-threading Various other minor changes When compiling REE, binaries are optimized for size (-Os), not speed. See below for an explanation. The tcmalloc memory allocator has been upgraded to version 1.3 RubyGems upgraded to version 1.3.5. The Debian package now installs files to /usr/local, so that one doesn’t have to mess with $PATH to get things working. In practice This release has been beta tested with Twitter. Evan Weaver reported significant improvements in Twitter’s throughput after having switched to this REE release. Twitter especially benefited from the MBARI patch set. We’ve found a few unexpected things during the beta testing phase: We’ve found that optimizing Ruby for code size (compilation with -Os) yields better results that optimizing Ruby for speed (compiling with -O2). Ruby is a very unconventional program and we suspect that optimizing for code size allows more instructions to fit in the CPU instruction cache, resulting in better performance. Microbenchmarks and synthetic benchmarks such as the Ruby benchmark suite do not show any performance improvements at all, although Twitter was able to measure a significant performance improvement. As always, benchmark results should be taken with a grain of salt. Other major beta testers include 37signals and Novell. They’re still in the progress of testing, and we’re eager to receive their test results. Sponsors Special thanks go out to LVS for acting as a gold sponsor for this release: LVS has been building and supplying enterprise platforms for the betting and gaming industry in South East Asia, Europe and the UK for more than 10 years. The team at LVS is a balanced mix of gaming industry specialists, technical experts and entrepreneurial leaders. With offices in London and Gibraltar we directly employ over 40 staff. Our Advanced Betting Platform (a fully internationalized and localized high-availability platform based on Java and Ruby on Rails) is now regarded as the most technically advanced sports book system available in the World, and has been customised to fit the requirements of companies like Victor Chandler in Gibraltar and La Française des Jeux in France. We also thank the following people and organizations for sponsoring this release: Matt Gillam, 300.00 USD Robert Harvey, 250.00 USD ASPgems SL, 200.00 USD Jason Lee, 200.00 USD Sung Jin Won, 200.00 USD Amedeo Paglione, 100.00 USD Bret Berger, 100.00 USD Ernest Ellingson, 100.00 USD Gavin Stark, 100.00 USD Hosting Playground Inc., 100.00 USD LUCAS MELIA, 100.00 USD Tesly, LLC, 100.00 USD 3hv, 50.00 USD David Balatero, 50.00 USD Francesc Esplugas Marti, 50.00 USD Frans Benders, 50.00 USD Invisible Jens Christian Fischer, 50.00 USD Layton Wedgeworth, 50.00 USD LeeCommerce, LLC, 50.00 USD Mikhail Platov, 50.00 USD Revolution Co., 50.00 USD Trevor Smith, 50.00 USD Steven Berg, 30.00 USD Alexey Kovyrin, 25.00 USD Chris Gaffney, 25.00 USD Cindy Studebaker, 25.00 USD Norman Clarke, 25.00 USD ARP Networks, 20.00 USD Frank Wöckener, 20.00 USD Istvan Hoka, 20.00 USD James Cribbs, 20.00 USD Luke Randall, 20.00 USD Roger Pack, 20.00 USD Trevor Turk, 20.00 USD Eric DUMINIL, 15.00 USD @bryanl, 10.00 USD declanmcgrath, 10.00 USD David Dollar, 5.00 USD Joshua Lund, 5.00 USD Kieran P, 5.00 USD Download & upgrade To install Ruby Enterprise Edition, please visit the download page. To upgrade from a previous version, simply install into the same prefix that you installed to last time. Please also refer to the documentation for upgrade instructions. UPDATE: Snow Leopard support It looks like tcmalloc has some issues on Snow Leopard, causing the compiler to crash. We’ve just been able to obtain Snow Leopard last weekend (it had been sold out for several weeks) and we haven’t gotten around installing it yet. To install REE on Snow Leopard, you should run the installer with --no-tcmalloc