Google’s MapReduce is now available for Ruby (via gem install starfish ). MapReduce is the technique used by Google to do monstrous distributed programming over 30 terabyte files.

Here is the basic code that will get you up and running with MapReduce in Starfish .

    # item.rb
    ActiveRecord::Base.establish_connection(
      :adapter  => "mysql",
      :host     => "localhost",
      :username => "root",
      :password => "",
      :database => "some_database"
    )

    class Item < ActiveRecord::Base; end

    server do |map_reduce|
      map_reduce.type = Item
    end

    client do |item|
      logger.info item.id
    end

Now just run:

    starfish item.rb

and Starfish takes care of the rest. The code above does the following:

• The server grabs all the items via: Item.find(:all)
• Each of the clients grab an item from the collection
• When there are no more items to be grabbed, everything shuts down

Just add REST (and it’s come by default with the Edge Rails) and you’ll have your own S3 or GDrive for free

mongrel_cluster makes it easy to manage multiple Mongrel processes behind a reverse-proxy server and load balancer such as Pound, Balance, Lighttpd, or Apache.

See also:

• Apache: Scaling Rails with Apache 2.2, mod_proxy_balancer and Mongrel .
• Lighttpd: Using Lighttpd With Mongrel
• Litespeed: setting up litespeed with mongrel

Must read articles for everybody, doing RoR systems administration:

• The adventures of scaling, Stage 1
• Questions and answers for Stage 1
• The adventures of scaling, Stage 2
• The adventures of scaling, Stage 3

03-Apr-2006 Update:

• The adventures of scaling, Stage 4