A library for making HTTP requests in parallel. evdispatch is Alpha software. I recommend using the curb gem and applying this patch if you need concurrency in your ruby application. For details check out this article.
tar -zxf curl-7.18.1.tar.gz cd curl-7.18.1/ ./configure && make && sudo make install sudo gem install evdispatch
The library provides an interface to a background posix thread running a libev event loop. HTTP requests are processed using libcurl and it’s multi interface. From Ruby requests and responses are passed back and forth to the event loop thread using a synchronized queue. In the future, responses may be processed in ruby by monitoring a file handle, making it possible to use IO.select.
The intended use of this library was to provide a method for web applications to make multiple concurrent service requests to satisify a single web request.
For example in rails this might look like:
class DashController < ApplicationController def index @blogs_id = $dispatcher.request("http://10.0.6.45/service/blogs") @news_id = $dispatcher.request("http://10.0.6.45/service/news") @messages_id = $dispatcher.request("http://10.0.6.45/service/messages") end end index.html.erb <% blogs = JSON.parse($dispatcher.response(@blogs_id)[:body]) %> <ul> <% for b in blogs %> <li><a href="<%= b['link'] %>" ><%= b['title'] %></a></li> <% end %> </ul> <% news = JSON.parse($dispatcher.response(@news_id)[:body]) %> <ul> <% for n in news %> <li><a href="<%= n['link'] %>" ><%= n['title'] %></a></li> <% end %> </ul>
What’s interesting is if the blogs response is not back and it blocks on $dispatcher.response(@blogs_id) chances are high that the news_id will have returned it’s response by the time it gets finished with the blogs. This is because the background thread does not block while ruby waits for the first response. One thing to keep in mind is the background thread will block if it has to resolve DNS names.
In my testing I’ve successfully run 3 million requests. Doing 10000 iterations of 300 concurrent requests, on a Linux FC7 duel P4 machine. Memory utilization stays flat at around 14 to 19 megs with about 2,000 – 12,000 objects.
Demonstration of usage
require 'rubygems' require 'evdispatch' # create a new dispatch loop d = Evdispatch::Loop.new # start the event loop thread d.start # send a dispatch http request and store a handle to the request google_id = d.request_http("http://www.google.com/") # do some processing and later on check for the response response = d.response( google_id ) puts response[:response_time] puts response[:body] # sometime later you can stop the event loop d.stop
How to submit patches
The trunk repository is http://evdispatch.rubyforge.org/svn/trunk for anonymous access.
I get an error when installing the gem: ev_http.cc:411: error: ‘CURLOPT_COPYPOSTFIELDS’ was not declared in this scope
The minimum version of libcurl required is 7.17.1. see the reference note here
I get an error when installing the gem: “error: ‘curl_socket_t’ has not been declared”
You need to have at least version 7.17.1 of libcurl. I recommend at least version: 7.18.1.
Mac OSX (Darwin) Crashing on HTTP POST request
There are some issues with crashes on darwin builds when making an HTTP POST. This doesn’t happen on Linux and I’m actively looking into the issue on my apple.
This code is free to use under the terms of the MIT license.
Comments are welcome. See the forum