I have a Rails(3.2.8)/Mongoid(3.0.6) app which uses Resque workers to generate and send emails each morning. Each Resque job has to query a few collections to generate the email and then perform the actual send.
We have compound indexes set up to support the queries and executing a single job manually is very fast. However, as we increase the number of concurrent workers the time that it takes to generate emails blows up quickly. Tailing the log shows that the actual execution of the mongo queries increases dramatically.
(note: I noticed this while evaluating Sidekiq, but the same happens with Resque; 3-4 Processors/Workers move along fine but throughput drops dramatically past that)
We have 4 Thin processes on one box, Resque on another, and the MongoDB server on a third (all hosted within the same datacenter on Rackspace). The difference is noticeable even when moving from 4 to 8 workers. The Thins are also affected.
Is there any low hanging fruit or obvious bits that I may be missing that could explain the performance issues I'm seeing? Any pointers or advice would be appreciated.