Rails Ruby on Rails

Using find_in_batches on a table of ~1MM rows generates queries that take >3s for a very simple query because the order clause has to consider the whole table greater than the start row.

http://github.com/chriseppstein/rails/commit/0a726f1b1447ddf105a3e5...

is a patch that changes the implementation of find_in_batches such that it selects a section of ids and only sorts that section.

With this patch, my queries are about 3ms on my local system - a 1000x speed improvement - but they are a constant-time query with the number of records instead of growing slower over time.

To do this, I made the assumption that the primary key is an integer -- I'm not sure if that's a valid assumption, but it's certainly the most common scenario.

This patch also adds support for :limit, because there's really no good reason to not support it in code.

Lastly, this patch breaks the behavior that only the last batch will have less than the batch size number of records. However, it does make sure that every batch has at least one record. This doesn't seem like a terribly important aspect of the api, but it's worth noting.