Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

This project is archived and is in readonly mode.

ActiveRecord#calculate broken for multiple fields in :group option

#497

The docs to ActiveRecord#calculate state:

:group - An attribute name by which the result should

be grouped. Uses the GROUP BY SQL-clause.

Most likely to be compatible to finders. But this statement is only partially true. The method's algorithm assumes that :group contains only one field. When specifying multiple values -- e.g. :group => 'foo, bar' -- this does not work anymore.

If one were to use ActiveRecord#count to get only the size of a result set (as the will_paginate-plugin does for example) this used to work in Rails 2.0 and was broken by http://preview.tinyurl.com/3l77f7 [Rails-Changeset]. Now the final result array with grouped key => aggregated value pairs is totally broken because keys are overwritten:

>> Item.count(:group => 'item_id').size

=> 687

>> Item.count(:group => 'quality').size

=> 2

>> Item.count(:group => 'item_id, quality').size

=> 2 # item_id is primary key, should return 687

>> Item.count(:group => 'quality')

=> [[5, 2], [4, 685]]

>> Item.count(:group => 'item_id, quality')

=> [[4, 1], [5, 1]]

I do not know how this should or can be fixed. Ideally every field in :group should be taken into account and used as some sort of combined key. The major problem I see here is splitting the :group string into fields because a simple split(/\s*,\s*/) won't work for more advanced things like 'COALESCE(col1, col2)'.

Another solution would be to allow :group being an array, thus putting the splitting in the hands of the caller:

Model.calculate(:op, :group => ['c', 'COALESCE(c1, c2)'])

This should be changed for finders as well then.

Otherwise the docs should be changed to reflect that multiple fields are not possible.

Reported by gix · June 27th, 2008 @ 03:45 AM

State: committed
Milestone: 3.0.5
Assigned to: Aaron Patterson Aaron Patterson
Importance: Low

Activity

  1. josh
    josh
    • State changed from new to stale

    September 30th, 2008 @ 05:53 PM

  2. Sebastian
    Sebastian

    +1 for fixing this issue.

    February 6th, 2009 @ 01:30 AM

  3. Seth Ladd
    Seth Ladd

    +1 for addressing this issue. or being much more explicit in the docs. I would want multiple group by support, achieved via an array as the original post suggests.

    March 27th, 2009 @ 10:22 PM

  4. Alex
    Alex

    +1 for having a way to group by multiple fields.

    May 6th, 2009 @ 10:39 PM

  5. jdwyah
    jdwyah

    +1 In the meantime, see my ugly hack: poor_mans_group_by.rb http://pastie.org/471055 coalesce deals with nulls and delimitters are chosen to be unicode values unlikely to be used in your code.

    May 7th, 2009 @ 01:13 PM

  6. Sebastian
    Sebastian

    Couldn't we use the returned row-hash (minus count_all) as the key?

    May 14th, 2009 @ 10:40 PM

  7. Matt White
    Matt White

    +1 for getting this fixed...

    June 22nd, 2009 @ 02:04 AM

  8. Scott Brown
  9. Jacob Kjeldahl
    Jacob Kjeldahl
    • Tag set to activerecord, calculations, group, patch

    I have found a patch by Eric Lindvall here http://tinyurl.com/luas8x it is an attachment to another lighthouse ticket which I am unable to find.

    The patch works against ActiveRecord v2.3.2 (For v2.1.0 check this gist http://gist.github.com/142343)

    I have corrected a tiny error in it and attached it to this ticket.

    From the test:
    @@@ Ruby def test_should_group_by_multiple_fields
    c = Account.count(1, :group => [:credit_limit, :firm_id]) assert_equal [[50, nil], [50, 1], [50, 6], [53, 9], [55, 6], [60, 2]], c.keys end

    
    

    July 8th, 2009 @ 03:26 PM

  10. Gavin Stark
    Gavin Stark

    Another +1. This patch would be very useful.

    December 1st, 2009 @ 02:51 AM

  11. Chris Taggart
  12. James Brooks
  13. Paul Holzberger
    Paul Holzberger

    +1

    for those looking for temporary support of this, there is a plugin i tracked down from some other references in this thread: http://github.com/patientslikeme/multi_field_group_by

    February 19th, 2010 @ 04:35 PM

  14. Rizwan Reza
    Rizwan Reza
    • No changes were found…

    May 19th, 2010 @ 10:33 AM

  15. Alex
    Alex
    • Importance changed from to Low

    Just created a patch that resolves this issue https://rails.lighthouseapp.com/projects/8994-ruby-on-rails/tickets.... It also works with COALESCE and other SQL functions. Please help verify the patch (see link above) if you guys are still interested in a solution.

    July 22nd, 2010 @ 11:42 PM

  16. Alex
    Alex

    Uploading a patch here in case any of you guys want to verify it.

    August 23rd, 2010 @ 04:39 PM

  17. Sebastian
    Sebastian

    I verified your patch and made it work against the 3-0-stable branch. I also added one more test case for using sql-functions in the group clause.

    September 11th, 2010 @ 03:24 AM

  18. Rafael Cardoso
  19. Santiago Pastorino
    Santiago Pastorino
    • State changed from stale to open
    • Milestone set to 3.0.2
    • Assigned user set to Aaron Patterson

    November 12th, 2010 @ 07:57 PM

  20. Alex
    Alex

    I updated the patch to work against master branch.

    November 13th, 2010 @ 10:52 PM

  21. Santiago Pastorino
  22. Aaron Patterson
    Aaron Patterson

    Hi everyone! I'd really like to push rails to do less SQL parsing. Parsing SQL is a dangerous game.

    If we could support this via a list passed to the :group clause (as @gix suggests), would that be OK with everyone?

    November 16th, 2010 @ 12:07 AM

  23. Alex
    Alex

    I agree that is better. The only reason it was done that way was because the documentation when this issue was first opened hinted that the correct usage was to pass in a SQL frag. I updated the patches for master and 2-3-stable such that there is no SQL parsing. They are attached.

    November 16th, 2010 @ 03:18 AM

  24. Alex
  25. Aaron Patterson
    Aaron Patterson

    @Alex excellent, thank you. I'll apply and push after verifying everything. Thanks for the patches!

    November 16th, 2010 @ 06:34 PM

  26. Aaron Patterson
    Aaron Patterson
    • State changed from open to committed

    Applied and pushed. Thanks!

    November 16th, 2010 @ 07:10 PM

  27. Alex
    Alex

    Great news! Thank you.

    November 16th, 2010 @ 07:55 PM

  28. Ludovic Gasc
    Ludovic Gasc

    Are you sure this bug is fixed ?

    I've tested with AR 3.0.3+your patch and AR 3.0.4.rc built with the Git repository, the SQL query is correct but not the response of the method.

    code:

    pp SavedForm.count(:group => 'agent_id, code_dispo')
    

    result:

    DEBUG -- :   SQL (1.4ms)  SELECT COUNT(*) AS count_all, agent_id, code_dispo AS agent_id_code_dispo FROM saved_forms GROUP BY agent_id, code_dispo
    {"bte_vocale"=>3, "CB"=>1, "nc"=>1, "pasok"=>4, "wln"=>2, "ok"=>3}
    
    # gem list
    
    *** LOCAL GEMS ***
    
    activemodel (3.0.4.rc)
    activerecord (3.0.4.rc)
    activesupport (3.0.4.rc)
    
    I can't test with the master branch, I've this error: /usr/local/lib/ruby/gems/1.9.1/gems/activerecord-3.1.0.beta/lib/active_record/relation/calculations.rb:169:in `perform_calculation': undefined method `grep' for # (NoMethodError) from /usr/local/lib/ruby/gems/1.9.1/gems/activerecord-3.1.0.beta/lib/active_record/relation/calculations.rb:152:in `calculate' from /usr/local/lib/ruby/gems/1.9.1/gems/activerecord-3.1.0.beta/lib/active_record/relation/calculations.rb:147:in `calculate' from /usr/local/lib/ruby/gems/1.9.1/gems/activerecord-3.1.0.beta/lib/active_record/relation/calculations.rb:58:in `count' from /usr/local/lib/ruby/gems/1.9.1/gems/activerecord-3.1.0.beta/lib/active_record/base.rb:430:in `count' Thanks for your feedback.

    January 4th, 2011 @ 11:06 AM

  29. Ludovic Gasc
    Ludovic Gasc

    Alex, I've just tested with your patch of November 13th, 2010 @ 10:52 PM, it runs correctly.

    Only your latest patch has a problem.

    If I can help, don't hesitate to contact me.

    Yours.

    January 4th, 2011 @ 11:17 AM

  30. Alex
    Alex

    The usage should be pp SavedForm.count(:group => [ 'agent_id', 'code_dispo']). The reason we do this is to avoid SQL parsing. This is not documented anywhere yet, so eventually we should be adding this to the rails doc.

    January 4th, 2011 @ 05:45 PM

  31. Santiago Pastorino
  32. Ben Woosley
    Ben Woosley

    I can confirm this isn't fixed. And +1 for getting it fixed.

    April 2nd, 2011 @ 09:18 PM

  33. Ben Woosley
    Ben Woosley

    Also, I'll be taking another look at it this weekend.

    April 2nd, 2011 @ 09:20 PM

  34. Ludovic Gasc
    Ludovic Gasc

    I confirm this bug isn't fixed in 3.0.7.

    I don't understand why the patch isn't applied on the source code since 6 months, what's the problem ?

    May 2nd, 2011 @ 03:36 PM