Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

This project is archived and is in readonly mode.

Fixtures data should be reset between tests

#3176

When I have two Tests:

class FirstTest < ActiveSupport::TestCase
  fixtures :foo

  test "some test case" do
    assert(Foo.count > 0)
  end
end

class SecondTest < ActiveSupport::TestCase
  # no fixtures loaded

  test "some other test" do
    assert(Foo.count == 0) # this can fail if FirstTest is run first
  end
end

When SecondTest is run after FirstTest, then the fixtures loaded in FirstTest are not reset after the test is run and they cause SecondTest to fail.

This gist demonstrates the problem in more detail:

http://gist.github.com/183881

The tests should be isolated against each other (especially when use_transactional_fixtures is set) and therefore the Fixture data should be reset after each test.

Reported by Levin Alexander · September 9th, 2009 @ 06:55 PM

State: wontfix
Milestone: 3.0.2
Assigned to: nobody
Importance: none

Activity

  1. Niels Meersschaert
    Niels Meersschaert
    • Tag changed from activecord, fixtures, testunit to 2.3.5, activecord, fixtures, testunit

    This appears to have become even worse under 2.3.5. I have self.use_transactional_fixtures = true & fixtures :all in my test_helper.rb file. With no other change other than 2.3.5 in environment.rb & running rake rails:update, tests which have passed before now fail. Switching back to 2.3.4, enables the tests to pass.

    November 30th, 2009 @ 05:51 PM

  2. Niels Meersschaert
    Niels Meersschaert

    After further investigation, it looks like 2.3.5 actually is working correctly, possibly solving this issue. It actually exposed bugs in my tests, causing the failures.

    November 30th, 2009 @ 06:58 PM

  3. Levin Alexander
    Levin Alexander
    • Tag cleared.

    I can't verify that the issue is fixed in 2.3.5.

    The test code in the gist still shows that Fixtures are not purged after tests are run.

    November 30th, 2009 @ 07:09 PM

  4. Levin Alexander
    Levin Alexander
    • Tag set to 2.3.x, activerecord, fixtures, testunit

    November 30th, 2009 @ 07:10 PM

  5. Tim Morton
    Tim Morton

    In my application, the transactional fixtures were working under 2.3.2, but broke on 2.3.3, 4, and 5.

    April 30th, 2010 @ 05:14 PM

  6. Ryan Bigg
    Ryan Bigg
    • Tag changed from 2.3.x, activerecord, fixtures, testunit to 2.3.x, activerecord, bugmash, fixtures, testunit
    • State changed from new to open

    I can confirm this issue still exists in 2.3.8.

    May 25th, 2010 @ 10:03 AM

  7. Neeraj Singh
    Neeraj Singh
    • Importance changed from to

    I can confirm that this problem exists in 2-3-stable (latest) and rails3 master(latest).

    June 27th, 2010 @ 04:45 AM

  8. Neeraj Singh
    Neeraj Singh

    LH can (and should) do better than that. Priority changed from “” to “” :-)

    I did not even change priority.

    June 27th, 2010 @ 04:48 AM

  9. Neeraj Singh
    Neeraj Singh
    • Tag changed from 2.3.x, activerecord, bugmash, fixtures, testunit to 2.3.x, activerecord, bugmash, fixtures, rails3, testunit

    June 27th, 2010 @ 04:48 AM

  10. Santiago Pastorino
    Santiago Pastorino
    • Milestone cleared.
    • Assigned user set to José Valim

    June 27th, 2010 @ 05:01 AM

  11. José Valim
    José Valim
    • Assigned user cleared.

    Can anyone work on a patch please?

    June 27th, 2010 @ 08:06 AM

  12. Neeraj Singh
    Neeraj Singh

    I spent some time looking at it and ,I think, rails is working as expected. However it might not be what user is expecting.

    Here is how test is run with self.use_transactional_fixtures = true .

    • test database has no records
    • fixture data is loaded into the database. transaction is committed.
    • before the test starts a transcation is started because of self.use_transactional_fixtures = true
    • tests are run
    • all the data created during test are stored in the database (note transaction is not yet committed)
    • tests finish
    • since self.use_transactional_fixtures = true is used transaction is rolled back.
    • note that data created during tests are rolled back. Not the data created by fixtures.
    • when the next time this fixture is used then fixture data is wiped out before fixture data is inserted.

    Note that when fixture data is loaded then first all the data is wiped out.

    In this case the second test created by Levin is not loading fixtures. It means the data created by previous fixtures are still there in database even though use is using self.use_transactional_fixtures = true .

    Fixing this issue would be mean that if self.use_transactional_fixtures = true is used then put fixture data also within transaction. This could have strong impact on the way a lot of tests are written.

    In order to run tests faster a lot of time users load all the fixture data at once in database and then the data created by tests are rollbacked. In this case one does not have to create data from fixtures time and again and this results in faster tests.

    If we bring loading of fixture data within transaction then tests will run much much slower because then fixture data would be loaded in every single test.

    June 27th, 2010 @ 06:42 PM

  13. José Valim
    José Valim
    • State changed from open to wontfix

    Neeraj, awesome explanation about the issue. I'm marking this as won't fix, according to Neeraj information.

    June 27th, 2010 @ 07:13 PM

  14. Levin Alexander
    Levin Alexander

    I am not convinced that this would slow down tests much. The fixtures need to be loaded in either case.
    I think (but don't know yet) that the slowdown should not be all that big. I'll try to investigate and get some numbers.

    José: would you accept a patch for this ticket if the slowdown is small?

    June 27th, 2010 @ 07:26 PM

  15. José Valim
    José Valim

    Avoid fixtures to be loaded is exactly the reason why we have transactional fixtures.

    http://clarkware.com/cgi/blosxom/2005/10/24

    That said, I cannot accept a patch that removes fixtures after running each test.

    My advice is: a solution that will always make the data be deleted, even if you don't want to, is going to make your tests slow. So my advice is: just do Foo.delete_all in the few tests you want to ensure it's empty.

    June 27th, 2010 @ 07:37 PM

  16. Levin Alexander
    Levin Alexander

    Got it, thanks.

    right now, if you're writing fixtures :table you are deleting fixtures also. The fix for this ticket would just need to shift this work from before each test is run to after. The total amount of work should remain the same.

    Could be that I'm wrong about that, though. I'll try to get some data.

    June 27th, 2010 @ 07:50 PM

  17. José Valim
    José Valim

    "If you're writing fixtures :table you are deleting fixtures also."

    This just happens once and just before running tests. Maybe is ok to delete the whole data after all tests executed, but it needs to be after all tests. And that is not going to solve your issue.

    June 27th, 2010 @ 07:54 PM

  18. Jeremy Kemper
    Jeremy Kemper

    Great analysis, Neeraj. This is precisely why fixtures are inserted outside the transaction.

    Since then, we added support for savepoints, though. We could begin a transaction for the whole test run, insert the fixtures, the mark a savepoint and rollback to it around each test case.

    June 27th, 2010 @ 08:12 PM

  19. Levin Alexander
    Levin Alexander

    The reason I think this ticket is a problem is that different test files can influence each other in unexpected ways. (running FooTest before BarTest will cause BarTest to fail)

    Only rolling back to a known state after each TestCase would be enough to fix that particular problem.

    June 27th, 2010 @ 08:17 PM

  20. Jeremy Kemper
    Jeremy Kemper
    • Milestone set to 3.0.2

    October 15th, 2010 @ 11:01 PM