#3176 Fixtures data should be reset between tests - Ruby on Rails

Type	To find
responsible:me	tickets assigned to you
tagged:"@high"	tickets tagged @high
milestone:next	tickets in the upcoming milestone
state:invalid	tickets with the state invalid
created:"last week"	tickets created last week
sort:number, importance, updated	tickets sorted by #, importance or updated
Combine keywords for powerful searching.
Use advanced searching »

This project is archived and is in readonly mode.

#3176 ✓wontfix

Fixtures data should be reset between tests

Reported by Levin Alexander | September 9th, 2009 @ 06:55 PM | in 3.0.2

When I have two Tests:

class FirstTest < ActiveSupport::TestCase
  fixtures :foo

  test "some test case" do
    assert(Foo.count > 0)
  end
end

class SecondTest < ActiveSupport::TestCase
  # no fixtures loaded

  test "some other test" do
    assert(Foo.count == 0) # this can fail if FirstTest is run first
  end
end

When SecondTest is run after FirstTest, then the fixtures loaded in FirstTest are not reset after the test is run and they cause SecondTest to fail.

This gist demonstrates the problem in more detail:

http://gist.github.com/183881

The tests should be isolated against each other (especially when use_transactional_fixtures is set) and therefore the Fixture data should be reset after each test.

Comments and changes to this ticket

You flagged this item as spam.
Niels Meersschaert November 30th, 2009 @ 05:51 PM
- Tag changed from “activecord, fixtures, testunit” to “2.3.5, activecord, fixtures, testunit”
This appears to have become even worse under 2.3.5. I have self.use_transactional_fixtures = true & fixtures :all in my test_helper.rb file. With no other change other than 2.3.5 in environment.rb & running rake rails:update, tests which have passed before now fail. Switching back to 2.3.4, enables the tests to pass.
You flagged this item as spam.
Niels Meersschaert November 30th, 2009 @ 06:58 PM
After further investigation, it looks like 2.3.5 actually is working correctly, possibly solving this issue. It actually exposed bugs in my tests, causing the failures.
You flagged this item as spam.
Levin Alexander November 30th, 2009 @ 07:09 PM
- Tag cleared.
I can't verify that the issue is fixed in 2.3.5.

The test code in the gist still shows that Fixtures are not purged after tests are run.
You flagged this item as spam.
Levin Alexander November 30th, 2009 @ 07:10 PM
- Tag set to “2.3.x, activerecord, fixtures, testunit”
You flagged this item as spam.
Tim Morton April 30th, 2010 @ 05:14 PM
In my application, the transactional fixtures were working under 2.3.2, but broke on 2.3.3, 4, and 5.
Ryan Bigg May 25th, 2010 @ 10:03 AM
- Tag changed from “2.3.x, activerecord, fixtures, testunit” to “2.3.x, activerecord, bugmash, fixtures, testunit”
- State changed from “new” to “open”
I can confirm this issue still exists in 2.3.8.
Neeraj Singh June 27th, 2010 @ 04:45 AM
- Importance changed from “” to “”
I can confirm that this problem exists in 2-3-stable (latest) and rails3 master(latest).
Neeraj Singh June 27th, 2010 @ 04:48 AM
LH can (and should) do better than that. Priority changed from “” to “” :-)

I did not even change priority.
Neeraj Singh June 27th, 2010 @ 04:48 AM
- Tag changed from “2.3.x, activerecord, bugmash, fixtures, testunit” to “2.3.x, activerecord, bugmash, fixtures, rails3, testunit”
Santiago Pastorino June 27th, 2010 @ 05:01 AM
- Milestone cleared.
- Assigned user set to “José Valim”
José Valim June 27th, 2010 @ 08:06 AM
- Assigned user cleared.
Can anyone work on a patch please?
Neeraj Singh June 27th, 2010 @ 06:42 PM
I spent some time looking at it and ,I think, rails is working as expected. However it might not be what user is expecting.

Here is how test is run with self.use_transactional_fixtures = true .
- test database has no records
- fixture data is loaded into the database. transaction is committed.
- before the test starts a transcation is started because of self.use_transactional_fixtures = true
- tests are run
- all the data created during test are stored in the database (note transaction is not yet committed)
- tests finish
- since self.use_transactional_fixtures = true is used transaction is rolled back.
- note that data created during tests are rolled back. Not the data created by fixtures.
- when the next time this fixture is used then fixture data is wiped out before fixture data is inserted.
Note that when fixture data is loaded then first all the data is wiped out.

In this case the second test created by Levin is not loading fixtures. It means the data created by previous fixtures are still there in database even though use is using self.use_transactional_fixtures = true .

Fixing this issue would be mean that if self.use_transactional_fixtures = true is used then put fixture data also within transaction. This could have strong impact on the way a lot of tests are written.

In order to run tests faster a lot of time users load all the fixture data at once in database and then the data created by tests are rollbacked. In this case one does not have to create data from fixtures time and again and this results in faster tests.

If we bring loading of fixture data within transaction then tests will run much much slower because then fixture data would be loaded in every single test.
José Valim June 27th, 2010 @ 07:13 PM
- State changed from “open” to “wontfix”
Neeraj, awesome explanation about the issue. I'm marking this as won't fix, according to Neeraj information.
You flagged this item as spam.
Levin Alexander June 27th, 2010 @ 07:26 PM
I am not convinced that this would slow down tests much. The fixtures need to be loaded in either case.
I think (but don't know yet) that the slowdown should not be all that big. I'll try to investigate and get some numbers.

José: would you accept a patch for this ticket if the slowdown is small?
José Valim June 27th, 2010 @ 07:37 PM
Avoid fixtures to be loaded is exactly the reason why we have transactional fixtures.

http://clarkware.com/cgi/blosxom/2005/10/24

That said, I cannot accept a patch that removes fixtures after running each test.

My advice is: a solution that will always make the data be deleted, even if you don't want to, is going to make your tests slow. So my advice is: just do Foo.delete_all in the few tests you want to ensure it's empty.
You flagged this item as spam.
Levin Alexander June 27th, 2010 @ 07:50 PM
Got it, thanks.

right now, if you're writing fixtures :table you are deleting fixtures also. The fix for this ticket would just need to shift this work from before each test is run to after. The total amount of work should remain the same.

Could be that I'm wrong about that, though. I'll try to get some data.
José Valim June 27th, 2010 @ 07:54 PM
"If you're writing fixtures :table you are deleting fixtures also."

This just happens once and just before running tests. Maybe is ok to delete the whole data after all tests executed, but it needs to be after all tests. And that is not going to solve your issue.
Jeremy Kemper June 27th, 2010 @ 08:12 PM
Great analysis, Neeraj. This is precisely why fixtures are inserted outside the transaction.

Since then, we added support for savepoints, though. We could begin a transaction for the whole test run, insert the fixtures, the mark a savepoint and rollback to it around each test case.
You flagged this item as spam.
Levin Alexander June 27th, 2010 @ 08:17 PM
The reason I think this ticket is a problem is that different test files can influence each other in unexpected ways. (running FooTest before BarTest will cause BarTest to fail)

Only rolling back to a known state after each TestCase would be enough to fix that particular problem.
Jeremy Kemper October 15th, 2010 @ 11:01 PM
- Milestone set to “3.0.2”
[bulk edit]