This project is archived and is in readonly mode.
multipart form upload: mongrel memory consumption
Reported by Markus Seeger | May 15th, 2009 @ 04:53 PM | in 2.x
When uploading big files with a multipart form, the mongrel
process will eat up lots of memory and not release it anytime, thus
permanently stalling the whole machine after some uploads.
Occured on Debian Linux and Windows XP Pro machines, also with a
mongrel cluster balanced by apache.
This didn't happen when switching back to rails 2.2.2.
Reproduction instructions:
rails: 2.3.2
mongrel: 1.1.5
$> rails upload_test
$> cd upload_test
$> script/generate controller upload index
Modify app/views/upload/index.html.erb to:
<% form_tag({}, { :multipart => true }) do %>
<label for="file">File to Upload</label>
<%= file_field_tag "file" %>
<%= submit_tag %>
<% end %>
then start the server, create a huge file
$> mongrel_rails start
$> dd if=/dev/random of=big_file.txt bs=1000000 count=300
and upload using the form:
http://localhost:3000/uploads/new
Comments and changes to this ticket
-
Daniel Guettler May 16th, 2009 @ 01:57 AM
I tracked this down to this commit:
{{{
1adc1496f9152c893e1f08abcb1e5e7272829899 is first bad commit
commit 1adc1496f9152c893e1f08abcb1e5e7272829899
Author: Joshua Peek josh@joshpeek.com
Date: Tue Jan 13 16:09:51 2009 -0600Add RewindableInput wrapper to fix issues with middleware that impolitely eat up non-rewindable input:040000 040000 dea255e41b37ac558b28750a1086ace2c9c35f46 8c6e1795511808984f105270a9ce5949bf128cd4 M actionpack
}}}
Which adds a RewindableInput to the standard Rack stack used internally in Rails. Now when you upload a file this input is not rewindable by itself. So it is read into a StringIO, which means your large file is read into memory.
The rest is pretty much depends on the GC. After ~700 request to /upload all memory got freed again under mongrel. With passenger memory did not get freed again (tried 4000 additional requests without change in memory consumption).Not sure if it's an option for you but the RewindableInput can be removed from the Rack stack which brings back memory to "normal". Not sure about the consequences of removing it. So do it on your own risk... would be nice to get a comment from somebody who knows more about why it was added and which middleware impolitely eats up non-rewindable inputs.
-
Daniel Guettler May 16th, 2009 @ 02:01 AM
Ah forgot to tell you how to remove it. Under config/initializers/ add a file containing (or use an existing one):
ActionController::Dispatcher.middleware.delete(ActionController::RewindableInput)This will remove remove RewindableInput from the Rack stack
-
Daniel Guettler May 26th, 2009 @ 04:23 AM
This ticket should be closed! RewindableInput has been removed in this commit: github.com/rails/rails/commit/61a14569379974564a98b229ab595dfec18d2059 which solves the problem described above.
-
Markus Seeger May 26th, 2009 @ 01:52 PM
I can confirm the fix (removing the RewindableInput).
Thank you very much! -
Steve St. Martin April 16th, 2010 @ 01:27 AM
- Assigned user set to “Ryan Bigg”
OP confirms fix, can be marked as resolved.
-
Ryan Bigg April 16th, 2010 @ 01:28 AM
- State changed from “new” to “resolved”
-
-
Create your profile
Help contribute to this project by taking a few moments to create your personal profile. Create your profile »
<h2 style="font-size: 14px">Tickets have moved to Github</h2>
The new ticket tracker is available at <a href="https://github.com/rails/rails/issues">https://github.com/rails/rails/issues</a>
Daniel Guettler
Jeremy Kemper
Ryan Bigg