This project is archived and is in readonly mode.

multipart form upload: mongrel memory consumption

#2652

When uploading big files with a multipart form, the mongrel process will eat up lots of memory and not release it anytime, thus permanently stalling the whole machine after some uploads.
Occured on Debian Linux and Windows XP Pro machines, also with a mongrel cluster balanced by apache.
This didn't happen when switching back to rails 2.2.2.

Reproduction instructions:

rails: 2.3.2
mongrel: 1.1.5

$> rails upload_test
$> cd upload_test
$> script/generate controller upload index

Modify app/views/upload/index.html.erb to:

<% form_tag({}, { :multipart => true }) do %>
  <label for="file">File to Upload</label>
  <%= file_field_tag "file" %>
  <%= submit_tag %>
<% end %>

then start the server, create a huge file

$> mongrel_rails start
$> dd if=/dev/random of=big_file.txt bs=1000000 count=300

and upload using the form:

http://localhost:3000/uploads/new

Reported by Markus Seeger · May 15th, 2009 @ 04:53 PM

Activity

Daniel Guettler
I tracked this down to this commit:

{{{

1adc1496f9152c893e1f08abcb1e5e7272829899 is first bad commit
commit 1adc1496f9152c893e1f08abcb1e5e7272829899
Author: Joshua Peek josh@joshpeek.com
Date: Tue Jan 13 16:09:51 2009 -0600
```
Add RewindableInput wrapper to fix issues with middleware that impolitely eat up non-rewindable input
```
:040000 040000 dea255e41b37ac558b28750a1086ace2c9c35f46 8c6e1795511808984f105270a9ce5949bf128cd4 M actionpack

}}}

Which adds a RewindableInput to the standard Rack stack used internally in Rails. Now when you upload a file this input is not rewindable by itself. So it is read into a StringIO, which means your large file is read into memory.
The rest is pretty much depends on the GC. After ~700 request to /upload all memory got freed again under mongrel. With passenger memory did not get freed again (tried 4000 additional requests without change in memory consumption).

Not sure if it's an option for you but the RewindableInput can be removed from the Rack stack which brings back memory to "normal". Not sure about the consequences of removing it. So do it on your own risk... would be nice to get a comment from somebody who knows more about why it was added and which middleware impolitely eats up non-rewindable inputs.
May 16th, 2009 @ 01:57 AM
Daniel Guettler
Ah forgot to tell you how to remove it. Under config/initializers/ add a file containing (or use an existing one):
```
ActionController::Dispatcher.middleware.delete(ActionController::RewindableInput)
```
This will remove remove RewindableInput from the Rack stack
May 16th, 2009 @ 02:01 AM
Daniel Guettler
This ticket should be closed! RewindableInput has been removed in this commit: github.com/rails/rails/commit/61a14569379974564a98b229ab595dfec18d2059 which solves the problem described above.

May 26th, 2009 @ 04:23 AM
Markus Seeger
I can confirm the fix (removing the RewindableInput).
Thank you very much!

May 26th, 2009 @ 01:52 PM
Steve St. Martin
- Assigned user set to Ryan Bigg
OP confirms fix, can be marked as resolved.

April 16th, 2010 @ 01:27 AM
Ryan Bigg
- State changed from new to resolved
April 16th, 2010 @ 01:28 AM
Ryan Bigg
- Tag cleared.
Automatic cleanup of spam.

October 9th, 2010 @ 09:55 PM
Jeff Kreeftmeijer
Automatic cleanup of spam.

November 7th, 2010 @ 04:57 PM