Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

This project is archived and is in readonly mode.

Ruby1.9: submitted string form parameters with non-ASCII characters cause encoding errors

#4336

(Using latest Rails 3 from git and latest ruby-1.9.2-head)

When one enters any non-ASCII character(s) into a form's text field and submits the form one gets an "incompatible character encodings: ASCII-8BIT and UTF-8" error, unless one does something like

@tags = params[:tags].force_encoding('UTF-8')

Reported by Michael Hasenstein · April 7th, 2010 @ 12:25 PM

State: duplicate
Milestone: 3.0.2
Assigned to: Jeremy Kemper Jeremy Kemper
Importance: Low

Activity

  1. Jeremy Kemper
    Jeremy Kemper
    • State changed from new to open
    • Assigned user set to Jeremy Kemper
    • Milestone cleared.

    We need to solve this at the Rack level by having a middleware force_encoding on incoming params and headers.

    April 7th, 2010 @ 06:36 PM

  2. Szymon Nowak
    Szymon Nowak

    In our projects (Rails 2.3.5) we're currently using something like this to solve the problem with params:

      before_filter :force_utf8_params
      
      def force_utf8_params
        traverse = lambda do |object, block|
          if object.kind_of?(Hash)
            object.each_value { |o| traverse.call(o, block) }
          elsif object.kind_of?(Array)
            object.each { |o| traverse.call(o, block) }
          else
            block.call(object)
          end
          object
        end
        force_encoding = lambda do |o|
          o.force_encoding(Encoding::UTF_8) if o.respond_to?(:force_encoding)
        end
        traverse.call(params, force_encoding)
      end
    

    but we want to move it into a middleware as well.

    April 8th, 2010 @ 12:08 PM

  3. Marcin Kulik
    Marcin Kulik

    Jeremy, I think this middleware should not blindly do force_encoding("utf8") because browser could send data in other encoding without telling which. It could be really in ISO-8859-1 and forcing UTF8 will damage the string. Maybe the middleware should use some heuristics to guess encoding if HTTP encoding header wasn't set. Wycats wrote in one if his recent posts that there are some ideas for such a middleware. Do you know if anyone tried to do sth with it?
    I'm willing to help as this is real PITA for me at the moment.

    May 12th, 2010 @ 10:32 AM

  4. Dan Pickett
    Dan Pickett
    • Tag changed from encoding, rails3, ruby19, utf8 to bugmash, encoding, rails3, ruby19, utf8

    May 15th, 2010 @ 01:48 AM

  5. Sasha Koss
    Sasha Koss

    @Szymon Nowak

    Thanks! It works fine for me!

    May 21st, 2010 @ 03:15 AM

  6. Marcin Kulik
    Marcin Kulik

    @Szymon

    Thanks! It works pretty well so far.

    May 31st, 2010 @ 02:23 PM

  7. David Morton
    David Morton

    My first attempt at creating a rails 3 project hit this one. @Szymon solution works, if a bit hackish. :)

    June 21st, 2010 @ 10:59 PM

  8. Rohit Arondekar
    Rohit Arondekar
    • State changed from open to duplicate
    • Importance changed from to Low

    I think this issue was resolved in this ticket https://rails.lighthouseapp.com/projects/8994/tickets/4807

    Can you please confirm? If there are still issues that remain then can you continue the discussion on that ticket?

    July 4th, 2010 @ 02:59 AM

  9. Jeremy Kemper