This project is archived and is in readonly mode.

#4336 ✓duplicate
Michael Hasenstein

Ruby1.9: submitted string form parameters with non-ASCII characters cause encoding errors

Reported by Michael Hasenstein | April 7th, 2010 @ 12:25 PM | in 3.0.2

(Using latest Rails 3 from git and latest ruby-1.9.2-head)

When one enters any non-ASCII character(s) into a form's text field and submits the form one gets an "incompatible character encodings: ASCII-8BIT and UTF-8" error, unless one does something like

@tags = params[:tags].force_encoding('UTF-8')

Comments and changes to this ticket

  • Jeremy Kemper

    Jeremy Kemper April 7th, 2010 @ 06:36 PM

    • State changed from “new” to “open”
    • Assigned user set to “Jeremy Kemper”
    • Milestone cleared.

    We need to solve this at the Rack level by having a middleware force_encoding on incoming params and headers.

  • Szymon Nowak

    Szymon Nowak April 8th, 2010 @ 12:08 PM

    In our projects (Rails 2.3.5) we're currently using something like this to solve the problem with params:

      before_filter :force_utf8_params
      
      def force_utf8_params
        traverse = lambda do |object, block|
          if object.kind_of?(Hash)
            object.each_value { |o| traverse.call(o, block) }
          elsif object.kind_of?(Array)
            object.each { |o| traverse.call(o, block) }
          else
            block.call(object)
          end
          object
        end
        force_encoding = lambda do |o|
          o.force_encoding(Encoding::UTF_8) if o.respond_to?(:force_encoding)
        end
        traverse.call(params, force_encoding)
      end
    

    but we want to move it into a middleware as well.

  • Marcin Kulik

    Marcin Kulik May 12th, 2010 @ 10:32 AM

    Jeremy, I think this middleware should not blindly do force_encoding("utf8") because browser could send data in other encoding without telling which. It could be really in ISO-8859-1 and forcing UTF8 will damage the string. Maybe the middleware should use some heuristics to guess encoding if HTTP encoding header wasn't set. Wycats wrote in one if his recent posts that there are some ideas for such a middleware. Do you know if anyone tried to do sth with it?
    I'm willing to help as this is real PITA for me at the moment.

  • Dan Pickett

    Dan Pickett May 15th, 2010 @ 01:48 AM

    • Tag changed from encoding, rails3, ruby19, utf8 to bugmash, encoding, rails3, ruby19, utf8
  • Sasha Koss

    Sasha Koss May 21st, 2010 @ 03:15 AM

    @Szymon Nowak

    Thanks! It works fine for me!

  • Marcin Kulik

    Marcin Kulik May 31st, 2010 @ 02:23 PM

    @Szymon

    Thanks! It works pretty well so far.

  • David Morton

    David Morton June 21st, 2010 @ 10:59 PM

    My first attempt at creating a rails 3 project hit this one. @Szymon solution works, if a bit hackish. :)

  • Rohit Arondekar

    Rohit Arondekar July 4th, 2010 @ 02:59 AM

    • State changed from “open” to “duplicate”
    • Importance changed from “” to “Low”

    I think this issue was resolved in this ticket https://rails.lighthouseapp.com/projects/8994/tickets/4807

    Can you please confirm? If there are still issues that remain then can you continue the discussion on that ticket?

  • Jeremy Kemper

    Jeremy Kemper October 15th, 2010 @ 11:01 PM

    • Milestone set to 3.0.2

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »

<h2 style="font-size: 14px">Tickets have moved to Github</h2>

The new ticket tracker is available at <a href="https://github.com/rails/rails/issues">https://github.com/rails/rails/issues</a>

Pages