This project is archived and is in readonly mode.

#3504 ✓invalid
Aljaz

String parameterize / normalize bug

Reported by Aljaz | November 23rd, 2009 @ 06:06 PM

If I create a String with special characters (ščž..):

str = "test ščž"
str.parameterize

I get the following error:

NoMethodError: undefined method normalize' for "test s\237\247s":String

The same thing happens I call normalize on the str.

Comments and changes to this ticket

  • Yaroslav Markin

    Yaroslav Markin April 10th, 2010 @ 08:11 PM

    • Tag changed from normalize, parameterize, string to bug, normalize, parameterize, string

    Can't reproduce on Ruby 1.8.7 and Rails 2.3.5

    >> str = "test ščž"
    => "test šč\305\276"
    >> str.parameterize
    => #
    >> str.parameterize.to_s
    => "test-scz"
    

    Can you please be more specific with Ruby and Rails versions?

  • Nathaniel Bibler

    Nathaniel Bibler April 11th, 2010 @ 03:36 AM

    • Tag changed from bug, normalize, parameterize, string to bug, invalid, normalize, parameterize, string

    I could not reproduce with Rails 2.3.5 and Ruby 1.8.7, either. HOWEVER! Attempting to recreate with 1.8.7p248 and Rails 3, I got:

    /gems/ruby-1.8.7-p248/gems/activesupport-3.0.0.beta/lib/active_support/inflector/transliterate.rb:26:in `transliterate': undefined method `normalize' for "test \305\241\304\215\305\276":String (NoMethodError)
        from /gems/ruby-1.8.7-p248/gems/activesupport-3.0.0.beta/lib/active_support/inflector/transliterate.rb:48:in `parameterize'
        from /gems/ruby-1.8.7-p248/gems/activesupport-3.0.0.beta/lib/active_support/core_ext/string/inflections.rb:102:in `parameterize'
        from /gems/ruby-1.8.7-p248/gems/activesupport-3.0.0.beta/lib/active_support/multibyte/chars.rb:102:in `__send__'
        from /gems/ruby-1.8.7-p248/gems/activesupport-3.0.0.beta/lib/active_support/multibyte/chars.rb:102:in `method_missing'
        from test.rb:12
    

    To fix it, I had to define the $KCODE (encoding) for the document:

    require 'rubygems'
    require 'active_support/all'
    
    $KCODE = 'UTF8'
    
    str = "test ščž"
    puts str.parameterize.inspect
    puts str.parameterize.to_s
    
    => #<ActiveSupport::Multibyte::Chars:0x101876110 @wrapped_string="test-scz">
    => test-scz
    

    The author should update their config/environment.rb or create an "encoding" initializer properly setting their $KCODE for their application. This is a known requirement of Ruby 1.8 when working with multibyte strings.

    Tagging with invalid.

  • Ryan Bigg

    Ryan Bigg April 11th, 2010 @ 02:16 PM

    • Tag cleared.

    Shouldn't the $KCODE be set by Rails to something that supports this, such as UTF8, by default? I do not think this ticket is invalid.

  • Nathaniel Bibler

    Nathaniel Bibler April 11th, 2010 @ 05:45 PM

    Perhaps it should.. does anyone know of any downside of setting the $KCODE to UTF8 all the time? Perhaps a time penalty when working with string comparisons and whatnot?

  • Yaroslav Markin

    Yaroslav Markin April 11th, 2010 @ 05:52 PM

    • Tag set to activesupport, utf8
    • Assigned user set to “Ryan Bigg”

    Rails 3:

    railties/lib/rails.rb:# For Ruby 1.8, this initialization sets $KCODE to 'u' to enable the
    railties/lib/rails.rb:  $KCODE='u'
    

    Rails 2.3:

    railties/lib/initializer.rb:    # For Ruby 1.8, this initialization sets $KCODE to 'u' to enable the
    railties/lib/initializer.rb:      $KCODE='u' if RUBY_VERSION < '1.9'
    

    And it was there for quite a long time. Unicode KCODE is default in Rails 2.3+ at least.

  • Ryan Bigg

    Ryan Bigg April 11th, 2010 @ 09:24 PM

    • State changed from “new” to “open”

    Nathaniel Bibler, you seem to be knowledgeable in this area, could you submit a test case and if you could, a fix for it? Thank you!

  • releod

    releod April 11th, 2010 @ 10:32 PM

    If it is relevant - use of $KCODE in 1.9 will produce a warning...

    warning: variable $KCODE is no longer effective; ignored

  • Yaroslav Markin

    Yaroslav Markin April 12th, 2010 @ 12:59 AM

    You can ensure $KCODE defaults to "UTF8" in Rails apps if you open up script/console in any of your applications and inspect $KCODE.

    @releod -- you are right, $KCODE is for Ruby 1.8 and less only.

  • Nathaniel Bibler

    Nathaniel Bibler April 12th, 2010 @ 08:13 PM

    Part of the problem is that the original author didn't post the Ruby/Rails versions they were using to cause the problem. So, they may actually be using an older version of Rails (<2.3). For the sake of argument, I assumed Rails 2.3.5 / Rails 3 for testing.

    You can see in my sample script, I didn't fully load Rails 3; instead loading only the ActiveSupport module, which didn't execute the Rails $KCODE setter that @Yaroslav mentioned.

    So, if this ticket isn't invalid or wontfix I would think it should at least be needs-more-info.

  • Ryan Bigg

    Ryan Bigg April 12th, 2010 @ 09:52 PM

    If it's working on 2.3.5 then that's fine.

    If it is broken in 3.0.0, then please write a parch.

  • Norman Clarke

    Norman Clarke April 13th, 2010 @ 01:24 PM

    • Tag changed from activesupport, utf8 to activesupport, invalid, utf8

    Jeremy committed a patch of mine today that modifies the way Inflector.parameterize and Inflector.transliterate work. The #transliterate method invoked by parameterize now always operates on an instance of ActiveSupport::Multibyte::Chars rather than a proxy class, so it should not be possible to reach this error by invoking #parameterize. Marking the ticket invalid.

  • Ryan Bigg

    Ryan Bigg April 13th, 2010 @ 09:41 PM

    • Tag changed from activesupport, invalid, utf8 to activesupport, utf8
    • State changed from “open” to “invalid”

    confirmed.

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »

<h2 style="font-size: 14px">Tickets have moved to Github</h2>

The new ticket tracker is available at <a href="https://github.com/rails/rails/issues">https://github.com/rails/rails/issues</a>

Referenced by

Pages