Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

This project is archived and is in readonly mode.

"redundant UTF-8 sequence" in String#to_json

#1112

Certain strings (which are otherwise valid utf-8 sequences) will cause String#to_json to raise an ArgumentError (redundant UTF-8 sequence). Upon investigating, it turns out to be due to String#to_json's use of String#unpack:


s.unpack("U*")...

Further investigating showed that any two byte sequence beginning with 0xC0 or 0xC1, with the second byte in the range 0x80..0xBF, would cause String#unpack("U*") to raise that exception. Because String#to_json explicitly includes 0xC0 and 0xC1 in it's gsub regex, it seems simplest to have it check only for 0xC2 and up, to avoid the ArgumentError. (The alternative would be to find some way to normalize the redundant sequences to their shorter equivalences...but I'm not clear on how to make that happen.)

I've attached a patch making this (trivial) change, as well as a script that can be used to demonstrate the error for the ranges mentioned above.

Reported by Jamis Buck · September 25th, 2008 @ 04:41 PM

State: stale
Milestone: 2.x
Assigned to: Jeremy Kemper Jeremy Kemper
Importance: none

Activity

  1. Jamis Buck
    Jamis Buck

    Here's the test script; I couldn't see a way to attach more than a single file to a ticket without using the comments. :(

    September 25th, 2008 @ 04:42 PM

  2. josh
    josh
    • Assigned user set to Jeremy Kemper

    STALE!

    j/k Jamis ;)

    January 2nd, 2009 @ 03:59 AM

  3. Pratik
    Pratik
    • State changed from new to stale

    Ok it's really STALE now :)

    March 10th, 2009 @ 11:28 AM

  4. Bira
    Bira

    I've updated the patch - the test attached above should still work for it, I believe.

    January 22nd, 2010 @ 02:03 PM

  5. Damien MATHIEU
    Damien MATHIEU
    • Tag changed from activesupport, bug, patch to activesupport, bug, bugmash, patch

    May 17th, 2010 @ 04:14 PM

  6. Damien MATHIEU
    Damien MATHIEU

    I wouldn't be so sure it's stale. Or at least, there's a bug somewhere else.

    May 17th, 2010 @ 04:53 PM

  7. Julien Sanchez
    Julien Sanchez

    Could the stale status be removed because it appears as closed whereas this bug is still valid?

    May 18th, 2010 @ 09:51 AM

  8. Damien MATHIEU
    Damien MATHIEU

    The patch won't work. It breaks the encoding on some characters (including è). The bug remains however.

    May 18th, 2010 @ 02:26 PM