#1112 "redundant UTF-8 sequence" in String#to_json - Ruby on Rails

Type	To find
responsible:me	tickets assigned to you
tagged:"@high"	tickets tagged @high
milestone:next	tickets in the upcoming milestone
state:invalid	tickets with the state invalid
created:"last week"	tickets created last week
sort:number, importance, updated	tickets sorted by #, importance or updated
Combine keywords for powerful searching.
Use advanced searching »

This project is archived and is in readonly mode.

#1112 ✓stale

"redundant UTF-8 sequence" in String#to_json

Reported by Jamis Buck | September 25th, 2008 @ 04:41 PM | in 2.x

Certain strings (which are otherwise valid utf-8 sequences) will cause String#to_json to raise an ArgumentError (redundant UTF-8 sequence). Upon investigating, it turns out to be due to String#to_json's use of String#unpack:


s.unpack("U*")...

Further investigating showed that any two byte sequence beginning with 0xC0 or 0xC1, with the second byte in the range 0x80..0xBF, would cause String#unpack("U*") to raise that exception. Because String#to_json explicitly includes 0xC0 and 0xC1 in it's gsub regex, it seems simplest to have it check only for 0xC2 and up, to avoid the ArgumentError. (The alternative would be to find some way to normalize the redundant sequences to their shorter equivalences...but I'm not clear on how to make that happen.)

I've attached a patch making this (trivial) change, as well as a script that can be used to demonstrate the error for the ranges mentioned above.

redundant-utf8-sequence.patch 697 Bytes

Comments and changes to this ticket

You flagged this item as spam.
Jamis Buck September 25th, 2008 @ 04:42 PM
Here's the test script; I couldn't see a way to attach more than a single file to a ticket without using the comments. :(
- test.rb 370 Bytes
josh January 2nd, 2009 @ 03:59 AM
- Assigned user set to “Jeremy Kemper”
STALE!

j/k Jamis ;)
Pratik March 10th, 2009 @ 11:28 AM
- State changed from “new” to “stale”
Ok it's really STALE now :)
You flagged this item as spam.
Bira January 22nd, 2010 @ 02:03 PM
I've updated the patch - the test attached above should still work for it, I believe.
- 0001-1112-fix-redundant-utf08-sequence-in-string-to_json.patch 1.2 KB
You flagged this item as spam.
Damien MATHIEU May 17th, 2010 @ 04:14 PM
- Tag changed from “activesupport, bug, patch” to “activesupport, bug, bugmash, patch”
You flagged this item as spam.
Damien MATHIEU May 17th, 2010 @ 04:53 PM
I wouldn't be so sure it's stale. Or at least, there's a bug somewhere else.
You flagged this item as spam.
Julien Sanchez May 18th, 2010 @ 09:51 AM
Could the stale status be removed because it appears as closed whereas this bug is still valid?
You flagged this item as spam.
Damien MATHIEU May 18th, 2010 @ 02:26 PM
The patch won't work. It breaks the encoding on some characters (including è). The bug remains however.