#1100 JSON decoding of \u escape sequences - Ruby on Rails

Type	To find
responsible:me	tickets assigned to you
tagged:"@high"	tickets tagged @high
milestone:next	tickets in the upcoming milestone
state:invalid	tickets with the state invalid
created:"last week"	tickets created last week
sort:number, importance, updated	tickets sorted by #, importance or updated
Combine keywords for powerful searching.
Use advanced searching »

This project is archived and is in readonly mode.

#1100 ✓resolved

JSON decoding of \u escape sequences

Reported by Tim Pope | September 24th, 2008 @ 01:04 AM

Currently ActiveSupport::JSON.decode doesn't handle "\u0000" style escape sequences. The attached patch rectifies this.

json_unicode.patch 1.8 KB

Comments and changes to this ticket

You flagged this item as spam.
Philip Hallstrom October 23rd, 2008 @ 06:32 PM
- Tag set to “2.1, activesupport, bug, patch, tested, tiny”
I'll admit I am not an expert about JSON. I applied your patch and the tests passed. However, I'm testing out Google's AJAX search API that returns JSON and it isn't passing. Not sure if Google is doing something wrong, the strings are being double escaped or if it's the patch. With your patch applied I get this (note the \u003's at the very end)
```
>> ActiveSupport::JSON.decode(open('http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=Ruby+on+Rails').read)['responseData']['results'].first['content']
=> "RoR home; full stack, Web application framework optimized for sustainable   programming productivity, allows writing sound code by favoring convention over   \\u003cb\\u003e...\\u003c/b\\u003e"
```
With an update to your patch (attached) I get this:
```
>> ActiveSupport::JSON.decode(open('http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=Ruby+on+Rails').read)['responseData']['results'].first['content']
=> "RoR home; full stack, Web application framework optimized for sustainable   programming productivity, allows writing sound code by favoring convention over   <b>...</b>"
```
This updated patch still passes the tests supplied by the first patch.

It also seems to work if you run this conversion at the very beginning on the original json string in it's entirety, but I'm not sure that's a good idea.
- json_unicode_2.patch 2.3 KB
You flagged this item as spam.
Tim Pope October 23rd, 2008 @ 08:27 PM
+1 on the improvement. I'm not a big fan of repeating the same gsub logic in two places but reusing the code is probably more trouble than it's worth.

#973 points out another JSON decoding issue. These two should probably be combined into one patch. I'll try to take a look at it this weekend if nobody beats me to it.
You flagged this item as spam.
Philip Hallstrom October 24th, 2008 @ 12:23 AM
Another patch. I was encountering situations where $1 would be valid (ie. "u003c") before the $1.start_with("u") and nil immediately after.

This patch fixes that by explicitly setting $1 to a local variable.
- json_unicode_3.patch 4.1 KB
You flagged this item as spam.
Nick Adams November 16th, 2008 @ 09:14 AM
Thank you so much. Patched my rails using include library until this is fixed in edge.
DHH November 16th, 2008 @ 02:49 PM
- Assigned user set to “Rick”
You flagged this item as spam.
Michael Siebert February 7th, 2009 @ 04:09 PM
what's the status of that patch? i just confirmed it applies and passes on master@2009-02-07.

+1 for patch 3 if anyone cares
You flagged this item as spam.
Ernesto Jiménez February 26th, 2009 @ 02:46 PM
PATCH 3: useful and test pass

+1
DHH February 27th, 2009 @ 12:41 PM
This doesn't apply cleanly against master right now. Could someone take a look at fixing?
You flagged this item as spam.
Bira March 10th, 2009 @ 12:39 PM
This is something I'm really interested in seeing on Rails, so I've applied it to master. This also solves http://dev.rubyonrails.org/ticke..., by the way.
- json_unicode_4.patch 2.6 KB
Repository March 10th, 2009 @ 05:08 PM
- State changed from “new” to “resolved”
(from [9b9b2937ce3bef3bca9d22821e76c40cc74fa689]) Properly decode \u escape sequences in JSON [#1100 state:resolved] [Tim Pope, Philip Hallstrom]

Signed-off-by: Pratik Naik pratiknaik@gmail.com http://github.com/rails/rails/co...
CancelProfileIsBroken March 10th, 2009 @ 05:18 PM
- State changed from “resolved” to “open”
Test fail with this patch applied on Ruby 1.9.1:
```
Name: test_json_decoding_{"matzue": Type: Failure Message: <{"matzue"=>"松江", "asakusa"=>"浅草"}> expected but was <{"matzue"=>"松江", "as akusa"=>"浅草 "}>. 
```
Previous to this, all ActiveSupport tests passed on 1.9.1.
You flagged this item as spam.
Bira March 11th, 2009 @ 09:23 AM
I see a part of the previous patch was untested, and so was removed from master. I've added a few tests to it, and I hope they make sense.

Unfortunately I can't do anything about the 1.9.1 issue from where I am right now. I'll try to install it and test later today.
- json_unicode_tests.diff 2.2 KB
You flagged this item as spam.
Akira Matsuda March 11th, 2009 @ 10:03 AM
Please take a look at #1969 for the Ruby 1.9 matter.

I guess,
```
json[left.succ..right_pos[i]]
```
won't work properly with multibyte chars on Ruby 1.9.
You flagged this item as spam.
Bira March 11th, 2009 @ 11:53 AM
Thank you very much, Akira. I've installed Ruby 1.9 and incorporated your changes from #1969 (which the patch from this ticket had overwritten). Tests now pass both for Ruby 1.8 and Ruby 1.9. The patch I'm attaching here has all the changes of the previous one, plus 1.9 support.
- json_unicode_tests_2.diff 2.2 KB
Jeremy Kemper March 11th, 2009 @ 09:35 PM
- State changed from “open” to “verified”
- Milestone cleared.
Repository March 11th, 2009 @ 10:02 PM
- State changed from “verified” to “resolved”
(from [7b382cb9e5c5706f8d15216159a2873375915c9c]) Reverted affe50105f7027a44eb6e9cfb56f5b3fc070b19b and added more JSON decoding tests.

Works on Ruby 1.8 and 1.9

[#1100 state:resolved]

Signed-off-by: Jeremy Kemper jeremy@bitsweat.net http://github.com/rails/rails/co...