#1100 JSON decoding of \u escape sequences - Ruby on Rails

Tag set to 2.1, activesupport, bug, patch, tested, tiny

I'll admit I am not an expert about JSON. I applied your patch and the tests passed. However, I'm testing out Google's AJAX search API that returns JSON and it isn't passing. Not sure if Google is doing something wrong, the strings are being double escaped or if it's the patch. With your patch applied I get this (note the \u003's at the very end)


>> ActiveSupport::JSON.decode(open('http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=Ruby+on+Rails').read)['responseData']['results'].first['content']
=> "RoR home; full stack, Web application framework optimized for sustainable   programming productivity, allows writing sound code by favoring convention over   \\u003cb\\u003e...\\u003c/b\\u003e"

With an update to your patch (attached) I get this:


>> ActiveSupport::JSON.decode(open('http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=Ruby+on+Rails').read)['responseData']['results'].first['content']
=> "RoR home; full stack, Web application framework optimized for sustainable   programming productivity, allows writing sound code by favoring convention over   <b>...</b>"

This updated patch still passes the tests supplied by the first patch.

It also seems to work if you run this conversion at the very beginning on the original json string in it's entirety, but I'm not sure that's a good idea.

json_unicode_2.patch 2.34 KB

October 23rd, 2008 @ 06:32 PM

Tim Pope

+1 on the improvement. I'm not a big fan of repeating the same gsub logic in two places but reusing the code is probably more trouble than it's worth.

#973 points out another JSON decoding issue. These two should probably be combined into one patch. I'll try to take a look at it this weekend if nobody beats me to it.

October 23rd, 2008 @ 08:27 PM

Philip Hallstrom

Another patch. I was encountering situations where $1 would be valid (ie. "u003c") before the $1.start_with("u") and nil immediately after.

This patch fixes that by explicitly setting $1 to a local variable.

json_unicode_3.patch 4.1 KB

October 24th, 2008 @ 12:23 AM

Nick Adams

Thank you so much. Patched my rails using include library until this is fixed in edge.

November 16th, 2008 @ 09:14 AM

DHH

Assigned user set to Rick

November 16th, 2008 @ 02:49 PM

Michael Siebert

what's the status of that patch? i just confirmed it applies and passes on master@2009-02-07.

+1 for patch 3 if anyone cares

February 7th, 2009 @ 04:09 PM

Ernesto Jiménez

PATCH 3: useful and test pass

+1

February 26th, 2009 @ 02:46 PM

DHH

This doesn't apply cleanly against master right now. Could someone take a look at fixing?

February 27th, 2009 @ 12:41 PM

Bira

This is something I'm really interested in seeing on Rails, so I've applied it to master. This also solves http://dev.rubyonrails.org/ticke..., by the way.

json_unicode_4.patch 2.55 KB

March 10th, 2009 @ 12:39 PM

Repository

State changed from new to resolved

(from [9b9b2937ce3bef3bca9d22821e76c40cc74fa689]) Properly decode \u escape sequences in JSON [#1100 state:resolved] [Tim Pope, Philip Hallstrom]

Signed-off-by: Pratik Naik pratiknaik@gmail.com http://github.com/rails/rails/co...

March 10th, 2009 @ 05:08 PM

CancelProfileIsBroken

State changed from resolved to open

Test fail with this patch applied on Ruby 1.9.1:


Name: test_json_decoding_{"matzue": Type: Failure Message: <{"matzue"=>"松江", "asakusa"=>"浅草"}> expected but was <{"matzue"=>"松江", "as akusa"=>"浅草 "}>.

Previous to this, all ActiveSupport tests passed on 1.9.1.

March 10th, 2009 @ 05:18 PM

Bira

I see a part of the previous patch was untested, and so was removed from master. I've added a few tests to it, and I hope they make sense.

Unfortunately I can't do anything about the 1.9.1 issue from where I am right now. I'll try to install it and test later today.

json_unicode_tests.diff 2.19 KB

March 11th, 2009 @ 09:23 AM

Akira Matsuda

Please take a look at #1969 for the Ruby 1.9 matter.

I guess,


json[left.succ..right_pos[i]]

won't work properly with multibyte chars on Ruby 1.9.

March 11th, 2009 @ 10:03 AM

Bira

Thank you very much, Akira. I've installed Ruby 1.9 and incorporated your changes from #1969 (which the patch from this ticket had overwritten). Tests now pass both for Ruby 1.8 and Ruby 1.9. The patch I'm attaching here has all the changes of the previous one, plus 1.9 support.