This project is archived and is in readonly mode.

#2273 ✓duplicate
Harm Aarts

ActiveSupport::JSON.decode dies on double escaped string

Reported by Harm Aarts | March 17th, 2009 @ 03:34 PM | in 2.x

Parsing a long file with ActiveSupport::JSON.decode dies with ActiveSupport::JSON::ParseError: Invalid JSON string. The underlying error is an ArgumentError. Running the decode with the attached file yields the error.

I believe the date extraction places the wrong marks or uses them prematurely. Inspecting the error showed: '... "ranAt": "2009-02-19 21:48:05" "encoding": "UTF-8 , ...'. This is clearly wrong with a missing comma and the UTF-8 string not correctly closed.

Comments and changes to this ticket

  • Harm Aarts

    Harm Aarts March 17th, 2009 @ 04:26 PM

    I believe the marks are counted wrong in the file activesupport-2.3.2/lib/active_support/json/decoding.rb line 41. The datetime field contains colons (:) and these are incorrectly counted. The same holds true for fields holding urls (http://).

  • Harm Aarts

    Harm Aarts March 18th, 2009 @ 10:24 AM

    • Title changed from “ActiveSupport::JSON.decode dies on long file” to “ActiveSupport::JSON.decode dies on double escaped string”

    Say the content of a file is this: {"a":"\/\/\/\/\/","b":"2009-03-17 14:20:42",some_key Then reading that string with File.open("file_name").read yields this string: "{"a":"\/\/\/\/\/","b":"2009-03-17 14:20:42",some_key Decoding this string with ActiveSupport::JSON.decode goes wrong, the resulting hash: {"a"=>"/////", "b"=>"2009 03-17 14:20:42", "s me_key"=>"some_value"}

    Notice the hyphen between 2009-03 is missing as well as the 'o' in the 'some_key' word.

    I believe this is due to the fact that the string read from the file is double escaped. The in combination with the date matching causes the wrong characters to be replaced with a space which in some cases leads to a mangled result and in other cases in invalid JSON strings.

    A quick fix is to replace the '\/' sequences in the string which resulted from the file read with '/'. Example: ActiveSupport::JSON.decode(File.open("some_file").gsub!('\/','/'))

    I still believe this is a bug.

  • okkez

    okkez March 20th, 2009 @ 08:40 AM

    I think this is a bug, too.

    I found reproduceable code, but it uses multibyte characters.

    >> h = { "data" => [{"key" => "漢字", "date" => "2008-01-01"}, {"key" => "あいう", "date" => "2009-01-01", "date2" => "2009-01-02"}]} >> ActiveSupport::JSON.decode(h.to_json) #=> {"data"=>[{"date"=>Tue, 01 Jan 2008, "key"=>"漢字"}, {"k y"=>"あいう", "date"=>"2009- 1-01", "d te2"=>"2009- 1-02"}]}

    I think this is a bug in ActiveSupport::JSON#convert_json_to_yaml.

    In convert_json_to_yaml, the local variable named 'times' is constructed in wrong timing.

  • Bas van Klinkenberg

    Bas van Klinkenberg August 1st, 2009 @ 01:43 AM

    Please have a look at #2831, which is very similar to this ticket. I submitted a patch to #2831 which should solve this ticket as well.

  • Michael Koziarski

    Michael Koziarski August 3rd, 2009 @ 05:59 AM

    • Tag changed from date, json to bugmash, date, json
  • Nikolay Petrachkov

    Nikolay Petrachkov August 8th, 2009 @ 01:34 PM

    • Assigned user set to “Michael Koziarski”

    bug by Harm Aarts is "not reproducible" .

    Can reproduce bug by okkez

    Before patch by Bas van Klinkenberg (https://rails.lighthouseapp.com/projects/8994/tickets/2831)

    h = { "data" => [{"key" => "漢字", "date" => "2008-01-01"}, {"key" => "あいう", "date" => "2009-01-01", "date2" => "2009-01-02"}]}
    => {"data"=>[{"key"=>"漢字", "date"=>"2008-01-01"}, {"key"=>"あいう", "date"=>"2009-01-01", "date2"=>"2009-01-02"}]}
    >> ActiveSupport::JSON.decode(h.to_json)
    IndexError: index 118 out of string
        from /home/jastix/projects/bugmash/bookclub/vendor/rails/activesupport/lib/active_support/json/backends/yaml.rb:77:in `[]='
        from /home/jastix/projects/bugmash/bookclub/vendor/rails/activesupport/lib/active_support/json/backends/yaml.rb:77:in `block in convert_json_to_yaml'
        from /home/jastix/projects/bugmash/bookclub/vendor/rails/activesupport/lib/active_support/json/backends/yaml.rb:77:in `each'
        from /home/jastix/projects/bugmash/bookclub/vendor/rails/activesupport/lib/active_support/json/backends/yaml.rb:77:in `convert_json_to_yaml'
        from /home/jastix/projects/bugmash/bookclub/vendor/rails/activesupport/lib/active_support/json/backends/yaml.rb:16:in `decode'
        from /home/jastix/projects/bugmash/bookclub/vendor/rails/activesupport/lib/active_support/json/decoding.rb:11:in `decode'
        from (irb):45
        from /usr/local/bin/irb:12:in `'
    
    

    After patch:

    h = { "data" => [{"key" => "漢字", "date" => "2008-01-01"}, {"key" => "あいう", "date" => "2009-01-01", "date2" => "2009-01-02"}]}
    => {"data"=>[{"key"=>"漢字", "date"=>"2008-01-01"}, {"key"=>"あいう", "date"=>"2009-01-01", "date2"=>"2009-01-02"}]}
    
    

    ActiveSupport::JSON.decode(h.to_json) => {"data"=>[{"key"=>"漢字", "date"=>Tue, 01 Jan 2008}, {"key"=>"あいう", "date"=>Thu, 01 Jan 2009, "date2"=>Fri, 02 Jan 2009}]}

    
    
  • Rizwan Reza

    Rizwan Reza August 8th, 2009 @ 10:51 PM

    not reproducible

    -1 I wasn't able to reproduce the error.

  • Dan Croak

    Dan Croak August 8th, 2009 @ 11:00 PM

    Should be marked as "resolved, duplicate". Fix merged in by Yehuda in [#2831].

  • Dan Croak

    Dan Croak August 8th, 2009 @ 11:02 PM

    uhhh... hardbap told me to say this is "verified"

  • CancelProfileIsBroken

    CancelProfileIsBroken August 8th, 2009 @ 11:33 PM

    • State changed from “new” to “duplicate”

    Duplicate of #2831

  • CancelProfileIsBroken

    CancelProfileIsBroken August 8th, 2009 @ 11:34 PM

    • Tag changed from bugmash, date, json to date, json

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »

<h2 style="font-size: 14px">Tickets have moved to Github</h2>

The new ticket tracker is available at <a href="https://github.com/rails/rails/issues">https://github.com/rails/rails/issues</a>

Attachments

Tags

Referenced by

Pages