This project is archived and is in readonly mode.

#3479 ✓resolved
Ethan

ActiveSupport::JSON fails to decode unicode-escaped newline and literal newlines

Reported by Ethan | November 10th, 2009 @ 11:08 PM | in 2.3.10

With the YAML backend, ActiveSupport::JSON decodes a unicode-escaped newline (or CRLF) as a space. It does the same with a literal newline. But, it deals correctly with a \n-escaped newline.

# unicode-escaped LF
>> ActiveSupport::JSON.decode("\"\\u000a\"")
=> " "
# unicode-escaped CRLF
>> ActiveSupport::JSON.decode("\"\\u000d\\u000a\"")
=> " "
# literal LF
>> ActiveSupport::JSON.decode("\"\n\"")
=> " "
# normal backslash-escaped LF
>> ActiveSupport::JSON.decode("\"\\n\"")
=> "\n"

The JSON gem does this correctly:

>> JSON.parse "[\"\\u000a\"]"
=> ["\n"]

(note that it's in an array because the JSON gem doesn't seem to like decoding just strings. but that's not at issue here; the YAML backend fails when it's in an array too)

This is because the method convert_json_to_yaml changes the escaped newline to a literal newline:

>> ActiveSupport::JSON::Backends::Yaml.send(:convert_json_to_yaml, "[\"\\u000a\"]")
=> "[\"\n\"]"

and then calls YAML.load("["\n"]") which changes the newline to a space:

>> YAML.load "[\"\n\"]"
=> [" "]

I'm not sure if that's correct behavior for YAML or not. But, the result is certainly incorrect for JSON.
A workaround could be to escape newlines in strings before passing them to YAML, after converting from unicode-escaped newline to literal newline. that is, change "\n" to "\n"; change "\r\n" to "\r\n", as YAML correctly converts those to literal newlines.

Comments and changes to this ticket

  • Ethan

    Ethan November 11th, 2009 @ 02:01 AM

    formatting seems to have gotten lost; that last line was supposed to say:

    change "\n" to "\\n"; change "\r\n" to "\\r\\n", as YAML correctly converts those to literal newlines.

    (maybe it will work this time? where's the preview function on this thing?)

  • Maxime RETY

    Maxime RETY June 14th, 2010 @ 03:41 PM

    • Tag changed from “activesupport, decode, json, newline, yaml” to “activesupport, decode, json, newline, patch, yaml”

    As you previously reported it, ActiveSupport::JSON fails to decode a unicode-escaped newline with the YAML backend :

    unicode-escaped LF

    ActiveSupport::JSON.decode(""\u000a"") => " "

    With the new Yajl backend, same code works :

    unicode-escaped LF (with Yajl backend)

    ActiveSupport::JSON.decode(""\u000a"") => "\n"

    I did not test with JSON gem, since it's not a default json backend since rails 2.3.6.

    I believe "\n" char is interpreted in YAML as a formatting instruction of the file.
    So newlines need to be escaped as "\n" before passing yaml string to "YAML.load" method.

    I also think it's a newline-only bug. Other "backslash-chars" like "\r" are correctly handled without extra escaping.

    I propose the attached patch to fix the issue in "convert_json_to_yaml" method.
    I added 3 tests to check that bug is fixed with any backend.

    Did you have any other feedback on this issue ?

  • Maxime RETY

    Maxime RETY June 14th, 2010 @ 03:59 PM

    Oops i should have read formatting help more carefully !

    As you previously reported it, ActiveSupport::JSON fails to decode a unicode-escaped newline with the YAML backend :

    # unicode-escaped LF
    ActiveSupport::JSON.decode("\"\\u000a\"")
    => " "

    With the new Yajl backend, same code works :

    # unicode-escaped LF (with Yajl backend)
    ActiveSupport::JSON.decode("\"\\u000a\"")
    => "\n"

    I did not test with JSON gem, since it's not a default json backend since rails 2.3.6.

    I believe "\n" char is interpreted in YAML as a formatting instruction of the file.
    So newlines need to be escaped as "\\n" before passing yaml string to "YAML.load" method.

    I propose the attached patch to fix the issue in "convert_json_to_yaml" method.
    I added 3 tests to check that bug is fixed with any backend.

    Did you have any other feedback on this issue ?

  • Michael Koziarski

    Michael Koziarski June 15th, 2010 @ 04:36 AM

    • Milestone set to “2.3.9”
  • Jeremy Kemper

    Jeremy Kemper August 30th, 2010 @ 02:28 AM

    • Milestone changed from “2.3.9” to “2.3.10”
  • Santiago Pastorino

    Santiago Pastorino February 2nd, 2011 @ 04:49 PM

    • State changed from “new” to “open”

    This issue has been automatically marked as stale because it has not been commented on for at least three months.

    The resources of the Rails core team are limited, and so we are asking for your help. If you can still reproduce this error on the 3-0-stable branch or on master, please reply with all of the information you have about it and add "[state:open]" to your comment. This will reopen the ticket for review. Likewise, if you feel that this is a very important feature for Rails to include, please reply with your explanation so we can consider it.

    Thank you for all your contributions, and we hope you will understand this step to focus our efforts where they are most helpful.

  • Santiago Pastorino

    Santiago Pastorino February 2nd, 2011 @ 04:49 PM

    • State changed from “open” to “stale”
  • Ethan

    Ethan February 2nd, 2011 @ 08:38 PM

    • State changed from “stale” to “open”

    This is still reproducing in ActiveSupport 3.0.3. Maxime's patch should be applied.

    [state:open]

  • Repository

    Repository February 2nd, 2011 @ 09:03 PM

    • State changed from “open” to “resolved”

    (from [68e3fb81090ba67575e513407fc2463dba3b002b]) Fix JSON decoding of newline character with Yaml backend [#3479 state:resolved]

    Signed-off-by: Santiago Pastorino santiago@wyeworks.com
    https://github.com/rails/rails/commit/68e3fb81090ba67575e513407fc24...

  • bingbing

    bingbing March 26th, 2011 @ 02:06 AM

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »

<h2 style="font-size: 14px">Tickets have moved to Github</h2>

The new ticket tracker is available at <a href="https://github.com/rails/rails/issues">https://github.com/rails/rails/issues</a>

People watching this ticket

Attachments

Tags

Referenced by

Pages