This project is archived and is in readonly mode.

#3479 ✓resolved

ActiveSupport::JSON fails to decode unicode-escaped newline and literal newlines

Reported by Ethan | November 10th, 2009 @ 11:08 PM | in 2.3.10

With the YAML backend, ActiveSupport::JSON decodes a unicode-escaped newline (or CRLF) as a space. It does the same with a literal newline. But, it deals correctly with a \n-escaped newline.

# unicode-escaped LF
>> ActiveSupport::JSON.decode("\"\\u000a\"")
=> " "
# unicode-escaped CRLF
>> ActiveSupport::JSON.decode("\"\\u000d\\u000a\"")
=> " "
# literal LF
>> ActiveSupport::JSON.decode("\"\n\"")
=> " "
# normal backslash-escaped LF
>> ActiveSupport::JSON.decode("\"\\n\"")
=> "\n"

The JSON gem does this correctly:

>> JSON.parse "[\"\\u000a\"]"
=> ["\n"]

(note that it's in an array because the JSON gem doesn't seem to like decoding just strings. but that's not at issue here; the YAML backend fails when it's in an array too)

This is because the method convert_json_to_yaml changes the escaped newline to a literal newline:

>> ActiveSupport::JSON::Backends::Yaml.send(:convert_json_to_yaml, "[\"\\u000a\"]")
=> "[\"\n\"]"

and then calls YAML.load("["\n"]") which changes the newline to a space:

>> YAML.load "[\"\n\"]"
=> [" "]

I'm not sure if that's correct behavior for YAML or not. But, the result is certainly incorrect for JSON.
A workaround could be to escape newlines in strings before passing them to YAML, after converting from unicode-escaped newline to literal newline. that is, change "\n" to "\n"; change "\r\n" to "\r\n", as YAML correctly converts those to literal newlines.

Comments and changes to this ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »

<h2 style="font-size: 14px">Tickets have moved to Github</h2>

The new ticket tracker is available at <a href=""></a>

Shared Ticket Bins