This project is archived and is in readonly mode.
tidy_bytes fails on 1.9.x
Reported by Norman Clarke | April 8th, 2010 @ 09:26 PM
ActiveSupport::Multibyte::Chars#tidy_bytes does not work on 1.9.x, because its implementation uses a Unicode regular expression that, in 1.9.x, can only operate on valid UTF-8 strings, and therefore always raises an error.
There's already a test named
test_tidy_bytes_is_broken_on_1_9_0 in ActiveSupport,
which shows that this is a known issue.
After opening this ticket I'll attach a patch which resolves the issue, and also doubles the performance of this method. Additionally, I've added more test cases, a few of which fail on the current master branch and are fixed by this patch.
The patch also adds a force option to
tidy_bytes, because some sequences of ISO-8859-1 or
CP-1252 characters form a single valid UTF-8 character, and end up
transformed to a single unprintable character without this
option.
If you want to see this code in isolation, I've also packaged this is as a separate library on Github.
Comments and changes to this ticket
-
Norman Clarke April 8th, 2010 @ 09:30 PM
- Tag changed from “active_support, multibyte” to “active_support, multibyte, patch”
Here is the patch.
-
Norman Clarke April 8th, 2010 @ 09:31 PM
Blech. Included the wrong ticket number in the commit message; new patch attached.
-
Norman Clarke April 9th, 2010 @ 12:13 AM
Eh, really sorry for the multiple patches. I just noticed I left out a small change I had intended to add before submitting this. This should be the last one. :)
-
Repository April 9th, 2010 @ 06:31 PM
- State changed from “new” to “resolved”
(from [e416f1d0ab71e3e720e147e5d0e7f6e8b36516a5]) Make tidy_bytes work on 1.9 and improve its performance. [#4350 state:resolved]
Signed-off-by: Jeremy Kemper jeremy@bitsweat.net
http://github.com/rails/rails/commit/e416f1d0ab71e3e720e147e5d0e7f6... -
Create your profile
Help contribute to this project by taking a few moments to create your personal profile. Create your profile »
<h2 style="font-size: 14px">Tickets have moved to Github</h2>
The new ticket tracker is available at <a href="https://github.com/rails/rails/issues">https://github.com/rails/rails/issues</a>
People watching this ticket
Norman Clarke
Attachments
Tags
Referenced by
-
4350
tidy_bytes fails on 1.9.x
(from [e416f1d0ab71e3e720e147e5d0e7f6e8b36516a5])
Make ti...