Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

This project is archived and is in readonly mode.

tidy_bytes fails on 1.9.x

#4350

ActiveSupport::Multibyte::Chars#tidy_bytes does not work on 1.9.x, because its implementation uses a Unicode regular expression that, in 1.9.x, can only operate on valid UTF-8 strings, and therefore always raises an error.

There's already a test named test_tidy_bytes_is_broken_on_1_9_0 in ActiveSupport, which shows that this is a known issue.

After opening this ticket I'll attach a patch which resolves the issue, and also doubles the performance of this method. Additionally, I've added more test cases, a few of which fail on the current master branch and are fixed by this patch.

The patch also adds a force option to tidy_bytes, because some sequences of ISO-8859-1 or CP-1252 characters form a single valid UTF-8 character, and end up transformed to a single unprintable character without this option.

If you want to see this code in isolation, I've also packaged this is as a separate library on Github.

Reported by Norman Clarke · April 8th, 2010 @ 09:26 PM

State: resolved
Milestone: none
Assigned to: nobody
Importance: none

Activity

  1. Norman Clarke
    Norman Clarke
    • Tag changed from active_support, multibyte to active_support, multibyte, patch

    Here is the patch.

    April 8th, 2010 @ 09:30 PM

  2. Norman Clarke
    Norman Clarke

    Blech. Included the wrong ticket number in the commit message; new patch attached.

    April 8th, 2010 @ 09:31 PM

  3. Norman Clarke
    Norman Clarke

    Eh, really sorry for the multiple patches. I just noticed I left out a small change I had intended to add before submitting this. This should be the last one. :)

    April 9th, 2010 @ 12:13 AM

  4. Repository
    Repository
    • State changed from new to resolved

    (from [e416f1d0ab71e3e720e147e5d0e7f6e8b36516a5]) Make tidy_bytes work on 1.9 and improve its performance. [#4350 state:resolved]

    Signed-off-by: Jeremy Kemper jeremy@bitsweat.net
    http://github.com/rails/rails/commit/e416f1d0ab71e3e720e147e5d0e7f6...

    April 9th, 2010 @ 06:31 PM