#4582 Multiple encoding support for Erubis - Ruby on Rails

Type	To find
responsible:me	tickets assigned to you
tagged:"@high"	tickets tagged @high
milestone:next	tickets in the upcoming milestone
state:invalid	tickets with the state invalid
created:"last week"	tickets created last week
sort:number, importance, updated	tickets sorted by #, importance or updated
Combine keywords for powerful searching.
Use advanced searching »

This project is archived and is in readonly mode.

#4582 ✓resolved

Multiple encoding support for Erubis

Reported by Cezary Baginski | May 12th, 2010 @ 01:04 PM | in 3.0.2

This just #2188 for Rails 3.0

I opened a new ticket because:

Ruby 1.9 encoding support in Rails is still an ongoing issue
Rails 3 uses Erubis, which might cause slight differences (like line numbers in errors)
final solution may need some discussion
the previous fix was only tested for Erb

I assume the solution should:

support templates with multiple encodings
support templates without magic comments as long as they match Encoding.default_internal
make encoding errors appear as soon as possible (before template compilation if possible)
set the final encoding to effectively be: (Encoding.default_internal || Encoding.default_external)
not try to support multibyte and ascii incompatible encodings
should not attempt any conversion in binary mode
should validate if the encoding magic matches the template
should avoid unnecessary calls to .encode and .dup if possible

Problems / ideas:

test cases for other templating engines could be provided by authors and added to Rails
Rails could generally disallow using multiple encodings for templates and raise errors if they don't match the current encoding
line numbers may need more test cases because of the way newlines are handled

Comments and changes to this ticket

Jeremy Kemper May 12th, 2010 @ 05:57 PM
- Milestone cleared.
- State changed from “new” to “open”
Cezary Baginski May 12th, 2010 @ 10:34 PM
Haml, etc

Haml doesn't handle magic comments, but Rails could do it instead(the #encoding type). This could apply to other templating engines and would be consistent with the way Erb works.

Status

I'll be finishing off with test cases for Haml and I would like to include them in the Rails tests (if Haml is installed). This is mainly for coverage in the AV:template.rb encoding handling and to make sure line numbering and related things work correctly.

Patch

Basically a port of #2188 to Rails 3, but I want to make sure I didn't break Haml or other templating engines and provide test cases to prove it. And suggestions for #2188 will also apply here.

Comments

This is basically implementing iconv functionality for templates in Rails and personally I did consider dropping magic tag support completely and showing an error with an iconv suggestion, if the encoding doesn't match default_internal.

Yehuda convinced me that this functionality is valuable enough to keep.
Cezary Baginski May 12th, 2010 @ 10:35 PM
- Tag changed from “encoding, erubis, templates, utf8” to “encoding, erubis, haml, templates, utf8”
Cezary Baginski May 15th, 2010 @ 07:28 PM
Phew ..

Patch done, rebased against master.

Things changed:
- line numbering should now work correctly (after a tiny Erubis patch: http://github.com/e2/erubis/commit/5fe702df9aa2b4acc4547d295d108b07...)
- magic comments are handled for all templates and are less strict (should work for Haml style comments, etc)
- resolver opens files in binary mode to prevent crashing when encoding doesn't match system
- template contents are validated
- fixed wrong trim parameter for Erubis and Erb (related to #2553 ?)
- replaced utf-8 tests with more strict sjis and euc tests (and makes use of Hyōgaiji character to make euc->sjis cause errors)
- refactoring in render tests
- did some testing with Haml
This patch changes quite a bit, so I wouldn't be too quick about backporting it to 2-3-stable.
- rails3_m17n_render.patch 25.3 KB
Jeremy Kemper August 30th, 2010 @ 04:10 AM
- Milestone cleared.
[bulk edit]
Rohit Arondekar September 13th, 2010 @ 06:51 AM
- Importance changed from “” to “Low”
Any updates here?
Cezary Baginski September 13th, 2010 @ 09:31 AM
Yehuda did a lot of work on this a long time ago and if there aren't any issues in Rails3, this can be closed.

I'll check this soon and close it. If anyone has issues, feel free to reopen it.
David Trasbo September 23rd, 2010 @ 10:01 AM
- State changed from “open” to “resolved”
See Cezary's comment.
Ryan Bigg October 9th, 2010 @ 09:46 PM
- Tag cleared.
Automatic cleanup of spam.
Jeremy Kemper October 15th, 2010 @ 11:01 PM
- Milestone set to “3.0.2”
[bulk edit]
You flagged this item as spam.
klkk May 23rd, 2011 @ 02:54 AM
www.louisvuittonwarehouse.com