This project is archived and is in readonly mode.
Multiple encoding support for Erubis
Reported by Cezary Baginski | May 12th, 2010 @ 01:04 PM | in 3.0.2
This just #2188 for Rails 3.0
I opened a new ticket because:
- Ruby 1.9 encoding support in Rails is still an ongoing issue
- Rails 3 uses Erubis, which might cause slight differences (like line numbers in errors)
- final solution may need some discussion
- the previous fix was only tested for Erb
I assume the solution should:
- support templates with multiple encodings
- support templates without magic comments as long as they match Encoding.default_internal
- make encoding errors appear as soon as possible (before template compilation if possible)
- set the final encoding to effectively be: (Encoding.default_internal || Encoding.default_external)
- not try to support multibyte and ascii incompatible encodings
- should not attempt any conversion in binary mode
- should validate if the encoding magic matches the template
- should avoid unnecessary calls to .encode and .dup if possible
Problems / ideas:
- test cases for other templating engines could be provided by authors and added to Rails
- Rails could generally disallow using multiple encodings for templates and raise errors if they don't match the current encoding
- line numbers may need more test cases because of the way newlines are handled
Comments and changes to this ticket
-
Jeremy Kemper May 12th, 2010 @ 05:57 PM
- Milestone cleared.
- State changed from new to open
-
Cezary Baginski May 12th, 2010 @ 10:34 PM
Haml, etc
Haml doesn't handle magic comments, but Rails could do it instead(the #encoding type). This could apply to other templating engines and would be consistent with the way Erb works.
Status
I'll be finishing off with test cases for Haml and I would like to include them in the Rails tests (if Haml is installed). This is mainly for coverage in the AV:template.rb encoding handling and to make sure line numbering and related things work correctly.
Patch
Basically a port of #2188 to Rails 3, but I want to make sure I didn't break Haml or other templating engines and provide test cases to prove it. And suggestions for #2188 will also apply here.
Comments
This is basically implementing iconv functionality for templates in Rails and personally I did consider dropping magic tag support completely and showing an error with an iconv suggestion, if the encoding doesn't match default_internal.
Yehuda convinced me that this functionality is valuable enough to keep.
-
Cezary Baginski May 12th, 2010 @ 10:35 PM
- Tag changed from encoding, erubis, templates, utf8 to encoding, erubis, haml, templates, utf8
-
Cezary Baginski May 15th, 2010 @ 07:28 PM
Phew ..
Patch done, rebased against master.
Things changed:
- line numbering should now work correctly (after a tiny Erubis patch: http://github.com/e2/erubis/commit/5fe702df9aa2b4acc4547d295d108b07...)
- magic comments are handled for all templates and are less strict (should work for Haml style comments, etc)
- resolver opens files in binary mode to prevent crashing when encoding doesn't match system
- template contents are validated
- fixed wrong trim parameter for Erubis and Erb (related to #2553 ?)
- replaced utf-8 tests with more strict sjis and euc tests (and makes use of Hyōgaiji character to make euc->sjis cause errors)
- refactoring in render tests
- did some testing with Haml
This patch changes quite a bit, so I wouldn't be too quick about backporting it to 2-3-stable.
-
Rohit Arondekar September 13th, 2010 @ 06:51 AM
- Importance changed from to Low
Any updates here?
-
Cezary Baginski September 13th, 2010 @ 09:31 AM
Yehuda did a lot of work on this a long time ago and if there aren't any issues in Rails3, this can be closed.
I'll check this soon and close it. If anyone has issues, feel free to reopen it.
-
David Trasbo September 23rd, 2010 @ 10:01 AM
- State changed from open to resolved
See Cezary's comment.
Create your profile
Help contribute to this project by taking a few moments to create your personal profile. Create your profile »
<h2 style="font-size: 14px">Tickets have moved to Github</h2>
The new ticket tracker is available at <a href="https://github.com/rails/rails/issues">https://github.com/rails/rails/issues</a>
People watching this ticket
Attachments
Referenced by
- 2188 Encoding error in Ruby1.9 for templates New ticket for the Rails 3 version of the patch: #4582