Multiple encoding support for Erubis

Haml, etc

Haml doesn't handle magic comments, but Rails could do it instead(the #encoding type). This could apply to other templating engines and would be consistent with the way Erb works.

Status

I'll be finishing off with test cases for Haml and I would like to include them in the Rails tests (if Haml is installed). This is mainly for coverage in the AV:template.rb encoding handling and to make sure line numbering and related things work correctly.

Patch

Basically a port of #2188 Encoding error in Ruby1.9 for templates to Rails 3, but I want to make sure I didn't break Haml or other templating engines and provide test cases to prove it. And suggestions for #2188 Encoding error in Ruby1.9 for templates will also apply here.

Comments

This is basically implementing iconv functionality for templates in Rails and personally I did consider dropping magic tag support completely and showing an error with an iconv suggestion, if the encoding doesn't match default_internal.

Yehuda convinced me that this functionality is valuable enough to keep.

Phew ..

Patch done, rebased against master.

Things changed:

• line numbering should now work correctly (after a tiny Erubis patch: http://github.com/e2/erubis/commit/5fe702df9aa2b4acc4547d295d108b07...)
• magic comments are handled for all templates and are less strict (should work for Haml style comments, etc)
• resolver opens files in binary mode to prevent crashing when encoding doesn't match system
• template contents are validated
• fixed wrong trim parameter for Erubis and Erb (related to #2553 ActionView::TemplateHandlers::ERB.erb_trim_mode= broken ?)
• replaced utf-8 tests with more strict sjis and euc tests (and makes use of Hyōgaiji character to make euc->sjis cause errors)
• refactoring in render tests
• did some testing with Haml

This patch changes quite a bit, so I wouldn't be too quick about backporting it to 2-3-stable.

Yehuda did a lot of work on this a long time ago and if there aren't any issues in Rails3, this can be closed.

I'll check this soon and close it. If anyone has issues, feel free to reopen it.