Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

This project is archived and is in readonly mode.

ASCII-8BIT and UTF-8 in hell

#4683

MySQL drive does not recognize nor enforce UTF-8 encoding for Ruby 1.9. There is a problem with MySQL text and varchar fields for Rails3 and Ruby 1.9. They are not correctly encoded. Instead of UTF-8 they come as ASCII-8BIT, binary data, and when it is mixed with other UTF-8 text, Rails will raise an exception.

field = MyModel.first.field

field.encoding # => ASCII-8BIT

currently it has to dirty fix:

field = MyModel.first.field.force_encoding('UTF-8').encoding # => UTF-8

The patch enforce UTF-8 encoding for Active Record and MySQL.

Reported by Jaroslaw Zabiello · May 24th, 2010 @ 06:26 PM

State: duplicate
Milestone: none
Assigned to: nobody
Importance: Low

Activity

  1. Jaroslaw Zabiello
    Jaroslaw Zabiello

    The previous patch file was wrong.

    May 24th, 2010 @ 06:45 PM

  2. Rafał Lisowski
    Rafał Lisowski
    1. I use it in my rails3 application and it works! The change is quite small and easy to understand. I like it.

    May 31st, 2010 @ 10:03 AM

  3. Simon Hafner
    Simon Hafner

    If you do an eval, put the RUBY_VERSION =~ /1.9/ somewhere outside the eval'd string.

    June 1st, 2010 @ 12:48 PM

  4. Jaroslaw Zabiello
    Jaroslaw Zabiello
    • Tag changed from rails3 encoding ruby1.9 mysql to rails3 encoding ruby1.9 mysql, error

    Simon: my patch is no more needed since that code has been completely rewritten in 3.0b4. But he problem with Ruby 1.9 and ASCII-8BIT, vs UTF-8 still exists and it breaks the application making Rails 3 useless for using with Ruby 1.9, MySQL and non-ASCII characters. You can replicate the error in the following way:

    $ rails new tescior -d mysql
$ cd tescior
$ bundle install
$ rails g model Message msg:string
$ rake db:migrate

    Small change in ~/.rvm/gems/ruby-1.9.2-preview3/gems/activesupport-3.0.0.beta4/lib/active_support/buffered_logger.rb file:

    def flush
  @guard.synchronize do
    unless buffer.empty?
      old_buffer = buffer
      p old_buffer # MY CHANGE
      @log.write(old_buffer.join)
    end
    # Important to do this even if buffer was empty or else @buffer will
    # accumulate empty arrays for each request where nothing was logged.
    clear_buffer
  end
end

    Now, let's start console:

    $ rails c
ruby-1.9.2-preview3 > msg =  "za\xc5\xbc\xc3\xb3\xc5\x82\xc4\x87 g\xc4\x99\xc5\x9bl\xc4\x85 ja\xc5\xba\xc5\x84"
=> "zażółć gęślą jaźń"
ruby-1.9.2-preview3 > Message.create :msg => msg

["  \e[1m\e[36mSQL (0.3ms)\e[0m  \e[1mINSERT INTO `messages` (`created_at`, `msg`, `updated_at`) VALUES ('2010-06-14 19:40:40', 'za\xC5\xBC\xC3\xB3\xC5\x82\xC4\x87 g\xC4\x99\xC5\x9Bl\xC4\x85 ja\xC5\xBA\xC5\x84', '2010-06-14 19:40:40')\e[0m\n", "Could not log \"sql.active_record\" event. Encoding::UndefinedConversionError: \"\\xC5\" from ASCII-8BIT to UTF-8\n", "Encoding::UndefinedConversionError: \"\\xC5\" from ASCII-8BIT to UTF-8: INSERT INTO `messages` (`created_at`, `msg`, `updated_at`) VALUES ('2010-06-14 19:40:40', 'za\xC5\xBC\xC3\xB3\xC5\x82\xC4\x87 g\xC4\x99\xC5\x9Bl\xC4\x85 ja\xC5\xBA\xC5\x84', '2010-06-14 19:40:40')\n", "  \e[1m\e[35mSQL (1.5ms)\e[0m  ROLLBACK\n", "Could not log \"sql.active_record\" event. Encoding::UndefinedConversionError: \"\\xC5\" from ASCII-8BIT to UTF-8\n", "Encoding::UndefinedConversionError: \"\\xC5\" from ASCII-8BIT to UTF-8: ROLLBACK\n"]

Encoding::UndefinedConversionError: "\xC5" from ASCII-8BIT to UTF-8
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activesupport-3.0.0.beta4/lib/active_support/buffered_logger.rb:106:in `write'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activesupport-3.0.0.beta4/lib/active_support/buffered_logger.rb:106:in `block in flush'
  from <internal:prelude>:10:in `synchronize'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activesupport-3.0.0.beta4/lib/active_support/buffered_logger.rb:101:in `flush'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activesupport-3.0.0.beta4/lib/active_support/buffered_logger.rb:123:in `auto_flush'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activesupport-3.0.0.beta4/lib/active_support/buffered_logger.rb:66:in `add'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activesupport-3.0.0.beta4/lib/active_support/buffered_logger.rb:77:in `debug'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/connection_adapters/abstract_adapter.rb:209:in `rescue in log'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/connection_adapters/abstract_adapter.rb:200:in `log'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/connection_adapters/mysql_adapter.rb:286:in `execute'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/connection_adapters/abstract/database_statements.rb:263:in `insert_sql'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/connection_adapters/mysql_adapter.rb:297:in `insert_sql'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/connection_adapters/abstract/database_statements.rb:44:in `insert'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/connection_adapters/abstract/query_cache.rb:16:in `insert'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/arel-0.4.0/lib/arel/engines/sql/engine.rb:36:in `create'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/arel-0.4.0/lib/arel/algebra/relations/writes.rb:20:in `call'
... 14 levels...
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/attribute_methods/dirty.rb:21:in `save'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/transactions.rb:235:in `block (2 levels) in save'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/transactions.rb:287:in `block in with_transaction_returning_status'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/connection_adapters/abstract/database_statements.rb:139:in `transaction'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/transactions.rb:202:in `transaction'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/transactions.rb:285:in `with_transaction_returning_status'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/transactions.rb:235:in `block in save'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/transactions.rb:246:in `rollback_active_record_state!'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/transactions.rb:234:in `save'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/base.rb:462:in `create'
  from (irb):3
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/railties-3.0.0.beta4/lib/rails/commands/console.rb:47:in `start'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/railties-3.0.0.beta4/lib/rails/commands/console.rb:8:in `start'
  from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/railties-3.0.0.beta4/lib/rails/commands.rb:23:in `<top (required)>'
  from script/rails:6:in `require'

    June 14th, 2010 @ 09:02 PM

  5. Jaroslaw Zabiello
    Jaroslaw Zabiello

    I found, that mentioned error dissapeared if I logger level was higher than 0.

    $ rails c
ruby-1.9.2-preview3 > Rails.logger.level
 => 0 
ruby-1.9.2-preview3 > Rails.logger.level = Logger::INFO
 => 1 
ruby-1.9.2-preview3 > Message.create :msg => 'zażółć gęślą jaźń'
 => #<Message id: 11, msg: "zażółć gęślą jaźń", created_at: "2010-06-14 21:54:31", updated_at: "2010-06-14 21:54:31"> 
ruby-1.9.2-preview3 > Rails.logger.level = Logger::DEBUG
 => 0 
ruby-1.9.2-preview3 > Message.create :msg => 'zażółć gęślą jaźń'
Encoding::UndefinedConversionError: "\xC4" from ASCII-8BIT to UTF-8
    from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activesupport-3.0.0.beta4/lib/active_support/buffered_logger.rb:104:in `write'
...

    June 14th, 2010 @ 11:00 PM

  6. Jaroslaw Zabiello
    Jaroslaw Zabiello
    • Tag changed from rails3 encoding ruby1.9 mysql, error to rails3 encoding ruby1.9 mysql, error, logger

    June 14th, 2010 @ 11:00 PM

  7. José Valim
    José Valim
    • State changed from new to duplicate

    I believe this is a duplicate of #4807 Add magic encoding comment to generated files now. Yehuda is working on a fix.

    June 22nd, 2010 @ 04:28 PM