This project is archived and is in readonly mode.

#4683 ✓duplicate
Jaroslaw Zabiello

ASCII-8BIT and UTF-8 in hell

Reported by Jaroslaw Zabiello | May 24th, 2010 @ 06:26 PM

MySQL drive does not recognize nor enforce UTF-8 encoding for Ruby 1.9. There is a problem with MySQL text and varchar fields for Rails3 and Ruby 1.9. They are not correctly encoded. Instead of UTF-8 they come as ASCII-8BIT, binary data, and when it is mixed with other UTF-8 text, Rails will raise an exception.

field = MyModel.first.field

field.encoding # => ASCII-8BIT

currently it has to dirty fix:

field = MyModel.first.field.force_encoding('UTF-8').encoding # => UTF-8

The patch enforce UTF-8 encoding for Active Record and MySQL.

Comments and changes to this ticket

  • Jaroslaw Zabiello
  • Rafał Lisowski

    Rafał Lisowski May 31st, 2010 @ 10:03 AM

    1. I use it in my rails3 application and it works! The change is quite small and easy to understand. I like it.
  • Simon Hafner

    Simon Hafner June 1st, 2010 @ 12:48 PM

    If you do an eval, put the RUBY_VERSION =~ /1.9/ somewhere outside the eval'd string.

  • Jaroslaw Zabiello

    Jaroslaw Zabiello June 14th, 2010 @ 09:02 PM

    • Tag changed from rails3 encoding ruby1.9 mysql to rails3 encoding ruby1.9 mysql, error

    Simon: my patch is no more needed since that code has been completely rewritten in 3.0b4. But he problem with Ruby 1.9 and ASCII-8BIT, vs UTF-8 still exists and it breaks the application making Rails 3 useless for using with Ruby 1.9, MySQL and non-ASCII characters. You can replicate the error in the following way:

    $ rails new tescior -d mysql
    $ cd tescior
    $ bundle install
    $ rails g model Message msg:string
    $ rake db:migrate
    

    Small change in ~/.rvm/gems/ruby-1.9.2-preview3/gems/activesupport-3.0.0.beta4/lib/active_support/buffered_logger.rb file:

    def flush
      @guard.synchronize do
        unless buffer.empty?
          old_buffer = buffer
          p old_buffer # MY CHANGE
          @log.write(old_buffer.join)
        end
        # Important to do this even if buffer was empty or else @buffer will
        # accumulate empty arrays for each request where nothing was logged.
        clear_buffer
      end
    end
    

    Now, let's start console:

    $ rails c
    ruby-1.9.2-preview3 > msg =  "za\xc5\xbc\xc3\xb3\xc5\x82\xc4\x87 g\xc4\x99\xc5\x9bl\xc4\x85 ja\xc5\xba\xc5\x84"
    => "zażółć gęślą jaźń"
    ruby-1.9.2-preview3 > Message.create :msg => msg
    
    ["  \e[1m\e[36mSQL (0.3ms)\e[0m  \e[1mINSERT INTO `messages` (`created_at`, `msg`, `updated_at`) VALUES ('2010-06-14 19:40:40', 'za\xC5\xBC\xC3\xB3\xC5\x82\xC4\x87 g\xC4\x99\xC5\x9Bl\xC4\x85 ja\xC5\xBA\xC5\x84', '2010-06-14 19:40:40')\e[0m\n", "Could not log \"sql.active_record\" event. Encoding::UndefinedConversionError: \"\\xC5\" from ASCII-8BIT to UTF-8\n", "Encoding::UndefinedConversionError: \"\\xC5\" from ASCII-8BIT to UTF-8: INSERT INTO `messages` (`created_at`, `msg`, `updated_at`) VALUES ('2010-06-14 19:40:40', 'za\xC5\xBC\xC3\xB3\xC5\x82\xC4\x87 g\xC4\x99\xC5\x9Bl\xC4\x85 ja\xC5\xBA\xC5\x84', '2010-06-14 19:40:40')\n", "  \e[1m\e[35mSQL (1.5ms)\e[0m  ROLLBACK\n", "Could not log \"sql.active_record\" event. Encoding::UndefinedConversionError: \"\\xC5\" from ASCII-8BIT to UTF-8\n", "Encoding::UndefinedConversionError: \"\\xC5\" from ASCII-8BIT to UTF-8: ROLLBACK\n"]
    
    Encoding::UndefinedConversionError: "\xC5" from ASCII-8BIT to UTF-8
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activesupport-3.0.0.beta4/lib/active_support/buffered_logger.rb:106:in `write'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activesupport-3.0.0.beta4/lib/active_support/buffered_logger.rb:106:in `block in flush'
      from <internal:prelude>:10:in `synchronize'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activesupport-3.0.0.beta4/lib/active_support/buffered_logger.rb:101:in `flush'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activesupport-3.0.0.beta4/lib/active_support/buffered_logger.rb:123:in `auto_flush'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activesupport-3.0.0.beta4/lib/active_support/buffered_logger.rb:66:in `add'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activesupport-3.0.0.beta4/lib/active_support/buffered_logger.rb:77:in `debug'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/connection_adapters/abstract_adapter.rb:209:in `rescue in log'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/connection_adapters/abstract_adapter.rb:200:in `log'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/connection_adapters/mysql_adapter.rb:286:in `execute'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/connection_adapters/abstract/database_statements.rb:263:in `insert_sql'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/connection_adapters/mysql_adapter.rb:297:in `insert_sql'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/connection_adapters/abstract/database_statements.rb:44:in `insert'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/connection_adapters/abstract/query_cache.rb:16:in `insert'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/arel-0.4.0/lib/arel/engines/sql/engine.rb:36:in `create'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/arel-0.4.0/lib/arel/algebra/relations/writes.rb:20:in `call'
    ... 14 levels...
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/attribute_methods/dirty.rb:21:in `save'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/transactions.rb:235:in `block (2 levels) in save'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/transactions.rb:287:in `block in with_transaction_returning_status'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/connection_adapters/abstract/database_statements.rb:139:in `transaction'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/transactions.rb:202:in `transaction'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/transactions.rb:285:in `with_transaction_returning_status'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/transactions.rb:235:in `block in save'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/transactions.rb:246:in `rollback_active_record_state!'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/transactions.rb:234:in `save'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activerecord-3.0.0.beta4/lib/active_record/base.rb:462:in `create'
      from (irb):3
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/railties-3.0.0.beta4/lib/rails/commands/console.rb:47:in `start'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/railties-3.0.0.beta4/lib/rails/commands/console.rb:8:in `start'
      from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/railties-3.0.0.beta4/lib/rails/commands.rb:23:in `<top (required)>'
      from script/rails:6:in `require'
    
  • Jaroslaw Zabiello

    Jaroslaw Zabiello June 14th, 2010 @ 11:00 PM

    I found, that mentioned error dissapeared if I logger level was higher than 0.

    $ rails c
    ruby-1.9.2-preview3 > Rails.logger.level
     => 0 
    ruby-1.9.2-preview3 > Rails.logger.level = Logger::INFO
     => 1 
    ruby-1.9.2-preview3 > Message.create :msg => 'zażółć gęślą jaźń'
     => #<Message id: 11, msg: "zażółć gęślą jaźń", created_at: "2010-06-14 21:54:31", updated_at: "2010-06-14 21:54:31"> 
    ruby-1.9.2-preview3 > Rails.logger.level = Logger::DEBUG
     => 0 
    ruby-1.9.2-preview3 > Message.create :msg => 'zażółć gęślą jaźń'
    Encoding::UndefinedConversionError: "\xC4" from ASCII-8BIT to UTF-8
        from /Users/zbiru/.rvm/gems/ruby-1.9.2-preview3/gems/activesupport-3.0.0.beta4/lib/active_support/buffered_logger.rb:104:in `write'
    ...
    
  • Jaroslaw Zabiello

    Jaroslaw Zabiello June 14th, 2010 @ 11:00 PM

    • Tag changed from rails3 encoding ruby1.9 mysql, error to rails3 encoding ruby1.9 mysql, error, logger
  • José Valim

    José Valim June 22nd, 2010 @ 04:28 PM

    • State changed from “new” to “duplicate”

    I believe this is a duplicate of #4807 now. Yehuda is working on a fix.

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »

<h2 style="font-size: 14px">Tickets have moved to Github</h2>

The new ticket tracker is available at <a href="https://github.com/rails/rails/issues">https://github.com/rails/rails/issues</a>

Referenced by

Pages