This project is archived and is in readonly mode.

from ASCII-8BIT to UTF-8 error with MySQL seed from CSV file

#5025

There seems to be a problem with UTF8 strings loaded from a file and inserted with ActiveRecord.
This is only a problem with MySQL, with SQLite it works fine.

Create a database in mysql:

CREATE DATABASE example CHARACTER SET 'utf8';
GRANT ALL PRIVILEGES ON example.* TO 'example'@'localhost' IDENTIFIED BY 'example';
FLUSH PRIVILEGES;

Then extract the attached app, migrate and seed:

rake db:migrate && rake db:seed

You'll see the error:
"\xC3" from ASCII-8BIT to UTF-8

Using rails3-beta4 and ruby-1.9.2.

Thanks. :)

example-utf8.zip 94.9 KB

Reported by humancopy · July 1st, 2010 @ 10:41 AM

Activity

Rohit Arondekar
- Tag changed from rails3, ruby-1.9.2, utf8 to mysql, rails3, ruby-1.9.2, utf8
- State changed from new to open
- Importance changed from to Low
If you can add unicode data into the Database via the MySQL console then I think this is a problem with the mysql gem and not Rails. Can you try opening a similar issue here http://rubyforge.org/tracker/?group_id=4550 ?

In the mean time can somebody confirm if this is indeed a Rails issue or not?

July 1st, 2010 @ 01:16 PM

humancopy

I can confirm that without rails it works. :)
For example, this works fine:

require 'mysql'

c = Mysql.init
# host, username, password, database
c.connect(nil, "example", "example", "example")
insert_new_country = c.prepare "INSERT INTO `countries` (name, code2, code3) VALUES (?, ? ,?)"
  
open('countries.csv', 'r:UTF-8') do |countries|
  countries.read.each_line do |country|
    code2, code3, name = country.chomp.split("|")
    insert_new_country.execute name, code2, code3
  end
end

c.close

July 1st, 2010 @ 01:57 PM

José Valim
Your example works fine outside Rails because your code is not iterating with any other IO or String. This is a bug in MySQL gem which is not marking the encoding of the strings.

July 1st, 2010 @ 02:07 PM
Rohit Arondekar
- Assigned user set to Yehuda Katz (wycats)
July 1st, 2010 @ 02:16 PM
Santiago Pastorino
- Tag changed from mysql, rails3, ruby-1.9.2, utf8 to mysql, rails3, ruby-192, utf8
This issue has been automatically marked as stale because it has not been commented on for at least three months.

The resources of the Rails core team are limited, and so we are asking for your help. If you can still reproduce this error on the 3-0-stable branch or on master, please reply with all of the information you have about it and add "[state:open]" to your comment. This will reopen the ticket for review. Likewise, if you feel that this is a very important feature for Rails to include, please reply with your explanation so we can consider it.

Thank you for all your contributions, and we hope you will understand this step to focus our efforts where they are most helpful.

February 2nd, 2011 @ 05:01 PM
Santiago Pastorino
- State changed from open to stale
February 2nd, 2011 @ 05:01 PM