This project is archived and is in readonly mode.

#5025 ✓stale
humancopy

from ASCII-8BIT to UTF-8 error with MySQL seed from CSV file

Reported by humancopy | July 1st, 2010 @ 10:41 AM

There seems to be a problem with UTF8 strings loaded from a file and inserted with ActiveRecord.
This is only a problem with MySQL, with SQLite it works fine.

Create a database in mysql:

CREATE DATABASE example CHARACTER SET 'utf8';
GRANT ALL PRIVILEGES ON example.* TO 'example'@'localhost' IDENTIFIED BY 'example';
FLUSH PRIVILEGES;

Then extract the attached app, migrate and seed:

rake db:migrate && rake db:seed

You'll see the error:
"\xC3" from ASCII-8BIT to UTF-8

Using rails3-beta4 and ruby-1.9.2.

Thanks. :)

Comments and changes to this ticket

  • Rohit Arondekar

    Rohit Arondekar July 1st, 2010 @ 01:16 PM

    • Tag changed from rails3, ruby-1.9.2, utf8 to mysql, rails3, ruby-1.9.2, utf8
    • State changed from “new” to “open”
    • Importance changed from “” to “Low”

    If you can add unicode data into the Database via the MySQL console then I think this is a problem with the mysql gem and not Rails. Can you try opening a similar issue here http://rubyforge.org/tracker/?group_id=4550 ?

    In the mean time can somebody confirm if this is indeed a Rails issue or not?

  • humancopy

    humancopy July 1st, 2010 @ 01:57 PM

    I can confirm that without rails it works. :)
    For example, this works fine:

    require 'mysql'
    
    c = Mysql.init
    # host, username, password, database
    c.connect(nil, "example", "example", "example")
    insert_new_country = c.prepare "INSERT INTO `countries` (name, code2, code3) VALUES (?, ? ,?)"
      
    open('countries.csv', 'r:UTF-8') do |countries|
      countries.read.each_line do |country|
        code2, code3, name = country.chomp.split("|")
        insert_new_country.execute name, code2, code3
      end
    end
    
    c.close
    
  • José Valim

    José Valim July 1st, 2010 @ 02:07 PM

    Your example works fine outside Rails because your code is not iterating with any other IO or String. This is a bug in MySQL gem which is not marking the encoding of the strings.

  • Rohit Arondekar

    Rohit Arondekar July 1st, 2010 @ 02:16 PM

    • Assigned user set to “Yehuda Katz (wycats)”
  • Santiago Pastorino

    Santiago Pastorino February 2nd, 2011 @ 05:01 PM

    • Tag changed from mysql, rails3, ruby-1.9.2, utf8 to mysql, rails3, ruby-192, utf8

    This issue has been automatically marked as stale because it has not been commented on for at least three months.

    The resources of the Rails core team are limited, and so we are asking for your help. If you can still reproduce this error on the 3-0-stable branch or on master, please reply with all of the information you have about it and add "[state:open]" to your comment. This will reopen the ticket for review. Likewise, if you feel that this is a very important feature for Rails to include, please reply with your explanation so we can consider it.

    Thank you for all your contributions, and we hope you will understand this step to focus our efforts where they are most helpful.

  • Santiago Pastorino

    Santiago Pastorino February 2nd, 2011 @ 05:01 PM

    • State changed from “open” to “stale”

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »

<h2 style="font-size: 14px">Tickets have moved to Github</h2>

The new ticket tracker is available at <a href="https://github.com/rails/rails/issues">https://github.com/rails/rails/issues</a>

Attachments

Pages