This project is archived and is in readonly mode.
validates_uniqueness_of unicode aware field length
Reported by Jacob Burkhart | May 15th, 2009 @ 06:27 PM | in 2.3.4
If I have a validates_uniqueness_of column with limit 50 and set it to:
ᏈᏉᏊᏈᏉᏊᏈᏉᏊᏈᏉᏊᏈᏉᏊᏈᏉᏊᏈᏉᏊᏈᏉᏊᏈᏉᏊᏈᏉᏊᏈᏉᏊᏈᏉᏊᏈᏉᏊᏈᏉᏊᏈᏉᏊᏈᏉᏊᏈᏉ
rails incorrectly runs a check for anything existing where that column equals
ᏈᏉᏊᏈᏉᏊᏈᏉᏊᏈᏉᏊᏈᏉᏊᏈ?
The problem is in validates_uniqueness_of string concatenation (implemented in #1042)
http://github.com/rails/rails/blob/b6bac73b282c7e500c43810f2a937fc0...
value = column.limit ? value.to_s[0, column.limit] : value.to_s
should be changed to
value = column.limit ? value.to_s.mb_chars[0, column.limit].to_s : value.to_s
NOTE: also to reproduce you must have set encoding: utf8 in database.yml, and your database table must be "convert to character set utf8"
Comments and changes to this ticket
-
CancelProfileIsBroken August 6th, 2009 @ 02:39 PM
- Tag set to bugmash
-
Elise Huard August 8th, 2009 @ 03:02 PM
there seems to be a problem indeed - the validates_uniqueness select statement doesn't seem to work with utf8
so verified with mysql
I get outright errors with postgresql, while the encoding is UTF8 and so should work.
linked to 2883 i thinkadded tests, and i'll now look for a fix
-
Elise Huard August 8th, 2009 @ 03:03 PM
sorry about the lack of punctuation, there were line breaks in there.
-
Elise Huard August 8th, 2009 @ 04:17 PM
well, i updated the patch: this fix works for postgresql and sqlite3, still not for mysql though.
-
Elise Huard August 8th, 2009 @ 10:43 PM
- no changes were found...
-
Pratik August 8th, 2009 @ 10:48 PM
- Assigned user set to Jeremy Kemper
-
Jeremy Kemper August 9th, 2009 @ 03:27 AM
- Milestone changed from 2.x to 2.3.4
Nice fix, +1. Needs rebase against master.
-
Rizwan Reza August 9th, 2009 @ 03:51 PM
not reproducible
-1 The patch fails on master.
Created an event whose title, with limit taken into account, is not unique. <false> is not true.
-
Elad Meidar August 9th, 2009 @ 05:18 PM
-1 Patch applies on master but fails test, i was unable to trace the generated SQL in the tests log to see what is really going on. Since mb_chars[x,x] is obviously working, i suspect the DB is the one to blame. (MySQL)
1) Failure: test_validate_uniqueness_with_limit_and_utf8(UniquenessValidationTest) [./test/cases/validations/uniqueness_validation_test.rb:246:in `test_validate_uniqueness_with_limit_and_utf8' ./test/cases/../../lib/active_record/test_case.rb:57:in `with_kcode' ./test/cases/validations/uniqueness_validation_test.rb:241:in `test_validate_uniqueness_with_limit_and_utf8' ./test/cases/../../lib/../../activemodel/lib/../../activesupport/lib/active_support/testing/setup_and_teardown.rb:62:in `__send__' ./test/cases/../../lib/../../activemodel/lib/../../activesupport/lib/active_support/testing/setup_and_teardown.rb:62:in `run']: Created an event whose title, with limit taken into account, is not unique. <false> is not true.
-
Elise Huard August 9th, 2009 @ 08:10 PM
exactly - it works for postgres and sqlite3, but not for mysql.
-
Jeremy Kemper August 9th, 2009 @ 08:55 PM
Works for mysql if you regenerate your test dbs. Drop both dbs and recreate with
rake mysql:build_databases
. This corrects the default charset and collation. -
Repository August 9th, 2009 @ 08:55 PM
- State changed from new to committed
(from [c5896bfd8432f6b7a1c6cb06486c5c85eafe9450]) validate uniqueness with limit in utf8
[#2653 state:committed]
Signed-off-by: Jeremy Kemper jeremy@bitsweat.net
http://github.com/rails/rails/commit/c5896bfd8432f6b7a1c6cb06486c5c... -
Repository August 9th, 2009 @ 08:55 PM
(from [f6f04f1549cea33385323b6a1fd2c58855927d15]) validate uniqueness with limit in utf8
[#2653 state:committed]
Signed-off-by: Jeremy Kemper jeremy@bitsweat.net
http://github.com/rails/rails/commit/f6f04f1549cea33385323b6a1fd2c5... -
Jeremy Kemper August 9th, 2009 @ 08:55 PM
- Tag cleared.
-
Elise Huard August 9th, 2009 @ 09:08 PM
I found what makes the test pass (and is also the condition for it to work for mysql: explicit utf8 encoding. I copied the little hack from another test.
-
Jeremy Kemper August 9th, 2009 @ 10:32 PM
Elise, tests already pass for me on mysql. Why is this patch needed?
-
Jeremy Kemper August 9th, 2009 @ 10:34 PM
(first do
rake mysql:rebuild_databases
to fix the old test database's charset and collation) -
Elise Huard August 9th, 2009 @ 10:39 PM
Jeremy, i think your DB is properly configured (for utf8 encoding) - as you can see from Elad's comments (and mine), the mysql still failed, as opposed to the other db's.
this patch checks for that, by doing a query and seeing if the select works properly. I use the same extra bit of code that is used in the test_validate_case_insensitive_uniqueness.
It's a bit of a hack, but i tried different things, like removing the 'BINARY' = connection.case_sensitive_equality_operator for is_utf8? , but that resulted in collation errors in the mysql database, which seems to have LATIN_C1 encoding.
-
Elise Huard August 9th, 2009 @ 10:45 PM
aha, sorry, missed your comment. Ignore the patch then, sorry.
-
Jeremy Kemper August 10th, 2009 @ 05:48 AM
Elise, this would be a useful bit of code for test/connections/native_mysql/connection.rb so we can fail fast and advise a database rebuild instead of leaving Rails testers with odd, unexplained test failures.
Create your profile
Help contribute to this project by taking a few moments to create your personal profile. Create your profile »
<h2 style="font-size: 14px">Tickets have moved to Github</h2>
The new ticket tracker is available at <a href="https://github.com/rails/rails/issues">https://github.com/rails/rails/issues</a>
People watching this ticket
Attachments
Referenced by
- 2653 validates_uniqueness_of unicode aware field length [#2653 state:committed]
- 2653 validates_uniqueness_of unicode aware field length [#2653 state:committed]
- 1042 validates_uniqueness_of does not take field length into account This needs to be fixed to support unicode, see #2653