rubygems loaded twice

Symptoms are this on STDERR from script/rails c

/path/to/ruby/site_ruby/1.9.1/rubygems.rb:14: warning: already initialized constant VERSION
/path/to/ruby/site_ruby/1.9.1/rubygems.rb:14: warning: already initialized constant RubyGemsVersion
/path/to/ruby/site_ruby/1.9.1/rubygems.rb:194: warning: already initialized constant MUTEX
/path/to/ruby/site_ruby/1.9.1/rubygems.rb:196: warning: already initialized constant RubyGemsPackageVersion
/path/to/ruby/site_ruby/1.9.1/rubygems.rb:202: warning: already initialized constant WIN_PATTERNS
/path/to/ruby/site_ruby/1.9.1/rubygems.rb:1079: warning: already initialized constant MARSHAL_SPEC_DIR
/path/to/ruby/site_ruby/1.9.1/rubygems.rb:1084: warning: already initialized constant YAML_SPEC_DIR
/path/to/ruby/site_ruby/1.9.1/rubygems/version.rb:72: warning: already initialized constant VERSION_PATTERN
/path/to/ruby/site_ruby/1.9.1/rubygems/requirement.rb:20: warning: already initialized constant OPS
/path/to/ruby/site_ruby/1.9.1/rubygems/requirement.rb:30: warning: already initialized constant OP_RE
/path/to/ruby/site_ruby/1.9.1/rubygems/version.rb:246: warning: already initialized constant Requirement
/path/to/ruby/site_ruby/1.9.1/rubygems/dependency.rb:18: warning: already initialized constant TYPES
/path/to/ruby/site_ruby/1.9.1/rubygems/platform.rb:171: warning: already initialized constant RUBY
/path/to/ruby/site_ruby/1.9.1/rubygems/platform.rb:177: warning: already initialized constant CURRENT
/path/to/ruby/site_ruby/1.9.1/rubygems/specification.rb:39: warning: already initialized constant NONEXISTENT_SPECIFICATION_VERSION
/path/to/ruby/site_ruby/1.9.1/rubygems/specification.rb:50: warning: already initialized constant CURRENT_SPECIFICATION_VERSION
/path/to/ruby/site_ruby/1.9.1/rubygems/specification.rb:56: warning: already initialized constant SPECIFICATION_VERSION_HISTORY
/path/to/ruby/site_ruby/1.9.1/rubygems/specification.rb:72: warning: already initialized constant MARSHAL_FIELDS
/path/to/ruby/site_ruby/1.9.1/rubygems/specification.rb:75: warning: already initialized constant TODAY
/path/to/ruby/site_ruby/1.9.1/rubygems/source_index.rb:593: warning: already initialized constant Cache
/path/to/ruby/site_ruby/1.9.1/rubygems/config_file.rb:14: warning: already initialized constant DEFAULT_BACKTRACE
/path/to/ruby/site_ruby/1.9.1/rubygems/config_file.rb:15: warning: already initialized constant DEFAULT_BENCHMARK
/path/to/ruby/site_ruby/1.9.1/rubygems/config_file.rb:16: warning: already initialized constant DEFAULT_BULK_THRESHOLD
/path/to/ruby/site_ruby/1.9.1/rubygems/config_file.rb:17: warning: already initialized constant DEFAULT_VERBOSITY
/path/to/ruby/site_ruby/1.9.1/rubygems/config_file.rb:18: warning: already initialized constant DEFAULT_UPDATE_SOURCES
/path/to/ruby/site_ruby/1.9.1/rubygems/config_file.rb:24: warning: already initialized constant OPERATING_SYSTEM_DEFAULTS
/path/to/ruby/site_ruby/1.9.1/rubygems/config_file.rb:30: warning: already initialized constant PLATFORM_DEFAULTS
/path/to/ruby/site_ruby/1.9.1/rubygems/config_file.rb:53: warning: already initialized constant SYSTEM_WIDE_CONFIG_FILE

This is caused by Runtime#crippple_rubygems which removes the /path/to/ruby/gems path to rubygems.rb from $LOAD_PATH but not the /path/to/ruby/site_ruby path. I'm not really sure what dark magic is happening here, so am not really keen on guessing at a patch.