Unify approach to gem dependencies

I think the way Rails handles gem dependencies, inside its own environment, is flawed and leads to problems. rake gems:install doesn't work for me because the environment fails to load due to a gem requirement.

@Steven - That shouldn't be happening as of 2.2; which gem was giving you this problem?

Ahh, my mistake. This project is still running 2.1.0. Will have to upgrade; thanks.

Note to self: consider this issue as well.

http://rubyforge.org/tracker/ind...

What about adopting Merb's dependency manager? It runs outside the environment and has no such load-order/dependency/etc issues that seem to plague rails & gem management.

The problem is while trying to load the environment, any code that starts requiring other code (cache_classes, observers, etc), and eventually gems, will always catastrophically fail. The whole problem would be gone if gem management wasn't inside the environment.

We should consider at this point also making the vendor/gems structure a full gem repository. Instead of having Rails do its own management. Just set GEM_PATH, and everything would work.

re: switching to Merb's dependency manager

It's a good idea, but there's a lot of legacy cruft in Rails that Merb doesn't have. For instance:

• vendor/rails: this requires special handling to make gems that depend on Rails load correctly.

• plugins: Merb has the advantage here, as they have always just used gems as plugins. Rails plugins (in vendor/plugins) can add gem dependencies during the load process. I somewhat doubt that the core team will want to break every non-gem plugin in 3.0.

• bin files: Merb drops unpacked gem executables into a bin directory under the project. This lets Merb freeze gems that we currently aren't able to, such as rake (for merb, thor).

The fixes are straightforward for some of these, but will break any hope of backward compatibility.

Sorry for adding so many people, but I'd like to get some feedback from the core team if possible - thanks!

Fundamentally I think the fact that two (previously) seperate projects came up with different solutions to the same problem, kinda indicates that this work could take place inside rubygems itself. This would obviously be more work, and require the coordination of more people, but if our gem freezing and loading code could all just be 'natural rubygems usage' then I think everyone would be better off.

Does anyone have a feel for what kind of solution we could send to the rubygems guys?

Also, yeah, not keen on breaking non gems plugins ;)

In my opinion, the merb code is extremely viable as a stand-alone library that could be partially integration with rubygems. Specifically, the parts of the Merb bundler that deal with multiple private repositories, as well as the installer extensions could be moved in.

In the long run, I'm not sure that the entire solution could be moved into rubygems, but who knows. In the meantime, the first step would be to fully separate the merb code from merb (it's basically done already with the exception of the code that reads in the dependencies in the first place) and make it available to Rails folks as a standalone library.

Right, if rubygems could nicely handle multiple private repositories with fallback to system repositories, then the stuff left in rails could just be the way that config.gems works, and some glue stuff for handling the installation.

If it's a seperate library at first, that's fine too. Just seems like a really natural fit for rubygems inclusion to me ;)

I've gotten most of this working in the past as a Rails plugin, mostly a direct port from merb's dependency system with slight changes.

http://github.com/ddollar/depend...

That being said, I'm a bit more of a fan of configuring the gems through config.gem as then you can take advantage of existing Rails systems (such as putting config.gem lines in environments/test.rb)

The rest of the system from merb I like a lot more. I like that merb removes gems you no longer need, and can keep things 'up to date' as opposed to just littering your app with new versions as upgrades happen. I also like the fact that merb has a bin/ directory to handle the binaries installed by gems.

The counterpoint to using config.gem is definitely that loading the entire rails initializer for gems leads to some serious chicken/egg issues.

It would definitely be helpful, though, if any final solution could help me avoid loading rspec in production :)

The chicken-and-egg config.gem problem is a killer imho.

The canonical way to avoid running rspec in production mode with Merb is to do:

dependency "rspec", :require_as => nil

That definitely works for Rspec because it loads itself in its rake tasks, but I can envision other cases where you want a gem to be loaded in development/test but not in production (shoulda comes to mind as an example, but I'm sure there are others)

  dependency "rspec", :env => %w(test staging)

Would work great. ;)

I added :env support to the above-linked Rails plugin.

I strongly prefer:

dependency "rspec", :only => %w(test staging)
dependency "rspec", :only => :test
dependency "rspec", :except => :production

Modified aforementioned rails plugin to use :only and :except

While I do understand the motivations here, it seems unfortunate that the 'normal' way of restricting some configuration to a particular environment is to use the config/environments/{test,production,development}.rb files, but we have to introduce another mechanism to satisfy this need when working with gems.

I feel like this inconsistency is hinting at the need to be able to treat the environment files in two different ways - sometimes as executable initialization instructions, and other times as very lightweight definition of application configuration.

So, would it makes sense to be able to evaluate 'environment.rb' in the second mode, to gather information about the configuration without actually loading the environment? There's some potential to do this already, hinted at by the first argument to Rails::Initializer#run.

This kind of approach has secondary benefits - it could possibly assist in finding rake tasks from non-vendored gems, without having to load the whole environment.

Any thoughts appreciated.

@James: I'm not a CS guy, but isn't this idea a variation of something turing proved impossible?

My initializers at present do things which rely on bits of the environment being set up already, authenticate with a CC gateway, verify a database is configured for utf-8 correctly etc. Seems that faking this out won't ever work right...

Having a simple dependencies manifest (a la Merb) with the added support for specified the mode(s) to use is significantly simpler than trying to solve the environments problem, and clearer too (imho) for the problem at hand.

@koz - ultimately, I think you're right. If there's momentum behind declaring dependencies in the way that Yehuda et al. are describing, that's great :)

david dollar and I have been working on the (merb inspired?) dependencies gem. I'm using it in a large project and it's been working really well for me. http://github.com/ddollar/depend...

I'd love to see it get some attention from the core team.

Hey Steven,

Are you going to RailsConf by any chance? Few people worked on different solutions and Yehuda has been looking/helping some projects. We are seriously trying to improve the current situation.

• Matt

I think both Steven and I will be at Railsconf, I'd love to talk to some folks about solutions to the gem problem, it's been near to my heart for a while now.

I'd love to meet you guys. Yehuda and Carl will also be there and I think Jacques worked on a new bundler, so we should all get together and try to look at this deps situation.

• Matt

I've been planning to add multiple configurable gem paths to ddollar dependencies plugin. This is needed to keep a clean tree hierachy when a plugin has its own vcs'd gem dependencies. If we are also going to improve the rails-gems code for 3.0 please consider multiple, configurable gem locations rather than hard coding a single path, like 'vendor/gems' was in the old code. Or if using a fixed relative path then please search it also within each subordinate gem or plugin. Thank you.

Some of you may be interested in this patch (ticket #2834) for the existing 2.3 code.

There's some code up for free in moonshine. With the moonshine rails plugin and the command rake moonshine:gems will generate a config/gems.yml file from the config/environment.rb.

Good for migrating the gem config in existing projects.

http://github.com/railsmachine/moonshine/blob/master/lib/moonshine/...