When loading a plugin via rubygems, rake tasks aren't included

In fact there's no provision for it doing anything but plugins in vendor/rails (eg if you have plugins in a non standard directory)

Here's the hackish patch to my project Rakefile that I've used to work around this bug.

Here's a patch for the issue - it includes tasks from frozen gems in exactly the same way they're included from plugins (e.g., /vendor/gems/*/**/tasks/**/*.rake)

Consider this a +1, but should there be a test for it?

+1, but I think there's potential issues with gems currently being used that store gem specific tasks that wouldn't be need in the rails project scope in the tasks directory (e.g. rdoc tasks, rcov tasks, rubyforge, etc.)

another possible glob to consider is to namespace it in the rails directory:

"#{RAILS_ROOT}/vendor/gems/*/rails/tasks/**/*.rake"

I agree with Brian, namespacing should occur, but then we should ensure that it also happens in classic plugins (to ensure consistency)

I'm not entirely in favor of this. IMO frozen gems should behave like external gems. Instead I think the convention should be to add a "require" statement to the Rakefile to include the gem's tasks.

For example, let's say we are moving the rspec-rails plugin to a gem. This plugin has many rake tasks. Someone may want to freeze the gem, but others might not - either way these tasks should be included in the same manner.

To do so, one could add a require statement to their rails project Rakefile.

# in Rakefile
require 'spec/tasks/rails' # or something similar

The included file would behave just like the rails internal tasks/rails.rb file - it simply loads all of the tasks for that gem.

I haven't tried this yet so I don't know how well it works, has anyone done so? Any opinions?

I did try something similar to what Ryan suggested, by having one file that could be required in a Rakefile. It works fine, and doesn't require any patching.

I think there's two ways to look at this:

• Using gems as a replacement for plugins: to start using a plugin-turned-gem, you have an extra step to get up on running, if it has Rake tasks.
• Using gems as gems: if you've wanted to use Rake stuff from gems, you've always had to require it at some point.

"Someone may want to freeze the gem, but others might not - either way these tasks should be included in the same manner." - I agree with that, there should be literally no difference.

What if it was done like generator lookup?

That is, installed gems could be searched for rails/tasks/* as well.

I was wondering if a global lookup would work too. This is similar to how generators work so it must be possible.

However, I think rake tasks are usually more project specific. They also sometimes need to inject or override behavior.

Going back to the rspec-rails example. This includes a lot of rake tasks which I don't want if my project isn't using rspec. Not only that, it overrides the default rake task. The "rake" command without specifying a task usually runs all tests, but with rspec installed it overrides this and runs the specs, not tests.

I don't see an easy way of solving that if you're always including rake tasks from all installed gems.

What about config.gem - if a gem is specified there (frozen or not) it could load tasks.

In the other case (gem installed on the system but neither mentioned in config.gem nor frozen) tasks whould not be loaded.

And in the edge case: gem not mentioned in config.gem but frozen - tasks would be also loaded.

That might work. This would mean rake would have to load the environment to determine the tasks, but it may already do that. Anyone know?

Some gems you don't want to always load (again, rspec as an example) so you don't want it in config.gem. However that's probably an edge case. In that case it can fallback to a require statement in Rakefile.

I'm a little confused how the rails/init.rb file works. Does this get executed for all frozen gems automatically (without needing the config.gem statement)? It makes sense that rake tasks loading follow the same logic as init.rb.

The problem is it doesn't work yet: #324 and #122.

Maybe we should wait for the rails/init.rb problems to be sorted out and then revisit this. To me it makes the most sense that rake tasks are included when init.rb loads, since that's technically when the gem is loaded/included.

The attached patch should achieve the results Ryan is talking about. It loads rake tasks in the rails/tasks directory inside gems that are loaded via config.gem.

re: "And in the edge case: gem not mentioned in config.gem but frozen - tasks would be also loaded." I do not think tasks should be loaded in this case. Loading a gem via config.gem sets up the load_paths for that gem and should be (is) required for that gem to be available in the rails app. This patch follows that convention.

re: "Some gems you don't want to always load (again, rspec as an example) so you don't want it in config.gem. However that's probably an edge case. In that case it can fallback to a require statement in Rakefile." The only case I can imagine is based on environment and the ability to load gems selectively based on environment already exists via the sub config/environment files. You can put config.gem statements in these files to get the desired results. This patch follows that convention: gems with rake tasks loaded in the test.rb environment file will only be available when running rake in the test environment.

Also note I added a patch to this ticket to solve the rails/init.rb issue: http://rails.lighthouseapp.com/p... so help me out and +1 that jam. I wanna get this stuff into rails as I believe gems are something like 14.8X cooler than plugins for most scenarios.

Just hit this brick wall myself. Would love to see this get into 2.2, so we can really start running with gemified plugins.

The latest patch still requires starting up the rails environment for rake tasks that possibly don't need it, or in some cases actively don't want it. It will also cause quite a lot of extra work to happen just to be able to do a rake -T for example.

Seems like a step in the wrong direction.

We need to have a way of knowing what gems we are using, while never executing the environment{.rb,s/*.rb} files. I for one have no good plan.

Is it worth considering moving gem requirements out of the environments files? A big step I know.

Anyways, my 2cents, feel free to ignore.

Can we do something about this before 2.2? It would be great to have it.

Maybe we can rely on a new initializer (a good practice IMO) like config/initializer/gems.rb. That way the file could be required (being passed a mock config object) without the environment. We can get the gems and include the rake tasks.

Thoughts?

Let's push for this before 2.2...

Why do we need a new initializer when you can accomplish the same thing via preinitializer.rb, and avoid this issue entirely?

I've been restraining from pimping GemInstaller in this thread, but making a new initializer seems too much added complexity - and ironically, lack of complexity was the original reason technoweenie gave for duplicating GemInstaller's functionality in config.gems and the gems rake tasks. Perhaps the difficulty of managing the chicken-egg nature of gem loading and dependency management is indicating that loading gems during initialization is not the best solution. This is the whole reason I put in the preinitializer.rb patch - so there is a hook that occurs before rails ever initializes, where you have a chance to load all your gem dependencies.

This approach works fine, we have been using GemInstaller to do it with many production over the past couple of years, even by hacking boot.rb directly before I pushed the preinitializer.rb patch. See docs at http://geminstaller.rubyforge.org, or the source for an example rails app: http://geminstaller.rubyforge.or...

GemInstaller is stable, actively maintained with up-to-date lighthouse - thanks Rick :) - status for all known bugs, backward compatible with several legacy versions of rubygems, and very well tested, including Continuous Integration against Rubygems trunk and major legacy versions of rubygems. And, and as I said, it in use in many production rails apps going on a couple of years now.

Or, if people don't want to use GemInstaller at all, that's fine, it is just a matter of invoking the proper "gem" methods from preinitializer.rb. Or the whole config.gems logic and rake tasks can be decoupled from the initialization process to make it easy to invoke from preinitializer.rb. I haven't confirmed lately, but when I looked at the initialization code, the rails initialization should be able to find any gem or gemified plugin, whether it was loaded by config.gems or prior to that in preinitialization. If that isn't the case, it should be...

The point is, the difficulty of solving this issue to me points that trying to load/activate gem dependencies during initialization is a misguided attempt. They should be activated BEFORE initialization, with is the exact and only purpose of preinitializer.rb. I believe this is true regardless of how you activate the gems, or what sugar rake tasks/etc you provide to install/freeze the gems.

Thanks, -- Chad

Here's a more concise summary of my take on the issue, from the mailing list thread:

Basically, I think preinitializer.rb should be used to activate gems, whether that is manually via the 'gem' method, through some call to an initialization-decoupled version of the config.gems code, or some third party tool like GemInstaller.

The rails initialization code should be structured (if it is not already) to process the loaded gems, init files, rake tasks, etc that are already on the load path, regardless of how they got there (it can be any method as long as it is called from preinitializer.rb before any initialization happens).

and one more thought :)

This approach (making initialization not care how gems/plugins were activated, but still able to process them) would probably also make it a lot easier for hosted rails environments, like Heroku. They can provide their own mechanism for configuring/specifying/installing gems, and just instruct their users to put an appropriate hook in preinitializer.rb. Everything else will work the same way - and developers can even provide an alternate mechanism in their local dev environment, just by switching on RAILS_ENV or some other environment flag in preinitializer.rb.

We use preinitializer.rb for ensuring that the proper versions of gems are activated in the app. Chad's suggestion has worked out well for us. (we're unabashed GemInstaller users). I never understood why we went straight to ruby (config.gem) for this type of configuration...

"It's probably easiest to think about refactoring the initializer from the huge glob of procedural code it is now, into something a little more flexible. So you could run a subset of the initializer then retrieve the results."

Yes, that's what I mean by decoupling it - so gem activation can go as early as you want.

However, I'm not sure if this approach can or should still be coupled with the inherent multi-environment support that comes with being part of rails init. If it is still part of rails init, you still have the possibility of chicken-egg situations that come with executing init logic before all dependencies are fully loaded.

It still seems cleaner to have the standard approach be to load/activate everything before any rails init starts (in preinitializer.rb or equivalent), and if people need environment-specific loading, they can manually switch on RAILS_ENV.

Also, keeping gem loading explicitly decoupled from rails init solves the issues related to people wanting decoupled and DRY dependency declaration/installing/loading, but not always wanting to initialize rails (such as for Rake tasks, or hosted ci or rails environments processing gems).

Finally, if you run your YAML through ERB (which GemInstaller does), you can still do any 'fancy' ruby conditional checks you want ;)

What would a solution with preinitalizer.rb look like?

Why would we call it preinitializer.rb instead of gems.rb? What other purposes would it have?

@damian:

It would look like this :)

http://geminstaller.rubyforge.or...

preinitializer.rb is already in rails, has been since about 1.0, so the name should not change. It's working fine for us in multiple apps and in the example above.

It is called preinitializer.rb because it is a generic hook, it could be used for gem loading/activation, or anything else you want to do before rails initialization starts. It is essentially just a hook that boot.rb calls before it does anything else.

-- Chad

sorry, I misspoke - preinitializer.rb support was added right before 2.0. Still pre-git, though ;)

I think I like the idea of using preinitializer. Some gems (or at least soap4r) already seem to require you to do something like that or they won't work properly.

One potential downside I see, though: would we lose "rake gems" and "rake gems:install"? Not that I've used them much, personally, but others?

@george:

If the logic for the rake gem[:*] tasks is decoupled from the initializer, it should be possible to keep the functionality with a single preinitializer hook.

In any case, the RubyGems API is not hard. The "GemRunner" class is pretty much an exact mirror of the 'gem' command line commands, and the Gem Specification class provides info about gems, such as whether they are installed, etc.

The frozen vs. not frozen reporting seems to be the only rails specific thing, which doesn't seem like it has to be tied to any initialization, it just involves looking under vendor to see if they exist, right?

Hmmm, I think I was misunderstanding how you'd use the preinitializer. I took it to mean you'd just require the gems in there, in which case the tasks in question would blow up if you didn't have the gems installed.

So you'd still use a config file of some sort to declare gem deps?

@george

You can do whatever you want. You can install and/or activate ('gem' method) the gems, but point both to the same list of gems to keep it DRY (just like config.gems already does). I usually always activate in all environments, but only auto-install in non development/test environments (e.g. ci, staging, prod) - because if you auto-install in dev/test it really slows down startup time. SO, this means you have to manually install gems on your dev box, but they automatically get installed in CI and deployable environments.

Also, don't confuse activating a gem with a require statement. 'Activating' is just putting the gem files on the load path (as defined in the gem spec), by issuing a 'gem' method call. That would be done in preinitializer.

'Requiring' is issuing a ruby 'require' or 'load' for one of the files that the gem put on the load path. That would be done during initialization or rake processing.

@Chad

Thanks for the explanation, but I think we're on different wavelengths. The point I was trying to make is simply that if evaluating the preinitializer does anything that depends on the gem being installed--be it #require or #gem--it'll raise an exception if the gem isn't there. This would thwart tasks like "rake gems" (which shouldn't need all the gems installed) if it tried to eval the preinitializer.

But really, I think I just misunderstood your proposal. I had thought you were proposing using preinitializer with direct #require (or #gem) calls instead of a configuration file (or some config object passed to preinitializer), which would've made getting the list of uninstalled gems a bit tricky/hacky. But I'm pretty sure I'm mistaken; sorry for the noise. X-(

+1 for this to work the same as standard plugins

This has actually been resolved in Rails 3 - you'll need a subclass of Rails::Railtie, and add a rake_tasks block. Inside that, load the appropriate task files. An example from my Thinking Sphinx library:

http://github.com/freelancing-god/thinking-sphinx/blob/rails3/lib/t...

+1 for making it work without copy-pasting some railtie(that will likely have a different api in future rails version) into every gem.