Rake tasks for run engine migrations

Here's a slightly different approach: http://github.com/lazyatom/rails...

One issue that needs to be taken into consideration is that migrations must be timestamped rather than sequential for migrations from multiple sources reliably. For example, if your app contains 001_create_table.rb and a plugins has 001_create_another_table.rb, once one of these migrations has been run, the other will be studiously ignored by ActiveRecord.

The danger with running migrations directly from the plugin directory, rather than copying to the main migration stream (as my change above does) or creating an intermediate migration (like the engines plugin does) is that we should try to keep an accurate record of the database state throughout the history of the application.

As you can see running migrations is an "engine atomic" operation, and it makes sense, because engines are pluggedin applications and you want to run all the migrations from a single engine, in order to make it full working.

The issues come when you 'update' a plugin, which brings new migrations, but then later try to revert the whole application back to its previous state.

I should also add that I love the incorporation of a Rails.plugins list (though might possibly call it Rails.configuration.plugins, or Rails.loaded_plugins, but that's just taste). This will make it easier to try plugins and gems equally for migrations, as well as a bunch of other stuff.

@James Rails.configuration.plugins is the configuration you specify in environment.rb, so it's nil if you leave it untouched. And Rails.loaded_plugins returns a NoMethodError. :)

Just recognized that schema_migrations tracks all the changes.

This means if you have two plugins with a 001 migration it will only run the first one (according to the Rails.plugins order).

A solution could be to add another column to schema_migrations, in order to specify the source of the migration.

I think it could work, but it has a big downside: what if we rename an already migrated plugin?

Or we can just keep out numeric migrations

Rails.loaded_plugins was a suggestion for the name of your Rails.plugins method (i.e. yes, neither of them really 'exists' yet), but as I said it's only a reflection of personal aesthetics. Good point about the other one.

I'd considered the 'adding a column' approach, but decided that simply copying the migrations across was a simpler first step. I don't see renaming the plugin as a real issue. However, without adding a 'source' column to the schema_migrations table, there is simply no way of supporting numeric migrations.

As I commented on your blog:

One of the big 'wins' with engines is being able to share updates and bug fixes across a bunch of applications. This implies that engine plugins will be updated over the lifecycle of an application, and these updates should be able to include modifications to the database. It should also be possible to 'roll back' such updates, so the application is restored to its pre-update state.

This patch doesn't achieve this, so for the moment, it's a -1 from me, in favour of either the rake task in my fork, or a new patch which adds a source column to schema_migrations.

The problem with renaming a plugin is that the change is not reflected in the schema_migrations column.

I don't agree with your comment, because if an engine need to modify the database over the time, it should add new migrations with schema alterations. And this patch actually supports this specific use case.

Copy migrations from vendor/plugins/*/db/migrate to db/migrate looks a bit hacky to me. So why don't copy controllers and models to app? The answer is: separation.

As we already argued, engines are tiny applications. I think it's better to keep all the stuff in vendor/plugins.

Don't get me wrong - I am really, really glad that you're spending time looking at this, and don't want to sound unappreciative. I just want to get it 'right' :)

I don't think 'renaming a plugin' is a valid concern here, as it's something that basically never happens. So let's put that aside.

I think what we're disagreeing about here is that as well as being able to upgrade an engine, I want to be able to downgrade to a previous state. Without any ability to 'undo' a change to my database state, the whole proposition becomes less compelling. Copying the migrations is merely the simplest possible solution that delivers this.

Certainly, separation is important, and I'd hope that the past few years of effort on my part might demonstrate that I appreciate that :) Copying the migrations wholesale isn't ideal - my preference would be to have an additional column, as I've said before, but I believed that it would be harder to 'sell' a more invasive patch to the powers-that-be. Any opinions, core team?

The key thing here is that I think we need to be able to rollback the database in a more robust and consistent way than everything-or-nothing. That's all, really.

I created a second patch for running plugins migrations.

I added a plugin column in the schema_definitions table: when NULL the migration belongs to the application, otherwise to a plugin. I created a rake task (rails:update:schema_migrations) for update from old Rails versions, the task has been added to the more generic rails:update task.

Two important changes are about db:migrate and test tasks: they both automatically check/run migrations from plugins. This means test suite will be halted if there are pending migrations from plugins.

Even if db:migrate will run all the migrations, you have a new set of Rake tasks for a full control on your schema:

db:migrate:application   # Run migrations from application
db:migrate:plugins       # Run migrations from plugins
db:migrate:plugins:down  # Runs the "down" for a given PLUGIN and a given migration PLUGIN_VERSION
db:migrate:plugins:up    # Runs the "up" for a given PLUGIN and a given migration PLUGIN_VERSION
db:plugins:redo          # Rollbacks the database one migration and re migrate up for a given PLUGIN. If you want to rollback more than one step, define STEP=x. Target specific version with PLUGIN_VERSION=x.
db:plugins:rollback      # Rolls the schema back to the previous version for a given PLUGIN. Specify the number of steps with STEP=n'
db:plugins:version       # Retrieves the current schema version number for a given PLUGIN

Talking about the plugin renaming issue, I added a new command script/plugin

script/plugin rename cached-models cached_models

I think it could be enough ;-)

I extracted the patch into a plugin: http://github.com/jodosha/plugin...

Enjoy!

Good thing we discussed that yesterday, Luca. Awesome patch! :-)

Great work. There are a few bugs in the patch (http://github.com/lazyatom/rails... for fixes), but having discussed this with @jodosha we feel there might be a few more fundamental problems with the approach. I'm working up a proper example, and will post back when I have it ready.

I'll try to provide a trivial example.

vendor/plugins/authentication/db/migrate/20090101000000_create_users.rb
db/migrate/20090101000001_add_users_openid.rb

This situation will cause a crash, because db:migrate run application's migrations first then from plugins.

Maybe the solution is to invert the order above.

I'll try to provide a trivial example.

vendor/plugins/authentication/db/migrate/20090101000000_create_users.rb
db/migrate/20090101000001_add_users_openid.rb

This situation will cause a crash, because db:migrate run application's migrations first then from plugins.

Maybe the solution is to invert the order above.

I've posted an initial version of our consideration of plugin migrations here: http://interblah.net/plugin-migr...

If time allows, I'll detail the other two solutions that I didn't outline. One of them still isn't good enough, and the other will work, but it quite complex.

As a summary, there needs to be a single timeline that describes the order in which migrations where applied to the database to avoid breaking rollback, and any viable solution needs to respect this.

My preferred solution at the moment is copying and re-timestamping migrations from the plugin into the applications' migration directory, and also editing the migration name to include the name of the plugin so we can tell which migrations have been copied in the past. This is described as 'Solution 6' on http://interblah.net/plugin-migr...>

I would love to get this functionality into Rails 2.3, so I'll modify my original simple-copy branch on github to work in the way I describe.

Ok, in the meanwhile I played a bit with my patch, it seems to work fine for me. The only change I would apply is related to migrations execution order: as I said in my previous comment, it's better to run first plugins migrations, then from the application.

Because each plugin could provide database alterations, and we would to reference them in our application migrations.

I attached a patch with the new execution order.

Ooops wrong patch! Here the right one.

Sorry this has taken so long, but I have a version of my simplest-possible solution that seems to be satisfactory in terms of not breaking the migration timeline, being fully rollback-able and so on.

See my earlier link to interblah.net for more details; I do still believe those issues are important and must be addressed in any solution. Running plugin migrations 'first' doesn't guarantee these factors.

Anyway, here's the patch. The gist is - migrations are copied, but with a new timestamp. This preserves the ordering in which they are applied, and so they can be rolled-back consistently.

However, we must be able to tell which migrations have already been copied from a plugin later. We do this by adding the plugin name to the migration, as part of the extension. So, 12345_add_jazz.rb from the 'acts_as_armstrong' plugin is copied as 45678_add_jazz.acts_as_armstrong.rb (or whatever the current timestamp might be).

This is a compromise in two ways. Firstly, some people will not like that migrations are copied. My response to that is that the amount of code required to support running the migrations from within the plugins is much larger, and I'm not sure the benefit is there.

The second compromise is the way in which the filename is modified. Some people, I am sure, will find the .acts_as_armstrong.rb ugly. Because of the way migrations are implemented in Rails, the basename of the file must match the class inside the file, and so this solution is the simplest way to 'tag' a migration with its source without having to employ complicated logic to rewrite the migration class name inside the file itself.

Anyway, I hope it's all very clear. The patch is a new rake task with some supporting methods, and a tiny tweak to ActiveRecord::Migration to support the slightly novel file naming.

Good compromise - nice simple patch.

I was wondering why it would be preferable to use the poor man's copying approach for plugin/engine migrations rather than using the metadata approach which the old engines plugin used to use (pre-Rails 2.3) for migrations?

The metadata approach appears to solve all issues from where I am standing and allows the application developer to control what version of the plugin/engine schema they need to migrate to (if they want that control). It also provides an auto discovery mechanism for application developers to just generate a "plugin_migration" (please refer to how the old engines plugin does this to demonstrate this further) that creates an application-level migration which migrates up to the latest schema versions of the various plugins/engines. It also generates the down method appropriately too. Therefore everything is kept in sync.

The blog post linked to above that discusses the available solutions for big problem conveniently glosses over the "pitfalls" of the metadata approach. How are any pitfalls associated with a metadata solution to plugin migrations even close in magnitude to the poor man's copying approach? Also the copying approach appears to only work for plugins/engines that are under vendor/plugins not Gemified plugins/engines, which in my view is short-sighted.

Thanks and would appreciate a more thorough explanation of the pitfalls of the metadata solution, because I am not seeing them right now, especially in comparison to the pitfalls of the copying solution, which are:
1. Only working for engines/plugins in vendor/plugins. It fails to provide a solution for cleaner Gemified plugins/engines. Thus penalizing those that want to create cleaner and better organized web applications and engines/plugins.
2. Duplicating migration code when IMHO it is unnecessary.
3. Long-ass migration file names.

Thanks and I look forward to feedback on my analysis above. This is only meant as a (hopefully) constructive critique.

To offer an answer to your question Susan, I would say one of the main pitfalls to the meta-data approach is that having migration files scattered in multiple places in an app adds an unhealthy level of indirection. It forces you to perform some mental gymnastics when you want to identify exactly which plugin migration corresponds to a generated migration. This becomes even more painful when you have more than one engine.

Having worked extensively with engines, for me, the generated 'plugin migration' mechanism just feels messy, and although copying is not a perfect solution, I'd much prefer to have a clear and obvious migration history in db/migrations without having to jump around my plugin's db/migrations folders to see exactly what's happened. This becomes even more difficult if the migrations are in gems.

The penalty may be duplicating migrations into your app, but I think this is a price worth paying to keep the migration system simple.

Also, I don't see why the patch couldn't be extended to support Gemified plugin engines.

@Tekin It seems I was not clear what I meant by metadata approach. I suggest we could use a metadata approach like that used in the old Engines plugin, but not necessarily by generating a plugin_migration. That does is not needed at all. My point was only that the Engines plugin used metadata and stored it in the database and that is provided a much cleaner way of managing this metadata internally. I agree with your assessment of the generated plugin_migrations, however, I see very little difference between generating a plugin_migration and copying files.

As far as my experience with engines goes, I think it would be wise to keep engine migrations with the engine code itself. There doesn't seem to be a need for copying files [again]. Of course, I never liked copying files into vendor/plugins either from the beginning. I am glad that at least on the latter issue Rails developers are starting to see the light.

If no plugin_migration needs to be generated for a generic solution to this problem, are there any other shortcomings to using a metadata approach? I have not seen this/these mentioned anywhere.

The only reason I could come up with was:
1. It adds either a whole new table to the database or a new column to the existing schema_migrations table. One could argue that this might be problematic, although I do not see that for 99.99% of usages out there. Comments?

I am asking because I would actually like to use a Rails sanctioned way of managing engines migrations, but I couldn't imagine using a "solution" that merely copies files. I am working on a client specific solution to this problem right now, but that only works because of the way we use schemas to separate engine models and is not a generic solution. Honestly, I like how compartmentalised this is, but wouldn't work for every environment, probably not even half:(

When I stop working insane hours (probably end of August) I would like to build a more complete solution myself. I am not criticizing without attempting to provide a solution, but I have little time at present to actively develop.

I simply want a well thought out solution rather than just copying files again.

Thanks.

I don't have a dog in this fight, so I'll just add my two cents and move on...

For my application, with timestamped migrations, the simplest possible solution was the best solution: I redefined the rake database migration tasks to pull migrations from **/db/migrate rather than db/migrate. And it works.

My rake task file is attached for anyone who's interested.

The crux of this problem is that you will need to make the assumption that either:

• Engines/plugins are independent of the application logic, or;

• Engines/plugins are NOT independent of the application logic.

Surely we need to make the assumption that engines/plugins are not independent of our application logic and that running migrations along parallel timelines is going to break something at some point? What we're talking about with engines/plugins integration is no different to having a dev team work on a single code base in parallel - at some point we need to bring all the components together through a merge process.

With this in mind, I can't see any other method besides forcing the application developer to manually merge migrations in to a single timeline. The alternative is that we automatically merge, but as far as I'm aware this isn't possible yet.

If you agree with me up to this point, then the only problem left to solve is how to implement the merging of the single timeline.

Ignoring engines/plugins for the time being, Rails currently keeps a chronologically ordered queue of migrations, and then a pointer to the current migration in the schema_migrations table. Adding more pointers in this table to the different engines/plugins will not work as that would make the assumption that engines/plugins are logically independent to the application, but this is not a safe assumption. A single pointer will suffice given that we must merge the migration timelines anyway, so my vote is to keep the schema_migrations system as it currently stands.

Given all of this, copying migrations in to the application's migration directory makes a lot of sense. Including the plugin name in the migration will go a long way to help keep things organised and prevent information loss. Re-timestamping the engine/plugin migration is required to effectively perform the merge procedure.

If everything I've said above is correct, then James Adam has not only suggested a solution, but probably the only correct solution to this problem.

While James has provided a solution I'd like to hope that I've shown that it's the only minimal solution to this problem. Hopefully it's now easier for the core team to integrate a similar solution in to core :)

Can somebody delete the comments marked as spam above? Apparently, any possibility of getting a link seen is enough to draw spammers...

+1 for @James Adam 's approach. Particularly I'd prefer copying the migrations into the hosting application, since it gives the host's maintainer an opportunity to review and alter the migrations before running. I don't know about you, but I'm rather paranoid about letting someone else's script alter my production database.

@LeoDev: Great, now the spammers have taken to writing comments. Just great.

During this summer I work on mountable apps project for rubysoc.org and one of the challenges that I needed to solve is running migrations in plugins. I went through that topic and attached posts (James' post was really helpful with summing up all the problems and approaches) and after discussions with my mentors we chose James Adam's approach with copying migrations and retimestamping them.

Unfortunately James' patch didn't apply cleanly and I wanted to move the entire implementation to ActiveRecord::Migration to test it easier, so I implemented it from zero.

Currently my work is not yet merged with rails master, so changes are in my fork: http://github.com/drogus/rails/compare/a5c8d892...5ba45959

I would be glad to hear feedback on these commits.

Automatic cleanup of spam.

Automatic cleanup of spam.

Not sure if this is still relevant or wanted, but I've had migrations running from engines for a while now.

Not the best solution but I found that overriding the ActiveRecord::Migrator.schema_migrations_table_name method to prefix the name of the engine made rails install each engine's migration independently so you can easily migrate engines completely separately from the rails app.

I created a bunch of methods (one for each rake task with migrations) in the namespace Migrations.

Automatic cleanup of spam.

The Rake tasks for run engine migrations is some thing which has good for knowledge and I think this is really nice to have such information for engine migration.

Term Papers

Originally an engine was a mechanical device that converted force into motion. Military devices such as catapults, trebuchets and battering rams are referred to as siege engines. The term "gin" as in cotton gin is recognised as a short form of the Old French word engin, in turn from the Latin ingenium, related to ingenious. Most devices used in the industrial revolution were referred to as engines, and this is where the steam engine gained its name.

Credit Card Debt Relief

Since version 2.0, Ruby on Rails by default offers both HTML and XML as output formats. The latter is the facility for RESTful web services.

Ruby on Rails 2.3 relies on Ruby 1.8.6. Ruby on Rails 3.0 has been designed to work with Ruby 1.8.7, Ruby 1.9.2, and JRuby 1.5.2+; earlier versions are not supported.

Debt Relief Company

The Nissan Engine Museum (日産エンジン博物館, Nissan Enjin Hakubutsukan?) is a automobile engine museum run by Nissan Motor Company. The museum is located at the first floor of the guest hall in Yokohama auto plant, Kanagawa-ku, Yokohama, Japan.

Aviation Insurance

Outstanding blog post, I have marked your site so ideally I’ll see much more on this subject in the foreseeable future...
mothers day flowers delivery Bangkok | send mothers day flowers Johannesburg

he Old French word engin, in turn from the Latin ingenium, related to ingenious. Most devices used in the industrial revolution were referred to as engines, and this is where the steam engine gained its name...leather lounge

Originally an engine was a mechanical device that converted force into motion. Military devices such as catapults, trebuchets and battering rams are referred to as siege engines. The term "gin" as in cotton gin is recognised as a short form of the Old French word engin,
logo design

Awesome collection of dog houses. I am impressed!

Regards,
Dissertation Help

plugin migrations even close in magnitude to the poor man's copying approach? Also the copying approach appears to only work for plugins/engines that are under vendor/plugins not Gemified plugins/engines, which in my view is short-sighted.iPhone deals

I am once again feeling happy and proud to say that this is my favorite web site.The postings are very unique and also out standing performance with the new creativity and excellency with the new different ideas and concepts.Really I am waiting for some more new posts from you.Keep up your excellency and efficiency in this same levels....

cell phone spyware

I am really enjoying reading
I am really enjoying reading your well written articles. I think you spend numerous effort and time updating your blog. I have bookmarked it and I am taking a look ahead to reading new articles. Please keep up the good articles!

cell phone spy software

This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information. Keep it up. Keep blogging. Looking to reading your next post. Austin Tx Real Estate