In a Rails app, I have a migration that takes a while to complete. First it updates a column's default, then it re-indexes a Solr index. The problem is, that while the migration is running, the old version of the app serves requests with the partial-new database and solr index. And because of the new default column, it throws some errors.
Is there a best practise method to avoid such a behaviour? Should we use a second database and solr index? We are deploying with capistrano.
This is a common problem with long-running, complicated rails database migration.
There're several methods:
(1) Before running your database migration, redirect your web traffic to a page with a message, eg "we are down for maintenance in the next 30 minutes". This would prevent any requests from coming in and hitting your application with a database only partially migrated. This is not ideal due to downtime.
(2) Run your database migration before code deployment, but ensure the database is backward and forward compatible. Then deploy your code to use the new schema. You may want to write to the old schema too (until the next release) in case you need to "rollback". Then later remove or rename the columns in another migration to clean up the schema. This is called "zero downtime database migration".
(3) You could have two databases, and once the new database is prepared, you can do the switch. But the issue here is how to keep them in sync, if there're frequent writes coming in from the application. You could run a script to sync them afterwards.
As a precaution, you should take a backup dump of the database before your migration in case you need to rollback. As some db migration is irreversible, for example, remove columns. Or you could copy affected tables into temporary tables as backup.
And of course, do the migration during a relatively quiet period.
For solr, it should handle incremental update without throwing an error.