Content
You are here:
Database Update Fails 11.2.2 to 11.3.0
Added by Rob A over 3 years ago
Hi all,
I've never had a single issue updating Openproject in the past but this time I am getting an error while upgrade to version 11.3 from version 11.2.2.
[openproject] ./bin/postinstall
I, [2021-06-07T11:56:51.309386 #8973] INFO -- : Migrating to MakeProjectIdentifierUnique (20210512121322)
== 20210512121322 MakeProjectIdentifierUnique: migrating ======================
-- remove_index(:projects, :identifier)
rake aborted!
Does anyone know how to solve this issue?
Full error text is below.
Thanks!
Rob
Multiple indexes found on projects columns [:identifier]. Specify an index name from index_projects_on_identifier, index_projects_on_identifier
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/schema_statements.rb:1408:in `index_name_for_remove'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/postgresql/schema_statements.rb:470:in `remove_index'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:929:in `block in method_missing'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:897:in `block in say_with_time'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:897:in `say_with_time'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:918:in `method_missing'
/opt/openproject/db/migrate/20210512121322_make_project_identifier_unique.rb:3:in `change'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:867:in `exec_migration'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:850:in `block in migrate'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:849:in `migrate'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1037:in `migrate'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1380:in `block in ddl_transaction'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/transaction.rb:310:in `block in within_new_transaction'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.3.2/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/transaction.rb:308:in `within_new_transaction'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/transactions.rb:209:in `transaction'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1380:in `ddl_transaction'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1302:in `each'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1302:in `migrate_without_lock'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1251:in `block in migrate'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1401:in `block in with_advisory_lock'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1416:in `with_advisory_lock_connection'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1397:in `with_advisory_lock'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1251:in `migrate'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1086:in `up'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/migration.rb:1061:in `migrate'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <top (required)>'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/railties/databases.rake:90:in `each'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.3.2/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <top (required)>'
/opt/openproject/vendor/bundle/ruby/2.7.0/gems/rake-13.0.3/exe/rake:27:in `<top (required)>'
bin/bundle:110:in `load'
bin/bundle:110:in `<main>'
Replies (29)
Ran with Trace:
I am having this exact same issue as well. Attached is my db:migrate error.
Not sure what to do either but thought I would add on as this might not be a one off issue. Running enterprise edition as well.
Same issue and same message when trying for 11.3.1.
Also fails going to 11.3.2
Michael,
Are you running Debian 10 (Buster)? I've also opened a bug report for this issue.
Rob
Hi Rob,
I am running Ubuntu 20.04 LTS.
Great, maybe we can find a fix then!
Hello,
I've got the same problem now. Is there a fix?
We are running Ubuntu 18.04 LTS.
Hi,
I've got the same problem. System runs on Debian Buster, Package-based installation. It looks like everything is working. Projects can be opened, tasks can be created, so far I haven't noticed anything that doesn't work.
But in the Administration section there is also a note that the migration was not successful. How do you deal with the error, do you stop the service or do you continue working on the system?
Andreas,
I'm running the same setup in a VM so I always snapshot it before doing an upgrade. I just rolled it back when the migration failed. I've since spun up a cloned test VM and like you said, it just seems to work even with the failed update. I don't think the next updates will succeed because they may rely on previous database migrations being complete.
Sorry that's not really an answer but it's the best I can do.
Hi Rob,
Thanks for the reply, I will also go back to the last working snapshot. So far I had Openproject automatically updated via the official repositories. But updating manually and making backups directly before is probably the better way :) (Although it has always worked great before)
Hi Michael,
you're running into this issue because we added a migration that prevents multiple projects receiving the same identifier. This stems from this work package:
There appears to be a rare race condition that allowed an instance of OpenProject to duplicate a project with the same identifier. As we could not reproduce it, the first step to identifying the issue is to prevent it happening on a db level. There however was no uniqueness constraint defined yet on the database level, just as a Rails validation.
What you can do to find the duplicates. Run a rails console with
openproject run console
Then, get a list of duplicate identifiers:
Project.group(:identifier).having("count(*) > 1").pluck(:identifier).uniq
This will give you a list of the identifiers that are duplicates. You can then find the affected projects
Project.where(identifier: '<IDENTIFIER FROM ABOVE OUTPUT>').map { |p| [p.id, p.name, p.updated_at, p.work_packages.count] }
This will output something like this
A good indication of which one is the duplicate output is the one that has a zero at the end (no work packages where created in that project). But it depends on how you use the project.
Once you're sure you can delete one of the projects, run this command where ID is the id of the project. In the above output, the ID is the first number in the list (1 and 2 in my examples).
Project.find(ID).destroy
After that, run
openproject configure
like in a normal update, and the migration should run through. We would have liked to automatically do this for you, but as we weren't sure which project would be the duplicate and which one would have been used, this might have resulted in unwanted data loss.Best
Oliver
Oliver,
I'm not sure about Michael but my installation does not contain any duplicate entries. I ran the same search with > 0 to double check and it returned all of my projects.
irb(main):003:0> Project.group(:identifier).having("count(*) > 1").pluck(:identifier).uniq
=> []
irb(main):004:0> Project.group(:identifier).having("count(*) > 1").pluck(:identifier).uniq
=> []
irb(main):005:0>
Thanks,
Rob
Hi Rob,
indeed the error message appears to hint at something different. That there are multiple constraints on :identifier already which shouldn't be the case.
Please run the following command and append the output:
(the command works only for a packaged installation)
psql $(openproject config:get DATABASE_URL)
And then run
\d+ projects
to describe the projects table with all indices
on my end, this looks like the following
And I assume under Indexes: You have another index that collides with the one we create for the migration. I am not sure why that would be. Did you migrate from MySQL at some point in time?
Best
Oliver
I have the same trouble.
Output of \d+ projects :
Looks like Rails is complaining because it finds two indexes, however only one is present according to your command.
I can't really reproduce this but I attached a patch that might fix this. Could you confirm that this works by applying the patch manually ? We will the release a patch release with this fix:
Download the patchfile and check whether it applies: From your OpenProject root folder:
Check whether the patch applies cleanly
⟩ git apply --check /path/to/project-identifier.patch
If nothing is output, it applies cleanly. Then, actually apply the patch:
⟩ git apply /path/to/project-identifier.patch
To reverse the patch again, run
⟩ git apply -R /path/to/project-identifier.patch
After applying the patch, please run
openproject configure
once more.Best
Oliver
Oliver,
I did originally install Openproject with MySQL and then migrated to Postgres following the official instructions.
Here's the output on my test VM (after trying to update).
On my production VM (before the update) the output looks like:
Oliver,
This instance was migrated from MySQL and unfortunately the patch does not fix the problem.
I get the same db:migrate error as before.
Thanks,
Rob
Hi Rob,
thanks for the reply. Can you also run this postgres database command (just like above with
\d+
)Best
Oliver
Oliver,
Here are the results:
Hi Rob,
it seems you still have the openproject and public schema present. During the migration, it looks like the search_path includes both schemas and fails for that reason.
Can you double check that the
openproject
schema is no longer in use? The migration script should automatically rename the schema fromopenproject
topublic
but for some reason, this isn't happening for you:https://github.com/opf/openproject/blob/v11.3.2/docker/prod/mysql-to-postgres/bin/migrate-mysql-to-postgres#L131-L143
OpenProject should run under the
public
schema by default, so I assume you can simply drop that schema:(WARNING: DATA LOSS if you're not sure the openproject schema is unused)
DROP SCHEMA openproject CASCADE;
Best
Oliver
Oliver,
That did indeed fix the update problem but it seems that all of my data was stored in the openproject schema. Openproject restarted and acted as if it was the first time.
Would it be safe to remove the public schema? Will the update process rename the current openproject schema to public?
Thanks,
Rob
Hi Rob,
in this case, you should retrace the step of the migration script that does the rename for you:
DROP SCHEMA public CASCADE
ALTER schema openproject RENAME to public;
OpenProject can also be taught to use the
openproject
schema by your DATABASE_URL withschema_search_path=openproject
. But I'd recommend moving it to the public schema instead, as that's more future proof and is what the migration script should have done.Best
Oliver
YES! That worked perfectly! Thanks so much for all your help!
For anyone else having this db migration issue follow the steps bellow:
psql $(openproject config:get DATABASE_URL)
You should only see the "public" indexes. If you see the openproject indexes AND public indexes below then you have this problem.
WARNING: DATA LOSS
The easiest way for me to check which schema was active was to delete one (after making a backup of course).
DROP SCHEMA openproject CASCADE;
If this reverts the installation to a "new" installation (admin is the only user, no projects, etc) then you restore and run:
DROP SCHEMA public CASCADE;
ALTER schema openproject RENAME to public;
The update process now works flawlessly.
Hi Rob,
thank you very much for helping me in identifying this bug. This is highly appreciated.
I'll try to extend the migration with some information in case this error is raised. There's probably not much we can automate at this point in time though.
The migration from MySQL to PostgreSQL should have automatically renamed the schema. It's probably hard to identify this now, but if you or any other user in this thread has any more information on their migration path, let me know.
I'll try to reproduce the migration issue.
Best
Oliver
Oliver,
I did the original install of version 8 or 9 of Openproject with MySQL and then migrated to Postgres at whichever version that became a requirement. I followed all of the upgrade procedures at the time. It was a long time ago that I migrated but from what I remember everything went smoothly and there weren't any error messages. I'm running a standard packaged install on Debian 10 (it was an older OS version back when I did the migration). Openproject is running on a barebones headless VM with no other software installed.
I can't think of anything else that might have affected things but I'm also not an expert on any of this by any means.
Thanks for all your help getting this sorted out.
Rob
Oliver Günther wrote:
+
sudo openproject configure
+
sudo openproject restart
Worked fine! Thanks!
Hi Vladimir,
thank you for confirming this as well!
Just to be sure, your database was also migrated from MySQL to PostgreSQL at one point in time?
Best
Oliver
Oliver Günther wrote:
Yes, we are using OpenProject started back from ChiliProject. )
Had the same problem here. But the OpenProject is running in a docker environment.
I could also solve the problem like described above.
History:
Initially the Openproject (6 or 7, don't remember) was installed natively in Debian 9 with MySQL. Before updating to Debian 10 I moved the Openproject to a docker instance (All-in-one container). Later when MySQL support was dropped by OpenProject, I migrated to PostgreSQL.
But I had to start a postgres docker image to get command line access to the PostgreSQL database (while the OpenProject docker image was stopped). I used the description "Restoring an OpenProject backup" from the docs to get there:
https://docs.openproject.org/installation-and-operations/operation/restoring/#restoring-a-dump
So starting a postgres docker image and start the psql:
Then on the psql promt:
Afterwards stopped the postgres container. And then the new OpenProject container (11.3.2 now) started and did the migration successfully.