Puppet Enterprise (Public)

Had a weird one today. Upgraded PE from 2.0.0 to 2.0.2 only to find the Console database updates failing:

root@mgrl001:/opt/puppet/share/puppet-dashboard# service pe-puppet-dashboard-workers restart
Stopping pe-dashboard-workers: Database isn't the current migration version: expected 20111118204121, got 20110818164856
You must either run 'rake db:migrate' or set environmental variable NO_MIGRATION_CHECK
[FAILED]
Starting pe-dashboard-workers: Database isn't the current migration version: expected 20111118204121, got 20110818164856
You must either run 'rake db:migrate' or set environmental variable NO_MIGRATION_CHECK
[FAILED]
db:migrate fails:
root@mgrl001:/opt/puppet/share/puppet-dashboard# rake db:migrate
(in /opt/puppet/share/puppet-dashboard)
An error has occurred, all later migrations canceled:
Mysql::Error: Cannot add or update a child row: a foreign key constraint fails (`console/#sql-1292_386`, CONSTRAINT `fk_reports_node_id` FOREIGN KEY (`node_id`) REFERENCES `nodes` (`id`) ON DELETE CASCADE): ALTER TABLE reports ADD CONSTRAINT fk_reports_node_id FOREIGN KEY (node_id) REFERENCES nodes(id) ON DELETE CASCADE;

On investigation it appeared we had a ‘hanging’ node_id in the reports DB. Which, when deleted, solved the upgrade issue:

mysql> select node_id from reports where node_id NOT IN (select id from nodes);
+---------+
| node_id |
+---------+
|      19 | 
+---------+
1 row in set (0.00 sec)
mysql> delete from reports where node_id='19';
Query OK, 1 row affected (0.01 sec)
rake db:migrate
[snip]
==  AddReportForeignKeyConstraints: migrated (20.4046s) =======================

I’m raising this ticket more as an awareness thing, in case it happens with other customers.