Bug #3234
err: Could not retrieve catalog: undefined method `fact_merge' for nil:NilClass
| Status: | Investigating | Start date: | 02/22/2010 | |
|---|---|---|---|---|
| Priority: | Normal | Due date: | ||
| Assignee: |  Joe McDonagh |
% Done: | 0% |
|
| Category: | - | |||
| Target version: | - | |||
| Affected Puppet version: | 0.25.4 | Branch: | ||
| Keywords: | storedconfigs | |||
| Votes: | 12 |
Description
Ever since I upgraded to .25.4 I keep getting the error on new nodes until I restart the master
bash-4.0# puppetd -dt debug: Creating default schedules debug: Failed to load library 'shadow' for feature 'libshadow' debug: /Settings[/etc/puppet/puppet.conf]/Settings[main]/File[/var/puppet/lib]: Autorequiring File[/var/puppet] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/puppet/ssl/public_keys]: Autorequiring File[/var/puppet/ssl] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/puppet/ssl/private_keys]: Autorequiring File[/var/puppet/ssl] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/puppet/ssl/private_keys/somemachine.com.pem]: Autorequiring File[/var/puppet/ssl/private_keys] debug: /Settings[/etc/puppet/puppet.conf]/Settings[puppetd]/File[/etc/puppet/puppet.conf]: Autorequiring File[/etc/puppet] debug: /Settings[/etc/puppet/puppet.conf]/Settings[main]/File[/var/puppet/state]: Autorequiring File[/var/puppet] debug: /Settings[/etc/puppet/puppet.conf]/Settings[main]/File[/var/puppet/facts]: Autorequiring File[/var/puppet] debug: /Settings[/etc/puppet/puppet.conf]/Settings[main]/File[/var/puppet/plugins]: Autorequiring File[/var/puppet] debug: /Settings[/etc/puppet/puppet.conf]/Settings[main]/File[/var/puppet/ssl]: Autorequiring File[/var/puppet] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/puppet/ssl/certs]: Autorequiring File[/var/puppet/ssl] debug: /Settings[/etc/puppet/puppet.conf]/Settings[main]/File[/etc/puppet/namespaceauth.conf]: Autorequiring File[/etc/puppet] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/puppet/ssl/private]: Autorequiring File[/var/puppet/ssl] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/puppet/ssl/certs/ca.pem]: Autorequiring File[/var/puppet/ssl/certs] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/puppet/ssl/somemachine.com.pem]: Autorequiring File[/var/puppet/ssl] debug: Finishing transaction 1053198210 with 0 changes debug: Calling puppetca.getcert warning: peer certificate won't be verified in this SSL session notice: Got signed certificate debug: Retrieved facts in 2.37 seconds debug: Retrieving catalog debug: Calling puppetmaster.getconfig err: Could not retrieve catalog: undefined method `fact_merge' for nil:NilClass warning: Not using cache on failed catalog
Weird? So then I restart the master.
bash-4.0# puppetd -dt debug: Creating default schedules debug: Failed to load library 'shadow' for feature 'libshadow' debug: /Settings[/etc/puppet/puppet.conf]/Settings[main]/File[/var/puppet/plugins]: Autorequiring File[/var/puppet] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/puppet/ssl/public_keys]: Autorequiring File[/var/puppet/ssl] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/puppet/ssl/private_keys]: Autorequiring File[/var/puppet/ssl] debug: /Settings[/etc/puppet/puppet.conf]/Settings[main]/File[/var/puppet/state]: Autorequiring File[/var/puppet] debug: /Settings[/etc/puppet/puppet.conf]/Settings[main]/File[/var/puppet/facts]: Autorequiring File[/var/puppet] debug: /Settings[/etc/puppet/puppet.conf]/Settings[main]/File[/var/puppet/lib]: Autorequiring File[/var/puppet] debug: /Settings[/etc/puppet/puppet.conf]/Settings[main]/File[/var/puppet/ssl]: Autorequiring File[/var/puppet] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/puppet/ssl/somemachine.com.pem]: Autorequiring File[/var/puppet/ssl] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/puppet/ssl/certs/somemachine.com.pem]: Autorequiring File[/var/puppet/ssl/certs] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/puppet/ssl/certs]: Autorequiring File[/var/puppet/ssl] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/puppet/ssl/private]: Autorequiring File[/var/puppet/ssl] debug: /Settings[/etc/puppet/puppet.conf]/Settings[puppetd]/File[/etc/puppet/puppet.conf]: Autorequiring File[/etc/puppet] debug: /Settings[/etc/puppet/puppet.conf]/Settings[main]/File[/etc/puppet/namespaceauth.conf]: Autorequiring File[/etc/puppet] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/puppet/ssl/certs/ca.pem]: Autorequiring File[/var/puppet/ssl/certs] debug: /Settings[/etc/puppet/puppet.conf]/Settings[ssl]/File[/var/puppet/ssl/private_keys/somemachine.com.pem]: Autorequiring File[/var/puppet/ssl/private_keys] debug: Finishing transaction -1015757078 with 0 changes debug: Retrieved facts in 2.03 seconds debug: Retrieving catalog debug: Calling puppetmaster.getconfig err: Could not retrieve catalog: Could not parse for environment development: Could not find file /etc/puppet/manifests/site.pp warning: Not using cache on failed catalog
OOPS forgot this is OpenBSD so it’s on a .24.8 or something and defaulting to the dev environment. However this has been happening even on .25.4 clients, which every Linux machine I have is on. Then I add —environment production and it runs fine except for the mounts without paths problem when you use .24.8 against .25 masters.
Please note that I have stored configs running on my master. This has happened on the four or so machines I have provisioned since I upgraded to .25.4. This is the first OpenBSD one, but it seems like that is an insignificant detail.
My process for new nodes is:
- Generate a key on my puppet dev server
- Add node def to nodes.pp in prod environment
- Run “cap deploy” which is basically just a distributed checkout, then rsync to /etc/puppet from the checkout
History
Updated by Joe McDonagh almost 2 years ago
Hey I forgot to mention in my detail of how I provision, the part where I copy the keys to the new server…
Updated by James Turnbull almost 2 years ago
- Status changed from Unreviewed to Investigating
- Assignee set to Markus Roberts
Markus – any ideas?
Updated by Joe McDonagh almost 2 years ago
No Love for this bug? Just deployed two new nodes today and got the same problem. Anything I can submit to help? They were Linux nodes by the way, running .25.4.
Updated by Michiel Fokke almost 2 years ago
I encountered the same error after I changed my default distribution from Debian Lenny to Debian Squeeze. It turned out that facter could not fill the lsb* facts due to a missing dependency with a package ‘lsb-release’. After I installed the named package, facter could fill all facts and the error disappeared.
Maybe a similar problem is causing this problem in OpenBSD.
Updated by Joe McDonagh almost 2 years ago
Well it’s happening on Ubuntu and OpenBSD, the error goes away when I restart the master, which is an Ubuntu machine.
Updated by Markus Roberts almost 2 years ago
- Status changed from Investigating to Needs More Information
- Assignee changed from Markus Roberts to Joe McDonagh
Joe —
Does the problem persist if you install lsb-release on the client (without restarting the master)?
— Markus
Updated by Joe McDonagh almost 2 years ago
lsb-release is standard on Hardy Server afaik. Obviously, the package is not available on OpenBSD. To be clear, it is happening across nodes of each OS, only new nodes get this problem, and my method of generating a key on the master, then pushing it out to the client is probably not the same method most people take (signing a request from the node).
I can probably reproduce this pretty easily. I just need a little time to spin up a new VM.
Updated by Robin Bowes almost 2 years ago
I’ve just run into this on CentOS 5.4
When I first provision a new machine (with cobbler – puppet running in the post-install) I get the 400 error. Running “puppetd —test” on the new machine after it has re-booted shows the same thing.
If I restart puppetmaster, “puppetd —test” works, and if I then re-provision the machine with cobbler, it works OK too.
I have 42 machines to provision in the next few days – this is rather annoying. :(
Updated by Joe McDonagh almost 2 years ago
Thank god someone else is finally running into this. Does cobbler do server-side cert generation??? Maybe that has something to do with it… I generate mine with a small shell script on a puppet dev server, and push them out via a separate small script.
Restarting the master fixes it for you Robin?
Updated by Scott Briggs almost 2 years ago
I just ran into this same problem as well when I tried deploying 2 new nodes. All the servers are using puppet 0.25.1 and running Ubuntu 8.04.
Restarting the masters solved this problem.
Updated by Markus Roberts over 1 year ago
- Status changed from Needs More Information to Investigating
Updated by Joe McDonagh over 1 year ago
FYI, I re-encounter this bug if I remove a node from my stored config db and go to re-run puppetd on the node.
Updated by Joe McDonagh over 1 year ago
- Affected Puppet version changed from 0.25.4 to 2.6.0
- Keywords set to storedconfigs
Hi guys, I upgraded to 2.6 and still have this issue. Any new node that doesn’t have info in stored configs throws that exception. Upon restarting the master, everything is fine. Is there something I can do to get more info to help you track this down?
Updated by Joe McDonagh over 1 year ago
I was just trying to work on this bug, and it might not be reproducible since I moved my masters to Ubuntu 10.04 LTS. Possibly a rails->mysql thing on 8.04?
Updated by bram vogelaar over 1 year ago
i have the same problem with ubuntu 10.04.1 lts puppetmaster and puppets when i start a new node that is not in the database yet i get the error above, the lbs is correctly show in the facter output if i turn off “storeconfigs” or add the node to the database it works ok
i use the debian sid puppet debs to install the puppet and puppet-common packages
Updated by Jesse Wolfe over 1 year ago
- Affected Puppet version changed from 2.6.0 to 0.25.4
I’m settings this version back to when the issue first appeared so it is not assumed to be a new issue.
Updated by Joe McDonagh over 1 year ago
Sorry; was under the impression that the affected version field should be updated if a bug persists.
Updated by Jesse Wolfe over 1 year ago
No problem, Joe, it’s new policy. In general, though, the fact that the ticket isn’t “closed” should be enough to know that it’s still open.
Updated by Joe McDonagh over 1 year ago
I just wanted to confirm this still exists in Ubuntu 10.04. I must have been running a pre-existing node during testing by accident.
Updated by Dane Gardner about 1 year ago
I can vouch that this issue exists in a Puppet master version 2.6.2 running on CentOS 5.5 with ruby packages from the EPEL repo and MySQL from the CentOS repos. Clients running on both Ubuntu and CentOS/RHEL. As Joe finds, new nodes that are added to the StoreConfigs database require a restart of the master.
Updated by Andrew Gaffney about 1 year ago
I’m encountering this one with 2.6.4 (master and agents) on CentOS 5.5. As part of my upgrade from 0.25.5 (where I didn’t have this issue), I also updated rails/active* to 2.3.10 from 2.2.2.
Updated by Andrew Gaffney about 1 year ago
When I downgrade rails/action* back to 2.2.2, this problem seems to go away.
Updated by Joe McDonagh about 1 year ago
ii rails 2.2.3-2 MVC ruby based framework geared for web application development
Updated by Joe McDonagh about 1 year ago
I didn’t mean to post that without a comment, I meant to add, looks like my rails is already 2.2…
Updated by Joe McDonagh 10 months ago
I have just completed a 2.6.7 rollout, along with the migration to passenger vs mod_proxy_balancer with mongrels. The bug appears to be fixed by one of these changes. I am guessing the move to passenger is what did it.
Updated by jared jennings 7 months ago
I think I’m seeing this one on RHEL 6.1 with Puppet 2.7.1 plus some patches (#8140), using Webrick. Maybe it’s time to do the Passenger thing.
Updated by Joe McDonagh 7 months ago
This bug completely disappeared after moving to 2.6.7 behind passenger.
Updated by vagn scott 6 months ago
This happened to me today, puppet 2.7.2rc2 master and 2.6.2 client.
Upgraded to 2.7.2rc2 client and it still happened.
Restarting master didn’t help.
Made it thin_storedconfigs and the problem went away.
Updated by Trey Dockendorf 6 months ago
I just experienced this problem. Right after enabling storedconfigs all my hosts began to fail with the following message
err Could not retrieve catalog from remote server: Error 400 on SERVER: undefined method `fact_merge' for nil:NilClass
err Could not retrieve catalog; skipping run Puppet
notice Using cached catalog
My server is running 2.6.9 and the clients are a mix of 2.6.8 and 2.6.9.
The puppetmaster has this in the logs..please let me know what other information may be useful
undefined method `fact_merge' for nil:NilClass
Updated by Joe Hillenbrand 5 months ago
Just had this issue today on a Debian Lenny 2.6.2-4 Master and a Squeeze 2.6.2-5 Node.
Restarting the master fixed the issue.
Updated by Nick Lewis 2 months ago
I haven’t tested my theory, yet, but it looks like this is happening because we set active_record as the cache terminus for nodes, then try to retrieve a node object to compile. We ask for it out of the database, it’s not there yet (because this is our first compile of the node), which should be fine. However, we don’t check whether we actually got one back in the terminus before calling fact_merge on it. Really we should just bail out if we couldn’t find one, and let the main terminus do its job.