Puppet

Hi Markus,

We’re actually not using external nodes, just regex in nodes.pp. re:IPv6, we have a bit of puppet code that disables IPv6 on the initial server provision, and we have not experienced any problems with our IPv6 disabler working in Debian5 thus far.

I ran the same test with —trace and —debug enabled for both puppetmaster and puppetd and found nothing of interest in the logs. Is there anyway to find out the query that activerecord sends to postgres? The fact that the table data becomes misalligned feels like a Rails bug to me (but what do i know?!)

Interesting. I’m surprised that running the puppetmaster with —trace & —debug did not produce anything about the “err: Could not run Puppet configuration client: Parameter ip failed: Invalid IP address”; there are only three places that message could have originated, and each of them should have produced a stack trace on the console if run with —trace.

Are you watching the console output (e.g. —no-daemonize) or just the logs?

— Markus

Thanks Markus, you are correct I was watching the logs. puppetmasterd —trace —no-damonize produces a lot more output. I will update this bug shortly with the new information.

puppetd —debug —trace http://pastebin.com/f76447bee

puppetmaster —debug —trace http://pastebin.com/f6db3d10c

However, I don’t see anything specific to the database node insert after the catalog is compiled:

notice: Compiled catalog for nagios2.domain.com in 4.24 seconds info: Caching catalog for nagios2.domain.com debug: Searched for resources in 0.00 seconds debug: Searched for resource params and tags in 0.00 seconds debug: Resource removal in 0.00 seconds debug: Resource merger in 0.00 seconds debug: Added resources(parameters) in 1.49 seconds debug: Added resources(tags) in 1.46 seconds debug: Added resources(initialization) in 0.49 seconds debug: Resource addition in 17.93 seconds debug: Performed resource comparison in 17.93 seconds debug: Saved catalog to database in 17.94 seconds

Also, do you know which facts are producing the “xx.xx.210.11x” values? I’ve not seen those before (which isn’t to say they might not be normal). It might help to do:

select * from fact_values,fact_names as hn where "host_id"= 40 and "fact_name_id" = hn.id order by "fact_name_id" asc;

Hi Marcus,

I’ll try that change and report back shortly. Sorry for misleading you about the xx.xx IPs; I intended to blank out proprietary IP information from the ticket. (the real IPs from facter and in the database are actually correct)

The only odd thing about the facts collected from this node is that when facts are initially inserted into fact_values, the data is misaligned.

nagios2:~# vim /usr/lib/ruby/1.8/puppet/type/host.rb (made change as requested)

nagios2:~# puppetd —test info: Retrieving plugin err: /File[/var/lib/puppet/lib]: Failed to retrieve current state of resource: Could not retrieve information from source(s) puppet://mgmt.domain.com/plugins info: Caching catalog for nagios2.domain.com err: Could not run Puppet configuration client: Parameter ip failed: Invalid IP address: “”

(subsequent runs return the same blank value for ip address)

Results from your suggested sql query:

working host: http://img.skitch.com/20091221-khxge4nu6b36ecihfse78pkppw.jpg

nagios2 broken host: http://img.skitch.com/20091221-1g5dby5iqij4drhefddqydwbqx.jpg

Sorry for misleading you about the xx.xx IPs; I intended to blank out proprietary IP information from the ticket. (the real IPs from facter and in the database are actually correct)

Ah, that makes much more sense. I was really scratching my head over those “x"s.

The only odd thing about the facts collected from this node is that when facts are initially inserted into fact_values, the data is misaligned.

Can you clarify what you mean by “misaligned”? Also, I note that the nagios2 node appears to have 7 fewer facts.

’m going to look at the query results now. I’m suspecting that the broken one are just dropping a value somewhere.

sorry I know it makes it a bit hard to troubleshoot this when IP and hostname information is redacted, i’ve included output from facter and ifconfig on the troubled nagios2 node:

facter: http://img.skitch.com/20091221-d8uk1xmbdjai546qc79i7rd1u3.jpg

ifconfig: http://img.skitch.com/20091221-gaaryqmkexsepesyui5dtsskih.jpg

So the non-working one is missing macaddress_eth1, netmask_eth1, and, most interestingly, ipaddress_eth1, but has four additional values (macaddress_dummy, processor1, processor2, processor3). I’d suspect that ipaddress_eth1 is the problem.

It would seem very odd to me if these changes were being made in the store config process. How is ipaddress_eth1 being set in the working case?

Here’s a thought: could it be that this box doesn’t have an eth1 but something in your manifest is assuming that there will be one and trying to use its ip address?

yes that’s a very good possibility! I’ll check the catalog yaml and the manifests assigned to this node and respond in a moment…

I’ve added this ugly hack to my site.pp for now:

case $fqdn { ‘nagios2.domain.com’: {

$ipaddress_eth1 = $ipaddress_eth0

} }

This is a terrible hack, but it has fixed the problem until we can rewrite the various manifests assigned to this node which reference the non-existent $ipaddress_eth1. We’re using exported resources that reference $ipaddress_eth1 as well as referencing that variable from within .erb files.

so it seems that the bug can be potentially summed up with:

if you realized exported resources which contain variables based on facts that are not present on the node where the exported resources were collected from, puppetmaster’s database becomes inconsistent.

Just to clarify, was the address being set before, by the explicit node, and (if so) would it make more sense to have the external node tool do “the same thing”?

In any case, this sounds like a site problem, not a puppet bug, correct?

Rob Terhaar wrote:

so it seems that the bug can be potentially summed up with:

if you realized exported resources which contain variables based on facts that are not present on the node where the exported resources were collected from, puppetmaster’s database becomes inconsistent.

I think it’s simpler than that – if you attempt to export an invalid resource, then it only fails on collection rather than export.

I think this is a real bug, but the bug is the above – invalid resources should fail on export, rather than on collection. However, in many cases this isn’t actually possible – we can’t do much resource validation until it reaches the client, which in the case of exported resources can take a while and seem terribly disconnected.

Markus Roberts wrote:

In this case it wasn’t so much a question of the resources being invalid as of them depending on variables that were never set. That would pose the same problems regardless of whether they were exported or not.

I think it should be opened as a new bug, though — validation should happen on export rather than collect.