Puppet

Markus Roberts wrote:

I can reproduce this, at least in part. It looks like there are minor differences between 1.8.7 and earlier versions.

Excellent! Anything I can do to help troubleshoot?

So one finding: it’s not related to #2598, and the yaml, though odd looking, is fine.

To see this, go into irb and type

class Puppet; class Node; class Facts; attr_accessor :expiration,:name,:values; end; end; end
x = YAML.load(File.read('/var/puppet/yaml/facts/YOURNODENAME.yaml'))
x.values

And you can see that it loads fine.

Josh —

Can you run facter stand-alone and post the results?

— Markus

Luke Kanies wrote:

Josh, I have two questions:

• What is the path if you don’t specify it? E.g., if you do puppetd —configprint path, what do you get?

# puppetd --configprint path
none
#

• You didn’t actually run puppetd for 24 hours until you fixed the path, right? I mean, if it’s broken you wouldn’t let it run that long, so you probably fixed it quickly, and once it was working let it run, and only then saw the problem. If this is the case, then I expect that the provider/path and Facter issues are orthogonal.

I allowed puppetd to run long enough to encounter the provider errors several times to verify that I could reproduce it. E.g., restart, run for 24 hours, check for errors. Digging around in the code and looking at tickets for similar issues made me think that the provider issue had to do with the path that puppetd was using, so, at that point, I added the path to puppet.conf. Then I started to get the template errors, so I restarted puppetd and watched it to see if it was reproducible. It was, so I opened this ticket.

I’m not sure whether or not the two errors have anything to do with each other, but it’s slightly suspicious that they both crop up after puppetd has been running for about the same length of time.

Markus Roberts wrote:

Josh —

Can you run facter stand-alone and post the results?

— Markus

Yes:

domain => foo.com
facterversion => 1.5.6
fqdn => puppetm.foo.com
hardwareisa => sparc
hardwaremodel => sun4v
hostname => puppetm
id => root
interfaces => lo0_17,e1000g0_5,e1000g201000_5
ipaddress => 10.13.23.49
ipaddress_e1000g0_5 => 10.8.23.49
ipaddress_e1000g201000_5 => 10.13.23.49
ipaddress_lo0_17 => 127.0.0.1
is_virtual => false
kernel => SunOS
kernelmajversion => Generic_127111-02
kernelrelease => 5.10
kernelversion => Generic_127111-02
netmask_e1000g0_5 => 255.0.0.0
netmask_e1000g201000_5 => 255.255.0.0
netmask_lo0_17 => 255.0.0.0
network_e1000g0_5 => 10.0.0.0
network_e1000g201000_5 => 10.13.0.0
network_lo0_17 => 127.0.0.0
operatingsystem => Solaris
operatingsystemrelease => 5.10
ps => ps -ef
puppetversion => 0.25.0
rubysitedir => /opt/ruby/lib/ruby/site_ruby/1.8
rubyversion => 1.8.7
sshdsakey => AAAAB3NzaC1kc3MAAACBAMAIzIP2T9LiVqKV/iyk...
sshrsakey => AAAAB3NzaC1yc2EAAAABIwAAAIEAqJ4c0t/M2w...
timezone => PDT
uniqueid => 84abda4c
uptime => 7 day
virtual => zone

My first, possibly naive attempt to reproduce this failed. Letting a 25.0 puppetmasterd/puppetd run for over 24hrs did not do anything interesting.

Well over 100 hours and nothing odd. I’m starting to look for various hypothetical ancillary causes (memory leaks, protocol issues, etc.). Any suggestions would be welcome.

Markus Roberts wrote:

Well over 100 hours and nothing odd. I’m starting to look for various hypothetical ancillary causes (memory leaks, protocol issues, etc.). Any suggestions would be welcome.

I’m going to run my puppet master with —debug and see if anything shows up there. In the meantime, where in the serialization code were you looking for problems?

I’m just poking. The idea was that something might change over time or by chance that would cause the server’s parsing of the facts to terminate prematurely and give the appearance that facts had been omitted. It was the result of a “how could this possibly happen” brainstorming session and not the result of any direct evidence.

Here’s that the puppetmaster logs:

Thu Sep 24 10:36:34 -0700 2009 Puppet (info): Expiring the node cache of puppetm.foo.com
Thu Sep 24 10:36:34 -0700 2009 Puppet (info): Not using expired node for puppetm.foo.com from cache; expired at Thu Sep 24 10:35:34 -0700 2009
Thu Sep 24 10:36:34 -0700 2009 Puppet (info): Caching node for puppetm.foo.com
Thu Sep 24 10:36:34 -0700 2009 Puppet (warning): Host is missing hostname and/or domain: puppetm.foo.com
Thu Sep 24 10:36:34 -0700 2009 Puppet (err): Failed to parse template templatetest/test.erb: Could not find value for 'hostname' at /etc/puppet/modules/templatetest/manifests/init.pp:3 on node puppetm.foo.com
Thu Sep 24 10:36:34 -0700 2009 Puppet (err): Failed to parse template templatetest/test.erb: Could not find value for 'hostname' at /etc/puppet/modules/templatetest/manifests/init.pp:3 on node puppetm.foo.com

I’m afraid that’s not very helpful. However, I can say for sure that this is something going wrong with puppetd, as restarting the puppetmaster doesn’t affect the problem at all.

Okay, it turns out I was shooting myself in the foot on this one. I had a fact that I wrote six-plus months ago that appended an application’s bin directory to ENV[‘PATH’]. Once ENV[‘PATH’] reached a certain length, fact resolution stopped working properly. This same problem was causing nil:nilClass errors on 0.24.8.