Puppet

FYI – on my environment, I do not see this. All home compiled. Most interesting would be the later version of passenger:

• Solaris 10 Update 8
• puppet (2.6.3rc3)
• ruby (1.8.7-p249)
• apache (2.2.15)
• fastthread (1.0.7)
• passenger (2.2.14)
• rack (1.1.0)
• rake (0.8.7)

I also tried upgrading passenger to 2.2.14, and also tried upgrading ruby to 1.8.6, could reproduce in both cases

I am having the same behavior with a stack similar to Dan’s:

• RHEL 5.5
• Puppet 2.6.1
• Ruby 1.8.5
• Passenger 2.2.15
• Rack 1.1.0
• Apache 2.2.3

Could it be the older version of Apache?

Reproduced with edp version 0.1 Centos x86_64 CentOS release 5.5 (Final) 2.6.18-194.el5 x86_64 GNU/Linux

I’ve tested and re-tested this and, despite recent discussions to the contrary; the problem appears repeatable on edp, including when taking “filetimeout = 15” into account. The problem is not 100% repeatable each test run. Will research further.

Reproduced on:

• Ubuntu LTS 10.04 Lucid
• Ruby 1.8.7.249-2
• Puppet 2.6.7
• Facter 1.5.9
• Apache 2.2.14-5ubuntu8.4
• Passenger 2.2.7debian-1
• Rails 2.2.3-2
• Rack 1.1.0-3

I’ve isolated the problem further.

If you have a custom environment defined, this problem does not occur when puppeting against that environment.

If you do any of the following, the problem is reproducible:

• Do not specify an environment
• Specify the environment as “production” (the default) without [production] being defined in puppet.conf on the master
• Specify the environment as “production” (the default) with [production] defined in puppet.conf on the master

then the problem is reproducible.

Ditto here.

Ubuntu LTS 10.04 Lucid
Ruby 1.8.7.249
Puppet 2.6.7
Facter 1.5.8
Apache 2.2.14-5ubuntu8.4
Passenger 2.2.7debian-1

The root cause of this problem is actually that the puppet master uses a current cache of the environment, which includes the data parsed from the manifests, but fails to update and manage that cache effectively.

Specifically, we flush that cache after the compile of a catalog is complete, but before we serialize it and send it to the client. This was done to satisfy the precondition of the compile function, that it would refresh the available type data for the compile. Which, indeed, is a weak way to ensure that this precondition was satisfied.

I suspect that it was also constructed this way because it would result (theoretically) in the environment being parsed at the start of compile, but would then stay consistent until the next compile run; it would try and avoid an update to a file in the middle of a compile run resulting in an incoherent set of definitions. (Note: it would absolutely not close the race, just reduce it in scope by only doing this once.)

When we changed to putting more data in the catalog, to reduce the need for type and provider code on the client during catalog manipulation, we actually started using the known_resource_types from the environment while we serialized the catalog to send to the client. This happened after the cache flush, and resulted in the cache being renewed with a new environment (and a second parse of the files) during serialization.

This, in turn, led to the precondition no longer being satisfied (although no one noticed because it was never documented or asserted as a precondition of compile). We would now parse changes from manifests only when we serialized the catalog, not when we started to compile – and the previous set of changes would leak over. Which, in turn, resulted in our taking two passes to notice the change; our sequence was that we would use the previous cache to compile, flush it, reparse in the serialization pass, then next time around would have the right input. Without looking at exact and correlated agent/master traces this looked to be doing the right things in the right places.

Further details: after thinking for a while, and checking the documentation, it has become clear why this problem is only evident under Passenger, and not under Webrick and similar servers. It circles around the thread cache, and the way these servers handle requests.

Specifically, Passenger doesn’t do threads, so every HTTP request lands in the same Ruby (green) thread. This means that the cache problem would manifest: because we used the same thread for multiple requests, the bad cache flush was exposed, and we did the wrong thing.

Webrick, and the other application servers we tested, are threaded. Specifically, they create a new thread for the request no matter what. This disguised the caching bug around the threads, both when running on small scale in production, and during our testing where we almost exclusively use the Webrick or Morgrel servers.

So, this was a constant bug, but was disguised by the runtime environment in most circumstances, leading to it appearing to be Passenger related.

Daniel, I’d like us to close off this bug and set up another one for the more complex solution. Is that possible?

For the record, this was also fixed in 2.7.3.