Puppet

Oliver Hookins wrote:

Quite frequently there will be cases where the providers think they have done the right thing and report success even though the end result is not successful. This results in continual runs where there are successful changes but the overall outcome is the same – the system state is not what you want it to be.

I assume you mean things like a service where a call to /etc/init.d/service status returns 0, even though the service failed to start? (Just to make sure we are looking at the same problem. :)

The unfortunate reality is that providers do have bugs and the number of possible situations a provider can encounter is infinite. No amount of combinatorial testing will uncover every possibilty, nor will you be able to code the providers to test for all possible problems they might encounter (e.g. files being set immutable in the filesystem would probably cause problems for most providers, silently).

Given this is the case, but we still want to be aware of the distinction between a failure of a provider to do its job versus a system in constant entropy, reporting back the success of a provider’s attempted changes fills that gap in functionality. The former case might be something like acpid which cannot be started inside an OpenVZ container (even if the init script reports it has been started), and the latter might be a an angry sysadmin who logs into the system and disables acpid – before the next Puppet run.

There are just too many buggy pieces of software to adequately handle, but if Puppet were to implement this feature we’d at least get a more reliable insight into bugs vs natural/unnatural system entropy. Does that make more sense?

Actually the main use case I had for it was already in my summary. When using Puppet as a deployment tool you really want to be 100% sure you have succeeded on the current machine before moving on to the next. If you accept the end status of a Puppet run as an indication of the success of all of the providers you can easily take down an entire cluster of machines serving an application simply because the init script of your application server incorrectly returned 0 when it shouldn’t have.

Doing some more obsessive checks gives any deployment system wrapping around Puppet a better view of what has actually been achieved on each system and opens up the possibility to stop a potentially devastating upgrade or deployment before it takes all the machines down.

No, most (all?) init scripts return a status based on being able to spawn a daemon process successfully, when you call ‘start’. Some probably just assume it has been started correctly if they can retrieve the PID of the child process successfully.

‘Status’ on the other hand should be testing (at very least) whether a process with a pid found in the pid file exists, and quite possibly will attempt to verify some basic IPC with that process to verify it is operating.

In any case, the execution paths for starting and checking the status of a running process are entirely different, and this is the difference I want to expose.