Bug #1527

puppet fails to run if store facts wont compile

Added by Ohad Levy over 3 years ago. Updated about 2 years ago.

Status:Closed Start date:08/24/2008
Priority:High Due date:
Assignee:James Turnbull % Done:

0%

Category:-
Target version:0.25.0
Affected Puppet version:0.24.4 Branch:
Keywords:
Votes: 0

Description

It seems that if puppet has synced a fact which does not compile, puppet will not run. fixing the fact on the puppet server does not solve the problem and one needs to delete the copied fact on the client to get factsync working again.

I would expect puppet to sync the fact again in a case of fact execution problem and not to exit

History

Updated by AJ Christensen over 3 years ago

  • Status changed from Unreviewed to Needs More Information
  • Assignee set to Ohad Levy

Need more details, steps to reproduce.

Does this happen w/ 0.24.5?

Updated by Ohad Levy over 3 years ago

If a custom fact which does not compile has been copied to the client, puppetd will fail instead of trying to resync the fact again.

steps to reproduce, edit one of the custom facts (on the client, not on the server), and “break it”. try to run puppet and puppet will fail to load the fact and exit.

ideally, puppet should check if there is a new version of the fact in case that it cant run it.

Updated by Ohad Levy over 3 years ago

  • Assignee changed from Ohad Levy to Puppet Community

Updated by Luke Kanies over 3 years ago

To be clear, the problem here is that you’ve got code to produce a fact value, and that code is somehow invalid, which is cause Facter to fail, which is in turn causing puppetd to fail. Right?

In that case, there are two bugs — one in Puppet (puppetd should be resilient to failures in Facter) and one in Facter (it should be resilient to broken facts).

Either way, though, it would be hard to have this kind of failure result in an automatic resyncing of the fact code, and, really, that wouldn’t normally fix the problem — it just pulled down broken code, and it’s reasonable to assume that, seconds later, the code is still broken.

The only reasonable response here is to cache the facts locally and pass them back up again; or, with more recent versions, the server can use its cached facts for the client.

Updated by Ohad Levy over 3 years ago

luke wrote:

To be clear, the problem here is that you’ve got code to produce a fact value, and that code is somehow invalid, which is cause Facter to fail, which is in turn causing puppetd to fail. Right?

Right

In that case, there are two bugs — one in Puppet (puppetd should be resilient to failures in Facter) and one in Facter (it should be resilient to broken facts).

Either way, though, it would be hard to have this kind of failure result in an automatic resyncing of the fact code, and, really, that wouldn’t normally fix the problem — it just pulled down broken code, and it’s reasonable to assume that, seconds later, the code is still broken.

well, the main problem is that puppetd will never run again on this machine unless you manually remove the fact, so ideally, puppet should try to check once in a while if the fact has changed on the server.

The only reasonable response here is to cache the facts locally and pass them back up again; or, with more recent versions, the server can use its cached facts for the client.

I would suggest that puppet

do not fail when facter fails, worst case you get a compile error because of a missing fact.

puppet resync facts if facter fail (which is almost the same like today when you have factsync, e.g. puppet syncs the facts and reloaded after factsync).

I think that have a cache of facts is way too complex for this problem… and not needed, I don’t really wants my clients to compile something with broken facts, i just want an automated way to fix it if it happens (e.g. resync of the new good facts).

Updated by Luke Kanies over 3 years ago

ohadlevy wrote:

well, the main problem is that puppetd will never run again on this machine unless you manually remove the fact, so ideally, puppet should try to check once in a while if the fact has changed on the server.

Really? I would expect the fact syncing to still work, even if there are broken facts. That’s not the case right now?

I would suggest that puppet

do not fail when facter fails, worst case you get a compile error because of a missing fact.

I actually think the run probably should fail if Facter fails — I think most people would agree it’s better to fail than to compile with incomplete information.

puppet resync facts if facter fail (which is almost the same like today when you have factsync, e.g. puppet syncs the facts and reloaded after factsync).

We should be resyncing facts anyway, so I don’t see how this helps or hurts. We should resync facts on every run, regardless of whether they’re working or not.

Can you confirm that a broken fact causes factsync and/or pluginsync not to work any longer?

Updated by Luke Kanies over 3 years ago

  • Target version set to 0.25.0

I’d like to try to fix this for 0.25, but we need to come up with the “right” design – how should Puppet behave if it cannot read its facts?

Updated by Luke Kanies over 3 years ago

  • Status changed from Needs More Information to Accepted
  • Priority changed from Normal to High

Actually, I think the way to fix this is to iterate over each fact, rather than using Facter.to_hash – that way we can protect ourselves from any individual fact failing.

Should we use some kind of ‘failed’ value for those facts that fail, or should we just skip them, or what? Anyone concerned about getting a broken configuration if a fact is missing?

Updated by James Turnbull over 3 years ago

  • Assignee changed from Puppet Community to Andrew Shafer

Updated by Luke Kanies over 3 years ago

  • Assignee changed from Andrew Shafer to Luke Kanies

Updated by Luke Kanies over 3 years ago

  • Status changed from Accepted to Ready For Checkin
  • Assignee changed from Luke Kanies to James Turnbull

Fixed in the tickets/master/1527 branch in my repo.

There are a couple of other small fixes here, but the main thing is that exceptions are now thrown more often (e.g., when facts cannot be found by the server) but they’re also dealt with more successfully.

The system should now behave sanely, but in the process I’ve discovered a couple of other issues.

Updated by James Turnbull over 3 years ago

  • Status changed from Ready For Checkin to Closed

Pushed in branch master.

Also available in: Atom PDF