Bug #2899

puppetd checks client files recursively even if they aren't specified

Added by Josh Endries about 2 years ago. Updated about 1 year ago.

Status:Rejected Start date:12/08/2009
Priority:Normal Due date:
Assignee:- % Done:

0%
Category:fileserving
Target version:-
Affected Puppet version:0.25.1 Branch:
Keywords:
Votes: 1

Description

We stumbled upon a puppetd using 100% CPU and 12 GB of RAM a few days ago. We have a DB app in /usr/local/fedora and a dev was doing work on migrating to a new version and had many files in /usr/local/fedora/migration. We copy /usr/local/fedora recursively from puppetmaster, so it (something) does descend into the folder, but /usr/local/fedora/migration is not part of those files. Nevertheless, the client’s puppetd seemed to descend into it, probably doing checksums or something, and ran for possibly days eating up more and more RAM and constant high CPU. We worked around it by shutting puppetd down until the migration is finished.

This may be worked around by using an ignore statement, I haven’t tested that since this situation is temporary, but that requires you know where the files will be placed beforehand. This seems to suggest a DoS could be done against Puppet or the system pretty easily by putting a directory somewhere with tons of files in it, if Puppet copies the parent recursively. It also generally seems like a bad design, the client reading things in that the server doesn’t need a check for, but maybe isn’t easy to fix.

Anyway, I think it should be discussed or at least brought to attention. :)

I may be able to set up a test environment if necessary.

Related issues

related to Puppet - Bug #2929: File rucurse true, checksums all the files with md5 Closed 12/14/2009
related to Puppet - Feature #3170: Please re-introduce checksum => nosum for file resources Closed 02/09/2010

History

#1
Updated by Josh Endries about 2 years ago

Oh I should add, running puppetd in debug mode doesn’t show anything, it just sits there forever, chewing up memory and CPU with no output. I guess whatever process is it doesn’t spit out a debug message, or does so afterward.

#2
Updated by James Turnbull about 2 years ago

  • Category set to fileserving
  • Status changed from Unreviewed to Investigating
  • Assignee set to Markus Roberts

Markus – I think this will ring bells with you.

#3
Updated by Josh Endries about 2 years ago

Sorry, ignore this comment.

#4
Updated by Markus Roberts about 2 years ago

  • Assignee changed from Markus Roberts to Jesse Wolfe
  • Target version set to 0.25.3

#5
Updated by Markus Roberts about 2 years ago

  • Target version changed from 0.25.3 to 0.25.4

#6
Updated by Peter Meier about 2 years ago

I can confirm that behavior, that if I set a maillist to absent, it will grow as well. This seems to happen because maillist removes the archive directories etc. which can be pretty huge. But if I remove first the archive directories by hand and then run puppet it removes a maillist immediately and as it should.

Unfortunately I can confirm as well the behavior that if you run it it with @—debug@ it won’t tell you anything.

Somehow this is pretty ugly and imho should have higher prio…

#7
Updated by James Turnbull about 2 years ago

  • Target version changed from 0.25.4 to 0.25.5

#8
Updated by Mark Plaksin about 2 years ago

Possibly related to this: With 0.25.4, if I use tidy on a directory with lots of files (100k) puppet eats up lots of and lots of RAM. It was running for so long I didn’t let it finish. When I killed puppet it was using 800M of RAM.

#9
Updated by Brice Figureau almost 2 years ago

Josh Endries wrote:

We stumbled upon a puppetd using 100% CPU and 12 GB of RAM a few days ago. We have a DB app in /usr/local/fedora and a dev was doing work on migrating to a new version and had many files in /usr/local/fedora/migration. We copy /usr/local/fedora recursively from puppetmaster, so it (something) does descend into the folder, but /usr/local/fedora/migration is not part of those files. Nevertheless, the client’s puppetd seemed to descend into it, probably doing checksums or something, and ran for possibly days eating up more and more RAM and constant high CPU. We worked around it by shutting puppetd down until the migration is finished.

/usr/local/fedora/migration is below /usr/local/fedora. Using recurse => true, will recurse on both side (ie source and destination), so the behaviour you are observing is correct. Use: @recurse => remote@ to recurse only on the server side (ignoring everything that is already on your client).

You should also be aware that recursing locally on a directory involves md5 checksumming every file which is costly (see #2929).

#10
Updated by James Turnbull almost 2 years ago

  • Target version changed from 0.25.5 to 49

#11
Updated by Jesse Wolfe over 1 year ago

  • Status changed from Investigating to Rejected
  • Assignee deleted (Jesse Wolfe)

Brice’s reasoning seems sound, so I’m calling this not-a-bug. (also, the fix for #2929 may have helped the situation)

#12
Updated by James Turnbull about 1 year ago

  • Target version deleted (49)

Also available in: Atom PDF