Puppet

I can also add my 2c, puppetmaster uses huge amount of memory on a 32bit platform as well, currently a puppetmaster with about 150 clients, would need to be restarted every two or three days as each puppetmaster process crosses the 2gb usage.
using a tool like monit can prevent puppet from abusing memory, however, I would prefer to find the root cause of the memory abuse (file server operations?)

I'll be stylin' and profilin'

A data point that may help track down where the issue lies.

We run puppetmaster in a VMware server 32-bit CentOS 5 VM, with 384 MB of memory allocated - tiny, but perfectly functional.

We split puppetmaster into two (I'm the D. Hill referenced in the ticket description) - one handling file serving, the other handling manifest serving. The file serving puppetmaster holds steady at 54M VIRT, 39M RES. The manifest serving puppetmaster hold steady at 74M VIRT, 55M RES until storeconfigs is enabled, with a backing server of PostgreSQL 8.1.11 (provided in the base CentOS 5 installation).

Once storeconfigs is enabled, the memory usage starts climbing, almost as if there's a leak. Once the VM starts swapping, we have no choice but to stop puppetmaster, wait a few minutes for the load to come back to a sensible level, and start puppetmaster again. Rinse, repeat.

I've checked file handles with lsof, and the number seems to hold steady around 50 - 60 files, whether the memory usage is 50 MB or 350 MB.

I found and fixed one leak in puppetd.

Puppetd was leaking report objects into an array.

I was running puppetd on one minute intervals and it grew about 20 MB over 8 hours.

Now running on 30 runintervals it has grown ~160K in about the same period, but it isn't gaining Ruby objects.

The patch is available from my github (littleidea) in the report_leak branch.

I don't want to close this issue, but I do want to give people something to work with.

I made extensive use of the memory_profiler.rb from: http://reductivelabs.com/trac/puppet/ticket/1131
(It will basically start a separate thread that keeps an inventory of ObjectSpace prints an audit to a file every so often. Make sure to add the 'start' after puppet daemonizes)

This told me I was leaking Puppet:Transaction::Reports.

I modified it a bit as I went but I got mileage doing stuff like this to mark when the new Object was created:

if o.class Puppet::Transaction::Report and !report_ids.include?(o.object_id)
file.puts('New report')
file.puts(o.inspect)
report_ids << o.object_id
end

And this to figure out what the structure of the whatever was holding it
if o.inspect =~ /Puppet::Transaction::Report/ and (o.class Hash or o.class Array)
file.puts(o.inspect)
end

I used this in conjunction with some basic dtrace scripts, mainly of the form:

sudo dtrace -s find_sync.d -p &lt;pid#&gt;

where find_sync.d looks like
ruby$target:::object-create-start
/copyinstr(arg0) "Sync"/ {
printf("%s was allocated in file %s, line number %d\n", copyinstr(arg0), copyinstr(arg1), arg2);
}

That will tell you the line number and file where the Sync objects get created, as an entry point to start looking at code.

I'll keep revisiting performance and memory problems, but I have to balance that with other work. I can solve problems much faster if I have a manifest or setup that can reliably reproduce the issue. Alternatively, if you are having memory problems try using the memory_profiler and dtrace with your setup.

I didn't look at storeconfigs yet, but I think there are more perf problems their than just memory.

If anyone in the community has strong dtrace fu, or learns a new trick share with the list please. The ruby probes are somewhat simplistic and I think we probably need to get deeper knowledge to solve some of the issues.

Also, I have a relatively simple setup right now, but I'm not getting growth in the puppetmasterd.

If someone can give me a setup that consistently produces the problems, I'll figure it out much faster.

help me help you :/

Oh yeah, when is Linux getting something like dtrace?

andrew wrote:

Also, I have a relatively simple setup right now, but I'm not getting growth in the puppetmasterd.

If someone can give me a setup that consistently produces the problems, I'll figure it out much faster.

help me help you :/

Hello,

I've got a similar problem, and I'd gladly provide you will all the required files. The only Problem, even though I compressed them with bzip2. The Package is 44M. I'm using the version shipped with Ubuntu 8.04: 2.24.4.

The Package contains all configuration files as well as the data files I want puppet to distribute: www.alexanderschaber.de/puppet.tar.bz2 (It will be fully uploaded in about 10 minutes)

Correcting .. it's 36MB .. ah and btw.: it's fully uploaded now..

I'm getting significant memory usage on 0.24.5 on puppetd clients that are sitting around waiting for the CA to sign their cert. I'm not sure if this is the same issue as reported but thought it worth mentioning. One machine running Debian Etch has had puppetd running as a daemon for only 5 days and it has already swallowed a significant chunk of memory:

# ps u -C ruby
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     17715  9.6 60.9 750952 626340 ?       Dsl  Sep12 689:18 ruby /usr/sbin/puppetd -w 0

about 600MB! I'm also finding puppetd tends to use a lot of CPU time while waiting for the CA to sign the cert, and attempts to talk to the puppetmaster a lot more frequently than it should... but perhaps that's a story for another ticket.