Puppet

I can also add my 2c, puppetmaster uses huge amount of memory on a 32bit platform as well, currently a puppetmaster with about 150 clients, would need to be restarted every two or three days as each puppetmaster process crosses the 2gb usage. using a tool like monit can prevent puppet from abusing memory, however, I would prefer to find the root cause of the memory abuse (file server operations?)

I’ll be stylin' and profilin'

A data point that may help track down where the issue lies.

We run puppetmaster in a VMware server 32-bit CentOS 5 VM, with 384 MB of memory allocated – tiny, but perfectly functional.

We split puppetmaster into two (I’m the D. Hill referenced in the ticket description) – one handling file serving, the other handling manifest serving. The file serving puppetmaster holds steady at 54M VIRT, 39M RES. The manifest serving puppetmaster hold steady at 74M VIRT, 55M RES until storeconfigs is enabled, with a backing server of PostgreSQL 8.1.11 (provided in the base CentOS 5 installation).

Once storeconfigs is enabled, the memory usage starts climbing, almost as if there’s a leak. Once the VM starts swapping, we have no choice but to stop puppetmaster, wait a few minutes for the load to come back to a sensible level, and start puppetmaster again. Rinse, repeat.

I’ve checked file handles with lsof, and the number seems to hold steady around 50 – 60 files, whether the memory usage is 50 MB or 350 MB.

I found and fixed one leak in puppetd.

Puppetd was leaking report objects into an array.

I was running puppetd on one minute intervals and it grew about 20 MB over 8 hours.

Now running on 30 runintervals it has grown ~160K in about the same period, but it isn’t gaining Ruby objects.

The patch is available from my github (littleidea) in the report_leak branch.

I don’t want to close this issue, but I do want to give people something to work with.

I made extensive use of the memory_profiler.rb from: http://reductivelabs.com/trac/puppet/ticket/1131 (It will basically start a separate thread that keeps an inventory of ObjectSpace prints an audit to a file every so often. Make sure to add the ‘start’ after puppet daemonizes)

This told me I was leaking Puppet:Transaction::Reports.

I modified it a bit as I went but I got mileage doing stuff like this to mark when the new Object was created:

if o.class == Puppet::Transaction::Report and !report_ids.include?(o.object_id)

file.puts('New report')
file.puts(o.inspect)
report_ids << o.object_id

end

And this to figure out what the structure of the whatever was holding it if o.inspect =~ /Puppet::Transaction::Report/ and (o.class == Hash or o.class == Array)

file.puts(o.inspect)

end

I used this in conjunction with some basic dtrace scripts, mainly of the form:

sudo dtrace -s find_sync.d -p <pid#>

where find_sync.d looks like ruby$target:::object-create-start /copyinstr(arg0) == “Sync”/ {

printf("%s was allocated in file %s, line number %d\n", copyinstr(arg0), copyinstr(arg1), arg2);

}

That will tell you the line number and file where the Sync objects get created, as an entry point to start looking at code.

I’ll keep revisiting performance and memory problems, but I have to balance that with other work. I can solve problems much faster if I have a manifest or setup that can reliably reproduce the issue. Alternatively, if you are having memory problems try using the memory_profiler and dtrace with your setup.

I didn’t look at storeconfigs yet, but I think there are more perf problems their than just memory.

If anyone in the community has strong dtrace fu, or learns a new trick share with the list please. The ruby probes are somewhat simplistic and I think we probably need to get deeper knowledge to solve some of the issues.

Also, I have a relatively simple setup right now, but I’m not getting growth in the puppetmasterd.

If someone can give me a setup that consistently produces the problems, I’ll figure it out much faster.

help me help you :/

Oh yeah, when is Linux getting something like dtrace?

andrew wrote:

Also, I have a relatively simple setup right now, but I’m not getting growth in the puppetmasterd.

If someone can give me a setup that consistently produces the problems, I’ll figure it out much faster.

help me help you :/

Hello,

I’ve got a similar problem, and I’d gladly provide you will all the required files. The only Problem, even though I compressed them with bzip2. The Package is 44M. I’m using the version shipped with Ubuntu 8.04: 2.24.4.

The Package contains all configuration files as well as the data files I want puppet to distribute: www.alexanderschaber.de/puppet.tar.bz2 (It will be fully uploaded in about 10 minutes)

Correcting .. it’s 36MB .. ah and btw.: it’s fully uploaded now..

I’m getting significant memory usage on 0.24.5 on puppetd clients that are sitting around waiting for the CA to sign their cert. I’m not sure if this is the same issue as reported but thought it worth mentioning. One machine running Debian Etch has had puppetd running as a daemon for only 5 days and it has already swallowed a significant chunk of memory:

# ps u -C ruby
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     17715  9.6 60.9 750952 626340 ?       Dsl  Sep12 689:18 ruby /usr/sbin/puppetd -w 0

about 600MB! I’m also finding puppetd tends to use a lot of CPU time while waiting for the CA to sign the cert, and attempts to talk to the puppetmaster a lot more frequently than it should… but perhaps that’s a story for another ticket.

In doing automated testing for several other bugs I’ve been able to reproduce the memory leak in 0.25.1

I’m attempting to isolate and diagnose.

i’m also getting leaks with storeconfigs and postgresql driver (puppetmaster=0.25.1; passenger=2.2.2). now i’m killing threads with >768mb addressed (my server is getting killed without it). webrick also was going as high as 2gig (but it was uncapable of handling the load). i haven’t tried without stored configs. currently 57 nodes connects to this master.

2009-12-08 18:02:01 killed: pid=14966 shared=1500mb (limit: 1024mb) private=1123mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 18:03:02 killed: pid=15061 shared=1480mb (limit: 1024mb) private=1108mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 18:03:02 killed: pid=15169 shared=3046mb (limit: 1024mb) private=1673mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 18:05:02 killed: pid=14927 shared=1844mb (limit: 1024mb) private=1541mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 18:35:02 killed: pid=20125 shared=3076mb (limit: 1024mb) private=1594mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 18:39:01 killed: pid=21464 shared=1389mb (limit: 1024mb) private=976mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 19:03:03 killed: pid=24896 shared=1490mb (limit: 1024mb) private=1078mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 19:05:01 killed: pid=24795 shared=2634mb (limit: 1024mb) private=1689mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 19:06:02 killed: pid=24908 shared=2623mb (limit: 1024mb) private=1616mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 19:08:02 killed: pid=24844 shared=1503mb (limit: 1024mb) private=1254mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 19:11:02 killed: pid=26456 shared=1391mb (limit: 1024mb) private=1077mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 19:34:01 killed: pid=29484 shared=2502mb (limit: 1024mb) private=1760mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:06:01 killed: pid=1890 shared=1732mb (limit: 1024mb) private=1179mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:07:02 killed: pid=1691 shared=989mb (limit: 1024mb) private=817mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:08:01 killed: pid=2834 shared=1470mb (limit: 1024mb) private=1031mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:11:01 killed: pid=3270 shared=1867mb (limit: 1024mb) private=1575mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:36:01 killed: pid=6839 shared=1518mb (limit: 1024mb) private=1101mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:39:01 killed: pid=7218 shared=1534mb (limit: 1024mb) private=1054mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:41:01 killed: pid=7271 shared=2603mb (limit: 1024mb) private=1538mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:43:01 killed: pid=7737 shared=1514mb (limit: 1024mb) private=1266mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:45:02 killed: pid=8394 shared=971mb (limit: 1024mb) private=832mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 21:05:02 killed: pid=10905 shared=1729mb (limit: 1024mb) private=1257mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 21:12:01 killed: pid=10966 shared=2635mb (limit: 1024mb) private=1534mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 22:44:11 killed: pid=11703 shared=2615mb (limit: 1024mb) private=1409mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 22:44:11 killed: pid=11732 shared=4178mb (limit: 1024mb) private=1970mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 22:46:01 killed: pid=12141 shared=1389mb (limit: 1024mb) private=917mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:08:01 killed: pid=16876 shared=1494mb (limit: 1024mb) private=1120mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:16:01 killed: pid=17667 shared=1742mb (limit: 1024mb) private=1068mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:16:01 killed: pid=17682 shared=1799mb (limit: 1024mb) private=1002mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:18:01 killed: pid=18584 shared=2151mb (limit: 1024mb) private=1318mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:21:01 killed: pid=19036 shared=1659mb (limit: 1024mb) private=1137mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:40:01 killed: pid=21413 shared=1522mb (limit: 1024mb) private=1109mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:41:02 killed: pid=21262 shared=1060mb (limit: 1024mb) private=775mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:48:01 killed: pid=22471 shared=1831mb (limit: 1024mb) private=1148mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:50:01 killed: pid=23236 shared=1475mb (limit: 1024mb) private=1042mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-09 00:11:01 killed: pid=26442 shared=1777mb (limit: 1024mb) private=1234mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-09 00:20:02 killed: pid=27678 shared=1843mb (limit: 1024mb) private=1115mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-09 00:22:01 killed: pid=28141 shared=2267mb (limit: 1024mb) private=1471mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-09 00:23:02 killed: pid=28457 shared=1174mb (limit: 1024mb) private=851mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd

/usr/local/bin/ruby /usr/local/bin/passenger-memory-stats | grep "/rack/"|grep "[0-9] MB"|while read procpid threads shared mb private mb crap ; do
        shared=$(echo "$shared"|cut -f1 -d.)
        private=$(echo "$private"|cut -f1 -d.)
        if [ $shared -gt 1512 ] || [ $private -gt 768 ] ; then
                kill -9 $procpid
                echo "$(date '+%Y-%m-%d %H:%M:%S') killed: pid=$procpid shared=${shared}mb (limit: 1024mb) private=${private}mb (limit: 512mb) process=$crap" >>/tmp/spuchlaki.log
        fi
done

Konrad —

Can you quantify how long (in wall clock time & puppet runs) a puppetmaster goes before reaching this point? And is the process gradual/continuous or abrupt?

Also, I note that the limits specified in the log file lines (1GB shared, 512MB private) are not the actual limits enforced by the script.

And finally, would you be willing to run test/instrumented versions to help pin the problem down?

— Markus

it’s countinous while it starts, goes ~30mb/sec after it started and never finishes (server memory exhausted or killed by script). i’m not sure – but can verify – it occurs after the run (ie. when no pupppet is connected).

can you provide info about test procedures/versions?

i know about limits, that’s why i provided a script (i forgot to change the number that goes to log). limit was 768 since 18:00.

EDIT: typo

I’m not sure – but can verify – it occurs after the run (ie. when no puppet is connected).

That would be very interesting if you can confirm. I’m not sure what it means though—there shouldn’t be anything much happening at that point. That’s generally the sign of a great clue.

If you can reliably reproduce this I’d love to work with you to nail it down.

interesting screenshot below. i’ve taken it just after doing “service passenger stop” – so leaked processed were not killed (despite the apache process exited) and they keept their active mysql connections (see netstat and mysqladmin). btw. i’ve switched postgres to mysql as stored configs backend, as it appears to work more stable (no sudden disconnects) and leak less.

Every 2.0s: passenger-memory-stats | grep -A50 "Passenger processes" ; netstat -nalp|grep ESTAB ; tail -10 /var/log/httpd/passenger_access.log ; mysqladmin -u puppet --password...  Thu Dec 10 23:22:45 2009

[44m[1m[33m----------- Passenger processes -----------
[37mPID    Threads  VMSize     Private    Name
-------------------------------------------[0m
1193   1        59.7 MB    0.1 MB     grep -A50 Passenger processes
26882  1        3028.7 MB  1756.4 MB  Rack: /usr/share/puppet/rack/puppetmasterd
26894  1        2974.9 MB  821.1 MB   Rack: /usr/share/puppet/rack/puppetmasterd
28462  1        1530.1 MB  719.4 MB   Rack: /usr/share/puppet/rack/puppetmasterd
### Processes: 4
### Total private dirty RSS: 3296.99 MB
tcp        0      0 127.0.0.1:3306              127.0.0.1:39207             ESTABLISHED 18452/mysqld
tcp        0      0 10.0.4.4:39634              10.0.100.2:3493             ESTABLISHED 2399/upsmon
tcp        0      0 127.0.0.1:39153             127.0.0.1:3306              ESTABLISHED 26894/puppetmasterd
tcp        0      0 127.0.0.1:39145             127.0.0.1:3306              ESTABLISHED 26882/puppetmasterd
tcp        0      0 127.0.0.1:39207             127.0.0.1:3306              ESTABLISHED 28462/puppetmasterd
tcp        0      0 10.0.4.4:22                 100.100.100.100:44106       ESTABLISHED 7157/sshd: kondi [p
tcp        0      0 127.0.0.1:3306              127.0.0.1:39145             ESTABLISHED 18452/mysqld
tcp        0      0 127.0.0.1:3306              127.0.0.1:39153             ESTABLISHED 18452/mysqld
172.16.101.98 - - [10/Dec/2009:23:20:59 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:00 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:00 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:01 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:01 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:01 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:02 +0100] "POST /RPC2 HTTP/1.1" 200 120 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:03 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:03 +0100] "POST /RPC2 HTTP/1.1" 200 120 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:04 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
+-----+--------+-----------------+--------+---------+------+-------+-----------------------+
| Id  | User   | Host            | db     | Command | Time | State | Info                  |
+-----+--------+-----------------+--------+---------+------+-------+-----------------------+
| 287 | puppet | localhost:39145 | puppet | Sleep   | 486  |       |                       |
| 293 | puppet | localhost:39153 | puppet | Sleep   | 487  |       |                       |
| 328 | puppet | localhost:39207 | puppet | Sleep   | 414  |       |                       |
| 389 | puppet | localhost       |        | Query   | 0    |       | show full processlist |
+-----+--------+-----------------+--------+---------+------+-------+-----------------------+

aaah, and sth more – if left not killed, those processes appear to return some of its memory – weird.

Every 2.0s: passenger-memory-stats | grep -A50 "Passenger processes" ; netstat -nalp|grep ESTAB ; tail -10 /var/log/httpd/passenger_access.log ; mysqladmin -u puppet --password...  Thu Dec 10 23:29:20 2009

[44m[1m[33m----------- Passenger processes -----------
[37mPID    Threads  VMSize     Private    Name
-------------------------------------------[0m
6987   1        59.7 MB    0.1 MB     grep -A50 Passenger processes
26882  1        3028.7 MB  976.7 MB   Rack: /usr/share/puppet/rack/puppetmasterd
26894  1        2974.9 MB  1684.7 MB  Rack: /usr/share/puppet/rack/puppetmasterd
28462  1        2051.1 MB  729.6 MB   Rack: /usr/share/puppet/rack/puppetmasterd
### Processes: 4
### Total private dirty RSS: 3391.05 MB
tcp        0      0 127.0.0.1:3306              127.0.0.1:39207             ESTABLISHED 18452/mysqld
tcp        0      0 10.0.4.4:39634              10.0.100.2:3493             ESTABLISHED 2399/upsmon
tcp        0      0 127.0.0.1:39153             127.0.0.1:3306              ESTABLISHED 26894/puppetmasterd
tcp        0      0 127.0.0.1:39145             127.0.0.1:3306              ESTABLISHED 26882/puppetmasterd
tcp        0      0 127.0.0.1:39207             127.0.0.1:3306              ESTABLISHED 28462/puppetmasterd
tcp        0      0 10.0.4.4:22                 100.100.100.100:44106       ESTABLISHED 7157/sshd: kondi [p
tcp        0      0 127.0.0.1:3306              127.0.0.1:39145             ESTABLISHED 18452/mysqld
tcp        0      0 127.0.0.1:3306              127.0.0.1:39153             ESTABLISHED 18452/mysqld
172.16.101.98 - - [10/Dec/2009:23:20:59 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:00 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:00 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:01 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:01 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:01 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:02 +0100] "POST /RPC2 HTTP/1.1" 200 120 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:03 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:03 +0100] "POST /RPC2 HTTP/1.1" 200 120 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:04 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
+-----+--------+-----------------+--------+---------+------+-------+-----------------------+
| Id  | User   | Host            | db     | Command | Time | State | Info                  |
+-----+--------+-----------------+--------+---------+------+-------+-----------------------+
| 287 | puppet | localhost:39145 | puppet | Sleep   | 890  |       |                       |
| 293 | puppet | localhost:39153 | puppet | Sleep   | 891  |       |                       |
| 328 | puppet | localhost:39207 | puppet | Sleep   | 818  |       |                       |
| 469 | puppet | localhost       |        | Query   | 0    |       | show full processlist |
+-----+--------+-----------------+--------+---------+------+-------+-----------------------+

(my server has died shortly after that – watchdog reboots it when free physical mem <2000mb)

good news… it seems not to be related to storeconfigs. i have small function on puppetmaster called @:template_search@ and it seems to be locking itself and eating all memory.

# /var/lib/puppet/lib/puppet/parser/functions/template_search.rb
 
Puppet::Parser::Functions::newfunction(:template_search, :type => :rvalue, :doc =>
    "Find first existing file and run puppet template function on it. If you want
    to concatenate multiple templates try var = template_search(file1,file2) +
    template_search(file3,file4) and then use var in your resources.") do |vals|
            ret = nil
            errors = Array.new
            vals.each do |file|
                debug "in :template_search find %s" % file
                ret = nil
                begin
                    ret = function_template(file)
                rescue Exception=>e
                    if /could.not.find.template/i =~ e
                        debug "in :template_search not found %s" % file
                    else
                        raise Puppet::ParseError, "in :template_search exception in %s: %s" % [file,e]
                        errors.push(e)
                    end
                end
                if ret
                    debug "in :template_search found %s" % file
                    break
                end
            end
            if ret
                ret
            else
                if errors.length > 0
                    raise Puppet::ParseError, "Errors from :template function: %s" %
                        errors.join(", ")
                else
                    raise Puppet::ParseError, "Could not find any files from %s" %
                        vals.join(", ")
                end
            end
end

i call it with:

    class backend inherits exim {
        file {"exim.conf":
            path    => "/etc/exim/exim.conf",
            content  => template_search("exim/$hostname/exim-backend.erb", "exim/$environment/exim-backend.erb"),
        }

in logs it hangs on:

Dec 14 19:10:40 punch puppetmasterd[16865]: (Scope(Class[exim::backend])) in :template_search find exim/mailser01/exim-backend.erb

(where @mailser01@ is substituted @$hostname@)

any idea why this is happening?

i’m making separate ticket for it (#2930) as it seems to be endless loop, not memory leak. i’ve fixed my functions and now my puppetmaster processes are stable (~100mb on x86_64 running passenger/ruby-enterprise).

Absent something definite & reproducible or a strategy for finding such, there’s not much we can expect to accomplish on this ticket in the next few days.

This issue is much more critical!

It can be reproducible in a period running of 30+ days, I have about 20+ machines running puppetd (version 0.25.4), after 30 days, 80% of their puppetd’s memory usage are up to 50%

restarting puppetd process can resolve the issue.

puppetd memory usage (the host has 1GB RAM & 1GB swap)

ELAPSED %CPU %MEM CMD

33-20:01:14 3.4 60.7 /usr/bin/ruby /usr/sbin/puppetd

Kevin

1. What platform?
2. What Ruby version?