Are you able to help us figure this out? We haven’t been able to reproduce any leaks in the master recently, with clients running at an interval of 60 seconds.

You’d need to be at least a little bit ruby-savvy, or be willing to ship us your configurations so we can run the compiles ourselves.

We’ve been having severe memory problems with puppetmasterd as well. Within a day, the memory usage of the puppetmasterd processes would grow until all memory was used, at which point either Linux' out-of-memory killer kicked in and killed the puppetmasterd processes, or it didn’t pick the correct processes and the entire server needed a hard reboot. I’ve attached a graph which shows the memory usage for the pupetmasterd processes for the past week.

I’ve recently (yesterday) upgraded our Rails version from 2.0.2 to 2.1, which seems to have improved things a lot. It’s a bit early to tell if the Rails upgrade completely fixed things for us, but things are looking good so far: the memory usage of the puppetmasterd processes is currently pretty steady at about 250 MB each for about a day now, whereas previously they would grow to 1GB within a day. So in our case at least it seems the Rails layer may have been the cause of the memory leaks.

To the reporter of this ticket: do you have Rails installed on the puppetmaster server, and if so, which version of Rails are you running?

bart wrote:

We’ve been having severe memory problems with puppetmasterd as well. Within a day, the memory usage of the puppetmasterd processes would grow until all memory was used, at which point either Linux' out-of-memory killer kicked in and killed the puppetmasterd processes, or it didn’t pick the correct processes and the entire server needed a hard reboot. I’ve attached a graph which shows the memory usage for the pupetmasterd processes for the past week.

I’ve recently (yesterday) upgraded our Rails version from 2.0.2 to 2.1, which seems to have improved things a lot. It’s a bit early to tell if the Rails upgrade completely fixed things for us, but things are looking good so far: the memory usage of the puppetmasterd processes is currently pretty steady at about 250 MB each for about a day now, whereas previously they would grow to 1GB within a day. So in our case at least it seems the Rails layer may have been the cause of the memory leaks.

To the reporter of this ticket: do you have Rails installed on the puppetmaster server, and if so, which version of Rails are you running?

My graphs looks really similar to yours, but we don’t use store configs at all… :(

ohadlevy wrote:

My graphs looks really similar to yours, but we don’t use store configs at all… :(

I seem to remember reading that if you’ve installed Rails, ActiveRecord always gets loaded by puppetmasterd and can cause problems even when you’re not using stored configs. I can’t remember where I read that so I may be misrembering though. I tried the Rails upgrade because I read about Rails causing problems, thinking our memory problems might be the same or related. So even though you’re not using stored configs, uninstalling or upgrading Rails may be worth a shot if you currently have it installed.

Puppetmasterd has now been running on our server for two days without problems, so although it may be a different problem, I’m pretty optimistic the Rails upgrade from 2.0.2 to 2.1 has fixed our memory problems.

We currently do not have rails installed, but we do have activesupport installed:

[root@intranet-repo ~]# yum search rails | egrep ‘(rails|activesupport)’ ruby-activesupport.noarch : Utility classes and extension to Ruby’s standard library rubygem-activesupport.noarch : Support and utility classes used by the Rails framework rubygem-rails.noarch : Web-application framework rubygem-activesupport.noarch : Support and utility classes used by the Rails framework ruby-activesupport.noarch : Utility classes and extension to Ruby’s standard library rubygem-activesupport.noarch : Support and utility classes used by the Rails framework [root@intranet-repo ~]# rpm -qa | egrep ‘(rails|activesupport)’ rubygem-activesupport-2.0.2-1.fc8

I’ll check to see if there’s an update available for it, but any way to isolate puppetmasterd from this module via configs?

I found the post about where the problem with ActiveRecord was mentioned: http://groups.google.com/group/puppet-users/msg/bd157c2b4b9f03c8

bretm wrote:

I’ll check to see if there’s an update available for it, but any way to isolate puppetmasterd from this module via configs?

As far as I can tell from a quick grep, Puppet doesn’t use ActiveSupport. I also had to upgrade some other ruby libraries before I could upgrade to Rails 2.1, so possibly one of those upgrades was the real fix… although it doesn’t look like the Debian package for puppet (directly or indirectly) depends on any of them. The ruby libraries I upgraded at the same time I did the Rails upgrade are:

• libxml-simple-ruby
• libjs-prototype
• libbuilder-ruby1.8

I also ugraded from Facter 1.3.8 to Facter 1.5, but that was about half a day after I upgraded Rails.

bart wrote:

We’ve been having severe memory problems with puppetmasterd as well. Within a day, the memory usage of the puppetmasterd processes would grow until all memory was used, at which point either Linux' out-of-memory killer kicked in and killed the puppetmasterd processes, or it didn’t pick the correct processes and the entire server needed a hard reboot. I’ve attached a graph which shows the memory usage for the pupetmasterd processes for the past week.

I’ve recently (yesterday) upgraded our Rails version from 2.0.2 to 2.1, which seems to have improved things a lot. It’s a bit early to tell if the Rails upgrade completely fixed things for us, but things are looking good so far: the memory usage of the puppetmasterd processes is currently pretty steady at about 250 MB each for about a day now, whereas previously they would grow to 1GB within a day. So in our case at least it seems the Rails layer may have been the cause of the memory leaks.

To the reporter of this ticket: do you have Rails installed on the puppetmaster server, and if so, which version of Rails are you running?

I looked on the system bretm is using and Rails is not being used:

irb(main):001:0> require ‘puppet’ => true irb(main):002:0> Puppet.features.rails? => false

luke wrote:

Are you able to help us figure this out? We haven’t been able to reproduce any leaks in the master recently, with clients running at an interval of 60 seconds.

You’d need to be at least a little bit ruby-savvy, or be willing to ship us your configurations so we can run the compiles ourselves.

I am available to help track down the issue bretm is hitting. I can reproduce the issue on Fedora 9 with puppet 0.24.4. I’m running a Xen guests with 1512 MB of ram with no puppet clients. The only modification to the stock Fedora puppet package is the addition of the “memory_profiler” library to the puppetmaster.

Here is my puppet-test command:

./puppet-test —server=.com —fqdn=db02.intranet.intdev.redhat.com —fork 4 —test getconfig —repeat 50 —suite remote_catalog

“db02.intranet.intdev.redhat.com” is the hostname of a machine that is being used in bretm’s environment. I have copied his manifests and wired up external_nodes to return the appropriate classes for that node. To reproduce the memory issue I ran the above command and after each run recorded the active memory usage (as reported by “free -m”).

The first few times I ran the test I was simply watching the ram grow. I eventually started recording them. The results were: 760, 885, 1009, 1134, 1259, 1328. After the next run I started seeing the errors bretm mentioned. I bounced the puppetmaster and that freed nearly 1200 MB of ram.

I’m currently trying out 0.24.5 and so far I’m seeing similar results. After each run I’m cat'ing out /proc/meminfo. I’ll gladly post the results along with /tmp/memory_log/memory_profiler.log if you think it will help.

If there’s anything else I can do to track down this problem just let me know.

brenton wrote:

I’m currently trying out 0.24.5 and so far I’m seeing similar results. After each run I’m cat'ing out /proc/meminfo. I’ll gladly post the results along with /tmp/memory_log/memory_profiler.log if you think it will help.

If there’s anything else I can do to track down this problem just let me know.

Using the same puppet-test oneliner I’m seeing the exact behavior with 0.24.5. Here is the active memory usage from a total of 9 runs of the puppet-test command mentioned previously:

[root@dhcp226-90 ext]# grep Active run*.txt | sort

run1.txt:Active: 417592 kB

run2.txt:Active: 542960 kB

run3.txt:Active: 665768 kB

run4.txt:Active: 789688 kB

run5.txt:Active: 912340 kB

run6.txt:Active: 1036872 kB

run7.txt:Active: 1161420 kB

run8.txt:Active: 1286612 kB

run9.txt:Active: 1321656 kB

bretm wrote:

We currently have a puppetmaster running between 10-20 clients, checking in on ~5min intervals. Once a day, we hit out-of-memory errors on the puppetmaster; the situation is eliminating by restarting the puppetmaster service.

The puppetmaster is running Fedora 8, and the leak has been seen in 0.24.4 and 0.24.5.

If 0.25 rpms are available, I’d be happy to test those.

I believe this to be related to the memory leak in 1.8.6 p230 (http://redmine.ruby-lang.org/issues/show/216). That happens to be the latest version for both Fedora 8 and 9. I’ve opened a bug against Fedora to see if they don’t mind upgrading to the latest 1.8.6 release. https://bugzilla.redhat.com/show_bug.cgi?id=458475. If that fixes this issue I’ll close this ticket.

Why is this closed? Issue gone? Issue not Puppet?