Bug #1965

File paths take up much memory

Added by Luke Kanies almost 3 years ago. Updated almost 3 years ago.

Status:Closed Start date:02/13/2009
Priority:Normal Due date:
Assignee:James Turnbull % Done:

0%

Category:plumbing
Target version:0.24.8
Affected Puppet version:0.24.7 Branch:
Keywords:
Votes: 0

Description

Larry Ludwig and I did some memory leak analysis, and found that, especially with multiple environments, file paths can take a significant amount of memory. Based on some simplistic analysis, they can take up to 50% of the memory in the master.

We need some kind of file collection we can use to store no more than one copy of each file path during parsing.

We should probably have a similar file collection in use on the client, since each resource and parameter also stores the file that it was defined in. Then we use a FileMangler module or something that interactions with this collection. Something like:

module Puppet::Util::FileMangler
  def file=(path)
    @file_reference = file_collection.file_reference(@path)
  end

  def file
    file_collection.file_path(@file_reference)
  end
end

History

Updated by Luke Kanies almost 3 years ago

  • Status changed from Accepted to In Topic Branch Pending Review

I’ve implemented an initial version of this in the refactor/0.24.x/fewerfiles branch in my repo.

If anyone experiencing RAM problems could give this a try, I’d like to hear if it has any impact on the server. I expect it’ll have a bigger impact there than on the client, especially for people with large amounts of code, which is particularly true on sites that use multiple environments.

I’ve done a bit of testing, and looks like there’s about a 7x benefit to using a simple hash to translate the filename to the index. This results in two copies of each file name, worst case.

Updated by Luke Kanies almost 3 years ago

Oh, and as my branch name suggests, my code is against 0.24.x, not master.

Updated by Pietro Ferrari almost 3 years ago

i created a package of 0.24.x branch + refactor/0.24.x/fewerfiles branch and deployed to master and clients. i’m monitoring memory on the master and will report back as soon as i get some more conclusive data.

Updated by Luke Kanies almost 3 years ago

  • Status changed from In Topic Branch Pending Review to Ready For Checkin
  • Assignee changed from Luke Kanies to James Turnbull
  • Target version changed from 0.25.0 to 0.24.8

Updated by James Turnbull almost 3 years ago

  • Status changed from Ready For Checkin to Closed

Pushed in commit:e2b406239eaa255c41acb31942169296bea71948 in branch 0.24.x

Also available in: Atom PDF