Bug #1443

Enable lazy loading of latest packages for yum

Added by Peter Meier over 3 years ago. Updated over 3 years ago.

Status:Closed Start date:07/21/2008
Priority:Normal Due date:
Assignee:James Turnbull % Done:

0%
Category:package
Target version:0.24.6
Affected Puppet version: Branch:
Keywords:yum
Votes: 0

Description

I could observe that yumhelper.py can run quite some time and use quite a lot of memory (up to 100MB!) if you’re using many repositories. As the yumhelper is only needed if you use latest as a ensure-keyword it makes no sense for people like me letting puppet fetch these details always, if you’re never going to use latest.

Therefor this patch changes the behaviour, that the yumhelper is only called, if @updates is accessed the first time. => lazy loading.

some observations I could do showed some significant perfomance boost:

without lazy loading:

notice: Finished catalog run in 61.45 seconds

with lazy loading and not using any latest definition:

notice: Finished catalog run in 34.99 seconds

Pushed in commit: “c24237”:http://github.com/duritong/puppet/commit/c24237daf14d22c858ff07fe88384333e0558bb6 Branch published @ “github”:http://github.com/duritong/puppet/tree/yumhelper_lazyloading

However maybe the yumhelper itself should be examined on the memory usage. But I think most of the stuff is coming from yum itself.

History

#1
Updated by Peter Meier over 3 years ago

  • Category set to package
  • Target version set to 0.24.5

#2
Updated by Luke Kanies over 3 years ago

  • Status changed from Unreviewed to Accepted
  • Assignee set to Darrell Fuhriman
  • Target version changed from 0.24.5 to 0.25.0

This isn’t going to make it into 0.24.5; we’re too late in the game for that.

As discussed on the list, I like the idea of doing this, but the specific patch isn’t quite sufficient.

#3
Updated by Peter Meier over 3 years ago

  • Assignee changed from Darrell Fuhriman to James Turnbull

As discussed on the list, I like the idea of doing this, but the specific patch isn’t quite sufficient.

published new patch in my “ticket/1443”:http://github.com/duritong/puppet/tree/ticket/1443 branch on github. Commit: “e6a1999”:http://github.com/duritong/puppet/commit/e6a1999cec765a340fbb650884291283c530dc0d

#4
Updated by James Turnbull over 3 years ago

  • Status changed from Accepted to Ready For Checkin
  • Assignee changed from James Turnbull to Luke Kanies
  • Keywords set to yum

#5
Updated by Peter Meier over 3 years ago

immerda wrote:

As discussed on the list, I like the idea of doing this, but the specific patch isn’t quite sufficient.

published new patch in my “ticket/1443”:http://github.com/duritong/puppet/tree/ticket/1443 branch on github. Commit: “e6a1999”:http://github.com/duritong/puppet/commit/e6a1999cec765a340fbb650884291283c530dc0d

new commit: “42c832”:http://github.com/duritong/puppet/commit/42c832cfe4fd222feaf78e85fca02ea3ec919462

#6
Updated by Peter Meier over 3 years ago

  • Assignee changed from Luke Kanies to Peter Meier

I realised, that his fix doesn’t actually work… :(

I redo the work. Sorry.

#7
Updated by Peter Meier over 3 years ago

  • Assignee changed from Peter Meier to James Turnbull

ok finally fixed it in commit “03937658”:http://github.com/duritong/puppet/commit/03937658a4b5d894dd05aa5c1025620492256bbd

#8
Updated by James Turnbull over 3 years ago

  • Assignee changed from James Turnbull to Luke Kanies

Pushed to my master in commit:818599db7dc6210cc015d8888d119726eb6c3323.

#9
Updated by James Turnbull over 3 years ago

  • Status changed from Ready For Checkin to Closed

#10
Updated by Peter Meier over 3 years ago

  • Status changed from Closed to Re-opened
  • Assignee changed from Luke Kanies to Peter Meier
  • Affected Puppet version deleted (0.24.4)

I could observe that this is somehow again a problem and therefor not (yet?) fixed. I reopen it, so I’ll work on it in the next few days. Maybe it was reintroduced by some other changes.

#11
Updated by Peter Meier over 3 years ago

  • Assignee changed from Peter Meier to James Turnbull
  • Target version changed from 0.25.0 to 0.24.6

immerda wrote:

I could observe that this is somehow again a problem and therefor not (yet?) fixed. I reopen it, so I’ll work on it in the next few days. Maybe it was reintroduced by some other changes.

James: this commit isn’t part of 0.24.6 any reason for that? It have only be pushed to the master branch. Should I provide a patch against 0.24.x so it gets included into 0.24.6? (which would be nice as yumhelper really eats a lot of memory)

#12
Updated by James Turnbull over 3 years ago

An 0.24.x patch is fine. Please submit and add. Thanks.

#13
Updated by Peter Meier over 3 years ago

An 0.24.x patch is fine. Please submit and add. Thanks.

pushed “65eafb7c”:http://github.com/duritong/puppet/commit/65eafb7c01e6b4c003066cf7c6dceb5910e91950 to “tickets/0.24.x/1443”:http://github.com/duritong/puppet/tree/tickets%2F0.24.x%2F1443 to “github”:http://github.com/duritong/puppet/

#14
Updated by James Turnbull over 3 years ago

  • Status changed from Re-opened to Closed

Pushed in commit:65eafb7c01e6b4c003066cf7c6dceb5910e91950 in branch 0.24.x

Also available in: Atom PDF