Bug #5200

Many-to-many implementation of stages needlessly inefficient

Added by Markus Roberts over 1 year ago. Updated about 1 month ago.

Status:Closed Start date:11/03/2010
Priority:Normal Due date:
Assignee:Nick Lewis % Done:

0%

Category:performance
Target version:2.7.x
Affected Puppet version:2.6.0 Branch:
Keywords:
Votes: 2

Description

Trevor Vaughan reports:

Stages appear, by default, to generate a many-to-many dependency graph. By adding 'pre' and 'post' stages that contain a single item, the graphs are vastly reduced in complexity and we're seeing something like a 3X speedup in graph execution in simple test cases. I've attached a couple of puppet files illustrating the issue with their associated graphs. If the graphs could be auto-reduced to a single point prior to execution automatically, it would be quite helpful.

I believe the savings could be even greater in complex situations. We had discussed addressing this internally by creating a whit (null resource) but the idea apparently did not make it into code. It should be an easy win.


Related issues

related to Puppet - Bug #4590: SimpleGraph is slow Closed 08/23/2010
related to Puppet - Bug #5012: File should autorequire Mounts Code Insufficient 10/15/2010
related to Puppet - Feature #5015: mount should autorequire mountpoint Code Insufficient 10/15/2010
related to Puppet - Bug #428: Deletion should invert sort order Rejected
related to Puppet - Bug #5369: Autorequire loop resolution inadequate Accepted 11/19/2010
related to Puppet - Feature #6911: Flexible, deterministic, unguessable application order Closed 03/30/2011
related to Puppet - Bug #9671: transaction eval_generate slow on many files in recurse m... Closed 09/26/2011

History

Updated by Markus Roberts over 1 year ago

  • Assignee set to Markus Roberts
  • Target version set to 2.7.x

Luke, Teyo & Nigel have all agreed that this would be very nice to get in Statler.

Updated by Markus Roberts over 1 year ago

Just to clarify, I linked the two autorequire tickets due to implicit conceptual collision between the issues discussed on the respective threads on puppet-dev; specifically, those tickets propose expanding / extending the autorequire functionality while Luke’s refactor/master/3691-no_relationship_graph branch, which he suggested as an alternate resolution on the thread for this ticket removes autorequire functionality and it’s not immediately clear (to me at least) how it could be restored in that model.

I think eliminating the second graph as Luke’s branch does is the right idea, and I think representing containment with bracket node as Tervor, Jesse and other have advocated is better than adding edge types. But I’m not yet seeing how either idea can work in actual code. Just adding bracket nodes as an added complication in splice! looks like the easiest, though significantly less efficient. In short, I’m finding myself in agreement with pretty much everyone on the details and with almost no one on the big picture.

Updated by Markus Roberts about 1 year ago

To update: Jesse & I have a working sketch of the bracket solution which keeps the “two graphs” structure for now. It’s turning out to be a needed step for #6911 and will probably be folded in with that.

Updated by Markus Roberts about 1 year ago

The branch on #6911 now contains a sketch of the fix for this, with one known bug and a small amount of cleanup left to do.

Updated by Joshua Lifton 5 months ago

  • Assignee deleted (Markus Roberts)

This issue was assigned to a former Puppet Labs employee. Adding back to the pool of unreviewed issues.

Updated by Joshua Lifton 5 months ago

This issue was assigned to a former Puppet Labs employee. Adding back to the pool of unreviewed issues.

Updated by Ben Hughes 4 months ago

  • Description updated (diff)
  • Status changed from Accepted to Unreviewed

Updated by Nigel Kersten 4 months ago

  • Description updated (diff)
  • Status changed from Unreviewed to Needs More Information
  • Assignee set to Nick Lewis

Nick, didn’t this get resolved when we had the same issue with class –> class relationships and the Whit implementation was meant to solve it?

Updated by Nick Lewis about 1 month ago

  • Status changed from Needs More Information to Closed

This has been resolved for a year or so.

Also available in: Atom PDF