Bug #7316

puppet face applications (subcommands) delivered via pluginsync and as modules should work

Added by Dan Bode about 1 year ago. Updated 14 days ago.

Status:Merged - Pending Release Start date:05/02/2011
Priority:Urgent Due date:
Assignee:Chris Price % Done:

0%

Category:Faces
Target version:3.0.0
Affected Puppet version: Branch:https://github.com/puppetlabs/puppet/pull/571
Keywords:face faces subcommand application module plugin pluginsync
Votes: 0

Description

If you deliver a new face that consists of:

  • application
  • face
  • action for face

via pluginsync, then the application isn’t actually found, and worse, it taunts you by showing it to you in the list of available subcommands.


Related issues

related to Puppet - Refactor #7749: Bootstrapping Puppet in Ruby code has nasty scope cycles... Merged - Pending Release 06/01/2011
related to Puppet - Refactor #6753: Faces should be available as `puppet ${face}` without a s... Accepted 03/17/2011
duplicated by Puppet - Bug #9018: Faces cannot add subcommands when distributed as a module Duplicate 08/15/2011

History

Updated by Daniel Pittman about 1 year ago

  • Status changed from Unreviewed to Needs Decision
  • Assignee set to Nigel Kersten
  • Affected Puppet version set to 2.7.0rc1

Nigel, I don’t know if we want to support this or not, but it is a genuine issue. The Faces code doesn’t make substantial effort to massage the module path into the Ruby load path, and uses only the later to try and locate code.

Given we extensively use require for loading, not having those library directories on the Ruby load path will require substantial extra work, as well as reimplementing the require mechanism ourselves.

On the other hand, obviously we don’t already put the lib directories on the module path into the Ruby load path, so I don’t know what side-effects that change might have on the system; that makes me nervous about putting this into the RC series for 2.7, and about putting it in post-2.7.0, since both of those might be the worst choice depending on what assumptions this tripped up.

Updated by Nigel Kersten about 1 year ago

  • Tracker changed from Feature to Bug
  • Subject changed from faces should support loading faces from modulepath to puppet applications delivered via pluginsync don't work.
  • Status changed from Needs Decision to Accepted
  • Assignee deleted (Nigel Kersten)
  • Target version set to 2.7.1

Updated by James Turnbull about 1 year ago

  • Priority changed from Normal to High

I actually think this is a higher priority – not only does pluginsync partially work giving you the disappointing illusion that things are okay :) But without it Faces are very manual to deploy and can’t really be developed and shared – which seems to defeat some of their purpose. YMMV. :)

Updated by Luke Kanies about 1 year ago

Can someone provide an idea of what’s actually wrong here? There are a lot of “doesn’t work” reports in this, but very little detail.

In talking with James, he says it looks like the Faces are found but not the applications. In looking at the application code, AFAICT it’s loading applications from the $LOAD_PATH, Puppet’s lib dir is in the $LOAD_PATH, and the applications should be copied to Puppet’s lib dir.

What actually goes wrong in this sequence?

Updated by James Turnbull about 1 year ago

So tested so far.

I pluginsync the face.

  1. When I run puppet as a normal (non-root) user I don’t see anything.
  2. When I run puppet as the root user – my face shows up in the output of puppet help.
  3. When I try to run the face I get:
$ puppet github
Error: Unknown Puppet subcommand 'github'

Updated by Daniel Pittman about 1 year ago

So tested so far. I pluginsync the face.

Did you sync the puppet/application/github.rb file as well?

When I run puppet as a normal (non-root) user I don’t see anything. When I run puppet as the root user – my face shows up in the output of puppet help. When I try to run the face I get:

$ puppet github Error: Unknown Puppet subcommand ‘github’

My best guess, given the (entirely uninvestigated, since we have no time scheduled to faces at the moment) reports, is that the application is getting synced, but not seen by the loader, so it fails with “unknown subcommand”.

Updated by Nigel Kersten about 1 year ago

Daniel has it. The synced application isn’t found by the autoloader.

Updated by Nigel Kersten about 1 year ago

  • Priority changed from High to Urgent

Updated by Nick Lewis 11 months ago

  • Assignee set to Nick Lewis

Updated by Daniel Pittman 10 months ago

  • Target version changed from 2.7.1 to 2.7.x

Updated by Nick Lewis 10 months ago

What’s happening is that command_line simply looks in $LOAD_PATH, rather than using the autoloader.

I’ve experimented with connecting it to the autoloader, which works great except for some interesting load order issues. Primarily around defaults.rb getting executed, which tries to set Puppet[:name] (which is read-only after that) to Puppet.application_name, which obviously hasn’t been set yet. It may work to make Puppet[:name] not read-only, and set it in command_line after loading the application (or in Application during the ‘run’ sequence), possibly with a meaningless default. Or without a default at all, since it’s not actually configurable.

Updated by Daniel Pittman 10 months ago

What’s happening is that command_line simply looks in $LOAD_PATH, rather than using the autoloader.

I’ve experimented with connecting it to the autoloader, which works great except for some interesting load order issues. Primarily around defaults.rb getting executed, which tries to set Puppet[:name] (which is read-only after that) to Puppet.application_name, which obviously hasn’t been set yet.

Ah! Another part of the fragile load ordering issue identified in

7749. Awesome.

I will connect the two tickets together, and see where that gets us to and all.

Updated by Carl Caum 10 months ago

It works if you set your RUBYLIB to puppet’s $vardir/lib

Updated by Jeff McCune 9 months ago

  • Keywords set to face faces subcommand application module plugin pluginsync

Updated by Jeff McCune 9 months ago

  • Subject changed from puppet applications delivered via pluginsync don't work. to puppet face applications (subcommands) delivered via pluginsync and as modules should work

Modules

Since #9018 was closed as a duplicate of this, I just want to make sure it’s clear that a solution to this ticket requires that puppet stand alone with a modulepath set correctly should also find the face application sub-commands. That is to say, the scope of this ticket is NOT simply limited to pluginsync.

Updated by Jeff McCune 9 months ago

  • Affected Puppet version deleted (2.7.0rc1)

Nick,

I’m trying to plan the remaining work for Cloud Provisioner in CK and this ticket impacts my work in that I may or may not be able to implement additional subcommands (application) beyond “puppet node” if this is resolved in 2.7.4.

Are you in a position to estimate if this ticket will be resolved in 2.7.4 or not?

Thanks, -Jeff

Updated by K Hightower 7 months ago

  • Assignee changed from Nick Lewis to K Hightower

Updated by Nick Lewis 6 months ago

  • Assignee changed from K Hightower to Nick Lewis

I’m working on this now. Making applications loadable is fairly straightforward, although there’s some weird issues around setting modulepath and libdir, because we can’t use the config file for those settings (because we don’t have a runmode, so where do we look?). Also, naturally, applications need to be run as a user that can read them. This means on the master they need to be run as root or the user the master runs as, and on the agent, they probably need to be run as root.

Updated by Nick Lewis 6 months ago

  • Status changed from Accepted to Investigating

While applications are loadable from modules, loading the faces on which they depend is proving much more difficult. The main problem is that we don’t allow actions to be redefined, so we can’t load a face or action more than once, so we use require. But require is out of our control as far as module path goes. We could require via the autoloader, but that would necessarily use the absolute path, which doesn’t do anything to prevent multiple loading in the case where the file IS in the regular load path, and gets required again with its relative path. Unfortunately, because faces are a public API, this is a real concern. Requiring an action would autoload the face, which would autoload the action and we get a conflict.

The closest thing to a solution I’ve got as yet is to add a hook to the modulepath setting to populate the load path with the lib dir of every module in each directory of the modulepath. Unfortunately, that doesn’t play well with environments, and would also mean that if a module is added, we wouldn’t find it until at least the filetimeout (when we refresh the modulepath).

Alternatively, we could have a mechanism to “pluginsync” the contents of module libs into the libdir, which would greatly simplify the situation.

Updated by Nigel Kersten 6 months ago

Great update Nick. Just pointing out that not playing well with environments is a huge problem, and I’d like to have a chat in person if that’s the only feasible solution we have.

Updated by Nick Lewis 6 months ago

So to clarify before I get myself further entangled in this mess: is it actually a requirement that we be able to run applications from the modulepath on the master without first having pluginsunc them? I guess the case where this matters is a master which is not also an agent. Do we need to support that case, or is sufficient to say in that case you must add the modules you need to your RUBYLIB?

Updated by Daniel Pittman 6 months ago

  • Status changed from Investigating to Needs Decision
  • Assignee changed from Nick Lewis to Nigel Kersten

Nigel Kersten wrote:

Great update Nick. Just pointing out that not playing well with environments is a huge problem, and I’d like to have a chat in person if that’s the only feasible solution we have.

After some discussion, it sounds like this is more complex: for a master/agent model it shouldn’t be using content from modules before they are pushed to the system with pluginsync. (That also helps address the case of environments, where you might have different versions of the same application in multiple environments.)

There might be some complexity around the puppet apply case, where we don’t do pluginsync, but we do want to see some code from modules. That, fairly obviously, doesn’t apply to other top level commands in Puppet though. (…as, if we are running apply we can’t also be running something else. :)

I have assigned this back to Nigel for now, since he can’t work out the details today…

Updated by Nigel Kersten 5 months ago

I’m not sure what actual decision is left here for me to make.

  • We don’t need to support loading apps from modules on a master that haven’t been pluginsynced to the local agent
  • We do need to play well with environments
  • We do need to work with puppet apply and a local module. If this means we need to implement “pluginsync” on puppet apply (from the local modulepath source) for consistency, I’m ok with that.

Updated by Daniel Pittman 5 months ago

  • Status changed from Needs Decision to Accepted
  • Assignee deleted (Nigel Kersten)

Thanks. That seems to cover any questions – perhaps just a restatement was all that was needed. :)

Updated by eric sorenson 4 months ago

  • Assignee set to Nick Lewis

Triage-a-thon: This is Accepted, marked as Urgent, and target for 2.7.x even — but has no Assignee. Sending it to Nick as he seems to be the one doing the actual work here. :)

Updated by Daniel Pittman 4 months ago

  • Assignee deleted (Nick Lewis)

Updated by Chris Price 3 months ago

  • Assignee set to Chris Price

Updated by Nick Lewis 3 months ago

I have a partial branch for this, that adds support for loading applications from $libdir, as well as from the modulepath, by using the autoloader. The latter behavior is undesired, so should be removed, but there is some work around refactoring the bootstrap process a bit which may be useful.

https://github.com/nicklewis/puppet/tree/ticket/2.7.x/7316

Updated by Chris Price 3 months ago

Initial acceptance test for this here:

https://github.com/puppetlabs/puppet/pull/518

These don’t deal with anything related to environments yet.

Updated by Matthew Black 3 months ago

Nick Lewis wrote:

I have a partial branch for this, that adds support for loading applications from $libdir, as well as from the modulepath, by using the autoloader. The latter behavior is undesired, so should be removed, but there is some work around refactoring the bootstrap process a bit which may be useful.

https://github.com/nicklewis/puppet/tree/ticket/2.7.x/7316

Not sure why it would need to load from the modulepath, wouldnt it load it from libdir for the environment that the server requested from? Even the server running the puppet master would download it to its libdir on a plugin sync for its environment.

Updated by Chris Price 3 months ago

correct. we won’t be loading from the module path.

Updated by Daniel Pittman 3 months ago

  • Status changed from Accepted to Merged - Pending Release
  • Target version changed from 2.7.x to 2.7.11

Updated by Chris Price 3 months ago

  • Status changed from Merged - Pending Release to Code Insufficient
  • Target version changed from 2.7.11 to 3.x

This isn’t actually fixed yet. Pull request 518 ( https://github.com/puppetlabs/puppet/pull/518 ) was merged, but that just contains a (failing) acceptance test that I wanted to get in in case anyone else besides myself was looking at this ticket.

Updated by Chris Price 2 months ago

  • Status changed from Code Insufficient to In Topic Branch Pending Review
  • Branch set to https://github.com/puppetlabs/puppet/pull/571

Updated by Chris Price 2 months ago

  • Status changed from In Topic Branch Pending Review to Merged - Pending Release

Updated by Daniel Pittman 14 days ago

  • Target version changed from 3.x to 3.0.0

Also available in: Atom PDF