Puppet

This is a missing feature, not a bug. I don’t think it will be implemented for a while yet, since it requires considerable changes in the library.

This is now not all that difficult, now that resources can generate new resources at run-time using ‘eval_generate’, but it will require additional configuration, maybe ‘recurse => upward’ or something, although that would make it impossible to recurse both directions.

Marked #668 as a duplicate, and fixed description.

I’m gonna take a crack at this.

I’m gonna have to admit defeat on this one; don’t currently have the bandwidth to get this fixed.

A usable workaround exists (courtesy of Volcane on #puppet):

file{ [“/foo”, “/foo/bar”, “/foo/bar/baz”]: ensure => directory }

I’d also like this fixed too…

a discussion on #puppet lead to the question “who owns the directories?”. If puppet creates directories recursive, should it own them (setting permissions and stuff) or not. And should it also set rights on existing folders?

To avoid implicit declaration one possible syntax would be:

file { “/var/long/path/here”: ensure => “directory”, recursive => “/var/long”, owner => “foo”, }

this would update ownership (and create directories if they are missing) for /var/long/path and /var/long/path/here

berge wrote:

A usable workaround exists (courtesy of Volcane on #puppet):

file{ [“/foo”, “/foo/bar”, “/foo/bar/baz”]: ensure => directory }

fragfutter wrote:

a discussion on #puppet lead to the question “who owns the directories?”. If puppet creates directories recursive, should it own them (setting permissions and stuff) or not. And should it also set rights on existing folders?

To avoid implicit declaration one possible syntax would be:

file { “/var/long/path/here”: ensure => “directory”, recursive => “/var/long”, owner => “foo”, }

this would update ownership (and create directories if they are missing) for /var/long/path and /var/long/path/here

The solution proposed by berge seems to be less generic but more explicit.

Here’s a suggestion that I made recently on IRC which addresses the issue of predictability.

Given a definition something like this:

file{“/some/very/deep/path”: ensure => directory, owner => ‘root’, create_parents => true }

This should create /some, /some/very, /some/very/deep with the same perms/ownership as / and create /some/very/deep/path with the specified perms/ownership

If the definition changes to:

file{“/some/very/deep/path”: ensure => directory, owner => ‘foo’, create_parents => true }

then the ownership of /some/very/deep/path would be changed, but the parent dirs would not be affected

eh, what’s the difference between:

file{“/some/very/deep/path”: ensure => directory, owner => ‘root’, create_parents => true }

and

file{“/some/very/deep/path”: ensure => directory, owner => ‘foo’, create_parents => true }

then the ownership of /some/very/deep/path would be changed, but the parent dirs would not be affected

as I don’t see the difference (except at one point root is the owner and @ the other foo) I assume you forgot to remove the create_parents?

Still, what about:

file{"/var/lib/cache/foo/bar": ensure => directory, mode => 0700, create_parents => true }

as far as I understood your proposal this would set permission even on /var/lib/cache and upwards, which definately wouldn’t be what you’d like to do. So something like a stop point: /var/lib/cache/foo is imho definately necessary. So something like fragfutter’s solution still seems to be necessary or you go with the current solution.

I’ll try to answer based on my understanding of the IRC conversation yesterday, but of course Robin can followup..

Peter Meier wrote:

as I don’t see the difference (except at one point root is the owner and @ the other foo) I assume you forgot to remove the create_parents?

no, the create_parents can stay. here is the scenario: file{“/some/very/deep/path”: ensure => directory, owner => ‘root’, create_parents => true } You have this code in place for a while and it deploys to 10 hosts.. then you change owner to ‘foo’ which gets updated on those 10 hosts and also deploys to 10 new hosts. With this strategy, any ancestor directories that need to be created get perms/owner/group copied from the deepest existing parent, and the leaf uses the resource settings. So say / has 755/root/root on all your systems. Then the first 10 hosts get /some, /some/very and /some/very/deep also created with 755/root/root. /some/very/deep/path originally gets created with owner root, and the update only changes that dir to owner ‘foo’. The next 10 systems will result in everything matching because /some, /some/very and /some/very/deep again copy upwards from /, and only the leaf gets owner ‘foo’. Hope this helps.

as far as I understood your proposal this would set permission even on /var/lib/cache and upwards, which definately wouldn’t be what you’d like to do. So something like a stop point: /var/lib/cache/foo is imho definately necessary. So something like fragfutter’s solution still seems to be necessary or you go with the current solution.

I think in this example /var/lib/cache/foo would be created with matching perms/owner/group as /var/lib/cache, and “bar” would be created with mode 700.

So far all these suggestions are kind of unappealing to me. The idea of perms being copied from the deepest existing parent is somewhat counter-intuitive to me.

Just my 2c. I realize I’m more than welcome not to use such a feature if it is implemented :)

I’m dead against anything that is non deterministic, giving file this ability means you make it on par with installing a package with yum/apt etc and getting a ton of unknown dependencies. You’d pick up on weird things like the package situation in dev ofcourse but what about file{} as suggested above?

file{“/some/very/deeply/nested/file”: create_parents => true}

in most cases this is fine as described, you get the parent dirs made once in a sane and predictable way.

should I add file{“/some/very”: ensure => directory, owner => “another” } though the file resource above would not fix it. ie. /some/very/deeply directory would not get adjusted to match the new parent dir ownership – see below.

We just do not have state anywhere to say a file as in the first block also created all the parents at some point months ago and store that for all future invocations to then also create these additional resources. Even creating resources on the fly to fill those gaps wont work because presumably on the 2nd run the logic that made all the parent resources will be skipped since hte parents exist already and as described we will leave existing dirs alone, else where do you draw the line between not editing / ? or between not editing /var/cache but letting it edit /var/cache/foo? there’s way too many edge cases.

This further breaks if you have something like class apache::install { } that sets up the parent dir for all your vhosts and such using this feature, on another machine perhaps you created one of the parent dirs in another class already using file{}. The result would be that if you included apache::install on different machines the result across different type of machine would not be the same and would not be predictable.

If you used specifically specified resources for each of the parent dirs you’d know you’re causing a screwup cos it just wont compile thanks to the duplicate resource checks rather than silently creating a false sense of certainty that your machines are all behaving in the same manner because puppet built them and puppet is declaring the full known state.

I can imagine many many cases where this behaviour would result in unpredictable results either through the life of a machine, through the life of other machines that’s supposedly just like this one (by including all the same classes) or through other machines that tries to reuse classes used on this one. The predictable behavior of puppets core types – like file, etc and unlike augeas for example – is what makes it a usable tool.

Any added features that downgrade core types like file{} to something that’s unpredictable should be very very carefully considered.

We just do not have state anywhere to say a file as in the first block also created all the parents at some point months ago and store that for all future invocations to then also create these additional resources. Even creating resources on the fly to fill those gaps wont work because presumably on the 2nd run the logic that made all the parent resources will be skipped since hte parents exist already and as described we will leave existing dirs alone, else where do you draw the line between not editing / ? or between not editing /var/cache but letting it edit /var/cache/foo? there’s way too many edge cases.

This further breaks if you have something like class apache::install { } that sets up the parent dir for all your vhosts and such using this feature, on another machine perhaps you created one of the parent dirs in another class already using file{}. The result would be that if you included apache::install on different machines the result across different type of machine would not be the same and would not be predictable.

good catch about getting unpredictable, even the one over time. running the same puppet manifest on 2 hosts should get the same result.

A better way than just saying ‘create_parents => true’, would be ‘parents_upto => “/some/very”’, and having that be exactly equivalent to explicitly specifying every path inbetween “/some/very” and “/some/very/long/path” as file resources with the same parameters.

A slightly different way of achieving this would be to have a function dirs_between() that takes a directory and a subdirectory as arguments and returns a list of all directories inbetween, including the two endpoints. dirs_between(“/some/very”, “long/path”) would return the list [“/some/very”, “/some/very/long”, “/some/very/long/path”].

One would then use that like

    $dirs = dirs_between("/some/very", "long/path")
    file {
        $dirs:
            ensure => directory, owner => "someuser", mode => 0755;
    }

(It would be nice to be able to have the dirs_between() call directly in the file declaration, and not have the temporary variable $dirs, but at least 0.25.0rc1 barfs on that.)

I’m attaching a quick and dirty implementation of dirs_between(). It is sorely lacking in error handling, but seems to work as long as you don’t pass it bad parameters.

I have created myseld a patch on version 0.25.4

Then you can create “mkdir -p”

file { “/data/mysql/$hostname/db”: ensure => directory, recurse => true }

diff -up type/file/ensure.rb_ORIG type/file/ensure.rb —– type/file/ensure.rb_ORIG 2010-02-09 10:01:29.000000000 +0100 +++ type/file/ensure.rb 2010-02-09 10:14:34.000000000 +0100 @@ -61,10 +61,14 @@ module Puppet

    newvalue(:directory) do
        mode = @resource.should(:mode)
        parent = File.dirname(@resource[:path])

•      unless FileTest.exists? parent
  

•          raise Puppet::Error,
  

•              "Cannot create %s; parent directory %s does not exist" %
  

•                  [@resource[:path], parent]
  

•      if @resource.recurse?
  

•          FileUtils.mkdir_p(parent)
  

•      else      
  

•          unless FileTest.exists? parent
  

•              raise Puppet::Error,
  

•                  "Cannot create %s; parent directory %s does not exist" %
  

•                      [@resource[:path], parent]
  

•          end       
      end       
      if mode   
          Puppet::Util.withumask(000) do
  

I have created myseld a patch on version 0.25.4

Then you can create “mkdir -p”

file { “/data/mysql/$hostname/db”: ensure => directory, recurse => true }

this is, hmm, pretty ugly to abuse recurse like that. recurse has basically a totally different meaning. Anyway, what do you do if you set this file resource to absent? As noted in different parts of the discussion, the behavior becomes like that very inconsistent.

Idd it can be confusing and inconsistent If absent you can’t permit to delete also the parents it could be very painful

but I want this feature and I now the limits of it

Maybe if you write into the documents that if you delete a recurse directory that the parents won’t be deleted

Likes fair to me

Lots of reasons why we can’t do this, but it would be nice to have. I think we should list the limitations and side effects and still have it, but not make “absent” work.

Even if we just had …

File { expand(“/tmp/path/one/two”,1): ensure => directory } as shorthand for listing the arrays of each directory without including part of the root (which might be a duplicate).

Or just add mkdir -p into the provider (as mentioned) and surround it will skulls and crossbones about how it determines permissions, following what mkdir -p does for the most part, and listing when it’s a bad idea. After all, even Haskell has some provision for side effects :)

Longer term, if we duplicate resources, and they are 100% the same, maybe that’s not an error and they resolve to the same resource. I don’t know.

—Michael

I disagree.

If it’s just a “nice to have” then we have it. exec { “mkdir -p /foo/bar/foo” }

Consistency is important and I don’t see a good way of offering this functionality in the file type and still being consistent.

Handling creating a directory tree should not be any different than “mkdir -p” when creating a directory tree.

Handling complex permissions should be a separate transaction/resource.

A stipulation can be made that when a directory tree is made, the “tree” only can have one set of permissions/owner/group – as specified in the resource. This would start with the directory that is missing, down to the leaf. –

ie – if /opt/local was there, and the resource called for creation of /opt/local/lib/mylib/mysubdir the permissions/owner/group will start with the newly created directories, starting with “lib”, and including “mylib” & “mysubdir”.

Absent could just be a dangerous “rm -rf” at the leaf level specified. ie – using the above example if one wants to get rid of mylib/mysubdir, one would say

path    => "/opt/local/lib/mylib",

ensure => absent, recurse => true,

As said above, any complex permissions/owner/group should be handled separately.

If people want the file resource to chown, chmod or chgrp with a “-R/r”, that could also be done, but as a separate bug/feature. (I would say something similar to absent… start at the point specified in “path” or “name”)

exec’s ok… but from the http://docs.puppetlabs.com/guides/types/exec.html page:

There is a strong tendency to use exec to do whatever work Puppet can’t already do; while

this is obviously acceptable (and unavoidable) in the short term, it is highly recommended to migrate work from exec to native Puppet types as quickly as possible. If you find that you are doing a lot of work with exec, please at least notify us at Reductive Labs what you are doing, and hopefully we can work with you to get a native resource type for the work you are doing.

Sounds like this is a feature that a lot of people want.

Just need to agree on what the feature does :)

Regards, -Roy

A stipulation can be made that when a directory tree is made, the “tree” only can have one set of permissions/owner/group – as specified in the resource. This would start with the directory that is missing, down to the leaf. –

and this is exactly where this resource becomes unpredictable if on one host /foo/bar/a exists but on one one host only /foo/bar and your tree should create /foo/bar/a/b/c one one host puppet would manage (owner, etc.) folder a on the other not, where it might be different and for example not chmod 0755 which would break access to your leave for the daemon, which would make the puppet-run incomplete and not able to set everything correctly. Or: what if you have 2 trees: /foo/bar/a/b/c and /foo/bar/a/e/f in one tree the owner is set to root in the other it is set to mail. On a vanilla host /foo doesn’t exist. So what is owner /foo? Maybe root maybe mail? depends on which host which tree resource would have been applied earlier.

Furthermore in the second run: how would puppet know from which part of the tree it manages that resource and what should it do if somebody changes in the above example permission of b ? In the first run it would set them correctly and then somebody changes it and what would it do then in the second run?

ie – if /opt/local was there, and the resource called for creation of /opt/local/lib/mylib/mysubdir the permissions/owner/group will start with the newly created directories, starting with “lib”, and including “mylib” & “mysubdir”.

Absent could just be a dangerous “rm -rf” at the leaf level specified. ie – using the above example if one wants to get rid of mylib/mysubdir, one would say

path    => "/opt/local/lib/mylib",

ensure => absent, recurse => true,

and this makes it somehow very inconsistent: you wouldn’t have the same state as if you first manage the tree and then set it to absent, you would have some parts of the tree left, which means that creating and removing a resource wouldn’t do the same.

As said above, any complex permissions/owner/group should be handled separately.

If people want the file resource to chown, chmod or chgrp with a “-R/r”, that could also be done, but as a separate bug/feature. (I would say something similar to absent… start at the point specified in “path” or “name”)

this can imho get very cumbersome. furthermore as puppet has the paradigma to manage a resource only once in a manifest you wouldn’t be able to set the flags with a second file resource.

[…] Sounds like this is a feature that a lot of people want.

Yes, I agree on that. On the other side I can say that within the > 100 modules I wrote I encountered that problem maybe 10 times and half of them I realized that I create anyway a common structure for my systems which I can easily facter out into a seperate module and include that everywhere. And due to the auto-require of paths and the possibility to pass an array to the file resource the overhead is minimal. I know that minimal is somehow ugly, but:

Just need to agree on what the feature does :)

and how it fits into the model that puppet tries to ensure and so far nobody came up with a solution that is consistent and is easy to use.

…

and this is exactly where this resource becomes unpredictable if on one host

/foo/bar/a exists but on one one host only /foo/bar and your tree should create /foo/bar/a/b/c

so – how do you do this now?

one one host puppet would manage (owner, etc.) folder a on the other not, where it might 

Yes, if one wanted/needed a specific set of ownership on a tree, or part of a tree, that could/should be managed separate from the “create”. This would mandate the use of a “before” or “require”…

be different and for example not chmod 0755 which would break access to your 

leave for the daemon, which would make the puppet-run incomplete and not able to set everything correctly. Or: what if you have 2 trees: /foo/bar/a/b/c and /foo/bar/a/e/f in one tree the owner is set to root in the other it is set to mail. On a vanilla host /foo doesn’t exist. So what is owner /foo? Maybe root maybe mail? depends on which host which tree resource would have been applied earlier.

see above.

Furthermore in the second run: how would puppet know from which part of the tree it manages that resource and what should it do if somebody changes in the above example permission of b ? In the first run it would set them correctly and then somebody changes it and what would it do then in the second run?

I see this (#86) as a “create” issue – not an issue of maintaining tree permissions. That should be done as a different resource, with this as a “before”, or be “required”.

I see the “create_dir” option (maybe of ensure => ?) to handle ensuring the directories are created.

but… I’m rather newish to puppet and target managing clients not servers so I may not be seeing this through the same pair of glasses :)

Regards, -Roy

Would it be possible to do something like:

file { “/foo/voo/bar/baz” :

ensure => directory, between => {

start => "/foo", end => "/foo/voo/bar", owner => "foo", group => "foo", mode => 755, }

between => {

start => "/foo/voo/bar/baz", end => "/foo/voo/bar/baz", owner => "baz", group => "baz", mode => 775, }

}

or would that require re-writing chunks of puppet?

Regards, -Roy