Bug #7959

augeas type_check RAM usage

Added by Markus Falb 11 months ago. Updated 11 months ago.

Status:Rejected Start date:06/16/2011
Priority:Normal Due date:
Assignee:Jacob Helwig % Done:

0%
Category:agent
Target version:-
Affected Puppet version: Branch:
Keywords:augeas
Votes: 1

Description

augeas { "ram fresser":
    type_check => true,
    context => "/files/etc/sysctl.conf",
    changes => "set net.ipv4.ip_forward 1",
}

and puppet needs 1GB RAM ? This looks wrong to me. I only see this behaviour if type_check is true.

History

#1
Updated by Adrien Thebo 11 months ago

Could you provide platform details for this, such as the operating system type and release, the version of puppet used, and the versions of augeas and the ruby augeas bindings installed?

#2
Updated by Markus Falb 11 months ago

CentOS release 5.6 (Final)
hardwareisa => x86_64

facter-1.5.8-1.el5
puppet-0.25.5-1.el5
ruby-augeas-0.4.1-1.el5
augeas-libs-0.8.1-2.el5
(all from EPEL with CentOS's ruby-1.8.5-5.el5_4.8)

#3
Updated by Dominic Cleal 11 months ago

  • Keywords set to augeas

Type checking is performed by the Augeas library itself. Puppet simply allows you to flip the flag on and off, so I think this is more of an upstream issue.

The src/augeas.h comments say “since it can be very expensive it is not done by default”. You can see this just from running augtool with type checking enabled:

$ /usr/bin/time augtool -c quit
55.34user 1.36system 0:57.23elapsed 99%CPU (0avgtext+0avgdata 786044maxresident)k
0inputs+0outputs (0major+773569minor)pagefaults 0swaps

It’s using up to 767MB to type check every lens and takes nearly a minute!

As the type checking will be performed for every lens loaded, then you’re creating a lot of work for the library. I’d suggest you use the Puppet type’s lens/incl attributes to reduce the workload to a single lens and see if that improves matters for you.

#4
Updated by Jacob Helwig 11 months ago

  • Status changed from Unreviewed to Investigating
  • Assignee set to Jacob Helwig

Markus: As Dominic pointed out, this really does look like an upstream issue with Augeas itself.

Dominic: I’ve seen suggestions that people use the lens/incl attributes come up fairly frequently recently. Should these be made required rather than optional? I’m afraid I don’t actually know that much about Augeas, but it seems like if it’s something that comes up as frequently as it has been then we should just be requiring it.

#5
Updated by Markus Falb 11 months ago

I tried setting incl and lens on a client with puppet 2.6.8 and it helps. puppet 0.25.5 does not have incl and lens parameters.

#6
Updated by Dominic Cleal 11 months ago

Jacob Helwig wrote:

Dominic: I’ve seen suggestions that people use the lens/incl attributes come up fairly frequently recently. Should these be made required rather than optional? I’m afraid I don’t actually know that much about Augeas, but it seems like if it’s something that comes up as frequently as it has been then we should just be requiring it.

I don’t think so. It’s only needed for two reasons:

  1. Supporting non-standard file locations or lenses
  2. Performance, as in this case: instead of Augeas loading every file it knows, you restrict it to speed up execution. Two other issues help this enormously, #2728 doubles speed when a change has to be made and #7285 would also make a big difference (another 50% for me).

I think the performance fixes will help the occasional query like this, and other queries are expected use cases for lens+incl. To make it mandatory would make specifying and using the type much harder for the large majority of cases with little benefit other than speed.

It also requires that the user work out which lens is used for their particular file, which requires some code diving or grepping.

#7
Updated by Dominic Cleal 11 months ago

Markus Falb wrote:

I tried setting incl and lens on a client with puppet 2.6.8 and it helps. puppet 0.25.5 does not have incl and lens parameters.

Glad it helps. I forgot to ask though – why are you setting it? Type checking is normally used at lens design or testing time (with augparse) and isn’t something you’d usually want to do at runtime (for the reasons we’ve seen!).

#8
Updated by Markus Falb 11 months ago

Dominic Cleal wrote:

Markus Falb wrote:

I tried setting incl and lens on a client with puppet 2.6.8 and it helps. puppet 0.25.5 does not have incl and lens parameters.

Glad it helps. I forgot to ask though – why are you setting it? Type checking is normally used at lens design or testing time (with augparse) and isn’t something you’d usually want to do at runtime (for the reasons we’ve seen!).

I really dont know. I have several augeas resources but this one is the only one with type_check set. I do not know why it is there. After this discussion and after glancing over http://augeas.net/page/Ambiguities_or_what_do_those_error_messages_from_the_typechecker_mean which states

once a module has been successfully typechecked, there's no need to typecheck it again

I am pretty sure that I will throw away this type_check. After all I fear I produced just noise. Thanks.

#9
Updated by Dominic Cleal 11 months ago

  • Status changed from Investigating to Rejected

No worries, probably best to remove it then.

Also available in: Atom PDF