Puppet

For a modest node with about 400 resources (ie not the largest one I have), storeconfig takes about 12s for a total compile time of 18s (and that's after not the first configuration run which should store everything in the database).
It is expected since there was no change in the manifests that this phase would have been immediate (ie nothing has changed, there is no reason to update anything in the database).

My database backend is a (rather powerful) MySQL server.

This is mainly because:

• the created schema has duplicated index (ie primary keys + another index on the primary key). This worsen such write workload. Another factor is that most varchar indexes contains the full field. Usually indexing other the 15 first chars is enough.
• for this specific node, 2400 SQL requests are generated :-O
  • Among those requests, there is only 1 UPDATE
  • 646 DELETE
    • 50% are DELETE FROM 'resource_tags'
    • 43% are DELETE FROM 'param_values'
  • 678 INSERT
    • 52% are INSERTs to the 'resource_tags' table
    • 42% are INSERTs to the 'param_values' table

The DELETE/INSERTs of resource_tags and param_values all comes from the DELETE and then INSERT again of only 36 resources.
All those 36 resources are of the form:
Mainresource::Subpart[title] or Mainresource::Subpart1::Subpart2[title].

The code that checks there are old resources in the database or new resources uses a case sensitive string comparison (see collection_merger)
of the keys of 2 hashes to be merged. If only one is present, the resource is either/deleted or inserted.
Unfortunately the resource ref of the resource coming from the database has the form:
Mainresource::subpart1::subpart2[title] (notice the lowercases 'S' of the subparts).
Thus the collection_merger thinks both are two different resources and deletes the one coming from the database and creates a new one.
This all comes from Puppet::Rails::Resource::ref which is defined as:

def ref
    "%s[%s]" % [self[:restype].capitalize, self[:title]]
end

Capitalize capitalizes the first character but lowercases everything else. The fix is (patch against head enclosed too):

def ref
    "%s[%s]" % [self[:restype].split("::").collect { |s| s.capitalize }.join("::"), self[:title]]
end

With this patch applied the storeconfig part for this specific node is down to 6s instead of 12s.
With this patch applied, there are still 1000 SQL requests issued (mainly deletion/insertion of resource parameters).
An analysis of those requests show that Puppet::Parser::Resource::Param::modify_rails_values is comparing compiled resource references
(of type Puppet::Parser::Resource::Reference) with Puppet::Parser::Resource::Reference coming form the database.
Those can't match because Puppet::Parser::Resource::Reference doesn't override .

I enclosed a temptative fix, that brings my testcase storeconfig to 4s instead of 6s.
The real fix might be to override in Puppet::Parser::Resource::Reference.

More important, now a storeconfig session for a node contains almost only SELECT when the manifests didn't change (only 9% of INSERT).

There are still some room for improvements (ie preload the tags while loading the host) to remove the high number of SELECTs issued.