Puppet

It appears that the catalog has characters with the high-bit set (e.g., UTF-8 characters, characters that have been through an overly helpful editor, raw binary data, or…).

• If you can find where that is, and it is an error / typo, try correcting it.
  If that fixes things we just have a less than helpful error message.
  If that doesn’t fix it, look for other occurrences ** If that still doesn’t fix it, tell me I’m wrong & I’ll try again.
• If it’s intentional/needed, please update the ticket with an explanation
• If you can’t find any high-bit character, and the puppet code is reasonably small, please post it.
• Otherwise, let me know why none of the above fit the case & I’ll try again.

so I dumped the response.body I got from the master, and then tried to debug the @PSON.parse@ from within irb. I put a puts edge on @/usr/local/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:417@ and this what I got:

[...]
sourceLine[enable_puppetd_on_boot]targetExec[perl -ni -e 'print if $_ ne "if [ -x /usr/local/bin/puppetd ]; then echo -n " puppetd"; /usr/local/bin/puppetd; fi
";' '/etc/rc.local']
ArgumentError: Could not convert from pson: Could not find relationship target ''
        from /usr/local/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:439:in `edge_from_pson'
        from /usr/local/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:418:in `from_pson'
        from /usr/local/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:416:in `each'
        from /usr/local/lib/ruby/site_ruby/1.8/puppet/resource/catalog.rb:416:in `from_pson'
        from /usr/local/lib/ruby/site_ruby/1.8/puppet/util/pson.rb:11:in `pson_create'
        from /usr/local/lib/ruby/site_ruby/1.8/puppet/external/pson/pure/parser.rb:256:in `parse_object'
        from /usr/local/lib/ruby/site_ruby/1.8/puppet/external/pson/pure/parser.rb:97:in `parse'
        from /usr/local/lib/ruby/site_ruby/1.8/puppet/external/pson/common.rb:133:in `parse'
        from (irb):3

This is the first kind of resource like that, but there are further. I tried to load the dumped catalog as well on other systems and it failed to parse there as well. So imho it is a general problem of pson.

The resource in question is the following:

http://git.puppet.immerda.ch/?p=module-puppet.git;a=blob;f=manifests/openbsd.pp;h=d08a50bdc4738c6cbd630d19659bc6328a7844e8;hb=e80e95edb9007440faa9251a6e6621116783ed91#l9

which is this define:

http://git.puppet.immerda.ch/?p=module-openbsd.git;a=blob;f=manifests/rc_local.pp;h=7f2ea8bb32d41c36fb938d8e06fdd83209b5f21d;hb=59d3fa06836d8d07109908ac1894002fde8e0f07

and the line in question is this one:

http://git.puppet.immerda.ch/?p=module-openbsd.git;a=blob;f=manifests/rc_local.pp;h=7f2ea8bb32d41c36fb938d8e06fdd83209b5f21d;hb=59d3fa06836d8d07109908ac1894002fde8e0f07

As this worked on 0.24.8 and seems to be a valid puppet statement I would say that it is a pson problem.

If you need any further infos, please let me know.

Could you post the pson you captured as well?

So far as I can tell the main issue is the semi-spurious warning; unless you care about UTF-8 support in your catalog (and I don’t see why you should) simply omitting ruby-iconv and ignoring the warning should work. Under that assumption, we should fix the handling of that error.

If you do need iconv/UTF-8 support the next thing to try would probably be to recompile ruby with current iconv available, as there appear to be issues with just dropping ruby-iconv in under openbsd. Before going that route, I’d be interested in hearing why you need UTF-8 support.

Markus Roberts wrote:

Could you post the pson you captured as well?

I sent it to you by Mail. Hope that’s fine.

So far as I can tell the main issue is the semi-spurious warning; unless you care about UTF-8 support in your catalog (and I don’t see why you should) simply omitting ruby-iconv and ignoring the warning should work. Under that assumption, we should fix the handling of that error.

The message of this error is printed with @warn@. As I encountered the ugliness of warn different times, I thought about writing to the list for a general discussion for the usage of warn.

If you do need iconv/UTF-8 support the next thing to try would probably be to recompile ruby with current iconv available, as there appear to be issues with just dropping ruby-iconv in under openbsd. Before going that route, I’d be interested in hearing why you need UTF-8 support.

I don’t yet see why it is a UTF-8 issue and I don’t know why I should need it. :/ For sure puppet prints out this warning, but I haven’t done anything special that this appears.

However, as mentioned above I can also not parse the captured catalog on other systems like CentOS. So I think it’s more a problem of serialization of this rather ugly resource. This resource is really only used on OpenBSD Systems to enable the service during boot. I suspect that if I would use this resource as well on other systems it would fail as well with the very same error.

So I change the title to address the fact that it is a general issue and not a problem related to openbsd. I would change the priority back to normal as well, as I think it’s not really a very special limited issue. I think it’s more an issue of serialization. But as I didn’t yet fully understand why you see a problem with utf-8 (maybe we have all been misleaded by the first warning) I leave it up to you to categorize and prioritize the bug.

I have now also been working directly on the master (which is a CentOS box), compiled the catalog and tried to apply it locally. And it fails there to load as well.

# puppetmasterd --compile foobar.example.com > compiled_catalog
# #cleaning out the puppetmaster messages from compiled_catalog
# puppet --noop --apply compiled_catalog 
Could not run: Could not deserialize catalog from pson: Could not convert from pson: Could not find relationship target ''

I send you the compiled_catalog in another mail.

The problem seems really to exist only if I use the openbsd::rc_local hence the rather complicated line statement. But I think this is just one example where such problems may rise.

The UTF-8 aspects are probably a red-herring (though I’m still not clear about your comment “Runnning puppet without ruby-iconv installed works fine so far”). Do you mean by this that the pson problems do not occur when iconv is absent?

Markus Roberts wrote:

The UTF-8 aspects are probably a red-herring (though I’m still not clear about your comment “Runnning puppet without ruby-iconv installed works fine so far”). Do you mean by this that the pson problems do not occur when iconv is absent?

yes, they don’t occur and puppet seems to work, except the nasty message print to stderr. I assumed puppet switches to yaml?

Markus Roberts wrote:

The UTF-8 aspects are probably a red-herring (though I’m still not clear about your comment “Runnning puppet without ruby-iconv installed works fine so far”). Do you mean by this that the pson problems do not occur when iconv is absent?

Peter Meier wrote:

yes, they don’t occur and puppet seems to work, except the nasty message print to stderr. I assumed puppet switches to yaml?

Ah. That puts a different complexion on it. Here’s what I think is going on:

• You have a resource with a namevar, title, or target that contains or consists of characters with the high bit set. Depending on the environment you may see these as the corresponding 7-bit ASCII character or as a unicode character that, to the human eye, looks like one.
• Without iconv, these are passed verbatum and everything works, apart from the warning, because what the client gets is exactly what’s on the server.
• With iconv installed (and assuming //IGNORE) it removes these and, on encoding, produces relations that have an empty target designation.
• This then fails

It matches all the evidence save one point: in the catalog you emailed there is no evidence of such a character.

However, the data was also split into multiple lines and showed other evidence that some process had massaged it in transit, and it is possible that this process also removed the odd character(s). Can you describe how you captured it?

Thanks, — Markus

Peter Meier wrote:

yes, they don’t occur and puppet seems to work, except the nasty message print to stderr. I assumed puppet switches to yaml?

Ah. That puts a different complexion on it. Here’s what I think is going on:

• You have a resource with a namevar, title, or target that contains or consists of characters with the high bit set. Depending on the environment you may see these as the corresponding 7-bit ASCII character or as a unicode character that, to the human eye, looks like one
• Without iconv, these are passed verbatum and everything works, apart from the warning, because what the client gets is exactly what’s on the server.
• With iconv installed (and assuming //IGNORE) it removes these and, on encoding, produces relations that have an empty target designation.
• This then fails

ok, so this would make sense.

It matches all the evidence save one point: in the catalog you emailed there is no evidence of such a character.

:/ I couldn’t find anyone either.

However, the data was also split into multiple lines and showed other evidence that some process had massaged it in transit, and it is possible that this process also removed the odd character(s). Can you describe how you captured it?

I sent you two. One I captured with the new compile option on the master. This one also fails to apply on the very same master (I tried it with —noop as it isn’t really the catalog for the master, but I think that doesn’t matter) as well on the failing client, as well other clients (openbsd or centos).

The other one I captured by putting

File.open('/tmp/catalog','w') { |f| f.puts response.body }

in /usr/local/lib/ruby/site_ruby/1.8/puppet/indirector/rest.rb on line 50.

I pack the 2 catalogs in a tar and send the archive to you, as they don’t consist of multiple lines on my systems. So maybe mail changed them :/

So I’m not seeing anything surprising/revealing in the catalogs you sent.

I think we’re on the right track, that running without iconv is the right thing to do in your environment, but I just can’t pin down the exact mechanism.

I’m inclined to note it as a bad interaction, remove/reduce the warning, and move on, unless you have any further insights.

Markus Roberts wrote:

So I’m not seeing anything surprising/revealing in the catalogs you sent.

But it’s also failing to parse for you? Or do you don’t even encounter these issues while loading the sent catalogs?

I think we’re on the right track, that running without iconv is the right thing to do in your environment, but I just can’t pin down the exact mechanism.

Well sending the iconv-message not to stderr can be a temporary solution, but if I use somehow ruby-iconv in the future for maybe some other ruby tool, I’m back to the same problem, as puppet seems to want to load iconv for some reason. I didn’t yet figure out where and why this happens, but there might be a valuable reason for it.

As well we still have the issue that I can compile a catalog on the puppetmasterd and if I try to apply it on the very same host (CentOS 5.4) it fails as well. And I can’t find any iconv library nor gem installed on this system. Currently I encounter this problem only on OpenBSD clients, but as I can reproduce the problem without having any OpenBSD involved I have to assume that it’s somewhere deeper and might come up in the future also on other hosts.

I’m inclined to note it as a bad interaction, remove/reduce the warning, and move on, unless you have any further insights.

reduce the warning would be anyway a nice thing, as messages to stderr aren’t imho that cool if you don’t fail at all. Anyway as noted above I fear that the problem is rather somewhere inside and it will pop up in the future as well if we use some special named resources.

Unfortunately I don’t have further insight. If you can reproduce the parse error with the sent catalogs I’ll try to look further into what might cause the problem. If you can’t even reproduce it I try to figure out if might be a serializing issue on Centos 5.4.

Markus Roberts wrote:

So I’m not seeing anything surprising/revealing in the catalogs you sent.

Peter Meier wrote: But it’s also failing to parse for you? Or do you don’t even encounter these issues while loading the sent catalogs?

I was not, but since I last posted I think I’ve gotten a lead on it. More when I know more.

The problem seems to be with resources such as

Exec[perl -ni -e 'print if $_ ne "if [ -x /usr/local/bin/puppetd ]; then echo -n " puppetd"; /usr/local/bin/puppetd; fi\n";' '/etc/rc.local']

which contain a newline in their title.

Try it without the \n after the fi.

Trying it myself, I discovered that the “\n"s were only part of the problem. The square brackets were the other part.

All symptoms accounted for, fix in the works.

The changes on this branch

http://github.com/MarkusQ/puppet/tree/ticket/master/2770

should fix it.

Markus Roberts wrote:

The changes on this branch

http://github.com/MarkusQ/puppet/tree/ticket/master/2770

should fix it.

the changes are only against master, or can they also be applied against 0.25?

I applied the patch to 0.25.x branch and it seems to work fine.

Pushed in commit:“3fdc8effbe25c9653d8bc86f2d4847984d4cb6f3” in branch 0.25.x