Bug #3019
Failed to generate resources / Could not retieve file metadata
| Status: | Needs More Information | Start date: | 01/08/2010 | |
|---|---|---|---|---|
| Priority: | Normal | Due date: | ||
| Assignee: | - | % Done: | 0% |
|
| Category: | fileserving | |||
| Target version: | - | |||
| Affected Puppet version: | 0.24.4 | Branch: | ||
| Keywords: | ||||
| Votes: | 4 |
Description
(Reported by Mark Plaksin)
I think I’m getting 3 different stack traces on our RHEL 4.7 box. (1) has “connect” at the top and is about a recursive file resource. (2) and (3) both start with “fail”. In (2) the file on the file server is symlink (and is inside a recursive file resource, in (3) the file is a normal file. The traces don’t always happen on the same files from run to run.
------------------------------------------------------------------------------ 1: /usr/lib/ruby/1.8/net/protocols.rb:49:in `connect' /usr/lib/ruby/1.8/net/protocols.rb:49:in `ssl_connect' /usr/lib/ruby/1.8/net/https.rb:181:in `on_connect' /usr/lib/ruby/1.8/net/http.rb:433:in `do_start' /usr/lib/ruby/1.8/net/http.rb:419:in `start' /usr/lib/ruby/1.8/net/http.rb:824:in `request' /usr/lib/ruby/1.8/net/http.rb:618:in `get' /usr/src/git/puppet/lib/puppet/indirector/rest.rb:73:in `search' /usr/src/git/puppet/lib/puppet/indirector/indirection.rb:240:in `search' /usr/src/git/puppet/lib/puppet/indirector.rb:59:in `search' /usr/src/git/puppet/lib/puppet/type/file.rb:595:in `perform_recursion' /usr/src/git/puppet/lib/puppet/type/file.rb:562:in `recurse_remote' /usr/src/git/puppet/lib/puppet/type/file.rb:561:in `collect' /usr/src/git/puppet/lib/puppet/type/file.rb:561:in `recurse_remote' /usr/src/git/puppet/lib/puppet/type/file.rb:483:in `recurse' /usr/src/git/puppet/lib/puppet/type/file.rb:385:in `eval_generate' /usr/src/git/puppet/lib/puppet/transaction.rb:349:in `send' /usr/src/git/puppet/lib/puppet/transaction.rb:349:in `generate_additional_resources' /usr/src/git/puppet/lib/puppet/transaction.rb:193:in `eval_generate' /usr/src/git/puppet/lib/puppet/transaction.rb:240:in `eval_children_and_apply_resource' /usr/src/git/puppet/lib/puppet/transaction.rb:207:in `eval_resource' /usr/src/git/puppet/lib/puppet/transaction.rb:296:in `evaluate' /usr/src/git/puppet/lib/puppet/transaction.rb:295:in `thinmark' /usr/src/git/puppet/lib/puppet/util.rb:405:in `measure' /usr/lib/ruby/1.8/benchmark.rb:342:in `realtime' /usr/src/git/puppet/lib/puppet/util.rb:405:in `thinmark' /usr/src/git/puppet/lib/puppet/transaction.rb:297:in `evaluate' /usr/src/git/puppet/lib/puppet/transaction.rb:289:in `collect' /usr/src/git/puppet/lib/puppet/transaction.rb:289:in `evaluate' /usr/src/git/puppet/lib/puppet/resource/catalog.rb:142:in `apply' /usr/src/git/puppet/lib/puppet/configurer.rb:153:in `run' /usr/src/git/puppet/lib/puppet/configurer.rb:152:in `benchmark' /usr/src/git/puppet/lib/puppet/util.rb:177:in `measure' /usr/lib/ruby/1.8/benchmark.rb:342:in `realtime' /usr/src/git/puppet/lib/puppet/util.rb:177:in `benchmark' /usr/src/git/puppet/lib/puppet/configurer.rb:152:in `run' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `run' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `lock' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `run' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `synchronize' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `run' /usr/src/git/puppet/lib/puppet/agent.rb:51:in `with_client' /usr/src/git/puppet/lib/puppet/agent.rb:58:in `run' /usr/src/git/puppet/lib/puppet/application/puppetd.rb:103:in `onetime' /usr/src/git/puppet/lib/puppet/application/puppetd.rb:94:in `onetime' /usr/src/git/puppet/lib/puppet/application.rb:226:in `send' /usr/src/git/puppet/lib/puppet/application.rb:226:in `run_command' /usr/src/git/puppet/lib/puppet/application.rb:217:in `run' /usr/src/git/puppet/lib/puppet/application.rb:217:in `exit_on_fail' /usr/src/git/puppet/lib/puppet/application.rb:217:in `run' sbin/puppetd:159 err: //allusers::bjones/Allusers::Userfile[bjones/bin]/File[/home/bjones/bin]: Failed to generate additional resources using 'eval_generate':
NOTE: There’s nothing after that final colon except for a single space.
------------------------------------------------------------------------------ 2: /usr/src/git/puppet/lib/puppet/parameter.rb:401:in `fail' /usr/src/git/puppet/lib/puppet/type/file/source.rb:157:in `init_metadata' /usr/src/git/puppet/lib/puppet/type/file/source.rb:149:in `each' /usr/src/git/puppet/lib/puppet/type/file/source.rb:149:in `init_metadata' /usr/src/git/puppet/lib/puppet/type/file/source.rb:146:in `init_metadata' /usr/src/git/puppet/lib/puppet/util/cacher.rb:106:in `send' /usr/src/git/puppet/lib/puppet/util/cacher.rb:106:in `cached_value' /usr/src/git/puppet/lib/puppet/util/cacher.rb:46:in `metadata' /usr/src/git/puppet/lib/puppet/util/cacher.rb:45:in `metadata' /usr/src/git/puppet/lib/puppet/type/file/source.rb:111:in `copy_source_values' /usr/src/git/puppet/lib/puppet/type/file.rb:630:in `retrieve' /usr/src/git/puppet/lib/puppet/type.rb:726:in `evaluate' /usr/src/git/puppet/lib/puppet/transaction.rb:62:in `apply' /usr/src/git/puppet/lib/puppet/transaction.rb:251:in `eval_children_and_apply_resource' /usr/src/git/puppet/lib/puppet/transaction.rb:250:in `thinmark' /usr/src/git/puppet/lib/puppet/util.rb:405:in `measure' /usr/lib/ruby/1.8/benchmark.rb:342:in `realtime' /usr/src/git/puppet/lib/puppet/util.rb:405:in `thinmark' /usr/src/git/puppet/lib/puppet/transaction.rb:252:in `eval_children_and_apply_resource' /usr/src/git/puppet/lib/puppet/transaction.rb:207:in `eval_resource' /usr/src/git/puppet/lib/puppet/transaction.rb:296:in `evaluate' /usr/src/git/puppet/lib/puppet/transaction.rb:295:in `thinmark' /usr/src/git/puppet/lib/puppet/util.rb:405:in `measure' /usr/lib/ruby/1.8/benchmark.rb:342:in `realtime' /usr/src/git/puppet/lib/puppet/util.rb:405:in `thinmark' /usr/src/git/puppet/lib/puppet/transaction.rb:297:in `evaluate' /usr/src/git/puppet/lib/puppet/transaction.rb:289:in `collect' /usr/src/git/puppet/lib/puppet/transaction.rb:289:in `evaluate' /usr/src/git/puppet/lib/puppet/resource/catalog.rb:142:in `apply' /usr/src/git/puppet/lib/puppet/configurer.rb:153:in `run' /usr/src/git/puppet/lib/puppet/configurer.rb:152:in `benchmark' /usr/src/git/puppet/lib/puppet/util.rb:177:in `measure' /usr/lib/ruby/1.8/benchmark.rb:342:in `realtime' /usr/src/git/puppet/lib/puppet/util.rb:177:in `benchmark' /usr/src/git/puppet/lib/puppet/configurer.rb:152:in `run' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `run' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `lock' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `run' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `synchronize' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `run' /usr/src/git/puppet/lib/puppet/agent.rb:51:in `with_client' /usr/src/git/puppet/lib/puppet/agent.rb:58:in `run' /usr/src/git/puppet/lib/puppet/application/puppetd.rb:103:in `onetime' /usr/src/git/puppet/lib/puppet/application/puppetd.rb:94:in `onetime' /usr/src/git/puppet/lib/puppet/application.rb:226:in `send' /usr/src/git/puppet/lib/puppet/application.rb:226:in `run_command' /usr/src/git/puppet/lib/puppet/application.rb:217:in `run' /usr/src/git/puppet/lib/puppet/application.rb:217:in `exit_on_fail' /usr/src/git/puppet/lib/puppet/application.rb:217:in `run' sbin/puppetd:159 err: //allusers::tss/Allusers::Userfile[tss/.screenrc]/File[/home/tss/.screenrc]: Failed to retrieve current state of resource: Could not retrieve file metadata for puppet://billy.bor.usg.edu/allusers/tss/.screenrc: at /usr/src/git/bor/puppet/modules/allusers/manifests/init.pp:23
------------------------------------------------------------------------------ 3: /usr/src/git/puppet/lib/puppet/parameter.rb:401:in `fail' /usr/src/git/puppet/lib/puppet/type/file/source.rb:157:in `init_metadata' /usr/src/git/puppet/lib/puppet/type/file/source.rb:149:in `each' /usr/src/git/puppet/lib/puppet/type/file/source.rb:149:in `init_metadata' /usr/src/git/puppet/lib/puppet/type/file/source.rb:146:in `init_metadata' /usr/src/git/puppet/lib/puppet/util/cacher.rb:106:in `send' /usr/src/git/puppet/lib/puppet/util/cacher.rb:106:in `cached_value' /usr/src/git/puppet/lib/puppet/util/cacher.rb:46:in `metadata' /usr/src/git/puppet/lib/puppet/util/cacher.rb:45:in `metadata' /usr/src/git/puppet/lib/puppet/type/file/source.rb:111:in `copy_source_values' /usr/src/git/puppet/lib/puppet/type/file.rb:630:in `retrieve' /usr/src/git/puppet/lib/puppet/type.rb:726:in `evaluate' /usr/src/git/puppet/lib/puppet/transaction.rb:62:in `apply' /usr/src/git/puppet/lib/puppet/transaction.rb:251:in `eval_children_and_apply_resource' /usr/src/git/puppet/lib/puppet/transaction.rb:250:in `thinmark' /usr/src/git/puppet/lib/puppet/util.rb:405:in `measure' /usr/lib/ruby/1.8/benchmark.rb:342:in `realtime' /usr/src/git/puppet/lib/puppet/util.rb:405:in `thinmark' /usr/src/git/puppet/lib/puppet/transaction.rb:252:in `eval_children_and_apply_resource' /usr/src/git/puppet/lib/puppet/transaction.rb:207:in `eval_resource' /usr/src/git/puppet/lib/puppet/transaction.rb:296:in `evaluate' /usr/src/git/puppet/lib/puppet/transaction.rb:295:in `thinmark' /usr/src/git/puppet/lib/puppet/util.rb:405:in `measure' /usr/lib/ruby/1.8/benchmark.rb:342:in `realtime' /usr/src/git/puppet/lib/puppet/util.rb:405:in `thinmark' /usr/src/git/puppet/lib/puppet/transaction.rb:297:in `evaluate' /usr/src/git/puppet/lib/puppet/transaction.rb:289:in `collect' /usr/src/git/puppet/lib/puppet/transaction.rb:289:in `evaluate' /usr/src/git/puppet/lib/puppet/resource/catalog.rb:142:in `apply' /usr/src/git/puppet/lib/puppet/configurer.rb:153:in `run' /usr/src/git/puppet/lib/puppet/configurer.rb:152:in `benchmark' /usr/src/git/puppet/lib/puppet/util.rb:177:in `measure' /usr/lib/ruby/1.8/benchmark.rb:342:in `realtime' /usr/src/git/puppet/lib/puppet/util.rb:177:in `benchmark' /usr/src/git/puppet/lib/puppet/configurer.rb:152:in `run' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `run' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `lock' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `run' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `synchronize' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `run' /usr/src/git/puppet/lib/puppet/agent.rb:51:in `with_client' /usr/src/git/puppet/lib/puppet/agent.rb:58:in `run' /usr/src/git/puppet/lib/puppet/application/puppetd.rb:103:in `onetime' /usr/src/git/puppet/lib/puppet/application/puppetd.rb:94:in `onetime' /usr/src/git/puppet/lib/puppet/application.rb:226:in `send' /usr/src/git/puppet/lib/puppet/application.rb:226:in `run_command' /usr/src/git/puppet/lib/puppet/application.rb:217:in `run' /usr/src/git/puppet/lib/puppet/application.rb:217:in `exit_on_fail' /usr/src/git/puppet/lib/puppet/application.rb:217:in `run' sbin/puppetd:159 err: //networker/File[/usr/local/bin/NSRpre_process]: Failed to retrieve current state of resource: Could not retrieve file metadata for puppet://billy.bor.usg.edu/networker/NSRpre_process: at /usr/src/git/bor/puppet/modules/networker/manifests/init.pp:26
Related issues
History
Updated by Markus Roberts about 2 years ago
Mark —
I tried to reproduce this on billy using the go.server / go.client scripts and did not see it or anything like it. Could you provide any hints as to how to reproduce this?
— Markus
Updated by Mark Plaksin about 2 years ago
billy’s running RHEL5 and is the server, squier is RHEL4. I got these running go.client on squier with your branch for #3013.
Updated by Markus Roberts about 2 years ago
Thanks, I’ll try to replicate.
Updated by Markus Roberts about 2 years ago
No luck so far, but I did get:
/usr/lib/ruby/1.8/net/protocols.rb:49:in `connect' /usr/lib/ruby/1.8/net/protocols.rb:49:in `ssl_connect' /usr/lib/ruby/1.8/net/https.rb:181:in `on_connect' /usr/lib/ruby/1.8/net/http.rb:433:in `do_start' /usr/lib/ruby/1.8/net/http.rb:419:in `start' /usr/lib/ruby/1.8/net/http.rb:824:in `request' /usr/lib/ruby/1.8/net/http.rb:618:in `get' /usr/src/git/puppet/lib/puppet/indirector/rest.rb:69:in `find' /usr/src/git/puppet/lib/puppet/indirector/indirection.rb:195:in `find' /usr/src/git/puppet/lib/puppet/indirector.rb:51:in `find' /usr/src/git/puppet/lib/puppet/configurer.rb:94:in `retrieve_catalog' /usr/src/git/puppet/lib/puppet/configurer.rb:93:in `thinmark' /usr/src/git/puppet/lib/puppet/util.rb:405:in `measure' /usr/lib/ruby/1.8/benchmark.rb:342:in `realtime' /usr/src/git/puppet/lib/puppet/util.rb:405:in `thinmark' /usr/src/git/puppet/lib/puppet/configurer.rb:95:in `retrieve_catalog' /usr/src/git/puppet/lib/puppet/configurer.rb:146:in `run' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `run' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `lock' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `run' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `synchronize' /usr/src/git/puppet/lib/puppet/agent.rb:53:in `run' /usr/src/git/puppet/lib/puppet/agent.rb:51:in `with_client' /usr/src/git/puppet/lib/puppet/agent.rb:58:in `run' /usr/src/git/puppet/lib/puppet/application/puppetd.rb:103:in `onetime' /usr/src/git/puppet/lib/puppet/application/puppetd.rb:94:in `onetime' /usr/src/git/puppet/lib/puppet/application.rb:226:in `send' /usr/src/git/puppet/lib/puppet/application.rb:226:in `run_command' /usr/src/git/puppet/lib/puppet/application.rb:217:in `run' /usr/src/git/puppet/lib/puppet/application.rb:217:in `exit_on_fail' /usr/src/git/puppet/lib/puppet/application.rb:217:in `run' sbin/puppetd:159 err: Could not retrieve catalog from remote server: block type is not 01 warning: Not using cache on failed catalog err: Could not retrieve catalog; skipping run
on one attempt. I’ll be trying again this morning.
Updated by Markus Roberts about 2 years ago
- Status changed from Investigating to Needs More Information
- Assignee changed from Markus Roberts to Mark Plaksin
I get the above error even with 0.25.1; have you seen this previosly or do you have any idea what I may be doing wrong?
Updated by Markus Roberts about 2 years ago
- Target version changed from 0.25.3 to 0.25.4
Updated by Mark Plaksin about 2 years ago
I think that 0.25.1 on RHEL4 had other problems which may have masked the ones that cropped up when I tested your #3013 branch. I just retested with 0.25.3 on both the client (RHEL4) and the server (RHEL5) and got the same “block type” error as you saw.
As I mentioned via email, we’re suddenly and unexpectedly not running any more RHEL4 boxes (except for a few stragglers which will got to RHEL5 soon). But I’m happy to help debug further.
Updated by James Turnbull about 2 years ago
- Target version changed from 0.25.4 to 0.25.5
Updated by Markus Roberts about 2 years ago
Mark —
I’ve been working the block type errors while Jesse tries to glean something from the stack traces. It appears that the block errors are a problem with the certificates, and when I retry starting with a clean ssl directory (see go.server.mqr and go.client.mqr) I do not get them anymore.
However it now fails on the puppetmaster (squier) with:
err: Failed to find billy.bor.usg.edu via exec: Execution of '/usr/local/bin/node billy.bor.usg.edu' returned 1: No such file or directory - /usr/local/bin/node
which doesn’t seem related to the initial error either.
Thoughts?
— Markus
Updated by Jesse Wolfe about 2 years ago
It looks to me like the errors either occurred on the server (and perhaps there’s useful debug information there) or on the network (which would suggest that our error messages aren’t conveying enough information)
Updated by Mark Plaksin about 2 years ago
Hi:
billy (RHEL5) is the server. squier (RHEL4) is the client. We’ve never run the server on RHEL4 or squier. The “/usr/local/bin/node not found” message must have come from running the server on squier.
I just created go.client.49 and pointed it at a different SSL directory and signed squier’s new cert on billy. When I run that against the server billy on billy I get sporadic stack traces that seem to match the ones in this ticket. I also get this message:
notice: //puppet/Cron[sleepyRestartPuppet]/ensure: created “–”:6: bad command errors in crontab file, can’t install.
And the cron job isn’t created.
I’m not sure what happened with the old SSL dir. We had used been using it to test against billy for a while and it worked fine. Then those “01” errors started showing up.
Updated by Mark Plaksin about 2 years ago
- Assignee changed from Mark Plaksin to Markus Roberts
- Affected Puppet version changed from 0.25.2 to 0.25.4rc2
With 0.24.5rc2 I get the same behavior as described in #11.
Updated by Josh Endries about 2 years ago
I get these errors (mostly eval_generate) all the time on my hosts (WEBRick). It’s sporadic, doesn’t happen every run. It seems like it happens in modules that deal with many small files (Tomcat is ~2100 and Fedora is ~1800). I have been dealing with it (ok ok, ignoring it) for months and haven’t found any pattern to the errors yet. Some have error messages, some don’t. Most are usually EOF errors, which would make sense to me if there were a problem in a manifest but most of the time and whenever I run it manually the same host seems to work fine. I’m 99% certain networking is fine as it’s only one or two hosts that complain when I have dozens, and if the network were flaky I’d have all sorts of other errors. This is on 0.25.1. I haven’t ran the server in debug mode yet but I’ll set that up today and hope one of these comes through quickly. If this is ticket hijacking let me know and ignore me. :)
Mon Jan 25 07:35:07 -0500 2010 //Node[repo1]/Fedora[fedora]/Tomcat[fedora]/File[/usr/local/tomcat-fedora/webapps] (err): Failed to generate additional resources using ‘eval_generate’: end of file reached
Sat Dec 19 22:58:21 -0500 2009 //Node[dds1]/Dds[dds]/Tomcat[dds]/File[/usr/local/tomcat-dds] (err): Failed to generate additional resources using ‘eval_generate’:
Tue Jan 05 17:57:50 -0500 2010 //Node[repo6]/Proai[proai]/Tomcat[proai]/File[/usr/local/tomcat-proai/webapps] (err): Failed to generate additional resources using ‘eval_generate’: Tue Jan 05 17:57:50 -0500 2010 //Node[repo6]/Proai[proai]/Tomcat[proai]/File[/usr/local/tomcat-proai/webapps] (err): Failed to retrieve current state of resource: Could not retrieve file metadata for puppet:///modules/tomcat/6.0.20/titles/proai/1.2/webapps: at /etc/puppet/production/modules/tomcat/manifests/init.pp:143
Wed Jan 06 07:41:48 -0500 2010 //wfi/Tomcat[wfi]/File[/usr/local/tomcat-wfi/webapps] (err): Failed to generate additional resources using ‘eval_generate’: end of file reached
Fri Jan 22 17:14:07 -0500 2010 //Node[repo1]/Fedora[fedora]/File[/usr/local/fedora] (err): Failed to generate additional resources using ‘eval_generate’: end of file reached
Fri Jan 22 17:53:22 -0500 2010 //Node[repo1]/Fedora[fedora]/File[/usr/local/fedora] (err): Failed to generate additional resources using ‘eval_generate’: end of file reached
Sat Jan 23 05:39:16 -0500 2010 //Node[repo1]/Fedora[fedora]/File[/usr/local/fedora] (err): Failed to generate additional resources using ‘eval_generate’: Could not intern_multiple from marshal: marshal data too short
Updated by James Turnbull almost 2 years ago
- Target version changed from 0.25.5 to 49
Updated by S K over 1 year ago
- Affected Puppet version changed from 0.25.4rc2 to 0.24.4
Hi everyone,
I have probably the same issue – getting about five false emails per day from various puppet hosts (triggered by various File resources):
Wed Jun 02 17:31:00 +0000 2010 //core::users::aduser/Module_file[/home/aduser/.ssh/adkey]/File[/home/aduser/.ssh/adkey] (err): Failed to generate additional resources using ‘eval_generate’:
running puppet 0.25.4 from EPEL on CentOS 5.4.
getting about five to ten messages per day from whole platform.
any ideas how to get rid of it?
thanks, Steve.
Updated by S K over 1 year ago
actually I can see the issue is related to the ‘source=>’ parameter of File resource. according to http://docs.puppetlabs.com/references/latest/type.html#source following syntax is supported
file { “/path/to/my/file”:
source => [
"/modules/nfs/files/file.$host",
"/modules/nfs/files/file.$operatingsystem",
"/modules/nfs/files/file"
]
}
and the first existing source should be picked up. however this seems be the case of random failures (in my case) – in about 30% cases it sends false report complaining about some missing file with the suffix (eg. /modules/nfs/files/file.$host) instead of continuing in checking the next source – in my case there is always at least the file without any suffix (/modules/nfs/files/file) but the processing doesn’t get to this file.
To point out – this happens randomly in about 30% of runs. if the issue occurs and I restart the puppet agent to rerun again the issue doesn’t repeat. it is always related to the file {‘something": source = [“…/file.$host”, “…/file.$operatingsystem”, “…/file” ] }
Steve.
Updated by Markus Roberts over 1 year ago
- Assignee deleted (
Markus Roberts) - Target version changed from 49 to 4
Updated by James Turnbull 6 months ago
- Target version deleted (
4)