Bug #3581
tagmail + err: PGError: server closed the connection unexpectedly
| Status: | Closed | Start date: | 04/20/2010 | |
|---|---|---|---|---|
| Priority: | Normal | Due date: | ||
| Assignee: | % Done: | 0% |
||
| Category: | stored configuration | |||
| Target version: | 2.7.14 | |||
| Affected Puppet version: | 0.25.4 | Branch: | https://github.com/puppetlabs/puppet/pull/620 | |
| Keywords: | ||||
| Votes: | 2 |
Description
Hello all,
got a strange issue with puppet 0.25.4, tagmail and postgresql connections. Having the following setup running – puppet0.25.4 and foreman. As DB backend works a PostgreSQL database. We are managing aprox. 50 Clients. OS is only SLES10SP2. Now, when running puppet i got after some minutes the following error: err: PGError: server closed the connection unexpectedly. This only happens when reports is set to tagmail. It seems that puppet tries to open a connection but then stalls.
Related issues
History
Updated by James Turnbull about 2 years ago
- Category set to stored configuration
- Status changed from Unreviewed to Needs More Information
So it never happens with any other report type set?
Updated by Jan Doleschal about 2 years ago
Tried reports = foreman, that works. Trying reports = foreman,tagmail or reports = tagmail get this error. Spoke with our databse admins, they guess that the client sends an unexpected eof. Tried updating active record from version 2.2.3 to 2.3.5 and passenger from 2.2.2 to 2.2.11 no change. Seen this erros under puppet-0.24.8 but they where very sporadic and the connection came always back again.
Updated by Martha Greenberg over 1 year ago
This also happens to me on Debian Lenny. It happens with both 0.25.4 and 2.6.1. If I turn off tagmail reports (and leave reports=log,store), it goes away.
Updated by Ken Bowley over 1 year ago
I can also confirm that this is still the case with 2.6.4. We are currently using 0.25.5 with sqlite in production because of this bug, and I was hoping it would have been fixed by now. 2.6.4 and postgresql works fine in our test environment without tagmail, but the connection broken as soon as tagmail is used.
Updated by Sirtaj Kang 12 months ago
I am facing this problem too, and it appeared when I enabled tagmail+err. Restarting the puppet master appears to make this go away for a while, but I haven’t detected a pattern yet.
Environment notes:
- OS: debian squeeze, x64
- puppet: 2.6.2-4 (squeeze apt)
- postgres:8.4.7 (squeeze apt)
master config:
[master]
storeconfigs=true
thin_storeconfigs=true
dbadapter=postgresql
dbhost=localhost
dbname=xxxx
dbuser=xxx
dbpassword=xxx
report=true
reports=tagmail,rrdgraph
tagmap=/etc/puppet/tagmail.conf
report_sender_email=puppet-report@xxxxx
/etc/puppet/tagmail.conf (file perms – puppet/puppet/mode 440):
err: root@localhost
Updated by Kevin Collas-Arundell 11 months ago
James Turnbull wrote:
So it never happens with any other report type set?
We are seeing this issue as well with puppet 2.6.9 Unfortunately we are not using any other report types
I will set up a development environment later this week and enable other reports
Updated by Stefan Schulte 7 months ago
- Status changed from Needs More Information to In Topic Branch Pending Review
- Branch set to https://github.com/stschulte/puppet/tree/ticket/2.7.x/3581
The error is still present in recent puppet versions. I use puppet agent 2.6.1, puppet master 2.6.11, the mysql2 gem, passenger 3.0.9 and activerecord 3.0.10. I have set the smtpserver configuration setting (so puppet uses Net::SMTP instead calling sendmail to do the job). I got about 20 error messages a day.
I think it is a forking issue where the child process closes the connection, leaving the parent process with an invalid database handle. The patch in the branch field removes the fork call in the tagmail report (causing the puppet process to block until the email is sent). I patched my puppet master two days ago and have not seen the error since.
Updated by James Turnbull 6 months ago
- Assignee set to Jason McKerr
Related to #4694 and impacting the same customer.
Updated by Jason McKerr 4 months ago
- Assignee changed from Jason McKerr to Daniel Pittman
Updated by Daniel Pittman 4 months ago
- Status changed from In Topic Branch Pending Review to Code Insufficient
This is a larger issue – anywhere that uses fork in the code can trigger it, and tagmail just happens to be the biggest offender.
There is the start of a discussion here about how best to handle this: https://groups.google.com/forum/#!topic/puppet-dev/M32pl_Jxgw4
If anyone here has a strong view on that, please let us know, but regardless of what that works out I will shortly fix the problem so that not only tagmail, but all current uses of fork in the product are safe.
I have marked this “code insufficient” because the change solves the tagmail issue, but doesn’t get to the roots of the problem, and because that is significant enough I want to make sure we go there.
Updated by Daniel Pittman 4 months ago
The conclusion of that discussion is that we are going to introduce a new method to fork, closing file handles, and encourage use of that through the code base. Given we review all submissions in core, finding the problem being reintroduces is less likely to cause grief than trying to change the behaviour of a core method.
Updated by Kelsey Hightower about 1 month ago
- Assignee changed from Daniel Pittman to Kelsey Hightower
Updated by Kelsey Hightower about 1 month ago
- Branch changed from https://github.com/stschulte/puppet/tree/ticket/2.7.x/3581 to https://github.com/kelseyhightower/puppet/tree/kelseyhightower/ticket/2.7.x/3581_fork_that
I have a WIP branch going now. Will have someone review this to make sure I’m on the right track.
Updated by Kelsey Hightower about 1 month ago
- Status changed from Code Insufficient to In Topic Branch Pending Review
- Target version set to 2.7.x
- Branch changed from https://github.com/kelseyhightower/puppet/tree/kelseyhightower/ticket/2.7.x/3581_fork_that to https://github.com/puppetlabs/puppet/pull/620
Updated by Daniel Pittman about 1 month ago
- Status changed from In Topic Branch Pending Review to Merged - Pending Release
- Target version changed from 2.7.x to 2.7.14
Updated by Matthaus Litteken about 1 month ago
- Status changed from Merged - Pending Release to Closed
Released in Puppet 2.7.14rc1