Bug #1095
Puppetmaster leaving half-open connections
| Status: | Duplicate | Start date: | ||
|---|---|---|---|---|
| Priority: | High | Due date: | ||
| Assignee: |  Luke Kanies |
% Done: | 0% |
|
| Category: | network | |||
| Target version: | 0.25.0 | |||
| Affected Puppet version: | 0.25.0beta2 | Branch: | ||
| Keywords: | ||||
| Votes: | 0 |
Description
After a period of time ranging from a few hours to several days, puppetmaster begins leaving half open TCP connections in a CLOSE_WAIT state. It usually seems to happen to connections from clients, though at least once I’ve seen it hit the database connection (MySQL). Here’s an example:
[root@lorien ~]# lsof -i | grep 8140 puppetd 13420 root 7u IPv4 48150014 TCP lorien.wpi.edu:52225->lorien.wpi.edu:8140 (ESTABLISHED) puppetmas 13744 puppet 10u IPv4 47981997 TCP *:8140 (LISTEN) puppetmas 13744 puppet 205u IPv4 48146861 TCP lorien.wpi.edu:8140->DELENN.WPI.EDU:63688 (CLOSE_WAIT) puppetmas 13744 puppet 206u IPv4 48145681 TCP lorien.wpi.edu:8140->IVANOVA.WPI.EDU:54630 (CLOSE_WAIT) puppetmas 13744 puppet 208u IPv4 48146636 TCP lorien.wpi.edu:8140->DELENN.WPI.EDU:63687 (CLOSE_WAIT) puppetmas 13744 puppet 210u IPv4 48146848 TCP lorien.wpi.edu:8140->IVANOVA.WPI.EDU:58605 (CLOSE_WAIT)
Once puppetmaster starts leaking sockets like this, it seems unable to answer any new requests. In this example, you can see that the puppet client on the local machine (lorien) has opened a connection to puppetmaster, but puppetmaster has not responded. None of the log files on either master or client show that any progress has been made.
Sending a HUP to the server generates “Restarting” and “Shutting down” messages in syslog, but it never restarts. lsof shows that there are puppetmaster processes hanging around keeping the original set of half open sockets open, but nothing is listening for new connections anymore:
[root@lorien ~]# lsof -i | grep 8140 puppetmas 13744 puppet 205u IPv4 48146861 TCP lorien.wpi.edu:8140->DELENN.WPI.EDU:63688 (CLOSE_WAIT) puppetmas 13744 puppet 206u IPv4 48145681 TCP lorien.wpi.edu:8140->IVANOVA.WPI.EDU:54630 (CLOSE_WAIT) puppetmas 13744 puppet 208u IPv4 48146636 TCP lorien.wpi.edu:8140->DELENN.WPI.EDU:63687 (CLOSE_WAIT) puppetmas 13744 puppet 210u IPv4 48146848 TCP lorien.wpi.edu:8140->IVANOVA.WPI.EDU:58605 (CLOSE_WAIT)
A full restart of puppetmaster appears to be the only way to get things flowing again.
This is on 0.24.1 plus the patch from ticket 959. Let me know what other debugging info you’d like me to gather up.
Related issues
History
Updated by David Schmitt about 4 years ago
Please supply at least your Distro and Ruby version. It might also help to run the puppetmasterd with —trace and watch whether some hidden error occurs.
Updated by Luke Kanies about 4 years ago
- Status changed from 1 to Closed
- 7 set to fixed
I expect that this is fixed now that we’re no longer using keep-alive.
Please reopen if it’s still a problem in 0.24.4.
Updated by Frank Sweetser about 4 years ago
It looks like I’m seeing the same problem creeping back in on 0.24.4 running on Fedora 8. I tried switching from webbrick to mongrel behind an apache load balancer, and still had the same symptoms occur.
[fs@lorien webrick]$ rpm -q puppet puppet-0.24.4-1.fc8 [fs@lorien webrick]$ rpm -q ruby ruby-1.8.6.114-1.fc8
I have puppetmaster running with debug and trace on right now; I’ll post the captured output once the problem reoccurs.
Updated by Frank Sweetser about 4 years ago
I was able to reproduce the problem with trace and debug, but there is no additional information or any kind of errors in the output. I can still post the entire transcript if you like, though.
Any suggestions on how else to go about gathering troubleshooting information?
Updated by Frank Sweetser about 4 years ago
More evidence that the OS is accepting the connection, but puppetmaster is never noticing:
[root@lorien ~]# lsof -i | grep 8140 puppetmas 14849 puppet 9u IPv4 121846444 TCP *:8140 (LISTEN) puppetmas 14849 puppet 202u IPv4 122681142 TCP lorien.wpi.edu:8140->NOC1.WPI.EDU:57493 (CLOSE_WAIT) puppetmas 14849 puppet 206u IPv4 122681360 TCP lorien.wpi.edu:8140->GKAR.WPI.EDU:61305 (CLOSE_WAIT) puppetmas 14849 puppet 208u IPv4 122681924 TCP lorien.wpi.edu:8140->NOC1.WPI.EDU:57494 (CLOSE_WAIT) puppetmas 14849 puppet 209u IPv4 122681930 TCP lorien.wpi.edu:8140->GKAR.WPI.EDU:61306 (CLOSE_WAIT) puppetd 14961 root 7u IPv4 123868883 TCP lorien.wpi.edu:60217->lorien.wpi.edu:8140 (ESTABLISHED) [root@lorien ~]# netstat --inet | grep 8140 tcp 0 0 lorien.wpi.edu:8140 lorien.wpi.edu:60217 SYN_RECV tcp 0 0 lorien.wpi.edu:8140 LENNIER.WPI.EDU:50599 SYN_RECV tcp 0 0 lorien.wpi.edu:8140 centos.wpi.edu:52918 SYN_RECV tcp 0 0 lorien.wpi.edu:8140 ERWIN.WPI.EDU:50847 SYN_RECV tcp 117 0 lorien.wpi.edu:8140 LONDO.WPI.EDU:54496 ESTABLISHED tcp 38 0 lorien.wpi.edu:8140 GKAR.WPI.EDU:61305 CLOSE_WAIT tcp 118 0 lorien.wpi.edu:8140 LENNIER.WPI.EDU:63643 CLOSE_WAIT tcp 38 0 lorien.wpi.edu:8140 NOC1.WPI.EDU:57494 CLOSE_WAIT tcp 117 0 lorien.wpi.edu:8140 NOC1.WPI.EDU:60061 ESTABLISHED tcp 0 117 lorien.wpi.edu:60217 lorien.wpi.edu:8140 ESTABLISHED tcp 38 0 lorien.wpi.edu:8140 GKAR.WPI.EDU:61306 CLOSE_WAIT tcp 38 0 lorien.wpi.edu:8140 NOC1.WPI.EDU:57493 CLOSE_WAIT tcp 117 0 lorien.wpi.edu:8140 GKAR.WPI.EDU:61311 ESTABLISHED tcp 117 0 lorien.wpi.edu:8140 TRAPEZE.WPI.EDU:56065 ESTABLISHED tcp 118 0 lorien.wpi.edu:8140 LENNIER.WPI.EDU:65165 CLOSE_WAIT
Updated by Frank Sweetser almost 4 years ago
Looks like puppetmaster is leaking file handles:
[root@lorien ~]# ps aux | grep puppetmaster | grep -v grep puppet 5202 0.1 5.0 186424 170856 ? Ssl 05:08 1:47 /usr/bin/ruby /usr/sbin/puppetmasterd --logdest=syslog [root@lorien ~]# lsof -p 5202 | wc -l 260 [root@lorien ~]# lsof -p 5202 | grep rails.log | wc -l 103 [root@lorien ~]# lsof -p 5202 | grep "can't identify protocol" | wc -l 100
Updated by Lawrence Ludwig almost 4 years ago
Based upon my experiences with our puppetmaster setup, I am also reporting the same issues as described in this ticket.
More details can be found in this thread.
http://groups.google.com/group/puppet-users/browse_thread/thread/b64a1f2502b11cea/5539256edfc4b67b#5539256edfc4b67b
Updated by Lawrence Ludwig almost 4 years ago
Adding storeconfigs= to the mix it makes the same issue occur MUCH faster (within a few min) Basic symtoms are: – Increased CPU load from puppetmasterd (running three instances in my case) behind mongrel/apache setup – many CLOSE_WAIT connections to the puppetmasters – many file handlers open
My setup with puppetmaster is: – Xen based VPS – 1 full 2.33Ghz Xeon allocation – 2 GB of RAM – 3 puppetmaster instances running with mongrel/apache load balancer setup – CentOS 4.6 64-bit – ruby-1.8.6.111-1
Updated by Luke Kanies almost 4 years ago
- Status changed from Closed to Unreviewed
- Priority changed from Normal to High
- 3 changed from Unknown to Medium
I think this is a pretty big deal, so I’m going to look at it this week.
Updated by Luke Kanies almost 4 years ago
- Status changed from Unreviewed to Accepted
Updated by Frank Sweetser almost 4 years ago
- Status changed from Accepted to Closed
It looks like the root of the problem is that rails (or possibly activerecord) ships with a very low quality MySQL connector. The giveaway was finding this log message in the rails.log logfile:
@WARNING: You’re using the Ruby-based MySQL library that ships with Rails. This library is not suited for production. Please install the C-based MySQL library instead (gem install mysql).@
Ensuring that the mysql gem was installed completely cleared up the problem for me.
Updated by Lawrence Ludwig almost 4 years ago
fs wrote:
It looks like the root of the problem is that rails (or possibly activerecord) ships with a very low quality MySQL connector. The giveaway was finding this log message in the rails.log logfile:
@WARNING: You’re using the Ruby-based MySQL library that ships with Rails. This library is not suited for production. Please install the C-based MySQL library instead (gem install mysql).@
Ensuring that the mysql gem was installed completely cleared up the problem for me.
This doesn’t explain in my case who’s not using storeconfigs=
Also the wiki should be updated to reflect the needing of mysql gem.
Updated by James Turnbull almost 4 years ago
lludwig – It’s already in the wiki – http://reductivelabs.com/trac/puppet/wiki/UsingStoredConfiguration.
Updated by Lawrence Ludwig almost 4 years ago
- Status changed from Closed to Unreviewed
Updated by Luke Kanies almost 4 years ago
- Status changed from Unreviewed to Needs More Information
Larry,
Are you sure you’re not running out of filehandles? Can you check the output of lsof for the process that’s hanging? It’s important that you check the whole fd list, not just network.
And please provide the output of ‘ulimit -n’, so we know what your fd limit is.
Updated by Lawrence Ludwig almost 4 years ago
luke wrote:
Larry,
Are you sure you’re not running out of filehandles? Can you check the output of lsof for the process that’s hanging? It’s important that you check the whole fd list, not just network.
I don’t think so but that could be an issue, keep in mind puppetmaster is on Xen and Virtuozzo, and with pretty beefy setup too.
I will check this once it happens again.
And please provide the output of ‘ulimit -n’, so we know what your fd limit is.
ulimit -n¶
1024
Updated by Luke Kanies almost 4 years ago
- Assignee changed from Puppet Community to Luke Kanies
- Target version changed from 0.24.5 to 0.25.0
We can’t really pin this on a release until we get more information. I think we might have multiple, conflated problems here, too.
This needs coordinated debugging with the help of someone suffering from the problem, and AFAIK Larry is the only one with the actual issue. I’m glad to help debug it — I worked with fsweeter on it, which is how we resolved it was the mysql gem.
Please contact me on irc and we can try to figure it out, but in the meantime, I’m bumping this until we get more info.
Updated by Lawrence Ludwig almost 4 years ago
Right now I’m restarting puppetmasterd 4 times a day via cron.. I’ll wait until next week to stop this and contact you then.
Updated by Lawrence Ludwig almost 4 years ago
Please contact me on irc and we can try to figure it out, but in the meantime, I’m bumping this until we get more info.
Ok thanks… what’s your irc handle? ‘luke’?
Updated by Luke Kanies almost 4 years ago
- Category set to network
Updated by Nigel Kersten almost 4 years ago
Just to confuse matters more…
I was seeing this issue too with apache/mongrel, which was the reason why we switched to using pound instead of Apache, which resolved it entirely for us.
This doesn’t explain why fs was seeing it with webrick alone, but I’ve never tested webrick with any serious load at all.
Updated by Jason Goodwin over 3 years ago
I think I’m suffering from this issue as well. I can’t say I’ve noticed the open sockets specifically, but ever since I enabled storeconfigs I can’t keep my puppetmaster running for more than a few minutes. Typically it starts up ok, handles a few requests, and then at some point it just starts consuming 100% CPU. At that point, it stops responding to new connections from puppetd, and if I leave it alone for awhile it’ll eventually just crash. I’m using PostgreSQL for the DB for storeconfigs. I tried turning on debugging / trace / etc, but puppetmaster never seems to log any real errors. I ran strace against it, and typically when it hits the 100% CPU point, all it ever logs after that are thousands of lines like this:
23775 select(16, [8 10 14], [], [], {0, 832767}) = -1 EBADF (Bad file descriptor)
I’ve tried to track down what filehandle that is, but I never did see an open() call returning 16 (I’m not great at reading strace dumps, maybe I’m not doing it quite right).
Oh, I should mention I’m running Ubuntu 8.04 using the 0.24.4-3 packages of puppet and puppetmaster. ulimit -n => 1024 I’m currently using Webrick, as I only have about 40 clients checking in at 30 minute intervals and didn’t think I really needed to add the load balancing.
Let me know if I can provide any more details or if there’s any debugging anyone wants me to try, I’ve got time for it since this is causing me a lot of pain. I really wanted storeconfigs to work, as I’m trying out the external resources collection to generate nagios configs.
Updated by Jos Backus over 3 years ago
demotivator wrote:
23775 select(16, [8 10 14], [], [], {0, 832767}) = -1 EBADF (Bad file descriptor)
I’ve tried to track down what filehandle that is, but I never did see an open() call returning 16 (I’m not great at reading strace dumps, maybe I’m not doing it quite right).
select(2) (on FreeBSD) says:
int select(int nfds, fd_set *readfds, fd_set *writefds, fd_set *exceptfds, struct timeval *timeout);
So 16 is the number of fds in the fd_set to check. In the above example EBADF indicates that one of the descriptor sets specified an invalid descriptor, so at least one of 8,10 or 14 is invalid.
Updated by Eric Dennis over 3 years ago
- Affected Puppet version set to 0.22.1
I am running into the same problem with my puppet setup. I have two separate puppet installations, one with ~150 nodes, and the other with ~300. Both puppetmasters are running on CentOS 5.2, using WEBrick, and have storeconfigs enabled. I am using the C-based MySQL libraries (MySQL gem, installed via the ruby-mysql RPM).
When puppetmaster hangs, I see inbound connections to puppetmaster start to stack up:
netstat -an | grep 8140 | grep SYN_RECV | wc -l¶
55
netstat -an | grep 8140 | wc -l¶
67
However, it doesn’t look like open filehandles are the problem here:
lsof -u puppet | wc -l¶
77
I do see multiple filehandles for the /var/log/puppet/rails.log and masterhttp.log files:
lsof -u puppet | grep rails.log | wc -l¶
10
lsof -u puppet | grep masterhttp.log | wc -l¶
3
puppetmaster is running puppet 0.24.7, clients are running versions between 0.24.4 and 0.24.7.
I’m going to try using Apache/Passenger to see if it alleviates the problem. Any other ideas?
Updated by Eric Dennis over 3 years ago
threetee wrote:
I’m going to try using Apache/Passenger to see if it alleviates the problem. Any other ideas?
I switched over to Apache/Passenger yesterday, and both puppetmaster servers are staying up so far. I’m going to continue to monitor for a while, but hopefully this has fixed it.
Updated by Luke Kanies over 3 years ago
- Status changed from Needs More Information to Closed
I think this was fixed with #961.
Updated by James Turnbull over 3 years ago
- Target version changed from 0.25.0 to 0.24.8
Updated by Ricky Zhou almost 3 years ago
- Status changed from Closed to Re-opened
- Affected Puppet version changed from 0.22.1 to 0.25.0beta2
Hi, I am currently testing 0.25beta2 and I am also seeing this issue (specifically when stored configs is enabled). I have ruby-mysql and I don’t see any warnings about using the built-in mysql functions, so that doesn’t seem to be the problem in this case.
Specifically, I am seeing the same symptom where after starting puppetmasterd (I use mongrel behind apache/mod_proxy_balancer) and attempting a puppet run, the puppet run times out, with the connection stuck in CLOSE_WAIT. The mongrel instance then stops handling requests (so something like a curl http://127.0.0.1:18141/ will hang after this happens).
The output of puppetd -t I get (after a long wait) is:
notice: Ignoring --listen on onetime run
/usr/lib/ruby/1.8/net/protocol.rb:132:in `rbuf_fill': execution expired (Timeout::Error)
from /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
from /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'
from /usr/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'
from /usr/lib/ruby/1.8/net/http.rb:2009:in `read_new'
from /usr/lib/ruby/1.8/net/http.rb:1050:in `request'
from /usr/lib/ruby/1.8/net/http.rb:1037:in `request'
from /usr/lib/ruby/1.8/net/http.rb:543:in `start'
from /usr/lib/ruby/1.8/net/http.rb:1035:in `request'
... 18 levels...
from /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:226:in `send'
from /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:226:in `run_command'
from /usr/lib/ruby/site_ruby/1.8/puppet/application.rb:217:in `run'
from /usr/sbin/puppetd:159
and a netstat -pan | grep 1814 gives:
tcp 0 0 127.0.0.1:18140 0.0.0.0:* LISTEN 23420/ruby tcp 0 0 127.0.0.1:18141 0.0.0.0:* LISTEN 23472/ruby tcp 0 0 127.0.0.1:18142 0.0.0.0:* LISTEN 23524/ruby tcp 0 0 127.0.0.1:18143 0.0.0.0:* LISTEN 23576/ruby tcp 0 0 127.0.0.1:18140 127.0.0.1:46870 ESTABLISHED 23420/ruby tcp 0 0 127.0.0.1:47823 127.0.0.1:18141 FIN_WAIT2 - tcp 0 0 127.0.0.1:46870 127.0.0.1:18140 ESTABLISHED 6050/httpd tcp 1 0 127.0.0.1:18141 127.0.0.1:47823 CLOSE_WAIT 23472/ruby tcp 1 0 127.0.0.1:18141 127.0.0.1:47829 CLOSE_WAIT 23472/ruby
I did one more test where I pointed mod_proxy_balancer at only one mongrel instance, then did an strace -p on it and tried a puppet run. The strace output I got from that is available at http://ricky.fedorapeople.org/puppetmaster.strace.
If I can help with providing any more debugging information, I’m ricky in #puppet on Freenode – feel free to ping me any time.
Updated by Luke Kanies almost 3 years ago
- Target version changed from 0.24.8 to 0.25.0
Do you get the half-open connection on every run, or what?
Updated by Ricky Zhou almost 3 years ago
Yup, this has happened on every run I’ve tried so far with storeconfigs = true.
Updated by Luke Kanies almost 3 years ago
There’s definitely something else screwy going on, then – this is working for lots of people. Is there any chance you’ve got two ruby installs, and one has the mysql gem and the other one is running puppetmasterd?
Or something like that?
Updated by Ricky Zhou almost 3 years ago
I only have one ruby install, and here the full list of packages I have installed (I’m running Fedora 11 and I don’t have anything installed outside of package management): http://ricky.fedorapeople.org/rpmpkgs. Puppet/Facter are the only ruby apps I run, so hopefully the set of ruby packages is pretty clean.
Updated by Ricky Zhou almost 3 years ago
Hi, I just tried to reproduce this on another F11 machine. At first, I wasn’t able to, but after looking at the differences, I noticed that SELinux was disabled on the machine that did not exhibit the behavior. When I enabled SELinux (permissive, not enforcing), I was able to reproduce the issue. Here are the steps that I took to do so:
yum install httpd mod_ssl mysql-server ruby ruby-augeas ruby-shadow which ruby-RRDtool rubygem-rails rubygem-mongrel ruby-mysql facter
rpm -Uvh \
http://tmz.fedorapeople.org/repo/puppet/fedora/11/x86_64/puppet-0.25.0-0.2.beta2.fc11.noarch.rpm \
http://tmz.fedorapeople.org/repo/puppet/fedora/11/x86_64/puppet-server-0.25.0-0.2.beta2.fc11.noarch.rpm
/etc/init.d/mysqld start
mysql -u root
create database puppet;
grant all privileges on puppet.* to puppet@localhost identified by 'puppet';
wget -O /etc/puppet/puppet.conf http://ricky.fedorapeople.org/puppet.conf
echo 'PUPPETMASTER_PORTS=( 18140 18141 18142 18143 )' > /etc/sysconfig/puppetmaster
puppetca --generate publictest1.fedoraproject.org
wget -O /etc/httpd/conf.d/puppetmaster-mongrel.conf http://ricky.fedorapeople.org/puppetmaster-mongrel.conf
echo 'notice("This is a test")' > /etc/puppet/manifests/site.pp
/etc/init.d/puppetmaster start
/etc/init.d/httpd start
puppetd -t
Updated by Ricky Zhou almost 3 years ago
Here’s the output of puppetmasterd —debug —trace —no-daemonize —servertype=mongrel —masterport=18140 —color false on one of the machines where I was able to reproduce. I ran this command, then did a puppetd -t, which timed out. I ended up having to kill -9 the process when it did not respond to ctrl-c. Unfortunately, nothing seems too out of the ordinary in the output.
debug: /File[/var/lib/puppet/ssl/crl.pem]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/crl.pem debug: /File[/var/lib/puppet/ssl/crl.pem]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/crl.pem debug: /File[/var/lib/puppet/ssl/crl.pem]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/crl.pem debug: /File[/var/lib/puppet/ssl/crl.pem]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/crl.pem debug: /File[/var/run/puppet]/seluser: Found seluser default 'system_u' for /var/run/puppet debug: /File[/var/run/puppet]/selrole: Found selrole default 'object_r' for /var/run/puppet debug: /File[/var/run/puppet]/seltype: Found seltype default 'var_run_t' for /var/run/puppet debug: /File[/var/run/puppet]/selrange: Found selrange default 's0' for /var/run/puppet debug: /File[/var/lib/puppet/ssl/public_keys]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/public_keys debug: /File[/var/lib/puppet/ssl/public_keys]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/public_keys debug: /File[/var/lib/puppet/ssl/public_keys]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/public_keys debug: /File[/var/lib/puppet/ssl/public_keys]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/public_keys debug: /File[/etc/puppet]/seluser: Found seluser default 'system_u' for /etc/puppet debug: /File[/etc/puppet]/selrole: Found selrole default 'object_r' for /etc/puppet debug: /File[/etc/puppet]/seltype: Found seltype default 'etc_t' for /etc/puppet debug: /File[/etc/puppet]/selrange: Found selrange default 's0' for /etc/puppet debug: /File[/var/lib/puppet/rrd]/seluser: Found seluser default 'system_u' for /var/lib/puppet/rrd debug: /File[/var/lib/puppet/rrd]/selrole: Found selrole default 'object_r' for /var/lib/puppet/rrd debug: /File[/var/lib/puppet/rrd]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/rrd debug: /File[/var/lib/puppet/rrd]/selrange: Found selrange default 's0' for /var/lib/puppet/rrd debug: /File[/var/lib/puppet/ssl/certs]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/certs debug: /File[/var/lib/puppet/ssl/certs]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/certs debug: /File[/var/lib/puppet/ssl/certs]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/certs debug: /File[/var/lib/puppet/ssl/certs]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/certs debug: /File[/var/lib/puppet/ssl/certs/ca.pem]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/certs/ca.pem debug: /File[/var/lib/puppet/ssl/certs/ca.pem]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/certs/ca.pem debug: /File[/var/lib/puppet/ssl/certs/ca.pem]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/certs/ca.pem debug: /File[/var/lib/puppet/ssl/certs/ca.pem]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/certs/ca.pem debug: /File[/var/lib/puppet/facts/]/seluser: Found seluser default 'system_u' for /var/lib/puppet/facts debug: /File[/var/lib/puppet/facts/]/selrole: Found selrole default 'object_r' for /var/lib/puppet/facts debug: /File[/var/lib/puppet/facts/]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/facts debug: /File[/var/lib/puppet/facts/]/selrange: Found selrange default 's0' for /var/lib/puppet/facts debug: /File[/var/log/puppet]/seluser: Found seluser default 'system_u' for /var/log/puppet debug: /File[/var/log/puppet]/selrole: Found selrole default 'object_r' for /var/log/puppet debug: /File[/var/log/puppet]/seltype: Found seltype default 'var_log_t' for /var/log/puppet debug: /File[/var/log/puppet]/selrange: Found selrange default 's0' for /var/log/puppet debug: /File[/var/lib/puppet/reports]/seluser: Found seluser default 'system_u' for /var/lib/puppet/reports debug: /File[/var/lib/puppet/reports]/selrole: Found selrole default 'object_r' for /var/lib/puppet/reports debug: /File[/var/lib/puppet/reports]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/reports debug: /File[/var/lib/puppet/reports]/selrange: Found selrange default 's0' for /var/lib/puppet/reports debug: /File[/var/lib/puppet/lib]/seluser: Found seluser default 'system_u' for /var/lib/puppet/lib debug: /File[/var/lib/puppet/lib]/selrole: Found selrole default 'object_r' for /var/lib/puppet/lib debug: /File[/var/lib/puppet/lib]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/lib debug: /File[/var/lib/puppet/lib]/selrange: Found selrange default 's0' for /var/lib/puppet/lib debug: /File[/var/lib/puppet/bucket]/seluser: Found seluser default 'system_u' for /var/lib/puppet/bucket debug: /File[/var/lib/puppet/bucket]/selrole: Found selrole default 'object_r' for /var/lib/puppet/bucket debug: /File[/var/lib/puppet/bucket]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/bucket debug: /File[/var/lib/puppet/bucket]/selrange: Found selrange default 's0' for /var/lib/puppet/bucket debug: /File[/etc/puppet/fileserver.conf]/seluser: Found seluser default 'system_u' for /etc/puppet/fileserver.conf debug: /File[/etc/puppet/fileserver.conf]/selrole: Found selrole default 'object_r' for /etc/puppet/fileserver.conf debug: /File[/etc/puppet/fileserver.conf]/seltype: Found seltype default 'etc_t' for /etc/puppet/fileserver.conf debug: /File[/etc/puppet/fileserver.conf]/selrange: Found selrange default 's0' for /etc/puppet/fileserver.conf debug: /File[/etc/puppet/puppet.conf]/seluser: Found seluser default 'system_u' for /etc/puppet/puppet.conf debug: /File[/etc/puppet/puppet.conf]/selrole: Found selrole default 'object_r' for /etc/puppet/puppet.conf debug: /File[/etc/puppet/puppet.conf]/seltype: Found seltype default 'etc_t' for /etc/puppet/puppet.conf debug: /File[/etc/puppet/puppet.conf]/selrange: Found selrange default 's0' for /etc/puppet/puppet.conf debug: /File[/var/lib/puppet/ssl/private]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/private debug: /File[/var/lib/puppet/ssl/private]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/private debug: /File[/var/lib/puppet/ssl/private]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/private debug: /File[/var/lib/puppet/ssl/private]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/private debug: /File[/etc/puppet/manifests]/seluser: Found seluser default 'system_u' for /etc/puppet/manifests debug: /File[/etc/puppet/manifests]/selrole: Found selrole default 'object_r' for /etc/puppet/manifests debug: /File[/etc/puppet/manifests]/seltype: Found seltype default 'etc_t' for /etc/puppet/manifests debug: /File[/etc/puppet/manifests]/selrange: Found selrange default 's0' for /etc/puppet/manifests debug: /Filecommit:/var/lib/puppet/ssl/certificate_requests/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/certificate_requests debug: /Filecommit:/var/lib/puppet/ssl/certificate_requests/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/certificate_requests debug: /Filecommit:/var/lib/puppet/ssl/certificate_requests/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/certificate_requests debug: /Filecommit:/var/lib/puppet/ssl/certificate_requests/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/certificate_requests debug: /File[/var/lib/puppet/ssl/private_keys/publictest1.fedoraproject.org.pem]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/private_keys/publictest1.fedoraproject.org.pem debug: /File[/var/lib/puppet/ssl/private_keys/publictest1.fedoraproject.org.pem]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/private_keys/publictest1.fedoraproject.org.pem debug: /File[/var/lib/puppet/ssl/private_keys/publictest1.fedoraproject.org.pem]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/private_keys/publictest1.fedoraproject.org.pem debug: /File[/var/lib/puppet/ssl/private_keys/publictest1.fedoraproject.org.pem]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/private_keys/publictest1.fedoraproject.org.pem debug: /File[/var/lib/puppet/yaml]/seluser: Found seluser default 'system_u' for /var/lib/puppet/yaml debug: /File[/var/lib/puppet/yaml]/selrole: Found selrole default 'object_r' for /var/lib/puppet/yaml debug: /File[/var/lib/puppet/yaml]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/yaml debug: /File[/var/lib/puppet/yaml]/selrange: Found selrange default 's0' for /var/lib/puppet/yaml debug: /File[/etc/puppet/manifests/site.pp]/seluser: Found seluser default 'system_u' for /etc/puppet/manifests/site.pp debug: /File[/etc/puppet/manifests/site.pp]/selrole: Found selrole default 'object_r' for /etc/puppet/manifests/site.pp debug: /File[/etc/puppet/manifests/site.pp]/seltype: Found seltype default 'etc_t' for /etc/puppet/manifests/site.pp debug: /File[/etc/puppet/manifests/site.pp]/selrange: Found selrange default 's0' for /etc/puppet/manifests/site.pp debug: /File[/var/lib/puppet]/seluser: Found seluser default 'system_u' for /var/lib/puppet debug: /File[/var/lib/puppet]/selrole: Found selrole default 'object_r' for /var/lib/puppet debug: /File[/var/lib/puppet]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet debug: /File[/var/lib/puppet]/selrange: Found selrange default 's0' for /var/lib/puppet debug: /File[/var/log/puppet/masterhttp.log]/seluser: Found seluser default 'system_u' for /var/log/puppet/masterhttp.log debug: /File[/var/log/puppet/masterhttp.log]/selrole: Found selrole default 'object_r' for /var/log/puppet/masterhttp.log debug: /File[/var/log/puppet/masterhttp.log]/seltype: Found seltype default 'var_log_t' for /var/log/puppet/masterhttp.log debug: /File[/var/log/puppet/masterhttp.log]/selrange: Found selrange default 's0' for /var/log/puppet/masterhttp.log debug: /File[/var/lib/puppet/ssl/private_keys]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/private_keys debug: /File[/var/lib/puppet/ssl/private_keys]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/private_keys debug: /File[/var/lib/puppet/ssl/private_keys]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/private_keys debug: /File[/var/lib/puppet/ssl/private_keys]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/private_keys debug: /File[/var/lib/puppet/ssl]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl debug: /File[/var/lib/puppet/ssl]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl debug: /File[/var/lib/puppet/ssl]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl debug: /File[/var/lib/puppet/ssl]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl debug: /File[/var/lib/puppet/state]/seluser: Found seluser default 'system_u' for /var/lib/puppet/state debug: /File[/var/lib/puppet/state]/selrole: Found selrole default 'object_r' for /var/lib/puppet/state debug: /File[/var/lib/puppet/state]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/state debug: /File[/var/lib/puppet/state]/selrange: Found selrange default 's0' for /var/lib/puppet/state debug: /File[/var/lib/puppet/ssl/certs/publictest1.fedoraproject.org.pem]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/certs/publictest1.fedoraproject.org.pem debug: /File[/var/lib/puppet/ssl/certs/publictest1.fedoraproject.org.pem]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/certs/publictest1.fedoraproject.org.pem debug: /File[/var/lib/puppet/ssl/certs/publictest1.fedoraproject.org.pem]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/certs/publictest1.fedoraproject.org.pem debug: /File[/var/lib/puppet/ssl/certs/publictest1.fedoraproject.org.pem]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/certs/publictest1.fedoraproject.org.pem debug: Failed to load library 'ldap' for feature 'ldap' debug: /File[/var/lib/puppet/yaml]: Autorequiring File[/var/lib/puppet] debug: /File[/etc/puppet/puppet.conf]: Autorequiring File[/etc/puppet] debug: /File[/etc/puppet/manifests]: Autorequiring File[/etc/puppet] debug: /File[/var/lib/puppet/ssl/private_keys/publictest1.fedoraproject.org.pem]: Autorequiring File[/var/lib/puppet/ssl/private_keys] debug: /File[/etc/puppet/manifests/site.pp]: Autorequiring File[/etc/puppet/manifests] debug: /File[/var/lib/puppet/bucket]: Autorequiring File[/var/lib/puppet] debug: /File[/var/log/puppet/masterhttp.log]: Autorequiring File[/var/log/puppet] debug: /File[/var/lib/puppet/ssl/public_keys]: Autorequiring File[/var/lib/puppet/ssl] debug: /File[/var/lib/puppet/reports]: Autorequiring File[/var/lib/puppet] debug: /File[/var/lib/puppet/lib]: Autorequiring File[/var/lib/puppet] debug: /File[/var/lib/puppet/ssl/private]: Autorequiring File[/var/lib/puppet/ssl] debug: /File[/var/lib/puppet/facts]: Autorequiring File[/var/lib/puppet] debug: /Filecommit:/var/lib/puppet/ssl/certificate_requests: Autorequiring File[/var/lib/puppet/ssl] debug: /File[/var/lib/puppet/ssl/certs/ca.pem]: Autorequiring File[/var/lib/puppet/ssl/certs] debug: /File[/var/lib/puppet/ssl/certs/publictest1.fedoraproject.org.pem]: Autorequiring File[/var/lib/puppet/ssl/certs] debug: /File[/var/lib/puppet/rrd]: Autorequiring File[/var/lib/puppet] debug: /File[/var/lib/puppet/ssl/private_keys]: Autorequiring File[/var/lib/puppet/ssl] debug: /File[/var/lib/puppet/state]: Autorequiring File[/var/lib/puppet] debug: /File[/var/lib/puppet/ssl/crl.pem]: Autorequiring File[/var/lib/puppet/ssl] debug: /File[/var/lib/puppet/ssl/certs]: Autorequiring File[/var/lib/puppet/ssl] debug: /File[/etc/puppet/fileserver.conf]: Autorequiring File[/etc/puppet] debug: /File[/var/lib/puppet/ssl]: Autorequiring File[/var/lib/puppet] debug: Finishing transaction 70360911177640 with 0 changes debug: /File[/var/lib/puppet/ssl/ca/signed]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca/signed debug: /File[/var/lib/puppet/ssl/ca/signed]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca/signed debug: /File[/var/lib/puppet/ssl/ca/signed]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca/signed debug: /File[/var/lib/puppet/ssl/ca/signed]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca/signed debug: /File[/var/lib/puppet/ssl/ca/private]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca/private debug: /File[/var/lib/puppet/ssl/ca/private]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca/private debug: /File[/var/lib/puppet/ssl/ca/private]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca/private debug: /File[/var/lib/puppet/ssl/ca/private]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca/private debug: /File[/var/lib/puppet/ssl/ca/ca_crt.pem]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca/ca_crt.pem debug: /File[/var/lib/puppet/ssl/ca/ca_crt.pem]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca/ca_crt.pem debug: /File[/var/lib/puppet/ssl/ca/ca_crt.pem]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca/ca_crt.pem debug: /File[/var/lib/puppet/ssl/ca/ca_crt.pem]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca/ca_crt.pem debug: /File[/var/lib/puppet/ssl/ca/ca_key.pem]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca/ca_key.pem debug: /File[/var/lib/puppet/ssl/ca/ca_key.pem]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca/ca_key.pem debug: /File[/var/lib/puppet/ssl/ca/ca_key.pem]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca/ca_key.pem debug: /File[/var/lib/puppet/ssl/ca/ca_key.pem]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca/ca_key.pem debug: /File[/var/lib/puppet/ssl/ca/ca_crl.pem]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca/ca_crl.pem debug: /File[/var/lib/puppet/ssl/ca/ca_crl.pem]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca/ca_crl.pem debug: /File[/var/lib/puppet/ssl/ca/ca_crl.pem]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca/ca_crl.pem debug: /File[/var/lib/puppet/ssl/ca/ca_crl.pem]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca/ca_crl.pem debug: /File[/var/lib/puppet/ssl/ca/inventory.txt]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca/inventory.txt debug: /File[/var/lib/puppet/ssl/ca/inventory.txt]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca/inventory.txt debug: /File[/var/lib/puppet/ssl/ca/inventory.txt]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca/inventory.txt debug: /File[/var/lib/puppet/ssl/ca/inventory.txt]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca/inventory.txt debug: /File[/var/lib/puppet/ssl/ca]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca debug: /File[/var/lib/puppet/ssl/ca]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca debug: /File[/var/lib/puppet/ssl/ca]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca debug: /File[/var/lib/puppet/ssl/ca]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca debug: /File[/var/lib/puppet/ssl/ca/requests]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca/requests debug: /File[/var/lib/puppet/ssl/ca/requests]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca/requests debug: /File[/var/lib/puppet/ssl/ca/requests]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca/requests debug: /File[/var/lib/puppet/ssl/ca/requests]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca/requests debug: /File[/var/lib/puppet/ssl/ca/private/ca.pass]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca/private/ca.pass debug: /File[/var/lib/puppet/ssl/ca/private/ca.pass]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca/private/ca.pass debug: /File[/var/lib/puppet/ssl/ca/private/ca.pass]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca/private/ca.pass debug: /File[/var/lib/puppet/ssl/ca/private/ca.pass]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca/private/ca.pass debug: /File[/var/lib/puppet/ssl/ca/serial]/seluser: Found seluser default 'system_u' for /var/lib/puppet/ssl/ca/serial debug: /File[/var/lib/puppet/ssl/ca/serial]/selrole: Found selrole default 'object_r' for /var/lib/puppet/ssl/ca/serial debug: /File[/var/lib/puppet/ssl/ca/serial]/seltype: Found seltype default 'var_lib_t' for /var/lib/puppet/ssl/ca/serial debug: /File[/var/lib/puppet/ssl/ca/serial]/selrange: Found selrange default 's0' for /var/lib/puppet/ssl/ca/serial debug: /File[/var/lib/puppet/ssl/ca/serial]: Autorequiring File[/var/lib/puppet/ssl/ca] debug: /File[/var/lib/puppet/ssl/ca/private]: Autorequiring File[/var/lib/puppet/ssl/ca] debug: /File[/var/lib/puppet/ssl/ca/signed]: Autorequiring File[/var/lib/puppet/ssl/ca] debug: /File[/var/lib/puppet/ssl/ca/ca_crl.pem]: Autorequiring File[/var/lib/puppet/ssl/ca] debug: /File[/var/lib/puppet/ssl/ca/private/ca.pass]: Autorequiring File[/var/lib/puppet/ssl/ca/private] debug: /File[/var/lib/puppet/ssl/ca/requests]: Autorequiring File[/var/lib/puppet/ssl/ca] debug: /File[/var/lib/puppet/ssl/ca/ca_crt.pem]: Autorequiring File[/var/lib/puppet/ssl/ca] debug: /File[/var/lib/puppet/ssl/ca/inventory.txt]: Autorequiring File[/var/lib/puppet/ssl/ca] debug: /File[/var/lib/puppet/ssl/ca/ca_key.pem]: Autorequiring File[/var/lib/puppet/ssl/ca] debug: Finishing transaction 70360910955680 with 0 changes debug: Using cached certificate for ca debug: Using cached certificate for ca debug: Using cached certificate for publictest1.fedoraproject.org notice: Starting Puppet server version 0.25.0 debug: Mongrel client debugging enabled. [$mongrel_debug_client = true]. debug: Finishing transaction 70360896283500 with 0 changes debug: No modules mount given; autocreating with default permissions debug: No plugins mount given; autocreating with default permissions debug: Creating interpreter debug: Finishing transaction 70360896268760 with 0 changes warning: Inserting default '~ ^/catalog/([^/]+)$'(auth) acl because none were found in 'no auth.conf file configured' warning: Inserting default '/file'(non-auth) acl because none were found in 'no auth.conf file configured' warning: Inserting default '/certificate_revocation_list/ca'(auth) acl because none were found in 'no auth.conf file configured' warning: Inserting default '/report'(auth) acl because none were found in 'no auth.conf file configured' warning: Inserting default '/certificate/ca'(non-auth) acl because none were found in 'no auth.conf file configured' warning: Inserting default '/certificate/'(non-auth) acl because none were found in 'no auth.conf file configured' warning: Inserting default '/certificate_request'(non-auth) acl because none were found in 'no auth.conf file configured'
The Apache error logs contained a bunch of lines like:
[Wed Jul 01 04:06:56 2009] [error] (111)Connection refused: proxy: HTTP: attempt to connect to 127.0.0.1:18140 (127.0.0.1) failed [Wed Jul 01 04:06:56 2009] [error] ap_proxy_connect_backend disabling worker for (127.0.0.1)
Updated by Luke Kanies almost 3 years ago
Does knowing that SELinux causes it leave us any closer? Is there an SELinux change that could fix this problem?
Updated by Ricky Zhou almost 3 years ago
I’m not sure, unfortunately. The most confusing part is that it happens even with SELinux set to permissive, which means that it shouldn’t block anything at all. If it’d be useful, I can try doing a git bisect to see if there was a certain commit that introduced this problem.
Updated by Luke Kanies almost 3 years ago
Ricky Zhou wrote:
I’m not sure, unfortunately. The most confusing part is that it happens even with SELinux set to permissive, which means that it shouldn’t block anything at all. If it’d be useful, I can try doing a git bisect to see if there was a certain commit that introduced this problem.
Definitely.
Updated by Luke Kanies almost 3 years ago
- Priority changed from High to Normal
At this point, by the way, I’m assuming it’s an SELinux problem so I’m comfortable bumping this if we can’t get a fix. Thus, I’m reducing it to normal priority.
Updated by Ricky Zhou almost 3 years ago
Hi, I tried a git bisect last night, but things got pretty hairy, especially as I ran into a bunch of broken commits that I couldn’t test. I can see why even having SELinux enabled but permissive triggers the problem, though. When I edited lib/puppet/util/selinux.rb and made selinux_support? always return false, puppet ran fine. I’m going to try adding a bunch of print statements to that file and see at what point this is hanging.
Updated by Ricky Zhou almost 3 years ago
OK, a little bit of lucky guessing with where to put the print statements shows that it is hanging on a read on /proc/mounts in the read_mounts function in lib/puppet/util/selinux.rb. This is starting to remind me a lot of bug #1963. Maybe these two could be related?
Updated by James Turnbull almost 3 years ago
- Status changed from Re-opened to Accepted
- Priority changed from Normal to High
Updated by Ricky Zhou almost 3 years ago
OK, the storeconfigs thing was a total red herring. It seems that having storeconfigs enabled just happened to trigger the read_mounts call somehow. I just what I think may be the root cause to this in #1963.
Updated by James Turnbull almost 3 years ago
- Status changed from Accepted to Duplicate
To be tracked in #1963.