Puppet

Can anyone else replicate this? Is anyone using a CRL with Apache?

Replying to [comment:1 luke]:

Can anyone else replicate this? Is anyone using a CRL with Apache? Yes, I have exactly the same problem. Setting Apache2 to ‘LogLevel DEBUG’ reveals the following in /var/log/apache2/error.log

[Thu Nov 29 14:28:40 2007] [warn] Invalid signature on CRL [Thu Nov 29 14:28:40 2007] [error] Certificate Verification: Error (8): CRL signature failure [Thu Nov 29 14:28:40 2007] [debug] ssl_engine_kernel.c(1770): OpenSSL: Write: SSLv3 read client certificate B [Thu Nov 29 14:28:40 2007] [debug] ssl_engine_kernel.c(1789): OpenSSL: Exit: error in SSLv3 read client certificate B [Thu Nov 29 14:28:40 2007] [debug] ssl_engine_kernel.c(1789): OpenSSL: Exit: error in SSLv3 read client certificate B [Thu Nov 29 14:28:40 2007] [info] [client <IP-address>] SSL library error 1 in handshake (server puppetmaster:8140) [Thu Nov 29 14:28:40 2007] [info] SSL Library Error: 67567722 error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01 [Thu Nov 29 14:28:40 2007] [info] SSL Library Error: 67530866 error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed [Thu Nov 29 14:28:40 2007] [info] SSL Library Error: 218910726 error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib [Thu Nov 29 14:28:40 2007] [info] SSL Library Error: 336105650 error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned [Thu Nov 29 14:28:40 2007] [info] [client <IP-address>] Connection closed to child 67 with abortive shutdown (server puppetmaster:8140)

The client which tries to connect does not have a vertificate yet. It should get one from the server. No signing request is generated, and also no signed certificate.

So I just verified this myself as we were trying to deal with the new ruby ssl library issue on Ubuntu clients.

This was verified on both puppetd 0.23.2 and 0.24.1, both running against our test server, which is using the same Apache/Mongrel setup as above.

The problem only occurs with the newer ruby libraries and when the client does not have a certificate, mine are as shown:

# dpkg -l |grep ruby
ii  libopenssl-ruby                           1.0.0+ruby1.8.2-1
ii  libopenssl-ruby1.8                        1.8.6.111-2ubuntu1
ii  libruby1.8                                1.8.6.111-2ubuntu1
ii  libshadow-ruby1.8                         1.4.1-8
ii  libxmlrpc-ruby                            1.8.2-1
ii  ruby                                      1.8.2-1 
ii  ruby1.8                                   1.8.6.111-2ubuntu1

and manifests itself on the client side as follows when running puppetd:

warning: peer certificate won't be verified in this SSL session
/usr/lib/ruby/1.8/net/http.rb:586:in @connect': sslv3 alert handshake failure (OpenSSL::SSL::SSLError)
    from /usr/lib/ruby/1.8/net/http.rb:586:in @connect'
    from /usr/lib/ruby/1.8/net/http.rb:553:in @do_start'
    from /usr/lib/ruby/1.8/net/http.rb:548:in @start'
    from /usr/lib/ruby/1.8/puppet/network/xmlrpc/client.rb:128:in @start'
    from /usr/lib/ruby/1.8/puppet/network/client.rb:99:in @initialize'
    from /usr/lib/ruby/1.8/puppet/network/client/master.rb:203:in @initialize'
    from /usr/sbin/puppetd:312:in @new'
    from /usr/sbin/puppetd:312

The relevant Apache errors in the logs are:

[debug] ssl_engine_kernel.c(1770): [[OpenSSL]]: Write: SSLv3 read client certificate B
[debug] ssl_engine_kernel.c(1789): [[OpenSSL]]: Exit: error in SSLv3 read client certificate B
[debug] ssl_engine_kernel.c(1789): [[OpenSSL]]: Exit: error in SSLv3 read client certificate B
[info] [client 172.27.71.100] SSL library error 1 in handshake (server XXX.XXX.XXX.XXX:8140)
[info] SSL Library Error: 336105671 error:140890C7:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:peer did not return a certificate No CAs known to server for verification?
[info] [client 172.27.71.100] Connection closed to child 65 with abortive shutdown (server XXX.XXX.XXX.XXX:8140

So it appears that something has changed in that Ruby SSL library version that changes behavior when talking to Apache with a CRL when you don’t have a certificate.

Just to make it obvious, this problem doesn’t exist if you talk straight to the webrick server by forcing masterport to be the same as the ca_port, which is to be expected.

I haven’t downgraded a client and tested whether this still occurs with 0.22.4, as that doesn’t seem to be such a high priority right now.

I’m running into this erro as well, though under slightly different circumstances: the puppetd client in my case is running on the puppetmaster server. Puppetd clients on other servers can connect to apache fine, but every time I run @puppetd —test@ on the puppetmaster, I get the following error:

/usr/lib/ruby/1.8/net/http.rb:586:in @connect': tlsv1 alert decrypt error (OpenSSL::SSL::SSLError)
    from /usr/lib/ruby/1.8/net/http.rb:586:in @connect'
    from /usr/lib/ruby/1.8/net/http.rb:553:in @do_start'
    from /usr/lib/ruby/1.8/net/http.rb:548:in @start'
    from /usr/lib/ruby/1.8/puppet/network/xmlrpc/client.rb:128:in @start'
    from /usr/lib/ruby/1.8/puppet/network/client.rb:99:in @initialize'
    from /usr/lib/ruby/1.8/puppet/network/client/master.rb:203:in @initialize'
    from /usr/sbin/puppetd:312:in @new'
    from /usr/sbin/puppetd:312

The apache logs contain:

[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1143): [client 172.16.0.2] handing out temporary 1024 bit DH key
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1760): [[OpenSSL]]: Loop: SSLv3 write key exchange A
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1760): [[OpenSSL]]: Loop: SSLv3 write certificate request A
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1760): [[OpenSSL]]: Loop: SSLv3 flush data
[2008 Mar 16 10:03:29] [debug] ssl_engine_io.c(1775): [[OpenSSL]]: read 5/5 bytes from BIO#86b930 [mem: 87a0d0] (BIO dump follows)
[2008 Mar 16 10:03:29] [debug] ssl_engine_io.c(1722): +-------------------------------------------------------------------------+
<...snip...>
[2008 Mar 16 10:03:29] [debug] ssl_engine_io.c(1753): +-------------------------------------------------------------------------+
[2008 Mar 16 10:03:29] [debug] ssl_engine_io.c(1775): [[OpenSSL]]: read 652/652 bytes from BIO#86b930 [mem: 87a0d5] (BIO dump follows)
[2008 Mar 16 10:03:29] [debug] ssl_engine_io.c(1722): +-------------------------------------------------------------------------+
<...snip...>
[2008 Mar 16 10:03:29] [debug] ssl_engine_io.c(1753): +-------------------------------------------------------------------------+
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1190): Certificate Verification: depth: 1, subject: /CN=puppet.mycorp.com, issuer: /CN=puppet.mycorp.com
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1382): CA CRL: Issuer: CN=puppet.mycorp.com, lastUpdate: Feb 24 23:40:40 2008 GMT, nextUpdate: Feb 22 23:40:40 2013 GMT
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1190): Certificate Verification: depth: 0, subject: /CN=puppet.mycorp.com, issuer: /CN=puppet.mycorp.com
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1382): CA CRL: Issuer: CN=puppet.mycorp.com, lastUpdate: Feb 24 23:40:40 2008 GMT, nextUpdate: Feb 22 23:40:40 2013 GMT
[2008 Mar 16 10:03:29] [warn] Invalid signature on CRL
[2008 Mar 16 10:03:29] [error] Certificate Verification: Error (8): CRL signature failure
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1770): [[OpenSSL]]: Write: SSLv3 read client certificate B
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1789): [[OpenSSL]]: Exit: error in SSLv3 read client certificate B
[2008 Mar 16 10:03:29] [debug] ssl_engine_kernel.c(1789): [[OpenSSL]]: Exit: error in SSLv3 read client certificate B

This is all on ubuntu gutsy, running puppet 0.24.1 from the debian .deb; until now we’ve been using WEBrick with the same certificates, CA, and CRLs without any SSL problems. The apache config (take mostly from the UsingMongrel wiki page:

Listen 8140

[[ProxyRequests]] Off


    <% (minport .. maxport).each do |port| %>
    [[BalancerMember]] http://127.0.0.1:<%= port %>
    <% end %>



    SSLEngine on
    SSLProtocol -all +SSLv3 +TLSv1
    SSLCipherSuite HIGH:MEDIUM:!aNULL:+SHA1:+MD5:+HIGH:+MEDIUM
    SSLCertificateFile /var/lib/puppet/ssl/certs/<%= fqdn %>.pem
    SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/<%= fqdn %>.pem
    SSLCertificateChainFile /var/lib/puppet/ssl/ca/ca_crt.pem
    SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
    SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
    SSLVerifyClient optional
    SSLVerifyDepth  1
    SSLOptions +StdEnvVars

    [[RequestHeader]] set X-Client-DN %{SSL_CLIENT_S_DN}e
    [[RequestHeader]] set X-Client-Verify %{SSL_CLIENT_VERIFY}e

    
       [[SetHandler]] balancer-manager
       Order allow,deny
       Allow from all
    

    [[ProxyPass]] / balancer://puppetmaster/
    [[ProxyPassReverse]] / balancer://puppetmaster/
    [[ProxyPreserveHost]] on

Replying to [comment:4 emerose]:

I guess I should also mention that the problem goes away when the CRL line is commented out of the apache config…

Replying to [comment:5 emerose]:

Replying to [comment:4 emerose]:

I guess I should also mention that the problem goes away when the CRL line is commented out of the apache config…

I had the exact same problem and solution. I’m using the apache proxy instead of webrick as well. Fixed by commenting out the “SSLCARevocationFile” line and reloading apache.

Replying to [comment:7 greenmoss]:

I had the exact same problem and solution. I’m using the apache proxy instead of webrick as well. Fixed by commenting out the “SSLCARevocationFile” line and reloading apache.

Strangely enough, after puppetd —test ran fine one time, I re-enabled SSLCARevocationFile and restarted Apache/Puppetmaster. The problem has not yet returned. Is puzzled

Brendan Beveridge wrote:

Following the guide on: http://reductivelabs.com/trac/puppet/wiki/UsingMongrel

running: puppetd —test —server localhost gives me: err: Could not retrieve configuration: Certificates were not trusted: tlsv1 alert decrypt error

in the apache balancer logs i get: [warn] Invalid signature on CRL [error] Certificate Verification: Error (8): CRL signature failure

If i comment out the revocation file all works fine.

This afternoon, ohadlevy (sorry I don’t know his real name) and I traced down this issue and found a work-around.

This is what I think is a bug in apache mod_ssl (other balancer, like Nginx or Pound are not affected). Mod_ssl implements its own CRL checking instead of using the OpenSSL one. This specific algorithm first tries to load a CRL whose name is the client subjectDN. If it finds one, it checks its signature against the cert public key. Unfortunately, when we run puppetd on the same host as the puppetmaster, puppetd uses the same cert as the puppetmaster server certificate. This certificate subject matches the CRL issuer, so apache loads this crl and tries to validate it with this public key. Unfortunately again, this CRL has been signed by the CA, not the server certificate, so it fails.

The workaround is to generate a new certificate on the puppetmaster, which uses a different subjectDN than the server cert, then when use the —certname puppetd option when running puppetd with this new name.

The long term solution I think is to either sign the CRL with the server public key, or issue a CA cert with a different subjectDN than the server certname (if that’s possible).

It seems reasonable to sign the CRL with the server cert instead of the CA cert, but it does mean that each server will need to make its own CRL, which probably changes how things work just a bit.

Provided a workaround for this in the code for #2617 – you can specify a different name for the CA cert, which should allow the CRL to work.