Puppet

Unfortunately these error messages come from the openssl ruby layer itself rather than puppet and it’s extremely difficult for puppet to determine why a certificate verification check failed in the openssl layer.

The same error cuts both ways apparently. We had issued a small number (~10) of certificates with overlapping serial numbers and revoked some of them as hosts were reimaged.

One of the revoked serial numbers belonged to a host and the certificate for the ‘puppet.mydomain.com’ VIP. So when clients downloaded the CRL, they stopped trusting the VIP.

This was only discoverable through packet analysis: (10.1.1.1 is the client, 10.2.2.2 is the VIP fronting the puppetmaster)

[root@host ~]# tshark -s1500 -i eth0 -n -d tcp.port==8140,ssl port 8140
  0.000000 10.1.1.1 -> 10.2.2.2 TCP 51532 > 8140 [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=3705949461 TSER=0 WS=7
  0.000527 10.2.2.2 -> 10.1.1.1 TCP 8140 > 51532 [SYN, ACK] Seq=0 Ack=1 Win=8190 Len=0 MSS=1460
  0.000543 10.1.1.1 -> 10.2.2.2 TCP 51532 > 8140 [ACK] Seq=1 Ack=1 Win=5840 Len=0
  0.000981 10.1.1.1 -> 10.2.2.2 SSLv2 Client Hello
  0.001640 10.2.2.2 -> 10.1.1.1 TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done
  0.001670 10.1.1.1 -> 10.2.2.2 TCP 51532 > 8140 [ACK] Seq=106 Ack=1358 Win=8142 Len=0
  0.002009 10.1.1.1 -> 10.2.2.2 TLSv1 Alert (Level: Fatal, Description: Certificate Revoked)

Jeff McCune wrote:

Unfortunately these error messages come from the openssl ruby layer itself rather than puppet and it’s extremely difficult for puppet to determine why a certificate verification check failed in the openssl layer.

I don’t think it’s right. You can define a verify_callback on any ssl context (and this can be done with Net::HTTPS too). This proc will be called after validation is done and you can ask the context for more information about the error.

The question is: isn’t it an information disclosure to let the client know his cert has been revoked?

Brice Figureau wrote:

The question is: isn’t it an information disclosure to let the client know his cert has been revoked?

I don’t think so, because “Cert Revoked” is an explicit message type during TLS negotiation, so a client would see the revocation error just by running verbose openssl negotiation.

http://en.wikipedia.org/wiki/Transport_Layer_Security#Alert_protocol

eric sorenson wrote:

Brice Figureau wrote:

The question is: isn’t it an information disclosure to let the client know his cert has been revoked?

I don’t think so, because “Cert Revoked” is an explicit message type during TLS negotiation, so a client would see the revocation error just by running verbose openssl negotiation.

http://en.wikipedia.org/wiki/Transport_Layer_Security#Alert_protocol

Then +1 for this bug.

Brice Figureau wrote:

eric sorenson wrote:

Brice Figureau wrote:

The question is: isn’t it an information disclosure to let the client know his cert has been revoked?

I don’t think so, because “Cert Revoked” is an explicit message type during TLS negotiation, so a client would see the revocation error just by running verbose openssl negotiation.

http://en.wikipedia.org/wiki/Transport_Layer_Security#Alert_protocol

Then +1 for this bug.

I’m afraid it won’t be fixed because of #4273 Maybe if we only display the extended error message it will work? (and log it at a level > debug to be useful).

One more problem that can cause a very similar looking error and cause the client to completely fail, with the following message during the pluginsync phase:

err: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify fai
led
err: /File[/var/lib/puppet/lib]: Could not evaluate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed Could not retrieve file metadata for pu
ppet://puppet/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

This happened because the CRL downloaded from the server has an issuer field that does not match the Subject field of CA.pem. You can tell with the following command:

# good
openssl crl -noout -CAfile certs/ca.pem -in crl.pem 
verify OK
# bad
openssl crl -noout -CAfile certs/ca.pem -in crl-fatal.pem 
Error getting CRL issuer certificate

In my case this happened because I had converted this puppetmaster from a standalone CA to using my site-wide CA certificate, but ‘puppet cert’ reused the Issuer field it found in ‘$ssldir/crl.pem’ (which had been created along with the original CA certificate) instead of the cleaned/cloned ‘$ssldir/ca/ca_crl.pem’ i.e. it used :hostcrl instead of :cacrl.

http://www.braintreepayments.com/devblog/sslsocket-verify_mode-doesnt-verify demonstrates the mechanism for extracting the error message and turning it into a more useful exception. We should do that.

FWIW this happened to another team here last week. It took out their SSL setup for a couple of days until they grew frustrated with debugging and came ‘round to ask me if I’d seen it before.