Using Mongrel with nginx
Alternatively, see: Using Mongrel and Using Mongrel Pound
This solution freshly baked by Jeff McCune.
Date: 2007-11-27
References
Why choose nginx
- Performance: nginx is very small and fast. Many claim it’s faster than Pound.
- Logging, Debugging: nginx appears to be “cleaner” than Pound.
- Flexibility: nginx handles SSL client verification at the application layer, rather than terminating the SSL connection.
- You don’t have to patch pound if you don’t want to upgrade Mongrel.
- You don’t have to upgrade Mongrel if you don’t want to patch Pound.
- nginx works “out of the box”
- Despite russian documentation, the configuration syntax is intuitive.
Why NOT choose nginx
- I see no way to configure a certificate revocation list. If you need revocation, consider Pound or Apache for your SSL proxy.
Overview
This sample configuration is based off nginx-0.5.33.
The back end puppet servers are configuration-compatible with the Using Mongrel Pound documentation. That is, you must configure ssl_client_header = HTTP_X_SSL_SUBJECT and take note of ticket #906.
Sample Configuration
This configuration appears to be working well for about 300 clients in production:
user daemon daemon;
worker_processes 4;
error_log /var/log/nginx-puppet.log notice;
pid /var/run/nginx-puppet.pid;
events {
worker_connections 1024;
}
http {
# include /etc/mime.types;
default_type application/octet-stream;
# no sendfile on OSX uncomment
#this if your on linux or bsd
sendfile on;
tcp_nopush on;
# Look at TLB size in /proc/cpuinfo (Linux) for the 4k pagesize
large_client_header_buffers 16 4k;
proxy_buffers 128 4k;
# if you adjust this setting to something higher
# you should as well update the proxy_read_timeout
# in the server config part (see below)
# Otherwise nginx will rerequest a manifest compile.
keepalive_timeout 65;
tcp_nodelay on;
ssl on;
ssl_certificate /Library/Puppet/Generated/Server/SSL/host_cert.pem;
ssl_certificate_key /Library/Puppet/Generated/Server/SSL/host_key.pem;
ssl_client_certificate /Library/Puppet/Generated/Server/SSL/ca/ca_crt.pem;
ssl_ciphers SSLv2:-LOW:-EXPORT:RC4+RSA;
ssl_session_cache shared:SSL:8m;
ssl_session_timeout 5m;
upstream puppet-production {
server 127.0.0.1:18140;
server 127.0.0.1:18141;
server 127.0.0.1:18142;
server 127.0.0.1:18143;
}
upstream puppet-test1 {
server 127.0.0.1:28140;
}
upstream puppet-test2 {
server 127.0.0.1:38140;
}
upstream puppet-test3 {
server 127.0.0.1:48140;
}
server {
listen 8140;
ssl_verify_client on;
root /var/empty;
access_log /var/log/nginx/access-8140.log;
rewrite_log /var/log/nginx/rewrite-8140.log;
# Variables
# $ssl_cipher returns the line of those utilized it is cipher for established SSL-connection
# $ssl_client_serial returns the series number of client certificate for established SSL-connection
# $ssl_client_s_dn returns line subject DN of client certificate for established SSL-connection
# $ssl_client_i_dn returns line issuer DN of client certificate for established SSL-connection
# $ssl_protocol returns the protocol of established SSL-connection
location / {
proxy_pass http://puppet-production;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Client-Verify SUCCESS;
proxy_set_header X-Client-DN $ssl_client_s_dn;
proxy_set_header X-SSL-Subject $ssl_client_s_dn;
proxy_set_header X-SSL-Issuer $ssl_client_i_dn;
proxy_read_timeout 65;
}
}
server {
listen 8141;
ssl_verify_client off;
root /var/empty;
access_log /var/log/nginx/access-8141.log;
rewrite_log /var/log/nginx/rewrite-8141.log;
location / {
proxy_pass http://puppet-production;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Client-Verify FAILURE;
proxy_set_header X-Client-DN $ssl_client_s_dn;
proxy_set_header X-SSL-Subject $ssl_client_s_dn;
proxy_set_header X-SSL-Issuer $ssl_client_i_dn;
proxy_read_timeout 65;
}
}
server {
listen 8150;
ssl_verify_client on;
root /var/empty;
access_log /var/log/nginx/access-8150.log;
rewrite_log /var/log/nginx/rewrite-8150.log;
location / {
proxy_pass http://puppet-test1;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Client-Verify SUCCESS;
proxy_set_header X-Client-DN $ssl_client_s_dn;
proxy_set_header X-SSL-Subject $ssl_client_s_dn;
proxy_set_header X-SSL-Issuer $ssl_client_i_dn;
proxy_read_timeout 65;
}
}
server {
listen 8160;
ssl_verify_client on;
root /var/empty;
access_log /var/log/nginx/access-8160.log;
rewrite_log /var/log/nginx/rewrite-8160.log;
location / {
proxy_pass http://puppet-test2;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Client-Verify SUCCESS;
proxy_set_header X-Client-DN $ssl_client_s_dn;
proxy_set_header X-SSL-Subject $ssl_client_s_dn;
proxy_set_header X-SSL-Issuer $ssl_client_i_dn;
proxy_read_timeout 65;
}
}
server {
listen 8170;
ssl_verify_client on;
root /var/empty;
access_log /var/log/nginx/access-8170.log;
rewrite_log /var/log/nginx/rewrite-8170.log;
location / {
proxy_pass http://puppet-test3;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Client-Verify SUCCESS;
proxy_set_header X-Client-DN $ssl_client_s_dn;
proxy_set_header X-SSL-Subject $ssl_client_s_dn;
proxy_set_header X-SSL-Issuer $ssl_client_i_dn;
proxy_read_timeout 65;
}
}
}
Certificate Signing
- You will need to add —ca_port 8141 to your first puppetd run in order to have the puppetmaster sign your certificate.
Nginx > 0.8.7
Since Nginx 0.8.7 or any 0.7 version with the patches from: http://www.masterzen.fr/2009/07/21/new-ssl-features-for-nginx/ (Note: not all 0.7 version need these patches, from a certain version on they are integrated upstream) Nginx supports out of the box
* Optional client verification, like apache * CRL certificate verification
Thus for those versions, the configuration becomes much simpler, and you don’t need to instruct your clients to use another port (—ca_port is not needed anymore):
...
upstream puppet-production {
server 127.0.0.1:18140;
server 127.0.0.1:18141;
}
server {
listen 8140;
ssl on;
ssl_session_timeout 5m;
ssl_certificate /var/lib/puppet/ssl/certs/puppetmaster.pem;
ssl_certificate_key /var/lib/puppet/ssl/private_keys/puppetmaster.pem;
ssl_client_certificate /var/lib/puppet/ssl/ca/ca_crt.pem;
# choose any ciphers
ssl_ciphers SSLv2:-LOW:-EXPORT:RC4+RSA;
# allow authenticated and client without certs
ssl_verify_client optional;
# obey to the Puppet CRL
ssl_crl /var/lib/puppet/ssl/ca/ca_crl.pem;
root /var/tmp;
location / {
proxy_pass http://puppet-production;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Client-Verify $ssl_client_verify;
proxy_set_header X-Client-DN $ssl_client_s_dn;
proxy_set_header X-SSL-Subject $ssl_client_s_dn;
proxy_set_header X-SSL-Issuer $ssl_client_i_dn;
proxy_read_timeout 65;
}
}
...
Fair Proxy Balancer
There is a module for nginx available that improves how requests are balanced to the backend mongrel servers: http://www.brainspl.at/articles/2007/11/09/a-fair-proxy-balancer-for-nginx-and-mongrel. Integrating and activating this module for your nginx can improve performance of puppet runs further. The reason is that some puppet requests are very long living such as the compile requests. As the mongrel instance is busy during that time compiling your catalog it won’t serve any other requests and they are stuck in nginx’s queue until this request is finished. If you have 4 mongrel instances running and one is currently busy compiling a huge catalog every 4th request from another client that is for example asking for file metadata will be stuck at the mongrel instance compiling the catalog. This will slowdown the puppet runs on the second client.
The fair proxy balancer will keep track which of your mongrel instances are busy and will direct new requests to other mongrel instances, which are probably free.
All you need to do is to compile nginx with this module and setup your proxy balancer like that:
...
upstream puppet-production {
fair;
server 127.0.0.1:18140;
server 127.0.0.1:18141;
}
...