Taco de Wolff

Data Scientist

The perfect webserver

Published on March 10, 2016

Outdated: see the new article here

To run a secure and performant webserver, you need to think through the layout and configuration of your services. This is what I learned when I replaced my first VPS with a better one.

My VPS runs on CentOS 7 with 2GB RAM and a dual core.

Software

HAProxy - a reverse proxy load balancer that redirects requests to a backend. We will use this to listen to ports 80 and 443. We will also use it for SSL termination, so that our backend can be solely HTTP. HAProxy also makes it easy to switch to a different backend without downtime. Simply setup the new backend, test it through and gradually increase the load. It’s an extra layer of indirection that adds a lot of flexibility.

Varnish - a reverse proxy cache that will store common requests that are cachable. This reduces load to the backend and speeds up your server. To ensure that Varnish works well, the backend must provide HTTP cache headers.

Apache - the backend that will process requests and return responses. The backend will redirect requests for each domain name to their directories.

PHP-FPM - execute PHP scripts in worker threads. This is a performant way to process PHP scripts, as the worker threads keep running to process new requests. This is unlike the historically more common mod_php, which spawns a worker for every request. PHP code should be cached in socache to reduce the overhead of parsing the script. For sessions we will use an in-memory cache.

Let’s Encrypt and lego - for our SSL certificates we will use the awesome Let’s Encrypt service that provides free SSL certificates. The ACME client we use to obtain the certificates is github.com/xenolf/lego, which is a Go library with a CLI with which we can automate the creation and renewal of certificates.

Security

The following software will not be configured in this article, but should nonetheless exist on your server.

selinux - paranoid security of your server. Run it in permissive mode until it fully works with all services. It can take some time to get it running, but it is definitively worth it.

fail2ban - setup fail2ban jails for all logins that happen on your server. SSH, FTP, HTTP Auth, email and more. This prevents any bruteforce password guessing.

Server setup

Webserver layout — Fig 1: webserver layout of services and ports. HAProxy performs SSL termination and passthrough for ACME certificate validation. Requests are send to Varnish which updates its cache from Apache, the actual webserver.

See Fig. 1 for an overview of the webserver we will set up. HAProxy is the only service that is listening for website requests, it passes them on to Varnish or other services (ACME in our case). HAProxy as our main entrance allows us to change any of the underlying services to another server or implementation. Instead of HAProxy you could also use Nginx I guess, which can be a webserver backend as well as a reverse proxy.

Varnish will cache all static requests to Apache, to reduce Apache load. Apache then retrieve website content from the disk and executes PHP files through PHP-FPM.

HAProxy

/etc/haproxy/haproxy.conf

frontend www-http
	bind 185.96.4.242:80
	mode http

	# send ACME requests to the ACME backend
	acl url_acme_http01 path_beg -i /.well-known/acme-challenge/
	use_backend acme if url_acme_http01

	default_backend http

frontend www-https
	bind 185.96.4.242:443 ssl crt /etc/pki/haproxy/default.pem crt /etc/pki/haproxy/
	mode http

	# set the protocol in a header so that Apache knows the original request was HTTPS
	http-request set-header X-Forwarded-Proto https

	# send ACME requests to the ACME backend
	acl url_acme_http01 path_beg -i /.well-known/acme-challenge/
	use_backend acme if url_acme_http01

	default_backend http

backend http
	balance roundrobin

	# sticky SSL sessions, only useful when you run with multiple backend servers
	# it ensures that once an SSL connection has been made by the client, the client will make all succesive requests to the same server
	stick-table type binary len 32 size 30k expire 30m
	acl clienthello req_ssl_hello_type 1
	acl serverhello rep_ssl_hello_type 2
	tcp-request inspect-delay 5s
	tcp-request content accept if clienthello
	tcp-response content accept if serverhello
	stick on payload_lv(43,1) if clienthello
	stick store-response payload_lv(43,1) if serverhello

	# same thing for PHP sessions
	appsession PHPSESSID len 64 timeout 30m request-learn prefix

	server varnish 127.0.0.1:8081 check

backend acme
	mode http

	# run the ACME client with the http01 challenge on port 8082
	server acme 127.0.0.1:8082

Lego ACME

I run lego through the CLI. I use bash scripts to automate creation and renewal, but I suppose you can build your own Go program to handle it.

It is important that we don’t listen on the default 80 and 443 ports, as those are used by HAProxy. Instead, we let HAProxy redirect all ACME requests to port 8082, which is what lego will listen to.

Certificate creation

We will save our certificates to /etc/pki/lego and disable the tls-sni-01 and dns-01 methods, as we won’t use those.

lego -m [email protected] -k rsa4096 --http ":8082" -x "tls-sni-01" -x "dns-01" --path /etc/pki/lego -d example.com run

Replace example.com with your own domain. You can add alternative names to the certificate by appending more -d sub.example.com options to the CLI.

The certificates are put into /etc/pki/lego/certificates/ and consist of a key and certificate file. HAProxy requires .pem files, which is a .crt and .key concatenated. We will save these into /etc/pki/haproxy/.

cat /etc/pki/lego/certificates/example.com.crt /etc/pki/lego/certificates/example.com.key > /etc/pki/haproxy/example.com.pem

I would suggest that you write a bash script to automate the creation of these if you run multiple domains.

Certificate renewal

Varnish

Varnish setup can be pretty default. We make sure it listens on 8081 and makes backend requests from 8080.

/etc/varnish/varnish.params

VARNISH_LISTEN_ADDRESS=127.0.0.1
VARNISH_LISTEN_PORT=8081

/etc/varnish/default.vcl

backend default {
	.host = "127.0.0.1";
	.port = "8080";
}

It would be wise to unset certain HTTP headers, for example cookies. Disable statistics tracking cookies like those from Google Analytics and make sure to cache static files such as gif|jpg|png|css|js and more.

Apache

We will use /var/www/example.com/public_html/subdomain/ as the document root for our websites. We also put mail files, logs and temporary files in the /var/www/example.com/ directory:

├── /var/www/
│   ├── example.com/
│   │   ├── log/
│   │   │   ├── access_log
│   │   │   ├── error_log
│   │   │   └── php_errors.log
│   │   ├── mail/
│   │   │   ├── info/
│   │   │   ├── admin/
│   │   │   └── contact/
│   │   ├── public_html/
│   │   │   ├── _/
│   │   │   └── www/
│   │   └── tmp/
│   ├── example2.com/
│   └── example3.com/

The _ subdomain is when there is no subdomain given (ie. example.com) and www is for the www subdomain (ie. www.example.com). We will use virtual document roots so that adding a new directory to public_html/ will automatically enable a new subdomain.

/etc/httpd/conf/httpd.conf

Listen 127.0.0.1:8080

<Directory />
	AllowOverride None
	Require all denied
	Options None
</Directory>

# the default directory if the website doesn't exist, folder doesn't have to exist
DocumentRoot "/var/www/html"

/etc/httpd/conf.d/httpd.conf

# the default virtual host when no other virtual host is found
<VirtualHost *:8080>
	ServerName example.com # root domain of the server
	VirtualDocumentRoot "/var/www/html"

	<FilesMatch \.php$>
		SetHandler "proxy:unix:/var/run/php-fpm/php-fpm-www.sock|fcgi://localhost"
	</FilesMatch>
</VirtualHost>

# macro for easy creation of new virtual hosts
<Macro VHost $domain>
	<VirtualHost *:8080>
		ServerName $domain
		ServerAlias *.$domain
		VirtualDocumentRoot "/var/www/$domain/public_html/%-3+"
		ErrorLog /var/www/$domain/log/error_log
		CustomLog /var/www/$domain/log/access_log combined

		<FilesMatch \.php$>
			SetHandler "proxy:unix:/var/run/php-fpm/php-fpm-$domain.sock|fcgi://localhost"
		</FilesMatch>
	</VirtualHost>
</Macro>

IncludeOptional vhost.d/*.conf

# reduce Apache headers
ServerSignature Off
ServerTokens Prod
TraceEnable Off

# allow access in all public_html folders
<Directory "/var/www/*/public_html">
	Require all granted
	AllowOverride All
	Options FollowSymLinks SymLinksIfOwnerMatch
</Directory>

Header unset Pragma
Header unset Cache-Control

# set REMOTE_ADDR to the client's IP instead of the proxy's IP
<IfModule mod_remoteip.c>
	RemoteIPHeader X-Forwarded-For
	RemoteIPInternalProxy 127.0.0.1/8
</IfModule>

# set HTTPS if the original request was a HTTPS request
<IfModule mod_setenvif.c>
	SetEnvIf X-Forwarded-Proto "^https$" HTTPS
</IfModule>

Example of a file in vhost.d/:

# enable new domain to use document root /var/www/example/com/public_html/[subdomain]/
Use VHost example.com

Just add files to the vhost.d/ directory, and rename them to example.com.conf.off to turn them off and reload the Apache configuration files.

PHP-FPM

Add the following file to /etc/php-fpm.d/. Your could make a template file with $domain and $user and replace those occurrences with sed.

/etc/php-fpm.d/example.com.conf

[example.com]
listen = /var/run/php-fpm/php-fpm-example.com.sock
listen.backlog = 128
listen.allowed_clients = 127.0.0.1
listen.owner = example
listen.group = apache
listen.mode = 0660

user = example
group = nobody

pm = ondemand
pm.process_idle_timeout = 10s
pm.max_children = 5
pm.max_requests = 4096

catch_workers_output = yes

env[TMP] = /var/www/example.com/tmp
env[TMPDIR] = /var/www/example.com/tmp
env[TEMP] = /var/www/example.com/tmp

php_admin_value[open_basedir] = /var/www/example.com/public_html/:/var/www/example.com/tmp/
php_admin_value[upload_tmp_dir] = /var/www/example.com/tmp
php_admin_value[error_log] = /var/www/example.com/log/php_errors.log
php_admin_value[session.name] = example
php_admin_value[session.cookie_domain] = ".example.com"