Me

Taco de Wolff

Data Scientist

The perfect webserver

Published on May 28, 2019

To run a secure and performant webserver, you need to think-through the layout and configuration of your services. You want to enable HTTP/2 and set secure SSL settings while setting the file permissions in a way that even in case a website gets hacked, no other part of the webserver is vulnerable. This is what I learned when I replaced my second VPS with a better one.

My VPS runs on CentOS 7 with 4GB RAM and a dual core.

Software

NGINX - the webserver that will process requests and return responses. It also does SSL termination and deferral to PHP-FPM.

PHP-FPM - execute PHP scripts in worker threads. This is a performant way to process PHP scripts, as the worker threads keep running to process new requests. This is unlike the historically more common mod_php, which spawns a worker for every request. PHP code should be cached in socache to reduce the overhead of parsing the script. For sessions we will use an in-memory cache.

Let’s Encrypt and lego - for our SSL certificates we will use the Let’s Encrypt service that provides free SSL certificates. The ACME client we use to obtain the certificates is github.com/xenolf/lego, which is a Go library with a CLI with which we can automate the creation and renewal of certificates.

Users and file permissions

We will use different user accounts to separate and secure the different websites that are hosted. That is to say, each website will have its own system account that owns the files. PHP-FPM will have a pool for each website to run using that user. All the files will be restricted so that only the user can edit the files and the group (NGINX) has read access.

The websites will be located under /var/www. Each website will have subdirectories to store the websites, the temporary files, logs, and emails for example. The website files will have subdirectories for all subdomains so that we can add subdomains automatically by adding a new subdirectory. The subdirectory _ refers to when no subdomain is given and www for example is the common www subdomain. The file layout is thus as follows:

/var/www/
├── default/  # HTML files displayed if no domain matches
├── jaap.nl/
│   ├── tmp/
│   ├── log/
│   └── html/
│       ├── _/
│       ├── www/
│       └── [more subdomains]
├── piet.nl/
│   ├── tmp/
│   ├── log/
│   └── html/
│       ├── _/
│       ├── zwarte/
│       └── [more subdomains]
└── [more websites]

To continue with the jaap.nl example, we create a new website and system user as follows:

mkdir -p /var/www/jaap.nl/html/_
mkdir /var/www/jaap.nl/tmp
mkdir /var/www/jaap.nl/log

# make sure the user is a system user and cannot login
useradd -d /var/www/jaap.nl -s /sbin/nologin jaap >/dev/null 2>&1
usermod -a -G jaap nginx

# set all permissions correctly, run whenever to make sure all is correct
chown -R jaap:jaap /var/www/jaap.nl
find /var/www/jaap.nl -type d -exec chmod 750 {} \;
find /var/www/jaap.nl -type d -exec chmod g+s {} \;
find /var/www/jaap.nl -type f -exec chmod 640 {} \;

Add server-wide account with write access

In order to enable a system-wide SSH account to be able to access and modify all websites, we’ll use ACLs to add another user with write access. This is useful if you need to manage a handful of websites but would like to use scp or rsync to transfer files. If this SSH account is called admin, we can give it write permissions as follows:

setfacl -m u:admin:rwx /var/www/jaap.nl
find /var/www/jaap.nl/html -type d -exec setfacl -m u:admin:rwx,d:u:admin:rwx,m:rwx {} \;
find /var/www/jaap.nl/html -type f -exec setfacl -m u:admin:rw,m:rw {} \;

This will give directories rwx access to admin, and rw access to files. Additionally, newly created files and directories will automatically be given rwx permissions, which in combination with the standard file mask will result in rw for files. Creating new files with this account will ofcourse change the ownership of those files to admin, but since we’ve set chmod g+s to all directories, the group will still be jaap so that jaap and NGINX still have read access.

NGINX

The brunt of the webserver will be done by NGINX: serving the static files, redirecting to PHP-FPM for the dynamic files, termination of SSL, handling of HTTP/2, redirection of automatic certificate renewal, etc.

In /etc/nginx/nginx.conf we set the following:

user nginx;
worker_processes 2;  # one per CPU core
worker_rlimit_nofile 8192;

error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
    worker_connections 4096;
    multi_accept on;
    use epoll;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    # set secure SSL settings
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_dhparam /etc/nginx/dhparam.pem;
    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
    ssl_ecdh_curve secp384r1;
    ssl_session_timeout 10m;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets off;
    ssl_stapling on;
    ssl_stapling_verify on;
    resolver 1.1.1.1 1.0.0.1 valid=300s ipv6=off;
    resolver_timeout 5s;
    ssl_certificate     /etc/pki/lego/certificates/fqdn.server.nl.crt;
    ssl_certificate_key /etc/pki/lego/certificates/fqdn.server.nl.key;

    server_names_hash_bucket_size 128;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" '
                      'req=$request_time '
                      'resp=$upstream_response_time';
    access_log off;

    sendfile    on;
    tcp_nopush  on;
    tcp_nodelay on;

    keepalive_timeout 30;

    # enable automatic GZIP compression
    gzip on;
    gzip_disable msie6;
    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 1;
    gzip_min_length 10240;
    gzip_types
        text/css
        text/plain
        text/javascript
        application/javascript
        application/json
        application/x-javascript
        application/xml
        application/rss+xml
        application/atom+xml
        application/xhtml+xml
        application/x-font-ttf
        application/x-font-opentype
        application/vnd.ms-fontobject
        image/svg+xml
        image/x-icon
        font/truetype
        font/opentype;

    # improve security
    server_tokens off;
    charset UTF-8;
    add_header X-Frame-Options "SAMEORIGIN";
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Content-Type-Options nosniff;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

    client_max_body_size 1024m;

    include /etc/nginx/conf.d/default.conf;  # load in the default handling
    include /etc/nginx/sites-available/*.conf;  # load in our websites
}

Note that you need to have a fqdn.server.nl.crt for your whole server. This is when someone tries to contact your server for an unknown domain name through HTTPS, in which case the default certificate will be used (which is unlikely to match with the requested domain name and will fail the security test in the browser).

In /etc/nginx/conf.d/default.conf we will set a couple of default handlers:

# Default server when nothing matched
server {
    listen 80 default_server;
    listen 443 ssl http2 default_server;
    server_name _;
    root /var/www/default;

    # accept certificate verification for (yet) unavailable websites
    location ^~ /.well-known/acme-challenge/ {
        proxy_pass http://127.0.0.1:8081;
        proxy_set_header Host $host;
    }

    location / {
        error_log off;
        return 404;
    }
}

# IP address access
server {
    listen 80;
    server_name 83.96.200.140;
    root /var/www/default;
    error_log off;
}

server {
    listen 443 ssl http2;
    server_name 83.96.200.140;
    root /var/www/default;
    error_log off;

    return 301 http://$host$request_uri;
}

# FQDN
server {
    listen 80;
    server_name fqdn.server.nl;
    root /var/www/default;

    location ^~ /.well-known/acme-challenge/ {
        proxy_pass http://127.0.0.1:8081;
        proxy_set_header Host $host;
    }

    location / {
        error_log off;
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl http2;
    server_name fqdn.server.nl;
    root /var/www/default;

    # set up routes for server stats or other utilities admins need
}

Note that we use /var/www/default as the default website without PHP support to load when accessing the IP or the FQDN. Also, some routes will accept the /.well-known/acme-challenge/ route to confirm domain names for SSL creation using lego.

Website configuration

To add a new website we could use the following configuration:

server {
    listen 80;
    listen 443 ssl http2;
    server_name jaap.nl;  # this is the domain name NGINX will use this for

    ssl_certificate     /etc/pki/lego/certificates/jaap.nl.crt;
    ssl_certificate_key /etc/pki/lego/certificates/jaap.nl.key;
    error_log /var/www/jaap.nl/log/error.log;

    root /var/www/jaap.nl/html/_;
    index index.php index.html index.htm;

    # accept SSL verification
    location ^~ /.well-known/acme-challenge/ {
        proxy_pass http://127.0.0.1:8081;
        proxy_set_header Host $host;
    }

    location / {
        # refuse access to non-existing subdomains
        if (!-d /var/www/jaap.nl/html/_) {
            return 404;
        }

        # force HTTPS scheme
        if ($scheme = "http") {
            return 301 https://$host$request_uri;
        }

        try_files $uri $uri.html $uri/ /index.php?$args;
        http2_push_preload on;
    }

    # handle dynamic pages
    location ~ \.php$ {
        try_files $uri =404;
        http2_push_preload on;

        fastcgi_pass unix:/var/run/php-fpm/jaap.nl.sock;
        fastcgi_index index.php;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        include fastcgi_params;
    }

    # use browser cache for images and other resources
    location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
        expires 7d;
        add_header Cache-Control "public";
    }
}

Website redirection

For a subdomain you obviously have to add the subdomain in the server_name and change the root as well. To redirect the www subdomain to the previous handler, we’d use:

listen 80;
listen 443 ssl http2;
server_name www.jaap.nl;

ssl_certificate     /etc/pki/lego/certificates/jaap.nl.crt;
ssl_certificate_key /etc/pki/lego/certificates/jaap.nl.key;
error_log /var/www/jaap.nl/log/error.log;

location ^~ /.well-known/acme-challenge/ {
    proxy_pass http://127.0.0.1:8081;
    proxy_set_header Host $host;
}

location / {
    error_log off;
    return 301 https://jaap.nl$request_uri;
}

Lego ACME

By using lego and its CLI we can automatically create SSL certificates and do auto-renewal through a cron job.

Certificate creation

We will save our certificates to /etc/pki/lego and disable the tls-sni-01 and dns-01 methods, as we won’t use those.

lego -m [email protected] -k rsa4096 --http ":8081" -x "tls-alpn-01" -x "dns-01" --path /etc/pki/lego -d jaap.nl run

You can add alternative names to the certificate by appending more domain names such as -d www.jaap.nl. The certificates are put into /etc/pki/lego/certificates/ and consist of a key and certificate file.

To create one for all subdomains, you could write a bash script such as:

DOMAIN="jaap.nl"
DOMAINS="-d $DOMAIN -d www.$DOMAIN"
for d in /var/www/$DOMAIN/html/*/; do
    d=$(basename "$d")
    if [ $d != "_" ] && [ $d != "www" ]; then
        DOMAINS="$DOMAINS -d $d.$DOMAIN"
    fi
done

echo "Generating certificates: $DOMAINS"
lego -m [email protected] -k rsa4096 --http ":8081" -x "tls-alpn-01" -x "dns-01" --path /etc/pki/lego $DOMAINS run

Certificate renewal

For automatic renewal we’ll create a script that is to be called by a cron job daily and renews certificates that are less then 30 days valid (recommended by Let’s Encrypt).

Create /etc/cron.daily/tls-renew-all.sh that contains:

echo "Renewing all certificates"
for d in /etc/pki/lego/certificates/*.key; do
	echo "Checking $d..."
	d=$(basename "$d")
	d=${d::-4}
	lego -m [email protected] -k rsa4096 --http ":8081" -x "tls-alpn-01" -x "dns-01" --path /etc/pki/lego -d $d renew --days 30
done
echo "Restarting NGINX..."
systemctl reload nginx

PHP-FPM

Create /etc/php-fpm.d/jaap.nl.conf with the following content. Note that here we ensure that PHP is being run using jaap:jaap as the user and group, as well as that temporary files and logs are saved under /var/www/jaap.nl/.

[jaap.nl]
listen = /var/run/php-fpm/jaap.nl.sock
listen.owner = nginx
listen.group = nginx

user = jaap
group = jaap

pm = ondemand
pm.process_idle_timeout = 10s
pm.max_children = 5
pm.max_requests = 4096

catch_workers_output = yes

env[TMP] = /var/www/jaap.nl/tmp
env[TMPDIR] = /var/www/jaap.nl/tmp
env[TEMP] = /var/www/jaap.nl/tmp

php_admin_value[open_basedir] = /var/www/jaap.nl/public_html/:/var/www/jaap.nl/tmp/
php_admin_value[upload_tmp_dir] = /var/www/jaap.nl/tmp
php_admin_value[error_log] = /var/www/jaap.nl/log/php_errors.log
php_admin_value[session.name] = jaap
php_admin_value[session.cookie_domain] = ".jaap.nl"
php_admin_value[sendmail_path] = "/usr/sbin/sendmail -t -i -f [email protected]"

Closing notes

This is an example setup with NGINX + PHP-FPM and secure file permissions. You’ll obviously need to install MariaDB for most websites. I also recommand installing fail2ban and enable it on all ports. If your websites run WordPress make sure to install the fail2ban plugin and enable the filter in fail2ban, this will prevent people bruteforcing their way into your client’s WordPress.

Some of the scripts displayed above could benefit from using a template file with variables such as $domain or $subdomain. That way you can easily replace all occurrences of these variables by new values using sed for example.