F5 BIGIP and HAProxy — Masking 2-Way “Mutual” SSL Authentication

Hello folks,

So a recent post I published talked about 1-Way vs 2-way SSL Authentication in some decent detail. We learned that 2-Way “Mutual” SSL Authentication can be used to enforce both parties attempting to communicate securely to provide authenticity. In other words, prove to each other that they are who they say they are. This can be very powerful from a security standpoint, but is it practical? The answer is, yes and no. The constraint comes from the aspect of administration (actually create certificates for each client) and manageability (keep accounting and maintaining actively lists of trusts) with the trade-off of proper authenticity. For example at first administering and managing 10 client certificates may be okay, but then imaging 100, or even a 1,000! So in this post I wanted to approach the idea of utilizing some tools we can use to offload some of this administration and management while maintaining Mutual Authentication with another entity. The idea revolves around one major assumption, users of a particular service (In this case a web-server) reside on a privately controlled and trusted network

My idea is if we have a group of clients residing on an internal privately addressed network, we can use either an F5 LTM or HAProxy to proxy our users’s connections destined for a service that is enforcing 2-Way SSL “Mutual” Authentication. The F5 LTM or HAProxy would perform the 2-Way SSL Mutual Authentication on behalf of each connecting user, eliminating the technical need to generate certificates for each client, while maintaining an element of mutual trust to the end service.

The basic idea is: (notice only our F5 LTM/HAproxy and the web-server perform 2-Way “Mutual” Authentication)

Preliminary Steps:

For the following steps please read my 1-Way vs 2-Way SSL Authentication Post.

  1. Create a the web-server’s CSR and Key
    root@ca:/opt# openssl req -config openssl-rootca.conf -extensions server_req_ext -new -nodes -newkey rsa:2048 -keyout web-server.key -out web-server.csr -days 365
Generating a 2048 bit RSA private key
................................+++
............................................................................+++
writing new private key to 'web-server2.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Enter Country [US]:
State or Province Name (full name) [Connecticut]:
Locality Name (eg, city) [Wethersfield]:
Organization Name [thejimmahknows]:
Unit Name [Test Unit]:
Common Name (e.g. server FQDN or YOUR name) []:web-server
Contact email for this Certificate [admin@example.com]:

     

  2. Create the F5 & HAproxy Server-Side CSR and Key
    root@ca:/opt# openssl req -config openssl-rootca.conf -extensions client_req_ext -new -nodes -newkey rsa:2048 -keyout ha-client1.key -out ha-client1.csr -days 365
Generating a 2048 bit RSA private key
...................+++
.....+++
writing new private key to 'ha-client1.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Enter Country [US]:
State or Province Name (full name) [Connecticut]:
Locality Name (eg, city) [Wethersfield]:
Organization Name [thejimmahknows]:
Unit Name [Test Unit]:
Common Name (e.g. server FQDN or YOUR name) []:ha-client1
Contact email for this Certificate [admin@example.com]:

     

  3. Create the F5 & HAproxy Client-Side (connection the client will actual connect to)
    root@ca:/opt# openssl req -config openssl-rootca.conf -extensions server_req_ext -new -nodes -newkey rsa:2048 -keyout virtual-service.key -out virtual-service.csr -days 365
Generating a 2048 bit RSA private key
........................+++
.............................................................+++
writing new private key to 'virtual-service.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Enter Country [US]:
State or Province Name (full name) [Connecticut]:
Locality Name (eg, city) [Wethersfield]:
Organization Name [thejimmahknows]:
Unit Name [Test Unit]:
Common Name (e.g. server FQDN or YOUR name) []:mytestvip
Contact email for this Certificate [admin@example.com]:
  4. Using our CA from my previous ariticle, sign all three of these certificates

    Sign the web-server’s CSR

    root@ca:/opt# openssl ca -config openssl-rootca.conf -extensions server_req_ext -in web-server2.csr -out web-server2.crt
Using configuration from openssl-rootca.conf
Enter pass phrase for /opt/rootCA.key:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'US'
stateOrProvinceName   :ASN.1 12:'Connecticut'
localityName          :ASN.1 12:'Wethersfield'
organizationName      :ASN.1 12:'thejimmahknows'
organizationalUnitName:ASN.1 12:'Test Unit'
commonName            :ASN.1 12:'web-server'
Certificate is to be certified until Feb 25 15:10:27 2017 GMT (365 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

    Sign the F5 & HAProxy Server-Side CSR (This is the connection the F5 makes to our backend pool members)

    root@ca:/opt# openssl ca -config openssl-rootca.conf -extensions client_req_ext -in ha-client1.csr -out ha-client1.crt
Using configuration from openssl-rootca.conf
Enter pass phrase for /opt/rootCA.key:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'US'
stateOrProvinceName   :ASN.1 12:'Connecticut'
localityName          :ASN.1 12:'Wethersfield'
organizationName      :ASN.1 12:'thejimmahknows'
organizationalUnitName:ASN.1 12:'Test Unit'
commonName            :ASN.1 12:'ha-client1'
Certificate is to be certified until Feb 25 15:12:16 2017 GMT (365 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

    Sign the F5 & HAProxy Client-Side CSR (This is the connection our users will connect to, the VIP)

    root@ca:/opt# openssl ca -config openssl-rootca.conf -extensions server_req_ext -in virtual-service.csr -out virtual-service.crt
Using configuration from openssl-rootca.conf
Enter pass phrase for /opt/rootCA.key:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'US'
stateOrProvinceName   :ASN.1 12:'Connecticut'
localityName          :ASN.1 12:'Wethersfield'
organizationName      :ASN.1 12:'thejimmahknows'
organizationalUnitName:ASN.1 12:'Test Unit'
commonName            :ASN.1 12:'mytestvip'
Certificate is to be certified until Feb 25 15:15:09 2017 GMT (365 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated

     

  5. Configure Apache web-server to enforce the 2-Way “Mutual” Authentication

    Apache Config from previous post.

    root@web-server:/opt# vi /etc/apache2/sites-available/default.conf
Listen 443
<VirtualHost *:443>
        DocumentRoot           "/var/www/"

        SSLEngine               on
        SSLCACertificateFile   /opt/rootCA.crt
        SSLCertificateFile     /opt/web-server.crt
        SSLCertificateKeyFile  /opt/web-server.key
       SSLCARevocationFile    /opt/rootCRL.crl
        SSLStrictSNIVHostCheck on
        SSLVerifyClient        require
        SSLVerifyDepth         1

# Allows PHP to read Certificate info
        SSLOptions +stdEnvVars

        LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
        CustomLog "/tmp/access.log" combined
        ErrorLog "/tmp/error.log"

</VirtualHost>

    Restart Apache

    root@web-server:/opt# service apache2 restart

    For Troubleshooting create this index.php file

    <?php
echo "Date       =   " . date('Y-m-d H:i:s') . "<br>";

echo "Client Cert =  " . $_SERVER['SSL_CLIENT_S_DN_CN'] . "<br>";
echo "Server Cert =  " . $_SERVER['SSL_SERVER_S_DN_CN'] . "<br>";
echo "Server Serial =  " . $_SERVER['SSL_SERVER_M_SERIAL'] . "<br>";
?>

     

Masking with F5 LTM:

  1. Importing Certificates

    • Import virtual-service certificate
    • Import ha-client1 certificate
      • Use the same steps from above.
    • Import the rootCA certificate (used to authenticate the web-server)
      • Use the same steps from above.
    • All 3 Certificates imported
  2. Create the SSL Profiles
    • Create the client-side-connection SSL profile
    • Create the server-side-connection SSL profile
  3. Create the Server Pool
    • Verify Pool Status is GOOD
  4. Create the Virtual Server aka VIP
    • Verify VIP (Vritual Server) looks good
  5. Test by using Chrome to connect to our virtual-service

  6. Certificate revocation
    • Revoke the ha-client1.crt (The certificate the F5 authenticates with when connecting to the web-server) 
      root@ca:/opt# openssl ca -config openssl-rootca.conf -revoke ha-client1.crt -crl_reason keyCompromise
Using configuration from openssl-rootca.conf
Enter pass phrase for /opt/rootCA.key:
Revoking Certificate 1005.
Data Base Updated
    • Re-generate the CRL 
      root@ca:/opt# openssl ca -config openssl-rootca.conf -gencrl -out rootCRL.crl
Using configuration from openssl-rootca.conf
Enter pass phrase for /opt/rootCA.key:

root@ca:/opt# openssl crl -noout -text -in rootCRL.crl
Certificate Revocation List (CRL):
        Version 2 (0x1)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: /C=US/ST=Connecticut/L=Wethersfield/O=thejimmahknows/OU=Test Unit/CN=MyRootAuthority/emailAddress=admin@example.com
        Last Update: Feb 26 18:42:33 2016 GMT
        Next Update: Mar 27 18:42:33 2016 GMT
        CRL extensions:
            X509v3 Authority Key Identifier: 
                keyid:E3:A6:FD:69:23:0A:25:AF:7B:77:7A:B8:03:0B:B6:8A:CF:F2:B2:B8

            Authority Information Access: 
                CA Issuers - URI:http://ocsp.thejimmahknows.com/rootCA.crt

            X509v3 CRL Number: 
                4100
Revoked Certificates:
    Serial Number: 1003
        Revocation Date: Jan 16 18:55:22 2016 GMT
    Serial Number: 1005
        Revocation Date: Feb 26 18:38:04 2016 GMT
        CRL entry extensions:
            X509v3 CRL Reason Code: 
                Key Compromise

      Notice the Serial Number 1005 is revoked in the CRL file now.

    • Replace the rootCRL.crl file on the web-server and restart Apache 
      scp /opt/rootCRL.crl root@web-server:/opt/rootCRL.crl

root@web-server:/opt# service apache2 restart
    • Test using the virtual-service VIP on the F5 again.

Masking with HAProxy:

If you are unfamiliar with HAProxy, I recommend checking out my article on setting up HAProxy. Or my articles on using HAProxy as a F5 LTM replacement.

  1. Before we being, we have to generate and sign another certificate and key because we revoked the ha-client.crt perviously. 
    root@ca:/opt# openssl req -config openssl-rootca.conf -extensions client_req_ext -new -nodes -newkey rsa:2048 -keyout ha-client2.key -out ha-client2.csr -days 365
Generating a 2048 bit RSA private key
.+++
....................................+++
writing new private key to 'ha-client2.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Enter Country [US]:
State or Province Name (full name) [Connecticut]:
Locality Name (eg, city) [Wethersfield]:
Organization Name [thejimmahknows]:
Unit Name [Test Unit]:
Common Name (e.g. server FQDN or YOUR name) []:ha-client2
Contact email for this Certificate [admin@example.com]:

    And Sign it

    root@ca:/opt# openssl ca -config openssl-rootca.conf -extensions client_req_ext -in ha-client2.csr -out ha-client2.crt
Using configuration from openssl-rootca.conf
Enter pass phrase for /opt/rootCA.key:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'US'
stateOrProvinceName   :ASN.1 12:'Connecticut'
localityName          :ASN.1 12:'Wethersfield'
organizationName      :ASN.1 12:'thejimmahknows'
organizationalUnitName:ASN.1 12:'Test Unit'
commonName            :ASN.1 12:'ha-client2'
Certificate is to be certified until Feb 25 19:47:40 2017 GMT (365 days)
Sign the certificate? [y/n]:y


1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
  2. Copy All 3 certificates to our HAProxy server 
    root@ca:/opt# scp virtual-service.* root@172.16.0.44:/opt/
root@ca:/opt# scp ha-client2.* root@172.16.0.44:/opt/
root@ca:/opt# scp rootCA.crt root@172.16.0.44:/opt/
  3. We need to chain the virtual-service certificate with the root CA certificate for HAProxy to accept it. (For help reference this) 
    root@test-haproxy:/opt# cat virtual-service.key >> virtual-service-chain.pem
root@test-haproxy:/opt# cat virtual-service.crt >> virtual-service-chain.pem
root@test-haproxy:/opt# cat rootCA.crt >> virtual-service-chain.pem

    And ha-client2

    root@test-haproxy:/opt# cat ha-client2.key >> ha-client2.pem
root@test-haproxy:/opt# cat ha-client2.crt >> ha-client2.pem
  4.  Edit your haproxy.conf file to match 
    global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
       user haproxy
       group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3
        tune.ssl.default-dh-param 2048

defaults
        log     global
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http


frontend vs_172.16.0.44_443
        bind 172.16.0.44:443 ssl crt /opt/virtual-service-chain.pem
        default_backend pool_test2waySSL


backend pool_test2waySSL
        server testweb01 172.16.0.25:443 ssl verify required ca-file /opt/rootCA.crt crt /opt/ha-client2.pem
  5. Success!!
  6. Revocation test
    • Revoke and re-generate the rootCRL.crl file 
      root@ca:/opt# openssl ca -config openssl-rootca.conf -revoke ha-client2.crt -crl_reason keyCompromise
Using configuration from openssl-rootca.conf
Enter pass phrase for /opt/rootCA.key:
Revoking Certificate 1007.
Data Base Updated

      And re-generate the CRL

      root@ca:/opt# openssl ca -config openssl-rootca.conf -gencrl -out rootCRL.crl
Using configuration from openssl-rootca.conf
Enter pass phrase for /opt/rootCA.key:
    • Reload Apache on our web-server to pick up the new CRL file 
      root@web-server:/opt# service apache2 restart
    • Now we test….

 

Tags:

February 26, 2016 by F5 1

You may also like...

1 Response

  1. webjockey says:
    January 23, 2019 at 12:33 AM

    Yet another comprehensive article on 2 way SSL, not many people understand and know how to make it work, thanks

    Reply

Leave a Reply to webjockey Cancel reply

Your email address will not be published. Required fields are marked *