Skip to content

opnsense transfer certificate to internal server

Description

This how to describes the automation of transfering a letsencrypt certificate, which is created on an opnsense firewall/gateway, to internal systems.

Why do I need this?

We are using a opnsense instance to connect the cloud resources to the internet. The use case is not only to connect resources to the internet. This could be also done easyly with a router. We use the instance also as a firewall, a vpn gate, internal dns cache, loadbalancer, mx-gate, etc.

In some use cases, the connection is forwarded to an internal resource. If the internal resource is providing a service based on TLS, the system needs to provide a certificate to the client from the internet. In all these cases the same problem appears.

The Trouble!

The client is connecting to the external IP address of the opnsense instance. The instance is connected to the internet, means has a public IP address and can use certificates from letsencrypt. The client can verify the certificate and establish the communication.

If the connection is forwarded to an internal resource, than the internal resource is providing its certificate, and this certificate is regularly self-signed or using internal PKI and can in close to all cases not a valid certificate for the client, because it is not able to verify the certificate.

The Solution

To come up with a solution after understanding the certificate mismatch is easy. The letsencrypt signed certificate on the opnsense instance needs to be transfered to the internal resource and it uses also the certificate from the opnsense instance. For the client than is only the external IP of the opnsense and its certificate on the internal resource. The match is done and no warnings are given to the client from his client applications.

Example

At the moment it is only used for mail communication (smtp, imap, pop), which will be described in the example. May be in future there will be more examples. We will see :)

Preparation

Some steps are not described, because they are at the moment out of the scope of this howto. But here is a list of the needed pieces to use this description:

On the opnsense instance

  • Configure the opnsense instance to be able to create letsencrypt signed certificates: install acme plugin and set it up.
  • Create a letsencrype certificate with the domain name, which is pointing to the IP of the opnsense IP address.
  • Find the certificate in the directory structure of opnsense. You will find them under: /var/etc/acme-client/certs/CERTIFICATE-ID/fullchain.pem. The ID looks like: 642e99463982d2.46857863
  • Create a ssh keypair to use for the transfer (we use scp) of the certificate. In this example, we are using the keyname "cert2mx".
  • Create a further administrator on the opnsense. In this example, I use the user "Hutee9quaeh9vaix". We do not want to work with root!!!
  • Enable shell and ssh access for this user to the opnsense instance.

On the internal mailserver

  • Create on the instance, which is providing the internal resource, the mail protocols, a new user, which will be used to transfer the certificate. In this example it will be "cert2mx".
  • Add the public key (generated on the opnsense) to the authorized_keys file in the home directory of the created cert2mx user.
  • Copy the key for the certificate (this one will not change, only the certificate generated based on this) and the recent certificate to the mailservers and place it in the home directory of the new user cert2mx, for example in file names mailserver.key and mailserver.crt.
  • Check the mail applications (postfix and dovecot), that the certificate file, they use, is a link to the certificate in the home directory of cert2mx. In this example, the configuration of postfix and dovecot are looking like this:

Postfix:

smtpd_tls_cert_file = /etc/postfix/smtpd.cert
smtpd_tls_key_file = /etc/postfix/smtpd.key

Dovecot:

ssl_cert = </etc/postfix/smtpd.cert
ssl_key = </etc/postfix/smtpd.key

Now create the links for the certificate and the key:

ln -s /home/cert2mx/mailserver.key /etc/postfix/smtpd.key
ln -s /home/cert2mx/mailserver.crt /etc/postfix/smtpd.cert

Restart the services for postfix and dovecot to be sure the new configuration works. At this timepoint we have solved the certificate mismatch issue. The service sould be provided without any warnings for the clients.

Configure and setup certificate changes

The copy process of the certificate could be done in both direction. We need to decide, which way could be more secure.

Allowing connections to the internal resources from the internet means, that there could be vulnerabilities and the could be misused to get access to the system. This is the risk we need to take. The opnsense on the other hand does not provide any services to internet and should be more trustable than the internal resource.

Furthermore, if a internal resource is compromised, we loose on ly the resource. But if the opnsense instance could be compromised, we loose everything!

So, we copy from the opnsense instance to the resource, to not open a way to access the firewall from an internal resource.

Setup tasks on the opnsense

On the opnsense instance, we need to create a cron job which will copy the certificate to the internal resources. We use two redundant mailservers, so it needs to copy the certificate to both servers.

Connect to the opnsense instance with the new administrator:

ssh Hutee9quaeh9vaix@10.8.0.1

You will see something like this by getting the shell:

Last login: Wed Mar 20 11:02:21 2024 from 10.0.0.2
----------------------------------------------
|      Hello, this is OPNsense 23.7          |         @@@@@@@@@@@@@@@
|                                            |        @@@@         @@@@
| Website:      https://opnsense.org/        |         @@@\\\   ///@@@
| Handbook:     https://docs.opnsense.org/   |       ))))))))   ((((((((
| Forums:       https://forum.opnsense.org/  |         @@@///   \\\@@@
| Code:         https://github.com/opnsense  |        @@@@         @@@@
| Twitter:      https://twitter.com/opnsense |         @@@@@@@@@@@@@@@
----------------------------------------------
Hutee9quaeh9vaix@ciprocRNCgate:~ $

Edit the crontab (it will be opened with vi):

crontab -e

Enter these lines to copy the certificate to the internal servers. Please replace IP_ADDRESS and NAME_OF_FILE with the IP addresses of the internal resources and the filename you prefer (if happy with fullchain.pem, remove .crt).

In this example the transfer is schedules for 04:17 am daily:

17 4 * * *      sudo scp -P999 -i /root/.ssh/cert2mx /var/etc/acme-client/certs/642e99463982d2.46857863/fullchain.pem cert2mx@<IP_ADDRESS>:~/<NAME_OF_FILE>.crt
17 4 * * *      sudo scp -P999 -i /root/.ssh/cert2mx /var/etc/acme-client/certs/642e99463982d2.46857863/fullchain.pem cert2mx@<IP_ADDRESS>:~/<NAME_OF_FILE>.crt

Setup tasks on the internal mailserver

Now we have finished the creation of the crontab on the opnsense, we need a job, which will check the certificate to detect a change and if the certificate has changed, the certificate will be copied to the configured directory for the mailserver to use. The reason to check the certificate for changes a head reduces the restarts of the mail applications and potential problems with restarting.

Have in mind, that we are copying and restating services on all systems. If something went wrong, it will be on all systems and the service is definitly down!!!

For this purpose we use a script reload_cert on the mailservers to check and implement the new certificate:

#!/bin/bash


HOME=/opt/reloadCert
PATH=$HOME:$PATH
DATUM=`date +%Y%m%d%H%M%S`
LOGFILE=$HOME/reload_cert.log
CERT2MX=/home/cert2mx

aelog () {

        echo "$(date +%Y%m%d-%H%M%S): $1"  >> $LOGFILE

}


aelog "====================================================="
aelog "Start: $DATUM"


OLD=$HOME/cert_hash_old
NEW=$HOME/cert_hash_new

sha256sum $CERT2MX/mailserver.crt > $NEW

OLDHASH=`cat $OLD | awk '{print $1}'`
NEWHASH=`cat $NEW | awk '{print $1}'`

aelog "oldhash: $OLDHASH"
aelog "newhash: $NEWHASH"


if [ $OLDHASH == $NEWHASH ]
    then
        aelog "unchanged - nothing to do"

    else
        aelog "changed - reloading certificate"
        systemctl reload postfix
        systemctl reload dovecot
        aelog "postfix, dovecot reload done"
        mv $NEW $OLD
fi


exit 0

You will find the script filed here.

Don't forget to change the sertificate file name in the script, if something else than mailserver.crt is used!

It will do the following: - sha256sum of the certificate file - compare it to the latest hash - restart the services, if certificate has changed - log the results to /home/cert2mx/reload_cert.log

Please adapt the script to your needs, make it executable and place it on the mailserver, where you will call it from the crontab for the cert2mx user on the mailservers.

In this example, the script is placed under /opt/reloadCert.

So the crontab entries for the cert2mx user on the mailserver looks like this:

0 5 * * * /opt/reloadCert/reload_cert

TEST IT!

Test the workflow through modifying the cron timestams and check the results.

Trouble shooting

  • I am at the moment not sure, but the script could stuck the first run, because it does not have an old hash file to load. Create an empty file with the name "cert_hash_old". This should be fine, to detect the needed files and get its first change.