opensense high-availabitlity with CARP and pfSync

• Description
• How does it work?
• Example
• Preparation
• Disable port security (OpenStack only)
• Configuration of Virtual IPs (uses CARP)
• Check the status of Virtual IPs
• Configuration of the config sync (uses pfSync/XMLRPC Sync)
• Check the status of High-Availability Settings

Description

This how to describes the setup of high-available OpnSense firewalls/gateways. In this description you will find only the configuration part of the high-availability. The installaion and initial configuration is not part of this howto.

The high-availability solution mentioned in this howto is based on a master instance and a backup one for failover in cases the master instance fails.

How does it work?

OpnSense uses CARP (Common Address Redundancy Protocol) to detect and enable hardware/instance failover.

Because CARP is only used for availability of the IP resources on the instances, we need to have both instances configured in exact the same way.

For example, it must have the same IP configurations, the same users, the same VPN clients, the same rules, ...

This can be challenging, because each change needs to be done on both instances, that everything can work further after a failover to the backup system.

For this reason pfSync and XMLRPC Sync is used to synchronize the configuration to the backup.

In the following example, we will do the following steps:

• Configuration of virtual IPs (uses CARP): On the web GUI you will find the settings under "Interfaces -> Virtual IPs -> Settings"

• Configuration of the config sync (uses pfSync/XMLRPC Sync): On the web GUI, it is under "System -> High Availability -> Settings"

Further details and screenshots will be show below in the example.

Example

The following example shows the configuration of a high-available OpnSense setup with two instances a master and a backup.

This example will have one WAN connection and three internal networks to connect to the internet and deliver services.

Preparation

We need to prepare the used IP addresses and the needed network connections.

Do not forget to activate the WebUI also on the pfsync_net!!!

If working on the OpenStack, do not forget to deactivate the port security on all network connection. There is no need to filter same packets with the same rules on different layers (Openstack port-security and later in the instance and its packet filters).

Here is a summarized list of the IP addresses, which will be used. In this case firewall01 is the master and firewall02 the backup system.

Network	firewall01	firewall02	VIP
WAN	213.206.190.46	213.206.175.56	213.206.175.60
service_net	192.168.117.11	192.168.117.12	192.168.117.1
backend_net	192.168.213.11	192.168.213.12	192.168.213.1
volume_net	192.168.251.11	192.168.251.12	192.168.251.1
pfsync_net	192.168.27.1	192.168.27.2	---

Disable port security (OpenStack only)

On the OpenStack WebUI edit the connected port and disable port security.

To edit the port open "Network -> Networks", which will deliver you a list of the networks of the project. Then choose the network you would like to edit the port and move to the "Ports" tab on the page. Finally click "Edit Port" in the row of the listed port and diasble it:

After clicking "Save" the port security should be disabled.

Configuration of Virtual IPs (uses CARP)

This needs to be done on both instances.

In the OpnSense WebUI go to "Interfaces -> Virtual IPs -> Settings"

Click on the plus (in orange on the right side) and add the first virtual IP.

NOTE: You should aktivate in the settings the advance mode to be able to modify the "advskew" for the virtual IP, if needed. Regularly OpnSense changes this after the pfsync and the XML config transfer. The important part is, that the master has a smaller value than the backup instance.

In this example, we start with the WAN VIP: 213.206.175.60

It does not matter in which order you create the virtual IPs. You need to take care to setup each VIP you prepared in its own VHID Group.

As the second one, we use here the service_net VIP: 192.168.117.1

Repeat this for further VIPs if needed.

After setting up the needed VIPs your view in the "Interfaces -> Virtual IPs -> Settings" should look like this.

On the master:

On the backup:

Check the status of Virtual IPs

Now go to "Interfaces -> Virtual IPs -> Status" on both systems and check it to be sure, things are running as expected.

It should look like this on the master:

It should look like this on the backup:

To test the setup, you can use the "Temporarily Disable CARP" button. This disables CARP and you should now see the status on the backup instance changing to "MASTER".

The behaviour of the VIPs, if the master takes over after activating CARP or not is depending on the high-availability settings. The next step is to configure the HA behaviour.

Configuration of the config sync (uses pfSync/XMLRPC Sync)

This step will only be done on the master!!!

For this to setup go to "System -> High Availability -> Settings".

• Activate "Synchronize States"
• Choose the sync network pfsync_net "Synchronize Interface"
• Set the "Synchronize Peer IP". If not set, the default 224.0.0.XXX will be used, which then need some fight with the firewall rules.

Configuration Synchronization Settings (XMLRPC Sync) Perform synchronization

• Setup "Synchronize Config to IP" - If not using default port 443. use full URL: https://:/
• Enter "Remote System Username"
• Enter "Remote System Password"

This should enable to transfer the configuration from the master to the backup system.

But there is one setting "Disable preempt" (the first one), you should enable or disable, it depends on your needs. Here the description from OpnSense:

"When this device is configured as CARP master it will try to switch to master when powering up, this option will keep this one slave if there already is a master on the network. A reboot is required to take effect."

The last step is to configure, which parts should be synced to the backup. The list will be find on the same page. You just need to scroll down and choose, what is needed:

Check the status of High-Availability Settings

To check this, just switch to "System -> High Availability -> Status"

Do not forget to make tests to be sure, everything is working as expected.

Create firewall rules, users, certificates, ... , klick on the sync icon on the High Availability Status page and then move to the backup system to see, that the items are transfered.

Do not forget to work on the master from now on. The sync is working in one direction!!!