How to upgrade Check Point Multi-Domain management from R71.20 to R75.30

The main reason for writing this post is due to the fact that there is absolutely no information regarding this topic. I had to reinstall whole LAB environment five times before I found a breakthrough. Check Point Multi-Domain management (Provider-1) is a centralized solution for firewall management used in big environments. You can assign global policies, IPS settings, separate countries/regions into CMAs and share objects between them. You also have the ability for primary-slave CMAs and CLMs (centralized logging server) so in case a datacenter goes down, you can manage firewalls via slave MDS. In R75.30 you can also export/import objects between CMAs but let’s talk about it later.

In this post I will describe the upgrade process on real data, real management which maintains around 25-30 firewall clusters and something about 100.000 objects. It’s not similar to available "HowTo Guide" where you only have some patching etc. on Any/Any firewall.

As usual it hasn’t been painless, but I will describe what issues can happen and how to fix them.

More information:
Multi-Domain Management / Provider-1 - https://supportcenter.checkpoint.com

// Preparation

There are two possible upgrade paths:
a] R71.20 -> R75 -> R75.20 -> R75.30
b] R71.20 -> R71.30 -> R75.20 -> R75.30

Preferable though is option A because R75 has the ability to upgrade from R71.20 from its beginning and it seems to be tested closely and deeper than option B.

You have to download these files:

Our environment (MDS/MLM) is running on OpenServers so re-select packages accordingly to your situation.

Please note the whole upgrade process occurs in CLI (bash - expert mode). I would not recommend using WebUI. I already experienced that the upgrade process "finished without problems" but box itself has been unstable and a lot of essential applications for management have had to be deleted (mdsstat, mdsstart, cpconfig, ...). Regarding logs I did find that at the end of upgrade process, the installer deletes those scripts, then it wants to check for licenses and after that wants to install new version of them. But script for checking licenses fails and new versions is never installed... so, be careful and follow me on CLI.

In my case I did the upgrade in three steps:

1] Upgrade of primary MDS
2] Upgrade of standby MDS and primary MLM
3] Upgrade of standby MLM

During upgrade process nobody should change anything on MDS environment.

LAB in VirtualBox (optional)

I was able to make whole upgrade in VirtualBox environment as well. Just ensure that your LAB can't reach firewalls which can possibly affect them. For this is best option Host-Only network where your testing server can reach only host running VirtualBox.

// Update MDS licenses

You are unable to update your actual licenses in Check Point support section to R75, YOU MUST contact account services to do that (believe me).
This part is essential and can’t be skipped. If you don’t update licenses your upgrade will fail and you have to recover it from backup. Again, this is not a joke. I had to recover MDS twice because of this.

Print your actual licenses by command:

Send output to Account Services if they can update your actual licenses or if provided one are already prepared for R75.

Here you have example of output:

// MDS Backup

First of all you have to make backup of MDS. Create some directory in /var, cd into it and run:

This will generate 4 files. All of them (!) copy to your PC or different server (compare md5sum after that).
Check Point also provides you something called "System Snapshot" for whole appliance/server backup with all partitions etc. DO NOT use it! It doesn’t work in some cases and if it happens you’re simply ... out of recovery :]

Exclude logs and db_versions from mds_backup (optional)

If you don’t need backup of logs you can add exception which tells mds_backup to not do so.

Edit file $MDSDIR/conf/mds_exclude.dat and append two lines:

Delete logs on MLM/CLM

Delete logs on MLM because otherwise they will be re-duplicated to every upgraded version (another bug) and MLM will be soon out of disk space.

Logs can be found here: /var/opt/CPmds-R71/customers/$CUSTOMER_NAME/CPsuite-R71/fw1/log/
You can safely leave last log file (fw.log) only and deleted or move the rest to different server.

Fixing DBImage issue

In my case mds_backup failed on standby MDS, because there is bug in R71.x which will create tons of DBImage files on standby CMA (one file for each policy installation). Due to this, mds_backup failed when compressing folder (too much arguments).

I have found SK describing this problem and only solution provided was delete those files.

Delete unused objects (optional)

Your management can be overloaded by many unused objects. It’s always a good habit to delete them from time to time.
It can speed-up policy verification, SmartDashboard loading and also this upgrade because during upgrade process Check Point rebuilding object database.
You can find Unused objects in SmartDashboard by Query object and then select "Unused objects"

Delete everything except the five items which can’t be deleted. So select all objects, remove from selection specified items and delete them. It can take a lot of time if you’re deleting (for example) 5k objects. If possible, make this on some server accessible via RDP where it can run over night. This step is not mandatory for the upgrade.

// Upgrade from R71.20 to R75

On MDS/MDM (at this moment still Provider-1) create dir in /var and allow scp transfers for the admin account.

Make sure that you have valid MDS backup saved somewhere else.

Now copy from your server/PC installation DVD for R75. For example via scp:

It’s also good idea to double-check file md5sum on destination MDS.

Mount DVD on MDS and run upgrade process

Example of ./mds_setup output:

Select first option and read all problems which could occur. Some of them can be irrelevant and can be ignored.
Otherwise don’t go through upgrade until all issues are resolved.

Run ./mds_setup again and select option 2 - Upgrade to R75

It can take 1-2 hours.
You will be then asked if you want to add another leading interface, MDS administrators, GUI clients, Groups, ... leave all as is.

If you think that upgrade is completed please double-check this file:

If you see that installation is really completed and no new lines appear then you can be sure that the installation is complete. I saw that post-installation process is still rebuilding CMA database even if upgrade process has to be finished...
Or you can check system load via "top" command. When you see that system isn’t working on something else, you can safely reboot system.

// Upgrade from R75 to R75.20

After reboot MDS will start all CMAs. It will take more than usual time so be patient.

When all CMAs are up:

Check license status:

You shouldn’t encounter a problem there. If so, you have to make rollback, update license and try upgrade again.
Rollback means that you will reinstall server to R71.10, upgrade ro R71.20 and then import previous mds_backup.

Run the script that checks (fixes) that database have correct version.

Where $mds-hostname is name of your MDS server (without domain).

Example output:

Make reboot once more (just to be sure).

Moving on to the next upgrade:

-> copy file to MDS

Example output:

Rest of output is same as in previous update. As I stated above backup image is not needed and just waste your time.

When all is done and system does not show some load (top command) you can safely reboot server to R75.20

// Upgrade to R75.30

When all CMAs are UP (mdsstat), run script checking version again.

Copy upgrade package to MDS

Example output:

Cool... after reboot we have to be on R75.30
Just check all as before: mdsstat, cplic print, delete /var/mdsupgrade

Now you can login via R75.30 SmartDomain Manager.

You need to install Check_Point_R75.30_SmartConsole.Windows.exe and after that Check_Point_R75.30_SmartDomain_Manager.Windows.exe

If you're able to login you can continue with upgrading rest of servers.

In next part I will focus on problems after upgrade and how to fix them.

Next part: How to fix problems after upgrade to Check Point Multi-Domain management R75.30