If you are running any version of Cisco ISE there is always going to be a time when you need to upgrade.
Cisco Identity Services Engine Upgrade
There are five basic steps to performing your upgrade
1. Verify your upgrade path
Depending on what version you are currently running will determine that steps you need to take to upgrade to the latest version. If you are upgrading from version 2.4 to 2.6 – this is a one step easy upgrade.
However if you are upgrading from a version earlier than 2.1 you will have to upgrade to 2.1 before going to 2.6
So I only have to download the latest ISE 2.6 upgrade software.
2. Prepare your system for the upgrade
This step requires creating a repository, uploading the upgrade image and the upgrade readiness tool if you want to use that.
3. Perform the upgrade
This step can be very easy if you are upgrading a standalone note or just a Primary & Secondary. It becomes a bit more involved if you have lots of policy service nodes, you just have to work out the correct sequence.
4. Install the latest patch for your version
Once you have upgraded ISE there will normally be a latest patch to install. This is a best practice step and advised to ensure you are running the latest stable version.
5. Verify correct operation
Once you have upgraded you need to verify that ISE is operating correctly, the best way is to just watch the live logs and make sure authentications are happening as normal.
OK, let’s start the ISE upgrade!
So I am going to be upgrading an ISE 2.4 to 2.6. At the time of writing 2.6 is the latest version.
This is a direct upgrade so I only have to download the ISE 2.6 upgrade image. This can be found on the Cisco download page for ISE 2.6 at the bottom.
https://software.cisco.com/download/home/283801620/type/283802505/release/2.6.0
Keep scrolling to the bottom!
You need to download the Upgrade bundle. This is for upgrading ISE from version 2.6 from 2.1 upwards. So it covers 2.1, 2.2, 2.3, 2.4 (there is no ISE 2.5)
While you are there you can also download the upgrade readiness tool (URT) this is not required but it is an ISE upgrade best practice.
The tool will check your current ISE installation and warn of any potential issues that could affect the upgrade.
Create a repository
We now need to create a repository on ISE so we can upload the upgrade bundle and the URT.
To create a repository in ISE head over to
Administration / System / Maintenance – then click on Repository
As you can see I have no repositories created so I am going to click on Add to create one. You will then have to enter the details of your server. This is however you plan to upload the images. I used FTP and use Filezilla server for this as I find it handles larger images better.
Click Submit
Now head over to
Administration / System / Upgrade
Click the Upgrade tab and you should see this screen
You have to agree to this screen that you have read and taken action on all the options before you proceed (you don’t have to action on every item, but you are well advised to do so!)
The main one here is a backup of the configuration and certificates as worse case scenario if the upgrade goes horribly wrong you can re-install ISE and restore from the backup.
I am not going to detail how to backup Cisco ISE here, but we will assume that you have been backing up your ISE installation every night since installation and you can tick that one off!
Once you have reviewed the check list and are happy you have covered everything, tick the box and the green Continue button will light up and you can click it.
You will now see that the bundle is not in the repository, as we have not uploaded it yet, so let’s do that.
Prepare the upgrade bundle
So you should now have downloaded 2 files from Cisco
- The Upgrade Bundle – in my case ise-upgradebundle-2.1.x-2.4.x-to-2.6.0.156.SPA.x86_64.tar.gz
- The URT – ise-urtbundle-2.6.0.156-1.0.0.SPA.x86_64.tar.gz
Make sure you have these in a folder where your FTP server can access them. I am using Filezilla Server and have set it up as follows.
I have created a user called roger and mapped that users shared directory to C:\users\roger\Downloads\ISE
This is where I will have my images saved.
You can now upgrade using the GUI or the CLI. I will detail both options here.
Upgrade via GUI
To upgrade using the GUI head over to Administration / System / Upgrade
Click on the Upgrade tab next to Overview and then select which node you want to upgrade first.
In my case it is a standalone node. Then click the Download button
You then need to select your repository and ISE will show you all files you have in that repository filtered by any that include ise-upgradebundle.
You should see your image, select it and then hit confirm, then begin download.
You should see your file start loading with a progress bar showing how much has been upgraded.
Once the image has been uploaded you are prompted to reload, your system should reload and come back running ISE 2.6
ISE Upgrade via CLI and using URT
The steps using the CLI are use the URT if you want to check everything before you proceed with the upgrade and then if your happy initiate the upgrade from the CLI
Cisco ISE Upgrade Readiness Tool
Now we can run the URT tool. This will perform a check if the system is ready for an upgrade. If it finds any issues it will report them so you can address before performing the real upgrade.
To run the upgrade readiness tool simply enter the command
application install ise-urtbundle-2.6.0.156-1.0.0.SPA.x86_64.tar.gz <repository-name>
It will upload the file and then run through a series of checks as below. It will warn you that it might take up some resources which you have to acknowledge.
It also warns you that the URT is 486 days old and it’s verion 1.0.0. It always says this! We have downloaded the latest version from Cisco.
80Save the current ADE-OS running configuration? (yes/no) [yes] ?
Generating configuration...
Saved the ADE-OS running configuration to startup successfully
Getting bundle to local machine...
Unbundling Application Package...
Verifying Application Signature...
Initiating Application Install...
###########################################
# Installing Upgrade Readiness Tool (URT) #
###########################################
Checking ISE version compatibility
- Successful
Checking ISE persona
- Successful
Along with Administration, other services (MNT,PROFILER,SESSION) are enabled on this node. Installing and running URT might consume additional resources.
Do you want to proceed with installing and running URT now (y/n):y
Checking if URT is recent(<45 days old)
- Note: URT is 486 days old and its version is 1.0.0. There might be a recent URT bundle on CCO, please verify on CCO
Do you want to proceed with this version which is 486 days old (y/n):y
Proceeding with this version of URT itself
Installing URT bundle
- Successful
########################################
# Running Upgrade Readiness Tool (URT) #
########################################
This tool will perform following tasks:
1. Pre-requisite checks
2. Clone config database
3. Copy upgrade files
4. Data upgrade on cloned database
5. Time estimate for upgrade
Pre-requisite checks
====================
Disk Space sanity check
- Successful
NTP sanity
- Failed
Appliance/VM compatibility
- Successful
Trust Cert Validation
The certificate has expired.
Trust certificate with friendly name 'Default self-signed server certificate' is invalid: The certificate has expired.
The certificate has expired.
Trust certificate with friendly name 'VeriSign Class 3 Secure Server CA - G3' is invalid: The certificate has expired.
% Error: One or more trust certificates are invalid (see above), please re-import valid CA Certificate(s) before continuing. Upgrade cannot continue.
- Failed
System Cert Validation
The certificate has expired.
System certificate with friendly name 'Default self-signed saml server certificate - CN=SAML_ise.securitydemo.net' is invalid: The certificate has expired.
The certificate has expired.
System certificate with friendly name 'Default self-signed server certificate' is invalid: The certificate has expired.
% Error: One or more system certificates are invalid (see above), please update with valid system certificate(s) before continuing. Upgrade cannot continue.
/opt/CSCOcpm/upgrade/bin/isedbupgrade-functions.sh: line 101: [: -le: unary operator expected
- Failed
Invalid MDMServerNames in Authorization Policies check
- Successful
3 out of 6 pre-requisite checks passed
Some pre-requisite checks have failed. Hence exiting...
Final cleanup before exiting...
Collecting log files ...
- Encrypting logs bundle...
Please enter encryption password:
Please enter encryption password again to verify:
Encrypted URT logs(urt_logs.tar.gpg) are available in localdisk. Please reach out to Cisco TAC to debug
% Post-install step failed. Please check the logs for more details.
ise/admin#
Disk Space sanity check
- Successful
NTP sanity
- Failed
Appliance/VM compatibility
- Successful
Trust Cert Validation
The certificate has expired.
Trust certificate with friendly name 'Default self-signed server certificate' is invalid: The certificate has expired.
The certificate has expired.
Trust certificate with friendly name 'VeriSign Class 3 Secure Server CA - G3' is invalid: The certificate has expired.
% Error: One or more trust certificates are invalid (see above), please re-import valid CA Certificate(s) before continuing. Upgrade cannot continue.
- Failed
System Cert Validation
The certificate has expired.
System certificate with friendly name 'Default self-signed saml server certificate - CN=SAML_ise.securitydemo.net' is invalid: The certificate has expired.
The certificate has expired.
System certificate with friendly name 'Default self-signed server certificate' is invalid: The certificate has expired.
% Error: One or more system certificates are invalid (see above), please update with valid system certificate(s) before continuing. Upgrade cannot continue.
/opt/CSCOcpm/upgrade/bin/isedbupgrade-functions.sh: line 101: [: -le: unary operator expected
- Failed
Invalid MDMServerNames in Authorization Policies check
- Successful
3 out of 6 pre-requisite checks passed
Some pre-requisite checks have failed. Hence exiting...
Final cleanup before exiting...
Collecting log files ...
- Encrypting logs bundle...
Please enter encryption password:
Please enter encryption password again to verify:
Encrypted URT logs(urt_logs.tar.gpg) are available in localdisk. Please reach out to Cisco TAC to debug
% Post-install step failed. Please check the logs for more details.
ise/admin#
I have 2 small problems
So after running the upgrade readiness tool it warned me that I have two issues. Firstly NTP is not working and secondly my self signed certificate has expired.
Both these issues will stop me upgrading this lab ISE node so need to be addressed.
Change NTP server
To change the NTP server in ISE head over to
Administration / System / Settings / System Time
Enter a valid NTP server in here and click save
As this is a lab node I have just entered pool.ntp.org, however in production ensure this is pointing to a valid NTP server.
If this is a production node, this should already be setup!
You can verify the NTP status from the CLI as below
ise/admin# sh ntp
Configured NTP Servers:
80.86.38.193
81.128.218.110
synchronised to NTP server (81.128.218.110) at stratum 2
time correct to within 16 ms
polling server every 128 s
remote refid st t when poll reach delay offset jitter
==============================================================================
127.127.1.0 .LOCL. 10 l 54m 64 0 0.000 0.000 0.000
+80.86.38.193 .GPS. 1 u 10 128 377 14.231 4.706 1.505
*81.128.218.110 .GPS. 1 u 17 64 377 18.403 3.744 2.710
* Current time source, + Candidate , x False ticker
Warning: Output results may conflict during periods of changing synchronization.
ise/admin#
The main things you need to pay attention to are certificates, but if these have expired they should be causing you issues now. Also disk space is another big problem. So I think we are good to proceed with the upgrade.
Deal with expired certificates
I had a self signed certificate that had expired, this will stop the upgrade process continuing. So I created a new self signed certificate and now the upgrade can continue.
Run the URT again
I now ran the upgrade readiness tool again and it completed and gave me an estimate that it will take 77 minutes to perform the upgrade.
This time will vary depending on the size of your database but it gives you a confidence check that the real upgrade will complete successfully.
########################################
# Running Upgrade Readiness Tool (URT) #
########################################
This tool will perform following tasks:
1. Pre-requisite checks
2. Clone config database
3. Copy upgrade files
4. Data upgrade on cloned database
5. Time estimate for upgrade
Pre-requisite checks
====================
Disk Space sanity check
- Successful
NTP sanity
- Successful
Appliance/VM compatibility
- Successful
Trust Cert Validation
- Successful
System Cert Validation
- Successful
Invalid MDMServerNames in Authorization Policies check
- Successful
6 out of 6 pre-requisite checks passed
Clone config database
=====================
[##--------------------------------------] 5% Validating connection to ISE data [####------------------------------------] 10% Validating available disk space [######----------------------------------] 15% Extracting base database files [##########------------------------------] 25% Cloning database [####################--------------------] 50% Exporting data from ISE database [##############################----------] 75% Importing data into cloned datab [########################################] 100% Successful
Copy upgrade files
==================
- N/A
Data upgrade on cloned database
===============================
Modifying upgrade scripts to run on cloned database
- Successful
Running schema upgrade on cloned database
- Running db sanity to check and fix if any index corruption
- Auto Upgrading Schema for UPS Model
- Upgrading Schema completed for UPS Model
- Successful
Running sanity after schema upgrade on cloned database
- Successful
Running data upgrade on cloned database
- Data upgrade step 1/20, NSFUpgradeService(2.5.0.129)... Done in 8 seconds.
- Data upgrade step 2/20, NSFUpgradeService(2.5.0.130)... Done in 4 seconds.
- Data upgrade step 3/20, NSFUpgradeService(2.5.0.168)... Done in 0 seconds.
- Data upgrade step 4/20, NSFUpgradeService(2.5.0.183)... Done in 0 seconds.
- Data upgrade step 5/20, NSFUpgradeService(2.5.0.196)... Done in 0 seconds.
- Data upgrade step 6/20, GuestAccessUpgradeService(2.5.0.199)... Done in 8 seconds.
- Data upgrade step 7/20, UPSUpgradeHandler(2.5.0.200)... Done in 5 seconds.
- Data upgrade step 8/20, LSDSettingsRegistration(2.5.0.225)... Done in 0 seconds.
- Data upgrade step 9/20, NSFUpgradeService(2.5.0.236)... Done in 0 seconds.
- Data upgrade step 10/20, CertMgmtUpgradeService(2.5.0.276)... Done in 11 seconds.
- Data upgrade step 11/20, ProfilerUpgradeService(2.5.0.288)... Done in 0 seconds.
- Data upgrade step 12/20, UPSUpgradeHandler(2.5.0.316)... Done in 2 seconds.
- Data upgrade step 13/20, UPSUpgradeHandler(2.5.0.320)... Done in 0 seconds.
- Data upgrade step 14/20, RegisterPostureTypes(2.6.0.103)... Done in 0 seconds.
- Data upgrade step 15/20, ProvisioningUpgradeService(2.6.0.103)... Done in 0 seconds.
- Data upgrade step 16/20, UPSUpgradeHandler(2.6.0.108)... Done in 0 seconds.
- Data upgrade step 17/20, UPSUpgradeHandler(2.6.0.154)... Done in 0 seconds.
- Data upgrade step 18/20, NSFUpgradeService(2.6.0.156)... Done in 0 seconds.
- Data upgrade step 19/20, ProfilerUpgradeService(2.6.0.156)... Done in 0 seconds.
- Data upgrade step 20/20, GuestAccessUpgradeService(2.6.0.156)... Done in 6 seconds.
- Successful
Running data upgrade for node specific data on cloned database
- Successful
Time estimate for upgrade
=========================
(Estimates are calculated based on size of config and mnt data only. Network latency between PAN and other nodes is not considered in calculating estimates)
Estimated time for each node (in mins):
ise(STANDALONE):77
Final cleanup before exiting...
Application successfully installed
ise/admin#
Now start the real CLI upgrade!
To upgrade Cisco ISE using the CLI enter the command
application upgrade prepare <ISE image> <repository>
ise/admin# application upgrade prepare ise-upgradebundle-2.1.x-2.4.x-to-2.6.0.156.SPA.x86_64.tar.gz FZ-FTP ?
<cr> Carriage return.
Hit return and ISE will start to upload the bundle, you should see the connection on our Filezilla server and the upload progress indicated.
You will not see any progress from within ISE
Once the upload has completed, the package will be unbundled and then checked. This should complete with the following message
Unbundling Application Package...
Verifying Application Signature...
Application upgrade preparation successful
ise/admin#
Now you can initiate the upgrade process with the command
ise/admin# application upgrade proceed
This process could take a long time (many hours) so if you are upgrading a VM I would advise you log into the console, so you can see what is going on.
In my URT test it suggested it would take 77 minutes – your mileage will vary and is mainly dependant on the size of the database.
The last production upgrade I performed took over 3 1/2 hours per node so please make sure you allow enough time for this process.
My upgrade completed 148 minutes so allowing for 44 minutes to upload the image the estimate of 77 minutes was not far off. This is more to do with the fact it’s running on an under powered lab server!
The server then goes for a final reboot, if you are logged into VMWare console you will the progress
You should be able to SSH into ISE now and if I do a show version I get this
ise/admin# sh version
Cisco Application Deployment Engine OS Release: 3.0
ADE-OS Build Version: 3.0.5.144
ADE-OS System Architecture: x86_64
Copyright (c) 2005-2019 by Cisco Systems, Inc.
All rights reserved.
Hostname: ise
% NOTICE: Identity Services Engine upgrade is in progress...
Version information of installed applications
---------------------------------------------
ise/admin#
Then finally you get the message you are wanting
ise/admin# sh version
Cisco Application Deployment Engine OS Release: 3.0
ADE-OS Build Version: 3.0.5.144
ADE-OS System Architecture: x86_64
Copyright (c) 2005-2019 by Cisco Systems, Inc.
All rights reserved.
Hostname: ise
Version information of installed applications
---------------------------------------------
Cisco Identity Services Engine
---------------------------------------------
Version : 2.6.0.156
Build Date : Tue Feb 12 00:45:06 2019
Install Date : Tue Jun 16 13:41:18 2020
ise/admin#
The upgrade is complete, now we just need to install the latest patch.
Install latest patch
Now you have upgraded your Cisco ISE to the latest suggested software version, as a best practice you should also install the latest patch.
Head back to the Cisco download site and get the latest patch level for your chosen download. In my case 2.6 patch level 6
The latest patch will normally be at the top of the downloads.
https://software.cisco.com/download/home/283801620/type/283802505/release/2.6.0
Once you have the patch downloaded head over to:
Administration > System > Maintenance > Patch Management > Install
Click on Choose file and browse to where you have downloaded the ISE 2.6 patch. Click on Install
You will see the upload progressing in your browser (I am using Chrome and it shows in the bottom left.
Verify
Once you have completed the ISE upgrade and latest patch install, you can verify the latest version with the CLI command
Also you need to perform any authentication tasks relevant to your environment to ensure ISE is operating correctly.
If you observe any issues you will have to troubleshoot or if the issue is more complex, you might have to log a call with Cisco TAC.
For the most part all the upgrades I have performed have gone very smoothly if all the preparation has been done.
I hope this short guide has helped you understand the Cisco ISE upgrade process. Whilst the best practices I have shared here will guide you through the process you should still perform this task with caution and not before you are sure you have a full backup. As, whilst in most cases the process runs smoothly, sometimes it does not and you need to be prepared for the scenario where your ISE might be down.
In the next post I will be covering ISE upgrades with multiple devices and covering which order to upgrade them in.
I will also cover the backup and restore upgrade.
Problems
If you get the error
“Failed to create upgrade preparation directory. Try cleanup first”
Something has gone wrong with the upgrade software, if you run the cleanup command you can then run the upgrade command again and you should proceed ok.
% Failed to create upgrade preparation directory. Try cleanup first.
ise/admin# application upgrade ?
<WORD> Application bundle file name (Max Size - 255)
cleanup Cleanup previous prepared bundle so as to prepare a new bundle
prepare Download and prepare application for upgrade
proceed Proceed with upgrade using local prepared bundle
ise/admin# application upgrade cleanup
Application upgrade preparation directory cleanup successful
ise/admin#
Now if you run the upgrade command again it will continue.
How to perform IOS upgrade on Cisco Switch using Ansible
»
Hi Roger,
Thanks for explaining the ISE upgrade from 2.4 to 2.6.
One thing more here after the upgrade, we can not see the live logs ( Radius & Tacacs) after the upgrade to ISE 2.6. To resolve the issue we need to uncheck the UDP section in Administration–> Logging. ( just below the option to reserve the logs in MnT, by default, it’s 1 day)
After unchecking the UDP selection setting, we can see the ogs.
Thanks Again.