Have you got 100’s or 1000’s of Cisco switches or routers to perform a software upgrade on?
Looking for a simple way to upgrade Cisco IOS remotely? Or perform the task without manually logging into each device?
Ansible is a great way to perform software upgrades on network devices.
This Ansible Playbook will upgrade your Cisco IOS switches with ease!
NOTE: The advice given here is purely for educational purposes. Please test this upgrade process out before pressing the button on 100’s of devices. Every network environment is different. Ensure you are 100% happy with the process before updating multiple Cisco switches in a production environment. I cannot be held responsible for any outages caused by Cisco switches failing to reboot!
IOS upgrade on Cisco switch using Ansible
For this example I will be using a Cisco 2960 switch. WS-C2960-24TT-L to be exact.
The upgrade steps will be different for your device so check the upgrade instructions and adjust the below to your specific task.
If you want to upgrade your Cisco IOS remotely you have 2 options.
- Copy the software to the remote switch
- Change the boot variable
- Reload the switch
- Wait patiently while it comes back online
or
- Get remote assistance to console into the switch locally
- Copy the software to the switch
- Change the boot variable
- Reload the switch
- Watch the switch reboot
Both options come with some risk, but having someone on site is a much safer option if the switch fails to reload. Or having out of band management to all your network devices has it’s benefits here.
Performing the task remotely is always a concern as switches always take longer to reboot when your not there! Also if the switch fails to reboot you have not option but to gain local console access.
Saying all that, the process to perform an IOS upgrade on a Cisco switch is basically very simple using Network Automation
The main factor is saving time / manpower.
Ansible Cisco IOS upgrade
Using Ansible to perform this task can take a lot of the manpower out of the task. But you still need to test every step to make sure it operates correctly.
Perform each step on the CLI and transfer the steps to your playbook. Then test to see it runs smoothly.
You can never be 100% sure everything will be ok so the steps below describe the process only to perform the upgrade on a 2960 switch. Whilst the steps are mostly the same for other IOS network devices I suggest you try this on a few local switches before upgrading your entire company’s switching overnight!
The Ansible playbook will be broken up into steps so you can see how each part works.
1. Verify the current image running on your device
Before you upgrade and reload all your switches, it would make sense to first check what software each switch is running and then eliminate all the compliant ones from the upgrade process.
2. Take a backup of the running config
This step in the Ansible playbook performs a show running-config and saves the live config of the switch before you reload it. There could be a case where changes have been made and not saved, which you will lose when the switch reboots.
3. Copy the software image to network device
Using SCP the Ansible playbook will copy the required upgrade image to the switch, ready for the upgrade.
Note: This playbook does not check to see if there is available space on the remote device. We will assume for this demonstration that there is sufficient space to take the new image.
4. Change the boot variable
It varies from platform to platform but in most cases you will have to change the boot variable to tell your network device to boot from the new image.
In most cases you will have already performed this operation yourself manually so will know the steps, we are just using Ansible here to save on the manual labour.
5. Reload the switch
All the steps so far have been non service impacting, we now come to reboot the switch remotely!
You just have to wait patiently while the switch reboots and to face on of 3 scenarios?
- The switch reboots and has the new image running – Hooray!
- The switch reboots and is still running the same image – What?
- The switch fails to reboot and you cannot log in again – ARGHG!
6. Verify the current image
Once the switch has successfully rebooted, we just need to verify that the correct image is running.
Cisco IOS Upgrade Automation Playbook step by step
This is the entire playbook we will use to perform the IOS upgrade.
The entire playbook can be found here:
https://github.com/rogerperkin/network-programmability/blob/master/SCRIPTS/Ansible/ios-upgrade.yml
The image that is currently running on this switch is 15.0(2)SE10a
Checking the Cisco website I can see the latest suggested version is 12.2.55 so we will be downgrading this switch.
I first need to download this image and put it in a folder on my Ansible host.
In my case /images
The first step in the playbook is to check the current running image on the switch.
Breakdown of Ansible Playbook to upgrade Cisco IOS
This is a breakdown of all the elements of the playbook.
Check current IOS version using Ansible
--- # Ansible Playbook to upgrade Cisco IOS - name: Upgrade CISCO IOS hosts: SWITCHES vars: upgrade_ios_version: 12.2(55)SE12 tasks: - name: CHECK CURRENT VERSION ios_facts: - debug: msg: - "Current version is {{ ansible_net_version }}" - "Upgrade image is 12.2.55-SE12" - debug: msg: - "Image is not compliant and will be upgraded" when: ansible_net_version != upgrade_ios_version
When we run this playbook we see this output
roger@ubuntu:~/network-programmability/SCRIPTS/Ansible$ ansible-playbook ios-upgrade.yml PLAY [Upgrade CISCO IOS] *************************************************************************************************************** TASK [CHECK CURRENT VERSION] *********************************************************************************************************** [WARNING]: default value for `gather_subset` will be changed to `min` from `!config` v2.11 onwards ok: [2960] TASK [debug] *************************************************************************************************************************** ok: [2960] => { "msg": [ "Current version is 15.0(2)SE10a", "Upgrade image is 12.2.55-SE12" ] } TASK [debug] *************************************************************************************************************************** ok: [2960] => { "msg": [ "Image is not compliant and will be upgraded" ] } PLAY RECAP ***************************************************************************************************************************** 2960 : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 roger@ubuntu:~/network-programmability/SCRIPTS/Ansible$
You can see that the version of the switch has been identified as 15.0(2)1SE10a – we can use this value later in the playbook.
Using the When statement I also issue a message that says image is not compliant.
If the running image was the same as the upgrade image we would skip the upgrade process. This will be shown later in the post.
You will also notice a warning, this is telling us that a default value for the ios_facts module will be changing in V2.11 onwards. This can be ignored. If you do not want to see warning messages (I personally like to see them) but for demonstration purposes we can tidy up the output by putting this line in your ansible.cfg file.
# Silence Action Warnings action_warnings = False
From now on we will not see any warning messages. This is personal preference and you don’t really see many warning messages in Ansible, but when they pop up it sometimes spoils the clean playbook run.
Particularly when doing a demo for a customer they will always ask
“What is that warning message?”
So we can now use ansible_net_version whenever we want to get the current image of our network device
Backup the running config
This is not a required step for an IOS upgrade but it is certainly a best practice!
Firstly I am going to take a backup of the running-config.
Then as another best practice I am going to get Ansible to save the config.
This will ensure that if any changes have been made to the switch and not saved that they will be retained on a reboot. You will also have a copy of the latest config to refer to in case of any issues.
## Create backup folder for today
- hosts: localhost
tasks:
- name: Get ansible date/time facts
setup:
filter: "ansible_date_time"
gather_subset: "!all"
- name: Store DTG as fact
set_fact:
DTG: "{{ ansible_date_time.date }}"
- name: Create Directory {{hostvars.localhost.DTG}}
file:
path: ~/network-programmability/backups/{{hostvars.localhost.DTG}}
state: directory
run_once: true
The first part of this backup section of the playbook creates a folder for today in /backups
Next we take a backup of the running-config
## Backup Running Config
- hosts: SWITCHES
tasks:
- name: Backup Running Config
ios_command:
commands: show run
register: config
- name: Save output to ~/network-programmability/backups/
copy:
content: "{{config.stdout[0]}}"
dest: "~/network-programmability/backups/{{hostvars.localhost.DTG}}/{{ inventory_hostname }}-{{hostvars.localhost.DTG}}-config.txt"
I now have a backup of the running-config
/backups/2960-2020-06-03-config.txt
Finally I am going to save the running-config
## SAVE the Running Config
- name: Save running config
ios_config:
save_when: always
So we have now verified the image is not compliant, backed up and saved the running config. The next step is to copy the new image to the switch.
Copy IOS image to switch using Ansible
We now need to get the IOS software image we have in /images to the flash of our switch. For this I am going to use the Ansible net_put module.
## Copy software to target device
- name: Copy Image // This could take up to 4 minutes
net_put:
src: "~/network-programmability/images/c2960-lanbasek9-mz.122-55.SE12.bin"
dest: "flash:/c2960-lanbasek9-mz.122-55.SE12.bin"
vars:
ansible_command_timeout: 600
This is very straightforward, you simply tell the net_put module the source and destination of what you want to copy. By default it will use SCP. So you will need to make sure you have SCP enabled on your remote switch.
This could be performed by the playbook at an earlier stage or as a pre-requisite task.
The command to enable SCP on a Cisco Switch is
ip scp server enable
You can also see I have added an ansible_command_timeout of 600 seconds. The reason for this is the default ansible_command_timeout is 30 seconds. This is not enough time to copy a Cisco image so the task will fail. Increasing it to seconds is enough time to copy the image.
Once the image starts copying if I login to the switch you can see the image file has started to be uploaded but sill has a way to go.
SW1#dir flash:
Directory of flash:/
2 -rwx 2686 Mar 1 1993 04:58:04 +00:00 config.text
3 -rwx 5144 Mar 1 1993 04:58:04 +00:00 multiple-fs
4 -rwx 616 Mar 1 1993 12:25:15 +00:00 vlan.dat
5 -rwx 11831953 Mar 1 1993 03:05:51 +00:00 c2960-lanbasek9-mz.150-2.SE10a.bin
6 -rwx 1920 Mar 1 1993 04:58:04 +00:00 private-config.text
7 -rwx 1208320 Mar 1 1993 04:58:33 +00:00 c2960-lanbasek9-mz.122-55.SE12.bin
32514048 bytes total (19460608 bytes free)
SW1#
Once the image has copied the task completes and we can verify it is now in the flash on our switch.
SW1#dir flash:
Directory of flash:/
2 -rwx 9827106 Mar 1 1993 04:10:52 +00:00 c2960-lanbasek9-mz.122-55.SE12.bin
3 -rwx 1920 Mar 1 1993 03:01:37 +00:00 private-config.text
4 -rwx 616 Mar 1 1993 12:25:15 +00:00 vlan.dat
5 -rwx 11831953 Mar 1 1993 03:05:51 +00:00 c2960-lanbasek9-mz.150-2.SE10a.bin
6 -rwx 2686 Mar 1 1993 03:01:37 +00:00 config.text
7 -rwx 5144 Mar 1 1993 04:06:24 +00:00 multiple-fs
32514048 bytes total (10841600 bytes free)
SW1#
There are two more steps to complete this upgrade and that is to change the boot variable so the switch will use the new image and then reload the switch.
Change the boot variable
You need to check the configuration guide for your particular platform but for a Cisco 2960 switch to change the boot variable you need to enter the following command into global configuration.
boot system flash:<cisco-ios-image>
So to achieve this with Ansible we are going to use the ios_config module
The playbook section looks like this
## Change the Boot Variable to the new image
- name: Change Boot Variable to new image
ios_config:
commands:
- "boot system flash:c2960-lanbasek9-mz.122-55.SE12.bin"
save_when: always
If we had configured this on the CLI of the switch it would prompt you to save the configuration before you reloaded (as below)
SW1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
SW1(config)#boot system flash:c2960-lanbasek9-mz.122-55.SE12.bin
SW1(config)#end
SW1#reload
System configuration has been modified. Save? [yes/no]:
So in our playbook I have to add one more line to save the configuration before we reload. save_when: always
We are now ready to reload the switch.
Reload the device
For this we will use the cli_command module as it can handle prompts and when you reload a Cisco network device from the CLI it will ask you to confirm. The Ansible cli_command module allows you to specify multiple prompts and also the response. In this case we just have the prompt as confirm and the answer as y
## Reload the device
- name: Reload the Device
cli_command:
command: reload
prompt:
- confirm
answer:
- 'y'
The switch will now reload and startup using the new image. We will have to wait a few minutes for it to come back online, so we will use wait_for module to test reachability to the device.
Test reachability to the device with wait_for
## Test Reachability to the device
- name: Wait for device to come back online
wait_for:
host: "{{ inventory_hostname }}"
port: 22
delay: 90
delegate_to: localhost
This part of the playbook basically waits for the device to be reachable again on port 22. I have also added a delay of 90 seconds before it starts trying. The playbook will not progress until this device is reachable.
Once the switch is reachable again we just need to validate that the image has been upgraded by running the ios_facts module again and comparing with the upgrade version.
Validate image upgrade has been successful
We are now going to run the ios_facts module again and compare the running image with our upgrade image and make sure they match.
I am going to finish with the assert module to ensure this tasks completes successfully, if the images to not match the playbook will fail.
## Check current image
- name: Check Image Version
ios_facts:
- debug:
msg:
- "Current version is {{ ansible_net_version }}"
- name: ASSERT THAT THE IOS VERSION IS CORRECT
vars:
upgrade_ios_version: 12.2(55)SE12
assert:
that:
- upgrade_ios_version == ansible_net_version
- debug:
msg:
- "Software Upgrade has been completed"
This playbook will either now finish if it has been successful or fail.
The full playbook can be found here –
https://github.com/rogerperkin/network-programmability/blob/master/SCRIPTS/Ansible/ios-upgrade.yml
Conclusion
Using Ansible to do Cisco IOS upgrade is one way to take care of your ios upgrade automation.
There are many other ways including Python, the Cisco IOS upgrade tool and many other solutions.
The main thing is whatever tool you use you understand fully what is happening at each step.
This playbook does not:
- Take into account different switches
- Check if there is enough space on the device
- Fail if the switch already has the upgraded image
- Handle Nexus
- Handle install mode
It is purely a starting point for you to progress into Cisco IOS upgrade automation!
I hope it has been helpful to you and you now understand the process better.
If you enjoyed this post, please check out my other posts on Ansible
Please comment below if you have anything to add to this process or struggled with any section. I want to keep improving this playbook to make it better!
Adam Watson
Can you tell me exactly where you are storing the cisco switch ios software image in this example? You reference the images folder. Where exactly is that folder located?
Roger Perkin
Hi Adam,
The images folder is just a folder I created on my Ubunutu machine it’s in the Ansible folder so I just reference it by using /images when running the playbook.
It can be anywhere you want as long as you can point the playbook to the location. The Image file is then copied from that folder to the IOS device.
ravishankar
can we copy code from local desktop folder. I have been trying but getting src not found.
please help.
Roger Perkin
You just need to supply an accessible path to the image, I use a local folder. You might need to supply the full path to the source.
Arp
Hi Roger, great article.
Trying to update a csr1000v with this, but get an error:
fatal: [vm-vpn002]: FAILED! => {“changed”: false, “destination”: “bootflash:csr1000v-universalk9.16.12.04a.SPA.bin”, “msg”: “Exception received: Invalid value ‘None’ set for ssh_type option. Excpected value is either ‘libssh’ or ‘paramiko'”}
Does this ring a bell with you?
Roger Perkin
Arp,
The process to update a CSR1000v is not exactly the same as a switch, I believe there are a few gothcas, I would suggest you go through the upgrade process manually and then try to re-create the steps in the playbook.
Scott
I’m just starting with Ansible, Thank you for the tutorials.
I’m having an issue getting started. Just using your first section, on checking the IOS version, I’m getting this issue:
scotto@9715mdf-bsd[560] ~ $ ansible-playbook ver.yaml
PLAY [Upgrade CISCO IOS] ************************************************************************************************************************************************
TASK [Gathering Facts] **************************************************************************************************************************************************
fatal: [hostname]: FAILED! => {“ansible_facts”: {}, “changed”: false, “failed_modules”: {“setup”: {“failed”: true, “module_stderr”: “Shared connection to fqdn closed.\r\n”, “module_stdout”: “\r\n\r\n\r\nLine has invalid autocommand \”/bin/sh -c ‘/usr/local/bin/python ‘\”‘\”‘Line has invalid autocommand \”/bin/sh -c ‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘( umask 77 && mkdir -p \”` echo Line has invalid autocommand \”/bin/sh -c ‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”‘echo ~ && sleep 0’\”‘\”‘\”‘\”‘\”‘\”‘\”‘\”\””, “msg”: “MODULE FAILURE\nSee stdout/stderr for the exact error”, “rc”: 0}}, “msg”: “The following modules failed to execute: setup\n”}
PLAY RECAP **************************************************************************************************************************************************************
hostname : ok=0 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
Is there a configuration setting I’ve missed?
Thanks in advance!
Roger Perkin
What does your playbook look like, very hard to see where the error might be?
Ivan Levanov
Good afternoon.
I am using your playbook to update a group of our devices. Unfortunately, when copying an image to devices, a timeout of 800 seconds is only enough to download the image to only one device. Is it possible to somehow modify the playbook so that many devices can be updated at once?
Thanks for your reply!
Roger Perkin
What does your hosts: say in the playbook? You can certainly fire it to more than one switch at a time, another option is to split the playbook and copy the image to all the switches, then do some verifications then perform the upgrade
Simon WW
Hi, Nice article!
Have you by any chance, looked into the newer switch IOS XE upgrades where you have a choice of installing IOS XE in install or bundle mode?
That one is a bit trickier.
Thanks.
Roger Perkin
Hi Simon, no after this article I have not updated any more switches. I no the IOS XE method is trickier, but it’s just a case of breaking the steps down and scripting each one. Sadly I can’t offer you any knowledge in this area
Jarno
Hi how can u implement a action that deletes the old image in the flash folder
Roger Perkin
Just add a task to perform the same command as you would if you were doing it manually you just need to decide at what stage in the process you delete the image
You could add in tasks to check space etc