Two Catalyst WS-C4503-E switches are set up as VSS with Sup8-E on IOS-XE 03.06.02E.
The goal is to upgrade the VSS with ISSU to version 03.06.03E without network disruption.
Current 03.06.02E is running in "Install mode" (not "Bundle mode"):
boot-start-marker
boot system bootflash:packages.conf
boot-end-marker
Configuration register is 0x102 on both switches.
New IOS-XE 03.06.03E was copied to the bootflashes of both switches:
cat4k#dir bootflash: | i 03.06.03
cat4500es8-universalk9.SPA.03.06.03.E.152-2.E3.bin
tll-lauteri-cat4k#dir slavebootflash: | i 03.06.03
cat4500es8-universalk9.SPA.03.06.03.E.152-2.E3.bin
The VSS active switch was using slot 1 for the Active Supervisor Engine and the VSS standby was using slot 11 for the Standby Supervisor Engine:
cat4k#sh issu state det
Slot = 1
RP State = Active
ISSU State = Init
Operating Mode = Stateful Switchover
Current Image = bootflash:packages.conf
Pre-ISSU (Original) Image = N/A
Post-ISSU (Targeted) Image = N/A
Slot = 11
RP State = Standby
ISSU State = Init
Operating Mode = Stateful Switchover
Current Image = bootflash:packages.conf
Pre-ISSU (Original) Image = N/A
Post-ISSU (Targeted) Image = N/A
I chose to use the 4-step ISSU procedure for the upgrade. For reference and guide I used pages:
http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst4500/XE3-6-0E/15-22E/configuration/guide/xe-360-config/issu_XE.html#pgfId-1091552
http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst4500/15-1-2/XE_340/configuration/guide/config/vss.html#wp1329888
I started with the first step:
cat4k#issu loadversion 1 bootflash:cat4500es8-universalk9.SPA.03.06.03.E.152-2.E3.bin 11 slavebootflash:cat4500es8-universalk9.SPA.03.06.03.E.152-2.E3.bin
% issu loadversion executed successfully, Standby is being reloaded
After that the following ISSU state could be read:
cat4k#show issu state detail
Slot = 1
RP State = Active
ISSU State = Load Version
Operating Mode = not reached
Current Image = bootflash:packages.conf
Pre-ISSU (Original) Image = bootflash:packages.conf
Post-ISSU (Targeted) Image = bootflash:cat4500es8-universalk9.SPA.03.06.03.E.152-2.E3.bin
Standby information is not available because it is in 'DISABLED' state
The standby VSS switch was installed the new 03.06.03E software and rebooted. 5 minutes later the standby switch booted the new IOS and this is what I saw in the active switch's terminal:
Dec 1 21:13:59.500: %C4K_IOSINTF-5-LMPHWSESSIONSTATE: Lmp HW session UP on slot 1 port 1.
Dec 1 21:13:59.535: %C4K_IOSINTF-5-LMPHWSESSIONSTATE: Lmp HW session UP on slot 1 port 2.
Dec 1 21:14:15.505: %VSLP-5-VSL_UP: Ready for control traffic
Dec 1 21:14:21.473: %C4K_REDUNDANCY-6-DUPLEX_MODE: The peer Supervisor has been detected
Dec 1 21:15:01.288: %C4K_REDUNDANCY-2-IOS_VERSION_CHECK_FAIL: IOS version mismatch. Active supervisor version is 15.2(2)E2 (cat4500es8-UNIVERSALK9-M). Standby supervisor version is 15.2(2)E3 (cat4500es8-UNIVERSALK9-M). Redundancy feature may not work as expected. (This should be normal behavior during ISSU upgrade on VSS)
Dec 1 21:15:01.289: %C4K_REDUNDANCY-6-MODE: ACTIVE supervisor initializing for rpr mode
Dec 1 21:15:01.495: %C4K_REDUNDANCY-3-COMMUNICATION: Communication with the peer Supervisor has been established
Dec 1 21:15:06.168: %VSLP-2-VSL_DOWN: All VSL links went down while switch is in ACTIVE role
Dec 1 21:15:06.532: %C4K_REDUNDANCY-3-COMMUNICATION: Communication with the peer Supervisor has been lost
Dec 1 21:15:06.549: %C4K_REDUNDANCY-3-SIMPLEX_MODE: The peer Supervisor has been lost
And the standby switch was sent to reboot and the same loop happened for 3 times in a row. Then the standby switch booted into rommon and needed serial console port access.
The logs from the standby switch from serial-console were:
*Dec 1 20:52:53.973: %C4K_REDUNDANCY-2-IOS_VERSION_CHECK_FAIL: STANDBY:IOS version mismatch. Active supervisor version is 15.2(2)E2 (cat4500es8-UNIVERSALK9-M). Standby supervisor version is 15.2(2)E3 (cat4500es8-UNIVERSALK9-M). Redundancy feature may not work as expected.
*Dec 1 20:52:53.973: %C4K_REDUNDANCY-2-NON_SYMMETRICAL_REDUNDANT_SYSTEM: STANDBY:STANDBY supervisor will operate in fallback redundancy mode rpr.
*Dec 1 20:52:57.014: %C4K_REDUNDANCY-3-COMMUNICATION: STANDBY:Communication with the peer Supervisor has been established
*Dec 1 20:52:57.937: %C4K_REDUNDANCY-2-VS_REBOOT_ON_RPR_FALLBACK: STANDBY:Supervisor in virtual-switch configuration cannot operate in redundancy mode RPR, will be rese
*Dec 1 20:52:58.035: %RF-5-RF_RELOAD: STANDBY:Self Reload. Reason: Virtual-switch fallback to RPR
*Dec 1 20:52:58.083: %SYS-5-RELOAD: STANDBY:Reload requested by Platform redundancy manager. Reload Reason: Virtual-switch fallback to RPR.
<Tue Dec 1 20:52:58 2015> Message from sysmgr: Reason Code:[3] Reset Reason:Reset/Reload requested by [console]. [Reload command]
The interesting part is that after the first ISSU command the auto-installer(?) extracted the new cat4500es8-universalk9.SPA.03.06.03.E.152-2.E3.bin into different .pkg packages and packages.conf (while renaming the old packages.conf to packages.conf.00-) and attempted to boot the new IOS also in "Install mode" from the new packages.conf.
With VSS down I could not proceed with the next ISSU steps and restored the old state with old software. ISSU upgrade failed.
No comments:
Post a Comment