Hotswapping management modules

Management module switchover

Events that cause a switchover

There are a number of events that can cause the active management module to switchover to the standby management module when management module redundancy is enabled:

  • The active management module crashes

  • The standby management module does not receive a heartbeat from the active management module

  • The redundancy switchover command is executed

  • The MM Reset button on the active management module is pressed

  • The MM Shutdown button on the active management module is pressed

  • The boot or boot active command is executed

  • The reload command is executed

  • There is a hardware failure on the active management module

In all of these cases, the standby management module takes control and performs the actual switchover. The reason for the switchover is entered in log messages on the newly active management module and to any configured Syslog servers.

What happens when switchover occurs

When a switchover occurs, the features that support nonstop switching continue to operate in an uninterrupted manner. See Nonstop switching features for a list of the supported features.

The features that do not support nonstop switching perform as if the switch had just finished booting; however, no actual boot time occurs.


[NOTE: ]

NOTE: When meshing configuration changes are made on a redundant management system, you must execute write mem and then the boot system command to boot both management modules for the changes to be activated.

Meshing is not supported by nonstop switching.



[NOTE: ]

NOTE: If the switch is a querier and a failover occurs, the querier continues to be the same on the standby management module; no new querier election process occurs on the standby management module.


When switchover will not occur

There are some events for which a switchover is not triggered:

  • When a boot system command is executed

  • When the Clear button on the System Support module is pressed

  • When management module redundancy is disabled, unless there is a hardware failure and the system is rebooted.

When a management module crashes while the other management module is rebooting

If the uncommon situation occurs where the active management module (MM1) is trying to reboot and the standby management module (MM2) also crashes, the switch attempts to recover from the crash and eventually the standby management module becomes the active management module if it passes self-test. However, traffic can be disrupted for as long as five minutes before the newly active management module (MM2) has finished rebooting.

Hotswapping out the active management module

You can hotswap out the active management module and have switch operations taken over by the standby management module by following the correct shutdown procedure on the active module using the MM Shutdown button. When the MM Shutdown button is pressed, any file synchronization in progress completes before the shutdown begins, and then a graceful shutdown of that management module occurs.

When the standby module is not available

If you have disabled management module redundancy with the no redundancy management-module command, or the standby module failed selftest, the Dwn LED does not turn green to indicate it is OK to hotswap out the active management module.


[NOTE: ]

NOTE: If you remove the active management module without pressing the MM Shutdown button, any files that may have been in the process of synchronizing will not finish synchronizing to the standby module and all file transfer is aborted.


Hotswapping in a management module

If another management module is hotswapped in while there is an active management module booted up, the newly hotswapped management module becomes the standby module.

No negotiating is needed as to which module becomes the active management module, because there is already a functioning active management module. However, the following conditions must be met to determine if the hotswapped module can become a standby management module:

  • The hotswapped module must pass selftest

  • Management module redundancy is not administratively disabled (using the no redundancy management-module command.) If the active management module's config file has redundancy administratively disabled, the hotswapped management module goes into "offline" mode.

In nonstop switching mode—The active management module's files and features are synced with the standby management module. Heartbeats are sent back and forth, and the standby management module is ready to quickly take over in the event of a switchover or a failure on the active management module.

In warm-standby mode—The standby management module partially boots up and heartbeats are sent back and forth with the active management module.

Software version mismatch between active and hotswapped module

If the software version in the hotswapped module does not match the software version in the active module, the following occurs:

  1. The active module sends the primary and secondary images in flash to the hotswapped module.

  2. The module that was hotswapped in then reboots if necessary to primary or secondary flash, whichever matches (if it does not already match.)

  3. After the hotswapped management module finishes booting, it is sent the config and other critical files from the active management module.

  4. The hotswapped management module goes into standby mode and is ready to take over in case of a switchover.


[NOTE: ]

NOTE: After the boot standby command is executed, if the software versions on the active management module and the standby management module are not compatible, the standby module does not sync with the active management module. The standby module then enters warm-standby redundancy mode.


Other software version mismatch conditions

The following steps describe the behavior that may when a new software image is installed in secondary flash of the AMM and a redundancy switchover command is executed.

  1. A new software image, K.15.04.0002 containing ROM upgrade K.15.12 is installed in secondary flash of the AMM/MM1.

  2. The AMM/MM1 automatically syncs the images to the secondary flash in the SMM/MM2. Now both AMM/MM1 and SMM/MM2 have identical software and ROM in secondary flash.

  3. The SMM/MM2 is booted from secondary. It boots into the new K.15.04.0002 software version. The new ROM is applied and the SMM/ MM2 reboots.

  4. After the SMM/MM2 finishes rebooting, it reconnects to the AMM/MM1 and prepares to take the standby role by rebooting.

  5. However, the AMM/MM1 is running software version K.15.03.0008 in its primary flash, and the SMM/MM2 is running software version K.15.04.0002 in its secondary flash, so the SMM/MM2 pauses its reboot because of the software mismatch.

  6. If a redundancy switchover command is executed, the AMM/MM1 will give control to the SMM/MM2, which can then finish booting and become the new AMM/MM2. This is the warm-start behavior.

  7. The SMM/MM1 (former AMM/MM1) reboots, but unless the reboot is executed from secondary flash, it reboots into primary flash, which contains the older software version K.15.03.0008 with no ROM upgrade.

  8. If the SMM/MM1 is forced to boot from secondary before executing the redundancy switchover command, it will boot into the new K.15.04.0002 software and upgrade the ROM. After the reboot that occurs with the ROM upgrade, the SMM/MM1 connects to the new AMM/MM2 and takes the standby role.