Friday, March 6, 2015

IOS-XR - Upgrading rommonA


The below is the procedure to upgrade rommonA of a MSC in IOS-XR. be sure that when you upgrade, you only upgrade one type of rommon then reload the location before upgrading to the other. meaning, if you decided to upgrade all rommonA, then you should reload all MSC/PLIM after upgrading rommonA then you can proceed with upgrading the other rommon.

Of course if you don't know what you're doing you might screw up the MSC and PLIM. make sure you consult Cisco if you don't know what's going on.

The only problem with upgrading rommonA is that the command is hidden, don't know why exactly. anyhow, here's how you do it

I decided to print a log of all goodies before till after the upgrade

RP/0/RP0/CPU0:CRS(admin)#show hw-module fpd location all 
Wed Mar  4 17:47:24.650 UTC

===================================== ==========================================
                                      Existing Field Programmable Devices
                                      ==========================================
                                        HW                       Current SW Upg/
Location     Card Type                Version Type Subtype Inst   Version   Dng?
============ ======================== ======= ==== ======= ==== =========== ====
0/0/SP       MSC_B                      0.9   lc   rommonA 0       2.10     No 
                                              lc   rommon  0       2.10     No 
--------------------------------------------------------------------------------
0/0/CPU0     CRS1-SIP-800               0.88  lc   fpga1   0       6.00     No 
                                              lc   rommonA 0       2.10     No 
                                              lc   rommon  0       2.10     No 
--------------------------------------------------------------------------------
0/0/1        SPA-1X10GE-L-V2            1.2   spa  fpga1   1       1.11     No 
--------------------------------------------------------------------------------
0/0/2        SPA-1X10GE-L-V2            1.2   spa  fpga1   2       1.11     No 
--------------------------------------------------------------------------------
0/4/CPU0     FP-140G                    0.6   lc   rommonA 0       2.07     Yes
                                              lc   rommon  0       2.10     No 
                                              lc   fpga1   0       0.08     No 
                                              lc   fpga2   0       0.36     No 
--------------------------------------------------------------------------------

RP/0/RP0/CPU0:CRS(admin)#upgrade hw-module fpd ? 
  all     All FPD
  bios    BIOS FPD
  cpld1   CPLD FPD #1
  cpld2   CPLD FPD #2
  cpld3   CPLD FPD #3
  cpld4   CPLD FPD #4
  cpld5   CPLD FPD #5
  cpld6   CPLD FPD #6
  fabldr  Fabric Downloader FPD
  fpga    All FPGA FPD
  fpga1   FPGA FPD
  fpga10  FPGA FPD #10
  fpga11  FPGA FPD #11
  fpga12  FPGA FPD #12
  fpga13  FPGA FPD #13
  fpga14  FPGA FPD #14
  fpga2   FPGA FPD #2
  fpga3   FPGA FPD #3
  fpga4   FPGA FPD #4
  fpga5   FPGA FPD #5
  fpga6   FPGA FPD #6
  fpga7   FPGA FPD #7
  fpga8   FPGA FPD #8
  fpga9   FPGA FPD #9
  ibmc    IBMC FPD
  rommon  Rommon FPD
  rxpod   Rx POD FPD
  txpod   Tx POD FPD
RP/0/RP0/CPU0:CRS(admin)#upgrade hw-module fpd rommonA ? 
                                                     ^
% Invalid input detected at '^' marker.

****you can see that the command doesn't auto-complete or show available commands, you'll have to type it right manually ****


RP/0/RP0/CPU0:CRS(admin)#upgrade hw-module fpd rommonA location 0/4/CPU0
Wed Mar  4 17:48:34.023 UTC

***** UPGRADE WARNING MESSAGE: *****
  *  This upgrade operation has a maximum timout of 160 minutes.  *
  *  If you are executing the cmd for one specific location and  *
  *  card in that location reloads or goes down for some reason  *
  *  you can press CTRL-C to get back the RP's prompt.           *
  *  If you are executing the cmd for _all_ locations and a node *
  *  reloads or is down please allow other nodes to finish the   *
  *  upgrade process before pressing CTRL-C.                     *

% RELOAD REMINDER:
  - The upgrade operation of the target module will not interrupt its normal
    operation. However, for the changes to take effect, the target module
    will need to be manually reloaded after the upgrade operation. This can
    be accomplished with the use of "hw-module <target> reload" command.
  - If automatic reload operation is desired after the upgrade, please use
    the "reload" option at the end of the upgrade command.
  - The output of "show hw-module fpd location" command will not display
    correct version information after the upgrade if the target module is
    not reloaded.
NOTE: Chassis CLI will not be accessible while upgrade is in progress.
Continue? [confirm]
RP/0/RP0/CPU0:Mar  4 17:48:52.391 : upgrade_fpd_cli[65858]: %PLATFORM-UPGRADE_FPD-6-STATUS : FPD upgrade started. 
RP/0/RP0/CPU0:Mar  4 17:48:52.402 : upgrade_fpd_cli[65858]: %PLATFORM-UPGRADE_FPD-6-STATUS : FPD upgrade started. 


FPD upgrade in progress on some hardware, reload/configuration change 
on those is not recommended as it might cause HW programming failure 
and result in RMA of the hardware.


Starting the upgrade/download of following FPD:
=========== ==== ======= ======= =========== =========
                                   Current    Upg/Dng
Location    Type Subtype Upg/Dng   Version    Version
=========== ==== ======= ======= =========== =========
0/4/CPU0    lc   rommonA upg         2.07        2.10   
------------------------------------------------------
RP/0/RP0/CPU0:Mar  4 17:48:52.458 : upgrade_fpd_cli[65858]: %PLATFORM-UPGRADE_FPD-6-STATUS_LOC : Upgrade is going on: FPD upgrade sent to location  node0_4_CPU0 
LC/0/4/CPU0:Mar  4 17:48:52.493 : lc_fpd_upgrade[250]: %PLATFORM-UPGRADE_FPD-6-START : Starting to upgrade rommonA subtype image from 2.07 to 2.10 for on this card on location 0/4/CPU0 
LC/0/4/CPU0:Mar  4 17:49:31.893 : lc_fpd_upgrade[250]: %PLATFORM-UPGRADE_FPD-6-PASSED : Successfully upgrade rommonA subtype image for on this card on location 0/4/CPU0 
Successfully upgraded   rommonA for              FP-140G on location    0/4/CPU0 from  2.07 to  2.10
RP/0/RP0/CPU0:Mar  4 17:49:31.896 : upgrade_fpd_cli[65858]: %PLATFORM-UPGRADE_FPD-6-STATUS_LOC : Upgrade is going on: FPD upgrade completed for location  node0_4_CPU0 
RP/0/RP0/CPU0:Mar  4 17:49:34.469 : upgrade_fpd_cli[65858]: %PLATFORM-UPGRADE_FPD-6-STATUS : FPD upgrade completed. 


FPD upgrade has ended.

#####AFTER THE upgrade is done, you should reload the location#####

RP/0/RP0/CPU0:CRS(admin)#reload location 0/4/CPU0
Wed Mar  4 17:50:03.802 UTC

Preparing system for backup. This may take a few minutes especially for large configurations.
[Done]
Proceed with reload? [confirm]
RP/0/RP0/CPU0:Mar  4 17:50:06.377 : shelfmgrv2[392]: %PLATFORM-SHELFMGRV2-4-BRINGDOWN_REQUEST_RCVD : process reload running on node0_RP0_CPU0 requested node node0_4_CPU0 to be PROGRAM RELOAD. reason: [User initiated reload] 
RP/0/RP0/CPU0:CRS(admin)#LC/0/4/CPU0:Mar  4 17:50:11.384 : ingressq[237]: %FABRIC-INGRESSQ-6-LINK_DOWN : Ingressq: Link 0 of Asic Instance 0 has been administratively shut down. 
LC/0/4/CPU0:Mar  4 17:50:11.399 : ingressq[237]: %PLATFORM-CIH-1-ASIC_ERROR_HARD_RESET_START : ingressq[0]: HARD_RESET needed 0x130a000b 
LC/0/4/CPU0:Mar  4 17:50:11.399 : ingressq[237]: %FABRIC-INGRESSQ_DLL-3-RESET : HARD RESET Resetting IQM ASIC Device with halt status 10 due to int_lns_all_link_down 
LC/0/4/CPU0:Mar  4 17:50:11.415 : ingressq[237]: %PLATFORM-CIH-2-ASIC_ERROR_HARD_RESET : ingressq[0]: A bp-err error has occurred causing  halt. 0x130a000b   
LC/0/4/CPU0:Mar  4 17:50:11.436 : fabricq_mgr[181]: %PLATFORM-CIH-1-ASIC_ERROR_RETRAIN_LINK : fabricq[1]: A link-err error has occurred causing  packet drop persistent. 0x111d0014   
RP/0/RP0/CPU0:Mar  4 17:50:16.381 : invmgr[259]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/4/CPU0, state: BRINGDOWN 
LC/0/4/CPU0:Mar  4 17:50:16.424 : i2c_server[58]: %PLATFORM-I2C-7-HANDLE_PARK_MUX : Putting the I2C mux in default state in preparation for card reload: started 
LC/0/4/CPU0:Mar  4 17:50:16.424 : i2c_server[58]: %PLATFORM-I2C-7-HANDLE_PARK_MUX : Putting the I2C mux in default state in preparation for card reload: finished 
LC/0/4/CPU0:Mar  4 17:50:16.424 : i2c_server[58]: %PLATFORM-I2C-7-HANDLE_PARK_MUX : Putting the I2C mux in default state in preparation for card reload: replied back after 
RP/0/RP0/CPU0:Mar  4 17:50:31.398 : shelfmgrv2[392]: %PLATFORM-MBIMGR-7-IMAGE_VALIDATED : Remote location 0/4/CPU0: MBI tftp:/disk0/hfr-os-mbi-5.1.3/lc/0x500064/mbihfr-lc-x86e.vm validated 
RP/0/RP0/CPU0:Mar  4 17:51:36.805 : invmgr[259]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/4/CPU0, state: IOS XR RUN 
LC/0/4/CPU0:Mar  4 17:50:49.794 : i2c_server[58]: %PLATFORM-I2C-7-BOARD_POWERUP : Sending DCDC Enable to shelfmgr. 
LC/0/4/CPU0:Mar  4 17:51:08.928 : i2c_server[58]: %PLATFORM-I2C-7-BOARD_POWERUP : Received DCDC Enable from shelfmgr. 
LC/0/4/CPU0:Mar  4 17:51:12.222 : i2c_server[58]: %PLATFORM-I2C-7-BOARD_POWERUP : MotherBoard Powered Up. 
LC/0/4/CPU0:Mar  4 17:51:12.344 : pciesvr[69]: %PLATFORM-PCIE-7-INIT_INFO : PCIe server bus enumeration started. 
LC/0/4/CPU0:Mar  4 17:51:12.360 : pciesvr[69]: %PLATFORM-PCIE-7-INIT_INFO : PCIe server initialization completed. 
LC/0/4/CPU0:Mar  4 17:51:12.399 : init[65540]: %OS-INIT-7-MBI_STARTED : total time 25.463 seconds 
LC/0/4/CPU0:Mar  4 17:51:34.413 : sysmgr[82]: %OS-SYSMGR-5-NOTICE : Card is COLD started  
LC/0/4/CPU0:Mar  4 17:51:34.770 : init[65540]: %OS-INIT-7-INSTALL_READY : total time 47.834 seconds 
LC/0/4/CPU0:Mar  4 17:51:35.038 : sysmgr[375]: %OS-SYSMGR-6-INFO : Backup system manager is ready  
LC/0/4/CPU0:Mar  4 17:51:56.523 : sysmgr[82]: %OS-SYSMGR-7-DEBUG : sysmgr_admin_plane_check:SYSMGR_PLANE_ADMIN Notification sent.   
LC/0/4/CPU0:Mar  4 17:52:07.201 : rsi_agent[315]: %OS-RSI_AGENT-6-CARD_ROLE_CHANGE : Based on the card configuration/type, the AFI IPv4 role of the card has changed from Invalid to Customer Facing 
LC/0/4/CPU0:Mar  4 17:52:07.207 : rsi_agent[315]: %OS-RSI_AGENT-6-CARD_ROLE_CHANGE : Based on the card configuration/type, the AFI IPv6 role of the card has changed from Invalid to Not Interested 
LC/0/4/CPU0:Mar  4 17:52:27.581 : plim_xge_otn_flex[295]: %L2-PLIM-6-XFP_OIR : Optic Interface Module inserted for port 4 
LC/0/4/CPU0:Mar  4 17:52:31.822 : plim_xge_otn_flex[295]: %L2-PLIM-6-XFP_OIR : Optic Interface Module inserted for port 0 
LC/0/4/CPU0:Mar  4 17:52:35.967 : plim_xge_otn_flex[295]: %L2-PLIM-6-XFP_OIR : Optic Interface Module inserted for port 1 
LC/0/4/CPU0:Mar  4 17:52:40.113 : plim_xge_otn_flex[295]: %L2-PLIM-6-XFP_OIR : Optic Interface Module inserted for port 2 
LC/0/4/CPU0:Mar  4 17:52:44.258 : plim_xge_otn_flex[295]: %L2-PLIM-6-XFP_OIR : Optic Interface Module inserted for port 3 
LC/0/4/CPU0:Mar  4 17:52:48.404 : plim_xge_otn_flex[295]: %L2-PLIM-6-XFP_OIR : Optic Interface Module inserted for port 5 
LC/0/4/CPU0:Mar  4 17:52:53.172 : egressq[160]: %PLATFORM-CIH-1-ASIC_ERROR : egressq[0]: A link-err error has occurred causing  packet drop transient. 0x151d0022   
LC/0/4/CPU0:Mar  4 17:53:10.330 : ifmgr[197]: %PKT_INFRA-LINK-5-CHANGED : Interface TenGigE0/4/0/5, changed state to Administratively Down 
LC/0/4/CPU0:Mar  4 17:53:10.333 : ifmgr[197]: %PKT_INFRA-LINK-5-CHANGED : Interface TenGigE0/4/0/4, changed state to Administratively Down 
LC/0/4/CPU0:Mar  4 17:53:10.333 : ifmgr[197]: %PKT_INFRA-LINK-5-CHANGED : Interface TenGigE0/4/0/3, changed state to Administratively Down 
LC/0/4/CPU0:Mar  4 17:53:10.333 : ifmgr[197]: %PKT_INFRA-LINK-5-CHANGED : Interface TenGigE0/4/0/2, changed state to Administratively Down 
LC/0/4/CPU0:Mar  4 17:53:10.333 : ifmgr[197]: %PKT_INFRA-LINK-5-CHANGED : Interface TenGigE0/4/0/1, changed state to Administratively Down 
LC/0/4/CPU0:Mar  4 17:53:10.333 : ifmgr[197]: %PKT_INFRA-LINK-5-CHANGED : Interface TenGigE0/4/0/0, changed state to Administratively Down 
LC/0/4/CPU0:Mar  4 17:53:10.335 : cfgmgr-lc[142]: %MGBL-CONFIG-6-OIR_RESTORE : Configuration for node '0/4/0' has been restored. 
LC/0/4/CPU0:Mar  4 17:53:23.309 : 6-10ge-wlo-flex[245]: %L2-SPA-5-STATE_CHANGE : SPA in bay 2 type 10xGE SPA Initing   
RP/0/RP0/CPU0:Mar  4 17:53:30.057 : invmgr[259]: %PLATFORM-INV-6-NODE_STATE_CHANGE : Node: 0/4/2, state: OK 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/4/2/9, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/4/2/9, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/4/2/8, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/4/2/8, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/4/2/7, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/4/2/7, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/4/2/6, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/4/2/6, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/4/2/5, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/4/2/5, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/4/2/4, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/4/2/4, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/4/2/3, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/4/2/3, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/4/2/2, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/4/2/2, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/4/2/1, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/4/2/1, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINK-3-UPDOWN : Interface GigabitEthernet0/4/2/0, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.099 : ifmgr[197]: %PKT_INFRA-LINEPROTO-5-UPDOWN : Line protocol on Interface GigabitEthernet0/4/2/0, changed state to Down 
LC/0/4/CPU0:Mar  4 17:53:31.101 : cfgmgr-lc[142]: %MGBL-CONFIG-6-OIR_RESTORE : Configuration for node '0/4/2' has been restored. 
LC/0/4/CPU0:Mar  4 17:53:31.750 : spa_ge_v2(2)[329]: %L2-OPTICS-2-TRANSCEIVER_LOC : Transceiver location is int Gige/4/2/1 
LC/0/4/CPU0:Mar  4 17:53:31.751 : spa_ge_v2(2)[329]: %L2-OPTICS-2-XCVR_CODE : xcvr code is 0x8 
LC/0/4/CPU0:Mar  4 17:53:31.751 : spa_ge_v2(2)[329]: %L2-OPTICS-2-VID : VID is  
LC/0/4/CPU0:Mar  4 17:53:31.751 : spa_ge_v2(2)[329]: %L2-OPTICS-2-PID : PID is  
LC/0/4/CPU0:Mar  4 17:53:31.751 : spa_ge_v2(2)[329]: %L2-OPTICS-2-PN : PN is  
LC/0/4/CPU0:Mar  4 17:53:31.853 : spa_ge_v2(2)[329]: %PLATFORM-XCVR-6-ENABLE : Transceiver(GigabitEthernet :rack:0/slot:4/bay:2/port:1) ENABLED 
LC/0/4/CPU0:Mar  4 17:53:32.024 : rsi_agent[315]: %OS-RSI_AGENT-6-CARD_ROLE_CHANGE : Based on the card configuration/type, the AFI IPv4 role of the card has changed from Customer Facing to Core Facing 
LC/0/4/CPU0:Mar  4 17:53:32.027 : rsi_agent[315]: %OS-RSI_AGENT-6-CARD_ROLE_CHANGE : Based on the card configuration/type, the AFI IPv6 role of the card has changed from Not Interested to Core Facing 
RP/0/RP0/CPU0:Mar  4 17:53:35.080 : invmgr[259]: %PLATFORM-INV-6-CARD_OIRIN : OIR: Card 0/4/2 inserted 

***The PLIM is up now and all SPAs and ports are detected. Now let's check the SW ver of the MSC***

RP/0/RP0/CPU0:CRS(admin)#show hw-module fpd location all 
Wed Mar  4 17:54:05.002 UTC

===================================== ==========================================
                                      Existing Field Programmable Devices
                                      ==========================================
                                        HW                       Current SW Upg/
Location     Card Type                Version Type Subtype Inst   Version   Dng?
============ ======================== ======= ==== ======= ==== =========== ====
0/0/SP       MSC_B                      0.9   lc   rommonA 0       2.10     No 
                                              lc   rommon  0       2.10     No 
--------------------------------------------------------------------------------
0/0/CPU0     CRS1-SIP-800               0.88  lc   fpga1   0       6.00     No 
                                              lc   rommonA 0       2.10     No 
                                              lc   rommon  0       2.10     No 
--------------------------------------------------------------------------------
0/0/1        SPA-1X10GE-L-V2            1.2   spa  fpga1   1       1.11     No 
--------------------------------------------------------------------------------
0/0/2        SPA-1X10GE-L-V2            1.2   spa  fpga1   2       1.11     No 
--------------------------------------------------------------------------------
0/4/CPU0     FP-140G                    0.6   lc   rommonA 0       2.10     No 
                                              lc   rommon  0       2.10     No 
                                              lc   fpga1   0       0.08     No 
                                              lc   fpga2   0       0.36     No 
--------------------------------------------------------------------------------


now you're good to go.