I had to upgrade some Juniper EX4200 switches this weekend. All went good, but one of the switches didn’t come back into the Virtual Chassis…
After attaching the console cable I came to this output:
# reboot Killed Killed Waiting (max 60 seconds) for system process `vnlru' to stop...done Waiting (max 60 seconds) for system process `vnlru_mem' to stop...done Waiting (max 60 seconds) for system process `bufdaemon' to stop...done Waiting (max 60 seconds) for system process `syncer' to stop... Syncing disks, vnodes remaining...0 0 done syncing disks... All buffers synced. Uptime: 14m2s Rebooting... U-Boot 1.1.6 (Mar 28 2011 - 04:05:40) Board: EX4200-24T 4.18 EPLD: Version 10.0 (0x82) DRAM: Initializing (1024 MB) FLASH: 8 MB Firmware Version: --- 01.00.00 --- USB: scanning bus for devices... 2 USB Device(s) found scanning bus for storage devices... 1 Storage Device(s) found ELF file is 32 bit Consoles: U-Boot console FreeBSD/PowerPC U-Boot bootstrap loader, Revision 2.4 ([email protected], Mon Mar 28 01:49:54 UTC 2011) Memory: 1024MB bootsequencing is enabled bootsuccess is not set old boot slice = 2, new boot slice = 1 new boot device = disk0s1: |/-\|/-\|/-\|/-\|/-\|/-\ |/-\|/-\can't load '/kernel' |/-\|/-\can't load '/kernel.old' Press Enter to stop auto bootsequencing and to enter loader prompt. Type '?' for a list of commands, 'help' for more detailed help. loader> ? Available commands: bcachestat get disk block cache stats boot boot a file or loaded kernel autoboot boot automatically after a delay help detailed help ? list commands show show variable(s) set set a variable unset unset a variable echo echo arguments read read input from the terminal more show contents of a file nextboot set next boot device lsdev list all devices install install JUNOS include read commands from a file ls list files load load a kernel or module unload unload all modules lsmod list loaded modules export export variables to U-Boot environment save save U-Boot environment heap show heap usage --more-- <space> page down <enter> line down <q> quit reboot reboot the system loader> load ?
So the system was hanging on the bootloader. I guessed the installation failed. So I did a manual installation with a tftp server. So I hung my computer on the management ethernet interface of the switch. And configured the switch to connect to my computer like so:
loader> set ipaddr=10.36.10.101 loader> set netmask=255.255.255.0 loader> set serverip=10.36.10.100 loader> install tftp://10.36.10.100/jinstall-ex-4200-12.3R12.4-domestic-signed.tgz Speed: 1000, full duplex Package /jinstall-ex-4200-12.3R12.4-domestic-signed.tgz is signed... /kernel data=0x660be8+0x768b8 -syms=[0x4+0x62340-+0x4+0x94ad7|] Kernel entry at 0x800000c0 ... GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1996-2016, Juniper Networks, Inc. All rights reserved. Copyright (c) 1992-2006 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. JUNOS 12.3R12.4 #0: 2016-01-20 04:06:27 UTC [email protected]:/volume/build/junos/12.3/release/12.3R12.4/obj-powerpc/junos/bsd/kernels/INSTALL-EX/kernel can't re-use a leaf (all_slot_serialid)! Timecounter "decrementer" frequency 50000000 Hz quality 0 cpu0: Freescale e500v2 core revision 2.3 cpu0: HID0 80004080<EMCP,TBEN,EN_MAS7_UPDATE> real memory = 1034944512 (987 MB) avail memory = 1012666368 (965 MB) ETHERNET SOCKET BRIDGE initialising Initializing EXSERIES properties ... nexus0: <PPC e500 Nexus device> ocpbus0: <on-chip peripheral bus> on nexus0 openpic0: <OpenPIC in on-chip peripheral bus> iomem 0xfef40000-0xfef600b3 on ocpbus0 memctl0: <mpc85xx memory ECC monitor> iomem 0xfef20000-0xfef20e5b,0xfef02000-0xfef02e5b irq 32,34 on ocpbus0 i2c0: <MPC85XX OnChip i2c Controller> iomem 0xfef03000-0xfef03014 irq 59 on ocpbus0 i2c1: <MPC85XX OnChip i2c Controller> iomem 0xfef03100-0xfef03114 irq 59 on ocpbus0 uart0: <16550 or compatible> iomem 0xfef04500-0xfef0450f irq 58 on ocpbus0 uart0: console (9600,n,8,1) uart1: <16550 or compatible> iomem 0xfef04600-0xfef0460f irq 58 on ocpbus0 lbc0: <Freescale 8533 Local Bus Controller> iomem 0xfef05000-0xfef05fff,0xff000000-0xffffffff irq 22 on ocpbus0 cfi0: <AMD/Fujitsu - 8MB> iomem 0xff800000-0xffffffff on lbc0 syspld0 iomem 0xff000000-0xff00ffff on lbc0 tsec0: <eTSEC ethernet controller> iomem 0xfef24000-0xfef24fff irq 45,46,50 on ocpbus0 tsec0: hardware MAC address 40:b4:f0:b0:19:3f miibus0: <MII bus> on tsec0 e1000phy0: <Marvell 88E1112 Gigabit PHY> on miibus0 e1000phy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX-FDX, auto pcib0: <Freescale MPC8544 PCI host controller> iomem 0xfef08000-0xfef08fff,0xf0000000-0xf3ffffff on ocpbus0 pci0: <PCI bus> on pcib0 ohci0: <OHCI (generic) USB controller> mem 0xf0000000-0xf0000fff irq 22 at device 18.0 on pci0 usb0: OHCI version 1.0 usb0: <OHCI (generic) USB controller> on ohci0 usb0: USB revision 1.0 uhub0: (0x1131) OHCI root hub, class 9/0, rev 1.00/1.00, addr 1 uhub0: 2 ports with 2 removable, self powered ehci0: <Philips ISP156x USB 2.0 controller> mem 0xf0001000-0xf00010ff irq 22 at device 18.2 on pci0 usb1: EHCI version 1.0 usb1: companion controller, 2 ports each: usb0 usb1: <Philips ISP156x USB 2.0 controller> on ehci0 usb1: USB revision 2.0 uhub1: Philips EHCI root hub, class 9/0, rev 2.00/1.00, addr 1 uhub1: 2 ports with 2 removable, self powered umass0: STMicroelectronics ST72682 High Speed Mode, rev 2.00/2.10, addr 2 pcib1: <Freescale MPC8544 PCI Express host controller> iomem 0xfef0a000-0xfef0afff,0xe0000000-0xe3ffffff,0xec000000-0xec0fffff irq 42 on ocpbus0 pci1: <PCI bus> on pcib1 pcib2: <PCI-PCI bridge> at device 0.0 on pci1 pci2: <PCI bus> on pcib2 pci2: <memory> at device 0.0 (no driver attached) pcib3: <Freescale MPC8544 PCI Express host controller> iomem 0xfef0b000-0xfef0bfff,0xe8000000-0xebffffff,0xec200000-0xec2fffff irq 43 on ocpbus0 pci3: <PCI bus> on pcib3 pcib4: <PCI-PCI bridge> at device 0.0 on pci3 pci4: <PCI bus> on pcib4 pci4: <memory> at device 0.0 (no driver attached) Initializing product: 41 .. bmeb: bmeb_lib_init done 0x84b23000, addr 0x806619b4 bme0:Virtual BME driver initializing ###PCB Group initialized for udppcbgroup ###PCB Group initialized for tcppcbgroup md0: Preloaded image </isofs-install-ex> 17772544 bytes at 0x807ce2c0 da0 at umass-sim0 bus 0 target 0 lun 0 da0: <ST ST72682 2.10> Removable Direct Access SCSI-2 device da0: 40.000MB/s transfers da0: 1000MB (2048000 512 byte sectors: 64H 32S/T 1000C) Kernel thread "wkupdaemon" (pid 44) exited prematurely. Trying to mount root from cd9660:/dev/md0 1+0 records in 1+0 records out 512 bytes transferred in 0.000207 secs (2471212 bytes/sec) Media check on da0 on ex platforms Zone 05 Block 0018 Addr 141200 : Bad read Recovering Block Zone 06 Block 0476 Addr 19dc00 : Bad read Recovering Block Zone 06 Block 0498 Addr 19f200 : Bad read Recovering Block Zone 06 Block 0502 Addr 19f600 : Bad read Recovering Block Zone 06 Block 0745 Addr 1ae900 : Bad read Recovering Block Computing slice and partition sizes for /dev/da0 ... 32+0 records in 32+0 records out 16384 bytes transferred in 0.203088 secs (80674 bytes/sec) Checking s1a label...please wait FILE SYSTEM CLEAN; SKIPPING CHECKS clean, 93586 free (34 frags, 11694 blocks, 0.0% fragmentation) tunefs: soft updates remains unchanged as enabled Checking s2a label...please wait Computing layout of partitions in /dev/da0s2... Labeling /dev/da0s2: bsdlabel: write to disk label supressed - label was as follows: # /dev/da0s2: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 382720 256 unused 0 0 c: 382976 0 unused 0 0 # "raw" part, don't edit /dev/da0s2a: 186.9MB (382716 sectors) block size 16384, fragment size 2048 using 4 cylinder groups of 46.72MB, 2990 blks, 6016 inodes. with soft updates super-block backups (for fsck -b #) at: 32, 95712, 191392, 287072 Checking s3d label...please wait FILE SYSTEM CLEAN; SKIPPING CHECKS clean, 133920 free (24 frags, 16737 blocks, 0.0% fragmentation) tunefs: soft updates remains unchanged as enabled Checking s3e label...please wait Cannot find file system superblock LOOK FOR ALTERNATE SUPERBLOCKS? yes 32 is not a file system superblock 32 is not a file system superblock 128032 is not a file system superblock 192032 is not a file system superblock 128032 is not a file system superblock 192032 is not a file system superblock SEARCH FOR ALTERNATE SUPER-BLOCK FAILED. YOU MUST USE THE -b OPTION TO FSCK TO SPECIFY THE LOCATION OF AN ALTERNATE SUPER-BLOCK TO SUPPLY NEEDED INFORMATION; SEE fsck(8). tunefs: /dev/da0s3e: could not read superblock to fill out disk Checking s4d label...please wait FILE SYSTEM CLEAN; SKIPPING CHECKS clean, 31442 free (258 frags, 3898 blocks, 0.8% fragmentation) tunefs: soft updates remains unchanged as enabled mount: /dev/da0s3e : Invalid argument *** The installer exited with status 1 *** *** The installation is unsuccessful!!! *** A shell has been started. type exit<cr> to reboot: # fs chk h exit Waiting (max 60 seconds) for system process `vnlru' to stop...done Waiting (max 60 seconds) for system process `vnlru_mem' to stop...done Waiting (max 60 seconds) for system process `bufdaemon' to stop...done Waiting (max 60 seconds) for system process `syncer' to stop... Syncing disks, vnodes remaining...0 0 done syncing disks... All buffers synced. Uptime: 59s Rebooting...
So corrupt partition table (broken partition)
How to fix?
loader> set ipaddr=10.36.10.0.101 loader> set netmask=255.255.255.0 loader> set serverip=10.36.10.100 loader> install --format tftp://10.36.10.101/jinstall-ex-4200-12.3R12.4-domestic-signed.tgz
After waiting a while 🙂
Wed Jan 20 07:06:34 UTC 2016
Boot media /dev/da0 has dual root support
** /dev/da0s2a
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 27783 free (15 frags, 3471 blocks, 0.0% fragmentation)
Amnesiac (ttyu0)
login:
Amnesiac (ttyu0)
login: s root
Logging to master
Password:
--- JUNOS 12.3R12.4 built 2016-01-20 04:27:51 UTC
root@xxxxxx:RE:0% cli
{master:0}
root@xxxxxx> show virtual-chassis status
Virtual Chassis ID: d4dd.7dd1.5711
Virtual Chassis Mode: Enabled
Mstr Mixed Neighbor List
Member ID Status Serial No Model prio Role Mode ID Interface
0 (FPC 0) Prsnt BM0208266xxx ex4200-24t 255 Master* N 1 vcp-0
6 vcp-1
1 (FPC 1) Prsnt BM0209485xxx ex4200-24t 128 Linecard N 2 vcp-0
0 vcp-1
2 (FPC 2) Prsnt BM0209484xxx ex4200-24t 128 Linecard N 3 vcp-0
1 vcp-1
3 (FPC 3) Prsnt BM0209485xxx ex4200-24t 255 Backup N 4 vcp-0
2 vcp-1
4 (FPC 4) Prsnt BM0209484xxx ex4200-24t 128 Linecard N 5 vcp-0
3 vcp-1
5 (FPC 5) Prsnt BM0209378xxx ex4200-24t 128 Linecard N 4 vcp-0
6 vcp-1
6 (FPC 6) Prsnt BM0212417xxx ex4200-24t 128 Linecard N 0 vcp-0
5 vcp-1
Member ID for next new member: 7 (FPC 7)
Problem fixed 🙂
Maybe a tip: try to put a tftp server on your computer and check if it works before doing upgrades or installs. It can always come in handy.