8.1 reboot loop -- not NIC related

[Mysphyt]

So I've acquired a second TiVo, and I want to make sure that I can get it up and running (by which I mean at least telnet, ftp, and superpatch) before I commit to a second service agreement. It's a SA S2 (TCD240080) running 8.1. This is my first experience with 8.1, but since my understanding is that the 3.1.5 kernel I'm running on my current TiVo (same model, running either 7.2.1 or 7.2.2--it's downstairs ) should still work. I've already got a computer open (upgrades) so I've pulled my drive and reattempted this several times already. I keep ending up in a reboot loop--without my NIC connected, so it's not that. Here's the procedure I've gone through.

The new Tivo drive is installed in my linux box as secondary master (/dev/hdc).

On the new, bootpage -p showed "root=/dev/hda4".

Originally, I tried pulling the kernel from my old box (via dd if=/dev/hda3 of=kernel, ftp to new box, dd if=kernel of=/dev/hdc3), but no joy. So I downloaded a fresh (but killhdinitrd's) copy from here and tried again, to the same result.

Copied over my rc.sysinit.author (stripping everything but path, serial, telnet, and ftp).
chmod 755 rc.sysinit.author

Gutted iptables--mv iptables iptables.old; edited iptables to

#! /bin/bash
exit 0

and chmod 755 iptables.

Copied AlphaWolf's AIO pack to /tivo-bin (where my path is pointing). At this point, only concerned with FTP.

bootpage -P "root=/dev/hda4 dsscon=true console=2,115200 upgradesoftware=false" -C /dev/hdc

and shutdown.

It reboots shortly after the TiVo "Just a moment longer" screen comes up, and when I reconnect it to my computer, everything's gone. Well. Except an empty /tivo-bin directory, and both iptables and iptables.old are missing (which may be why it's rebooting after the first time).

My guess from all of this is that I've got a problem with my kernel, since it's removing all of my hacks. What am I missing here?

[ScanMan]

I'm confused on a couple of points, if you're getting a reboot and no hacks, how are you FTP'ing your "kernel" up to the new box? I would think you should be doing all the prep work (i.e, dd kernel, mod bootpage parms, create .author file, fix iptables, copy hacks, etc.) while the drive is in the PC. Lastly, serial console output is invaluable in troubleshooting reboot problems.

[Mysphyt]

Sorry if I wasn't clear on this, but yes, all of the work I've been doing has been while my drive has been in the PC. I've just upgraded one of my linux boxes, so it's open next to the tivo already, so it's been easy for me to move the drive back and forth. "FTP to new box" was my rather clumsy attempt at shortening "FTP kernel from current working Tivo to linux box where new Tivo drive is installed." I've tried two or three times with the 3.1.5 kernel copied from my 7.2 tivo as indicated, and once with the fresh 3.1.5 kernel, to the same results.

As I said, my hacks get wiped in the reboot loop, so I'm willing to bet that the second and subsequent reboots are caused by a missing iptables (because my modified one gets deleted). So I'm going to need to repull the drive and rehack before I can get meaningful serial output, which I should be able to do this afternoon. So silly, that people are asking me to actually "do" my "job" when there's a Tivo to be worked on. I'll get serial output up as soon as I can.

[ScanMan]

So your telnetting in to your good tivo, dd the kernel from hda3 to kernel file, FTP'ing kernel file to linux box w/ tivo drive 2-B-hacked mounted as primary master, and then dd the kernel file to primary master (hda3) of your 2-B-hacked drive? Nothing jumps out from that scenario; make sure you're FTP'ing binary. Yes, the iptables is getting deleted b/c the killhdinitrd kernel is not properly in place and the security checks are pruning your filesystem.

[Mysphyt]

Well, actually, I've got my tivo drive as secondary master (hdc), and the hda3 appearing in my first post was a typo in the post only, now repaired. But apart from that, that's exactly what I'm doing. And it is using binary mode.

[ScanMan]

One other thing, what message does dd give when complete? Based on a 2M kernel partition and your command line above (with no blocksize argument, i.e. no bs=??) it should report something like "4 records in, 4 records out" IIRC.

[Mysphyt]

Okay. First off, you're correct, I'm not using a blocksize argument. The output of the dd is

4096+0 records in
4096+0 records out

.

Serial output for first boot (iterated file checking skipped):

CPU revision is: 00005430
FPU revision is: 00005410
Primary instruction cache 32kb, linesize 32 bytes.
Primary data cache 32kb, linesize 32 bytes.
Linux version 2.4.20 (build@buildmaster59) (gcc version 3.3.4) #1 Thu Oct 26 01:35:03 PDT 2006
Determined physical RAM map:
 memory: 01e2b000 @ 001d5000 (usable)
Initial ramdisk at: 0x80187000 (316399 bytes)
On node 0 totalpages: 8192
zone(0): 8192 pages.
zone(1): 0 pages.
zone(2): 0 pages.
Kernel command line: root=/dev/hda4 dsscon=true console=2,115200 upgradesoftware=false
Monotonic time calibrated: 81.01 counts per usec
Calibrating delay loop... 161.79 BogoMIPS
Contiguous region 1: 1048576 bytes @ address 0x81900000
Contiguous region 2: 524288 bytes @ address 0x81a00000
Contiguous region 8: 5767168 bytes @ address 0x81a80000
Contiguous region of 7340032 bytes total reserved at 0x81900000.
Memory: 23364k/30892k available (1228k kernel code, 7528k reserved, 69k data, 64k init, 0k highmem)
Dentry cache hash table entries: 4096 (order: 3, 32768 bytes)
Inode cache hash table entries: 2048 (order: 2, 16384 bytes)
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
Buffer-cache hash table entries: 1024 (order: 0, 4096 bytes)
Page-cache hash table entries: 8192 (order: 3, 32768 bytes)
Checking for 'wait' instruction...  unavailable.
POSIX conformance testing by UNIFIX
PCI: Probing PCI hardware
ttyS00 at iomem 0xb4100100 (irq = 79) is a 16550A
ttyS00 at port 0xbc010000 (irq = 133) is a unknown
ttyS00 at iomem 0xb4100140 (irq = 81) is a 16550A
ttyS00 at iomem 0xb4100120 (irq = 80) is a 16550A
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
Starting kswapd
Serial driver version 5.05c (2001-07-08) with MANY_PORTS SHARE_IRQ SERIAL_PCI enabled
ttyS00 at 0xb4100100 (irq = 79) is a 16550A
ttyS01 at 0xbc010000 (irq = 133) is a unknown
ttyS02 at 0xb4100140 (irq = 81) is a 16550A
ttyS03 at 0xb4100120 (irq = 80) is a 16550A
Uniform Multi-Platform E-IDE driver Revision: 6.31
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
hda: Maxtor 4R080L0, ATA DISK drive
ide0 at 0x400-0x407,0x438 on irq 87
hda: 160086528 sectors (81964 MB) w/2048KiB Cache, CHS=9964/255/63
Partition check:
 hda: [mac] hda1 hda2 hda3 hda4 hda5 hda6 hda7 hda8 hda9 hda10 hda11[M] hda12 hda13[M]
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
PPP generic driver version 2.4.2
PPP Deflate Compression module registered
Software Watchdog Initialized
Kernel Panic Logger registered
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP
IP: routing cache hash table of 512 buckets, 4Kbytes
TCP: Hash tables configured (established 2048 bind 4096)
ip_conntrack version 2.1 (256 buckets, 2048 max) - 152 bytes per conntrack
ip_tables: (C) 2000-2002 Netfilter core team
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
RAMDISK: Compressed image found at block 0
Freeing initrd memory: 308k freed
VFS: Mounted root (romfs filesystem) readonly.
Running as /linuxrc - autoscan!
Loading signatures file
3974 valid entries loaded
Scan /mnt
[...finds and identifies all modified files...]
The filesystem needs to be repaired
At this point we will repair the filesystem
Scan /mnt
[...finds and removes all modified files...]
No hash found for /mnt/tivo-bin/zcip
Removing /mnt/tivo-bin/zcip
Iteration complete
flushing ide devices: hda 
Restarting system.

Serial output for first boot (iterated file checking skipped)

CPU revision is: 00005430
FPU revision is: 00005410
Primary instruction cache 32kb, linesize 32 bytes.
Primary data cache 32kb, linesize 32 bytes.
Linux version 2.4.20 (build@buildmaster59) (gcc version 3.3.4) #1 Thu Oct 26 01:35:03 PDT 2006
Determined physical RAM map:
 memory: 01e2b000 @ 001d5000 (usable)
Initial ramdisk at: 0x80187000 (316399 bytes)
On node 0 totalpages: 8192
zone(0): 8192 pages.
zone(1): 0 pages.
zone(2): 0 pages.
Kernel command line: root=/dev/hda4 dsscon=true console=2,115200 upgradesoftware=false
Monotonic time calibrated: 81.01 counts per usec
Calibrating delay loop... 161.79 BogoMIPS
Contiguous region 1: 1048576 bytes @ address 0x81900000
Contiguous region 2: 524288 bytes @ address 0x81a00000
Contiguous region 8: 5767168 bytes @ address 0x81a80000
Contiguous region of 7340032 bytes total reserved at 0x81900000.
Memory: 23364k/30892k available (1228k kernel code, 7528k reserved, 69k data, 64k init, 0k highmem)
Dentry cache hash table entries: 4096 (order: 3, 32768 bytes)
Inode cache hash table entries: 2048 (order: 2, 16384 bytes)
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
Buffer-cache hash table entries: 1024 (order: 0, 4096 bytes)
Page-cache hash table entries: 8192 (order: 3, 32768 bytes)
Checking for 'wait' instruction...  unavailable.
POSIX conformance testing by UNIFIX
PCI: Probing PCI hardware
ttyS00 at iomem 0xb4100100 (irq = 79) is a 16550A
ttyS00 at port 0xbc010000 (irq = 133) is a unknown
ttyS00 at iomem 0xb4100140 (irq = 81) is a 16550A
ttyS00 at iomem 0xb4100120 (irq = 80) is a 16550A
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
Starting kswapd
Serial driver version 5.05c (2001-07-08) with MANY_PORTS SHARE_IRQ SERIAL_PCI enabled
ttyS00 at 0xb4100100 (irq = 79) is a 16550A
ttyS01 at 0xbc010000 (irq = 133) is a unknown
ttyS02 at 0xb4100140 (irq = 81) is a 16550A
ttyS03 at 0xb4100120 (irq = 80) is a 16550A
Uniform Multi-Platform E-IDE driver Revision: 6.31
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
hda: Maxtor 4R080L0, ATA DISK drive
ide0 at 0x400-0x407,0x438 on irq 87
hda: 160086528 sectors (81964 MB) w/2048KiB Cache, CHS=9964/255/63
Partition check:
 hda: [mac] hda1 hda2 hda3 hda4 hda5 hda6 hda7 hda8 hda9 hda10 hda11[M] hda12 hda13[M]
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
PPP generic driver version 2.4.2
PPP Deflate Compression module registered
Software Watchdog Initialized
Kernel Panic Logger registered
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP
IP: routing cache hash table of 512 buckets, 4Kbytes
TCP: Hash tables configured (established 2048 bind 4096)
ip_conntrack version 2.1 (256 buckets, 2048 max) - 152 bytes per conntrack
ip_tables: (C) 2000-2002 Netfilter core team
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
RAMDISK: Compressed image found at block 0
Freeing initrd memory: 308k freed
VFS: Mounted root (romfs filesystem) readonly.
Running as /linuxrc - autoscan!
Loading signatures file
3974 valid entries loaded
Scan /mnt
[...this time everything is in order...]
Scan /mnt/tivo-bin
The filesystem seems to be OK
Scanner main is done
VFS: Mounted root (ext2 filesystem) readonly.
Trying to move old root to /initrd ... okay
Freeing unused kernel memory: 64k freed
Starting rc.sysinit
Running boot Stage A_PreKickstart scripts
Scanning for configuration files
Loading tivoconfig.o
Invoking startup scripts for:
    platform 'gryphon'
    implementation 'Series2'
    implementer 'TiVo'
Releasing /initrd and clearing ramdisk, if they exist
Activating swap partitions
Adding Swap: 65532k swap-space (priority -1)
Loading core system drivers
Loading fpga.o
Parallel modem driver loaded, 1 ports starting at /dev/ttyS1
Loading i2c_Series2.o
Loading ircatch.o
Found rev 'F' Si2433 modem on /dev/cua1
Checking for Kickstart panic signal
Running boot Stage B_PostKickstart scripts
Cleanup /dev/hda9 pass 1
ext2fs_check_if_mount: No such file or directory while determining whether /dev/hda9 is mounted.
/dev/hda9 was not cleanly unmounted, check forced.
Inode 2056, i_blocks wrong 362 (counted=352).  Set i_blocks to counted? yes
Inode 2050, i_blocks wrong 540 (counted=526).  Set i_blocks to counted? yes
Inode 2057, i_blocks wrong 72 (counted=60).  Set i_blocks to counted? yes
Fix summary information? yes
/dev/hda9: 95/32768 files (13.7% non-contiguous), 12968/131072 blocks
Cleanup /dev/hda9 pass 2
ext2fs_check_if_mount: No such file or directory while determining whether /dev/hda9 is mounted.
/dev/hda9: clean, 95/32768 files, 12968/131072 blocks
/dev/hda9 is clean after pass 2
Mounting /var
/dev/hda9 on /var type ext2 (rw)
Cleaning up files in /var
Checking space in /var
Mounting initial environment
Starting logging daemons
Checking Panic log
Found Silicon Labs "Si2433" modem, rev F, skipping modem patch
Scanning for phase1 repair scripts
Running boot Stage C_MediaInitialization scripts
Loading input section drivers
Loading tvinput.o
Assume SAA7114 video decoder. CodeID = 4
/dev/input loaded
Loading kfirR0m.o
KFir ucode version 3.18
Loading output section drivers
Loading brcmdrv-rb.o
Linux BCM7020 STB interface: version 510082104
******** versionInfo->major is 1 ********
Splash the screen
Running boot Stage D_PreMfs scripts
Remote control is  TIVO
MFS partition on  /dev/hda10
Not loading DSS processor for this platform
Look for debug board
Not loading ATSC processor for this platform
Loading irblast.o
Loading ideturbo.o
Loading fan.o
Loading therm.o
Loading TvBus router
Updating system clock
Time set to: Wed Jan  3 01:43:32 2007
/etc/netfilter-enable: /sbin/iptables: No such file or directory
/etc/netfilter-enable: /sbin/iptables: No such file or directory
/etc/netfilter-enable: /sbin/iptables: No such file or directory
/etc/netfilter-enable: /sbin/iptables: No such file or directory
/etc/netfilter-enable: /sbin/iptables: No such file or directory
/etc/netfilter-enable: /sbin/iptables: No such file or directory
/etc/netfilter-enable: /sbin/iptables: No such file or directory
/etc/netfilter-enable: /sbin/iptables: No such file or directory
/etc/netfilter-enable: /sbin/iptables: No such file or directory
/etc/netfilter-enable: /sbin/iptables: No such file or directory
/etc/netfilter-enable: /sbin/iptables: No such file or directory
/etc/netfilter-enable: /sbin/iptables: No such file or directory
/etc/netfilter-enable: /sbin/iptables: No such file or directory
/etc/netfilter-enable: /sbin/iptables: No such file or directory
/etc/netfilter-enable: /sbin/iptables: No such file or directory
Enabling local route
Setting TCP keepalive parameters
Checking for additional disk
Start fan control
First temperature parameters set:
  Terminal temp: 71
  Critical temp: 62
  Logging temp: 60
  Target temp: 50
  Lowest fan speed: 7
/tvbin/fancontrol is running in the background.
Starting TvLauncher
assert: Tmk Assertion Failure: 
assert: TmkServer, line 379 ()
Tmk Fatal Error: Activity TvBusProxy <103> strayed!
Paste the following into a shell to get a backtrace...
bt -t /tvbin/tivoapp <<END_OF_BT
  tcd 1
  hpk Series2
  build b-8-1-mr @249679 2006.10.30-2206 release-mips []
  pack 8.1-01-2
  read 0x00400000 /tvbin/tivoapp
  read 0x02000000 /lib/libc.so.6
  read 0x02200000 /lib/libm.so.6
  read 0x02400000 /lib/libpthread.so.0
  read 0x02600000 /lib/libutil.so.1
  read 0x02800000 /lib/libtvutil.so
  read 0x02a00000 /lib/libtmk.so
  read 0x02c00000 /lib/libtvstructures.so
  read 0x2aaa8000 /lib/ld.so.1
  read 0x2ab04000 /lib/libhpkoss.so
  read 0x2ab50000 /platform/lib/libhpkhl.so
  read 0x2abf0000 /platform/lib/libhpkll.so
  read 0x2ac34000 /lib/libdl.so.2
  0x02a82758 0x02a86d48 0x02a25f30 0x02a50b54 0x00676e14 0x02a2a518 0x02a2a144 
  0x02a5fe60 0x02a2a02c 0x02a29f40 0x02a2c94c 0x02a3e070 0x02a3df58 0x02a3de08 
  0x02a3d394 
END_OF_BT
Tmk Fatal Error: Activity TvBusProxy <103>: assertion failure
Waiting for launcher to start.
Launcher is running.
flushing ide devices: hda 
Restarting system.

To me this looks for all the world like I'm booting the stock kernel, but I'm definitely not--this is using the kernel I dd'd off of my working box. Is it possible that somehow that the bootpage points to /dev/hda4 (which it did when I started all of this) but that /dev/hda6 is actually where the kernel's being pulled from?

[Jamie]

...

Linux version 2.4.20 (build@buildmaster59) (gcc version 3.3.4) #1 Thu Oct 26 01:35:03 PDT 2006

To me this looks for all the world like I'm booting the stock kernel, but I'm definitely not--this is using the kernel I dd'd off of my working box. Is it possible that somehow that the bootpage points to /dev/hda4 (which it did when I started all of this) but that /dev/hda6 is actually where the kernel's being pulled from?

That's the signature of the stock 8.1 Series2 kernel, so indeed you are booting the stock kernel. You somehow blew it when dd'ing the kernel in.

You can always get the signature of a kernel via:

% strings vmlinux.px | grep "Linux version"
Linux version 2.4.20 (build@buildmaster59) (gcc version 3.3.4) #1 Thu Oct 26 01:35:03 PDT 2006

You can substitute a kernel partition (e.g. /dev/hda3 or /dev/hda6) for the vmlinux.px. The build date/time will generally uniquely identify the kernel.

Never hurts to verify that the kernel is the one you think it is before allowing it to boot. It can be a mess to clean up after the stock initrd has done it's evil deed.

[Mysphyt]

Weird, and interesting. I did not, in fact, blow it when dd'ing the kernel in:

mysphyt-server tivo # strings kernel | grep "Linux version" //the kernel I dd'd in earlier
Linux version 2.4.20 (build@buildmaster5) (gcc version 3.0) #22 Fri Feb 20 18:19:25 PST 2004

mysphyt-server tivoapps # strings /dev/hdc3 | grep "Linux version"
Linux version 2.4.20 (build@buildmaster5) (gcc version 3.0) #22 Fri Feb 20 18:19:25 PST 2004

mysphyt-server tivoapps # strings /dev/hdc6 | grep "Linux version"
Linux version 2.4.20 (build@buildmaster59) (gcc version 3.3.4) #1 Thu Oct 26 01:35:03 PDT 2006

mysphyt-server tivoapps # ./bootpage -p /dev/hdc
root=/dev/hda4 dsscon=true console=2,115200 upgradesoftware=false

So it looks like, despite the bootpage, I'm booting the kernel on /dev/hda6, and using the root on /dev/hda4. Any ideas how to fix that?

---

Idea, and disclosure. I thought I did this right, but perhaps I didn't. Since I knew (or thought I knew) what I was doing, I rather unwisely started with this Tivo by just straight pulling the drive, without checking the software version, assuming that since it wasn't hacked, it would be up to date (as in >7.1, since I believe everything since then has worked with the 3.1.5 kernel). I ended up in a reboot loop (just like this one) on boot, but assumed it was something I'd done. So I manually flipped the bootpage (which was "root=/dev/hda4") via 'bootpage -P "root=/dev/hda7" -C /dev/hdc'. Booted without incident, upgraded to 8.1, which brings us to the top of this story.

Was I operating under a false understanding of bootpage? If the bootpage is just the kernel parameters, then I was actually booting with the kernel on hda3 and the root on hda7. When the upgrade completed, could it have properly switched the bootpage, and effectively flip-flopped my mistake, now booting with the root on hda6 and the kernel on hda3?

[PlainBill]

I'm not a Linux guru, but both /etc/fstab and the boot parameters (bootpage -p /dev/hda) must be correct.

Just to drive it home, NEVER write new boot parameters or set the kernel partition without checking the existing settings. The TiVo flips the root and boot partitions with every software update. Just because Joe Nobody wrote a guide that says root is always /dev/hda4 doesn't make it so.

PlainBill

[Jamie]

...

mysphyt-server tivoapps # ./bootpage -p /dev/hdc
root=/dev/hda4 dsscon=true console=2,115200 upgradesoftware=false

...

You aren't looking at the part of the bootpage that tells you what kernel it will boot. You're just looking at the parameters passed to the kernel.

Try bootpage -b to see which partition the kernel will be booted from. It is *usually* one less than the root partition, but it doesn't have to be, and there are many ways to screw it up. Your manual flip of the bootpage was one way to screw it up (changing kernel parameters but not the boot partition.)

[ScanMan]

Yes, I think you are on the right track. You didn't "flip" the bootpage, that's the problem; all you did was write new bootpage parameters to point root=/dev/hda4. So what you are currently doing is booting the kernel at /dev/hda6 and loading the root filesystem at /dev/hda4. This can be very problematic if the software versions on the active/alternate are notably different and the MFS changes made by the more recent software release are incompatible with the older software. Check out the bootpage man page here; reset the bootpage to match where the latest root is and make your fixes, boot, capture the console output and see where you're at.

[Mysphyt]

Thanks, PlainBill, and of course you're right. The problem isn't that I didn't check my settings beforehand, it's that I didn't know what I thought I did about bootpage. I used bootpage -P to change my kernel parameters, but I didn't use bootpage -f to flip my boot partitions. So my tivo booted as 3/7, upgraded, and flipped to 6/4. I dd'd the killhdinitrd'd kernel into 3, which wasn't touched. I've since used bootpage -f get my boot partitions back how they're supposed to be, so I'm on 3/4 again. In all my toolings around on 3/4 I'm still unable to get the fool thing to boot, but I did get it to boot the right kernel, and I think it's just that I forgot to chmod iptables this time 'round that's holding me back. One more try.

[Mysphyt]

That was it. I'm happily booted on 3/4 now. Thanks to all who helped.