/var occassionally corrupted, system rebuilds

[TheSaint]

After the 6.3a upgrade, I find that on occassion my /var gets wiped and rebuilt. The logs say that the filesystem was corrupted. It doesn't wipe it every time after a reboot however. I did a customary extended disk diagnostics with Western Digital's software and it found no disk errors. I also ran e2fsck while in the tivo to see what it would report. Several unreferenced inodes were found, which was while the partition was still mounted.
What could cause this partition to get occasionally corrupted? Suggestions?

TheSaint
H10-250

[Jamie]

This is normal tivo behavior. Any ext2 file systems mounted readwrite can and occasionally will get corrupted when you reboot without a clean shutdown. I use a journalled file system (ext3) to avoid the problem.

[TheSaint]

Jamie,

What's not considered a clean shutdown? I would think pulling the plug would qualify. I would think tivo menu generated restarts and bash prompt reboot would be clean. In the previous tivo version, I never had /var wiped. It's disconcerting that it's happening now and more than just a one time occurence.


This is normal tivo behavior. Any ext2 file systems mounted readwrite can and occasionally will get corrupted when you reboot without a clean shutdown. I use a journalled file system (ext3) to avoid the problem.

[Narf54321]

Just to hazard a guess, but it might be worth checking if is something spewing log files in /var causing it to fill or something.

[cheer]

Jamie,

What's not considered a clean shutdown? I would think pulling the plug would qualify. I would think tivo menu generated restarts and bash prompt reboot would be clean. In the previous tivo version, I never had /var wiped. It's disconcerting that it's happening now and more than just a one time occurence.
Pulling a plug is hardly a clean shutdown!

The Tivo has no mechanism for a clean shutdown as it never truly "shuts." Root is read-only and var is wipe-able.

This is not new behavior -- Tivo has done this for a long time. Tivo will also wipe /var if it seems to fill up (logs, whatever).

[Jamie]

Jamie,

What's not considered a clean shutdown? I would think pulling the plug would qualify. I would think tivo menu generated restarts and bash prompt reboot would be clean. In the previous tivo version, I never had /var wiped. It's disconcerting that it's happening now and more than just a one time occurence.To do a clean shutdown, you'd want to cleanly terminate running processes and remount all file systems readonly before shutting down. In the past, the tivo software really didn't have any way to do that, since there was no way to cleanly shut down the tivo processes without forcing a reboot. That may have changed in current software versions.

A stock tivo doesn't really need a clean shutdown mechanism. The root file system is mounted readonly (so no danger of corruption) and /var can be wiped at will without causing any problems. MFS is journalled.

BTW: it is never safe to run e2fsck on a file system mounted readwrite. The startup scripts run e2fsck on /var before it is mounted. There's no nead for you to do it yourself. The root file system, on the other hand, is never scanned by the tivo software, since it is always suppose to be readonly. It's not a bad idea to run e2fsck on the root once in a while if you regularly leave it mounted readwrite. Just be sure you remount it readonly before you scan it.

[Cheezmo]

Some of the audio issues people are having do result in lots of log file output. If you are using fakecall or otherwise not doing phone calls, you need to be clearing out your logs regularly or 6.3 definitely will generate some large log files, fill var, and cause it to get wiped.

[TheSaint]

I was just diagnosing, so I just ran e2fsck to see if the drive was currently corrupt and on the next reboot it would be wiped.

I also have been experiencing the audio dropouts mentioned. There is log traffic on that. I need to see how close I'm getting to filling up var. Over time I've let a lot of hacks pile up in there. Probably a good time to clean it up.


TheSaint

To do a clean shutdown, you'd want to cleanly terminate running processes and remount all file systems readonly before shutting down. In the past, the tivo software really didn't have any way to do that, since there was no way to cleanly shut down the tivo processes without forcing a reboot. That may have changed in current software versions.

A stock tivo doesn't really need a clean shutdown mechanism. The root file system is mounted readonly (so no danger of corruption) and /var can be wiped at will without causing any problems. MFS is journalled.

BTW: it is never safe to run e2fsck on a file system mounted readwrite. The startup scripts run e2fsck on /var before it is mounted. There's no nead for you to do it yourself. The root file system, on the other hand, is never scanned by the tivo software, since it is always suppose to be readonly. It's not a bad idea to run e2fsck on the root once in a while if you regularly leave it mounted readwrite. Just be sure you remount it readonly before you scan it.

[pab]

Really, though, does it matter that much? /var has never been a safe place for hacks, and nothing important and irreplaceable should ever be placed there. I learned that years ago when the common practice was to put tivoftpd and a version of tnlite into /var/hack when you hacked the box.

[Narf54321]

Its a pain to set up, but eventually I started creating a /hack directory from root, putting all my hacks there, and then have a series of if..then statements in rc.sysinit.author to copy them at boottime into /var. Anything which relies heavily on read-write access is happy, and if it gets blown away it all gets recreated on the next reboot.

[cheer]

Its a pain to set up, but eventually I started creating a /hack directory from root, putting all my hacks there, and then have a series of if..then statements in rc.sysinit.author to copy them at boottime into /var. Anything which relies heavily on read-write access is happy, and if it gets blown away it all gets recreated on the next reboot.
My general approach has been to put everything in /hacks and create symlinks for the things needing rw (mfs_ftp's cache director, for example). But now I'm starting to just use a separate partition altogether.

[mike_s]

My general approach has been to put everything in /hacks and create symlinks for the things needing rw (mfs_ftp's cache director, for example). But now I'm starting to just use a separate partition altogether.Yep. TiVo can rebuild /var all it wants.


#part of rc.sysinit.author
#Make sure our hacks are always around...
if [ -d /var/hack ]
then
echo "$(date +"%d %b %T") /var/hack OK." >> /var/log/hack
else
echo "$(date +"%d %b %T") /var/hack mount point missing, recreating." >> /var/log/hack
mkdir /var/hack
fi
echo "$(date +"%d %b %T") Mounting /dev/hda2 at /var/hack" >> /var/log/hack
mount /dev/hda2 /var/hack

[cheer]

Yep. TiVo can rebuild /var all it wants.
Just curious...is there a reason you don't put your mountpoint right off of root (/hacks or whatever)? Then you don't even have to worry about the mount point disappearing.

[mike_s]

Just curious...is there a reason you don't put your mountpoint right off of root (/hacks or whatever)? Then you don't even have to worry about the mount point disappearing.Except during an upgrade, and then the script would need to rw/ro the root FS. ;) It's mostly a matter of *nix convention (http://www.pathname.com/fhs/pub/fhs-2.3.html), plus it's the historical path for TiVo. It just seemed to be the right place for me.

[Narf54321]

I noticed you put your hacks in partition 2, one of the 'bootstrap' partitions. Does this have a negative effect?

(I'm asking because on one of my units I created a large swap space and the used pdisk to reallocate some room to partition 5 for the same idea, but I haven't actually used it yet.)

[Jamie]

I noticed you put your hacks in partition 2, one of the 'bootstrap' partitions. Does this have a negative effect?

(I'm asking because on one of my units I created a large swap space and the used pdisk to reallocate some room to partition 5 for the same idea, but I haven't actually used it yet.)2 and 5 are good partition #'s to snag for a hack partition. The only downside I am aware of is that mfstools won't back them up, so you need to back them up separately.

[mgoddard]

I was seeing /var/hack getting wiped too with 6.3a so I now have my hacks on a 512MB USB flash sticking out the back of my tivo and I mount it to /hack. Seems to work pretty well for me.

[satjunkee]

I was seeing /var/hack getting wiped too with 6.3a so I now have my hacks on a 512MB USB flash sticking out the back of my tivo and I mount it to /hack. Seems to work pretty well for me.

That's an interesting idea. Can you post the startup mount script for that? Are there additional drivers needed to use the USB stick?

[mgoddard]

I have this in the top of my rc.sysinit.author script and these modules are part of the tivo OS.

#install drivers for usb external devices like disk drives
insmod /lib/modules/scsi_mod.o
insmod /lib/modules/usb-storage.o
insmod /lib/modules/sd_mod.o
insmod /lib/modules/fat.o
insmod /lib/modules/vfat.o
sleep 10
mount -t vfat /dev/sda1 /hack


My USB stick is formatted fat32 which makes it easier to transfer files when I stick it in my windows machine.



That's an interesting idea. Can you post the startup mount script for that? Are there additional drivers needed to use the USB stick?

[Greubin]

This is brilliant, I gotta try this:D

I have this in the top of my rc.sysinit.author script and these modules are part of the tivo OS.

#install drivers for usb external devices like disk drives
insmod /lib/modules/scsi_mod.o
insmod /lib/modules/usb-storage.o
insmod /lib/modules/sd_mod.o
insmod /lib/modules/fat.o
insmod /lib/modules/vfat.o
sleep 10
mount -t vfat /dev/sda1 /hack


My USB stick is formatted fat32 which makes it easier to transfer files when I stick it in my windows machine.