PDA

View Full Version : TiVo (SA 3.0.1 w/ TiVoNET) freeze / crash when extracting w/ TyTool, mfs_ftp



RangerOne
06-22-2004, 04:13 AM
I have an extra hacked Philips HDR112 (the original standalone 14 hr units) with TiVo software 3.0(-01-1-000). It has been hard drive upgraded and has a TiVoNET adapter and ISA NE2000 clone installed. It's not subscribed, generally used as a glorified VCR to catch series marathons, etc. It's been capacity upgraded using the Hinsdale Guide and mod'd (mostly) using of the Steve Jenkins' How-To guides, with telnet and FTP access, TiVoWeb 1.9.4, plus some enhancements like auto r/w mounting upon login and unmounting at logout, command aliases, etc. All these work fine.

I've been hacking TiVos for some time now, and am comfortable getting around.

Recently, I tried to extract a few shows off it using TyTool (9r14) and mfs_ftp (1.2.9p) and keep running into a hard freeze/crash. It's a 'hard' freeze/crash in the sense that the entire TiVo freezes, including the MyWorld/UI--the TV display shows the unit stuck on whatever video (frame) was showing at the time of the crash, whether it's a live broadcast or one of the menu pages. The unit no longer responds to the remote, network commands or pings; any active telnet sessions are hung and TyTool or the FTP client in use indicates no more network activity. Put briefly, it's a hard freeze/crash and the only remedy is a cold reboot using the power strip it is attached to.

The interesting bit is, the first show (~1.5 GB) I tried extracted fine without any trouble using TyTool. When I tried to extract the second (~2.5 GB), it hung about halfway into the download. Then, on subsequent reboots, the hang occured at various spots along the way, sometimes within the first 15 - 20% of the way, sometimes as late as upwards of 90% of the way. Thinking it was potentially a problem with that particular recording, I tried others and also newly recorded shows. In all cases, a freeze/crash occurs unpredictably somewhere along the line. I switched to mfs_ftp to see if it was a TyTool issue; I was able to get the ~2.5 GB show after a few tries without the identical freeze/crash this one time only, but overall, the issue remains; the freeze/crash occurs almost every single time.

I did some researching on the forums, and followed some of the suggestions:

1) I tried getting only parts of a recording instead of the whole in TyTool, albeit with the same results.
2) Getting TMFs instead of TYs via mfs_ftp did not help either, with the same crashes every time.
3) I tried Single Socket mode in TyTool, with again a hard freeze/crash.
4) I then tried to see if I can narrow it down to a network/NIC issue vs. a recording/mfs issue; I did a

/var/hack/mfs_ftp/mfs_stream -s <fsid_of_a_recording_part> > /dev/null

and that went through fine, with different fsids and different recordings, so it doesn't seem there are disk/filesystem/recording corruption issues here.

Here are the relevant posts I found:

mfs_ftp crash & fsid break (http://www.dealdatabase.com/forum/showthread.php?t=35272) - This one seems to be the *exact* same problem I have.

tytool: tserver_mfs7 & 3.0.1b (http://www.dealdatabase.com/forum/showthread.php?t=28314) and Network problems when extracting (http://www.dealdatabase.com/forum/showthread.php?t=28510) - These are *similar*; the solution for one user was to replace the NIC (though it was a wireless one). I am beginning to suspect my NE2000 clone.

tserver_mfs7 Locks up. (http://www.dealdatabase.com/forum/showthread.php?t=28524) - This is another *similar* post; commented on by those on the earlier posts, again not much help.

I have another NE2000 clone NIC from another manufacturer, so I am going to replace the one in the TiVo with this one to see if there is any difference (since they are both NE2000 clones, I expect little or no reconfiguration). However, in the meanwhile, I am open to suggestions as to what to try next.

Thanks!
RangerOne

RangerOne
06-22-2004, 05:44 AM
OK, no luck.

I did install the second NE2000 clone I had, a UMC chipset one. This is a jumperless ISA PnP card, so I had to use the DOS-based config tool to set it to I/O 300h and IRQ 5 like the original NE2000 clone which was jumpered that way. I then replaced the card and rebooted.

The interface came up fine, as I expected. Again, I was able to telnet, FTP, view TiVoWeb, etc. I started tserver_mfs7 and started downloading a 2 hr program recorded at the standard Medium Quality (~2.3 GB). The TiVo froze / crashed 31% of the way in exactly the same way, with the video frozen and no response from anything.

I found the manual for the old NE2000 clone NIC--on this jumpered card, there is a jumper called 'PCMODE'; the ancient manual says this:

"Some 286-class AT compatible computers using a Chips & Technology 286 chipset are not AT timing compatible when using the I/O CH RDY signal. The NE-12, NE-12T and NE-12CT adapters provide a way for these computers to be able to accept an Ethernet card. The PCMODE jumper may be enabled if your computer seems to be having communications problems with the NE-12 family of adapters. By enabling this jumper, you force the adapter to respond to C&T motherboard timing instead of standard AT timing. NOTE: This problem is generally only encountered in 286-class AT compatible PC's. 386- and 486-class machines do not normally use a non-standard timing scheme."

Interestingly enough, I had this on the ON position previously (i.e. for the so-called 'non-standard 286-class timing' option). I am now going to try it with that off and using this original NE2000 clone back in the TiVo. Let's see if there is any difference.

BTW, IIRC Tridge's original TiVoNET document (http://samba.org/tridge/tivo-ethernet/isa_adapter_howto.txt) says a 3com 3C503 may also work, though the NE2000 compatibles are better...I might have one lying around and if I can dig it out, I might give that a try to see if there's any difference--i.e., whether it is a TiVoNET/TiVoNET + software or NE2000 incompatibility.

A couple of other theories (unresolved variables):

1) I wonder if the network switch I am going through (a D-Link 8-port switch) could be related? Perhaps some weird incompatibility triggering something bad on the TiVo? Sounds unlikely to me.

2) The only time I've seen similar behavior in electronics, inc. computers, they have always been heat-related--i.e., something overheating. Since the transfer is potentially CPU intensive (I am not sure about this, just speculating), could it be overheating something on the board and crashing the system? I've seen the 'Internal Temperature' as reported by the TiVo GUI as high as 43 C post-reboot immediately after a crash, but TiVo GUI also annotates this as being 'normal' (It's running at 34 C idle right now). This would also explain the apparent randomness of where the freeze / crashes occur.

Still can't get the shows off via the network. Grrr...

RangerOne

eastwind
06-22-2004, 06:30 AM
If you're worried that it might be a heat issue, pop the top off the TiVo and run some tests that way (with excess heat able to excape vertically). And if you suspect the switch, move the TiVo closer to the router/PC and eliminate the switch. I didn't read through the referenced threads, but have you checked your CAT-5? Maybe replaced the wire runs just to make sure you don't have a bad cable? (I doubt this would cause the problems you're having with the hard freeze though.)

ew

rfc
06-22-2004, 06:45 AM
OK, no luck.


1) I wonder if the network switch I am going through (a D-Link 8-port switch) could be related? Perhaps some weird incompatibility triggering something bad on the TiVo? Sounds unlikely to me.



RangerOne


At the risk of stating the obvious, why not connect the pc directly to the tivo with a crossover cable (a new, pre-made one, not home made, lol). That would completely rule out any issue with the switch, or any other weirdness in your network.

I commend you on the homework done. A good model for others to follow.

RangerOne
06-22-2004, 07:26 AM
eastwind, rfc: Thanks for the suggestions.

I did run another test with the cover off and the NE2000 PCMODE jumper off, but still a freeze / crash. This time it happened about 60% into the ~2.3 GB recording I was getting via mfs_ftp.

I am going to start logging when and where the freeze / crashes occur; this one was at 1.38 GB (60%), 28:13 min in. WS_FTP last reported a 7.81 Mbps transfer rate...I wonder if a high enough transfer rate causes some sort of saturation condition--this was one of the problems with the person with the Aironet (D-Link 802.11b wireless) card, the crash occurring when the transfer hit a high enough rate.

I will give a crossover and/or another set of cables/switches to be thorough. I do have cable tester kits on hand, so I can verify both the current and new cabling.

Re- the heat suspicion. I noticed now, after the last freeze / crash that the MPEG2 decoder chip was very, very hot to the touch. Now, I've done my share of overclocking, etc. and know from experience CPUs and other ICs can run at acceptable temperatures that will be too hot to the touch, but this just felt wrong. I've got one of those Holmes desk fans that I keep around (they're the best remedy when you're mixing up audio for a party and need to cool your rig in a tight space!). I'm going to run that blowing over the TiVo to try and give some extra cooling and see if that makes a difference.

rfc: BTW, thanks for the compliment. I've had long stints in IT support and I know those who want to tweak things but don't want to do the requisite work beforehand (RTFM, etc.--or RTFB in this case, I suppose) are the most annoying.

Will post further findings.

RangerOne

RangerOne
06-22-2004, 10:07 AM
OK, some more findings, but no resolution yet...

First, I tried several runs with the fan blowing over the TiVo for some extra cooling. It was helping at least cool down the HDD and the unit, as I monitored the TiVo 'internal temperature' run steady at 28 - 29 C with the fan on (whereas it went as high as 43 C before, both values declared 'normal' by the TiVo GUI). The freeze/crash still happened every time.

I don't have a crossover cable handy, so I went down to a very simple setup--just the PC and the TiVo hooked up via an old NetGear hub (not even a switch) I had lying around. At first it seemed like it was going to make it, but TyTools froze/crashed the TiVo about a quarter of the way through.

So, now I suspect it's not a heat issue (though the MPEG2 decoder chip is still very hot to the touch immediately after freezes, though I don't suspect mfs_stream uses it since it seems to dump the ty stream instead of decoding anything--I could be wrong).

To summarize, so far I've eliminated
- heat issues & overheating
- network topology (at least with the exception of a crossover cable)
- NE2000 clone card (tried two different brands)
- stream issues (mfs_stream dumps fine to /dev/null)
- physical contact, seating, etc. -- I know the TiVoNET adapter is a very finnicky piece of hardware (low tolerances/clearances), so I secure it in place very tightly so it absolutely doesn't move (though I wonder if somehow some heat is causing it to expand and break contact or something!!!)

What's left? A bug in mfs_ftp and/or TyTools net routines? A conflict with some other TiVo app? This isn't a heavily hacked TiVo, just telnet, FTP prompts, TiVoWeb 1.9.4, and the capacity upgrade as I spoke. No subs, and mfs_ftp.tcl and tserver_mfs7 are only run when attempting to extract...A problem with the TiVo itself?

Suggestions welcome--I've run out of ideas.

RangerOne

eastwind
06-22-2004, 12:20 PM
To summarize, so far I've eliminated
- heat issues & overheating
- network topology (at least with the exception of a crossover cable)
- NE2000 clone card (tried two different brands)
- stream issues (mfs_stream dumps fine to /dev/null)
- physical contact, seating, etc. -- I know the TiVoNET adapter is a very finnicky piece of hardware (low tolerances/clearances), so I secure it in place very tightly so it absolutely doesn't move (though I wonder if somehow some heat is causing it to expand and break contact or something!!!)

What's left? A bug in mfs_ftp and/or TyTools net routines? A conflict with some other TiVo app? This isn't a heavily hacked TiVo, just telnet, FTP prompts, TiVoWeb 1.9.4, and the capacity upgrade as I spoke. No subs, and mfs_ftp.tcl and tserver_mfs7 are only run when attempting to extract...A problem with the TiVo itself?

Suggestions welcome--I've run out of ideas.

RangerOne
Run it without TiVoWeb. And if you suspect the network is being overrun, throttle back mfs_ftp so it runs slower. I've never done it, but it is covered in the readme.txt file.

ew

RangerOne
06-23-2004, 01:06 PM
OK, no definitive solution, but more troubleshooting.

I've tried running the transfer via mfs_ftp without TiVoWeb running (and after a reboot after stopping and disabling it). Same kind of crash happens.

Interesting thing; today, I started testing fresh with mfs_ftp and FileZilla on a rebooted TiVo that had just been running idle overnight (i.e., no transfers or attempts since the last reboot, just sitting there, caching whatever channel it was last left on, for 10 - 12 hours). I set FileZilla to limit the connection from the PC side to 1000 KB/s and disabled keepalives, active mode (as before), and attempted a transfer of the same ~2.3 GB recording. This *completed fine*. I then attempted the second ~2.3 GB recording, which froze with the same symptoms 1/3 of the way.

I rebooted, and this time, I tried the second ~2.3 GB recording again, but it froze about halfway. After another reboot, I was this time able to *complete* the transfer of the second recording. Since then, I've been running into crashes consistently while trying to get the remaining ~4.6 GB recording.

It is at this point that I tried disabling TiVoWeb, which had been running until then. No luck; same crash. It's interesting that this appears to happen when a recent transfer has been attempted/failed or the TiVo has had significant network access--if I leave the TiVo alone after rebooting and try this "fresh", it seems to work; subsequent transfers, persistently across reboots, tend to fail. I bet if I leave it untouched for several hours after a reboot and try again, it will work...it's as if some sort of "buffer" is being filled up.

Anyway, right now, ftpd and tnlited are the only things left running. I'm not certain if mfs_ftp requires ftpd (e.g., does it use ftpd or spawn its own), but I obviously would like to keep tnlited for troubleshooting and access. I am now going to try the ithrottle setting mentioned in the mfs_ftp ReadMe (that eastwind mentioned), though from the sound of it, that's intended more for pc -> tivo transfers. We'll see.

I also found The Jake's post to AVS/TiVoCommunity Forums (http://www.tivocommunity.com/tivo-vb/showthread.php?threadid=179196&highlight=TiVoNET+freeze) about the issue about the same time he posted his thread here (http://www.dealdatabase.com/forum/showthread.php?p=170848#post170848) (which I quoted in my original post). I should send him a message directly through the board to see if he can shed some more light, as he is so far the only other person with/instance of the exact same issues.

Will keep the thread posted.

RangerOne

RangerOne
06-23-2004, 01:28 PM
Nope, no luck with the ithrottle set to 7, will try 10 (highest/slowest).

Interesting thing is, none of the usual log files (/var/log/kernel, port.3105.log in the mfs_ftp directory) seem to show any failure or interesting that would indicate a precursor to the freeze.

Here are some empirical data:

Bytes xferred prior to crash / Xfer rate last reported / Xfer time (start to) crash

1,380,880,020 bytes / 7.81 Mbps / 28:13 mins:secs (mfs_ftp)
1,751,748,408 bytes / 7.83 Mbps / 35:41 (mfs_ftp)
464,325,644 bytes / 8.12 Mbps / 9:07 (mfs_ftp)
129,241,088 bytes / 8.07 Mbps / 2:33 (mfs_ftp)
344,723,456 bytes / 8.10 Mbps / 6:47 (mfs_ftp)
631,111,680 / 0.65 Mbps / 15:20 (tserver_mfs7)
304,425,104 / 727 KB/s / 6:24 (mfs_ftp)
726,718,648 / ? / ? (mfs_ftp)

Hmmm...nothing of a pattern jumps at me, with perhaps the exception of the transfer rate...but these are all before throttling attempts from either PC and/or the TiVo side.

RangerOne

eastwind
06-23-2004, 02:59 PM
It is at this point that I tried disabling TiVoWeb, which had been running until then. No luck; same crash. It's interesting that this appears to happen when a recent transfer has been attempted/failed or the TiVo has had significant network access--if I leave the TiVo alone after rebooting and try this "fresh", it seems to work; subsequent transfers, persistently across reboots, tend to fail. I bet if I leave it untouched for several hours after a reboot and try again, it will work...it's as if some sort of "buffer" is being filled up.
RangerOne
mfs_ftp doesn't need ftpd AFAIK--been a long time since I started using it. I do know that TyTool has it's own server. And it can take the FSIDs one at a time (I'm pretty sure mfs_ftp has this capabilities also). If you can pull each 512 meg FSID one at a time it might be a work around until you get the real problem tracked down.
ew

RangerOne
06-23-2004, 03:40 PM
A few more additions...

Tried a known-good 80-conductor IDE cable, just in case, instead of the TiVo stock IDE cable (I know TiVo won't do high speed transfers, but at least this is a reliable cable). This is especially on account of the drive currently there being a UDMA133 capable drive which by now has downgraded itself to UDMA33 or slower operations. This was recommended by a friend who recalled misc freezes tracked down to bad IDE cables on other TiVos. Did not help.

I am now trying without ftpd (eastwind was right, mfs_ftp doesn't seem to need it to be running), as well as without TiVoWeb and ithrottle set to 10 in mfs_ftp.tcl. Let's see how this fares.

I have not yet been able to coax my 3C503 card into working (no pun intended). I'll try that if I can get it recognized.

RangerOne

RangerOne
06-23-2004, 06:30 PM
Here's another older post with the exact same setup as The Jake and I, plus the same issue: Extraction locks up TiVo (http://www.dealdatabase.com/forum/showthread.php?t=25169). Now if I can only get these folks talking to track down the common factors...:)

RangerOne

RangerOne
06-23-2004, 06:51 PM
I e-mailed Matt (from the last thread I found) to see if he can comment.

Here is another older thread that seems to be somewhat related if you read into it; the person was using older extraction methods and mfs_stream over NFS mounts, but still, the symptoms are similar and so are the hardware:

Extraction crashes my SA (http://www.dealdatabase.com/forum/showthread.php?t=13360)

BTW, sorry if I seem to be posting too much, but I'd like to collect as much info on this as possible so that the next guy doesn't have to...even though this is an old NIC adapter, there must still be people using them.

RangerOne

RangerOne
06-24-2004, 07:20 AM
Some additional findings before I run off to work:

I found a few additional posts with identical or similar problems. See

Tivonet Card Failure? (http://www.dealdatabase.com/forum/showthread.php?t=16373)

Tivonet Question (http://www.dealdatabase.com/forum/showthread.php?t=16357)

(though from the kernel parameters used in rc.sysinit for the NIC module as the suggested solution, these last two sound like TurboNET and not TiVoNET.)

Poor performance extracting video (http://www.dealdatabase.com/forum/showthread.php?t=16431) is similar, but also Turbonet.

Fixed: Tivo crash during extraction (http://forum.ptvupgrade.com/showthread.php?t=388) points to a NIC failure.

Just to be sure I was sane and the problem was probably hardware or driver related, I let mfs_stream run through all FSIDs of the longest recordings I have 3 times, consecutively, piping to /dev/null without a single hitch.

Yesterday, I tried the timing=16 option (apparently only for TurboNET) on the /etc/rc.d/rc.arch line that loads 8390tridge.0; predictably, that parameter was not in its syntax, and it failed to load, prompting me to remove the drive and return to the PC to fix /etc/rc/d/rc.arch with the Jenkins' boot CD. I was able to do so, however, lo and behold, the drive failed on me after a couple of reboots on the PC, and in a particular way. According to the links I found Googling for the Maxtor PowerMax diagnostics error code, it most likely indicates an error with the drive circuitry/electronics.

This seemed to make sense; if perhapse the drive was hanging up when the electronics could not respond to a particular command, that could easily be the culprit. Sadly, I lost one 2 hr recording on the disk, but at least it's something likely to be shown again. So, I arranged for an RMA on that disk (was still under warranty) and switched to another disk I have with recordings on it, also a similar Maxtor model.

Rebooted, followed the Jenkins' guide to get telnetd, tivoftpd, tivoweb-tcl 1.9.4, mfs_ftp 1.2.9p, tserver_mfs7 on the disk. Connected via mfs_ftp, did a ~600 MB transfer fine without a hitch. Things were looking up.

I next tried a ~1.2 GB recording via mfs_ftp (everything so far with the defaults, and tivoftpd and TiVoWeb running alongside). This failed a couple of times, but did not crash the TiVo--just timed out the FTP connection. Unlike before, however, I was able to resume transfers where I left off using FileZilla. This way, I got the ~1.2 GB recording off after 3 tries.

On to the ~1.5 GB, and bam(!), another crash. Exact same symptoms. Now, I've been trying overnight and several times this morning, again, unpredictable crashes along the way. Setting mfs_ftp ithrottle all the way up to 10, limiting download speed to 700 KB/s on the PC side, no luck. Same issue.

In the interim Matt, whose post I mentioned above, got back to me, saying he replaced the TiVoNET adapter and the NE2000 with "the new TiVo NIC card" by which I assume he means a TurboNET, and that his problems were solved.

Though I've tried two different NE2000 NICs so far, the current thinking is to:

o See if I can borrow another TiVoNET/NE2000 clone and/or TurboNET from a friend and try those with the same setup.

o Try the same network hardware and drive on another identical TiVo to eliminate TiVo hardware issues.

Which leaves behind:

o Potential problem with these Maxtor DiamondMax (newer, thinner) IDE drives.

o Potential problem with my 3.0.1 image/binaries somehow (anyone got a virgin/clean 3.1?--yes, I know about the image begging thread).

o Potential, unresolved issue with the TiVoNET and 8390 modules and the current version of TiVo Software.

I'll let people know what I find.

RangerOne

sanderton
06-24-2004, 09:20 AM
Looking like the TiVoNet and drivers are the suspects. Unfortunately very few people run TiVoNet anymore, so you may be out of luck.

Invest in a new CacheCard - they extract faster too!