6.4a driving me NUTS - Broken extraction, elseed, TWP SP's

[cwilkins]

...or maybe the subject should be "Hacking your Series 2 in 2009."

The basic question is whether I can fix the broken bits on this 6.4a box, or should I just cut my losses and run screaming back to 6.2?

Background:
I've had a hacked SD-DVR40 dtivo running 6.2 for ages. Upped the drive size three times, stopping at 250GB. Occasional problems with random reboots and taming mfs_ftp, but got all that sorted and life was good. And then recently the second tuner went flaky. From the "check signal levels" interface, you could watch the 2nd tuner signal drop to zero and back to 90% every few seconds...

So, ordered another SD-DVR40 off eBay. It showed up, I pulled the 40gb drive, stuffed an instantcaked/PTVnet'ed 500gb drive in it. Ran tweak.sh, didn't like the results, pulled the drive, re-instantcaked/PTVnet'd, start over. Couldn't get it on the net with a Belkin F5D5050 and the PTVnet drivers, so installed the latest backported drivers for 2.4.20. *poof* Life was good (at 500K/s...) Or so I thought...

And the older 6.2 box has been relegated to the basement where its single tuner, TWP, and mfs_ftp still render it quite useful -- even when managed/used purely remotely, thanks to TWP, MovieLoader and mfs_ftp. (You guys rock!)

v6.4 Hell Ensues:

MRV?: First think I tripped over was no MRV. Ugh. Ok, HME/MovieLoader to the rescue! Still, I think I'd rather have MRV. The HME/MovieLoader interface (at least in 6.4a) is glitchy and clunky. For starters, the folders/shows don't display properly. They appear and disappear at random as you scroll up/down the list. Second, my wife is a heavy user of the Select button when navigating NPL. So she quickly found that pressing Select on an HME/MovieLoader folder has surprising results. (*Poof* multiple queued transfers!) Any thoughts/opinions/fixes?

elseed: Next was battling with elseed. Various versions of elseed, both dynamic and static. All had some issue or other, many of them relating to how OSD works on 6.4a apparently. The static copy was my best bet and my only problem with it is that it will simply "quit working." I'm now using a shell "while loop" wrapped around elseed running with the -e option and it's *still* unreliable. When I started pondering a cron script to kill the running copy periodically, I realized I needed another option/clue/whatever.

Typical elseed debug output:

Initializing modem...'AT+VCID=1
'
DEBUG: Setting alarm for 10 seconds
DEBUG: AT+VCID=1
DEBUG:
DEBUG: OK
Modem: OK
DEBUG: Setting alarm for 10 seconds
DEBUG:
DEBUG: RING
Modem: RING
DEBUG: Setting alarm for 10 seconds
DEBUG:
DEBUG: RING
Modem: RING
DEBUG: Setting alarm for 10 seconds
DEBUG:
DEBUG: RING
Modem: RING
DEBUG: Setting alarm for 10 seconds
DEBUG:
DEBUG: RING
Modem: RING
DEBUG: Setting alarm for 10 seconds
DEBUG:
DEBUG: RING
Modem: RING
DEBUG: Setting alarm for 10 seconds
DEBUG:

I should mention that elseed does work great sometimes. When it quits working seems to be fairly random though. Maybe after 1 call, maybe 5, maybe none.

Season Passes and TWP: Specifically NpkChannelDefinition and the goofiness that causes with the NPL. Good grief! Using the latest TWP I could find (2.1.b3 oztivo-090522) seems to have improved things slightly, but it appears I still can't reliably set up an SP from TWP.

I've deleted the errant TWP season passes and I'm still getting stuff like this in /var/log/tverr every time I try to navigate NPL into the detail for certain (not all) shows:

Oct 19 22:12:10 (none) findActionsInfo[278]: FindProgramSourcesForRecordingNoRetry: Can't get display title, err: 0xffff
Oct 19 22:12:10 (none) findActionsInfo[278]: FillInRestNoRetry: FindProgramSourcesForRecordingNoRetry, err: 0xffff

Pressing Play still works thankfully.

Extraction: First, I've used MovieLoader successfully for imports/exports to/from the 6.4a box. So there is that. But extraction to the PC is giving me grief. Whether I use mfs_ftp (same heavily patched 1.2.9p-pl-20070717b I have working nicely on 6.2) or TWP (2.1.b3 oztivo-090522), the result is the same. Lots of complaining from tytools 10r4 along these lines:

T00:23:33.317
    largestVideoPTS is: 00:23:32.683
BBB the PTS was bad, but the new SEQ check lines up|!?
Found an OOB packet... The Audio Diff is: 00:00:00.744
               sAud is: 00:23:32.599
    largestAudioPTS is: 00:23:31.855
Is it in sequence??? It is OFF by exactly 31.000000 frames.
It is in the right range. Let's see if it lines up.
It is in sequence. Starting back up after the 'hole'...
... 2000

.
Found an OOB packet... The Video Diff is: 00:00:00.367
               sVid is: 00:23:52.586
    largestVideoPTS is: 00:23:52.219
BBB the PTS was bad, but the new SEQ check lines up|!?
Found an OOB packet... The Audio Diff is: 00:00:00.408
               sAud is: 00:23:52.183
    largestAudioPTS is: 00:23:51.775
Is it in sequence??? It is OFF by exactly 17.000000 frames.
It is in the right range. Let's see if it lines up.
It is in sequence. Starting back up after the 'hole'...
.
Found an OOB packet... The Video Diff is: 00:00:00.300
               sVid is: 00:23:56.223
    largestVideoPTS is: 00:23:55.922
BBB the PTS was bad, but the new SEQ check lines up|!?
Found an OOB packet... The Audio Diff is: 00:00:00.432
               sAud is: 00:23:55.903
    largestAudioPTS is: 00:23:55.471
Is it in sequence??? It is OFF by exactly 18.000000 frames.
It is in the right range. Let's see if it lines up.
It is in sequence. Starting back up after the 'hole'...
.

So I'm guessing yet another stumbling block between 6.2 and 6.4a? Funny thing is, a 6.4a show exported to the 6.2 box plays fine, no glitches in the video. But if I extract a 6.4a show from either dtivo to the PC, I get the same complaining in tytools (above). So is reliable extraction for tytools consumption possible? Do I need to use something besides tytools on my WinXP/Vista PCs?

Fish or cut bait?

Do I need to run from v6.4a and not look back? (I think I can survive without a "Recently Deleted" folder.) And of course if the answer is "yes, get out while you can!!", the next question is whether I can extract or otherwise save shows from 6.4a before/during the downgrade.

One nice thing was I didn't have to monte 6.4a. I think I monted (it's been awhile) 6.2 to get LBA48 support and to turn off iptables support for faster networking. I can live without faster networking (heck, I'm using a USB 1.0 network adapter), but did 6.2 include LBA48?

Any clues appreciated -- after two+ solid weekends of this nonsense, my googlefoo is worn down. I'm frazzled. Feel free to just point me at what I couldn't find, or if no one's really pulled all these 6.4 issues together in one thread yet, I'll do my part to help fill in all the bits I can.

Edit: Yes, I know about the "6.4 has arrived" thread - just read through all 161 posts. I've also read through various tytool, elseed, MovieLoader, mfs_ftp, and extraction-related threads. Sure I probably overlooked something, but it's not a case of "I don't feel like searching, please just tell me what to click on and/or type." I'm still searching while hoping for a reply here, and will update here with any located clues.

Sidebar:

Instantcake/PTVnet... Did that because I "didn't have time to hack by hand" and I guess it was still quicker than starting from scratch, but not without its pitfalls and aggrevations. I found myself spending much time undoing/reworking things that apparently someone at DVRUpgrade thought were clever ideas. The automatic behind-the-scenes twiddling of bootpage (in /init/002_62start.init) that kept undoing my faster serial port settings was a prime example. And be warned about running tweak.sh, especially since much of what it does conflicts with or otherwise gums up the PTVnet layout. (Wanna guess what rc.sysinit.author looks like after that? Not pretty!) Not sure what they were thinking there, but kind of reminds me of what Redhat/Fedora do to Linux, and I'm used to that.

-cw-

[cwilkins]

HME / MovieLoader:

A reboot fixed the screen painting glitchiness with the Movieloader Folder / Show menus. [headscratch...]

Elseed / CallerID support:

I don't know if it's a Trinity mainboard thing or what, but the modem on this SD-DVR40 does not do CID reliably. My final, failed attempt to use the internal Tivo modem involved NCID (ncidd & tivocid), and same problem -- it would work fine once, twice, five times, then just quit giving CID data.

However, I do have a working, reliable CID setup now. Yay! It involves tivocid running on the Tivo and ncidd running on one of my Linux servers, with an old SupraFaxModem plugged into it.

The only downside I've found is that tivocid (ncid) apparently doesn't do any CID logging, so TWP has no handy "local" CID log data to display. It seems TWP will work with an FTP URL and pull CID data from that, so I'm hoping to make the CID log on the Linux server available to TWP on the Tivo via FTP. It's just that easy!

For anyone else in this same pickle (but lacking a handy Linux server), apparently some folks have enjoyed success running a copy of ncidd on the Tivo, pointed at the Tivo serial line wired to an old external modem that supports CID. Not sure what that means for normal serial console use...

TWP Season Passes vs. v6.4a:

No luck locating a fix yet, and the mess left behind in MFS from previous attempts to create SP's from within TWP mucked up various aspects of accessing recorded shows, and I was getting Error #32 all over the place. Finally did a "Clear program information & To Do list" and an hour later, sanity was restored.

So TWP SP's and 6.4a... Seriously, years have gone by and nobody's cracked this nut yet? Did everyone who placed a priority on that function just go back to 6.2, or decide creating SP's remotely wasn't that important, or what?

Inquiring minds want to know!

Extraction Adventures:

A few more clues, but still no joy. I can move shows between the 6.2 and 6.4a Tivo using MovieLoader and they play fine. But shows extracted to PC from either Tivo retain the characteristics of whichever version Tivo they were recorded on. That is, a show recorded on the 6.2 Tivo can be extracted from either Tivo and will convert/play fine. A show recorded on the 6.4a Tivo will not convert/play properly on the PC, regardless of which Tivo I extract it from. .ty files, .ty+ files, .tmf files... mfs_ftp or TWP... Doesn't matter. Given a little more time, I'll take some of these files apart and have a closer look.

So all that and not a peep out of anyone? Really? I'm left wondering if despite my searching, these problems are so well known to the regulars that I'm just another drooling noob not worth responding to, or maybe, possibly, could it be everyone's stumped? ...or I'm the last guy running an S2 dtivo? ...or some combination of the above?

Any clues appreciated, or if/when I find one, I'll beat it into submission and drag it back here in hopes of making the next poor slob's life a little easier.

-cw-

[darrin75]

Well i upgraded recently to 6.4a, even though I like some of the features I decided its not for me. No mrv, or tivoserver for mrv. Even though 6.4a looks and feels nicer, it does not hold a bucket of water to 6.2 or 6.2a for that matter. 6.2 is more stable, and does everything I would want..So goodbye 6.4a. For now at least..

[whitepelican]

The HME/MovieLoader interface (at least in 6.4a) is glitchy and clunky. For starters, the folders/shows don't display properly. They appear and disappear at random as you scroll up/down the list.

This is a problem caused by Elseed, not MovieLoader. I had to give up trying to run YAC or Elseed with MovieLoader because they both cause problems with the display. Stop running Elseed and you will see that MovieLoader appears perfectly.

Second, my wife is a heavy user of the Select button when navigating NPL. So she quickly found that pressing Select on an HME/MovieLoader folder has surprising results. (*Poof* multiple queued transfers!) Any thoughts/opinions/fixes?

It sounds like you aren't running the most recent version of MovieLoader if you're having these problems. I remember complaining about my wife doing the same thing and GFB107 made changes to MovieLoader to prevent the "re-queueing" problem. Are you running MovieLoader v.0.17?

[cwilkins]

Well i upgraded recently to 6.4a, even though I like some of the features I decided its not for me. No mrv, or tivoserver for mrv. Even though 6.4a looks and feels nicer, it does not hold a bucket of water to 6.2 or 6.2a for that matter. 6.2 is more stable, and does everything I would want..So goodbye 6.4a. For now at least..

Thanks for the input darrin75!

That makes sense and there seems to be a consensus along those lines. People don't prefer 6.4a, they put up with it. So it seems pretty clear that my 6.2 Tivo is staying at 6.2 for now. No "upgrade" for that one. Not gonna!

I'd probably like MRV better than HME/MovieLoader, but honestly I've never seen MRV. All I know it is shows up on the NPL which is more intuitive right off the bat. Have you used both? Any glaring differences come to mind?

As I continue to beat my head repeately against 6.4a, I'm kind of getting used to to pain. Well, most of it...

In short:

1. I have not (yet) run into stability issues. (Maybe that was 6.4 and mostly fixed in 6.4a?)
2. I've got CallerID tamed nicely now. (ncid)
3. There's promise of smoothing over the HME/MovieLoader problems (thanks to whitepelican).
4. I can survive without twiddling Season Passes in TivoWebPlus. Yeah, it's a cool and useful feature to be sure, but practically speaking, as long as I can at least remotely set up a new show to record, that'll hold me until I get home. (And who knows, someone may figure out how to fix that pesky NpkChannelDefinition problem!)

Then there's extraction...

Did you get as far as extraction, or go back to 6.2a before trying that? This one is the real deal-breaker feature for me. Extraction (and going LBA48 for more space) was the main reason I started hacking Tivos in the first place -- the ability to backup/watch shows on the PC, and burn shows to DVD to free up space on the Tivo. And nowadays, converting shows to mp4 and dropping them on an iPod is pretty useful too. So if I can't get extraction working, then it's definitely back to 6.2.

But here's a nice dose of irony... I've been running 6.4a for a few weeks now on our main Tivo -- long enough to accumulate some stuff I don't want to just blow away as part of a downgrade to 6.2. Meanwhile I've got issues with extraction. Extraction/Insertion is probably my best option for handling the content I don't want to lose during the downgrade. Yet... (and here's the irony) If I solve the 6.4a extraction problem, I may be ok staying with 6.4a on that Tivo. (It's a question of which path is least painful...)

My brain hurts... I blame DVRUpgrade for selling me a 6.4a InstantCake without appropriate warnings -- not even an obvious option for getting other versions instead.

And I blame society. Society made me what I am!!

-cw-

[tivo4mevo]

I believe that jkozee posted a patch for the NpkChannelDefinition issue, but it still takes a little while for the season pass to kick in (unless you poke it by fiddling with some other season pass). There's discussion here: link, and jkozee's patch here: link (which appears to have been committed).

Regarding extraction, I am assuming that you're trying to extract with mfs_ftp? If so, mfs_ftp requires patching, which can get messy. I'm assuming you read this thread, but if not, it might be worth a look. It provides nice links to patches: link.

Short of that, have you looked into whether or not tserver will work for extraction?

[cwilkins]

This is a problem caused by Elseed, not MovieLoader. I had to give up trying to run YAC or Elseed with MovieLoader because they both cause problems with the display. Stop running Elseed and you will see that MovieLoader appears perfectly.

Ahhh... It all makes sense now. Thanks for clearing that up! I had already quit using Elseed in favor of ncid, but I hadn't rebooted yet. Apparently it was the reboot some days later that actually cleared up my HME/Movieloader display problems. ncid/out2osd seems to work very nicely, but the jury is still out -- haven't yet gotten a call while in a menu, which I've read can be dodgy.

It sounds like you aren't running the most recent version of MovieLoader if you're having these problems. I remember complaining about my wife doing the same thing and GFB107 made changes to MovieLoader to prevent the "re-queueing" problem. Are you running MovieLoader v.0.17?

Yes, I'm up to v0.17. Honestly I'm now scratching my head over this one now. I can't get a coherent description from the females in the house on how exactly they managed to get multiple queued transfers, and my earlier theories were wrong. I have not been able to recreate the problem. So I'll just have to wait until they manage to do it again, then grill them for information under the bright lights.

-cw-

[djl]

Just wondering.. did you get the cid log straightened out too? FTP actually works well for a few people that can't run an ncid server on their Tivo. You don't have VOIP by any chance? Then you could bypass the modem entirely.

HME / MovieLoader:

A reboot fixed the screen painting glitchiness with the Movieloader Folder / Show menus. [headscratch...]

Elseed / CallerID support:

I don't know if it's a Trinity mainboard thing or what, but the modem on this SD-DVR40 does not do CID reliably. My final, failed attempt to use the internal Tivo modem involved NCID (ncidd & tivocid), and same problem -- it would work fine once, twice, five times, then just quit giving CID data.

However, I do have a working, reliable CID setup now. Yay! It involves tivocid running on the Tivo and ncidd running on one of my Linux servers, with an old SupraFaxModem plugged into it.

The only downside I've found is that tivocid (ncid) apparently doesn't do any CID logging, so TWP has no handy "local" CID log data to display. It seems TWP will work with an FTP URL and pull CID data from that, so I'm hoping to make the CID log on the Linux server available to TWP on the Tivo via FTP. It's just that easy!

For anyone else in this same pickle (but lacking a handy Linux server), apparently some folks have enjoyed success running a copy of ncidd on the Tivo, pointed at the Tivo serial line wired to an old external modem that supports CID. Not sure what that means for normal serial console use...

[cwilkins]

I believe that jkozee posted a patch for the NpkChannelDefinition issue, but it still takes a little while for the season pass to kick in (unless you poke it by fiddling with some other season pass). There's discussion here: link, and jkozee's patch here: link (which appears to have been committed).

I followed your link into the TWP thread through to the end, to get a good idea of what kinds of changes were being made, up to current. Ultimately I grabbed the cvs-mips binary, stuck it in my PATH, and after beating on TWP at length, was able to use the cvsupdate module to pull the very latest. So far (fingers crossed), I seem to have regained the ability to create SP's via TWP (with the caveat of needing to give the SP a gentle nudge by twiddling a priority via the TiVo UI).

Regarding extraction, I am assuming that you're trying to extract with mfs_ftp? If so, mfs_ftp requires patching, which can get messy. I'm assuming you read this thread, but if not, it might be worth a look. It provides nice links to patches: link.

Short of that, have you looked into whether or not tserver will work for extraction?

I've used mfs_ftp (1.2.9p-patchlevel-20070717b -- works great on v6.2) and TWP, which I think both use mfs_extract. This is the next thing I need to really dig into and try both tserver and other conversion tools on the PC (Tytools 10r4 is all I've ever used).

Pretty sure this isn't a problem with the extraction tools, but rather, there's something new in the 6.4a Tystreams that at least Tytools 10r4 muxing can't deal with. Of course I could be completely wrong. All these problems with v6.4a are new to me and a complete surprise, hence the frantic posts...

Thanks for the tips/links!

-cw-

[cwilkins]

Just wondering.. did you get the cid log straightened out too? FTP actually works well for a few people that can't run an ncid server on their Tivo. You don't have VOIP by any chance? Then you could bypass the modem entirely.

Nope, plain ol' copper-based telephone service here. (Ah, so the SIP stuff is for VoIP support, cool!)

Haven't tried FTP yet, definitely on my Todo list. Thanks for confirming it's a workable option.

The past couple of days have been spend upgrading my daughter's 3.1.1b dtivo to 6.4a. (After much fretting about 6.2 vs. 6.4) Then I went to import some of her shows back from the archive on a Linux box to other Tivo's. They would play fine until the import was completed. After that I'd get an error trying to play them. Heard this one before? The 3.1.1b box had the kmem hack, so the streams were unscrambled, but all the CSO keys were still in place. Ouch.

And if anyone wants a Perl-based, off-Tivo batch method for removing CSO keys from .tmf files, I posted it here.

-cw-

[tas3086]

Nope, plain ol' copper-based telephone service here. (Ah, so the SIP stuff is for VoIP support, cool!)

Haven't tried FTP yet, definitely on my Todo list. Thanks for confirming it's a workable option.

The past couple of days have been spend upgrading my daughter's 3.1.1b dtivo to 6.4a. (After much fretting about 6.2 vs. 6.4) Then I went to import some of her shows back from the archive on a Linux box to other Tivo's. They would play fine until the import was completed. After that I'd get an error trying to play them. Heard this one before? The 3.1.1b box had the kmem hack, so the streams were unscrambled, but all the CSO keys were still in place. Ouch.

And if anyone wants a Perl-based, off-Tivo batch method for removing CSO keys from .tmf files, I posted it here.

-cw-

See post http://www.dealdatabase.com/forum/sh...postcount=1312 through 1318 for more information, but here is a summary of how to fix mfs_ftp:

Comment out the "CommercialSkipOffset { append info(parts_csos)..." in the "foreach csoc insert grouping in set_rec_csos proc". I'd prefer to make a patch with the correct check, rather than just going with the temporary solution. This is a quick fix to MFS_FTP's set_rec_csos proc that fixed the problem, and has been working for years! All it needed was to have one existing line commented out. Transfers now are working fine for my situation.

A better solution would be to test the old xml version number in the header, and see if it is a down level version, then ignore the csos insert.

[cwilkins]

See post http://www.dealdatabase.com/forum/sh...postcount=1312 through 1318 for more information, but here is a summary of how to fix mfs_ftp:

Comment out the "CommercialSkipOffset { append info(parts_csos)..." in the "foreach csoc insert grouping in set_rec_csos proc". I'd prefer to make a patch with the correct check, rather than just going with the temporary solution. This is a quick fix to MFS_FTP's set_rec_csos proc that fixed the problem, and has been working for years! All it needed was to have one existing line commented out. Transfers now are working fine for my situation.

A better solution would be to test the old xml version number in the header, and see if it is a down level version, then ignore the csos insert.

Ok, so I could've commented out a single line of code in mfs_ftp instead of writing 243 lines Perl script. Apparently I'm having way too much fun. *sigh*

You definitely hit on the sweet solution though -- having the insertion/extraction tools clean that crap out automagically. That hadn't occurred to me at all until you mentioned. After having a "DUH!!" moment, I started looking at mfs_ftp to see how do something more "elegant", but alas I don't know TCL that well yet.

As I understand it, the basic foolproof process is:

Check for magic number "f5477abd" (hex) as first 4 bytes of each part. If found, the stream is not encrypted and any CSOs can be safely removed.

I'm far from fluent in TCL, but I was able to get enough clue (the above process) out of AlphaWolf's ciphercheck and CSOScout programs to build an "offline" Perl tool that safely strips CSOs out of .tmf files. I was feeling pretty smug after putting that together, but now, not so much...

In theory, it wouldn't be hard to stick something like that in both the insertion and extraction routines in mfs_ftp. That way, the user is covered in either direction. They wouldn't have to fret at all about CSOs and whatnot, unless the stream is actually encrypted. And if I understand correctly, they're still fine as long as they don't need the streams to be portable to other Tivos and provided of course that the CSOs are left intact.

After looking over the mfs_ftp code (I can now read TCL more or less. Writing it is another story), it looks like it would be pretty easy to stick in some code to safely deal with unwanted CSOs while leaving needed CSOs in place. Worst case is a more "surgical" implementation of CSOScout is called on the recording just before it's extracted, or just after it's inserted.

This may be just my own stupid pet peeve, but what I didn't like about CSOScout and ciphercheck was that they call the mfs_export binary, simply for the purpose of getting the first 4 bytes from the tystream parts. Sure it gets the job done, but now there's an extra dependency to worry about (which bit me). There's got to be a way to get four bytes from a tystream, natively within TCL. Right? I just don't have a clue how yet.

-cw-