Direct extraction from a tivo drive in a pc (not networked)

[das Monkey]

The edit to the first post indicates that the binaries will not work with 3.1. I wandered through the steps to this process as best as I could piece them together with a 2-drive 3.1 setup (booted into MFSTools with "vmlnodma hda=bswap hdb=bswap") and got as far as the mfs_streams command, which replied "Cannot find fsid=1". I'm not sure if that error is related to 3.1 or a flaw elsewhere in the process, but I very much need to access some recordings on a drive pair that will not boot up for whatever reason (boot cycles at "Updating Database"). Is there another method I should use for a 3.1 setup? I have a T60. I do not have a linux machine, but I am familiar with Knoppix.

I would very much appreciate some assistance with this. I am willing to try most anything. Thank you.

Oh, I did have mfs_ftp running, but for the life of me, I cannot find that cache directory.

das

[jonbig]

I have just finished hacking together bits and pieces from various toolsets to create a new tool that extracts recordings in .tmf format directly from a TiVo drive attached to a PC. It's based on mfs_tarstream (used by mfs_ftp) and the vserver-1.2 distribution. It contains C code that generates the needed showing.xml part of a .tmf file directly from the MFS partitions on the drive.

The nice thing about this tool is that it works for extracting scrambled recordings that you can then upload back to your TiVo via mfs_ftp. At least it worked for the first recording that I just extracted from my non-bootable TiVo drive.

Give me a couple of more days to test and clean up the source and I'll release it.

I guess I are now a certifiable TiVo hacker!

[FredThompson]

Are you saying this is an all-in-one package to recover stored recordings from "dead" drives or drive sets?

Linux? Windows? Mac? FAT? FAT32? NTFS?

[jonbig]

Are you saying this is an all-in-one package to recover stored recordings from "dead" drives or drive sets?

Linux? Windows? Mac? FAT? FAT32? NTFS?

It runs on Linux currently and accesses the drives directly, so you don't need any hacks to be able to understand the TiVo disk partitioning. I'm not familiar with running the tools on MacOS X, but if vserver runs there, it probably would too.

You can store the extracted stream anywhere you can access from Linux, as long as the filesystem supports files the size of the stream you are extracting. So FAT32 will fail for streams > 2GB.

I will be releasing the source, since the sources I worked with are GPL'd.

[AlphaWolf]

I have just finished hacking together bits and pieces from various toolsets to create a new tool that extracts recordings in .tmf format directly from a TiVo drive attached to a PC. It's based on mfs_tarstream (used by mfs_ftp) and the vserver-1.2 distribution. It contains C code that generates the needed showing.xml part of a .tmf file directly from the MFS partitions on the drive.

Excellent, this is what I have been trying to push for for a very long time, but lack the knowledge of the C language to do myself.

I hate to just request features out of nowhere, but here are a few things that you might consider adding later on down the line if you don't already do them (no other TMF/tar utilities do it thus far):

Detect if the tystream is scrambled by checking the first four bytes of each tystream part. If its unscrambled, the first four bytes will be: 0xF5467ABD. Anything else will be a scrambled stream. The purpose of doing this is that you don't need to include the CommercialSkipOffset attribute, that way when you insert it into another tivo, it will play it back whether the tivo is in noscramble mode or not. (including CSO makes the stream fail in scramble mode)

Also, if you want to keep scrambled recordings workable from a dead tivo hard disk, you might want to include a means of getting the /State/Media.DiskConfiguration attribute (you can include it in the TMF's XML data outside of the normal object tags, whatever reads the TMF can decide what it wants to do with it from there, be that to ignore it or whatever else.)

[FredThompson]

Yup, this does look promising.

Does it support a 2 drive pair or just 1 drive?

How will it be packaged? May I suggest 2 ways; Linux executable (and source) and an ISO with a minimum install of a recent Linux build to auto-load for people who aren't Linux saavy? (You know, similar to how people use MFS Tools to enlarge their drive space. It protects Linuxophiliacs from hurting themselves.)

[AlphaWolf]

I would just have sleeper include it with his bootable ISO.

[FredThompson]

Good idea.

[jonbig]

Detect if the tystream is scrambled by checking the first four bytes of each tystream part. If its unscrambled, the first four bytes will be: 0xF5467ABD. Anything else will be a scrambled stream. The purpose of doing this is that you don't need to include the CommercialSkipOffset attribute, that way when you insert it into another tivo, it will play it back whether the tivo is in noscramble mode or not. (including CSO makes the stream fail in scramble mode)

Also, if you want to keep scrambled recordings workable from a dead tivo hard disk, you might want to include a means of getting the /State/Media.DiskConfiguration attribute (you can include it in the TMF's XML data outside of the normal object tags, whatever reads the TMF can decide what it wants to do with it from there, be that to ignore it or whatever else.)

I'll add the unscramble detection to my TODO list.

I just used the mfs_dumpobj tool to get the DiskConfiguration attribute from my broken drive and hand edited it into XML format. That's pretty trivial, so the need for a tool is minimal.

[jonbig]

Yup, this does look promising.

Does it support a 2 drive pair or just 1 drive?

How will it be packaged? May I suggest 2 ways; Linux executable (and source) and an ISO with a minimum install of a recent Linux build to auto-load for people who aren't Linux saavy? (You know, similar to how people use MFS Tools to enlarge their drive space. It protects Linuxophiliacs from hurting themselves.)

I have a patch to the mfs tools that handles 2 drives properly. I wrote my own before I discovered somone else wrote one too.

I just set MFS_DEVLIST to '/dev/hdc /dev/hdd' and everything works great.

I will talk to Sleeper eventually about including it. I had to make a couple of other patches to the vserver-1.2 distribution mfs source as well, so maybe I can coordinate to get a new release of that package to include my tool.

[FredThompson]

I have a patch to the mfs tools that handles 2 drives properly.

There are 2 S1 drives sitting on a shelf here which are now very, very excited. Uh...so am I

[jonbig]

Testing on my new tool is going well, but I'm finding various recordings are missing different MFS object attributes, (most make sense, but a few seem odd), so I'm having to go back and update the source to handle the non-existence of these attributes.

Does anyone have a good pointer to documentation on the MFS object schema and .tmf that might indicate what attributes are *absolutely* necessary vs which ones could be left out without messing up the extracted .tmf?

[rc3105]

title, episode, description & duration are really the only biggies. the first 3 for convenience & the last so the playback bar is accurate

the DC key isn't really the sort of thing you want embedded in each recording, a seperate util/script to extract it would be simpler all around

[AlphaWolf]

the DC key isn't really the sort of thing you want embedded in each recording

Why not? It wouldn't hurt anything. Just only add it if the stream is known to be scrambled, much like how CSO keys should be handled.

[rc3105]

Why not? It wouldn't hurt anything. Just only add it if the stream is known to be scrambled, much like how CSO keys should be handled.

that's great if you only have 1 scrambled stream, or all your scrambles have the same dc key. if the insert util restores a key from a scramble then anything else on that drive scrambled with a different key just quit working... (and did it backup the old key? did it know which streams were associated with the old key?)

for more than one key or a few streams what you're proposing requires a full blown caching system to keep track of which key goes with which stream, then substituting it in when tivoapp tries to play it AND a reboot for it to take effect. alternatly you can tweak noscramble to directly provide tivoapp with mfs_scramble_stream style keys looked up from a cache

any way you slice it it's a headache. got utils to do it either way, but releasing that tarbaby was not one of my new years resolutions...