Proven Concept: Tivo Stream Importing

[AlphaWolf]

First let me say: yes, it finaly has been done without the use of ele2pestripple.

We can export a tystream from one tivo, and import it directly into another, and it works perfectly.

But before I provide details, the forum link I am providing will not now, nor ever, get you anything that you can use for this, that stuff will be posted here when its deemed ready for release. The forum I am about to link to is primarily for developers.

Does that mean you need to be able to write any software? No, god knows that I can't even do that. That forum is only for developmental posts, meaning information that is useful. Questions about "I cant get this to work", "how do I do this", or statements about what you want to do with it, etc, are not allowed there, and will be deleted. You either ask those questions here, or on AVS, when the software is available (which will no doubt be a matter of only a few days at most).

Now getting down to the point, I don't have any software that can do this yet, I can only tell you that its been done, (I know this first hand, and I don't think that I have ever lied to any of you) and that the theory I described has been proven to work. We need some of you talented developers out there to help come up with some code that the end users can use.

http://alt.org/forum/index.php?t=thr...2a2a5d74c62ad1

[AlphaWolf]

Just a quick heads up: If you want to save any tystreams for future use on another tivo, do not extract them using any current tystream extraction tools. The reason why is because the versions of mfs_stream that are going around currently are known to mangle the tystreams to where tivo can't read them. Sometimes some extra garbage gets put into the tystream through stderr, and also the "master chunk" of the tystream (which contains important information about the video) is completely ommited.

The best extraction tool I have seen so far was the one from edpuffedmonster, which simply was mfs_export that queued each tystream part into one big file, without ommiting anything or adding anything, but it seems to have been deleted from the sticky thread.

[captain_video]

I just added a link for mfs_export in the HowTos thread in the File Downloads post. I'm not sure if it's the same one and it doesn't have a readme file with it but there it is. Run a search in the forum to get more info.

[AlphaWolf]

Another heads up, those of you who have extracted scrambled tystreams in the hope that they can be descrambled later are a bit out of luck. Embeem, dtype, and a few others have pointed out to me that for each scrambled tystream is a key thats associated with it, but that key is not stored in the tystream itself. When you extract the stream, you are leaving that key behind.

When you reinsert the stream, its believed that tivo sees no key associated with it and assumes that its a stream that was recorded in 2.0 where streams were not scrambled (the 2.5 upgrade does NOT scramble old recordings, they remain unscrambled), and hence doesn't try to descramble it, nor could it if it even tried since it doesn't have the key (get it?). I have done a few tests to confirm this, so AFAIK, extracted scrambled tystreams are basicaly useless, unless somebody wants to write some code to brute force descramble them. Tystreams that you haven't extracted but are still scrambled aren't useless of course, but as of yet I haven't seen any developer incentive to work on this, nor do I see any reason for anybody to want to work on it, as most of us just assume record the show the next time its on again

[psperry]

ok, probably missed something obvious, but here it is anyway...

set up my nfs server on my xp machine ... works great - have it mounted to /var/tmp/contents.

downloaded rcinsert.tcl and mfs_insert to /var/insert and did chmod +x on both files .

added /var/insert to my path as suggested

while in /var/insert I get the following:


declare -x DEBUG_BOARD="false"
declare -x HDA_ID="354029229304"
declare -x HDB_ID="Unknown"
declare -x HOME="/"
declare -x HOSTNAME="(none)"
declare -x HOSTTYPE="i686"
declare -x IrdSerialNumber="532F0898"
declare -x MACHTYPE="i686-pc-linux-gnu"
declare -x MFS_DEVICE="/dev/hda10"
declare -x OSTYPE="linux-gnu"
declare -x PATH="/bin:/sbin:/tvbin:/devbin"
declare -x PROMVERSION="

TiVoProm Monitor, release version 2.05 (ntsc)
"
declare -x PWD="/var/tmp"
declare -x SHELL="/bin/sh"
declare -x SHLVL="2"
declare -x SerialNumber="0010000202F0898"
declare -x TERM="linux"
declare -x TIVO_REMOTE="TIVO"
declare -x TIVO_ROOT=""
declare -x TIVO_SVR_ADDR="192.168.50.1:80"
declare -x TV_STD="NTSC"
declare -x _="/bin/bash"
declare -x chmem="0x806029C0"
declare -x prom="r"
declare -x root="/dev/hda7"
declare -x runideturbo="false"
declare -x varpartition="/dev/hda9"
bash-2.02# cd /var/insert
bash-2.02# ls
dtv02.ty mfs_insert rcinsert.tcl usage.txt
dtv02h.ty rcinsert.tcl
bash-2.02# rcinsert.tcl /var/tmp/contents 'test' 'test'
bash: rcinsert.tcl: command not found
bash-2.02#

I haven't gotten an error like this since I was a wee newbie; I am still a pretty big newbie so can someone please tell me where I have gone wrong?

[kshep]

Well it works for me on my first single part test file! (SAT-T60, 2.5)

Only hitch was that rcinsert.tcl calls mfs_insert with the argument "dtvh02.ty", but the file as distributed is called "dtv02h.ty".

If you don't rename the file, you'll get a "inserting a dummy header block in fsid #0" error when it gets to the first mfs_insert call.

I scratched my head over that one for a while until I removed the devnull output redirect from the mfs_insert call and saw that it was causing a file not found error.

[AlphaWolf]

as a side note though, I am glad to see that interest in this has picked back up.

[kshep]

Well I really just stumbled into this as I was trying to figure out how I could exchange tystreams with a friend who just got a SAT-T60. I'll admit it's a little tricky following who's working on what, what with all the different threads on a couple different boards.

This seems like the simplest solution around at the moment, and it seems like your further ahead than anybody else, Riley.

I've checked out the alt.org/forum thread, but it doesn't really seem like there's a concerted effort to develop a full blown tool to do this. Am I just missing something?

[AlphaWolf]

It was being worked on, but some of us had other things get in the way. We intend on building a tivoweb module where you can basicaly do a one step extraction or insertion via your web browser, and it automaticaly copies over all of the MFS objects/attributes (the guide data relavent to the stream) and the stream into one single file so that when you reinsert it, its exactly as it was when you exported it.

On a side note, there are also other tools sorta like this under development, I heard mbm and lightn talking about adding a feature to make it possible to back up all of your configurations/settings/thumbs data/season passes, etc etc etc in one step from tivoweb, so well that you could actualy do a clear and delete everything, then put it all back on, and it would be exactly as you left it. Pretty cool IMO as this will leave the possibility of doing all kinds of other things in the future.

Tivoweb definitely has a lot more work going into it. All of this will be in the official tivoweb distributions when they are completed. Don't associate the tivoweb developers with the AVS guys that easily, they have a lot more interests in doing the more "cool" things with tivos than you may think, in fact had it not been for them, our stream insertion project wouldn't be anywhere near where it is today.

[keith721]

I had kinda wondered if all the discussion had moved completely to the IRC channel, since the alt.org/forum message threads hadn't been updated in a couple of weeks. It's nice to hear you guys are having success, and moving forward. Thanks !!

[kshep]

Thanks for this background info.

I might be able to lend a hand either on the full blown tivoweb module side or, if it's not a significant duplication of effort, on a sort of tivoweb module wrapper to your script, Riley. I'm picturing something that scans the source directory looking for .ty files with names in the TyTools "Get Part" format, then presents the user with a list of files to choose from. It could extract the title and episode info from the file, then iterate though the parts, even splitting the program into multiple Now Showing entries if the program has more than six parts. Perhaps before building this as a tivoweb module it'd make sense to do it as a command line tool, then make both versions available.

I'm new to tivo development, but not to Tcl or web development in general. Perhaps building this simple version of an insert module would be good preparation to help in the full blown effort...if you folks could use a hand, that is.

[Torg]

It was being worked on, but some of us had other things get in the way. We intend on building a tivoweb
module where you can basicaly do a one step extraction or insertion via your web browser, and it
automaticaly copies over all of the MFS objects/attributes (the guide data relavent to the stream) and the
stream into one single file so that when you reinsert it, its exactly as it was when you exported it.

This has already been done. I am attaching the ui.itcl I use as well as the program it uses. Be advised this is for tivoweb 1.9.3, but it can be easily added or adopted to newer verstions.

The reason it needs to be in ui.itcl is it has to display links with the "Now Showing" parts of the script.

It is essentialy one line added to put in the export link. Three procs to monitor and do the export. And finaly the last line to register it as a part of the menu.

[pfl2000]

This is great, I backed up this weeks Sopranos (great episode BTW) to my Linux box via NFS, and then restored it back using rcinsert. It worked flawlessly.

I've not tried any of the automation scripts, but would certainly like to see all of the show's information come out with the export and then be restored with the insert.

I've got some free time, if there are any parts to this project that have yet to be tackled let me know and I can give it a try. I know enough TCL to be dangerous and have my cross compiler setup from my mods to elseed and yac.

[AlphaWolf]

If any of you are good at TCL and/or C, we need more assistance in the project.

Torg: We were working on having a file format thats basicaly a tarball containing each individual tystream part, as well as a plaintext file containing an XML dump of the streams guide data. Just one file that contains all of the info, much easier to deal with that way. See alt.org for details.

[drewhallward]

I am getting a file not found error on rcinsert.tcl too, I chmod +x it, it is in my path, I am still getting the error however. Do I need to set the source $tcl_library path to anywhere, I noticed on the second line of the script. I have looked for a while on the error and came up empty, any help is appreciated.