Proven Concept: Tivo Stream Importing

[OvitivO]

rc3105: very nice.

Is there a console script out there that can simply list the now showing fsids (as opposed to the video blocks)?

(i.e., I don't use TivoWeb, and I'm looking for methods to do all of my extraction/insertion via shell.)

[OvitivO]

Stop looking over my shoulder when typing.

NowShowing.tcl is what I originally used, but I had an error with using what I thought was the proper fsid.

I'll generate the error again and post.

Well, it wasn't exactly an "error," but more of an oddity that will likely not work for re-insertion.

Everytime I use the script I *always* get px.ty file sizes of 1310720 bytes.

# /tvbin/tivosh rcxtract-with-info.tcl 1054287
madedir /var/bols/{Edited Title}/{Edited Episode}
created /var/bols/{Edited Title}/{Edited Episode}/nsinfo.dbt
{Edited Title}
{Edited Episode}

3600
12011
12012
AutoSyncNo
original air date 11-20-2002
1060329
536870912
1060331
536870912
1060333
343932928

# ls -al /var/bols/{Edited Title}/{Edited Episode}
total 3864
drwxr-xr-x 2 65534 65534 4096 Nov 29 05:12 .
drwxr-xr-x 3 65534 65534 4096 Nov 29 05:12 ..
-rw-r--r-- 1 65534 65534 160 Nov 29 05:13 nsinfo.dbt
-rw-r--r-- 1 65534 65534 1310720 Nov 29 05:12 p1.ty
-rw-r--r-- 1 65534 65534 1310720 Nov 29 05:12 p2.ty
-rw-r--r-- 1 65534 65534 1310720 Nov 29 05:13 p3.ty

Unless you've reversed entropy, I don't see how those 3 tystreams are going to provide the data necessary to re-insert a recording back into a Tivo.

[compwiz312]

Hi, I have just been reading all of this and I was wondering, is there any way for me to convert MPEG2 files (such as DVD or SVCD encodes) to Tivo streams so that I could import them successfully to my Tivo?

Thanks for your help,
Justin

[AlphaWolf]

I think we just need to finnish the tivoweb interface

EDIT:

Speaking of which, have you used tivodvlprs mfs_tarexport yet? It already extracts shows perfectly in our tar format that we are going to use. May want to start working on code that will take that tarfile and rebuild its contents (xml file typarts and all) back into MFS instead.

[OvitivO]

An extraction with this script will take roughly 2 hours for every 512MB segment, however a full insertion will take roughly 15 minutes per 512MB segment.

What could possibly be causing this discrepancy?

[OvitivO]

Note the timestamps:

-rw-r--r-- 1 nfsnobody nfsnobody 512M Dec 4 02:59 p1.ty
-rw-r--r-- 1 nfsnobody nfsnobody 512M Dec 4 04:55 p2.ty
-rw-r--r-- 1 nfsnobody nfsnobody 512M Dec 4 06:48 p3.ty
-rw-r--r-- 1 nfsnobody nfsnobody 512M Dec 4 08:41 p4.ty
-rw-r--r-- 1 nfsnobody nfsnobody 512M Dec 4 10:34 p5.ty
-rw-r--r-- 1 nfsnobody nfsnobody 512M Dec 4 12:27 p6.ty

[OvitivO]

Beefy Linux Box <-> Tivo (i.e., no XP)

I'll check with some additional mounts to see if I have the same problem with writing, but that would seem odd if that were the case. (btw, I remembered that mfs_stream by itself does not have this problem.)

One other issue I have (which may be related) is that during an extraction, if I have the Tivo turned "off," then I am not able to power the unit back up until the extraction is finished.

In addition, when I telnet to the Tivo (during an extraction) and attempt to run something like tivoftp, I get a fork error.

# /var/hack/tivoftpd 
tivoftpd: error forking; Resource temporarily unavailable

The only other thing I've noticed is that if I kill the extraction before it is finished (e.g., ^C), then the Tivo will reboot.

[keith721]

mine is mounted using 8K buffers, and seems to perform reasonably well (Sony SVR-2000 Standalone TiVo)

bash-2.02# mount -t nfs -o rsize=8192,wsize=8192 192.168.1.10:/g/ /mnt/nfs/g

[OvitivO]

Good point, my buffer sizes were set to 4096. I'll test again w/ 8192.

EDIT: 8192 definitely improved the performance

[AlphaWolf]

Just want to pass word around that we are open for suggestions as to what we should name the extension for our tarballed tystream format.

In case anybody doesn't know how it works yet, basicaly its a unix .tar (Tape ARchive) file that contains each tystream part as an individual file, along with an XML'ed dump of all of the information in your tivo thats relavent to that specific recording that you have extracted. This was origionaly designed with re-insertion in mind, but it also has other uses. Namely, it could help automate the title information when you move the tystream to another medium - such as DVDs. This is a very clean, neat format. Much so, that anybody who wants to tinker with it can use an everyday application such as winrar, winzip, or a number of mac/unix applications, to alter its contents. Not only that, but every time you extract a tystream, you get all of this in just one neat little file, no directories or a mess of specialy named files that make a more complicated way of accomplishing the same thing, which makes dealing with tystreams a lot easier for the user. And yes, for once you can have peace of mind that files extracted into this format are perfect and can be used for anything we can forseeably do with tystreams later.

Naturaly, I also (passively ) suggest that anybody who writes any software that processes tystreams in any way support our new format once it becomes finalized (probably already is finalized actualy, it is a pretty simple format when you think about it, just needs a name). The full source codes to everything we have done so far can be found on alt.org.

[FreydNot]

How about .TarVo ? It kind of rolls off the toungue

[olaf_sc]

Do you know if there is a lib that has functions to parse a tar file? I would be pretty nice if we could read the tystreams directly in the tar files. It would aviod the unpacking and I could e.g. demux directly from the tar file.

Cheers Olaf

PS: Yes it's open etc but I rather not code the parser if I can avoid it

[jkeegan]

Hmm..

.tya (TYstream Archive)
.tyb (TYstream Bundle)

(I think I like tya better)

..Jeff

[kshep]

I'll nominate:

.tva

...which you could read as TiVo Video Archive or just TiVo Archive.

I can imagine a "tva" file would have a required XML data file up front that includes not only program information but could also specify the format(s) of video segments to follow. These could be tystream chunks, "joined" tystream files ala TyTool, or mpeg files. In fact, a TiVo Archive (tva) file could contain multiple programs...in mixed formats, even...so the format could be used to archive not just a single program but an entire TiVo.

You might even generalize it a bit to something like ".var" (ala ".tar" and ".jar") and submit it as a W3C standard for video archives.

BTW, I'd be happy to start using this format in code I write...if someone would publish a draft DTD.

[AlphaWolf]

Well, bear in mind its also not realy intended as an archive. The intent is that one file holds all of the information relavent to one recording. We could have just kinda globbed it all together in a single file, but felt a tarball would be a nice way to separate each peice of the recording.

You can see what we have so far by downloading tivodvlprs tarexport server from alt.org, it will extract a tystream in our format through tivoweb.