PDA

View Full Version : Extract mpeg2-ts directly from series 3



bcc
03-24-2008, 02:55 AM
Enclosed is a mips binary of s3tots and diffs to mfs_ftp which allow one to extract unencrypted recordings in mpeg2 .ts format directly. Ie no need to deal with extracting TY/TMF.
The changes to mfs_ftp are minimal, causing it to invoke a wrapper to mfs_uberexport if one attempts to retrieve a .ts file.

The wrapper in turn is a trivial, simply invoking mfs_uberexport and s3tots.

To install:
patch your mfs_ftp.tcl with the enclosed diffs, using patch.
Place s3tots and mfs_tsexport in your mfs_ftp directory.

I suggested this approach a few times in the past and received 0 feedback; this tool is meant as a fully functional proof of concept.

Disclaimer: Minimally tested; warnings about any transmission errors are thrown out by mfs_ftp.

Note that other servers such as dserver could leverage this same approach, not just mfs_ftp.

ronnythunder
03-24-2008, 12:02 PM
sounds good - i'd bet my buddy with a prommed tivohd would like this. we'll check it out.

ronny

tiver
03-24-2008, 12:28 PM
bcc,

Thanks for this.

I am curious - how much transformation/math is involved in going from the tmf to the ts ?

My initial guess is "very little", as I think tivo stores the shows in something very close to an mpeg anyway, and your ability to do it on the fly like that inside the tivo suggests that all you are doing is stripping/replacing headers, etc. (I haven't looked at your code).

But then I am struck by the fairly significant size difference - the ts ends up about 10% smaller than the tmf. Why is this ? Am I incorrect in my assumption of the (small) amount of work I perceive s3tots to be doing ?

StanSimmons
03-24-2008, 01:21 PM
Very cool, I look forward to installing and testing this on my TiVoHD v9.2a soon.

What patches are required to mfs_ftp before your diff is applied? I quit using mfs_ftp on S2-DTiVo boxes when tytool became fairly stable, so I'm not up to speed on it... I still use the original, unpatched mfs_ftp on my old S1 boxes w/ v3 on the rare occasions that I need to pull something off of them.

StanSimmons
03-24-2008, 02:40 PM
How did the download speed using your app compare to using the TiVo approved .tivo download method?

Jamie
03-24-2008, 02:41 PM
I prefer to run s3tots on the client side. For example:
MFS_DEVLIST=":s3" mfs_uberexport -v -R 1676598 | s3tots - foo.tsThis still extracts direct to .ts. I did have to hack the s3tots source slightly to add "-" as a synonym for stdin.

For me, this runs about twice as fast as extracting ts using the mips s3tots, presumably because my PC is faster than my TiVo. I see ~ 70% cpu time going to s3tots in top when extracting .ts via the mips s3tots. So the load isn't negligible. Still, it's a nice alternative to have. Thanks bcc!

captain_video
03-24-2008, 02:57 PM
Always nice to see more tools added to the extraction toolbox. Unfortunately, I stopped using mfs_ftp years ago when I lost track of all the patches required to get it running. Call me old school but TyTools still works great for extraction and it doesn't take a rocket scientist to set it up and use it. Thanks for s3tots so us old farts can convert the ty files to transport streams on the PC side. It's an extra step and can be somewhat time consuming for large files but I usually run it in the background and then walk away from the PC while letting it do its thing.

bcc
03-24-2008, 03:25 PM
bcc,

Thanks for this.

I am curious - how much transformation/math is involved in going from the tmf to the ts ?

My initial guess is "very little", as I think tivo stores the shows in something very close to an mpeg anyway, and your ability to do it on the fly like that inside the tivo suggests that all you are doing is stripping/replacing headers, etc. (I haven't looked at your code).There is no transformation or math required. Complete transport stream packets are contained in the series3 TY format, it's just a matter of unbundling the extra container (and rebundling in some cases, for example tivocast). There's a bit of extra processing to provide transport stream PAT&PMT tables, as those are stripped out of the TY by tivo.


But then I am struck by the fairly significant size difference - the ts ends up about 10% smaller than the tmf. Why is this ? Am I incorrect in my assumption of the (small) amount of work I perceive s3tots to be doing ?There's a bug I identified a long time ago in the extraction tools where they work in terms of the number of pre-allocated chunks, instead of the number of chunks actually recorded. In older tivo models, this results in a small number of extra garbage chunks being extracted. On s3 systems, chunk allocation chaged and this number of garbage chunks can now be quite large. For example, a recording with 521 recorded chunks may transfer as 8192 chunks instead.

I assume this is why you see a large size discrepancy.

I gave Jamie an mfs-utils fix for this back in Sep. but it hasn't been incorporated. I can post the fix here as well.

bcc
03-24-2008, 03:30 PM
VWhat patches are required to mfs_ftp before your diff is applied?I can't keep up with all the scattered mfs_ftp patches myself. As far as I know, a vanilla 1.2.9p patched with jamie's mfs_ftp.jamie_patch from mfs-utils is the only pre-requisite.

bcc
03-24-2008, 03:58 PM
Ok, it sounds like you guys are gonna make me go benchmark this :) I only did 1 test before releasing. This was a proof of concept release, remember? Anyways, my test was on a box that was already busy recording so it's less than ideal. In that test I got 24Mbit/sec, which is enough for real time streaming, on par with TY extraction from a busy box, and already faster than best case .tivo extraction (11Mbit/sec on an unloaded box last I checked).

There is some sanity checking s3tots does which is completely unnecessary if one is throwing out the warnings anyways. This could improve performance. Also, I currently have s3tots streaming from a pipe which likely adds a lot of overhead. I'd rather link with mfs-utils directly but I don't want my code GPL encumbered. If someone would like to sort out the licensing issue, making this work without a pipe would be really easy. LGPL version of mfs-utils maybe?

bcc
03-24-2008, 03:59 PM
I did have to hack the s3tots source slightly to add "-" as a synonym for stdin. I did the same for this project (and also stdout support). Will update the s3tots source release to reflect this.

Jamie
03-24-2008, 04:10 PM
... LGPL version of mfs-utils maybe?You'd have to get the vplay original author (tridge) to sign off on this.

I do think passing that data through a pipe between two processes adds significant overhead to the overburdened tivo processor.

bcc
03-24-2008, 04:20 PM
Call me old school but TyTools still works great for extraction and it doesn't take a rocket scientist to set it up and use it.Ok, you win the cave-man award for still using tytools with an s3. I still occasionally do things such as use a line-editor to edit files but I wouldn't brag about it :) BTW, tytools doesn't run on my linux systems so I never use it.

I personally am happy to do conversions off-tivo as well, but the ease of use crowd has been pushing for a solution like this. It has many benefits such as allowing one to treat an s3 tivo like an HD Homerun box. It eliminates the extra disk i/o overhead involved in TY3->.ts conversion for those who have been throwing stones about i/o overhead.
It also addresses concerns about lack of TY3 support in ffmpeg/mplayer/tyshow. Finally one can contain the arcane knowledge of TY3/TMF3 formats in one place instead of spreading those details into other applications that are better off sticking with standard a/v formats such as .ts.

bcc
03-24-2008, 04:24 PM
You'd have to get the vplay original author (tridge) to sign off on this.I'll defer on that to whomever wants higher performance levels than what one gets via a pipe.

bcc
03-24-2008, 08:07 PM
There's a bug I identified a long time ago in the extraction tools where they work in terms of the number of pre-allocated chunks, instead of the number of chunks actually recorded. In older tivo models, this results in a small number of extra garbage chunks being extracted. On s3 systems, chunk allocation chaged and this number of garbage chunks can now be quite large. For example, a recording with 521 recorded chunks may transfer as 8192 chunks instead.

Here is the mfs-utils change I was referring to. With this change, s3 extraction of unscrambled recordings will only include the number of chunks actually used in a recording, instead of the number of pre-allocated chunks.
Note that I'm picking up the number of used chunks in the recording from the master chunk header, not from the inode metadata. The later is theoretically cleaner, but it turns out this number is left at 0 for recordings that are in progress, whereas the number I'm using is up to date while recordings are in progress.

I've been using this change since last Sept. with no problems. This change works for all of TY/TMF/TS extraction. I recommend this change even if one is not using my new TS extraction.

Also enclosed is a pre-compiled mfs_uberexport binary with the change.

jt1134
03-12-2009, 07:12 PM
Thanks for the mfs_uberexport fix bcc. I've been using it for almost a year now with no problems. Here's a few numbers for anyone who's interested.

I made a 15 second SD test recording. The tivo and its built-in web server reported it at 1.01GB. Filezilla showed it as roughly the same. I first extracted it with the 64bit mfs_uberexport from here (http://dealdatabase.com/forum/showpost.php?p=290822&postcount=10). The resulting file came out at 1080042496 bytes (~1.0GB). Then I extracted it using a version with your fix that I compiled myself. The resulting file came out at 15213568 bytes (~14.5MB). I ran them both through s3tots, and both came out at 15038308 bytes (~14.3MB). So there's definitely plenty of junk getting thrown out.

pdd
03-13-2009, 07:01 PM
I first extracted it with the 64bit mfs_uberexport from here (http://dealdatabase.com/forum/showpost.php?p=290822&postcount=10). The resulting file came out at 1080042496 bytes (~1.0GB). Then I extracted it using a version with your fix that I compiled myself.

Would you mind posting your 64 bit mfs_uberexport here jt1134?

I have not yet got to the point where I can compile my own MIPS binaries yet. Still got alot of reading to do before I get there.

jt1134
03-13-2009, 07:21 PM
Would you mind posting your 64 bit mfs_uberexport here jt1134?
attached.

I have not yet got to the point where I can compile my own MIPS binaries yet. Still got alot of reading to do before I get there.
jamie has posted 2 build scripts that make setting up a cross-compiler on linux/cygwin brutally easy, here (http://dealdatabase.com/forum/showthread.php?t=46361) and here (http://dealdatabase.com/forum/showthread.php?t=54047).

bcc
03-13-2009, 08:15 PM
Great, thanks for building that.

Note that there is a pre-compiled mips cross-compiler environment already available so you don't have to compile it yourself. That's what I've always used actually. It's at:

http://prdownloads.sourceforge.net/tivoutils/usr.local.mips-tivo.tar.bz2?download

pdd
03-28-2009, 08:53 PM
Attached is a mips binary of the most recent s3tots 1.3 which now works with the Australian Tivo's .ty files.

With this plus the mfs_ftp patch which can be found in the first post of this thread you can download .ts files directly from the tivo.

Just patch your mfs_ftp.tcl file and place this binary in the same directory and it should all work.

It took a bit of trial and error getting this to compile... after alot of blindly stumbling around in the dark it eventually worked :) It is twice the size of the original in the first post of this thread??? Probably due to me not knowing what I was doing with the compiler. Either way... it works!!!

Edit: Thanks for that bcc I just stripped out the debug and replaced the binary here.

bcc
03-29-2009, 04:31 PM
It is twice the size of the original in the first post of this thread??? Probably due to me not knowing what I was doing with the compiler. Either way... it works!!!Normally strip is used to remove the debugging symbol tables from a binary before publishing. That'd explain the difference. You won't need those symbols since s3tots isn't known to core dump :)
strip is in the pre-compiled mips-cross compiler distro I pointed you to.

Richard Berg
08-04-2009, 05:10 AM
I can't keep up with all the scattered mfs_ftp patches myself. As far as I know, a vanilla 1.2.9p patched with jamie's mfs_ftp.jamie_patch from mfs-utils is the only pre-requisite.
This doesn't work for me. Jamie's patch applies fine, but trying to add yours on top gives a few "invalid patch" warnings & promptly segfaults.

bcc
08-04-2009, 08:14 PM
This doesn't work for me. Jamie's patch applies fine, but trying to add yours on top gives a few "invalid patch" warnings & promptly segfaults.Works for me & others. That's not enough info to help you any really. The patch is pretty simple you could always apply it by hand.

Pyrotek
12-06-2009, 11:32 PM
Just socketed my HD, it's been a glorious day. :) This was my first SMT rework experience, and was a lot easier then I expected.

It seems that mips s3tots would allow direct streaming from the tivo, similar to vserver functionality in the past. Am I wrong in assuming this?

Perhaps with a client-side cache handler for the codec?

bcc
12-08-2009, 03:50 PM
Just socketed my HD, it's been a glorious day. :) This was my first SMT rework experience, and was a lot easier then I expected.

It seems that mips s3tots would allow direct streaming from the tivo, similar to vserver functionality in the past. Am I wrong in assuming this?

Perhaps with a client-side cache handler for the codec?Sure, you could write code to do that. You'd have to take care to handle the master chunk records that are not filled in completely while the corresponding chunk segment is still being written. You'd have to worry about underrun processing. And you might not have enough CPU/bandwidth for high data rate streams such as OTA recordings that can be up to 19Mbit/sec. Not that I've ever seen anyone broadcast at that rate...

bcc
12-08-2009, 04:15 PM
xxxxxxxxxxxxxx