Page 1 of 2 12 LastLast
Results 1 to 15 of 26

Thread: Extract mpeg2-ts directly from series 3

  1. #1
    Join Date
    Nov 2002
    Posts
    1,076

    Extract mpeg2-ts directly from series 3

    Enclosed is a mips binary of s3tots and diffs to mfs_ftp which allow one to extract unencrypted recordings in mpeg2 .ts format directly. Ie no need to deal with extracting TY/TMF.
    The changes to mfs_ftp are minimal, causing it to invoke a wrapper to mfs_uberexport if one attempts to retrieve a .ts file.

    The wrapper in turn is a trivial, simply invoking mfs_uberexport and s3tots.

    To install:
    patch your mfs_ftp.tcl with the enclosed diffs, using patch.
    Place s3tots and mfs_tsexport in your mfs_ftp directory.

    I suggested this approach a few times in the past and received 0 feedback; this tool is meant as a fully functional proof of concept.

    Disclaimer: Minimally tested; warnings about any transmission errors are thrown out by mfs_ftp.

    Note that other servers such as dserver could leverage this same approach, not just mfs_ftp.
    Last edited by bcc; 03-24-2008 at 02:02 AM.

  2. #2
    Join Date
    Sep 2001
    Location
    West of Bermuda
    Posts
    1,017
    sounds good - i'd bet my buddy with a prommed tivohd would like this. we'll check it out.

    ronny

  3. #3
    Join Date
    Feb 2007
    Posts
    70
    bcc,

    Thanks for this.

    I am curious - how much transformation/math is involved in going from the tmf to the ts ?

    My initial guess is "very little", as I think tivo stores the shows in something very close to an mpeg anyway, and your ability to do it on the fly like that inside the tivo suggests that all you are doing is stripping/replacing headers, etc. (I haven't looked at your code).

    But then I am struck by the fairly significant size difference - the ts ends up about 10% smaller than the tmf. Why is this ? Am I incorrect in my assumption of the (small) amount of work I perceive s3tots to be doing ?

  4. #4
    Join Date
    Jan 2002
    Posts
    214
    Very cool, I look forward to installing and testing this on my TiVoHD v9.2a soon.

    What patches are required to mfs_ftp before your diff is applied? I quit using mfs_ftp on S2-DTiVo boxes when tytool became fairly stable, so I'm not up to speed on it... I still use the original, unpatched mfs_ftp on my old S1 boxes w/ v3 on the rare occasions that I need to pull something off of them.

  5. #5
    Join Date
    Jan 2002
    Posts
    214
    How did the download speed using your app compare to using the TiVo approved .tivo download method?

  6. #6
    Join Date
    Aug 2004
    Posts
    4,075
    I prefer to run s3tots on the client side. For example:
    Code:
    MFS_DEVLIST=":s3" mfs_uberexport -v -R 1676598 | s3tots - foo.ts
    This still extracts direct to .ts. I did have to hack the s3tots source slightly to add "-" as a synonym for stdin.

    For me, this runs about twice as fast as extracting ts using the mips s3tots, presumably because my PC is faster than my TiVo. I see ~ 70% cpu time going to s3tots in top when extracting .ts via the mips s3tots. So the load isn't negligible. Still, it's a nice alternative to have. Thanks bcc!

  7. #7
    Join Date
    Feb 2002
    Posts
    6,413
    Always nice to see more tools added to the extraction toolbox. Unfortunately, I stopped using mfs_ftp years ago when I lost track of all the patches required to get it running. Call me old school but TyTools still works great for extraction and it doesn't take a rocket scientist to set it up and use it. Thanks for s3tots so us old farts can convert the ty files to transport streams on the PC side. It's an extra step and can be somewhat time consuming for large files but I usually run it in the background and then walk away from the PC while letting it do its thing.
    Please don't PM me or any other members looking for personal assistance. You'll do better by posting (after you've exhausted the search feature, of course) and taking advantage of the collective expertise of the membership instead of a single individual that may or may not be able to help you. Thank you and enjoy your stay at DDB!

  8. #8
    Join Date
    Nov 2002
    Posts
    1,076
    Quote Originally Posted by tiver View Post
    bcc,

    Thanks for this.

    I am curious - how much transformation/math is involved in going from the tmf to the ts ?

    My initial guess is "very little", as I think tivo stores the shows in something very close to an mpeg anyway, and your ability to do it on the fly like that inside the tivo suggests that all you are doing is stripping/replacing headers, etc. (I haven't looked at your code).
    There is no transformation or math required. Complete transport stream packets are contained in the series3 TY format, it's just a matter of unbundling the extra container (and rebundling in some cases, for example tivocast). There's a bit of extra processing to provide transport stream PAT&PMT tables, as those are stripped out of the TY by tivo.
    Quote Originally Posted by tiver View Post
    But then I am struck by the fairly significant size difference - the ts ends up about 10% smaller than the tmf. Why is this ? Am I incorrect in my assumption of the (small) amount of work I perceive s3tots to be doing ?
    There's a bug I identified a long time ago in the extraction tools where they work in terms of the number of pre-allocated chunks, instead of the number of chunks actually recorded. In older tivo models, this results in a small number of extra garbage chunks being extracted. On s3 systems, chunk allocation chaged and this number of garbage chunks can now be quite large. For example, a recording with 521 recorded chunks may transfer as 8192 chunks instead.

    I assume this is why you see a large size discrepancy.

    I gave Jamie an mfs-utils fix for this back in Sep. but it hasn't been incorporated. I can post the fix here as well.

  9. #9
    Join Date
    Nov 2002
    Posts
    1,076
    Quote Originally Posted by StanSimmons View Post
    VWhat patches are required to mfs_ftp before your diff is applied?
    I can't keep up with all the scattered mfs_ftp patches myself. As far as I know, a vanilla 1.2.9p patched with jamie's mfs_ftp.jamie_patch from mfs-utils is the only pre-requisite.

  10. #10
    Join Date
    Nov 2002
    Posts
    1,076

    performance

    Ok, it sounds like you guys are gonna make me go benchmark this I only did 1 test before releasing. This was a proof of concept release, remember? Anyways, my test was on a box that was already busy recording so it's less than ideal. In that test I got 24Mbit/sec, which is enough for real time streaming, on par with TY extraction from a busy box, and already faster than best case .tivo extraction (11Mbit/sec on an unloaded box last I checked).

    There is some sanity checking s3tots does which is completely unnecessary if one is throwing out the warnings anyways. This could improve performance. Also, I currently have s3tots streaming from a pipe which likely adds a lot of overhead. I'd rather link with mfs-utils directly but I don't want my code GPL encumbered. If someone would like to sort out the licensing issue, making this work without a pipe would be really easy. LGPL version of mfs-utils maybe?

  11. #11
    Join Date
    Nov 2002
    Posts
    1,076
    Quote Originally Posted by Jamie View Post
    I did have to hack the s3tots source slightly to add "-" as a synonym for stdin.
    I did the same for this project (and also stdout support). Will update the s3tots source release to reflect this.

  12. #12
    Join Date
    Aug 2004
    Posts
    4,075
    Quote Originally Posted by bcc View Post
    ... LGPL version of mfs-utils maybe?
    You'd have to get the vplay original author (tridge) to sign off on this.

    I do think passing that data through a pipe between two processes adds significant overhead to the overburdened tivo processor.

  13. #13
    Join Date
    Nov 2002
    Posts
    1,076
    Quote Originally Posted by captain_video View Post
    Call me old school but TyTools still works great for extraction and it doesn't take a rocket scientist to set it up and use it.
    Ok, you win the cave-man award for still using tytools with an s3. I still occasionally do things such as use a line-editor to edit files but I wouldn't brag about it BTW, tytools doesn't run on my linux systems so I never use it.

    I personally am happy to do conversions off-tivo as well, but the ease of use crowd has been pushing for a solution like this. It has many benefits such as allowing one to treat an s3 tivo like an HD Homerun box. It eliminates the extra disk i/o overhead involved in TY3->.ts conversion for those who have been throwing stones about i/o overhead.
    It also addresses concerns about lack of TY3 support in ffmpeg/mplayer/tyshow. Finally one can contain the arcane knowledge of TY3/TMF3 formats in one place instead of spreading those details into other applications that are better off sticking with standard a/v formats such as .ts.

  14. #14
    Join Date
    Nov 2002
    Posts
    1,076
    Quote Originally Posted by Jamie View Post
    You'd have to get the vplay original author (tridge) to sign off on this.
    I'll defer on that to whomever wants higher performance levels than what one gets via a pipe.
    Last edited by bcc; 03-24-2008 at 03:27 PM.

  15. #15
    Join Date
    Nov 2002
    Posts
    1,076
    Quote Originally Posted by bcc View Post
    There's a bug I identified a long time ago in the extraction tools where they work in terms of the number of pre-allocated chunks, instead of the number of chunks actually recorded. In older tivo models, this results in a small number of extra garbage chunks being extracted. On s3 systems, chunk allocation chaged and this number of garbage chunks can now be quite large. For example, a recording with 521 recorded chunks may transfer as 8192 chunks instead.
    Here is the mfs-utils change I was referring to. With this change, s3 extraction of unscrambled recordings will only include the number of chunks actually used in a recording, instead of the number of pre-allocated chunks.
    Note that I'm picking up the number of used chunks in the recording from the master chunk header, not from the inode metadata. The later is theoretically cleaner, but it turns out this number is left at 0 for recordings that are in progress, whereas the number I'm using is up to date while recordings are in progress.

    I've been using this change since last Sept. with no problems. This change works for all of TY/TMF/TS extraction. I recommend this change even if one is not using my new TS extraction.

    Also enclosed is a pre-compiled mfs_uberexport binary with the change.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •