Page 1 of 7 123 ... LastLast
Results 1 to 15 of 94

Thread: Re-Encoding DTV source

  1. #1
    Join Date
    Jun 2001
    Posts
    44

    Re-Encoding DTV source

    I generally do all my extraction and re-encoding from a SA TiVo, and as long as I select Top Field first, in the source, everything turns out great. However, in order to complete my collection, I got a couple files from a DTiVo.

    As jdiner pointed out in one of these threads, DTV sources can switch field order in the middle! Thus, there's no correct setting to use for TMPGenc - if I select Bottom first, some parts will be messed up, some will be fine. If I select top, the exact opposite clips will be messed up vs fine.

    Is there a utility that will mangle the field order of a MPG to make it consistant throughout (even convert it to an interlaced AVI) without re-encoding it, so I can send it through TMPGenc?

  2. #2
    Join Date
    Jun 2001
    Posts
    44
    Well, after searching most of the day for a utility and trying everything under the sun, I finally broke down and modified pulldown.exe so it would copy the field order from my source m2v file into my destination (encoded with TMPGenc) m2v file. (I also had to correct an off-by-1 bug in the pulldown source. *sigh*) Worked like a charm.

    My source (a DTV stream edited with TyTool I believe) ended up switching field orders 7 times (in 40 minutes). One time it looks like it actually switched to 3:2 pulldown for a second or two, then reverted back to bottom field first.

  3. #3
    Join Date
    Sep 2002
    Posts
    1,735
    Quote Originally Posted by Saturn
    Well, after searching most of the day for a utility and trying everything under the sun, I finally broke down and modified pulldown.exe so it would copy the field order from my source m2v file into my destination (encoded with TMPGenc) m2v file. (I also had to correct an off-by-1 bug in the pulldown source. *sigh*) Worked like a charm.
    Huh? How are you feeding the source stream and what happens when you get partial frames?

    DVD2AVIdg is supposed to support the order flips and partials frames while feeding out a stabilized version. I've yet to find a VFAPI that will feed to it, though.
    Collecting 9/11, Afghan/Iraq, Mail Call, Trains, Cooking, Woodworking, Fighting Illini - Let's chat
    A/V links: neuron2 doom9 VideoHelp DigitalMediaNet CreativeCow DVDShrink PgcEdit Streambox WMRecorder
    other links: SnapFiles NoNags HackADay Engadget Fontleech OfflineExplorerPro TechBargains PriceWatch

  4. #4
    Join Date
    Jun 2001
    Posts
    44
    Quote Originally Posted by FredThompson
    Huh? How are you feeding the source stream and what happens when you get partial frames?

    DVD2AVIdg is supposed to support the order flips and partials frames while feeding out a stabilized version. I've yet to find a VFAPI that will feed to it, though.
    Basically there's a flag in each mpg frame that says whether it is top field first or bottom field first. In my source, it was mixed, and switched back and forth. So I just copied those bits to the re-encoded mpg, overwriting the bits that were there (which had all top-field first, since that's what TMPGenc creates). What the player does with the data is up to it, but since DVDs do this all the time, DVD players should support it just fine (and it plays fine on my PS2, though it looked a little funny on PowerDVD, which is trying to de-interlace anyway...) I really don't know what the DVD player does on the transitions, I suspect it plays a field twice or skips one, depending on which way it is going. But since it generally only happens between scenes, and then only for a single field, it doesn't really matter.

  5. #5
    Join Date
    Jun 2001
    Posts
    44
    Quote Originally Posted by FredThompson
    Huh? How are you feeding the source stream?
    I didn't answer this above. I opened the source stream in TMPGenc directly, no need to frameserve it from anywhere. I selected top-field first so that TMPGenc wouldn't mangle the field order, just re-encode each field and put them in the destination in the same order, though it marks EVERY frame as top field first.

    Then as a final post-processing step I copied the source's field orders to the destinations.

  6. #6
    Join Date
    Sep 2002
    Posts
    1,735
    Ah, that's why we're having different results. I'm trying to go through AviSynth so filtering can be done. That's why DVD2AVIdg would be so helpful.

    Any suggestions how to get it to work? Been pounding my head over this for quite a while. Would really appreciate help.
    Collecting 9/11, Afghan/Iraq, Mail Call, Trains, Cooking, Woodworking, Fighting Illini - Let's chat
    A/V links: neuron2 doom9 VideoHelp DigitalMediaNet CreativeCow DVDShrink PgcEdit Streambox WMRecorder
    other links: SnapFiles NoNags HackADay Engadget Fontleech OfflineExplorerPro TechBargains PriceWatch

  7. #7
    Join Date
    Jun 2001
    Posts
    44
    Out of all the utilities I tried, DVD2AVI wasn't one of them, so I can't help you there. Are you trying to de-interlace your video? TMPGenc can do that..., and that's an option that will essentially ignore the top or bottom field first, and will look great on a monitor, but not so good back on a TV...

  8. #8
    Join Date
    Sep 2002
    Posts
    1,735
    No, I never deinterlace unless I'm re-processing something which was originally progressive.

    In the 2 cases I'm working with right now, I'm going to overlay black on the edges to cover the corruption and data squiggly to better allocated bandwidth to the clean portion of the image. Sometimes I'll do quality filtering. In any event, the challenge was getting the darn file to go into AviSynth.

    I've tried serving frames from VirtualDub-MPEG2 but it doesn't handle the partial frames properly, AFAIK. I've asked fcchandler about it and he said DVD2AVIdg was far better for this type of source.

    If there's a way to get a DTiVo MPEG or VOB into DVD2SVCD the final encode can be better tuned for size. Every time I've tried, VFAPI pukes. There is a way to do this, I've had it work before, just can't remember what I did.
    Collecting 9/11, Afghan/Iraq, Mail Call, Trains, Cooking, Woodworking, Fighting Illini - Let's chat
    A/V links: neuron2 doom9 VideoHelp DigitalMediaNet CreativeCow DVDShrink PgcEdit Streambox WMRecorder
    other links: SnapFiles NoNags HackADay Engadget Fontleech OfflineExplorerPro TechBargains PriceWatch

  9. #9
    Join Date
    Sep 2002
    Posts
    1,735
    Belay my help request.

    I'd rebuilt the machine and forgot to run vipset.bat to get the VFAPI codec loaded. Geez, that was dumb.
    Last edited by FredThompson; 03-30-2004 at 05:41 AM.
    Collecting 9/11, Afghan/Iraq, Mail Call, Trains, Cooking, Woodworking, Fighting Illini - Let's chat
    A/V links: neuron2 doom9 VideoHelp DigitalMediaNet CreativeCow DVDShrink PgcEdit Streambox WMRecorder
    other links: SnapFiles NoNags HackADay Engadget Fontleech OfflineExplorerPro TechBargains PriceWatch

  10. #10
    Join Date
    Sep 2002
    Posts
    1,735
    OK, am (finally) testing the new beta of DVD2SVCD on DTiVo streams. Directly serving DTiVo frames is making Canopus ProCoder choke. The only way I've managed to get things to work is to do a VirtualDub-MPEG2 fast recompress through HuffYUV set for YUY output instead of RGB. Audio won't pipe properly and I'm still testing.

    Would you please post your modified pulldown? For that matter, how about documenting at doom9.org what you changed and why? It will probably help other folks. DVD2SVCD uses pulldown so this might be real helpful.
    Last edited by FredThompson; 03-30-2004 at 11:43 PM.
    Collecting 9/11, Afghan/Iraq, Mail Call, Trains, Cooking, Woodworking, Fighting Illini - Let's chat
    A/V links: neuron2 doom9 VideoHelp DigitalMediaNet CreativeCow DVDShrink PgcEdit Streambox WMRecorder
    other links: SnapFiles NoNags HackADay Engadget Fontleech OfflineExplorerPro TechBargains PriceWatch

  11. #11
    Join Date
    Jun 2001
    Posts
    44
    Quote Originally Posted by FredThompson
    Would you please post your modified pulldown? For that matter, how about documenting at doom9.org what you changed and why? It will probably help other folks. DVD2SVCD uses pulldown so this might be real helpful.
    My pulldown.exe really isn't in a releasable condition. But if anyone is interested, here's exactly how it works (documented sort-of above).

    Lets call the source S.MPV.

    When you pull a source off a DTiVo, there's no way to reliably tell which way it is encoded - top field first or bottom. Certain shows probably generally end up one way or another. Which is why some people swear by one or the other. The truth is that it depends on the encoder at DTV. My source, however, was mixed. If you looked at the "which field first" bit (assuming you had a program that could look at each frame individually), you'd find something like this:

    Top field first:
    1111111111111

    Bottom field first:
    0000000000000

    Mixed (S.MPV):
    1111000011110

    where each 1 or a zero corresponds to a frame.
    If you look at the individual fields, they'd be layed out like this (A is the one that is supposed to be displayed first, then B, "top" would be first, "bottom" is second)

    Top field first:
    AB|AB|AB|AB|AB|AB|AB|AB|AB|AB|AB|AB|AB

    Bottom field first:
    BA|BA|BA|BA|BA|BA|BA|BA|BA|BA|BA|BA|BA

    Mixed (S.mpv):
    AB|AB|AB|AB|BA|BA|BA|BA|AB|AB|AB|AB|BA


    In reality, there are several thousand of these bits (~76,000 for me), one for each frame. But essentially my source looked like the Mixed one above. So any tools that tried to detect field order generally only looked at the first few fields and discovered it as top field first. Which worked. For the first minute or so.


    So when I used TMPGenc to encode the video into D.mpv to change the resolution and bitrate, (or any other tool for that matter), the result of the bits was this:

    (D.mpv)
    1111111111111

    but the frames were still in mixed order.
    (D.mpv)
    AB|AB|AB|AB|BA|BA|BA|BA|AB|AB|AB|AB|BA

    This created jittery output for the "BA" frames, whereas the "AB" frames looked fine.

    Now there are potentially two solutions to this problem:

    1. Re-order the frames to fit the bits and make everything consistent as to top (or bottom) field first, or 2. set the field-first bits properly. The 1st solution seemed to be what I wanted to do, and tried various ways to try to convert S.mpv into something consistent before encoding it with TMPGenc. But there's a problem. The frame listed first corresponds (lets say) to the even lines, and the second frame corresponds to the odd lines. So you can't just swap the A's and B's around, or your picture would look VERY strange. There may, however, be a way to add or remove a field to get around this, shifting everything over to look like the AB|AB|AB|AB sequence of top field-first. This requires logic and complexity that I really didn't want to get into.

    So here's what I did instead:

    1. I used a modifed pulldown.exe to read the top-field bits from the DTV source and write them to a simple text file consisting of 0's and 1's:
    (S.mpv -> output.txt)

    2. I modified pulldown.exe again to read that text file and copy those bits into the destination file:
    (D.mpv + output.txt -> D2.mpv)

    The resulting D2.mpv looked like the original S.mpv did:

    (D2.MPV):
    1111000011110

    AB|AB|AB|AB|BA|BA|BA|BA|AB|AB|AB|AB|BA


    However, the aspect ratio and bitrate was that of D.mpv - the one that had been re-encoded by TMPGenc.

    The reason my pulldown.exe isn't really in a releasable state is the following:

    pulldown.exe is designed to take in a single MPV file and produce a single MPV file. What I'm trying to do is take two MPV files (S.mpv and D.mpv) to produce a correct MPV file (D2.mpv) by taking all the information from D.mpv except the field order, which is taken from S.mpv. Modifying pulldown.exe to read two MPV files simultaneously looked like it would be an ENORMOUS task riddled with bugs, considering the design of the code. So I did it in two steps. The result is that step 1 above (S.mpv -> output.txt) actually produces another .mpv file along the way, even though it is really only reading S.mpv for its field order bits. There's also no checking in place to see that S.mpv and D.mpv contain exactly the same number of frames, etc. I'm also not sure that all the bits in the D2 header are set properly in the end - there may be a need to set the 3:2 flag or some such for the mixed-field order D2.mpv to play correctly on all DVD players.
    Last edited by Saturn; 04-03-2004 at 12:06 AM.

  12. #12
    Join Date
    Jun 2001
    Posts
    44
    Thus, I can't really release pulldown.exe in its current state - it was very kludged. The only reason I used pulldown.exe at all as a starting point was:

    1. The source was available
    2. It had existing code to read and write .mpv files while tweaking individual bits of the frame along the way. My changes were minimal at best.

    I may get around to making it releasable, but it'll probably still be a 2-part solution, and dedicated to this particular application. Right now I have a thesis to finish though.

  13. #13
    Join Date
    Sep 2002
    Posts
    1,735
    Understood. In thinking about it, there's no need for a modified pulldown if you properly use DVD2AVIdg which will create proper full frames from the partials in a DTiVo stream.
    Collecting 9/11, Afghan/Iraq, Mail Call, Trains, Cooking, Woodworking, Fighting Illini - Let's chat
    A/V links: neuron2 doom9 VideoHelp DigitalMediaNet CreativeCow DVDShrink PgcEdit Streambox WMRecorder
    other links: SnapFiles NoNags HackADay Engadget Fontleech OfflineExplorerPro TechBargains PriceWatch

  14. #14
    Join Date
    Jun 2001
    Posts
    44
    Quote Originally Posted by FredThompson
    Understood. In thinking about it, there's no need for a modified pulldown if you properly use DVD2AVIdg which will create proper full frames from the partials in a DTiVo stream.
    Are you working off a HOW-TO or something? I do have 1 more mixed episode to deal with...

  15. #15
    Join Date
    Sep 2002
    Posts
    1,735
    I'm trying to figure out how to properly serve frames for re-encoding or other processing. Use DVD2AVIdg's preview function and look at the farme counters. You'll see DTiVo streams have lots of partials.
    Collecting 9/11, Afghan/Iraq, Mail Call, Trains, Cooking, Woodworking, Fighting Illini - Let's chat
    A/V links: neuron2 doom9 VideoHelp DigitalMediaNet CreativeCow DVDShrink PgcEdit Streambox WMRecorder
    other links: SnapFiles NoNags HackADay Engadget Fontleech OfflineExplorerPro TechBargains PriceWatch

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •