Nonlinear Editing (Vegas5) of ty streams

[LowPingBoy]

I need to have greater capabilities than provided with the standard TyTools editor; transitions, 5.1 surround, etc. However, the audio is consistently out of synch with the video in ty files, or vob, or mpeg files I get using TyTools. They play fine in Nero, but just get trashed when I open them in Vegas. I have tried just about every combination of switches, audio, stream, mplex, etc. but cannot get them synched. Is there a better way to move my Tivo (SAT60) files to my Win2000 pc so that I can edit them in Vegas?

Thanks - LPB

[satguymtl]

Me too... sync problem moving any of the files to Cleaner XL for encoding, no matter which combination of settings I've tried... I'm almost ready to give up and just accept 480x480 and get a new DVD player... ;)

[AlphaWolf]

Try using tydemux, it was designed so that the resultant mpeg streams would work in just any editor, in contrast to tytool which doesn't produce mpeg streams that are exactly kosher.

[LowPingBoy]

Thanks for the tip about Tydemux. I'll try that and load the video bus with the -v file and the #1 audio bus with the -a file and hope they're synched up. Vegas prefers AVI files but those I have created with DVD2AVI suffer from the same synch problem. Hope the TYdemux trick solves this. Have only recently been communicating via the net, previously just captured the video I wanted while playing the show on the Tivo. Great results, but ties up the Tivo and is slow compared to Tytools.

Thanks again -- LPB

[ronnythunder]

lpb, it might be worth your while to read up some on the ty files produced by dtivos. josh (creator of tytools) and others have written reams about it. in short, the mpeg that comes down from the birds is some pretty whacked out stuff, and it's not surprising at all that many tools and some dvd players don't like it. it's certainly *way* out of normal dvd spec, but hey, directv runs the show and they don't care about us making dvds (well, actually they do - they'd rather we *not*), and they make the stream suit their needs rather than ours.

there's also a big discussion thread about re-encoding the streams here; i don't keep up with it much because my dvd players can play the dvds that tytools makes with no problems.

ronny

[AlphaWolf]

Well, they can be converted into a proper stream, but the primary issue is when you get a bad signal - even if it is only for a split second - the stream timings can get all out of whack, which is what causes the sync issues. Olaf_SC wrote tydemux to correct these in just about any situation (personally I have yet to see a single tystream that is unable to stay in sync after being processed by tydemux.) The problem is there are a few bugs remaining in it that were never fixed. It should work just fine for a series 1 unit though.

I recommend opening the tystream file in tystudio and processing it with generic mpeg-2 and no audio processing, and don't make any edits. Once that is done, open the resulting mpeg file with VideoReDo (an awesome little program) and make your frame accurate edits with that, then go ahead and send the edited mpeg stream straight to your favorite DVD creation software. You can't go wrong with that method, or at least it has never failed me yet (this is the whole reason why to this day I still prefer tystudio over tytool; I can edit the streams with just any arbitrary video editing software that has a lot more features, and I don't run into a single issue.)

[contemplativo]

I too need NLE capability. If tystudio's output is compatible with commercial NLE's, then that seems promising. I thought, though, that people were reporting sync issues after using tystudio's editor. So it doesn't follow that the same or worse sync issues will arise with NLE's?

Alpha, it's good to hear of your success with VideoReDo. The tystudio threads for some time have been featuring trouble reports rather than positive feedback. So it hasn't been obvious that tystudio is still being actively used for producing mpegs to be processed with commercial editors. What other editors have you used with success on tydemux output?

[AlphaWolf]

I just use videoredo. I am not really a hardcore video enthusiast, I just extract things now and then and then archive them somewhere. I've only made one tyDVD thus far (friend of mine missed the second presidential debate which I tivo'ed, so I handed him a copy.) As for the editing sync issues, my guess is that it may or may not happen, but I wouldn't know since I don't use the editor in tystudio, I just use it to convert the stream to standard mpeg, then I edit it with videoredo (its a lot easier to use and seems to work faster, not to mention having a lot more features as well as frame accurate editing.)

The reason I might never run into sync issues might have to do with videoredo and its automatic sync correction feature (everytime I finish it tells me how many sync frames it deleted, and I never notice any hiccups in the resulting streams.) I don't think its automatic sync correction will fix big errors in tystreams (rather just the little ones left behind,) but it does have an option to offset the audio manually, which may help with bigger sync problems if they remain (I've never needed to use it thus far, so I am not sure.)

[malfunct]

I just use videoredo. I am not really a hardcore video enthusiast, I just extract things now and then and then archive them somewhere. I've only made one tyDVD thus far (friend of mine missed the second presidential debate which I tivo'ed, so I handed him a copy.) As for the editing sync issues, my guess is that it may or may not happen, but I wouldn't know since I don't use the editor in tystudio, I just use it to convert the stream to standard mpeg, then I edit it with videoredo (its a lot easier to use and seems to work faster, not to mention having a lot more features as well as frame accurate editing.)

The reason I might never run into sync issues might have to do with videoredo and its automatic sync correction feature (everytime I finish it tells me how many sync frames it deleted, and I never notice any hiccups in the resulting streams.) I don't think its automatic sync correction will fix big errors in tystreams (rather just the little ones left behind,) but it does have an option to offset the audio manually, which may help with bigger sync problems if they remain (I've never needed to use it thus far, so I am not sure.)

Part of the key to keeping sync (at least when using tytool as I've never used anything else) is to not make any cuts. When you cut the timecodes are never patched and so lots of editing software gets confused.

[AlphaWolf]

Part of the key to keeping sync (at least when using tytool as I've never used anything else) is to not make any cuts. When you cut the timecodes are never patched and so lots of editing software gets confused.

Well, that is just the thing, tytool relies on patching the timecodes in order to artificially sync the video. Tydemux makes the video correct to begin with, thus no timecode patching is even necessary, so AFAIK (there may be a few exceptions out there, but I haven't seen any) you can edit it with pretty much anything.

I've got several tystreams I could show you where editing with tytool processed video causes sync loss, whereas editing with tydemux results in a perfect stream.

[FredThompson]

Mux to VOB using TyTool's new method, open with AviSynth's DirectShowSource and frameserve.

http://forum.doom9.org/showthread.php?s=&threadid=47194
http://www.debugmode.com/frameserver/

Don't use VirtualDubMPEG to serve frames. It's decoder is screwed up.
DGMPGDec might work properly with DTiVo streams now.

[AlphaWolf]

Mux to VOB using TyTool's new method, open with AviSynth's DirectShowSource and frameserve.

http://forum.doom9.org/showthread.php?s=&threadid=47194
http://www.debugmode.com/frameserver/

Don't use VirtualDubMPEG to serve frames. It's decoder is screwed up.
DGMPGDec might work properly with DTiVo streams now.

I think he might be looking for a method that doesn't involve re-encoding the video (unless frameserving can be done without re-encoding? *shrug*)

[FredThompson]

I think he might be looking for a method that doesn't involve re-encoding the video (unless frameserving can be done without re-encoding? *shrug*)Vegas uses RGB internally and he'll have to re-encode any time the video is altered. I should have mentioned to use AviSynth's ConvertToRGB32 to serve uncompressed RGB frames. Actually, if he's going to do that, may as well pipe it through VirtualDub to have access to those filters. MSU's denoise and deblocking filters would be quite useful if you're going to RGB anyhow.

http://www.compression.ru/video/denoising/index_en.html
http://www.compression.ru/video/deblocking/index_en.html

FWIW, I don't mess with RGB filters because the colorspace isn't a 1:1 relationship so, whenever possible, I'd do as much as possible with AviSynth.

[AlphaWolf]

Hmm...why not just re-encode the video anyways then? It'll always iron out any issues. The reason I don't do it myself is b/c I don't like the lossiness, plus it takes longer, but if that doesn't matter to anybody else then it is probably the way to go.

[FredThompson]

Read the first post again. He talks about using transitions. There's no way to modify the video and store it without re-encoding.

I'm not sure what he's confused about wrt 5.1 since TyTool supports it.

IIWM, I'd chop the original TY into pieces and edit just the sections which are to be transitions then stitch it all back together with TyTool.

[ronnythunder]

there are things other than sync issues in the streams if you're talking about dtivo streams. there's the "bazillion frames in a gop" issue, the strange repeat_last_field/frame handling, out of order fields and other stuff. again, josh has written extensively on the topic.

ronny

[AlphaWolf]

Read the first post again. He talks about using transitions. There's no way to modify the video and store it without re-encoding.

Dude, you don't have to start flaming. All I was saying is if you prefer it that way, then do it that way.

there are things other than sync issues in the streams if you're talking about dtivo streams. there's the "bazillion frames in a gop" issue, the strange repeat_last_field/frame handling, out of order fields and other stuff. again, josh has written extensively on the topic.

ronny

*shrug* possibly, just the thing is I have plugged these streams into several different apps and I've never run into a single problem. Although I do recall olaf_sc saying that it is possible make these streams compatible with anything...as far as I can tell that is what he did (all of my streams that go out of sync with tytool stay in sync with typrocess) :confused:

WRT to editing in tystudio itself, I have seen issues (namely the dreadded make your cut somewhere else message) but those streams can be edited without any problems at all in videoredo once they have been through typrocess, so as far as I can tell the bugs are just in the tystudio editor itself, but the streams are fine (they are perfectly in sync and the edits are frame accurate and perfect as well.)

BTW what is the difference between a linear editor and a nonlinear editor? I just noticed that this videoredo program calls itself a nonlinear editor. I have no idea what that means (at first I was thinking it was something related to re-encoding, but this editor doesn't re-encode at all.)

[FredThompson]

There wasn't any flame.

Your comment about serving frames without re-encoding doesn't make sense. They're two different things. If he's going to use transitions, there's no option but to re-encode at least those portions which will be modified.

...or were you just making a reference to the "dude" fiasco at doom9 a while ago?

I agree with you that re-encoding will reduce the quality. The streams are MPEG2, just not DVD standard. Given the vast majority of software and hardware players can handle the streams, re-encoding just to change GOP size, make closed GOPs, or other similar changes doesn't make sense to me.

The difference between an NLE and a LE is similar to the difference between a full-screen editor and a line editor. NLEs typically have a few timelines and sections can be selectively processed, copied, cut, etc. TyTool's editor is a very simple linear editor. You can remove parts and cut the stream into pieces but you cannot rearrange them or selectively apply processing.

There's a nice NLE for Linux from Herroine Warrior called Cinelerra. At least, that's what it was called the last time I looked at it.

The "problem" with most editors is they'll render the entire final stream. That's why I was suggesting cutting the source into pieces and only re-rendering the modified parts.

[AlphaWolf]

Well, I dunno about all of the stuff involved. All I know is that I just used the trial version of videoredo (http://www.drdsystems.com/VideoReDo/FeaturesOverview.htm) to cut up and master about nine hours worth of dtivo tystreams processed by typrocess in somewhere around thirty minutes using a 1ghz laptop. If you can do without the fading then I highly recommend it as seems it'll do everything else you need, and it was easy enough for me to learn and fully understand (even to setup a batch process) in about 2 minutes without reading any help files, and I rarely use editing software so that says a lot. Thus far the streams look perfect and skipping to the end shows zero sync problems. That's my final 2 cents.

[contemplativo]

VideoReDo seems to have great support staff. Two product inquiries of mine were answered in the space of 3.5 hours, including one today (on a Saturday?).

Malfunct, they tell me that "output files from VideoReDo are totally MPEG compliant. No matter how many cuts and/or joins are made the sequence headers and timestamps are recalculated and correct." That seems promising if one wants to do post-processing after an initial round of cutting, like I need to do.

Unfortunately, cuts are frame-accurate not field-accurate, unlike tytool (I want access to the field information). Also, the program is Windows only.

[AlphaWolf]

Yeah thats something else I noticed, the time information displayed by powerdvd is correct on all of these streams. Also there are absolutely no seek issues whatsoever, no matter what I play it in.

Not only that, but something else a bit surprising is happening as well. I am not quite sure how this is managed, but it seems that this program is also able to fix some (but far from all) visual distortions such as artefacting and/or pausing/skipping of the video. (?!) I have two streams that when played on the tivo you see a lot of pixelation and distortion at one point, yet in the edited version of these streams you barely see any pixelation and the video doesn't even flinch.

I also have one stream where the sound goes out of sync for about a split second and immediately recovers when played in powerdvd. Yet when I play that same stream in winamp with the elecard codec, it plays perfectly.

[digiital]

Unless I'm missing something. I'm JUST in the process to create my first DVD(It's rendering the DVD right now as I type this, 5 hrs on a 2500XP) for my little one(Fisher Prices - Little People) and what I did was using TyTool. Download the files off the Tivo to MPG(set the audio to 48K), and then I edited the files in Vegas5, cut out the commericals. I then rendered the video back to AVI. Then in the DVD Architect, I created menus, chapters and what not, and no problems with audio sync at all.

[FredThompson]

PowerDVD has a few bugs in its decoder, even with the upgrade patch. I made a menu a week ago which would not display yet it played just fine in hardware players and WinDVD.

If you mean pixellation during the a passage, the app must be filtering and re-encoding. If you mean pixellation at a GOP point, are you editing within this ap? These streams are open GOP which means a forward reference can create pixellation if the data it references is different.

It's also possible the decoders in the DTiVos are buggy. Their encoders certainly are.

[AlphaWolf]

PowerDVD has a few bugs in its decoder, even with the upgrade patch. I made a menu a week ago which would not display yet it played just fine in hardware players and WinDVD.

Actually this gets even more interesting, I went through the videoredo options and added a few checkmarks to things that were minor fixes to make it even more compatible overall. It fixed ALL of the problems in this stream; it plays flawlessly in powerdvd now.

EDIT: just found that the only option that is necessary is: start video frame in new packet. It actually reduces the filesize rather than increasing it too! In looking at the help file, it also indicates that this option is used to make the streams compatible with other editors.

If you mean pixellation during the a passage, the app must be filtering and re-encoding. If you mean pixellation at a GOP point, are you editing within this ap? These streams are open GOP which means a forward reference can create pixellation if the data it references is different.

*shrug* I have no idea. All I know is that there were once flaws that are now gone, and the website says that the editor doesn't re-encode the video. Given the time it takes for it to process the cuts, I don't see how it could possibly re-encode the video in that short of time. Plus the picture quality looks true to the original tystream when I play it in mplayer.

It's also possible the decoders in the DTiVos are buggy. Their encoders certainly are.

Well, the pixelation I see also shows up in mplayer, yet not in my final streams.

Dunno what else to say, it just works, and it works good.

EDIT: Just read the feature page of videoredo in full, and it lists that it does almost everything I have talked about in this thread :D

Also, if anybody wants to duplicate exactly what I have done here without having to use tyeditor, run typrocess as follows:

typrocess -d 4 -la tyeditor-process.log -it directivoextractedstream.ty -d 4 -demux { -s 2 } -mplex { -f 3 -V } -om directivoextractedstream.mpg

edit directivoextractedstream.mpg with videoredo, watch what it can do.

[FredThompson]

That's odd. Pixellation is either a bad encode or a forward reference to data which has changed. Maybe there's something in the non-image data which is being tweaked a little bit.

I didn't think to look fro more advanced settings in PowerDVD. CLEV is off because it has too much of a tendency to make ty source look overly garish. I'll give those settings a try. Thanks.

If your playback is with a Windows app, try using ffdshow as the override decoder and play with the filters. There's some really nice stuff there. http://htpcnews.com/main.php?id=ffdshowdvd_1

@digiital, suggest you try TyTool-only DVD first and see if it meets your needs. Rendering will reduce the quality of both image and sound and takes a LOT longer. Additionally, if you keep things in TyTool muxed format, you can always remux when the next muxer is released and probably have some more cleanup performed.

[AlphaWolf]

I didn't think to look fro more advanced settings in PowerDVD. CLEV is off because it has too much of a tendency to make ty source look overly garish. I'll give those settings a try. Thanks.

I didn't adjust anything at all in powerdvd, this was all in videoredo.

[FredThompson]

Oh, that wasn't clear when I read your post. I thought you were replying to my comment about PowerDVD's busted decoder.

Hmmm...I'll let Josh know about this. He's been working on HD bugaboos lately but maybe this will help him work ont he muxer. He's gotten about 30 "problem" streams from me which will break TyTool's process at various points. Maybe VideoReDo can help shed some light on how he could improve his muxer.

[AlphaWolf]

Oh, that wasn't clear when I read your post. I thought you were replying to my comment about PowerDVD's busted decoder.

Hmmm...I'll let Josh know about this. He's been working on HD bugaboos lately but maybe this will help him work ont he muxer. He's gotten about 30 "problem" streams from me which will break TyTool's process at various points. Maybe VideoReDo can help shed some light on how he could improve his muxer.

FWIW I also just noticed something else, the option that actually fixed my problem in powerdvd was just the first one, the other ones didn't do anything. I got a smaller filesize by not checking those two (smaller even than without checking the first option.) In looking at the help file, it also indicates that this option is used to make the streams compatible with other editors.

Try demuxing those streams with tydemux, then run it through videoredo with mux only and that option checked (as its muxer seems to repair streams,) then demux it with (insert your favorite demuxer here) and see if it will run through tytools.

EDIT: Just did some more poking around, it seems that particular sync issue doesn't exist in the post typrocess stage (the unedited mpeg plays fine in powerdvd,) but it did exist in the post videoredo edit stage, so whatever was going wrong was actually *introduced* by videoredo. Since checking that option fixed it as I mentioned earlier all is all good anyways, just letting you know in any case though.

[AlphaWolf]

I think I have an idea as to what is going on here wrt to the artefacting. It seems that it is actually deleting the frames that were displaying the artefacting, as they may have been frames that were useless and simply didn't belong. This might have to do with what somebody mentioned earlier with the excess "bazillion" frames thing.

On a hunch I decided to process a stream that is affected by this with tytools and run it through videoredo. At the end it indicated that it deleted 84 video frames, and 96 audio frames for the purpose of resyncing.

With this same stream in tystudio it indicated that it deleted 22 video frames and it didn't delete any audio.

When this stream is played on the tivo, there was a certain point in the video where you normally would see a bit of artefacting, and the audio continues without a hitch.

In the tytool processed stream, this particular section was cut out entirely, and theres a very distinct jump in both the video and the audio (theres about three seconds of video missing, and even more audio missing.) The good thing is that it still maintains sync.

In the tystudio processed stream it just continues smoothly as I mentioned earlier (no gaps, cuts, artefacting or anything, just flawless playback.)

Normally if you mux this stream with tytools, and then run it through e.g. mpeg2vcr, there is no jump, but there is artefacting, and the stream permanently loses sync by a few seconds at that exact point. Videoredo seems to compensate for that by deleting a calculated number of video or audio frames, hence there is the jump, but it stays in sync no matter what.

My guess is that in both scenarios, it is simply removing frames that thinks shouldn't be there, and these frames are also the ones that cause the artefacting which is removed in the videoredo muxed stream. Not all of the frames it is deleting are actually bad frames in the tytools case, as evidenced by the results of tystudio where it leaves many more of those frames intact and the video and audio can continue without jumping (just goes through it smoothly and unnoticeably.)

The good thing though, is that videoredo can keep pretty much anything you give it in sync. With typrocess it seems to need to discard fewer frames than with tytools in order to accomplish this. In either case it can clear up some artefacting, and it makes the streams compatible with any other editor you want to put the stream through. That makes videoredo the editor of choice IMO. :D

I would like to see an expert oppinion on these observations though, as it is a bit interesting.

[FredThompson]

Hmmm...that seems rather odd. I wonder how it can tell which are blocky and then how it makes the stream stable. It can't just remove a frame because of the intra-frame pointers. I know Josh is out of town for a couple of days but why don't you drop him a message and make the original and 3 edited versions available to him? If he's getting into the trouble streams I've sent him, it might be an opportune time to fix more of the muxer.

What you describe is similar to what I've noticed about recording the same time frame and channel on an S2 and S1. My S1 recordings are always smaller. I have no idea why.

[AlphaWolf]

Sure, I'll get on it next weekend.

[AlphaWolf]

Ok here is a sample of the skip behavior I was talking about with tytool and videoredo.

The source tystream is here (http://alphawolf.0sync.com/test.ty).

Note in this stream theres a small section where the video distorts heavily, then recovers normally on the tivo.

When you run this stream through tytool, then run it through any editor other than videoredo, you get a permanent sync loss immediately after this point. When you run this stream through tystudio, you can run it through any editor you want (or at least any that I have seen) and it will stay in sync perfectly.

When you run this stream through tystudio, then through videoredo, you get pretty much everything (if not everything?) you sent through it back without any skips, as seen here (http://alphawolf.0sync.com/tystudiovr.mpg). Videoredo doesn't report any anomalies nor removing of any video or audio frames.

When you run this stream through tytool, then through videoredo, you get a skip. Note that in this case the video pixelates less, however you are missing some audio because instead of fixing it, it just cuts it out entirely, as seen here (http://alphawolf.0sync.com/tytoolvr.mpg). Videoredo reports the following with the tytool stream: Input sequence errors: 32 Audio resync frames removed: 57

These skips can get much worse with tytool, I have seen it cut off around 2 or 3 seconds worth of video before, and this wasn't pixelated video as seen in this stream, it was perfectly good video. Tystudio on the other hand kept all of this video, just like you saw in the sample.

*Note that I cut both of the videoredo processed streams at the 9.01 second mark (wanted to save bandwidth.)

[LowPingBoy]

This thread kind of got away from the initial inquiry about taking a TY stream and converting in a way so as to use it in a NLE, specifically Vegas 5.0. The suggestions and introduction to a number of other useful tools has been worthwhile but did not produce a file that worked for me. But going through a lot of the material here I have found a solution. There may be other alternatives, but this works for me.

Take the stream from your Tivo as a TY. Using TYTools convert this file to VOB, not the New VOB option. Use DVD2AVI to generate separate audio and video files. Do not convert 48k to 44.1k for the audio, use the DivX codec with resolution set to 720 x 480 for the video. The AVI file created will have no audio, find that in the mpa file. Import both files to the timeline in Vegas. Since you cannot place more than one file in the trimmer you have to render them back out of Vegas as one AVI file that will contain both audio and video. Figure about 13G for a one hour show. Load the new AVI file back to Vegas in the trimmer and edit as desired.

This is a whole lot of churning and burning, but it produces an editable file for Vegas that has the audio and video synched.

Thanks to all for the comments.

LPB :) :)

[FredThompson]

DVD2AVI will not properly handle DTiVo source. The old muxers in TyTool are not as accurate as the new ones. Use the new mux format to make a VOB.

Again, DirectShowSource through AviSynth to serve frames. If you MUST use a DVD2VI, use Donald Grafts equivalent.

[LowPingBoy]

Yeah, based on an earlier post I gave AVISynth a try for solving the only real problm I had, keeping the audio and video in synch. Inasmuch as I use Vegas to perform all edits, transitions, splicing, 5.1 with surround panner, multiple audio tracks, placing *jpg in the middle of a video, etc., a lot of the fuctionality is nothing I need. From avisynth.org, the five main reasons to use this tool are: 1 Join Videos, 2 Filter Videos, 3 Break the 2 GB Barrier, 4 Open Unsupported Formats, and, 5 Save Disk Space. None of these are of any concern to me whatsoever. Vegas does all of this and more and is screen based, not command line/script. Kind of nice to edit with a slider, select filers from a drop down, etc. I only care about getting a good AVI file that I can synch with audio within Vegas. And all of that works with DVD2AVI, quite well in fact. Use the DivX codec, properly set up, and a nice AVI file is created as well as an audio file that when placed on tracks in Vegas are synched as well as the AVI file created by capturing the file via a IEEE-1394 card. Furthermore, using an external reference monitor, differences between a captured AVI and one created using the route I described are indistinguishable.

I don't know what the problems with DVD2AVI are. It delivers the outcome I need and I think Occum's razor would find my straightforward solution preferable. I need quality video and synched audio in Vegas -- I get that. Vegas is entirely capable of handling everything else. BTW, I have no interest in Vegas, I'm just a user, nothing more.

[FredThompson]

Sigh...

DTiVo streams are valid MPEG2 but non-standard. If you want properly constructed frames, you must use a decoder which uses all of the MPEG2 spec. It's impossible to have proper audio sync if frames of the video portion are dropped by the decoder.

Edit and mux a VOB with TyTool then note the total length from the chp file. Load that same VOB into another ap. Almost without exception the other ap will report a length which is less because the MPEG2 decoding in that other ap doesn't handle aspects of MPEG2 which are rarely used except in bandwidth-tweaked transports like satellite transmission.

If you take DTiVo and run it through anything other than Donald Graft's DVD2AVI variant or AviSynth as a frame server you will NOT get the full set of fields that are present in a DTiVo stream.

It appears TyStudio and VideoReDo change the structure of the original stream. They can only do this by dropping or degrading some of the data. There's no way to losslessly change a set of inter-dependent data.

[rc3105]

There's no way to losslessly change a set of inter-dependent data.
sure there is. if you make up a set of cliff notes (mpg packets) on index cards (the file) and shuffle (reorder) no info is actually lost. a professor (mpeg decoder with a good buffer) will process all the information correctly. a clueless student (cheapo dvd player, poorly written editing sw, etc) will get confuzled pretty quick


edit: changing timing values within the stream to make it more apparent which packets go where is effectivly a lossless technique. the original timing data is lost, but that's irrelevant because new values have been calculated.

that method doesn't require altering any of the video data which is what most people refer to when speaking of lossless conversion

[AlphaWolf]

Sigh...

DTiVo streams are valid MPEG2 but non-standard. If you want properly constructed frames, you must use a decoder which uses all of the MPEG2 spec. It's impossible to have proper audio sync if frames of the video portion are dropped by the decoder.

I am not sure what you are basing this off of to call it "standard" but everything I throw these streams through accepts them without any issues whatsoever.

It appears TyStudio and VideoReDo change the structure of the original stream. They can only do this by dropping or degrading some of the data. There's no way to losslessly change a set of inter-dependent data.

What are you basing this off of? There is not any re-encoding done whatsoever, the whole thing is completely lossless. Where specifically are you seeing any degrading? I haven't seen any data get dropped with the method I described. If you look in my example, tytool drops a huge segment of video and audio, whereas tystudio/videoredo leave every single bit of video and audio intact. You seem to have things backwards.

[FredThompson]

(Damnit, I hate how this board loses typing if you accidently navigate away from the page!!!)

MPEG2 video consists of periodic full-frame images and 2 types of deltas which are applied in forwards and backwards references to construct adjacent frames. The only way they work properly is if the data is not changed.

rc3105's note card example doesn't work because it destroys the positional component of the data set. Good point about the use of the term "lossless".

Use TyTool to edit just with GOP boundaries then examine the result. You will probably see some momentary pixelated corruption where some of the GOPs were removed. That is because the previous forward-looking pointers are now referencing data which has changed. They are performing properly but the data they rely on is different.

DTiVo also uses some data structures which are MPEG2 valid but rarely used. The overwhelming majority of decoders used in the types of video aps we use don't properly process those so you lose frames.

By "standard" I meant most aps expect shorter, consistent-length GOPs and don't use flags to duplicate fields. Maybe I should have typed something more along the lines of "commonly used MPEG2 sub-sets" because "standard" implies more than what I meant.

I'm not sure about AlphaWolf's sample. It looks to me like a corrupted reception such as what happens during heavy rain. I think we're both "correct" about this. The other tools will work "better" with the damaged data. Working with corrupt data is different than properly decoding all the valid aspects of MPEG2.

btw, I pointed Josh to your samples so they're now part of the bugaboo testing.

[AlphaWolf]

By "standard" I meant most aps expect shorter, consistent-length GOPs and don't use flags to duplicate fields. Maybe I should have typed something more along the lines of "commonly used MPEG2 sub-sets" because "standard" implies more than what I meant.

As far as I know this only applies to DVD, which is inherantly different than generic MPEG-2.

You can shorten the GOPs too however, and it only involves re-rendering one or two frames in the GOP depending on how large the GOP was vs how small you are reducing its size to. Since these frames are very far apart and only come far and few in between, you'd probably never notice any loss in picture quality without playing the video in slow-mo and really looking at the picture. Not only that, but the conversion process is still much faster than a full re-encode, and the video becomes 100% DVD compliant (save for the possibility of the resolution being nonstandard, which is also possible to re-scale without a full re-encode, as olaf mentioned a long time ago.)

FWIW VideoReDo can do exactly this if you configure it properly.

[FredThompson]

I'm sorry if what I typed implied I meant re-encoding the whole stream. That wasn't what I was thinking at all.

IIRC, DVD is about 18 frames per GOP, something like that, and doesn't allow for things like the flag which duplicates a field. Most aps want both fields for every frame so they choke when a field isn't present. Some also don't like the variable long GOP lengths of DTiVo source. DGMPGDec was specifically modified to handle both of those issues properly.

[AlphaWolf]

BTW, what exactly are you referring to wrt to degrading? I don't see any symptoms of it.

I have seen all kinds of programs that handle tystudio processed streams perfectly, yet not a single one can handle tytool processed content without losing sync or cutting out video and audio somewhere along the line. If theres any kind of degrading going on with tystudio, it must be a good kind of degrading. :D

[FredThompson]

Aargh, multiple proper uses of the same word in English strikes again...

I'm not sure which post you're looking at but I was probably talking about what happens when the data a pointer points to is changed. Forward-looking references over a GOP boundary are the easiest to force to show that type of corruption.

FWIW, I've talked with Josh quite a bit about muxing issues and IIRC, he toyed with a method of muxing which made the streams jmore compatible with "outside" editors but there was a valid technical reason not to use it. I'm not sure what he's up to now. He had some time-heavy work requirements and the last he mentioned it to me was looking at some HD issues then was planning to get into things like usdating the DVDAuthor version, multiple menus, etc. That was before he got the 20-30 streams I've gathered which crash TyTool in various ways. I told him about your samples and he asked for the info so I pointed him to this thread.

Oh, now I remember, there was something about padding with extra fields in an attempt to change field dominance flipping. (DG's decoder is supposed to handle this properly. It does handle long GOPs.) Most non-DTiVo MPEG2 streams will have the top field first. DV is BFF. Josh said the stream from the birds will flip dominance. That must be one way they save bandwidth, similar to duplicating fields and using variable-length GOPs. I'm pretty sure the challenge is accounting for the dominance flipping.

That last bit is based on my recollection of a few weeks ago...

[captain_video]

Sorry to interject with an OT post but I just ran a search for VideoReDo and found most of the hits were in this thread. VideoReDo seems to work wonders for editing mpegs and is widely used by Dish PVR owners for editing extracted videos. I found it interesting that when you set up the program for use it asks you whether or not the mpeg came from a Tivo or a ReplayTV. Apparently, all PVRs use a similar non-standard GOP sizing technique to facilitate the PVR features like FF and REW during playback and probably others as well.

VideoReDo has quite an assortment of features built into the software that used to require several different programs to accomplish. It can fix timecode errors (MPEG2VCR), perform frame-accurate editing (TyTools), and many other features that I have yet to explore. I've used it to repair several damaged tystreams that had audio synch issues.

AW - I didn't see any mention of which extraction tool you used to get your ty files and was wondering if you found one to be better than any other for your process. I always hated TyStudio for this because it liked to crash a lot on me. I still prefer TyTools for extraction as it is by far the easiest to set up and use, IMO.

[Rowan]

I use tytools to extract and then tystudio to convert to the ty to mpeg no cuts. I have not tryed tytoold to convert to mpeg because I have read that there are audio sync problems.

EDIT: I just tryed it with a tytool output and did not have any sync problems so it looks like the output from both programs will work. I guess if it does not work with one try the other.

Rowan

[AlphaWolf]

I extract with mfs_ftp, then use typrocess to do a direct ty to mpeg conversion:

typrocess -d 4 -it tivostream.ty -d 4 -demux { -s 2 } -mplex { -f 3 -V } -om tivostream.mpg

Then I edit that mpeg with videoredo.

No matter what you process it with, be it tytool or tystudio, you'll always get proper sync. The issue with tytool though is that it is common to lose huge segments of video or audio (or both) during a period of rainfade. Tystudio is able to make sense of the rainfade areas so that videoredo doesn't have to remove them entirely in order to maintain sync.

[Rowan]

AlphaWolf

What GOP size do you use in videoredo 15?

Edit: Opps I mean 18

Rowan

[bato]

Rowan, do you have problems with the large GOPs with Dtivo streams? or just want to make them more compatible for the future (new dvd player, friends, etc.)? You should select 18 frames for your GOPs.

[Rowan]

Rowan, do you have problems with the large GOPs with Dtivo streams? or just want to make them more compatible for the future (new dvd player, friends, etc.)? You should select 18 frames for your GOPs.


Just trying to make the them more compatible.

Thanks,

Rowan

[mrshock]

I extract with mfs_ftp, then use typrocess to do a direct ty to mpeg conversion:

typrocess -d 4 -it tivostream.ty -d 4 -demux { -s 2 } -mplex { -f 3 -V } -om tivostream.mpg

Then I edit that mpeg with videoredo.



AW, I am trying this approach, but with no luck in getting typrocess to execute thus far. I have assumed that after installing the win32 version of tystudio on my xp machine that I could simply put a .ty file in the same directory as the program files and use this text command (rather then the GUI of tystudio) since the typrocess approach is merely doing a straight to MPEG2 conversion. I have opened up a command prompt on my machine, gone to the program directory and run the command but with no joy. Does your approach work in DOS? At first it appears as though the program runs, as information about the application is announced, etc. but no ouput file is created.

[gyre]

Alphawolf... I'm also a videoredo user.

If I understood one of your earlier posts correctly, you suggested ticking the 'start video frame in new packet' option.

Could you explain if this is indeed the case, and if so why you needed to? I don't currently have this option ticked and wondered if I should.

Thanks!

-- gyre --

[phoenixZed]

AW, I am trying this approach, but with no luck in getting typrocess to execute thus far. I have assumed that after installing the win32 version of tystudio on my xp machine that I could simply put a .ty file in the same directory as the program files and use this text command (rather then the GUI of tystudio) since the typrocess approach is merely doing a straight to MPEG2 conversion. I have opened up a command prompt on my machine, gone to the program directory and run the command but with no joy. Does your approach work in DOS? At first it appears as though the program runs, as information about the application is announced, etc. but no ouput file is created.

I had this issue after I upgraded from XP Pro to XP MCE, the command line would only display copyright info and not process the file. It turns out I had installed an older beta version of tyStudio rather than the final 0.5.0 version. Once I installed the version from 6-1-03 at "sourceforge.net/projects/dvd-create", everything ran as before. Just something to check...

-pZ