MUX'ing, VSplit, and MPG2 files.

[jdiner]

Ok. This is the second new Thread I am going to create. This thread is most definately a discussion of how to produce to mux'ing/working/finished/finalized/ready-to-use MPEG2 files.

Further it is to be a place where we can all trade ideas, discuss code and applications, be it for the Tivo or just for MPEG2 in general. However there are some rules for this thread.

Just like before these are rules for this thread, not the forum. If you want to discuss things deemed off limits here make your own threads and set your own rules. But if you decide to participate here then please follow the guidelines.

So the rules for this thread are:

1- Leave the flames and arguments elsewhere... Again I am interested in getting things working better and moving forward, not a re-hash of the problems covered in other threads. So here be nice or stay out.

2- We discuss ANY tools dealing with mux'ing or viewing or editing MPEG2 files or TyStreams... But let's focus on things that are built for TiVo output as this is a Tivo Forum. I am not the only one working on this, and while I intend to have my discussions of things here I have no arguments again others doing the same. If you find a tool that is better that any you have seen before, then share.

3- NO DISCUSSION OF EXTRACTION. This thread is about MUX'ing and making working MPEG2 files. To do that you have to have elementary streams. So no matter how you get them you have to have them before coming here with ideas or problems.

4- Feel free to discuss various OS wishes/desires. At some point we should support all of them or at least as many as is possible.

5- Let's limit the discussion of VSplit to how it interracts with MUX'ing rather than usability or general feature set. We have to discuss VSplit as it is the core of my current MUX'ing engine. Also in the works is a MUX'er for exisitng VSplit output that will stitch it all back together as appropriate for a tivo file that can be done long after the a/v split has been done.

6- Feel free to ask for features but be aware that I am making no promises except to work on things in general... I mean it. Ask. But don't hound me or any other author, that accomplishes nothing. If you think of something that would be cool let me know. I will add it to the list.

I would like this thread to be about the previously mentioned "technical amazement" rather than usability in my tools, ExtractStream, etc... That is what the other thread and this forum is for. Here if you find something sweet dealing with MUX'ing/editing then let's all talk about it and move forward in getting to that "perfect" point where everyone is happy.

The technical discussion of the mux'ing and what is going on behind the scenes will be held in here. Some don't care about it. If all you want to discuss is tools for extraction/splitting then go to the other threads.

So if anyone has things on their wish list for MUX'ing now is the time and this is the place.

--jdiner

[RxMan]

Obviosly everyone wants muxing in a hope that it will eliminate audio/video sync problems. Has any progress been made with the timestamps and how the tivo handles them?

[jdiner]

Yes. I have had timestamps perfected for a long time. That is why the A/V split via my VSplit code works "perfectly".

There are some having problems with it, but as I explained before, that is an issue with the mux'ing mechanism getting off rather than problems with VSplit.

Once before I mentioned some facts. I have not kept up on them so they are old. But I have had right around 20,000 prefectly split streams reported to me. I myself have done about 300 since January that were perfect.

I use this information in my mux'ing and things stay just as expected. Sync'ed all the way through. Right now I just have to fix the PES streams I am generating and the first cut at mux'ing should be usable.

--jdiner

[FreydNot]

I have been playing with burning SA extracted files to DVD. My goal is to cut out the commercials so that I can fit 4 "1 hour" 3,500kps episodes on a single 4.7Gig DVD.

I've been using DVDMaestro for authoring. I can extract elementry stream using vsplit. That works great. If I use those streams and go strait to DVD, everything stays in sync. If I run them through and editor (M2edit has been what I use most) the audio always drifts.

The last stratagy I tried was triming out 5 pieces which represented the show without the commercials. Then I placed the 5 clips back to back to form a single move in Maestro. Maestro will create a single audio file from a group of clips (and even has a switch to keep them in sync). This function only works on AC3 files.

So I used BeSweet to convert the .m2a files from vsplit into 48k ac3 files. That worked, but the sync was hopelessly lost. Loading the m2a file and the ac3 file into an editor (Goldwave is my favorite) showed they were clearly a different length (even after accounting for the 32k to 48k upsample).

It seems like a simple trim using M2edit should produce a single file with good sync. I'm not sure why it doesn't.

So my question is this: What should I expect from muxing? Will the fabled muxing version of vsplit add some kind of information to the program stream which will let m2edit (or mpeg-vcr or whatever) not get out of sync? I'm starting to wonder if I've been waiting for the wrong tool.

Comments?

[scarabus]

Everybody seems to be looking to muxing as the holy grail of A/V Sync. But as far as I can see it's really only going to introduce additional unnecessary steps.

While the basic entry-level packages will accept multiplexed MPEGs, most of the high end stuff requires elementary streams. So having split our streams and multiplexed them we have to demux again before we can use them, with all the hassle that entails.

On the other hand there are plenty of perfectly good tools that will take elementary streams and multiplex them to form program streams. If we have good, synched elementary streams we will get good synched program streams. GIGO.

We all want accurate sync God knows how many times I've tried to make perfect Max Headroom discs but is muxing really the best/only way to do it? Would it not make more sense to improve the way that the splitter tools handle the raw tystream so that hopefully we don't get bad sync in the first place? And at the same time maybe we can fix some of the other stream handling problems. I've got several instances of streams that I can't use but play fine on the TiVo. I'm getting about a 90% success rate on extraction.

So in my opinion it would be a more sensible use of resources to try and better understand and decode the tystream.

BUT having said that I'm not the custodian of the knowledge and the tools, so if this is the direction that development is heading then I'll be happy to help if I can.

[Pr.Sinister]

The reason why we are all waiting for the muxing is because a lot
of us have edited commercials out of shows we recorded with a
capture card and we never had any A/V Sync problems after editing
out commercials. So we are inclined to think that maybe the way
we mux right now is not perfect.

If i transcode the streams with TMPGEnc and i check the box that
says to create a GOP structure suitable for editing, i have no
problems. But obviously i don't want to waste TIME and QUALITY
transcoding the streams i get off the TiVo... It beats the purpose
of me getting a TurboNet to not have to record from TiVo to
my capture card.

So i will continue to hold my breath for a perfect Muxed stream
that can be edited by ANY MPEG-2 editor.

Pr.Sinister

[jdiner]

Originally posted by FreydNot
I have been playing with burning SA extracted files to DVD. My goal is to cut out the commercials so that I can fit 4 "1 hour" 3,500kps episodes on a single 4.7Gig DVD.


That pretty much sums up my goals as well. Pretty much perfectly in fact.


I've been using DVDMaestro for authoring. I can extract elementry stream using vsplit. That works great. If I use those streams and go strait to DVD, everything stays in sync. If I run them through and editor (M2edit has been what I use most) the audio always drifts.


The problem is, as I mentioned a long long time ago in thread far away, that MPEG-2 is a very very... forgiving spec. Things that are perfectly legal are honestly... a bad way to do it. The much tighter specs used for a DVD force limits that make recovery after bad data much much easier.

For instance. In MPEG-2 it is possible and strictly speaking legal to have hundreds of Frames (I, B, or P) in a single Group-Of-Pictures (GOP). For a DVD standard spec is fixed at either 16 or at 18 if I remember the spec correctly. Anything other than the exact number is just plain not in the DVD spec.

Now since most modern DVD players are also built to play VCD and SVCD disks this is not a problem. For most as long as it is a valid MPEG-2 file, not just a DVD spec file, it is playable. In my slightly modified APEX-600a player you can play just about anything right up to and including muddy-water.

Back to the point... The drift most people are running into is a simple matter of "assumptions". The MPEG-2 editors I have tested all have assumptions about not only what is legal but what form and layout it will be in. The output of a tivo TyStream is NOT INVALID, make no mistake about that. But for whatever the reason it is seriously "non-standard". When the assumptions get in the way you get errors.


It seems like a simple trim using M2edit should produce a single file with good sync. I'm not sure why it doesn't.


While I can't answer this definatively having not written this tool... I hope I have provided some answer to this questions.


So my question is this: What should I expect from muxing? Will the fabled muxing version of vsplit add some kind of information to the program stream which will let m2edit (or mpeg-vcr or whatever) not get out of sync? I'm starting to wonder if I've been waiting for the wrong tool.


Yeah. Once you build all of the absolutely correct information into an mpeg stream m2edit appears to edit it perfectly. At the moment the size of the file has bloated a bit to get to this stable point but it does work.

Now all the normal caveats... It works for me. Your mileage may very right up to the point of complete failure. But since it does work for me on what I am doing it is reasonable to expect that at some point soon it will work for you.

As for waiting for the wrong tool. Keep in mind that this is all steps:

Step 1 - Getting a good mux produces files that can be played on a PC without errors.

Step 2 - Expand this fix so that it can be played on a DVD/VCD player either as a VCD/SVCD disk or a DVD disk.

Step 3- Add the promised, and somewhat fabled, visual GOP editor. This will allow the cutting of commercials and other things from the Tivo streams. The output of this will either be a single mpeg-2 file that is everything but what is cut, or a set of files broken on each cut point...

Step 4 - Extend the mux'ing output file format to be a DVD... crap. I forget the name of the DVD record format for a single "block" on the disk. Anyway, so that what is produced is directly burnable with no processing by other programs needed.

This is what I intend to do. If this fits your needs, then someday soon you will have what you need. If not, then keep looking elsewhere or suggest to me another step that will work for what you need.

--jdiner

[jdiner]

Originally posted by scarabus
On the other hand there are plenty of perfectly good tools that will take elementary streams and multiplex them to form program streams. If we have good, synched elementary streams we will get good synched program streams. GIGO.


If you have a stream that VSplit does not produce a good split on then send me a copy of it. And I will fix vsplit. But please before we get to that point prove to yourself that it is really not splitting correctly.

Do the following as exactly as possible since it is what I do to prove such things:
1- Grab VirtualDub's latest version. Grab WinAmp's latest version.

2- Grab DVD2AVI version 1.74. And the related VFAPIConv program.

3- Get Vsplit. The latest version is old now but still the latest.

4- Run the TyStream through VSplit.

5- Check the size of the audio and video to make sure you really have a file for each of about the right size. Video+Audio file size should be 95% or so of the size fo the TyStream itself. Also check the output VSplit to make sure no "real" errors were reported.

6- Check the audio file in WinAmp by playing part of it.

7- Check the video file in WM or some other player to make sure it is playable.

8- Reconfigure WinAmp to play to an output file and send the output there.

9- Load the video file into DVD2AVI and with NO configuration changes save a project with it.

10- Convert the project to a "fake avi" with VFAPIConv.

11- Load this "fake avi" into VirtualDub. Set the video compression to RGB uncompressed. This is technically "no compression". Since we won't be making a output AVI this is fine if we were the file size would be massive.

12- Open the audio file. And set the sync with the Audio Interleaving option.

13- Play it at very start of the file and make sure A/V lip-sync is correct.

14- Then seek to whatever location in the files you wil and play starting there. And watch for sync. I have done this countless times with tons of test clips. (Each of the 300 clips I have done for my own archives I have done exactly this with to remove commericals...)

That's it. For me sync is perfectly maintained.

What can can sync to get off in the above process:

1- The biggest culprit is the audio compression. For some unknown reason, unknown to me because I have never ever looked into and have no plans whatsoever to do so, if you compress audio into MP3 format to make it smaller (which I do) it gets off. The playback speed of the MP3 audio stream is actually off. You can correct for this and maintain sync.

2- If you use a video compressor depending on the type and strength of compression the sync can be lost as well. Frames sizes to large? I have no idea I just know that it happens. Neither DivX nor XVid have ever gotten off for me, but some of the others have during testing...

These are the only 2 culprits I know of.

I meant it when I said use the above. I know there are a ton of other tools that can be used at each step. But I can't predict the interraction of the programs or any bugs/problems they might introduce. So use the above, if it is still off you have found a problem I have not seen before. If it is on you are suffering from the same problem as everyone else when it comes to dealing with the elementary streams themselves and having the software MUX them.



We all want accurate sync God knows how many times I've tried to make perfect Max Headroom discs but is muxing really the best/only way to do it? Would it not make more sense to improve the way that the splitter tools handle the raw tystream so that hopefully we don't get bad sync in the first place?


If you have found a problem in a TyStream not related to a bad extraction directly, i.e. you got it all or to an emu/hack card that once before was just randomly dropping chunks and VSplit fails on it then I can fix it once I get the TyStream itself.

So run the above and we can decide on a direction to move in at that point.

--jdiner

[scarabus]

Originally posted by jdiner
If you have a stream that VSplit does not produce a good split on then send me a copy of it. And I will fix vsplit.

I have several. See my response in your other thread.

I'll be happy to snail-mail the streams to you on DVD - send me a PM with an address to mail to.

Alternatively I can FTP them if you have a site with enough storage. It'll probably take the same length of time :-/


Originally posted by jdiner
So run the above and we can decide on a direction to move in at that point.

I will do that thing.

[jdiner]

Originally posted by scarabus

I have several. See my response in your other thread.

I'll be happy to snail-mail the streams to you on DVD - send me a PM with an address to mail to.

Alternatively I can FTP them if you have a site with enough storage. It'll probably take the same length of time :-/


Yeah. That is why I want you to use the TyFileSplit tool I wrote. It will let you identify a single small block that has a problem. 20 meg is easy to upload if you have a decent link. If not then dvd is probably best.

--jdiner

[scarabus]

For the benefit of everyone else you can find TyFileSplit
HERE (http://dealdatabase.com/forum/showthread.php?s=&postid=38137&highlight=tyfilesplit#post38137)

[RxMan]

I guess I am a little confused. Is TyFileSplit different than using TyTool in VSplit mode? I have one file that has sync issues. I still have the file on my Tivo so I can extract again and follow the steps above if necessary. Just let me know if there is a difference and I will proceed.

[jdiner]

Ah ok. To try and clear that up as people might get confused.

All that TyFileSplit does is split a TyStream file into a smaller valid TyStream file. I.e. it stays on the byte-count correct boundries etc...

It does not do the A/V split just cuts something that can then be processed later. Namely by me when looking for problems.

Using split in both names might have been a bad idea... :(

--jdiner

[laserfan]

Originally posted by jdiner
...
14- Then seek to whatever location in the files you wil and play starting there....That's it. For me sync is perfectly maintained.

What can can sync to get off in the above process:

1- The biggest culprit is the audio compression. Uff da! 14 steps, wow. Here are 4+ steps that work for me, noting I have an SA and need to convert audio (you're right of course about audio compression as a key problem):

1. Mux the m2a and m2v files using MPEG2VCR. If the streams are "off" it may take a couple tries to get the AV sync right (quick though, as a "try" need only take 10-20secs to output enuf program to check sync), but it's done here, right up front.

2. If there's any trimming to do, I use MPEG Cutter from Canopus, remember my audio is still 32kHz.

3. Import the resulting trimmed mpg to MPEG2VCR again, this time use "Record" and set Audio to 48kHz (if for DVD) or 44.1kHz (if for SVCD), and change audio bitrate also if you like, but this results in the export taking a little longer.

4. Test! Done! Burn w/Nero or import to SpruceUp to author a DVD. This whole process takes only a few minutes for an hourlong program; MPEG2VCR is really fast, fast, fast both for muxing and for transcoding audio. Still, for a movie I do walk away from it and do something else for a while.

Note that MPEG2VCR has mpg-cutting ability, but altho I have waxed eloquent about this tool in another thread, alas I've discovered that if you use IT to cut it results in audio drift. I can only think that while Canopus' tool cuts on I-frames, and MPEG2VCR appears to NOT, that this somehow is hosing the audio. Also note that converting the audio before cutting doesn't work--sync is lost again. So I leave the audio alone until the very end.

It is the case for me now that the lengthiest part of the creation process is the transfer from Tivo-to-PC over my 10baseT Tivonet board. These other steps have become almost trivially easy, tho occasionally I find a pair where it's difficult to find the audio "sweet spot" especially for the music programs I'm archiving.

A final caveat is that while the above allows front-and-back trimming, it does not allow (or I have not found out how to do) excising from within e.g. to cut commercials. But an easy thing for me to do, if for example I want to save two songs from a Sat. Nite Live, is to cut each song out of the same muxed (again 32kHz) file and save them separately, and import each separately to MPEG2VCR to transcode the audio.

[Wes]

Originally posted by laserfan
Uff da! 14 steps, wow. Here are 4+ steps that work for me, noting I have an SA and need to convert audio (you're right of course about audio compression as a key problem):


The 14 steps is to prove the sync from vsplit was perfectly maintained. It's not the process that we usually would do.

I've never have sync problem with my SA. I only have the sync problem very occasionally with my DTivo. (This is a DTivo forum, not SA. Please don't compare apple and orange here. ;))

[lmurray]

Jdiner,
over in this thread:

http://www.dealdatabase.com/forum/showthread.php?s=&threadid=16133

we've been talking about tools for macosX. I've compiled some of the basic tools for macosX, but would really like to get a binary of vsplit13 for splitting ty files. Can we work to compile vsplit13 on macosX? I'd be glad to do this if given the source. Most of the other programs I've compiled (dare i say port) required little to no work to compile on macosX.


Thanks in advance,
-lloyd-

[jdiner]

Originally posted by laserfan
Uff da! 14 steps, wow. Here are 4+ steps that work for me, noting I have an SA and need to convert audio (you're right of course about audio compression as a key problem):

1. Mux the m2a and m2v files using MPEG2VCR. If the streams are "off" it may take a couple tries to get the AV sync right (quick though, as a "try" need only take 10-20secs to output enuf program to check sync), but it's done here, right up front.


Ok. Back to what I was addressing. This is not an optimal solution. It was never advertised as that. IT IS A TEST SOLUTION THAT ALWAYS PRODUCES IN SYNC A/V FOR VERIFYING THE OUTPUT OF VSPLIT AS CORRECT... I am not making DVDs here. I am not making MPEGs here. I am not making avi's here. I wanted people to know how to detect if it is the mux'er/software, or the output itself that was getting off.

--jdiner

[justintraining]

I have benefited greatly from those on this board, and wish I could contribute more to the "cause".

At this point, the most I can contribute is an FTP site for sharing files as needed.

ftp://63.165.143.229
Anonymous access is allowed.

I am also looking for a few individuals to help maintain the site. You know sort, filter and generally keep the site clean. Please e-mail me for a name/password with more rights.

As we progress with this thread, and others, I plan to setup a website with downloads and “How-To” information. Again, I will be looking for help in this area as well.

My e-mail address: justintraining@hotmail.com

[jdiner]

Ok. Here is what I need to get on to the next step. Some MPEG-2 multiplex pieces.

Out there are people that are using mux'ing tools to recombine a TyStream split with my tools that are maintaining sync. I need some. Not all of them probably a 20meg chunk is enough.

Now I am not talking about streams that can be edited. All I am talking about is streams that when mux'ed with whatever work in playback and keep sync.

OS doesn't matter to me. The tool doesn't matter to me. etc... etc... etc... All I need is a 20meg or so piece of a working MPEG-2 file from a DTIVO.

I can obviously find MPEG-2 files from other sources. But these need to be DTIVO specific sources. Further I need if at all possible multiple pieces of the "same type" of show. I.e. the first 20 meg of 3 of 4 episodes of Show-X. As they would by and large all be similar in size and content.

To start with what I want is a show of hands. Who has, without editing, produced good MPEG-2 files and with what tools. What OS. Etc... If I can make my own I will and not need uploads. If not I will need some uplaods.

The reason for this is that I have found a problem in some of my calculations for the multiplex fields. If I can find values that work I can hopefully determine what the difference is and get things working accordingly.

So chime in and help us all out. :)

--jdiner

[dlang]

I have a bunch of mpegs from a dtivo, but I don't have many I would say have good sync on them :-(

one thing I have noted is that if the recording happens across a program break (i.e. I start my recording of thunderbirds 1 min early) the av sync error reported by vsplit is _very_ different if I have it split the entire show vs if I have it skip stuff at the beginning (to cut out the tail of the previous show)

I don't know if commercial breaks cause similar problems

[Rotten]

jdiner,


Ok I have been able to produce good snyced video from tytool5 muxing with Multiplex (Not simple Multiplex) in Mpeg tools in TMPGEnc Plus. This gives me a great MPEG every time. However This file can only maintain good sync if played independently or if burned to DVD without editing in Spruce. As soon as you introduce it to any other program like DVD workshop or MPEG2VCR the snyc is gone and gone bad!. I will continue testing as I would prefer to use other tools to burn with. If I can ever be of service please let me know. Keep up the great work.

[homer08]

I just tried TEMPGEnc Plus Version 2.57.41.146 using the multiplex function described above and it worked great. Weeks ago I had tried a couple of other tools (can't remember which) and was not able to get a sync'd mpg.

[bronco13]

Where is everybody getting TEMPGEnc Plus Version 2.57...... from???

Is it on justingtraining's site???

Could someone upload it there?

[jfkjfk]

I can't find Canopus MPEG cutter any where.. I'm understanding it's part of they're software/hardware package.. I've also read that it only install on machines with a Canopus card in them.. anyhow.. anyone know if this is true or not.. I've played with everything I can find.. M-2 Edit works ok, but it CHUGS so slowly and crashes when I try to use files over about 600megs...
ANyhow.. Can anyone help me out with MPEG Cutter.. Does it work well with large files 1-2gig..

thanks.

jfkjfk

perhaps someone could u/l to the ftp site mentioned above..

[jfkjfk]

Just played with MPG2VCR for the first time.. You guys are mentioning sync problems with it.. When I preview files from DTIVO d/led and split with tytool.. I remux using mplex.. Audio sync at beginning of file is on.. When I get further into a 1.5GIG file say an hour in I notice that the preview is WAY OFF sync.. Where talking a couple seconds.. I tried extracting a portion of clip.. Same result..
You guys are talking sync problems.. I just don't think these mpeg editors are properly built, and they're flaws just show up when we use these HUGE tivo files.. Who else is chopping up 2gig MPEG files? Same issues I had with M2-Edit.. Works fine on small files, can't handle huge files..
Currently there a program I use to convert MPEGs to divx.. Sync seems very good if not perfect.. There is no audio in the priview, and the frame accuracy for edits is horrible.. In fact even scrolling through is not very accurate, but it's UI flys.. And it's open source.. I've considered trying to get back into programming, but I haven't done windows or any C++ since 1994.. So perhaps someone with a little more experience would consider adding a few thngs to it... anyhow.. it's a little hard to use at first.. There is an export n frames function that start at the frame your preview and exports n frames.. It's very fast.. Uses flask type plugins.. anyhow take a look.. Perhaps we can mold this into a nice custom tool...
<http://www.mpeg-mediator.com/>
I use XMpeg to scan and find my in and out points(File-info-current video variable), and then I compute the n frames for export of a divx avi.. Then I chop further with VirualDub..
--edited this link so you get enlish page---
<http://www.mp3guest.com/Xmpeg_Index.asp?l=US>
these are the best two tools I could find... All other tools I've used including FlaskMpeg, crash, or lock up for strange reasons...
Unfortuately I still haven't got a good MPEG cutter.. After seeing the ease at which these two programs fly though an mpeg, and the accuracy/sync of MPEG MEdiators there has to been something out there.. But I've given up hope, and think that trying to add a little more functionalitly to MPEG mediator may be the answer.. anyhow bla, bla, bla.. I might be telling you all stuff you already know.. But I have spent at least 30 hours over the last couple week messing with tools trying to find something to CUT these large MPEGs well..

anyhow.. that's probably enough for now...

jfkjfk

[laserfan]

Originally posted by jfkjfk
...Can anyone help me out with MPEG Cutter...I'm the culprit, having mentioned this a couple of times. Sorry, but I've looked into this, and not only does it not INSTALL without a Canopus card in the PC, I found out yesterday that if it's installed and then you remove the card, it still looks for the card and therefore doesn't run without it. I don't know how to defeat this or I would, so I could do some editing on other PC(s) I have here.

[dattrax]

I've been getting great results from TyTool

1, TyTool
2, bbmpeg (use the offset)
3, Mediaware solutions m2-edit
4, TMPEG to reencode to DVD compatible file

takes about 10minutes to step 4 and then the cost of a TMPGenc encode.

I also use ssrc and toolame to get the audio, as tmpgenc is lacking.

The only feature I could do with is the 'bbmpeg' removed

Jim

[jfkjfk]

dattrax,
what kind of file size are you throwing at M2-edit?
I had problems with M2-Edit and large files.. It's way too slow... Pause of 5-10 seconds or so every time I try to seek into the file.. It worked ok for files of 500Megs or so... but was still pretty slow..
Is the UI slow for you?

jfkjfk

[jfkjfk]

laserfan,
Does mpeg cutter work well with large files? I would have no problem paying for something that works well... I don't want to buy something that makes me struggle.. Crashses. locks up.. or takes a 10 second pause every time I try to move the seek pointer...
what type of card does it require? How much $$

thanks..

jfkjfk

[jdiner]

Ok. I have run a bunch of tests. With BBMpeg muxer I get massive errors. Off in the first 3 minutes by over a second.

With the latest version of tmpgenc's muxer I get what appears to be good sync all the way through. It was hard for me to tell if it was getting off or had just stayed off. The test clip I used was off by 11ms and I can see that. I expect most can.

My next set of tests go with the new mjpeg mux'er. I got it installed on my FreeBSD box. Man the /usr/ports stuff is very very nice... :)

Once I determine what is really working the best it will help move things forward. My own works. It just doesn't work everywhere in each player. I am trying to figure out why...

--jdiner

[dattrax]

jfkjfk

No problem in m2-edit with any files at all, UI is not blazingly fast, but I wouldn't call it slow.

I can chop out a commercial in about 30 secs.

Jim

[dattrax]

jdiner,

I should have mentioned how I multiplex for bbmpeg. I select the option for multiplexing a DVD stream, and then add (or subtract) the audio offset given by TyTool.

If you use both of these then bbmpeg should keep it in sync

Jim

[jdiner]

the bbmpeg output is still bad for me. But that could be any number of things. However both the tmpgenc and the mjpeg mux correctly as long as no editing takes place. I have no clear idea yet how either affect the editors as I am now using new versions of both and my editing tests are old.

--jdiner

[jfkjfk]

jdnire,
are you using the dos compiled mjpeg muxer or unix muxer?

what settings are you using?? what's you command line?

thanks..

jfkjfk

[jdiner]

I used the Dos compiled one posted here and am getting ready to use the unix one...

The command line I used was:

mplex -f8 -V -o ..\da-mjpeg.mpg ..\da.m2v ..\da.m2a | more

The more is just so I could watch the output. The rest is pretty simple. -f8 means mux for a DVD. The -V puts it into verbose mode. And the -o sets the output file.

--jdiner

[jdiner]

Very interesting. The same jerkiness that I was seeing in the output from my mux'er code is in the BBMpeg output. Since that is where I used to determine a few things that makes sense. Perhaps it is version of bbmpeg. Are those that are being successful with it using something other than:

version: AVI2MPEG - v 1.24 Beta 13.

It was the newest thing I could find.

--jdiner

[dlang]

actually I think the -V makes it work with variable bit rate files, I think you want to look at -v (lower case) to get more verbose output.

[jfkjfk]

Originally posted by jdiner
I used the Dos compiled one posted here and am getting ready to use the unix one...

The command line I used was:

mplex -f8 -V -o ..\da-mjpeg.mpg ..\da.m2v ..\da.m2a | more

The more is just so I could watch the output. The rest is pretty simple. -f8 means mux for a DVD. The -V puts it into verbose mode. And the -o sets the output file.

--jdiner
Yeah.. That's about same here.. Isn't -V for variable bit rate? I do use it.. I was trying to use -S to split the file up but it wouldn't work with -f8 but would with -f3?!?! (I don't know the difference)
Why when using -f8 does it say...
INFO: [mplex] Selecting DVD output profile (INCOMEPLETE!!!!)

Also I ran some muxes that I did through a demo of m2-edit's pro 5, DR M tool.. It complains about all sorts of problems.. Maybe this will help you?!? I attatched it.. I dunno if this tools is right or not, but here it is..


jfkjfk

[wh1212]

As we all know, Dtivos are vbr,and SA's are cbr or vbr,Your choice.
There is nothing wrong with, Tytool5 it does one thing great,everytime WOO HOO, Yes it's out of sync -33ms for every 30mins recorded,-66ms for 1hr and so on, BUT to notice, AVsync would have to be out by at laest, +or- 500ms.
Ok back to cbr vbr and muxing,You must mux cbr in cbr,M2edit
vbr in vbr,mpeg2vcr etc.
Play in powerdvd,etc.
Out of sync,HUMAN ERROR, Do it again.
Are we in sync, If not find a guide.
Ok editing,The key,Do not RE-ENCODE,You need to cut and join.
Guides at vcdhelper.com(How to cut and join)
Play movie, out of sync,HUMAN ERROR,Do it again.
Success.
Ok do yourself a favor,DVD media 4.7gigs,love it.
Mpeg2 hate it,It has to be,10Mb per sec or it looks like a cheap vcd.
Mpeg4 is now open source, Which means set top boxe's,coming to a store near you.Why have only 2 shows, on a dvd,when you
can have, 10 or more,at better quality. Try it,use gordian knot,Guide at Doom9.org or net ?.

[pbar]

Originally posted by wh1212
As we all know, Dtivos are vbr,and SA's are cbr or vbr,Your choice.
There is nothing wrong with, Tytool5 it does one thing great,everytime WOO HOO, Yes it's out of sync -33ms for every 30mins recorded,-66ms for 1hr and so on, BUT to notice, AVsync would have to be out by at laest, +or- 500ms.


Personally, I find that 100ms out-of-sync is almost unwatchable... (seems that you buid up some amount of error for each 'part' of the tystream?)

I may just be imagining this, but if I extract a program as 'separate parts' using tytool and mux them seperately (I use mplex) then the sync error doesn't build up and I can then burn them to DVD as seperate chapters.

Paul

[jdiner]

Originally posted by wh1212
As we all know, Dtivos are vbr,and SA's are cbr or vbr,Your choice.
There is nothing wrong with, Tytool5 it does one thing great,everytime WOO HOO, Yes it's out of sync -33ms for every 30mins recorded,-66ms for 1hr and so on, BUT to notice, AVsync would have to be out by at laest, +or- 500ms.

I notice a whole lot less thatn 500ms. But how did you determine how far it is off? I have tested and tested and tested and I never see anything off at all. If I get right at the start I stay on. Fill me in. Maybe I am missing something. Maybe in your case the extract is actually bad.

--jdiner

[laserfan]

Originally posted by jdiner
...I notice a whole lot less thatn 500ms. But how did you determine how far it is off? I have tested and tested and tested and I never see anything off at all. If I get right at the start I stay on. Fill me in. Maybe I am missing something. Maybe in your case the extract is actually bad.First you say "I never see anything at all" and then you say "If I get it right at the start I stay on"? What are you saying? Have you seen the problem or haven't you?

I have seen m2a and m2v extracted via TyTool that are off at the start by as much as almost a second (1000ms last time I checked). These can be muxed to stay in sync, unless any edits are attempted. I have also had extracts that are close enough to not notice any problem at all, but...if edits are done, all bets are off--sync goes to heck at edit points.

[jdiner]

Originally posted by laserfan
First you say "I never see anything at all" and then you say "If I get it right at the start I stay on"? What are you saying? Have you seen the problem or haven't you?


Ok. I can tell I was not clear enough. The first comment stands alone. I don't see any of the problems being mentioned. I ran another 5 shows through last night as a test. No problems. No loss of sync. Nothing...

The second statement should have been a seperate sentence:
"I have tested and tested and tested and I never see anything off at all. If I get right at the start I stay on."

Way back when all of this started. There were times when things were off at the start and stayed off. I no longer see any of these. (See statement #1) So my followup comment was to mean, "I start on, I stay on, I finish on..."

So if you have something that does not. Then clip it and send it to me.

Bottom line, there should be no way to get off by 1000ms... The code and the Tivo doesn't allow for more than 33ms. And in the TyTool5 and VSplit #13 there should be no way to be off by more than 17ms.



I have seen m2a and m2v extracted via TyTool that are off at the start by as much as almost a second (1000ms last time I checked). These can be muxed to stay in sync, unless any edits are attempted. I have also had extracts that are close enough to not notice any problem at all, but...if edits are done, all bets are off--sync goes to heck at edit points.

Editing is a totally different issue. It can and will smash things and has nothing to do with current splitting issues. As for the getting 1000ms off. Send me the clip. I would love to figure out what is going wrong...

--jdiner

[jdiner]

Well I am right back where I started. I am getting so tired of this...

Nothing makes any sense. I have been working over the mux'ing output. The 2 things tha work "almost" for me in my testing are the DVD form in BBMpeg and the tmpgenc muxer.

However in disecting the files to see what is really going on within them to find out what I did wrong I can't believe they work now or ever have. The timestamps in them are completely smashed. They have things set to decode long after they set to be presented.

I just don't get it. Nothing seems to follow the docs/specs. What doesn't run well seems to be closer than the rest of it.

I am lost as to what to try next. It should not be this hard. I have said that before. I will say it again. It should not be that hard.

I am open to suggestions on where to look or what to try. Anyone?

--jdiner

[omf]

Originally posted by jdiner

I am open to suggestions on where to look or what to try. Anyone?


Are you always muxing DTivo audio/video streams to check your output? You might want to start with source from somewhere else to see if it makes any more sense. For example, capture some DV video and create elementary audio/video streams from it. See if the muxing of those streams looks as screwy as what you're getting when you start from the DTivo streams.

[laserfan]

Originally posted by jdiner
...I am open to suggestions on where to look or what to try...I am a rank beginner at all this so take the following for what it is worth:

MPEG2VCR (Womble) can take your vsplit-output m2a and m2v files and mux them together so that they are in sync. This proggie also has a Repair utility built-in for subsequent checking of these files, though it only works with 720x480 programs (which I get out of my SA Tivo). It detects GOP time code errors, audio PTS errors, and GOP size errors and repairs them. The resulting files can be imported into various authoring tools e.g. SpruceUp, which does some <significant?> checking of these files on import.

All I am saying is that it appears that these muxed files are DVD-compliant, therefore I would assume that someone <much smarter than I> could look at streams before & after and perhaps this would help you with what is going on...

Note I say this without knowing what tool one might use to "look at" an interlaced mpg program, much less make sense of it. Only throwing out ideas here hoping to give you a fresh "angle" to look from.
:confused:

[Elwood12]

I am a not very knowledgable on the whole Muxing thing. And I hope this is not too newbie or off topic but it seems that the problem everyone is having with whole audio syncing is that something is missing after you spilt the ty file into m2v and m2a files.

After searching these forums and the web it seem that the reason is to get rid of the tivo PES headers. These PES headers from tivo I have read contain the time stamp info. I also have read that the tystream is a form of Mpeg2 file already. That said, can we replace the tivo PES headers with the correct Mpeg2 headers we need and skip the splitting to m2a and m2v?

If I have made any wrong statements please tell me so.

[jdiner]

Yeah. More programs I fear is the key. I am grabbing a few more now to try it out with.

The output from the nanoPeg program is garbage. The Xmuxer won't install for me for some reason.

I was hoping to find something good to emulate rather than continue to try brute force with the long road to getting things working that that can be.

As for trying other things. I have. I have downloaded a number of things from the net as source mpeg-2 examples. Most of them are "off" in similar ways. But nothing ever really repeats.

Like I said I don't get it. The frame orders make no sense. The docs are clear. It should go: IPBBPBBPBBPBB in the first GOP and then in every GOP after that it should be IBBPBBPBBPBB. Which gives PTS and DTS timestamps in the normal fashion.

In looking at the current output it is not even close to that. I am going to keep digging into things. It is just not matching up to anything that is expected.

--jdiner

[jdiner]

Originally posted by Elwood12
I am a not very knowledgable on the whole Muxing thing. And I hope this is not too newbie or off topic but it seems that the problem everyone is having with whole audio syncing is that something is missing after you spilt the ty file into m2v and m2a files.

After searching these forums and the web it seem that the reason is to get rid of the tivo PES headers. These PES headers from tivo I have read contain the time stamp info. I also have read that the tystream is a form of Mpeg2 file already. That said, can we replace the tivo PES headers with the correct Mpeg2 headers we need and skip the splitting to m2a and m2v?


No this is in essence exactly what I am trying to do.

One thing that should be made clear. The TIVO has headers in it that are part of what is going on. But they ARE NOT PES HEADERS. They are "close" and contain some similar data. But they can't be used as is. And the replacement process is what is causing so much trouble. If they were just there for real then this would be easy. Just dump the data in a new way and we would be there.

--jdiner

[FreydNot]

We might find some hints by looking at the hardware encoders. Does anyone have access to the datasheets for the Sony encoder? I know the TiVo still changes the data it gets from the encoder, but I'm thinking there might be some similarties in the specs of the encoder. Maybe it will ring some bells.

Also, I know there has been discussions about the first block in the ty file in the recent video insertion threads. Something to the effect of there being data in there that most all extraction programs jump over and ignore. I know the data in that block is key in getting data back into the TiVo. Maybe some secrets of the stream lie in there?

Just my thoughts...

[Elwood12]

I am starting to get a handle on this. I guess another way to go at it may be to see what is it you are trying to get and go backward.

If we have a valid mpeg2 file can you convert it into a tystream?

And if you can't what are you missing?

I don't suggest you actually spend too much time trying to do this as there would be many problems doing it but it may shake some ideas loose.

[FreydNot]

I haven't personally done any work with inserting video back into TiVo, but I've read about it here http://dealdatabase.com/forum/showthread.php?s=&threadid=16238. I know they are starting with standard mpeg2 files and using a progam on the TiVo called ele2pestripple (which I assume means Element To PES Tripple). Does PES Tripple mean anything to anyone?

Jdiner, you might want to look at example MPeg2 files from the mpeg.org web site. You can download reference test patterns from http://www.mpeg.org/MPEG/video.html#video-test-bitstreams

It might also be worth looking at the output of the TMPGEnc encoder. Could be hints there.

[TheDoctor]

Originally posted by jdiner
Like I said I don't get it. The frame orders make no sense. The docs are clear. It should go: IPBBPBBPBBPBB in the first GOP and then in every GOP after that it should be IBBPBBPBBPBB. Which gives PTS and DTS timestamps in the normal fashion.

In looking at the current output it is not even close to that. I am going to keep digging into things. It is just not matching up to anything that is expected.

--jdiner

Most of what I know about the stream I got from stare and compare, but I see very little difference in the stream I look at from a tivo, dtivo or replay once you get past the format changes, replay just uses a few diffrent headers and breaks up the data more. From you discription you I am not sure what you mean about the frame order and time stamps?

The frame ordering/time stamps I have seen on the Dtivo under 2.0 have always seemed consistant. I have never really tried to match it back to the spec's. I think the way they write it the DTV transport stream would look something like:

1I 0B 3B 2P 4B 6B 5P 7B 9B 8P...

With the numbers being the timestamp seq. Of course this is from memory so I may have botched it.

[jdiner]

Originally posted by TheDoctor
The frame ordering/time stamps I have seen on the Dtivo under 2.0 have always seemed consistant. I have never really tried to match it back to the spec's. I think the way they write it the DTV transport stream would look something like:

1I 0B 3B 2P 4B 6B 5P 7B 9B 8P...

With the numbers being the timestamp seq. Of course this is from memory so I may have botched it.

No, you are right. You just took my comments out of context. No worries there. What was being said was that when I used one of the "standard" mux'er software programs out there to try and figure out why mine jerks and theirs is smooth the resulting file is garbage. It is mux'ed. It is a DTivo source. But the output frame orders and the SCR, SCR_EXT, PTS/DTS values are all jumbled.

If anyone really wants to see it I can print them out for you. Often things are set to decode almost a full second after they are displayed etc... Junk like that.

I continue to work on mine irreguardless of what I see in those but I was looking for a bit of a short-cut in figuring out how some of those values are calculated... But all in all it is kind of wierd in what is going on inside of the mux'ed files.

--jdiner

[Elwood12]

What are you guys using to view the file? I assume it is not some sort of movie player but a hex editor? I would like to take a gander at the streams to see if I can notice anything.

[jdiner]

Some people were using a program or set of programs from the bbmpeg author.

I use a pile of tools I wrote myself. Don't think they will mean much to anyone but me. They are just the better part of an mpeg-2 decoder.

I also use HexEdit V1.0 from Andrew Phillips.

--jdiner

[TheDoctor]

For this kind of work a good hex editor is your friend, as is a good compiler.

[pbar]

I found it useful to add a few printfs to the mjpegtools mplex to dump the PTS and DTS of each stream while it's muxing them. It's fairly easy to visualise them using something like gnuplot by plotting them as offsets from the system clock...

I haven't had chance yet to work out why sync sometimes seems to drift for me over a long stream. (my guess is that it's the muxing program trying to aim for a particular bitrate and not dealing with buffer under/overflows very well)

[stealthdave]

Given the thread, I thought I'd share my experiences with Tivo stream-splitting and muxing.

First, here's my setup. I have a Phillips 14 SA Tivo that has been expanded to 80 hours by adding a 60G drive, and is running 3.0 Tivo software. I have the TivoNET adapter installed (not the newer TurboNET, unfortunately). My desktop system is running Mandrake Linux 9.0Beta3 on an nForce/Athlon system (I'm running a beta due to hardware compatibility issues, and it's actually quite stable). I used to dual-boot with various flavors of Windows, but I no longer have a Windows partition. I use various flavors of Wine for my Windows compatibility needs.

I use the latest version of jdiner's vsplit on my Ty streams, and when it can read the stream, it works flawlessly. I have some streams recorded on an earlier software rev (2.0 or possibly 1.4) before various upgrades that can't be read by vsplit, and I've sent jdiner samples of them a while back.

For muxing, I use one of two programs: mplex from the mjpegtools or tcmplex from transcode. I believe that tcmplex is based on bbtools, but I'm not certain. The transcode website can be found here (http://www.theorie.physik.uni-goettingen.de/~ostreich/transcode/).

I've found in general that tcmplex works better at keeping sync while seeking through an MPEG stream than mplex, but tcmplex doesn't always work split Ty streams. When it works, it tends to keep sync better when jumping around in a player. The main downside or upside of tcmplex (depending on how you look at it) is the lack of configurability of tcmplex. There's fewer options to play around with, but it's simpler to use and the default options tend to work best.

Here's the command-line that I typically use with mplex:

mplex -f 3 -b 260 myprog.m2a myprog.m2v myprog.mpg

The -f 3 specifies MPEG2 and the -b 260 sets the buffer available to the player. -b by default is 46, which is appropriate for VCD. Around 260 is what's recommended for SVCD. In my experience, this leads to better sync when seeking through an mpeg stream. Adjust to taste, YMMV.

btw, another developer of Tivo tools, Warren Toomey, recently posted a Ty stream splitting tool to the ExtractStream Yahoo! Group based on his playitsam tool which he calls "vsplit". This is obviously not the same as jdiner's tool (which works much better, IMO; I can only seem to get segfaults from Toomey's tool), but I thought I'd point it out in case anyone else gets confused. I didn't look closely enough at the author when I first saw it and thought that it was a new version of jdiner's tool. The source for Toomey's tool is available from the mailing list archives (you must be a member of the list to view the archives). His source is released under GPL, and so might provide some useful code for others to build on.

I hope that you find at least some of my lengthy discussion both useful and relevant. :)

- Stealth Dave

[edpuffmonster]

If I recall correctly, Jdiner's vsplit is based on Warren Toomey's.

Technically he's violating the GPL by releasing the binaries with no source available. He has repetedly said, however, that he will eventually include the source, so I guess it's not so bad. (his reasoning for not releasing the source yet is arguable, but sound)

[jdiner]

Originally posted by edpuffmonster
If I recall correctly, Jdiner's vsplit is based on Warren Toomey's.

Technically he's violating the GPL by releasing the binaries with no source available. He has repetedly said, however, that he will eventually include the source, so I guess it's not so bad. (his reasoning for not releasing the source yet is arguable, but sound)

I thought I had addressed this before. But was in another thread. So one more time. My work is all original. (I have no idea who Warren Toomey is but I will look at the link in the above message about his stuff...) That is why it actually works. No slam to previous authors. But there are reasons that nothing else has been as successful. I will release source, but since I wrote it I get to set the time frame.

--jdiner

[KRavEN]

and for the record again. Stop friggin whining about source!!!

It is his code to do with as he pleases. We are all lucky that he is contibuting his time and effort to make the tools AND also making them available for us to use. It is his CHOICE to do this and he could have just as easily made tools for himself and never bothered with the rest of us.

So once again let it drop, move on, find something else to talk about.

If you want source so bad, write your own.

[digitalAir]

Originally posted by KRavEN
and for the record again. Stop friggin whining about source!!!

It is his code to do with as he pleases. We are all lucky that he is contibuting his time and effort to make the tools AND also making them available for us to use. It is his CHOICE to do this and he could have just as easily made tools for himself and never bothered with the rest of us.

So once again let it drop, move on, find something else to talk about.

If you want source so bad, write your own.

To heck with the source... I WANT A PONY!!! :)

[edpuffmonster]

Originally posted by KRavEN
and for the record again. Stop friggin whining about source!!!
It is his code to do with as he pleases.
Narf. If it were based on GPL'd code, as I thought I had read, people would have every right to whine about it :P

Stop friggin whining about my whining! :)

Besides, it's fun to pretend I'd actually do something with the source if I had it.

[jdiner]

The only portion of anything that I have written that is based on GPL code. Is the mfs_stream program. It's source has been released.

I suppose in that light the server portion of the TyTool also contains the same code. However it is just the mfs_stream code run in a secondary thread. i.e. no changes to the source. But I suppose I should release that server source.

I dunno if anyone is interested in that part or not. Litterally it is a small parser with some socket code...

--jdiner

[jdiner]

Oh man. Well that explains why I could not fix it. The TyStream file itself on one of the recent bad clips I was sent is actually wrong. This is a first for me. Never seen one where it is simply wrong before.

Having decided on that. I have removed it from my test pool. There is nothing at present that I can do about a bad tyStream file. When it is corrupt it is over...

Having said that it means I can remove the latest debug code and re-run the latest testbed and if all is well the next version of vsplit will be ready to roll...

--jdiner

[RxMan]

What features will the new version contain? How will a person know if the tystream is bad or the cause of problems.

THANKS

[rpongett]

jdiner:

Excellent. Can't wait.

Do you think the tystream as it exists on the TIVO is bad, or is it a rare problem with etraction? (i.e., could it be "fixed" by re-extracting the same program).

[jdiner]

Originally posted by RxMan
What features will the new version contain? How will a person know if the tystream is bad or the cause of problems.

From the types of errors. If things get off and stay off all the way through that is a TyStream file error. If it is a -1x4 error report then those should now be fixed.

I can't tell if it is a bad extraction that is the problem with this one clip that I have or if it is something else. But the file itself is smashed. No way to recover and keep going. The "key" to seeing this is in getting a "fix it dear henry" message. This message is displayed if a header shows up for decoding with a totally wrong header type. I.e. we want time and position information on something that is neither audio nor video data. As evidenced by the current "bad" clip.

Think of it this way. Inside the TyStream file things line up like this:

Rec 1 ----> VHeader ----> 16 bytes
Rec 2 ----> VBody ----> 7000 bytes
Rec 3 ----> VHeader ----> 16 bytes
Rec 4 ----> VBody ----> 7000 bytes
Rec 5 ----> AHeader ----> 16 bytes
Rec 6 ----> ABody ----> 480 bytes

Now these sizes made up to some extent. I have seen video records as large as almost 128k bytes.

Things that are "always" the case. A header record points to header data, and a body record points to body data.

Now in going through this new "really bad" clip this is the case to a certain point in the file. (In everything else I have this is always the case...) Then suddenly this is not the case. Where the header records point is suddenly and inexplicably body data. Something like this:

Rec 1 ----> VHeader ----> 16 bytes
Rec 2 ----> VBody ----> 7000 bytes
Rec 3 ----> VHeader ----> 16 bytes
Rec 4 ----> VBody ----> 7000 bytes + 16 bytes of the next header + 42 bytes of the next body.
Rec 5 ----> AHeader ----> 16 bytes of the next body.
Rec 6 ----> ABody ----> 398 bytes of this body. The 16 for the next header, and then some of the body after it.

I have checked and rechecked my code and manually gone through the problem chunk by hand. And it is not a calculation errors the TyStream information itself is simply wrong.

Anyway it continues like this. Once off it stays off for several chunks. Things eventually appear to line up. But there is no way to maintain sync in the current form. We have a hole and there is nothing reasonable that can be automated to fix/find these holes. I would have to "restart" the mechanism and build a new set of files once things are back on track in the TyStream file. This is possible just very time consuming.

I am not going to do that at this point. For several reasons:

1- This is the one and only case that I have seen myself where this happened. I can't explain it but I do know from history that it is rare.

2- Why bog down in a fixes for such a rare case when we can move on with the mainstream things.

3- While I do have time now it is limited and as a result I have to prioritize.

I hope this clears a few things up. I do wonder what happened to the Tivo it was occuring on to make it happen...

--jdiner

[jdiner]

Ok. A simple poll. There is a choice facing me right now and it affects most of the rest of you so I want some feedback.

I am working on the mux'ing as most of you know. Plain and simple right now I am having problems. I think I have it hammered it out and then something else breaks. So this could take awhile.

Now having said that I have in playing just fine as an mpeg right now. But the editing of them is still smashed.

Now I can either spend my time trying to get "busted in the same way" as something else and in so doing get back on track for editing via another program. OR I can call it good and start moving on with the rest of this stuff including the editor I have been creating for my own stuff.

My editor is not a full blown editor. Never will be. What it is is a GOP level editor for the cutting of commercials. When I did it manually some time ago, read months and months ago now, I got it working. A few visual sniggles took place but I have a technique to try to get rid of that. Since it is not a frame level editor it is not nor will it ever really be "perfect". It is possible there will be a few frames on either side that are unwanted.. But it will cut commercials and maintain sync.

This is "good enough" for me. And it is much much easier to do. Which means that I should be able to get it out the door much sooner than tweaking the dark to get things to a point where the editors out there will like it.

Ok. So the poll.

1- Keep going with tweaking the mpeg file output until m2-edit and mpeg2vcr will frame/gop edit the output mpeg successfully.

or

2- Keep making the mpeg file play in the most places i.e. software and hardware mpeg-2/dvd players. With our own custom cutting/editing software...

I was going to ask that people PM me with their suggestiong but this does affect what so many are waiting/hoping for. So let's do it here in this thread. Anyone with an option or a dream chime in. But please keep it civil...

--jdiner

[bronco13]

jdiner,

I'm not sure I fully understand either option, but if it's of any use to you I'd rather have the mux'ing. I have like 50 movies piled up on my Hughes and can't get anything to sync properly not matter how I mux!

[rpongett]

Even though I don't have an MPEG-2 editor, I would say #2.

Getting output of a sync'd and widely playable MPEG-2 file that can be edited by other programs if desired would appear to be a higher priority. Especially if you already have a rough-cut GOP level commercial editor to throw out for optional use.

That's my opinion, anyway.

Thanks for all of the effort on this.

[jdiner]

Ok. Perhaps I was not clear enough.

What can happen here is this.

Option #1 - Keep working on the generation of the mpeg-2 file. In the hopes that at some point it will work to edit it with one of the commercial editors. Thus at some point something could be done with what is coming out of those to burn to disk. What ever type of disk people are using.

Option #2 - Do it myself and in so doing forget about ever being compatible with one of the other editors. Thus the ONLY way to ever edit it at that point will be with the internal one I am building. The benefit here is that things are and stay sync'ed even after the cuts have been made.

--jdiner

[artships]

1- Keep going with tweaking the mpeg file output until m2-edit and mpeg2vcr will frame/gop edit the output mpeg successfully.


To me, this sounds like you want to create a program stream that any DVD player can play and any DVD authoring program can edit without losing sync, so I vote for this.


2- Keep making the mpeg file play in the most places i.e. software and hardware mpeg-2/dvd players. With our own custom cutting/editing software...


This sounds like a nearly-standard program stream that some DVD players or authoring programs can't handle, so I don't vote for this, despite the nifty advert-zapping editor.

John,
Who realizes, of course, that the main criteria for what you're to work on next should be what sounds like the most fun to you.

[rpongett]

jdiner:

My mistake. Misread your initial post.

I would still stick with #2, as long as the simple GOP level editor permitted users to easily scan through (like, with a dragable bar) the video to cut where they wanted (i.e., not just sensing all black I-frames and alowing cuts there, but whatever else they wanted to cut, as well).

If its something like, say, the simple chapter marking mechanism in SpruceUp (just a small video screen, dragable bar for position in the video and a chapter mark button) where you could essentially cut "chapters" (segments), that would probably be more than acceptable to most. The downside of being off a few extra frames on either side on cuts with GOP level editing is pretty low.

Then, perhaps at some point later on, you might try to go back and see if you can get tytool to massage the data such that commercial editors could work on it (option #1).

[rpongett]

artships:

I think that option #2 allows most DVD players to play the sync'd file.

I may be reading this wrong, but I think that jdiner right now is able to produce on his system a mux'd sync'd MPEG-2 that is playable basically everywhere. However, commercial editors like M2-Edit can't properly edit that outuput.

I think he is contemplating whether to spend his time:

1. Keep working to try to (if ever) get the output such that professional frame-level editing programs such as M2-Edit can properly edit that output; or

2. Building into tytool6 (or a separate program) a simple GOP level (not frame level) editor that can edit these files properly (and they can be played anywhwere).

If I'm reading that correctly, my vote is for #2 first. Then try to perfect #1 later.

[stealthdave]

jdiner,

It sounds like you pretty much have option 2 ready to roll, or nearly so. As I stated before, I'm running a Linux operation, so even a simple editor probably won't work even through WINE, so that part of option 2 probably won't affect me unless your editor is cross-compilable. ;) (I've found that WINE can't run most programs that use windows codecs, or I at least have trouble running them.) Not to fear, though. GOPchop (http://outflux.net/unix/software/GOPchop/) seems to work pretty well on vsplit13 produced mpegs, so I'm set there. (Though I'd love to try your editor. GOPchop can be pretty buggy at times.) Over time I'd definitely like to see option 1 pursued, but I tend to re-encode everything to SVCD anyways, so I vote for option 2.

btw, GOPchop is GPL, so his source is available for download. :)

[stealthdave]

Originally posted by jdiner
Having said that it means I can remove the latest debug code and re-run the latest testbed and if all is well the next version of vsplit will be ready to roll...

--jdiner

Does this mean that we might see a new version of vsplit soon? Woohoo! I'm really looking forward to trying it out! I've got some Battlestar Gallactica streams that are just dying to be decoded!

- Stealth Dave

[laserfan]

I strongly vote for #2, provided it works for SA Tivos as well.

If as you said it includes audio sync, and even the most rudimentary "trim-front-and-back" editing that would be wonderful. An "embedded commercial exciser" would be positively fantastic, a step beyond what I might have hoped for in TyTool.

I guess I don't care if "commercial editing tools" recognize the files as "perfectly MPEG2-compliant" or not; I only care, as an SA Tivo owner, that I'm able somehow to transcode the audio to 48kHz before burning to disk. Since this is CURRENTLY easily do-able w/muxed TyTool streams I'm gonna assume it will be possible w/the next gen system you're working on.

Thanks for all your hard work jdiner.

[gtschance]

Output recordable into DVD/SVCD/VCD that stays in sync would be my vote.

Quick question

Is the ty file exported from a DTivo likely to change in any of your scenarios?

I ask to determine if it is safe to archive them off my DTivo and get back around to splitting and DVD'ing them later.

Great work by the way

[KRavEN]

Originally posted by rpongett
jdiner:

My mistake. Misread your initial post.

I would still stick with #2, as long as the simple GOP level editor permitted users to easily scan through (like, with a dragable bar) the video to cut where they wanted (i.e., not just sensing all black I-frames and alowing cuts there, but whatever else they wanted to cut, as well).

If its something like, say, the simple chapter marking mechanism in SpruceUp (just a small video screen, dragable bar for position in the video and a chapter mark button) where you could essentially cut "chapters" (segments), that would probably be more than acceptable to most. The downside of being off a few extra frames on either side on cuts with GOP level editing is pretty low.

Then, perhaps at some point later on, you might try to go back and see if you can get tytool to massage the data such that commercial editors could work on it (option #1).


I agree with this completely. This is what I vote for.

[FreydNot]

Since I have never been able to convert 32k audio to 48k without messing up the sync, I have never been able to burn an in sync DVD from a simple pair of vsplit files.

Since my goal is to burn DVD's from SA files, I will vote for option #1 which I believe will continue to work towards this goal.

Option #2 is a function that we can pretty much do now via a labor intisive manual task. I just did this for several ty files, but it all ended up being for not since I still couldn't get the audio to a DVD compliant frequency. What I did by hand is make up batch files to use vsplit to extract 11 chunks on 100 chunk intervuls. From there I found the chunks that blocked out the commercials and then did it again with 10 chunk resolution between there edit points. Once I got it down to the nearest 10 chunks, I'd do it again with a resolution of 1. Then I'd use vsplit to cut the ty file in program segements down to the nearest chunk.

The proposed #2 option should have better resolution (GOP boundries instead of chunk boundries), but chunks where pretty reasonable for what I was doing.

I should also say that I don't really need a full blown #1 solution. If besweet or some other tool can convert the SA audio from 32k to 48k without loosing sync, that is all I need.

If somone can point me towards a way to get SA streams onto a DVD, then I'll switch my vote to #2.

[Hoochster]

I personally go for what is easiest for the person programming. As I have mentioned in the past this is all beyond me and I am grateful for anything that anyone shares with the community. So if you already have something that basically fits the task that everyone wants and it works for you and is easier for you and you are willing to share it then by all means that is what works for me! Since I personally can't do no better then I am happy to get what someone was willing to share!

So thanks for your time and efforts on this project and looking forward to whatever you give us!

Hoochster

[edpuffmonster]

Sorry for posting this here, but jdiner, your mailbox has been full for quite some time now. :)

[Rotten]

2.

Lets get it on. By The way I would contribute financially to the cause if needed

[FreydNot]

Originally posted by rc3105
Freydnot: virtual dub, select audio full processing mode, go to conversion (cntrl-a) and set the options


Here is what I tried...

I muxed a m2v and m2a pair using TMPGEnc V2.57.41.146 full mux (not simple) to a mpg file. I tried to load that file into VirtualDub 1.4.10 but I get the error "MPEG Import Filter: pack synchronization"

I tried using DVD2AVI and VFAPIConv to get the file into VirtualDub, and while that did load the file up, it was going down a transcoding road a didn't want to. The only way I could see this working would be to transcode the video+audio into an AVI and then re-encode the avi into mpeg. Not really a solution.

Did I miss something?

I did try to burn a muxed mpg file into a ISO data DVD disk as you described. My Pioneer player didn't like it at all (not much of a suprise).

[koreth]

The only thing I really wanted to use M2-Edit for was to chop out commercials. If I can do that some other way without major headaches, fine by me. Actually it won't surprise me if you end up with a better UI than M2-Edit -- it was clearly not designed with 60-minute-long clips in mind!

Obviously it'll be nice to have the muxed files in good editable condition at some point, but that seems like a project that could stretch on indefinitely, so best to get something useful out the door earlier rather than later.

I watch my DVDs on a PC so audio sample rates and so forth aren't an issue. (Pretty sure my standalone DVD player will handle different audio rates too.)

Looking forward to burning some DVDs!

[Fugg]

Originally posted by rc3105
...
if you want to convert the audio seperately, try using the winamp disk writer plug in to convert the mpa to a wav, then any sound converter util to change the sampling rate and or re-compress it
--
Riley

the audio editor in mpeg2vcr will convert the .m2a file to 44.1 or 48khz directly, no need to change formats.

[Fugg]

Originally posted by rc3105
... my apex plays mpeg files written directly to disk as though they were individual tracks on an audio cd. ...
--
Riley

your apex 1100 will play the files from a cd-r or a dvd-r?

mine will play them as individual files from a cd but not from a dvd. i thought it was because it sensed the media type and read the disk appropriately. (so it thinks)

ie. Looks for individual files on cd and looks for a standard dvd format on dvd’s.

have you changed the stock firmware?

[Fugg]

...and as for which option, I have to agree with Hoochster,

Bless your heart jdiner, whatever solution would be easier for you. I would think that #2 would be the easiest, but I don't know how far along you are with your editor.

I keep pretty good sync with good extractions on my dtivo, but anytime I try to pull streams from my sa that are over 2 fsids, I loose approximately a half a second of audio at every transition.

Any solution that would keep that from happening without extracting each fsid individually would be most excellent.

[jdiner]

Originally posted by stealthdave

btw, GOPchop is GPL, so his source is available for download. :)

Don't need it. I don't intend to sound smug, and I can explain in detail if anyone wants, but it is a VERY simple thing to do. Basically, you get Frames (I, B, P, and a 4th that is an add-on that neither type of tivo uses...) These frames are then group'ed in a GOP. With a little bit of tweaking you just get rid of everything between GOP headers you cut and shazam. You are there.

--jdiner

[jdiner]

Originally posted by gtschance
Output recordable into DVD/SVCD/VCD that stays in sync would be my vote.

Quick question

Is the ty file exported from a DTivo likely to change in any of your scenarios?

I ask to determine if it is safe to archive them off my DTivo and get back around to splitting and DVD'ing them later.


No. I won't be changing things. Tivo might... But I won't be. That is what I have done. I have archived a ton of things in defference to my being able to finish this so that I can make nice archives...

--jdiner

[FreydNot]

Originally posted by rc3105
if you want to convert the audio separately, try using the winamp disk writer plug in to convert the mpa to a wav, then any sound converter until to change the sampling rate and or re-compress it

I haven't tried using winamp 2.0 for this (winamp 3.0 doesn't seem to have the same diskwriter output option), but I have used goldwave to save the mpa to wav and then use besweet to convert the wave back to 48k mp2. I have also tried using BeSweet to directly convert the 32k mpa to 48k mp2, and 48k ac3.

All of the things I have tired result in an audio file that is out of sync from the original audio file. Is there something magical about winamp's diskwriter plugin?


Originally posted by Fugg
the audio editor in mpeg2vcr will convert the .m2a file to 44.1 or 48khz directly, no need to change formats.

I have used Womble's Mpeg-VCR V3.12 (6/2002) to convert the 32k audio to 48k audio. While the process worked fine and the audio quality was excellent, the resultant audio file was out of sync from the original audio (and the video of course).

Maybe I should start a new thread for this...

[dlang]

a simple editor is all I expect to need (trimming commercials out of a video or dropping everything except for a really good commerical out of a video :-)

however long term I need to save these things to removeable media (my 100G addon drive in my tivo is full and my 160G video storage drive in my fileserver is already up to 80% full..) it would be really nice if I was able to burn DVDs that can be played in normal players.

the only thing is that I will need to be able to run this code on one of my linux systems as I don't have anything suitable running windows.

[scarabus]

Originally posted by jdiner
Oh man. Well that explains why I could not fix it. The TyStream file itself on one of the recent bad clips I was sent is actually wrong. This is a first for me. Never seen one where it is simply wrong before.

Bad blocks on the hard drive perhaps?

So what will VSplit say if it encounters a bad stream like this?
Will there be an error message that says "this stream is toast - bin it and try to record it again"?

[und420_247]

I recorded the new DirecTV Freeview (Bruce Hornsby Live) and attempted to extract it today.

TYTool does NOT show a filesize for the show (it works for all other items in my Now Playing).

It started extracting, and got as far as 3 GB when I stopped it by killing TYTool.

What I got was a 3GB .mpv (which is 27 minutes of video) and a 0 byte audio file.

Is this program strange in some way - I will attempt to record it again and see if this happens again. Can someone else try this?

und420_247

[scarabus]

Personally I'd prefer Option #1 - I'd really like to be able to import streams into a frame accurate editor and do perfect commercial cuts.

But it seems to me that we're more likely to get #2 sooner and I'll be happy to get "good enough" soon and wait for perfect later on.

[Wooly]

Me vote #2. Mongo say #2. Mongo LIKE #2, as Mongo hates most editors, and Mongo just wants his MacGyver on DVD ASAFP.


Option #2 - Do it myself and in so doing forget about ever being compatible with one of the other editors. Thus the ONLY way to ever edit it at that point will be with the internal one I am building. The benefit here is that things are and stay sync'ed even after the cuts have been made.

--jdiner [/B]

[jdiner]

Originally posted by scarabus


Bad blocks on the hard drive perhaps?

So what will VSplit say if it encounters a bad stream like this?
Will there be an error message that says "this stream is toast - bin it and try to record it again"?

I had not thought about the bad block type of thing... I had been thinking software issues. But you never know. I mean a truely loaded down and "freaking out" tivo might be the reason to...

Way back when I started using all of this stuff I have been writing I found that without the priority patch when I went to get the list of playing shows and off and on during the extraction it would smash the display on the screen. I thought it might just be playback but upon testing it back then it was indeed bad on disk. Backing up and playing it again showed the same bad corrupted data.

That type of thing might explain the issue as well. Just "gets behind". But like I said. I have never seen this before. But if it was bad on the screen I have never tried to process it... So I am just guessing here.

Lastly, I will adding a message that is better than "fix it dear henry"... Bottom line. When I have been programming for days and days of 14 hour sessions I get a little... punchy. And for some reason when I was working over that part I found the childrens song "There's a hole in the bucket" running through my head... And you see the result... :)

--jdiner

[FreydNot]

with what shal we fix it? ...

[jdiner]

Ok. No need to keep voting. It appears to be pretty solidly on the side of "doing it oursleves" for now.

So my plans are to drop the idea of munging the output further to make it look better and work better going with a commercial cutter/editor.

Instead I will focus on just getting something that will work for most of us as a simple commercial cutter. There are a ton of techniques for this. And to be honest I intend to enlist the aid of some others that are good/better at linux/unix type GUI's and we can get together on producing something that can be used on a variety of platforms. (But keep in mind that editing is just a step in this process and not currently the next one... Although it should be soon... :)

Perhaps in doing all of the above work I will figure/find out enough to get better at producing something that can be used to fix the problems with the output so that something else can be used. You never know...

--jdiner

[jdiner]

Originally posted by FreydNot
with what shal we fix it? ...

Fix what? The one where the framing in the TyStream file is smashed? In terms of what is being done right now. There is no way to fix it. It would take a different approach. And for 1 stream in 20,000... A different approach is not worth it. At least not to me.

--jdiner

[FreydNot]

Sorry, that's the next line in the "hole in the bucket" song. Didn't mean to confuse you with it.

[Ellipse]

Oh good...I just thought it was my odd mind thinking of "Hole in the Bucket" when I saw that comment. :)

[laserfan]

Originally posted by jdiner
Ok. No need to keep voting. ...keep in mind that editing is just a step in this process and not currently the next one... Although it should be soon....I want to vote once more though--and in favor of the next step being a muxed/in-sync mpg2 program stream. You have said that you can do this and I hope you will release it to us before getting wrapped around the edit step axle?:)

[jdiner]

Originally posted by FreydNot
Sorry, that's the next line in the "hole in the bucket" song. Didn't mean to confuse you with it.

You are most correct. Sorry. See what happens when there is too much sleep going on? The song stops and things are just not right.

--jdiner

[laserfan]

Originally posted by rc3105
getting a nice neat PROGRAM stream will require transcoding. (partial at least)...the mpeg coming out of the tivo is a TRANSPORT stream...find a good stream editor or a dvd authoring prog that accepts transport streams as input, problem solvedI can state with conviction that after some googling I still do not understand the difference :confused: . But I thought what jdiner was after was a way to store Tivo programs to DVD, which implies (I believe) a more-or-less "compliant" MPEG2 Program Stream. If instead it means he makes a more-or-less compliant Transport stream that some authoring program will accept & transcode & burn, well, whatever the shortest path is I will accept, even if it means from jdiner "you have to acquire thus-and-such Authoring program (that I use myself) for this to work".

Do I still not understand the difference between options 1 & 2... Ok, then I'll just take what I can get! :D

[jdiner]

Originally posted by rc3105


getting a nice neat PROGRAM stream will require transcoding. (partial at least)

whether you use dvd2avi or jdiner's app does it, that's the bottom line here

the mpeg coming out of the tivo is a TRANSPORT stream

they're pretty similiar, but it's still apple's and oranges

find a good stream editor or a dvd authoring prog that accepts transport streams as input, problem solved


You are correct. But do keep in mind that it is only "almost" a transport stream. There are mpeg-2 records that are missing that keep it from being a true transport stream. There are also some changes, depending on Tivo type, that make some of what is there incorrect. But it would be easier to go to a transport stream than to a Program stream.

However, you have to define your use of transcoding here. Nothing of what I am doing reencodes even a single frame of the stream. I am just changing how it is "blocked and mux'ed" to make it a program stream. The end result is a playable mpeg-2 program stream file.

--jdiner

[kyle]

If this is a TRANSPORT stream, the MUXing in MPEG2VCR allows for this--but wants me to speicify a PID. What is this? What values should I use? The defaults are

Video Elementary Stream 34
Audio Elementary Stream 33

Thanks,
-Kyle

[jdiner]

Yeah. Mostly playable would be pretty accurate. That is the last part that I am working on now. I will get it. Sadly I have been almost there for a very long time. "whee"... :)

--jdiner

[laserfan]

Originally posted by kyle
...If this is a TRANSPORT stream, the MUXing in MPEG2VCR allows for this--but wants me to speicify a PID...kyle it is my understanding that the MPEG-VCR Transport Stream option is for the Creation of transport streams; the discussion was about "what kind of streams come out of Tivo" and I think the answer is "it's a kind of transport stream w/a few new wrinkles thrown-in". So the challenge before jdiner & others working this is to translate these streams into something that is useable for editing and burnable to a DVD (both of which require that certain rules are met). So I don't think using this option in MPEG-VCR buys you anything, but someone smarter than me here will correct me if I'm wrong!

My simplistic take-away from the "program stream vs. transport stream" discussion is that Transport streams are created such that they can be reliably "broadcast" (whether that means thru-the-air or across a network) given the interruptions that occur in the broadcasting process (whether that be a thunderstorm or an IP traffic problem). So I'm thinking of Transport streams in terms of pieces of information about the video/audio that can suffer interruptions yet be pieced-back together at the final destination (playback) point. Program streams OTOH don't have to fret about the possibility of major interruptions and thus aren't "packetized" like Transport streams. There's probably an overhead/bandwidth associated w/Transport streams that PS don't have to be saddled with/bound by. My story and I'm sticking to it until somebody explains it better/more accurately.

[jdiner]

Ahh... And the latest testing failures start to make sense finally. I was keying off of some values that used to be constant and now seem to be well... not constant. That makes what I was trying to do harder.

Not sure yet what would cause this. I have not upgraded anything on my tivo from one extraction to the next... Has there been a "stealth" upgrade done by anyone or anything on the DTivo front?

Trying to use another method to detect a good or bad status for the -1x4 failures. (When there is not PES information for either audio or video in an entire chunk...)

The good thing is that in doing some more exhaustive testing it is just "another set". So not random as I had first thought but rather just changed from what it has been for years. (Go figure...)

--jdiner

[jdiner]

OK. Well. I don't know if something changed or what it truly was. But I can see that there is something going. Good thing I have a large test bed. I ran through a ton of clips checking this out and found a few interesting things. There now appear to be 2 ranges of values used to indicate what it is, video header or body and audio header or body, and what is going on with it...

The old set worked in this range:


VIDEO HEADER RESETS AT > 00 CB 10 --> 0 BB 20 == diff of FF0 (4080)

VIDEO BODY RESETS AT > 31 8e 6c --> 20 50 40


This was the norm for every clip I had older than about 5 months. But then I started grabbing clips from "The Dead Zone" on usa during the marathon. So no more than 2 weeks ago and for almost 2 full weeks 1 episode a day. On the 8 I got thanks to tennis, does anyone really watch that anyway? :), I got 3 with this range scheme and 5 with the new one...


VIDEO HEADER RESETS AT > 01 4f c4 --> 01 3f cc == diff of FF8 (4088)

VIDEO BODY RESETS AT > 6d 7d 9c --> 5c 30 40 == diff of 11 4d 5c (1133916)



So 2 totally different ranges. I don't get it... But these are the only 2 ranges I have seen so far. The rollback point changes by a few bytes in either direction for both header and body but it stays in roughly the same range. So... if that remains the case we can still use this technique the fixing of the -1x4 problems...

--jdiner

[newlooper]

OK I have the thought planted in my head, let's see if I can get it out.

code:
--------------------------------------------------------------------------------
VIDEO HEADER RESETS AT > 00 CB 10 --> 0 BB 20 == diff of FF0 (4080)

VIDEO BODY RESETS AT > 31 8e 6c --> 20 50 40
--------------------------------------------------------------------------------

If it the video header resets at FF0, could you use FFx to detect a video header reset? That would kind of cover all bases.

The video body would have to be > 11 3E xx though and I am not sure that would meet the criteria.

It seems that there still remains a constant, just a little different than normal.

[daveinfla]

Running a Phillips DTIVO w/
Extreme 2.5 Upgrade (TivoWeb1.93, noscramble, all the defaults)
TurboNet Card

Added mfswebmodule97 last night and was able to click on and download a single .ty stream. The file is a one hour show at around 1G in size.

I ran Vsplit13.exe filename.ty filename.m2v filename.m2a and after about 30 seconds, and a bunch of messages, vsplit stops and generates a Windows XP application error (you know the one that wants to report to Mickeysoft).

I tryed using Tytool5r2, opening an existing file and got the same results.

Any ideas?

[Wes]

Did you try to use Tytool5r2 to extract as m2v & m2a directly instead?

[daveinfla]

Wes,

I tried using Tytools after I had already extracted the .ty file with Tivoweb.

Are you suggesting I NOT use tivoweb to download the file and install TYtools on my Tivo and use the client to extract and split at the same time?

[jdiner]

Originally posted by newlooper
OK I have the thought planted in my head, let's see if I can get it out.

code:
--------------------------------------------------------------------------------
VIDEO HEADER RESETS AT > 00 CB 10 --> 0 BB 20 == diff of FF0 (4080)

VIDEO BODY RESETS AT > 31 8e 6c --> 20 50 40
--------------------------------------------------------------------------------

If it the video header resets at FF0, could you use FFx to detect a video header reset? That would kind of cover all bases.

The video body would have to be > 11 3E xx though and I am not sure that would meet the criteria.

It seems that there still remains a constant, just a little different than normal.


Not quite. For the header counting it resets at 00 CB 06 through about 00 CB 1F. It fluctuates a bit.

The body counting fluctuates much more wildly. But I have these values through observations. What I don't like is that after having things all be in the same place I have now identified 3 seperate bands of values. This just makes it problematic to work on as these values if they are indeed changing are how I can tell when things line up and no "better" data is available.

--jdiner

[jdiner]

Originally posted by daveinfla

I ran Vsplit13.exe filename.ty filename.m2v filename.m2a and after about 30 seconds, and a bunch of messages, vsplit stops and generates a Windows XP application error (you know the one that wants to report to Mickeysoft).

I tryed using Tytool5r2, opening an existing file and got the same results.

Any ideas?

Find out where it is crashing in terms of the number of chunks. Use TyFileSplit to start roughly 40 chunks before that point and then cut a 100 chunk portion it and send it to me. I can at least stop the crashing, and maybe figure out what is wrong and help it continue to process correctly.

--jdiner

[wh1212]

OK I've alway's extracted from my SA,720x480 cbr etc, looks great to me, Have a dsr6000,give it a try.
Completly differant streams,Disney DVD like, Interlaced 48%-67% the rest progressive,sometimes top field first,or not. has the audio always been 4800 ?Mpg2 muxers &editors make a mess(Tmpegenc beta 12a best mux) So send streams to dvd2avi(gordian knot)Inverse Telicide, Perfect Mpeg4 23.976 and insync.
Need more keyframes for editing though,Anyone know of a frame acurate Avi editor,Also second extraction with Tytool5,got a .gl file,not text ? anyone know what it is, or want it.
Happy with streams, seem more crisp and vivid, Have the streams always been like this, or is it a merger thing.

[Wes]

daveinfla, yea, that's what i mean. tivoweb MAY be the reason of your problem. I'm just guessing though.

[tdavis80]

I get the app error running tytool 5 if I try to 'get' Coldplay / vines Special. I even recorded it twice. Both versions (1 erase #1 and re-recorded it the next day) cause the app to fail & close.

OS here is XP SP1.



Getting NowShowing data...
Total Size = 5258kbytes in 7.581000 seconds...
Unknown tag <166>
tag data <p 14 22:58:31 tcl[362]: Tcl created pool of 1458176 bytes>

Name = 'c:\tivo\Coldplay / Vines Special - FREEVIEW-'
fsIDs = '1179967/1179969/1179970/1179971/1179972'
Tivo Address = '192.168.1.222'
Connected...
cmd => 'TYSTREAM 1179967/1179969/1179970/1179971/1179972'
Detected Tivo Type: DTivo
Detected Audio Stream Type: MPEG Layer II
Final standardAudioSize = 588
Final standardFrameLength = 576
Final standardAudioDiff = 2160 or 00:00:00.024
First Video PTS: 00:02:35.057

[Wes]

It could be because the show's name has a "/" in it. I had to rename the "9/11" shows to "9-11" to download mine.

[jdiner]

No the / character shouldn't be a problem. Lemme grab the full list from the source:


while ((ptr = strstr(fname2, "/"))) ptr[0] = '-'; // Get rid of any /s...
while ((ptr = strstr(fname2, "\\"))) ptr[0] = '-'; // Get rid of any \s...
while ((ptr = strstr(fname2, ":"))) ptr[0] = '-'; // Get rid of any colons...
while ((ptr = strstr(fname2, "*"))) ptr[0] = '-'; // Get rid of any *s...
while ((ptr = strstr(fname2, "?"))) ptr[0] = '-'; // Get rid of any Quesion marks...
while ((ptr = strstr(fname2, "\""))) ptr[0] = '-'; // Get rid of any "s...
while ((ptr = strstr(fname2, "<"))) ptr[0] = '-'; // Get rid of any <s...
while ((ptr = strstr(fname2, ">"))) ptr[0] = '-'; // Get rid of any >s...


That is the list of what I cut out to protect the names of files...

Perhaps it should be a "\/" string to "protect" that character but I don't think that it is special in C/C++.

--jdiner

[jdiner]

Alright. It looks like VSplit Alpha #14. Is pretty much ready to roll...

It took a lot longer than I thought it would but the chunk alignment matching engine is now extremely solid. And since that is what will fix the temporal aligment issues as well as the missing data do to -1x4 chunks it was a needed thing.

There are a number of sundry other fixes in it that will make it much more stable for some. I still don't see (ever) many of the problems being sent to me from others but hey... them's the breaks.

Right now I have it running over my test bed one last time. There are over 40 1 hour shows in the test bed now so it takes awhile. One of these days...

Anyway, just wanted to let me people know what is going on. I actually want to test and fix one more thing but it is pretty minor so it should not need re-running the test bed. Recently people have been sending me bad clips that contain reseting audio PTS values. Which is wierd. Once again mine never does it, so... Anyway the new alignment code will make it possible to catch these and correctly determine what is going on.

--jdiner

[jdiner]

Oh man. Corner case after corner case. I am getting really tired of trying to fix this one. When the PTS timestamps reset... I don't have enough test clips so I have to make due with kind of guessing and what types of things can happen. No idea if they ever really do.

Anyone out there have a TyStreams where the PTS values reset in the middle? I do believe it will be a stand-alone issue only.

--jdiner

[FreydNot]

How can we tell? I assume vsplit (v13) will give some error message?

I know when 3.0 came along, the green bar went away. Could your changes have happened at the same time?

[jdiner]

Ah that is a very good point. :)

You can tell because all of a sudden the audio and video PTS data will be off. You will start to see OOB messages from VSplit that indicate that every single chunk past a certain point is off.

I have found 2 more bad ones from someone that put it on my staging FTP site. So I have a bit better notion of what can happen now. But do keep in mind that it appears it can happen in either a SATivo or a DTivo stream.

--jdiner

[jknapp]

Originally posted by jdiner
No the / character shouldn't be a problem. ...--jdiner
Jdiner--I'm having exactly the same problem as tdavis80, but with a different show that has the "/" character. The show I'm trying to extract is yesterday's Smallville, which is titled "Pilot/Metamorphosis". I have downloaded lots of other shows, including one after this problem with no trouble.
You can get around the crash by setting Tytool in Tystream mode instead of VSplit, but then the client just gives an error that it can't open the output file. (Sorry at work, so don't have the exact language in front of me.)
I'm using 5r2 with WinXP SP1 also. I see your code above, but from my admitedly uneducated view it looks like it either isn't working or the "/" is tripping it up somewhere else in the process.

[jdiner]

I will protect it in the next release. It is easy enough to do for a cost of 1 more byte in size...

--jdiner

[jdiner]

Ah ha. I begin to see why people have not thought that mux'ing and proper MPEG-2 Program Stream output was the holy grail of all of this.

We have been talking 2 totally different things. And now I am not sure which is better. Or if there is even a better.

I have been planning on making something like the DVD-VCD... In essence a VCD formatted disk that is just 4.7 gig in size and not 640-700meg in size. www.vcdhelp.com has a good explaination of this.

This is why I have always planned and worked towards good output. Then take mux'ing 1 step further to a program stream that includes RIFF headers, a .dat file, and you can just burn the pig.

However in looking over some of the FAQs on TyDVD's and what not it appears that people are not doing this. But are making DVDs that are tricked out to look like real DVDs...

So 1 question... Why? Why go through the headaches of changing a few header values if the other style works? Sync would be maintained etc...

What are the Pro's and Con's that people see that made them choose 1 over the other?

--jdiner

[laserfan]

Originally posted by jdiner
...Why go through the headaches of changing a few header values if the other style works? Sync would be maintained etc...Well, here's one answer, I DIDN'T KNOW THERE WAS ANY OTHER WAY! i.e. if there is a way to create a "VCD" on a DVD platter that will play in a set-top player then I don't know about it. But I will look at vcdhelp when it's working again.

I think there are some players that will play a DVD that's just had an mpeg2 file copied onto it, but I don't think my player will read such a thing. In fact my player thinks a VCD is an 1150kbps bitrate and 44.1kHz audio etc. So I take the MPEG2 file and prettify it for DVD by running it thru an authoring program.

I must confess to some concern about why such a disconnect exists between what you are doing and what people are expecting. Maybe somewhere in your 500-odd posts there is one that explains exactly what You have been doing to archive your Tivo recordings for both posterity and easy playback?

[SirTrini]

My hope was that jdiners work was going towards VOB files that are DVD compliant.