PDA

View Full Version : MUX'ing, VSplit, and MPG2 files.



Pages : [1] 2 3 4 5

jdiner
08-22-2002, 05:59 PM
Ok. This is the second new Thread I am going to create. This thread is most definately a discussion of how to produce to mux'ing/working/finished/finalized/ready-to-use MPEG2 files.

Further it is to be a place where we can all trade ideas, discuss code and applications, be it for the Tivo or just for MPEG2 in general. However there are some rules for this thread.

Just like before these are rules for this thread, not the forum. If you want to discuss things deemed off limits here make your own threads and set your own rules. But if you decide to participate here then please follow the guidelines.

So the rules for this thread are:

1- Leave the flames and arguments elsewhere... Again I am interested in getting things working better and moving forward, not a re-hash of the problems covered in other threads. So here be nice or stay out.

2- We discuss ANY tools dealing with mux'ing or viewing or editing MPEG2 files or TyStreams... But let's focus on things that are built for TiVo output as this is a Tivo Forum. I am not the only one working on this, and while I intend to have my discussions of things here I have no arguments again others doing the same. If you find a tool that is better that any you have seen before, then share.

3- NO DISCUSSION OF EXTRACTION. This thread is about MUX'ing and making working MPEG2 files. To do that you have to have elementary streams. So no matter how you get them you have to have them before coming here with ideas or problems.

4- Feel free to discuss various OS wishes/desires. At some point we should support all of them or at least as many as is possible.

5- Let's limit the discussion of VSplit to how it interracts with MUX'ing rather than usability or general feature set. We have to discuss VSplit as it is the core of my current MUX'ing engine. Also in the works is a MUX'er for exisitng VSplit output that will stitch it all back together as appropriate for a tivo file that can be done long after the a/v split has been done.

6- Feel free to ask for features but be aware that I am making no promises except to work on things in general... I mean it. Ask. But don't hound me or any other author, that accomplishes nothing. If you think of something that would be cool let me know. I will add it to the list.

I would like this thread to be about the previously mentioned "technical amazement" rather than usability in my tools, ExtractStream, etc... That is what the other thread and this forum is for. Here if you find something sweet dealing with MUX'ing/editing then let's all talk about it and move forward in getting to that "perfect" point where everyone is happy.

The technical discussion of the mux'ing and what is going on behind the scenes will be held in here. Some don't care about it. If all you want to discuss is tools for extraction/splitting then go to the other threads.

So if anyone has things on their wish list for MUX'ing now is the time and this is the place.

--jdiner

RxMan
08-22-2002, 06:17 PM
Obviosly everyone wants muxing in a hope that it will eliminate audio/video sync problems. Has any progress been made with the timestamps and how the tivo handles them?

jdiner
08-22-2002, 07:02 PM
Yes. I have had timestamps perfected for a long time. That is why the A/V split via my VSplit code works "perfectly".

There are some having problems with it, but as I explained before, that is an issue with the mux'ing mechanism getting off rather than problems with VSplit.

Once before I mentioned some facts. I have not kept up on them so they are old. But I have had right around 20,000 prefectly split streams reported to me. I myself have done about 300 since January that were perfect.

I use this information in my mux'ing and things stay just as expected. Sync'ed all the way through. Right now I just have to fix the PES streams I am generating and the first cut at mux'ing should be usable.

--jdiner

FreydNot
08-22-2002, 07:48 PM
I have been playing with burning SA extracted files to DVD. My goal is to cut out the commercials so that I can fit 4 "1 hour" 3,500kps episodes on a single 4.7Gig DVD.

I've been using DVDMaestro for authoring. I can extract elementry stream using vsplit. That works great. If I use those streams and go strait to DVD, everything stays in sync. If I run them through and editor (M2edit has been what I use most) the audio always drifts.

The last stratagy I tried was triming out 5 pieces which represented the show without the commercials. Then I placed the 5 clips back to back to form a single move in Maestro. Maestro will create a single audio file from a group of clips (and even has a switch to keep them in sync). This function only works on AC3 files.

So I used BeSweet to convert the .m2a files from vsplit into 48k ac3 files. That worked, but the sync was hopelessly lost. Loading the m2a file and the ac3 file into an editor (Goldwave is my favorite) showed they were clearly a different length (even after accounting for the 32k to 48k upsample).

It seems like a simple trim using M2edit should produce a single file with good sync. I'm not sure why it doesn't.

So my question is this: What should I expect from muxing? Will the fabled muxing version of vsplit add some kind of information to the program stream which will let m2edit (or mpeg-vcr or whatever) not get out of sync? I'm starting to wonder if I've been waiting for the wrong tool.

Comments?

scarabus
08-22-2002, 08:04 PM
Everybody seems to be looking to muxing as the holy grail of A/V Sync. But as far as I can see it's really only going to introduce additional unnecessary steps.

While the basic entry-level packages will accept multiplexed MPEGs, most of the high end stuff requires elementary streams. So having split our streams and multiplexed them we have to demux again before we can use them, with all the hassle that entails.

On the other hand there are plenty of perfectly good tools that will take elementary streams and multiplex them to form program streams. If we have good, synched elementary streams we will get good synched program streams. GIGO.

We all want accurate sync God knows how many times I've tried to make perfect Max Headroom discs but is muxing really the best/only way to do it? Would it not make more sense to improve the way that the splitter tools handle the raw tystream so that hopefully we don't get bad sync in the first place? And at the same time maybe we can fix some of the other stream handling problems. I've got several instances of streams that I can't use but play fine on the TiVo. I'm getting about a 90% success rate on extraction.

So in my opinion it would be a more sensible use of resources to try and better understand and decode the tystream.

BUT having said that I'm not the custodian of the knowledge and the tools, so if this is the direction that development is heading then I'll be happy to help if I can.

Pr.Sinister
08-22-2002, 08:43 PM
The reason why we are all waiting for the muxing is because a lot
of us have edited commercials out of shows we recorded with a
capture card and we never had any A/V Sync problems after editing
out commercials. So we are inclined to think that maybe the way
we mux right now is not perfect.

If i transcode the streams with TMPGEnc and i check the box that
says to create a GOP structure suitable for editing, i have no
problems. But obviously i don't want to waste TIME and QUALITY
transcoding the streams i get off the TiVo... It beats the purpose
of me getting a TurboNet to not have to record from TiVo to
my capture card.

So i will continue to hold my breath for a perfect Muxed stream
that can be edited by ANY MPEG-2 editor.

Pr.Sinister

jdiner
08-22-2002, 08:53 PM
Originally posted by FreydNot
I have been playing with burning SA extracted files to DVD. My goal is to cut out the commercials so that I can fit 4 "1 hour" 3,500kps episodes on a single 4.7Gig DVD.


That pretty much sums up my goals as well. Pretty much perfectly in fact.



I've been using DVDMaestro for authoring. I can extract elementry stream using vsplit. That works great. If I use those streams and go strait to DVD, everything stays in sync. If I run them through and editor (M2edit has been what I use most) the audio always drifts.


The problem is, as I mentioned a long long time ago in thread far away, that MPEG-2 is a very very... forgiving spec. Things that are perfectly legal are honestly... a bad way to do it. The much tighter specs used for a DVD force limits that make recovery after bad data much much easier.

For instance. In MPEG-2 it is possible and strictly speaking legal to have hundreds of Frames (I, B, or P) in a single Group-Of-Pictures (GOP). For a DVD standard spec is fixed at either 16 or at 18 if I remember the spec correctly. Anything other than the exact number is just plain not in the DVD spec.

Now since most modern DVD players are also built to play VCD and SVCD disks this is not a problem. For most as long as it is a valid MPEG-2 file, not just a DVD spec file, it is playable. In my slightly modified APEX-600a player you can play just about anything right up to and including muddy-water.

Back to the point... The drift most people are running into is a simple matter of "assumptions". The MPEG-2 editors I have tested all have assumptions about not only what is legal but what form and layout it will be in. The output of a tivo TyStream is NOT INVALID, make no mistake about that. But for whatever the reason it is seriously "non-standard". When the assumptions get in the way you get errors.



It seems like a simple trim using M2edit should produce a single file with good sync. I'm not sure why it doesn't.


While I can't answer this definatively having not written this tool... I hope I have provided some answer to this questions.



So my question is this: What should I expect from muxing? Will the fabled muxing version of vsplit add some kind of information to the program stream which will let m2edit (or mpeg-vcr or whatever) not get out of sync? I'm starting to wonder if I've been waiting for the wrong tool.


Yeah. Once you build all of the absolutely correct information into an mpeg stream m2edit appears to edit it perfectly. At the moment the size of the file has bloated a bit to get to this stable point but it does work.

Now all the normal caveats... It works for me. Your mileage may very right up to the point of complete failure. But since it does work for me on what I am doing it is reasonable to expect that at some point soon it will work for you.

As for waiting for the wrong tool. Keep in mind that this is all steps:

Step 1 - Getting a good mux produces files that can be played on a PC without errors.

Step 2 - Expand this fix so that it can be played on a DVD/VCD player either as a VCD/SVCD disk or a DVD disk.

Step 3- Add the promised, and somewhat fabled, visual GOP editor. This will allow the cutting of commercials and other things from the Tivo streams. The output of this will either be a single mpeg-2 file that is everything but what is cut, or a set of files broken on each cut point...

Step 4 - Extend the mux'ing output file format to be a DVD... crap. I forget the name of the DVD record format for a single "block" on the disk. Anyway, so that what is produced is directly burnable with no processing by other programs needed.

This is what I intend to do. If this fits your needs, then someday soon you will have what you need. If not, then keep looking elsewhere or suggest to me another step that will work for what you need.

--jdiner

jdiner
08-22-2002, 09:11 PM
Originally posted by scarabus
On the other hand there are plenty of perfectly good tools that will take elementary streams and multiplex them to form program streams. If we have good, synched elementary streams we will get good synched program streams. GIGO.


If you have a stream that VSplit does not produce a good split on then send me a copy of it. And I will fix vsplit. But please before we get to that point prove to yourself that it is really not splitting correctly.

Do the following as exactly as possible since it is what I do to prove such things:
1- Grab VirtualDub's latest version. Grab WinAmp's latest version.

2- Grab DVD2AVI version 1.74. And the related VFAPIConv program.

3- Get Vsplit. The latest version is old now but still the latest.

4- Run the TyStream through VSplit.

5- Check the size of the audio and video to make sure you really have a file for each of about the right size. Video+Audio file size should be 95% or so of the size fo the TyStream itself. Also check the output VSplit to make sure no "real" errors were reported.

6- Check the audio file in WinAmp by playing part of it.

7- Check the video file in WM or some other player to make sure it is playable.

8- Reconfigure WinAmp to play to an output file and send the output there.

9- Load the video file into DVD2AVI and with NO configuration changes save a project with it.

10- Convert the project to a "fake avi" with VFAPIConv.

11- Load this "fake avi" into VirtualDub. Set the video compression to RGB uncompressed. This is technically "no compression". Since we won't be making a output AVI this is fine if we were the file size would be massive.

12- Open the audio file. And set the sync with the Audio Interleaving option.

13- Play it at very start of the file and make sure A/V lip-sync is correct.

14- Then seek to whatever location in the files you wil and play starting there. And watch for sync. I have done this countless times with tons of test clips. (Each of the 300 clips I have done for my own archives I have done exactly this with to remove commericals...)

That's it. For me sync is perfectly maintained.

What can can sync to get off in the above process:

1- The biggest culprit is the audio compression. For some unknown reason, unknown to me because I have never ever looked into and have no plans whatsoever to do so, if you compress audio into MP3 format to make it smaller (which I do) it gets off. The playback speed of the MP3 audio stream is actually off. You can correct for this and maintain sync.

2- If you use a video compressor depending on the type and strength of compression the sync can be lost as well. Frames sizes to large? I have no idea I just know that it happens. Neither DivX nor XVid have ever gotten off for me, but some of the others have during testing...

These are the only 2 culprits I know of.

I meant it when I said use the above. I know there are a ton of other tools that can be used at each step. But I can't predict the interraction of the programs or any bugs/problems they might introduce. So use the above, if it is still off you have found a problem I have not seen before. If it is on you are suffering from the same problem as everyone else when it comes to dealing with the elementary streams themselves and having the software MUX them.




We all want accurate sync God knows how many times I've tried to make perfect Max Headroom discs but is muxing really the best/only way to do it? Would it not make more sense to improve the way that the splitter tools handle the raw tystream so that hopefully we don't get bad sync in the first place?


If you have found a problem in a TyStream not related to a bad extraction directly, i.e. you got it all or to an emu/hack card that once before was just randomly dropping chunks and VSplit fails on it then I can fix it once I get the TyStream itself.

So run the above and we can decide on a direction to move in at that point.

--jdiner

scarabus
08-22-2002, 09:19 PM
Originally posted by jdiner
If you have a stream that VSplit does not produce a good split on then send me a copy of it. And I will fix vsplit.

I have several. See my response in your other thread.

I'll be happy to snail-mail the streams to you on DVD - send me a PM with an address to mail to.

Alternatively I can FTP them if you have a site with enough storage. It'll probably take the same length of time :-/



Originally posted by jdiner
So run the above and we can decide on a direction to move in at that point.

I will do that thing.

jdiner
08-22-2002, 09:30 PM
Originally posted by scarabus

I have several. See my response in your other thread.

I'll be happy to snail-mail the streams to you on DVD - send me a PM with an address to mail to.

Alternatively I can FTP them if you have a site with enough storage. It'll probably take the same length of time :-/


Yeah. That is why I want you to use the TyFileSplit tool I wrote. It will let you identify a single small block that has a problem. 20 meg is easy to upload if you have a decent link. If not then dvd is probably best.

--jdiner

scarabus
08-22-2002, 09:36 PM
For the benefit of everyone else you can find TyFileSplit
HERE (http://dealdatabase.com/forum/showthread.php?s=&postid=38137&highlight=tyfilesplit#post38137)

RxMan
08-22-2002, 10:41 PM
I guess I am a little confused. Is TyFileSplit different than using TyTool in VSplit mode? I have one file that has sync issues. I still have the file on my Tivo so I can extract again and follow the steps above if necessary. Just let me know if there is a difference and I will proceed.

jdiner
08-22-2002, 11:02 PM
Ah ok. To try and clear that up as people might get confused.

All that TyFileSplit does is split a TyStream file into a smaller valid TyStream file. I.e. it stays on the byte-count correct boundries etc...

It does not do the A/V split just cuts something that can then be processed later. Namely by me when looking for problems.

Using split in both names might have been a bad idea... :(

--jdiner

laserfan
08-22-2002, 11:49 PM
Originally posted by jdiner
...
14- Then seek to whatever location in the files you wil and play starting there....That's it. For me sync is perfectly maintained.

What can can sync to get off in the above process:

1- The biggest culprit is the audio compression. Uff da! 14 steps, wow. Here are 4+ steps that work for me, noting I have an SA and need to convert audio (you're right of course about audio compression as a key problem):

1. Mux the m2a and m2v files using MPEG2VCR. If the streams are "off" it may take a couple tries to get the AV sync right (quick though, as a "try" need only take 10-20secs to output enuf program to check sync), but it's done here, right up front.

2. If there's any trimming to do, I use MPEG Cutter from Canopus, remember my audio is still 32kHz.

3. Import the resulting trimmed mpg to MPEG2VCR again, this time use "Record" and set Audio to 48kHz (if for DVD) or 44.1kHz (if for SVCD), and change audio bitrate also if you like, but this results in the export taking a little longer.

4. Test! Done! Burn w/Nero or import to SpruceUp to author a DVD. This whole process takes only a few minutes for an hourlong program; MPEG2VCR is really fast, fast, fast both for muxing and for transcoding audio. Still, for a movie I do walk away from it and do something else for a while.

Note that MPEG2VCR has mpg-cutting ability, but altho I have waxed eloquent about this tool in another thread, alas I've discovered that if you use IT to cut it results in audio drift. I can only think that while Canopus' tool cuts on I-frames, and MPEG2VCR appears to NOT, that this somehow is hosing the audio. Also note that converting the audio before cutting doesn't work--sync is lost again. So I leave the audio alone until the very end.

It is the case for me now that the lengthiest part of the creation process is the transfer from Tivo-to-PC over my 10baseT Tivonet board. These other steps have become almost trivially easy, tho occasionally I find a pair where it's difficult to find the audio "sweet spot" especially for the music programs I'm archiving.

A final caveat is that while the above allows front-and-back trimming, it does not allow (or I have not found out how to do) excising from within e.g. to cut commercials. But an easy thing for me to do, if for example I want to save two songs from a Sat. Nite Live, is to cut each song out of the same muxed (again 32kHz) file and save them separately, and import each separately to MPEG2VCR to transcode the audio.

Wes
08-23-2002, 02:56 AM
Originally posted by laserfan
Uff da! 14 steps, wow. Here are 4+ steps that work for me, noting I have an SA and need to convert audio (you're right of course about audio compression as a key problem):


The 14 steps is to prove the sync from vsplit was perfectly maintained. It's not the process that we usually would do.

I've never have sync problem with my SA. I only have the sync problem very occasionally with my DTivo. (This is a DTivo forum, not SA. Please don't compare apple and orange here. ;))

lmurray
08-23-2002, 10:33 AM
Jdiner,
over in this thread:

http://www.dealdatabase.com/forum/showthread.php?s=&threadid=16133

we've been talking about tools for macosX. I've compiled some of the basic tools for macosX, but would really like to get a binary of vsplit13 for splitting ty files. Can we work to compile vsplit13 on macosX? I'd be glad to do this if given the source. Most of the other programs I've compiled (dare i say port) required little to no work to compile on macosX.


Thanks in advance,
-lloyd-

jdiner
08-23-2002, 02:23 PM
Originally posted by laserfan
Uff da! 14 steps, wow. Here are 4+ steps that work for me, noting I have an SA and need to convert audio (you're right of course about audio compression as a key problem):

1. Mux the m2a and m2v files using MPEG2VCR. If the streams are "off" it may take a couple tries to get the AV sync right (quick though, as a "try" need only take 10-20secs to output enuf program to check sync), but it's done here, right up front.


Ok. Back to what I was addressing. This is not an optimal solution. It was never advertised as that. IT IS A TEST SOLUTION THAT ALWAYS PRODUCES IN SYNC A/V FOR VERIFYING THE OUTPUT OF VSPLIT AS CORRECT... I am not making DVDs here. I am not making MPEGs here. I am not making avi's here. I wanted people to know how to detect if it is the mux'er/software, or the output itself that was getting off.

--jdiner

justintraining
08-23-2002, 03:22 PM
I have benefited greatly from those on this board, and wish I could contribute more to the "cause".

At this point, the most I can contribute is an FTP site for sharing files as needed.

ftp://63.165.143.229
Anonymous access is allowed.

I am also looking for a few individuals to help maintain the site. You know sort, filter and generally keep the site clean. Please e-mail me for a name/password with more rights.

As we progress with this thread, and others, I plan to setup a website with downloads and “How-To” information. Again, I will be looking for help in this area as well.

My e-mail address: justintraining@hotmail.com

jdiner
08-23-2002, 05:56 PM
Ok. Here is what I need to get on to the next step. Some MPEG-2 multiplex pieces.

Out there are people that are using mux'ing tools to recombine a TyStream split with my tools that are maintaining sync. I need some. Not all of them probably a 20meg chunk is enough.

Now I am not talking about streams that can be edited. All I am talking about is streams that when mux'ed with whatever work in playback and keep sync.

OS doesn't matter to me. The tool doesn't matter to me. etc... etc... etc... All I need is a 20meg or so piece of a working MPEG-2 file from a DTIVO.

I can obviously find MPEG-2 files from other sources. But these need to be DTIVO specific sources. Further I need if at all possible multiple pieces of the "same type" of show. I.e. the first 20 meg of 3 of 4 episodes of Show-X. As they would by and large all be similar in size and content.

To start with what I want is a show of hands. Who has, without editing, produced good MPEG-2 files and with what tools. What OS. Etc... If I can make my own I will and not need uploads. If not I will need some uplaods.

The reason for this is that I have found a problem in some of my calculations for the multiplex fields. If I can find values that work I can hopefully determine what the difference is and get things working accordingly.

So chime in and help us all out. :)

--jdiner

dlang
08-23-2002, 06:28 PM
I have a bunch of mpegs from a dtivo, but I don't have many I would say have good sync on them :-(

one thing I have noted is that if the recording happens across a program break (i.e. I start my recording of thunderbirds 1 min early) the av sync error reported by vsplit is _very_ different if I have it split the entire show vs if I have it skip stuff at the beginning (to cut out the tail of the previous show)

I don't know if commercial breaks cause similar problems

Rotten
08-23-2002, 07:48 PM
jdiner,


Ok I have been able to produce good snyced video from tytool5 muxing with Multiplex (Not simple Multiplex) in Mpeg tools in TMPGEnc Plus. This gives me a great MPEG every time. However This file can only maintain good sync if played independently or if burned to DVD without editing in Spruce. As soon as you introduce it to any other program like DVD workshop or MPEG2VCR the snyc is gone and gone bad!. I will continue testing as I would prefer to use other tools to burn with. If I can ever be of service please let me know. Keep up the great work.

homer08
08-23-2002, 11:00 PM
I just tried TEMPGEnc Plus Version 2.57.41.146 using the multiplex function described above and it worked great. Weeks ago I had tried a couple of other tools (can't remember which) and was not able to get a sync'd mpg.

bronco13
08-25-2002, 04:31 AM
Where is everybody getting TEMPGEnc Plus Version 2.57...... from???

Is it on justingtraining's site???

Could someone upload it there?

jfkjfk
08-25-2002, 05:36 AM
I can't find Canopus MPEG cutter any where.. I'm understanding it's part of they're software/hardware package.. I've also read that it only install on machines with a Canopus card in them.. anyhow.. anyone know if this is true or not.. I've played with everything I can find.. M-2 Edit works ok, but it CHUGS so slowly and crashes when I try to use files over about 600megs...
ANyhow.. Can anyone help me out with MPEG Cutter.. Does it work well with large files 1-2gig..

thanks.

jfkjfk

perhaps someone could u/l to the ftp site mentioned above..

jfkjfk
08-25-2002, 06:19 AM
Just played with MPG2VCR for the first time.. You guys are mentioning sync problems with it.. When I preview files from DTIVO d/led and split with tytool.. I remux using mplex.. Audio sync at beginning of file is on.. When I get further into a 1.5GIG file say an hour in I notice that the preview is WAY OFF sync.. Where talking a couple seconds.. I tried extracting a portion of clip.. Same result..
You guys are talking sync problems.. I just don't think these mpeg editors are properly built, and they're flaws just show up when we use these HUGE tivo files.. Who else is chopping up 2gig MPEG files? Same issues I had with M2-Edit.. Works fine on small files, can't handle huge files..
Currently there a program I use to convert MPEGs to divx.. Sync seems very good if not perfect.. There is no audio in the priview, and the frame accuracy for edits is horrible.. In fact even scrolling through is not very accurate, but it's UI flys.. And it's open source.. I've considered trying to get back into programming, but I haven't done windows or any C++ since 1994.. So perhaps someone with a little more experience would consider adding a few thngs to it... anyhow.. it's a little hard to use at first.. There is an export n frames function that start at the frame your preview and exports n frames.. It's very fast.. Uses flask type plugins.. anyhow take a look.. Perhaps we can mold this into a nice custom tool...
<http://www.mpeg-mediator.com/>
I use XMpeg to scan and find my in and out points(File-info-current video variable), and then I compute the n frames for export of a divx avi.. Then I chop further with VirualDub..
--edited this link so you get enlish page---
<http://www.mp3guest.com/Xmpeg_Index.asp?l=US>
these are the best two tools I could find... All other tools I've used including FlaskMpeg, crash, or lock up for strange reasons...
Unfortuately I still haven't got a good MPEG cutter.. After seeing the ease at which these two programs fly though an mpeg, and the accuracy/sync of MPEG MEdiators there has to been something out there.. But I've given up hope, and think that trying to add a little more functionalitly to MPEG mediator may be the answer.. anyhow bla, bla, bla.. I might be telling you all stuff you already know.. But I have spent at least 30 hours over the last couple week messing with tools trying to find something to CUT these large MPEGs well..

anyhow.. that's probably enough for now...

jfkjfk

laserfan
08-25-2002, 10:10 AM
Originally posted by jfkjfk
...Can anyone help me out with MPEG Cutter...I'm the culprit, having mentioned this a couple of times. Sorry, but I've looked into this, and not only does it not INSTALL without a Canopus card in the PC, I found out yesterday that if it's installed and then you remove the card, it still looks for the card and therefore doesn't run without it. I don't know how to defeat this or I would, so I could do some editing on other PC(s) I have here.

dattrax
08-25-2002, 12:29 PM
I've been getting great results from TyTool

1, TyTool
2, bbmpeg (use the offset)
3, Mediaware solutions m2-edit
4, TMPEG to reencode to DVD compatible file

takes about 10minutes to step 4 and then the cost of a TMPGenc encode.

I also use ssrc and toolame to get the audio, as tmpgenc is lacking.

The only feature I could do with is the 'bbmpeg' removed

Jim

jfkjfk
08-25-2002, 06:14 PM
dattrax,
what kind of file size are you throwing at M2-edit?
I had problems with M2-Edit and large files.. It's way too slow... Pause of 5-10 seconds or so every time I try to seek into the file.. It worked ok for files of 500Megs or so... but was still pretty slow..
Is the UI slow for you?

jfkjfk

jfkjfk
08-25-2002, 08:15 PM
laserfan,
Does mpeg cutter work well with large files? I would have no problem paying for something that works well... I don't want to buy something that makes me struggle.. Crashses. locks up.. or takes a 10 second pause every time I try to move the seek pointer...
what type of card does it require? How much $$

thanks..

jfkjfk

jdiner
08-25-2002, 08:15 PM
Ok. I have run a bunch of tests. With BBMpeg muxer I get massive errors. Off in the first 3 minutes by over a second.

With the latest version of tmpgenc's muxer I get what appears to be good sync all the way through. It was hard for me to tell if it was getting off or had just stayed off. The test clip I used was off by 11ms and I can see that. I expect most can.

My next set of tests go with the new mjpeg mux'er. I got it installed on my FreeBSD box. Man the /usr/ports stuff is very very nice... :)

Once I determine what is really working the best it will help move things forward. My own works. It just doesn't work everywhere in each player. I am trying to figure out why...

--jdiner

dattrax
08-26-2002, 04:48 AM
jfkjfk

No problem in m2-edit with any files at all, UI is not blazingly fast, but I wouldn't call it slow.

I can chop out a commercial in about 30 secs.

Jim

dattrax
08-26-2002, 04:53 AM
jdiner,

I should have mentioned how I multiplex for bbmpeg. I select the option for multiplexing a DVD stream, and then add (or subtract) the audio offset given by TyTool.

If you use both of these then bbmpeg should keep it in sync

Jim

jdiner
08-26-2002, 07:11 PM
the bbmpeg output is still bad for me. But that could be any number of things. However both the tmpgenc and the mjpeg mux correctly as long as no editing takes place. I have no clear idea yet how either affect the editors as I am now using new versions of both and my editing tests are old.

--jdiner

jfkjfk
08-26-2002, 08:18 PM
jdnire,
are you using the dos compiled mjpeg muxer or unix muxer?

what settings are you using?? what's you command line?

thanks..

jfkjfk

jdiner
08-26-2002, 08:33 PM
I used the Dos compiled one posted here and am getting ready to use the unix one...

The command line I used was:

mplex -f8 -V -o ..\da-mjpeg.mpg ..\da.m2v ..\da.m2a | more

The more is just so I could watch the output. The rest is pretty simple. -f8 means mux for a DVD. The -V puts it into verbose mode. And the -o sets the output file.

--jdiner

jdiner
08-26-2002, 08:35 PM
Very interesting. The same jerkiness that I was seeing in the output from my mux'er code is in the BBMpeg output. Since that is where I used to determine a few things that makes sense. Perhaps it is version of bbmpeg. Are those that are being successful with it using something other than:

version: AVI2MPEG - v 1.24 Beta 13.

It was the newest thing I could find.

--jdiner

dlang
08-26-2002, 08:56 PM
actually I think the -V makes it work with variable bit rate files, I think you want to look at -v (lower case) to get more verbose output.

jfkjfk
08-26-2002, 11:02 PM
Originally posted by jdiner
I used the Dos compiled one posted here and am getting ready to use the unix one...

The command line I used was:

mplex -f8 -V -o ..\da-mjpeg.mpg ..\da.m2v ..\da.m2a | more

The more is just so I could watch the output. The rest is pretty simple. -f8 means mux for a DVD. The -V puts it into verbose mode. And the -o sets the output file.

--jdiner
Yeah.. That's about same here.. Isn't -V for variable bit rate? I do use it.. I was trying to use -S to split the file up but it wouldn't work with -f8 but would with -f3?!?! (I don't know the difference)
Why when using -f8 does it say...
INFO: [mplex] Selecting DVD output profile (INCOMEPLETE!!!!)

Also I ran some muxes that I did through a demo of m2-edit's pro 5, DR M tool.. It complains about all sorts of problems.. Maybe this will help you?!? I attatched it.. I dunno if this tools is right or not, but here it is..


jfkjfk

wh1212
08-27-2002, 04:03 AM
As we all know, Dtivos are vbr,and SA's are cbr or vbr,Your choice.
There is nothing wrong with, Tytool5 it does one thing great,everytime WOO HOO, Yes it's out of sync -33ms for every 30mins recorded,-66ms for 1hr and so on, BUT to notice, AVsync would have to be out by at laest, +or- 500ms.
Ok back to cbr vbr and muxing,You must mux cbr in cbr,M2edit
vbr in vbr,mpeg2vcr etc.
Play in powerdvd,etc.
Out of sync,HUMAN ERROR, Do it again.
Are we in sync, If not find a guide.
Ok editing,The key,Do not RE-ENCODE,You need to cut and join.
Guides at vcdhelper.com(How to cut and join)
Play movie, out of sync,HUMAN ERROR,Do it again.
Success.
Ok do yourself a favor,DVD media 4.7gigs,love it.
Mpeg2 hate it,It has to be,10Mb per sec or it looks like a cheap vcd.
Mpeg4 is now open source, Which means set top boxe's,coming to a store near you.Why have only 2 shows, on a dvd,when you
can have, 10 or more,at better quality. Try it,use gordian knot,Guide at Doom9.org or net ?.

pbar
08-27-2002, 01:10 PM
Originally posted by wh1212
As we all know, Dtivos are vbr,and SA's are cbr or vbr,Your choice.
There is nothing wrong with, Tytool5 it does one thing great,everytime WOO HOO, Yes it's out of sync -33ms for every 30mins recorded,-66ms for 1hr and so on, BUT to notice, AVsync would have to be out by at laest, +or- 500ms.


Personally, I find that 100ms out-of-sync is almost unwatchable... (seems that you buid up some amount of error for each 'part' of the tystream?)

I may just be imagining this, but if I extract a program as 'separate parts' using tytool and mux them seperately (I use mplex) then the sync error doesn't build up and I can then burn them to DVD as seperate chapters.

Paul

jdiner
08-27-2002, 04:11 PM
Originally posted by wh1212
As we all know, Dtivos are vbr,and SA's are cbr or vbr,Your choice.
There is nothing wrong with, Tytool5 it does one thing great,everytime WOO HOO, Yes it's out of sync -33ms for every 30mins recorded,-66ms for 1hr and so on, BUT to notice, AVsync would have to be out by at laest, +or- 500ms.

I notice a whole lot less thatn 500ms. But how did you determine how far it is off? I have tested and tested and tested and I never see anything off at all. If I get right at the start I stay on. Fill me in. Maybe I am missing something. Maybe in your case the extract is actually bad.

--jdiner

laserfan
08-27-2002, 08:18 PM
Originally posted by jdiner
...I notice a whole lot less thatn 500ms. But how did you determine how far it is off? I have tested and tested and tested and I never see anything off at all. If I get right at the start I stay on. Fill me in. Maybe I am missing something. Maybe in your case the extract is actually bad.First you say "I never see anything at all" and then you say "If I get it right at the start I stay on"? What are you saying? Have you seen the problem or haven't you?

I have seen m2a and m2v extracted via TyTool that are off at the start by as much as almost a second (1000ms last time I checked). These can be muxed to stay in sync, unless any edits are attempted. I have also had extracts that are close enough to not notice any problem at all, but...if edits are done, all bets are off--sync goes to heck at edit points.

jdiner
08-27-2002, 10:19 PM
Originally posted by laserfan
First you say "I never see anything at all" and then you say "If I get it right at the start I stay on"? What are you saying? Have you seen the problem or haven't you?


Ok. I can tell I was not clear enough. The first comment stands alone. I don't see any of the problems being mentioned. I ran another 5 shows through last night as a test. No problems. No loss of sync. Nothing...

The second statement should have been a seperate sentence:
"I have tested and tested and tested and I never see anything off at all. If I get right at the start I stay on."

Way back when all of this started. There were times when things were off at the start and stayed off. I no longer see any of these. (See statement #1) So my followup comment was to mean, "I start on, I stay on, I finish on..."

So if you have something that does not. Then clip it and send it to me.

Bottom line, there should be no way to get off by 1000ms... The code and the Tivo doesn't allow for more than 33ms. And in the TyTool5 and VSplit #13 there should be no way to be off by more than 17ms.




I have seen m2a and m2v extracted via TyTool that are off at the start by as much as almost a second (1000ms last time I checked). These can be muxed to stay in sync, unless any edits are attempted. I have also had extracts that are close enough to not notice any problem at all, but...if edits are done, all bets are off--sync goes to heck at edit points.

Editing is a totally different issue. It can and will smash things and has nothing to do with current splitting issues. As for the getting 1000ms off. Send me the clip. I would love to figure out what is going wrong...

--jdiner

jdiner
08-28-2002, 10:19 PM
Well I am right back where I started. I am getting so tired of this...

Nothing makes any sense. I have been working over the mux'ing output. The 2 things tha work "almost" for me in my testing are the DVD form in BBMpeg and the tmpgenc muxer.

However in disecting the files to see what is really going on within them to find out what I did wrong I can't believe they work now or ever have. The timestamps in them are completely smashed. They have things set to decode long after they set to be presented.

I just don't get it. Nothing seems to follow the docs/specs. What doesn't run well seems to be closer than the rest of it.

I am lost as to what to try next. It should not be this hard. I have said that before. I will say it again. It should not be that hard.

I am open to suggestions on where to look or what to try. Anyone?

--jdiner

omf
08-28-2002, 10:38 PM
Originally posted by jdiner

I am open to suggestions on where to look or what to try. Anyone?


Are you always muxing DTivo audio/video streams to check your output? You might want to start with source from somewhere else to see if it makes any more sense. For example, capture some DV video and create elementary audio/video streams from it. See if the muxing of those streams looks as screwy as what you're getting when you start from the DTivo streams.

laserfan
08-29-2002, 12:52 AM
Originally posted by jdiner
...I am open to suggestions on where to look or what to try...I am a rank beginner at all this so take the following for what it is worth:

MPEG2VCR (Womble) can take your vsplit-output m2a and m2v files and mux them together so that they are in sync. This proggie also has a Repair utility built-in for subsequent checking of these files, though it only works with 720x480 programs (which I get out of my SA Tivo). It detects GOP time code errors, audio PTS errors, and GOP size errors and repairs them. The resulting files can be imported into various authoring tools e.g. SpruceUp, which does some <significant?> checking of these files on import.

All I am saying is that it appears that these muxed files are DVD-compliant, therefore I would assume that someone <much smarter than I> could look at streams before & after and perhaps this would help you with what is going on...

Note I say this without knowing what tool one might use to "look at" an interlaced mpg program, much less make sense of it. Only throwing out ideas here hoping to give you a fresh "angle" to look from.
:confused:

Elwood12
08-29-2002, 01:12 AM
I am a not very knowledgable on the whole Muxing thing. And I hope this is not too newbie or off topic but it seems that the problem everyone is having with whole audio syncing is that something is missing after you spilt the ty file into m2v and m2a files.

After searching these forums and the web it seem that the reason is to get rid of the tivo PES headers. These PES headers from tivo I have read contain the time stamp info. I also have read that the tystream is a form of Mpeg2 file already. That said, can we replace the tivo PES headers with the correct Mpeg2 headers we need and skip the splitting to m2a and m2v?

If I have made any wrong statements please tell me so.

jdiner
08-29-2002, 01:15 AM
Yeah. More programs I fear is the key. I am grabbing a few more now to try it out with.

The output from the nanoPeg program is garbage. The Xmuxer won't install for me for some reason.

I was hoping to find something good to emulate rather than continue to try brute force with the long road to getting things working that that can be.

As for trying other things. I have. I have downloaded a number of things from the net as source mpeg-2 examples. Most of them are "off" in similar ways. But nothing ever really repeats.

Like I said I don't get it. The frame orders make no sense. The docs are clear. It should go: IPBBPBBPBBPBB in the first GOP and then in every GOP after that it should be IBBPBBPBBPBB. Which gives PTS and DTS timestamps in the normal fashion.

In looking at the current output it is not even close to that. I am going to keep digging into things. It is just not matching up to anything that is expected.

--jdiner

jdiner
08-29-2002, 01:17 AM
Originally posted by Elwood12
I am a not very knowledgable on the whole Muxing thing. And I hope this is not too newbie or off topic but it seems that the problem everyone is having with whole audio syncing is that something is missing after you spilt the ty file into m2v and m2a files.

After searching these forums and the web it seem that the reason is to get rid of the tivo PES headers. These PES headers from tivo I have read contain the time stamp info. I also have read that the tystream is a form of Mpeg2 file already. That said, can we replace the tivo PES headers with the correct Mpeg2 headers we need and skip the splitting to m2a and m2v?


No this is in essence exactly what I am trying to do.

One thing that should be made clear. The TIVO has headers in it that are part of what is going on. But they ARE NOT PES HEADERS. They are "close" and contain some similar data. But they can't be used as is. And the replacement process is what is causing so much trouble. If they were just there for real then this would be easy. Just dump the data in a new way and we would be there.

--jdiner

FreydNot
08-29-2002, 01:31 AM
We might find some hints by looking at the hardware encoders. Does anyone have access to the datasheets for the Sony encoder? I know the TiVo still changes the data it gets from the encoder, but I'm thinking there might be some similarties in the specs of the encoder. Maybe it will ring some bells.

Also, I know there has been discussions about the first block in the ty file in the recent video insertion threads. Something to the effect of there being data in there that most all extraction programs jump over and ignore. I know the data in that block is key in getting data back into the TiVo. Maybe some secrets of the stream lie in there?

Just my thoughts...

Elwood12
08-29-2002, 01:38 AM
I am starting to get a handle on this. I guess another way to go at it may be to see what is it you are trying to get and go backward.

If we have a valid mpeg2 file can you convert it into a tystream?

And if you can't what are you missing?

I don't suggest you actually spend too much time trying to do this as there would be many problems doing it but it may shake some ideas loose.

FreydNot
08-29-2002, 02:22 AM
I haven't personally done any work with inserting video back into TiVo, but I've read about it here http://dealdatabase.com/forum/showthread.php?s=&threadid=16238. I know they are starting with standard mpeg2 files and using a progam on the TiVo called ele2pestripple (which I assume means Element To PES Tripple). Does PES Tripple mean anything to anyone?

Jdiner, you might want to look at example MPeg2 files from the mpeg.org web site. You can download reference test patterns from http://www.mpeg.org/MPEG/video.html#video-test-bitstreams

It might also be worth looking at the output of the TMPGEnc encoder. Could be hints there.

TheDoctor
08-29-2002, 06:53 PM
Originally posted by jdiner
Like I said I don't get it. The frame orders make no sense. The docs are clear. It should go: IPBBPBBPBBPBB in the first GOP and then in every GOP after that it should be IBBPBBPBBPBB. Which gives PTS and DTS timestamps in the normal fashion.

In looking at the current output it is not even close to that. I am going to keep digging into things. It is just not matching up to anything that is expected.

--jdiner

Most of what I know about the stream I got from stare and compare, but I see very little difference in the stream I look at from a tivo, dtivo or replay once you get past the format changes, replay just uses a few diffrent headers and breaks up the data more. From you discription you I am not sure what you mean about the frame order and time stamps?

The frame ordering/time stamps I have seen on the Dtivo under 2.0 have always seemed consistant. I have never really tried to match it back to the spec's. I think the way they write it the DTV transport stream would look something like:

1I 0B 3B 2P 4B 6B 5P 7B 9B 8P...

With the numbers being the timestamp seq. Of course this is from memory so I may have botched it.

jdiner
08-29-2002, 07:44 PM
Originally posted by TheDoctor
The frame ordering/time stamps I have seen on the Dtivo under 2.0 have always seemed consistant. I have never really tried to match it back to the spec's. I think the way they write it the DTV transport stream would look something like:

1I 0B 3B 2P 4B 6B 5P 7B 9B 8P...

With the numbers being the timestamp seq. Of course this is from memory so I may have botched it.

No, you are right. You just took my comments out of context. No worries there. What was being said was that when I used one of the "standard" mux'er software programs out there to try and figure out why mine jerks and theirs is smooth the resulting file is garbage. It is mux'ed. It is a DTivo source. But the output frame orders and the SCR, SCR_EXT, PTS/DTS values are all jumbled.

If anyone really wants to see it I can print them out for you. Often things are set to decode almost a full second after they are displayed etc... Junk like that.

I continue to work on mine irreguardless of what I see in those but I was looking for a bit of a short-cut in figuring out how some of those values are calculated... But all in all it is kind of wierd in what is going on inside of the mux'ed files.

--jdiner

Elwood12
08-30-2002, 02:56 PM
What are you guys using to view the file? I assume it is not some sort of movie player but a hex editor? I would like to take a gander at the streams to see if I can notice anything.

jdiner
08-31-2002, 02:19 AM
Some people were using a program or set of programs from the bbmpeg author.

I use a pile of tools I wrote myself. Don't think they will mean much to anyone but me. They are just the better part of an mpeg-2 decoder.

I also use HexEdit V1.0 from Andrew Phillips.

--jdiner

TheDoctor
08-31-2002, 02:03 PM
For this kind of work a good hex editor is your friend, as is a good compiler.

pbar
09-01-2002, 06:04 AM
I found it useful to add a few printfs to the mjpegtools mplex to dump the PTS and DTS of each stream while it's muxing them. It's fairly easy to visualise them using something like gnuplot by plotting them as offsets from the system clock...

I haven't had chance yet to work out why sync sometimes seems to drift for me over a long stream. (my guess is that it's the muxing program trying to aim for a particular bitrate and not dealing with buffer under/overflows very well)

stealthdave
09-04-2002, 05:40 PM
Given the thread, I thought I'd share my experiences with Tivo stream-splitting and muxing.

First, here's my setup. I have a Phillips 14 SA Tivo that has been expanded to 80 hours by adding a 60G drive, and is running 3.0 Tivo software. I have the TivoNET adapter installed (not the newer TurboNET, unfortunately). My desktop system is running Mandrake Linux 9.0Beta3 on an nForce/Athlon system (I'm running a beta due to hardware compatibility issues, and it's actually quite stable). I used to dual-boot with various flavors of Windows, but I no longer have a Windows partition. I use various flavors of Wine for my Windows compatibility needs.

I use the latest version of jdiner's vsplit on my Ty streams, and when it can read the stream, it works flawlessly. I have some streams recorded on an earlier software rev (2.0 or possibly 1.4) before various upgrades that can't be read by vsplit, and I've sent jdiner samples of them a while back.

For muxing, I use one of two programs: mplex from the mjpegtools or tcmplex from transcode. I believe that tcmplex is based on bbtools, but I'm not certain. The transcode website can be found here (http://www.theorie.physik.uni-goettingen.de/~ostreich/transcode/).

I've found in general that tcmplex works better at keeping sync while seeking through an MPEG stream than mplex, but tcmplex doesn't always work split Ty streams. When it works, it tends to keep sync better when jumping around in a player. The main downside or upside of tcmplex (depending on how you look at it) is the lack of configurability of tcmplex. There's fewer options to play around with, but it's simpler to use and the default options tend to work best.

Here's the command-line that I typically use with mplex:

mplex -f 3 -b 260 myprog.m2a myprog.m2v myprog.mpg

The -f 3 specifies MPEG2 and the -b 260 sets the buffer available to the player. -b by default is 46, which is appropriate for VCD. Around 260 is what's recommended for SVCD. In my experience, this leads to better sync when seeking through an mpeg stream. Adjust to taste, YMMV.

btw, another developer of Tivo tools, Warren Toomey, recently posted a Ty stream splitting tool to the ExtractStream Yahoo! Group based on his playitsam tool which he calls "vsplit". This is obviously not the same as jdiner's tool (which works much better, IMO; I can only seem to get segfaults from Toomey's tool), but I thought I'd point it out in case anyone else gets confused. I didn't look closely enough at the author when I first saw it and thought that it was a new version of jdiner's tool. The source for Toomey's tool is available from the mailing list archives (you must be a member of the list to view the archives). His source is released under GPL, and so might provide some useful code for others to build on.

I hope that you find at least some of my lengthy discussion both useful and relevant. :)

- Stealth Dave

edpuffmonster
09-04-2002, 09:19 PM
If I recall correctly, Jdiner's vsplit is based on Warren Toomey's.

Technically he's violating the GPL by releasing the binaries with no source available. He has repetedly said, however, that he will eventually include the source, so I guess it's not so bad. (his reasoning for not releasing the source yet is arguable, but sound)

jdiner
09-04-2002, 11:17 PM
Originally posted by edpuffmonster
If I recall correctly, Jdiner's vsplit is based on Warren Toomey's.

Technically he's violating the GPL by releasing the binaries with no source available. He has repetedly said, however, that he will eventually include the source, so I guess it's not so bad. (his reasoning for not releasing the source yet is arguable, but sound)

I thought I had addressed this before. But was in another thread. So one more time. My work is all original. (I have no idea who Warren Toomey is but I will look at the link in the above message about his stuff...) That is why it actually works. No slam to previous authors. But there are reasons that nothing else has been as successful. I will release source, but since I wrote it I get to set the time frame.

--jdiner

KRavEN
09-04-2002, 11:34 PM
and for the record again. Stop friggin whining about source!!!

It is his code to do with as he pleases. We are all lucky that he is contibuting his time and effort to make the tools AND also making them available for us to use. It is his CHOICE to do this and he could have just as easily made tools for himself and never bothered with the rest of us.

So once again let it drop, move on, find something else to talk about.

If you want source so bad, write your own.

digitalAir
09-05-2002, 02:12 PM
Originally posted by KRavEN
and for the record again. Stop friggin whining about source!!!

It is his code to do with as he pleases. We are all lucky that he is contibuting his time and effort to make the tools AND also making them available for us to use. It is his CHOICE to do this and he could have just as easily made tools for himself and never bothered with the rest of us.

So once again let it drop, move on, find something else to talk about.

If you want source so bad, write your own.

To heck with the source... I WANT A PONY!!! :)

edpuffmonster
09-05-2002, 06:05 PM
Originally posted by KRavEN
and for the record again. Stop friggin whining about source!!!
It is his code to do with as he pleases.
Narf. If it were based on GPL'd code, as I thought I had read, people would have every right to whine about it :P

Stop friggin whining about my whining! :)

Besides, it's fun to pretend I'd actually do something with the source if I had it.

jdiner
09-05-2002, 06:39 PM
The only portion of anything that I have written that is based on GPL code. Is the mfs_stream program. It's source has been released.

I suppose in that light the server portion of the TyTool also contains the same code. However it is just the mfs_stream code run in a secondary thread. i.e. no changes to the source. But I suppose I should release that server source.

I dunno if anyone is interested in that part or not. Litterally it is a small parser with some socket code...

--jdiner

jdiner
09-05-2002, 08:54 PM
Oh man. Well that explains why I could not fix it. The TyStream file itself on one of the recent bad clips I was sent is actually wrong. This is a first for me. Never seen one where it is simply wrong before.

Having decided on that. I have removed it from my test pool. There is nothing at present that I can do about a bad tyStream file. When it is corrupt it is over...

Having said that it means I can remove the latest debug code and re-run the latest testbed and if all is well the next version of vsplit will be ready to roll...

--jdiner

RxMan
09-06-2002, 09:35 AM
What features will the new version contain? How will a person know if the tystream is bad or the cause of problems.

THANKS

rpongett
09-06-2002, 12:19 PM
jdiner:

Excellent. Can't wait.

Do you think the tystream as it exists on the TIVO is bad, or is it a rare problem with etraction? (i.e., could it be "fixed" by re-extracting the same program).

jdiner
09-06-2002, 01:09 PM
Originally posted by RxMan
What features will the new version contain? How will a person know if the tystream is bad or the cause of problems.

From the types of errors. If things get off and stay off all the way through that is a TyStream file error. If it is a -1x4 error report then those should now be fixed.

I can't tell if it is a bad extraction that is the problem with this one clip that I have or if it is something else. But the file itself is smashed. No way to recover and keep going. The "key" to seeing this is in getting a "fix it dear henry" message. This message is displayed if a header shows up for decoding with a totally wrong header type. I.e. we want time and position information on something that is neither audio nor video data. As evidenced by the current "bad" clip.

Think of it this way. Inside the TyStream file things line up like this:

Rec 1 ----> VHeader ----> 16 bytes
Rec 2 ----> VBody ----> 7000 bytes
Rec 3 ----> VHeader ----> 16 bytes
Rec 4 ----> VBody ----> 7000 bytes
Rec 5 ----> AHeader ----> 16 bytes
Rec 6 ----> ABody ----> 480 bytes

Now these sizes made up to some extent. I have seen video records as large as almost 128k bytes.

Things that are "always" the case. A header record points to header data, and a body record points to body data.

Now in going through this new "really bad" clip this is the case to a certain point in the file. (In everything else I have this is always the case...) Then suddenly this is not the case. Where the header records point is suddenly and inexplicably body data. Something like this:

Rec 1 ----> VHeader ----> 16 bytes
Rec 2 ----> VBody ----> 7000 bytes
Rec 3 ----> VHeader ----> 16 bytes
Rec 4 ----> VBody ----> 7000 bytes + 16 bytes of the next header + 42 bytes of the next body.
Rec 5 ----> AHeader ----> 16 bytes of the next body.
Rec 6 ----> ABody ----> 398 bytes of this body. The 16 for the next header, and then some of the body after it.

I have checked and rechecked my code and manually gone through the problem chunk by hand. And it is not a calculation errors the TyStream information itself is simply wrong.

Anyway it continues like this. Once off it stays off for several chunks. Things eventually appear to line up. But there is no way to maintain sync in the current form. We have a hole and there is nothing reasonable that can be automated to fix/find these holes. I would have to "restart" the mechanism and build a new set of files once things are back on track in the TyStream file. This is possible just very time consuming.

I am not going to do that at this point. For several reasons:

1- This is the one and only case that I have seen myself where this happened. I can't explain it but I do know from history that it is rare.

2- Why bog down in a fixes for such a rare case when we can move on with the mainstream things.

3- While I do have time now it is limited and as a result I have to prioritize.

I hope this clears a few things up. I do wonder what happened to the Tivo it was occuring on to make it happen...

--jdiner

jdiner
09-06-2002, 01:27 PM
Ok. A simple poll. There is a choice facing me right now and it affects most of the rest of you so I want some feedback.

I am working on the mux'ing as most of you know. Plain and simple right now I am having problems. I think I have it hammered it out and then something else breaks. So this could take awhile.

Now having said that I have in playing just fine as an mpeg right now. But the editing of them is still smashed.

Now I can either spend my time trying to get "busted in the same way" as something else and in so doing get back on track for editing via another program. OR I can call it good and start moving on with the rest of this stuff including the editor I have been creating for my own stuff.

My editor is not a full blown editor. Never will be. What it is is a GOP level editor for the cutting of commercials. When I did it manually some time ago, read months and months ago now, I got it working. A few visual sniggles took place but I have a technique to try to get rid of that. Since it is not a frame level editor it is not nor will it ever really be "perfect". It is possible there will be a few frames on either side that are unwanted.. But it will cut commercials and maintain sync.

This is "good enough" for me. And it is much much easier to do. Which means that I should be able to get it out the door much sooner than tweaking the dark to get things to a point where the editors out there will like it.

Ok. So the poll.

1- Keep going with tweaking the mpeg file output until m2-edit and mpeg2vcr will frame/gop edit the output mpeg successfully.

or

2- Keep making the mpeg file play in the most places i.e. software and hardware mpeg-2/dvd players. With our own custom cutting/editing software...

I was going to ask that people PM me with their suggestiong but this does affect what so many are waiting/hoping for. So let's do it here in this thread. Anyone with an option or a dream chime in. But please keep it civil...

--jdiner

bronco13
09-06-2002, 01:37 PM
jdiner,

I'm not sure I fully understand either option, but if it's of any use to you I'd rather have the mux'ing. I have like 50 movies piled up on my Hughes and can't get anything to sync properly not matter how I mux!

rpongett
09-06-2002, 01:44 PM
Even though I don't have an MPEG-2 editor, I would say #2.

Getting output of a sync'd and widely playable MPEG-2 file that can be edited by other programs if desired would appear to be a higher priority. Especially if you already have a rough-cut GOP level commercial editor to throw out for optional use.

That's my opinion, anyway.

Thanks for all of the effort on this.

jdiner
09-06-2002, 01:57 PM
Ok. Perhaps I was not clear enough.

What can happen here is this.

Option #1 - Keep working on the generation of the mpeg-2 file. In the hopes that at some point it will work to edit it with one of the commercial editors. Thus at some point something could be done with what is coming out of those to burn to disk. What ever type of disk people are using.

Option #2 - Do it myself and in so doing forget about ever being compatible with one of the other editors. Thus the ONLY way to ever edit it at that point will be with the internal one I am building. The benefit here is that things are and stay sync'ed even after the cuts have been made.

--jdiner

artships
09-06-2002, 02:01 PM
1- Keep going with tweaking the mpeg file output until m2-edit and mpeg2vcr will frame/gop edit the output mpeg successfully.


To me, this sounds like you want to create a program stream that any DVD player can play and any DVD authoring program can edit without losing sync, so I vote for this.



2- Keep making the mpeg file play in the most places i.e. software and hardware mpeg-2/dvd players. With our own custom cutting/editing software...


This sounds like a nearly-standard program stream that some DVD players or authoring programs can't handle, so I don't vote for this, despite the nifty advert-zapping editor.

John,
Who realizes, of course, that the main criteria for what you're to work on next should be what sounds like the most fun to you.

rpongett
09-06-2002, 02:09 PM
jdiner:

My mistake. Misread your initial post.

I would still stick with #2, as long as the simple GOP level editor permitted users to easily scan through (like, with a dragable bar) the video to cut where they wanted (i.e., not just sensing all black I-frames and alowing cuts there, but whatever else they wanted to cut, as well).

If its something like, say, the simple chapter marking mechanism in SpruceUp (just a small video screen, dragable bar for position in the video and a chapter mark button) where you could essentially cut "chapters" (segments), that would probably be more than acceptable to most. The downside of being off a few extra frames on either side on cuts with GOP level editing is pretty low.

Then, perhaps at some point later on, you might try to go back and see if you can get tytool to massage the data such that commercial editors could work on it (option #1).

rpongett
09-06-2002, 02:19 PM
artships:

I think that option #2 allows most DVD players to play the sync'd file.

I may be reading this wrong, but I think that jdiner right now is able to produce on his system a mux'd sync'd MPEG-2 that is playable basically everywhere. However, commercial editors like M2-Edit can't properly edit that outuput.

I think he is contemplating whether to spend his time:

1. Keep working to try to (if ever) get the output such that professional frame-level editing programs such as M2-Edit can properly edit that output; or

2. Building into tytool6 (or a separate program) a simple GOP level (not frame level) editor that can edit these files properly (and they can be played anywhwere).

If I'm reading that correctly, my vote is for #2 first. Then try to perfect #1 later.

stealthdave
09-06-2002, 02:31 PM
jdiner,

It sounds like you pretty much have option 2 ready to roll, or nearly so. As I stated before, I'm running a Linux operation, so even a simple editor probably won't work even through WINE, so that part of option 2 probably won't affect me unless your editor is cross-compilable. ;) (I've found that WINE can't run most programs that use windows codecs, or I at least have trouble running them.) Not to fear, though. GOPchop (http://outflux.net/unix/software/GOPchop/) seems to work pretty well on vsplit13 produced mpegs, so I'm set there. (Though I'd love to try your editor. GOPchop can be pretty buggy at times.) Over time I'd definitely like to see option 1 pursued, but I tend to re-encode everything to SVCD anyways, so I vote for option 2.

btw, GOPchop is GPL, so his source is available for download. :)

stealthdave
09-06-2002, 02:37 PM
Originally posted by jdiner
Having said that it means I can remove the latest debug code and re-run the latest testbed and if all is well the next version of vsplit will be ready to roll...

--jdiner

Does this mean that we might see a new version of vsplit soon? Woohoo! I'm really looking forward to trying it out! I've got some Battlestar Gallactica streams that are just dying to be decoded!

- Stealth Dave

laserfan
09-06-2002, 02:39 PM
I strongly vote for #2, provided it works for SA Tivos as well.

If as you said it includes audio sync, and even the most rudimentary "trim-front-and-back" editing that would be wonderful. An "embedded commercial exciser" would be positively fantastic, a step beyond what I might have hoped for in TyTool.

I guess I don't care if "commercial editing tools" recognize the files as "perfectly MPEG2-compliant" or not; I only care, as an SA Tivo owner, that I'm able somehow to transcode the audio to 48kHz before burning to disk. Since this is CURRENTLY easily do-able w/muxed TyTool streams I'm gonna assume it will be possible w/the next gen system you're working on.

Thanks for all your hard work jdiner.

gtschance
09-06-2002, 02:49 PM
Output recordable into DVD/SVCD/VCD that stays in sync would be my vote.

Quick question

Is the ty file exported from a DTivo likely to change in any of your scenarios?

I ask to determine if it is safe to archive them off my DTivo and get back around to splitting and DVD'ing them later.

Great work by the way

KRavEN
09-06-2002, 02:53 PM
Originally posted by rpongett
jdiner:

My mistake. Misread your initial post.

I would still stick with #2, as long as the simple GOP level editor permitted users to easily scan through (like, with a dragable bar) the video to cut where they wanted (i.e., not just sensing all black I-frames and alowing cuts there, but whatever else they wanted to cut, as well).

If its something like, say, the simple chapter marking mechanism in SpruceUp (just a small video screen, dragable bar for position in the video and a chapter mark button) where you could essentially cut "chapters" (segments), that would probably be more than acceptable to most. The downside of being off a few extra frames on either side on cuts with GOP level editing is pretty low.

Then, perhaps at some point later on, you might try to go back and see if you can get tytool to massage the data such that commercial editors could work on it (option #1).


I agree with this completely. This is what I vote for.

FreydNot
09-06-2002, 04:32 PM
Since I have never been able to convert 32k audio to 48k without messing up the sync, I have never been able to burn an in sync DVD from a simple pair of vsplit files.

Since my goal is to burn DVD's from SA files, I will vote for option #1 which I believe will continue to work towards this goal.

Option #2 is a function that we can pretty much do now via a labor intisive manual task. I just did this for several ty files, but it all ended up being for not since I still couldn't get the audio to a DVD compliant frequency. What I did by hand is make up batch files to use vsplit to extract 11 chunks on 100 chunk intervuls. From there I found the chunks that blocked out the commercials and then did it again with 10 chunk resolution between there edit points. Once I got it down to the nearest 10 chunks, I'd do it again with a resolution of 1. Then I'd use vsplit to cut the ty file in program segements down to the nearest chunk.

The proposed #2 option should have better resolution (GOP boundries instead of chunk boundries), but chunks where pretty reasonable for what I was doing.

I should also say that I don't really need a full blown #1 solution. If besweet or some other tool can convert the SA audio from 32k to 48k without loosing sync, that is all I need.

If somone can point me towards a way to get SA streams onto a DVD, then I'll switch my vote to #2.

Hoochster
09-06-2002, 05:04 PM
I personally go for what is easiest for the person programming. As I have mentioned in the past this is all beyond me and I am grateful for anything that anyone shares with the community. So if you already have something that basically fits the task that everyone wants and it works for you and is easier for you and you are willing to share it then by all means that is what works for me! Since I personally can't do no better then I am happy to get what someone was willing to share!

So thanks for your time and efforts on this project and looking forward to whatever you give us!

Hoochster

edpuffmonster
09-06-2002, 08:46 PM
Sorry for posting this here, but jdiner, your mailbox has been full for quite some time now. :)

Rotten
09-06-2002, 09:50 PM
2.

Lets get it on. By The way I would contribute financially to the cause if needed

FreydNot
09-07-2002, 04:30 AM
Originally posted by rc3105
Freydnot: virtual dub, select audio full processing mode, go to conversion (cntrl-a) and set the options


Here is what I tried...

I muxed a m2v and m2a pair using TMPGEnc V2.57.41.146 full mux (not simple) to a mpg file. I tried to load that file into VirtualDub 1.4.10 but I get the error "MPEG Import Filter: pack synchronization"

I tried using DVD2AVI and VFAPIConv to get the file into VirtualDub, and while that did load the file up, it was going down a transcoding road a didn't want to. The only way I could see this working would be to transcode the video+audio into an AVI and then re-encode the avi into mpeg. Not really a solution.

Did I miss something?

I did try to burn a muxed mpg file into a ISO data DVD disk as you described. My Pioneer player didn't like it at all (not much of a suprise).

koreth
09-07-2002, 05:39 AM
The only thing I really wanted to use M2-Edit for was to chop out commercials. If I can do that some other way without major headaches, fine by me. Actually it won't surprise me if you end up with a better UI than M2-Edit -- it was clearly not designed with 60-minute-long clips in mind!

Obviously it'll be nice to have the muxed files in good editable condition at some point, but that seems like a project that could stretch on indefinitely, so best to get something useful out the door earlier rather than later.

I watch my DVDs on a PC so audio sample rates and so forth aren't an issue. (Pretty sure my standalone DVD player will handle different audio rates too.)

Looking forward to burning some DVDs!

Fugg
09-07-2002, 02:54 PM
Originally posted by rc3105
...
if you want to convert the audio seperately, try using the winamp disk writer plug in to convert the mpa to a wav, then any sound converter util to change the sampling rate and or re-compress it
--
Riley

the audio editor in mpeg2vcr will convert the .m2a file to 44.1 or 48khz directly, no need to change formats.

Fugg
09-07-2002, 03:04 PM
Originally posted by rc3105
... my apex plays mpeg files written directly to disk as though they were individual tracks on an audio cd. ...
--
Riley

your apex 1100 will play the files from a cd-r or a dvd-r?

mine will play them as individual files from a cd but not from a dvd. i thought it was because it sensed the media type and read the disk appropriately. (so it thinks)

ie. Looks for individual files on cd and looks for a standard dvd format on dvd’s.

have you changed the stock firmware?

Fugg
09-07-2002, 03:19 PM
...and as for which option, I have to agree with Hoochster,

Bless your heart jdiner, whatever solution would be easier for you. I would think that #2 would be the easiest, but I don't know how far along you are with your editor.

I keep pretty good sync with good extractions on my dtivo, but anytime I try to pull streams from my sa that are over 2 fsids, I loose approximately a half a second of audio at every transition.

Any solution that would keep that from happening without extracting each fsid individually would be most excellent.

jdiner
09-07-2002, 05:02 PM
Originally posted by stealthdave

btw, GOPchop is GPL, so his source is available for download. :)

Don't need it. I don't intend to sound smug, and I can explain in detail if anyone wants, but it is a VERY simple thing to do. Basically, you get Frames (I, B, P, and a 4th that is an add-on that neither type of tivo uses...) These frames are then group'ed in a GOP. With a little bit of tweaking you just get rid of everything between GOP headers you cut and shazam. You are there.

--jdiner

jdiner
09-07-2002, 05:04 PM
Originally posted by gtschance
Output recordable into DVD/SVCD/VCD that stays in sync would be my vote.

Quick question

Is the ty file exported from a DTivo likely to change in any of your scenarios?

I ask to determine if it is safe to archive them off my DTivo and get back around to splitting and DVD'ing them later.


No. I won't be changing things. Tivo might... But I won't be. That is what I have done. I have archived a ton of things in defference to my being able to finish this so that I can make nice archives...

--jdiner

FreydNot
09-07-2002, 06:25 PM
Originally posted by rc3105
if you want to convert the audio separately, try using the winamp disk writer plug in to convert the mpa to a wav, then any sound converter until to change the sampling rate and or re-compress it

I haven't tried using winamp 2.0 for this (winamp 3.0 doesn't seem to have the same diskwriter output option), but I have used goldwave to save the mpa to wav and then use besweet to convert the wave back to 48k mp2. I have also tried using BeSweet to directly convert the 32k mpa to 48k mp2, and 48k ac3.

All of the things I have tired result in an audio file that is out of sync from the original audio file. Is there something magical about winamp's diskwriter plugin?



Originally posted by Fugg
the audio editor in mpeg2vcr will convert the .m2a file to 44.1 or 48khz directly, no need to change formats.

I have used Womble's Mpeg-VCR V3.12 (6/2002) to convert the 32k audio to 48k audio. While the process worked fine and the audio quality was excellent, the resultant audio file was out of sync from the original audio (and the video of course).

Maybe I should start a new thread for this...

dlang
09-07-2002, 08:17 PM
a simple editor is all I expect to need (trimming commercials out of a video or dropping everything except for a really good commerical out of a video :-)

however long term I need to save these things to removeable media (my 100G addon drive in my tivo is full and my 160G video storage drive in my fileserver is already up to 80% full..) it would be really nice if I was able to burn DVDs that can be played in normal players.

the only thing is that I will need to be able to run this code on one of my linux systems as I don't have anything suitable running windows.

scarabus
09-08-2002, 04:21 PM
Originally posted by jdiner
Oh man. Well that explains why I could not fix it. The TyStream file itself on one of the recent bad clips I was sent is actually wrong. This is a first for me. Never seen one where it is simply wrong before.

Bad blocks on the hard drive perhaps?

So what will VSplit say if it encounters a bad stream like this?
Will there be an error message that says "this stream is toast - bin it and try to record it again"?

und420_247
09-08-2002, 04:22 PM
I recorded the new DirecTV Freeview (Bruce Hornsby Live) and attempted to extract it today.

TYTool does NOT show a filesize for the show (it works for all other items in my Now Playing).

It started extracting, and got as far as 3 GB when I stopped it by killing TYTool.

What I got was a 3GB .mpv (which is 27 minutes of video) and a 0 byte audio file.

Is this program strange in some way - I will attempt to record it again and see if this happens again. Can someone else try this?

und420_247

scarabus
09-08-2002, 04:26 PM
Personally I'd prefer Option #1 - I'd really like to be able to import streams into a frame accurate editor and do perfect commercial cuts.

But it seems to me that we're more likely to get #2 sooner and I'll be happy to get "good enough" soon and wait for perfect later on.

Wooly
09-08-2002, 08:21 PM
Me vote #2. Mongo say #2. Mongo LIKE #2, as Mongo hates most editors, and Mongo just wants his MacGyver on DVD ASAFP.



Option #2 - Do it myself and in so doing forget about ever being compatible with one of the other editors. Thus the ONLY way to ever edit it at that point will be with the internal one I am building. The benefit here is that things are and stay sync'ed even after the cuts have been made.

--jdiner [/B]

jdiner
09-08-2002, 11:29 PM
Originally posted by scarabus


Bad blocks on the hard drive perhaps?

So what will VSplit say if it encounters a bad stream like this?
Will there be an error message that says "this stream is toast - bin it and try to record it again"?

I had not thought about the bad block type of thing... I had been thinking software issues. But you never know. I mean a truely loaded down and "freaking out" tivo might be the reason to...

Way back when I started using all of this stuff I have been writing I found that without the priority patch when I went to get the list of playing shows and off and on during the extraction it would smash the display on the screen. I thought it might just be playback but upon testing it back then it was indeed bad on disk. Backing up and playing it again showed the same bad corrupted data.

That type of thing might explain the issue as well. Just "gets behind". But like I said. I have never seen this before. But if it was bad on the screen I have never tried to process it... So I am just guessing here.

Lastly, I will adding a message that is better than "fix it dear henry"... Bottom line. When I have been programming for days and days of 14 hour sessions I get a little... punchy. And for some reason when I was working over that part I found the childrens song "There's a hole in the bucket" running through my head... And you see the result... :)

--jdiner

FreydNot
09-08-2002, 11:32 PM
with what shal we fix it? ...

jdiner
09-08-2002, 11:33 PM
Ok. No need to keep voting. It appears to be pretty solidly on the side of "doing it oursleves" for now.

So my plans are to drop the idea of munging the output further to make it look better and work better going with a commercial cutter/editor.

Instead I will focus on just getting something that will work for most of us as a simple commercial cutter. There are a ton of techniques for this. And to be honest I intend to enlist the aid of some others that are good/better at linux/unix type GUI's and we can get together on producing something that can be used on a variety of platforms. (But keep in mind that editing is just a step in this process and not currently the next one... Although it should be soon... :)

Perhaps in doing all of the above work I will figure/find out enough to get better at producing something that can be used to fix the problems with the output so that something else can be used. You never know...

--jdiner

jdiner
09-08-2002, 11:34 PM
Originally posted by FreydNot
with what shal we fix it? ...

Fix what? The one where the framing in the TyStream file is smashed? In terms of what is being done right now. There is no way to fix it. It would take a different approach. And for 1 stream in 20,000... A different approach is not worth it. At least not to me.

--jdiner

FreydNot
09-08-2002, 11:37 PM
Sorry, that's the next line in the "hole in the bucket" song. Didn't mean to confuse you with it.

Ellipse
09-08-2002, 11:48 PM
Oh good...I just thought it was my odd mind thinking of "Hole in the Bucket" when I saw that comment. :)

laserfan
09-09-2002, 12:52 AM
Originally posted by jdiner
Ok. No need to keep voting. ...keep in mind that editing is just a step in this process and not currently the next one... Although it should be soon....I want to vote once more though--and in favor of the next step being a muxed/in-sync mpg2 program stream. You have said that you can do this and I hope you will release it to us before getting wrapped around the edit step axle?:)

jdiner
09-09-2002, 12:42 PM
Originally posted by FreydNot
Sorry, that's the next line in the "hole in the bucket" song. Didn't mean to confuse you with it.

You are most correct. Sorry. See what happens when there is too much sleep going on? The song stops and things are just not right.

--jdiner

laserfan
09-09-2002, 12:45 PM
Originally posted by rc3105
getting a nice neat PROGRAM stream will require transcoding. (partial at least)...the mpeg coming out of the tivo is a TRANSPORT stream...find a good stream editor or a dvd authoring prog that accepts transport streams as input, problem solvedI can state with conviction that after some googling I still do not understand the difference :confused: . But I thought what jdiner was after was a way to store Tivo programs to DVD, which implies (I believe) a more-or-less "compliant" MPEG2 Program Stream. If instead it means he makes a more-or-less compliant Transport stream that some authoring program will accept & transcode & burn, well, whatever the shortest path is I will accept, even if it means from jdiner "you have to acquire thus-and-such Authoring program (that I use myself) for this to work".

Do I still not understand the difference between options 1 & 2... Ok, then I'll just take what I can get! :D

jdiner
09-09-2002, 12:46 PM
Originally posted by rc3105


getting a nice neat PROGRAM stream will require transcoding. (partial at least)

whether you use dvd2avi or jdiner's app does it, that's the bottom line here

the mpeg coming out of the tivo is a TRANSPORT stream

they're pretty similiar, but it's still apple's and oranges

find a good stream editor or a dvd authoring prog that accepts transport streams as input, problem solved


You are correct. But do keep in mind that it is only "almost" a transport stream. There are mpeg-2 records that are missing that keep it from being a true transport stream. There are also some changes, depending on Tivo type, that make some of what is there incorrect. But it would be easier to go to a transport stream than to a Program stream.

However, you have to define your use of transcoding here. Nothing of what I am doing reencodes even a single frame of the stream. I am just changing how it is "blocked and mux'ed" to make it a program stream. The end result is a playable mpeg-2 program stream file.

--jdiner

kyle
09-09-2002, 09:47 PM
If this is a TRANSPORT stream, the MUXing in MPEG2VCR allows for this--but wants me to speicify a PID. What is this? What values should I use? The defaults are

Video Elementary Stream 34
Audio Elementary Stream 33

Thanks,
-Kyle

jdiner
09-10-2002, 01:42 AM
Yeah. Mostly playable would be pretty accurate. That is the last part that I am working on now. I will get it. Sadly I have been almost there for a very long time. "whee"... :)

--jdiner

laserfan
09-10-2002, 10:21 AM
Originally posted by kyle
...If this is a TRANSPORT stream, the MUXing in MPEG2VCR allows for this--but wants me to speicify a PID...kyle it is my understanding that the MPEG-VCR Transport Stream option is for the Creation of transport streams; the discussion was about "what kind of streams come out of Tivo" and I think the answer is "it's a kind of transport stream w/a few new wrinkles thrown-in". So the challenge before jdiner & others working this is to translate these streams into something that is useable for editing and burnable to a DVD (both of which require that certain rules are met). So I don't think using this option in MPEG-VCR buys you anything, but someone smarter than me here will correct me if I'm wrong!

My simplistic take-away from the "program stream vs. transport stream" discussion is that Transport streams are created such that they can be reliably "broadcast" (whether that means thru-the-air or across a network) given the interruptions that occur in the broadcasting process (whether that be a thunderstorm or an IP traffic problem). So I'm thinking of Transport streams in terms of pieces of information about the video/audio that can suffer interruptions yet be pieced-back together at the final destination (playback) point. Program streams OTOH don't have to fret about the possibility of major interruptions and thus aren't "packetized" like Transport streams. There's probably an overhead/bandwidth associated w/Transport streams that PS don't have to be saddled with/bound by. My story and I'm sticking to it until somebody explains it better/more accurately.

jdiner
09-11-2002, 03:08 AM
Ahh... And the latest testing failures start to make sense finally. I was keying off of some values that used to be constant and now seem to be well... not constant. That makes what I was trying to do harder.

Not sure yet what would cause this. I have not upgraded anything on my tivo from one extraction to the next... Has there been a "stealth" upgrade done by anyone or anything on the DTivo front?

Trying to use another method to detect a good or bad status for the -1x4 failures. (When there is not PES information for either audio or video in an entire chunk...)

The good thing is that in doing some more exhaustive testing it is just "another set". So not random as I had first thought but rather just changed from what it has been for years. (Go figure...)

--jdiner

jdiner
09-11-2002, 11:54 AM
OK. Well. I don't know if something changed or what it truly was. But I can see that there is something going. Good thing I have a large test bed. I ran through a ton of clips checking this out and found a few interesting things. There now appear to be 2 ranges of values used to indicate what it is, video header or body and audio header or body, and what is going on with it...

The old set worked in this range:



VIDEO HEADER RESETS AT > 00 CB 10 --> 0 BB 20 == diff of FF0 (4080)

VIDEO BODY RESETS AT > 31 8e 6c --> 20 50 40


This was the norm for every clip I had older than about 5 months. But then I started grabbing clips from "The Dead Zone" on usa during the marathon. So no more than 2 weeks ago and for almost 2 full weeks 1 episode a day. On the 8 I got thanks to tennis, does anyone really watch that anyway? :), I got 3 with this range scheme and 5 with the new one...



VIDEO HEADER RESETS AT > 01 4f c4 --> 01 3f cc == diff of FF8 (4088)

VIDEO BODY RESETS AT > 6d 7d 9c --> 5c 30 40 == diff of 11 4d 5c (1133916)



So 2 totally different ranges. I don't get it... But these are the only 2 ranges I have seen so far. The rollback point changes by a few bytes in either direction for both header and body but it stays in roughly the same range. So... if that remains the case we can still use this technique the fixing of the -1x4 problems...

--jdiner

newlooper
09-11-2002, 01:06 PM
OK I have the thought planted in my head, let's see if I can get it out.

code:
--------------------------------------------------------------------------------
VIDEO HEADER RESETS AT > 00 CB 10 --> 0 BB 20 == diff of FF0 (4080)

VIDEO BODY RESETS AT > 31 8e 6c --> 20 50 40
--------------------------------------------------------------------------------

If it the video header resets at FF0, could you use FFx to detect a video header reset? That would kind of cover all bases.

The video body would have to be > 11 3E xx though and I am not sure that would meet the criteria.

It seems that there still remains a constant, just a little different than normal.

daveinfla
09-12-2002, 03:29 PM
Running a Phillips DTIVO w/
Extreme 2.5 Upgrade (TivoWeb1.93, noscramble, all the defaults)
TurboNet Card

Added mfswebmodule97 last night and was able to click on and download a single .ty stream. The file is a one hour show at around 1G in size.

I ran Vsplit13.exe filename.ty filename.m2v filename.m2a and after about 30 seconds, and a bunch of messages, vsplit stops and generates a Windows XP application error (you know the one that wants to report to Mickeysoft).

I tryed using Tytool5r2, opening an existing file and got the same results.

Any ideas?

Wes
09-12-2002, 03:34 PM
Did you try to use Tytool5r2 to extract as m2v & m2a directly instead?

daveinfla
09-12-2002, 03:57 PM
Wes,

I tried using Tytools after I had already extracted the .ty file with Tivoweb.

Are you suggesting I NOT use tivoweb to download the file and install TYtools on my Tivo and use the client to extract and split at the same time?

jdiner
09-12-2002, 06:34 PM
Originally posted by newlooper
OK I have the thought planted in my head, let's see if I can get it out.

code:
--------------------------------------------------------------------------------
VIDEO HEADER RESETS AT > 00 CB 10 --> 0 BB 20 == diff of FF0 (4080)

VIDEO BODY RESETS AT > 31 8e 6c --> 20 50 40
--------------------------------------------------------------------------------

If it the video header resets at FF0, could you use FFx to detect a video header reset? That would kind of cover all bases.

The video body would have to be > 11 3E xx though and I am not sure that would meet the criteria.

It seems that there still remains a constant, just a little different than normal.


Not quite. For the header counting it resets at 00 CB 06 through about 00 CB 1F. It fluctuates a bit.

The body counting fluctuates much more wildly. But I have these values through observations. What I don't like is that after having things all be in the same place I have now identified 3 seperate bands of values. This just makes it problematic to work on as these values if they are indeed changing are how I can tell when things line up and no "better" data is available.

--jdiner

jdiner
09-12-2002, 06:36 PM
Originally posted by daveinfla

I ran Vsplit13.exe filename.ty filename.m2v filename.m2a and after about 30 seconds, and a bunch of messages, vsplit stops and generates a Windows XP application error (you know the one that wants to report to Mickeysoft).

I tryed using Tytool5r2, opening an existing file and got the same results.

Any ideas?

Find out where it is crashing in terms of the number of chunks. Use TyFileSplit to start roughly 40 chunks before that point and then cut a 100 chunk portion it and send it to me. I can at least stop the crashing, and maybe figure out what is wrong and help it continue to process correctly.

--jdiner

wh1212
09-13-2002, 12:48 AM
OK I've alway's extracted from my SA,720x480 cbr etc, looks great to me, Have a dsr6000,give it a try.
Completly differant streams,Disney DVD like, Interlaced 48%-67% the rest progressive,sometimes top field first,or not. has the audio always been 4800 ?Mpg2 muxers &editors make a mess(Tmpegenc beta 12a best mux) So send streams to dvd2avi(gordian knot)Inverse Telicide, Perfect Mpeg4 23.976 and insync.
Need more keyframes for editing though,Anyone know of a frame acurate Avi editor,Also second extraction with Tytool5,got a .gl file,not text ? anyone know what it is, or want it.
Happy with streams, seem more crisp and vivid, Have the streams always been like this, or is it a merger thing.

Wes
09-13-2002, 02:45 AM
daveinfla, yea, that's what i mean. tivoweb MAY be the reason of your problem. I'm just guessing though.

tdavis80
09-14-2002, 07:00 PM
I get the app error running tytool 5 if I try to 'get' Coldplay / vines Special. I even recorded it twice. Both versions (1 erase #1 and re-recorded it the next day) cause the app to fail & close.

OS here is XP SP1.



Getting NowShowing data...
Total Size = 5258kbytes in 7.581000 seconds...
Unknown tag <166>
tag data <p 14 22:58:31 tcl[362]: Tcl created pool of 1458176 bytes>

Name = 'c:\tivo\Coldplay / Vines Special - FREEVIEW-'
fsIDs = '1179967/1179969/1179970/1179971/1179972'
Tivo Address = '192.168.1.222'
Connected...
cmd => 'TYSTREAM 1179967/1179969/1179970/1179971/1179972'
Detected Tivo Type: DTivo
Detected Audio Stream Type: MPEG Layer II
Final standardAudioSize = 588
Final standardFrameLength = 576
Final standardAudioDiff = 2160 or 00:00:00.024
First Video PTS: 00:02:35.057

Wes
09-14-2002, 08:01 PM
It could be because the show's name has a "/" in it. I had to rename the "9/11" shows to "9-11" to download mine.

jdiner
09-14-2002, 08:44 PM
No the / character shouldn't be a problem. Lemme grab the full list from the source:



while ((ptr = strstr(fname2, "/"))) ptr[0] = '-'; // Get rid of any /s...
while ((ptr = strstr(fname2, "\\"))) ptr[0] = '-'; // Get rid of any \s...
while ((ptr = strstr(fname2, ":"))) ptr[0] = '-'; // Get rid of any colons...
while ((ptr = strstr(fname2, "*"))) ptr[0] = '-'; // Get rid of any *s...
while ((ptr = strstr(fname2, "?"))) ptr[0] = '-'; // Get rid of any Quesion marks...
while ((ptr = strstr(fname2, "\""))) ptr[0] = '-'; // Get rid of any "s...
while ((ptr = strstr(fname2, "<"))) ptr[0] = '-'; // Get rid of any <s...
while ((ptr = strstr(fname2, ">"))) ptr[0] = '-'; // Get rid of any >s...


That is the list of what I cut out to protect the names of files...

Perhaps it should be a "\/" string to "protect" that character but I don't think that it is special in C/C++.

--jdiner

jdiner
09-15-2002, 02:33 PM
Alright. It looks like VSplit Alpha #14. Is pretty much ready to roll...

It took a lot longer than I thought it would but the chunk alignment matching engine is now extremely solid. And since that is what will fix the temporal aligment issues as well as the missing data do to -1x4 chunks it was a needed thing.

There are a number of sundry other fixes in it that will make it much more stable for some. I still don't see (ever) many of the problems being sent to me from others but hey... them's the breaks.

Right now I have it running over my test bed one last time. There are over 40 1 hour shows in the test bed now so it takes awhile. One of these days...

Anyway, just wanted to let me people know what is going on. I actually want to test and fix one more thing but it is pretty minor so it should not need re-running the test bed. Recently people have been sending me bad clips that contain reseting audio PTS values. Which is wierd. Once again mine never does it, so... Anyway the new alignment code will make it possible to catch these and correctly determine what is going on.

--jdiner

jdiner
09-16-2002, 11:35 AM
Oh man. Corner case after corner case. I am getting really tired of trying to fix this one. When the PTS timestamps reset... I don't have enough test clips so I have to make due with kind of guessing and what types of things can happen. No idea if they ever really do.

Anyone out there have a TyStreams where the PTS values reset in the middle? I do believe it will be a stand-alone issue only.

--jdiner

FreydNot
09-16-2002, 11:55 AM
How can we tell? I assume vsplit (v13) will give some error message?

I know when 3.0 came along, the green bar went away. Could your changes have happened at the same time?

jdiner
09-16-2002, 12:33 PM
Ah that is a very good point. :)

You can tell because all of a sudden the audio and video PTS data will be off. You will start to see OOB messages from VSplit that indicate that every single chunk past a certain point is off.

I have found 2 more bad ones from someone that put it on my staging FTP site. So I have a bit better notion of what can happen now. But do keep in mind that it appears it can happen in either a SATivo or a DTivo stream.

--jdiner

jknapp
09-16-2002, 02:23 PM
Originally posted by jdiner
No the / character shouldn't be a problem. ...--jdiner
Jdiner--I'm having exactly the same problem as tdavis80, but with a different show that has the "/" character. The show I'm trying to extract is yesterday's Smallville, which is titled "Pilot/Metamorphosis". I have downloaded lots of other shows, including one after this problem with no trouble.
You can get around the crash by setting Tytool in Tystream mode instead of VSplit, but then the client just gives an error that it can't open the output file. (Sorry at work, so don't have the exact language in front of me.)
I'm using 5r2 with WinXP SP1 also. I see your code above, but from my admitedly uneducated view it looks like it either isn't working or the "/" is tripping it up somewhere else in the process.

jdiner
09-16-2002, 04:55 PM
I will protect it in the next release. It is easy enough to do for a cost of 1 more byte in size...

--jdiner

jdiner
09-16-2002, 05:17 PM
Ah ha. I begin to see why people have not thought that mux'ing and proper MPEG-2 Program Stream output was the holy grail of all of this.

We have been talking 2 totally different things. And now I am not sure which is better. Or if there is even a better.

I have been planning on making something like the DVD-VCD... In essence a VCD formatted disk that is just 4.7 gig in size and not 640-700meg in size. www.vcdhelp.com has a good explaination of this.

This is why I have always planned and worked towards good output. Then take mux'ing 1 step further to a program stream that includes RIFF headers, a .dat file, and you can just burn the pig.

However in looking over some of the FAQs on TyDVD's and what not it appears that people are not doing this. But are making DVDs that are tricked out to look like real DVDs...

So 1 question... Why? Why go through the headaches of changing a few header values if the other style works? Sync would be maintained etc...

What are the Pro's and Con's that people see that made them choose 1 over the other?

--jdiner

laserfan
09-16-2002, 05:55 PM
Originally posted by jdiner
...Why go through the headaches of changing a few header values if the other style works? Sync would be maintained etc...Well, here's one answer, I DIDN'T KNOW THERE WAS ANY OTHER WAY! i.e. if there is a way to create a "VCD" on a DVD platter that will play in a set-top player then I don't know about it. But I will look at vcdhelp when it's working again.

I think there are some players that will play a DVD that's just had an mpeg2 file copied onto it, but I don't think my player will read such a thing. In fact my player thinks a VCD is an 1150kbps bitrate and 44.1kHz audio etc. So I take the MPEG2 file and prettify it for DVD by running it thru an authoring program.

I must confess to some concern about why such a disconnect exists between what you are doing and what people are expecting. Maybe somewhere in your 500-odd posts there is one that explains exactly what You have been doing to archive your Tivo recordings for both posterity and easy playback?

SirTrini
09-16-2002, 06:06 PM
My hope was that jdiners work was going towards VOB files that are DVD compliant.

chimera
09-16-2002, 06:09 PM
VCDs can not have multichannel audio, such as Dolby Digital. That's a good reason, in my opinion, to use the DVD format instead.

koreth
09-16-2002, 06:09 PM
I want menu screens on the DVDs I make for my TV shows. Nothing fancy, just the names of the episodes so I can quickly choose the one I want. As far as I know, VCDs don't have any notion of menus; the best you could do would be chapter stops, which I'd instead want to use (on some shows) for scene access within a show.

That said, as long as the DVDs work on my equipment, I'm not too concerned if they break on half the DVD players out there. These discs will be more for my use than for loaning out to other people.

The experience I want to end up with is to pop a DVD into the player, get a menu of the three or four episodes of show X, choose the one I want, and hit the chapter advance button a couple times to get to the particular scene I'm interested in. Not seeing any hint of a commercial would be swell too!

Oh, and Chimera's point about multichannel audio is a great one too. Several of the shows I want to archive have Dolby Digital audio.

scarabus
09-16-2002, 06:28 PM
Well my project is to make the perfect Max Headroom DVD collection.

Currently I'm burning two shows on to 1 DVD-R with a menu that has the episode names.

I'm using Maestro for authoring; patching the header using DVDPatcher and creating chapter points to automatically skip the commercial breaks. It's not an optimal solution since you can only do chapter breaks at the start of a GOP, but it has the benefit that the MPEG is never transcoded.

Transcoding takes a significant amount of time and you're always going to lose some quality, increase the file size, or both.

The perfect solution for me would be to be able to do frame-accurate editing to remove the commercial breaks. Obviously the GOPs where the cuts appear will need to be re-encoded but the rest of the streams should be untouched. I'd then patch the header and import to Maestro as before.

I've tried doing this by using TMPGenc to mux the existing streams and editing with m2Edit or Womble.

For one thing I always get an error when muxing the streams using TMPGenc; this is probably because the length of the video and audio streams never match up accurately. And then the other issue is that sync gets lost during the editing; sometimes the edits go completely awry.

Kythorn
09-16-2002, 06:30 PM
Edit: Axed the entirety of the original post. Rambled way too much, and never got the point across.

Here's the short version instead:

I will personally be content when tytool produces synced mpeg2 files. I can slap these on a dvd and play them back on my hardware no problem.

Letting me select three episodes to turn into a ready to burn DVD-SVCD image would be even better yet.

Allowing frame accurate editing of commercials would force me to name all of my children jdiner.

Identifying and suggesting likely commercial break points would force me to renounce all my worldly goods (aside from my dtivo and a pc to run tytool on) and become the high priest of the newly founded church of our lord jdiner.


PS - Any suggestions on the best way to store my tystreams until at least synced mpegs are possible? I have approximately 40 episodes of summer shows sitting on my tivo that I haven't transcoded - and the fall season's already started. I have a 240 gig tivo, and a gigantic amount of accessible network storage. I'm not afraid of running out of space, but I am very afraid of my files becoming corrupt while I try to wait for a better solution. Running RAID on the NAS is not an option at this time.

My current thought is to extract a copy of everything I haven't encoded yet and store md5sums to make sure bitrot doesn't set in. Should I wait to do this? I became confused on whether or not TyTool is currently discarding a header that may be valuable later on.

scarabus
09-16-2002, 06:32 PM
Originally posted by koreth
I want menu screens on the DVDs I make for my TV shows. Nothing fancy, just the names of the episodes so I can quickly choose the one I want. As far as I know, VCDs don't have any notion of menus; the best you could do would be chapter stops, which I'd instead want to use (on some shows) for scene access within a show.


You can have menus on VCDs. I've done that using Nero. What you don't have is the complex authoring that DVD allows.

stealthdave
09-16-2002, 08:49 PM
It may not be as good as DVD, but VCD and SVCD authoring can do some pretty complicated things if you take the time to do them. You can make motion menus, multiple layers of menus, transitions between menus, chapters, branching programs, etc., all on standard VCD/SVCD. You can even store computer files on a VCD/SVCD that are accessible via CD-ROM.

What you can't do is have interactive menus (menu selection is pressing numbers 0-9, FF, RW, or ENTER/PLAY; no up/down highlighted menus here), multiple audio tracks (you *may* actually be able to do this, I'm not sure), and many other whiz-bang features that I'm probably forgetting.

The biggest drawbacks are video quality and space. DVD has much higher quality and capacity than either SVCD or VCD standards. However, we're talking about non-standard VCDs/SVCDs burned onto a DVD-R/-RW/+R/+RW, so capacity and quality aren't the issue. Compatibility is. That's where your biggest drawback is going to be.

Two places to learn about making VCDs and SVCDs (standard or otherwise) are the aforementioned http://www.vcdhelp.com and http://www.vcdimager.org. These are also good places to look for information on DVD authoring. The latter is the home page for a free (GPL'd) VCD/SVCD creation software that works on Windows, Linux, and Mac. You'll need additional software to do the actual burning.

Of course, if you're going to all the trouble of making such a complicated VCD to burn onto a DVD-R, you might as well make a DVD that will work on most commercial players. Even if you just want the programs archived for yourself, compatibility helps when your player that plays every non-standard variation of VCD barfs and goes to that big electronic junkyard in the sky.

And that's my long-winded take on VCDs on DVD. Apparently, brevity is not in my nature. ;)

- Stealth Dave

Attack
09-17-2002, 02:36 AM
Originally posted by Kythorn
PS - Any suggestions on the best way to store my tystreams until at least synced mpegs are possible?

My current thought is to extract a copy of everything I haven't encoded yet and store md5sums to make sure bitrot doesn't set in. Should I wait to do this? I became confused on whether or not TyTool is currently discarding a header that may be valuable later on.


If you want to make sure the files are perfect you need to extract them with mfs_export read this thread to understand why. (http://www.dealdatabase.com/forum/showthread.php?s=&threadid=16238)
then I would use a WinRar 3.0 to make smaller set of files split at about 100 megs each. This will allow you to use FSRaid (http://www.fluidstudios.com/fsraid.html) to make parity files of the rar. Now you get to decide how much data could be corrupt before losing all your data.


The other way you could do this is take out your current HD's and buy 2 of the exactly same drives. Then use the old dd backup option on each drive, and put one set of the drives away for later extraction.

Fugg
09-17-2002, 10:02 AM
All of this final disk-proper-files generation is fine, but...

I have always thought that mux'ing and proper MPEG-2 Program Stream output was the holy grail of all of this.

I'm not looking for vob's, vcd's, dvd's.... at least not yet.

I want to walk before I run.

Can I vote for just a good muxed, playable, synced mpegs?

...to be able to get a good synced mpeg out of a multiple fsid high rate capture?
...something that does not loose 1/4 to 1/2 second of audio at every fsid transistion?

Wooly
09-17-2002, 10:12 AM
I think I'd have to send Jdiner a couple hundred dollars as a thank-you if he could get it to this final point (although I'd be completely happy to just have it muxing and editing in a way that I can create SVCD-DVD discs).

As for storing .TY files, I'm in your boat also - I currently have roughly 700 gigs of raw .TY files I've stored on a 1.4 TB server (taking advantage of good prices on Maxtor 160 gig drives). I simply store them on the RAID, and I randomly test them every now and then - no problems with any of them.

In case you're wondering, I have EVERY episode of Babylon 5, MacGyver, and Farscape all grabbed from my DTV, waiting...patiently...for the holy grail from Jdiner!


Originally posted by Kythorn

Identifying and suggesting likely commercial break points would force me to renounce all my worldly goods (aside from my dtivo and a pc to run tytool on) and become the high priest of the newly founded church of our lord jdiner.


PS - Any suggestions on the best way to store my tystreams until at least synced mpegs are possible? I have approximately 40 episodes of summer shows sitting on my tivo that I haven't transcoded - and the fall season's already started. I have a 240 gig tivo, and a gigantic amount of accessible network storage. I'm not afraid of running out of space, but I am very afraid of my files becoming corrupt while I try to wait for a better solution. Running RAID on the NAS is not an option at this time.

My current thought is to extract a copy of everything I haven't encoded yet and store md5sums to make sure bitrot doesn't set in. Should I wait to do this? I became confused on whether or not TyTool is currently discarding a header that may be valuable later on.

Kythorn
09-17-2002, 10:29 AM
A couple of things:

1) I'd be very very happy with a simple sync'd, muxed mpeg2 file. I'll probably start converting my collection ASAP when we hit this stage. I have a small footprint ultra quiet high quality AV output PC in my stereo cabinet that I traditionally have used for DivX/XviD playback.

I plan to simply convert to mpegs, burn 2-3 to a DVD and pop them in and play as the whim strikes me. I can handle fast forwarding through the commercials with a remote the same way I do on the tivo now. I no longer possess the amount of time required to strip out all the commercials and transcode to XviD.

I will be very content when we hit this stage. Really, for me, anything past this point is just pure bonus, as I compulsively buy all the DVDs of the shows I watch when they come out anyway. (Including importing two seasons of dark angel from japan.... ouch, my poor wallet!)

2) This question is specifically for jdiner:

I've scoured the vcdhelp.com site, and I can't figure out what exactly you mean by VCD-DVD. Tystreams are the proper res for an SVCD, but not a VCD.

I found this article http://www.vcdhelp.com/svcddvdr.htm , but it actually appears to do actually be what others in the forum are already doing, changing the initial header to fool SpruceUP or Maestro, and then burning the non compliant 480x480 DVD disc.

Are you referring to something else completely? I'm honestly not aware of any other method besides this for using vsplit output on a DVD short of transcoding.

scarabus
09-17-2002, 12:33 PM
Originally posted by Kythorn
I can handle fast forwarding through the commercials with a remote the same way I do on the tivo now. I no longer possess the amount of time required to strip out all the commercials and transcode to XviD.

I've scoured the vcdhelp.com site, and I can't figure out what exactly you mean by VCD-DVD. Tystreams are the proper res for an SVCD, but not a VCD.


The technique of creating chapter points before and after commercials then authoring the DVD so that those chapters are ommitted from playback is fairly trivial and does not add much to the overall time.

From what JDiner said in his other post my take is that instead of using a DVD authoring program to create VOBs he intended to create a homebrew app that formats the MPEG as a SVCD .dat file.

This should work fine in a PC. However I doubt that would play in any set-top players (well Okay, maybe in an APEX) for exactly the same reason that they can't play DVD formats on CD media.

Since I think that the holy grail for many of us is something that plays in a set-top player - be it VCD/SVCD/DVD or whatever - that doesn't interest me.

wh1212
09-17-2002, 12:58 PM
Since vsplit is a Dvd-rip tool, and part of tytool, would'nt it be best
to rip tystream to a .vob. ?

Kythorn
09-17-2002, 01:41 PM
Originally posted by scarabus

The technique of creating chapter points before and after commercials then authoring the DVD so that those chapters are ommitted from playback is fairly trivial and does not add much to the overall time.


I've done this. I probably would have done it a lot more if it wasn't for the $#@$# GOP temporal errors.

As I said, I'm flexible. I have the hardware to author/play just about any format I need. I'll take all the other neat stuff under discussion if I can get it, but I'll be satisfied as soon as we hit the point of synced mpeg2 program streams as a direct output of tytool/vsplit.

Fugg
09-17-2002, 03:04 PM
Originally posted by wh1212
Since vsplit is a Dvd-rip tool, and part of tytool, would'nt it be best
to rip tystream to a .vob. ?

THUNK!!!

the sound of Fugg falling out of his chair and hitting the floor

jdiner
09-17-2002, 05:06 PM
Originally posted by Wooly
In case you're wondering, I have EVERY episode of Babylon 5, MacGyver, and Farscape all grabbed from my DTV, waiting...patiently...for the holy grail from Jdiner!


Forget the money. I will just send you disks or money for disks and you can make me a full set of MacGyver episodes. One of my favorite shows as a kid. :)

--jdiner

jdiner
09-17-2002, 05:24 PM
Originally posted by rc3105

so fas as I know I'm the only person with a dvd drive in my tivo, and vsplit on the pc's got nothing to do with that....


Ummm. Just out of curiosity... Why? Why a DVD drive in a tivo? What do you use it for?

--jdiner

KRavEN
09-17-2002, 05:40 PM
He probably has a way to mount and play the raw tystreams directly from a DVD-R, at least that would be my guess.

Be really usefull if there were a way to mount a network drive and play the raw tystreams from there. Probably is.....

gosquad
09-17-2002, 06:26 PM
Well I'm with you, jd! What i'd do for a 100mbit link to this guy's dtivo! :) Now if he had all the Quincy eps, too! haha (a personal vice of mine) :)

-gosquad


Originally posted by jdiner


Forget the money. I will just send you disks or money for disks and you can make me a full set of MacGyver episodes. One of my favorite shows as a kid. :)

--jdiner

Wooly
09-17-2002, 07:59 PM
Re: MacGyver episodes:

Forget you sending me disks or money - I'd be insulted. You'll have the full series, on DVD-R's, in DVD Jackets and shipped all at my expense, as soon as we get to a point where I can start ripping these things off. It's the LEAST I can do for your providing a moron like me with the ability to enjoy a hobby as challenging as this.


Originally posted by jdiner


Forget the money. I will just send you disks or money for disks and you can make me a full set of MacGyver episodes. One of my favorite shows as a kid. :)

--jdiner

scarabus
09-17-2002, 08:14 PM
Originally posted by Wooly
In case you're wondering, I have EVERY episode of Babylon 5, MacGyver, and Farscape all grabbed from my DTV, waiting...patiently...for the holy grail from Jdiner!


Season 1 of Babylon 5 is due out on DVD next month - that should free up about 40Gb and save you a fair amount of time.

Already got my copy on order...

hancocks
09-17-2002, 09:56 PM
Jdiner,

My $0.02 --

- Shows extractable as one-stop-shopping as VOBs, either for instant burning to DVD-r or easy commercial removal followed by burning to DVD-R, or maybe DVD-RW if they work on my set-top box(hey, nice two- or four-show menu, that's fine...nothing fancy). Put 'em in your 800-DVD juke, and life's looking good (no, I don't own that one yet).

DVD-Rs are running about a buck, and are headed for CDR-type prices. So, DVD quality (or thereabouts) for 2-hour storage at $0.07/hr. when the prices bottom out. Right now, a little over $0.50/hr. What's not to like?

- Stu

jdiner
09-18-2002, 01:56 AM
Originally posted by rc3105
*jdiner, what type of standalone player do you have?

I have an APEX ad-600a. One of the old original ones... But let's just say it seen certain upgrades over the years. :)

--jdiner

captain_video
09-18-2002, 08:26 AM
I've been storing tystreams as well as split and processed (i.e. patched & imported in SpruceUp with PRV files) on DVDs as DVD isos. I occasionally miss an episode of shows that I'm archiving and I like to keep them in a specific order when I burn them to DVD. I use episode guides culled from the net as a checklist to see which ones I already have and which ones I'm missing. If I miss an episode that is intended to be included on a DVD with other episodes, I copy the episodes I already have to DVD-RW and wait until the missing episode is rebroadcast. When it is, I can simply copy the episode back to my hard drive for authoring to DVD. It seems that storing the raw tystreams works better than the processed files. I've had problems importing files back into SpruceUp and ended up repatching and processing them again. Its probably just a matter of changing permissions on some of the files. I don't think the PRV files are being recognized after being restored to the PC.

Note that the permissions have to be changed because once you burn a file to DVD it automatically becomes read-only. Just right-click and change the properties and you're good to go. I've had to do this a lot with Seinfeld episodes lately because they've been broadcasting them all out of sequence for some reason. They used to broadcast them in numerical order by episode number but now they seem to be jumping all over the place. My PC hard drive is limited to about 50GB of working space for extracting and processing tystreams. When the drive starts filling up and I see that there are holes in the continuity of episodes I have to start off-loading the tystreams to DVD until needed.

jdiner
09-18-2002, 12:06 PM
Ok. It has taken a lot longer than I thought it due to some majorly screwy logic in things. At least from the outside it is rather insane...

I have the new byte counting mechanism ready to roll for the VSplit program. It is a much better way of determining if things are as they should be in terms of always having the right data and not getting out of sync or using bad chunks. It still needs a bit more testing by me but I wanted to give everyone a heads up...

Once I get it posted I need people to run it. Right now the action that takes place is based on the old code. And the new code is just informational. It will print messages about what should be done based on both sets of code. What I need most is for people to run it and let me know if they ever report differences.

I.e. the BC code says it is good, and the PTS code reports it as bad and vice versa. If they are the same then it is all good and there is nothing to be concerned about.

For those with the time and inclination to run it I recommend using the -n option. This will not write anything out. With less disk access things just run a bit faster...

--jdiner

koreth
09-18-2002, 12:35 PM
Looking forward to trying it out! I have a bunch of streams from Showtime ready to try (all Dolby Digital audio).

rpongett
09-18-2002, 01:56 PM
Great.

I have a lot of items of varying lengths (and probably video quality) on my TIVO, including Pay-Per-View programming.

I'll run it all whenever you post it.

Hoochster
09-18-2002, 03:51 PM
Thanks for your time on this JDLINER, I have lots of movies to dub off once this is ready so I should be able to give it a good run.

Hoochster

Pr.Sinister
09-19-2002, 09:44 AM
Well, i tried something this morning before going
to work and i got some interesting results.

1st i have to set this up with a bit of background...

Before i got my TurboNet, archiving to VCD/SVCD
was done by capturing the program from S-VIDEO
with CyberLink PowerVCR II v3.0, Editing out commercials
at blazing speeds with M2-Edit Pro v4.0 and burning
VCD/SVCDs with fancy menus with Nero v5.x.x.

I never had a problem with A/V Sync...

Then i bought the TurboNet, started extracting, installed
ALL KINDS OF CODECS AND MUXERS , and a bunch
of editing software. All this to try and make the process as
easy as it was with Video Capturing.

Well last week i recorded a show on my Canadian Dish/PVR
combo and wanted to archive it to CD. So obviously i have
no network card in that one so i go back to VidCap. All is well
with the capture but to edit out commercials, i have a problem!

None of the smaller clips where in sync!!! i tried M2-Edit,
MPEG2-VCR, Ulead VideoStudio, CyberLink PowerDirector,
etc...

Now why would this no longer work when it was always
perfect before? What's changed? ALL THE NEW CODECS I INSTALLED!!!

So to test my theory, i installed a fresh copy of WinXP on a
different partition, installed CyberLink PowerDVD, PowerVCR II,
and MPEG2-VCR. THAT'S ALL.

I then loaded a TMPGenc SVCD Muxed MPEG2 of the season
premiere of The Real World : Las Vegas (It was awesome BTW!)
and tried to make smaller clips.

THE CLIPS WERE IN SYNC!

Now it just might be a fluke but when i get home tonight,
i plan on extending my test by taking the same MPEG
and cutting it with the same software but on the WinXP
install that has all the junk installed. I will alternate back
and forth so i can definately see what's up with that.

I will let you know what i figure out tomorrow.

jdiner
09-19-2002, 01:11 PM
I posted this over in the TyTool thread. But then realized that it is far more of a VSplit core function than the GUI it is wrapped. So I put it here as well for comments from the rest of you.

---------------------------------------------------------------

I just had the coolest idea! :)

Every now and again we run into holes in the extracted TyStream files. These are problematic because they throw off the data. Even if we get back to the right stream we are still missing 24-32 ms of data...

I have been toying this morning with the idea of adding into the output file enough empty audio packets, of the right type, to fill in the hole. So for 24-32ms there would be 0 sound but things would stay lined up.

Now that would be cool. This could also be extrapolated, I believe at the moment, to "save" the sync from the very start. In essence if we are off by 7ms then generate a chunk that is set to play 7ms in length. But that plays nothing. Thuse the output files would be truly "sync'ed perfectly" from the very start. This might or might not work.

The reason I say that is that I am not sure yet if I can make it play for a smaller segment of time. Or if it has to be the fill 24ms or 32ms that I see from the SATivo and the DTivo.

But I just had the thought that we could go the other way with almost as much ease. Add a GOP to the start of the video segment that had enough frames (at the standare 29.97 the tivo uses) to get to that same "perfect" sync point. You would have a black lead-in on the video but it would mean the sync would dead on for the DVD-burning people amoung us.

--jdiner

gosquad
09-19-2002, 01:26 PM
Now that's a "Why didn't I think of that?" post! :)

gosquad

laserfan
09-19-2002, 01:45 PM
Originally posted by Pr.Sinister
...Now why would this no longer work when it was always
perfect before? What's changed? ALL THE NEW CODECS I INSTALLED!!!I remember somewhere around here someone expressed a concern about the possibility of conflicting codecs. I also asked a question about "how you find them" and it seems trickier that just looking in the Multimedia Control Panel (what about WMP filters for example).

Fer sher I have downloaded junk that I later removed and am always concerned that residual cr*p gets left behind. If you or anyone else can offer pointers on cleaning-up codecs beyond the obvious of re-formatting/reinstalling from scratch, they would be most welcome.

keith721
09-19-2002, 03:58 PM
can't recall who said it where, but seem to remember something about the Elecard/Moonlight MPEG codec causing problems with TMPGenC and synchronization...

"I'm not losing my memory -- I've just got more to remember, now that I'm older."
:D :rolleyes: ;)

jdiner
09-19-2002, 08:10 PM
Alright folks here it is. A major release for the first time in a very long time.

This is VSplit #13 test a (VSplit13a.exe) for the Windows PC only at this point. I apologize to everyone that is using Linux and what not at this point but relax. This is just a test phase to try out a some new code en mass. Soon to come will be a true release of VSplit #14.

This release includes a boat load of fixes. This should solve a great many problems. Namely:

1- The -1x4 stuff should be working correctly now.
2- A lot of timestamp issues have been fixed.
3- A new byte-count method of checking things has been added.

However havine said that what I need looked at is the new byte-count method code. Primarily what I am looking for is differences. Something the old code found that the new one does not and vice versa. So read over what is below and if you find something during testing that does not line up let me know...

This new code runs before the standard existing checks on things via the PTS values. The output from this code for clarity sake is shown with 5 repeating numbers:

11111 == BAD! There is NO audio or video data in the chunk. his would normally stop things but for now processing continues...

33333 == The video data does not line up. These are rare... Processing does not stop here.

55555 ==The audio data does not line up. These are not so rare. In a DTivo stream you will see a few. In an SATivo stream you will see tons. Processing does not stop here.

77777 == Processing DOES stop here. Or at least it will. For right now it continues on.

Now a set of examples:

1- This is the classic data is out-of-bounds... First you see the new bytecount code... And you see the OOB packet pretty much as it used to be. The key here is that it ends in a 77777 flag. So the new code things it is OOB and in looking at what follows so does the old code...



33333 -> old Vid ByteCount = 28 E1 50 to 2A BC 60
33333 -> new Vid ByteCount = 5C 45 48 to 5E 2B A4
33333 -> NOT ALIGNED on the Video... It is an OOB chunk!
55555 -> old Aud ByteCount = 33 C2 E8 to 33 E3 B8
55555 -> new Aud ByteCount = 6F C9 B8 to 6F E0 90
55555 -> NOT ALIGNED on the Audio... It is an OOB chunk!

77777 -> Yep. We are out of here! Neither lined up!

Found an OOB packet... The Video Diff is: 00:14:20.844
getBoth: old Vid ByteCount = 28 E1 50 to 2A BC 60
-1x4: We are NOT aligned on the Video... It is an OOB chunk!
We got one that is just to far away!
getBoth: old Vid ByteCount = 28 E1 50 to 2A BC 60
getBoth: old Aud ByteCount = 33 C2 E8 to 33 E3 B8
Found an OOB packet... The Audio Diff is: 00:14:20.522
Is it in sequence??? It is OFF by exactly 35855.083333 frames.
Nope... Not in sequence... Skipping it...


2- Somthing that doesn't line up perfectly but is close enough... Again, first the new lines and then the same old ones. Since these are not different, I.e. they both find it to be good data. This is nothing to worry about.



33333 -> old Vid ByteCount = 2A E1 80 to 2C C5 84
33333 -> new Vid ByteCount = 2C C5 88 to 2E A4 7C
33333 -> NOT ALIGNED on the Video... It is an OOB chunk!

Found an OOB packet... The Audio Diff is: 00:00:00.024
It is in sequence... Everything is fine...


3- There is no audio or video data in the chunk... Again, first the new lines and then the same old ones. We start with the 11111 flag so it should stop here. But for now we fall through. Then the we get the -1x4 older code (with some new lines) that show that it is an out of sequence chunk. They are the same so no worries.



11111 -> Chunk contains neither audio nor video data. Stop here...

33333 -> There is no video data to compare...
55555 -> There is no audio data to compare...

-1x4 - No Audio or Video timestamps in this chunk... Checking byte counts...
-1x4: New Vid ByteCount = 0 0 0 to 0 0 0
-1x4: New Aud ByteCount = 0 0 0 to 0 0 0
-1x4: Old Vid ByteCount = 6A 26 58 to 6B F7 A8
-1x4: Old Aud ByteCount = 6D D8 4C to 6D FF B4
-1x4 - The chunk contains neither audio nor video data... It is an OOB chunk!
Nope... Not in sequence... Skipping it...


That's pretty much it.

--jdiner

Kythorn
09-19-2002, 08:41 PM
Got 26 semi random running in -n mode.

I'll get back to you with results.

Just a note: I've already seen a 77777 in a file that I know vsplit13 didn't die on. Is this signifigant?

koreth
09-19-2002, 08:54 PM
I've run the code against five .ty files so far, all DD5.1 shows from Showtime. It looks like all the errors are flagged by both the old and new code, except for one instance:

First Video PTS: 00:02:11.065
......... 100....55555 -> There is no audio data to compare...
..... 200......... 300......... 400......... 500

Not sure if that's expected or not but it's the only discrepancy I see.

I do see several 77777 errors per file -- does that mean these files won't be muxable since vsplit will stop when it hits them?

jdiner
09-19-2002, 09:12 PM
Originally posted by Kythorn
Got 26 semi random running in -n mode.

I'll get back to you with results.

Just a note: I've already seen a 77777 in a file that I know vsplit13 didn't die on. Is this signifigant?

No. Sorry I must not have been quite clear enough. It does not terminate the file. It marks a single chunk that had a truly out-of-bound status.

--jdiner

jdiner
09-19-2002, 09:14 PM
Originally posted by koreth
I've run the code against five .ty files so far, all DD5.1 shows from Showtime. It looks like all the errors are flagged by both the old and new code, except for one instance:

First Video PTS: 00:02:11.065
......... 100....55555 -> There is no audio data to compare...
..... 200......... 300......... 400......... 500

Not sure if that's expected or not but it's the only discrepancy I see.

I do see several 77777 errors per file -- does that mean these files won't be muxable since vsplit will stop when it hits them?

No. Check the last message from me. The 77777 should just match up with a "not in sequence" message from the old code. If you have a 77777 without the other then it is significant.

A single 55555 line is nothing worry about. It is actually good news since it means that I am now more accurately trapping things.

--jdiner

koreth
09-19-2002, 09:22 PM
For the record, the 77777 errors I've seen do match up with errors from the old code so it looks like all is well.

I've now checked all the files I most urgently want to archive to DVD and they all look good -- aside from that 55555 message the old and new code flag all the same errors.

jdiner
09-19-2002, 09:27 PM
One more request for peoples. For those that had TyStream files that had termporal fixup errors, or whatever they were exactly with Spruce UP, please try them with this new version. By and large they should be fixed.

--jdiner

Kythorn
09-19-2002, 10:01 PM
Here's the most amusing log output I've seen in a while.

Note: this spans 3 FSIDs, but is reported as 0m in TyTool.

I've watched this entire episode in the past, and just went through the entire thing again at the lowest level of fast forward, and it's there, beginning to end.

Damn, the tivo's good at error correction.

Kythorn
09-19-2002, 10:02 PM
Ok, so I guess I can't actually attach that without compressing it first. Here's attempt two.

Kythorn
09-19-2002, 10:45 PM
Ran through 26 tystreams. Aside from the 1 fun log (which is really beyond hope I guess) I posted, all looked fairly well. In every case I saw, the old and new code matched up, with the exception of some of the 55555's already mentioned.

I did catch a few
### The Vid Body is off! 20 50 50 -> 31 B4 50 diff == 0xFFEE9C00 (-1139712)'s

hanging out by themselves, I don't recall seeing these before.

stealthdave
09-19-2002, 11:06 PM
Got me an error! Here's the log output:


Processing 'bsg_ll2.ty': (10 chunks per tick)
Detected Tivo Type: Standalone
Detected Audio Stream Type: MPEG Layer II
Final standardAudioSize = 880

Final standardFrameLength = 864

Final standardAudioDiff = 3240 or 00:00:00.036
First Video PTS: 01:14:46.801
......... 100......... 200......... 300......... 400......... 500
......... 600......... 700......... 800......... 900......... 1000
......... 1100......... 1200......... 1300.### The Vid Body is off! 20 56 40 -> 25 7B DC diff == 0xFFFADA64 (-337308)
$$$ The Aud Body is off! 2D 1C 40 -> 2D 84 E0 diff == 0xFFFF9760 (-26784)

Found an OOB packet... The Video Diff is: 01:22:53.458
getBoth: old Vid ByteCount = 23 59 78 to 25 31 C4
-1x4: We are aligned on the Video so it is good!!!
Found an OOB packet... The Audio Diff is: 00:00:00.036
It is in sequence... Everything is fine...

Found an OOB packet... The Video Diff is: 01:22:53.124
getBoth: old Vid ByteCount = 25 31 C4 to 21 DF B8
-1x4: We are aligned on the Video so it is good!!!
We got one that is just to far away!
getBoth: old Vid ByteCount = 25 31 C4 to 21 DF B8
getBoth: old Aud ByteCount = 2D 69 E0 to 2D 26 60
Found an OOB packet... The Audio Diff is: 01:22:53.400
Is it in sequence??? It is OFF by exactly 138150.000000 frames.
Nope... Not in sequence... Skipping it...


33333 -> old Vid ByteCount = 25 31 C4 to 21 DF B8
33333 -> new Vid ByteCount = 23 BB EC to 25 93 F0
33333 -> NOT ALIGNED on the Video... It is an OOB chunk!
55555 -> old Aud ByteCount = 2D 69 E0 to 2D 26 60
55555 -> new Aud ByteCount = 2D 44 C0 to 2D 66 54
55555 -> NOT ALIGNED on the Audio... It is an OOB chunk!

77777 -> Yep. We are out of here! Neither lined up!


Found an OOB packet... The Video Diff is: 01:22:52.790
getBoth: old Vid ByteCount = 25 31 C4 to 21 DF B8
-1x4: We are NOT aligned on the Video... It is an OOB chunk!
We got one that is just to far away!
getBoth: old Vid ByteCount = 25 31 C4 to 21 DF B8
getBoth: old Aud ByteCount = 2D 69 E0 to 2D 26 60
Found an OOB packet... The Audio Diff is: 01:22:53.076
Is it in sequence??? It is OFF by exactly 138141.000000 frames.
Nope... Not in sequence... Skipping it...


33333 -> old Vid ByteCount = 25 31 C4 to 21 DF B8
33333 -> new Vid ByteCount = 25 93 F0 to 27 5D 8C
33333 -> NOT ALIGNED on the Video... It is an OOB chunk!
55555 -> old Aud ByteCount = 2D 69 E0 to 2D 26 60
55555 -> new Aud ByteCount = 2D 66 54 to 2D 95 C0
55555 -> NOT ALIGNED on the Audio... It is an OOB chunk!

77777 -> Yep. We are out of here! Neither lined up!


etc. btw, I'm using Linux/Wine to run the test. :) I had sent you a sample of this stream a while back (it failed with vsplit13 as well), but you may not have it around anymore. I had used dd to split it before. I can determine the chunk where the problem starts. How much data do you want, what do you want me to use to split it with (is dd fine, or should I use your tysplit tool?), and where should I send it? This problem occurs with about half of my Battlestar Galactica streams, so I can get multiple samples if you need them.

- Stealth Dave

FreydNot
09-19-2002, 11:18 PM
Here is the only one that was truly different for me. I'm attaching the whole thing zipped. The funny part is the line that starts ###.

Also, I'm seeing a lot of these "55555 -> There is no audio data to compare". I assume they are normal?



......... 100......... 200.........

33333 -> old Vid ByteCount = 2E 33 94 to 30 F F0
33333 -> new Vid ByteCount = 23 44 10 to 25 20 9C
33333 -> NOT ALIGNED on the Video... It is an OOB chunk!
Found an OOB packet... The Audio Diff is: 00:00:00.036
It is in sequence... Everything is fine...
300......... 400......... 500
......... 600......... 700......... 800......... 900......... 1000
......... 1100......... 1200......... 1300.......55555 -> There is no audio data to compare...
.. 1400......... 1500
......... 1600......... 1700......... 1800......... 1900......... 2000
......... 2100......... 2200......... 2300......... 2400......... 2500
......... 2600......... 2700......... 2800..55555 -> There is no audio data to compare...
....... 290055555 -> There is no audio data to compare...
......... 3000
......... 3100....55555 -> There is no audio data to compare...
..... 3200........55555 -> There is no audio data to compare...
. 3300......... 3400......... 3500
......... 3600......... 3700......... 3800......... 3900......... 4000
......... 4100......... 4200......... 4300......... 4400......... 4500
......... 4600......... 470055555 -> There is no audio data to compare...
....55555 -> There is no audio data to compare...
..... 4800.....### The Vid Body is off! 23 44 18 -> 30 F F8 diff == 0xFFF33420 (-838624)
.... 4900......... 5000
......... 5100......... 5200......... 5300......... 5400......... 5500
....55555 -> There is no audio data to compare...
..... 5600......... 5700......... 5800......... 5900......... 6000
......... 6100......... 6200......... 6300......... 6400......... 6500
......... 6600......... 6700......... 6800....55555 -> There is no audio data to compare...
..... 6900......... 7000
55555 -> There is no audio data to compare...
......... 7100......55555 -> There is no audio data to compare...
... 7200.55555 -> There is no audio data to compare...
........ 7300......... 7400......... 7500
......... 7600......... 7700......... 7800......... 7900......... 8000
......... 8100.........

jdiner
09-20-2002, 02:12 AM
Originally posted by Kythorn
Ok, so I guess I can't actually attach that without compressing it first. Here's attempt two.

Wow. That disintegrated nicely. I would like a copy of that one. Actually just the piece where it is on and then gets off so badly.

Use the TyFileSplit program to grab just that section. Please start 40 or so chunks before it fails, and then go 100-150 chunks in length from that point. That will great a file of roughly 20-30 meg in size. Hopefully you have a decent network link and can upload it to me.

--jdiner

jdiner
09-20-2002, 02:15 AM
Originally posted by Kythorn

I did catch a few
### The Vid Body is off! 20 50 50 -> 31 B4 50 diff == 0xFFEE9C00 (-1139712)'s

hanging out by themselves, I don't recall seeing these before.

Darn. I did not mention these because I was hoping that no one would see them. It means that my "perfect" byte count code isn't. These never show up for me any more but it did for you here. I would like a clip of this one as well. So I can figure out what is going on and how to catch it more carefully.

Best guess at this point this. One of the reset values for the byte counting is 20 50 40. This clip reset but then had add-ons before the first real byte-count after the reset was seen to lock onto. This is not that big a deal it is just frustrating as it is yet another case I have never seen.

--jdiner

jdiner
09-20-2002, 02:17 AM
Originally posted by stealthdave

etc. btw, I'm using Linux/Wine to run the test. :) I had sent you a sample of this stream a while back (it failed with vsplit13 as well), but you may not have it around anymore. I had used dd to split it before. I can determine the chunk where the problem starts. How much data do you want, what do you want me to use to split it with (is dd fine, or should I use your tysplit tool?), and where should I send it? This problem occurs with about half of my Battlestar Galactica streams, so I can get multiple samples if you need them.


I still have it. But the one I have works. This is a related but somehow different error.

Please split it with the TyFileSplit. Getting clips that are split on the wrong boundry just makes them tedious for me to fix. Then upload it to the same ftp site as before.

--jdiner

jdiner
09-20-2002, 02:20 AM
Originally posted by FreydNot


33333 -> old Vid ByteCount = 2E 33 94 to 30 F F0
33333 -> new Vid ByteCount = 23 44 10 to 25 20 9C
33333 -> NOT ALIGNED on the Video... It is an OOB chunk!
Found an OOB packet... The Audio Diff is: 00:00:00.036
It is in sequence... Everything is fine...


Crap. Ok. Remember the 20 50 40 I have mentioned at times. This is a reset in the byte counter of the TyStream file to 23 44 10. This is the first time I have seen this. But then there have been 2 new ones tonight. But that is why I release the pre-test version. To see if these would keep cropping up.





### The Vid Body is off! 23 44 18 -> 30 F F8 diff == 0xFFF33420 (-838624)


This is the same reset value again. Just not in a position in the chunk to cause the rest of the error/debug messages.

--jdiner

Mike
09-20-2002, 03:37 AM
Hi,

New to the forum but wondered if I could help test from the UK TIVO perspective. I tried out the new version but on WinXP it GPF's on every ty I have tried. An example of the output with /V on is shown below. Of course it may be me in which case flame away! but if it isn't I am happy to help try fixes or provide more info. I can FTP the file (or a part of it if someone tells me how and where to split it). The example is small about 131MB so I could ftpit as is on cable. Thanks for all the great work.

X:\newshows>vsplit13a -V test2.ty hmm.m2v hmm.m2a
Processing 'test2.ty': (10 chunks per tick)

checkNumRecs: Number of Records: 18165 (f5 46 7a bd)
Startup: Found a bad chunk in the first Analysis-10. Skipping until there are 10 good ones.

checkNumRecs: Number of Records: 92 (5c 0 16 80)

checkNumRecs: Number of Records: 95 (5f 0 ff ff)

checkNumRecs: Number of Records: 93 (5d 0 2 80)

checkNumRecs: Number of Records: 96 (60 0 3 80)

checkNumRecs: Number of Records: 96 (60 0 3 80)

checkNumRecs: Number of Records: 76 (4c 0 22 80)

checkNumRecs: Number of Records: 95 (5f 0 ff ff)

checkNumRecs: Number of Records: 107 (6b 0 7 80)

checkNumRecs: Number of Records: 81 (51 0 b 80)

checkNumRecs: Number of Records: 97 (61 0 ff ff)
Read the first 10 chunk's for analysis...
Detected Tivo Type: Standalone
Detected Audio Stream Type: MPEG Layer II
We found the SEQ header start!
***** Setting first Video PTS to: 00:00:01.402
Final standardAudioSize = 880
Final standardFrameLength = 864
Final standardAudioDiff = 3240 or 00:00:00.036
First Video PTS: 00:00:01.402


last 4 = 0 0 0 0 -- first 4 = 5c 0 16 80

X:\newshows>

Kythorn
09-20-2002, 08:10 AM
### The Vid Body is off! 20 50 50 -> 31 B4 50 diff == 0xFFEE9C00 (-1139712)

### The Vid Body is off! 6A 66 B4 -> 6A 66 A4 diff == 0x10 (16)

### The Vid Body is off! 5D A3 60 -> 5D A3 50 diff == 0x10 (16)

### The Vid Body is off! 5C D7 50 -> 5C D7 40 diff == 0x10 (16)

### The Vid Body is off! 63 A6 18 -> 63 A6 8 diff == 0x10 (16)

### The Vid Body is off! 5C 30 64 -> 6D 94 64 diff == 0xFFEE9C00 (-1139712)

You want clips of any of these? BTW, PM me the specific FTP you want these uploaded to.

I'm going to try to PM you the entire log file from my 26 tests, and you can point out any particular chunks you want.

rd001
09-20-2002, 08:26 AM
Originally posted by Kythorn
Here's the most amusing log output I've seen in a while.
Note: this spans 3 FSIDs, but is reported as 0m in TyTool.
I've watched this entire episode in the past...
The only time I've seen this is when I'm replaying a stored video from disk while running Tytool. If I put the Dtivo back on live TV, then I get proper file sizes. I assume DTivo somehow locks it's database file while you're viewing it the same way that it locks it while recording it.

Do you ever get the "0M" from anything else?

Kythorn
09-20-2002, 08:38 AM
Originally posted by rd001

The only time I've seen this is when I'm replaying a stored video from disk while running Tytool. If I put the Dtivo back on live TV, then I get proper file sizes. I assume DTivo somehow locks it's database file while you're viewing it the same way that it locks it while recording it.

Do you ever get the "0M" from anything else?

Tivo isn't playing anything. Tuner 1 -> channel 0. Tuner 2 -> Channel 1. (both videoless channels)

Just fired up tytool and refreshed, it's still 0 megs. And no, I've never seen this before.

jdiner
09-20-2002, 11:15 AM
Originally posted by Kythorn


Tivo isn't playing anything. Tuner 1 -> channel 0. Tuner 2 -> Channel 1. (both videoless channels)

Just fired up tytool and refreshed, it's still 0 megs. And no, I've never seen this before.

I have seen this before. If it was being played and was stopped at some point in the middle of the show. I.e. "resume playing option" the size is oftern reported a 0... I don't know why. But the size records that come back from the MFS DB in the tivo don't report anything... I.e. not just wrong I actually get back 0's...

--jdiner

jdiner
09-20-2002, 12:20 PM
Ok. Well. Something things are just not right still. But I have a new test version pretty much ready to roll. I am just running some big tests of it here before I post it.

So everyone get ready to run things again...

--jdiner

jdiner
09-20-2002, 07:48 PM
Ok. A few more things have been cleaned up and trapped out now.

Please run this one through again. What I most need people to look for at this point are the ### and $$$ flags. These are part of the lineup checking code in the new byte counting stuff. They show up when things are off. For now with this release ignore the rest of the messages.

They should never show up... But do on occassion for some unknown reason in the audio. I have seen 3 of them in 50 test clips. If you find a file with them please drop me a quick email at:

joshua@best.com

A PM or a post here might be easier but I would like to archive everything a touch more permanently.

--jdiner

jdiner
09-20-2002, 07:48 PM
Anyone having problems with the temporal references had any luck with the 13a release and getting rid of them?

--jdiner

scarabus
09-20-2002, 08:07 PM
I've tried it with two streams that were problematic before.
One stream previously had the temporal references problem, and it still does.

The other previously had no audio (though it plays OK on TiVo)
and it shows an absolute slew of errors.

Overall, I saw no improvement over the previous version.

I think I've already submitted samples of these streams,
but can do so again if it helps.

scarabus
09-20-2002, 08:12 PM
Here's the output from vsplit13b
Maestro complains of temporal distortion 4% into the m2v file


Processing 'grobags.ty': (10 chunks per tick)
Detected Tivo Type: DTivo
Detected Audio Stream Type: MPEG Layer II
Final standardAudioSize = 588
Final standardFrameLength = 576
Final standardAudioDiff = 2160 or 00:00:00.024
First Video PTS: 00:01:55.392
.....$$$ The Aud Body is off! 33 AD 70 -> 33 A9 5C diff == 0x414 (1044)
.... 100......... 200......... 300......... 400......... 500
......... 600........55555 -> old Aud ByteCount = 33 F4 28 to 34 F AC
55555 -> new Aud ByteCount = 34 14 C to 34 2D 40
55555 -> NOT ALIGNED on the Audio... It is an OOB chunk!
We got one that is just to far away!
getBoth: old Vid ByteCount = 28 5D E8 to 2A 3E 44
getBoth: old Aud ByteCount = 33 F4 28 to 34 F AC
Found an OOB packet... The Audio Diff is: 00:00:00.288
Is it in sequence??? It is OFF by exactly 12.000000 frames.
It is in the right range. Let's see if it lines up.
It is in sequence. Starting back up after the 'hole'...
. 700......... 800......... 900.........

[then everything OK until]

8000
......... 8100.........

33333 -> old Vid ByteCount = 2D 73 D0 to 2F 5C CC
33333 -> new Vid ByteCount = 28 FC 70 to 2A E2 7C
33333 -> NOT ALIGNED on the Video... It is an OOB chunk!
55555 -> old Aud ByteCount = 32 35 B8 to 32 4A 5C
55555 -> new Aud ByteCount = 32 C4 F4 to 32 DB E4
55555 -> NOT ALIGNED on the Audio... It is an OOB chunk!

77777 -> Yep. We are out of here! Neither lined up!


Found an OOB packet... The Video Diff is: 01:22:09.609
getBoth: old Vid ByteCount = 2D 73 D0 to 2F 5C CC
-1x4: We are NOT aligned on the Video... It is an OOB chunk!
We got one that is just to far away!
getBoth: old Vid ByteCount = 2D 73 D0 to 2F 5C CC
getBoth: old Aud ByteCount = 32 35 B8 to 32 4A 5C
Found an OOB packet... The Audio Diff is: 01:22:09.498
Is it in sequence??? It is OFF by exactly 205395.750000 frames.
Nope... Not in sequence... Skipping it...

8200.........

[then everything OK until]

.........15000
...

33333 -> old Vid ByteCount = 24 A0 AC to 26 7C E8
33333 -> new Vid ByteCount = 28 93 20 to 2A 65 50
33333 -> NOT ALIGNED on the Video... It is an OOB chunk!
55555 -> old Aud ByteCount = 33 FE 58 to 34 1E 74
55555 -> new Aud ByteCount = 32 F9 5C to 33 22 A4
55555 -> NOT ALIGNED on the Audio... It is an OOB chunk!

77777 -> Yep. We are out of here! Neither lined up!


Found an OOB packet... The Video Diff is: 00:37:02.885
getBoth: old Vid ByteCount = 24 A0 AC to 26 7C E8
-1x4: We are NOT aligned on the Video... It is an OOB chunk!
We got one that is just to far away!
getBoth: old Vid ByteCount = 24 A0 AC to 26 7C E8
getBoth: old Aud ByteCount = 33 FE 58 to 34 1E 74
Found an OOB packet... The Audio Diff is: 00:37:02.926
Is it in sequence??? It is OFF by exactly 92621.916667 frames.
Nope... Not in sequence... Skipping it...

......15100.........15200.........15300.........15400.........15500
.........15600.........

33333 -> old Vid ByteCount = 2F 65 DC to 31 4E 88
33333 -> new Vid ByteCount = 26 9 78 to 27 DA 0
33333 -> NOT ALIGNED on the Video... It is an OOB chunk!
55555 -> old Aud ByteCount = 32 A2 68 to 32 B7 C
55555 -> new Aud ByteCount = 34 F 10 to 31 BD D4
55555 -> NOT ALIGNED on the Audio... It is an OOB chunk!

77777 -> Yep. We are out of here! Neither lined up!


Found an OOB packet... The Video Diff is: 00:41:29.122
getBoth: old Vid ByteCount = 2F 65 DC to 31 4E 88
-1x4: We are NOT aligned on the Video... It is an OOB chunk!
We got one that is just to far away!
getBoth: old Vid ByteCount = 2F 65 DC to 31 4E 88
getBoth: old Aud ByteCount = 32 A2 68 to 32 B7 C
Found an OOB packet... The Audio Diff is: 00:41:29.557
Is it in sequence??? It is OFF by exactly 103731.541667 frames.
Nope... Not in sequence... Skipping it...

15700...

A/V Sync Offset: 4ms (i.e. plays 4ms early!)

Wooly
09-20-2002, 09:35 PM
Sound good to me - I live on a lake, so we'll water-ski and drink beer all day, edit and setup batches for the computers to process in the evening, and then let the puter's work on the next-day's batches. Not a bad way to spend September in Florida!


Originally posted by rc3105
wooly:

I'm working on editing utils so I can chop the commercials out of my b5 episodes and archive them to dvd. I should head over there in my RV, camp out and make a set of mcgyvers for jdiner and a set of b5's for us.


:D

--
Riley


*jdiner, what type of standalone player do you have?

Kythorn
09-20-2002, 11:00 PM
Ran through the same 26 streams with 3 more thrown in for good measure.

Overall same results, as far as I could see, the new code and the old code agreed.

The 7 occurences I previously had of ### The Vid Body is off! did not occur in 13b.

No better luck with the clip that vsplit just completely disintegrates on though. I'll make and upload as many clips as you like, just tell me which ones, and where to put them. Bandwidth on my side is not an issue.

If you can't put the location here, PM me or shoot an email, I'll toss the ones you requested so far and any additional ones up tomorrow.

Kythorn
09-20-2002, 11:20 PM
Originally posted by jdiner


Wow. That disintegrated nicely. I would like a copy of that one. Actually just the piece where it is on and then gets off so badly.

Use the TyFileSplit program to grab just that section. Please start 40 or so chunks before it fails, and then go 100-150 chunks in length from that point. That will great a file of roughly 20-30 meg in size. Hopefully you have a decent network link and can upload it to me.

--jdiner

This is kind of interesting.

I did what you said, and split off from did tyfilesplit from 8000 for 150 blocks. Ran vsplit on it for the hell of it... processed fine.

Got curious, and split off from 8000 to the end of the file, then ran vsplit on it again.

The errors don't start until another solid 2000 chunks in, and they're completely different errors.

I tossed a mail to you of this log file as well.

This is not due to vsplit13b, the except for some ### vid body's that didn't happen in 13b, 13a and 13b's output is identical - if I start at the beginning and run all the way through.

Let me know if you want a different or additional range of chunks.

jdiner
09-21-2002, 12:30 AM
Kythorn, check your PMs...

--jdiner

pbar
09-21-2002, 04:32 AM
Originally posted by jdiner

(at the standard 29.97 the tivo uses)

...sounds like a great idea, but it would be very nice if there was an option for 25fps as well for us poor impoverished PAL users! :-)

jdiner
09-21-2002, 11:38 AM
I had not thought about it. But yeah. I can do that at the same time.

--jdiner

jdiner
09-21-2002, 02:35 PM
Ok. I found some documentation on the temporal_reference issues some are having with other software. If I am right in what I have read then it should be fixable.

If someone who has a clip that has one could please cut a section that contains a temproral reference and send it to me I would really appriciate it. With a known bad to test against I can't go any further with that part.

--jdiner

FreydNot
09-21-2002, 05:10 PM
Originally posted by jdiner
Please run this one through again. What I most need people to look for at this point are the ### and $$$ flags. These are part of the lineup checking code in the new byte counting stuff. They show up when things are off. For now with this release ignore the rest of the messages.

I didn't see any ### or $$$ on the 13 1 hour ty files I have handy.

There was an occastion where I had old style messages with out the matching new style messages

Fugg
09-21-2002, 06:15 PM
My first weird one!
this is a hour program that just flat finishes at 21:43, but it's a 1.25gig ty!??!?

D:\ty>VSplit13b.exe "Stargate SG-1-Prometheus-seasF.ty" "Stargate SG-1-Prometheus-seasF.m2v" "Stargate SG-1-Prometheus-seasF.m2a"
Processing 'Stargate SG-1-Prometheus-seasF.ty': (10 chunks per tick)
Detected Tivo Type: DTivo
Detected Audio Stream Type: MPEG Layer II
Final standardAudioSize = 588
Final standardFrameLength = 576
Final standardAudioDiff = 2160 or 00:00:00.024
First Video PTS: 00:01:45.817
........### The Vid Body is off! 5C 30 40 -> 6D 94 50 diff == 0xFFEE9BF0 (-1139
728)
### The Vid Body is off! 5C 30 6C -> 5C 30 4C diff == 0x20 (32)
. 100......... 200......... 300......... 400......... 500
......... 600......... 700......... 800......... 900......... 1000
......... 1100......... 1200......... 1300......... 1400......... 1500
......... 1600......... 1700......... 1800......... 1900......... 2000
......... 2100......... 2200......... 2300......... 2400......... 2500
......... 2600......... 2700......... 2800......... 2900......... 3000
......... 3100......... 3200......... 3300.....

A/V Sync Offset: 11ms (i.e. plays 11ms early!)

the rest(11 so far) are ok. (no ###'s)

Fugg
09-21-2002, 06:51 PM
another one:
30min, 496meg

Processing 'King of the Hill-A Man Without a Country Club.ty': (10 chunks per tick)
Detected Tivo Type: DTivo
Detected Audio Stream Type: MPEG Layer II
Final standardAudioSize = 492Final standardFrameLength = 480Final standardAudioDiff = 2160 or 00:00:00.024
First Video PTS: 00:01:08.711
......... 100......... 200......... 300......... 400......... 500
......... 600......... 700......... 800......... 900......... 1000
......... 1100......... 1200......... 1300......... 1400......... 1500
......... 1600......... 1700......... 1800......... 1900......... 2000
......... 2100......... 2200......... 2300......... 2400.$$$ The Aud Body is off! 6F 5C D8 -> 6F 59 90 diff == 0x348 (840)
$$$ The Aud Body is off! 6F BF 20 -> 6F 6C 30 diff == 0x52F0 (21232)
### The Vid Body is off! 6D 2 DC -> 69 A1 AC diff == 0x36130 (221488)
$$$ The Aud Body is off! 6F CB 8C -> 6F C8 B4 diff == 0x2D8 (728)

Found an OOB packet... The Video Diff is: 00:18:31.708
getBoth: old Vid ByteCount = 66 76 4 to 68 2E EC
-1x4: We are aligned on the Video so it is good!!!
Found an OOB packet... The Audio Diff is: 00:00:00.024
It is in sequence... Everything is fine...
.$$$ The Aud Body is off! 6F E0 28 -> 6F DC D0 diff == 0x358 (856)
Unknown PES packet. Fix it dear henry...
type = 0x80
### The Vid Body is off! 61 4 E0 -> 5D 28 44 diff == 0x3DC9C (253084)
$$$ The Aud Body is off! 6D BB 90 -> 6F E2 14 diff == 0xFFFDD97C (-140932)

Found an OOB packet... The Video Diff is: 00:19:56.922
getBoth: old Vid ByteCount = 68 2E EC to 6D 60 4C
-1x4: We are aligned on the Video so it is good!!!
We got one that is just to far away!
getBoth: old Vid ByteCount = 68 2E EC to 6D 60 4C
getBoth: old Aud ByteCount = 6F 4F FC to 6F D3 38
Found an OOB packet... The Audio Diff is: 00:18:31.998
Is it in sequence??? It is OFF by exactly 46333.250000 frames.
Nope... Not in sequence... Skipping it...



33333 -> old Vid ByteCount = 68 2E EC to 6D 60 4C
33333 -> new Vid ByteCount = 61 B6 F4 to 63 84 60
33333 -> NOT ALIGNED on the Video... It is an OOB chunk!
55555 -> old Aud ByteCount = 6F 4F FC to 6F D3 38
55555 -> new Aud ByteCount = 6D CC D0 to 6D F7 0
55555 -> NOT ALIGNED on the Audio... It is an OOB chunk!

77777 -> Yep. We are out of here! Neither lined up!


Found an OOB packet... The Video Diff is: 00:18:29.672
getBoth: old Vid ByteCount = 68 2E EC to 6D 60 4C
-1x4: We are NOT aligned on the Video... It is an OOB chunk!
We got one that is just to far away!
getBoth: old Vid ByteCount = 68 2E EC to 6D 60 4C
getBoth: old Aud ByteCount = 6F 4F FC to 6F D3 38
Found an OOB packet... The Audio Diff is: 00:18:29.790
Is it in sequence??? It is OFF by exactly 46241.250000 frames.
Nope... Not in sequence... Skipping it...

$$$ The Aud Body is off! 6E B 1C -> 6E 8 44 diff == 0x2D8 (728)
### The Vid Body is off! 67 3B EC -> 65 5 4C diff == 0x236A0 (145056)
$$$ The Aud Body is off! 6E 5F 50 -> 6E 10 DC diff == 0x4E74 (20084)


33333 -> old Vid ByteCount = 68 2E EC to 6D 60 4C
33333 -> new Vid ByteCount = 63 84 60 to 67 83 34
33333 -> NOT ALIGNED on the Video... It is an OOB chunk!
55555 -> old Aud ByteCount = 6F 4F FC to 6F D3 38
55555 -> new Aud ByteCount = 6D F7 0 to 6E 7A 28
55555 -> NOT ALIGNED on the Audio... It is an OOB chunk!

77777 -> Yep. We are out of here! Neither lined up!
and on and on......

the txt log is 1.24meg

21 more, 36 total, 2 bad ones (above)

That's all I got here!:)

pbar
09-22-2002, 08:35 AM
Originally posted by laserfan
If you or anyone else can offer pointers on cleaning-up codecs beyond the obvious of re-formatting/reinstalling from scratch, they would be most welcome.

The best way I've found to work out what codecs are installed and being used is to use graphedt.exe (part of the DirectX SDK but you can find it all over the web).

They are basically COM objects and usually in a file ending .ax (though it's just a DLL). You can often register and unregister them using regsvr32.exe (use the /u option to unregister).

They are located via various registry entries under HKEY_CLASSES_ROOT.

Attack
09-22-2002, 07:45 PM
Here are the logs from vsplit12 and vsplit13b all video was extracted from my 14 Hour SA TiVo. All shows have been recorded within the last month.


I didn't notice anything odd in the logs but I'm not sure if I am reading this correctly.


I will extract everything from my other SA TiVo tonight, it still has a recording almost one year old (9/30/01) on it. Once it is done extracting I will run vsplit12 and vsplit13b on them.

Attack
09-24-2002, 12:12 AM
Here are the logs from vsplit12 and vsplit13b for my larger TiVo for all but 4 shows since I am running out of disk space.

I do have a large number of these error messages

"### The Vid Body is off!" with vsplit13b


Should I split all these streams and send them to you or can you tell from the logs if you only need a few of the streams? Where should I upload the split ty files to?

Also would the .html file from TiVo Web about each episode be enough info? If you need more info what else might be needed?

gifford
09-26-2002, 06:27 PM
I have not seen mention of ffmpeg (http://sourceforge.net/projects/ffmpeg) in this forum, but I have wondered if would be feasible to write a tystream-parsing module for ffmpeg.

This would make it easy (?) to transcode into other formats (e.g., MPEG4, ASF, etc.) and might even help with some synchronization issues (ffmpeg uses PTS values internally for muxes).

For a skeletal example, I have "ported" http://www.samba.org/ftp/unpacked/tivo/vplay/vsplit.c to the ffmpeg libav framework. I do not know if this even does anything productive, since my TiVo is not yet extract-enabled, but the code does compile. ;)

jdiner
09-26-2002, 10:11 PM
Ok. I took another approach to the extended chunk checking code. It works great here. But there are a few area where I expect it will still croak but my Tivo's never make those streams so I am waiting for others to find problems and send me the streams that cause them. To that end here is the next release.

The text output has changed a bit, but is really compeletely different.

Notes:

1- You may still see lines like:

$$$ The Aud Body is off! 33 EA 7C -> 33 3B 78 diff == 0xAF04 (44804)
### The Vid Body is off! 20 64 C4 -> 23 FB 80 diff == 0xFFFC6944 (-235196)

But they are almost alway directly before an OOB chunk now. These are just my code trying to lock onto things. So if you see them they are NOT and indication of failure but rather just "weirdness" in the TyStream. Please send me just these 2 lines via email or a PM. I am collecting the "reset to" values.

2- The 33333/55555/OOB output has been changed pretty thoroughly. You will now see the OOB warning first, followed potentially by 33333 lines for a video OOB report and a 55555 lines for an audio OOB report. Something like the following:



Found an OOB packet... The Video Diff is: 00:34:25.811
33333 -> + old Vid ByteCount = 21 16 64 to 22 F4 50
33333 -> + new Vid ByteCount = 22 F4 50 to 21 3C 54
33333 -> + WE ARE ALIGNED on the Video so it is good!!!


And in some cases a really long one like:



Found an OOB packet... The Video Diff is: 00:34:25.811
33333 -> + old Vid ByteCount = 22 F4 50 to 21 3C 54
33333 -> + new Vid ByteCount = 21 3C 54 to 23 24 10
33333 -> + WE ARE ALIGNED on the Video so it is good!!!
Found an OOB packet... The Audio Diff is: 00:34:25.811
Is it in sequence??? It is OFF by exactly 86075.458333 frames.


#1 55555 -> + old Aud ByteCount = 33 30 0 to 33 FC D0
#1 55555 -> + new Aud ByteCount = 33 FC D0 to 34 11 74
55555 -> + WE ARE ALIGNED on the Audio so it is good!!!


Just like before in the old code this last one is all 1 piece... You get byte count output for video and the statmake that it is "good". Then there is the old audio diff warning and a report of how many frames off it is followed by the new byte count code and finally a report that it is aligned so it is good.

NOTE: The above happens when the PTS values in the file reset. For the DTivo these should now be handled correctly and the splitting problems with it happens should be done. THIS IS NOT READY FOR THE SATIVO YET!!! I am waiting for some streams from some tester to finish up the SATivo stuff.

3- There was a fix to the SATivo vs DTivo detection code. Someone had sent me a clip that showed a DTivo stream as an SATivo stream. It was a first for me. But it is now handled. (So to the user that sent me badmax.ty/badcriminal.ty/badshow.ty try them again. They work here now...)

Ummm. That is about it. keep in mind this is a testing release. Try it and see if it is any better on your clips. If you have some that still have problems then let me and get ready to send them to me. I will need them as my test bed now all work correctly except for the SATivo on with the PTS reset'ing problem.

--jdiner

jdiner
09-26-2002, 10:13 PM
Originally posted by gifford


For a skeletal example, I have "ported" http://www.samba.org/ftp/unpacked/tivo/vplay/vsplit.c to the ffmpeg libav framework. I do not know if this even does anything productive, since my TiVo is not yet extract-enabled, but the code does compile. ;)

While I think that this idea has merit. Don't confuse the vsplit you found with mine. The one you have works correctly on pretty much nothing... At some point it could be a good idea to take what I have done and plug it into ffmpeg or other tools that do "the rest" of the work.

Also don't be fooled by the concept of PTS's being used. They have to be. They are a fundament part of mpeg. It is like saying, the Viper is better than the Yugo because it uses... gasoline! :)

--jdiner

gifford
09-26-2002, 10:56 PM
Originally posted by jdiner
While I think that this idea has merit. Don't confuse the vsplit you found with mine. The one you have works correctly on pretty much nothing... At some point it could be a good idea to take what I have done and plug it into ffmpeg or other tools that do "the rest" of the work.

Also don't be fooled by the concept of PTS's being used. They have to be. They are a fundament part of mpeg. It is like saying, the Viper is better than the Yugo because it uses... gasoline! :)

--jdiner

It was quite obvious that the vsplit that I used did not have any of the intelligence of your program. That is why I called my example skeletal. :rolleyes:

Let's just say I am hopeful that ffmpeg might offer some support with the synchronization efforts. I certainly did not mean to imply that the presence of PTS values made anything a slam dunk.

That having been said, I can vouch for ffmpeg's ability to do what it does very well, and I think its output quality (esp. for MPEG4) is very good and CPU-efficient.

jdiner
09-27-2002, 01:43 AM
Originally posted by gifford
It was quite obvious that the vsplit that I used did not have any of the intelligence of your program. That is why I called my example skeletal. :rolleyes:


For the record my comment was supposed to be humorous... Guess I worked to long today.

--jdiner

FreydNot
09-27-2002, 03:05 AM
I ran vsplit13c on several 1 hour SA streams. No errors or strangeness.

Kythorn
09-27-2002, 10:04 AM
Running through my 40+ clip testbed, a handful of which have been proven difficult in the past - will get back to you when it's finished.

jdiner
09-27-2002, 10:54 AM
Originally posted by FreydNot
I ran vsplit13c on several 1 hour SA streams. No errors or strangeness.

Ah. I was not that clear I guess. It will work on SATivo streams just like always. However... I have seen in 1 clip from 1 user a reseting PTS problem. I.e. it counts up, and then suddenly at a little over 1 hour in the PTS it resets to a basically starting over value. That single condition is what I was refering too when I said it would not work. I am glad it worked on the standard streams as well...

--jdiner

Fugg
09-27-2002, 12:53 PM
w/vsplit13c, 34 good(excellent even!), 2 bad. same as before.
The 2 bad ones ended up being bad rips. (resumed from paused position during recording)

Good To Go!

:D

btw, the 2 are still good, i just have to jump around the bad chunk.

jdiner
09-27-2002, 04:12 PM
Ok. Things seem good and perhaps better than they were. The reset'ing PTS problem which got me once, for once, this last week :) which makes it much easier to test appears to have been solved.

So I am going to move onto the next issue. Back to mux'ing...

And in speaking of which. I would like some help from those with fast connections and large testbeds... I need people to make up some short clips that are as much a possible talking with closeups of faces.

That's right folks we are back to working on A/V sync with muxing... What is needed are some clips that start out with something we can determine sync on and after say roughly 5 minutes have more... So that we can detect and slip in sync.

To start with I don't want them FTP'ed to me. But it might come down to that. To start with I want people to make them. So that they are always the same and then start testing the output if we get on, then I won't need any sent to me. But if we have problems I am going to need some sent to me so I can pour over them and try to figure out why...

--jdiner

Kythorn
09-27-2002, 05:11 PM
Testbed.. check
Bandwidth.. check
Motivation to reclaim hundreds of gigs unprocessed tystreams are taking up.. check

Count me in.

newlooper
09-27-2002, 06:11 PM
I would like to participate as well.

jdiner
09-27-2002, 07:24 PM
Wow. Someone just sent me a 1.3 clip that VSplit couldn't do anything with. And now I know why. The constructs I have been using for 2.5 and up are just not there in the old streams. Nothing is as "it should be"... Wild. With the lack of PTS values I am amazed much of anything worked...

--jdiner

laserfan
09-27-2002, 08:47 PM
Might I suggest some "in concert" clips. Nothing like even a background drummer to detect sync problems...talking heads have fooled me in the past. What kind of music do you like jdiner?

jdiner
09-28-2002, 12:08 AM
Originally posted by laserfan
Might I suggest some "in concert" clips. Nothing like even a background drummer to detect sync problems...talking heads have fooled me in the past. What kind of music do you like jdiner?

Pretty much you name it. 80's, 3 Doors Down, Nickleback, The Boss, Sam Cooke... I have ecclectic tastes I guess...

But I must say I like that idea. It was not something that I had thought of. Someone else mentioned to me making a video stream manually. Put audio and video elements into that would show absolute sync. I like that idea but I don't have the equipment to do it...

--jdiner

FreydNot
09-28-2002, 12:58 AM
I can create a timing video in mpeg if someone else can get it into the tivo. I'd play it in, but I don't have any Tivo's set for s-video input.

FreydNot
09-28-2002, 01:35 AM
Got the following on a SA tystream using vsplit13c:



Processing 'Late Night With Conan O'Brien-.ty': (10 chunks per tick)
Detected Tivo Type: Standalone
Detected Audio Stream Type: MPEG Layer II
Final standardAudioSize = 880

Final standardFrameLength = 864

Final standardAudioDiff = 3240 or 00:00:00.036
First Video PTS: 00:00:01.193
......... 100......... 200......... 300......... 400......... 500
......... 600......... 700......... 800......... 900......... 1000
......... 1100......... 1200......... 1300......... 1400......... 1500
......... 1600......... 1700......... 1800......... 1900......... 2000
......... 2100......... 2200......... 2300......... 2400......... 2500
......... 2600......... 2700......... 2800......... 2900......... 3000
......... 3100......... 3200......... 3300......... 3400......... 3500
......... 3600......... 3700......... 3800......... 3900......... 4000
.........
Found an OOB packet... The Video Diff is: 00:16:26.806
33333 -> - old Vid ByteCount = 24 64 3C to 26 35 8C
33333 -> - new Vid ByteCount = 2A CF 20 to 2C AB 64
33333 -> - NOT ALIGNED on the Video... It is an OOB chunk!
Found an OOB packet... The Audio Diff is: 00:16:26.760
Is it in sequence??? It is OFF by exactly 27410.000000 frames.


#1 55555 -> + old Aud ByteCount = 21 79 10 to 21 A1 90
#1 55555 -> + new Aud ByteCount = 21 B2 70 to 21 D0 D0
55555 -> - NOT ALIGNED on the Audio... It is an OOB chunk!
Nope... Not in sequence... Skipping it...

4100......... 4200......... 4300......... 4400......... 4500
Unknown PES packet. Fix it dear henry...
type = 0x0
Unknown PES packet. Fix it dear henry...
type = 0x0
Unknown PES packet. Fix it dear henry...
type = 0x80
$$$ The Aud Body is off! 21 4D 30 -> 22 E8 F0 diff == 0xFFFE6440 (-105408)
### The Vid Body is off! 23 4A 10 -> 26 E 1C diff == 0xFFFD3BF4 (-181260)

Found an OOB packet... The Video Diff is: 00:24:08.073
33333 -> + old Vid ByteCount = 30 B F4 to 25 28 0
33333 -> + new Vid ByteCount = 25 28 0 to 24 1D FC
33333 -> + WE ARE ALIGNED on the Video so it is good!!!
Found an OOB packet... The Audio Diff is: 00:24:06.696
Is it in sequence??? It is OFF by exactly 40186.000000 frames.


#1 55555 -> + old Aud ByteCount = 22 C0 70 to 21 5E 10
#1 55555 -> + new Aud ByteCount = 21 5E 10 to 21 7C 70
55555 -> + WE ARE ALIGNED on the Audio so it is good!!!
......... 4600......... 4700......... 4800......... 4900......... 5000
......... 5100......... 5200......... 5300......... 5400......... 5500
......... 5600......... 5700......... 5800......... 5900......... 6000
......... 6100......... 6200......... 6300......... 6400......... 6500
......... 6600......... 6700......... 6800......... 6900......... 7000
......... 7100......... 7200......... 7300......... 7400......... 7500
......... 7600......... 7700......... 7800......... 7900......... 8000
......... 8100.........
Found an OOB packet... The Video Diff is: 00:00:33.756
33333 -> - old Vid ByteCount = 2F E8 98 to 24 F0 DC
33333 -> - new Vid ByteCount = 2C B0 6C to 2E 9B F4
33333 -> - NOT ALIGNED on the Video... It is an OOB chunk!
Found an OOB packet... The Audio Diff is: 00:00:33.768
Is it in sequence??? It is OFF by exactly 938.000000 frames.


#1 55555 -> + old Aud ByteCount = 21 5A B0 to 21 7F D0
#1 55555 -> + new Aud ByteCount = 22 DE D0 to 22 EF B0
55555 -> - NOT ALIGNED on the Audio... It is an OOB chunk!
Nope... Not in sequence... Skipping it...

8200......... 8300......... 8400......... 8500
......... 8600......... 8700......... 8800......... 8900......... 9000
......... 9100......... 9200

A/V Sync Offset: 1ms (i.e. plays 1ms early!)

scarabus
09-28-2002, 02:24 PM
JDiner, you're the man.

VSplit13c fixes the problem I've been seeing with streams that claim to have no audio packets.

As soon as that's finished I'll try it on my bad "temporal references" stream.

Do you plan to release a new version of TyTool?

jdiner
09-28-2002, 03:15 PM
Originally posted by rc3105
I can turn that timing mpeg into a dtivo stream, but a sa tivo or dtivo stream (via ele2pestriple) won't have exactly the same structure as dave's feed off the sat.

--
Riley

I can. If you can make it and get it to me. I can play it in via my computer with a nice SVideo output jack... :)

--jdiner

jdiner
09-28-2002, 03:54 PM
Ok. There were 3 things that I was not sure how to "build" when creating the mux'ed output.

I have spent the last 24 hours working over one of them. The System_Clock_Reference field in an MPEG-2 Pack header.

The docs for MPEG-2 were hard to follow so I went back to the MPEG-1 docs to see what they had to say. By starting there I figured out exactly how to calculate this field for an MPEG-1 file. Not what we really need, but a good starting point.

Then I moved back to the MPEG-2 docs and they made sense finally. So I felt I had that version in the bag as well. So I started dumping values out of what I have as mpeg-2 clips. Many I have downloaded so the source is not a DTivo and some I have mux'ed myself using a variety of tools...

This is where it gets wierd. Nothing much matches up. Using the standard calculation from the docs. The output of tmpgenc is just wrong. Massively so. The output of many others is wrong for me as well.

By wrong I mean that the values there don't match the formulas from the docs. The ones I can make myself that do are the ones that are the worst "off" in terms of sync. :)

I asked this once before but never really got clear on any of it. Does anyone have a program they use to mux streams, besides tmpgenc that produces a stream that will play cleanly and with sync. I am not talking about being edited, just mux and play.

--jdiner

dattrax
09-28-2002, 05:42 PM
Hi,

to be honest, I just use bbmpeg 12418. I extract using Tytool from my SA, set the program stream type to ~DVD and either add or subtract the number of ms from the 180ms delay for startup. This file is always in sync, but it even stays in sync if I use m2-edit to cut the ad breaks.

The only 'problem' you get is that it takes about 1 second after a seek in powerdvd for the audio and video to get to the same position, but it doesn't happen on my standalone player.

Jim

Kythorn
09-28-2002, 05:42 PM
I can't attest to how technically "right" it is, and I know others have had issues, but almost everything I've made using mplex plays fine, generally better than what I've made with tmpgenc.

Now, I'm also aware that you have used mplex in the past, so I can only assume you've not had satisfactory results with it.

I think something that's been mentioned that that might bear investigating, is when playback is out of sync, what codecs are involved?

The situation isn't the same, but I've seen mpeg4 clips that were out of synch in dshow, that worked fine in vfw.

I've also had tivo clips that were out of synch in powerdvd that played fine in a hardware player (generally considered more inflexible, I would have guessed this situation would have been reversed)

If we're going to be trying to figure whether or not something is sync'd, I think we need to eliminate as many variables as possible - I just don't have any real constructive suggestions on how to accomplish this.

FreydNot
09-28-2002, 05:44 PM
Originally posted by jdiner
I asked this once before but never really got clear on any of it. Does anyone have a program they use to mux streams, besides tmpgenc that produces a stream that will play cleanly and with sync. I am not talking about being edited, just mux and play.

The muxer in mpeg-vcr has given me bad sync. I say it failed.

I have had sucess with bbMpeg (AKA avi2mpg2) when used as just a muxer. Its a bit tricky to get configured, but there are instructions here on the board.

I thought I remembered some people having sucess using some basic unix tools too (mplex?).

jdiner
09-28-2002, 05:48 PM
Originally posted by dattrax
Hi,

to be honest, I just use bbmpeg 12418. I extract using Tytool from my SA, set the program stream type to ~DVD and either add or subtract the number of ms from the 180ms delay for startup. This file is always in sync, but it even stays in sync if I use m2-edit to cut the ad breaks.

The only 'problem' you get is that it takes about 1 second after a seek in powerdvd for the audio and video to get to the same position, but it doesn't happen on my standalone player.

Jim

This makes sense to me as the SA Tivo is CBR. And constant bit rate makes some things so much easier. For the DTivo with the seriously VBR nature of it all it seems as if all bets are off.

Just to give you all a better idea as to why I am wondering on this... The SCR field is used to sync the playback of each seperate Elementary Stream to the others, in our case the other one. Given the sync problems that would seem important wouldn't it... :)

The calculation for MPEG-1 is pretty simple. The calculation for mpeg-2 is more complex but related pretty directly. The calculation for a VBR mpeg-2 stream is undocumented...

--jdiner

dattrax
09-28-2002, 05:56 PM
sorry I forgot to mention that my tivo is set to VBR encoding mode, or space saving as it called.

certainly when I looked at this before the PTS values where in the stream. If these are used natively it should stay in sync.

Jim

FreydNot
09-28-2002, 06:12 PM
Originally posted by FreydNot
The muxer in mpeg-vcr has given me bad sync. I say it failed.


I take this back. My test was flawed.

[Edit: I checked it again and mpeg-vcd program stream didn't keep the audio in sync no matter what I did.]


Here is how I am testing sync...

Extract ty file
use vsplit to make element streams
use a muxer (bbMpeg or TMPGenc) to mux the element streams into a program stream.
open the program stream in zoom player and look for sync issues.

there are some issues with zoom player (maybe all players). If I start the clip at the begining and don't jump around, the audio is in sync. If I jump around while it is playing, the audio is out of sync. If I stop the player, jump to a position later (or earlier) in the clip, and then press play, the audio stays in sync.

This is all direct show based. I am using the latest version 2 beta of the elecard mpeg decoder and it appears to be using the moonlight odeo dekoda (which I assume is the moonlight audio decoder) for audio.


Oh, and I have my SA tivo set to VBR also.

jdiner
09-28-2002, 06:26 PM
Originally posted by FreydNot
The muxer in mpeg-vcr has given me bad sync. I say it failed.

I have had sucess with bbMpeg (AKA avi2mpg2) when used as just a muxer. Its a bit tricky to get configured, but there are instructions here on the board.

I thought I remembered some people having sucess using some basic unix tools too (mplex?).

Ok. To be clear. Running tests with the CBR output of my SA Tivo and BBMpeg. it works like a charm. I get nice sync all the way through. But that isn't the problem. I suppose I should have made myself clear at the start. Using the VBR setting has the same good result, because the ES data itself is CBR.

With output from VSplit and my DTivo... Using bbmpeg I am off by roughly 5 seconds about 20 seconds in... Not 5ms, but 5 full seconds. And it degrades from there until at the end I am off by... so much I can really tell at least 2-3 minutes. The end being 45-60 minutes into a clip. Very strange. As near as I can tell I have it configured right as it does work with the SATivo, but not with CBR or VBR for the DTivo. Nor does mpeg-2 versus DVD make any difference.

--jdiner

pbar
09-29-2002, 04:37 AM
Originally posted by rc3105
mplex on mac & pc & in tivo, tmpgenc, bbmpeg,
ele2pestriple

all produce perfectly sync'd mpg here with tytool split streams from dtivo's running 2.5.01


As I've mentioned before in this thread, I believe that for many people perceived sync issues are down to which codecs are being used by the mpeg player they use to view the program stream and transcode it.
http://www.dealdatabase.com/forum/showthread.php?s=&threadid=16645

jdiner
09-29-2002, 11:50 AM
I can't realy figure t out either... Why I am having so much trouble. What I am now producing claims to match the specs I have. BUT... and this is a big but... there is nothing in the specs that defines what to do if you are in a VBR setup. Everything I have references fixed bitrates.

I can understand having things be off. I can understand having it be the codecs. But I have used 4 different players to try this and sync is maintained in all with some output, notably Tempgenc, and not in all with others, sadly my stuff.

Now one reason some codec work and some don't seems to be tied to the issue of the SCR values. Which is why I am spending so much time on them. Many codecs, at M$ suggestion, ignore this value and use only picture timestamps. And again some do not. Beyond that I can't explain that much about what others are doing.

And lastly perhaps I should just get a Mac... I can't believe the mess with the PC for all of this.

To try and make it more clear I will write up a real explanation of what I have found and seen and where I think things are going wrong and why. That way everyone can try to get a feel for what I am talking about. Maybe I am just missing something...

--jdiner

edpuffmonster
09-29-2002, 02:11 PM
Originally posted by pbar
As I've mentioned before in this thread, I believe that for many people perceived sync issues are down to which codecs are being used by the mpeg player they use to view the program stream and transcode it.
http://www.dealdatabase.com/forum/showthread.php?s=&threadid=16645

I agree. Moonlight Cordless's codecs do a grand job of messing all that stuff up. Even some DiVX videos fall out of synch if I have their codecs loaded.

zqfmbg
09-29-2002, 02:34 PM
Originally posted by jdiner
To try and make it more clear I will write up a real explanation of what I have found and seen and where I think things are going wrong and why. That way everyone can try to get a feel for what I am talking about. Maybe I am just missing something...

Works for me; I vaguely remember my copy of the specs coming with VBR-related stuff... :D

jdiner
09-29-2002, 07:26 PM
Originally posted by zqfmbg
Works for me; I vaguely remember my copy of the specs coming with VBR-related stuff... :D

Well. Look at it, if it does, send the info my way... :)

--jdiner

jdiner
09-30-2002, 01:53 AM
Well. I have now tried a host of commercial mux'ing programs. Just to see what the output would be. Almost universally nothing does VBR right. Infact the way in which most of them fail is quite related... I.e. the same pauses in the same place show up in many cases.

One of the programs I got a demo to costs $3k. What a joke. Anyway, it only runs for 10 seconds worth of output but during that time things appear to be sync'ed. But in taking a look at their output things are kind of wierd. Thousands, literally, of 1024byte padding packets. Which make things needlessly huge. In this case almost double what everything else produces.

In going through all of these programs only 1 actually appeared to be even close on output. So I am not alone in the problem area... I was hoping to get closer to a solution than that. But hey at least I have rule some more options... :)

--jdiner

KRavEN
09-30-2002, 10:14 AM
jdiner, whats the nam of this $3K program?

jdiner
09-30-2002, 11:37 AM
Originally posted by KRavEN
jdiner, whats the nam of this $3K program?

Ok. I tested:

1- mpspsmd - The demo from Manzanita, it is a Windows Muxer... It only produced 10 seconds worth. Their price $995. NOTE: I suppose I should metion that while it was only 10 seconds long it appeared to be in sync. The real problem was that I just don't have enough to tell whether it is really on or not. It claims to produce 10 seconds worth of output but when it is played it is only about 6 seconds long?!!? The nature of a free demo I guess.

2- MPEGRepair - From PixelTools. This one wanted to "rebuild" rather than just mux the output. But I did no with no changes. Their price $2,995.00. They also offer a TransMux program which is what I really wanted to try. But no one I know has this program. TransMux is $995 as well. Or you can get both for a major savings of $3,495.

3- Then I tried the edit features of MyFlix_XE. While it does not have a direct mux'er in it. once again it will "dub" audio onto the video. Which is in essence mux'ing by a different name. Do be honest I don't remember what the list price of this one was to be..

4- Then I tried Vitec's DVD toolbox v2.0. This one had it's own set of features including a mux'er. It produced a file almost exactly identical to BBMpeg's output. Which means that for me the sync was as smashed as earlier... List price $299.

5- I then tried something mentioned here called XMuxer. But it would not install. Claimed it had already been installed and the eval period was over?!!? I don't remember ever having it installed but you just never know.


All of my visual testing is being done with PowerDVD v2.5, which came with my Video card, and MS MediaPlayer. Everything produced by Tmpgenc, version 2.57.41.146, plays correctly in both out to the very end of the clips. Everything else is junk... Now the question is WHY is everything else junk.

So on the free front I have tried BBMpeg, stdmux (from here way back when...), old mplex, MJpeg's mplex, another version of MPLex that was originally extended for the Mac, Tmpgenc, etc..

--jdiner

Kythorn
09-30-2002, 12:57 PM
Out of curiosity, does anyone know what mplex attempts to do differently with -V specified? I haven't had a chance to wade through the source.

lmurray
09-30-2002, 01:44 PM
looking at mplex and....

vbr only does a few things in the multiplex code (that I can see).

first, theres a method call RunInSectors. the comment follows:

/*
* Compute the number of run-in sectors needed to fill up the buffers to
* suit the type of stream being muxed.
*
* For stills we have to ensure an entire buffer is loaded as we only
* ever process one frame at a time.
*
*/


in which there's a if statement for computing sector_delay:

else if( vbr )
sectors_delay += 3*(*str)->BufferSize() / ( 4 * sector_size );
else
sectors_delay += 5 *(*str)->BufferSize() / ( 6 * sector_size );


not sure what the meaning of the math is, but hopefully it'll make sense to someone.

then later in the code they create sys_headers. not sure whats going on here. I need to look at it more.

then in the OutputMultiplex method, theres a big if statement again about vbrs.
comments are:
//
// If we got here no stream could be muxed out.
// We therefore generate padding packets if necessary
// usually this is because reciever buffers are likely to be
// full.
//


code is about 10 extra lines. seems to be about judging how to do the output padding for vbr.
check out line 986 of multplex.cc


seems worthwhile for someone w/ mpg knowledge to look into this a little more. there's plenty of comments in this code.

-lloyd-

jdiner
09-30-2002, 03:09 PM
The frustration with source code like that is that what is being produced is wrong. Why is it wrong? Who knows. Where is it wrong? Again who knows...

I use PowerDVD, as I mentioned, to play these things. Is it the source of some of the problems? Again who knows... What I need is setup that we can all agree works so I can start lifting information from it. But no one really seems to have one. I know I don't. I have had so many codecs come and go on this machine that MediaPlayer is dodgy at best these days.

--jdiner

Kythorn
09-30-2002, 03:33 PM
You could always try zoomplayer and build your own graphs to control exactly which codecs get used - if you think that's really an issue.

I've done it in the past to troubleshoot crazy dshow type things.

Kythorn
09-30-2002, 04:37 PM
jdiner:

Just a shot in the dark, but do you have the tmpgenc vfapi plugin setting that takes over playback of mpeg2 related things turned on?

It could possibly explain why tmpgenc output plays back the 'best' for you.

I'd rate this one as a one in a million shot, but I figured I'd mention it.

Fugg
10-01-2002, 04:52 PM
Originally posted by jdiner
...
I use PowerDVD, as I mentioned, to play these things. Is it the source of some of the problems? Again who knows... What I need is setup that we can all agree works so I can start lifting information from it. But no one really seems to have one. I know I don't. ...
--jdiner

I think I do...

From my dtivo running 2.5.1,
I use Tytool to extract.
I use vsplit13c to process.
I use Tmpgenc to simple mux.

I never use any offset value to "re-sync". I just mux 'em as vsplit spits 'em out.

Other than when I try to do frame accurate edits(gop edits with Tmpgenc's merge and cut are fine), I never have a sync problem at all with any file from my dtivo. I haven't since the arrival of vsplit-x.

I use windvd 3 and 4 and hollywood station to view, as well as an apex 1100w.

No sync probs at all.

Other than the occasional bad rip(due to reasons stated in earlier posts), I am working at almost 100%.

Am I just the lucky one? Did I simply luck into "The Setup That Works"???!??!?!

or am i just hosed and missed the point entirely?

Wooly
10-01-2002, 08:34 PM
You're not the only one - I know several people who follow that process (the only thing I do differently is that I use TivoApp to extract...other than that I'm identical to you) and have no problems whatsoever. I believe that Jdiner has the vsplit down-pat, and is now working on a Muxing component that will allow us to eliminate that last step as well.

If we could get the muxing figured out through TyTool/Vsplit/Whatever and just find a frame-accurate way of SIMPLY removing commercials I'd be ready to start burning my MacGyver's to DVD and fulfill my promise to Jdiner (I promised that when we got Muxing and a way for me to simply edit out the commercials and therefore create DVD's with shows on them, I'd burn the entire MacGyver series for Jdiner onto DVD). I'm a man of my word, so as soon as we get to that point I'm ready to rock. I just don't relish spending weeks of my time muxing and editing several hundred gigs of .TY files!


Originally posted by Fugg
I think I do...

From my dtivo running 2.5.1,
I use Tytool to extract.
I use vsplit13c to process.
I use Tmpgenc to simple mux.

I never use any offset value to "re-sync". I just mux 'em as vsplit spits 'em out.

.....

Am I just the lucky one? Did I simply luck into "The Setup That Works"???!??!?!

or am i just hosed and missed the point entirely?

jdiner
10-01-2002, 11:56 PM
Ok. Here is what I have discovered and here is what is going on.

Mux'ing is most definately possible. But VBR streams are as near as I can tell seriously under-documented. I have thought about buying the latest and greatest docs from the ISO group but at $145 for just the one file I want to try and make sure that it will cover what I am looking for first.

As near as I can tell almost everyone does VBR mux'ing wrong. And here is why...

Let's start first with a description of mux'ing and how mpeg-2 works as a system, i.e. Program Stream. There are 0 or more Video Elementary Streams and 0 or more Audio Elementary streams. Each of these streams has a series of PTS/DTS values associated with it. It should be decoded at a certain time and should "played", i.e. presented, at a certain time. For audio there is only a presentation timestamp. For video there is both for an I or P frame and only a PTS timestamp for a B frame. The reason for this in video is the "this frame is based off of these other frames" method of compression. Just trust me that this is what is going on and that it is both needed and works.

I figured out a long time ago how to create these PTS and DTS values for both audio and video. To be quite honest it was simple once I stepped back and started thinking about it.

For audio you get a PTS every 24ms or 32ms or... Basically I just lift these values from the TyStream. You should at some point have noticed that at the top of the output from VSplit/TyTool is "stream" information. One of the fields is what this timestamp offset is. So I just use it and all is well. Infact I got a bit "better" for Dolby Digital and I calculate each frame off of each audio packet, since they change size and therefore duration.

For video you calculate them off of 29.97 FPS for NTSC and 25 FPS for PAL. Again this is simple arithmatic and is ease to generate. A wide variety of tests both by myself and others on the forum have shown that this part works.

Now we get to the last past. And this is where things are still... wonky. Mostly because I can find no clear definition of how to do it correctly to match the MPEG spec...

Each of these streams works on a different base clock. You have all seen this. Think about it... When you VSplit the TyStream file you get seperate audio and video files. Then VSplit dumps out at the end a print out of "how far off" these 2 are from each other. Sometimes it is 0ms, sometimes it is 1-4 and sometimes it is 16-17ms off. Each clock, or timestamp starting point, is completely valid because they are completely seperate from each other. If you have 2 audio streams it would be perfectly legal to have one and -16ms from the video and one at 16ms...

So... at long last the key is HOW??? does mpeg-2 really synchronize them. Rather than lining them all up to one of the Elementary Streams (ES) it lines them up to a non-stream releated clock, the System_Clock_Reference (SCR). This SCR value is really a measure of how many bytes have gone through the system based off of some simple calculations. Simple at least for a CBR system. As my sister would say... "Let me break it down for you Fred..."

The SCR is based off of an expected data rate.



mux_rate = (video data rate +audio data rate)(1+packet header +pack header * packs/packet)/data

20 + 12/3
mux_rate = (150 000 + 24 000)(1 + -------------- )
2 028

mux_rate = 176 059 bytes/sec


This mux_rate value is quite literally how many bytes should go by during a second of playback. 150,000 bytes of video data, 24,000 bytes of audio data, and a 20 byte pack header with a PES packet header, and so on. Giving a grand total of 176,059 bytes per second.

This value used with the number of bytes that have "gone past" are used to generate the SCR value put in each PACK header. For CBR data this is correct. I built a counting tool and each pack is off by no more than about a 1000 bytes in either direction. So the timestamps line up nicely.

Now using that same counting tool, I ran a bunch of VBR streams through it. Average data rates... Anywhere from 90,000 bytes/sec to 700,000 bytes/sec...

I hope you begin to see the problem. Things just don't line up. After having observed this the one comment I have seen that applies to all of this is... "Using VBR things are no longer linear". I believe this came from the bbmpeg source itself.

contiuned....

jdiner
10-01-2002, 11:56 PM
Continued here...

Now if they are not linear how and when do you determine that the SCR value need to be changed or set and then set to what? I don't know at present.

One thing I can tell you is that if it is wrong it causes jerky playback. And the reason why is obvious. If the central clock counts from 1 to 2 ... to 5 seconds but the PTS value for the next video frame is at 4 seconds, it is skipped, in favour of catching up with the next frame. And vice versa... If you are at 6 seconds for the PTS value and the SCR is at 5 it is "held" until you catch up. Slight milisecond fluctuations are common during playback and not a problem. But in timing things I have occasionally seen them be off by over a second. Very noticable... :)

So all we need to be done with mux'ing is to figure out how to generate a proper SCR value from the VBR data.

So your mission should you choose to accept it, is look for docs newer than mine (ISO 13818-1 for me is from Nov. 1994 and has nothing about VBR in it), or a book that describes it... Or (fill in the blank here...) that will give me some clue.

Source is not a bad thing. But the problem is that it needs to be working source. My own testing has shown that mplex. bbmpeg, etc... simply aren't doing it right. Some codecs play them back but in looking into it and reading up on groups.google.com many PC codecs ignore the SCR and just sync audio to the video. This is easier. Much much easier but again from what I have read most likely won't work on a standalone player that is really doing it right.

--jdiner

jdiner
10-02-2002, 12:00 AM
I supose I should have pointed out the rest of the SCR formula. It is basically:



scr_increment = #byte_gone_past * mux_rate * clock_freq / 50


Thus if 2048 bytes have gone by you get a fraction of a second increment. If the full 170k or 300k or 700k or whatever has gone by then you see a full second increment in the SCR.

When in CBR mode this is simple and straight forward. But in VBR... it is not. I have tried a few brute for methods and have thought about a few more ways.... Basically pool up a full second of video, however much it is, and then write out packs that use the right mux_rate values for that block.

But as near as I can tell this value can and should be calculatable. i.e. some formula is used to spit things out. But who knows... I am open to any and all suggestions.

Hopefully this description helps people in some fashion. I have what I think is a solid handle on mux'ing and how and why it is done. But I am definately missing this one last piece... However if you are certain I am wrong about something then please let me know, and I will refine things as needed.

--jdiner

laserfan
10-02-2002, 12:47 AM
Well, I for one think you should concentrate on the "known quantity" first, i.e. CBR SA Tivo streams, to get a solid base to work from, and to prove that muxing can work. But then maybe I'm prejudiced... ;) (how to erase my sig on-the-fly?)