PDA

View Full Version : tyremux a GPLed TyStream demuxer and to be remuxer



Pages : [1] 2 3

olaf_sc
12-04-2002, 04:02 AM
tydemux/demux - the future and present

Demuxing, transcoding and remuxing and external muxing

This is to be a place where we can all trade ideas, discuss the tydemux code base (and the up comming tyremux) and MPEG coding in general. It's GPLed application so you are all welcomed to partisipate/develop or just look at the code in general. Let it inspire so that ideas, code and every thing flows freely.

NOTE: There is however some rules for this thread. The rules are for this thread, not the forum. If you want to discuss things deemed off limits here make your own threads and set your own rules. But if you decide to participate here then please follow the guidelines.

The rules for this thread:

1- Leave the flames and arguments elsewhere... Again I am interested in getting things working better and moving forward

2- While we discuss any tools dealing with mux'ing or viewing or editing MPEG2 files or TyStreams... but lets keep the focus on tydemux/remux and things that are that are built for TiVo.

3- No discussion of extraction. This thread is about demuxing, remuxing, transcoding and external muxing of Tivo TyStreams files to MPEG-2.

4- Feel free to discuss various OS wishes/desires. At some point we should support all of them or at least as many as is possible. However be moderate and if you can code please help out - download the source code and send patches.

5- It's a technical/coding and to some extent usage thread - hence if you want to just talk in general think twice before posting.

6- Don't post applications without the source, this thread is following the Open Source guide lines.

Cheers Olaf

olaf_sc
12-04-2002, 04:08 AM
LATEST VERSION IS 0.4.2

Get it here (http://www.dealdatabase.com/forum/showthread.php?s=&postid=79714&highlight=tydemuxW0.4.2.zip#post79714).


It's time for a new release of tydemux - I hope you will find it useful!

It's a nearly complete rewrite and it should be much more robust now. It also has a lot more functionality that I think you will find very handy.

The current features of tydemux is:
---------------------------------------

Extraction of video and audio (both MPEG and AC3/a52) from all Tivo types made - SA Tivo Series 1, SA TIVO Series 2, DTivo Series 1 and UK Tivo - using software revision 1.3 and up. NOTE: DTivo Series 2 should in principle be supported but I have not yet be able to obtain a TyStream from it.

Tivo Type/Series probe you don't need to worry about what type your TyStream is. tydemux will detect it and use the right engine to demux the audio and video.

Audio probe, you don't need to worry about what type of audio a recordings has. tydemux will probe the whole TyStream, determine audio type and skip to the first chunk that has that type of audio.

TyStream will repair the TyStream if it detects holes/gaps in the TyStream. Up on detection of a hole/gap tydemux it will align video and audio in such way that there is no or very ittle loss of sync. Further more tydemux will fix the start and end of the hole/gap in such way that Video and Audio doesn't deteriorate in quality.

tydemux is minimizing the Audio/Video sync offset. At the end of the execution it will tell you the offset in milli seconds.


The tydemux_W_0.3.0.zip holds Windows binaries and sources/project files from MS Visual Studio 6.x.

The tydemux_L_0.3.0.zip holds Linux_x86 binaries and source/make files for gcc.

While both has the exact same codebase I still think it's easier to post to different packages.

NOTE: I'm using the latest as in CVS version of mplex from mjpeg-tools as a reference muxer for my streams.

Mplayer is my std mpeg video and mpeg program stream player. While I use mpg123 as reference player for mpeg audio files and a52dec as reference player for ac3 audio files.


Cheers Olaf

olaf_sc
12-04-2002, 04:09 AM
Linux binaries and sources

olaf_sc
12-04-2002, 04:12 AM
To make this thread as useful as possible we need documentation. Hence I have gatherered some docs that I hope you will find useful. It's MPEG, AC3 and TyStream documentation. The latter one is a write in progress by me in StarOffice and Word format. It's currently describing the TyStream syntax.


tystream.zip == tystream docs
mpeg-1.zip == MPEG-1 part 1, 2 and 3
mpeg-2.zip == MPEG-2 part , 2, 3 and 4
a52.zip == AC3 specs

olaf_sc
12-04-2002, 04:13 AM
MPEG-1 part 1, 2 and 3

olaf_sc
12-04-2002, 04:14 AM
MPEG-2 part 1, 2, 3 and 4

olaf_sc
12-04-2002, 04:15 AM
AC3 specs

racingclub
12-04-2002, 07:37 AM
I've tried your Windows Binary with a ty stream extracted from a UK SA TiVo.

With a 2.5gb stream I get the error as soon as I run the commandline - 'TyStream is to small to demux We need at least 40 chunks'

With a 400mb stream the process gets so far as to start the demux process but then crashes with a Windows GPF

olaf_sc
12-04-2002, 11:39 AM
Hello


Originally posted by racingclub
I've tried your Windows Binary with a ty stream extracted from a UK SA TiVo.

With a 2.5gb stream I get the error as soon as I run the commandline - 'TyStream is to small to demux We need at least 40 chunks'


tydemux doesn't yet support filesizes over 2GB, it's pretty trivial but I thought we better get it working 100% before we enable it.



With a 400mb stream the process gets so far as to start the demux process but then crashes with a Windows GPF

It would be really nice if you could upload this stream to my ftp server.
ftp://66.121.15.34/UK/ so I can see why it makes tydemux crash.

I have been testing a lot of streams but the fact is that I only have a few very small UK streams.


Cheers Olof

laserfan
12-04-2002, 11:44 AM
I tried this quickly on a 340Mb ty this a.m. and while I only did a demux, both the audio and the video seemed to play OK separately. My Win Media Player crashed right away on the video, but maybe I have an old version (7.01). But it appeared to play fine w/CyberDVD though there was a strange loop it went into at the end (it didn't just stop playing, it looped around the last few frames continuously).

Anyway I will play some more as it worked pretty well for me at least on the surface.

US SA Tivo v3.0

I will upload a file to your ftp site if you want one this big? It's a 1/2 hour program.

olaf_sc
12-04-2002, 11:54 AM
Hello

Thanks for your reply, please go a head and upload it to ftp://66.121.15.34/SA_3x/ so I can see what is going wrong. It might be a Windows only error. I'm doing more or less every thing under Linux so I need bug testing under Windows.

Anyways I will try to solve this problem and the one with the UK stream as quickly as possible.

Cheers Olaf




Originally posted by laserfan
I tried this quickly on a 340Mb ty this a.m. and while I only did a demux, both the audio and the video seemed to play OK separately. My Win Media Player crashed right away on the video, but maybe I have an old version (7.01). But it appeared to play fine w/CyberDVD though there was a strange loop it went into at the end (it didn't just stop playing, it looped around the last few frames continuously).

Anyway I will play some more as it worked pretty well for me at least on the surface.

US SA Tivo v3.0

I will upload a file to your ftp site if you want one this big? It's a 1/2 hour program.

AlphaWolf
12-04-2002, 12:09 PM
Olaf_SC: I would like to see converting an mpeg file to a tystream in the far future todo list, as this would be the other step necessary to be able to put whatever you want into your tivo.

Of course there would probably need to be a better understanding of tystream files first, so this would probably be a post 1.0 feature :)

racingclub
12-04-2002, 12:19 PM
It would be really nice if you could upload this stream to my ftp server.

OK Olof - am uploading 'm.ty' to your UK dir :)

If I can be of help by providing more UK streams then I will more than happy to up them to you. :)

Keep up the great work!

olaf_sc
12-04-2002, 02:18 PM
Hello,

Thanks for your feedback - the main problem coverting a mpeg to a TyStream is that we don't have all the data about the master chunk (Chunk 0) and that we don't understand the all of the data present in the record header.

If you have any further knowledge about the subject I very much welcome you to edit the TyStream document - it's currently doesn't cover the master chunk.

Cheers Olaf

PS: Yes, if I get the knowledge needed I will definitly implement it :)


Originally posted by AlphaWolf
Olaf_SC: I would like to see converting an mpeg file to a tystream in the far future todo list, as this would be the other step necessary to be able to put whatever you want into your tivo.

Of course there would probably need to be a better understanding of tystream files first, so this would probably be a post 1.0 feature :)

olaf_sc
12-04-2002, 02:24 PM
Hello

Thanks and YES, please - I definitly need more UK streams!! Please go a head and upload the once you have or create new once :).

There is currently 19GB of free space in the ftp area so I don't think we will run out of space at once :).

Cheers Olaf


Originally posted by racingclub
OK Olof - am uploading 'm.ty' to your UK dir :)

If I can be of help by providing more UK streams then I will more than happy to up them to you. :)

Keep up the great work!

artships
12-04-2002, 02:47 PM
Hey, Olaf!

Glad to see this tool right about now. For the last two weeks everything I've extracted from my SA tivo vsplits just fine, but the resulting audio contains stuff that makes TMPGEnc act like it's suffered a buffer overflow problem - Miscellaneous errors that don't make sense and occasionally even cause a TMPGEnc window to become an immovable black square on the screen. I think it's the audio because it also causes toolame and ssrc (optionally invoked from within TMPGEnc) to abort with something like "exception error".

I very much look forward to seeing how tydemux copes with my thus afflicted Andromeda episodes. And to seeing how they take to editing in TMPGEnc, too. I'll edit this with my findings tonight.

See results a few posts down.

John

olaf_sc
12-04-2002, 03:19 PM
Hi,

Thanks for your support - it would be very nice if you upload one of those streams to my ftp server ftp://66.121.15.34/SA_3x/

This is no matter if it works or not, since it can be a particular way your Tivo is encoding this and it can be nice to have a look at the TyStream.


Cheers Olaf


Originally posted by artships
Hey, Olaf!

Glad to see this tool right about now. For the last two weeks everything I've extracted from my SA tivo vsplits just fine, but the resulting audio contains stuff that makes TMPGEnc act like it's suffered a buffer overflow problem - Miscellaneous errors that don't make sense and occasionally even cause a TMPGEnc window to become an immovable black square on the screen. I think it's the audio because it also causes toolame and ssrc (optionally invoked from within TMPGEnc) to abort with something like "exception error".

I very much look forward to seeing how tydemux copes with my thus afflicted Andromeda episodes. And to seeing how they take to editing in TMPGEnc, too. I'll edit this with my findings tonight.

John

FreydNot
12-04-2002, 04:39 PM
My one and only short term feature request is inline audio frequency resampeling. This is a very important step in getting from an SA tystream (32k audio) to DVD compliant (48k audio).

Since the tystreams are very sensitive to being de-muxed and re-muxed, its hard to convert the audio externaly while keeping the sync.

olaf_sc
12-04-2002, 04:51 PM
Hello FreydNot

It's on the Todo list for 0.4.0 (see todo list in the README). I allready had this working in an internal version of tydemux 0.2.1.

When 0.4.0 hits the streets you will both have internal remuxing and internal audio transcoding. The audio transcoding will handle all types of trancodings - e.g. AC3 to MPEG, MPEG to AC3, 32k to 48k, 48k to 44.1k etc...

Cheers Olaf


Originally posted by FreydNot
My one and only short term feature request is inline audio frequency resampeling. This is a very important step in getting from an SA tystream (32k audio) to DVD compliant (48k audio).

Since the tystreams are very sensitive to being de-muxed and re-muxed, its hard to convert the audio externaly while keeping the sync.

AlphaWolf
12-04-2002, 04:52 PM
olaf_sc: sorry, I am not very advanced when it comes to all of this (my only knowledge about computers comes from tinkering I do, I haven't taken any programming courses or whatnot ever, in fact I am actualy working on a law degree :) ), so I can't help you much other than to come up with ideas. That said, I did have a thought, since its possible to extract the master chunk from MFS while the recording is in progress, wouldn't it help you guys further understand the master chunk by logging the changes to it every minute or so during the recording process? Or something like that?

olaf_sc
12-04-2002, 07:41 PM
Hello,

That would be a good idea (extracting it while it's recording) - what would be even better is if you could extract it each time a new chunk is created along with that chunk. Then we could see exactly how it comes together.

Cheers Olaf


Originally posted by AlphaWolf
olaf_sc: sorry, I am not very advanced when it comes to all of this (my only knowledge about computers comes from tinkering I do, I haven't taken any programming courses or whatnot ever, in fact I am actualy working on a law degree :) ), so I can't help you much other than to come up with ideas. That said, I did have a thought, since its possible to extract the master chunk from MFS while the recording is in progress, wouldn't it help you guys further understand the master chunk by logging the changes to it every minute or so during the recording process? Or something like that?

AlphaWolf
12-04-2002, 07:53 PM
hmm...you know, if you can get a PPC debugger of some sort that allows you to step through an already compiled programs operation, and use it on ele2pestripple, which creates tystreams from regular mpeg files, couldn't you do something like that?

olaf_sc
12-04-2002, 08:18 PM
Yes, actually been thinking about doing that - however when I coded tydemux 0.3.0 I simply didn't have the time. The master chunk was in my view not a essential for a sucessful demux/remux.

On the subject of converting mpeg to tystream, I must say it's something I will do if I get help with at least the knowledge - specs for the tyrecord and tystream master chunk. If not then it's definitly beyond version 1.0 of tydemux/remux.

Cheers Olaf




Originally posted by AlphaWolf
hmm...you know, if you can get a PPC debugger of some sort that allows you to step through an already compiled programs operation, and use it on ele2pestripple, which creates tystreams from regular mpeg files, couldn't you do something like that?

laserfan
12-04-2002, 09:56 PM
Originally posted by laserfan
I tried this quickly on a 340Mb ty this a.m...I had posted problems w/my SA version 1 stream, but I forgot the key issue: tydemux exited (crashed) at the end so I got no info about the audio offset. So the files appeared to demux OK but tydemux crashed.

I reported also a problem at the end w/the video part--I'm convinced it is a player problem only as it ran OK using DVD Station (w/my Hollywood Plus card).

I'm uploading my SA ty now & hope it gets up there OK as the connection is really slow.

olaf_sc
12-05-2002, 12:12 AM
Hello Folks

Found the bug that made tydemux core dump at the end of the TyStream. Basically if the TyStream wasn't properly extracted and the last chunk has parts of it missing. Tydemux countinued to read out in thin air, not good :)

Anyways it's corrected and I have prepared a new release I just wait for laserfan to finish his upload so I can make a last test with his file. If nothing bad happes I will release 0.3.1 by the end of the evening.

Cheers Olaf

artships
12-05-2002, 03:00 AM
Originally posted by olaf_sc Thanks for your support - it would be very nice if you upload one of those streams to my ftp server ftp://66.121.15.34/SA_3x/

This is no matter if it works or not
Alas, Olaf, it didn't work.
tydemux.exe -j 500 -i nw.ty -v nw2.m2v -a nw2.m2a > nw2.log resulted in a usage message.

tydemux.exe -i nw.ty -v nw2.m2v -a nw2.m2a resulted in, "Warning: Probe faild - specify audio type".

tydemux.exe -t2 -i nw.ty -v nw2.m2v -a nw2.m2a resulted in that grossly annoying WinXP message, "tydemux.exe has encountered a problem and needs to close."

Oh, and my copy bannered as "tydemux 0.2.1".

Running in an rxvt window courtesy cygwin and at a dos prompt, WinXP Home. I may have uploaded the first 450000 bytes of nw.ty - I can't tell, as a "150 directory listing" message is followed by "226 Tranfer done(but failed to open directory) (probably because I'm not supposed to be able to read anything there).

HTH

John

olaf_sc
12-05-2002, 03:05 AM
Hello

If it banner as 0.2.1 then you are using a old and out dated version of tydemux please download the latest version of tydemux.

You will find it in the second post from the start in this thread.

Cheers Olaf



Originally posted by artships
Originally posted by olaf_sc Thanks for your support - it would be very nice if you upload one of those streams to my ftp server ftp://66.121.15.34/SA_3x/

This is no matter if it works or not
Alas, Olaf, it didn't work.
tydemux.exe -j 500 -i nw.ty -v nw2.m2v -a nw2.m2a > nw2.log resulted in a usage message.

tydemux.exe -i nw.ty -v nw2.m2v -a nw2.m2a resulted in, "Warning: Probe faild - specify audio type".

tydemux.exe -t2 -i nw.ty -v nw2.m2v -a nw2.m2a resulted in that grossly annoying WinXP message, "tydemux.exe has encountered a problem and needs to close."

Oh, and my copy bannered as "tydemux 0.2.1".

Running in an rxvt window courtesy cygwin and at a dos prompt, WinXP Home. I may have uploaded the first 450000 bytes of nw.ty - I can't tell, as a "150 directory listing" message is followed by "226 Tranfer done(but failed to open directory) (probably because I'm not supposed to be able to read anything there).

HTH

John

artships
12-05-2002, 03:09 AM
Originally posted by olaf_sc
If it banner as 0.2.1 then you are using a old and out dated version of tydemux

Vexing, as that's what was in "tydemux_w_0.3.0.zip".

I uploaded part of the offending tystream to http://artships.com/nw.ty

Luck!

John

olaf_sc
12-05-2002, 03:49 AM
Hmm, strange I just downloaded the tydemux_W_0.3.0.zip and execed it.

The banner says :

tydemux 0.3.0 usage:
tydemux [option] -i infile -a outfile_audio -v outfile_video
[snip]

Another indication that you use a old version is that you use the -j switch which isn't there in 0.3.0. Please try to download it again. Before you do the test make sure that you don't have a the old version in your path.

Cheers Olaf

PS: The uploaded file is a bit to small I need around 6MB to test it. Although your upload to my ftp did go just fine - you aren't supposed to see whats in there :).



Originally posted by artships
Originally posted by olaf_sc
If it banner as 0.2.1 then you are using a old and out dated version of tydemux

Vexing, as that's what was in "tydemux_w_0.3.0.zip".

I uploaded part of the offending tystream to http://artships.com/nw.ty

Luck!

John

olaf_sc
12-05-2002, 03:56 AM
Hello Folks

- Tydemux 0.3.1 -

The example streams you send to me (thanks laserfan and racingclub) highlighted two minor bugs that can make tydemux crash.

1: Turncated chunks at the end of the TyStream
2: I was printing data from a released chunk hence I was reading data that was unavalible, not good.

Anyways it's fixed now and as usuall I will post two packages one for Windows and one for Linux.

Cheers Olaf

tydemux_W_0.3.1.zip == Windows
tydemux_L_0.3.1.zip == Linux

olaf_sc
12-05-2002, 03:57 AM
The Linux version

(By misstake I uploaded a version with extensive debug output it's corrected now - one person downloaded that version - please download the updated zip)

artships
12-05-2002, 01:49 PM
Originally posted by olaf_sc
Hmm, strange I just downloaded the tydemux_W_0.3.0.zip and execed it.

The banner says :

tydemux 0.3.0 usage:
Oops. You're right, it sure sounds like a PATH problem on my part. I should have checked that. I plead insanity, to which all the friends in my head can readily atest.

I'm looking forward to trying 0.3.1 tonight, Olaf. And so am I. And I'll upload a longer clip. Or, I will.

John

racingclub
12-05-2002, 02:21 PM
0.31 seems to work on streams that crashed 0.30 - the resulting files muxed up OK & kept in sync (something I've never managed with vsplit) - nice one Olof!!!!

now just need to remove the 2gb limit & up the release to 1.0? :)

olaf_sc
12-05-2002, 03:40 PM
Hello

Thanks for your (and others) feedback and support, the demuxing engine should be pretty solid in 0.3.x I habe been feeding it pretty nasty TyStreams that it still will manage to demux.

However we will most probably bump in to bugs that makes it crash. I'm talking about programming glitches not logical errors. I try to run tydemux in valgrind (a programing tool telling me when I do something ilegal) the problem is that it takes x10 as long to run tydemux in valgrind than stand alone. Hence I only run tydemux with valgrind on a half of my test streams.

The engine it self; does the demuxing by seeking start and end of all components, it will not assume that anything is aligned. Instead it takes the record type e.g. GOP header and does a seek for the start of it and a seek for the end of it. The in it extracts the the GOP header.

Doing it like this makes the engine much more rubust than a engine taking advantage of the fact that e.g. all video PES Headers in a TyStream are starting a byte zero in a TyRecord. See sometimes those assumtions aren't working and a "static" engine will fail. For those how are interested, take a look at the get_video.c file and you will see how my engine is working.

0.3.2 will be released shortly removing the 2GB limit on systems supporting it. However I will not jump to 1.0 :) it's still a lot of things to do.

I have started to code what will become tydemux/remux 0.4.0, the feature set planned for 0.4.0 is as follows:

Internal audio transcoding:
You will be able to transcode from AC3 to MPEG-II and vice versa. You will also be able to change the sample rate so it complies to either SVCD 44.1kHz or DVD 48kHz.

Internal remuxing:
Intrenal remuxing from TyStream to a std MPEG-2 program stream. This is the first step towards making DVD and SVCD compliant MPEG files. The MPEG file that 0.4.0 will make will play in any MPEG-2 player but it will not be something you can burn to a DVD or SVCD. tydemux/remux 0.5.0 will take care of that.

Cheers Olaf


Originally posted by racingclub
0.31 seems to work on streams that crashed 0.30 - the resulting files muxed up OK & kept in sync (something I've never managed with vsplit) - nice one Olof!!!!

now just need to remove the 2gb limit & up the release to 1.0? :)

fredisdead
12-05-2002, 05:38 PM
Olaf:

I tried the new version (Under windows xp) and had 2 problems.

1) Abnormal termination. After completing the split, the program crashed.

2) The split video portion of the ty file could not be imported into DVD Maestro due to a Temporal Reference Error, which as I understand is a backward time reference in a gop. Jdiner does not like these errors (see his posts) blaming them on 'shuffeled' frames from the dtivo. HOWEVER, when split with vsplit13c, there is no Temporal Reference error so I must assume it is either repaired by vsplit or introduced by tydemux. Perhaps when you pad out a hole, there are incorrect time stamps in the created frames?

Anyway, keep up the good work, I will try and ftp the file (VG 03x21 Before & After.ty 840mg) to your ftp site..

laserfan
12-05-2002, 07:01 PM
Originally posted by fredisdead
...The split video portion of the ty file could not be imported into DVD Maestro due to a Temporal Reference Error...I should add that I tried to import my split (SA) video file into SpruceUp (same mfr) and got the same error.

I like SU but at least jdiner (whom I respect of course) has called it a "steaming pile of crap" so maybe I need to be able to switch horses e.g. IfoEdit or something...

olaf_sc
12-05-2002, 07:26 PM
Hello

Thanks for your feedback, let us see what we can do with it :).


Originally posted by fredisdead
1) Abnormal termination. After completing the split, the program crashed.


First what version are you running? Version 0.3.0 has a bug making it crash at the end of the mux if the tystream has been abnormaly turncated.

If you are running 0.3.1 well then we have another bug :(. The good thing is that we find them, what I need from you is the last 6MB of your stream. The error has to do with the ending of the muxing. My ftp server is as always 66.121.15.34.



2) The split video portion of the ty file could not be imported into DVD Maestro due to a Temporal Reference Error, which as I understand is a backward time reference in a gop. Jdiner does not like these errors (see his posts) blaming them on 'shuffeled' frames from the dtivo. HOWEVER, when split with vsplit13c, there is no Temporal Reference error so I must assume it is either repaired by vsplit or introduced by tydemux. Perhaps when you pad out a hole, there are incorrect time stamps in the created frames?


Well I dunno if vsplit13c is fixing "Temporal Reference Errors", and I haven't read what jdiner has to say about it. Maybe you can give me specific URL to the post so I don't need to search around for it.

Anyways I guess that you mean when frames comes out of order or frames has been dropped. The whole thing pretty well described in this page (http://www.zvon.org/tmRFC/RFC2250/Output/chapter5.html). Frames comeing out order is actually leagal in the MPEG std (it's not in DVD). However droped frames are naturally not legal unless you make the cut it in the GOP boundaries and take care of it in the GOP header.

My guess is that Maestro DVD wants MPEG video that is complies to the DVD std. Hence it will complain about frames that comes in the wrong order.

Now I could include a Temporal Reference counter in tydemux and correct the problem when it happens. It's a little work but it not that hard. I.e. let me hack it over the weekend and you will have a working version on Monday.



Anyway, keep up the good work, I will try and ftp the file (VG 03x21 Before & After.ty 840mg) to your ftp site..

Yes, please it's a bit hard to fix the problem if I don't have example streams with it :). Hence I need a stream that is covering the part that has the "error".

Cheers Olaf

fredisdead
12-05-2002, 08:36 PM
Olaf:

I was using 0.3.0 so I guess that the abend is not an issue.

As to the Temporal Reference Error, Most of us look to burning the video to DVD.

IFOEDIT does not care about the errors, but sometimes loses synch. Maestro appears to be the most robust (never burned a coaster) tool available, Spruceup (which is a piece of S**T) if you can live through the constant crashes, actually uses Maestro's multiplexing engine (same author) and also chokes on the Error.

Anyway, correcting the Errors would be very useful to those of us burning DVD's.

Thx, will be watching for your next release....

olaf_sc
12-05-2002, 10:42 PM
Hello Folks,

Started coding the algo to preform the "temporal reference" check in all GOP present on in TyStream.

Now this problem is parsially my own fault - I'm by default making a closed GOP as the start of the video stream. This is how it supposed to be. However what I don't do is fix the temporal refernece in all the frames in that start GOP.


Now instead of just fixing that small error which is relativly easy I will instead fix the whole TyStream hopefully I will have something working either later (real late) today or tomorrow evening (if I don't go and see the new Harry Potter).

Cheers Olaf




Originally posted by fredisdead
Olaf:

I was using 0.3.0 so I guess that the abend is not an issue.

As to the Temporal Reference Error, Most of us look to burning the video to DVD.

IFOEDIT does not care about the errors, but sometimes loses synch. Maestro appears to be the most robust (never burned a coaster) tool available, Spruceup (which is a piece of S**T) if you can live through the constant crashes, actually uses Maestro's multiplexing engine (same author) and also chokes on the Error.

Anyway, correcting the Errors would be very useful to those of us burning DVD's.

Thx, will be watching for your next release....

laserfan
12-05-2002, 11:22 PM
Olaf, I have now tried v0.3.1 on another SA ty stream (1.3Gb music program) and it has impressively re-muxed w/full sync (I'm using MPEG-VCR 3.13 to mux & off set audio). This was on streams that refused to sync under vsplit.

A couple comments:

1. The audio offset reported by 0.3.1 has not been useful. The last stream I remuxed was reported to be 7ms early, and MPEG-VCR needed -230ms to achieve perfect sync. Note I've never found vsplit to be any good either FWIW

2. The remuxed program has frequent "freeze frame" (pauses) which seem to occur precisely at scene changes. That is, the video frame pauses for perhaps 4 frames or so, then resumes with a new scene. Audio remains in sync of course

I don't know if both of these issues are with MPEG-VCR muxing process and not the mpv and mpa that tydemux produces. In any case I am very impressed with what you have accomplished in what seems to be an extremely short period of time! Please keep up the great work! Looking forward to your remuxing version with audio resample.

Thanks Olaf!!!

laserfan
12-06-2002, 12:44 AM
And now I've tried re-muxing with IfoEdit, and it again maintains perfect sync to the end! The "end of scene" freezes that I experienced w/MPEG-VCR muxing are now Start of scene freezes (though fewer in frequency), so apparently this is a muxer issue of some sort.

But this is pretty darn good. Except that you are moving so fast w/updates, I might be inclined to burn some DVDs already (yes, I converted the audio to 48kHz using MPEG-VCR). But I am going to tinker some more instead for now.

olaf_sc
12-06-2002, 02:59 AM
Hello Laserfan,

Thanks for your feedback and encouragements - I have now implemented a algo for checking temporal reference. I'm continuing hacking along on a algo actually fixing the problem.

Anyways I think it's worth while to mention that there is a lot of things that makes a Tivo stream not DVD compliant besideds the obvious once.

E.g. it has more frames per GOP than allowed (seen up to 48 frames while DVD only allows 18/15 depening if it's NTSC or PAL). There is alot of those small details that makes DVD creation of TyStreams a bit hard.

My goal for 0.4.0 is to make internal muxing work but not having it comply to DVD. In 0.5.0 I will make it comply except for the fact that it will remain at it's present resolution and frame rate.

Ahh! It would be really nice if you could upload your test stream laserfan so I could take a look at those frame freez.

Cheers Olaf



Originally posted by laserfan
And now I've tried re-muxing with IfoEdit, and it again maintains perfect sync to the end! The "end of scene" freezes that I experienced w/MPEG-VCR muxing are now Start of scene freezes (though fewer in frequency), so apparently this is a muxer issue of some sort.

But this is pretty darn good. Except that you are moving so fast w/updates, I might be inclined to burn some DVDs already (yes, I converted the audio to 48kHz using MPEG-VCR). But I am going to tinker some more instead for now.

olaf_sc
12-06-2002, 03:47 AM
Hmm the algo used to detect temp ref error as described at http://www.zvon.org/tmRFC/RFC2250/Output/chapter5.html.
Is actually flawed bummer after implementing it but on the flip side I made my own - it avalible here (ftp://66.121.15.34/tydemux/temp_ref_algo.txt).

Anyways it will handle both closed and open GOPs and more funcy GOPs that e.g. has P-Frame B-Frame P-Frame instead of the more usuall P B B P.

Back to implementing it.

Cheers Olaf

racingclub
12-06-2002, 04:37 AM
Hi again.... :)

I tried 031 with a 30min show - the program executed fine and gave me the relevant audio & video files. Muxing these up with IfoEdit with the appropriate audio offset settings gave me a video thats starts off sync'd but seems to progressively lose audio sync over the course of the show (something I always seem to get from vsplit).

I have uploaded the file 'royle.ty' to your FTP.......

Cheers :D

BTW - I'm more interested in SVCD creation (rather than DVD).

artships
12-06-2002, 10:23 AM
Olaf,

I extracted an fsid from the middle of a Nero Wolfe episode. tydemux worked great, indicating a mere -001ms offset. Fed the resulting m2v to DVD2AVI, fed that d2v to TMPGEnc along with the m2a, cut the adverts from either end, transcoded/muxed it to DVD-standards, played the resulting mpg in PowerDVD.

Perfect. Thank you!

I'll try it with the whole episode and edit this with the results. Um, it sure would be handy to be able to do a whole hour show as one tystream, instead of breaking it in two to get around the 2G limit. Just a thought.

John

laserfan
12-06-2002, 11:31 AM
Originally posted by olaf_sc
...It would be really nice if you could upload your test stream laserfan so I could take a look at those frame freez...I can send you an mpg file with the frame freezing in it, but as I said it appears the freezing varies depending on what method I use to mux the mpv and mpa back together again. So I'm not sure how it will help you, but in any case if you want such a thing I am inclined to make a small program for you from scratch (versus the 1.3Gb file which would take a day or two to upload!), i.e. record a short SA Tivo program (a music video perhaps), apply tydemux to it, remake (remux) these files as an mpg file using any of a number of muxers I have here, observe that it shows the freeze frame problem, and send it to you. Is this what you want?

As I re-read my post, I would probably send you both the original ty and the new mpg file?

olaf_sc
12-06-2002, 12:18 PM
Hello

Yes, a 1.3G is a bit to big to send if you have a dedicated pip :). It would be really nice if you could send me a small exampel (i.e a smaller recording) as you say.

Cheers Olaf


Originally posted by laserfan
I can send you an mpg file with the frame freezing in it, but as I said it appears the freezing varies depending on what method I use to mux the mpv and mpa back together again. So I'm not sure how it will help you, but in any case if you want such a thing I am inclined to make a small program for you from scratch (versus the 1.3Gb file which would take a day or two to upload!), i.e. record a short SA Tivo program (a music video perhaps), apply tydemux to it, remake (remux) these files as an mpg file using any of a number of muxers I have here, observe that it shows the freeze frame problem, and send it to you. Is this what you want?

As I re-read my post, I would probably send you both the original ty and the new mpg file?

olaf_sc
12-06-2002, 02:08 PM
Hello

Thanks for your feedback and also for uploading the file. I will check it out later today.

Why loosing sync gradually? Well sync is dependant on relationship between specified framerate and actual framerate. If e.g. the specified one is 30/s and the actual is 31/s well then you will gradually loose sync since the muxer will count frames when it does the mux.

There is two ways to get around this problem - either we do an internal mux or we tweak the mpeg stream to actually specified framerate (there is several ways to do that).

Anyways I will as said before make a internal muxer but I will also thinking about fixing the framerate.

As for SVCD yes 0.5.0 will have both SVCD and DVD support.

Cheers Olaf



Originally posted by racingclub
Hi again.... :)

I tried 031 with a 30min show - the program executed fine and gave me the relevant audio & video files. Muxing these up with IfoEdit with the appropriate audio offset settings gave me a video thats starts off sync'd but seems to progressively lose audio sync over the course of the show (something I always seem to get from vsplit).

I have uploaded the file 'royle.ty' to your FTP.......

Cheers :D

BTW - I'm more interested in SVCD creation (rather than DVD).

olaf_sc
12-06-2002, 02:12 PM
Hello John,

Thanks for your feedback I'm glad that it works so fine for you.

I will remove the 2GB limit in 0.3.2 but before I release 0.3.2 I want to implment a temporal reference check and fix functionality so we don't get those pesky Temporal Reference errors.

Cheers Olaf

PS: If it wasn't for the fact that the inital check algo was flawed it would been released this evening now it looks more like Saturday.


Originally posted by artships
Olaf,

I extracted an fsid from the middle of a Nero Wolfe episode. tydemux worked great, indicating a mere -001ms offset. Fed the resulting m2v to DVD2AVI, fed that d2v to TMPGEnc along with the m2a, cut the adverts from either end, transcoded/muxed it to DVD-standards, played the resulting mpg in PowerDVD.

Perfect. Thank you!

I'll try it with the whole episode and edit this with the results. Um, it sure would be handy to be able to do a whole hour show as one tystream, instead of breaking it in two to get around the 2G limit. Just a thought.

John

artships
12-06-2002, 03:55 PM
Olaf,

The first half of a Nero Wolfe episode has, I'm guessing, some bad data in it. This is the kind of bad data I've been seeing the last two weeks with stuff that's been vsplited, too.

Edit:
The problem I think was that I was extracting using tserver_mfs3, instead of tserver_mfs5 with the lower scheduling priority, consequently, the tystream was getting intermittant errors in it.

I fetched a fresh tystream. There's still a glitch in it, but a subtle one. About a third of the way into the show, it starts losing sync. The offset grows as the show progresses. I don't see the sync problem until after I burn it to DVD. PowerDVD plays each section between adverts (separate files) just fine. Muxing with TMPGEnc, the resulting file plays just fine. Ulead DVD Movie Factory made VOBs that have the sync drift.

Why this process started messing-up two weeks ago I've no idea, but it is very, very annoying. All of a sudden tivo-to-dvd has stopped being an automatic. Major disappointment.

TMGEnc uses either it's own audio conversion routines, or toolame. At point 39000 (of 69000) the audio contains something that kills toolame. I fixed it by running it through winamp and creating a wav file, and feeding that to TMPGEnc and toolame.

I can't fix the video error:
Read error occured at address 1000636B of modle 'DVD2AVI.vfp' with 01faaf3c

It occured 2.21 minutes into the show (which would be 3 or 4 minutes into the tystream). I'm uploading to your site emmm.ty

Edit:
To get you a too-small clip took my modem two hours, so I doubt I'll be able to send you the offending stream.

olaf_sc
12-06-2002, 04:25 PM
Hello John,

Will take a look at the TyStream as soon as your upload is finished. It will be interesting to see what is really going on.

Cheers Olaf



Originally posted by artships
Olaf,

The first half of a Nero Wolfe episode has, I'm guessing, some bad data in it. This is the kind of bad data I've been seeing the last two weeks with stuff that's been vsplited, too.

TMGEnc uses either it's own audio conversion routines, or toolame. At point 39000 (of 69000) the audio contains something that kills toolame. I fixed it by running it through winamp and creating a wav file, and feeding that to TMPGEnc and toolame.

I can't fix the video error:
Read error occured at address 1000636B of modle 'DVD2AVI.vfp' with 01faaf3c

It occured 2.21 minutes into the show (which would be 3 or 4 minutes into the tystream). I'm uploading to your site emmm.ty

Sorry!

John

olaf_sc
12-06-2002, 06:17 PM
Hi Riley,

Thanks for your post and also for the intersting observation. Please see my inline comments.



Originally posted by rc3105
I have serveral clips that have intact audio but missing video. (my tivo compiler runs natively in the tivo & video gets dropped because of excessive disk activity when I'm compiling) same sort of thing happens with the original (slow) quantum with some scripts on db opens or even a fast drive with excessive insert/extract activity

these clips split/mux fine until they hit one of these dropouts, then the sync is lost. audio is continous but video gets shifted (vsplit has the same problems only more pronounced)

A--A--A--A--A--A--A-- --A--A--A--A-- --A--A--A--A--A--
V--V--V--V--V--V-- --V-- --V-- -- -- -- --V--V--V--V--

becomes

A--A--A--A--A--A--A--A--A--A--A--A--A--A--A--A--
V--V--V--V--V--V--V--V--V--V--V--V--


Hmm, so what you are saying is that Tivo is dropping frames internally on high loads. In other words the TyPackets (Chunks) are okay but the content isn't. The sync from one TyPacket to another is also okay/within bounderies. What needs to be done is to check if all frames are present in the mpeg stream that we demux/extract.

Anyhow my new temporal reference check will detect those kind missing frames. The problem is what to do about it.

Normally when we have a error in the temporal reference we can seach for the correct frame and correct the order of the frame.

Say that the frames are comming like this (very simplyfied*).

1 2 3 4 5 6 9 7 8 10
| error

All what is needed here is to search for frame 7 and when you find it (in this case the next frame) you just shift them around. Sure there is more effective algos how to do this but I want to compicate it more than nessesary.

Anyhow the probelm with missing frames is that they are missing :).

1 2 3 5 6 7 8 9 10

Now we can either throw away the whole GOP and realign the audio - this natrually a very ugly solution but effective.

Now if we have a MPEG stream like this (M == missing frame)

I B B P B M P B B P ......

then I can just copy the B frame next to the missing B frame (make some small adjust to the header of the picture) and voiala we are home free.

The problem is when we have a P frame or several B frames missing.

* If a P frame is missing I need to recreate it or do the ugly one where I throw away to whole GOP.
* If all B frames are missing between P frames then I can use P frames to cover the gap.

What ever we do will make the video a bit funky (repeated frames) but it's probably better than audio out of sync.



--A--A--A--A--
--V--V--V--V--

stays

--A--A--A--A--
--V--V--V--V--

as much as I hate to suggest discarding packets, is it feasable to continouisly verify sync & discard/resync if necessary by comparing timestamps? I know some packets arrive out of order but if it's off by more than a few chunks or outside the gop it I doubt it's ever going to arrive.


It's actually not such a good idea to check timestamps in this case it's easier and faster to check the temporal reference. However this is naturally dependant on the fact that that the temporal reference is written correctrly by Tivo. Hence that tivo when skipping frames under heavy load is updating the temp ref but not writing the frame.

We will see if it's like that when I get your segments - Ahh, yes when you upload please make each segment 6MB or more. tydemux need that amount of space to probe the tystream.

Cheers Olaf

* Yes, mpeg frames doesn't come in that order but like this (open gop) 2 0 1 5 3 4 8 6 7 .......

olaf_sc
12-06-2002, 09:12 PM
Hello John,

I tested your tystream emmm.ty but it doesn't hold the Nero Wolfe, just a preview.

Both demuxed video and audio is normal in the TyStream you sent me. I think I need more of your stream so I get at least to the beginning of your problem.

Cheers Olaf



Originally posted by artships
Olaf,

The first half of a Nero Wolfe episode has, I'm guessing, some bad data in It occured 2.21 minutes into the show (which would be 3 or 4 minutes into the tystream). I'm uploading to your site emmm.ty
[snip]
Sorry!

John

du1jec
12-07-2002, 02:20 AM
Olaf,

I attempted to compile tydemux in MacOS 10.2 (Jaguar) and here is what I got. It gives me a compiled binary, but when I try to run it against a tyfile I get bus error.

See below for compiler output and subsequent error.

Thanks,

Jojo


[du1jec-powermac:~/Documents/tydemux] du1jec% gcc -v
Reading specs from /usr/libexec/gcc/darwin/ppc/3.1/specs
Thread model: posix
Apple Computer, Inc. GCC version 1151, based on gcc version 3.1 20020420 (prerelease)

[du1jec-powermac:~/Documents/tydemux] du1jec% make all
rm -f *.o
gcc -g -Wall -W -c check_audio_sync.c -o check_audio_sync.o
gcc -g -Wall -W -c find_start_code_offset.c -o find_start_code_offset.o
find_start_code_offset.c: In function `find_start_code_offset':
find_start_code_offset.c:55: warning: implicit declaration of function `memcpy'
find_start_code_offset.c:133: warning: implicit declaration of function `memcmp'
gcc -g -Wall -W -c get_audio.c -o get_audio.o
get_audio.c: In function `get_audio':
get_audio.c:163: warning: implicit declaration of function `memcpy'
gcc -g -Wall -W -c get_next_record.c -o get_next_record.o
get_next_record.c: In function `get_next_record':
get_next_record.c:59: warning: implicit declaration of function `memcpy'
gcc -g -Wall -W -c get_time_X.c -o get_time_X.o
get_time_X.c: In function `get_time_X':
get_time_X.c:131: warning: implicit declaration of function `memcpy'
gcc -g -Wall -W -c get_video.c -o get_video.o
get_video.c: In function `get_video':
get_video.c:190: warning: implicit declaration of function `memcpy'
gcc -g -Wall -W -c chunk.c -o chunk.o
gcc -g -Wall -W -c check_chunk.c -o check_chunk.o
gcc -g -Wall -W -c check_junk_chunks.c -o check_junk_chunks.o
gcc -g -Wall -W -c probe_audio.c -o probe_audio.o
probe_audio.c: In function `copy_chunk':
probe_audio.c:60: warning: implicit declaration of function `memcpy'
gcc -g -Wall -W -c misc.c -o misc.o
gcc -g -Wall -W -c tystream.c -o tystream.o
gcc -g -Wall -W -c pes_holder.c -o pes_holder.o
gcc -g -Wall -W -c parse_chunk.c -o parse_chunk.o
parse_chunk.c: In function `parse_chunk_video_remainder_S1':
parse_chunk.c:152: warning: implicit declaration of function `memcpy'
gcc -g -Wall -W -c tydemux.c -o tydemux.o
gcc -g -Wall -W -c tystream_repair.c -o tystream_repair.o
tystream_repair.c: In function `repair_audio':
tystream_repair.c:228: warning: unused parameter `audio_times'
gcc -g -Wall -W -c mpeg.c -o mpeg.o
gcc -g -Wall -W -c tystream_init.c -o tystream_init.o
gcc -o tydemux check_audio_sync.o find_start_code_offset.o get_audio.o get_next_record.o get_time_X.o get_video.o chunk.o check_chunk.o check_junk_chunks.o probe_audio.o misc.o tystream.o pes_holder.o parse_chunk.o tydemux.o tystream_repair.o mpeg.o tystream_init.o

[du1jec-powermac:~/Documents/tydemux] du1jec% ./tydemux -i ../mpeg/808865-Andromeda-Slipfighter_the_Dogs_of_War.ty -a ../mpeg/slip.mpa -v ../mpeg/slip.mpv
Probing TyStream .....
Bus error

pbar
12-07-2002, 05:30 AM
Originally posted by rc3105
Olaf:
I have serveral clips that have intact audio but missing video. these clips split/mux fine until they hit one of these dropouts, then the sync is lost. audio is continous but video gets shifted (vsplit has the same problems only more pronounced)

A--A--A--A--A--A--A-- --A--A--A--A-- --A--A--A--A--A--
V--V--V--V--V--V-- --V-- --V-- -- -- -- --V--V--V--V--

becomes

A--A--A--A--A--A--A--A--A--A--A--A--A--A--A--A--
V--V--V--V--V--V--V--V--V--V--V--V--

I have this problem with pretty much every recording from my UK TiVo ... even when my TiVo isn't doing anything else! A graph of the video frame number (i.e. count of MPEG2 PICTURE start codes) against TiVo video PTS drifts by about a second over a 1-hour recording! This makes both vsplit and tydemux unusable for me.

Even if I use vsplit -m to produce an MPEG with the correct A/V timestamps, it plays in sync on the PC, but as soon as you transcode or feed the file into a DVD authoring package the above problem happens and you end up losing sync again!

The only way I have found to work around this is to split the tystream into lots of small-ish chunks, demux them individually and add them as separate chapters when building a DVD.


Originally posted by rc3105
Olaf:
as much as I hate to suggest discarding packets, is it feasable to continouisly verify sync & discard/resync if necessary by comparing timestamps? I know some packets arrive out of order but if it's off by more than a few chunks or outside the gop it I doubt it's ever going to arrive.


Surely it's *much* easier to drop one or more audio segments to keep the timestamps approximately in step? Audio frames have no forwards or backwards dependencies, and a small 36ms glitch in the audio is way preferable to throwing away a 2-3 second GOP.

Ideally you'd like to drop only 40ms of audio samples (or 33.3ms for NTSC) - but I'm not sure how easy it is to change the size of an MPEG audio frame?

Now that there's a well-written open-source splitting tool, I've been thinking about having a go at coding this up (assuming I ever get the time) but I expect it wouldn't be a very big job for somebody familiar with the source?

This would make such a big difference to my TiVo experience that I'd be off to paypal like a shot!


Paul

olaf_sc
12-07-2002, 11:18 AM
Hello du1jec,

Great :) well great that you have a Mac I mean, porting it to Mac is pretty trivial.

What I need from you is what include files that memcmp and memcpy needs. Normally you get that by typing man memcpy and man memcmp. In the SYNOPSIS it will tell you what files to include.

Now don't think that just by including those files it will run just fine under Mac - the app is currently little endian only! However if you give me the include I will make the small changes that is needed to make it bigendian/littleendian independant in the next release.

Cheers Olaf



Originally posted by du1jec

[du1jec-powermac:~/Documents/tydemux] du1jec% gcc -v
find_start_code_offset.c:55: warning: implicit declaration of function `memcpy'
find_start_code_offset.c:133: warning: implicit declaration of function `memcmp'

du1jec
12-07-2002, 12:18 PM
Olaf,

Here are the man pages for memcmp and memcpy.

I have also attached the string.h include file. It is called stringh.txt so the board would take it.

Thanks,

Jojo


MEMCMP(3) System Library Functions Manual MEMCMP(3)

NAME
memcmp - compare byte string

LIBRARY
Standard C Library (libc, -lc)


SYNOPSIS
#include <string.h>

int
memcmp(const void *b1, const void *b2, size_t len);

DESCRIPTION
The memcmp() function compares byte string b1 against byte string b2.
Both strings are assumed to be len bytes long.

RETURN VALUES
The memcmp() function returns zero if the two strings are identical, oth-
erwise returns the difference between the first two differing bytes
(treated as unsigned char values, so that `\200' is greater than `\0',
for example). Zero-length strings are always identical.

SEE ALSO
bcmp(3), strcasecmp(3), strcmp(3), strcoll(3), strxfrm(3)

STANDARDS
The memcmp() function conforms to ISO/IEC 9899:1990 (``ISO C89'').

BSD June 4, 1993 BSD

[du1jec-powermac:~/Documents/tydemux] du1jec% man memcpy
MEMCPY(3) System Library Functions Manual MEMCPY(3)

NAME
memcpy - copy byte string

LIBRARY
Standard C Library (libc, -lc)

SYNOPSIS
#include <string.h>

void *
memcpy(void *dst, const void *src, size_t len);

DESCRIPTION
The memcpy() function copies len bytes from string src to string dst.

RETURN VALUES
The memcpy() function returns the original value of dst.

SEE ALSO
bcopy(3), memccpy(3), memmove(3), strcpy(3)

STANDARDS
The memcpy() function conforms to ISO/IEC 9899:1990 (``ISO C89'').

BUGS
In this implementation memcpy() is implemented using bcopy(3), and there-
fore the strings may overlap. On other systems, copying overlapping
strings may produce surprises. A simpler solution is to not use
memcpy().

BCOPY(3) System Library Functions Manual BCOPY(3)

NAME
bcopy - copy byte string

LIBRARY
Standard C Library (libc, -lc)

SYNOPSIS
#include <string.h>

void
bcopy(const void *src, void *dst, size_t len);

DESCRIPTION
The bcopy() function copies len bytes from string src to string dst. The
two strings may overlap. If len is zero, no bytes are copied.

SEE ALSO
memccpy(3), memcpy(3), memmove(3), strcpy(3), strncpy(3)

HISTORY
A bcopy() function appeared in 4.2BSD.

BSD June 4, 1993 BSD

olaf_sc
12-07-2002, 12:33 PM
Hello Paul,


Originally posted by pbar
I have this problem with pretty much every recording from my UK TiVo ... even when my TiVo isn't doing anything else! A graph of the video frame number (i.e. count of MPEG2 PICTURE start codes) against TiVo video PTS drifts by about a second over a 1-hour recording! This makes both vsplit and tydemux unusable for me.

Hmm, not sure if I get you here. Let me assume that what you are talking about is: You count frames and give them a "virtual PTS" based on the frame rate specified in the video stream. Now you compare that to the actuall PTS given in the video stream and you find that they drift 1 second over an hour recording. Right?

Now this is how it supposed to be:) - well not but in the Tivo case it is like this due to the fact that they have a variable frame rate (not allowed in the MPEG std).



Even if I use vsplit -m to produce an MPEG with the correct A/V timestamps, it plays in sync on the PC, but as soon as you transcode or feed the file into a DVD authoring package the above problem happens and you end up losing sync again!


Naturally the whole idea with internal muxing is to force sync, by using the PTS present in the video/audio stream in the TyStream. vsplit -m doesn't do any magic to fix the video stream. Which means that when you trancode it (which includes a demux) you will end up with the same problem. Too few or too may frames per second compared to what the video stream specifies it to be.



The only way I have found to work around this is to split the tystream into lots of small-ish chunks, demux them individually and add them as separate chapters when building a DVD.


Ugh, bulky but workable work around. The thing is that the muxer is counting frames and it simply doesn't get enough of them to produce out of sync video/audio.



Surely it's *much* easier to drop one or more audio segments to keep the timestamps approximately in step? Audio frames have no forwards or backwards dependencies, and a small 36ms glitch in the audio is way preferable to throwing away a 2-3 second GOP.


Firstly what I assume RC3105 was talking about was when we actually had missing frames due to heavy load not that we have "missing"/"too many" frames due to variable frame rate.


Secondly this is actually easier to do in the video stream.

1: We need to count frames in the video stream in order to detect when we have under/over run in frames.

2: As soon as we detect a over/under run we can simply add or delete a B-Frame.

This is much simpler than seeking and removing audio frames.

Just for the sake of it I actually made an experiment this moring.

I removed every 7:th B-Frame and I added a B-Frame every 25:th B-Frame. An as I suspected the video clip is playing just perfectly

Well it's a bit jerky but considering that we removed such an amout of frames and added quite a bit of frames it's totally understandable.

The key is that B-Frames has bidirectional dependencies to the previous P-Frame and the P-Frame a head of it. But it doesn't care as long as if "finds" the P-Frames that it depends up on

I naturally need to check this in more detail (don't have that deep knowledge of the mpeg video stream - mostly been working with the program stream (muxed audio/video).

Now if I fix the frame rate problem which I intend to do. Then we will have a video/audio stream that will keep in sync even if you edit it, transcode it etc..

Cheers Olaf

pbar
12-07-2002, 01:19 PM
Olaf,

Thanks for the reply.


Originally posted by olaf_sc
Hmm, not sure if I get you here. Let me assume that what you are talking about is: You count frames and give them a "virtual PTS" based on the frame rate specified in the video stream. Now you compare that to the actuall PTS given in the video stream and you find that they drift 1 second over an hour recording. Right?

Yep, exactly. I was trying to work out if the sync drifted gradually or if there were 2 or 3 sudden jumps where video or audio was dropped.



Now this is how it supposed to be:) - well not but in the Tivo case it is like this due to the fact that they have a variable frame rate (not allowed in the MPEG std).

I thought this was just a DTivo issue... but I've written firmware for a motion-jpeg video/audio compression board before and it's easy to imagine numerous reasons why you might occasionally miss a frame-sync if a scene is a bit complex!



Naturally the whole idea with internal muxing is to force sync, by using the PTS present in the video/audio stream in the TyStream. vsplit -m doesn't do any magic to fix the video stream. Which means that when you trancode it (which includes a demux) you will end up with the same problem. Too few or too may frames per second compared to what the video stream specifies it to be.

True... but I'd been hoping (perhaps a bit naively) that any DVD authoring package which accepted MPEG2 transport streams directly would be capable of recoding the video frames without discarding the presentation timestamps. Certainly you'd expect to lose sync if you demuxed to audio and video elementary streams... ho hum. Spose most of the video tools out there are used to dealing with MPEG2 from DVDs :D



Ugh, bulky but workable work around. The thing is that the muxer is counting frames and it simply doesn't get enough of them to produce out of sync video/audio.

Not ideal... but necessity is the mother of invention.




... this is actually easier to do in the video stream.
I removed every 7:th B-Frame and I added a B-Frame every 25:th B-Frame. An as I suspected the video clip is playing just perfectly

I naturally need to check this in more detail (don't have that deep knowledge of the mpeg video stream - mostly been working with the program stream (muxed audio/video).

Now if I fix the frame rate problem which I intend to do. Then we will have a video/audio stream that will keep in sync even if you edit it, transcode it etc..


That'll do very nicely... :cool:

Inserting/deleting B frames has the big advantage that duplicate video frames are almost impossible to notice, whereas an audio glitch is sometimes quite irritating... just thought it seemed a lot more fiddly and 'stateful' than binning a single independent audio packet every so often.

Nice work, nice tool. Looking forward to cutting some nice DVDs!

Thanks for all the hard work,

Paul

OS X
12-07-2002, 09:47 PM
Originally posted by du1jec
Olaf,

I attempted to compile tydemux in MacOS 10.2 (Jaguar) and here is what I got. It gives me a compiled binary, but when I try to run it against a tyfile I get bus error.


Nice to hear there's another Mac OS X user playing with this stuff. I'm a newbie but so far I've managed to extract to my PC using Tytool5. My ultimate goal is to edit and burn commercial free DVD's with Final Cut Pro and DVD Studio Pro.

I've never tried to compile source. I'll be watching to see how this works out for you guys. Any suggestions for me to come up to speed to participate in getting this working on OS X would be appreciated.

Keep up the good work! :)

rd001
12-07-2002, 09:49 PM
Originally posted by racingclub

I tried 031 with a 30min show - the program executed fine and gave me the relevant audio & video files. Muxing these up with IfoEdit with the appropriate audio offset settings gave me a video thats starts off sync'd but seems to progressively lose audio sync over the course of the show (something I always seem to get from vsplit). You may have noticed that some programs played directly on the DTivo live have sync problems too.

DTV doesn't always maintain perfectly A/V. I've noticed it in the last few years. It even happens with early runs of hot PPVs (channels 100-110) where they typically allocate a lot of bandwidth but it does happen much less often with those.

I think that there is no way to make absolutely perfect A/V sync with the DTV streams as they are currently. You can get close enough that it isn't distracting.

I assume that you're talking about a drift problem that gets progressively worse. That is a different problem but I was trying to point out that it's unlikely that any muxing program is going to produce completely perfect A/V sync.

olaf_sc
12-08-2002, 12:49 AM
Hello Folks,

After doing some analyze of streams from DTivo 1* and I found some intersting facts.

Now as said before the frame rate is variable but that is not entierly true. A more correct term should be that it shifts.

MPEG specifies - seven std framerates. From a quick glance Tivo uses two 23.976 frames per sec and 29.97 frames per sec 2*.

Now MPEG "specifies" time in ticks and when it comes to the presentation time of a video frame then MPEG says that there will be 90 000 ticks per second.

If we combine the ticks with the frame rates we find that a 29.97 f/s will have 3003 ticks between each frame and 23.97 f/s will have 3753 ticks between each frame.

Now lets take a look at the actual diff between frames,


Starting demux process
.Checking tmp ref
First closed gop
SEQ Header Frame rate: 29.97 frames/sec
[snip 1]
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
SEQ Header Frame rate: 29.97 frames/sec
Diff is 3003
[snip 2]
Diff is 3003
Diff is 4505
Diff is 3003
Diff is 4504
Diff is 3003
Diff is 4505
Diff is 3003
Diff is 4504
Diff is 3003
[snip 3]
Diff is 4504
Diff is 3003
Diff is 4505
Diff is 3003
Diff is 4504
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
[snip-4]
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 4505
Diff is 3003
Diff is 4504
Diff is 3003
Diff is 4505
Diff is 3003
Diff is 4504
Diff is 3003
Diff is 4505
Diff is 3003
Diff is 4504
[snip -5]
Diff is 4505
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003
Diff is 3003

Demux process finished

A/V Sync Offset: -014ms (i.e. audio plays 014ms late use -O 014 in mplex)

(Sorry for the flood :) )

Anyways what we see here is 29.97 f/s (normal NTSC) in the begining then it shifts to 24.97 f/s (normal NTSC from film 3-2 downpulled effect 3*) then it makes a shift to 29.97 and the it shifts back and so on.

Now what is happening in the movie when it shifts/each snip?

1: HBO Intro
2: Film entry
3: My guess movie into
4: Normal movie (this and to the last snip is around 1h 50 min of film)
5: Ending and so on

Now since the SEQ header is specifiying 29.97 f/s we will most defitly get sync issues.

If it was so simple as changeing the frame rate in the SEQ header this would be a piece of cake. However most muxer will take a look at the first seq header and then start muxing (e.g. mplex is doing this). Hence they will mux the video as 29.97 f/s which is wrong.

Now how do we detect the right frame rate? Well we have to probe the stream, at regular interval and there by determine what frame rate we have.

Then what can we do about it? We have a mix of frame rates after all. Well we can open for suggestion what you folks think is best:


Remove video of unwanted frame rate, a side effect of this is that we for sure will remove comersials, previews etc 4*. There might however be side effects that we loose a bit of the start/end in each break.

Adjust the minority frame rate to the major frame rate of the TyStream. Pretty easy just add and remove B-Frames.


As you might have notice 3003 + 4505 is not == 2*3753 we have so on top of the shifting of frame rate we have a slight drift in frames small but over 3753/2 frames it will make a hole frame (around 300ms) and over a whole 2h movie it will make 45 frames in difference i.e a lot of diff. Now you shold now that there every now and then come a 3001 and maybe a 4503 so the drift is not as big as I state but it's large enough to be iritated over :).

Ahh, yes I naturally hacking on correcting this junk. However it will take a little time it's not as trival as it might look.

Hence the next version will be 0.4.0 and the the other features will be bumped up one notch in versioning.

Cheers Olaf


*1: Will examine SA Tivos too but the algo is slightly different. However the nice thing is that the algo I use to examine is the one I will use to repair the mpeg stream.

*2: No worries I will cover UK/PAL Tivo to :)

*3 Yes, the number of ticks is not 3753 but 3003 + 4505 is nearly 2 * 3753

*4 I'm allready doing this to some extent on recordings that has a mix of MPEG and AC3 sound - the clips with AC3 will be keept while I ditch the MPEG sound. All comersials are sent in MPEG.

AlphaWolf
12-08-2002, 01:34 AM
I would be concerned about what video we might be losing if we did the first option, but then, that would be nice to remove the unneeded portions of the stream just like that. I can't decide myself!

But then, what about having our own special *modified* version of mplex which is designed to detect changes in the frame rate? (mplex is OSS isn't it?) That way, when you edit the tystreams, then remux em, they are in sync.

hell...I dunno :)

rd001
12-08-2002, 11:55 AM
Originally posted by olaf_sc

If it was so simple as changeing the frame rate in the SEQ header this would be a piece of cake. However most muxer will take a look at the first seq header and then start muxing (e.g. mplex is doing this). Hence they will mux the video as 29.97 f/s which is wrong.

Now how do we detect the right frame rate? Well we have to probe the stream, at regular interval and there by determine what frame rate we have.
I would think that you'd want to add/remove the B-frames and leave the stream intact. Maybe you can offer the option of removing these segments in other frame rates later.

You know, this matter of detecting stream changes by frame rates is very interesting as it might relate to automated commercial-cutting during extraction or automated commercial- skipping like ReplayTV. Any comments, Riley (over on your own thread)?

You can't do anything except make a best-guess, what-works-most-often solution. Maybe you could later offer the user the option of a preliminary scan of the tystream and allow them to adjust the offsets manually at the key points. Tytool seems to just use a single best-fit approach which has worked well enough but still isn't always perfect. There are times I wish I could see and manually adjust the A/V sync at key points. I am in no way criticizing Tytool but it's just an inherent problem given the stream defects. GIGO, like always.

dlang
12-08-2002, 04:55 PM
the problem with adding frames is that you distort the timing of the ones that are already there.

if you want the frame to be timed like this
1 1 1 1 1 1 1 1 1 1
but they are timed like this
1 1.2 1 1.5 1.5 1 1.3 1.5
(note that both of these streams cover the same time, even though one has 8 frmaaes and the other 10)

exactly what timestamps will you put on your new frames and where in the stream will you put them?

olaf_sc
12-08-2002, 07:27 PM
Hello,

Yes,the observation is both right and wrong however it's a interesting observation that needs to be delt with.

If we talk about a demux to ES streams
(raw audio/video) well then there is no time stamps hence the matter is of no interest.

If we talk about a internal mux or an demux to PES streams (ES streams with PES headers). Well then there will be a issue. However my intensions is that when we detect a framerate shift we will adjust all PTS time stamps until we shift back to the correct framerate again.



Originally posted by dlang
if you want the frame to be timed like this
1 1 1 1 1 1 1 1 1 1
but they are timed like this
1 1.2 1 1.5 1.5 1 1.3 1.5
(note that both of these streams cover the same time, even though one has 8 frmaaes and the other 10)


Nope you are wrong, they will cover different time unless you change the frame rate in the SEQ header. It's only when you have PTS time stamp present that they will cover the same time. However having the wrong amount of frames compared to the frame rate and forcing sync with PTS time stamps is simply not a good thing (tm).

Cheers and thanks for the feedback

Olaf

olaf_sc
12-08-2002, 07:33 PM
Hello RC


Originally posted by rc3105
the same info needed to maintain sync on-the-fly is tremendously usefull for auto detecting commercials. by combining that with info about the cc data, frame rates, ratio of I vs P & B frames & AC3/mpeg - video aspect ratio shifts we can probably get it automated quite nicely.


Hmm, have not yet probed a SA stream for framerates but since they aren't aware of what the content is - well I think they will not shift in frame rate hence no detection. Well SA probably drift in sync although but thats another problem however delt with in the same fashion.



olaf has said he also intends to release his utils as a set of libraries. that will make writing these sort of utils SO much easier.


Yes, it's even in the TODO-README, anyways the c file holding the main loop is just 100 lines or so the rest is "library" functions. All I need to do is to clean it up and have some values in private data access holders so you don't thinker with them without knowing what you do.

Cheers Olaf

dlang
12-08-2002, 11:03 PM
Originally posted by olaf_sc
Nope you are wrong, they will cover different time unless you change the frame rate in the SEQ header. It's only when you have PTS time stamp present that they will cover the same time. However having the wrong amount of frames compared to the frame rate and forcing sync with PTS time stamps is simply not a good thing (tm).

Cheers and thanks for the feedback

Olaf [/B]

however the VFR of the dtivo means that in case #2 these are the only frames that you have, and these 8 frames are played in the same 1/3 of a second that the 10 frames of case #1 are played in.

yes this is in violation of the MPEG specs, but VFR is in violation of the MPEG specs and it's reality in the dtivos.

the headers claim 29.97 frames/sec, but the reality is something less then that, changing without notice. unless the result of your muxing is a mpeg stream that contains a timestamp for each frame you have no chance of keeping the sync through playback.

now if you want to move the frames around and add additional frames to make it CFR instead of VFR then my question is valid

the two cases stated another way both take 1/3 of a second to play

frames should be displayed at
case #1
.03 .06 .1 .13 .16 .2 .23 .26 .3 .33 seconds

case #2
.03 .07 .11 .16 .21 .24 .28 .33 seconds

if you just duplicate frames and sorten the display time of the existing frames to make room for them you will end up with the video stuttering

olaf_sc
12-09-2002, 12:58 AM
Hello Folks,

Let me clarify MPEG and the sync issues we have in regards to it.


First there is no such thing a VFR!!!
Instead we have a shift it the frame rate from e.g. 29.97 to 23.976, period!
This is totally leagal and in compliance with the MPEG std for Transport Streams (which DTivo is derivate of). 1*
The only thing Dtivo is breaking is that they

Don't end the video sequence when they shift frame rate
Don't specifiy it in the seq header
Shift frame rate in the middle of a GOP/SEQ



Now over to the details about adding removing frames in order to comply with the frame rate.

Let me give an example, let us say that we have probed the stream and found out that the stream is mostly has a frame rate of 29.72f/s. For one reason or the other DTivo mixing in small portions that has a frame rate of 23.976 f/s.

Now let us first think that we a frame rate of 29.72 f/s how the heck is that displayed on your TV that has a frame rate of 59.94f/s?

Simple answer every frame is duplicated i.e. showed twise.

Now how is 23.976 f/s encoded mpeg showed on your TV which has a frame rate of 59.94 f/s (by a good mpeg decorder)?

Simple two film frames will be down pulled to 5 frames this is called 3-2 down pull. This is done every time you e.g. watch a DVD on your TV (unless the source is of the film is TV and not film).

See this is common stuff and has been done over and over again and there is no reason what so ever that we can't do it :). Allthough we will do it between 29.97 and 23.976 ane we do it both up and down so to say. We will simply up/down pull 5 to 4 and 4 to 5.

The bottom line is that if you don't use PTS timestams i.e. you do a simple demux you will never have to worry about frame rates, duplicated frames and so forth, period!

Tydemux will produce a perfect MPEG Video stream and audio stream (MPEG or AC3).

Tydemux will check and repair every thing, gaps/holes, temporal reference, video drift, frame rate and so forth.

Yes, internally I have taken care of adjusting the PTS time stamps but that's for the reason that I want to do an internal mux later on in version 0.5.0.

Now the advantage of having a perfect MPEG stream instead of one that forcefully is muxed with help of PTS time stamps is:


You will be able to use your favorite muxer
You will be able to transcode, edit, alter etc as much as you want and still have sync.
It will be a legal MPEG program stream - shifting anything (beside the bitrate) is not allowed in a MPEG Program Stream!! 2*



Cheers Olaf

PS: Yes version 0.4.0 will have a switch which makes you able to choose if you want to fix the frame rate or drop segments with faulty frame rate.

PS: MPEG Program Streams are e.g. used in a DVD, MPEG video file, VSCD etc..., MPEG Transport Streams are used in DSS/DVB, HDTV, ATM video networks, i.e. in places where we can have loss of data! (Yes, DSS/DirecTV is not really MPEG Transport Stream there are small differences).


PS:Hmm, just a small side note. calculating the frame rate by counting frames over one second in a BIG no no. Since the frame rate is not even i.e. 29.97 (or 23.976). Counting over one second will make it drift! Some seconds simply must have 29, 30, or 31 frames per second it will not add up other wise.


1* The std says for a decoder to be in compliance with a Transport Stream it must be able to allow some discontinuites at the video sequence boundery. Namely any parameter set in the sequence header which e.g. includes the frame rate!

2* Most of us want to produce a DVD or SVCD of our Tivo recording. Having a invalid program stream is not a good thing even if most players will play it just fine. No, let us instead fix the stream and produce valid program streams!

rd001
12-09-2002, 06:18 AM
Originally posted by olaf_sc


The only thing Dtivo is breaking is that they[list=1] Don't end the video sequence when they shift frame rate
Don't specifiy it in the seq header
Shifts framerate in the middle of a GOP/SEQ[/list=1]So, it's all perfectly legal except for these horrible little omissions that break the MPEG standard in much the same sense that a roadmap is a perfectly good map even if it omits those portions of the road that lead the driver straight over a cliff.

You are really too kind to those DTV MPEG perverts. Olaf, you should be canonized as the patron saint of long-suffering charity toward infidels.

But thank you for that explanation. It reminds me very much of some earlier discussion of these broken streams.

The real question I always have after reading explanations like this from you or jdiner is this: how the hell does a DTivo or any other DTV box decode this stream as well as it does given the state of the input stream? After all, they can't be using much in the way of stream look-ahead scanning. Somewhere in those DTivo boxen and other lesser IRDs, there's a simple and nasty kludge to make all this MPEG confusion work without gross visible effect.

dlang
12-09-2002, 02:37 PM
for playback it's simple, instead of having the playback code look at the framerate and calculate when to show each frame they just have it look at the timestamp of each frame and display it then (in relative terms that is, when you switch to a steam the first timestamp you find is 'now')

in many ways it's a simpler job to do, and since they control the streams and can therefor be sure that there will always be timestamps in it this will always work

dlang
12-09-2002, 02:40 PM
If the framerate is as consistant as olaf thinks it is it should be very useful for spotting commercials at least during movies where the programe frame rate is 24 fps

olaf can you have your program print timestamps of when the frame rate shift happens so we can compare that to the commercial slots?

olaf_sc
12-09-2002, 03:51 PM
Hello dlang

You nailed it :). Thats just how they do it.

Cheers Olaf



Originally posted by dlang
for playback it's simple, instead of having the playback code look at the framerate and calculate when to show each frame they just have it look at the timestamp of each frame and display it then (in relative terms that is, when you switch to a steam the first timestamp you find is 'now')

in many ways it's a simpler job to do, and since they control the streams and can therefor be sure that there will always be timestamps in it this will always work

olaf_sc
12-09-2002, 03:53 PM
Hello Dlang

Yes, thats a walk in the park to implement (will provide a switch to turn it on and off).

I will make tydemux print the first time stamp and then print a time stamp every time the frame rate shifts.

Cheers Olaf



Originally posted by dlang
If the framerate is as consistant as olaf thinks it is it should be very useful for spotting commercials at least during movies where the programe frame rate is 24 fps

olaf can you have your program print timestamps of when the frame rate shift happens so we can compare that to the commercial slots?

AlphaWolf
12-09-2002, 07:28 PM
olaf_sc: I'm not a coder but heres my contribution, I fixed a big amount of spelling and grammatical errors in both your code and the documentation as requested per the README, without interfering with the program itself. Heres the diff -u output dumped for all files that were changed, and then .bzipped, just whack off the .txt extension of course.

Also not being rude, just it looks like English isn't your first language, and two words you commonly misspell are commercial and twice. ;)

Enjoy :D

olaf_sc
12-09-2002, 07:52 PM
Hello AlphaWolf,

BIG thanks for the patch it was really needed. I didn't prioritized spelling, grammar for five seconds. Hence a check/fix was badly needed.

To you all this is the kind of help I really need besided bug reports. It's really helpful and offloads me quite a lot.


Originally posted by AlphaWolf
Also not being rude, just it looks like English isn't your first language


No shit Sherlock :D, besidedes I'm also slightly dyslectic.

Cheers Olaf

AlphaWolf
12-09-2002, 08:15 PM
Didn't thoroughly fix all of the grammatical errors mind you though :P

Just fixed the ones I noticed while looking for spelling errors. I'll probably take a deeper look after the next release, b/c the changes there were big enough already :D

BillyZ
12-10-2002, 09:56 AM
olaf -- Kudos on Tydemux -- ive been waiting months for a program like this -- now im starting to extract/burn gigs and gigs of .ty files ive saved and archived that couldn't be split with the older apps.


Question: I have run into several anomolies with various .ty files that you may want to look at, what would be the best way to get these to you....posting a snippit of the out put or uploading a 1.5gb .ty file to an FTP site?

any timeline for the 0.32 release? Im eager to break the 2gb barrier (i have 4 'Taken' Eps that need to be demuxed ;) )


Cheers

=-BillyZ

rd001
12-10-2002, 11:19 AM
Originally posted by olaf_sc
[Olaf responding to dlang's inquiry about outputting a list of frame rate changes to aid commerical detection]: Yes, thats a walk in the park to implement (will provide a switch to turn it on and off).

I will make tydemux print the first time stamp and then print a time stamp every time the frame rate shifts. Given the relatively uniform practice of upping the volume of commercials, sometimes to ridiculous levels, would monitoring the audio stream and calculating its relative volume be of any help in confirming where commercials are placed?

If you found a place where the frame rates change and the volume suddenly surges, that would be a pretty strong indicator. Maybe it would be of limited usefulness though. I just keep thinking that ReplayTV seems to have a pretty good way to detect commericals and it can hardly be rocket science.

olaf_sc
12-10-2002, 12:57 PM
Hello BillyZ

In regards to a time line for 0.3.2, there will be no 0.3.2 :). There will be a 0.4.0.

The version bump is due to the large amount of new functionality. Hence next version will be 0.4.0, which means that 0.5.0 will have internal MPEG muxer and internal audio transcoding and 0.6.0 will have muxed DVD/SVCD compliant MPEG program streams.

I expect to release 0.4.0 before the 14 of Dec. Now what will 0.4.0 bring us?


Check and fix of temporal reference
Check and fix of frame rate
Check and fix of drift in audio/video sync
Check and fix of missing frames
Check and fix of max bit rate
Port to Mac OS 10.x, Solaris etc..
Removed 2GB barrier








Originally posted by BillyZ
olaf -- Kudos on Tydemux -- ive been

Thanks :)


Question: I have run into several anomolies with various .ty files that you may want to look at, what would be the best way to get these to you....posting a snippit of the out put or uploading a 1.5gb .ty file to an FTP site?


May I ask what kind of anomolies? Then I think we can figure out how to get those anomolies to me :). I have a ftp site 66.121.15.34 but before we start uploading it might be good to figure out what is happening and then start upload.

Cheers Olaf

olaf_sc
12-10-2002, 01:02 PM
There is several ways of detecting commercials, volume, frame rate change, closed captions, channel log on/off, black frames, etc...

Taken all of those together and you will be able to detect and remove commercials with high precision. Now this is something for past 0.7.0 or soo since I better get other things going. However I preparing for it by 1: writing the code in such way that it's possible 2: Allready providing hocks for some of the info in the present code base.

Cheers Olaf


Originally posted by rd001
Given the relatively uniform practice of upping the volume of commercials, sometimes to ridiculous levels, would monitoring the audio stream and calculating its relative volume be of any help in confirming where commercials are placed?

If you found a place where the frame rates change and the volume suddenly surges, that would be a pretty strong indicator. Maybe it would be of limited usefulness though. I just keep thinking that ReplayTV seems to have a pretty good way to detect commericals and it can hardly be rocket science.

und420_247
12-10-2002, 02:24 PM
Wow, can't believe the progress being made here. Incredible job, Olaf!

Also, the news of a MacOSX port really makes me happy. Just got a Mac and I'd love to start using it for this purpose.

racingclub
12-10-2002, 02:46 PM
I expect to release 0.4.0 before the 14 of Dec. Now what will 0.4.0 bring us?

Check and fix of temporal reference

Check and fix of frame rate

Check and fix of drift in audio/video sync

Check and fix of missing frames

Check and fix of max bit rate

Port to Mac OS 10.x, Solaris etc..

Removed 2GB barrier


roll on the 14th :) - great work Olaf! :D

saltydog4791
12-10-2002, 05:02 PM
The port for OS X will make my life so much simpler. The 14th can't come fast enough. I think I might be able to finally ditch my windoze box. :)

saltydog4791

AlphaWolf
12-10-2002, 05:07 PM
Hopefuly we can finaly edit our resulting mpegs with any editor, without losing sync as well :D. Sounds like we are upon it.

icculus
12-10-2002, 05:10 PM
Originally posted by AlphaWolf
Hopefuly we can finaly edit our resulting mpegs with any editor, without losing sync as well :D. Sounds like we are upon it.

I'm not sure how this tool helps that. Do any editing tools support mpeg input? I've been converting to .avi with DVD2AVI and VFAPIConv. What would you be using to, say, edit commercials out?

Olaf - been lurking for a while. I'm very excited to see the end product!

BillyZ
12-10-2002, 05:17 PM
I already got a jump-start on a decent front-end, just waiting to include the new "Oh Four Oh" code....

What do you suppose are the odds of programs like SonicFoundry's VideoFactory and TMPGenc having the abillity to open and read .ty files....


then my life would be complete.


god i need a life.

AlphaWolf
12-10-2002, 05:54 PM
icculus: womble mpeg vcr does

olaf_sc
12-10-2002, 06:57 PM
Hello BillyZ,

You should wait with building a frontend until tydemux is modularized, i.e. when it's a lib and a small frontend (tydemux). Then you (and others) can start making really cool frontends. I will try to make the clean up after the 14:th so that it's a lib in version 0.4.1.

Cheers Olof



Originally posted by BillyZ
I already got a jump-start on a decent front-end, just waiting to include the new "Oh Four Oh" code....

What do you suppose are the odds of programs like SonicFoundry's VideoFactory and TMPGenc having the abillity to open and read .ty files....


then my life would be complete.


god i need a life.

BillyZ
12-11-2002, 12:15 PM
Sounds like a plan...im going to wait until .4 to see if any of those anomolies dissapear -- Most of them are GOP errors anyway.




Originally posted by olaf_sc
Hello BillyZ,

You should wait with building a frontend until tydemux is modularized, i.e. when it's a lib and a small frontend (tydemux). Then you (and others) can start making really cool frontends. I will try to make the clean up after the 14:th so that it's a lib in version 0.4.1.

Cheers Olof

Pr.Sinister
12-11-2002, 01:26 PM
Originally posted by icculus
I'm not sure how this tool helps that. Do any editing tools support mpeg input? I've been converting to .avi with DVD2AVI and VFAPIConv. What would you be using to, say, edit commercials out?

Olaf - been lurking for a while. I'm very excited to see the end product!

MediaWare Solutions' M2-Edit does it,
Womble's MPEG2VCR does it,
Honestech's MPEG Editor does it,
Adobe's Premiere does it,
Sound Foundry's Vegas Video & Video Factory do it,

etc...

There are a LOT of editing tools for MPEG out there...
MPEG is the most widely used format for video
(actually that just speculation on my part) ;)

-Pr.

icculus
12-11-2002, 01:54 PM
Originally posted by Pr.Sinister
MediaWare Solutions' M2-Edit does it,
Womble's MPEG2VCR does it,
Honestech's MPEG Editor does it,
Adobe's Premiere does it,
Sound Foundry's Vegas Video & Video Factory do it,
-Pr.

Thanks for the info. I'm sorry to drag this more off topic, but are any of these freeware?

I don't really want to invest hundereds of dollars in such a package. I did pony up for Pinnacle Studio for my DV work, and otherwise use Virtualdub/TMPGenc, and the other motley crew of (free/cheap) tools for other work.

thanks!

artships
12-11-2002, 02:08 PM
Originally posted by Pr.Sinister
MediaWare Solutions' M2-Edit does it,
Womble's MPEG2VCR does it,
Honestech's MPEG Editor does it,
Adobe's Premiere does it,
Sound Foundry's Vegas Video & Video Factory do it


Um, I think TMPGEnc, when frame-served by DVD2AVI, and even by itself, does it, too. And it is an editor. I've removed many an advert with it.
Here's how I do it. (http://tivo.30below.com/jdouglass/)

John,
Much anticipating .0.4.0...

icculus
12-11-2002, 02:10 PM
Originally posted by artships
Um, I think TMPGEnc, when frame-served by DVD2AVI, and even by itself, does it, too.


Right, but TMPGenc is not, AFAIK, an editor. This was in regards to editing the MPEG2 file (to edit out commercials). I'm looking for something with the functionality of Virtualdub (an awesome app once you get used to the interface)

BillyZ
12-11-2002, 05:46 PM
Womble has many issues when it comes to Muxing with out drivting the audio too much...i noticed that it does not do a good job of calculating the filesize properly...a 800MB video + a 70KB audio <> 6149GB...

Sonic Foundry's (both vegas video and Video factory) will NOT import Mpeg2 videos right from the tivo...they have to be processed which either A.) takes too much time or B.) loses quality.

so far the best one ive used that causes the least headaches is TMPgenc plus -- but still it doesn't always work correctly...which is why i gave up re-encoding all together, and just Author DVDs and 'blank' out the comercials by using chapters.




Originally posted by Pr.Sinister
MediaWare Solutions' M2-Edit does it,
Womble's MPEG2VCR does it,
Honestech's MPEG Editor does it,
Adobe's Premiere does it,
Sound Foundry's Vegas Video & Video Factory do it,

etc...

There are a LOT of editing tools for MPEG out there...
MPEG is the most widely used format for video
(actually that just speculation on my part) ;)

-Pr.

BillyZ
12-11-2002, 05:51 PM
Excellent site -- its good to know that other ppl are suffering thru this as I -- I use DVD Maestro to author the movies -- only because i can tell it to skip over the chapters that are commercials -- but I will try using TMPGENc with the DVD2AVI part in it -- that would be nice to speed up that specific step.

-BillyZ


Originally posted by artships
Um, I think TMPGEnc, when frame-served by DVD2AVI, and even by itself, does it, too. And it is an editor. I've removed many an advert with it.
Here's how I do it. (http://tivo.30below.com/jdouglass/)

John,
Much anticipating .0.4.0...

icculus
12-11-2002, 05:51 PM
Originally posted by BillyZ
Sonic Foundry's (both vegas video and Video factory) will NOT import Mpeg2 videos right from the tivo...they have to be processed which either A.) takes too much time or B.) loses quality.


Yes, but do they import an MPEG that has been muxed by vsplit or tyremux? Neither of those takes much time, and I wouldn't see how that would lose quality, either.



so far the best one ive used that causes the least headaches is TMPgenc plus -- but still it doesn't always work correctly...

Right, but again, TMPgenc is just an encoder, it's not an editor, so how would you edit out commercials with it??

(I posted just as BillyZ was above; ignore me) =)

olaf_sc
12-11-2002, 10:53 PM
Hello Folks,

Okay a little update :) and reminders.

I just bought a Mac G3 400MHz box and MacOS X (jaguar) to go with it. Hence I should now be able not just to port tydemux but also test and develop under Mac.

I already have a Sun box so the next release will have binaries for Windows Linux, Solaris and Mac. Anybody using AIX?? If there is then I will start up my RS6000 after X-mas and make a port :).

Code updata is that all temporal reference checks and fix functions are in place - and that I'm now finishing of the frame rate and frame drift functions. As I said before I really want to have this out the door before the 14th.

Now, a small reminder I know it's easy to start talking about all kinds of apps and what they are capable of. However this thread is about tydemux, I therefore in a very soft and mild tone say "please keep it to a minimum" :). NOTE: that it's very!! okay to discuss such things as this app does or doesn't work with tydemux and why it is like that!.

Cheers Olaf

OS X
12-12-2002, 02:40 AM
Originally posted by olaf_sc
Hello Folks,

Okay a little update :) and reminders.

I just bought a Mac G3 400MHz box and MacOS X (jaguar) to go with it. Hence I should now be able not just to port tydemux but also test and develop under Mac.



Great news!

You've got about the lowest end hardware that's still capable of running Jaguar. I have a Dual G4 800 with 1G RAM. I'm a bit of a newbie with TiVo and *nix, but I'll help however I can if you need anything.

BillyZ
12-12-2002, 10:34 AM
Yes, and im thinking because they re-write the header which is what VideoFactory is looking for, but you can import some split mpegs because you are able to import seperate audio and video.


Originally posted by icculus


Yes, but do they import an MPEG that has been muxed by vsplit or tyremux? Neither of those takes much time, and I wouldn't see how that would lose quality, either.

[/B]


Originally posted by icculus


Right, but again, TMPgenc is just an encoder, it's not an editor, so how would you edit out commercials with it??

(I posted just as BillyZ was above; ignore me) =) [/B]

because Mpegvcr can read the file and can scan through it very fast, i use it to mark the times for the comercial start & stop.

then i either A.) use tmpgenc to merge and cut the mpeg (after muxing) or i import it into DVD maestro and import the list as a chapter file.

olaf_sc
12-13-2002, 01:23 AM
Maybe you have better connections but I can't pickup a Blue & White G3 400MHz / 256 MB / 9GB SCSI - for that kind of a price. At least not when I need it the same day as I did in this case (test and compile tydemux for mac). Maybe I paid to much for it ($460)? In all with a visit to CompUSA to get Jaguar and new keyboard/mouse the whole thing costed me $720. Anyways I'm happy needed to learn MacUnix anyway, and it's nice to get tydemux running on Mac since there are some Mac users out there :).

Hmm, inorder to write something about tydemux - functions for fixing 29.97 f/s frame rate in a 23.97 f/s movie is now ready. Over to the other way around. Ahh, UK folks this is only DTivo - SA and SA UK only have problem with sync drift which is a whole lot easier - walk in a park compared to frame rate altering.

Cheers Olaf


Originally posted by rc3105
well, not quite the lowest end

but why the heck not when you can pick one up for less than a turbonet card

:D

--
Riley

salfter
12-13-2002, 02:48 PM
Originally posted by icculus

I don't really want to invest hundereds of dollars in such a package. I did pony up for Pinnacle Studio for my DV work, and otherwise use Virtualdub/TMPGenc, and the other motley crew of (free/cheap) tools for other work.


Lately, I've been using Avisynth (http://www.avisynth.org/) with an MPEG-2 decoder plugin (http://www.avisynth.org/index.php?page=Section+3%3A+Filters+and+colorspaces#q34). You use DVD2AVI to create a .d2v from an .m2v or .mpg file, use LAME to convert from .m2a to .wav, then create a script that looks something like this:


LoadPlugin("c:\path\to\mpeg2dec.dll")
DelayAudio(AudioDub(MPEG2Source("c:\foo.d2v"),WAVSource("c:\foo.wav")),delay)


Save this to foo.avs, then load that into VirtualDub, TMPGEnc, Windows Media Player, etc. (Note that you'll need to disable the "DirectShow Multimedia File Reader" VFAPI plugin in TMPGEnc for this to work.)

Avisynth also provides editing capabilities of its own that are comparable to VirtualDub (better in some ways, such as its non-linear editing capabilities). Nowadays, I mainly use VirtualDub just to find the cutpoints for some ripped video and to verify that the audio and video are synced. I do all of the editing with Avisynth and create a script that's ready to encode with TMPGEnc with no additional processing.

rc3105
12-13-2002, 07:00 PM
Originally posted by olaf_sc
Maybe you have better connections but I can't pickup a Blue & White G3 400MHz / 256 MB / 9GB SCSI - for that kind of a price...
Cheers Olaf

I hit the computer shows ocassionally. (halfway tween a trade show and a swap meet) I used to make that circuit as part of my computer business but the last couple of years I'm mostly just a spectator.

last fall I picked up 5 beige g3-333 tower servers for $225 for the whole lot. (installing X on an 8500 was a nuisance) they're a notch down from the blue & white but I like them because they were the last model with motherboard scsi, com ports and a floppy drive.

there's a show here in san antonio tomorrow, I'll let ya know what stuff's goin for. I'm trolling for a g4 notebook or an emac.

--
Riley

olaf_sc
12-13-2002, 07:41 PM
Update:

Will not be able to release tydemux on the 14:th :( , well well not the end of the day I hope. It will just be delayed a couple of days good estimate is probably around the 16:th to 19:th of December. My kids are coming over and I want to spend some time with them, hence my eager try to make it until the 14:th :).

After releasing I will spend a couple of days fixing bug reports (sure they will come) then it's time to clean up tydemux. So it can be a library (libtydemux) and a small frontend app (i.e. tydemux it self). Will try to provide both a cli frontend app and a GUI (QT) one as examples for you how want to make your own cool frontends.

Cheers Olof

rd001
12-14-2002, 04:09 PM
Go do the family thing. No need to explain anything, Olaf.

It's not like anyone is paying you a salary or something. It's just a hobby. We'll be happy with anything you distribute, whenever you get around to it. If people are too impatient, they can work on your source code, can't they?

FredThompson
12-14-2002, 05:18 PM
Originally posted by salfter
Lately, I've been using Avisynth (http://www.avisynth.org/) with an MPEG-2 decoder plugin (http://www.avisynth.org/index.php?page=Section+3%3A+Filters+and+colorspaces#q34).

There are better MPEG2DEC routines. Try this one: http://nic.dnsalias.com/ It doesn't need the whole d2v aspect.

salfter
12-14-2002, 08:45 PM
Originally posted by FredThompson
There are better MPEG2DEC routines. Try this one: http://nic.dnsalias.com/ It doesn't need the whole d2v aspect.

I think I tried that before and found that it didn't work as well. With some sources, you still need DVD2AVI to separate the audio and video anyway. I'm not sure where I got the DLL that I'm using...it's 108K and is dated 25 Sep 02, if that narrows it down any. (I've put it up at http://alfter.us/files/mpeg2dec_pp_mod_b3.zip, if anybody's interested.)

FredThompson
12-14-2002, 09:19 PM
Different DLL. Try Nic's and see if it works better. He's one of the tech guys involved in developing XviD. It's just possible he might know what he's doing.

The nice thing about his is you don't need DVD2AVI.

Take a look at this stuff also, it's an offshoot of what has become a very active doom9 focus on adding proper MPEG2 to VirtualDub/AVISynth:

http://smokeslikeapoet.d2g.com/

There's also this little gem that's not that visible. The code is GPL'ed so I'm wondering if it would be helpful making a Windows client for editing.

http://www.flaskmpeg.net/

VirtualDub code is also available but both editing panes are in the same window. That's really awkward with large source.

Personally, I'd like a version of VirtualDub that works in MPEG2 non-destructively but maintains everything else. Basically, if there are no filters applied and the source is MPEG-2, it stays MPEG-2 with the exception of truncated GOPs being converted to I frames. Oh, and with a few other GUI changes (floating windows, user-defined crop bar color, etc.)

That really shouldn't be too hard. It needs a flag to denote processing (other than cutting) and an alternate path for the save routine.

Oh, keep an eye on this thread:

http://forum.doom9.org/showthread.php?threadid=39941

neuron2 is Donald Graft. He runs the VirtualDub/AVISynth board that hosts my link lists. Uh...he's also very knowledgable about commercial MPEG-2 streams in North America. trbarry and Marc FD are also quite experienced with this stuff. It's a good thread.

And, while I'm at it, here's the Tektronic MPEG reference:

http://www.tek.com/Measurement/App_Notes/mpegfund/

laserfan
12-17-2002, 12:00 AM
Olaf gently suggested some time back that posters should kindly stay on-topic. Remember please that Olaf is forced to read every one of these looking for some relevance to tydemux.

If you want to start another thread by all means do it! And now I have contributed to the "noise" myself! Sigh.

FredThompson
12-17-2002, 01:15 AM
What should be obvious if you follow the links:

Two Open source GUIs for video editing
Open source MPEG-2 stream debugging and cleaning of buggy streams
Commercial MPEG-2 streams in North America. Gee, what might I be implying?

The non-TiVo-specific community working on MPEG-2 is far greater than just a few people hanging out on this board.

You could look at post #1 in this thread:

This is to be a place where we can all trade ideas, discuss the tydemux code base (and the up comming tyremux) and MPEG coding in general. It's GPLed application so you are all welcomed to partisipate/develop or just look at the code in general. Let it inspire so that ideas, code and every thing flows freely.

AlphaWolf
12-17-2002, 12:53 PM
hmmmph...I took one of the videos coming from the menu background on a series 1 directivo with s/w version 3.1, and I looked at it with a hex editor, and the magic number for a tystream header is there, as well what appears to be a very basic master chunk. Look what happens when you have tydemux process it:



Probing TyStream .....

Tystream recorded on:
SA Tivo Series 2
Software rev 1.3 (these two contradict)

Tystream recoding audio stats:
MPEG Layer II audio
Average tyrecord (audio) size: 864
Audio frame size: 864
Audio frame time: 3240 (ticks)

Seeking TyStream start of MPEG Layer II audio
Found start of MPEG Layer II audio
Skipping to chunk 1 - reseting chunk numbering

Starting demux process
......... 100..
Demux process finished

A/V Sync Offset: 000ms (i.e. audio plays 000ms early use -O -000 in mplex)


weird...heh...also the output is just a zero length file. vsplit1n doesn't process this file at all, complains that it can't find the first 10 chunks.

olaf_sc
12-19-2002, 05:22 AM
Yes they contradict each other :), will make a extra check and skip when I find results like this.

What is determining if it's a 1.3 or 2.x or newer is the first four bytes in the chunk. If byte 3 bit 4 (bit with a value of 8) is one then we have a 2.x or higer stream.

Now we probe the rest of the chunks if we find no pes headers in the stream well then we have a S2 Tivo.

My guess is that there is no PES in the background stream kind of useless since it's a circluar buffer. Hence tydemux thinks it's a S2 stream and uses the S2 video demux engine and the 1.3 audio demux engine. Well non of it will work really well as you discovered.

I have to take a closer look at this very special stream.

Cheers Olaf


Originally posted by AlphaWolf
hmmmph...I took one of the videos coming from the menu background on a series 1 directivo with s/w version 3.1, and I looked at it with a hex editor, and the magic number for a tystream header is there, as well what appears to be a very basic master chunk. Look what happens when you have tydemux process it:



Probing TyStream .....

Tystream recorded on:
SA Tivo Series 2
Software rev 1.3 (these two contradict)

Tystream recoding audio stats:
MPEG Layer II audio
Average tyrecord (audio) size: 864
Audio frame size: 864
Audio frame time: 3240 (ticks)

Seeking TyStream start of MPEG Layer II audio
Found start of MPEG Layer II audio
Skipping to chunk 1 - reseting chunk numbering

Starting demux process
......... 100..
Demux process finished

A/V Sync Offset: 000ms (i.e. audio plays 000ms early use -O -000 in mplex)


weird...heh...also the output is just a zero length file. vsplit1n doesn't process this file at all, complains that it can't find the first 10 chunks.

olaf_sc
12-19-2002, 05:27 AM
Progress update:

Well kids take thier time so to say so things are going slowly. It's not a total stopp and I'm actually wrapping up things i.e. testing the upcomming version (0.4.0).

It's not a small addition this, 0.4.0 has more than 4000 new lines compared to 0.3.x hence it takes a while to nail down every bug.

Hopefully we will have 0.4.0 out before X-mas it looks promising but I will not release any half baked product.

Cheers Olaf

racingclub
12-19-2002, 11:19 AM
Thanks for keeping us informed Olaf :)

BillyZ
12-25-2002, 10:16 AM
I hope all have a Merry, Happy and Joyous Safe Holiday.

;)

olaf_sc
12-26-2002, 05:00 AM
Hello Folks,

Progres update :). Nope 0.4.0 is not ready yet - X-mas and family and kids makes things go really slow.

However good news, it's debuged and I have made pre tests with 0.4.0 and it works like sharm. The frame rate and sync drift corrections made by tydemux works really good and so far i have managed to keep sync on really difficult streams.

I will now enter code clean up stage and I'm counting on releasing the prog the 31:th of Dec as a New Years gift.


Cheers and sorry for all delays

Olaf

racingclub
12-26-2002, 07:36 AM
looking forward to it Olaf - hope you had a good Xmas :D

davidblack
01-01-2003, 02:14 AM
Hope everyone had a good christmas and new year

Olaf - any update on ETA ??

Cheers

David

BillyZ
01-01-2003, 10:59 AM
yeah, im glad Im not the only one very excited about this...oh the work i have to do afterwards....

Ive collected all the 'i love the 80s' and the old tydemux or tytool has issues with a few of them...


tons of alf eps too....what a great day


Originally posted by davidblack
Hope everyone had a good christmas and new year

Olaf - any update on ETA ??

Cheers

David

bbarry
01-01-2003, 08:56 PM
Same here. With over 450 new Thread Views in the last 24 hours I guess others
are waiting too!

Hi8
01-01-2003, 11:18 PM
Hoping to see 4.0 soon... not much of my holiday vacation left! need some time to BURN!

also picking up my Sony 715 set-top player.. tomorrow morning, TyDVD compliant!

In/Sync DVD ready!

olaf_sc
01-01-2003, 11:39 PM
Hello Folks,

Okay I'm late but not due to the fact that the code isn't "working" and ported to both MacOS X and Solaris.

I got everything to "working" last week, but then I ran into trouble with a/v sync when I multiplex it with mplex (mjpeg-tools) both v 1.6.0 and the latest cvs version.

Hmm, the fact is that I can view how the frame rate is shifting back and forth from 29.97 to 23.976 in my DTivo TyStreams and correct it.

Here is how I do it:

First I executing a temporal reference check to see that every frame is in the right order (if not I fix it).

After that I fetch the PTS time in temporal reference order

MPEG temp refs for a small SEQ can look like this:
2I 0B 1B 5P 3B 4B 8P 6B 7B
Where I == I Frame, P == P Frame and B == B Frame, The I and P frames comes before the B frames but you will show them in temp ref order hence you should get the PTS time for 0 before 2 and so on.

(PTS presentation time stamp i.e. the time the frame should be displayed).

Now for a frame rate of 29.976 the diff in PTS time between two frames e.g. 0B and 2I should be 7506 (plus minus a few ticks). If it's 6006 we have a frame rate of 29.97.

I have determine the frame rate of my Tystream during the initial probe of the stream. If it's set to 23.976 and I bounce into a frame rate of 29.97 during the check I will return a error.

When I get a error I preform a frame rate fix in this case (29.97 to 23.976) I remove frames and adjust the PTS times accordingly.

If I have made a frame rate correction I will preform a final check of the frame rate. Just so I can check if I corrected it, I'm also doing a secod temp reference check just for the sake of it.

Anyways I can see the frame rate both before check/fix and after and I correct it. The final check shows that the frame rate I write to file is near perfect but still I get problem with the A/V sync when I multiplex it :( :(.

I can mention that I on top of this make a check for drift in A/V sync - you see it's not always the diff is e.g. 7506 but instead 7509 or so. It's not much but over couple of hundred frames it can build up to a a/v sync drift of 30ms or so. Hence I make a correction of this too. In fact I can mend sync drifts of as little as 14ms depending on the frame rate.

At the moment I simply don't get what is wrong but I guess I will figure it out during the week that comes (been sick the last couple of days on top of this #$%$#% ).

Nope it's not the a/v drift check that is the touble - it's only has a very small effect and I have tested both with and without it.

It wasn't a small task to fix all this temp ref, frame rate, drift and so on - it's more or less doubled the amount of code lines from 8500 to over 15000 lines of code.

Anyways I hope you can wait a little while longer.

Cheers Olaf

jmayes
01-02-2003, 08:51 AM
First off, great work Olaf!

When you are done, we will finally be able to burn instead of wasting time chopping/rmplexing/etc on these $%^& files. I think you finally explained why some streams mplex ok and others loose sync. Get well, you can't think right when the Crud is doing it's thing.

Meantime I will continue to archive TY files.

Thankx again,
Jmayes

olaf_sc
01-02-2003, 11:54 AM
Hello Folks,

Hmm, after some tinkering/hacking of the mjpeg-tools mplex source code I think I'm on to it - I mean why I have a/v sync problem with my present code.

It has to do with repeated frames, no not the one I add or subtract but if the mpeg stream contains frames that should be repeated or not (which also depends on the chroma format).

Anyways things are geting slightly more complex :) the more I dig into the core of the problem. Back with an update soon.

Cheers Olaf

BillyZ
01-02-2003, 01:02 PM
THanks for the update Olaf--

Why is this so Non-Standard, I mean, i know that MPEG is a very forgiving format, but why do you think the tivo ppl decided to go this route? Do you think it was to help compensate for various signal strengths from the Input??


Originally posted by olaf_sc
Hello Folks,

Hmm, after some tinkering/hacking of the mjpeg-tools mplex source code I think I'm on to it - I mean why I have a/v sync problem with my present code.

It has to do with repeated frames, no not the one I add or subtract but if the mpeg stream contains frames that should be repeated or not (which also depends on the chroma format).

Anyways things are geting slightly more complex :) the more I dig into the core of the problem. Back with an update soon.

Cheers Olaf

AlphaWolf
01-02-2003, 01:33 PM
Its anybodies guess. I would suppose they strayed so far from spec because when directv started using mpeg2, there was no mpeg2 standard yet. That combined with they never intended for their transport streams to be recorded/streamed from any set top boxes, and, tivos own tystream format is not valid mpeg2 on its own anyways.

Hi8
01-02-2003, 01:57 PM
I've been experimenting with the tydemux (3.1) and IFOEdit .096B1 --

I did an extract of a SA stream, limited to one part, 512megs worth of a movie that I recorded to my SA - modified fomat/quality of 720x480 32khz 3.5mbps - original source was a DVD via the S-Video input.

ran tydemux against the .ty file, and just followed my normal procedure with IFOEdit, creating a few chapters vis CELLTIMES.TXT -- and to my amazment - it was perfectly insync. I've tried previously the same procedure with VSPLIT and upsampling the audio to 48khz - with A/V sync problems.. using the same .ty file as the source. I did notice that IFOEdit reported that it was offsetting , or starting the audio 1300 or so forgot the exact log message. I'll pay attention more closely next time.

I'm going to do some more tests, and see if I can create a complete extract via parts, and demux & burn..... I can do this perfectly with tytool5/vsplit13c and IFOEdit -- with my DTiVo streams but my SA extracts always seem to be problematic with A/V sync. I figured I'd try the burn using the 32khz audio and see if my /DVD set-top player cared... it didn't and boy am I happpy.

I'll use this method to convert some of my VHS and more so Hi8 tapes to DVD if it proves repeatable.

:p

olaf_sc
01-02-2003, 04:39 PM
Okay I more or less know* what is causing the problem.

I will do some test coding this evening to see if my theory is correct if so then I will most probably make a tmp hack and release 0.4.0 shortly even if it will not be perfect.

If it's like I think it is well then there still is a lot todo :(, and some that I have done that wasn't necessary.

Cheers Olaf

*Better than saying that I know just in case :) ).

racingclub
01-02-2003, 05:58 PM
Cheers olaf - everyone appreciates your hard work!! :)

MrBassMan
01-02-2003, 07:18 PM
Sorry if this is a stupid question but as I understand it, the problem outstanding is syncing audio and video over a long period of time?

Surely this is a by-product of spliting and then remuxing the ty stream.

Is the ty stream understood well enough now to produce a stream based codec to turn a ty stream directly into mpeg2? i.e avoid the split/remux but instead go from ty straight to MPEG2

This would avoid the long term sync problems that are seen at present.

It seems so obvious that I'm sure it's been thought of and found not possible but I would like to understand the reasons.

AlphaWolf
01-02-2003, 08:01 PM
Originally posted by MrBassMan
Sorry if this is a stupid question but as I understand it, the problem outstanding is syncing audio and video over a long period of time?

Surely this is a by-product of spliting and then remuxing the ty stream.

Is the ty stream understood well enough now to produce a stream based codec to turn a ty stream directly into mpeg2? i.e avoid the split/remux but instead go from ty straight to MPEG2

This would avoid the long term sync problems that are seen at present.

It seems so obvious that I'm sure it's been thought of and found not possible but I would like to understand the reasons.

This is not what causes the sync problems, afaik the reason you split the tystream into elementary streams is because from there its easier to make any changes to it. Even if you don't do this, the sync issues will still remain if you do not make the stream conform to mpeg2 program stream standards.

MrBassMan
01-02-2003, 08:24 PM
Originally posted by AlphaWolf
This is not what causes the sync problems, afaik the reason you split the tystream into elementary streams is because from there its easier to make any changes to it. Even if you don't do this, the sync issues will still remain if you do not make the stream conform to mpeg2 program stream standards.
1. There are loads of video editors out there that can handle MPEG2 input and loads of converters if you want a different format or want it splt. Most people would probably want MPEG2 for DVD generation.
2. Making it conform to the standards is what I suggest. The standards are well defined, I was wondering if the ty stream, was understood well enough now to do the necessary reformating. I do not understand why the long term sync issues remain when converted to an MPEG2 stream, the method of synchronising MPEG2 audio and video seems to be well defined in the standards. Is it getting the input ty stream audio/video sync sorted the problem?

If the input ty stream sync is the problem, would modeling a player help? i.e. Process the TY stream into a 'virtual player' that recodes to a MPEG2 output stream. This virtual player would have a fixed play buffer (0.5 second max) so the furthest out of sync the stream could ever get is 1/2 second. The 'virtual player' need not work in real time, the player's clock could be sped up to process streams as fast as your hardware could handle it, i.e. data is clocked in and out at the same speed, whatever clock speed you care to choose.

As you can probably guess I know very little about video streams, however I do know a lot about writing real time software to control and/or model hardware so this is why I would select that approach.

olaf_sc
01-02-2003, 08:53 PM
Hi RC

If this was just a small problem i.e. the tydemux basically worked I would release it. The sad fact that when I say problems with a/v sync then I really mean it (i.e. the present non released codebase is not producing streams that are useful).

Now as I said earlier this morning I kind of know what is wrong, I will do the test coding this evening. It's a relativly simple hack (tops out to 20 lines of code or so) - if this work I will do some final adjustments and release the code.

However note that it means that we do some things the wrong way which is a thing I generally don't like at all. Hence I need to continue coding and release a version that make the right thing as soon as possible but a interim pre version will at least give people something that some what works as it should. Anyways I will see when I make my test this evening - will as I said be back with an update.

Cheers Olaf







Originally posted by rc3105
Olaf, don't get too hung up trying to make things perfect. we're still a loooong ways from 1.0, and the more eyeballs involved the more likely sombody'll have a trick up their sleeve that solves the problem.

thanks for all the hard work and boy are we all glad you took the high road. :D

--
Riley

dlang
01-02-2003, 09:00 PM
we only have sync problems with the dtivo, the stand alone version produces a nice standard mpeg that is easy to split/mux without sync problems

if you decode/encode the data you solve all the sync problems.

however it takes a HUGE amount of processor time and losses a significant amount of quality in the process (remember that mpeg is a lossy compression)

we start with sort-of mpeg streams that are encoded with some of the best equipment you can get, from the best quality source available, for free (the studios do it)

this project and jdiner's project both are trying to take the tivo-mangled mpeg and convert it to a normal mpeg.

initially it was thought that just extracting the video and audio data out wold be good enough, but it has since been discovered that frame rate differences break this. the two projects have differing opinions on how extreme this frame rate change is and how best to manipulate the file to produce a standard mpeg file. that's where the interesting problems are happening.

looking just at frame rate, the mpeg standard says that the datastream reports the framerate and all frames will be at that rate. tivo changes the rate without reporting any change. tivo just includes a timestamp on each frame and the playback hardware just plays each frame at the appropriate time. unfortunantly DVD hardware doesn't do this and honors the framerate it's told. As a result it's not a trivial thing to convert this. Olaf thinks that there are only two frame rates in use and the datastream toggles back and forth between them and that he can duplicate frames to change the slow framerate to the fast one, jdiner thinks that frame rates are far more variable and he is working on his own approach to the problem. both are getting there, but both are running into different problems.

Hi8
01-02-2003, 10:07 PM
Originally posted by dlang
we only have sync problems with the dtivo, the stand alone version produces a nice standard mpeg that is easy to split/mux without sync problems


I've been really confused by these statements, those even by jdiner. Always stating that the SA splitting and muxing is a piece of cake and is already there. The DTivo streams are the issue.

Well it must be me ... all my Dtivo stream extractions and DVD burns are perfectly in-sync best I can detect. Using IFOEdit and Nero produces a GREAT DVD. I have been burning off my SAT60 for weeks now with a routine that is building quite a large number of disks! However getting the same consistancy from my SA is still a challange. mplex is the only muxer that seems to keep things in-sync. However it's limited to 2gig output files, which tends to be a pain for full length movies at 5.8mbps - have to extract in pieces. The other problem being the 32khz upsample stage.

jdiner's muxer seemed to be the answer, but the last release isstill crippled to limit 2000chunk output. Just when I thought jdiner was getting somewhere his releases seemed to dry-up. I hope he's ok; last I recall he had an ear infection and still had a ringing in his ear.

All this is quite alot of fun; no matter what the out come - I just wish it was abit more predictible.

AlphaWolf
01-02-2003, 10:44 PM
Originally posted by MrBassMan

2. Making it conform to the standards is what I suggest.

(in response to Hi8 as well)

And this is exactly what this software is attempting to accomplish. From what I can see, the tystreams do keep proper temporal records, however the issue at hand is that tystreams don't keep a constant framerate, which causes the loss of sync, so what olaf is doing here is compensating for this. Splitting or not, that is something that must be resolved for anything to properly sync if you want to be able to edit a tystream with just any editor you chose. If you don't care about editing, then thedoctor and jdiners muxers work fine, because they know how to properly sync video that was formerly a tystream, however no other muxing software does, so when you try to edit it, you lose sync.

dlang
01-02-2003, 11:00 PM
Hi8,
when muxing dtivo streams a lot is going to depend on what you are muxing. if you take something like a movie without commercials then you stand a good chance of keeping in sync (or if you record broadcast TV that's a constant framerate)

however if you record a channel that doesosomething like airing the program at 24 fps and the commercials at 30 fps then every time you go through 1 min of commercial the video will lag by by 15 seconds (1800 frames where 1440 were expected)

MrBassMan
01-02-2003, 11:45 PM
Originally posted by Hi8
I've been really confused by these statements, those even by jdiner. Always stating that the SA splitting and muxing is a piece of cake and is already there. The DTivo streams are the issue.

Well it must be me ...

I have a UK TIVO which I believe is the same as what you call an SA and I get a drift that after about an hour of video is very noticably out of sync, at least 2 seconds.

So it's not just you.

olaf_sc
01-03-2003, 01:06 AM
Hello


Originally posted by dlang
we only have sync problems with the dtivo, the stand alone version produces a nice standard mpeg that is easy to split/mux without sync problems


Correct and it's also the DirecTV "mpeg" video stream that is the problem. Getting that one to comfirm to std mpeg and the problem is solved. Not that easy although.




looking just at frame rate, the mpeg standard says that the datastream reports the framerate and all frames will be at that rate. tivo changes the rate without reporting any change. tivo just includes a timestamp on each frame and the playback hardware just plays each frame at the appropriate time. unfortunantly DVD hardware doesn't do this and honors the framerate it's told. As a result it's not a trivial thing to convert this. Olaf thinks that there are only two frame rates in use and the datastream toggles back and forth between them and that he can duplicate frames to change the slow framerate to the fast one, jdiner thinks that frame rates are far more variable and he is working on his own approach to the problem. both are getting there, but both are running into different problems.

Okay, not to be to much of "I know best" but jdiners way of messuring frame rate is not correct (or at least) I get the impression that he is doing it the wrong way.

Let me try to explain it. Imagine that you drive your car and look at the speedometer it says 65mph no this is the speed you drive at that precise moment. Now imagine that you only can drive in two different speeds 65mph and 45mph. During a hour long trip you drive 20min at 45mph and 40min at 65mph - now your speed during the trip is 58mph.

Now imagine that we count frames under one second instead say that we count 27 frames during one second hence the frame rate is 27 f/s - this is not true instead the instant frame rate of 25 frames was 29.97 f/s and the two remaining frames had a instant frame rate of 23.976 f/s.

Now this explains why jdiner see frame rates between 24 f/s and 29 f/s - but he also see 22, 23, 30 and 31 f/s how that that add up? Well since the two frame rates in use are 23.976 and 29.97 f/s we simply can't count frames during one second in the case of 23.976 we will count down to 22 frames and in the case of 29.97 we will count up to 31 frames during one secod due to the fact that the frame rate is not even it's not 24 f/s but nearly 24 f/s.

I hope this explains the frame rate issue so we can consentrate on what to do about it :).

Now to compicate it even more the DTivo TyStream is using frame duplication technics avalible in the mpeg specs - hence we can have a frame rate that at a glance look like 23.976 f/s but in fact is 29.97 f/s due to the fact that they show a frame twise or three times in a row. It's here my problem lies I haven't taken that into consideration when I check the frame rate.

Cheers Olaf

olaf_sc
01-03-2003, 01:49 AM
Okay folks,

I'm definitly on to it :). Now I get it in some what in sync :).

NOTE: Not perfect sync but if I have a sync offset it's some what constant throw out the recording. Hence it can be some what fixed by applying the right offset when you start the muxing.

This result I got when I started to convert the stream to progressive non interlaced from a mix of progressive/interlaced - the thing is that the DirecTV mpeg is shifting back and forth between interlaced and progressive mpeg while still useing frames and not field pictures. Further more they are shifting back and forth between duplication of pictures (done with help of repeate_first_field in the picture_coding_extension in the mpeg-2 spec).

I don't know how odd this is remember the mpeg spec is very loose and DirecTV is not mpeg-2 since they started to send before the std was finished. Anyways I think all of the issues (odd behaviour) that is present in the "mpeg" stream makes most mulitplexers go south.

I know that some of you have managed to mux DTivo streams with perfect sync. So I have I but I have also streams that aren't possible to mux properly - and we don't want that kind of problems do we?

Now my way of doing the correction at the moment is not correct in my view. Besides it's not working as properly as it should. I need to take the double frame display etc.. in to the calculation instread of removing it force fully without really checking or fixing it the correct way.

Yes it will make things even more complicated but what the heck better to make it right than to fake it so to say.

What I mean is that yes I could go a head a make a internal muxer trust me it would been a whole lot easier. Even experimented with it and it's definitly the easy way of doing it. However it's not the right way since you will not be able to edit the result. As soon as you start edit (i.e. demux/remux) you will loose sync.

When I have fixed the mpeg stream then I will make a internal muxer etc. so we don't need to write more files than necessary.

Cheers Olaf

Ahh, I have included a source "release" of tydemux - NOTE this is a developer only thing and mostly too keep the spirit of open source going. If you compile and use it it's on your own risk. It will only work on DTivo streams due to my hacks this evening. Hence please don't use it!! But you might be interested to look at it :). I will not answer any bug reports and as I stated it's not working properly!

olaf_sc
01-03-2003, 01:51 AM
Yes, you have drift of A/V sync in SA/UK tivo too but it's very easy to fix compared to the DirecTV stuff.

Cheers Olaf


Originally posted by MrBassMan
I have a UK TIVO which I believe is the same as what you call an SA and I get a drift that after about an hour of video is very noticably out of sync, at least 2 seconds.

So it's not just you.

dlang
01-03-2003, 02:52 AM
Olf, re: gframe rate. I understand both your opinion and jdiners and I don't have enough exposure to the raw streams to say which of you is right. (I suspect that in the vast majority of the situations your way is correct, but given that the dtv hardware will play things based strictly on timestamps it wouldn't surprise me if someone had started optimizing things to the point that your assumption breaks either)

in my post above I was attempting to give a overview of the problem for newcomers, not take sides :)

let the code tell the story, if you are right it's a much easier job, if jdiner is right your code will work on most streams and as it progresses the problem streams will come out of the woodwork when everything else works

BubbleLamp
01-03-2003, 03:54 AM
Not to sound like a complete *****, but do the problems you see have anything to do with 3:2 pulldown used on movies, and the way they are encoded in the stream? I know there is a lot of info regarding this and how it relates to different DVD players and the methods they use to try and sense it properly. (Seems the codecs used by the content generators (studios) don't always follow the rules, and sometimes set the flags incorrectly, or not at all.)

AlphaWolf
01-03-2003, 04:15 AM
It processes hardly anything I give it ;)

The one stream it did process though, returned repeated errors like this near the end:



PLEASE SEND A BUG REPORT - SAVE THE TYSTREAM - KNOW TYPE - UNKNOWN CONT
Check Chunk: chunk 4462 record: 31 - junk record type: e06 - size is 4028
DATA:
01 00 5e 23 28 01 08 00 45 00 0f b4 c2 5a 00 00


The 13th byte mentioned in the "DATA" was the only one that varied. It started at 5b, then incrimented by 1 until it got to the last message (when it reached FF, rolled over to 00), which had 5A. Might be something worthwhile there, I'll hold on to this stream just in case. Yeah, I know, no bug reports, just keeping our conversation active I am :)

Good news is, there are no buffer over/underruns, at all! First time I've seen it happen, and the stream plays with perfect sync, where it never did with vsplit or any older versions of tydemux, and I can remove the commercials with no problem!

dlang
01-03-2003, 04:16 AM
similar idea,except that DTV doesn't convert the movies to 30fps before broadcasting them (why waste the bandwidth) instead they let the player do it.

however they don't stick consistantly to any framerate which makes it even harder

MrBassMan
01-03-2003, 06:01 AM
Originally posted by olaf_sc
Yes, you have drift of A/V sync in SA/UK tivo too but it's very easy to fix compared to the DirecTV stuff.

Cheers Olaf

At the risk of angering the DTIVO owners, would it be possible to work towards a release for SA/UK and then return to the DTIVO problems?

Thanks

Nugget
01-03-2003, 07:18 AM
you have drift of A/V sync in SA/UK tivo too but it's very easy to fix


Umm.. as a uk tivo owner attempting to transcode to svcd/vcd/divx from tivo streams, Im very keen to find out how to 'fix' the sync drift too. (havent got much hair left from trying so far)

One workaround at the mo is to extract & demux each fsid individually, but this takes a while & isnt ideal.

vsplitmux (with minor modifications) can produce properly synced mpgs, but they (for various reasons discussed elsewhere) dont seem to be processable without losing sync. (Something I'd really like to do, and that tydemux seems to be aiming for)

btw, the current code is using fopen etc for accessing files, which doesnt work for 'large' files under windows. I did code up some #ifdef WIN32 changes to the last version to make it use the win32 api's for 'large' file access, but they didnt quite work (it worked with large files, and could create large output, but at the boundary the mpeg reset & contained data from the beginning again), I suspect I missed a place somewhere storing a file offset in an int. Is it worth porting this stuff up to 0.40 ? or is the code likely to change lots again ? I'd like to move all the file access code to a new file with the unix/win source in it, then have the rest of the code use the functions available there. (any chance of a nice cvs repository ?? ;-) )

(And maybe I was daft to try using the 0.40 dtivo only release on SA Uk tivo streams, but it seems to process ok, excepting that it consumes huge amounts of memory and cpu, with the processing getting slower and slower as it gets further into the stream (testing with a 1.6gig stream) )

Suggest next version is called 0.4.1 to better enable discussions of which version people are using, also update the version info given in the usage info, which at least on the version of 0.4.0 I just built, still reports as 0.3.1

Many thanks for the work so far.. hope to be able to start contributing some code soon...

olaf_sc
01-03-2003, 09:14 AM
Hello AlphaWolf,

It's not really a bug, but you did the right thing posting a message about it.

It's a new record type that I suspect to be present in TyStream but that I haven't encountered my self.

Is it possible for you to upload the stream to my ftp site? The IP of the site is changed to 66.121.15.35. I only neeed the end of the stream where you encounter the record type.


Cheers Olaf



Originally posted by AlphaWolf
It processes hardly anything I give it ;)

The one stream it did process though, returned repeated errors like this near the end:



PLEASE SEND A BUG REPORT - SAVE THE TYSTREAM - KNOW TYPE - UNKNOWN CONT
Check Chunk: chunk 4462 record: 31 - junk record type: e06 - size is 4028
DATA:
01 00 5e 23 28 01 08 00 45 00 0f b4 c2 5a 00 00


The 13th byte mentioned in the "DATA" was the only one that varied. It started at 5b, then incrimented by 1 until it got to the last message (when it reached FF, rolled over to 00), which had 5A. Might be something worthwhile there, I'll hold on to this stream just in case. Yeah, I know, no bug reports, just keeping our conversation active I am :)

Good news is, there are no buffer over/underruns, at all! First time I've seen it happen, and the stream plays with perfect sync, where it never did with vsplit or any older versions of tydemux, and I can remove the commercials with no problem!

olaf_sc
01-03-2003, 09:24 AM
Hello BubbleLamp

Some of has to do with that but it's not that whole truth, if it was just that it would be a minor problem.

Cheers Olaf



Originally posted by BubbleLamp
Not to sound like a complete *****, but do the problems you see have anything to do with 3:2 pulldown used on movies, and the way they are encoded in the stream? I know there is a lot of info regarding this and how it relates to different DVD players and the methods they use to try and sense it properly. (Seems the codecs used by the content generators (studios) don't always follow the rules, and sometimes set the flags incorrectly, or not at all.)

olaf_sc
01-03-2003, 09:42 AM
Hello Folks,

First of 0.4.0 is NOT RELEASED, the source code I posted doesn't work and should not be used, period! It's purely for the sake of letting people see the code progress I made so far.




Originally posted by Nugget

btw, the current code is using fopen etc for accessing files, which doesnt work for 'large' files under windows. I did code up some #ifdef WIN32 changes to the last version to make it use the win32 api's for 'large' file access, but they didnt quite work (it worked with large files, and could create large output, but at the boundary the mpeg reset & contained data from the beginning again), I suspect I missed a place somewhere storing a file offset in an int. Is it worth porting this stuff up to 0.40 ? or is the code likely to change lots again ? I'd like to move all the file access code to a new file with the unix/win source in it, then have the rest of the code use the functions available there. (any chance of a nice cvs repository ?? ;-) )


You most probably missed the file offset issue, otherwise it's the right
way to do it. btw, not using fopen but open. It's also not nessesary to use ifdefs since Windows and Unix will have the same code for accessing large files. When I release 0.4.0 it will be large file enabled as stated above it's not yet released.





(And maybe I was daft to try using the 0.40 dtivo only release on SA Uk tivo streams, but it seems to process ok, excepting that it consumes huge amounts of memory and cpu, with the processing getting slower and slower as it gets further into the stream (testing with a 1.6gig stream) )


As stated the code doesn't work for SA/UK Tivo you will get this type of error with the present code.



Suggest next version is called 0.4.1 to better enable discussions of which version people are using, also update the version info given in the usage info, which at least on the version of 0.4.0 I just built, still reports as 0.3.1

Many thanks for the work so far.. hope to be able to start contributing some code soon...

As stated above 0.4.0 IS NOT RELEASED Sorry for sounding harsh but I want to make every thing crystal clear. I will set up the CVS server as soon as I can just now coding has been much higher up on my priority list :).

Cheers Olaf

olaf_sc
01-03-2003, 09:45 AM
When the DTivo problem is slowed SA/UK Tivo will also work. Instead of coding a lot of code that only work with SA/UK I want to code generic functions that works with both SA/UK/DirecTV Tivo:s. Hence you make it work with the most complicated case and exclude some checks fixes for the more simple cases.

Cheers Olaf


Originally posted by MrBassMan
At the risk of angering the DTIVO owners, would it be possible to work towards a release for SA/UK and then return to the DTIVO problems?

Thanks

Nugget
01-03-2003, 12:38 PM
You most probably missed the file offset issue, otherwise it's the right
way to do it. btw, not using fopen but open. It's also not nessesary to use ifdefs since Windows and Unix will have the same code for accessing large files. When I release 0.4.0 it will be large file enabled as stated above it's not yet released.

On win32 to access files beyond 4gb (or 2gb, depending on use of signed/unsigned 32bit vars to store offsets...), you have to use the win32 api to be able to read/write beyond the limit. This means using CreateFile, ReadFile, WriteFile, SetFilePointer, rather than open/read/write/fstat. This is why I was coding using #ifdef WIN32, as this code isnt needed under unix where you can just recompile with the -D_FILE_OFFSET_BITS=64 type of magic to get the open/close/read to work with huge files.
(Might also be possible to code the windows stuff to do memory mapped file i/o too as a bonus)
The fstat call in tydemux.c is the current first point of failure with large files under windows, where the off_t field is only 32bits wide. Indeed, Ive seen you use open/read/write rather than fopen/fread/fwrite, I usually use the latter, and got muddled in my last post without your source to hand.

Apologies over calling the last zip of source (avoiding calling it a 'release' here) 0.4.0
I wasnt trying to label it, just needed a name by which to refer to that version of the code. The zip of source contained a readme that referred to that version as 0.4.0, ok, it wasnt released as a compiled binary, and when compiled didnt report itself as 0.4.0.. but it wasnt the 0.3.1 it claimed to be either.

When suggesting the next zip be referred to as 0.4.1, I was just trying to say that its always best to have some way of identifying code thats been released (as in, 'in the wild' -release rather then 'this is a release'-release) so when someone tries it out N months down the line, that you have some way of knowing which codebase they have run on.

So I went with the readme, and called it 0.4.0, but hey, I dont plan on wanting to start a flame war centered around versioning of code. Especially not when the time would be much better spent developing it, rather than debating version no's.

(And Im not too fussed over the last source release (what should it be called if not 0.4.0?) not working (after all, you said it wouldnt when you posted it) just wanted to give some feedback on the way it was running here)

Back to SA/UK tivo discussion.. Its nice to know the generic routine should fix the gradual sync drift seen. And yes, its always nicer to have a generic solution rather than a hack for one purpose.... But (hey, there had to be a but ;-) ) could you explain why the UK sync drift is easy to fix?
Ive seen it referred to as being 'easy' a couple of times, and have seen lengthy discussions on why dtivo streams are tricky to solve (gaps, multiple framerates, etc), but havent read anything on how/if/why the SA sync drift is there, or methods for resolving it.

Cheers,
Nugget

olaf_sc
01-03-2003, 02:23 PM
Hello Nugget,

Please see my inline comments,


Originally posted by Nugget
On win32 to access files beyond 4gb (or 2gb, depending on use of signed/unsigned 32bit vars to store offsets...), you have to use
fopen/fread/fwrite, I usually use the latter, and got muddled in my last post without your source to hand.
[snip]
Apologies over calling the last zip of source (avoiding calling it a 'release' here) 0.4.0
I wasnt trying to label it, just needed a name by which to refer to that version of the code. The zip of source contained a readme that referred to that version as 0.4.0, ok, it wasnt released as a compiled binary, and when compiled didnt report itself as 0.4.0.. but it wasnt the 0.3.1 it claimed to be either.


Hmm, I will actually use feek64 and friends under Unix, while lseeki64 under Windows - if you do the right defines this will be transparent. Now I'm not a big Windows programmer and I know that this is the older interface than the new one that you mention. Maybe it's the wrong way (in Windows) if so enlighten me, as said I'm not a big Windows programmer.

I know you can just do a define under Unix but it's not always that easy hence I rater explicitly use 64 bit file access to avoid nasty surprices :).




When suggesting the next zip be referred to as 0.4.1, I was just trying to say that its always best to have some way of identifying code thats been released (as in, 'in the wild' -release rather then 'this is a release'-release) so when someone tries it out N months down the line, that you have some way of knowing which codebase they have run on.

So I went with the readme, and called it 0.4.0, but hey, I dont plan on wanting to start a flame war centered around versioning of code. Especially not when the time would be much better spent developing it, rather than debating version no's.

(And Im not too fussed over the last source release (what should it be called if not 0.4.0?) not working (after all, you said it wouldnt when you posted it) just wanted to give some feedback on the way it was running here)


Got your point is valid maybe I should have given the zip a names such as 0.4.0-pre-not-working :). Maybe I shouldn't have let the source out in the first place but I wanted to so people don't think it's all talk and no show (Code) so to say.

When I release it will be as 0.4.0 i.e. with a name on the zip as tydemux-X-0.4.0.zip where X is the OS. Ahh, note I didn't try to bash you I just wanted to make it clear to everybody that I have not yet released 0.4.0.



Back to SA/UK tivo discussion.. Its nice to know the generic routine should fix the gradual sync drift seen. And yes, its always nicer to have a generic solution rather than a hack for one purpose.... But (hey, there had to be a but ;-) ) could you explain why the UK sync drift is easy to fix?
Ive seen it referred to as being 'easy' a couple of times, and have seen lengthy discussions on why dtivo streams are tricky to solve (gaps, multiple framerates, etc), but havent read anything on how/if/why the SA sync drift is there, or methods for resolving it.


SA sync drift explanation:

In a SA with a frame rate of 29.97 f/s the displaytime of a single frame should be 3003 ticks (3003/90 ms). Now say that we have a recording that consists of say 10000 frames - 80% of those frames has a display time of 3003 ticks while 20% has a display time of 3005 ticks (due to the fact that the tivo didn't encode properly, tymuxed properly or what ever).

Now the muxer doesn't know that fact and mux the video and audio stream as all frames had a display time of 3003 ticks hence all 3005 frames will be displayed slightly to short :(.

In all we had 2000 frames that is displayed 2 ticks to short hence we will have a drift of 4000 ticks which is a bit over one frame. Now there is no way of telling the muxer what frames that has to be displayed slightly longer and which ones that should be displayed at std lenght. What we need to do is to add a frame when we detect that the drift is larger that 3003 ticks.

Now that is not much it's only 30ms of drift, but e.g. a two hour recoding has 215784 frames and a 20% 3005 frames here will result in a drift of 86313 ticks which is a little less than a second of drift which is very noticible. (There is 90000 ticks on a second).

This explanation is also applicable to UK tivo just that it's using a frame rate of 25 f/s.

Cheers Olaf

Nugget
01-03-2003, 03:27 PM
Thanks for the reply, Im off to go read about mpeg stuffs now to make a bit better sense of what you've said :-).

From just reading it alone I understand that theres a time unit called a tick at work, thats used as a unit of measurement of how long a frame is displayed on the screen.

I guess a frame is supposed to be displayed for N ticks, then the next frame displayed.

This information is then lost somehow during muxing so that every frame ends up being marked as being displayed for an identical number of ticks (the value being calculated from the indicated fps of the stream?)

Again, a guess, that the information is lost because somehow the ty stream contains the correct number of ticks per frame, and the mpeg (should have? / assumes?) has the ticks per frame to be identical. (or is more that the ticks per frame in the mpeg must be identical if the mpeg is to be editable using normal tooling without losing sync?)

So if I understand you correctly, you say that its just a case of keeping track of the number of frames that have gone past that were requested to be displayed for too long (and possibly too little? not mentioned, but might be possible?) and from that calculate the drift that would be expected from where the video should be at that point. And if the drift has grown to greater than 1 frames worth of ticks, then add (or remove if sync too positive?) an additional frame to level off the difference somewhat.

Would the distribution of these frames with non standard display tick counts affect the visual perception of the sync ? I suppose worse case scenario is the sync is pushed out to the limit just below where a frame would be inserted, and left there. If undersized tick counts do occur, then possibly the extra frame could be inserted when the sync drifts more than half a normal frame display count from where it should be.

Interesting game, need to find out more. I'm guessing all this tickcount stuff is related to the 'presentation timestamp' stuff Ive read about in other threads. Hopefully I'll find some nice mpeg info via google.

During import of an mpeg to rempeg2, it shows a nice overview of the GOP's with their content just shown as I,B,P's after the GOP. I notice the number if I,B,P's in each GOP varies quite a lot, should this matter? Would a 'normal' mpeg (if such thing exists) have identical numbers of I,B,P's (which I think are the frame types, one of which I assume to be a full frame, another a delta, and I have no clue what the third could be (yet)).

Cheers,
Nugget

olaf_sc
01-03-2003, 04:11 PM
Hello Nugget,

Please see my inline comments,


Originally posted by Nugget
From just reading it alone I understand that theres a time unit called a tick at work, thats used as a unit of measurement of how long a frame is displayed on the screen.


The time unit tick is my very own temonlogy normaly you messure this in ms (milli seconds). Now it's so that PTS (Presentation Time Stamp) present in the MPEG PES header that is packeting the frame use a 90kHz clock. This means that there will be 90000 increments of this clock for every second. The PTS time field present in a PES header is a 33 bit integer (using 64 bit internaly in tydemux). Now I could have a function "converting" the PTS time stamp to ms but I rater just use the time stamp directly hence my new term "ticks".



I guess a frame is supposed to be displayed for N ticks, then the next frame displayed.

This information is then lost somehow during muxing so that every frame ends up being marked as being displayed for an identical number of ticks (the value being calculated from the indicated fps of the stream?)


In a Tystream the PTS time stamp is present hence Tivo know when to display a frame and will display it until it should display the next frame stated by that frames PTS. Now when we extract the stream to a raw video stream a MPEG ES stream we loose the PTS time stamp, since a PES packet which the PES header is a part of only can be present in either a MPES PES, Program or Transport stream.

Hence it's during the demuxing of the tystream we loose the sync information. You might now ask why I don't do an internal muxing or write a MPEG PES stream to a file.

Well there is very few progam that is handling editing in MPEG Program streams and if they do they most of the time is doing an internal demuxing and we loose sync anyway.

But what about a PES stream? Well there is no mulitplexer yet that can mux two or more PES streams and there are no editing programs that can edit PES streams.



So if I understand you correctly, you say that its just a case of keeping track of the number of frames that have gone past that were requested to be displayed for too long (and possibly too little? not mentioned, but might be possible?) and from that calculate the drift that would be expected from where the video should be at that point.


Yes, that is correct and yes we can have both a over run and a under run in sync - you have assumed/guess right so far :). Pretty simple acctually.



And if the drift has grown to greater than 1 frames worth of ticks, then add (or remove if sync too positive?) an additional frame to level off the difference somewhat.


Yes basically but there are ways of correcting sync drift that is only half of the display time of a frame - i.e. I will be able to fix drifts in sync that is only 16ms ((3003/90) /2 ). Further more if we see that we are slipping in sync well then we should fix it when only half of that time in drift sync is messured. In other words we will have a maximum sync drift of 8ms which should be more or less imposible to notice when you watch the recording.



Would the distribution of these frames with non standard display tick counts affect the visual perception of the sync ? I suppose worse case scenario is the sync is pushed out to the limit just below where a frame would be inserted, and left there. If undersized tick counts do occur, then possibly the extra frame could be inserted when the sync drifts more than half a normal frame display count from where it should be.


As you see we can fix sync drift when it's as little as a 1/4 + 1 tick of the display time for one frame.



During import of an mpeg to rempeg2, it shows a nice overview of the GOP's with their content just shown as I,B,P's after the GOP. I notice the number if I,B,P's in each GOP varies quite a lot, should this matter? Would a 'normal' mpeg (if such thing exists) have identical numbers of I,B,P's (which I think are the frame types, one of which I assume to be a full frame, another a delta, and I have no clue what the third could be (yet)).


No there is no need to have the same amount of frames per GOP. Infact GOPs are optional in a std MPEG stream. Most of the time you will need them and they are mandotory in VCD, SVCD, DVD MPEG program streams. There is also definitions of the maximum number of frames per GOP in a DVD MPEG program stream.

Cheers Olaf

Nugget
01-03-2003, 04:55 PM
Ok, finally I understand why theres all the fuss regarding remuxing / why sync is lost / damaged mpegs :-)

So for a ty stream to be an mpeg that plays correctly in sync, notice must be taken of this presentation timestamp information. The crucial fact being this can only be present in a full multiplexed mpeg, not in an mpeg2 video only stream.

This is why multiplexed mpegs from vsplitmux can be in sync when first created, and then promptly break once split, audio / video resampled, and remuxed. (argh!)

So for the video stream to survive muxing / remuxing, then it must be built of frames that can be displayed for identical frame counts, for the entire duration of the stream. And as the tivo doesnt record it like that, tydemux tinkers with the video stream inserting/removing frames (or partial frames) to keep the expected a/v sync on target. (and I think I read elsewhere that potentially audio can go missing too ? with suggestions of dropping video or inserting blank audio frames to maintain sync there also).

A few questions :
What should an ideal mpeg2 video stream look like then ?
Which tools are best for checking sync ? I dont trust mediaplayer (using elecard I think) as Ive seen sync differences caused just by seeking into the file (and I dont have 2hrs to wait each time to see if its drifting...) Otherwise Ive tried power dvd which seems quite good. Maybe I could use tv out from a pc to record a ty stream with sound & video pulses that could be post processed when back on the pc to see how the sync is doing.. (a sort of digital clapper board) but its a lot of work for little gain.

I suppose if the GOP structures were maintained, (as with rempeg2) there might be a way to parse a ty file whilst multiplexing an previously split & upsampled audio mpg with a rempeg processed video mpg using the GOP structures to line up the video mpg with the one in the ty file to code correct timestamps into the resulting mpeg.

With a little expansion, it might be possible to save the timestamp info & some identifying info from the ty file to a timestamp file , which could later be used to sync an mpeg. (assuming the frames could still be matched between the two.. )

Anyways, if the video mpg can survive remuxing and demuxing then with luck it should survive running thru tmpgenc also :-)

Will check back tommorow, thanks for taking the time to explain stuff so far. I'll read thru the tydemux code again and see if it makes more sense in context of what Ive learnt so far. Only thing Ive seen so far that I could improve quickly is a function called bsort that appears to be doing a bubble sort (evil) which could probably speed up a little by swapping it for quicksort.

G'Night
Nugget

tungsten2k
01-03-2003, 08:16 PM
olaf - thanks for posting the code... even though you may be thinking there are a bunch of people complaining of problems, i'm sure they don't mean it that way... we're just excited to be playing with it and want to share our experiences (we're all egotistical humans afterall ; )

and with that, it is quite exciting to have tydemux on my OSX box for the first time. did a quick compile and ran a 560mb cartoon episode through it and it went about 380mb through before breaking and giving up on the rest with a bunch of "BIG WARNING"'s. resulting sync was pretty much dead on until then.

i've had a set of scripts ready to drop this into for a few weeks now and pass off to various utils and pop out a DVD on the other side... near-zero interaction. exciting times. thx for all the hard work.

saltydog4791
01-03-2003, 10:28 PM
Hey tungsten2k,

I am a fellow OS X user as well. What kind of scripts are you referring to, and what tools are you using after converting the ty files? TIA.

saltydog4791

tMk
01-03-2003, 11:38 PM
Trying to compile the pre-release on my Linux 7.3 machine. I run a make and get an executable - but when I try to run it, I get:

./tydemux: Permission denied.


Yes - the executable bit is set - from my ls:

-rwxrwxr-x 1 kerlitm kerlitm 277274 Jan 3 21:49 tydemux

Thanks for your help in advance!

olaf_sc
01-04-2003, 12:39 AM
Sorry for sounding rude but please get 0.3.1 it has a compiled and ready to use version.

Cheers Olaf

PS: The pre release is totally useless unless you are a developer or just want to view the code.


Originally posted by tMk
Trying to compile the pre-release on my Linux 7.3 machine. I run a make and get an executable - but when I try to run it, I get:

./tydemux: Permission denied.


Yes - the executable bit is set - from my ls:

-rwxrwxr-x 1 kerlitm kerlitm 277274 Jan 3 21:49 tydemux

Thanks for your help in advance!

tMk
01-04-2003, 01:10 AM
Olaf,

Even with 0.3.1 pre-compiled version I get the same error (Permission denied) - this must be a Linux thing I am doing wrong - sorry to be such a Linux dullard :-).

Thanks tMk

BillyZ
01-04-2003, 02:02 AM
Originally posted by tMk
Olaf,

Even with 0.3.1 pre-compiled version I get the same error (Permission denied) - this must be a Linux thing I am doing wrong - sorry to be such a Linux dullard :-).

Thanks tMk

yeah....sounds like a permissions problem...


j/k


are you running this as root? try chown user.group and chmod +rwx 7777

tungsten2k
01-04-2003, 09:49 AM
at the risk of posting yet another off-topic msg... can we please keep this thread clear of support questions (especially completely unrelated linux support questions ? :D ). a better place to post newbie questions is here : http://www.dealdatabase.com/forum/forumdisplay.php?s=&forumid=37 . good luck. and fwiw, the 0.3.1 is the best working version out there and will be until you see a 0.4.0 binary release posted by olaf.

re: osx scripts - the "scripts i'm referring to" are a bunch i wrote to do post-processing of *.ty streams. trust me, at this point my scripts would probably require more coaxing to get working on someone else's system and be less understood than tydemux ;) . as for the tools, take a look at the "About" dialog inside ffmpegX for a list of most of the tools i'm using in OSX.

okay now, back to the program...

OS X
01-04-2003, 04:17 PM
Originally posted by tungsten2k
at the risk of posting yet another off-topic msg...

re: osx scripts...


tungsten2k,

It sounds like you're (at least) an OS X poweruser. I recently installed the (free and awesome) Developer Tools with the goal of figuring out how to compile open source such as tydemux and develop a pretty UI to tydemux for OS X. I bet there are some other newbies like me who could use a hand.

Let's get together and do some real kick ass Mac OS X stuff! Whaddaya say you get a thread going for Mac OS X specific issues, how-to's...?

AlphaWolf
01-04-2003, 04:23 PM
while you are at it, use gtk+ so that it would be easy to port to linux.

MrBassMan
01-06-2003, 08:47 AM
I have ported the last source that Olaf posted to Win32. The code builds without errors and appears to run (progress info on screen), however the audio and video files are empty.

The following changes have been made:

1. 64 bit versions of lseek and stat functions used to cope with >2GB files
2. 64 bit version of the abs function created
3. All compiler warnings fixed.
4. Increased storage size for some variables such as the chunk number (was int).
5. Visual Studio 6 project and workspace files created

Olaf, you can use diffs of these files compared to your last source release to find the changes. This should fix a lot of problems with large files. I have NOT compiled on other platforms but have tried to make the changes as platform independant as possible. You may need to investigate linux & MAC versions of the _lseeki64 and _fstati64 functions. For now I have defined them to lseek and fstat on non Win32 platforms.

Note to everyone else: These changes do NOT result in a working demux program. There are other issues with processing the stream that are beyond my understanding. I have posted these changes for Olaf in the hope it speeds up his development.

John Barrett

olaf_sc
01-06-2003, 12:07 PM
Thanks,

I will definitly take a look at your change and also fix the empty output :). Anyways thanks!! It's always great to get dev help.

I will post a progress report later and an explanation to why the last source release is not working as it should.

Cheers Olaf





Originally posted by MrBassMan
I have ported the last source that Olaf posted to Win32. The code builds without errors and appears to run (progress info on screen), however the audio and video files are empty.

The following changes have been made:

1. 64 bit versions of lseek and stat functions used to cope with >2GB files
2. 64 bit version of the abs function created
3. All compiler warnings fixed.
4. Increased storage size for some variables such as the chunk number (was int).
5. Visual Studio 6 project and workspace files created

Olaf, you can use diffs of these files compared to your last source release to find the changes. This should fix a lot of problems with large files. I have NOT compiled on other platforms but have tried to make the changes as platform independant as possible. You may need to investigate linux & MAC versions of the _lseeki64 and _fstati64 functions. For now I have defined them to lseek and fstat on non Win32 platforms.

Note to everyone else: These changes do NOT result in a working demux program. There are other issues with processing the stream that are beyond my understanding. I have posted these changes for Olaf in the hope it speeds up his development.

John Barrett

olaf_sc
01-06-2003, 04:22 PM
Hello Folks,

Time for some progress update and also some explanation to why the source code I released doesn’t work, as it should.

As said before in different other threads I’m not much of a MPEG guru. As a matter of fact I’m doing this work to get a better understanding of MPEG, and I’m slowly getting there.

Now ready for the bomb? There is no shift in frame rate! DirectTV tystreams are 29.97 f/s and that's it!

Now why did I think there was a shift in frame rate? Well as said before the display time at 29.97 f/s should be 3003 ticks and the display time at 23.976 f/s should be 3754 ticks.
If you now with help of the PTS time stamp look at the display time for each frame 1*, we see frames that has a displaytime of 3003 and 4505 (and 4504). If we only had frames with a display time of 3003 ticks I interpreted that as a frame rate of 29.97 f/s. And if we had frames that alternated between 3003 and 4505 I interpreted as a frame rate of 23.976 f/s after all 2 * 3754 = 3003 + 4505.

This is however way wrong. Why? Well I didn’t take into consideration that frames produces fields in a non-progressive (i.e. interlaced video sequence). Two fields makes one picture that is displayed on your TV. As we all know TV is interlaced first it shows a “top field” and then it shows a “bottom field” and your eyes will do the rest combining those two fields (pictures) into one picture. From now on when I say a picture I mean the combination of two fields to a display picture that your TV will display.

Okay so far so what was wrong? Well a frame will produce fields and it can produce two or three fields depending on the setting of repeat_first_field and progressive_frame in the picture coding extension header and that was what I missed!

The display time for a picture (two fields) at 29.97 f/s is 3003 – but since a frame can produce three fields the display time for frames that produces three fields will be 3003 + 3003/2 which is 4505 or 4504 (we can’t divide int 3003 so we will alternate 1501 - 1502).

If you in the light of the above check the Dtivo MPEG stream we will find that all frames that had a displaytime of 4505 produced three fields and all frames that produced two fields had a display time of 3003. Conclusion is that the Dtivo MPEG stream is a perfect 29.97 f/s interlaced MPEG video stream.

Now if you look at a frame the MPEG video stream has to have some sort of way telling what type of field that a frame produces or to be precise what order the fields are coming. It does that by setting the top_field_first if this flag is set the first field is a top field other wise it’s a bottom field. Hence if we have a frame producing two fields they can either come as top bottom (TB) or bottom top (BT) and a frame producing three fields can come as top bottom top (TBT) or bottom top bottom (BTB).
So besides checking that the three fields has a 4505 display time and that two fields has a 3003 display time we also need to check if the field order is right. In other words that a TB frame is followed by a TBT or TB frame, if not something is definitely funky.

As before we will naturally also need to do a check of drift but in the light of the fact that we have three and two field producing frames. And to top of the cake we will need to make a temporal reference check.

Hmm, so what is wrong with the stream why does it get out of sync when we mux it? Well seen so far is either gradual drift in sync or missing frames. Gradual drift in sync will not generate any errors while demuxing at the end of the show I just get a report how much the drift was and the sad thing is that I so far not seen a drift of more that 15 ticks on a Dtivo stream.

However if we have a missing frame we will get errors in at least the temporal reference check – this is also where so far has spotted the errors that makes us drop out of sync.

olaf_sc
01-06-2003, 04:23 PM
(the rest of the message).

As side note many people think that the temporal reference is just a number incremented by one for each frame modulo 1024. Well it is but in practically it’s not as soon as we have GOP headers in the video stream it’s not like that.

Consider this note from 6.3.9 temporal reference:

The temporal_reference of each coded frame shall increment by one modulo 1024 when examined in display order at the output of the decoding process, except when a group of pictures header occurs. After a group of pictures header, the temporal_reference of the first frame in display order shall be set to zero.

Combined with this one from 6.1.1.7:

Group of picture header is an optional header that can be used immediately before a coded I-frame to indicate to the decoder if the first consecutive B-pictures immediately following the coded I-frame can be reconstructed properly in the case of a random access.

In English this means that we need to reset the temporal reference at each GOP and it also means that there must be a I frame in each GOP. If we study video sequence header (SEQ) and GOP’s in a DirecTV mpeg video stream we will find that there is only a GOP at the start of each SEQ. This is nice since we can detect missing SEQ/GOP/I-Frames as soon as we get a reset in the temporal reference without seeing SEQ/GOP/I-Frames.

So what is wrong with the video stream why do we loose sync?

Well we have the gradual drift and this is probably most frequent in SA streams remember DirecTV have state of the art MPEG encoders so they should have drift more or less.

We also have missing frames, SEQ/GOP headers and error in the temporal reference – if any of those errors happens the muxer will go south for sure.

However what I probably think is the most common error is bad muxers if they fail to properly detect three field frames well then we will get a drift in the sync.

Hence what comes to mind is that we should fix:

Drift in sync (which I already do)
Errors in tmp ref (which I already do)
Field order errors (TB TB and not TB BT)
Missing frames/SEQ/GOP:s.


The two latter once are naturally not that easy :).

Now this is naturally what I’m hacking and been hacking since Friday and I have concept code that is mostly working.

But what about my frame rate fix code – well I have to dump it :( - just 1500 lines or so to ditch. Naa not really with a bit of polishing I can actually use it later on to enable us to turn a 29.97 f/s video stream into a 23.976 f/s progressive non interlaced video stream. Hence it’s not totally wasted time.


Cheers Olaf

1* What I define as a frame is a frame picture as defined by the MPEG 13812-2 paragraph 6.3.10 - picture_structure == 2 (11 binary).

olaf_sc
01-06-2003, 04:35 PM
Hello Folks,

Okay what I would like to get help with is to get hold of a SA or UK tivo stream that is not displaying any repairs while doing a tydemux with tydemux 0.3.1.

Further more it should loose sync gradually during e.g. a 2h recording and to top it of you should have muxed it with mplex from mjpeg tools.

If any of you have such a recoding: i.e.

looses sync gradually when muxed with mpex from mjpeg tools and demuxed with tydemux 0.3.1
Is relativly long 1 to 2 hours
Is in perfect health i.e. no repair while doing a demux with tydemux 0.3.1


If so please PM me and when okayed from me upload it to 66.121.15.35 with your name/nick included in the file name.

Cheers Olaf

PS: I really need this to be able to verify my sync fixes of SA/UK tivo streams.

Nugget
01-06-2003, 05:02 PM
Hi

I have plenty of uk SA streams to try, anything over about an hour is >2gb in size though, and windows tydemux 0.3.1 cant deal with those.

I have a few that would be good candidates though, to be sure this is all done right, what flags would you expect to be present on the tydemux command (ie what, if any, level of debug), and which flags for the mplex command ( -f 3 ? )

With no -d set at all, there are no references to any repair being performed on the streams I have, but most output something similar to this at the end. (Which I guess is just superflous data)

Check Chunk: chunk 12772 record: 26 - junk record type: 000 - skipping
Check Chunk: OOS chunk - 25 chunk(s) in a row skiped
Demux process finished.

Cheers,
Nugget

olaf_sc
01-06-2003, 05:50 PM
Hello Nugget,

Thanks for your reply,

1: Don't turn on debug - if the stream is repaird you will get a report about it. It looks like the one you get at the end the only diff is that it then goes in to repair mode. Something like this is printed on the screen.

Check Chunk: OOS chunk - 100 chunk(s) in a row skiped
Trying to repair tysteam
Tystream repaird

Hence the thing you have in the end is just fine - in fact I actually want to take a look at that stream. Yes, it superflous data but I still want to take a look at that stream.

2:
When you mplex (mjpeg-tools) do it with -f 3 (generic MPEG 2) and the sync offset that tydemux produced.

3: Now when you get one that is loosing sync then upload it to ftp://66.121.15.35/UK and put nugget in the file name. NOTE: You will not be able to see the file you due to sec.

Cheers and Thanks Olof




Originally posted by Nugget
Hi

I have plenty of uk SA streams to try, anything over about an hour is >2gb in size though, and windows tydemux 0.3.1 cant deal with those.

I have a few that would be good candidates though, to be sure this is all done right, what flags would you expect to be present on the tydemux command (ie what, if any, level of debug), and which flags for the mplex command ( -f 3 ? )

With no -d set at all, there are no references to any repair being performed on the streams I have, but most output something similar to this at the end. (Which I guess is just superflous data)

Check Chunk: chunk 12772 record: 26 - junk record type: 000 - skipping
Check Chunk: OOS chunk - 25 chunk(s) in a row skiped
Demux process finished.

Cheers,
Nugget

FreydNot
01-06-2003, 09:55 PM
Sounds like the best candidate for your testing would be a low bitrate long recording. I'm sure if I don't have something already, I could record something (are two samples better then one?)

Is there any special version of mplex to use, or can I just do a web search and download whatever I find?

It sure would be nice to be able to handle files larger then 2Gb with the windows version of TyDemux. I'll have to look into getting the linux version running on my redhat box...

olaf_sc
01-06-2003, 11:58 PM
Hello FreydNot

Thanks for your reply, please see my inline comment.


Originally posted by FreydNot
Sounds like the best candidate for your testing would be a low bitrate long recording. I'm sure if I don't have something already, I could record something (are two samples better then one?)


Yes, that would be the ideal candidate, since there will be a lot of frames per MB so to say.



Is there any special version of mplex to use, or can I just do a web search and download whatever I find?


You should download the latest version from the mjpeg-tools project at SourceFourge - the ideal one is natually the one from CVS but thats is totally optional.



It sure would be nice to be able to handle files larger then 2Gb with the windows version of TyDemux. I'll have to look into getting the linux version running on my redhat box...

2G will come just bare with me, 2GB is trivial compared to DTivo MPEG hell :).

Cheers Olaf

FreydNot
01-07-2003, 12:40 AM
Originally posted by olaf_sc
Hello FreydNot

You should download the latest version from the mjpeg-tools project at SourceFourge - the ideal one is natually the one from CVS but that is totally optional.

Since I'm trying to do this all in windows, it was challenging. I was able to find a windows port (using cygwin.dll). I found it at http://themurrays.homeip.net/downloads/tivo/tivo.html

I've got a 1.6Gig file, but its only 30 minutes. I'll have to record something to get a long file that is longer then 1 hour...



2G will come just bare with me, 2GB is trivial compared to DTivo MPEG hell :).

I didn't mean it like it sounded :) I was trying to say that it would be easier to get a file matching your request if I could use ty files larger then 2Gigs. I spent about a hour trying to get an NFS mount working between my windows box (that has the big HD) and my linux box (which runs RH7.3). I think my problem lies somewhere in my firewall arrangement. Anyway, it looks like I'm stuck using windows for now.

pbar
01-07-2003, 03:59 AM
Originally posted by olaf_sc
Hello Folks,

Okay what I would like to get help with is to get hold of a SA or UK tivo stream that is not displaying any repairs while doing a tydemux with tydemux 0.3.1.

Further more it should loose sync gradually during e.g. a 2h recording and to top it of you should have muxed it with mplex from mjpeg tools.

If any of you have such a recoding: i.e.

looses sync gradually when muxed with mpex from mjpeg tools and demuxed with tydemux 0.3.1
Is relativly long 1 to 2 hours
Is in perfect health i.e. no repair while doing a demux with tydemux 0.3.1


If so please PM me and when okayed from me upload it to 66.121.15.35 with your name/nick included in the file name.

Cheers Olaf

PS: I really need this to be able to verify my sync fixes of SA/UK tivo streams.

I have several recordings from a UK tivo of approx 1 hour duration which do not produce errors from tydemux 0.3.1 and where the sync drifts by several seconds over the duration of the recording.

Unfortunately they are all about 1.6GB in size and at 128kbps upstream they would take about 28hours to upload from home!

Are you still interested?

(I could always try uploading them from work I suppose.... )

Paul

olaf_sc
01-07-2003, 11:08 AM
Hello

Thanks for your reply and yes I'm very interested.

However I would suggest uploading them from work due to the long duration it very likely fail otherwise.

Cheers Olaf


Originally posted by pbar
I have several recordings from a UK tivo of approx 1 hour duration which do not produce errors from tydemux 0.3.1 and where the sync drifts by several seconds over the duration of the recording.

Unfortunately they are all about 1.6GB in size and at 128kbps upstream they would take about 28hours to upload from home!

Are you still interested?

(I could always try uploading them from work I suppose.... )

Paul

FreydNot
01-08-2003, 12:35 AM
You might consider using WinRar to make several smaller files from the one big one. That way if it has a problem, you can just restart from the last file.

Gruph
01-08-2003, 12:35 PM
Olaf,

I attempted to use tydemux 0.3.1 to demux a ty stream of Oz from HBO on Sunday from my DirecTivo... It was unsuccessful. I was wondering if you wanted me to wait for the new version, or if you wanted any debugging information from 0.3.1.

I am running tydemux under Mac OS X 10.2.3 at the command line.

If you want any debugging information, or the ty stream itself (1.2GB), or any other info, let me know. I work at an isp so bandwidth is not an issue to upload it.

Thanks for all your work on this project.. It's very cool! I'm very much looking forward the next release :)

Gruph

tungsten2k
01-08-2003, 02:17 PM
you actually have "0.4.0 (development)" as that is the only version that has been released as source other than the code cleanup and windows project files posted a few days ago. it claims it is 0.3.1 but that is incorrect.

olaf, would prob be a good idea to pop an arbitrary version designation in two places... the filename, and in the "usage" that's printed when no arguments are passed just to curb the spread of confusion. i like your idea of "tydemux_0.4.0d_does_not_work_will_break_your_machine_and_erase_your_hard_drive_so_don't_complain.zip" as a naming convention ;)

btw- your posts on the fps issue was quite and interesting read. reverse-engineering can be such a pain in the (_|_) but when you hit that breakthru, it can be so satisfying :D

olaf_sc
01-08-2003, 04:00 PM
Hello Folks,

Time for a ETA, this is not set in stone so take it with a pinch of salt.

Hacking and algo creation for solving the problems with the stream has so far been progressing in good pace. I estimate that 0.4.0 as in release version will be ready early next week. I'm aiming to put the last code in place this weekend plus internal testing.

Cheers Olaf

newlooper
01-08-2003, 06:26 PM
Ok, so I am a boob (wish I had a twin:D). I have been following the thread closely but I think I missed something. What do we use to get the stream off the Tivo. Can we use TyTool or do we use mfs_streams?

:(

Gruph
01-08-2003, 08:37 PM
Originally posted by rc3105
tytool is ok to extract with if you have the latest version. the tivoweb mfsstream module works nicely.

tivodvlpr's tytar server also works nicely but it's a pain to untar & cat the individual fsid's before you can process the video. the advantage with that approach though is that the show info is also extracted as an xml file. I'll be posting some utils to automate that when olaf posts 4.0

--
Riley

Riley, what kind of tools? command line / linux stuff? or ?

I'll be working to get 100% of the extraction/editing/authoring/burning process done on my Mac. I have some experience with command line scripting but not much programming, so maybe we can put something together on the Mac side of things as well.

I'm excited to see what comes out here in the next week :)

olaf_sc
01-09-2003, 05:06 PM
Hello

The return value from tydemux could be used to give the offset - one drawback is that most progs retruns 0 on sucess and every thing else is a fail. Workaound and 1000 above is error codes every thing below is sync offset.

I can send it out on stderr (not that nice) but then we don't need to worry about stdout parsing.

I will setup a link to what progs I use to extract together with a howto in the beginning of the thread. Will do that when I release 0.4.0 (not 4.0 :) )

Cheers Olaf


Originally posted by rc3105
Olaf_sc:

two requests

1. how about a command line option that will return the a/v sync offset to stdout or stderror (& perhaps suppress/divert other output to a log/logs) so a script can retrieve the offset and feed it to mplex easily. (parsing the log's not a big deal but it is terribly inelegant)

2. could you post a link in the thread (preferrably in the first post) to reference versions of the source & binaries of the muxers you are using for mac/win/linux? that way everybody is on the same page when it comes to reporting/uploading problem streams.

--
Riley

Gruph
01-10-2003, 01:28 AM
a simple command line argument would be cool to use for "script-friendly" output.

-Gruph

olaf_sc
01-10-2003, 05:51 PM
Will do :)

Cheers Olaf


Originally posted by Gruph
a simple command line argument would be cool to use for "script-friendly" output.

-Gruph

rd001
01-12-2003, 11:51 AM
Regards, Olaf.

I hope you're making progress toward your 0.4.0 release. Please don't worry quite so much over source releases just because you're GPL. Just make it the way you want it and release the working binaries and source when you're ready. Releasing interim source code is a non-issue anyway except to the very few people who are actively testing compiling the Mac versions or giving a hand on Windows portability issues.

It should be fun for you. It's still just a hobby. I don't like to see anyone bug you or Josh on your releases and the management of this board has forbidden anyone to demand source from any author. I'm just glad for anything you guys produce. And the source will be a great bonus to the hobby.

rd001
01-12-2003, 01:15 PM
Originally posted by rc3105
sittin here watching a tivo stream being broadcast by videolan server thinkin... "yep, source can be handy"Gee, Riley, I hope none of that unreleased code is GPLed. :rolleyes:

What do you mean, I can't have my cake and eat it too?

Nugget
01-12-2003, 01:40 PM
The unreleased code can be GPL'd provided the binary is also unreleased.

Just because you alter a GPL program, you are not forced to distribute your changes, provided you do not distribute the altered result in any way.

For more info see
'[GNU FAQ on GPL] Does the GPL require that source code of modified versions be posted to the public?' (http://www.gnu.org/licenses/gpl-faq.html#GPLRequireSourcePostedPublic)

Cheers,
Nugget

rd001
01-12-2003, 02:31 PM
I was just ribbing Riley a little. I don't care if he releases it (but I'd download it in a flash if he did).

olaf_sc
01-14-2003, 02:48 AM
It's time for a new release of tydemux 0.4.0 I hope you will find it useful!

The main news is that it will check and correct A/V sync - both in terms of missing frames and gradual drift of sync! You will most probably find this very useful. I have also made a port to MacOS 10.2 (jaguar).

Although not a rewrite the code base has grown with over 100% or roughly 11000 lines of code. Even if frame correction, temporal reference etc is relatively easy :) (grin) it still takes a lot of code and thinking to implement it :).

Hence we will bounce into bugs, some of the code is more or less impossible to test unless you actually have TyStreams that has that particular error. Now this doesn't mean that tydemux isn't useful for a wider audience it only means that you should report bugs as soon as you find them. As usually I have a ftp site (66.121.15.35) where you can upload TyStreams that triggers bugs or other peculiarities in tydemux. Please email me (olaf_sc@yahoo.com) if you have really big files (1GB and above) that you want to upload.



The current features of tydemux:
--------------------------------


Demultiplex of video and audio (both MPEG and AC3/a52) from all Tivo types made - SA Tivo Series 1, SA TIVO Series 2, DTivo Series 1 and UK Tivo using software revision 1.3 and up. NOTE: DTivo Series 2 should in principle be supported but I have not yet be able to obtain a TyStream from it.

Tivo Type/Series probe you don't need to worry about what type your TyStream is tydemux will detect it and use the right engine to demux the audio and video.

tydemux will probe the tystream for the correct frame rate regardless of what the sequence header in the MPEG stream is stating.

Check and correction when we have a overflow or underflow of frames.
- Overflow - This is more or less always a video sequence (gop) header and corresponding I frame that is actually missing. What happens is that tydemux doesn't catch the seq/gop hence when we count frames for the seq/gop it looks like a over flow. Tydemux repairs this problem by adding both seq, gop and I frame at the right location in the stream.

- Underflow - This is when one or several B or P frames is missing from the video sequence (gop). Tydemux will repair this by inserting a B or a P frame at the appropriate location in the stream.


Correction of A/V sync drift in TyStream, tydemux will correct gradua drift on A/V sync. Depending on frame rate tydemux will mend A/V drifts of as little as 8ms (max 20ms).

Correction of temporal reference i.e. frames out of order. tydemux will detect any error in the temporal reference in a TyStream and mend all types of temporal reference errors that can happen.

Audio probe, you don't need to worry about what type of audio a recordings has. tydemux will probe the whole TyStream, determine audio type and skip to the first chunk that has that type of audio.

TyStream repair will repair the TyStream if it detects holes/gaps in the TyStream. Up on detection of a hole/gap tydemux will align video and audio in such way that there is no or very little loss of sync. Further more tydemux will fix the start and end of the hole/gap in such way that Video and Audio doesn't deteriorate in quality.

tydemux is minimizing the Audio/Video sync offset. At the end of the execution it will tell you the offset in milli seconds.

2GB file limit removed on platforms supporting large files. NOTE: Windows version still not 2GB - need to do more research how to enable it properly. Help wanted :). NOTE: I have not tested 2GB support extensively since I don't have any files that large (they just upgraded my Tivo and I have not yet hacked it).

The audio delay that sould be used when muxing the files are the return value from tydemux. A return value of 1000 indicates usage error and a return value of -1000 indicates a error. Anyother return value is the A/V sync offset. This is very handy when scripting. Under Unix you can easily use $? after tydemux finished to get the return value and feed that variable to the command line execution of e.g. mplex.



The tydemux_W_0.4.0.zip holds Windows binaries and sources/project files from MS Visiual Studio 6.x.

The tydemux_L_0.4.0.zip holds Linux x86 binaries and source/make files for gcc.

The tydemux_M_0.4.0.zip holds MacOS 10.2 binaries and source/make files for gcc.


While all of them has the "exact" same codebase I still think it's easier to post to different packages.

Cheers Olaf

olaf_sc
01-14-2003, 02:49 AM
Notes about missing frames and drift in A/V sync:

Gradual drift of A/V sync is more or less only a problem on SA/UK Tivo. The maximum gradual drift of A/V sync I have seen on a DTivo is 50 ticks i.e. about 0.5ms (a half milli second) during a 2h recording.

SA/UK Tivos can drift a lot although, more than 2 seconds in not unusual. UK streams drift more than SA (US) once and I really don't know why it's like that. It's just something I have noticed.

SA (US) once tend to have PTS stamps / displaytimes that are more or less impossible. I have e.g. seen display times for one frame of only one tick i.e. 1/90000 of a sec then to be followed by a frame that has a display time of 6005. Something is wrong there since the normal display time is 3003 ticks. However it's not always it's like this sometimes the SA (US) simply has frames that has a displaytime of e.g. 6006 . The temporal reference is still okay so this indicates that the SA Tivo was over loaded and didn't get a chance to encode one frame. It's here we loose sync especially when it happens a lot.

UK Tivos is not like this (at least as far as I have seen) they drift in a more gradual way say 10 ticks per gop or so. However this is a lot over a one hour recording.

Anyways tydemux is naturally fixing this drift in sync that is present on some SA and UK Tivos.

Over to DTivo streams :). They are actually missing frames - when this happens the temporal reference in the gop gets out of order and any muxer will go south (loose sync) when muxing the streams. It's also here programs like Spruce etc. bails out on a temporal reference error.

Why are we actually missing frames, I mean isn't the mpeg stream sent by DirecTV proper? Yes, it is - trust me on that one. However there is a number of reasons why we miss a frame or more:

Hacked DirecTV card - when they switch crypt you loose a frame or so since your 3M card (or what ever it's called - I'm a good boy and subscribe) can't keep up.

Badly aligned antenna - if you are on the brink of getting good reception due to this matter you will most probably loose frames.

Bad weather - same here you will loose frames.

You are standing in front of the antenna :) go figure what will happen :).

Anyways if it's a badly aligned antenna etc. align it properly and you will at least minimise the problem.

Tydemux will naturally do it's best fixing the problem but if you are missing I or P frames you will see degradation (blocky) video for a sec or so. However the sync will be perfect and you will be able to edit the recording in any mpeg editor.

Note: Tydemux can report both over and under run in frames please see feature list why.

Ahh, why do I make such a trouble fixing the streams when I can just have an internal muxer? Well I want the stream to be in the right order and I want you to be able to use what ever tool you want to edit, transcode, mux etc. 0.5.0 will have an internal muxer (please see todo list) but that is more for the convenient and not because we need to force sync with help of PTS time stamps.

Cheers Olaf

olaf_sc
01-14-2003, 02:50 AM
And finally the MAC file :).

artships
01-14-2003, 04:50 AM
Originally posted by olaf_sc
It's time for a new release of tydemux 0.4.0 I hope you will find it useful!

Olaf,
It's fast! It did 9020 chunks of a 5,349,376 tyfile in just seconds. Alas, the show was actually 41600 chunks in length. The video output was 1,107,070KB, but should have been 5,141,098KB. I take it there's still some rate-limiting in the windows binary?

John,
Working on tystreams tytooled from a Sony 2k SA to a WinXP NTFS disk.

MrBassMan
01-14-2003, 05:22 AM
I have just tydemuxed a 1.6GB 1 hour UK recording, remuxed it with mplex and played the resulting mp2 file in Windows Media player with NO loss of audio sync. This is the first time EVER that I have done this. At the end there is a trailer for wild wild west with a gunshot - that was a true test of the sync - and tydemux passed with flying colours.

A BIG thank-you to Olaf, at last I can start processing the GB of ty files sitting on my server.

Some observations:
Every 1000 or so chunks, I get a message saying A/V sync is drifting beyond allowable paremeters - repairing. This is as we would expect on a UK stream and shows tydemux is working as Olaf intended.

The final message is:
A/V Sync Offset: -011ms (i.e. audio plays 000ms late use -O 011 in mplex)
The 000ms is a bit odd, probably just a simple coding error.

Next I tried a 4GB two hour recording and the old "TyStream is too small" message is back. Looking at the code it looks as though Olaf has added my >2GB file support but then commented it out again. I will reactivate it and see what happens.

John

Nugget
01-14-2003, 05:47 AM
Tested here with a 1.6gb uk sa tivo stream, mplex with -f3.
Result plays fine in PowerDVD, random seeks remain in sync where they used to lose sync toward the end.

Media player must be using a bad codec or something ( I think Ive read others having an issue with this somewhere ) as it plays ok, (at least for as long as I can wait for ), but random seeking causes loss of sync again.

I get the A/V Sync Drifted / A/V Sync now Corrected messages too. Although its nice to know its happening (and even better to know its working!! ) It might be nice to not print that all the time, and maybe collate how many times it was done at the end.

I notice the -d option vanished in this build ? Although that was configuring the level of debug output, maybe we could add a new flag for level of (non debug) detail required in the output ?

Huge thanks for finally resolving the UK sync drift issue for all us UK ppl who were affected.

(Re UK sync, is it possible that some sort of clock chip is calibrated badly in UK tivos, and its supposed to beat at exactly 50hz or something, and its slightly off. If its only used by either the audio or the video encoder, and the other has correct sync, then the two would drift apart over time (like playing two copies of the same audio cassette in two different tape decks, they will start in sync but drift apart more and more the longer they are left to play)

If I get some time, I'll look into large file support also.

Cheers,
Nugget

MrBassMan
01-14-2003, 06:12 AM
Originally posted by Nugget
Media player must be using a bad codec or something ( I think Ive read others having an issue with this somewhere ) as it plays ok, (at least for as long as I can wait for ), but random seeking causes loss of sync again.


I have the freely downloadable DivX bundle on my PC. This may be the difference you need.

p.s.
Another difference is that I used -f 8 as my ultimate target is DVD authoring

rc3105
01-14-2003, 06:28 AM
I've run 50+gig of video though 0.4 tonight and everything it processed had absolutely perfect sync.

that's including nonstandard video created with ele2pestriple, inserts that were chopped / edited (creating ALL sorts of extra wierdness) crappy signal quality recordings and other oddities not generally discussed here...


STANDARD MPEG EDITORS WORK PERFECTLY ON THESE STREAMS!!!


I chopped an hour of stargate into 40 segments, shuffled them, jigsawed it back together at random and not a segment lost audio sync. VERY impressive.

tmpgenc & spruceup are happy campers.

I've run this under windows 98, rh 7.3 & mac os-x 10.2 with as little as 128 meg of ram and mplex 1.6. ff/rw are dead on in windows media player, videolan client, powerdvd and apex/sony/emerson dvd players I tried around here.


GREAT WORK OLAF!


--
Riley

MrBassMan
01-14-2003, 07:25 AM
Hi Olaf,

I have re-enabled my >2GB code and I am trying to process a UK stream 4GB big. It starts OK but just after Chunk 3600 I get the messages:
ERROR Didn’t find last I –Frame
ERROR didn’t find a I–Frame
ERROR Didn’t find last I –Frame

It seemed to recover from this and continued processing chunks until around chunk 4450 when I start getting the error:
Repairing TyStream
Faild to repair tystream

This error message repeats forever – or rather as long as I was willing to wait.

Running under a debugger, the function get_audio_times called from within repair_tystream is failing.
I followed the code back through to the main processing loop in main() where it does a seek and loops back to read the next chunk.
I have no idea about the structure of the chunk header but what is read looks like junk to me. However, it returned from read_chunk thinking it has successfully read the chunk and tries to process it. Back in the main loop tystream->repair is still set so it tries again to call repair_tystream which again fails – this continued until I got bored of watching the error message (at least a minute).

As I said I have no idea how the stream is formatted but it appears to me as if either the whole file is corrupt from that point onwards (unlikely) or that the file is not fixed multiples of CHUNK_SIZE bytes so the seek in the main loop is not jumping to the start of a chunk so the read chunk header read code is screwed.

If this is the case, you may need to do two things:

Put more stringent chunk header validitity checks in read_chunk
Trap that you have seen a valid chunk for a while and attempt to find a chunk header by scanning from the last valid chunk seen.

This would mean the seek code in the main loop would need some sort of offset to work from to re-align it to the chunk headers.

I can send you the file but it is 4.2GB. Alternatively if you can suggest a suitable method of chopping the end of the file after the first 5000 chunks I can do that.

I will next try a different ty stream in the hope I have just picked a bad one to start with.

(Also emailed to Olaf)

Update: I have now tried another >4GB stream with similar results but in different places in the file. The I-Frame error orccurs around chunk 150 and the Repairing Stream error at chunk 5545. To me this implies it is stream data related rather than file size related as a) It occurs in a different place for each file and b) This is well before we reach 2GB when I would expect file size problems to show up.

Dibblah
01-14-2003, 09:02 AM
BuggerBuggerBugger :-)

Spent 3 hours coding up large-file support for Win32 :-(

Gotta check here more regularly.


BTW, I'm getting the same errors you are coming across. Wanna compare code?

(I've written wrappers around seek(), etc to use the native Win32 functions)

Cheers,

Allan.

MrBassMan
01-14-2003, 10:32 AM
Originally posted by Dibblah
BuggerBuggerBugger :-)

Spent 3 hours coding up large-file support for Win32 :-(

Gotta check here more regularly.


BTW, I'm getting the same errors you are coming across. Wanna compare code?

(I've written wrappers around seek(), etc to use the native Win32 functions)

Cheers,

Allan.

Olaf has already integrated my code into the 0.4.0 source release. To activate it, edit common.h to remove the commented out code and comment out Olaf's replacement definitions immediately below my code.
i.e.:
/* 64 bit file access */
#define fstat64(h,s) _fstati64(h,s)
#define lseek64(h,off,orig) _lseeki64(h,off,orig)
//#define fstat64(h,s) fstat(h,s)
//#define lseek64(h,off,orig) lseek(h,off,orig)

#define off64_t __int64
typedef struct _stati64 stat64_t;
//#define off64_t off_t
//typedef struct stat stat64_t;

BTW - As a double-check of my code, I ran a 1.6GB stream through with my modified version of tydemux that had previously worked with Olaf's released version. It worked fine. It definitely looks like large files have something odd about the stream data. My current guess is that there are blocks of data missing from the file which results in chunks not being at the boundaries assumed in the program. A possible cause of this may be at the boundary of two fsids joined by the TivoWeb export module or packet loss when transfering over my network (I export to an NFS share located on a Windows 2000 server).

Nugget
01-14-2003, 10:37 AM
Could you validate the correctness of your >2gb streams by extracting individual fsids and concatenating them together, then comparing the result against the complete extracted ty stream.

TivoApp comes with source, and uses the win32 api calls (CreateFile/ReadFile/WriteFile etc) to deal with large files, so should be able to write ok past 2gb.

I think its been said in the past that the ty file is just fsids concatenated together, of course, you may wish to test this with ty streams of less than 2gb to start with.

Cheers,
Nugget

MrBassMan
01-14-2003, 10:43 AM
Originally posted by Nugget
Could you validate the correctness of your >2gb streams by extracting individual fsids and concatenating them together, then comparing the result against the complete extracted ty stream.

TivoApp comes with source, and uses the win32 api calls (CreateFile/ReadFile/WriteFile etc) to deal with large files, so should be able to write ok past 2gb.

I think its been said in the past that the ty file is just fsids concatenated together, of course, you may wish to test this with ty streams of less than 2gb to start with.

Cheers,
Nugget

Are there UK Tivo users who already use TivoApp willing to try this?

I have not had much luck in the past getting TivoApp to work so I use the TivoWeb mfs export module as this worked perfectly first time for me. The tivoweb module only extracts full streams, not individual fsids.

lmurray
01-14-2003, 10:50 AM
finally, a splitter that works under macosX.

THANKS!!!!!!!!!!!

-lloyd-

moshmothma
01-14-2003, 11:18 AM
Hmm, I'm having no success so far. Every DTIVO stream I try to edit gets further out of sync the deeper into the stream I go. I have tried to edit with a number of tools but with the same results. I can tydemux and then mux but editing results in serious a/v synch issues. Is anyone else having success with DTIVO files?

BTW, thanks a million OLAF for your efforts!

pbar
01-14-2003, 02:29 PM
Originally posted by pbar
I have several recordings from a UK tivo of approx 1 hour duration which do not produce errors from tydemux 0.3.1 and where the sync drifts by several seconds over the duration of the recording.

Unfortunately they are all about 1.6GB in size and at 128kbps upstream they would take about 28hours to upload from home!

Are you still interested?

(I could always try uploading them from work I suppose.... )

Paul

I never manged to upload these successfully, but it looks like tydemux 0.4.0 fixes the sync problems with all of my files! I get roughly 10 messages about 'sync out of allowed range - repairing' and the result seems to be great.

Now to try transcoding them to DVD! :D


Many Thanks!

Paul

artships
01-14-2003, 03:32 PM
Am I the only one to fail to get tydemux on a WinXP box to work more than a quarter of the way through a 2-hour show from a Sony SA? I mean, it merrily did it's thing, quiting after 9020 chunks as if there weren't another 30000.

racingclub
01-14-2003, 03:33 PM
NICE ONE OLAF!!!!

This seems to work great with tystreams I've never managed to sync before - its nice to see us UK'ers being looked after!!!! :D :D :D

Thanks a lot !!

alunj
01-14-2003, 04:05 PM
Hi Olaf,
Uk tivo starts ok until about chunk 3000 odd then I get a message fixing temporal ref at which point the programs hangs.
Also the m2v produced will not play in power dvd (it will seek)
I can try to ftp the stream to u as its aboy 600M
So far all of my streams will not work :(

I know it doesnt help but these stream demux ok with vsplit


Alun

olaf_sc
01-14-2003, 04:32 PM
Hello

The windows doesn't support 2GB (and over) files - I tried to make it work but it doesn't at the moment. Stay tune for a update.

Cheers Olof




Originally posted by artships
Originally posted by olaf_sc
It's time for a new release of tydemux 0.4.0 I hope you will find it useful!

Olaf,
It's fast! It did 9020 chunks of a 5,349,376 tyfile in just seconds. Alas, the show was actually 41600 chunks in length. The video output was 1,107,070KB, but should have been 5,141,098KB. I take it there's still some rate-limiting in the windows binary?

John,
Working on tystreams tytooled from a Sony 2k SA to a WinXP NTFS disk.

Gruph
01-14-2003, 04:36 PM
Running Mac OS X. I've tried a 1.5G ty stream. Works PERFECT!

I'll be playing with another one tonight (when I get home from work) on a 3G file.. but so far, it's looking beautiful!

Thanks Olaf!!!

-Gruph

olaf_sc
01-14-2003, 04:37 PM
Originally posted by MrBassMan

The final message is:
A/V Sync Offset: -011ms (i.e. audio plays 000ms late use -O 011 in mplex)
The 000ms is a bit odd, probably just a simple coding error.


That is actually a bug in Windows - take a look at the pritint_audio_delay function in misc.c and you will see what I mean.





Next I tried a 4GB two hour recording and the old "TyStream is too small" message is back. Looking at the code it looks as though Olaf has added my >2GB file support but then commented it out again. I will reactivate it and see what happens.

John

John, first thanks for the code - it was a big help - made the Unix (Linux) part of large file support really easy. However when I tried the windows version it didn't write to file. Just 0 sized files under win98 (also compiled under win98). I don't know why and help is really wellcommed.

Cheers Olaf

olaf_sc
01-14-2003, 04:49 PM
Originally posted by Nugget
I get the A/V Sync Drifted / A/V Sync now Corrected messages too. Although its nice to know its happening (and even better to know its working!! ) It might be nice to not print that all the time, and maybe collate how many times it was done at the end.


I will do that in the future but for the "first" version I wanted a bit more extensive out put.



I notice the -d option vanished in this build ? Although that was configuring the level of debug output, maybe we could add a new flag for level of (non debug) detail required in the output ?


The debug system was beyond "repair" when I was finished with 0.4.0 - hence I disabled it. It's on my todo list to make it more useful. I hope that I will manage to do so by version 0.4.2 or 0.4.3., 0.4.1 will most probably be just a bug fix release.



(Re UK sync, is it possible that some sort of clock chip is calibrated badly in UK tivos, and its supposed to beat at exactly 50hz or something, and its slightly off. If its only used by either the audio or the video encoder, and the other has correct sync, then the two would drift apart over time (like playing two copies of the same audio cassette in two different tape decks, they will start in sync but drift apart more and more the longer they are left to play)


I don't know at all actually I just know that the drift is different from the US SA - Ahh Nugget thanks for your stream very useful even if the recoding was on the cheeasy side :).




If I get some time, I'll look into large file support also.


Would be very nice if you can do that.

Cheers Olaf

alunj
01-14-2003, 04:51 PM
Hi again
Stream stil break at 3000 chunks but after remuxing with ifo edit
sync is 100% to the point where the demux hung. (which is nice )

I have a 4 G ty too .. do i need to do that under linux ?

Alun

olaf_sc
01-14-2003, 04:51 PM
Allan,

I'm very interested to see your wrappers!! and your code in general - the present large file support for Windows doesn't work properly.

Cheers Olaf


Originally posted by Dibblah
BuggerBuggerBugger :-)

Spent 3 hours coding up large-file support for Win32 :-(

Gotta check here more regularly.


BTW, I'm getting the same errors you are coming across. Wanna compare code?

(I've written wrappers around seek(), etc to use the native Win32 functions)

Cheers,

Allan.

olaf_sc
01-14-2003, 04:57 PM
Hello

Please upload it to my ftp server - just be sure to put your name in the file name so I know what stream it is.

Cheers Olof


Originally posted by alunj
Hi Olaf,
Uk tivo starts ok until about chunk 3000 odd then I get a message fixing temporal ref at which point the programs hangs.
Also the m2v produced will not play in power dvd (it will seek)
I can try to ftp the stream to u as its aboy 600M
So far all of my streams will not work :(

I know it doesnt help but these stream demux ok with vsplit


Alun

olaf_sc
01-14-2003, 05:16 PM
Hello

Please see my inline comments,


Originally posted by MrBassMan
Hi Olaf,

I have re-enabled my >2GB code and I am trying to process a UK stream 4GB big. It starts OK but just after Chunk 3600 I get the messages:
ERROR Didn’t find last I –Frame
ERROR didn’t find a I–Frame
ERROR Didn’t find last I –Frame

It seemed to recover from this and continued processing chunks until around chunk 4450 when I start getting the error:
Repairing TyStream
Faild to repair tystream

This error message repeats forever – or rather as long as I was willing to wait.


Since I don't have the stream I can't really pin point this problem but my guess is that the chunk offset boundary is not alinged. I know that vspit is some how seeking the boundary of a chunk but since there is no starts codes I really don't know a 100% acturate way of doing this at the moment.

Even if the file is really big it would be nice if you could upload it. There is a windows version of dd but I don't know if it's working with large files. Anyhow the URL is http://www.weihenstephan.de/~syring/win32/UnxUtils.html.

If win dd is working with large file you can chop up the stream with a comand like this:
dd if=your.ty of=send-1.ty bs=131072 count=1000 skip=0
dd if=your.ty of=send-2.ty bs=131072 count=1000 skip=1000

and so on

Sedning a 4.2GB file is somewhat begging for trouble since the transmission can break and my ftp server can do resume. Hence if possible chop it up.





As I said I have no idea how the stream is formatted but it appears to me as if either the whole file is corrupt from that point onwards (unlikely) or that the file is not fixed multiples of CHUNK_SIZE bytes so the seek in the main loop is not jumping to the start of a chunk so the read chunk header read code is screwed.



If this is the case, you may need to do two things:

Put more stringent chunk header validitity checks in read_chunk
Trap that you have seen a valid chunk for a while and attempt to find a chunk header by scanning from the last valid chunk seen.

This would mean the seek code in the main loop would need some sort of offset to work from to re-align it to the chunk headers.


Hmm will try to implement it ;).


Update: I have now tried another >4GB stream with similar results but in different places in the file. The I-Frame error orccurs around chunk 150 and the Repairing Stream error at chunk 5545. To me this implies it is stream data related rather than file size related as a) It occurs in a different place for each file and b) This is well before we reach 2GB when I would expect file size problems to show up.

As we think I'm very sure that it's a chunk boundary issue, now what to do about it is the question.

Cheers Olaf

olaf_sc
01-14-2003, 05:19 PM
Hello

If you please could upload one of your failing streams so I can debug tydemux and correct the problem or at least explain it.

Cheers Olaf


Originally posted by artships
Am I the only one to fail to get tydemux on a WinXP box to work more than a quarter of the way through a 2-hour show from a Sony SA? I mean, it merrily did it's thing, quiting after 9020 chunks as if there weren't another 30000.

olaf_sc
01-14-2003, 05:22 PM
Hmm very strange what tools do you use except tydemux?

If it's possible could you upload one of your files to my ftp server?

Cheers Olof






Originally posted by moshmothma
Hmm, I'm having no success so far. Every DTIVO stream I try to edit gets further out of sync the deeper into the stream I go. I have tried to edit with a number of tools but with the same results. I can tydemux and then mux but editing results in serious a/v synch issues. Is anyone else having success with DTIVO files?

BTW, thanks a million OLAF for your efforts!

MrBassMan
01-14-2003, 05:33 PM
Just to update you on what I have been up to:
I have now processed about 20 UK .ty streams with varying success.
Some process without problems.
Some bomb out with the unrepairable stream error I have already reported
Some cause a invalid pointer error crash in tydemux
Some cause tydemux to go into an infinite loop with no error messages

Size seems to be irrelevant - at least two of the successful streams were over 3GB. One of the failures was 200MB.

mplex cannot handle >2GB files so the merged audio and video files are truncated at 2,048,002 bytes. If the source for mplex is around, perhaps this is an area someone else can look at.

I think we therefore have at least 2, possibly 3 symptoms for tydemux failures:
1. Unrepairable streams, probably related to chunk misalignment
2. GPF type crashes
3. tydemux hanging in an infinite loop with no error message.

What's next:
1. I will try to isolate the sections of the .ty files causing the problems and send them to Olaf for analysis.

2. I will immediately send the 200MB file that fails to Olaf

3. I will try to debug the crashes myself in the hope of giving Olaf more information.

olaf_sc
01-14-2003, 05:47 PM
Originally posted by MrBassMan
I think we therefore have at least 2, possibly 3 symptoms for tydemux failures:
1. Unrepairable streams, probably related to chunk misalignment
2. GPF type crashes
3. tydemux hanging in an infinite loop with no error message.

What's next:
1. I will try to isolate the sections of the .ty files causing the problems and send them to Olaf for analysis.

2. I will immediately send the 200MB file that fails to Olaf

3. I will try to debug the crashes myself in the hope of giving Olaf more information.

Sound like a good idea - please if you can isolate the problem and extract that part of the chunk it would be very helpful for me (if you send them to me :) ).

One question did you just enable the large file support and every thing worked fine? If so are you running winNT/2000/XP or Win9x/ME . It didn't work at all for me under Win98.

Cheers Olaf

Hi8
01-14-2003, 06:16 PM
Originally posted by olaf_sc


One question did you just enable the large file support and every thing worked fine? If so are you running winNT/2000/XP or Win9x/ME . It didn't work at all for me under Win98.

Cheers Olaf


Windows OS versions WinNT/2000 /XP are the only ones that support <2gig files. in NTFS --

don't even bother trying on FAT/FAT32

it's just a mystery to me why anyone even still runs a PC with win9x/ME -- it's always been junk, in the world of real OS's.

this is 2003 by the way ... the win9x generation and the cosmetic upgrades -- were NEVER intended to be dealing with Video editting, anything serious anyway ...

olaf_sc
01-14-2003, 06:33 PM
Hello Folks,


When I come home from work today I will implment a chunk verifier and chunk seek function.

Been thinking a bit :) - yes I'm actually capable of that :). Anyhow even if there is no real good start code to verify the start of a chunk there still is some nice things we can check.


1: Byte 4, bit 8 should always be 1 in Tivo version 2.x and up (counting from the byte 0 in the chunk).
2: There can be more that 256 records in a chunk that's why they extended the number of records from 8 bit int to a 16 bit int. However the smallest records we have are audio recods of 320 bytes. Plus 4 byte records in front and after such a recod. Given a 131072 byte chunk this gives us a maximum of around 1200 records in a chunk. Now it's very unlikely that it's like this and I think we can safely asumme that the max is never above 500.
3: There can't be any records bigger than 131068 bytes
4: The total size of all recods can't exceed 131068 - (16 * nr_or_recods)


1: Is a bit funky since I have seen chunks that are valid how doesn't have this flag set - namlely junks chunks in the boundary between fsid.

I hope I can use all this info to enable a func that will verify that we are reading valid data and that chunk is alined with the rest of the stream or not. If it's not aliged we can try to seek and see if we find the right offset that it's missalinged with.

Cheers Olof

olaf_sc
01-14-2003, 06:37 PM
Hello,

I know that but the system calls are still there and if the OS doesn't support the call it should map it to the correct one. What is happening under Win9x is that files written to disk is Zero in size (even if the files I open is only 500MB).

This is the thing I don't get not the fact that Win9x can't seek/write/open files larger than 2GB.


Cheers Olaf


Originally posted by Hi8
Windows OS versions WinNT/2000 /XP are the only ones that support <2gig files. in NTFS --

AlphaWolf
01-14-2003, 07:26 PM
Originally posted by MrBassMan

mplex cannot handle >2GB files so the merged audio and video files are truncated at 2,048,002 bytes. If the source for mplex is around, perhaps this is an area someone else can look at.


Theres a flag to enable 2gb support in the ./configure script. See the INSTALL document for details.


Originally posted by olaf_sc

This is the thing I don't get not the fact that Win9x can't seek/write/open files larger than 2GB.


I wouldn't even bother with 2gb support in any non-NT version of windows. :) The reason why is because the best filesystem they support is FAT32, which simply cannot exceed 2gb filesizes. The only true way that anybody who uses these OSes can get around that is by breaking up the streams.

MrBassMan
01-14-2003, 07:27 PM
Originally posted by olaf_sc
Sound like a good idea - please if you can isolate the problem and extract that part of the chunk it would be very helpful for me (if you send them to me :) ).

One question did you just enable the large file support and every thing worked fine? If so are you running winNT/2000/XP or Win9x/ME . It didn't work at all for me under Win98.

Cheers Olaf

I have written a program called tychopper to extract the dodgy parts of a stream:
tychopper infile startchunk endchunk outfile

I am currently testing it and will release it here when I am happy with it. It should allow me and others to just give you the interesting parts of a ty stream. The process will be:

Run tydemux and note the chunk numbers when it goes wrong.
Run tychopper to extract the bad section.
Re-run tydemux on the extract to make sure you have the bad bit.
Send the extract to Olaf for analysis.

I am running WinXP and yes I have succesfully extracted several streams > 3GB. Note though that mplex cannot work with large files yet, I have not tried tmpgenc yet.

JeffDogg1979
01-14-2003, 07:45 PM
Olaf, I have a problem with tydemux...it's probably just my configuration, but what does this mean?

C:\tydemux>tydemux -i "d:\dtivo\Crusade-War Zone.ty" -a "d:\dtivo\Crusade-War Zone.m2a" -v "d:\dtivo\Crusade-War Zone.m2v"

TyStream is to small to demux
We need at least 40 chunks

This is a piece of content recorded at Best quality...on a SA Tivo.

Hi8
01-14-2003, 07:45 PM
Originally posted by AlphaWolf
Theres a flag to enable 2gb support in the ./configure script. See the INSTALL document for details.



I wouldn't even bother with 2gb support in any non-NT version of windows. :) The reason why is because the best filesystem they support is FAT32, which simply cannot exceed 2gb filesizes. The only true way that anybody who uses these OSes can get around that is by breaking up the streams.

I'm assuming you are referring to the unix ports... mplex does NOT support large file support under win32 ... even though it provides a commandline switch (which does nothing)

--max-segment-size|-S size
Maximum size of output file(s) in Mbyte (default: 2000) (0 = no limit)

just by you reffering to ./ indicates to me you are talking about a unix port... if you HAVE a version of mplex working under win32 that works >2gig PLEASE provide the cmd examples.... I've asked this several times in different threads without any responses.

I've tried using the .040 version of tydemux after editting a DTVIO stream mux'd with MPEG2VCR and all seems fine till I re-mux (with MPEG2CR) and I lose sync as always.

I'm sure it would be fine if I mux'd with mplex, but the 2gig barrier is there.

MrBassMan
01-14-2003, 07:47 PM
Originally posted by JeffDogg1979
Olaf, I have a problem with tydemux...it's probably just my configuration, but what does this mean?

C:\tydemux>tydemux -i "d:\dtivo\Crusade-War Zone.ty" -a "d:\dtivo\Crusade-War Zone.m2a" -v "d:\dtivo\Crusade-War Zone.m2v"

TyStream is to small to demux
We need at least 40 chunks

This is a piece of content recorded at Best quality...on a SA Tivo.

Your ty stream is > 2GB
We are working on a solution.

JeffDogg1979
01-14-2003, 07:51 PM
Originally posted by MrBassMan
Your ty stream is > 2GB
We are working on a solution.

I figured that's what the problem was. I scanned 12 or so pages of information so I probably missed the answer :)

I'd love to help you guys fix the code or contribute code, but lately I've been completely hosed after work to even look at a compiler outside of work. Must be the winter weather...

MrBassMan
01-14-2003, 07:57 PM
Here it is, it is unsupported and you are free to do what you like with it, the source is included. The linux & Mac users are free to port it to their environments, I have only compiled & tested under Win32. The Windows .exe and the .c file are in thie attached zip file.

If you have a bad stream, use this to extract the bad bit so you end up with a smaller file to upload to Olaf for analysis.

Olaf: I have started extracting the bad streams and I am uploading them to your ftp server in the UK directory. They will be named MrBassMan_Badnn.ty

Update: The files are uploading as I type. There are currently 3 representing the main types of errors I see when running tydemux. There will be 2 more, one for crashes and one for infinite loops. I will upload these when I have isolated the bad chunks.

Update 15/1/03 00:55GMT: I found a bug in tychopper which I have fixed. The new version has been uploaded to this post.

olaf_sc
01-14-2003, 08:32 PM
Hello Folks,

It looks like the large file support code is working just fine under Win2000/XP. But enabling it under Win9X makes tydemux write zero size files.

Now I have located a nice pice of code at msdn that lets me detect OS version at run time. This means that I will be able to use the right functions depending on what type of Windows we use pretty neat I think :).

Cheers Olaf

olaf_sc
01-14-2003, 08:37 PM
Great Bassman -

Really neat - a note to all of you uploading files. If you detect a error at e.g. chunk 1466 then I need at least 30 chunks before the error to be able to analyze the stream in a effective manner. Hence you should start the chopping at chunk 1436.

Bassman, I'm really looking fwd to analyze the streams - so we solve the problem. Will keep you all updated.

Cheers Olaf



Originally posted by MrBassMan
Here it is, it is unsupported and you are free to do what you like with it, the source is included. The linux & Mac users are free to port it to their environments, I have only compiled & tested under Win32. The Windows .exe and the .c file are in thie attached zip file.

If you have a bad stream, use this to extract the bad bit so you end up with a smaller file to upload to Olaf for analysis.

Olaf: I have started extracting the bad streams and I am uploading them to your ftp server in the UK directory. They will be named MrBassMan_Badnn.ty

Update: The files are uploading as I type. There are currently 3 representing the main types of errors I see when running tydemux. There will be 2 more, one for crashes and one for infinite loops. I will upload these when I have isolated the bad chunks.

tungsten2k
01-14-2003, 08:52 PM
first : is everyone as excited as i am ? :D olaf, i want to first offer my astonishment that you have made such a large amount of progress in such an amazingly short time. no other grassroots project i've ever followed has had movement like this. very exciting.

okay, on with the report... I just spent the last hour or so playing with the released 0.4.0 win32 and darwin binaries. i ran my std benchmark suite of 6 ty streams consecutively through it and interestingly found a hiccup exhibited in the win32 binary. the problem does not appear to be in the darwin binary, nor was it in the 0.3.1 win32 binary. read on...

at approx chunk 3300, nearly done with the ~500MB stream, it came to a screaching halt and began to chew thru memory like it was free (it pretty much *is* isn't it ? ; ) then once it exhausted vm the system would enter "sink or swim" mode and began to dump all resources. once once back down to a reasonable level, tydemux would start chewing up processor, and again memory use would climb. it went through this cycle for the remaining hundred or so chunks until it finally finished.

each machine has 512MB ram so i took one 256 stick out of each thinking i would get the same prob on the darwin box... nope... bliped right past it without flinching... no appreciable memory usage at all. attached is a picture of the win32 behavior for the curious. i would have taken more memory out of the OSX box but i don't have any sticks smaller than 256MB so i couldn't try it with any lower real RAM.

olaf, let me know if you want me to run any other test on this stream, or if you want me to upload it somewhere. even if it is just garbage in the stream from a goose flying past the dish ;) it might be nice to find out why tydemux on win32 got slapped so hard.

as a side note, to add to the "absolutely worthless benchmarks" pile... to process 6 ty streams totalling 2.8GB:

0:14:49 - 0.3.1 win32 (724MHz/112MHz/256kB PIII)
0:11:35 - 0.4.0 darwin (292MHz/83MHz/512kB G3)

except for the video cards (GForce2MX in pc vs Rage128Pro in mac) everything else is the same on each machine from the RAM right down to the the 29160 controllers and Atlas10k hds.

Disconnect
01-14-2003, 08:52 PM
Not just for win9x types - it'd make my life easier too (since a bunch of my streams are in single-fsid pieces. I know they can be catted together, but olaf suggested avoiding that for various reasons that don't spring immediately to mind..)

JeffDogg1979
01-14-2003, 10:51 PM
Originally posted by olaf_sc
Hello Folks,

It looks like the large file support code is working just fine under Win2000/XP. But enabling it under Win9X makes tydemux write zero size files.

Now I have located a nice pice of code at msdn that lets me detect OS version at run time. This means that I will be able to use the right functions depending on what type of Windows we use pretty neat I think :).

Cheers Olaf

I don't want to sound ungrateful, but any chance you can post a newly compiled binary with the large file support code with the caveat of "Win2k/XP" only?

If not, I'll just have to bust out my Visual Studio discs and figure out where to enable it, unless someone can point me to the file to uncomment it.

laserfan
01-15-2003, 12:46 AM
Has anyone been successful resampling US/SA tys to 48kHz? I just tried BeSweet and the sync is quite a bit off.

olaf_sc
01-15-2003, 02:42 AM
Hello MrBassMan

Got all five of your files,

Found and corrected the bug present in 1 2 3 and 4.

File 4 is the one that your chunks are not aligned - they are off by 0x400 bytes (hex). However I need some chunks before the error to see what is really happening. If you could be most kind and upload some additional chunks before that error.

File 5 has a very strange temporal reference and I actuall have't seen before (seen a lot of wired once that are valid) - it's valid but my check is determining it to be invalid - I also fail to repair it :(. I should be able to dissasmble any gop and put it back together. This bug is definitly hard to crack :(. Will however make some sort of escape so tydemux doesn't hang but exits at least as a first aid so to say.

RC3105 also uploaded some files that made tydemux crash in another place Those bugs has also been mended.

However Dissconnect is reporting sync drift in two DTivo streams - my guess here is that the audio is some what wacky since it for sure is not the video I will detect that 100%. Hence I need to check that out also.

I hope to be able to fix the remaining bugs tomorrow.

Cheers Olaf



Originally posted by olaf_sc
[Great Bassman -

Really neat - a note to all of you uploading files. If you detect a error at e.g. chunk 1466 then I need at least 30 chunks before the error to be able to analyze the stream in a effective manner. Hence you should start the chopping at chunk 1436.

Bassman, I'm really looking fwd to analyze the streams - so we solve the problem. Will keep you all updated.

Cheers Olaf

olaf_sc
01-15-2003, 02:46 AM
I have asked MrBassMan to post his binary (no source since I want to integrate that to 0.4.1 first).

Cheers Olaf


Originally posted by JeffDogg1979
I don't want to sound ungrateful, but any chance you can post a newly compiled binary with the large file support code with the caveat of "Win2k/XP" only?

If not, I'll just have to bust out my Visual Studio discs and figure out where to enable it, unless someone can point me to the file to uncomment it.

olaf_sc
01-15-2003, 02:53 AM
Hello

Please see my inline comments,


Originally posted by tungsten2k
first : is everyone as excited as i am ? :D olaf, i [snip]
very exciting.


Thanks :)



okay, on with the report... I just spent the last hour or so playing with the released 0.4.0 win32 and darwin binaries. i ran my std benchmark suite of 6 ty streams consecutively through it and interestingly found a hiccup exhibited in the win32 binary. the problem does not appear to be in the darwin binary, nor was it in the 0.3.1 win32 binary. read on...

at approx chunk 3300, nearly done with the ~500MB stream, it came to a screaching halt and began to chew thru memory like it was free (it
[snip]
stream from a goose flying past the dish ;) it might be nice to find out why tydemux on win32 got slapped so hard.


Hmm it would be very nice if you could upload that file to my ftp server ftp:/66.121.15.34 please put your nick in the file name som I can locate the file. Before that it's not much I can do but guess but let us identify the problem and fix it :).



as a side note, to add to the "absolutely worthless benchmarks" pile... to process 6 ty streams totalling 2.8GB:

0:14:49 - 0.3.1 win32 (724MHz/112MHz/256kB PIII)
0:11:35 - 0.4.0 darwin (292MHz/83MHz/512kB G3)

except for the video cards (GForce2MX in pc vs Rage128Pro in mac) everything else is the same on each machine from the RAM right down to the the 29160 controllers and Atlas10k hds.


Well the drawin binaries are optimized with -O while the Win32 aren't at all - might make that diff.

Cheers Olaf

MrBassMan
01-15-2003, 04:41 AM
Originally posted by olaf_sc
I have asked MrBassMan to post his binary (no source since I want to integrate that to 0.4.1 first).

Cheers Olaf

Here it is.
Please note: this does NOT contain the stream fixes Olaf mentioned in an earlier post. This is simply 0.4.0 with large file support enabled. I have found it successfully processes 1 in 5 of my UK streams.

Dibblah
01-15-2003, 04:51 AM
Originally posted by olaf_sc
Allan,

I'm very interested to see your wrappers!! and your code in general - the present large file support for Windows doesn't work properly.

Cheers Olaf

It does work properly - Under W2k, XP, etc. It's <Win98 that's broken. I've sent you an email with the code.

Cheers,

Allan.

artships
01-15-2003, 04:56 AM
Originally posted by olaf_sc
If you please could upload one of your failing streams so I can debug tydemux and correct the problem or at least explain it.

No.

Funny thing. As I said, it thought it was done after chunk 9200. So I used tychopper to create a 43001-sized chunk, starting with chunk 9000. tydemux told me, "Tystream is too small to demux" (it was just over 4G in size).

Fine. I tychoppered a 20000-sized piece, 2.44G, starting at 9000. Again, "Too small..." at 2.44G.

Ok, I tried a 12000-sized piece. It ran to completion, well past whatever was at the original chunk 9200.

alunj
01-15-2003, 05:06 AM
Hi olaf I have the two streams that are giving me grief in work but I cannot connect to your ftp
do I have the right address ?
66.121.15.34

Alun

MrBassMan
01-15-2003, 05:15 AM
Originally posted by olaf_sc
That is actually a bug in Windows - take a look at the pritint_audio_delay function in misc.c and you will see what I mean.
Cheers Olaf

You are right, it looks like a bug in the Windows stdio library. I have emailed you a revised version that gets around the problem. It also copes with an offset of zero.

MrBassMan
01-15-2003, 05:21 AM
Originally posted by alunj
Hi olaf I have the two streams that are giving me grief in work but I cannot connect to your ftp
do I have the right address ?
66.121.15.34

Alun

Olaf is probably asleep so I thought I would respond on his behalf - It should be 66.121.15.35.

I logged on as anonymous and put my email address as the password.

olaf_sc
01-15-2003, 06:23 AM
Okay so this is a large file support error under Windows, please test the version posted by MrBassMan.

Cheers Olaf



Originally posted by artships
Funny thing. As I said, it thought it was done after chunk 9200. So I used tychopper to create a 43001-sized chunk, starting with chunk 9000. tydemux told me, "Tystream is too small to demux" (it was just over 4G in size).

Fine. I tychoppered a 20000-sized piece, 2.44G, starting at 9000. Again, "Too small..." at 2.44G.

Ok, I tried a 12000-sized piece. It ran to completion, well past whatever was at the original chunk 9200.

racingclub
01-15-2003, 06:53 AM
Hi All,

I've now tested a few more UK streams - and they seem to cause TMPG to fail (crash).

I'm trying to re-encode the .m2v and .m2a as SVCD's using the simple TMPG project wizard with no 'frills'. I seem to be able to encode small sections with no problems.

I can use the TMPG MPEG tools to mux the 2 files together without any problems - but when I run the resulting .mpg through the TMPG SVCD wizard I get a crash in the same place.

I've also tried authoring a DVD with IfoEdit then running the VOBs through DVD2SVCD (which uses CCE) - this works fine and gives me a 'good' SVCD with proper sync.

Is anyone else (maybe just in the UK) trying SVCD creation?

alunj
01-15-2003, 09:14 AM
Uk streams are on the way 1st has finished
alunj1.ty and alunj2.ty ongoing !
Alun

Nugget
01-15-2003, 10:45 AM
Originally posted by racingclub

I've now tested a few more UK streams - and they seem to cause TMPG to fail (crash).

Is anyone else (maybe just in the UK) trying SVCD creation?

Yes, 'trying' being the key word here, hitting the same problem as you, feed it into tmpgenc, and try to encode to svcd or vcd, and it will crash, always at the same place per stream, but the place does vary per mpeg.

I found a MPEG2 plugin on vcdhelp.com that you can use with tmpgenc to get it to read mpeg2 files. That stops the crashing, but I'm not happy with the resulting mpeg, which seems to glitch on playback every 10 secs or so.

I'm looking into 'frameserving' next, using DVD2AVI, a method that seems popular elsewhere on here, but I havent tried it yet.

Will try the IFOEdit route you suggest, and see how it gets on.

Cheers,
Nugget