PDA

View Full Version : hdemux: New tystream demuxer



Pages : [1] 2

bcc
07-22-2004, 01:48 PM
hdemux is a ty stream demux program. It will convert a .ty/.ty+ file into its elementary audio and video streams. The results of this program can be fed to an open source multiplexer, to remux the streams into (hopefully) standard mpeg2. There are two multiplexers that have been known to work with hdemux output: mjpegtools' (http://mjpeg.sourceforge.net) mplex, and ffmpeg (http://ffmpeg.mplayerhq.hu/)

For both multiplexers, a patch is needed to allow their mpeg2 multiplexers to employ enough virtual buffering to avoid under-runs at standard HD data rates. Ie you *will* need to run a specially patched version of these programs if you intend to multiplex HD recordings. I have pre-built versions of these available in other DDB threads.

For ffmpeg, another special patch is needed to deal with some encoder's habit of encoding a variable number of fields per frame. (directv is the most notable encoder, but some OTA encoders such as FOX's employ this encoding as well). Standard DVD-compliant muxers/demuxers do not handle this.

For mplex, there is no such patch, and the resultant mpeg video will not play at the correct rate.

Latest version of hdemux is released as part of the tytompg package, here. (http://www.dealdatabase.com/forum/showpost.php?p=268791&postcount=1)
Mplex binary compiled for windows with HD support is here. (http://www.dealdatabase.com/forum/showpost.php?p=192328&postcount=52)
ffmpeg binary here. (http://www.dealdatabase.com/forum/showpost.php?p=258392&postcount=2)

GOALS:
A tool that can convert an HD tystream to standard mpeg format as simply as possible, with solid A/V sync. This conversion should not require transcoding. Once converted, your favorite mpeg editing/transcoding/playback tools will hopefully be able to do the rest (conversion to vob, re-transcoding, non-linear editing). I say hopefully in part because it is not safe to assume that other tools work with 1920x1080i resolution. Also because I don't really know what will/won't work :)

Non-goals:
Editing support (editing typically produces discontinuities in the stream, and I believe this is best handled after the content has been converted from tystream format to a standard, more portable format).

Typical Usage:
hdemux -i <show>.ty -v <show>.m2v -a <show>.m2a
then:
mplex -r <data-rate> -O <delay> -f 8 <show>.m2a <show>.m2v -o <show>.mpg
OR
ffmpeg -acodec copy -vcodec copy [-f ac3] -i <show>.m2a -itsoffset <delay> -i <show>.m2v <show>.vob

where,

<data-rate> is the target muxing data rate, as reported from the demux stage
<delay> is the a/v timestamp delta, as reported from the demux stage

See the ffmpeg/mplex threads at DDB for more details on the multiplex step.

example:


% hdemux -i cbs.ty -v c.m2v -a c.m2a
Version 0.17
Source is cbs.ty
Video frame size is 1920x1080 - high definition. Frame rate 29.970030
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
AC3 audio at 48kHz, 1280 bytes/sync frame
6246400 audio stream bytes 299222905 video stream bytes
Reported datarate 65320 Kbit/sec. (65000+320)
Proceed with remuxer:
mplex -f 8 -O 358ms -r 153701 c.m2a c.m2v -o <outfile>.mpg
OR
ffmpeg -acodec copy -vcodec copy -f ac3 -i c.m2a -itsoffset 0.3587 -i c.m2v <outfile>.vob
% file c.m2*
c.m2a: ATSC A/52 aka AC-3 aka Dolby Digital stream, 48 kHz,, complete main (CM) 2 front/0 rear, LFE on,, 320 kbit/s reserved Dolby Surround mode
c.m2v: MPEG sequence, v2, MP@HL interlaced Y'CbCr 4:2:0 video, HD-TV 1920P, 29.97 fps
% ffmpeg -acodec copy -vcodec copy -f ac3 -i c.m2a -itsoffset 0.3587 -i c.m2v c.vob
FFmpeg version SVN-r6543, Copyright (c) 2000-2006 Fabrice Bellard, et al.
[text deleted]
Input #0, ac3, from 'c.m2a':
Duration: 00:02:36.1, start: 0.000000, bitrate: 320 kb/s
Stream #0.0: Audio: ac3, 48000 Hz, mono, 320 kb/s
Input #1, mpegvideo, from 'c.m2v':
Duration: 00:00:36.8, start: 0.000000, bitrate: 64977 kb/s
Stream #1.0: Video: mpeg2video, yuv420p, 1920x1080, 65000 kb/s, 29.97 fps(r)
Output #0, vob, to 'c.vob':
Stream #0.0: Video: mpeg2video, yuv420p, 1920x1080, q=2-31, 65000 kb/s, 29.97 fps(c)
Stream #0.1: Audio: ac3, 48000 Hz, mono, 320 kb/s
...

To-do:
Robustness so that the tool can handle streams with data errors.


Disclaimer:
This program is release-candidate quality, and is being provided in the interest of information sharing. Any errors in the ty stream and it'll probably blow up instead of correcting the a/v discontinuities.

redstone
07-22-2004, 02:07 PM
This is exactly what the HDTIVO folks need to make playable mpeg2 files.

That is the most useful format. One can make VOB files if one wants for DVD authoring,Play the mpeg2 directly in WinDVD, dump it to a D-VHS for direct playback by the deck or use on one of the PC HDTV cards to play it back.

Your effort is much appreciated!!

Hate to ask this but:

I do Unix at work and Windows at home so I would like to recompile this in Windows.

Looking at your makefile, I don't see any dependencies on any Linux libraries.

Am I correct in that assumption?

Thanks a lot

Oops - almost missed something. Is there a windows version of mplex?


JQ

bcc
07-22-2004, 05:16 PM
Thanks for the appreciation. No, there's no linux binary dependency except for the mplex program. Hmm, I hadn't noticed but it looks like mplex wasn't designed to support windows. One might have to look at what tystudio did to get mplex compiled for windows.

I take it back: I see there is win32 support in mplex code & config scripts. I got mislead by a readme comment about lack of support for anything but linux.

redstone
07-23-2004, 09:44 AM
I take it back: I see there is win32 support in mplex code & config scripts. I got mislead by a readme comment about lack of support for anything but linux.


Where?

I have been hunting through sourceforgre for 2 hours but the only windows mplex (source or binary) I found is in this forum in a zip file.

It is the binary but an older version.

redstone
07-23-2004, 11:41 AM
bcc,
Any chance you could try to build hdmux in windows?

I have tried but there are too many differences between the data types in linux vs windows so I am getting a zillion compile errors.


I am using Visual Studio 6

eg:
uint64_t is undefined
long long syntax is not supported so I tried long double but that trips a bunch of errors.

I did find many of the data types (like u_char) in different .h files like <winsock.h> and <basetsd.h> under windows



Kinda stuck here.

alldeadhomiez
07-23-2004, 12:08 PM
bcc,
Any chance you could try to build hdmux in windows?

I have tried but there are too many differences between the data types in linux vs windows so I am getting a zillion compile errors.


I am using Visual Studio 6

eg:
uint64_t is undefined
long long syntax is not supported so I tried long double but that trips a bunch of errors.

I did find many of the data types (like u_char) in different .h files like <winsock.h> and <basetsd.h> under windows



Kinda stuck here.

Try cygwin.

bcc
07-23-2004, 10:37 PM
bcc,
Any chance you could try to build hdmux in windows?Here you go. No data type errors using cygwin (just had to fix a portability problem with open()). Have I mentioned that I think the windows development environment blows? :)

The latest windows release is bundled into the hdemux zip. Just extract the .exe from that release, and also cygwin1.dll from the cygwin.zip. See:
http://www.dealdatabase.com/forum/showpost.php?p=192328&postcount=52

bcc
07-24-2004, 01:02 AM
Where?The source code, per my install instructions (file INSTALL) is at http://prdownloads.sourceforge.net/mjpeg/mjpegtools-1.6.2.tar.gz I have yet to get it to compile with cygwin (some undefined references at link time).

rung
07-24-2004, 10:51 AM
(just had to fix a portability problem with open()) Yeah, I saw that too when I tried to compile. How did you fix, just remove the LARGEFILE option?

Thanks.

bcc
07-24-2004, 02:25 PM
Yeah, I saw that too when I tried to compile. How did you fix, just remove the LARGEFILE option?I made its use #ifdef linux. I wish LBA48 support was clean&portable... See updated source zip if you want to see exactly.

redstone
07-24-2004, 08:10 PM
The source code, per my install instructions (file INSTALL) is at http://prdownloads.sourceforge.net/mjpeg/mjpegtools-1.6.2.tar.gz I have yet to get it to compile with cygwin (some undefined references at link time).


I gave up on Windows so I set up another PC with Linux.

I can not get past the first step in building mjpegtools.

Could you please tell me what options you put for ./configure ?

or better yet, please build the patched mplex and post it?

Thanks,
JQ

bcc
07-24-2004, 11:23 PM
I gave up on Windows so I set up another PC with Linux.Ah, good.
I can not get past the first step in building mjpegtools.Probably you're missing a required program or library. The compilation was straightforward:
tar xzf mjpegtools-1.6.2.tar.gz
cd mjpegtools-1.6.2
./configure
cd mplex
patch < $path_to_hdemux/mplex.patch
cd ..
make

or better yet, please build the patched mplex and post it?Here you go. Make sure you put the shared libraries in some place already in your shared library path (/usr/local/lib?)

redstone
07-25-2004, 12:50 PM
Thanks for the help.

I finally got mplex to build. It was not a Linux issue. Turns out that one should not use WINZIP in Windows to untar the mjpegtools distribution cause it somehow messes up the configure script.

So, I have built the patched mplex and your hdemux under Linux.

FYI for folks that don't want to fool with building a complete Linux system.
Get the Kloppix Linux CD IMage. You boot your PC from that CD image and you have a complete Linux system.
I built the 2 tools under that.


Results are very good but as bcc noted, work still needs to be done.

I converted the following .ty files into .mpg files with bcc's tools:
1)Six Feet Under
2)HDNET test pattern
3)Biker
4) In and Out
5) Law and Order
6) Rides
7) Dion Live

Results:
I am playing these all with my MYHD card cause that is already connected to my HDTV and my Audio Receiver.

If using MYHD, please make you use at least Version 1.63 of the SW.

All seven of these short clips are nearly perfect or have at most a few <1 second audio discontinuties in them.

First time I have ever seen the biker clip play in sync:)


Great job bcc.

JQ

bcc
07-25-2004, 03:45 PM
1) Six Feet Under is basically unplayable - it only plays for 3 seconds and no AV.
5) Law and Order has serious lip sync problemsThat's pretty strange, I still maintain that I have neither of these problems with these clips using MYHD. I can even play them back using windows media player/powerdvd/windvd, tho 2.4ghz is not quite enough to keep the audio from dropping out sometimes (when using software playback).

bcc
07-25-2004, 04:19 PM
I'm running the 1.63 release of MYD (driver&app). Maybe you're running one of the million betas?
A couple more ideas:
You applied my patch to mplex before compiling right? I'm actually thinking that's less likely to be the problem as you said you had problems with the law.mpg I posted as well (yet I do not).
Have you tried software playback? wm9/powerdvd/windvd/elecard?

redstone
07-25-2004, 06:30 PM
I'm running the 1.63 release of MYD (driver&app). Maybe you're running one of the million betas?

Thanks for the tip. I was running an older version (1.6.2.3) and did update to 1.63.

Now law and order plays perfectly in MYHD.

I also found that somehow my copy of Six Feet Under had gotten corrupted and even though the file size of the .ty file was the same, it would not play in mplayer.

Grabbed a copy I had saved onto a DVDRW and re-converted that one.
Six feet under now plays fine.

Now we are batting 100.

Thanks for all the help.

JQ

redstone
07-25-2004, 06:59 PM
I did discover that MYHD or D-VHStool will only write native .tp or .ts files to D-VHS.

D-VHS is my intended archive medium so in order to archive these mpeg2 files, I need a program that does the opposite of HDTVtoMPEG2 ie. One that takes an mpeg2 file and converts it to a .tp file.

Does anyone know where I can find such a program?

Thanks

mikemav
07-25-2004, 09:53 PM
It sounds like from Redstone's experience, it is a long shot to get both the demuxer and remuxer working in Windows? I would rather not have to run linux (though thanks for the tip on the boot CD version; that would be simpler.

If anyone with a compiler wants to take another shot at a Win version, us linux-less souls would appreciate it.

I know Mac is the least likely OS to be invited to this party, but being based in unix, I wonder if an OS-X version would be easier? I installed X11, the developer tools, and managed to get KDE desktop running tonight on my Powerbook. I have yet to try to compile something though. Total newbie there. There is a big thread on AVS Forum/HDTV Recorders about extracting and archiving from HD cable boxes via firewire over OS-X. I wonder if they have compiled tools that I might find useful with HD streams? I will check into it. I can't be the only Steve Jobs disciple here!

bcc
07-25-2004, 11:01 PM
I did discover that MYHD or D-VHStool will only write native .tp or .ts files to D-VHS. So the D-VHS decks (just mitsubishi & JVC, I believe) will only decode transport streams, right?

Seems a bit backwards to go from ps back to ts, but the specs do provide for it. I don't see a util for it over at doom9.net, but there may be one. Might try asking over at forum.doom9.org or the avs forum.

bcc
07-25-2004, 11:06 PM
It sounds like from Redstone's experience, it is a long shot to get both the demuxer and remuxer working in Windows? I would rather not have to run linux (though thanks for the tip on the boot CD version; that would be simpler. Looks to me like the 1.6.2 release is broken w.r.t. windows compilation due to libmjpegutils not being compiled as a shared library. Looks to me like this is fixed in the CVS repository, but I wasn't easily able to get the mainline code to compile (couldn't get autoconf under cygwin to configure it right). I could probably get it to work but it'd take a few hours of fiddling. Someone at the mjpegtools mailing list should be able to more easily handle this.
I know Mac is the least likely OS to be invited to this partyActually the mjpegtools sourceforge web page has MAC/OS compilation instructions here: http://sourceforge.net/docman/display_doc.php?docid=19044&group_id=5776

bcc
07-26-2004, 03:17 AM
I saw the response over at avsforums. This worked for me under linux (plays back fine with MYHD, checktp says its OK too):
vlc -vvv law.ty --sout '#standard{access=file,mux=ts,url="law.ts"}'Talk about acrane arguments :) Now we really need a GUI wrapper.

redstone
07-26-2004, 08:51 AM
So the D-VHS decks (just mitsubishi & JVC, I believe) will only decode transport streams, right?

Seems a bit backwards to go from ps back to ts, but the specs do provide for it. I don't see a util for it over at doom9.net, but there may be one. Might try asking over at forum.doom9.org or the avs forum.

Actually, it is MYHD and DVHStool that is the culprit.

The way you archive a file to a D-VHS with MYHD is to start playing the .tp files and then tell it to record to the D-VHS.
If I try that same thing with the mpeg2 file, MYHD pops up a dialog box that says something to the effect of "Can not write DVD's to tape"


DVHSTOOL simply does not understand the mpeg2 format. It only understands the ATSC transport stream format.
The files that HIpix/MYHD/AccessDTV write all contain the same exact TS formatted data but just with different extensions and accessdtv added xml info.

redstone
07-26-2004, 08:59 AM
It sounds like from Redstone's experience, it is a long shot to get both the demuxer and remuxer working in Windows? I would rather not have to run linux (though thanks for the tip on the boot CD version; that would be simpler.

If anyone with a compiler wants to take another shot at a Win version, us linux-less souls would appreciate it.
!


bcc gave a good answer about porting but you have not tried the Knoppix bootable LInux CD. It is REALLY simple.

As I said to you in a PM:

I found a way to make dealing with LINUX truly trivial for everyone.

You can download an ISO image of a complete Linux distribution and burn it as a bootable CD.
Boot your PC with the CD, Linux is up and running and it sees all your drives.
It does NOT hurt or change your disk partitions.

If you want to try it, go to: http://www.knoppix.org/ and click on the American flag for english.

I downloaded it and used Easy CD Creator to build a bootable CD from that ISO image. PM me if anyone gets stuck with Easy CD Creator.

I booted 3 machines with it:
My laptop where it worked perfectly.
My 1.4 Ghz P4 ASUS P4TE system where it worked perfectly except it does not recognize the ethernet adaptor I am using.

My 3 Ghz ASUS P4G8X system (the one with the MYHD card) but it hung when it was probing the HW and saw my Serial ATA Raid system.

So for now, I will just make sure that the disk on the P4TE has the .ty files on it,boot from the Knoppix CD and make the .mpg files.

Since all the machines in my house are networked together, it's trivial to move these files around.

JQ

redstone
07-26-2004, 11:39 AM
I saw the response over at avsforums. This worked for me under linux (plays back fine with MYHD, checktp says its OK too):
vlc -vvv law.ty --sout '#standard{access=file,mux=ts,url="law.ts"}'Talk about acrane arguments :) Now we really need a GUI wrapper.

I merely used the windows version cause it had a GUI and that Stream to file Wizard makes it really simple.

Good new and bad news.

The good/GREAT news is that it does convert the mpeg's to a .ts/.tp file that is playable perfectly by MYHD.

The bad news is that when I try to have that .ts file recorded to my D-VHS, it hangs MYHD. That was the whole point of converting it to a .ts file:(

I double checked the recorder,etc. by having MYHD record a .tp file that MYHD had recorded to disk before. That worked so I know it is not MYHD's fault.

Really puzzled here because checktp passes.

Do you know what these PIDS are?


BTW, I assume in your post, you meant "vlc -vvv law.mpg ...." and not .ty as you wrote.

redstone
07-26-2004, 04:58 PM
bcc,


I emailed the master of all things video/audio related (Cliff Watson) and here is what he said about the .ts file we made using vlc of Lawandorder.mpg :




I'm saying they are not standard. Here are the rules for PAT/PMT;

ATSC encoders automatically set PID assignments. The automatic assignment of PIDs is the same across MPEG, DVB, and ATSC table modes, and follows the ATSC PID assignment rules, which are compatible with DVB and MPEG.

Automatic PID assignment is performed according to the following rules.

* PID calculations are performed in hex (h).
* A program's base PID is assigned by multiplying the program number by 10h.
* The program mapping table (PMT) is created from the program's base PID, according to the following rules:

* PMT equals the base PID.
* Video and program clock reference (PCR) are base +1.
* Audio A is base +4
* Audio B is base +5
* Audio C is base +6
* Audio D is base +7
* Auxiliary Data A is base + A hex (10 decimal).
* Auxiliary Data B is base + B hex (11 decimal).

For example, for a service with program number of 2 the following PIDs are automatically assigned:

* Program's base PID is 20 hex (2h x 10h = 20 hex).
* PMT, which is equivalent to the base PID, is PID 20 hex or 32 decimal.
* Video and PCR are 21 hex or 33 decimal (20h + 1 = 21hex).
* Audio A is 24 hex or 38 decimal (20h +4 = 24 hex).
* Auxiliary Data A is 2A hex or 42 decimal (20h +10d =2A hex).

Using your Transport Stream Continuity Checker image I see the following PIDs:

0x0000 ?
0x0042 (Unknown PID for program 4 base)
0x0044 (Audio PID for program 4)
0x0045 ( SAP Audio PID for program 4)

There is no PID (0x0041) listed for Video and may be the reason you can't record to tape using MyHD.

MyHD is rebust in that if an error is found in the PMT/PAT it will revert to reading headers to find and decode A/V PIDs.


I do not know where to go from here. Any ideas?

malfunct
07-26-2004, 05:07 PM
Is the tool setting the PID or is it reading it from the tystream? If its setting it on its own then you need to look into fixing the tool to set a correct PID for the tystream. If the PID is pulled from the tystream you need to figure out if you are getting it from the wrong place, or if tivo is just using a different set of ID's and you need to reverse engineer the translation. The blurb you posted mentioned the ability to calculate the correct PID from the audio and video included in the stream, maybe thats what the tool needs to do. Anyways I don't know a whole ton about what PID's are about or what they are for but these are just general suggestions of things to look for.

bcc
07-26-2004, 09:34 PM
0x0000 ?
0x0042 (Unknown PID for program 4 base)
0x0044 (Audio PID for program 4)
0x0045 ( SAP Audio PID for program 4)
PID 0x0000 is "Program Association Table" per table 2-3 of 13818-1.
Looks like bbdmux has an easier time decoding the PIDs:
% bbdmux law.ts | grep PID
Scanning for PID's, press control-c to quit ...
Found PID 0x0000, Program Association Table Stream
Found PID 0x0042, Other Stream
Found PID 0x0044, stream id 0xBD = Private Stream 1
Found PID 0x0045, stream id 0xE0 = Video Stream 0
PID 0x0000, Program Association Table packets = 354, total bytes = 6018
PID 0x0042, Other packets = 354, total bytes = 9558
PID 0x0044, Private Stream 1 packets = 20034, total bytes = 3419136
PID 0x0045, Video stream 0 packets = 829139, total bytes = 152150137
%(stream BD is the AC3 audio stream)
There is no PID (0x0041) listed for Video and may be the reason you can't record to tape using MyHD.vlc is synthesizing the PIDs so if these PIDs are non-standard vlc is the issue. bbdmux is able to identify the video PID based upon the stream ID. I'm curious where the strict PID assignment rules come from? I couldn't find them in the atsc or 13818 standards.

bcc
07-26-2004, 10:14 PM
bcc gave a good answer about porting but you have not tried the Knoppix bootable LInux CD. It is REALLY simple.Or maybe just ask nicely over at mjpegtools.sourceforge.net for a release that builds cleanly with cygwin?

mikemav
07-26-2004, 10:58 PM
I think I found my Mac OS-X dream GUI tool. check this (http://homepage.mac.com/major4/index.html) out. ffmpegX. Options for everything under the sun; making a Mac OS-X GUI for about 20 unix video tools (menoder, mplayer, mpeg2enc, etc...) Looks like it will remux video and audio mpeg elements, change resolution if I ever want to make a DVD, etc.. I will play around with bcc's Win port of the demuxer for the ty and then bring the video elements over to this to re-encode. If this works I will not need to install linux. BTW, when it first runs, it looks for binaries for mpeg2end, mencoder, mplayer, and real-libs (real player?.) If there are modified binaries that may work better, I wonder if this program will work with them? Anyway, I will play around and report back.

Also wanted to report I had no crashes with TyTools 9r15 using the multiplex option (unlike 9r14.) I have not tried the resulting mpeg HD file listening w/ sound, but I know it does play on MyHD. I like the idea of this new tool, but if TyTools latest version works to output playable mpeg (not just VOB), that gives us all the more options.

redstone
07-27-2004, 06:07 AM
Also wanted to report I had no crashes with TyTools 9r15 using the multiplex option (unlike 9r14.) I have not tried the resulting mpeg HD file listening w/ sound, but I know it does play on MyHD. I like the idea of this new tool, but if TyTools latest version works to output playable mpeg (not just VOB), that gives us all the more options.

I ran tytool9r15 on law and order.ty as well as IN and Out.ty+

Tryed both of the resultant .mpg files with MYHD.

Law and order only makes a 6 MB .mpg file
In and out makes a playable, correct length .mpg file but it has the classic lip sync problem.

I tried all the variations of multiplex in tytools but no difference.
Which files did you try and can you tell me which options you cose in tytools?

Thanks

redstone
07-27-2004, 07:45 AM
vlc is synthesizing the PIDs so if these PIDs are non-standard vlc is the issue. bbdmux is able to identify the video PID based upon the stream ID. I'm curious where the strict PID assignment rules come from? I couldn't find them in the atsc or 13818 standards.

Looks like vlc is the culprit.

I ran Checktp on a MYHD recorded .tp file.
I then converted that .tp file with hdtvtompeg2 to a .mpg file and it plays fine in MYHD.

Then I ran vlc on the hdtvtompeg2 to regenerate a .tp file
Ran checktp on that file and I get the same exact results as noted in the post above.

See the enclosed pic.
Upper left are the PID's in the recorded MYHD .tp file
Upper Right is .tp to .mpg with hdtvtompeg2. Notice the 21 and 24
Bottom right is that .mpg converted back to a .tp with vlc. Notice that it has changed the PID's.

Time to see if we have access to the vlc source code or find another .mpg to .ts convertor.

bcc
07-27-2004, 01:34 PM
vlc has some PID values hard-coded here:
% grep pid vlc-0.7.2/modules/mux/mpeg/ts.c | grep 0x4
p_sys->pmt.i_pid = 0x42;
p_sys->i_pid_free = 0x43;
p_ts->p_buffer[1] = ( b_new_pes ? 0x40 : 0x00 )|( ( p_stream->i_pid >> 8 )&0x1f );
%Dunno if it's configurable or what.

redstone
07-27-2004, 04:50 PM
I am really puzzled. It seems as though these programs that convert mpeg2 to transport stream files re-assign the Pid's.
I have tried VLC and MpegVCR2 as they both will convert a mpeg2 file to a transport stream.

The interesting thing is that these two programs come out with different PID's from each other.
Both .ts files hang up MYHD when trying to write to the D-VHS.


I found a 30 day trial ware set of programs that really let you see the format of mpeg and transport stream files. Thery are at:
http://www.manzanitasystems.com/products/products.htm

I converted the Dion Live mpeg2 to a transport stream with VLC and MpegVCR2. I then dumped them with the tranasport stream dump tool from manzanita systems.
This mpeg2 file was built from a .ty file using the hdemux tool set and play nicely in MYHD.

I also dumped a 'standard' MYHD recorded .tp file.

Please see the following snapshots to see if you can make sense of why the MYHD .tp file will record to D-VHS but the other two converted .ts files hang it.

Getting a bit lost here.

bcc
07-27-2004, 09:48 PM
Here's a new image that fixes the audio for OTA HD. In OTA HD, the audio PES packets can contain multiple ac3 frames per payload. In satellite I've only seen one frame (or less). I was simply mishandling the OTA case.

saltydog4791
07-28-2004, 03:08 PM
Hello and thanks for a great tool so far. I am trying to get this to work on OS X and am having a problems. I think I was able to compile it fine for OS X but I can't figure out how to patch mplex. The version I of mplex I got is 1.7.0 which I grabbed from missingmpegtools. I pulled the binary out of there and dropped it into /usr/local/bin but I don't know how to apply the patch properly. Thanks for any help anyone can provide, and is there anyone else who is trying to get this going on mac also?

saltydog4791

saltydog4791
07-28-2004, 03:13 PM
Forgot to write this before but this is the command I tried and the erroneous output.

dhcp32:/usr/local/bin saltydog$ sudo patch -p0 < mplex.patch mplex
patching file mplex
Hunk #1 FAILED at 73.
1 out of 1 hunk FAILED -- saving rejects to file mplex.rej



saltydog4791

mikemav
07-28-2004, 03:20 PM
Hello and thanks for a great tool so far. I am trying to get this to work on OS X and am having a problems. I think I was able to compile it fine for OS X but I can't figure out how to patch mplex. The version I of mplex I got is 1.7.0 which I grabbed from missingmpegtools. I pulled the binary out of there and dropped it into /usr/local/bin but I don't know how to apply the patch properly. Thanks for any help anyone can provide, and is there anyone else who is trying to get this going on mac also?

saltydog4791

saltydog-
Another Mac user! If you get it working, please let me know. This has been my goal but I have not had a chance to mess with it much yet. BTW, have you used ffmpegx? Great GUI for a lot of these tools, but does not look like it works with HD content using the usual binaries. Maybe if you get mplex patched that could be incorporated. The GUI sure is nice. Anyway, good luck. If you need any HD-tivo content to test it out on, I would be glad to send you some (or see the other post about HD extraction, where redstone has been kind enough to post my HD files for ftp.)

redstone
07-28-2004, 05:43 PM
I think I was able to compile it fine for OS X but I can't figure out how to patch mplex. The version I of mplex I got is 1.7.0 which I grabbed from missingmpegtools.
saltydog4791


I got a cryptic error message when I tried that patch in Linux so I just edited the file.

I don't have it handy but the patch is a one line mod to one module. It does a multiply by 8 on a single line.
Just look at the patch and edit the file.

redstone
07-28-2004, 05:56 PM
Here's a new image that fixes the audio for OTA HD. In OTA HD, the audio PES packets can contain multiple ac3 frames per payload. In satellite I've only seen one frame (or less). I was simply mishandling the OTA case.


Thanks - you dah man:)

Other related items to this project:

1) I posted the above .ts maps and the text on avsforum so that I could get more mpeg2 experts's eyeballs looking into the MPEG2 --> ts --> MYHD ---D-VHS issue.

Until that nut is cracked,I can archive the mpeg2 files onto my DDS4 dat drive when I run out of disk space on my RAID. The DDS-4 tapes are 20 GB uncompressed - enough for a 2.5 hour movie and not expensive


2) I have 2 more tests to run on a BIG 900 MB file that mikemav is going to upload to my ftp site and then hopefully my HDTIVO is off to get the prom mod done.

Thanks!!

redstone
07-28-2004, 09:24 PM
Here's a new image that fixes the audio for OTA HD. In OTA HD, the audio PES packets can contain multiple ac3 frames per payload. In satellite I've only seen one frame (or less). I was simply mishandling the OTA case.


Could you please try the In and Out clip please.

mplex kicks out with an error and generates a 0 byte file when I use the output of this new version.

If I run the previous version of hdemux, mplex generates a mpeg file just fine.

Here is the output of mplex showing the error:

Deanzsyclone
07-29-2004, 01:05 AM
Tech savi new HD TIVO (my first tivo )owner that wants to archive, I'm reading and reading and learning, do I understand correctly that hdemux runs only on a lenux os? and archiving my tivo (.ts?) HD files with all the programs neccesary need to and only can be done on Lenux OS?

Thanks for the answer.
Dean
Now I have to figure how to get the .ts files to my computer! :)

redstone
07-29-2004, 08:49 AM
It seems as though these programs that convert mpeg2 to transport stream files re-assign the Pid's.
I have tried VLC and MpegVCR2 as they both will convert a mpeg2 file to a transport stream.


By posting this on avsforum, I knew I could find an mpeg2 format expert and have done so.

I am told that the generated table's by these 2 mpeg2 to .ts conversion tools have NON-standard ATSC tables

Here is what I was told:
"The recommended PID layout for ATSC is:
PAT = 0
PMT = 0x20
Program Number = 2
PCR = 0x21
Video = 0x21
Audio = 0x24

But these PID values are just a recommendation. Many OTA stations use the
older recommendation which was:
PAT = 0
PMT = 0x10
Program Number = 1
PCR = 0x11
Video = 0x11
Audio = 0x14

Therefore, no decent software program or decoder should be hard coded to any set
of PID's. Too many streams will not work."




I responded to him after looking more carefully at the headers and you will see that both vlc and megtovcr2 generate totally wacky headers but VLC is much,much worse:

"The dump of the the MYHD generated transport stream looks exactly like recommened ATSC layout that you described.

This file of course, goes to D-VHS with MYHD just fine.

On the other hand both of the other conversion tools are NOT generating the recommened ATSC layout.
Therefore, it crashes MYHD when trying to write these to the D-VHS.

The cconverted mpeg2 file with vlc violates the ATSC spec for 2 reasons.
The first is that it is showing 2 program streams when there is only one and the first does not even have an audio PID.
The second is that neither one looks like the standard.

This is the table for the vlc converted one:


PAT = 0
PMT = 0x42
Program Number = 1
PCR = 0x44
Video = 0x44
Audio MISSING

and

PAT = 0
PMT = 0x42
Program Number = 1
PCR = 0x45
Video = 0x44
Audio = 0x45


Here is the table for the mpeg2vcr converted file:

PAT = 0
PMT = 0x10
Program Number = 10
PCR = 0x22
Video = 0x22
Audio = 0x23

Now that I have wrriten them down this way, I see that BOTH of the converted streams have totally wacked out tables.
Mpeg2VCR's is closer to correctness though."


I will continue to work with these folks on avsforum to crack this nut.

redstone
07-31-2004, 06:57 PM
One can not help but compare jdiner's HDTYtool with bcc's HDemux tool BUT this post is not intended to pit one against the other.

Just looking for answers.

I am having nearly a 100% successs rate in converting all the test files that I have with hdemux whereas I have about a 25% success rate with HDtytool.

However, hdemux/mplex are REALLY slow compared to HDtytool.
I am running the Linux version of hdemux on an almost identical PC to the one that I am running HDtytool with.

Any idea why hdemux/mplex are so slow?

By slow, I am talking about 5 - 10 minutes compared to less than 1 minute.

May not seem like a big deal but these are just short clips of around 150MB each.
When I get my box modded in a week or so , I will be converting full length HD movies that run ~14 Gbytes for a 1.5 hour movie.

Thanks

bcc
08-09-2004, 04:24 PM
I apologize for the lack of support. I haven't intentionally abandoned things here - I'm just on the road & away from the tivo for a couple more weeks.
PS: The diving in little cayman is real nice - I highly recommend it if you like to see healthy coral reefs.

redstone
08-12-2004, 08:08 PM
I know you are aay for most of August but I thought I would ask this question anyway.

Instead of generating mpeg2 files from .ty files, what would it take to generate Transport Streams (.ts files) from .ty files?

Also, your continued development of hdemux would be greatly appreciated since it is Open Source

redstone
08-13-2004, 05:34 PM
I ran hdemux which sucessfully ran, then I ran mplex with the suggested values but it almost immediately bombs out.

This is on a 1 GB clip that I extracted and which does convert to a playable mpeg2 file with the other tool.

Enclosed is a screen snapshot of running mplex in Linux

bcc
09-07-2004, 08:40 PM
I know you are aay for most of August but I thought I would ask this question anyway.

Instead of generating mpeg2 files from .ty files, what would it take to generate Transport Streams (.ts files) from .ty files?

Also, your continued development of hdemux would be greatly appreciated since it is Open SourceHi, I'm back (travel delays kept me away into Sept.). So now I'm finally getting caught up on things. In theory there's not much difference between the mpeg program streams (.mpg) and transport streams (.ts). But generating these combined streams would be a job for the re-multiplexor, ie the mplex program, not the demuxer. Mplex has support for outputting to several different mpeg formats, but not to188 byte transport stream packets. In practice, it's probably not so easy to hack mplex to output a transport stream (or maybe I'm just not so good at reading others' object oriented code). I would think it'd be easier to code something fresh for this particular purpose. Which makes me wonder how worthwhile that would be since I believe the only motivator for converting back to a transport stream is to support D-VHS. Are you still having troubles with D-VHS? Seems like DVD+R DL will be a more reliable and cheaper storage media once those media prices go mainstream.

bcc
09-07-2004, 08:57 PM
Forgot to write this before but this is the command I tried and the erroneous output.

dhcp32:/usr/local/bin saltydog$ sudo patch -p0 < mplex.patch mplex
patching file mplex
Hunk #1 FAILED at 73.
1 out of 1 hunk FAILED -- saving rejects to file mplex.rej



saltydog4791Oops, the recommended arguments to patch weren't quite right. Works fine with:
% pwd
/u3/mjpegtools-1.6.2
% patch --verbose -p0 < ../hdemux/mplex.patch
Hmm... Looks like a new-style context diff to me...
The text leading up to this was:
--------------------------
|*** mplex/stream_params.cpp.orig Sat Dec 20 09:33:38 2003
|--- mplex/stream_params.cpp Thu Jul 22 01:10:16 2004
--------------------------
Patching file mplex/stream_params.cpp using Plan A...
Hunk #1 succeeded at 73.
done
%

bcc
09-07-2004, 10:08 PM
Could you please try the In and Out clip please.

mplex kicks out with an error and generates a 0 byte file when I use the output of this new version.

If I run the previous version of hdemux, mplex generates a mpeg file just fine.

Here is the output of mplex showing the error:Ah my last change to detect multiple AC3 frames per payload was buggy, and now misdetecting the case where the payload has extra padding bytes. Fixed...

bcc
09-07-2004, 10:13 PM
I ran hdemux which sucessfully ran, then I ran mplex with the suggested values but it almost immediately bombs out.

This is on a 1 GB clip that I extracted and which does convert to a playable mpeg2 file with the other tool.

Enclosed is a screen snapshot of running mplex in LinuxI suspect this is the same problem as I just found (result is hdemux's .m2a file contains extra 2 bytes of padding per audio packet and mplex bombs out). I can't tell for sure without the sample clip. Maybe you could just retry with a fixed image if you're interested.

bcc
09-07-2004, 10:20 PM
Any idea why hdemux/mplex are so slow?

By slow, I am talking about 5 - 10 minutes compared to less than 1 minute.
I suspect with tytool you didn't configure it to write the split streams to disk before remuxing, but with hdemux/mplex you are doing that extra disk I/O. You should be able to use mkfifo to create two fifos for the .m2a and .m2v files, and then have hdemux/mplex write/read to the fifos. At that point your hard disk performance should be less of a bottleneck, and the performance should be much more similar.

bcc
11-05-2004, 09:49 PM
Here are the latest windows binaries for mplex.
mplex.zip contains a windows binary of the mplex program patched for HD multiplexing. cygwin.zip contains a support .dll for the mplex windows binary.

bcc
11-26-2004, 09:34 PM
Some folks were having problems with hdemux doing file IO in text mode instead of binary mode, resulting in bad .m2a/.m2v files or errors reading the ty file. I've posted a new release - 0.4 which fixes this cygwin pitfall.

bcc
11-28-2004, 12:35 AM
I put up a new version of hdemux - 0.5, which fixes some atypical OTA audio. Specifically I now handle the case where the ac3 audio sync. frame is not aligned with the beginning of transport stream payload. I also fixed mpeg2 audio for sat. recordings for data rates other than the typical 192kb.

bcc
11-28-2004, 08:08 PM
Okay, so 0.5 didn't do a great job of keeping sync for AC3 audio streams that are unaligned. Now I keep track of where we expect the next sync bytes to be, so we don't get faked out by "phony" sync bytes in the payload. New release: 0.6.

bcc
11-29-2004, 02:35 AM
Alright; another new version: 0.7. This fixes a problem where the audio data may contain the magic "start_code_prefix" pattern, which was faking out my packet start logic. I had thought this was "illegal" per the spec, but it turns out it's only illegal for the video stream not the audio. Amazingly someone found a recording that hits this.

slimjime17
11-29-2004, 05:56 AM
I keep getting error message when I run 0.7

Version 0.7
assertion "astream.buf[payoffset] == 0xff" failed: file "hdemux.c", line 766
3 [sig] hdemux 2788 open_stackdumpfile: Dumping stack trace to hdemux.exe.
stackdump

Any ideas what I'm doing wrong?

redstone
11-29-2004, 05:56 PM
G:\hdemux\hdemux>hdemux -i t:\master_and_commander.ty -v t.m2v -a t.m2a
Version 0.6
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
AC3 audio at 48kHz, 1536 bytes/sync frame
Audio timestamp offset: 530.777778 ms
Reported datarate 65384 Kbit/sec. (65000+384)
Proceed with remuxer:
mplex -f 8 -O 530ms -r 153701 t.m2a t.m2v -o <outfile>.mpg
assertion "found_sync" failed: file "hdemux.c", line 614
2 [sig] hdemux 2788 open_stackdumpfile: Dumping stack trace to hdemux.exe.
stackdump

G:\hdemux\hdemux>




Is there a place where the stack dump is going so I can get it to you?
This was NOT an OTA program

bcc
11-29-2004, 11:29 PM
I keep getting error message when I run 0.7

Version 0.7
assertion "astream.buf[payoffset] == 0xff" failed: file "hdemux.c", line 766
3 [sig] hdemux 2788 open_stackdumpfile: Dumping stack trace to hdemux.exe.
stackdump

Any ideas what I'm doing wrong?My first guess would be that you fed hdemux something that you extracted from a low-definition S2 tivo, not an HD-Tivo. The low-def models generate audio records of type 3/c0, that are "malformed" if you look into the PES header. I did not bother to special case older tivo file formats for this program - it only supports HD-Tivo.

bcc
11-29-2004, 11:56 PM
Is there a place where the stack dump is going so I can get it to you?
This was NOT an OTA programYou berate me over at the yahoo hdtivo forum, after I walk you guys through upgrading the OS without pulling the disk. And then you expect continued personal support? Sorry I have to make an exception here and not help.

AlphaWolf
11-30-2004, 12:09 AM
You berate me over at the yahoo hdtivo forum, after I walk you guys through upgrading the OS without pulling the disk. And then you expect continued personal support? Sorry I have to make an exception here and not help.

Isn't it amazing how supportive (http://groups.yahoo.com/group/hdtivo/message/1) they are over there? :D

On a more serious note, I wonder if these are all of the people that just got banned from DDB for being stupid and sought refuge by setting up a reject sanctuary (http://groups.yahoo.com/group/hdtivo/message/3).

bcc
11-30-2004, 12:30 AM
Isn't it amazing how supportive (http://groups.yahoo.com/group/hdtivo/message/1) they are over there? :DI found it ironic that he complains about DDB (badmouths it in-fact), and then is the one doing the attacking over there. Over some mis-perceived notion of attitude no less. Steered a perfectly good technical discussion into the sewer.

bcc
11-30-2004, 04:04 AM
Just put up version 0.8. Added a bunch of error detection and correction logic. Of special note, the code now tries to find time discontinuities in the ty stream. This helps prevent us from erroring out from garbage chunks at the end of FSIDs.

Rowan
11-30-2004, 07:54 PM
bcc,

I have made a few small changes to allow your code to be natively compiled for windows using the MS VC++ 6.0 compiler. It should also still compile under linux with out any problems. I have included the source code, native windows binarys for hdemux as well as mplex so that means that the cygwin1.dll is no longer needed and should run a little faster.

I think I read that you do not have access to the windows compiler, if you merge in the few changes I will make the windows side of things if you want. Just let me know.

Rowan

malfunct
11-30-2004, 08:24 PM
bcc,

I have made a few small changes to allow your code to be natively compiled for windows using the MS VC++ 6.0 compiler. It should also still compile under linux with out any problems. I have included the source code, native windows binarys for hdemux as well as mplex so that means that the cygwin1.dll is no longer needed and should run a little faster.

I think I read that you do not have access to the windows compiler, if you merge in the few changes I will make the windows side of things if you want. Just let me know.

Rowan

BCC, if you want VC6 and are willing to pay the $4 postage I have a copy (legally licenced VS6 enterprise I think) I no longer use. Catch me in PM.

bcc
11-30-2004, 08:52 PM
I have made a few small changes to allow your code to be natively compiled for windows using the MS VC++ 6.0 compiler. It should also still compile under linux with out any problems. I have included the source code, native windows binarys for hdemux as well as mplex so that means that the cygwin1.dll is no longer needed and should run a little faster.

I think I read that you do not have access to the windows compiler, if you merge in the few changes I will make the windows side of things if you want. Just let me know.
RowanGreat! Ok if I try to rework those diffs to minimize the special cases?
I assume you applied the HD mplex patch before compiling?
There's got to be a better way to get htonl() in visual c without resorting to assembly...

Actually I have visual C7, it's just that my best machine for it usually runs linux so I'd have to reboot to compile. Also I haven't taken the time to really learn the environment, being a linux person myself...

bcc
11-30-2004, 08:55 PM
BCC, if you want VC6 and are willing to pay the $4 postage I have a copy (legally licenced VS6 enterprise I think) I no longer use. Catch me in PM.Thanks for the offer. I'm already set up with vc7 - mostly. Probably should just resize my laptop's C: so I can more conveniently run it over there.

Rowan
11-30-2004, 09:09 PM
Ok if I try to rework those diffs to minimize the special cases?
Sure no problem, I tryed to make as few changes as posible but when you have both enviorments setup you can optimize it a little more. I do not have a linux setup (at home) so have no way to test my changes.


I assume you applied the HD mplex patch before compiling?
Yes and I tested it with a test stream I have.


There's got to be a better way to get htonl() in visual c without resorting to assembly...
Yes there is, at the top of hdemux.c in the WIN32 include section add '#inlcude <windows.h>' and then the ws2_32.lib needs to be included. Let me know if you want me to make the change. I always use the asm code because it is faster than the winsock library version and works on almost all I386 processors.


Actually I have visual C7
I also have VC7 but most people still use VC6 so I did it in that version, you can import the VC6 project file (.dsp file) and it should then work for you in VC7.

bcc
04-20-2005, 10:57 PM
I've posted a new version of hdemux - 0.9, with some improvments to over the air (OTA) audio processing. For some OTA recordings, there can be < 1 audio frame per PES packet. There was code to handle this before, but it was busted. Should be fixed now.
This release also includes portability changes for 64 bit processors, and some source cleanup. Hopefully I didn't break anything in the process.

patriot
04-21-2005, 07:24 PM
:D Nice!

I cant wait to see how it works out. Thanks for keeping the tools updated! :D

mr_zorg
06-15-2005, 02:52 AM
I've been playing with hdemux on my Mac OS X for a while, and it works quite nicely. Though to be honest, I've been using it mostly on SD streams, not HD...

Recently I was attempting to burn a stream to DVD that seemed to have mixed content. The video would flit back and forth between 24fps and 30fps (looking at the output from mplayer). I also think it has a mix of m2a and ac3 audio. Splitting with hdemux and remuxing with mplex (as hdeumux instructs) gives a nice solid mpg file with good a/v sync. But when I try to scale the video to 720x480 and then remux it, as I do most of the time, a/v sync gets shot to hell. Even if I don't try to scale it, and just burn it directly, the DVD software will demux and remux to VOB automatically, and that blows the a/v sync too.

I'm hoping someone has some idea on how I might deal with something like this in the future.

bcc
06-16-2005, 05:25 PM
I can't tell what's going wrong in your case, but I'd first look to see whether your scaling or DVD authoring steps involve regenerating of the recording's timestamps. If so, that may be what is causing the loss of sync between streams.

mr_zorg
06-18-2005, 04:11 AM
I can't tell what's going wrong in your case, but I'd first look to see whether your scaling or DVD authoring steps involve regenerating of the recording's timestamps. If so, that may be what is causing the loss of sync between streams.
The DVD authoring, I would hope not. I really don't know, though. It should, I would hope, just demux the mpeg and remux as a VOB. No reason for it to kill the timestamps.

My processes are a bit non-standard as there really aren't any great ty utilities for the Mac. The only real ones I can compile and get to work are yours and an old version of vsplit. For "normal" satellite streams (SD, mpeg audio), my process works a bit like this:

1) Demux with vsplit or hdemux.
2) Convert the m2a into aiff using madplay
3) Import the m2v and aiff into Final Cut Pro. (Damn thing just refuses to use the audio in mux'd mpeg streams, and won't use m2a audo files either.)
4) Edit, scale, etc., and export using it's Compressor engine back to a new m2v and aiff file.
5) Convert the aiff file to ac3 using A.Pack.
6) Import the m2v and ac3 files into DVD Studio Pro and burn away...

And it works beautifully. Except in the case I originally described. I'm totally open to suggestions for better ("works better") tools to use on a Mac... Just in case you're not familiar, Final Cut Pro, Compressor, A.Pack, and DVD Studio Pro are all part of Apple's pro-level tools. These programs are great, but apparently geared more toward generating mpeg (and other format) files, not working with them as they all pretty much refuse to work with full mpg or vob files... Sad.

mr_zorg
08-30-2005, 01:12 AM
Using hdemux (under Mac OS X) and its suggested mplex command, I've created a series of MPG files of SD material pulled from my TiVo. They play beautifully on the Mac (using either QuickTime or mplayer), but only get about 5 seconds into it on Windows before locking up, or freezing the video while the audio keeps going. I've tried Windows Media Player, QuickTime, Real Player and Winamp, all with similarly bad results. An MPG is an MPG is an MPG, right? Any idea what's going on here?

bcc
08-30-2005, 05:54 PM
An MPG is an MPG is an MPG, right? Any idea what's going on here?You'd think, but under windoze, directshow filters can have all sorts of issues. For example, the codec actually used for playback depends upon what software you've installed, and often in what order. Recommend you try a dvd player such as powerdvd or windvd. You could also try troubleshooting the .m2v stream separately, and working with graphedit to check your filters.

mr_zorg
08-31-2005, 12:57 AM
You'd think, but under windoze, directshow filters can have all sorts of issues. For example, the codec actually used for playback depends upon what software you've installed, and often in what order. Recommend you try a dvd player such as powerdvd or windvd. You could also try troubleshooting the .m2v stream separately, and working with graphedit to check your filters.

Huh. My wife's laptop had WinDVD and that worked just fine. Imagine that. Thanks Microsoft for mucking things up so badly that a simple MPG won't play in any of the leading media players... And thanks to you, bcc, for the solution.

bcc
08-31-2005, 01:10 PM
Thanks. Time to revise that list of "leading" media players :)

mr_zorg
08-31-2005, 10:42 PM
Huh. My wife's laptop had WinDVD and that worked just fine. Imagine that. Thanks Microsoft for mucking things up so badly that a simple MPG won't play in any of the leading media players... And thanks to you, bcc, for the solution.
Actually, I stand corrected. It still doesn't work very well, though they DO play, the audio is still off by a fair bit. Odd thing is, when I tried it on my Mac again the audio is off there too. I swear they were playing fine before on my Mac, perhaps I was hallucinating...

konfoo
09-13-2005, 03:39 AM
Actually directshow filters are chosen by their merit, and if the application that is doing the directshow graph building is worth its salt, it will manually build the filter graph and force the use of a codec chain that it knows to work.

Here's a handy tool:
http://www.divx-digest.com/software/radlight_filter_manager.html
NOTE microsoft's core codec dlls will sometimes take priority no matter what you set their merit values to. So be forewarned...

bcc
09-13-2005, 03:58 AM
Actually directshow filters are chosen by their merit, and if the application that is doing the directshow graph building is worth its salt, it will manually build the filter graph and force the use of a codec chain that it knows to work.Right, and we all know how well applications agree upon their relative merits. gspot is the best app. I've seen for troubleshooting merits&filters.

bcc
09-13-2005, 06:01 PM
Here's a new version of hdemux that computes the recommended a/v offset differently. Roughly, this version computes the offset based upon the pts values written to the start of the audio/video streams instead of trying to compute the offset based upon packet arrival times in the tivo records. Let me know if this works better. Source&linux binary below:

[See later version below]

Cheezmo
09-13-2005, 07:10 PM
Something can't be right, I get vastly different offsets just by changing how many initial chunks to skip...

hdemux -d -s 0 -n 2000 -i tivotool.ty -a ks.ac3 -v ks.m2v
Proceed with remuxer:
mplex -f 8 -O 1696ms -r 153701 ks.ac3 ks.m2v -o <outfile>.mpg

hdemux -d -s 1 -n 2000 -i tivotool.ty -a ks.ac3 -v ks.m2v
Proceed with remuxer:
mplex -f 8 -O 1696ms -r 153701 ks.ac3 ks.m2v -o <outfile>.mpg

hdemux -d -s 10 -n 2000 -i tivotool.ty -a ks.ac3 -v ks.m2v
Proceed with remuxer:
mplex -f 8 -O 3835ms -r 153701 ks.ac3 ks.m2v -o <outfile>.mpg

hdemux -d -s 20 -n 2000 -i tivotool.ty -a ks.ac3 -v ks.m2v
Proceed with remuxer:
mplex -f 8 -O 2715ms -r 153701 ks.ac3 ks.m2v -o <outfile>.mpg

bcc
09-13-2005, 07:47 PM
That's expected behavior because the code is now just looking at the beginning presentation timestamps from the streams, not at the delivery times to the buffer. Since chunks differ on where the first video sequence is delivered, the results vary from chunk to chunk.

bcc
09-21-2005, 09:47 PM
New version of hdemux. I've done a lot more testing of the a/v sync offset. The last version was producing a bad value because I was not correctly finding the first video timestamp (PTS). I've changed the video frame processing so that we now buffer them as I was for audio, and this allows me to better parse the PES packets (and thus find the first PTS. I've tested the results, and the a/v sync now seems solid for both low-def and hi-def recordings.

johnsolo
10-02-2005, 02:35 AM
Excellent program, I like it a lot.

Seems to have some issues with named pipes though.

tydemux is the named pipe.


mfs_uberexport -R 150888 -o ~/.tivotool/tydemux
stormy:~/.tivotool john$ hdemux -i tydemux -v oi.m2v -a oi.m2a
This is what happens when I execute the above command(s):

Version 0.9
Source is tydemux
0 audio stream bytes 0 video stream bytes

vsplit does work using the same method, but has other issues. (/outputting/ to a named pipe doesn't work).

I'll be looking over the source later tonight to see if I can figure out what's going on. Thanks again for this nice ty demuxer =)

bcc
10-02-2005, 03:32 AM
Excellent program, I like it a lot.

Seems to have some issues with named pipes though.

tydemux is the named pipe.


mfs_uberexport -R 150888 -o ~/.tivotool/tydemux
stormy:~/.tivotool john$ hdemux -i tydemux -v oi.m2v -a oi.m2a
This is what happens when I execute the above command(s):

Version 0.9
Source is tydemux
0 audio stream bytes 0 video stream bytes

vsplit does work using the same method, but has other issues. (/outputting/ to a named pipe doesn't work).

I'll be looking over the source later tonight to see if I can figure out what's going on. Thanks again for this nice ty demuxer =)
Thanks, glad you like :) I've also partially coded some unreleased support for older (low-def tivos) which you're probably interested in if you're trying to hook this into tivotool. I see you're not using the latest version - the latest version would have given you the error message:
error: lseek failed
System error string: Illegal seekNow this lseek() is really only handy if you specified a skip offset (and could even be avoided in that case if one wanted). Would be trivial to avoid the seek if no offset was specified. As for vsplit, I think it is way too rudimentary in its parsing of tivo packets to hook into any automated script.

johnsolo
10-02-2005, 10:14 PM
Thanks, glad you like :) I've also partially coded some unreleased support for older (low-def tivos) which you're probably interested in if you're trying to hook this into tivotool.
This point alwayed confused me, as hdemux seems to work with fine with SD streams, as least S2SA and S2DTivo streams. I do not have any S1 .ty files to test. But yeah, I am interested in what you are talking about, even if I'm not sure what else needs to be done for SD streams =)


I see you're not using the latest version - the latest version would have given you the error message:
error: lseek failed
System error string: Illegal seekNow this lseek() is really only handy if you specified a skip offset (and could even be avoided in that case if one wanted). Would be trivial to avoid the seek if no offset was specified. As for vsplit, I think it is way too rudimentary in its parsing of tivo packets to hook into any automated script.
I see the seek, and commenting it out goes back to the behavior of 0.9. I understand what you are saying about avoiding the seek, but the harder problem is figuring out this pipe issue. I think I need to somehow buffer the pipe so hdemux can get at least a chunk to start with. I need to read up.

bcc
10-04-2005, 12:42 AM
This point alwayed confused me, as hdemux seems to work with fine with SD streams, as least S2SA and S2DTivo streams.1) I only use an HD tivo 2) Lack of tester prospects for low-def 3) lack of sample images 4) under-documented differences between .ty file versions 5) not wanting to mess up the code trying to support everything like tystudio (to the determent of its maintainability IMO).
More specifically, for SA tivo's I only have 1 test stream and it would not play until I added a special case for an SA specific packet type, and also fixed the MPEG audio frame size equation. And that code has yet to be released, so if things work with S2SA, then I wouldn't know due to lack of any samples or feedback. S1 Dtivo won't play without special mpeg1 PES header processing logic (I just added that as well). All of this is special purpose stuff that I didn't care about until the HD stuff worked well (correctness first). Luckily low-def are simple special cases once you can handle HD frame processing. I think I've actually busted low-def video frame processing with the newer video buffering code as well, but that shouldn't be hard to clean up...
I see the seek, and commenting it out goes back to the behavior of 0.9. I understand what you are saying about avoiding the seek, but the harder problem is figuring out this pipe issue. I think I need to somehow buffer the pipe so hdemux can get at least a chunk to start with. I need to read up.Oh right it still doesn't work does it? I was quick&dirty with the file reading; it should really look something like the following. No need to do external buffering.
static ssize_t
read_internal(int fd, char *buf, size_t size)
{
size_t remain;
ssize_t stat;

for (remain = size; remain; remain -= stat) {
stat = read(fd, buf, remain);
if (stat <= 0) {
/* EOF */
return size - remain;
}
buf += stat;
}
/* Successfully read everything */
return size;
}

johnsolo
10-04-2005, 07:57 PM
1) I only use an HD tivo 2) Lack of tester prospects for low-def 3) lack of sample images 4) under-documented differences between .ty file versions 5) not wanting to mess up the code trying to support everything like tystudio (to the determent of its maintainability IMO).
More specifically, for SA tivo's I only have 1 test stream and it would not play until I added a special case for an SA specific packet type, and also fixed the MPEG audio frame size equation. And that code has yet to be released, so if things work with S2SA, then I wouldn't know due to lack of any samples or feedback. S1 Dtivo won't play without special mpeg1 PES header processing logic (I just added that as well). All of this is special purpose stuff that I didn't care about until the HD stuff worked well (correctness first). Luckily low-def are simple special cases once you can handle HD frame processing. I think I've actually busted low-def video frame processing with the newer video buffering code as well, but that shouldn't be hard to clean up...

Understandable, . I would be glad to test any builds, ( aim: johnsolo1701), I also have small .tys from an S2SA and an S2DTivo. Those might help you with the testing. If you want those, I can setup an FTP.



Oh right it still doesn't work does it? I was quick&dirty with the file reading; it should really look something like the following. No need to do external buffering.
This works and named pipes now work as input. Excellent work, again.

johnsolo
10-04-2005, 10:21 PM
Here is what I am trying to do:

Open pipes for reading, a/v offset ignored for testing:
mplex -f 8 -o pipetest.mpg ~/.tivotool/pipe.m2v ~/.tivotool/pipe.m2a &> mplex.log &

Run two copies of hdemux, one for video, one for audio:
./hdemux -i ~/.tivotool/tydemux -v ~/.tivotool/pipe.m2v -a /dev/null &> demux.vid.log & && ./hdemux -i ~/.tivotool/tydemux -a ~/.tivotool/pipe.m2a -v /dev/null &> demux.aud.log &

Finally, feed the pipes:
mfs_uberexport -R 149616 -o ~/.tivotool/tydemux &> export.log

Mplex gets fed, and it produces pipetest.mpg. This file has artifacts (see image), and skips around, with audio dropping in and out.


video process:
Version 0.11
Source is /Users/john/.tivotool/tydemux
MPEG2 audio at 48kHz, 576 bytes/sync frame
Video frame size is 352x480 Frame rate 29.970030
Video bit rate is 5800000 bits/sec., 5800 Kbit/sec.
Audio timestamp offset: 66.233333 ms
Reported datarate 5992 Kbit/sec. (5800+192)
Proceed with remuxer:
mplex -f 8 -O 66ms -r 153701 /dev/null /Users/john/.tivotool/pipe.m2v -o <outfile>.mpg
1520568 audio stream bytes 47034330 video stream bytes

audio process:
Version 0.11
Source is /Users/john/.tivotool/tydemux
MPEG2 audio at 48kHz, 576 bytes/sync frame
Video frame size is 352x480 Frame rate 29.970030
Video bit rate is 5800000 bits/sec., 5800 Kbit/sec.
Audio timestamp offset: 230.733333 ms
Reported datarate 5992 Kbit/sec. (5800+192)
Proceed with remuxer:
mplex -f 8 -O 230ms -r 153701 /Users/john/.tivotool/pipe.m2a /dev/null -o <outfile>.mpg
1583288 audio stream bytes 45245258 video stream bytes

Two things I notice that are odd, the a/v offset changes and the stream sizes are slightly different. Doing this without pipes works fine, no video artifacts, etc. Trying to do e.g. -v pipe.m2v -a pipe.m2a doesn't seem to work at all, mplex won't recognize the pipes as open.

I am playing with leaving out -v or -a, but I can't seem to redirect the output, either through pipes (|) or >, 1>, 2>, 3>, etc. It always ends up getting fed to my terminal (stdout). hrm

bcc
10-05-2005, 12:59 AM
Wow, two hdemux processes. But both processes are reading from the same input pipe at the same time? Not sure how that could possibly work right - I think that's your problem. Each process is probably getting about 50% of the source chunks. The reason why hdemux has separate a/v file switches was so that you could theoretically output to 2 named pipes at the same time (not out of trying to make the UI cumbersome like some might think :) Personally I never did use it that way (read: untested). I actually at one point merged mplex&hdemux together into one program (with some c++ glue and memory buffering between them). That lets you go ty->mpg without extra intermediate disk or pipe I/O. Result was a bit messy and I didn't want to get into the mplex support+troubleshooting business. Not to mention mplex failing to be sufficiently portable (won't compile with visual c++ for example). But if you do want to get into that business maybe I could get you some code... Problem is I'm on the road the rest of the week.

johnsolo
10-07-2005, 04:11 PM
Thanks bcc, I got this sort of working with tees. Gotta hack on it this weekend and see what becomes of it.


hdemux -i /Users/john/.tivotool/tyfile1 -a /Users/john/.tivotool/pipe.m2a -v /dev/null &
sleep 1
hdemux -i /Users/john/.tivotool/tyfile2 -v /Users/john/.tivotool/pipe.m2v -a /dev/null &
sleep 1
tee ~/.tivotool/tyfile1 < ~/.tivotool/tyfile > ~/.tivotool/tyfile2 &
sleep 1
cat /Users/john/Desktop/ty\ Collection/Samurai\ Jack\ -\ IV.ty > ~/.tivotool/tyfile &
sleep 1

# This works like it should.. output the two streams..
cat ~/.tivotool/pipe.m2v > test.m2v
cat ~/.tivotool/pipe.m2a > test.m2a

# Seems like this should work, two streams, valid command line
#mplex -f 8 -o ~/test.mpg ~/.tivotool/pipe.m2v ~/.tivotool/pipe.m2a

Thats where I'm at. Not really an hdemux issue, but this is the same behavior I got when I did something like


hdemux -i pipe.ty -a pipe.m2a -v pipe.m2v
mplex -f 8 -o ~/test.mpg pipe.m2v pipe.m2a

I would assume mplex is broke, but this line is used in the TivoServer program with success:

mplex -f 10 -r $bitplus -o $homed/pipe.tys $homed/pipe.m2v $homed/pipe.m2a &> $homed/mplex.log &



mplex support+troubleshooting business.
No, thanks =) My goal here is to get vsplit out of Tivotool to HELP with all the support questions.

bcc
10-11-2005, 01:36 PM
I would assume mplex is broke,Well broke probably isn't the right word for it... The a/v streams don't generate data at the same data rate, and so you're gonna get an underrun pretty readily if you try to read from them at a constant rate like mplex does initially. Here, underrun means the fifo gets blocked in pipe_wait. You can verify this with ps. Splitting the stream generation with tee and an extra hdemux doesn't get rid of the blocking problem - you still hit the same problem through back-pressure. If you're gonna do this with pipes, you do need to at least add buffering like you thought. Perhaps increasing the socket buffer sizes to 64k or 128k will be enough.

johnsolo
10-13-2005, 06:54 PM
Perhaps increasing the socket buffer sizes to 64k or 128k will be enough.
Alright, I understand about 75% of that. =) Lots of questions coming your way. First, what is the switch for ps to show if something is in pipe_wait?
I am playing with buffering commands. Not sure what you mean by the socket buffer sizes. Do you mean inside the hdemux source?

Do you know how I would seperate the audio and video stdout streams of hdemux? Leaving out -a and -v would send both streams to stdout right? How do I figure out which is which? Also, "hdemux -i input.ty > out.streams" doesn't seems to actually redirect for me. Any ideas?

Finally, theres no way to get the total size of the stream in hdemux, is there? I imagine you are processing it as a stream, so you don't know where the end is. Anyways just checking (need max value for a deterministic progress bar =)

bcc
10-13-2005, 08:54 PM
Alright, I understand about 75% of that. =) Lots of questions coming your way. First, what is the switch for ps to show if something is in pipe_wait?You can use ps axlw and look at the WCHAN column. 'Course it's truncated, so you might want something like ps axwo pid,wchan:12,comm to see the full pipe_wait name.
I am playing with buffering commands. Not sure what you mean by the socket buffer sizes. Do you mean inside the hdemux source?To be precise, I mean the socket buffer(s) for the named pipe. Those buffers are maintained by the kernel, but could be set via setsockopt(). I believe you could normally do this thru a script, if one was simply piping between commands. See mfs_ftp's use of fconfigure for an example, using 128k buffers. But this case is more complicated, where hdemux should really have 2 output pipes. In this case I don't know how you could do it thru a script. And so, yes, I'd do the setsockopt() from hdemux.
Do you know how I would seperate the audio and video stdout streams of hdemux? Leaving out -a and -v would send both streams to stdout right? How do I figure out which is which? Also, "hdemux -i input.ty > out.streams" doesn't seems to actually redirect for me. Any ideas?Well I guess if you really wanted to make the output redirectable, and you wanted to keep the a/v streams separate, one could be sent to stdout, and one to stderr. But that would be a strange UI.
Finally, theres no way to get the total size of the stream in hdemux, is there? I imagine you are processing it as a stream, so you don't know where the end is. Anyways just checking (need max value for a deterministic progress bar =)Of the input stream? hdemux can stat the input file and know the size that way. But last I knew you were streaming into hdemux so that wouldn't work. Otherwise hdemux could see the size of each segment from the master chunk record, but it wouldn't know how many segments are being streamed to it.

johnsolo
11-04-2005, 05:58 AM
Hi bcc, thanks for all your helpful advice. Just wanted to let you know I am using hdemux in TivoTool now as an optional demuxer for S2 users. I'll keep an eye out in this thread if you ever release that S1 code. If you would want to share that, I could try to patch and compile that in (for OSX at least..) and see how it goes with my S1 testers.

bcc
11-06-2005, 03:55 PM
Here is a new release of hdemux. Changes include:
Add support for S1 dtivo, S2 standalone tivo
Change parsing of video frames to accommodate these older models which do not place PES frames adjacent to video sequence headers.
Change read processing to allow input from blocking device such as a pipe or fifo.

johnsolo
11-06-2005, 08:22 PM
Huge update. great job bcc.

Tested on a couple UK S1 streams and it works fine. (Also fine w/ S2SA and S2DTivo) Gonna play with named pipes soon.

Looking forward to ditching vsplit and all the headaches it carries with it....

khmann
11-17-2005, 10:19 AM
Just a quick note - I am using hdemux to demux music channels using s1 DTIVO 3.1.0b. The "timestamp discontinuity" stuff causes problems. the extracted audio only plays for 5-10 seconds before it skips forward about 5 minutes, plays for another 5 seconds, skips forward again...

so I disabled the timestamp discontinuity whatever stuff, and OH YEAH BABY!
perhaps this check could become optional in future versions of hdemux? since this check deals primarily with VTS, maybe make the -v flag optional and this check along with it?


--- hdemux.c-orig 2005-11-17 08:39:29.000000000 -0500
+++ hdemux.c 2005-11-17 08:39:54.000000000 -0500
@@ -1323,7 +1323,7 @@
}
chunk_parse_rec(p, &type, &major, &time);
if (!i) {
- if (!time_validate(time, lasttime, vstream.ticks_per_chunk)) {
+ if (0) {
if (errcnt++ < CONTINUITY_MAX_ERROR) {
/*
* Be silent about first discontinuity.

BEFORE:
D:\xtract>hdemux -i bpm-.ty -a test.m2a -v test.m2v
Version 0.12
Source is bpm-.ty
MPEG2 audio at 48kHz, 576 bytes/sync frame
Warning: Skipping chunk 4 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity
Warning: Skipping chunk 68 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity
Warning: Skipping chunk 132 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity
Warning: Skipping chunk 196 due to timestamp discontinuity
637466 audio stream bytes 0 video stream bytes

THANKYOU!

bcc
11-17-2005, 03:02 PM
The timestamp continuity check has sometimes caused problems, on the other hand, it also has done a good job of fining bad chunks/discontinuities. Yes, a 'loose' switch that disables the check is probably in order.
since this check deals primarily with VTSWell the code is a general PTS timestamp check, but was presuming that your data rate included a video stream. The code probably should adjust the data rate if no video stream is found. Making -v/-a optional makes sense too, and the following accomplishes that
--- hdemux.c~ 2005-11-06 17:04:08.000000000 -0800
+++ hdemux.c 2005-11-17 10:48:28.000000000 -0800
@@ -121,6 +121,9 @@
printf("Writing %d to stream %s\n", size, stream->name);
}
ofd = stream->fd;
+ if (!ofd) {
+ return;
+ }
ret = write(ofd, buf, size);
if (ret != size) {
error("error on write: requested: %d got: %d", size, ret);

khmann
11-22-2005, 12:06 AM
nice. I'm finding occasional glitches in the xm audio streams and am having problems after muxing with locally encoded video - seeking during playback, issues with DVD authoring apps, etc. I don't have any of these problems when using locally encoded audio.

for PTS checking/correction you might find something interesting in "replex" @ http://www.metzlerbros.org/dvb/index.html

For my application (combine extracted music with "timecode" video to make music DVDs so I don't have to transcode or buy mp3 players) a mplex mux > replex demux with PTS regeneration seems to give me an audio stream which behaves better, dunno why. I'm going to try to hack replex to operate on ES'...

anyway, thanks again for hdemux. Apologies for hijacking the thread for a completely unrelated purpose :)

MelvinPurvis
12-01-2005, 01:07 AM
While working with TivoTool under OS X, I encountered two issues detected by mplex in the ac3 audio streams:
1. Periodically, an extra pair of nulls (two bytes of zero) would appear in the next to last byte of the AC3 frame, making it 2 bytes too long.

I found this was the result of some code beginning at line 1158 that starts:
if (astream.syncoffset && (astream.syncoffset < 4) && overlap)

This special case caused the extra bytes to be written when it appears there is no legitimate special case. This code is commented out.

2. After resolving this issue, a further error appeared in mplex indicating that no audio packet was found at a particular instant. This corresponded roughly to a message "Warning: Tossing AC3 audio frame with bad sync data."

I also noted that as many as a quarter of the ac3 packs were marked as having recovered synchronization. This is clearly inappropriate for a stream the was visually flawless.

Each of these 'recovered' cases involved a scan of the AC3 data to locate the sync bytes. Unfortunately, unlike most protocols, AC3 data does not exclude the sync pattern from appearing in valid data. This means that having found a sync word may not indicate that a valid packet start has been found.

Because the sync recovery operation was being invoked far more often than it should have been, it would occasionally find a sync word in the AC3 data and conclude that a packet should be discarded.

The sync recovery code was executed because the audio stream syncOffset value was only set in one of the two main cases. To maintain sync it needs to be set every time a frame is extracted.

For whatever reason, neither of these paths seem to be exercised with satellite video, but of the air HD hits them immediately. I have seen no indication that these problems would be unique to either use with TivoTool or with an OS X compiled version.

A patch file for hdemux.c in the 0.12 version is attached. Hopefully it is constructed correctly.

john brisbin

bcc
12-01-2005, 02:46 AM
Actually no, that "fix" is not correct, as there is a legit case being covered by the condition you commented out. What is the actual problem you seeing with mplex, or is it just a message you get when you enable verbose diagnostics (-v 2)?
Now, I tried to describe that case in the comment right there:
* Sometimes OTA PES payload overlaps with next PES
* packet's start code prefix.In the problem case I've seen, the next PES header's start code is "sharing" two bytes of zeros with the payload from the AC3 frame. Sharing in the sense that thse 2 bytes are required to complete the 1536 byte ac3 frame, and later expected to not have already been consumed by my code that parses the start code. Your change would result in mplex bombing out on this stream due to the ac3 frame being truncated.
Here's an example:
gdb /u3/tivo/tyconv/hdemux
GNU gdb Red Hat Linux (6.3.0.0-1.21rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/libthread_db.so.1".

(gdb) br hdemux.c:1169
Breakpoint 1 at 0x804a538: file hdemux.c, line 1169.
(gdb) run -i oc-rager.ty -v o.m2v -a o.m2a
Starting program: /u3/tivo/tyconv/hdemux -i oc-rager.ty -v o.m2v -a o.m2a
Reading symbols from shared object read from target memory...done.
Loaded system supplied DSO at 0x40000000
Version 0.12
Source is oc-rager.ty
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
Audio timestamp offset: 374.044444 ms
AC3 audio at 48kHz, 1536 bytes/sync frame
Recovered AC3 sync, offset 142, chunk 2
Reported datarate 19384 Kbit/sec. (19000+384)
Proceed with remuxer:
mplex -f 8 -O 374ms -r 29076 o.m2a o.m2v -o <outfile>.mpg

Breakpoint 1, parse_audio_frame (start_code=445, major=9, chunk=7,
headlen=0xbff4d41a, prepos=0xbff4d410, size=0xbff4d43c,
framesize=0xbff4d414) at hdemux.c:1169
1169 if (dbg(1)) {
(gdb) p payoffset
$1 = 122
(gdb) p paybytes
$2 = 1642
(gdb) p *framesize
$3 = 1536
(gdb) x/20b &astream.buf[122+1536-4]
0x4001267e: 0xf1 0xb0 0x00 0x00 0x01 0xbd 0x05 0xba
0x40012686: 0x80 0x80 0x05 0x2d 0xb4 0x17 0xa9 0x05
0x4001268e: 0xa3 0xce 0x0b 0x77
(gdb) I could probably handle your case by making the above special case check consider whether the subsequent data actually has a 000001bd start code, but I'd like to understand the case fully before doing that.

As for case 2, yes that warning means that hdemux is in a world of hurt (maybe it should just assert), and you shouldn't see that in the middle of a healthy stream. Perhaps you're causing that situation thru your case 1 change, but I'd have to see your stream to be sure.

MelvinPurvis
12-01-2005, 07:17 AM
Actually no, that "fix" is not correct, as there is a legit case being covered by the condition you commented out. What is the actual problem you seeing with mplex, or is it just a message you get when you enable verbose diagnostics (-v 2)?
Now, I tried to describe that case in the comment right there:
* Sometimes OTA PES payload overlaps with next PES
* packet's start code prefix.In the problem case I've seen, the next PES header's start code is "sharing" two bytes of zeros with the payload from the AC3 frame. Sharing in the sense that thse 2 bytes are required to complete the 1536 byte ac3 frame, and later expected to not have already been consumed by my code that parses the start code. Your change would result in mplex bombing out on this stream due to the ac3 frame being truncated.


That is a novel idea, but it is hard to justify coding to make a broken stream work and a legal one fail. Byte sharing is not legal (and I would argue does not happen in practice, even bad practice.)

The case that I saw (many times) is one where the last two bytes are not 0 but are properly placed after the header information following the 0x000001bd marker (just as they are in the gdb example you provided). Two bytes are just as legitimate in this position as the times when 50 bytes overlap into the next packet. You can not assume that less than 4 is somehow magic.

The fix you talk about would only work if the first data byte of the next packet were the first byte of the sync code. If it is not, you will write two additional bytes and the 0x700 byte frame becomes 0x702 which breaks any attempt to remux the data. In other words, this trick breaks any legal frame that extends into the next packet by anywhere from 1 to 3 bytes based on the value of astream.syncoffset.

astream.syncoffset was just determined by verifying the location of the next sync byte a few lines above in function ac3_syncoffset. The data exists in the correct place, not sharing space with the start code. ac3_syncoffset found it there!

The commented out lines just insert 1 to three bytes (2, in practice, because everything is even sized) of extra data in an otherwise perfectly good frame.

In your gdb example, the now commented out code is trying to ignore the valid data bytes at offset 0xe and 0xf (0xa3 0xce) into the new pack. Instead a pair of zeros and the 0xa3 and 0xce get written to output and mplex rightly complains.

If you throw away the idea of byte sharing instead of the bytes themselves, the special case disappears and everything works better.


As for case 2, yes that warning means that hdemux is in a world of hurt (maybe it should just assert), and you shouldn't see that in the middle of a healthy stream. Perhaps you're causing that situation thru your case 1 change, but I'd have to see your stream to be sure.

To refute the assertion that the first fix might have caused this, the now commented out code only ran about once in twenty frames or only a fourth as often as the rescanning occurred in the same file. It could not be responsible for all the rescanning that occurs.

As I explained in the first message, it only gets in a 'hurt' because it is not maintaining the astream.syncoffset except when the expression (paybytes > (*framesize +4)) is true. When that is false (about 1 in 5 times in the case I was looking at) the field value misinforms find_ac3_frame (which depends on it for the in sync case) and it does a byte by byte scan of the data to locate the 0xb77.

This extra scanning is innocuous when the next instance of 0xb77 is the sync word, but eventually, with enough data, the scan picks up a 0xb77 in the audio data and throws the packet away when it cannot validate the AC3 configuration info that follows it.

I would ask you to look at your code and explain how the function find_ac3_frame can work properly for the subsequent frame if the prior one does not set astream.syncoffset?

All theory and argument aside, I am enjoying having OTA streams that remux without error.

Thanks for making the tool available.

bcc
12-01-2005, 04:15 PM
That is a novel idea, but it is hard to justify coding to make a broken stream work and a legal one fail. Byte sharing is not legalActually no, this is not an illegal case. It just sounds that way if you think of it as byte sharing. If you look at the way start code processing is defined in iso13818, you must merely have 23 zero bits, and a 1 bit before the start code. The spec doesn't say that you must insert 23 zero bits and a 1 bit before the next start code. In other words there is no requirement that those bits must not overlap the end of an access unit.
But since I don't parse the stream in such a general bit stream fashion, I had to code a special case.
(and I would argue does not happen in practice, even bad practice.)I guess I can't stop you, but I've already shown you an example stream proving otherwise. It's a real OTA case from fox's 'The OC' (no I don't watch that show I just got the stream sent to me :) I also have the same special case from 'jay leno'.

Now, you've shown me some diffs that break the above cases. Even if you're unable to admit that the above cases are "legal" your changes violate the "be liberal in what you accept" rule, and thus break real world streams. You're supposedly fixing another case, but you have not made fully clear the circumstances for that case.

I'll come up with a more general fix that'll probably cover your case. If you could provide a sample stream I could be sure. The existing code has worked fine for all streams reported thus far (a fairly wide variety IMO).

bcc
12-01-2005, 09:19 PM
Here is a new release of hdemux. Changes:

Fix PES audio payload size computation
Relax timestamp validity checking to not kick-in until we've seen a valid video sequence. This should allow audio-only recordings to demux without any user intervention.
Don't require both a&v output files for processing (allow audio only)
Fix processing for PAL resolutions and frame rates

MelvinPurvis
12-02-2005, 01:40 AM
Actually no, this is not an illegal case. It just sounds that way if you think of it as byte sharing.
Byte sharing is your phrase, not mine.


If you look at the way start code processing is defined in iso13818, you must merely have 23 zero bits, and a 1 bit before the start code. The spec doesn't say that you must insert 23 zero bits and a 1 bit before the next start code. In other words there is no requirement that those bits must not overlap the end of an access unit.
That is absurd. MPEG is a nested containment architecture. Nothing ever overlaps.


Now, you've shown me some diffs that break the above cases. Even if you're unable to admit that the above cases are "legal" your changes violate the "be liberal in what you accept" rule, and thus break real world streams. You're supposedly fixing another case, but you have not made fully clear the circumstances for that case..

The problem you have in your code is that you inserted this hack based on a misunderstanding and created the idea of byte sharing to explain it. In fact, as I showed you with your own gdb example, the correct data is in the stream immediately prior to the 0xb77 code. There is no need to resort to such fictions.

Its clear that nothing I can do will 'show' you these bugs.

I have given you specifics which you have chosen to ignore.

I have asked you specific questions about your code, which if answered truthfully, would require you to admit these bugs. You ignored that.

It is sad that your pride will prevent you from improving the quality of your software which had potential.

People who have programmed for some time generally learn to deal with their own fallibility. I write code. It generally has bugs, and they must be fixed. This is life. You will do better work when you realize this.

You will be glad to hear that I will not bother to inform you of other bugs in your software.

bcc
12-02-2005, 02:45 PM
Byte sharing is your phrase, not mine.Yes, it was a convenient way to explain the special case processing. Not so convenient for explaining how it isn't forbidden from a protocol point of view unfortunately. Sorry to confuse you.
In fact, as I showed you with your own gdb example, the correct data is in the stream immediately prior to the 0xb77 code. There is no need to resort to such fictions.Oh right, damn I didn't capture the failure scenario with that gdb output. The real failure case, from NBC (leno), is:
(gdb) p payoffset
$4 = 14
(gdb) p paybytes
$5 = 6142
(gdb) p *framesize
$6 = 1536
(gdb) x/20b &astream.buf[14+paybytes-4]
0x40013810: 0x84 0xf3 0x70 0x00 0x00 0x00 0x01 0xbd
0x40013818: 0x18 0x08 0x85 0x80 0x05 0x21 0x52 0x83
0x40013820: 0xd1 0x87 0x0b 0x77
(gdb)
In this case the advertised payload size is again 2 bytes too short (given where the next_start_code is), and the 2 bytes before the next ac3_sync are part of the PES header. Ie you can't pick up the leftover payload from the next PES packet. The special case code kicks in and handles this. With your diffs we fail to write a 1536 byte ac3 frame, and mplex blows up. Clearly worse behavior.

Anyways, so you're right in that I had a bug here (introduced in more recent changes). And that bug was causing the special case code to go off far too often. Fixed in 0.13 already.
I have asked you specific questions about your code, which if answered truthfully, would require you to admit these bugs. You ignored that.Right, I did ignore you after case 1., because your understanding of the code was off track at that point, and thus a waste of my time to counter further.
It is sad that your pride will prevent you from improving the quality of your software which had potential.

People who have programmed for some time generally learn to deal with their own fallibility. I write code. It generally has bugs, and they must be fixed. This is life. You will do better work when you realize this.

Spare me the inappropriate lecture, I don't know you from Joe content pirate.
Nice of you to have introduced yourself with a: Hi I'm sure your code is buggy here, I commented it out without fully understanding, never mind any complete details about the actual problem I'm fixing.
You come in here with a diff that "fixes" some undisclosed scenario that you have and blows up my test suite, and you expect me to integrate your diff? You should know that's not how things work. Here, in this thankless user "community" your "must be fixed" assumption about bugs do not even hold true. This code is all just for the hell of it.

You will be glad to hear that I will not bother to inform you of other bugs in your software.I am. Bugs that are only seen by you, should now matter to me?

johnsolo
12-05-2005, 04:33 AM
0.13 compiles on Mac OSX perfect for me. Tested on S2SA, S2DTivo and S1UK test streams. All worked fine. Thanks for the new release again, I really appreciate it bcc. :o :o

Cheezmo
12-05-2005, 09:05 AM
[never mind, may have had a disk full problem]...

zooppoop
12-24-2005, 12:04 PM
After reading many forums on this server I finially found this thread. THANK YOU. :) So I got hdemux installed and I first tried running this on a SD recording I did as it said and ran mplex as the command was printed. Playing it on a different computer preduced some weird Video effects. It just had lines through it the entire time they didn't move. I kinda thought it might have been the computer It was playing on since it's been having some problems, so I figured I would try it later on another computer. I then tried to demux an HD stream and got a Segmentation fault with the program.


Version 0.13
Source is Video.ty
Video frame size is 1280x1088 - high definition. Frame rate 29.970030
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
Audio timestamp offset: 625.255556 ms
AC3 audio at 48kHz, 1536 bytes/sync frame
Reported datarate 65384 Kbit/sec. (65000+384)
Proceed with remuxer:
mplex -f 8 -O 625ms -r 153701 Video.m2a Video.m2v -o <outfile>.mpg
Warning: Skipping chunk 4098 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity
Segmentation fault


So I thoguht I would apply some of the Hacks in the message board about this. The one I found that was related was the Audio stream problem I didn't know if it would fix it or not but Hey give it a shot. So this is the result from that.


zoop@yours /files2 $ hdemux -i Video.ty -v Video.m2v -a Video.m2a
Version 0.13
Source is Video.ty
Video frame size is 1280x1088 - high definition. Frame rate 29.970030
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
Audio timestamp offset: 625.255556 ms
AC3 audio at 48kHz, 1536 bytes/sync frame
Reported datarate 65384 Kbit/sec. (65000+384)
Proceed with remuxer:
mplex -f 8 -O 625ms -r 153701 Video.m2a Video.m2v -o <outfile>.mpg
Segmentation fault


That didn't help might might give you some info on where it's failing. not sure what the problem might be but this is a 64bit proc, and it's installed on a system compiled as 64bit (Gentoo). I'm happy to install any debugging software to help figure this out or if you have any info that could help. I tried an strace frist as that's what I normally use, but I didn't have it installed. So I'll give that a whoorl later. Also I was going to try this on a Non 64 bit system too to see if that made a difference. I'll let you know.

Thanks again.

OK did an strace I guess a little useless, but here yea go. Shoot or maybe not I didn't see that priority set crap. that's gotta be it. I'll get that cleaned out of the MFS tools.



read(3, "\21\0\377\377\0\0O *\255X\f\0\0\0j\21\353\371\305\7\365"..., 131072) = 131072
write(5, "\vwH\7\0340\341\347\354\354\222@\0\3\377o\355KP0!\241\n"..., 1536) = 1536
write(5, "\vw\33\343\0340\341\347\354\354\222@\0\3\377o\355KP@Aa"..., 1536) = 1536
write(5, "\vw<)\0340\341\347\354\354\222@\0\3\377o\355KP\20\21\221"..., 1536) = 1536
write(4, "\0\0\1\0\2\327\377\373\200\0\0\1\265\206_\373]\200\0\0"..., 60084) = 60084
write(4, "\0\0\1\0\2_\377\373\270\0\0\1\265\206V[\335\200\0\0\1\262"..., 35276) = 35276
write(4, "\0\0\1\0\2\237\377\373\270\0\0\1\265\206V[\337\200\0\0"..., 33704) = 33704
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 131072) = 131072
read(3, "Priority set...\n\365Fz\275\0\0\0\2\0\2\0\0\0\17\377\4"..., 131072) = 131072
read(3, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\37\0\16\200\0\0O *\255"..., 131072) = 131072
read(3, "/\346\326\361\240\206\0\255\345\234\215\3527\372\260\20"..., 131072) = 131072
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++

bcc
12-24-2005, 03:22 PM
read(3, "Priority set...\n\365Fz\275\0\0\0\2\0\2\0\0\0\17\377\4"..., 131072) This shows that you're using an ancient version of the mfs_export utility to extract your recording, and that program is corrupting your stream. I wonder where you got it. I recommend you replace the mfs_export program you installed with the one here
http://www.dealdatabase.com/forum/showthread.php?t=39487
You'll have to either re-extract the recording, or use something like unpriority to repair: http://www.dealdatabase.com/forum/showpost.php?p=172899&postcount=646
For debugging under linux, gdb's backtrace command will show where the error occured. For example: gdb hdemux core.11354<ret>backtrace<ret>

zooppoop
12-25-2005, 12:13 AM
Kewl that was it. Yea don't know where I got those tools from. I belive it was one of these forms. Would be nice if this was a wiki or something that could have one document that could be updated. Took quite a while to go through the forms. Thanks for your help, it's much appreciated.

zooppoop
12-25-2005, 11:05 PM
I'm running into a problem with some of my streams, this is the error that I get from mplex.


zoop@yours /files2 $ mplex -f 8 -O -10169ms -r 23076 m2a m2v -o Video.mpg
INFO: [mplex] mplex version 1.6.2 (2.2.3 $Date: 2004/01/13 20:45:26 $)
INFO: [mplex] File m2a looks like an AC3 Audio stream.
INFO: [mplex] File m2v looks like an MPEG Video stream.
INFO: [mplex] Video stream 0: profile 8 selected - ignoring non-standard options!
INFO: [mplex] Found 1 audio streams and 1 video streams
INFO: [mplex] Selecting dvdauthor DVD output profile
INFO: [mplex] Multiplexing video program stream!
INFO: [mplex] Scanning for header info: AC3 Audio stream 00 (m2a)
INFO: [mplex] AC3 frame size = 1536

INFO: [mplex] AC3 AUDIO STREAM:
INFO: [mplex] Bit rate : 49152 bytes/sec (384 kbit/sec)
INFO: [mplex] Frequency : 48000 Hz
INFO: [mplex] Scanning for header info: Video stream e0 (m2v)
INFO: [mplex] VIDEO STREAM: e0
INFO: [mplex] Frame width : 480
INFO: [mplex] Frame height : 480
INFO: [mplex] Aspect ratio : 4:3 display
INFO: [mplex] Picture rate : 29.970 frames/sec
INFO: [mplex] Bit rate : 15000000 bits/sec
INFO: [mplex] Vbv buffer size : 229376 bytes
INFO: [mplex] CSPF : 0
INFO: [mplex] SYSTEMS/PROGRAM stream:
INFO: [mplex] rough-guess multiplexed stream data rate : 15710000
INFO: [mplex] target data-rate specified : 23076000
INFO: [mplex] Setting specified specified data rate: 23076000
INFO: [mplex] Run-in Sectors = 698 Video delay = 44602 Audio delay = 968821
INFO: [mplex] New sequence commences...
INFO: [mplex] Audio bd: buf= 16384 frame=000000 sector=00000000
INFO: [mplex] Video e0: buf=1900544 frame=000000 sector=00000000
**ERROR: [mplex] Can't find next AC3 frame: @ 231936 we have fffd - broken bit-stream?


This one I get right away in another stream that I d/l'ed it converted most of the video then had a failure. I have successfully converted other streams from this tivo. Any help would be appreciated. I know this isn't your software just thought you could give me some direction to trouble shoot it.

Thanks agian.

bcc
12-26-2005, 12:54 AM
Yes, a wiki *could* be nice. But, there is already one over at alt.org, and it suffers from wiki spam, and so would have to be actively administrated. Maintaining 3rd party how-to guides on this stuff has proven rather thankless, and usually the guide gets abandoned and becomes stale.

**ERROR: [mplex] Can't find next AC3 frame: @ 231936 we have fffd - broken bit-stream?That fffd looks suspiciously like an mpeg2 audio sync code. I would suspect your audio stream switched from ac3 to mpeg2 mid stream, and mplex is not able to handle that. Maybe you can play back the audio stream with xine and check whether it is flip flopping (check alt-i)

bigcat400
12-30-2005, 08:43 PM
Folks, any ideas on how to troubleshoot the following error?

$> hdemux -i show.ty+ -v show.m2v -a show.m2a
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
Audio timestamp offset: 389.522222 ms
AC3 audio at 48kHz, 1792 bytes/sync frame
Recovered AC3 sync, offset 152, chunk 1
Reported datarate 19448 Kbit/sec. (19000+448)
Proceed with remuxer:
mplex -f 8 -O 389ms -r 29172 ws2005.m2a ws2005.m2v -o <outfile>.mpg
Warning: Tossing AC3 audio frame with bad sync data. Chunk 943 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 1965 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 4576 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 4848 length 1840
Warning: Skipping chunk 5760 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity
Warning: Skipping chunk 5826 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity
error: Cannot fit Audio buffer, size 473313 offset 4928

bigcat400
12-31-2005, 10:28 AM
I got past my hdemux error using a different version of the mfs_* tools (unified ones). But my mplex command is now failing with the following error:

**ERROR: [mplex] Can't find next AC3 frame: @ 3596544 we have 886b - broken bit-stream?

Very similar to what zooppoop posted previously. The full output here:

$ mplex -f 8 -O 389ms -r 29172 show.m2a show.m2v -o f:/show.mpg
INFO: [mplex] mplex version 1.8.0 (2.2.4 $Date: 2005/08/28 17:50:54 $)
INFO: [mplex] File show.m2a looks like an AC3 Audio stream.
INFO: [mplex] File show.m2v looks like an MPEG Video stream.
INFO: [mplex] Video stream 0: profile 8 selected - ignoring non-standard options!
INFO: [mplex] Found 1 audio streams and 1 video streams
INFO: [mplex] Selecting dvdauthor DVD output profile
INFO: [mplex] Multiplexing video program stream!
INFO: [mplex] Scanning for header info: AC3 Audio stream 00 (show.m2a)
INFO: [mplex] AC3 frame size = 1792
INFO: [mplex] AC3 AUDIO STREAM:
INFO: [mplex] Bit rate : 57344 bytes/sec (448 kbit/sec)
INFO: [mplex] Frequency : 48000 Hz
INFO: [mplex] Scanning for header info: Video stream e0 (show.m2v)
INFO: [mplex] VIDEO STREAM: e0
INFO: [mplex] Frame width : 1280
INFO: [mplex] Frame height : 720
INFO: [mplex] Aspect ratio : 16:9 display
INFO: [mplex] Picture rate : 59.940 frames/sec
INFO: [mplex] Bit rate : 19000000 bits/sec
INFO: [mplex] Vbv buffer size : 999424 bytes
INFO: [mplex] CSPF : 0
INFO: [mplex] SYSTEMS/PROGRAM stream:
INFO: [mplex] rough-guess multiplexed stream data rate : 19858896
INFO: [mplex] target data-rate specified : 29172000
INFO: [mplex] Setting specified specified data rate: 29172000
INFO: [mplex] Run-in Sectors = 89 Video delay = 39508 Audio delay = 9003
INFO: [mplex] New sequence commences...
INFO: [mplex] Audio bd: buf= 0 frame=000000 sector=00000000
INFO: [mplex] Video e0: buf= 0 frame=000000 sector=00000000
**ERROR: [mplex] Can't find next AC3 frame: @ 3596544 we have 886b - broken bit-stream?



Folks, any ideas on how to troubleshoot the following error?

$> hdemux -i show.ty+ -v show.m2v -a show.m2a
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
Audio timestamp offset: 389.522222 ms
AC3 audio at 48kHz, 1792 bytes/sync frame
Recovered AC3 sync, offset 152, chunk 1
Reported datarate 19448 Kbit/sec. (19000+448)
Proceed with remuxer:
mplex -f 8 -O 389ms -r 29172 ws2005.m2a ws2005.m2v -o <outfile>.mpg
Warning: Tossing AC3 audio frame with bad sync data. Chunk 943 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 1965 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 4576 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 4848 length 1840
Warning: Skipping chunk 5760 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity
Warning: Skipping chunk 5826 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity
error: Cannot fit Audio buffer, size 473313 offset 4928

bcc
01-10-2006, 02:36 AM
bigcat400,

**ERROR: [mplex] Can't find next AC3 frame: @ 3596544 we have 886b - broken bit-stream?
So it looks like that error is about 60 seconds into your audio stream. Perhaps it is a real audio glitch in your source. Can you play back the source .ty with no glitch at 62 seconds? What about the demultiplexed audio stream? Can you play it back by itself OK?

bigcat400
01-10-2006, 10:07 AM
Thanks for your response bcc.

Yes this .ty+ was from HD OTA which has glitches every once in a while (due to my antenna probably). I was able to play the full audio stream with GoldWave so at least I knew it was complete, with some issues along the way I guess.

Anyway I was able to put the audio and video back together (mpeg) using TMPEnc and created my DVD.

I've realized that with HD OTA streams the thing is to try different tools until finding something that will work :) .

I have used hdemux/mplex successfully with many HD extractions. But, yesterday for example I was working with one that neither mplex nor tmpenc liked, they both aborted claiming audio issues... but then I tried tytool and worked great (ty+ directly to mpeg) :) . It'd be nice to always use one tool, but oh well, as long as something works we should be ok.

I always use mfs_ftp to extract (ty+ files).


bigcat400,

**ERROR: [mplex] Can't find next AC3 frame: @ 3596544 we have 886b - broken bit-stream?
So it looks like that error is about 60 seconds into your audio stream. Perhaps it is a real audio glitch in your source. Can you play back the source .ty with no glitch at 62 seconds? What about the demultiplexed audio stream? Can you play it back by itself OK?

ricka
05-15-2006, 08:04 PM
Hello,

I'm trying to downloads hdmux and it keeps asking me to login, accepting my credentials, then asking me to login again. Is there something I'm missing? Is there anywhere else I can download hdmux?

edit: Nevermind. I switched from Safari to Firefox and it downloaded fine.

Thanks,

--Rick

cheer
06-15-2006, 10:18 PM
OK. Had a number of files that exhibit the "Can't find next AC3 frame" problem in mplex, but after perusing this thread I'm gonna play with the audio files a bit and see what I can see. However, new problem, and this one is actually in hdemux itself:

Version 0.13
Source is er.ty
Video frame size is 1920x1080 - high definition. Frame rate 29.970030
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
Audio timestamp offset: 461.400000 ms
AC3 audio at 48kHz, 1536 bytes/sync frame
Reported datarate 65384 Kbit/sec. (65000+384)
Proceed with remuxer:
mplex -f 8 -O 461ms -r 153701 er.m2a er.m2v -o <outfile>.mpg
error: Cannot fit Video buffer, size 120704 offset 412740
Any ideas? It's an OTA recording (of course)...the error comes up almost immediately.

cheer
06-16-2006, 08:49 AM
Update on the audio issues.

OK, so first I grabbed besweet and tried converting the audio to mp2 audio. That changed my error -- now I get a comment saying it needed to create a new file but no %d was present. So I added %d to the output filename and tried again, and I got three MPG files.

Not liking that, I tried converting the audio to mp3. Same problem. So now I tried converting it to wav and then back to ac3. Back to the "can't find next AC3 frame" problem.

So I downloaded ac3fix and ran the file through that. Same problem. So then I downloaded besplit and ran the file through that. Died with a different error (don't remember and didn't log it).

So then I tried running through ac3fix and then besplit. Now I get
++ WARN: [mplex] No seq. header starting new sequence after seq. end!
**ERROR: [mplex] Sequence split detected 3 but no following sequence found...
Augh! Anyone have any other ideas on how to fix a b0rked ac3 stream? I'm sure it's what bcc speculated: it's flipping between m2a and ac3 in the stream, or something similar.

FWIW, I tried TyTool10R4...couple of bad dumps, but it keeps processing...until it just stops and says complete, far short of the actual file end.

Note that hdemux did indeed spit out errors when demuxing:

Version 0.13
Source is house.ty
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
Audio timestamp offset: 408.544444 ms
AC3 audio at 48kHz, 1792 bytes/sync frame
Recovered AC3 sync, offset 80, chunk 1
Reported datarate 19448 Kbit/sec. (19000+448)
Proceed with remuxer:
mplex -f 8 -O 408ms -r 29172 house.m2a house.m2v -o <outfile>.mpg
Warning: Tossing AC3 audio frame with bad sync data. Chunk 1546 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 1818 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 2107 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 5206 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 5547 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 6523 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 8454 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 9892 length 1656
Warning: Tossing AC3 audio frame with bad sync data. Chunk 10719 length 1656
Warning: Tossing AC3 audio frame with bad sync data. Chunk 11224 length 1656
Warning: Tossing AC3 audio frame with bad sync data. Chunk 11264 length 1656
Warning: Tossing AC3 audio frame with bad sync data. Chunk 11333 length 1656
Warning: Tossing AC3 audio frame with bad sync data. Chunk 11850 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 12520 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 13547 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 13886 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 17868 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 19551 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 21256 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 21729 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 22667 length 1656
Warning: Tossing AC3 audio frame with bad sync data. Chunk 22824 length 1472
Warning: Tossing AC3 audio frame with bad sync data. Chunk 23183 length 1472
Warning: Tossing AC3 audio frame with bad sync data. Chunk 23239 length 1656
Warning: Tossing AC3 audio frame with bad sync data. Chunk 23408 length 1656
Warning: Tossing AC3 audio frame with bad sync data. Chunk 23764 length 1656
Warning: Tossing AC3 audio frame with bad sync data. Chunk 24484 length 1840
Warning: Tossing AC3 audio frame with bad sync data. Chunk 26227 length 1840
Warning: Skipping chunk 30384 due to timestamp discontinuity
201142728 audio stream bytes 3763501022 video stream bytes
What sucks is that OTA is the best quality HD out there...but the flakiness of the streams (esp. on audio) makes it such a PITA to deal with.

cheer
06-16-2006, 06:01 PM
OK. I pulled every OTA HD show off my HR10-250 that was there. Not tons -- it is rerun season after all.

Of the nine shows that I pulled off, I was successfully able to hdemux/mplex two of them. Of the seven that failed:
2 failed during hdemux with "Cannot fit video buffer"
2 errored during hdemux (repeated tossing audio frames with bad sync, as well as skipping chunk due to timestamp discontinuity) and failed during mplex (Can't find next AC3 frame, broken bitstream?)
1 errored during hdemux (repeated tossing audio frames with bad sync, as well as skipping chunk due to timestamp discontinuity) and failed during mplex (no seq. header starting new sequence after seq. end! and sequence split detected 3 but no following sequence found)
1 errored during hdemux (tossing audio frames w/bad sync, skipping chunks due to timestamp discont., AND attempting resync due to repeated time discontinuity) and failed during mplex (Can't find next AC3 frame, broken bitstream?)
1 errored during hdemux (just skipping chunk due to timestamp discontinuity) and failed during mplex (Can't find next AC3 frame, broken bitstream?)I tried all sorts of things on the audio files, including running them through ac3fix, bbsplit, delaycut, mixtures of the above, and re-encoding to various other formats.

Any ideas? Also, any suggestions for a different muxing tool?

Just for a sanity check I'll run all of these through TyTool and post the results.

EDIT: Attached are logs from my tests. They include the output of hdemux, the output of mplex (if I got that far), and a file listing to show the ending file sizes.

bcc
06-16-2006, 10:09 PM
Where to start? You seem to be reporting 3 actual errors, below:

In general, "Can't find next AC3 frame" errors from mplex could be due to mplex issues, hdemux bugs, or problems with the actual stream. From your later messages, it seems like we can rule out the first scenario, as you're getting errors at the hdemux step. For a healthy ac3 stream, I'd expect to see the '0b 77' marker bytes repeating at a constant interval. You haven't sent me a stream, so I really can only guess what happened here. My guess would be that you truly have problems with your source stream, but that hdemux could do a lot better job in recovering from those problems. For example it should probably split the audio stream into segments if the stream changes between mpeg2 and ac3. If hdemux could assume a multiplexor that was able to handle such changes, this might not be necessary. But mplex is not such a multiplexor. You can work around this issue by pre-editing your .ty file with a chunk editor. Your "lost" demux log looks like a more complicated failure than that - but I can only guess from the log. Would be interesting to see that stream.

Timestamp discontinuities:
It is "expected" to see this from hdemux concerning chunks at the end of the file. Compare the # of chunks in the source vs the chunk # where the error is reported. Later versions of hdemux fix this so that you don't see this cosmetic warning. If you're seeing this from hdemux at some other point besides the end of the file, then you can expect problems with the demuxed streams as well. For those cases, I would verify with a .ty player such as mplayer whether or not there is a hiccup in the stream at the point that hdemux is claiming. Use tychopper or dd to splice to the problem section. Looks like most of your logs are from the cosmetic case, the exception being the two with the "attempting resync" messages.

"Cannot fit video buffer":
This can either be a symptom of an algorithm bug or a simple buffer sizing problem. I believe I did find the video buffer undersized since 0.13, so I probably simply need to make a new release. Again, lacking a sample, all I can give is my best guess.


Also, any suggestions for a different muxing tool?There's always ffmpeg, but your problems seem to be before you get to multiplexing.

cheer
06-17-2006, 03:33 PM
bcc, thanks for the detailed response.

So far as the AC3 frame stuff...yeah, I'm sure it's the source streams that are hosed. You mention pre-editing my tystreams with a chunk editor...never done such a thing, but I'll give it a shot. Are you suggesting that I make edit points around the spots where the problems are and then process separately? I'll play around and see what I can find.

The time seq. stuff...I'll fire up mplayer and see if I can spot stream errors, etc., near the spots hdemux identifies. If the other remarks (from the end of the file) are cosmetic, then I presume my mplex probs stem from something else (again, the fubar ac3 streams being a likely culprit).

I can easily give you samples of the streams...in fact, PM me a mailing address and you'll have some DVD-Rs magically appear forthwith. :)

Thanks for all your help!

--chris

P.S. Would it be of any use to see logs from something like ac3fix? Prolly not since by that point hdemux is already done, but...

P.P.S. For laughs, I ran all the streams through TyTool10R4. That doesn't do me much good (because I need to be able to batch process these things), but...more information better than less, I suppose. Anyway, all but two appear to have spit out valid MPG files (I haven't watched 'em yet). The two that failed were House -- no errors (apart from Bad Dump messages, but I get those all the time on successful OTA HD muxes) and Lost. Lost actually caused TyTool to GPF (or whatever they call it now) -- dialog box offering to send a report to MS, yadda yadda yadda. The .txt file generated by TyTool is full of all sorts of issues -- OOB packets, unknown PES packets, damaged video frames, and finally (just before the crash) an audio type change.

If those logs would be of any value, let me know; not sure if they mean anything to anyone but Josh.

bcc
06-17-2006, 10:11 PM
You mention pre-editing my tystreams with a chunk editor...never done such a thing, but I'll give it a shot. Are you suggesting that I make edit points around the spots where the problems are and then process separately? I'll play around and see what I can find.Since you're using OTA streams, they are likely fully of transitions in&out of HD, and varying AC3 audio rates due to commercial breaks and the like. Those transitions are likely screwing up hdemux and mplex both, as neither was designed for streams that change like that. Since you're probably going to edit out those transitions anyways, if you do so by pre-chopping your .ty, such that the resulting pieces are all one format, you could avoid the problem with hdemux&mplex. Personally I don't know of anything OTA that I'd want to archive so I haven't made hdemux handle this better. Yes, movies OTA are higher bit rate than directv's HD offerings but the commercial break interruptions make them unsuitable for archival IMO.

By pre-chopping I'm thinking of tychopper (windows) or dd (unix)

P.S. Would it be of any use to see logs from something like ac3fix? Prolly not since by that point hdemux is already done, but...I'm not familar with ac3fix, it probably would be easier for me to just look at a sample .ty.

If those logs would be of any value, let me know; not sure if they mean anything to anyone but Josh.Right, he's got his own private debug messages that are only going to be fully meaningful to him. Lacking source, the rest of our hands are tied when tytool barfs.

cheer
06-18-2006, 06:22 AM
Since you're using OTA streams, they are likely fully of transitions in&out of HD, and varying AC3 audio rates due to commercial breaks and the like. Those transitions are likely screwing up hdemux and mplex both, as neither was designed for streams that change like that. Since you're probably going to edit out those transitions anyways, if you do so by pre-chopping your .ty, such that the resulting pieces are all one format, you could avoid the problem with hdemux&mplex. Personally I don't know of anything OTA that I'd want to archive so I haven't made hdemux handle this better. Yes, movies OTA are higher bit rate than directv's HD offerings but the commercial break interruptions make them unsuitable for archival IMO.
Agreed -- but in fact I'm not going to be either archiving or editing these.

My son is a long-distance trucker. I'm using (or would like to use) EtiVo as an automated mechanism to allow him and his fiancee (she's on the truck too) to pull down their favorite shows. Most of them I just record on a non-HD Tivo, but some I do in HD because we watch them as well.

EtiVo by default uses WMV (ugh) but bluewomble wrote a nifty addin that allows one to use Divx (or Xvid or...well, it just uses mencoder, so whatever). His addin runs vsplit prior to mencoder to turn the .ty into a .mpg. Vsplit doesn't handle HD streams at all, really, but santa8claws wrote a "vsplit.bat" file designed to call hdemux/mplex. This didn't work reliably for me, and it had the additional limitation of not using the -O -r values...so I wrote a wrapper program called fakevsplit. Fakevsplit runs your hdemux and redirects the output to a file, then parses the file, extracts the -O and -r values, and runs mplex. From there the resulting .mpg file is handed off to bluewomble's addin, which fires off mencoder to finish the job.

This works remarkably well on all SD videos I've tried, and all of D*'s HD stuff.

I should also point out, for the record, that the above success rate (2 out of 9) is atypical -- usually hdemux seems to do better for me than anything else (including TyTool). That it can easily be used in a batch fashion is the icing on the cake, here.

By pre-chopping I'm thinking of tychopper (windows) or dd (unix)
I'm not familar with ac3fix, it probably would be easier for me to just look at a sample .ty.
Understood. I grabbed tychopper and will play with it today.

Right, he's got his own private debug messages that are only going to be fully meaningful to him. Lacking source, the rest of our hands are tied when tytool barfs.
Yep, that's what I figured. I'll play with tychopper and get some samples together for you.

Thanks again!

bcc
06-19-2006, 08:31 PM
Agreed -- but in fact I'm not going to be either archiving or editing these.

My son is a long-distance trucker. I'm using (or would like to use) EtiVo as an automated mechanism to allow him and his fiancee (she's on the truck too) to pull down their favorite shows. Most of them I just record on a non-HD Tivo, but some I do in HD because we watch them as well.If I was place-shifting HD tivo content, I'd probably just watch it via xvid compressed mpeg4 on an HTPC
I grabbed tychopper and will play with it today.
Thanks again!Great, let me know how it works out.

cheer
06-19-2006, 09:04 PM
If I was place-shifting HD tivo content, I'd probably just watch it via xvid compressed mpeg4 on an HTPC
Well, yeah, that's sort of the idea, except replace HTPC with a laptop. But to get Xvid I need to get .MPG first. :) (At least I don't think I can go .ty -> Xvid...)

Great, let me know how it works out.
Father's Day intervened. And of course work has piled up on me. But tomorrow I should be able to chop 'em up.

bcc
06-19-2006, 11:12 PM
Well, yeah, that's sort of the idea, except replace HTPC with a laptop. But to get Xvid I need to get .MPG first. :) (At least I don't think I can go .ty -> Xvid...)Right; I had mpg->ty on my brain when I mentioned that, due to ffmpeg hacking.

bcc
06-28-2006, 01:50 AM
Here's a new release of hdemux. Changes include
Allow video sequences to be unbounded in length (fixes "Cannot fit Video buffer" error).
No more cosmetic "Skipping chunk .. due to timestamp discontinuity" errors at end of file
I've fixed a bug in ac3 audio packet processing when audio packet lengths oscillate, and don't match the frame length (Fox TV). (fixes "Tossing AC3 audio frame with bad sync data." errors).

Binary release only for now.

bcc
06-28-2006, 01:53 AM
Father's Day intervened. And of course work has piled up on me. But tomorrow I should be able to chop 'em up.I made a bunch more OTA recordings of my own, and I think I found the 3 main errors I mentioned. Should be fixed in the 0.15 image I just posted.

lennier
07-02-2006, 02:05 AM
Think you could see your way clear to posting source? I'm working on my own homebrew archiving system (on an OS I wouldn't expect any of you guys to support) and would like to take a stab at it, as I can't get tytools to work without barfing.

bcc
07-07-2006, 04:01 PM
Think you could see your way clear to posting source? I'm working on my own homebrew archiving system (on an OS I wouldn't expect any of you guys to support) and would like to take a stab at it, as I can't get tytools to work without barfing.I'm not inclined to.
I've integrated hdemux with my own multiplexer that I'm not ready to give out source for.
Also hdemux hasn't really been working as an open source project in that it's simply meant extra support for me, with virtually no source contributions back. This makes it not worth the effort for me to maintain source releases.

tivoq
08-03-2006, 01:15 PM
I'm trying to convert a 9GB ty file to mpg on windows xp. The hdemux goes without trouble, but the mplex seems to wrap around at the 2GB mark. The output file never gets bigger than 2GB and the beginning gets overwrriten.

Any ideas?

cheer
08-03-2006, 01:19 PM
Is your partition NTFS or FAT32?

kmt
08-04-2006, 12:44 AM
Is your partition NTFS or FAT32?

Neither, it was a remote filesystem on a Unix host. Oddly hdemux had no trouble writing the 9GB m2v file, but mplex wouldn't go over 2GB for the mpg file. I guess it has something to do with how the cygwin dll deals with the writes.

I went ahead and made mplex do the actual write to a local ntfs and the problem went away.

--
I can add one more issue that occured. The resulting mpg had little skips in it particularly noticeable on the audio when I used my usual player Windvd. I generally multiplex with tytool and play with Windvd. This ty file caused tytool to abort, which is why I used hdemux.

Since the skipping was a problem with Windvd, I used vlc to play the same mpg and it didn't have the skipping. So using vlc was a sufficent work around for this movie, but I still wonder why I got the skipping in hdemux->mplex->windvd and not hdemux->mplex->vlc.

wbelhaven
09-06-2006, 02:19 PM
Hi,

I'm having problems similar to those reported by cheer (http://www.dealdatabase.com/forum/showthread.php?p=258190#post258190) muxing a pretty horrible HD .ty file with hdemux/mplex that TyTool also fails to mux. I've narrowed the troublesome section down to a single FSID using captain_video's method (http://www.dealdatabase.com/forum/showthread.php?p=254352#post254352). With respect to that FSID, TyTool mux's ~263.7 MiB (http://en.wikipedia.org/wiki/MiB) of the 512 MiB total, but mplex gives up after only ~15.7 MiB.

I get several warnings of the following form when running this version (http://www.dealdatabase.com/forum/showthread.php?p=258792#post258792) of hdemux:



D:\>hdemux -i input.ty -v video.m2v -a audio.ac3
.
.
Warning: Attempting resync due to repeated time discontinuity, at chunk 348
Warning: Skipping chunk 369 due to timestamp discontinuity
Warning: Skipping chunk 370 due to timestamp discontinuity

And I get the following error when running this version (http://www.dealdatabase.com/forum/showpost.php?p=192328&postcount=52) of mplex:



D:\>mplex -f 8 -O 440ms -r 153701 audio.ac3 video.m2v -o muxed.mpg
.
.
INFO: [mplex] Video e0: buf=1900544 frame=000000 sector=00000000
**ERROR: [mplex] Can't find next AC3 frame: @ 531456 we have aef1 - broken bit-stream?

As I mentioned, mplex only makes it ~15.7 MiB before bombing. When viewing the resulting muxed.mpg in mplayer, it has tons of mpeg blocking artifacts (i.e., looks like a poor satellite reception with tons of errors), but the TyTool muxed output looks fine ... until it abruptly ends, of course. By the way, when I "jump" far enough into this FSID before muxing in TyTool, it produces output, but that output contains NO audio.

Any idea what can be done to fix this beast? Which version of hdemux and mplex should I be using?

Thanks,
WB

phat_bastard
09-15-2006, 11:28 AM
First off, thanks for all the work you put into this hobby. I wish there were some way I could compensate you but alas, we've touched on that before. Anyway here's my 'trouble', a word which I'm sure you're sick of hearing me repeat. Sorry...

Please forgive any errors in my terminology - I hope this explains my problem well enough. I've been using hdemux 0.15 to split streams for a tool I put together to do automatic xvid transcodes of some SD dtivo material. The problem I'm running into on the streams I've tested (that incidentally contain 112kbps mono audio) appears to be related to hdemux's reporting of the audio offset. It looks like it's giving me the correct absolute offset, but after muxing using the reported offset as an audio delay and a video delay and then watching it's evident that the delay should favor video in this instance (or as dgindex reports it, a negative audio delay). Could this be an issue with your calculation of the offset or is it more likely a result of the source I'm feeding it?


...I've integrated hdemux with my own multiplexer that I'm not ready to give out source for.

Hmmm, sounds interesting. How's that coming along?

cheer
09-15-2006, 11:36 AM
I'd guess it's the source, as I've used hdemux to transcode craploads of SD and HD material...I pull the offset from the hdemux output and use it when I mux. Have yet to have an offset or sync issue.

lgkahn
09-15-2006, 01:27 PM
I have been unable to find a demuxer for files that were created with ffmpeg and mplex...

both vsplit9r16 and hdemux crash

here is the hdemux error.. any ides..

video bit rate is 104857200 bits/sec., 104857 Kbit/sec.
MPEG2 audio at 32kHz, 864 bytes/sync frame
assertion failed: pts != -1, file demux.c, line 567

his application has requested the Runtime to terminate it in an unusual way.
lease contact the application's support team for more information.

:\tivo-store>

phat_bastard
09-15-2006, 02:10 PM
I'd guess it's the source, as I've used hdemux to transcode craploads of SD and HD material...I pull the offset from the hdemux output and use it when I mux. Have yet to have an offset or sync issue.

If you're bored, grab something off of CNN (SD) and report back your findings - I'd be interested to see if might be related to the low bitrate mono audio.

At any rate I'd still like to see bcc's comments on the subject.

cheer
09-18-2006, 11:12 PM
bcc:

Here's a tiny edit out of an OTA recording that's giving me grief. Hdemux seems to go OK, but during mplex I get:
**ERROR: [mplex] Sequence split detected 3 but no following sequence found...Let me know if I've chopped this up too tiny -- took some trial and error to get piece that bombed as mplex isn't real good about telling me where it died. :)

Cheezmo
09-21-2006, 08:31 PM
I have one from an HD-Tivo 6.3 ota extraction that is similar. TyTool crashes trying to mutiplex it, hdemux seems to go fine, but then mplex reports:

"Can't find next AC3 frame: @ 7680 we have 0000 - broken bitstream?"

I can make the head of the ty available if needed.

bcc
09-21-2006, 09:11 PM
I'm still catching up on old DDB messages...
Hi,

I'm having problems similar to those reported by cheer (http://www.dealdatabase.com/forum/showthread.php?p=258190#post258190) muxing a pretty horrible HD .ty file with hdemux/mplex that TyTool also fails to mux. I've narrowed the troublesome section down to a single FSID using captain_video's method (http://www.dealdatabase.com/forum/showthread.php?p=254352#post254352). With respect to that FSID, TyTool mux's ~263.7 MiB (http://en.wikipedia.org/wiki/MiB) of the 512 MiB total, but mplex gives up after only ~15.7 MiB.From your hdemux output, hdemux is not finding a problem until 348 chunks into the recording, or 43MB into the recording. At that point the audio stream doesn't parse right, and so it's no surprise that mplex fails on the audio stream as well. I think it'd make sense to try and verify the correctness of the elementary a/v streams before moving onto a multiplexer (mplex or ffmpeg). You should be able to play back the .m2v with mplayer, xine or elecard. For the .m2a/.ac3, xine should be able to play it back. A hex dump of the .ac3 stream at the problem point may be in order.
**ERROR: [mplex] Can't find next AC3 frame: @ 531456 we have aef1 - broken bit-stream?It may simply be that your source recording really does have a broken audio stream. Theoretically I could try to catch that sort of situation and fill in audio holes; I believe that is what tytool tries to do.

By the way, when I "jump" far enough into this FSID before muxing in TyTool, it produces output, but that output contains NO audio.I assume by it you mean mplayer? If you chop off the problem section of chunks using dd or tychopper, I suspect you could verify whether or not the remaining chunks demux OK with hdemux.
Any idea what can be done to fix this beast? Which version of hdemux and mplex should I be using?For hdemux, the latest should be the best. You could narrow down the problem to just a few chunks and send them in. For mplex, a 1.6.2 based version is probably still best. I assume your problem recording is an HD OTA one. If so, you'll want a version of mplex with my HD patch. If low def., a vanilla version of mplex.

bcc
09-21-2006, 09:31 PM
First off, thanks for all the work you put into this hobby. I wish there were some way I could compensate you but alas, we've touched on that before. Anyway here's my 'trouble', a word which I'm sure you're sick of hearing me repeat. Sorry...

Please forgive any errors in my terminology - I hope this explains my problem well enough. I've been using hdemux 0.15 to split streams for a tool I put together to do automatic xvid transcodes of some SD dtivo material. The problem I'm running into on the streams I've tested (that incidentally contain 112kbps mono audio) appears to be related to hdemux's reporting of the audio offset. It looks like it's giving me the correct absolute offset, but after muxing using the reported offset as an audio delay and a video delay and then watching it's evident that the delay should favor video in this instance (or as dgindex reports it, a negative audio delay). Could this be an issue with your calculation of the offset or is it more likely a result of the source I'm feeding it?
Sounds like TNT movies? For the a/v offset, I'm basing it upon the PTS values for the elemental streams that the shows are broadcast with. Whether or not the audio stream is mono or stereo wouldn't make a difference as far as I know. So I'm guessing there is a problem with the source. Perhaps the audio stream is not starting with PTS values? You could take a closer look at the delta between the audio and video stream PTS values and see if it makes sense to you.
Hmmm, sounds interesting. How's that coming along?It got to the "works for me"(tm) state, but I stopped work on it a year or so ago. At that point I figured out that ffmpeg's multiplexer didn't seem as hopeless as mplex's, and since there was an open source multiplexer that is good enough, I didn't need to do more. So I've been thinking one should really test hdemux+ffmpeg and assuming that can be made to produce good results for all recordings, then hdemux could be made into a codec for ffmpeg.

bcc
09-21-2006, 09:45 PM
I have been unable to find a demuxer for files that were created with ffmpeg and mplex...

both vsplit9r16 and hdemux crash

here is the hdemux error.. any ides..I do not have this problem. I assume you're generating a .ty file with ffmpeg. Are you using -vcodec copy, -acodec copy? Try muxing just the audio or video stream to narrow down which of the two is causing problems.

As for vsplit, ha, that tool doesn't really know how to parse .ty correctly, so it's suprising it works on anything these days.

bcc
09-22-2006, 02:51 AM
If you're bored, grab something off of CNN (SD) and report back your findings - I'd be interested to see if might be related to the low bitrate mono audio.

At any rate I'd still like to see bcc's comments on the subject.OK, I checked CNN HLN (SD), and it did have mono audio. When I multiplexed with mplex, mplex reported SCR underruns. Playback with xine had audio hiccups but a/v appeared in sync. Playback with mplayer resulted in bad a/v sync. I think this shows how xine is more accurate about enforcing a/v presentation times.

Then I multiplexed with ffmpeg instead, and the a/v sync was fine.

Looks like mplex is screwing up here.

bcc
09-22-2006, 03:16 AM
bcc:

Here's a tiny edit out of an OTA recording that's giving me grief. Hdemux seems to go OK, but during mplex I get:
**ERROR: [mplex] Sequence split detected 3 but no following sequence found...Let me know if I've chopped this up too tiny -- took some trial and error to get piece that bombed as mplex isn't real good about telling me where it died. :)So mplex is complaining that you have a video sequence that ends with a B frame, which seems like a legit gripe to me. (It also matches what I see in your stream). Probably videoredo can clean up your video stream for you.
Also I'd be curious if ffmpeg handles this case.

bcc
09-22-2006, 03:19 AM
I have one from an HD-Tivo 6.3 ota extraction that is similar. TyTool crashes trying to mutiplex it, hdemux seems to go fine, but then mplex reports:

"Can't find next AC3 frame: @ 7680 we have 0000 - broken bitstream?"

I can make the head of the ty available if needed.Looks like the audio stream is broken right at the beginning. This is with 0.15 hdemux? The first few chunks from the .ty recording should reveal what's going on.

cheer
09-22-2006, 08:49 AM
So mplex is complaining that you have a video sequence that ends with a B frame, which seems like a legit gripe to me. (It also matches what I see in your stream). Probably videoredo can clean up your video stream for you.
Also I'd be curious if ffmpeg handles this case.
I'll give ffmpeg a try. Not sure how I'd use VideoReDo as VRD won't take an elementary stream.

Cheezmo
09-22-2006, 09:02 AM
Looks like the audio stream is broken right at the beginning. This is with 0.15 hdemux? The first few chunks from the .ty recording should reveal what's going on.

Let me know if this isn't enough.

bcc
09-22-2006, 12:41 PM
I'll give ffmpeg a try. Not sure how I'd use VideoReDo as VRD won't take an elementary stream.I didn't realize that. I was just thinking that since videoredo can rebuild GOPs, it would be able to clean up the video stream. A bit of a catch 22. How about ffmpeg then? Your sample was too small for ffmpeg to do anything with.

bcc
09-22-2006, 07:15 PM
Let me know if this isn't enough.OK, in the case where the OTA stream has multiple AC3 frames per PES packet, I was not allowing for extra alignment padding of the TY packet. Fix is coming right up...

bcc
09-22-2006, 07:55 PM
Here's a new release of hdemux. Changes:
Fix PES audio payload size computation (allow padding) in case where PES payload < TY payload and payload includes multiple ac3 frames per PES header

Cheezmo
09-22-2006, 08:37 PM
Could you check the hdemux.exe in that rar? I can't get it to execute.

cheer
09-22-2006, 08:45 PM
The Windows binary works for me. On the other hand, it says version is still 0.15 and seems to be the exact same size as my old 0.15 binary...

Cheezmo
09-22-2006, 08:49 PM
Works for me too, but as you said it says it is .15 which was the source of my confusion I guess. mplex is chugging away now so it looks good.

bcc
09-22-2006, 08:50 PM
The Windows binary works for me. On the other hand, it says version is still 0.15 and seems to be the exact same size as my old 0.15 binary...I tried it and it works for me. Yes, my version.h didn't get updated so it unfortunately reports the same version. The windows binaries are the same size but the contents are different.
I wonder why DDB still says "0 views" for both rar files...

Cheezmo
09-22-2006, 08:52 PM
What are the chances of getting a Mac (Universal) binary? I used to compile it myself but see you aren't providing the source any more.

Cheezmo
09-22-2006, 09:21 PM
Well, it processed it just fine. I use MPEGStreamClip on the Mac to edit and when I select a section of the resulting .mpg, and try to exit it reports "Error: can't read the frame size. If I convert the whole file it works fine. Is there an mplex option that might generate mpg's with the frame size info it is looking for embedded throughout? I tried -h, but that may be one of the options that -f 8 won't let me override.

bcc
09-22-2006, 09:24 PM
What are the chances of getting a Mac (Universal) binary? I used to compile it myself but see you aren't providing the source any more.Know of a good cross compiler for Macs? Maybe this? http://ranger.befunk.com/fink/darwin-cross/
How about: I'll trade the source for an s3 software exploit :)

Cheezmo
09-22-2006, 09:51 PM
Looks like the "can't read the frame size" problem is not related to mplex since it happens working with the demuxed video. I'll try a pre 6.3 file and see if it is something that changed in the tivo datastream or something with hdemux all along.

cheer
09-22-2006, 10:32 PM
I didn't realize that. I was just thinking that since videoredo can rebuild GOPs, it would be able to clean up the video stream. A bit of a catch 22. How about ffmpeg then? Your sample was too small for ffmpeg to do anything with.
Re-demuxed with your latest binary, then muxed w/ffmpeg. It goes all the way through but the resulting file is unplayable -- just garbage. Brought it into VideoReDo and VRD couldn't seek and suggested quickstreamfix. Ran quickstreamfix and got a file about half the size with constant skipping.

bcc
09-23-2006, 01:55 AM
Re-demuxed with your latest binary, then muxed w/ffmpeg. It goes all the way through but the resulting file is unplayable -- just garbage. Brought it into VideoReDo and VRD couldn't seek and suggested quickstreamfix. Ran quickstreamfix and got a file about half the size with constant skipping.After your demux the .m2v, can you play it back with xine or elecard? Sounds like you're going to need to re-encode the video stream with tmpgenc or the like, as the GOPs are too wacked out.

bcc
09-23-2006, 02:18 AM
Well, it processed it just fine. I use MPEGStreamClip on the Mac to edit and when I select a section of the resulting .mpg, and try to exit it reports "Error: can't read the frame size. If I convert the whole file it works fine. Is there an mplex option that might generate mpg's with the frame size info it is looking for embedded throughout? I tried -h, but that may be one of the options that -f 8 won't let me override."can't read the frame size" for which stream, the audio or the video? I assume your mac is complaining that the sequence_header() is not occuring often enough in the video stream. This is typical of streams on the tivo, if I remember right. I suppose the demuxer or muxer could insert extra copies tho I'm not aware of ready made code that does that. (Except for nova1's code in tymplex.)

bcc
09-23-2006, 02:27 AM
I tried -h, but that may be one of the options that -f 8 won't let me override.Oh right, but then mplex clears the -h for -f 8 and -f 9 (see the always_sys_header_in_pack = 0 statements in multiplexor.cpp).
You could easily patch mplex to make it include redundant copies of that header.

Cheezmo
09-23-2006, 10:03 AM
For the video. I tried -h in mplex with a generic mpeg profile and it still didn't help. FWIW, (can I mention it) TyTool generated files never had this problem but since it totally craps out on files from 6.3 I don't know if this has always been a problem (when using hdemux) or if it is new.


"can't read the frame size" for which stream, the audio or the video? I assume your mac is complaining that the sequence_header() is not occuring often enough in the video stream. This is typical of streams on the tivo, if I remember right. I suppose the demuxer or muxer could insert extra copies tho I'm not aware of ready made code that does that. (Except for nova1's code in tymplex.)

bcc
09-23-2006, 10:21 PM
I think I was confused in what I wrote last night. I thought you were having a problem with one of the video stream headers. But then you mentioned the -h option to mplex, which is just meant to control the rate of generation of the system_header() info, start code 0x000001bb (tho that switch does not do anything when used with -f 8/9). Maybe we could back up... What does the "can't read the frame size" error you are seeing mean? It doesn't make sense to me that your application would require this start code in every packet.

For the video. I tried -h in mplex with a generic mpeg profile and it still didn't help. FWIW, (can I mention it) TyTool generated files never had this problem but since it totally craps out on files from 6.3 I don't know if this has always been a problem (when using hdemux) or if it is new.I don't mind mention of tytool :) But it doesn't help me much as I don't use it and any diagnostic messages it generates are underdocumented.

As for 6.3 vs older .ty files, the only difference I've seen is that as of 6.3, the audio&video streams have independent tivo timestamps. This means the tivo timestamp is no longer monotonic between the audio & video records. This has not posed a problem for hdemux in any of my tests. To be clear, I'm not experiencing any problems extracting and demuxing 6.3 .ty.

Cheezmo
09-24-2006, 02:28 PM
I just noticed there is a Windows version of MPEG Streamclip...

http://www.squared5.com/svideo/mpeg-streamclip-win.html

Perhaps you could play with it a little and see if you can figure out what it is looking for. I could give you an MPEG sample generated by tytool that it likes and you could compare it to what hdemux generates that it doesn't.

cheer
09-25-2006, 09:35 AM
After your demux the .m2v, can you play it back with xine or elecard? Sounds like you're going to need to re-encode the video stream with tmpgenc or the like, as the GOPs are too wacked out.
OK, been playing with several streams.

As near as I can tell, hdemux is doing its job -- at least, it isn't throwing errors. Remuxing is what is killing me. Mplex barfs, usually immediately, so I am experimenting with ffmpeg (both the ty-enabled one and the latest Windows binary I could find). I'm a ffmpeg newb, so maybe I'm botching things, but I try something like this:
ffmpeg -i frog.m2v -itsoffset 0.766 -i frog.ac3 -acodec copy -vcodec copy frog.mpg(The offset came from hdemux, which gave an offset of 766ms on this particular stream.)

The audio sounds fine and the video looks clear and smooth -- so this is a big improvement over other things I've tried. However, a/v sync is utterly hosed -- the video is behind the audio by several seconds. Also, the aspect ratio looks wrong (video is 720x480 4:3 but both media player classic and zoom player play it as stretched out); I tried adding "-aspect 4:3" to the command line but it didn't make any difference. Note that the elementary stream plays normally (not stretched).

Here's something curious (and maybe I mentioned this before; I've lost track of what I've said to whom) -- if I just put the .m2v and .ac3 files in the same folder and play the .m2v w/media player classic, it auto-finds the .ac3 file and plays both -- in perfect sync/aspect.

Like I said, I suspect this is all me learning how to use ffmpeg, but the audio sync bothers me a lot since I infer that the offset output from hdemux won't mean much when I remux w/ffmpeg.

bcc
09-25-2006, 12:22 PM
Looks like you got the a/v arguments backwards. Order matters with ffmpeg. If the a/v offset reported by hdemux is positive, then you'll want to specify the audio stream before the itsoffset argument. Otherwise specify the video stream first. I've made the next hdemux report the arguments for you:
% hdemux -i nbr.ty -v n.m2v -a n.m2a
Version 0.17
Source is nbr.ty
AC3 audio at 48kHz, 768 bytes/sync frame
Recovered AC3 sync, offset 336, chunk 1
Video frame size is 704x480 Frame rate 29.970030
Video bit rate is 10000000 bits/sec., 10000 Kbit/sec.
3387808 audio stream bytes 42257516 video stream bytes
Reported detonate 10192 Kbit/sec. (10000+192)
Proceed with remuxer:
mplex -f 8 -O 1319ms -r 15288 n.m2a n.m2v -o <outfile>.mpg
OR
ffmpeg -acodec copy -vcodec copy -f ac3 -i n.m2a -itsoffset 1.3193 -i n.m2v <outfile>.vob
%
Note that because the audio stream was ac3, I'm using -f ac3 (instead of renaming the .m2a file to .ac3, tho either would work).

bcc
09-25-2006, 12:23 PM
Also note that I specified '.vob' instead of .mpg for the output file with ffmpeg. If you specify .mpg, ffmpeg will default to mpeg1 not mpeg2. With .vob you get mpeg2. That may explain some of your remaining problems.

cheer
09-26-2006, 10:48 AM
Also note that I specified '.vob' instead of .mpg for the output file with ffmpeg. If you specify .mpg, ffmpeg will default to mpeg1 not mpeg2. With .vob you get mpeg2. That may explain some of your remaining problems.
OK, did a bunch more experiments using multiple recordings, and still no joy.

First, I tried the .vob suggestion, but although the aspect ratio seems to be correct, MPC shows NO video and PowerDVD skips and stutters (I'm guessing 'cos the video isn't DVD-compliant -- wacky GOPs and stuff). I will try pulling one into VideoReDo but I'm not optimistic. (Given the skip/stutter I couldn't test a/v sync.)

So I tried again with .mpg (and btw would I really end up with mpeg1, given that I'm not encoding?). Same results: aspect ratio off (minor) and a/v sync off horribly. The order I use in the ffmpeg cmdline really doesn't enter into it, as we're talking about a reported offset of under a second per hdemux, and the sync is off by anywhere from 15 to 30 SECONDS.

Gonna try some more recordings -- I think everything I've done so far has been OTA and I want to see how a sat-based recording does again.

Is there a way to get ffmpeg to output some more detailed logging or diagnostic info?

Oh, just for reference, one of my command lines:
ffmpeg -acodec copy -vcodec copy -i s2e22.ac3 -itsoffset 0.449 -i s2e22.m2v s2e22.mpg

For what it's worth, I tried a muxer I dug up on doom9 called "MPEG Multiplexer 1.0." Windows GUI, so not my first choice, but I'm trying to eliminate things. It spit out the correct aspect ratio, but still had the horrible a/v sync.

bcc
09-26-2006, 01:21 PM
If your a/v is off by 15-30 seconds, then it shouldn't be hard to see where things are going wrong. Let me guess, the video is playing 15-30 seconds early? (As if there is 15-30 seconds of video missing?) You could narrow it down by verifying whether the individual demuxed streams have the "missing" 15-30 seconds. Then you could try muxing just the individual streams and see whether or not you get the missing seconds back.

I would be suspect of your codecs as well. Might want to make sure you're using known good ones with graphedit/gspot.

As for .vob vs .mpg, yes if you're using -vcodec copy the stream itself wouldn't be re-encoded but with .mpg you're getting V1 headers. Look:
% ffmpeg -acodec copy -vcodec copy -f ac3 -i n.m2a -itsoffset 1.3193 -i n.m2v n.mpg
...
Input #0, ac3, from 'n.m2a':
Duration: 00:02:21.1, start: 0.000000, bitrate: 191 kb/s
Stream #0.0: Audio: ac3, 48000 Hz, stereo, 192 kb/s
Input #1, mpegvideo, from 'n.m2v':
Duration: 00:00:33.8, start: 0.000000, bitrate: 10001 kb/s
Stream #1.0: Video: mpeg2video, yuv420p, 704x480, 10000 kb/s, 29.97 fps(r)
Output #0, mpeg, to 'n.mpg':
Stream #0.0: Video: mpeg2video, yuv420p, 704x480, q=2-31, 10000 kb/s, 29.97 fps(c)
Stream #0.1: Audio: ac3, 48000 Hz, stereo, 192 kb/s
Stream mapping:
Stream #1.0 -> #0.0
Stream #0.0 -> #0.1
Press [q] to stop encoding
frame= 4221 q=0.1 Lsize= 44808kB time=141.2 bitrate=2600.5kbits/s
video:41267kB audio:3308kB global headers:0kB muxing overhead 0.521913%
% file n.mpg
n.mpg: MPEG sequence, v1, system multiplex
% ffmpeg -acodec copy -vcodec copy -f ac3 -i n.m2a -itsoffset 1.3193 -i n.m2v n.vob
...
Input #0, ac3, from 'n.m2a':
Duration: 00:02:21.1, start: 0.000000, bitrate: 191 kb/s
Stream #0.0: Audio: ac3, 48000 Hz, stereo, 192 kb/s
Input #1, mpegvideo, from 'n.m2v':
Duration: 00:00:33.8, start: 0.000000, bitrate: 10001 kb/s
Stream #1.0: Video: mpeg2video, yuv420p, 704x480, 10000 kb/s, 29.97 fps(r)
Output #0, vob, to 'n.vob':
Stream #0.0: Video: mpeg2video, yuv420p, 704x480, q=2-31, 10000 kb/s, 29.97 fps(c)
Stream #0.1: Audio: ac3, 48000 Hz, stereo, 192 kb/s
Stream mapping:
Stream #1.0 -> #0.0
Stream #0.0 -> #0.1
Press [q] to stop encoding
frame= 4221 q=0.1 Lsize= 45144kB time=141.2 bitrate=2620.0kbits/s
video:41267kB audio:3308kB global headers:0kB muxing overhead 1.275693%
% file n.vob
n.vob: MPEG sequence, v2, program multiplex
%

bcc
09-26-2006, 01:41 PM
Oh, and I assume you have at least a 3ghz P4 or equivalent cpu power for software playback of 1080i HD. Also, for HD, with radeon or geforce based video cards, my experience is that windvd's codecs use significantly less cpu than powerdvd's.

cheer
09-26-2006, 03:36 PM
Not doing 1080i with these -- these are just SD OTA 720x480 w/DD2.0 audio. I do see what you mean re: MPG1 headers which would likely explain my weird aspect ratio stuff, etc.

I highly doubt it's a CPU issue (though I do have a P4 3.2 Ghz) or a codec issue -- because until the 6.3 update rolled around I had none of these problems. No, let me clarify that: occasionally an OTA recording would refuse to multiplex with mplex and on the one time I did try to use ffmpeg I again ended up with a bad a/v sync issue, but I didn't troubleshoot it much and attributed the problem to my ffmpeg inexperience.

Up until the 6.3 rollout, this was rare - 95% of the time either hdemux/mplex or TyTool worked great. (The little bad stream I posted back on 9/18 is an example of something that broke both.) Now, TyTool seems dead in the water (I'm guessing this is related to your comment above that the audio and video now have independent timestamps), and I've so far been unsuccessful in muxing anything via ffmpeg or mplex. I can certainly investigate the codec issue, but it seems rather unlikely.

Actually, the audio is way ahead of the video. Not really sure how to tell if the 30 seconds are "missing" -- it's hard to tell as the video starts with the closing credits of a prior show, then commercials, etc...it takes a couple of minutes before there's dialog spoken by someone onscreen, and at that point it's already well off.

bcc
09-26-2006, 09:01 PM
I've so far been unsuccessful in muxing anything via ffmpeg or mplex. I can certainly investigate the codec issue, but it seems rather unlikely.So far I've been unable to reproduce what you're talking about with any recordings.
Actually, the audio is way ahead of the video. Not really sure how to tell if the 30 seconds are "missing" -- it's hard to tell as the video starts with the closing credits of a prior show, then commercials, etc...it takes a couple of minutes before there's dialog spoken by someone onscreen, and at that point it's already well off.

You should have mentioned before that your problem case was a recording where the a/v sync problem is only measurable after a show change. At the show change point, the OTA station may very well be warping the timestamps and there may easily be other discontinuities in the streams.
Handling these transitions wasn't a design goal for hdemux as one wouldn't normally want to archive a recording that included these discontinuities.

I suggest you try tychopper or dd to skip the previous show and commercials and then check the a/v sync when you demux&remux just the second show. If this is inconvenient, and the problem is reproducible by you, then you could just get a recording of some talking heads. That should be handy for closely examining the a/v sync.

mr_zorg
09-30-2006, 03:12 AM
Know of a good cross compiler for Macs? Maybe this? http://ranger.befunk.com/fink/darwin-cross/
How about: I'll trade the source for an s3 software exploit :)
I second the call for the source... I used to compile it myself too, but can't anymore. I'll be happy to compile it for you and then delete the source if you really don't want to distribute the source. Why the change of heart?

bcc
09-30-2006, 08:16 PM
I've integrated hdemux with my own multiplexer that I'm not ready to give out source for. A professional quality multiplexer is not easy code to write, and I'm not convinced it makes sense to give away such work. Yes I could cleanly stub out the (re)multiplexer portion. That goes to the next point about extra work.
Open source is a two way street. Source level releases of hdemux are extra work for me. Build questions, cosmetic source code issues, etc. distract from the project goals, and are only of interest to a minority. Most tivo users simply want to run binaries, for which I've pre-built the most common formats.
For the project to be a successful two way street, there must be contributions from others. After over 2 years, developer contributions have been almost non-existent. As for non-development contributions, it's often difficult just to get basic technical troubleshooting information from users here. (Is it so hard to provide stream samples to reproduce problems? Sorry, I'm getting off track...)
hdemux technology has been used to advance other projects, still without contributions back to the core, or clear attribution of credit. These other projects actually serve to impede hdemux somewhat by drawing testing resources elsewhere. These other projects even solicit contributions, in that sense capitalizing on my work that was intended to be free.
Synergy with reverse engineering efforts. This project stands on the shoulders of other core tools, such as mfs-utils, mfs_ftp, tivoapp and prom patches. To the extent that advancements to these core tools are withheld, my project suffers. For example, hdemux can be a good solution for demuxing music channels, but I cannot test that support, as a tivoapp patch has not been made available for the hr10-250. I could re-disassemble tivoapp myself but it'd be more productive for me to work on higher level tools. "You scratch my back, I scratch yours". I know many have argued that releasing of reverse engineering efforts is at odds with making sure that holes don't get plugged. But if those results are kept from even those who contribute then we all lose.

It all comes down to one thing: collaboration. Enough positive results in that area and I can be convinced to change back.

mr_zorg
10-01-2006, 12:22 AM
For the project to be a successful two way street, there must be contributions from others. After over 2 years, developer contributions have been almost non-existent. As for non-development contributions, it's often difficult just to get basic technical troubleshooting information from users here. (Is it so hard to provide stream samples to reproduce problems? Sorry, I'm getting off track...)
I sympathize with your frustrations. I'd love to help out, but working with MPEG streams seems like a black art to me. I've tried looking into it years ago, and found documentation severely lacking. If there's any other way I can help out (I am a senior software engineer), just let me know. I'm game. For example, I would be more than willing to compile Mac versions for you while keeping your source closed...

hdemux technology has been used to advance other projects, still without contributions back to the core, or clear attribution of credit. These other projects actually serve to impede hdemux somewhat by drawing testing resources elsewhere. These other projects even solicit contributions, in that sense capitalizing on my work that was intended to be free.
Dude, that sucks.


For example, hdemux can be a good solution for demuxing music channels, but I cannot test that support, as a tivoapp patch has not been made available for the hr10-250. I could re-disassemble tivoapp myself but it'd be more productive for me to work on higher level tools.
Hmm. I wasn't even aware of such a patch in the first place. Sounds like a challenge. Let me see what I can do. :)

bcc
10-02-2006, 04:21 PM
I sympathize with your frustrations. I'd love to help out, but working with MPEG streams seems like a black art to me. I've tried looking into it years ago, and found documentation severely lacking. If there's any other way I can help out (I am a senior software engineer), just let me know. I'm game. For example, I would be more than willing to compile Mac versions for you while keeping your source closed...Yes the iso mpeg specs are opaque, intentionally so from what I can tell. (They are also patent encumbered.) For mac versions, if you could just point me to a solid linux cross compiler setup... Also doesn't the windows binary work fine on macs under emulation?

Dude, that sucks.Just contributing factors

Hmm. I wasn't even aware of such a patch in the first place. Sounds like a challenge. Let me see what I can do. :)The music channel case is just a simple for-example, not an important case for me. I'll probably shut off directv service soon now that the s3 tivo is out, so I won't even be able to test satellite music channels at that point.

LtngStalker
10-02-2006, 11:05 PM
Thank you bcc for all of your hard work. I just want to let you know that hdemux worked flawlessly on my video files that were crashing TyTool.

I think the problem with the videos is that the time base was a bit screwed up because there were some times when the tape player was stopped or when there was no video on a section of the tape during the recording. I'll have to try out VideoReDo and see what happens.com

mr_zorg
10-03-2006, 05:48 PM
For mac versions, if you could just point me to a solid linux cross compiler setup...

I wish I could. I do know that the Apple dev tools use GCC. But what switches, etc., you need to cross-compile is beyond me. I do mostly Java work. :) (Hmm, actually, a Java port of hdemux would be awesome!)


Also doesn't the windows binary work fine on macs under emulation?

Probably... But I don't run such an emulator. :(


The music channel case is just a simple for-example, not an important case for me. I'll probably shut off directv service soon now that the s3 tivo is out, so I won't even be able to test satellite music channels at that point.

I've been thinking about doing the same thing... Very tempting.

cheer
10-04-2006, 08:06 AM
Sorry about taking so long to respond...real life, fighting with 6.3a, etc.

So far I've been unable to reproduce what you're talking about with any recordings.Understood...but again, given the timing of things, it's hard for me to believe it's a codec issue (on multiple PCs, yet) or whatever. More below.

You should have mentioned before that your problem case was a recording where the a/v sync problem is only measurable after a show change. At the show change point, the OTA station may very well be warping the timestamps and there may easily be other discontinuities in the streams.
Handling these transitions wasn't a design goal for hdemux as one wouldn't normally want to archive a recording that included these discontinuities.Er, no...I wouldn't want to archive such a recording either.

But I dunno how your Tivo works...on mine, it's extremely common to end up with a bit of the closing credits of the previous show or whatever, even without running endpadplus -- and to be honest, if I know I'm going to archive something, I'll usually pad both ends by a minute or two to make sure nothing gets clipped.

I suggest you try tychopper or dd to skip the previous show and commercials and then check the a/v sync when you demux&remux just the second show. If this is inconvenient, and the problem is reproducible by you, then you could just get a recording of some talking heads. That should be handy for closely examining the a/v sync.
Well...things are weirder, now.

For some reason that I cannot remember, I pulled another HD (OTA) recording and demuxed w/hdemux. I then tried to mux w/mplex for my control...and it muxed just fine. Sync is perfect, video is gorgeous, etc.

Excellent, I think to myself, now I can confirm or rule out ffmpeg issues. So I took the same elementary streams and muxed with ffmpeg...and I got a mess. I used the VOB output and it was unplayable, and VideoReDo found all sorts of errors.

So...for whatever reason, ffmpeg doesn't behave for me. At this point it pretty much has to be a system thing, but for the life of me I can't figure out what it could be.

Anyway. At this point I really can't draw too many conclusions, as it seems my sample size is still too small...but for now, my personal situation is that I can use hdemux so long as the resultant streams don't break the finicky mplex, but if not I don't really have an alternative until I figure out what's up with ffmpeg.

LtngStalker
10-04-2006, 06:09 PM
Hello. What command lines is everyone using for hdemux and mplex to make DVD video? I'm running into problems where the original ty file will play just fine on the PC (using TyShow), but the m2v outputted by hdemux and the subsequent mpeg produced by mplex have black video output on PowerDVD and Nero ShowTime, and cause an infinite "Connecting..." message in WMP. I'm thinking it could be a funky option I'm using in hdemux/mplex, so I'm wondering what the "proper" command lines should look like.

cheer
10-04-2006, 09:13 PM
Hdemux is very simple:
hdemux -i <infile>.ty -v <outfile>.m2v -a <outfile>.ac3
And mplex isn't much more complex...
mplex -f 8 -O ###ms -r @@@@@ <outfile>.ac3 <outfile>.m2v -o <outfile>.mpgWhere ### is the offset reported by hdemux and @@@@@ is the rate reported by hdemux. If that's all you're doing, you might check your MPG codec(s).

LtngStalker
10-04-2006, 09:40 PM
Thanks, cheer. hdemux was recommending a rate of 2493, but mplex complained that "data will arrive too late", so I bumped it up to 4000 and it went fine. It still gives a black screen (sound is normal) on Nero ShowTime and PowerDVD though which worries me because if it won't play on those that means it's probably not going to play on a DVD player either. Could it be because I used "basic quality" when I recorded it in the first place?

bcc
10-05-2006, 07:46 PM
Sorry about taking so long to respond...real life, fighting with 6.3a, etc.Yes 6.3 provides hours of reverse engineering fun, doesn't it?

But I dunno how your Tivo works...on mine, it's extremely common to end up with a bit of the closing credits of the previous show or whatever, even without running endpadplus -- and to be honest, if I know I'm going to archive something, I'll usually pad both ends by a minute or two to make sure nothing gets clipped.Sure, but I would pre-chop the recording at the chunk level before demuxing&remuxing. Also, when I mention discontinuities at transitions, that's not to say it's always a problem, just sometimes. Some discontinuities, typically from OTA stations, are bad enough that even the tivo loses the video for a few seconds. Those are the sort that you would certainly want to edit out before attempting a demux/remux. I can only speculate whether you're have a problem with this or not, as I lack example problem streams.

Excellent, I think to myself, now I can confirm or rule out ffmpeg issues. So I took the same elementary streams and muxed with ffmpeg...and I got a mess. I used the VOB output and it was unplayable, and VideoReDo found all sorts of errors.When you use ffmpeg, I hope you're using the version I built (the 1.4 version). For directv streams, it's important as a stock version won't deal with directv's habbit of encoding a variable number of fields per frame. Also, I've done some testing and have noticed that stock ffmpeg won't handle HD video stream rates (just as stock mplex does not - they both blow their virtual video buffers). The symptom would be errors from ffmpeg that look like:
[vob @ 0x83b62c8]packet too large, ignoring buffer limits to mux it
[vob @ 0x83b62c8]buffer underflowI have a fix. ffmpeg's mpeg2 multiplexer also has a subtle bug in its muxer that would cause it to be slightly off in its packet interleaving. This could theoretically lead to further under/over runs during playback. I have a fix for that as well. Guess it's time for a 1.5 release of my ffmpeg changes.

drez
10-05-2006, 08:12 PM
I have a fix. ffmpeg's mpeg2 multiplexer also has a subtle bug in its muxer that would cause it to be slightly off in its packet interleaving. This could theoretically lead to further under/over runs during playback. I have a fix for that as well. Guess it's time for a 1.5 release of my ffmpeg changes.

if you decide to do a 1.5 release, will it be built with the latest ffmpeg svn? (wmv3/wmv9 support would be great)


do you accept donations btw?

bcc
10-05-2006, 09:14 PM
if you decide to do a 1.5 release, will it be built with the latest ffmpeg svn? (wmv3/wmv9 support would be great)That's the one labeled vc1/wmv3 decoder? I wasn't planning on syncing, for stability sake but I suppose I could. On the other hand it doesn't even look like there is a config sscript option for wmv3 so it's probably still alpha, and you'd have to roll your own image.
do you accept donations btw?No, but I'm still interested in a software exploit for the s3, and MRV/HME/TTG hacks for 6.3 might be nice...

drez
10-05-2006, 10:31 PM
That's the one labeled vc1/wmv3 decoder? I wasn't planning on syncing, for stability sake but I suppose I could. On the other hand it doesn't even look like there is a config sscript option for wmv3 so it's probably still alpha, and you'd have to roll your own image.

yea, wmv3 is handled by the vc-1 decoder. wmv3 implements the Simple & Main profiles of the VC-1 codec standard (http://en.wikipedia.org/wiki/VC-1)

According to Kostya (main guy working on VC-1 for ffmpeg (http://svn.mplayerhq.hu/ffmpeg/trunk/libavcodec/vc1.c?view=log)), he's about 93% done with the VC-1 Simple and Main Profiles (http://codecs.multimedia.cx/?cat=8)


I've been using wmv3 decoding in the latest svn's (using compiled binaries from here, in case anybody else wants them (http://ffdshow.faireal.net/mirror/ffmpeg/)) and it's been pretty solid with all the video's I've tried. I convert them to mpeg2 with those binaries and then run do a stream copy with tyffmpeg. I think it's stable enough to be put in ty-ffmpeg.

94SupraTT
10-05-2006, 10:32 PM
Hdemux is very simple:
hdemux -i <infile>.ty -v <outfile>.m2v -a <outfile>.ac3
And mplex isn't much more complex...
mplex -f 8 -O ###ms -r @@@@@ <outfile>.ac3 <outfile>.m2v -o <outfile>.mpgWhere ### is the offset reported by hdemux and @@@@@ is the rate reported by hdemux. If that's all you're doing, you might check your MPG codec(s).

I got this for output.


Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Light>e:

E:\>hdemux -i Seahawks.ty -v Seahawks.m2v Seahawks.ac3
Version 0.15
Source is Seahawks.ty
Video frame size is 1920x1080 - high definition. Frame rate 29.970030
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
AC3 audio at 48kHz, 1536 bytes/sync frame
567482998 audio stream bytes 137655595 video stream bytes
Reported datarate 65384 Kbit/sec. (65000+384)
Proceed with remuxer:
mplex -f 8 -O 199ms -r 153701 (null) Seahawks.m2v -o <outfile>.mpg

E:\>

Is it normal to have no ac3 file? Isn't that the sound file.

bcc
10-05-2006, 10:51 PM
yea, wmv3 is handled by the vc-1 decoder. wmv3 implements theThanks, I hadn't noticed some of the new configure script changes that pull it in automatically.
I've synced my ffmpeg to the head now...

bcc
10-05-2006, 10:51 PM
I got this for output.



Is it normal to have no ac3 file? Isn't that the sound file.You missed the -a argument on the command line for hdemux

bcc
10-05-2006, 11:49 PM
Thanks, I hadn't noticed some of the new configure script changes that pull it in automatically.
I've synced my ffmpeg to the head now...Argh, the latest changes in fmpeg have bad frame rate detection code. 60fps mpeg2 is detected as 25fps, for example. Now I feel like a sucker for having synced...

cheer
10-06-2006, 12:40 AM
Yes 6.3 provides hours of reverse engineering fun, doesn't it?Good golly, Miss Molly.
Sure, but I would pre-chop the recording at the chunk level before demuxing&remuxing. Also, when I mention discontinuities at transitions, that's not to say it's always a problem, just sometimes. Some discontinuities, typically from OTA stations, are bad enough that even the tivo loses the video for a few seconds. Those are the sort that you would certainly want to edit out before attempting a demux/remux. I can only speculate whether you're have a problem with this or not, as I lack example problem streams.Right -- trying to experiment on my own as much as possible without wasting your time, at least until I do enough tests to conclusively demonstrate that a stream is problematic. Got a bunch planned for this weekend. I will try chopping out at the chunk level.
When you use ffmpeg, I hope you're using the version I built (the 1.4 version). For directv streams, it's important as a stock version won't deal with directv's habbit of encoding a variable number of fields per frame. Also, I've done some testing and have noticed that stock ffmpeg won't handle HD video stream rates (just as stock mplex does not - they both blow their virtual video buffers).I was at first, but then I think I replaced it with a recent build of the stock ffmpeg to try and rule things out. I will re-download your latest.

bcc
10-06-2006, 04:16 AM
Good golly, Miss Molly... I will re-download your latest.
For 6.3, I see lots of confusion, but mostly just folks needing to find and apply 4.x and 6.2 hacks to 6.3.

I've merged my HD mpeg2 tweaks into my ty-ffmpeg diffs, and made a new version, so my ffmpeg changes should be in order now.

cheer
10-06-2006, 12:52 PM
For 6.3, I see lots of confusion, but mostly just folks needing to find and apply 4.x and 6.2 hacks to 6.3.The dolby digital thing is killing me.
I've merged my HD mpeg2 tweaks into my ty-ffmpeg diffs, and made a new version, so my ffmpeg changes should be in order now.Well I dunno if it's your latest fixes or what...but now EVERYTHING is muxing fab for me, including the real problem stream I was having trouble with before.

The weird thing? It's muxing OK with mplex too, now.

I swear, nothing has changed, except that I re-downloaded your 0.16 hdemux binary (the one that still shows 0.15) and downloaded the latest ffmpeg binary (both from ddb of course). So either (A) I thought I was using the 0.16 hdemux binary before but wasn't, (B) something has changed on my PC without my knowledge that has "unbroken" things, or (C) it's all attributable to spurious anomalies.

I vote C.

Regardless...thanks very much for your patience. I'm going to test a bunch more streams to be sure, but at the moment things look awesome.

94SupraTT
10-06-2006, 01:48 PM
You missed the -a argument on the command line for hdemux

Thanks. :)

Now I am getting this when I try to mplex.



Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Light>e:

E:\>hdemux -i Seahawks.ty -v Seahawks.m2v -a Seahawks.ac3
Version 0.15
Source is Seahawks.ty
Video frame size is 1920x1080 - high definition. Frame rate 29.970030
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
AC3 audio at 48kHz, 1536 bytes/sync frame
567482998 audio stream bytes 137655595 video stream bytes
Reported datarate 65384 Kbit/sec. (65000+384)
Proceed with remuxer:
mplex -f 8 -O 199ms -r 153701 Seahawks.ac3 Seahawks.m2v -o <outfile>.mpg


E:\>mplex -f 8 -O 199ms -r 153701 Seahawks.ac3 Seahawks.m2v -o Seahawks.mpg
INFO: [mplex] mplex version 1.6.2 (2.2.3 $Date: 2004/01/13 20:45:26 $)
INFO: [mplex] File Seahawks.ac3 looks like an AC3 Audio stream.
INFO: [mplex] File Seahawks.m2v looks like an MPEG Video stream.
INFO: [mplex] Video stream 0: profile 8 selected - ignoring non-standard opti
ons!
INFO: [mplex] Found 1 audio streams and 1 video streams
INFO: [mplex] Selecting dvdauthor DVD output profile
INFO: [mplex] Multiplexing video program stream!
INFO: [mplex] Scanning for header info: AC3 Audio stream 00 (Seahawks.ac3)
INFO: [mplex] AC3 frame size = 1536

INFO: [mplex] AC3 AUDIO STREAM:
INFO: [mplex] Bit rate : 49152 bytes/sec (384 kbit/sec)
INFO: [mplex] Frequency : 48000 Hz
INFO: [mplex] Scanning for header info: Video stream e0 (Seahawks.m2v)
INFO: [mplex] VIDEO STREAM: e0
INFO: [mplex] Frame width : 1920
INFO: [mplex] Frame height : 1080
INFO: [mplex] Aspect ratio : 16:9 display
INFO: [mplex] Picture rate : 29.970 frames/sec
INFO: [mplex] Bit rate : 65000000 bits/sec
INFO: [mplex] Vbv buffer size : 999424 bytes
INFO: [mplex] CSPF : 0
INFO: [mplex] SYSTEMS/PROGRAM stream:
INFO: [mplex] rough-guess multiplexed stream data rate : 66735000
INFO: [mplex] target data-rate specified : 153701200
INFO: [mplex] Setting specified specified data rate: 153701200
INFO: [mplex] Run-in Sectors = 698 Video delay = 24606 Audio delay = 15705
INFO: [mplex] New sequence commences...
INFO: [mplex] Audio bd: buf= 16384 frame=000000 sector=00000000
INFO: [mplex] Video e0: buf=1900544 frame=000000 sector=00000000
**ERROR: [mplex] Can't find next AC3 frame: @ 8471040 we have 1e08 - broken bit-
stream?

bcc
10-06-2006, 03:09 PM
AC3 audio at 48kHz, 1536 bytes/sync frame
567482998 audio stream bytes 137655595 video stream bytesThis part dosn't look right, your audio stream should be smaller than your video stream. What are your file sizes on the .ty,.m2v, and .m2a? When I do the math on the audio stream I get a recording of 3hours, 12 minutes. That sound right? Audio error is at 2 minutes, 52 sec. Can you play back the .ac3 fine across this point? How about the source .ty (reception problem at this point)?

bcc
10-06-2006, 04:00 PM
Here's a new, basically cosmetic release of hdemux. Changes:
Program shows ffmpeg usage guidelines
Dust off documentation, describe ffmpeg usage
Remember to bump revision properly this time :)

Windows32&linux binaries provided. Unpack with winrar or the like.

cheer
10-06-2006, 04:34 PM
This part dosn't look right, your audio stream should be smaller than your video stream. What are your file sizes on the .ty,.m2v, and .m2a? When I do the math on the audio stream I get a recording of 3hours, 12 minutes. That sound right? Audio error is at 2 minutes, 52 sec. Can you play back the .ac3 fine across this point? How about the source .ty (reception problem at this point)?
There must be a digit or two dropped -- 138 megs of 1920x1080i video would be, well, not much at all. :)

Might be worth trying this through ffmpeg instead; this is the kind of bad .ac3 stream that has biffed in mplex for me every once in a while. (Haven't had one lately to try ffmpeg on, but...)

94SupraTT
10-06-2006, 05:40 PM
This part dosn't look right, your audio stream should be smaller than your video stream. What are your file sizes on the .ty,.m2v, and .m2a? When I do the math on the audio stream I get a recording of 3hours, 12 minutes. That sound right? Audio error is at 2 minutes, 52 sec. Can you play back the .ac3 fine across this point? How about the source .ty (reception problem at this point)?


I'm not sure why it is reporting that. The .ty is 22,229,811,200 bytes. The m2v is 21,612,494,848 bytes and the ac3 is 567,484,416 bytes

Oh yeah the file is around 3hrs 12 mins. Its a NFL game.

94SupraTT
10-06-2006, 05:58 PM
There must be a digit or two dropped -- 138 megs of 1920x1080i video would be, well, not much at all. :)

Might be worth trying this through ffmpeg instead; this is the kind of bad .ac3 stream that has biffed in mplex for me every once in a while. (Haven't had one lately to try ffmpeg on, but...)

What settings do you recommend for ffmpeg.

bcc
10-06-2006, 06:04 PM
What settings do you recommend for ffmpeg.The version of hdemux I posted today will tell you what arguments to use for ffmpeg. Also be sure to use the ffmpeg version I posted yesterday :)

94SupraTT
10-07-2006, 12:44 AM
The version of hdemux I posted today will tell you what arguments to use for ffmpeg. Also be sure to use the ffmpeg version I posted yesterday :)

Thanks. I'll give it a whirl.

94SupraTT
10-07-2006, 02:00 AM
I'm getting a "point_impure_ptr" error when trying to run ffmpeg.

bcc
10-07-2006, 02:59 AM
I'm getting a "point_impure_ptr" error when trying to run ffmpeg.You installed the cygwin .dll's that I provided in the same directory as the ffmpeg.exe binary?

94SupraTT
10-07-2006, 11:40 AM
You installed the cygwin .dll's that I provided in the same directory as the ffmpeg.exe binary?

yes. Thats the first thing I double checked.

bcc
10-07-2006, 02:57 PM
yes. Thats the first thing I double checked.The full error message is a pop-up window that claims, "The procedure entry point_impure_ptr could not be located in the dynamic link library cygwin1.dll", right? This means you've got something wrong with your cygwin1.dll installation. Assuming you're using the binaries I provided, Windoze should report that your cygwin1.dll is version 1005.19.0.0, size 1,805,448 bytes.

mgoddard
10-07-2006, 11:11 PM
BCC,

Nice work on your hdemux program and it seems to work well on 6.3a HD streams. However, the calculated itsoffset doesn't always seem to be right. Here's what I got with "Walk The Line" off of HBOHD:
C:\tivo\tmf>..\hdemux\hdemux -i "{Walk the Line}{2004-12-31}{}{07.30 PM Sun Oct
01, 2006}{HBOH}.ty+" -v wtl.m2v -a wtl.ac3
Version 0.17
Source is {Walk the Line}{2004-12-31}{}{07.30 PM Sun Oct 01, 2006}{HBOH}.ty+
AC3 audio at 48kHz, 1536 bytes/sync frame
Video frame size is 1280x1088 - high definition. Frame rate 29.970030
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
431909376 audio stream bytes 1125714686 video stream bytes
Reported datarate 65384 Kbit/sec. (65000+384)
Proceed with remuxer:
mplex -f 8 -O 160196ms -r 153701 wtl.ac3 wtl.m2v -o <outfile>.mpg
OR
ffmpeg -acodec copy -vcodec copy -f ac3 -i wtl.ac3 -itsoffset 160.1968 -
i wtl.m2v <outfile>.vob


A 160 second offset was way off and in fact after some experimenting the offset should be about 0.5 seconds. I realize hdemux is a work in progress but I thought you should be aware of this. Keep up the good work.

bcc
10-08-2006, 01:58 AM
mplex -f 8 -O 160196ms -r 153701 wtl.ac3 wtl.m2v -o <outfile>.mpg
OR
ffmpeg -acodec copy -vcodec copy -f ac3 -i wtl.ac3 -itsoffset 160.1968 -
i wtl.m2v <outfile>.vobRight, that's clearly completely wrong. I tested HBO HD with 6.3 (not 6.3a, but...) I assume this is a one-of error? Here's a "secret" switch that should tell what is going on: "-dtr". Add that to your command line and cut&paste till it first shows the ffmpeg usage statement. Should only be a couple pages of output, and it should reveal what is up with the timestamps in your case.

johnsolo
10-08-2006, 02:32 AM
hdemux 0.13 Mac OSX Universal Binary

This hasn't been tested extensively on Intel. Works fine on my PPC Mac.

bcc
10-08-2006, 02:50 AM
The dolby digital thing is killing me.I must be lucky as I'm not seeing dropouts with 6.3 or 6.3a. Then again the old hr10-250 isn't getting too much use anymore... I haven't seen any technical troubleshooting of this, but here's my guess anyways. If the CPU is busy when dealing with difficult OTA decoding, you'll get the dropouts. The CPU is more easily busy now with 6.3 code (due to the guide caching and such). Some OTA broadcasting stations are more problematic than others because some encode audio frames in very inefficient ways. Folks who run a lot of hacked services on their box would more likely see problems as well (because those hacks are wasting cpu cycles at high priority).

I've seen 6.3 is very chatty in its kernel log concerning audio errors. I have to tune to sat. channels to keep /var/log from filling too quickly.

mgoddard
10-08-2006, 09:53 AM
BCC,

Here's the log with the -dtr switch. Hopefully this will help.


Version 0.17
Source is {Walk the Line}{2004-12-31}{}{07.30 PM Sun Oct 01, 2006}{HBOH}.ty+
Skipping master chunk
Vid SEQ packet at index 22
Chunk 1 Record 0 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: 180ca9 PES len: 1544
First Audio PTS set to 1576105
Chunk 1 Record 1 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: 180ca9 PES len: 1544
Pes len 1550 headlen 14 paybytes 1536
AC3 audio at 48kHz, 1536 bytes/sync frame
Chunk 1 Record 2 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f28bf6 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 3 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f29736 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 4 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f2a276 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 5 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f2adb6 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 6 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f2b8f6 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 7 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f2c436 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 8 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f2cf76 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 9 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f2dab6 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 10 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f2e5f6 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 11 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f2f136 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 12 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f2fc76 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 13 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f307b8 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 14 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f312f8 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 15 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f31e38 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 16 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f32978 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 17 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f334b8 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 18 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f33ff8 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 19 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f34b38 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 20 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f35678 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 21 Type 9/c0 1552 Timestamp: 0:00 (0)
PTS: f361b8 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 22 Type 7/e0 28 Timestamp: 0:00 (57b7831)
Chunk 1 Record 23 Type 2/e0 92 Timestamp: 0:00 (57b7831)
Video frame size is 1280x1088 - high definition. Frame rate 29.970030
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
PTS: f40bdc DTS: f3e8ab PES len: 0
First video DTS set to 15984811
First Video PTS set to 15993820
Audio timestamp offset: 160196.833333 ms
Reported datarate 65384 Kbit/sec. (65000+384)
Proceed with remuxer:
mplex -f 8 -O 160196ms -r 153701 wtl.ac3 wtl.m2v -o <outfile>.mpg
OR
ffmpeg -acodec copy -vcodec copy -f ac3 -i wtl.ac3 -itsoffset 160.1968 -i wtl.m2v <outfile>.vob

bcc
10-08-2006, 02:32 PM
BCC,

Here's the log with the -dtr switch. Hopefully this will help.

Well that's "interesting". The relevant lines are:
First Audio PTS set to 1576105
First Video PTS set to 15993820
Audio timestamp offset: 160196.833333 ms
(That's (15993820-1576105)/90000=160 seconds).
So yes there really is a 160 second skew in the mpeg timestamps.
If one was to work off the tivo timestamps instead, you'd get
(91977777-0)/1000000000=.0920 seconds, or a 92ms offset. But you said it should really have been about 500ms. So I'm at a loss as to how one is supposed to algorithmically see the right a/v sync in this stream.

mgoddard
10-08-2006, 04:43 PM
BCC,

My guess of 0.5 seconds was more of a rough estimate, not a precise measurement by me so I'm going to do some more testing to figure out what really should be. That's really odd that there is such an offset between the audio and video timestamps in that stream.

bcc
10-08-2006, 05:22 PM
My guess of 0.5 seconds was more of a rough estimate, not a precise measurement by me so I'm going to do some more testing to figure out what really should be. That's really odd that there is such an offset between the audio and video timestamps in that stream.It'd be good to test whether or not that unexpected delta only occurs at the beginning of the stream or whether chunks later on have the same situation. Did you start the recording from the live buffer? That changes things from the perspective of timestamps.

mgoddard
10-08-2006, 08:14 PM
I set it to record it ahead of time so I'm not sure if it was already on that channel before it started recording or not. It seems to me that a 160 offset couldn't possibly be right otherwise if you switched to that channel it would take 160 seconds before the audio and video would sync up.
I'll also do hdemux again and check the offsets further in the stream.

dizzave
10-08-2006, 08:48 PM
hdemux 0.13 Mac OSX Universal Binary

This hasn't been tested extensively on Intel. Works fine on my PPC Mac.

I tested this on a MacBook Pro with a directivo stream. The audio doesn't work at all with quicktime player, and with VLC, there seems to be a 1-2 second delay. This is essentially the same result as with the PPC version using Rosetta. I usually just use the mpeg-alternate with the latest TT.

bcc
10-08-2006, 08:59 PM
I set it to record it ahead of time so I'm not sure if it was already on that channel before it started recording or not. It seems to me that a 160 offset couldn't possibly be right otherwise if you switched to that channel it would take 160 seconds before the audio and video would sync up.
I'll also do hdemux again and check the offsets further in the stream.With hdemux, you can use the option "-s x" to skip x number of chunks from the input stream before demuxing. Should make testing easy.

mgoddard
10-08-2006, 10:18 PM
This stream was 77735 chunks long so I started it at 77,000 and it came back with an offset of 1.5519 seconds:

Version 0.17
Source is {Walk the Line}{2004-12-31}{}{07.30 PM Sun Oct 01, 2006}{HBOH}.ty+
Chunk 0 Record 0 Type f/20 4 Timestamp: 148:49 (81f02794807)
Chunk 0 Record 1 Type 2/e0 131032 Timestamp: 148:49 (81efe7f02f1)
Chunk 1 Record 0 Type f/20 4 Timestamp: 148:49 (81f02794807)
Chunk 1 Record 1 Type 2/e0 300 Timestamp: 148:49 (81efe7f02f1)
Chunk 1 Record 2 Type e/1 0 Timestamp: 148:49 (81efe7f02f1)
Chunk 1 Record 3 Type e/2 0 Timestamp: 148:49 (81efe7f02f1)
Chunk 1 Record 4 Type b/e0 28 Timestamp: 148:49 (81efe7f02f1)
Chunk 1 Record 5 Type 2/e0 24768 Timestamp: 148:49 (81f007c257c)
Chunk 1 Record 6 Type e/1 0 Timestamp: 148:49 (81f007c257c)
Chunk 1 Record 7 Type e/2 0 Timestamp: 148:49 (81f007c257c)
Chunk 1 Record 8 Type b/e0 28 Timestamp: 148:49 (81f007c257c)
Chunk 1 Record 9 Type 2/e0 25424 Timestamp: 148:49 (81f0870afa7)
Chunk 1 Record 10 Type e/1 0 Timestamp: 148:49 (81f0870afa7)
Chunk 1 Record 11 Type e/2 0 Timestamp: 148:49 (81f0870afa7)
Chunk 1 Record 12 Type f/20 4 Timestamp: 148:49 (81f0870afa7)
Chunk 1 Record 13 Type 9/c0 1552 Timestamp: 148:48 (81ee1a30b80)
PTS: 30d9a952 PES len: 1544
First Audio PTS set to 819571026
Chunk 1 Record 14 Type 9/c0 1552 Timestamp: 148:48 (81ee38b5380)
PTS: 30d9a952 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
AC3 audio at 48kHz, 1536 bytes/sync frame
Chunk 1 Record 15 Type 9/c0 1552 Timestamp: 148:48 (81ee5739b80)
PTS: 30d9b492 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
Chunk 1 Record 16 Type a/e0 28 Timestamp: 148:49 (81f0870afa7)
Chunk 1 Record 17 Type 2/e0 53920 Timestamp: 148:49 (81f04766a91)
Chunk 1 Record 18 Type e/1 0 Timestamp: 148:49 (81f04766a91)
Chunk 1 Record 19 Type e/2 0 Timestamp: 148:49 (81f04766a91)
Chunk 1 Record 20 Type b/e0 28 Timestamp: 148:49 (81f04766a91)
Chunk 1 Record 21 Type 2/e0 21528 Timestamp: 148:49 (81f06738d1c)
Chunk 2 Record 0 Type f/20 4 Timestamp: 148:49 (81f0870afa7)
Chunk 2 Record 1 Type 2/e0 9120 Timestamp: 148:49 (81f06738d1c)
Chunk 2 Record 2 Type e/1 0 Timestamp: 148:49 (81f06738d1c)
Chunk 2 Record 3 Type e/2 0 Timestamp: 148:49 (81f06738d1c)
Chunk 2 Record 4 Type b/e0 28 Timestamp: 148:49 (81f06738d1c)
Chunk 2 Record 5 Type 2/e0 30376 Timestamp: 148:49 (81f0e681747)
Chunk 2 Record 6 Type e/1 0 Timestamp: 148:49 (81f0e681747)
Chunk 2 Record 7 Type e/2 0 Timestamp: 148:49 (81f0e681747)
Chunk 2 Record 8 Type a/e0 28 Timestamp: 148:49 (81f0e681747)
Chunk 2 Record 9 Type 2/e0 39672 Timestamp: 148:49 (81f0a6dd231)
Chunk 2 Record 10 Type e/1 0 Timestamp: 148:49 (81f0a6dd231)
Chunk 2 Record 11 Type e/2 0 Timestamp: 148:49 (81f0a6dd231)
Chunk 2 Record 12 Type b/e0 28 Timestamp: 148:49 (81f0a6dd231)
Chunk 2 Record 13 Type 2/e0 22560 Timestamp: 148:49 (81f0c6af4bc)
Chunk 2 Record 14 Type e/1 0 Timestamp: 148:49 (81f0c6af4bc)
Chunk 2 Record 15 Type e/2 0 Timestamp: 148:49 (81f0c6af4bc)
Chunk 2 Record 16 Type f/20 4 Timestamp: 148:49 (81f0e681747)
Chunk 2 Record 17 Type 9/c0 1552 Timestamp: 148:48 (81ee75be380)
PTS: 30d9bfd2 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 2
Pes len 1550 headlen 14 paybytes 1536
Chunk 2 Record 18 Type 9/c0 1552 Timestamp: 148:48 (81ee9442b80)
PTS: 30d9cb12 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 2
Pes len 1550 headlen 14 paybytes 1536
Chunk 2 Record 19 Type 9/c0 1552 Timestamp: 148:48 (81eeb2c7380)
PTS: 30d9d652 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 2
Pes len 1550 headlen 14 paybytes 1536
Chunk 2 Record 20 Type 9/c0 1552 Timestamp: 148:48 (81eed14bb80)
PTS: 30d9e192 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 2
Pes len 1550 headlen 14 paybytes 1536
Chunk 2 Record 21 Type b/e0 28 Timestamp: 148:49 (81f0c6af4bc)
Chunk 2 Record 22 Type 2/e0 22644 Timestamp: 148:49 (81f145f7ee7)
Chunk 3 Record 0 Type f/20 4 Timestamp: 148:49 (81f145f7ee7)
Chunk 3 Record 1 Type 2/e0 588 Timestamp: 148:49 (81f145f7ee7)
Chunk 3 Record 2 Type e/1 0 Timestamp: 148:49 (81f145f7ee7)
Chunk 3 Record 3 Type e/2 0 Timestamp: 148:49 (81f145f7ee7)
Chunk 3 Record 4 Type a/e0 28 Timestamp: 148:49 (81f145f7ee7)
Chunk 3 Record 5 Type 2/e0 41776 Timestamp: 148:49 (81f106539d1)
Chunk 3 Record 6 Type e/1 0 Timestamp: 148:49 (81f106539d1)
Chunk 3 Record 7 Type e/2 0 Timestamp: 148:49 (81f106539d1)
Chunk 3 Record 8 Type b/e0 28 Timestamp: 148:49 (81f106539d1)
Chunk 3 Record 9 Type 2/e0 22656 Timestamp: 148:49 (81f12625c5c)
Chunk 3 Record 10 Type e/1 0 Timestamp: 148:49 (81f12625c5c)
Chunk 3 Record 11 Type e/2 0 Timestamp: 148:49 (81f12625c5c)
Chunk 3 Record 12 Type b/e0 28 Timestamp: 148:49 (81f12625c5c)
Chunk 3 Record 13 Type 2/e0 25056 Timestamp: 148:49 (81f1a56e687)
Chunk 3 Record 14 Type e/1 0 Timestamp: 148:49 (81f1a56e687)
Chunk 3 Record 15 Type e/2 0 Timestamp: 148:49 (81f1a56e687)
Chunk 3 Record 16 Type f/20 4 Timestamp: 148:49 (81f1a56e687)
Chunk 3 Record 17 Type 9/c0 1552 Timestamp: 148:48 (81eeefd0380)
PTS: 30d9ecd2 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 3
Pes len 1550 headlen 14 paybytes 1536
Chunk 3 Record 18 Type 9/c0 1552 Timestamp: 148:48 (81ef0e54b80)
PTS: 30d9f812 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 3
Pes len 1550 headlen 14 paybytes 1536
Chunk 3 Record 19 Type 9/c0 1552 Timestamp: 148:49 (81ef2cd9380)
PTS: 30da0352 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 3
Pes len 1550 headlen 14 paybytes 1536
Chunk 3 Record 20 Type a/e0 28 Timestamp: 148:49 (81f1a56e687)
Chunk 3 Record 21 Type 2/e0 35864 Timestamp: 148:49 (81f165ca171)
Chunk 4 Record 0 Type f/20 4 Timestamp: 148:49 (81f1a56e687)
Chunk 4 Record 1 Type 2/e0 5776 Timestamp: 148:49 (81f165ca171)
Chunk 4 Record 2 Type e/1 0 Timestamp: 148:49 (81f165ca171)
Chunk 4 Record 3 Type e/2 0 Timestamp: 148:49 (81f165ca171)
Chunk 4 Record 4 Type b/e0 28 Timestamp: 148:49 (81f165ca171)
Chunk 4 Record 5 Type 2/e0 25936 Timestamp: 148:49 (81f1859c3fc)
Chunk 4 Record 6 Type e/1 0 Timestamp: 148:49 (81f1859c3fc)
Chunk 4 Record 7 Type e/2 0 Timestamp: 148:49 (81f1859c3fc)
Chunk 4 Record 8 Type b/e0 28 Timestamp: 148:49 (81f1859c3fc)
Chunk 4 Record 9 Type 2/e0 29512 Timestamp: 148:49 (81f204e4e27)
Chunk 4 Record 10 Type e/1 0 Timestamp: 148:49 (81f204e4e27)
Chunk 4 Record 11 Type e/2 0 Timestamp: 148:49 (81f204e4e27)
Chunk 4 Record 12 Type a/e0 28 Timestamp: 148:49 (81f204e4e27)
Chunk 4 Record 13 Type 2/e0 45136 Timestamp: 148:49 (81f1c540911)
Chunk 4 Record 14 Type e/1 0 Timestamp: 148:49 (81f1c540911)
Chunk 4 Record 15 Type e/2 0 Timestamp: 148:49 (81f1c540911)
Chunk 4 Record 16 Type f/20 4 Timestamp: 148:49 (81f204e4e27)
Chunk 4 Record 17 Type 9/c0 1552 Timestamp: 148:49 (81ef4b5db80)
PTS: 30da0e92 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 4
Pes len 1550 headlen 14 paybytes 1536
Chunk 4 Record 18 Type 9/c0 1552 Timestamp: 148:49 (81ef69e2380)
PTS: 30da19d2 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 4
Pes len 1550 headlen 14 paybytes 1536
Chunk 4 Record 19 Type 9/c0 1552 Timestamp: 148:49 (81ef8866b80)
PTS: 30da2512 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 4
Pes len 1550 headlen 14 paybytes 1536
Chunk 4 Record 20 Type 9/c0 1552 Timestamp: 148:49 (81efa6eb380)
PTS: 30da3052 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 4
Pes len 1550 headlen 14 paybytes 1536
Chunk 4 Record 21 Type b/e0 28 Timestamp: 148:49 (81f1c540911)
Chunk 4 Record 22 Type 2/e0 18012 Timestamp: 148:49 (81f1e512b9c)
Chunk 5 Record 0 Type f/20 4 Timestamp: 148:49 (81f204e4e27)
Chunk 5 Record 1 Type 2/e0 16476 Timestamp: 148:49 (81f1e512b9c)
Chunk 5 Record 2 Type e/1 0 Timestamp: 148:49 (81f1e512b9c)
Chunk 5 Record 3 Type e/2 0 Timestamp: 148:49 (81f1e512b9c)
Chunk 5 Record 4 Type b/e0 28 Timestamp: 148:49 (81f1e512b9c)
Chunk 5 Record 5 Type 2/e0 32824 Timestamp: 148:49 (81f2645b5c7)
Chunk 5 Record 6 Type e/1 0 Timestamp: 148:49 (81f2645b5c7)
Chunk 5 Record 7 Type e/2 0 Timestamp: 148:49 (81f2645b5c7)
Chunk 5 Record 8 Type a/e0 28 Timestamp: 148:49 (81f2645b5c7)
Chunk 5 Record 9 Type 2/e0 43816 Timestamp: 148:49 (81f224b70b1)
Chunk 5 Record 10 Type e/1 0 Timestamp: 148:49 (81f224b70b1)
Chunk 5 Record 11 Type e/2 0 Timestamp: 148:49 (81f224b70b1)
Chunk 5 Record 12 Type b/e0 28 Timestamp: 148:49 (81f224b70b1)
Chunk 5 Record 13 Type 2/e0 26824 Timestamp: 148:49 (81f2448933c)
Chunk 5 Record 14 Type e/1 0 Timestamp: 148:49 (81f2448933c)
Chunk 5 Record 15 Type e/2 0 Timestamp: 148:49 (81f2448933c)
Chunk 5 Record 16 Type f/20 4 Timestamp: 148:49 (81f2645b5c7)
Chunk 5 Record 17 Type 9/c0 1552 Timestamp: 148:49 (81efc56fb80)
PTS: 30da3b92 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 5
Pes len 1550 headlen 14 paybytes 1536
Chunk 5 Record 18 Type 9/c0 1552 Timestamp: 148:49 (81efe3f4380)
PTS: 30da46d2 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 5
Pes len 1550 headlen 14 paybytes 1536
Chunk 5 Record 19 Type 9/c0 1552 Timestamp: 148:49 (81f00278b80)
PTS: 30da5212 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 5
Pes len 1550 headlen 14 paybytes 1536
Chunk 5 Record 20 Type b/e0 28 Timestamp: 148:49 (81f2448933c)
Chunk 5 Record 21 Type 2/e0 6000 Timestamp: 148:49 (81f2c3d1d67)
Chunk 6 Record 0 Type f/20 4 Timestamp: 148:49 (81f2c3d1d67)
Chunk 6 Record 1 Type 2/e0 22968 Timestamp: 148:49 (81f2c3d1d67)
Chunk 6 Record 2 Type e/1 0 Timestamp: 148:49 (81f2c3d1d67)
Chunk 6 Record 3 Type e/2 0 Timestamp: 148:49 (81f2c3d1d67)
Chunk 6 Record 4 Type a/e0 28 Timestamp: 148:49 (81f2c3d1d67)
Chunk 6 Record 5 Type 2/e0 46176 Timestamp: 148:49 (81f2842d851)
Chunk 6 Record 6 Type e/1 0 Timestamp: 148:49 (81f2842d851)
Chunk 6 Record 7 Type e/2 0 Timestamp: 148:49 (81f2842d851)
Chunk 6 Record 8 Type b/e0 28 Timestamp: 148:49 (81f2842d851)
Chunk 6 Record 9 Type 2/e0 38824 Timestamp: 148:49 (81f2a3ffadc)
Chunk 6 Record 10 Type e/1 0 Timestamp: 148:49 (81f2a3ffadc)
Chunk 6 Record 11 Type e/2 0 Timestamp: 148:49 (81f2a3ffadc)
Chunk 6 Record 12 Type f/20 4 Timestamp: 148:49 (81f2c3d1d67)
Chunk 6 Record 13 Type 9/c0 1552 Timestamp: 148:49 (81f020fd380)
PTS: 30da5d52 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 6
Pes len 1550 headlen 14 paybytes 1536
Chunk 6 Record 14 Type 9/c0 1552 Timestamp: 148:49 (81f03f81b80)
PTS: 30da6892 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 6
Pes len 1550 headlen 14 paybytes 1536
Chunk 6 Record 15 Type 9/c0 1552 Timestamp: 148:49 (81f05e06380)
PTS: 30da73d2 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 6
Pes len 1550 headlen 14 paybytes 1536
Chunk 6 Record 16 Type 9/c0 1552 Timestamp: 148:49 (81f07c8ab80)
PTS: 30da7f12 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 6
Pes len 1550 headlen 14 paybytes 1536
Chunk 6 Record 17 Type b/e0 28 Timestamp: 148:49 (81f2a3ffadc)
Chunk 6 Record 18 Type 2/e0 16496 Timestamp: 148:50 (81f32348507)
Chunk 7 Record 0 Type f/20 4 Timestamp: 148:50 (81f32348507)
Chunk 7 Record 1 Type 2/e0 17744 Timestamp: 148:50 (81f32348507)
Chunk 7 Record 2 Type e/1 0 Timestamp: 148:50 (81f32348507)
Chunk 7 Record 3 Type e/2 0 Timestamp: 148:50 (81f32348507)
Chunk 7 Record 4 Type a/e0 28 Timestamp: 148:50 (81f32348507)
Chunk 7 Record 5 Type 2/e0 44776 Timestamp: 148:50 (81f2e3a3ff1)
Chunk 7 Record 6 Type e/1 0 Timestamp: 148:50 (81f2e3a3ff1)
Chunk 7 Record 7 Type e/2 0 Timestamp: 148:50 (81f2e3a3ff1)
Chunk 7 Record 8 Type b/e0 28 Timestamp: 148:50 (81f2e3a3ff1)
Chunk 7 Record 9 Type 2/e0 36056 Timestamp: 148:50 (81f3037627c)
Chunk 7 Record 10 Type e/1 0 Timestamp: 148:50 (81f3037627c)
Chunk 7 Record 11 Type e/2 0 Timestamp: 148:50 (81f3037627c)
Chunk 7 Record 12 Type f/20 4 Timestamp: 148:50 (81f32348507)
Chunk 7 Record 13 Type 9/c0 1552 Timestamp: 148:49 (81f09b0f380)
PTS: 30da8a52 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 7
Pes len 1550 headlen 14 paybytes 1536
Chunk 7 Record 14 Type 9/c0 1552 Timestamp: 148:49 (81f0b993b80)
PTS: 30da9592 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 7
Pes len 1550 headlen 14 paybytes 1536
Chunk 7 Record 15 Type 9/c0 1552 Timestamp: 148:49 (81f0d818380)
PTS: 30daa0d2 PES len: 1544
Warning: PES payload had 2 bytes end pad, chunk 7
Pes len 1550 headlen 14 paybytes 1536
Chunk 7 Record 16 Type b/e0 28 Timestamp: 148:50 (81f3037627c)
Chunk 7 Record 17 Type 2/e0 27456 Timestamp: 148:50 (81f382beca7)
Vid SEQ packet at index 14
Chunk 8 Record 0 Type f/20 4 Timestamp: 148:50 (81f382beca7)
Chunk 8 Record 1 Type 2/e0 4080 Timestamp: 148:50 (81f382beca7)
Chunk 8 Record 2 Type e/1 0 Timestamp: 148:50 (81f382beca7)
Chunk 8 Record 3 Type e/2 0 Timestamp: 148:50 (81f382beca7)
Chunk 8 Record 4 Type a/e0 28 Timestamp: 148:50 (81f382beca7)
Chunk 8 Record 5 Type 2/e0 39832 Timestamp: 148:50 (81f3431a791)
Chunk 8 Record 6 Type e/1 0 Timestamp: 148:50 (81f3431a791)
Chunk 8 Record 7 Type e/2 0 Timestamp: 148:50 (81f3431a791)
Chunk 8 Record 8 Type b/e0 28 Timestamp: 148:50 (81f3431a791)
Chunk 8 Record 9 Type 2/e0 38160 Timestamp: 148:50 (81f362eca1c)
Chunk 8 Record 10 Type e/1 0 Timestamp: 148:50 (81f362eca1c)
Chunk 8 Record 11 Type e/2 0 Timestamp: 148:50 (81f362eca1c)
Chunk 8 Record 12 Type b/e0 28 Timestamp: 148:50 (81f362eca1c)
Chunk 8 Record 13 Type 2/e0 36796 Timestamp: 148:50 (81f3e235447)
Chunk 8 Record 14 Type 7/e0 28 Timestamp: 148:50 (81f3e235447)
Chunk 8 Record 15 Type 2/e0 88 Timestamp: 148:50 (81f3e235447)
Video frame size is 1280x1088 - high definition. Frame rate 29.970030
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
PTS: 30dbcaea DTS: 30dba7b9 PES len: 0
First video DTS set to 819701689
First Video PTS set to 819710698
Audio timestamp offset: 1551.911111 ms
Reported datarate 65384 Kbit/sec. (65000+384)
Proceed with remuxer:
mplex -f 8 -O 1551ms -r 153701 wtl.ac3 wtl.m2v -o <outfile>.mpg
OR
ffmpeg -acodec copy -vcodec copy -f ac3 -i wtl.ac3 -itsoffset 1.5519 -i wtl.m2v <outfile>.vob

mgoddard
10-08-2006, 10:19 PM
I also tried starting at chunk 70,000 and it came back with a different offset of 1.4156 seconds. Also I muxed that sample with ffmpeg with that recommended offset and the timing was perfect.

I also tried that offset from the beginning of the movie at least within the first 15 minutes the video is about 1/2 second before the audio.

bcc
10-08-2006, 10:38 PM
I also tried starting at chunk 70,000 and it came back with a different offset of 1.4156 seconds. Also I muxed that sample with ffmpeg with that recommended offset and the timing was perfect.Right, you're going to get a different a/v offsets depending upon what chunk you look at, as the a/v streams aren't multiplexed into chunks at an exact rate.

So I assume the problem is with some mpeg discontinuity at the beginning of your recording, and you merely need to pre-skip over that junk.

mgoddard
10-08-2006, 11:05 PM
There is definitely some sort of discontinuity with this stream. I muxed the first 15 minutes with ffmpeg with an offset of 0.9 seconds and the sync was about perfect. I'll try doing the whole movie with this offset tonight and see if it stays in sync completely through.

94SupraTT
10-09-2006, 07:33 PM
The full error message is a pop-up window that claims, "The procedure entry point_impure_ptr could not be located in the dynamic link library cygwin1.dll", right? This means you've got something wrong with your cygwin1.dll installation. Assuming you're using the binaries I provided, Windoze should report that your cygwin1.dll is version 1005.19.0.0, size 1,805,448 bytes.

I'll check to see what version I am running.

mgoddard
10-09-2006, 07:39 PM
Well I tried some other HD movies I recorded such as 'Block Party' and I got an offset of 0.716 and when I muxed it the synch was dead on. The fact that the first stream I tried had a problem led me to think there was a problem with hdemux but as the long as the stream is ok hdemux works perfect. Thanks for creating this useful tool BCC and also the command line switches you mentioned are very useful for determining if there is a problem with the stream.

bcc
10-10-2006, 01:57 AM
Well I tried some other HD movies I recorded such as 'Block Party' and I got an offset of 0.716 and when I muxed it the synch was dead on. The fact that the first stream I tried had a problem led me to think there was a problem with hdemux but as the long as the stream is ok hdemux works perfect. Thanks for creating this useful tool BCC and also the command line switches you mentioned are very useful for determining if there is a problem with the stream.Thanks, it seems like hdemux with my hacked ffmpeg is working better than hdemux+mplex did for folks.

cheer
10-10-2006, 07:03 AM
Thanks, it seems like hdemux with my hacked ffmpeg is working better than hdemux+mplex did for folks.

Quick question -- is there a way to make ffmpeg use a .mpg container instead of .vob? I ask because I'd like to use ffmpeg instead of mplex as part of a batch process I use in conjunction with EtiVo and bluewomble's Divx add-in (really mencoder), but it expects to be fed .mpg.

Also, the very first frame of the mux is generally garbage (mostly just green) -- is this normal? The same elementary streams muxed with mplex don't show this.

LtngStalker
10-12-2006, 02:06 AM
I gave up and tried TyTool again with success. Part of the problem is that my streams came from a poor quality VHS which, to TiVo's credit, was captured remarkably well. There were a lot of what VideoReDo called resync frames, which I'm guessing are "frames" of data to readjust the audio so it keeps in sync with the video despite dropped or repeated frames. Anyways, good luck with hdemux. It's getting better every day ;)

bcc
10-12-2006, 02:43 AM
Quick question -- is there a way to make ffmpeg use a .mpg container instead of .vob?Both should be the same container format - mpeg2 protocol stream. .vob merely implies a more strict subset of the mpeg2 standard that's dvd compliant. Ok well there's some extensions too like ac3 encapsulation for dvds. I don't know what your tools are getting uptight about, but under windoze, there's a good chance you can merely rename the .vob to .mpg.
Also, the very first frame of the mux is generally garbage (mostly just green) -- is this normal? The same elementary streams muxed with mplex don't show this.That is strange because hdemux goes to some trouble to throw out video packets until you're at a clean start of GOP boundary. I think i've seen that too, but I didn't get as far as figuring out whether it was the player's "fault" or what.

bcc
10-12-2006, 02:44 AM
I gave up and tried TyTool again with success.As far as I know you wouldn't have had a problem with current hdemux+ffmpeg.

cheer
10-12-2006, 12:37 PM
Weird situation.

I have something I recorded on my HR10-250 when it was running 6.3a. I then decided to try down-revving to 3.1.5, so I extracted my recordings as .ty files. After I got 3.1.5 up and running, I re-inserted a few recordings. I tried to play one of them but the Tivo rebooted -- I chalked this up to 3.1.5 issues at the time. I then re-upgraded to 6.3a.

Now when I try to play the file on the Tivo, again I get a reboot. I re-extracted and tried to hdemux, but got the following:
Version 0.17
Source is b0rked.ty
error: Cannot fit Audio buffer, size 158819 offset 0First 16 chunks attached. Any ideas? Is this lost forever? I tried skipping several chunks in but that didn't help.

bcc
10-12-2006, 01:15 PM
Weird situation.

I have something I recorded on my HR10-250 when it was running 6.3a. I then decided to try down-revving to 3.1.5, so I extracted my recordings as .ty files. After I got 3.1.5 up and running, I re-inserted a few recordings. I tried to play one of them but the Tivo rebooted -- I chalked this up to 3.1.5 issues at the time. I then re-upgraded to 6.3a.

Now when I try to play the file on the Tivo, again I get a reboot. I re-extracted and tried to hdemux, but got the following:The master chunk looks OK, the data chunks look scrambled or somesuch, with no TY records that parse.

cheer
10-12-2006, 02:30 PM
The master chunk looks OK, the data chunks look scrambled or somesuch, with no TY records that parse.
Gah. Wonder how I managed that. Ain't just one recording either - I've got half a dozen.

Ah well. I'll find 'em elsewhere. Thanks for looking at it. By the way...I don't know whether I fixed something on my PC or whether the latest hdemux is just that good, but since I stopped having all that trouble I have yet to find a situation where mplex dies now. (I'm ready to use ffmpeg JustInCase but haven't needed it once.) Even when hdemux finds timestamp discontinuities, everything just ends up fine.

Now I'm batching. I redirect the output of hdemux to a file, then parse it to extract the delay, etc. and run mplex. Turn the whole mess loose on a directory full of stuff and walk away for several hours. :)

Thanks again for all of your hard work on this.

Cheezmo
10-12-2006, 04:51 PM
I'm not sure why, but I'm having no success at all with the hdemux->ffmpeg path.

hdemux->mplex provides output that MPEG StreamClip can usually deal with although audio sync issues crop up partway into them quite frequently. Still has the problem that if you try to extract a clip from the middle it can't find the frame size for some reason.

hdemux->ffmpeg can not be opened by MPEG Streamclip and play audio only on the Roku Photobridge HD1000.

HR10-250 6.3a

cheer
10-12-2006, 10:27 PM
OK, new issue. (It's because I posted that things were going so well!)

On one video hdemux is crashing (Windows issues a GPF) with the following on-screen error:
Assertion failed: pts != -1, file demux.c, line 580

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
I ran it with the -dtr switch and logged it to a file, then figured out which chunk was the last processed. I used tychopper to chop around it then re-ran hdemux just to make sure it still crashed (it did).

Attached are the little choplet and the -dtr output right before it died (I can give you the whole log if you want but it's huge). Nothing in the log jumps out at me, but then I don't know what I'm doing anyway.

bcc
10-12-2006, 11:31 PM
I ran it with the -dtr switch and logged it to a file, then figured out which chunk was the last processed. I used tychopper to chop around it then re-ran hdemux just to make sure it still crashed (it did).Good job narrowing down the failure point. Looks like you have a problem with your video stream at this point. Can you see the video glitch when you try to play back the stream at this point (either on a pc or the tivo)?

bcc
10-12-2006, 11:34 PM
I'm not sure why, but I'm having no success at all with the hdemux->ffmpeg path.

hdemux->mplex provides output that MPEG StreamClip can usually deal with although audio sync issues crop up partway into them quite frequently. Still has the problem that if you try to extract a clip from the middle it can't find the frame size for some reason.

hdemux->ffmpeg can not be opened by MPEG Streamclip and play audio only on the Roku Photobridge HD1000.

HR10-250 6.3aYou really want me to debug mpeg streamclip, don't you? :) Do you have problems playing back the output from mplex or ffmpeg with a mpeg2 compliant media player?

LtngStalker
10-12-2006, 11:43 PM
As far as I know you wouldn't have had a problem with current hdemux+ffmpeg.

The version I have is 0.16 and it was producing black video. :confused:

Cheezmo
10-13-2006, 12:03 AM
Yes, the Photobridge HD1000. Mplay on it has handled just about every MPG I've ever thrown at it. If it was just MPEG Streamclip, I would have left it as an issue for him, but since I'm not able to play it on my other HD playback device, I figure something more general is going on.

I'm more than willing to do whatever I can to help debug, etc.

I had a workflow that worked TyTool's multiplexing -> MPEG Streamclip or to playback on my HD1000. Since that is no longer an option I'm looking for an alternative and hdemux looks to be it. Since the files it produces don't work with that target editor and playback device I'm 1) pointing that out, and 2) offering to help in whatever manner I can.

I do have one theory that I'll experiment with tonight. I've been using TyTool to download the .ty files. I'll try using mfs_ftp and see if that changes anything.

Cheezmo
10-13-2006, 01:10 AM
Here is a test I just did...

recent HD OTA recording with HR10-250 6.3a

Downloaded .ty with mfs_ftp

hdemux -> mplex
MPEG Streamclip plays OK, can't edit
Photobridge HD1000, no video
VLC plays fine
Quicktime (w/MPEG2 decoder) plays fine

hdexmux->ffmpeg
MPEG Streamclip (hangs loading)
Photobridge HD1000 no video
VLC plays fine
Quicktime (reports movie could not be opened)

tytool10r4 (lots of errors processing, known HR10-250 6.3a problem).
MPEG Streamclip Plays with breakups, can edit
Photobridge HD1000 plays with breakups
VLC plays with breakups
Quicktime plays with breakups

cheer
10-13-2006, 08:53 AM
Good job narrowing down the failure point. Looks like you have a problem with your video stream at this point. Can you see the video glitch when you try to play back the stream at this point (either on a pc or the tivo)?
Yep, absolutely. I chopped "around" the last chunk in the log, hdemuxed/mplexed the two segments separately, edited, and stitched together w/VideoReDo and it looks fine (momentary burp at the stitch point).

bcc
10-13-2006, 12:41 PM
Yep, absolutely. I chopped "around" the last chunk in the log, hdemuxed/mplexed the two segments separately, edited, and stitched together w/VideoReDo and it looks fine (momentary burp at the stitch point).What I meant was did you verify that the tivo "sees" this same video error in the source .ty at the point that you identified? Looks clearly like an error to me, as the video PES packet is malformed.

Looks like I probably could have skipped over the error and you'd only notice a bad macroblock.

bcc
10-13-2006, 01:03 PM
The version I have is 0.16 and it was producing black video. :confused:Not enuf info to go on here; may be a problem with your codecs.

cheer
10-13-2006, 01:05 PM
What I meant was did you verify that the tivo "sees" this same video error in the source .ty at the point that you identified? Looks clearly like an error to me, as the video PES packet is malformed.

Looks like I probably could have skipped over the error and you'd only notice a bad macroblock.
That I did not do -- I mean I see it visually when playing with TyShow/MPC, but haven't re-upped the .ty to the Tivo. Assuming I didn't delete it, I'll do that this afternoon and let you know what happens.

bcc
10-13-2006, 01:34 PM
Looks like I probably could have skipped over the error and you'd only notice a bad macroblock.I have a "fix" that will recover from the bad frame and keep processing. However if I'm really going to get into the business of recovering from bad a/v frames I'll have to start worrying about filling the resulting "holes" in the streams. Sound familiar? :)

cheer
10-13-2006, 01:41 PM
I have a "fix" that will recover from the bad frame and keep processing. However if I'm really going to get into the business of recovering from bad a/v frames I'll have to start worrying about filling the resulting "holes" in the streams. Sound familiar? :)
Yes indeedy. By the way, that -dtr switch is insanely useful; any other "secret" switches? :)

94SupraTT
10-13-2006, 06:39 PM
E:\>ffmpeg -acodec copy -vcodec copy -f ac3 -i Seahawks.ac3 -itsoffset 0.194 -i
Seahawks.m2v Seahawks.vob
FFmpeg version SVN-r6543, Copyright (c) 2000-2006 Fabrice Bellard, et al.
configuration: --enable-gpl --enable-a52 --disable-v4l --disable-dv1394 --dis
able-ffplay --disable-ffserver
libavutil version: 49.0.1
libavcodec version: 51.16.0
libavformat version: 50.5.0
built on Oct 6 2006 00:36:08, gcc: 3.4.4 (cygming special) (gdc 0.12, using d
md 0.125)
Input #0, ac3, from 'Seahawks.ac3':
Duration: 03:17:00.8, start: 0.000000, bitrate: 384 kb/s
Stream #0.0: Audio: ac3, 48000 Hz, mono, 384 kb/s
Input #1, mpegvideo, from 'Seahawks.m2v':
Duration: 00:44:20.0, start: 0.000000, bitrate: 64999 kb/s
Stream #1.0: Video: mpeg2video, yuv420p, 1920x1080, 65000 kb/s, 29.97 fps(r)
Output #0, vob, to 'Seahawks.vob':
Stream #0.0: Video: mpeg2video, yuv420p, 1920x1080, q=2-31, 65000 kb/s, 29.97
fps(c)
Stream #0.1: Audio: ac3, 48000 Hz, mono, 384 kb/s
Stream mapping:
Stream #1.0 -> #0.0
Stream #0.0 -> #0.1
[vob @ 0x6fd594]Setting video buffer for HD
Press [q] to stop encoding
frame=350428 q=19378.2 Lsize=21925422kB time=11820.8 bitrate=15194.7kbits/s
video:21105949kB audio:554100kB global headers:0kB muxing overhead 1.225171%

E:\>


It looks like hdemux/ffmpeg worked but I can't get VideoRedo to open the .vob now. :( vlc player will play the .vob