tytompg: TY to mpg conversion in one step, alpha release

[bcc]

Here's a beta release of tytompg. tytompg converts from TY/TY+/TMF to mpeg2 program streams in a single step. Ie it combines the TY demuxing operation with a loss-less re-multiplex operation. Inside it uses the mature hdemux code for demuxing, and a purpose-built muxer on the muxing side. For multiplexing, I'm preserving the timestamps from the original TY stream instead of trying to regenerate them, as was required with hdemux. This has some benefits and some drawbacks (mostly benefits I believe). Some benefits:

• Preserving a/v sync regardless of errors in the audio or video streams. For example, the current tivo bug that plagues much of the FOX OTA broadcasts with audio dropouts does not phase tytompg.
• tytompg's ability to pass-thru video with varying frame rates (these have historically been a deal-breaker for using the mplex muxer).
• mpeg2 editors such as videoredo work with tytompg a lot better than hdemux+ffmpeg, as tytompg includes timestamps at all video I frame boundaries (so long as the original recording included them), whereas ffmpeg would often leave out some of these timestamps.

Some issues:

• Because tytompg does not try to re-generate the timestamps, this can cause problems for players and mpeg2 editors that assume that an mpeg2 program stream has no errors and/or is dvd compliant. For example, I've seen mplayer unable to properly recover from audio holes, but xine does so just fine. YMMV.
• I'm not at this time generating DVD navigation records, so players that are unable to perform trick-play-like operations without them (ie simplistic players like MyHD's) will have trouble. This really should only be a real concern for DVD output, in which case you'll need to post-process the mpeg anyways (to fix other issues including the video GOPs not being DVD compliant).

Typical usage:
tytompg -y <show>.ty
Output is to <show>.mpg by default. Some switches:
-y Allow overwrite of output file without confirmation, even if it already exists
-V Omit video stream from output file
-A Omit audio stream from output file
-s <number> Skip <number> chunks on input before processing. Note: if input file is a .tmf file, this option will not work correctly
-n <number> Process <number> chunks from input


Example:

% tytompg -y game7cut.ty
tytompg: Copyright (c) 2004-2007 B.C. <bcc24x7@gmail.com>
Multiplexer version 0.19, Demuxer version 0.26
Source is game7cut.ty 
AC3 audio at 48kHz, 384 kbps, 1536 bytes/sync frame
Recovered AC3 sync, offset 134, chunk 0
Video frame size is 1280x720  - high definition. Frame rate 59.940060
Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
Recovered AC3 sync, offset 134, chunk 0
Warning: temporal_reference order problem.  Value: 3 group_start: 810 last: 836
audio: Warning: Decoder buffer empty, scr ahead by 81 (0.81) 0.0
        ... and encoder buffer full
audio: Warning: Dropping frame -- too old by 198662065 (662206.265) 7.32206
Done!
Stream audio: 1673 frames (62 seconds aprox.)
Stream video: 3737 frames, 7474 fields (62 seconds aprox.)
%

Windows 32, linux-x86, linux-x86-64, mac-x86 binaries attached.

Enjoy,

B.C. <bcc24x7@gmail.com>

[bcc]

<Info about troubleshooting, diagnostics to go here.>

-d is the debugging switch. It has a lot of arguments, (some of which don't necessarily do anything right now). -dtr, for example, means 't' and 'r' options are enabled. The full list (subject to change):

t timestamps
r TY records
a audio records
v video records
b buffering
i i/o
m multiplexer buffering
p padding
f virtual buffer flow 
q queueing
D packet dump
F full packet dump
R TY record details
T Tivo timestamps

Other useful command line options include '-s chunk-count' to skip chunk-count chunks before demuxing, and '-c chunk-count' to demux chunk-count chunks.

[mr_zorg]

Originally Posted by bcc

Windows & linux binaries attached.

This sounds awesome. I won't ask about the source, as we've already had that discussion over in your hdemux thread. However, I do want to reiterate my offer to compile an OS X binary for you, if you want. If you PM me the source, I will compile a universal binary (PPC & Intel) and send it back. I promise to delete the source immediately afterwards!

[RavenStL]

Well I tried it on FOX OTA and got this

Multiplexer version 0.1
Demuxer version 0.19
Source is NLCSg6.ty
AC3 audio at 48kHz, 448 kbps, 1792 bytes/sync frame
Recovered AC3 sync, offset 110, chunk 1
Video frame size is 1280x720  - high definition. Frame rate 59.940060
Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
Recovered AC3 sync, offset 110, chunk 1
audio: Warning: Decoder buffer empty, scr ahead by 2560 (8.160) 0.8
        ... and encoder buffer full
Assertion failed: FALSE, file demux.c, line 1259

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

Is that 8.160 seconds? Cause my dropouts are around 9 seconds long.

[bcc]

Originally Posted by RavenStL

Is that 8.160 seconds? Cause my dropouts are around 9 seconds long.

The warning shows an audio underrun, which is what I'd expect you to see during those FOX recording glitches. Unfortunately you can't see the whole duration of the hole from that diagnostic message.

The assertion failure probably happened a bit later. I should be able to fix that one fairly easily.

[bcc]

Originally Posted by mr_zorg

This sounds awesome. I won't ask about the source, as we've already had that discussion over in your hdemux thread. However, I do want to reiterate my offer to compile an OS X binary for you, if you want. If you PM me the source, I will compile a universal binary (PPC & Intel) and send it back. I promise to delete the source immediately afterwards!

Easiest would be to just cross compile. Nobody responded to my cross compiler question...

[bcc]

Originally Posted by bcc

The assertion failure probably happened a bit later. I should be able to fix that one fairly easily.

Try the 0.2 version. Unfortunately I don't have a test case to cover this, so you're the guinea pig (unless you want to send a clip).

[RavenStL]

That got it going, but I accidentally deleted the file and have to re downlaod it from the HR10-250. Dont ask, I thought I was deleting somehting else............

I tried on another and got this though. FOX OTA still, differnt .ty

Multiplexer version 0.2
Demuxer version 0.19
Source is f:\WSG1.ty
AC3 audio at 48kHz, 384 kbps, 1536 bytes/sync frame
Video frame size is 1280x720  - high definition. Frame rate 59.940060
Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
Assertion failed: stream->aucnt < stream->autotcnt, file mplex.c, line 405

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

[bcc]

Originally Posted by RavenStL

Assertion failed: stream->aucnt < stream->autotcnt, file mplex.c, line 405

Hmm, I did pre-test this on a half a dozen different fox cases. Guess I need to download more fox recordings. I assume you got some good mpeg before this error hit.

[RavenStL]

I got a good 6 gigs out of 24. 25% so far.

Another stream after 11.1 gigs of 20+

Multiplexer version 0.2
Demuxer version 0.19
Source is NLCSg7.ty
AC3 audio at 48kHz, 384 kbps, 1536 bytes/sync frame
Recovered AC3 sync, offset 34, chunk 1
Video frame size is 1280x720  - high definition. Frame rate 59.940060
Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
Recovered AC3 sync, offset 34, chunk 1
audio: Warning: Decoder buffer empty, scr ahead by 877098 (2923.198) 0.2923
        ... and encoder buffer full
audio: Warning: Decoder buffer empty, scr ahead by 299636 (998.236) 0.998
        ... and encoder buffer full
audio: Warning: Decoder buffer empty, scr ahead by 740 (2.140) 0.2
        ... and encoder buffer full
audio: Warning: Decoder buffer empty, scr ahead by 1213454 (4044.254) 0.4044
        ... and encoder buffer full
Warning: Tossing AC3 audio frame with bad sync data at end of packet.  Chunk 906
57 length 1450
Warning: Tossing AC3 audio frame with Rate out of bounds.  Chunk 90657 length 29
48
Warning: Tossing AC3 audio frame with Rate out of bounds.  Chunk 90657 length 27
48
Warning: Tossing AC3 audio frame with Rate out of bounds.  Chunk 90657 length 28
32
Warning: Tossing AC3 audio frame with Rate out of bounds.  Chunk 90657 length 29
16
Warning: Tossing AC3 audio frame with Rate out of bounds.  Chunk 90657 length 28
16
Warning: Tossing AC3 audio frame with Rate out of bounds.  Chunk 90657 length 29
00
Assertion failed: id != -1, file mplex.c, line 1542

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

[mr_zorg]

Originally Posted by bcc

Easiest would be to just cross compile. Nobody responded to my cross compiler question...

I know I didn't respond because I don't ever do it. I actually have a Mac. I'd love to tell you how to cross compile -- if I knew. :P

[mr_zorg]

Originally Posted by mr_zorg

I know I didn't respond because I don't ever do it. I actually have a Mac. I'd love to tell you how to cross compile -- if I knew. :P

It just dawned on me what I can do though. How about if I set you up with your own SSH/SFTP account on my Mac Mini Core Duo server? Would that be agreeable? If so, let me know and I'll PM you the information you need. Of course, once logged on you may change the password.

[bcc]

Originally Posted by mr_zorg

I know I didn't respond because I don't ever do it. I actually have a Mac. I'd love to tell you how to cross compile -- if I knew. :P

Right, and it hasn't been a priority for me to go research the best way to support equipment I *don't* have. Perhaps the distcc approach but gcc would be preferable.

[artships]

Wonderful tool, bcc. Tried it on an Ubuntu box, feeding it a 544x480 Sony Stand Alone (ancient, I know) tystream. Man is it fast! Alas, playback of the mpg was choppy in VLC, while playback of the original TY file was smooth. Listing:

Multiplexer version 0.2
Demuxer version 0.19
Source is The X-Files-Improbable.ty
Video frame size is 544x480  Frame rate 29.970030
Video bit rate is 5800000 bits/sec., 5800 Kbit/sec.
MPEG2 audio at 32kHz, 192 kbps, 864 bytes/sync frame
MPEG2 audio at 32kHz, 192 kbps, 864 bytes/sync frame
Done!
Stream audio: 100039 frames (2701 seconds aprox.)
Stream video: 107939 frames (3601 seconds aprox.)

Did I mention it was fast? Couldn't have lagged much behind cp for speed. And kept sync throughout, too. Quite impressive.

[Cheezmo]

I'm getting lots of assertion failed in line 428 of mplex.c on my tests. So far they are all about 1Gb in, so I don't know a good way to send you a test case.