ffmpeg with MPEG to TY+ encoding support

[bcc]

Enclosed are changes to ffmpeg which allows it to encode to .ty+ format. ffmpeg is superior to mplex for encoding to .ty+ format in that:

• ffmpeg's multiplexor, with the enclosed changes, can handle directv streams with video frames that sometimes contain 3 fields/frame. This is a common mpeg encoding scheme employed by directv but not seen in dvd compliant mpeg. mplex malfunctions in such cases because it fails to synthesize the correct timestamps to match the variable frame duration.
• With ffmpeg, one can now encode directly from .mpg or .vob to .ty+ without any separate demuxing step.

CREDITS:

• bcc for: The new libavformat encoding library for ffmpeg, TY master chunk processing library, XML record generation, TY chunk record streaming library.
• Nova1 for:
  Original code demonstrating the relatively straightforward mapping from an elementary mpeg video stream to TY records
  Original changes to more efficiently pack TY chunks

USAGE:
Notes: When specifying an a/v sync offset with -itsoffset, the order of the arguments matter.
Unlike mplex, ffmpeg won't work hard to glean the audio stream format, so if you're specifying that separately, you'll need to indicate whether it's ac3 or mpeg. In the following examples I do so by file name extension.
Converting from a .vob to .ty directly requires ffmpeg to identify the a/v streams. If it fails to do so (common it seems) you may still have to demux the streams first.
ffmpeg -i src.mpg -acodec copy -vcodec copy dest.ty
The above converts .mpg to .ty directly, without transcoding the video or audio
ffmpeg -i src.ac3 -itsoffset 0.798 -i src.m2v -acodec copy -vcodec copy dest.ty
In the above, the video stream start delayed 798ms after AC3 audio stream.
ffmpeg -i src.m2v -itsoffset 0.637 -i src.m2a -acodec copy -vcodec copy dest.ty
MPEG audio stream start delayed by 637ms after video stream.
COMPILING:
ffmpeg compiles under linux, and for windows, using cygwin.

1. Unfortunately the ffmpeg project does not seem to be maintaining snapshots of their code, so first checkout the code from their SVN repository. The diffs were made against revision 5510, so if you checkout the same revision, the diffs should patch cleanly even if the mainline has diverged since then:
   svn checkout -r 6543 svn://svn.mplayerhq.hu/ffmpeg/trunk ffmpeg
   Or you could try the tivoserver snapshot of ffmpeg source.
2. cd ffmpeg
3. patch -p1 < ffmpeg-diffs-xx.txt
4. ./configure --enable-gpl --enable-a52 --disable-v4l --disable-dv1394 --disable-ffplay --disable-ffserver
5. make
6. make install

There is a support thread over here: http://www.dealdatabase.com/forum/sh...ad.php?t=50010

[bcc]

Windows & linux binaries are here.
For windows, you'll need the DLL files from cygwin.rar as well. For windows, unpack the two .rar files, and put the .dll and .exe files in the same dir.

[Jamie]

Very cool. I'm embarrassed that I volunteered to look at this, then dropped the ball. Glad someone got to it. I'll give it a try.

[bcc]

Very cool. I'm embarrassed that I volunteered to look at this, then dropped the ball. Glad someone got to it. I'll give it a try.

Thanks. You weren't the only other person who said you were going to try tackling it But you should probably be glad you didn't try - I had to do a bunch of wading through the ffmpeg code before I could get things to work and there were a couple nasty issues. In the end the diffs to ffmpeg came out pretty clean (cleaner than with mplex). So the ffmpeg API is fairly modular but their internals are poorly documented.

[Jamie]

So the ffmpeg API is fairly modular but their internals are poorly documented.

Yeah, searching for good libav* documentation is about as far as I got.

To make another possibly empty promise, maybe I can look at integrating this into tivoserver. Tivoserver requires that ffmpeg can write to an unseekable stream, but, it doesn't need the at-end master chunk trickplay update. It seems to me they named a file extension type for ty without masterchunks, but I can remember what it is. Maybe another AVOutputFormat for ty without masterchunks is the way to go.

[bcc]

For tivoserver, I was going to make the code skip the master chunk processing if the file descriptor was a fifo, but punted because I think that would make the source less portable. A little less clean would be to make the code assume it's dealing with a fifo if the lseek fails.

[Jamie]

For tivoserver, I was going to make the code skip the master chunk processing if the file descriptor was a fifo, but punted because I think that would make the source less portable. A little less clean would be to make the code assume it's dealing with a fifo if the lseek fails.

tmf would also be a nice addition, since, IME, mfs_ftp works better with tmf than it does with ty+. Naturally, that would require a seekable output stream. I can probably look at this. It doesn't sound too hard and should be able to reuse all the ty+ code with just a little extra processing at the end and between file parts.

[phat_bastard]

What I've been able to watch has been flawless! The only issue I'm seeing is the master chunks after the first segment (chunk 4097 and up) don't seem to be valid so tivoserver is unable to parse the streams correctly and insertion (through tivoserver) causes the tivo to reboot when the transfer reaches the first corrupt master chunk. They're also crashing my ty chunk analyzer at the moment but what I saw before the app died was all FF bytes. I'll post additional details when I get a chance to dig into this a little further. I think everything is kosher with streams that aren't big enough to fill > 4096 chunks, though I haven't tested one just yet.

[bcc]

The only issue I'm seeing is the master chunks after the first segment (chunk 4097 and up) don't seem to be valid

Oops, I fudged the sanity checking trying to hush a cosmetic warning. Here's the fix:

*** libavformat/tymaster.c	2006-06-21 09:33:27.000000000 -0700
--- libavformat/tymaster.c	2006-06-22 13:22:58.000000000 -0700
***************
*** 70,82 ****
      boolean expected, have_seq;
  
      recs = getshort(buf);
      if (recs == 0xffff) {
! 	return FALSE;
      }
      p = &buf[4];
      time = get64(&p[8]);
      if (!time_validate(time, lasttime, ticks_per_chunk)) {
- 	expected = (chunkcnt == (segsize+1));
  	if (dbg(2)) {
  	    dbgprint("Hit %stime discontinuity at chunk %d\n",
  		     expected ? "expected " : "",
--- 70,82 ----
      boolean expected, have_seq;
  
      recs = getshort(buf);
+     expected = (chunkcnt == (segsize+1));
      if (recs == 0xffff) {
! 	return expected;
      }
      p = &buf[4];
      time = get64(&p[8]);
      if (!time_validate(time, lasttime, ticks_per_chunk)) {
  	if (dbg(2)) {
  	    dbgprint("Hit %stime discontinuity at chunk %d\n",
  		     expected ? "expected " : "",

[phat_bastard]

After further investigation, the problem seems to be that the master chunk for the second fsid is empty - chunk size and chunk count are both zero. The crashing of my packet analyzer was happening on a padding chunk at the end of the sequence (no problems there).

[phat_bastard]

I think it might be something here:

tymaster.c::rewrite_master()
 	if (!parse_chunk(buf, paychunk, &lastseq, segsize)) {
 	    if (chunkcnt && (chunkcnt-1) == pos) { // < ???
 		/* expected end of data chunks */
 	    } else {
 		dbgprint("Warning: Found invalid chunk at %u\n", pos);
 	    }
 	    break;
        }

Because I'm getting this:

Warning: Found invalid chunk at 4095
Generating master chunk for chunks 0:4094
Successfully wrote new XML

from muxing a stream that has 5972 chunks

Maybe just need to move the break up 3 lines?

[phat_bastard]

That makes more sense, thanks! Testing...

Incidentally the ffmpeg-tivoserver.tgz snapshot has a few changes in libavformat that prevent merging your patch, but it's not a big deal. The svn snapshot seemed to compile okay under cygwin after I got off my duff and installed it (svn).

[bcc]

Incidentally the ffmpeg-tivoserver.tgz snapshot has a few changes in libavformat that prevent merging your patch, but it's not a big deal. The svn snapshot seemed to compile okay under cygwin after I got off my duff and installed it (svn).

Hmm, I can tell that maintaining diffs against an ever changing baseline is going to be a problem.

[bcc]

I've made version 1.1 of the diffs, which fix the >4095chunk bug that phat_bastard saw, and also allows ffmpeg to write .ty to a fifo/pipe for tivoserver. (Diffs attached to first post of this topic).

[darrin75]

Does this seem to be more reliable and stable for tivoserver??