TY enabled mplex for mpeg to TY stream conversion

[nova1]

In the seq_pos = -1 case, the patch is bad: the code still falls through to the original memcpy.

Does this imply that the video stream does not have any sequence header? mplex searches for a sequence header in videostrm_in.cpp ScanFirstSeqHeader() and then fills in the horiz, vert size and other info. The code creates a sequence header of 12 bytes from the first sequence header seen and stores it into seqhdr[]. in multiplexor.cpp, if it sees that the seq_pos == -1 then it'll use this seqhdr for the memcpy. The code asserts if gop_pos or pic_pos == -1 so it should always have a sequence header if there was a first one.

[bcc]

Does this imply that the video stream does not have any sequence header? mplex searches for a sequence header in videostrm_in.cpp ScanFirstSeqHeader() and then fills in the horiz, vert size and other info. The code creates a sequence header of 12 bytes from the first sequence header seen and stores it into seqhdr[]. in multiplexor.cpp, if it sees that the seq_pos == -1 then it'll use this seqhdr for the memcpy. The code asserts if gop_pos or pic_pos == -1 so it should always have a sequence header if there was a first one.

I haven't seen a failure, I was just noticing that if seq_pos == -1, then the code after the if-then still uses the negative value in its pointer arithmetic which is surely not what you intended.

[nova1]

Ahh, I forgot to delete those lines after cut-n-pasting. I've uploaded a new patch3.txt that deletes the memcpy() after the seq_pos == -1 check. That probably explains the version 0.2 failures.

[nova1]

I'm thinking of changing tymaster.c generate_gop() to allow for master chunks without any GOP timestamp records in it if we have more than 1 master chunk in the file.

    extern out_stream_t *outs;
    assert(i >= 0 || (i < 0  && outs->chnk > CHUNK_VALID_PER_SEG));
    if (i >= 0) {
      put64((u_char *)&gop_stamps[i], gop_stamps[i]);
      write_internal(ofd, (u_char *)&gop_stamps[i], sizeof(tivo_tstamp_t));
    }

I also added CFLAGS += -D_FILE_OFFSET_BITS=64 to the utils/Makefile to handle the large TY output file(3GB).

[bcc]

Here's the build environment I've been using for mplex under cygwin. mplex's automake and shared library environment seems to be pretty broken in 1.6.2 so I had to resort to drastic measures. I wrote my own makefiles (included), and moved the source to the following layout:

Top level: C++ code, includes
Utils dir: c code, private includes
include dir: shared includes

See file directory in the zip file for layout details, and makefile.

PS: Pick up the revised patch2.txt if you want to compile for windows - I had 1 booboo in the initial copy.

[bcc]

I'm thinking of changing tymaster.c generate_gop() to allow for master chunks without any GOP timestamp records in it if we have more than 1 master chunk in the file.

Huh? I doubt it is "legal" to have a master chunk record without any GOP timestamps. In the case that there are no GOP bits to set in the table, you would still have the trailing timestamp I believe. But, do you really have a case where there is a segment with no TY sequence records?

rewrite_master() already handles updating multiple master chunks. It assumes the source stream has layed out placeholder master chunks for it to rewrite. Is there a problem?

I also added CFLAGS += -D_FILE_OFFSET_BITS=64 to the utils/Makefile to handle the large TY output file(3GB).

Is this something specific to the fopen()? Maybe this should just be replaced with an open() as I've already provided OS portable largefile support for that (tyos.h).

[nova1]

Huh? I doubt it is "legal" to have a master chunk record without any GOP timestamps. In the case that there are no GOP bits to set in the table, you would still have the trailing timestamp I believe. But, do you really have a case where there is a segment with no TY sequence records?

rewrite_master() already handles updating multiple master chunks. It assumes the source stream has layed out placeholder master chunks for it to rewrite. Is there a problem?Is this something specific to the fopen()? Maybe this should just be replaced with an open() as I've already provided OS portable largefile support for that (tyos.h).

I was redoing the dvd ty file and I got

   INFO: [lt-mplex] Generating master chunk for chunks 16384:16384
lt-mplex: tymaster.c:155: generate_gop: Assertion `i >= 0' failed.

so I thought since we are now packing the data into the TY chunks, it's possible to have a continuation record at the start of the next master chunk. I then found out that I needed to rebuild the utils lib with the CFLAGS. I did not see the assertion after rebuilding for large file support. So it isn't an issue I've seen. I believe in the ty1to2 thread, there was a note on lseek() not quite working but pos has been redefined as off_t.

[bcc]

INFO: [lt-mplex] Generating master chunk for chunks 16384:16384
lt-mplex: tymaster.c:155: generate_gop: Assertion `i >= 0' failed.

Ok, you're right it shouldn't assert in this case. Looks like this case is possible when your last segment only has 1 or 2 data chunks. I think the code should look like:

    stamp = 0;
    if (i >= 0) {
	stamp = gop_stamps[i];
    }
    /* Write out final timestamp */
    put64(&stamp, stamp);
    write_internal(ofd, (u_char *)&stamp, sizeof(tivo_tstamp_t));

So that we write the final timestamp regardless.

I did not see the assertion after rebuilding for large file support.

Makes sense since the chunk #, 16384, is at 2^31. But actually, this file already had laregfile support thanks to tyos.h, so the largefile issue was probably with the mplex stream reading code.

I believe in the ty1to2 thread, there was a note on lseek() not quite working but pos has been redefined as off_t.

Yes the lseek() issue was taken care of and isn't a problem in any of the newer code you/I have been tossing about. I also took care of the 64 bit compiler issues.

[nova1]

I added a patch4.txt that incorporates the xml code from ty1to2-1.1 so you can add the -x xml-file option to mplex to tack on a TY+ XML file at the end of the TY output file.

[bcc]

Hi, I'm back from my trip & almost recovered.
Looks like your patch is missing xml.c. I guess this is useful for the ty->mpg->ty case, but for the mpg->ty case, where you don't have some initial xml already, it'd be real cool if we could auto-generate (and/or prompt for) the important XML fields.

I'm a bit surprised over the interest in using tymplex for ty->mpg->ty, other than in testing. If that was the ultimate goal, the code could be written a lot differently. For example, we could keep the real tivo timestamps, and also leave the streams interleaved at the rate found in the initial ty stream. Ie basically leave out the middle step and just process the ty against a GOP-accurate cut-list. The point of ty->mpg->ty is just to get GOP-accurate editing of the ty stream, for playback on the tivo, right?

[darrin75]

I never could to get this work right..I try it and the video size is 0 bytes..

[nova1]

ok, I've uploaded a new patch4.txt that includes xml.c from ty1to2-1.1.

I didn't realize I had the "Priority Set..." string right before the '#' marker in my test ty+ file so ty1to2 just saved the xml file. I just took the file and then hand modified certain fields for use with a different upload. Uploads don't happen
that frequently for me. You can also edit fields once on the Tivo using
the universaledititle.tcl script which is probably easier or as you say, have mplex prompt for this info.

In my limited testing, I haven't seen any issues doing ty->mpeg->ty or going from ty->tytools edit->mpeg program stream->demux a,v -> mplex->ty. I use ty2ty(modified ty1to2-1.1) that retains Tivo timestamps for testing and as a chunk editor using the -s and -n options.

[bcc]

I never could to get this work right..I try it and the video size is 0 bytes..

Could you send some details to the support thread then? Output from mplex, kind of source file (ty file?), command line, how you're creating the .m2v/.m2a. Perhaps upload the .m2a/.m2v and or the .ty or .mpg. You could use tychopper to shrink the sample to a minimal size if the source was .ty.
http://www.dealdatabase.com/forum/sh...ad.php?t=42006

[nova1]

I looked at a sample 720x480 clip provided by T u r b o that works on both a HDVR2 and a 810HS. It looked like a standard transmission order frame list of IPBBPBB:

TY time:0:00:000 (0), AC3 PES time: 21 00 05 d2 1d 00:00:01.27 (403)
TY time:0:00:000 (0), VSH PES time: 31 00 05 d9 f7 00:00:01.38 (40e)
TY time:0:00:000 (0), GOP
TY time:0:00:011 (aa63ca), IFRAME
TY time:0:00:032 (1e87367), AC3 PES time: 21 00 05 e8 9f 00:00:01.59 (423)
TY time:0:00:064 (3d0bb67), AC3 PES time: 21 00 05 ff 1f 00:00:01.91 (443)
TY time:0:00:096 (5b90367), AC3 PES time: 21 00 07 15 9f 00:00:01.123 (463)
TY time:0:00:111 (6a1cb6a), PFRAME PES time: 31 00 07 20 59 00:00:01.138 (472)
TY time:0:00:128 (7a14b67), AC3 PES time: 21 00 07 2c 1f 00:00:01.155 (483)
TY time:0:00:044 (2a78655), BFRAME PES time: 21 00 05 f1 6d 00:00:01.71 (42f)
TY time:0:00:077 (4a4a8e0), BFRAME PES time: 21 00 07 08 e3 00:00:01.104 (450)
TY time:0:00:160 (9899367), AC3 PES time: 21 00 07 42 9f 00:00:01.187 (4a3)
TY time:0:00:192 (b71db67), AC3 PES time: 21 00 07 59 1f 00:00:01.219 (4c3)
TY time:0:00:211 (c99330a), PFRAME PES time: 31 00 07 66 bb 00:00:01.238

The AC3 frames were ordered after the last I-frame or P-frame. In the mplex -f 10 output, it seems the AC3 frames are just output interleaved with the video frames. There is a MuxPossible(currentSCR) call but this is only used to check if the AC3 stream input has any frames in the buffer. The standard ac3 buf size is found in ac3strm_in.cpp:
const unsigned int AC3Stream::default_buffer_size = 16*1024;

I adjusted this up and around 50KB got the best results but it still was not correct. This is output from mplex with the default buf of 16KB:
TY time:0:00.011 (a7d8c0), VSH PES time: 31 00 01 1f 33 00:00:00.44 (2c)
TY time:0:00.011 (a7d8c0), GOP
TY time:0:00.011 (a7d8c0), IFRAME
TY time:0:00.000 (0), AC3 PES time: 21 00 01 17 77 00:00:00.33 (21)
Warning: Source ty stream already has complete PES header (no need to convert)
TY time:0:00.111 (69db9c0), PFRAME PES time: 31 00 01 65 95 00:00:00.144 (90)
TY time:0:00.032 (1e84800), AC3 PES time: 21 00 01 2d f7 00:00:00.65 (41)
TY time:0:00.044 (29f6300), BFRAME PES time: 21 00 01 36 a9 00:00:00.77 (4d)
TY time:0:00.064 (3d09000), AC3 PES time: 21 00 01 44 77 00:00:00.97 (61)
TY time:0:00.078 (4a62f80), BFRAME PES time: 21 00 01 4e 1f 00:00:00.111 (6f)
TY time:0:00.096 (5b8d800), AC3 PES time: 21 00 01 5a f7 00:00:00.129 (81)
TY time:0:00.211 (c939ac0), PFRAME PES time: 31 00 01 ab f7 00:00:00.244 (f4)

TY time:0:00.511 (1e753dc0), VSH PES time: 31 00 03 7f 1d 00:00:00.544 (220)
TY time:0:00.511 (1e753dc0), GOP
TY time:0:00.511 (1e753dc0), IFRAME
TY time:0:00.416 (18cba800), AC3 PES time: 21 00 03 3b f7 00:00:00.449 (1c1)
TY time:0:00.445 (1a862940), BFRAME PES time: 21 00 03 50 31 00:00:00.478 (1de)
TY time:0:00.448 (1ab3f000), AC3 PES time: 21 00 03 52 77 00:00:00.481 (1e1)
TY time:0:00.478 (1c7db380), BFRAME PES time: 21 00 03 67 a7 00:00:00.511 (1ff)
TY time:0:00.480 (1c9c3800), AC3 PES time: 21 00 03 68 f7 00:00:00.513 (201)
TY time:0:00.611 (246b1ec0), PFRAME PES time: 31 00 03 c5 7f 00:00:00.644 (284)
TY time:0:00.545 (207c0a40), BFRAME PES time: 21 00 03 96 93 00:00:00.578 (242)
TY time:0:00.578 (22739480), BFRAME PES time: 21 00 03 ae 09 00:00:00.611 (263)
TY time:0:00.712 (2a704200), PFRAME PES time: 31 00 05 0b e1 00:00:00.745 (2e9)
TY time:0:00.645 (2671eb40), BFRAME PES time: 21 00 03 dc f5 00:00:00.678 (2a6)
TY time:0:00.678 (28697580), BFRAME PES time: 21 00 03 f4 6b 00:00:00.711 (2c7)
TY time:0:00.812 (30662300), PFRAME PES time: 31 00 05 52 43 00:00:00.845 (34d)
TY time:0:00.512 (1e848000), AC3 PES time: 21 00 03 7f 77 00:00:00.545 (221)

Eventually, it would reach a point where it would not output an AC3 frame until much later.

Also, the current mplex TY record timestamp algorithm is to associate the record with the last VSH(MPEG I-FRAME) seen and does not account for the -O A/V sync offset parameter. The 810HS output seems to associate the TY record time with the PES timestamp.

[nova1]

I've added a patch5.txt which should fix the sync issues seen when generating a TY file from AVI or DVD sources.