PDA

View Full Version : TY enabled mplex for mpeg to TY stream conversion



bcc
02-27-2005, 05:06 PM
Here are some changes to mplex that allow it to generate TY streams as output. This is being presented as two sets of diffs to 1.6.2 version of the mplex program. Input to this program are elementary mpeg video and an audio stream. The audio stream may be either mpeg2 or ac3. For video, I've tested with resolutions of 1920x1088, 1280x720, 480x480, and 720x480, and all work with my hr10-250. So you can use this mpeg->ty conversion starting with DVD, previously converted tivo mpeg, SVCD, or synthetic video streams (avisynth for example).

We haven't yet looked into whether audio sync drifts for longer recordings. Good chance there is an issue there.
I've included linux and windows binaries for mplex.

CREDITS:
The ifdef TY part is nearly all written (and GPL licensed) by nova1. The second part is taken from my ty1to2 program, with the copy_rec() chunk spanning changes that nova1 previously posted. I contributed some c++ vs c glue and some of the AC3 audio support as well. Thanks also to rung for some early discussions.
USAGE:
Use '-f 10' argument to mplex to select TY stream format output. For example, below I take some HD ty, convert to mpeg, and back:
% hdemux -i Fox-OC.ty -v f.m2v -a f.m2a
Version 0.9
AC3 audio at 48kHz, 1792 bytes/sync frame
Video frame size is 1280x720 - high definition
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
Audio timestamp offset: 513.533333 ms
Reported datarate 65448 Kbit/sec. (65000+448)
Proceed with remuxer:
mplex -f 8 -O 513ms -r 153701 f.m2a f.m2v -o <outfile>.mpg
426688 audio stream bytes 10857140 video stream bytes
% mplex -f 10 -O 513ms f.m2a f.m2v -o f.ty
INFO: [lt-mplex] mplex version 1.6.2 (2.2.3 $Date: 2004/01/13 20:45:26 $)
INFO: [lt-mplex] File f.m2a looks like an AC3 Audio stream.
INFO: [lt-mplex] File f.m2v looks like an MPEG Video stream.
INFO: [lt-mplex] Video stream 0: profile 10 selected - ignoring non-standard options!
INFO: [lt-mplex] Found 1 audio streams and 1 video streams
INFO: [lt-mplex] Selecting TY and generic MPEG2 output profile
INFO: [lt-mplex] Multiplexing video program stream!
INFO: [lt-mplex] Scanning for header info: AC3 Audio stream 00 (f.m2a)
INFO: [lt-mplex] AC3 frame size = 1792

INFO: [lt-mplex] AC3 AUDIO STREAM:
INFO: [lt-mplex] Bit rate : 57344 bytes/sec (448 kbit/sec)
INFO: [lt-mplex] Frequency : 48000 Hz
INFO: [lt-mplex] Scanning for header info: Video stream e0 (f.m2v)
INFO: [lt-mplex] VIDEO STREAM: e0
INFO: [lt-mplex] Frame width : 1280
INFO: [lt-mplex] Frame height : 720
INFO: [lt-mplex] Aspect ratio : 16:9 display
INFO: [lt-mplex] Picture rate : 59.940 frames/sec
INFO: [lt-mplex] Bit rate : 65000000 bits/sec
INFO: [lt-mplex] Vbv buffer size : 999424 bytes
INFO: [lt-mplex] CSPF : 0
INFO: [lt-mplex] SYSTEMS/PROGRAM stream:
INFO: [lt-mplex] rough-guess multiplexed stream data rate : 66801896
INFO: [lt-mplex] Setting best-guess data rate.
INFO: [lt-mplex] Run-in Sectors = 998 Video delay = 68199 Audio delay = 26533
INFO: [lt-mplex] New sequence commences...
INFO: [lt-mplex] Audio bd: buf= 16384 frame=000000 sector=00000000
INFO: [lt-mplex] Video e0: buf=2719744 frame=000000 sector=00000000
INFO: [lt-mplex] STREAM e0 completed @ frame 150.
INFO: [lt-mplex] STREAM bd completed @ frame 235.
INFO: [lt-mplex] Multiplex completion at SCR=677450.
INFO: [lt-mplex] Audio bd: buf= 256 frame=000235 sector=00000236
INFO: [lt-mplex] Video e0: buf=2719744 frame=000150 sector=00000171
INFO: [lt-mplex] AUDIO_STATISTICS: bd
INFO: [lt-mplex] Audio stream length 422912 bytes.
INFO: [lt-mplex] Frames : 236
INFO: [lt-mplex] BUFFERING min 256 Buf max 256
INFO: [lt-mplex] VIDEO_STATISTICS: e0
INFO: [lt-mplex] Video Stream length: 10855544 bytes
INFO: [lt-mplex] Sequence headers: 11
INFO: [lt-mplex] Sequence ends : 0
INFO: [lt-mplex] No. Pictures : 151
INFO: [lt-mplex] No. Groups : 11
INFO: [lt-mplex] No. I Frames : 11 avg. size 82887 bytes
INFO: [lt-mplex] No. P Frames : 40 avg. size114992 bytes
INFO: [lt-mplex] No. B Frames : 101 avg. size 52686 bytes
INFO: [lt-mplex] Average bit-rate : 22830800 bits/sec
INFO: [lt-mplex] Peak bit-rate : 28366400 bits/sec
INFO: [lt-mplex] BUFFERING min 192 Buf max -11760
INFO: [lt-mplex] MUX STATUS: no under-runs detected.
INFO: [lt-mplex] Generating master chunk for chunks 0:86

%Latest version is here:
http://www.dealdatabase.com/forum/showpost.php?p=226704&postcount=63

bcc
02-27-2005, 05:17 PM
Download mplex 1.6.2 source from:http://prdownloads.sourceforge.net/mjpeg/mjpegtools-1.6.2.tar.gz?download
Project page:http://mjpeg.sf.net
patch with:
patch -p1 < patch1.txt
patch -p1 < patch2.txt
and for gcc 3.4 users, you probably also need this enclosed patch:
patch -p1 < patch-gcc.txt

bcc
02-28-2005, 05:08 AM
Ok, I've added a windows binary.
Now, does anybody know of a good mpeg2 encoder for HD? I've been unable to get tmpgenc or cce to encode 1920x1080. If I could get that to work I could insert some top quality test patterns into the tivo. The HDNet test pattern is only 1280x1080 so it's not a good resolution test.

bcc
02-28-2005, 01:18 PM
Looks like someone's support questions got deleted. Lets try a support thread over here (http://www.dealdatabase.com/forum/showthread.php?t=42006)

nova1
02-28-2005, 08:37 PM
Here are some diffs on bcc's mplex with patch1 and 2 applied. These patches handle the case where an MPEG SEQUENCE_END header is seen and where a GOP header is not preceded by a SEQUENCE_HEADER(seen on a couple of VCDs).

Edit: Forgot to remove some code when handling the seq_pos == -1 case.

Added patch4 which adds a '-x xml-file' option to read in a TY+ XML file and adjust the StreamFileSize setting and then append it to the TY file.

Edit: forgot to include xml.c from ty1to2-1.1 in patch4.txt

Edit: added patch5.txt to generate better TY record timestamps. This should fix the sync issues seen when generating TY files from AVI and DVD sources.

Edit: uploaded new patch5.txt that changes definition of TIVO_SD_PER_CHUNK and modified time_validate().

Edit: uploaded patch6.txt that fixes bug in copy_rec() function that manifests itself when the current chunk has a VID_SEQ record already and the last record is a VID_SEQ record which continues on to the next chunk.

All the patches have been rolled into a larger patch file. Start at this thread http://www.dealdatabase.com/forum/showpost.php?p=234847&postcount=101

bcc
03-01-2005, 04:25 AM
Ok, I've built 0.2 versions that include the patch3 changes.

nova1
03-01-2005, 10:55 AM
I was working with a 3.2GB file under Fedora Core 2 and I needed to replace the fopen() calls in main.cpp with fopen64() to get rid of the file too large error.

Jamie
03-01-2005, 11:35 AM
I was working with a 3.2GB file under Fedora Core 2 and I needed to replace the fopen() calls in main.cpp with fopen64() to get rid of the file too large error.
It's usually better to enable LFS through the gcc command line, typically in your Makefile:
CFLAGS += -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCELarge File Support in Linux (http://www.suse.de/~aj/linux_lfs.html).

nova1
03-01-2005, 12:52 PM
It's usually better to enable LFS through the gcc command line, typically in your Makefile:
CFLAGS += -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCELarge File Support in Linux (http://www.suse.de/~aj/linux_lfs.html).
Thanks. I'll look to see what i did wrong.
I added the following to bbtool19 bits.cpp to dmux the vob and the fopen() still would not work unless I changed it to fopen64().


#define __USE_FILE_OFFSET64
#define _LARGEFILE64_SOURCE
#define __USE_LARGEFILE64
#define __REDIRECT

Jamie
03-01-2005, 01:18 PM
I added the following to bbtool19 bits.cpp to dmux the vob and the fopen() still would not work unless I changed it to fopen64().


#define __USE_FILE_OFFSET64
...
__USE_FILE_OFFSET64 doesn't look right to me.

On a POSIX system, getconf LFS_CFLAGS should tell you the right flags to use on your platform. If it's like mine (FC3), you'll get:
fedora[101] ~ % getconf LFS_CFLAGS
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64A typical thing to do for portability is to wrap this in the Makefile:
CFLAGS += `getconf LFS_CFLAGS`

bcc
03-01-2005, 01:26 PM
Here are some diffs on bcc's mplex with patch1 and 2 applied. These patches handle the case where an MPEG SEQUENCE_END header is seen and where a GOP header is not preceded by a SEQUENCE_HEADER(seen on a couple of VCDs).

Apply using patch -p1 <patch3.txt.In the seq_pos = -1 case, the patch is bad: the code still falls through to the original memcpy.

bcc
03-01-2005, 01:31 PM
__USE_FILE_OFFSET64 doesn't look right to me.

On a POSIX system, getconf LFS_CFLAGS should tell you the right flags to use on your platform. If it's like mine (FC3), you'll get:
fedora[101] ~ % getconf LFS_CFLAGS
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64A typical thing to do for portability is to wrap this in the Makefile:
CFLAGS += `getconf LFS_CFLAGS`We need something that is portable to windoze as well.

bcc
03-01-2005, 01:32 PM
Also I'm already setting _LARGEFILE64_SOURCE in the TY I/O code, and that's been working fine with open() under linux, cygwin, and microsoft visual studio.

Jamie
03-01-2005, 01:42 PM
We need something that is portable to windoze as well.That's messier. Here's a good reference (http://ac-archive.sourceforge.net/largefile/win32.html).

bcc
03-01-2005, 01:45 PM
Thanks. I'll look to see what i did wrong.
I added the following to bbtool19 bits.cpp to dmux the vob and the fopen() still would not work unless I changed it to fopen64().


#define __USE_FILE_OFFSET64
#define _LARGEFILE64_SOURCE
#define __USE_LARGEFILE64
#define __REDIRECT
Try just #define _LARGEFILE64_SOURCE /* Needs to precede inclusion on features.h */
then #include <sys/types.h> before the other includes.

nova1
03-01-2005, 06:40 PM
In the seq_pos = -1 case, the patch is bad: the code still falls through to the original memcpy.
Does this imply that the video stream does not have any sequence header? mplex searches for a sequence header in videostrm_in.cpp ScanFirstSeqHeader() and then fills in the horiz, vert size and other info. The code creates a sequence header of 12 bytes from the first sequence header seen and stores it into seqhdr[]. in multiplexor.cpp, if it sees that the seq_pos == -1 then it'll use this seqhdr for the memcpy. The code asserts if gop_pos or pic_pos == -1 so it should always have a sequence header if there was a first one.

bcc
03-01-2005, 06:54 PM
Does this imply that the video stream does not have any sequence header? mplex searches for a sequence header in videostrm_in.cpp ScanFirstSeqHeader() and then fills in the horiz, vert size and other info. The code creates a sequence header of 12 bytes from the first sequence header seen and stores it into seqhdr[]. in multiplexor.cpp, if it sees that the seq_pos == -1 then it'll use this seqhdr for the memcpy. The code asserts if gop_pos or pic_pos == -1 so it should always have a sequence header if there was a first one.I haven't seen a failure, I was just noticing that if seq_pos == -1, then the code after the if-then still uses the negative value in its pointer arithmetic which is surely not what you intended.

nova1
03-01-2005, 08:23 PM
Ahh, I forgot to delete those lines after cut-n-pasting. I've uploaded a new patch3.txt that deletes the memcpy() after the seq_pos == -1 check. That probably explains the version 0.2 failures.

nova1
03-02-2005, 10:35 AM
I'm thinking of changing tymaster.c generate_gop() to allow for master chunks without any GOP timestamp records in it if we have more than 1 master chunk in the file.


extern out_stream_t *outs;
assert(i >= 0 || (i < 0 && outs->chnk > CHUNK_VALID_PER_SEG));
if (i >= 0) {
put64((u_char *)&gop_stamps[i], gop_stamps[i]);
write_internal(ofd, (u_char *)&gop_stamps[i], sizeof(tivo_tstamp_t));
}


I also added CFLAGS += -D_FILE_OFFSET_BITS=64 to the utils/Makefile to handle the large TY output file(3GB).

bcc
03-02-2005, 03:35 PM
Here's the build environment I've been using for mplex under cygwin. mplex's automake and shared library environment seems to be pretty broken in 1.6.2 so I had to resort to drastic measures. I wrote my own makefiles (included), and moved the source to the following layout:

Top level: C++ code, includes
Utils dir: c code, private includes
include dir: shared includes

See file directory in the zip file for layout details, and makefile.

PS: Pick up the revised patch2.txt if you want to compile for windows - I had 1 booboo in the initial copy.

bcc
03-02-2005, 03:48 PM
I'm thinking of changing tymaster.c generate_gop() to allow for master chunks without any GOP timestamp records in it if we have more than 1 master chunk in the file.Huh? I doubt it is "legal" to have a master chunk record without any GOP timestamps. In the case that there are no GOP bits to set in the table, you would still have the trailing timestamp I believe. But, do you really have a case where there is a segment with no TY sequence records?

rewrite_master() already handles updating multiple master chunks. It assumes the source stream has layed out placeholder master chunks for it to rewrite. Is there a problem?

I also added CFLAGS += -D_FILE_OFFSET_BITS=64 to the utils/Makefile to handle the large TY output file(3GB).Is this something specific to the fopen()? Maybe this should just be replaced with an open() as I've already provided OS portable largefile support for that (tyos.h).

nova1
03-02-2005, 04:21 PM
Huh? I doubt it is "legal" to have a master chunk record without any GOP timestamps. In the case that there are no GOP bits to set in the table, you would still have the trailing timestamp I believe. But, do you really have a case where there is a segment with no TY sequence records?

rewrite_master() already handles updating multiple master chunks. It assumes the source stream has layed out placeholder master chunks for it to rewrite. Is there a problem?Is this something specific to the fopen()? Maybe this should just be replaced with an open() as I've already provided OS portable largefile support for that (tyos.h).

I was redoing the dvd ty file and I got


INFO: [lt-mplex] Generating master chunk for chunks 16384:16384
lt-mplex: tymaster.c:155: generate_gop: Assertion `i >= 0' failed.

so I thought since we are now packing the data into the TY chunks, it's possible to have a continuation record at the start of the next master chunk. I then found out that I needed to rebuild the utils lib with the CFLAGS. I did not see the assertion after rebuilding for large file support. So it isn't an issue I've seen. I believe in the ty1to2 thread, there was a note on lseek() not quite working but pos has been redefined as off_t.

bcc
03-02-2005, 07:19 PM
INFO: [lt-mplex] Generating master chunk for chunks 16384:16384
lt-mplex: tymaster.c:155: generate_gop: Assertion `i >= 0' failed.Ok, you're right it shouldn't assert in this case. Looks like this case is possible when your last segment only has 1 or 2 data chunks. I think the code should look like:
stamp = 0;
if (i >= 0) {
stamp = gop_stamps[i];
}
/* Write out final timestamp */
put64(&stamp, stamp);
write_internal(ofd, (u_char *)&stamp, sizeof(tivo_tstamp_t));So that we write the final timestamp regardless.
I did not see the assertion after rebuilding for large file support.Makes sense since the chunk #, 16384, is at 2^31. But actually, this file already had laregfile support thanks to tyos.h, so the largefile issue was probably with the mplex stream reading code.
I believe in the ty1to2 thread, there was a note on lseek() not quite working but pos has been redefined as off_t.Yes the lseek() issue was taken care of and isn't a problem in any of the newer code you/I have been tossing about. I also took care of the 64 bit compiler issues.

nova1
03-19-2005, 11:52 AM
I added a patch4.txt that incorporates the xml code from ty1to2-1.1 so you can add the -x xml-file option to mplex to tack on a TY+ XML file at the end of the TY output file.

bcc
03-19-2005, 04:57 PM
Hi, I'm back from my trip & almost recovered.
Looks like your patch is missing xml.c. I guess this is useful for the ty->mpg->ty case, but for the mpg->ty case, where you don't have some initial xml already, it'd be real cool if we could auto-generate (and/or prompt for) the important XML fields.

I'm a bit surprised over the interest in using tymplex for ty->mpg->ty, other than in testing. If that was the ultimate goal, the code could be written a lot differently. For example, we could keep the real tivo timestamps, and also leave the streams interleaved at the rate found in the initial ty stream. Ie basically leave out the middle step and just process the ty against a GOP-accurate cut-list. The point of ty->mpg->ty is just to get GOP-accurate editing of the ty stream, for playback on the tivo, right?

darrin75
03-19-2005, 05:44 PM
I never could to get this work right..I try it and the video size is 0 bytes..

nova1
03-19-2005, 05:52 PM
ok, I've uploaded a new patch4.txt that includes xml.c from ty1to2-1.1.

I didn't realize I had the "Priority Set..." string right before the '#' marker in my test ty+ file so ty1to2 just saved the xml file. I just took the file and then hand modified certain fields for use with a different upload. Uploads don't happen
that frequently for me. You can also edit fields once on the Tivo using
the universaledititle.tcl script which is probably easier or as you say, have mplex prompt for this info.

In my limited testing, I haven't seen any issues doing ty->mpeg->ty or going from ty->tytools edit->mpeg program stream->demux a,v -> mplex->ty. I use ty2ty(modified ty1to2-1.1) that retains Tivo timestamps for testing and as a chunk editor using the -s and -n options.

bcc
03-19-2005, 11:36 PM
I never could to get this work right..I try it and the video size is 0 bytes..Could you send some details to the support thread then? Output from mplex, kind of source file (ty file?), command line, how you're creating the .m2v/.m2a. Perhaps upload the .m2a/.m2v and or the .ty or .mpg. You could use tychopper to shrink the sample to a minimal size if the source was .ty.
http://www.dealdatabase.com/forum/showthread.php?t=42006

nova1
04-04-2005, 11:35 AM
I looked at a sample 720x480 clip provided by T u r b o that works on both a HDVR2 and a 810HS. It looked like a standard transmission order frame list of IPBBPBB:

TY time:0:00:000 (0), AC3 PES time: 21 00 05 d2 1d 00:00:01.27 (403)
TY time:0:00:000 (0), VSH PES time: 31 00 05 d9 f7 00:00:01.38 (40e)
TY time:0:00:000 (0), GOP
TY time:0:00:011 (aa63ca), IFRAME
TY time:0:00:032 (1e87367), AC3 PES time: 21 00 05 e8 9f 00:00:01.59 (423)
TY time:0:00:064 (3d0bb67), AC3 PES time: 21 00 05 ff 1f 00:00:01.91 (443)
TY time:0:00:096 (5b90367), AC3 PES time: 21 00 07 15 9f 00:00:01.123 (463)
TY time:0:00:111 (6a1cb6a), PFRAME PES time: 31 00 07 20 59 00:00:01.138 (472)
TY time:0:00:128 (7a14b67), AC3 PES time: 21 00 07 2c 1f 00:00:01.155 (483)
TY time:0:00:044 (2a78655), BFRAME PES time: 21 00 05 f1 6d 00:00:01.71 (42f)
TY time:0:00:077 (4a4a8e0), BFRAME PES time: 21 00 07 08 e3 00:00:01.104 (450)
TY time:0:00:160 (9899367), AC3 PES time: 21 00 07 42 9f 00:00:01.187 (4a3)
TY time:0:00:192 (b71db67), AC3 PES time: 21 00 07 59 1f 00:00:01.219 (4c3)
TY time:0:00:211 (c99330a), PFRAME PES time: 31 00 07 66 bb 00:00:01.238

The AC3 frames were ordered after the last I-frame or P-frame. In the mplex -f 10 output, it seems the AC3 frames are just output interleaved with the video frames. There is a MuxPossible(currentSCR) call but this is only used to check if the AC3 stream input has any frames in the buffer. The standard ac3 buf size is found in ac3strm_in.cpp:
const unsigned int AC3Stream::default_buffer_size = 16*1024;

I adjusted this up and around 50KB got the best results but it still was not correct. This is output from mplex with the default buf of 16KB:
TY time:0:00.011 (a7d8c0), VSH PES time: 31 00 01 1f 33 00:00:00.44 (2c)
TY time:0:00.011 (a7d8c0), GOP
TY time:0:00.011 (a7d8c0), IFRAME
TY time:0:00.000 (0), AC3 PES time: 21 00 01 17 77 00:00:00.33 (21)
Warning: Source ty stream already has complete PES header (no need to convert)
TY time:0:00.111 (69db9c0), PFRAME PES time: 31 00 01 65 95 00:00:00.144 (90)
TY time:0:00.032 (1e84800), AC3 PES time: 21 00 01 2d f7 00:00:00.65 (41)
TY time:0:00.044 (29f6300), BFRAME PES time: 21 00 01 36 a9 00:00:00.77 (4d)
TY time:0:00.064 (3d09000), AC3 PES time: 21 00 01 44 77 00:00:00.97 (61)
TY time:0:00.078 (4a62f80), BFRAME PES time: 21 00 01 4e 1f 00:00:00.111 (6f)
TY time:0:00.096 (5b8d800), AC3 PES time: 21 00 01 5a f7 00:00:00.129 (81)
TY time:0:00.211 (c939ac0), PFRAME PES time: 31 00 01 ab f7 00:00:00.244 (f4)

TY time:0:00.511 (1e753dc0), VSH PES time: 31 00 03 7f 1d 00:00:00.544 (220)
TY time:0:00.511 (1e753dc0), GOP
TY time:0:00.511 (1e753dc0), IFRAME
TY time:0:00.416 (18cba800), AC3 PES time: 21 00 03 3b f7 00:00:00.449 (1c1)
TY time:0:00.445 (1a862940), BFRAME PES time: 21 00 03 50 31 00:00:00.478 (1de)
TY time:0:00.448 (1ab3f000), AC3 PES time: 21 00 03 52 77 00:00:00.481 (1e1)
TY time:0:00.478 (1c7db380), BFRAME PES time: 21 00 03 67 a7 00:00:00.511 (1ff)
TY time:0:00.480 (1c9c3800), AC3 PES time: 21 00 03 68 f7 00:00:00.513 (201)
TY time:0:00.611 (246b1ec0), PFRAME PES time: 31 00 03 c5 7f 00:00:00.644 (284)
TY time:0:00.545 (207c0a40), BFRAME PES time: 21 00 03 96 93 00:00:00.578 (242)
TY time:0:00.578 (22739480), BFRAME PES time: 21 00 03 ae 09 00:00:00.611 (263)
TY time:0:00.712 (2a704200), PFRAME PES time: 31 00 05 0b e1 00:00:00.745 (2e9)
TY time:0:00.645 (2671eb40), BFRAME PES time: 21 00 03 dc f5 00:00:00.678 (2a6)
TY time:0:00.678 (28697580), BFRAME PES time: 21 00 03 f4 6b 00:00:00.711 (2c7)
TY time:0:00.812 (30662300), PFRAME PES time: 31 00 05 52 43 00:00:00.845 (34d)
TY time:0:00.512 (1e848000), AC3 PES time: 21 00 03 7f 77 00:00:00.545 (221)

Eventually, it would reach a point where it would not output an AC3 frame until much later.

Also, the current mplex TY record timestamp algorithm is to associate the record with the last VSH(MPEG I-FRAME) seen and does not account for the -O A/V sync offset parameter. The 810HS output seems to associate the TY record time with the PES timestamp.

nova1
04-07-2005, 12:56 PM
I've added a patch5.txt which should fix the sync issues seen when generating a TY file from AVI or DVD sources.

mbellot
04-07-2005, 01:40 PM
I've added a patch5.txt which should fix the sync issues seen when generating a TY file from AVI or DVD sources.

Any chance of getting an updated windows binary?

I'm still trying to sort out what I need from the cygwin install to rebuild this myself (it looks like most of the development stuff is NOT installed by default). :eek:

bcc
04-08-2005, 01:53 PM
So you found that adding DTS is necessary for the 810hs, or is that change necessary for all models (even tho most models don't store a DTS on their own)?

nova1
04-08-2005, 09:04 PM
DTS is not needed. I just left it in. The streams are now multiplexed according to the PTS.

mbellot,
you'll need gcc-core, gcc-g++, make, libpcre0, patch, jpeg, libxml2, libxml2-devel, libiconv packages. use the setup.exe utility to grab them in
addition to the base.

mbellot
04-08-2005, 11:33 PM
DTS is not needed. I just left it in. The streams are now multiplexed according to the PTS.

mbellot,
you'll need gcc-core, gcc-g++, make, libpcre0, patch, jpeg, libxml2, libxml2-devel, libiconv packages. use the setup.exe utility to grab them in
addition to the base.

Thanks! I'll queue it all up and let it rip while I sleep.

mbellot
04-09-2005, 08:55 PM
DTS is not needed. I just left it in. The streams are now multiplexed according to the PTS.

mbellot,
you'll need gcc-core, gcc-g++, make, libpcre0, patch, jpeg, libxml2, libxml2-devel, libiconv packages. use the setup.exe utility to grab them in
addition to the base.

Also need nasm, or ./configure says performance "will be crap".

Thanks again.

mbellot
04-10-2005, 12:14 AM
Now I'm getting errors when I try to compile.

I downloaded all the packages you listed plus nasm and installed it all to c:\cygwin

I put the source for mjpegtools in /usr/src/mjpegtools-1.6.2 using gunzip and tar and applied all five patches from /usr/src (patch -p1 <patchN.txt).

I then ran configure from the mjpegtools "root" directory and let it do its thing.

When it finished I typed make and hit enter. It goes through several directories but dies while building mplex with a bunch of errors like the following:



../mplex/.libs/libmplex2.a(libmplex2_la-lpcmstrm_in.o)(.text+0x5d8):/usr/src/mjp
egtools-1.6.2/mplex/lpcmstrm_in.cpp:186: undefined reference to `_mjpeg_info'
../mplex/.libs/libmplex2.a(libmplex2_la-lpcmstrm_in.o)(.text+0x5ee):/usr/src/mjp
egtools-1.6.2/mplex/lpcmstrm_in.cpp:187: undefined reference to `_mjpeg_info'
../mplex/.libs/libmplex2.a(libmplex2_la-lpcmstrm_in.o)(.text+0x617): In function
`_ZN10LPCMStream13OutputHdrInfoEv':
/usr/src/mjpegtools-1.6.2/mplex/lpcmstrm_in.cpp:199: undefined reference to `_mj
peg_info'
../mplex/.libs/libmplex2.a(libmplex2_la-lpcmstrm_in.o)(.text+0x640):/usr/src/mjp
egtools-1.6.2/mplex/lpcmstrm_in.cpp:201: undefined reference to `_mjpeg_info'
../mplex/.libs/libmplex2.a(libmplex2_la-lpcmstrm_in.o)(.text+0x656):/usr/src/mjp
egtools-1.6.2/mplex/lpcmstrm_in.cpp:203: more undefined references to `_mjpeg_in
fo' follow
../mplex/.libs/libmplex2.a(libmplex2_la-dtsstrm_in.o)(.text+0x35f): In function
`_ZN9DTSStream4InitEi':
/usr/src/mjpegtools-1.6.2/mplex/vector.hpp:18: undefined reference to `_mjpeg_er
ror_exit1'


The two that seem to occur regularly are undefined references to _mjpeg_info and _mjpeg_error_exit1

The really weird thing is that there are references to these defines in other areas that get compiled just fine (like in jpeg2yuv.exe in the lavtools directory).

Any ideas? Did I screw up the order of something when patching/configuring?

BTW - I already tried deleting the entire source tree and starting over, but I get the same result.

bcc
04-10-2005, 05:32 AM
I never got the stock mplex 1.6.2 build environment to work under windows. See http://www.dealdatabase.com/forum/showpost.php?p=213393&postcount=20
for what I did.

nova1
04-10-2005, 09:04 AM
I don't think you need nasm for mplex.

Here are some notes on what I did:
untar mjpegtools-1.6.2.
run ./configure.
apply patches 1-5.
copy build-env-win Makefiles into mjpegtools-1.6.2/mplex and utils directories.
need ln -s ../utils done in mplex subdir.
Edit utils/Makefile
LIB+=xml.$(OBJ)
CFLAGS+=-I. -I.. -I/usr/include/libxml2

Edit mplex/Makefile
LIB+=xml.$(OBJ)
CFLAGS+=-I.. -Iutils -I/usr/include/libxml2
Also add -lxml2 at end of compile cmd for mplex target.
${CC+} ${CFLAGS} ${LDFLAGS} -o $@ $^ -lxml2

mbellot
04-10-2005, 03:38 PM
Thanks Nova, only one odd thing now...



tymaster.c: In function `rewrite_master':
tymaster.c:207: warning: integer constant is too large for "long" type


I'm not sure why its being reported as too large for "long", since secs_per_chunk is defined as type tivo_tstamp_t, which is in turn defined as type uint64_t. I even added a check that printed the sizeof(tivo_tstamp_t) and it came back as 8 (bytes).

I assume since its a warning its OK (?), but just thought I'd run it past the experts first.

Its the only error/warning I see in the entire process, and I end up with an mplex.exe so it looks like everything worked.

Thanks for all the help, I guess I am just way too used to the Visual Studio interface. :eek:

bcc
04-11-2005, 03:53 AM
I'm not sure why its being reported as too large for "long", since secs_per_chunk is defined as type tivo_tstamp_t, which is in turn defined as type uint64_t. I even added a check that printed the sizeof(tivo_tstamp_t) and it came back as 8 (bytes).Yes, seems a bit silly for gcc to issue a warning in this case when there is no possible overflow. ALso looks like it's not easy to portably mark the constant as 64 bit. Now, it looks like nova1 is just bumping up that constant to a nonsense value so that time_validate() doesn't do anything. Personally I would have done this differently, such as making secs_per_chunk = 0 a flag to indicate that time_validate() need not be called.
I assume since its a warning its OK (?), but just thought I'd run it past the experts first.Yes, that's the bottom line.
Its the only error/warning I see in the entire process, and I end up with an mplex.exe so it looks like everything worked.So I assume you got it all working.

bcc
04-11-2005, 04:03 AM
Could this be ported to visual studio..Yes, but it could be rather thankless maintaining build environment changes to (third party) mjpegtools just so that it can be compiled with visual studio. That build environment wouldn't be portable either.

In other words, be my guest. :) mjpegtools has been compiled with visual studio before so it can't be hard.

mbellot
04-11-2005, 09:54 AM
Yes, seems a bit silly for gcc to issue a warning in this case when there is no possible overflow. ALso looks like it's not easy to portably mark the constant as 64 bit. Now, it looks like nova1 is just bumping up that constant to a nonsense value so that time_validate() doesn't do anything. Personally I would have done this differently, such as making secs_per_chunk = 0 a flag to indicate that time_validate() need not be called.Yes, that's the bottom line.So I assume you got it all working.

So we could change the time_validate check to:



if ((!prev) | (!delta_per_chunk)) {


And then change



#define TIVO_SD_PER_CHUNK 0xffffffffffffffff /* # secs. of SD data per chunk */


to



#define TIVO_SD_PER_CHUNK 0x0 /* # secs. of SD data per chunk */


I realize it still requires the call(s) to time_validate, but it does seem like a cleaner approach than trying to max out the time counter.

For anyone that wants to try it with patch5 integrated and the above change made here's the windows binary.

If I can figure out how to make a patch file I'll post that as well, assuming nova1 and bcc agree that the change I made to the time_validate routine is OK.

:cool:


Thanks again to both Nova1 and bcc for helping me get this set up.

nova1
04-11-2005, 11:13 AM
I've incorporated the change to TIVO_SD_PER_CHUNK and time_validate() and uploaded a new version of patch5.txt.

bcc
04-11-2005, 01:48 PM
f I can figure out how to make a patch file I'll post that as well, assuming nova1 and bcc agree that the change I made to the time_validate routine is OK.Two problems with that change: You used the wrong OR operator (the arithmetic one instead of the binary one), and so the if statement now does surprising things, if the delta_per_chunk arg. is passed in non-zero. And you're still changing the #define such that its semantics make no sense.

hxmiller
04-11-2005, 09:14 PM
For anyone that wants to try it with patch5 integrated and the above change made here's the windows binary.



Getting the error on the patch5 windows binary.

"The procedure piont_impure_ptr could not be located in the dynamic
linked library cygwin1.dll"

mbellot
04-11-2005, 10:39 PM
Getting the error on the patch5 windows binary.

"The procedure piont_impure_ptr could not be located in the dynamic
linked library cygwin1.dll"

You probably need to update your cygwin1.dll, I built it with the most recent cygwin distro - the others may not have.

mbellot
04-11-2005, 10:49 PM
Two problems with that change: You used the wrong OR operator (the arithmetic one instead of the binary one), and so the if statement now does surprising things, if the delta_per_chunk arg. is passed in non-zero.

:o I guess my C is even rustier than I would have liked to believe. I spend most of my day designing hardware, so crossing over to the dark side is usually only under duress (unless I'm playing around, like now). On the bright side (and from the brighter coder) nova1 got the correct (||) operator into the new patch5.txt


And you're still changing the #define such that its semantics make no sense.

Now you have really lost me. I think it makes as much sense as it can given the context. Or are you saying there is a fixed number of seconds per chunk for SD data?

bcc
04-11-2005, 11:17 PM
Now you have really lost me. I think it makes as much sense as it can given the context. Or are you saying there is a fixed number of seconds per chunk for SD data?Yes. Likewise for HD. If I rewrite the constants to be a function of the max. audio+video stream data rate, then it'd be more obvious.

bcc
04-11-2005, 11:36 PM
You probably need to update your cygwin1.dll, I built it with the most recent cygwin distro - the others may not have.Why don't you post the one you used, since it's not obvious, and otherwise folks just have to hunt down what version they think you might have used.
Shesh, hardware engineers :)

mbellot
04-11-2005, 11:39 PM
Yes. Likewise for HD. If I rewrite the constants to be a function of the max. audio+video stream data rate, then it'd be more obvious.

So then they are not really constants per se, since the data rate changes with resolution (quality, whatever you like to call it).

If thats true then it does seem wrong to ignore the fact (that seems to be what we are doing...)

mbellot
04-11-2005, 11:46 PM
Why don't you post the one you used, since it's not obvious, and otherwise folks just have to hunt down what version they think you might have used.
Shesh, hardware engineers :)

How hard is it to download the latest?

You lazy bit-heads always want things handed to you... :D

I am interested why my mplex.exe is just about double the 0.3 version, but not interested enough to try decyphering your makefile to see if its got some weird debug mode turned on.


And in the vein (of calling people lazy), how about you go ahead and re-write the datastream stuff as a function instead of constants? :eek:

gottahavit
04-11-2005, 11:54 PM
using that cygwin dll it now complains itwant cygxml2-2.dll. Does the new cygwin have other dependencies or are you using stuff from other libraries?

bcc
04-12-2005, 12:51 AM
How hard is it to download the latest?Much of the point of providing binary builds is to assist those users who don't know how to build source themselves. Most windows users fall into that category. They wouldn't know where to download the specific shared libraries your particular build happened to depend upon (of which there are several). Your idea of "latest" may mean latest pre-release, latest stable build, and besides "latest" doesn't stay correct for long.
I am interested why my mplex.exe is just about double the 0.3 version, but not interested enough to try decyphering your makefile to see if its got some weird debug mode turned on.You probably forgot to strip it.


And in the vein (of calling people lazy), how about you go ahead and re-write the datastream stuff as a function instead of constants? :eek:In the next release of master.c

mbellot
04-12-2005, 09:24 AM
You probably forgot to strip it.

Indeed? Who would have ever thought of such a thing... :eek:

I look forward to your next tymaster.c release, it will be interesting to see the details of the datastream chunking.

mbellot
04-12-2005, 09:32 AM
using that cygwin dll it now complains itwant cygxml2-2.dll. Does the new cygwin have other dependencies or are you using stuff from other libraries?

It probably complains because the xml stuff was added after the 0.3 binary was posted.

Here is cygxml2-2.dll and and updated mplex to correct the arithmetic vs. logical OR problem that bcc found.

hxmiller
04-12-2005, 03:41 PM
It probably complains because the xml stuff was added after the 0.3 binary was posted.

Here is cygxml2-2.dll and and updated mplex to correct the arithmetic vs. logical OR problem that bcc found.

It now complains about missing cygz.dll and cygiconv-2.dll.

I downloaded these binaries. Now it runs. It crashes with the following error:

INFO: [mplex] BUFFERING min 2658310 Buf max 2718545
INFO: [mplex] MUX STATUS: no under-runs detected.
assertion "major == VID_SEQ" failed: file "tymaster.c", line 91


Edit: It must have been a bad clip. I've been successful extracting, cutting, and reinserting. Great work!!

gottahavit
04-12-2005, 05:55 PM
Yeah there are several other dlls it needed, that I got from the distribution at which point it ran fine. Seems to work much better than the .3 binary the only obvious issue was a small audio sync problem. It started out of sync and stayed the same amount out all through.

I extracted from HDVR2 edited with tytools, split with tmpgenc and then used mplex and re-inserted to same HDVR2.

I will try your latest binary and see if it helps the audio issue, sure seems like this stuff is REAL close, great work!!!

gottahavit
04-12-2005, 07:49 PM
Yeah there are several other dlls it needed, that I got from the distribution at which point it ran fine. Seems to work much better than the .3 binary the only obvious issue was a small audio sync problem. It started out of sync and stayed the same amount out all through.

I extracted from HDVR2 edited with tytools, split with tmpgenc and then used mplex and re-inserted to same HDVR2.

I will try your latest binary and see if it helps the audio issue, sure seems like this stuff is REAL close, great work!!!

ok, that didn't help, it plays in sync for about 30 seconds then there is a quick jitter in the picture and then it is about 1/2 a second out of sync. It does play fine on the PC. I will try some other methods of producing the a/v streams.

AlphaWolf
04-13-2005, 01:52 AM
It's for this reason that I strongly suggest you use typrocess instead of tytool on the original tystream when you export it. Tytool only builds the tystreams so that they work well enough for DVD players; it doesn't correct certain types of stream errors that trigger sync loss when processed with standard mpeg software, thus it's very unlikely that you'll ever see proper sync with that video using the method you are currently exporting it with. Typrocess does fix these errors on the other hand. Also, if you want to do any editing on the typrocess'ed stream, I suggest you use videoredo.

BTW, these things should probably be in the support thread.

http://www.dealdatabase.com/forum/showpost.php?p=213436&postcount=38

nova1
05-04-2005, 01:13 AM
I posted a patch6.txt that fixes a bug that I recently encountered. The bug manifests itself when the current chunk already has a VID_SEQ record and the last record of this chunk is a VID_SEQ record and it continues on to the next chunk. The seq flag is not cleared so the next chunk and continuation record is marked as being a VID_SEQ record. mplex outputs this error "tymaster.c:91: parse_chunk: Assertion `major == 0x7'" because it is expecting the record type to be VID_SEQ but it is actually a VID_CONT record.

bcc
05-04-2005, 01:40 AM
I posted a patch6.txt that fixes a bug that I recently encountered. The bug manifests itself when the current chunk already has a VID_SEQ record and the last record of this chunk is a VID_SEQ record and it continues on to the next chunk. The seq flag is not cleared so the next chunk and continuation record is marked as being a VID_SEQ record. mplex outputs this error "tymaster.c:91: parse_chunk: Assertion `major == 0x7'" because it is expecting the record type to be VID_SEQ but it is actually a VID_CONT record.Ah right, it's now clear that your previous patch2 introduced this issue. I assume you saw the bug report from compwiz312 reporting this problem. Care to make linux&windows binaries?

mbellot
05-04-2005, 10:14 AM
Care to make linux&windows binaries?

Can't help with linux, but here's the windows binary.

bcc
06-19-2005, 05:25 PM
I've updated tymplex to generate XML on its own. This addresses the problems seen with the trick play bar seen upon insertion with mfs_ftp. The XML objects that are relevant to this include duration, starttime, and stoptime.
For example, in my test case, tymplex created a .ty file with the following XML at the end:
<?xml version="1.1" tivoversion="3.1.5e-01-2-357"?>
<Object type="Recording" id="_top">
<StreamFileSize>125568</StreamFileSize>
<Duration>76</Duration>
<StartTime>0</StartTime>
<StopTime>76</StopTime>
<CallSign>tymplex</CallSign>
<Object>
Enclosed are the changes to mplex as a single patch. (It was becoming difficult to maintain these as a set of logical patches). Also included are linux and windows binaries. For the windows binary, I compiled without libxml2, so -x won't quite work. This saves us from .dll dependency hell. The new XML generating code is probably more handy anyways.
Building from source for linux:
tar xzf mjpegtools-1.6.2.tar.gz
cd mjpegtools-1.6.2
patch -p1 < ../tymplex1.1.patch.txt
./configure
make

bcc
06-21-2005, 02:11 AM
I've simplified the build process for windows. Basically:
Unpack mjpegtools-1.6.2.tar.gz
cd mjpegtools-1.6.2
patch -p1 < tymplex1.2.patch.txt
unzip tymplex-build-win.zip
cd mplex
make
See attachments for the patch & zip files.

nova1
07-08-2005, 12:42 AM
bcc,
I think XML_FOOT should be defined as </Object> in tyxmladd.c.

bcc
07-08-2005, 02:53 AM
Oops, you're right. And technically the Duration and Callsign objects should be sub-subobjects, not at the level I put them at. But luckily mfs_ftp is populating the MFS db correctly without all that.

nova1
07-08-2005, 11:42 AM
Right. I just use a template and fill in the Title and Description and let
mplex -x rewrite/update the duration, stoptime and streamfilesize.


<?xml version="1.1" tivoversion="3.1.5f-01-2-357"?>
<Object type="Recording" id="_top">
<SubObject type="Showing" id="Showing">
<Date>12971</Date>
<Duration>600</Duration>
<Object type="Program" id="Program">
<DescLanguage>English</DescLanguage>
<Description></Description>
<Title></Title>
</Object>
<Object type="Station" id="Station">
<CallSign></CallSign>
<Name></Name>
</Object>
<Time>0</Time>
</SubObject>
<StartDate>12971</StartDate>
<StartTime>0</StartTime>
<StopDate>12971</StopDate>
<StopTime>600</StopTime>
<StreamFileSize>1680</StreamFileSize>
</Object>

bcc
07-08-2005, 01:39 PM
If you want to always have that extra xml sugar, you could just have the template compiled in so that you don't have to pass it in each time. Personally I like how tyxmladd avoids the dependency on XML support libraries.

nova1
07-22-2005, 09:04 PM
I found another bug in copy_rec(). The code should be


assert(((size+postpad) % 4) == 0); // Length must be multiple of 4
if (size) {
// Reset TY record length and retain major
putshort(&srchead[0],((size+postpad) >> 4) & 0xFFFF);
srchead[2] = ((size+postpad) & 0x0F) << 4 | (srchead[2] & 0x0F);
}

Otherwise it can't handle the TY records that have payload size 0 with extended data within the size bits.

bcc
07-22-2005, 11:52 PM
Otherwise it can't handle the TY records that have payload size 0 with extended data within the size bits.Makes sense if one were using this version of copy_rec() with ty1to2. But in the context of tymplex, when would we be synthesizing TY records with 0 payload and extended data? I can't see what actual problem this is solving for tymplex.

nova1
07-23-2005, 12:16 AM
It's useful if you want to generate Closed Caption TY records from the MPEG2 user data(0x000001b2) found on DVD's or from the satellite stream.

bcc
07-23-2005, 12:50 AM
So you're adding closed captioning support, at which point copy_rec() will need changing. Sounds good.

bcc
07-23-2005, 01:14 AM
BTW, Nova1, have you looked at the ffmpeg code? Looks to me to be a good deal cleaner than mplex. If one ported the TY encoding support to ffmpeg, as an encoding codec, this all could be easier to maintain. It could also remove the demux step requirement for users.

nova1
07-25-2005, 03:25 PM
I've glanced at the ffmpeg code but haven't done anything. I also lost the last 3 months of notes. I've given up on the DVD CC code. I'm able to search for the DVD MPEG2 private/user_data stream and extract the 2 bytes of CC data but I'm not sure where or what timestamp it should have. According to the SCC_TOOLS docs, the CC data is for the frames within this GOP. In my sample,
it seems to have more CC data then the B/P Frames in this GOP. It sort of worked when I assigned the tivo timestamp for the CC TY record to be the same
as the preceding I,B,P frame but some of the words didn't appear correctly
on the TV monitor.

nova1
07-28-2005, 03:21 PM
I looked at the DVD CC info again and created TY CC record timestamps based on the CC data record index. TY CC timestamp = TY GOP timestamp + 1E9*(index/29.97) since each CC record is per frame. This seems to work much better although I do see some issues. I'm currently outputting all of the TY CC records at once right before the TY GOP record. I now need to interleave them throughout the following frames within the GOP. I believe mplex is coalescing similar frames together and that's why I'm seeing less B and P frames than expected.

nova1
07-29-2005, 06:36 PM
Here's some code(along with the copy_rec() change) that'll generate TY CC data records from DVD CC data. It seems to work but I don't really know if what it is showing on the TV is correct or not as I'm not familiar with CC.



idmajor = 0x0c; // Video GOP Header
putshort(&vidhead[0],(total_packet_data_size >> 4) & 0xFFFF);
vidhead[2] = (total_packet_data_size & 0x0F) << 4;
vidhead[2] |= idmajor; // major
vidhead[3] = idtype; // type

// New Section
// Look for DVD CC data between GOP and IFRAME
int i, userdata_pos, dvdcc_pos;
uint32_t extdata;
uint8_t *cap, capcount, field, d0, d1;
uint8_t cchead[CHUNK_RECLEN];
clockticks cc_time;
#define START_CODE_USR (START_CODE_PREFIX | VID_CODE_USER_DATA)
#define START_CODE_CC 0x434301f8

userdata_pos = start_code_search(sector_buf,START_CODE_USR,
0,total_packet_data_size);
if (userdata_pos != -1) {
dvdcc_pos = start_code_search(sector_buf+userdata_pos,START_CODE_CC,
0,total_packet_data_size-userdata_pos);
if (dvdcc_pos != -1) {
capcount = (sector_buf[userdata_pos+8] >> 1) & 0x0f;
cap = &sector_buf[userdata_pos+9];
field = sector_buf[userdata_pos+8] >> 7;
// pattern == 0 means f2 then f1.
// pattern == 1 means f1 then f2.
// only f1 seems to be used so when pat==0 get bytes 4,5
// when pat==1 get bytes 1,2
for (i=0; i<capcount; i++) {
d0 = field ? cap[i*6+1] : cap[i*6+4];
d1 = field ? cap[i*6+2] : cap[i*6+5];
cc_time = tivo_timestamp + (clockticks) (TIVO_TIMESCALE*(i/29.97));
if (d0 != 0x80 || d1 != 0x80) {
memset(cchead,0,sizeof(cchead)); // zero out TY Record metadata
put64(&cchead[8],cc_time);
// 0x80000 means Embedded extended data
extdata = 0x80000 | (d0<<8) | d1;
putshort(&cchead[0],(extdata >> 4) & 0xFFFF);
cchead[2] = (extdata & 0x0f) << 4;
cchead[2] |= 0x0e; // major
cchead[3] = 0x01; // type Closed Caption
copy_rec(&cchead[0], 0, 0, FALSE);
}
} // end of for loop on capcount
}
}
// End of DVD CC data between GOP and IFRAME

nova1
07-29-2005, 09:34 PM
I noticed a small issue when analyzing a DVD. The calculation of the TY timestamp for a BFRAME is incorrect. The stream starts out with an IFRAME and then is followed by a BFRAME. However, this BFRAME has PTS earlier than the IFRAME. Since mplex uses the first video frame IFRAME as a starting reference point of 0, the TY timestamp is off. If this occurs, I just set the cur BFRAME timestamp to be the same as the first video pes timestamp.


cur_ts = PTS/(300*90);
// with a BFRAME, it is possible cur_ts < first_vid_pes_timestamp
if (cur_ts < first_vid_pes_timestamp)
cur_ts = first_vid_pes_timestamp;
tivo_timestamp = (cur_ts - first_vid_pes_timestamp) *
(TIVO_TIMESCALE/MSEC) + first_vid_tivo_timestamp;

johnsolo
07-29-2005, 10:53 PM
After patching mjpeg with bcc's patch, I did a ./configure, and then a make. It starts okay but dies with a string of errors. The first error is:


In file included from xml.c:31:
tydefs.h:22: error: parse error before "boolean"


lines 20 & 21 of tydefs.h:

typedef uint64_t pts_t; /* PES timestamps are 33 bits */
typedef uint64_t tivo_tstamp_t; /* Tivo timestamps are 64 bits */


gcc 4.0 gives the same error.

SteveT
08-12-2005, 05:19 PM
Here is a patch to make the -x <filename>.xml work (to import xml vs. using the internally-generated xml). The xml is expected to be the same as that downloaded by mfs_ftp. With this patch, mplex will now update the Duration, Times, etc. in the imported xml with calculated values for the generated .ty.

It is based on the tymplex1.2.patch.txt posted by bcc in this thread. No new libraries or dlls are required.

I use it to create edited .ty files which have all the episode (and other) details for use with tivoserver. Makes the MRV list much more useful and increases the WAF and KAF.

This is my first attempt at C, and my first patch file, so please let me know of any problems or successes.

Many thanks to bcc, nova1, and others who created and continue to enhance this killer app.

Jamie
08-14-2005, 08:19 PM
I'm using the mplex-linux from this (http://www.dealdatabase.com/forum/showpost.php?p=226704&postcount=63) post and am finding that it doesn't correctly update the master chunks and xml above 2GB. I tried both the binary there, and compiling it myself from source. This seems to be a Large File Support issue.

I know we did a round on LFS earlier (http://www.dealdatabase.com/forum/showthread.php?p=213085#post213085). It appears that _LARGEFILE64_SOURCE is set in utils/tyos.h, however, that by itself does not enable 64 bit seek offsets. You either need to use off64_t and lseek64, or define _FILE_OFFSET_BITS with value 64, so that the off_t and lseek are aliased to the 64 bit versions. See the descriptions of _LARGEFILE64_SOURCE and _FILE_OFFSET_BITS here (http://www.delorie.com/gnu/docs/glibc/libc_13.html). See also the "Using LFS" section here (http://www.suse.de/~aj/linux_lfs.html).

Since the package is already using autoconf, the easiest solution is to just let the configure script handle it: include <config.h> at the top of tyos.h. I did this and it now works fine for me (gcc 4.0 on FC4). I suspect it will solve the problem on cygwin too. I don't know about the visual C++ compiles.

Another LFS issue: the third argument to rewrite_xml in ty.c + tyexterns.h should be an off_t (assuming you're using the LFS options that force that to be 64 bit) rather than size_t. Otherwise the StreamFileSize is wrong in the xml for files > 4GB. This also affects originate_xml and copy_xml in SteveT's code.

bcc
08-15-2005, 04:57 AM
I'm using the mplex-linux from this (http://www.dealdatabase.com/forum/showpost.php?p=226704&postcount=63) post and am finding that it doesn't correctly update the master chunks and xml above 2GB.Nice, thanks for finding. You got me. In my defense, the LFS documentation doesn't make this too clear. In the doc, here (http://www.suse.de/~aj/linux_lfs.html), under "Using LFS" it lists 3 compile time choices saying you can choose "either" of them, and I set 2 of the 3 choices... Not that having to set 3 or 4 flags to turn on 1 feature is such a good API... :)

Jamie
08-15-2005, 01:00 PM
Not that having to set 3 or 4 flags to turn on 1 feature is such a good API... :)Blame Posix. It's in the IEEE Posix.2 standard.

There are really three different things you might want to do: use the old 32 bit api's (e.g. for compatability with libraries compiled with only the 32bit api), use the new 64 bit apis for everything, or mix and match using the 64 bit function names when you want the 64 bit version. You really only have to set one thing to get option 2: _FILE_OFFSET_BITS =64, or better yet, let autoconf handle it. Microsoft visual C/C++ doesn't attempt to be Posix compliant, AFAIK, so all bets are off on what works there.

I ran into another little bug:

In mplex/systems.hpp: line 131: (after applying tymplex1.1.patch.txt)
inline off_t SegmentSize() { output_strm.SegmentSize(); }should be
inline off_t SegmentSize() { return output_strm.SegmentSize(); }Thanks, BTW, for you work on the tool!

darrin75
08-18-2005, 03:09 AM
How to use steve's -x option. Using the x option I don't think is documented anywhere. Is it used in the cmd line during mplexing or after


mplex -f 10 VTS_01_1.m2v VTS_01_1.ac3 -o example.ty

results in a ty file.

Where does the x option come into play here?


thanks

SteveT
08-18-2005, 08:18 AM
How to use steve's -x option. Using the x option I don't think is documented anywhere. Is it used in the cmd line during mplexing or after
mplex -f 10 VTS_01_1.m2v VTS_01_1.ac3 -o example.ty
results in a ty file.
Where does the x option come into play here?
thanksJust add "-x example.xml" to the command above, where example.xml is your xml file.
mplex -f 10 VTS_01_1.m2v VTS_01_1.ac3 -o example.ty -x example.xmlmplex will read example.xml, replace the StreamFileSize, Duration, StartTime and StopTime values, and attach this to the end of the .ty file. Without the -x, mplex will create the xml internally with those same values.

makinrain
08-18-2005, 08:02 PM
Here is a patch to make the -x <filename>.xml work (to import xml vs. using the internally-generated xml). The xml is expected to be the same as that downloaded by mfs_ftp. With this patch, mplex will now update the Duration, Times, etc. in the imported xml with calculated values for the generated .ty.

Is this different than mplex_pl6 ?

Can you post the windows binary ?

Thx

darrin75
08-18-2005, 08:19 PM
yes its different the patch has to be applied and then recompiled. Will post a new binary later. ;)

stealthdave
08-22-2005, 01:49 AM
After patching mjpeg with bcc's patch, I did a ./configure, and then a make. It starts okay but dies with a string of errors. The first error is:


In file included from xml.c:31:
tydefs.h:22: error: parse error before "boolean"


lines 20 & 21 of tydefs.h:

typedef uint64_t pts_t; /* PES timestamps are 33 bits */
typedef uint64_t tivo_tstamp_t; /* Tivo timestamps are 64 bits */


gcc 4.0 gives the same error.
Just to confirm, I'm getting the same compile error on my Tiger setup. I'm able to compile mjpegtools on its own just fine through Fink.

bcc
08-22-2005, 03:17 PM
This is a simple OS porting issue regarding 64 bit integers. For unix based platforms, I wrote the code to expect stdint.h to define uint64_t. Apparently that doesn't happen in OSX. For windows, this was handled via the included win32/stdint.h. Perhaps there is a special define one needs with OSX to get the expected definitions out of stdint.h or maybe it would be easiest to use the win32/stdint.h for that platform. Someone with an OSX box needs to evaluate.

stealthdave
08-22-2005, 08:54 PM
Okay, I was able to compile tymplex on OS X Panther, but not Tiger, with the attached patch. All the patch does is add ${prefix}/include/libxml2 to the LDFLAGS, as opposed to just the hard-coded /usr/include/libxml2 path. This allows for libxml2 installations other than /usr, such as Fink's /sw path.

The patch is still not perfect, though. The compile will continue until it gets to mpeg2enc and crash here:

Making all in mpeg2enc
/bin/sh ../libtool --mode=link g++ -I/sw/include -L/sw/lib -o mpeg2enc mpeg2enc.o ../mpeg2enc/libmpeg2encpp.la ../utils/libmjpegutils.a -lpthread -lm
g++ -I/sw/include -o .libs/mpeg2enc mpeg2enc.o -Wl,-bind_at_load -L/sw/lib ../mpeg2enc/.libs/libmpeg2encpp.dylib ../utils/libmjpegutils.a -lpthread -lm
ld: Undefined symbols:
_xmlCheckVersion
_xmlCleanupParser
_xmlDocDumpFormatMemoryEnc
_xmlFreeDoc
_xmlNodeSetContent
_xmlReadMemory
_xmlXPathEvalExpression
_xmlXPathFreeContext
_xmlXPathNewContext
make[2]: *** [mpeg2enc] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

Again, I believe that it's a problem with the "make" finding the xml2 libs. However, all is not lost! You've already compiled the mplex portion, which is all we're really interested in. cd to the mplex folder and run "sudo make install" from there, and the patched mplex will install. My test stream played correctly in MPlayer, which detected it as a TiVo stream! I haven't tried uploading it to my TiVo yet. (Does tymplex work for Series 1 TiVos?) I'll also look into the stdint.h issue on Tiger when I get home. (Tiger is installed on my iMac, Panther on my iBook.)

- Stealth Dave

stealthdave
08-23-2005, 04:30 PM
This is a simple OS porting issue regarding 64 bit integers. For unix based platforms, I wrote the code to expect stdint.h to define uint64_t. Apparently that doesn't happen in OSX. For windows, this was handled via the included win32/stdint.h. Perhaps there is a special define one needs with OSX to get the expected definitions out of stdint.h or maybe it would be easiest to use the win32/stdint.h for that platform. Someone with an OSX box needs to evaluate.
I started looking into the issue, and I don't think that the issue is necessarily with uint64_t, but with the following line 22:

typedef u_char boolean;
uint64_t is defined in stdint.h on OSX Tiger, and if I comment out the above line (the line that actually throws the error), the compile continues on and throws an error at line 49: "error: parse error before 'u_int'".

This is obviously something to do specifically with the OSX Tiger environment, as my Panther setup does successfully compile (well, mostly). Unfortunately, I think I've reached the end of what I can do to figure out the problem. Any ideas?

bcc
08-23-2005, 05:50 PM
I started looking into the issue, and I don't think that the issue is necessarily with uint64_t, but with the following line 22:

typedef u_char boolean;Oh ok, I thought the compiler was already lost from the previous line. Those data types "should" be defined in <sys/types.h> for OSes that are trying to be BSD compatible. You may have to #define __BSD_SOURCE to pick up the definitions.

stealthdave
08-23-2005, 11:23 PM
Oh ok, I thought the compiler was already lost from the previous line. Those data types "should" be defined in <sys/types.h> for OSes that are trying to be BSD compatible. You may have to #define __BSD_SOURCE to pick up the definitions.
That did it! I added "#include <sys/types.h>" to utils/xml.c and I was able to compile enough to get mplex completed! Something in the patch still prevents mjpegtools from completely compiling when patched, though. Here's the error that gets returned:

Making all in mpeg2enc
/bin/sh ../libtool --mode=link g++ -I/sw/include -o libmpeg2encpp.la -rpath /sw/lib -version-info 2:2:2 -release 1.6 -export-dynamic conform.lo elemstrmwriter.lo encoderparams.lo macroblock.lo motionest.lo mpeg2coder.lo mpeg2encoptions.lo mpeg2encoder.lo picture.lo picturereader.lo predict.lo putpic.lo putseq.lo quantize.lo ratectl.lo stats.lo synchrolib.lo tables.lo transfrm.lo writepic.lo fdctref.lo idct.lo predict_ref.lo quantize_ref.lo transfrm_ref.lo ../utils/libmjpegutils.a ../utils/libcpuaccel.la ../utils/libmotion.la

*** Warning: Linking the shared library libmpeg2encpp.la against the
*** static library ../utils/libmjpegutils.a is not portable!
g++ -dynamiclib -single_module -flat_namespace -undefined suppress -o .libs/libmpeg2encpp-1.6.0.2.2.dylib .libs/conform.o .libs/elemstrmwriter.o .libs/encoderparams.o .libs/macroblock.o .libs/motionest.o .libs/mpeg2coder.o .libs/mpeg2encoptions.o .libs/mpeg2encoder.o .libs/picture.o .libs/picturereader.o .libs/predict.o .libs/putpic.o .libs/putseq.o .libs/quantize.o .libs/ratectl.o .libs/stats.o .libs/synchrolib.o .libs/tables.o .libs/transfrm.o .libs/writepic.o .libs/fdctref.o .libs/idct.o .libs/predict_ref.o .libs/quantize_ref.o .libs/transfrm_ref.o -all_load ../utils/.libs/libcpuaccel.a ../utils/.libs/libmotion.a ../utils/libmjpegutils.a -install_name /sw/lib/libmpeg2encpp-1.6.0.dylib -compatibility_version 3 -current_version 3.2
ld: warning multiple definitions of symbol ___eprintf
/usr/lib/gcc/powerpc-apple-darwin8/4.0.0/libgcc.a(_eprintf.o) private external definition of ___eprintf in section (__TEXT,__text)
/usr/lib/gcc/powerpc-apple-darwin8/4.0.0/libstdc++.dylib(single module) definition of ___eprintf
ld: multiple definitions of symbol ___divdi3
/usr/lib/gcc/powerpc-apple-darwin8/4.0.0/libgcc.a(_divdi3.o) private external definition of ___divdi3 in section (__TEXT,__text)
/usr/lib/gcc/powerpc-apple-darwin8/4.0.0/libgcc_s.dylib(_divdi3.o) definition of ___divdi3
ld: multiple definitions of symbol ___udivdi3
/usr/lib/gcc/powerpc-apple-darwin8/4.0.0/libgcc.a(_udivdi3.o) private external definition of ___udivdi3 in section (__TEXT,__text)
/usr/lib/gcc/powerpc-apple-darwin8/4.0.0/libgcc_s.dylib(_udivdi3.o) definition of ___udivdi3
ld: multiple definitions of symbol ___umoddi3
/usr/lib/gcc/powerpc-apple-darwin8/4.0.0/libgcc.a(_umoddi3.o) private external definition of ___umoddi3 in section (__TEXT,__text)
/usr/lib/gcc/powerpc-apple-darwin8/4.0.0/libgcc_s.dylib(_umoddi3.o) definition of ___umoddi3
/usr/bin/libtool: internal link edit command failed
make[2]: *** [libmpeg2encpp.la] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
Any clues as to what I can do to get a complete compile? It's not a huge deal as I have mjpegtools installed vi Fink prior to installing the patched mplex, but it'd be nice to have something that installs cleanly.

Thanks,
- Stealth Dave

bcc
08-24-2005, 02:04 PM
That did it! I added "#include <sys/types.h>" to utils/xml.cAh ok. On my linux box this was pulled in already via the stdio.h include. No idea about your mplex linking issue. mjpegtools seems to need an overhaul when it comes to linking libraries.

petec1234
08-29-2005, 08:55 PM
OK, I give up. I have tried to compile tymplex under OS X 10.3.9 and 10.4.2. I believe I may have gotten past the libxml2 issue by *gasp* installing the fink download of libxml2 into the /usr/lib folder. I was using an external drive with a fresh install so it didn't matter much. I patched with the 1.1 patch, then the 1.2 OS X fix patch. I got by the libjpeg issue and then the libxml2 error. Now I'm getting a dylib.o error.
Can anyone who was able to compile post the OS binary of mplex or post how you got it to compile?

Thanks in advance.
-petec1234

stealthdave
08-30-2005, 07:09 PM
OK, I give up. I have tried to compile tymplex under OS X 10.3.9 and 10.4.2. I believe I may have gotten past the libxml2 issue by *gasp* installing the fink download of libxml2 into the /usr/lib folder. I was using an external drive with a fresh install so it didn't matter much. I patched with the 1.1 patch, then the 1.2 OS X fix patch. I got by the libjpeg issue and then the libxml2 error. Now I'm getting a dylib.o error.
Can anyone who was able to compile post the OS binary of mplex or post how you got it to compile?

Thanks in advance.
-petec1234
For some reason, the patched mjpegtools does fail at some point. For me, it failed after it had actually compiled mplex, the piece that's needed. After the compile fails, cd to the mplex folder and run make install like so:

$ cd mplex
$ sudo make install
It should install the modified mplex. The reason that I haven't just posted the binary is because it isn't a single binary, but a binary and several libraries that need to be installed. Unfortunately, I don't know enough about the compile process to attempt a static binary. If someone wants to help me with that, I'd be glad to post a binary for all the Mac users out there. :)

petec1234
09-02-2005, 09:15 PM
For some reason, the patched mjpegtools does fail at some point. For me, it failed after it had actually compiled mplex, the piece that's needed. After the compile fails, cd to the mplex folder and run make install like so:

$ cd mplex
$ sudo make install
It should install the modified mplex. The reason that I haven't just posted the binary is because it isn't a single binary, but a binary and several libraries that need to be installed. Unfortunately, I don't know enough about the compile process to attempt a static binary. If someone wants to help me with that, I'd be glad to post a binary for all the Mac users out there. :)

Thanks Stealth Dave. It worked!. I used 10.4.2. I had to
1. Install the libjpeg libraries from source. Downloaded source and created a link to glibtool called libtool so libjpeg would compile.
2. Patch using the "tymplex1.1.patch.txt" patch.
3. Patch using the "tymplex-1.2-osx-fix.diff" patch
4. Change the xml.c file to include "types.h"
5. Compiled the source until it failed.
6. Compiled and installed the source for the utils folder.
7. Compiled and installed the source for the mplex folder.

That seemed to do it. I created a ty file from an edited mpeg and inserted it back onto the TiVo. It worked fine except something went wrong after about 45 minutes. I think my demuxing went wrong somewhere, so I'm still looking into that.
I am currently looking at all the dependencies and will post the binary and the needed libraries (and their locations) within a few days if I can't figure out how to make a static binary.

-petec1234

stealthdave
09-04-2005, 04:55 PM
Here's an updated patch to fix OS X compilation. This patch should not affect the compile process on other platforms, so I'm submitting it for inclusion into the main source. All that the patch really does is add more places to look for libraries and headers, so that software installed in non-standard locations (i.e., fink's installation default of /sw) can be found and included. It also fixes the missing header for OS X Tiger that on most other operating systems (including OS X Panther).

The compile issue with OS X not completing its compile appears to be an issue with gcc-4.0 that comes with Tiger (although why it doesn't compile on Panther, I don't know). Setting CC to gcc-3.3 and CXX to g++-3.3 allows the compile to complete on OS X Tiger.

So, for those still trying to compile on OS X, here's my process from start to finish:


Install Fink (http://fink.sf.net)
install mjpegtools via fink
install libxml2 from Fink
download source for mjpegtools (http://mjpeg.sf.net) and patch with tymplex and os x patches
set your compiler to gcc-3.3 and g++-3.3:

$ export CC=/usr/bin/gcc-3.3
$ export CXX=/usr/bin/g++-3.3
configure with "./configure --prefix=/sw"
run "make" and "sudo make install"

On OS X 10.4.2, this compiled completely and installed cleanely! Woohoo!!!

- Stealth Dave

jelwell
09-08-2005, 02:40 PM
I get this on OS 10.4.2 and 10.3.9:
checking for jpeg_start_compress in -ljpeg-mmx... no
checking jpeglib.h usability... no
checking jpeglib.h presence... no
checking for jpeglib.h... no
configure: error: jpeglib.h not found - please install the libjpeg headers

CLEARLY, jpeglib.h is in my "/sw/include/" directory. And I'm passing "--prefix=/sw" to configure. *shrug*. I tried adding "--exec-prefix=/sw" as well.

I had to setup fink to use unstable (http://fink.sourceforge.net/faq/usage-fink.php?phpLang=en#unstable), since there's no mjpegtools in stable.

You can do "sudo gcc_select 3.3" to set gcc to the right version. Fink will ask you to do this in order to install mjpegtools.

For whatever reason your mjpegtools link doesn't work, and there appears to be two projects on sourceforge with that name. So download mjpegtools here (https://sourceforge.net/project/showfiles.php?group_id=5776&package_id=5823).

Thanks for all the work so far. Any suggestions on how to fix my jpeglib problem? Or even more bluntly, does anyone have a precompiled binary of tymplex for the Macintosh? :)
Joseph Elwell.

stealthdave
09-08-2005, 04:31 PM
Sounds like you need to set up your dev environment to use Fink libraries. Edit your .profile in your home directory to include this text:

test -r /sw/bin/init.sh && . /sw/bin/init.sh
export CPATH=/sw/include
export LIBRARY_PATH=/sw/lib
export RSYNC_RSH=ssh
export CFLAGS=-I/sw/include
export CXXFLAGS=$CFLAGS
export CPPFLAGS=$CXXFLAGS
export ACLOCAL_FLAGS="-I/sw/share/aclocal"
export PKG_CONFIG_PATH="/sw/lib/pkgconfig"
Then open a new Terminal and try again. It should find your jpeglib as installed by Fink.

jelwell
09-08-2005, 08:16 PM
Sounds like you need to set up your dev environment to use Fink libraries. Edit your .profile in your home directory to include this text:

test -r /sw/bin/init.sh && . /sw/bin/init.sh
export CPATH=/sw/include
export LIBRARY_PATH=/sw/lib
export RSYNC_RSH=ssh
export CFLAGS=-I/sw/include
export CXXFLAGS=$CFLAGS
export CPPFLAGS=$CXXFLAGS
export ACLOCAL_FLAGS="-I/sw/share/aclocal"
export PKG_CONFIG_PATH="/sw/lib/pkgconfig"
Then open a new Terminal and try again. It should find your jpeglib as installed by Fink.

That was it. The reason for it was quite nefarious. I changed my default shell to tcsh some time ago. Fink install that init.sh call in tcsh. But rather than me convert the export commands you listed (to setenv) I just started a bash shell, which doesn't call init.sh (since fink installed the call into my .tcshrc file).

Thanks a ton!

Now I'm working on getting (ty)mplex working (i.e. not crashing) and I'll test transferring my movies over. :)
Joseph Elwell.

nova1
09-16-2005, 10:34 PM
I've built a cygwin mplex_cc.exe with the DVD CC patches. I haven't tested this particular build. This build includes the tymplex1.2.patch.txt, tymplex1.2.stevet.patch.txt and the tymplex1.2.nova1.patch.txt.

whitepelican
09-19-2005, 03:45 PM
I've built a cygwin mplex_cc.exe with the DVD CC patches. I haven't tested this particular build. This build includes the tymplex1.2.patch.txt, tymplex1.2.stevet.patch.txt and the tymplex1.2.nova1.patch.txt.

Thanks so much for your work on this, Nova1. I have tested the captions a bit so far (and will continue to do so), but upon first look it seems to work pretty good.

Edited out my stupidity.

whitepelican
09-26-2005, 04:44 PM
After further review, the Nova1 patch to convert DVD to TY CC's works absolutely perfectly! It seems the first DVD that I tried it on had some problems with the Closed Captioning information, so I mistakenly thought the program wasn't working correctly. I've tried a few more DVDs and it works beautifully. Also, in the case where the original DVD didn't have CC's and only had subtitles, I used the SCC Tools from here (http://www.geocities.com/mcpoodle43/SCC_TOOLS/DOCS/SCC_TOOLS.HTML) to convert subtitles to DVD style CC's first and then used Nova1's mplex build to create TY files with CCs. That also worked surprisingly well. Kudos Nova1, and thanks again!

buddy3000
11-19-2005, 07:52 PM
Does anyone know where I can get a windows binary of tymplex that has the functioning xml import option?

TivoTyro
12-01-2005, 08:11 AM
Has anyone been able to compile this on RedHat??

Please bear with me as this is my first real venture into c++ compiling (have an extensive java background). Here are the steps I have taken:



1. Download mjpegtools-1.6.2.tar.gz and expand: http://prdownloads.sourceforge.net/mjpeg/mjpegtools-1.6.2.tar.gz?download
2. Download tymplex1.1.patch.txt from this thread: http://www.dealdatabase.com/forum/showpost.php?p=226704&postcount=63
2. cd mjpegtools-1.6.2
3. patch -p1 < ../tymplex1.1.patch.txt
4 ./configure
5 make


It starts to churn and then I end up getting:



xml.c: In function `parse_xml':
xml.c:129: `XML_PARSE_NOWARNING' undeclared (first use in this function)
xml.c:129: (Each undeclared identifier is reported only once
xml.c:129: for each function it appears in.)
xml.c:129: `XML_PARSE_RECOVER' undeclared (first use in this function)
xml.c:131: `XML_PARSE_NOERROR' undeclared (first use in this function)
xml.c:133: warning: implicit declaration of function `xmlReadMemory'
xml.c:133: warning: assignment makes pointer from integer without a cast
make[3]: *** [xml.o] Error 1
make[3]: Leaving directory `/tivo/tymplex/mjpegtools-1.6.2/utils'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/tivo/tymplex/mjpegtools-1.6.2/utils'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/tivo/tymplex/mjpegtools-1.6.2'
make: *** [all] Error 2


It looks like I am getting something similar to a ClassCastException?

Now there are other patches I see in the thread, but they appear to be specific to the windows binary (i was able to compile under cygwin following the instructions specific to that).

Can someone advise where I went wrong? If there is more information you need, I will gladly provide it.

bcc
12-01-2005, 08:58 PM
Has anyone been able to compile this on RedHat??
My instructions were based upon compiling on a fedora system.
xml.c:129: `XML_PARSE_NOWARNING' undeclared (first use in this function)That should be defined in your libxml2/parser.h. I assume you have the libxml2-devel package installed. I have:
libxml2-devel-2.6.19-1
If compiling with libxml2 is too difficult, you could compile without it using my tmplex1.2.patch.txt , and take out the #define HAVE_XML_LIB from config.h.in.

TivoTyro
12-02-2005, 12:59 PM
My instructions were based upon compiling on a fedora system.That should be defined in your libxml2/parser.h. I assume you have the libxml2-devel package installed. I have:
libxml2-devel-2.6.19-1
If compiling with libxml2 is too difficult, you could compile without it using my tmplex1.2.patch.txt , and take out the #define HAVE_XML_LIB from config.h.in.


I have only 2 words for you:


YOU ROCK!!!!!!

That was it.... Thanks for the help.

bcc
12-02-2005, 01:49 PM
Thanks, glad it was that easy (it usually isn't).

TivoTyro
12-05-2005, 11:14 PM
Thanks, glad it was that easy (it usually isn't).

Well, now I went and did it!

I rebuilt my system over the weekend with FC4 as I was having some other OS level instability.

Now i get:



Making all in lavtools
make[2]: Entering directory `/tivo/tymplex/mjpegtools-1.6.2/lavtools'
if /bin/sh ../libtool --mode=compile gcc -DHAVE_CONFIG_H -I. -I. -I.. -I.. -I/usr/local/include -DG_LOG_DOMAIN=\"lavtools\" -DLAVPLAY_VERSION=\"1.6.2\" -I/usr/local/include -I/usr/X11R6/include -I /usr/X11R6/include -I../utils -I/usr/include/gtk-1.2 -I/usr/include/glib-1.2 -I/usr/lib/glib/include -I/usr/X11R6/include -mcpu=i686 -march=i686 -g -O2 -Wall -Wunused -MT liblavrec_la-liblavrec.lo -MD -MP -MF ".deps/liblavrec_la-liblavrec.Tpo" \
-c -o liblavrec_la-liblavrec.lo `test -f 'liblavrec.c' || echo './'`liblavrec.c; \
then mv -f ".deps/liblavrec_la-liblavrec.Tpo" ".deps/liblavrec_la-liblavrec.Plo"; \
else rm -f ".deps/liblavrec_la-liblavrec.Tpo"; exit 1; \
fi
gcc -DHAVE_CONFIG_H -I. -I. -I.. -I.. -I/usr/local/include -DG_LOG_DOMAIN=\"lavtools\" -DLAVPLAY_VERSION=\"1.6.2\" -I/usr/local/include -I/usr/X11R6/include -I /usr/X11R6/include -I../utils -I/usr/include/gtk-1.2 -I/usr/include/glib-1.2 -I/usr/lib/glib/include -I/usr/X11R6/include -mcpu=i686 -march=i686 -g -O2 -Wall -Wunused -MT liblavrec_la-liblavrec.lo -MD -MP -MF .deps/liblavrec_la-liblavrec.Tpo -c liblavrec.c -fPIC -DPIC -o .libs/liblavrec_la-liblavrec.o
`-mcpu=' is deprecated. Use `-mtune=' or '-march=' instead.
In file included from liblavrec.h:33,
from liblavrec.c:76:
frequencies.h:107: error: array type has incomplete element type
liblavrec.c: In function 'lavrec_hardware_init':
liblavrec.c:1234: warning: comparisons like X<=Y<=Z do not have their mathematical meaning
make[2]: *** [liblavrec_la-liblavrec.lo] Error 1
make[2]: Leaving directory `/tivo/tymplex/mjpegtools-1.6.2/lavtools'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/tivo/tymplex/mjpegtools-1.6.2'
make: *** [all] Error 2


Hopefully, this is another EASY one!

BTW: This was based on libxml2-devel.i386 0:2.6.20-1.FC4

andryan
12-06-2005, 12:18 PM
Using the great work done by those in this forum I've been converting and inserting a lot of video to my Tivo from various sources, with excellent success. I wrote a helper script to manage the conversions which others might find useful -- when getting started it took me quite a while to get going and even after I figured stuff out, converting a variety of video files (formats and resolutions) was still time-consuming. This script should help confused newbies and lazy experts through the process. Feedback welcome, especially on suggestions for improving quality and consistency of the videos.

I'm attaching the script, although for more details check the page I set up for it:
http://www.nam-shub.com/tivo-hacks/video2ty.html

If you want to know what it does here's a sample usage which should give you an idea:

video2ty.py --title="Man Pokes Bear" --description="A man goes to the wilderness with a sharp stick, bearlarity ensues." --input=/home/video/bear-poking.avi --output=/home/tivo/bearpoke.ty

There's a new feature in this version (1.4), which is the --ftphost and --ftpport parameters. You can now do set-and-forget conversions/uploads with video2ty.py, e.g.:

video2ty.py --title="Man Pokes Bear" --description="A man goes to the wilderness with a sharp stick, bearlarity ensues." --input=/home/video/bear-poking.avi --output=/home/tivo/bearpoke.ty --ftphost=192.168.0.200 --ftpport=1234

This is very handy for doing overnight batch conversions of a few files at a time.

nova1
12-12-2005, 01:07 PM
I looked into the -O video_offset and -x xml issue a bit more and it seems to be an issue with the time_validate() function exiting because of a time discontinuity in the stream. I used -O -3s and it detected an error with the current time of 0:30:180 and the last time was 0:25:033. The tivo_ticks_per_chunk or time delta allowed was 0:05:00. I then used -O -9s and it exited at an earlier chunk with current chunk time 0:05:090, last time of 0:13:447. Negative time. Using -O -2s, mplex runs fine with no time discontinuities detected.

I haven't looked into the timestamps in depth on this stream. I'll have to see what timestamps it is generating to cause the time discontinuities.

I thought about this some more driving into work and I think the reason the chunk position changes is because of the -O offset. The time discontinuity is detected when the first chunk record is NOT a audio record type. If it was a video record type, the timestamps time line would be inline and correct but if the current record type was audio, it's timestamp would be off by the -O offset value and a time discontinuity would be detected if the difference > 5s. I'll look into it more when I get home.

bcc
12-12-2005, 03:27 PM
Wow, things are broken at many different levels here I see.
First, in tymaster.c, rewrite_master() should not fail silently when an invalid chunk (bad timestamp) is hit. This was fixed in later versions of ty1to2's master.c with a warning. But nobody has updated tymplex's copy. A warning is appropriate for ty1to2, but for tymplex, the warning should probably instead be changed to an abort. This is because the mplex code shouldn't build chunks with timestamp discontinuities in the first place. Which goes to the question of what is going wrong at the lower levels...

There were other improvements made to ty1to2 as well, that have not been incorporated into tymplex. For example, the XML changes. Personally I'm not excited about doing the 3 way merge required to keep tymplex up to date. Anyone else want to?

bcc
12-12-2005, 03:44 PM
From the support thread...
rewrite_master() would return a short # and then the XML routine would insert the XML data in the wrong place(i.e. seek to chunk offset and start writing the xml there in the middle of the TY file).Yes, all that is a symptom of rewrite_master() silently failing as above. Saw the same thing with ty1to2 (and it's fixed there).

nova1
12-16-2005, 02:38 AM
I printed out the timestamps/major/type of the first TY record for each chunk and it does show that it will fail the time_validate() function when an audio record is the first TY record in a chunk. The time_validate() function uses the timestamp of the first TY record. When using the -O option, the timestamps may be off by more than the allowed ticks_per_chunk. If all the first TY records are of the same type(video or audio), the timeline will be ok. However, once you hit a different record type, the timestamp timeline will be off by the offset and time_validate() will return invalid. So it looks like the offset used needs to be added to ticks_per_chunk. Another related issue is that rewrite_master() returns the # of seconds based on the first video packet timestamp and the last TY chunk timestamp seen. The last timestamp may not be a video record and will throw that calculation off.

So basically, time validation should occur on a per stream(audio,video,etc) basis and not be based on the first TY record timestamp seen.

I worked out some patches to fix the above issues I've seen. I'll do some more testing on some streams and then post a patch. I also looked at ty1to2-1.4 and merged in the minor changes I noticed to tymaster.c, tyutils.c, tydefs.h and tyexterns.h.

I've attached tymplex1.2.nova2.patch.txt which should fix the -O issue. This build includes the tymplex1.2.patch.txt, tymplex1.2.stevet.patch.txt, tymplex1.2.nova1.patch.txt and the tymplex1.2.nova2.patch.txt.

pikespeakhiker
01-01-2006, 01:58 AM
Thanks, nova1, for your work on the merge. Does this patch/mplex rebuild also contain the fix for the -x xml support? Sorry if this is already answered. Wasn't quite sure what you meant by "above."


*** EDIT *** Answered my own question by trying this out. -x worked like a champ. Thank you!

ewieling
01-20-2006, 02:58 PM
What should I run this patch against? I assumed mjpegtools-1.6.2, but the patch tries to modify utils/ty.c which does not exist. It looked from your post that your patch included all patches needed, rolled into one.

Edit: I ran the following and it seemed to work. I'm just checking to make sure this is correct.

[root@fs-1 mjpegtools-1.6.2]# patch -p1 < ../tymplex1.1.patch_plus-lfs-fix.txt
[root@fs-1 mjpegtools-1.6.2]# patch -p1 -F 5 < ../tymplex1.2.nova2.patch.txt

I assume the above two patches include everything I need?

Edit: I'm trying to do this on Linux, of course. I seem to be having problems with ffmpeg. This is what I get. The overwrite question is odd.

[root@fs-1 ffmpeg-0.4.9-pre1]# ffmpeg -debug -i /home/eric/bob.vob -b 2000 -f mpeg2video -s 720x480 -padtop 0 -padbottom 0 -padleft 0 -padright 0 -r 29.97 /home/eric/tmp/tmpseZDUi/video.m2v -f mp2 -ab 192 -ar 32000 /home/eric/tmp/tmpseZDUi/audio.m2a
ffmpeg version 0.4.9-pre1, build 4718, Copyright (c) 2000-2004 Fabrice Bellard
built on Jan 20 2006 15:54:37, gcc: 3.3.1 (Mandrake Linux 9.2 3.3.1-2mdk)
File '/home/eric/bob.vob' already exists. Overwrite ? [y/N] Y
Unknown input or output format: mpeg2video
[root@fs-1 ffmpeg-0.4.9-pre1]#

Anyone have any ideas what I might be doing wrong?

The outout of "ffmpeg -formats" does show mpeg2video as a supported format. Thanks in advance.

rbautch
01-26-2006, 11:18 PM
Using the latest build, I'm getting asserts related to seq_pos = -1 described at the beginning of the thread. Seem to happen only on 20% of my DVD rips. How can I fix this?


I:\DVDRips>mplex.exe -f 10 40*.m2v 40*.mp2 -o Naval.Treaty.ty
INFO: [mplex] mplex version 1.6.2 (2.2.3 $Date: 2004/01/13 20:45:26 $)
INFO: [mplex] File 40 - NavalTreaty.m2v looks like an MPEG Video stream.
INFO: [mplex] File 40 - NavalTreaty.mp2 looks like an MPEG Audio stream.
INFO: [mplex] Video stream 0: profile 10 selected - ignoring non-standard opt
ions!
INFO: [mplex] Found 1 audio streams and 1 video streams
INFO: [mplex] Selecting TY and generic MPEG2 output profile
INFO: [mplex] Multiplexing video program stream!
INFO: [mplex] Scanning for header info: Video stream e0 (40 - NavalTreaty.m2v
)
INFO: [mplex] VIDEO STREAM: e0
INFO: [mplex] Frame width : 720
INFO: [mplex] Frame height : 480
INFO: [mplex] Aspect ratio : 4:3 display
INFO: [mplex] Picture rate : 29.970 frames/sec
INFO: [mplex] Bit rate : 7000000 bits/sec
INFO: [mplex] Vbv buffer size : 229376 bytes
INFO: [mplex] CSPF : 0
INFO: [mplex] Scanning for header info: Audio stream c0 (40 - NavalTreaty.mp2
)
INFO: [mplex] MPEG AUDIO STREAM: c0
INFO: [mplex] Audio version : 1.0
INFO: [mplex] Layer : 2
INFO: [mplex] CRC checksums : yes
INFO: [mplex] Bit rate : 24576 bytes/sec (192 kbit/sec)
INFO: [mplex] Frequency : 48000 Hz
INFO: [mplex] Mode : 0 stereo
INFO: [mplex] Mode extension : 0
INFO: [mplex] Copyright bit : 0 no copyright
INFO: [mplex] Original/Copy : 0 copy
INFO: [mplex] Emphasis : 0 none
INFO: [mplex] SYSTEMS/PROGRAM stream:
INFO: [mplex] rough-guess multiplexed stream data rate : 7345344
INFO: [mplex] Setting best-guess data rate.
INFO: [mplex] Run-in Sectors = 998 Video delay = 200346 Audio delay = 203349
INFO: [mplex] New sequence commences...
INFO: [mplex] Video e0: buf=2719744 frame=000000 sector=00000000
INFO: [mplex] Audio c0: buf= 4096 frame=000000 sector=00000000
INFO: [mplex] seq_pos=-1 gop_pos=-1 pic_pos=0
assertion "gop_pos != -1" failed: file "multiplexor.cpp", line 1768

Jamie
01-27-2006, 07:07 PM
BTW, Nova1, have you looked at the ffmpeg code? Looks to me to be a good deal cleaner than mplex. If one ported the TY encoding support to ffmpeg, as an encoding codec, this all could be easier to maintain. It could also remove the demux step requirement for users.I'm thinking of taking on this project if no one else is working on it and if it still seems to be a good idea. I'm thinking of supporting both tystreams with no master chunks (for streaming ala tivoserver) as well as seekable files with master chunks.

Any comments? Anybody else working on this?

rc3105
01-27-2006, 07:26 PM
an actual ty codec would be handy. mplayer/mencoder allready has halfway decent ty parsing (not that there's any earthshaking diff between mencoder & ffmpeg) adding ty read/write to ffmpeg or ty write to mencoder would allow one step hdty->sdty w/o pipes

bcc
01-27-2006, 07:50 PM
I'm thinking of taking on this project if no one else is working on it and if it still seems to be a good idea. I'm thinking of supporting both tystreams with no master chunks (for streaming ala tivoserver) as well as seekable files with master chunks.

Any comments? Anybody else working on this?Still sounds like a fine idea by me :) I don't think I heard back from nova1

bcc
01-27-2006, 07:54 PM
an actual ty codec would be handy. mplayer/mencoder allready has halfway decent ty parsing (not that there's any earthshaking diff between mencoder & ffmpeg) adding ty read/write to ffmpeg or ty write to mencoder would allow one step hdty->sdty w/o pipesmplayer is a bit quick&loose with its ty decoding, and isn't providing a/v sync. I'd have to recommend hdemux if one wants to look at decoding.

nova1
01-28-2006, 12:48 PM
I'm thinking of taking on this project if no one else is working on it and if it still seems to be a good idea. I'm thinking of supporting both tystreams with no master chunks (for streaming ala tivoserver) as well as seekable files with master chunks.

Any comments? Anybody else working on this?
Sounds cool. I only glanced at the code structure in ffmpeg and never did anything with the code.

rc3105
01-28-2006, 02:38 PM
mplayer is a bit quick&loose with its ty decoding, and isn't providing a/v sync. I'd have to recommend hdemux if one wants to look at decoding.
oh sure, I was just observing that mplayer/mencoder has some other cool features that would benefit from ty compatability

(http/ftp/vserver network streaming, video playback under win/mac/linux/bsd, direct playback through some hardware decoders & passthrough/capture to file/screen from various encoder cards)


call me optomistic but I don't imagine it'd be any harder to transplant the ty-enabled-mplex functionality into mplayer instead of ffmpeg. then the other features are gravy for us ty enthusiasts ;)


edit: crud, now I have to try & remember (or go look at) what code is common to mplayer & ffmpeg - perhaps they can both use the same ty codec. like there's not enough hours in the day allready... :eek:

Jamie
01-28-2006, 02:59 PM
oh sure, I was just observing that mplayer/mencoder has some other cool features that would benefit from ty compatabilityIt seems like there are several places where a bi-directional ty codec could be useful. ffmpeg/libavformat/libavcodec is one. mplayer/mencoder is another. vlc is a third. There are probably others. In an ideal world, a common code base could be shared between all. I haven't looked at this enough yet to know if that's at all practical. I thought of ffmpeg first, because it was on the tivoserver developers wish list.

bcc
01-28-2006, 09:24 PM
call me optomistic but I don't imagine it'd be any harder to transplant the ty-enabled-mplex functionality into mplayer instead of ffmpeg. then the other features are gravy for us ty enthusiasts ;)The devil's in the details. I don't think mplayer's code is as maintainable, and thus would be harder to work with. But I haven't tried hacking on ffmpeg.

HuMan321
02-16-2006, 08:11 PM
It seems like there are several places where a bi-directional ty codec could be useful. ffmpeg/libavformat/libavcodec is one. mplayer/mencoder is another. vlc is a third. There are probably others. In an ideal world, a common code base could be shared between all. I haven't looked at this enough yet to know if that's at all practical. I thought of ffmpeg first, because it was on the tivoserver developers wish list.


ffmpeg would be awesome for the tivoserver reasons you have given above.
Thanks for your efforts.

Besides, reencoding to mpeg will result in unnecessary loss of quality if series 1 ty files could work with Tivoserver

zippy7272
02-21-2006, 05:28 AM
Using the great work done by those in this forum I've been converting and inserting a lot of video to my Tivo from various sources, with excellent success. I wrote a helper script to manage the conversions which others might find useful -- when getting started it took me quite a while to get going and even after I figured stuff out, converting a variety of video files (formats and resolutions) was still time-consuming. This script should help confused newbies and lazy experts through the process. Feedback welcome, especially on suggestions for improving quality and consistency of the videos.

I'm attaching the script, although for more details check the page I set up for it:
http://www.nam-shub.com/tivo-hacks/video2ty.html

If you want to know what it does here's a sample usage which should give you an idea:

video2ty.py --title="Man Pokes Bear" --description="A man goes to the wilderness with a sharp stick, bearlarity ensues." --input=/home/video/bear-poking.avi --output=/home/tivo/bearpoke.ty

There's a new feature in this version (1.4), which is the --ftphost and --ftpport parameters. You can now do set-and-forget conversions/uploads with video2ty.py, e.g.:

video2ty.py --title="Man Pokes Bear" --description="A man goes to the wilderness with a sharp stick, bearlarity ensues." --input=/home/video/bear-poking.avi --output=/home/tivo/bearpoke.ty --ftphost=192.168.0.200 --ftpport=1234

This is very handy for doing overnight batch conversions of a few files at a time.

Sorry new to some of these things.

I've got cygwin and python installed (after reading this). But can't see how to run the .py script.

Do I do it from cygwin - which doesn't recognise the script. OR do I run it from python screen. I can seemingly get it to run (via the file open / run options) but it doesn't allow to put any parameters anywhere (not even file names etc)

I've tried down loading the unix version of python, but when I run the ./configure it can't find a compiler (cc / gcc etc). Where would I get 1 from?

Any help appreciated.

zippy7272
02-23-2006, 04:17 AM
Guess I may have originally posted this in the wrong forum, so I'll try here.

Quote:
Originally Posted by andryan
Using the great work done by those in this forum I've been converting and inserting a lot of video to my Tivo from various sources, with excellent success. I wrote a helper script to manage the conversions which others might find useful -- when getting started it took me quite a while to get going and even after I figured stuff out, converting a variety of video files (formats and resolutions) was still time-consuming. This script should help confused newbies and lazy experts through the process. Feedback welcome, especially on suggestions for improving quality and consistency of the videos.

I'm attaching the script, although for more details check the page I set up for it:
http://www.nam-shub.com/tivo-hacks/video2ty.html

If you want to know what it does here's a sample usage which should give you an idea:

Code:
video2ty.py --title="Man Pokes Bear" --description="A man goes to the wilderness with a sharp stick, bearlarity ensues." --input=/home/video/bear-poking.avi --output=/home/tivo/bearpoke.ty
There's a new feature in this version (1.4), which is the --ftphost and --ftpport parameters. You can now do set-and-forget conversions/uploads with video2ty.py, e.g.:

Code:
video2ty.py --title="Man Pokes Bear" --description="A man goes to the wilderness with a sharp stick, bearlarity ensues." --input=/home/video/bear-poking.avi --output=/home/tivo/bearpoke.ty --ftphost=192.168.0.200 --ftpport=1234
This is very handy for doing overnight batch conversions of a few files at a time.


Sorry new to some of these things.

I've got cygwin and python installed (after reading this). But can't see how to run the .py script.

Do I do it from cygwin - which doesn't recognise the script. OR do I run it from python screen. I can seemingly get it to run (via the file open / run options) but it doesn't allow to put any parameters anywhere (not even file names etc)

I've tried down loading the unix version of python, but when I run the ./configure it can't find a compiler (cc / gcc etc). Where would I get 1 from?

Any help appreciated.

BTUxNine
02-23-2006, 04:24 AM
from a cygwin command line, you should just be able to type ./video2ty.py

if that doesn't work, check that /usr/bin/python2 exists

zippy7272
02-23-2006, 06:24 AM
from a cygwin command line, you should just be able to type ./video2ty.py

if that doesn't work, check that /usr/bin/python2 exists

Thanks - Tried - failed

I don't think Python is installed properly, when I tried to install Python I ran ./configure - but it can't find a compiler (cc / gcc etc).

Where do I get one from / why doesn't Cygwin install one?

nova1
02-23-2006, 11:04 AM
Using the latest build, I'm getting asserts related to seq_pos = -1 described at the beginning of the thread. Seem to happen only on 20% of my DVD rips. How can I fix this?


INFO: [mplex] seq_pos=-1 gop_pos=-1 pic_pos=0
assertion "gop_pos != -1" failed: file "multiplexor.cpp", line 1768
mplex expects the video stream to have at least 1 mpeg sequence header and a Group of Pictures(GOP) header before every Picture Start header. mplex found the picture start code at offset 0(start of this record buffer) but didn't find the GOP header. Usually a sequence header is also included but if not found, mplex will re-use the first sequence record at the start of the stream.

AlphaWolf
03-03-2006, 09:25 PM
This is working great for me so far, and I am not having any audio drift issues. Having DVD's on the tivo is a hell of a lot more convenient, and the seek features on the tivo are a hell of a lot better than most DVD players. Instant replay and 30 sec skip are nice to have as well, even when there are no commercials, and hardly any DVD players out there have a playback bar to indicate where you are in the video (other than a timer which sucks,) and none of them have any "tick" marks that you can skip to. Plus, playing these DVD's on the HR10-250 is pretty nice in lieu of a progressive DVD player.

That said though, the HR10-250 doesn't support MRV, so therefore there's no tivoserver support. So that means I relie on mfs_ftp. My only beef though is that failed .ty transfers can be a huge pain in the ass to remove, you can't watch them while they are importing, and the show title is limited to the filename and you can't add a description. TMF doesn't have these problems on the other hand. So my solution was to make a .ty to .tmf converter. You can find it attached.

ty2tmf requires bash, and should work fine with cygwin. Note that it probably shouldn't be used with tivo created tystreams (see the comments in the script for an explanation) and rather should only be used with tymplex created streams. I only spent about 10 minutes writing it, and since this is mostly a development thread I haven't added any sanity checks anywhere.

It could be made to be more robust to use hexdump to parse the .ty master chunk headers in order to calculate the exact size of each typart thus enabling it to work with all .ty files, however I don't think I will bother with that since this is only a bash script afterall, and it already supports my purposes anyways (I don't have much time to write software that is intended for anybody other than myself lately.) I would like to see direct tmf creation added to tymplex though, as it would be a huge time saver that way. I am not smart enough to be able to code in C though :D

To use it, pass the source .ty file as the first parameter and the .tmf file to be created as the second parameter. You can go ahead and edit the showing.xml to reflect the desired title and description of the DVD that you are importing. You can also adjust the start and end time markers in order to adjust the shown length of the program according to the summary screen (otherwise it just says 0:00 (Partial)). That is optional though, as the playback bar will show the correct stream length once it is finished uploading.

But that said, here is the process I use for importing DVD's onto my HR10-250:

Use dvd decrypter according to Narff54321's instructions (seen nya) (http://www.dealdatabase.com/forum/showpost.php?p=241382&postcount=5) , with one exception. Before clicking the Disc-to-HDD button, click on the Input tab and then right click on anything in that sub-window, go to file splitting, and then check none. This makes it so that you get one big stream instead of several 1GB streams, which makes things much easier.

Then use nova1's patched mplex, seen nya (http://www.dealdatabase.com/forum/showpost.php?p=234847&postcount=101), to convert the m2v and ac3 files to a .ty stream. (e.g. ./mplex -f10 tivomovie.ty movie.m2v movie.ac3)

Then edit the showing.xml file to have the description and title that you want to see in nowshowing (you can also add actors, directors, genres, tvratings for parental controls, stream length, etc,) and then something like:

./ty2tmf.sh tivomovie.ty tivomovie.tmf

Then just start the ftp transfer with mfs_ftp, and start watching the show.

(FWIW for those wondering, yes, I use windows and linux simultaneously, though they aren't both installed on the same machine)

Jamie
03-03-2006, 09:37 PM
So my solution was to make a .ty to .tmf converter. You can find it attached.
Say, that idea sounds familar. (http://www.dealdatabase.com/forum/showthread.php?p=231695#post231695) :D

AlphaWolf
03-03-2006, 09:52 PM
Ah, I didn't see that. I tried to search for something like it but ddb doesn't like the 'ty' keyword :D

bcc
03-04-2006, 10:32 PM
Plus, playing these DVD's on the HR10-250 is pretty nice in lieu of a progressive DVD player.For me, my HTPC does a better job as tivo doesn't output native rate for my display.
you can't watch them while they are importing, and the show title is limited to the filename and you can't add a description.
You can supply XML to tymplex directly via the -x switch (if you compiled a version of tymplex with that support). In my more recent XML support, the streamfilesize, duration, stoptime are updated in any XML that you pass in.
Yes, looks like we should have just made tymplex capable of generating tmf. Wouldn't be hard. Didn't realize mfs_ftp would have such a difficult and ongoing problem with inserting .ty+ correctly.

AlphaWolf
03-05-2006, 04:14 AM
For me, my HTPC does a better job as tivo doesn't output native rate for my display.
You can supply XML to tymplex directly via the -x switch (if you compiled a version of tymplex with that support). In my more recent XML support, the streamfilesize, duration, stoptime are updated in any XML that you pass in.
Yes, looks like we should have just made tymplex capable of generating tmf. Wouldn't be hard. Didn't realize mfs_ftp would have such a difficult and ongoing problem with inserting .ty+ correctly.

You know what would also be nice, is to be able to pass an encapsulated title and description as a parameter so that it can be put into the XML file on the fly, rather than have to regenerate the TY+ or TMF from scratch.

BTW, there aren't ongoing problems in mfs_ftp so much as TMF was just designed specifically for on the fly tystream insertion, whereas ty and ty+ require more complexity in the insertion code.

jbstix
03-16-2006, 07:07 PM
I'm am trying to use this to convert an .mpg file to .ty
I really don't have any need to pull files off the Tivo, convert them twice then put them back...
My main goal is to convert already existing .mpg files into .ty files and watch on the Tivo.
These are downloaded mpg files, and I'm using TMPGenc to split them into 2 files. It gives me the option of ".m1v and mp2" when splitting.
When trying to covert w/ mplex heres what i get:
C:\>mplex -f 10 movie.m1v movie.mp2 -o movie.ty
INFO: [mplex] mplex version 1.6.2 (2.2.3 $Date: 2004/01/13 20:45:26 $)
INFO: [mplex] File movie.m1v looks like an MPEG Video stream.
INFO: [mplex] File movie.mp2 looks like an MPEG Audio stream.
INFO: [mplex] Video stream 0: profile 10 selected - ignoring non-standard opt
ions!
INFO: [mplex] Found 1 audio streams and 1 video streams
INFO: [mplex] Selecting TY and generic MPEG2 output profile
INFO: [mplex] Multiplexing video program stream!
INFO: [mplex] Scanning for header info: Video stream e0 (jenna.m1v)
INFO: [mplex] VIDEO STREAM: e0
INFO: [mplex] Frame width : 320
INFO: [mplex] Frame height : 240
INFO: [mplex] Aspect ratio : 1:1 pixels
INFO: [mplex] Picture rate : 30.000 frames/sec
INFO: [mplex] Bit rate : 1150000 bits/sec
INFO: [mplex] Vbv buffer size : 49152 bytes
INFO: [mplex] CSPF : 0
INFO: [mplex] Scanning for header info: Audio stream c0 (jenna.mp2)
INFO: [mplex] MPEG AUDIO STREAM: c0
INFO: [mplex] Audio version : 1.0
INFO: [mplex] Layer : 2
INFO: [mplex] CRC checksums : no
INFO: [mplex] Bit rate : 24576 bytes/sec (192 kbit/sec)
INFO: [mplex] Frequency : 44100 Hz
INFO: [mplex] Mode : 3 single channel
INFO: [mplex] Mode extension : 0
INFO: [mplex] Copyright bit : 0 no copyright
INFO: [mplex] Original/Copy : 0 copy
INFO: [mplex] Emphasis : 0 none
INFO: [mplex] SYSTEMS/PROGRAM stream:
INFO: [mplex] rough-guess multiplexed stream data rate : 1375448
INFO: [mplex] Setting best-guess data rate.
INFO: [mplex] Run-in Sectors = 998 Video delay = 1069913 Audio delay = 107291
3
INFO: [mplex] New sequence commences...
INFO: [mplex] Video e0: buf=2719744 frame=000000 sector=00000000
INFO: [mplex] Audio c0: buf= 4096 frame=000000 sector=00000000
INFO: [mplex] seq_pos=-1 gop_pos=0 pic_pos=8
INFO: [mplex] seq_pos=-1 gop_pos=0 pic_pos=-1
assertion "pic_pos != -1" failed: file "multiplexor.cpp", line 1767
3 [sig] mplex 2796 open_stackdumpfile: Dumping stack trace to mplex.exe.st
ackdump

Same error every time... I've tried using different programs as well.
Please help... how can I get usable .m2v and .m2a files from and existing .mpg file???
Thanks for the help

Cheezmo
03-16-2006, 08:47 PM
I've been giving this a try with some success on an HD Tivo. Much thanks to the OSX work by petec1234 and of course all the others.

I've converted a few .mpg files, even at strange resolutions such as 624x352 and they play back fine, but with one caviat. I have a widescreen TV and the shows are widesceen (originally HD) but when the Tivo is at 1080i, they play back as 4:3 shows with gray bars on the side. I can get them to play fullscreen by switching to 480p and telling the Tivo I have a 4:3 TV, but they aren't being treated like normal HDTV programs and played back widescreen at 1080i.

I tried scaling one to 1280x720p and it still played back 4:3.

Looking through the MFS data, the only thing that seems reasonable that it might be keying off of is the ApgStreamType field which is 2 for normal video (and I believe anything mfs_ftp inserts, but 197 for recorded HD programs. I notice that that field does not exist in the XML so it may not be something that can be set through mfs_ftp without some modifications.

I also survived one of the worst Tivo crashes I've ever experienced after a failed insert. After I rebooted, things were all crazy, and I finally discovered that /dev/null was being treated as read only file. Sure enough, it looked like a normal file. Turns out I must have left / in rw state and somehow /dev/null got corrupted. I bit of searching turned up how to recreate it, but it was pretty hairy for a while there. But that is another story...

bcc
03-17-2006, 02:07 AM
cheezmo -
Sounds like your video streams do not have their aspect ratios set to 16:9. Check your .m2v. Before releasing mplex, I tested with HD streams and did not have problems with aspect ratios. I even converted microsoft's demo HD .wmv files to .ty and those inserted fine.

jbstix
03-17-2006, 02:19 AM
...deleted...
moved my question(s) to the support forum.
thanks

bcc
03-17-2006, 02:26 AM
BTW, there aren't ongoing problems in mfs_ftp so much as TMF was just designed specifically for on the fly tystream insertion, whereas ty and ty+ require more complexity in the insertion code.
Actually there are outstanding problems with mfs_ftp support of .ty+. I was thinking of these when I made my comment: http://www.dealdatabase.com/forum/showthread.php?p=247282#post247282

http://www.dealdatabase.com/forum/showpost.php?p=246420&postcount=70
Personally I prefer the .ty+ file format as .tmf is a problem for any application that wants to stream. Perhaps if chunk segment boundary processing had been better built into the mfs_utils library (made more private to it), then mfs_ftp wouldn't have such a hard time getting it right, and .tmf wouldn't have been necessary.

BTUxNine
03-17-2006, 02:30 AM
Actually there are outstanding problems with mfs_ftp support of .ty+. I was thinking of these when I made my comment: http://www.dealdatabase.com/forum/showthread.php?p=247282#post247282

http://www.dealdatabase.com/forum/showpost.php?p=246420&postcount=70
Personally I prefer the .ty+ file format as .tmf is a problem for any application that wants to stream. Perhaps if chunk segment boundary processing had been better built into the mfs_utils library (made more private to it), then mfs_ftp wouldn't have such a hard time getting it right, and .tmf wouldn't have been necessary.
compounding the problem was that most tools have the part size off by one... tmf may have been developed to try to fix that.

bcc
03-17-2006, 02:38 AM
compounding the problem was that most tools have the part size off by one... tmf may have been developed to try to fix that.
The actual chunk segment length that comes out of the tivo filesystem seems to be off by 1. Luckily everybody seems to handle this in an interoperable way (tho it does result in garbage chunks in .ty files).

Cheezmo
03-17-2006, 08:59 AM
cheezmo -
Sounds like your video streams do not have their aspect ratios set to 16:9. Check your .m2v. Before releasing mplex, I tested with HD streams and did not have problems with aspect ratios. I even converted microsoft's demo HD .wmv files to .ty and those inserted fine.

I bet you are right. I was trying to avoid rescaling, but haven't figured out how to get ffmpegX to specify 16:9 for encoding generic MPEG-2. I'll try with a DVD profile. I looked at some of my previous files and they all did have 1:1 or 4:3 aspect ratios embedded.

shstevens
03-23-2006, 08:20 PM
sorry was a support issue - moved to correct thread....

thanks,
shawn

phat_bastard
05-02-2006, 06:05 PM
I think I've located an issue in Mutiplexor:WriteTyRecord() that would explain the problems I've been having with choppy playback of tivo sourced streams. I originally thought this was an issue with field dominance being lost, and if my theory is correct, that is (or is closely related to) the problem. Bear in mind that I'm looking at the mjpegtools source from the tivoserver sourceforge site, so there may be some differences between this and the latest CC enabled build.

In Multiplexor:WriteTyRecord() basically what's happening is the sequence, gop and picture headers are getting re-written to 'tivo format' with what I'd presume are tivo specific start codes. What I noticed here is that extension headers (specifically picture coding extension) aren't being re-written in this manner. I don't know for a fact that these are in a different format in a ty stream, but it would explain the problems I've been seeing with DTivo streams remuxed using ty mplex.

To check that I wasn't just nuts, I added this:


// find ext_pos
if (pic_pos == -1) {
ext_pos = start_code_search(tmp_buf,START_CODE_EXT,0,2048);
} else {
ext_pos = start_code_search(tmp_buf,START_CODE_EXT,pic_pos,2048);
}
if (ext_pos != -1) {
mjpeg_info("found extension header at %d, picture header at %d",ext_pos,pic_pos);
}

and lo and behold I'm seeing extension headers 8 bytes after pic_pos:


INFO: [mplex] found extension header at 9, picture (B-frame) at 0
INFO: [mplex] found extension header at 9, picture (P-frame) at 0
INFO: [mplex] found extension header at 52, picture header at 44

If I then try to demux these synthetic ty streams using tytool, it blows chunks on the first video frame.

Anyone have any idea how these headers look in a ty stream?

phat_bastard
05-05-2006, 10:17 AM
After building a very basic ty stream analyzer and working through the problem streams, it's evident that the tivo stream format doesn't include ty records for extension headers. I guess this means they need to be dropped by mplex. I'm trying to figure out the simplest way to handle this, but my skills in C are juvenile at best. Anyone have any suggestions or care to discuss?

phat_bastard
05-10-2006, 11:04 AM
After painful hours of prodding around in multiplexor.cpp I've thrown in the towel for a bit. I'd really appreciate any help or intelligent discussion of this topic though. In my initial analysis I guess I'd left out the possibility that extension headers are encapsulated in the frame payload data in the tivo stream, in which case there's no problem with the way it's being packaged back up. If that's the case, what other possibilities are there? Please, someone, throw me a bone...:confused:

mrdeals
05-10-2006, 01:07 PM
I got the TyTool and it seems to be mostly (all) about streaming from Tivo to Pc. I use Tivo desktop and wonder if there is a good way to convert them to DVD. I tried the sonic whateveritwas that Tivo said would work and it always hung up. What format are the shows in on Tivo it's self? Do they get converted in to the .Tivo WMA format by Tivo desktop?

Suggestions.

PS if I install any software on Tivo it'sself and scew it up my wife will KILL me, Slowly.

cheer
05-10-2006, 03:22 PM
They're stored on the Tivo in MPEG2-ish chunks. TyTool isn't at all about streaming -- in fact, it's exactly about transferring files from the Tivo and making DVDs out of them. However, it does involve hacking your Tivo and installing software on it.

Tivo Desktop transfers the files and puts them into DRM-protected .tivo files. I've seen mention of something called DirectShowDump that can dump them into normal files, but I've never used it.

bcc
05-13-2006, 02:07 PM
phat,
If you don't know C, I'm surprised you're trying to troubleshoot your problems from the code. Why not step back and try to isolate what is different about what you're doing vs what most others are doing that are having no problems with tymplex. In my "Some general troubleshooting ideas for tymplex" post, over in the support thread, I detailed some things you can do diagnose problems; I've lost track, but I don't think you've tried a couple of those ideas.

Failing all that, you could provide a short stream sample that others can use to reproduce your problem, and then hopefully figure out what's going wrong.

phat_bastard
05-15-2006, 11:58 AM
Thanks for taking the time to respond. I was beginning to think I'd done something to offend you. If I have, I sincerely apologize.


If you don't know C, I'm surprised you're trying to troubleshoot your problems from the code.I'm not fluent in C. I have written apps, and I've even done some hacking on the windows port of mjpegtools so I'm at least somewhat familiar with the inner workings of mplex. After a protracted attempt to remove extension headers from the frame payload, I was stuck wondering whether or not you (and josh / tytool) are sythesizing the extension headers I'm seeing in the PES streams hdemux generates? I wasn't able to locate any sources to this tool, so I couldn't verify this on my own. If the answer is no, then there's likely no need to look into that further.


Why not step back and try to isolate what is different about what you're doing vs what most others are doing that are having no problems with tymplex.I'd like to think that I have, but ultimately it doesn't seem like there are many ppl using mplex to do what I'm attempting. This happens with everything I have that I've taken off my DTivo, split with (hdemux, dgindex, tytool) and directly remux to play on the same DTivo. The problem is more apparent on certain streams, but if you watch closely it's apparent in differing degrees in everything I've run this process on. It's highly possible that I've overlooked the one or two posts from others that have tried this specific mix, had these type of problems and posted their breakthrough, so maybe I just need to spend the next few days reading everything here. (gulp)... Cheer has been one that has mentioned similar problems, and I've tried everything he's suggested to no avail.


In my "Some general troubleshooting ideas for tymplex" post, over in the support thread, I detailed some things you can do diagnose problems; I've lost track, but I don't think you've tried a couple of those ideas.

The only item I haven't tested with certinty here is #3. I'll go over a couple and explain why I ruled them out, please point out if my logic is flawed.



2. Verify that the .m2v and .m2a streams are the same playback length.

Since I can reproduce this issue in numerous streams even when I clip 5 minutes out using tytools, I highly doubt exploring this avenue is going to yield much of value.



3. Try multiplexing just the .m2v or the .m2a stream. You should be able to play back the multiplexed file .ty or .mpg on a computer.

I will try this tonight and report back.



4. If mplex is giving you errors about underruns or SCR values, you may need to adjust your audio delay or data rate.

Never seen any errors multiplexing the tivo sourced streams, or my Dish sourced streams.



6. Try transcoding the .m2a stream to standard rate mpeg2 audio.
7. For playback on a low-def tivo, make sure your .m2v is the expected rate.
8. ffmpeg can be handy to transcode the video to a low-enough rate and resolution for your low-def tivo.

Since I'm dealing with streams that came off the same tivo I'm trying to play them on, this probably isn't relevant?



9. If the above checks out but you still have tivo playback problems, check your /var/log/tverr

For anything specific? When I first began troubleshooting this issue I scoured them immediately after playing one of the problem streams, but nothing jumped out that seemed to be related to decoding.


Failing all that, you could provide a short stream sample that others can use to reproduce your problem, and then hopefully figure out what's going wrong.I thought you'd never ask. Where and how?

The only constant in this scenario that I've not really tried to eliminate is that I'm feeding the 'synthetic' ty streams back to the tivos using tivoserver (0.4.3, 0.4.4-a3, CVS build on slackware linux, etc.) and MRV. I had the impression that tivoserver didn't do anything with the streams if they are already in ty chunks, but maybe I've been mislead?

phat_bastard
05-16-2006, 12:17 PM
Using only the video stream resulted in the same stuttering / jerkiness. It may have been slightly less pronounced, but it was still there and in the same scenes.

Do you think I should setup mfs_ftp and try inserting one of the synthetic ty streams just to rule out tivoserver?

cheer
05-16-2006, 01:26 PM
Absolutely.

Deep in my heart I still think this is either (A) a field dominance issue or (B) an interlacing issue.

phat_bastard
05-18-2006, 10:06 AM
After giving it some thought, I recalled that when I inserted the unedited .ty file with tivoserver it plays back correctly, so that's not the issue either. I'll retest and confirm on build I'm running now, but I doubt anything's changed with handling of .ty files.

bcc
05-20-2006, 02:07 AM
Thanks for taking the time to respond. I was beginning to think I'd done something to offend you. If I have, I sincerely apologize.No; I've been traveling 3 of the last 4 weeks and not checking dealdatabase. tymplex is not much of a priority for me anyways as it's largely 3rd party software that others should be able to support as well as I. Also I don't get much out of supporting it as there have been nearly zero contributions since nova1 and I posted the patches; just questions and issues (exception: jamie).

I was stuck wondering whether or not you (and josh / tytool) are sythesizing the extension headers I'm seeing in the PES streams hdemux generates?There are many extension headers in iso 13818, I'm not sure which one you're speculating about. hdemux does not have to add extension headers. In the video stream, it parses the sequence_extension() to get information about the stream, not to modify it.
it doesn't seem like there are many ppl using mplex to do what I'm attempting.So what are you trying to do that is different? tivotool and tivoserver use tymplex to do their ty->mpg work. I believe many successfully use tivoserver to do what you are attempting.


Since I'm dealing with streams that came off the same tivo I'm trying to play them on, this [items 6, 7, 8] probably isn't relevant?Right, those ideas were for the expected tymplex use - converting non-tivo recordings to .ty format. In your case, I think you'd want to do chunkedit type editing to avoid re-multiplexing. But then you said you didn't want to do that out of concern for the STB's time (more important than yours? :) In that case you could implement chunkedit level editing on the host, assuming you don't need seemless cuts...

For anything specific?Check tverr for errors about alignment, PES headers, frmsizecod, overruns, start-codes, etc.

The only item I haven't tested with certinty here is #3.Perhaps the most important part of the troubleshooting info is the testing of the streams using known-to-work host-based players.
I downloaded the sample streams from you, and do not see any jerkiness that you are indicating when I play them using mplayer or xine. With xine I can notice some video combing problems, but then again I always do with my desktop. I assume you're having more serious playback issues than just combing? When I run your edited .mpg thru tymplex to make a .ty, it still plays back OK (same as .mpg but different a/v sync) on my desktop with mplayer. On my hr10-250, the video stutters a couple times as if there is an underrun. It would be nice to have the source .ty clip that goes with your .mpg samples to most easily tell if that stutter is a relatively basic underrun/overrun error due to mplex or a problem introduced from your editing, or ???

phat_bastard
05-22-2006, 10:46 AM
Also I don't get much out of supporting it as there have been nearly zero contributions since nova1 and I posted the patches; just questions and issues

I understand completely. If I could pinpoint the problem I'd be more than happy to invest the time as long as I know I'm not chasing my tail.


There are many extension headers in iso 13818, I'm not sure which one you're speculating about. hdemux does not have to add extension headers. In the video stream, it parses the sequence_extension() to get information about the stream, not to modify it.

If the extension headers are inside the frame payload then I doubt there's a need to further investigate that avenue.


Right, those ideas were for the expected tymplex use - converting non-tivo recordings to .ty format. In your case, I think you'd want to do chunkedit type editing to avoid re-multiplexing.

Aside from having seamless cuts, the other reason I was hoping to figure this out is so I can re-mux the 1/2 TB of DishNet sourced mpeg that I have that exhibit the same symptoms. Frameserving and decombing these sources makes them a good bit more bearable to watch, as well as considerably larger. :( I get in enough trouble with the wife as it is for buying 1/4 TB of storage space every three months.:D


On my hr10-250, the video stutters a couple times as if there is an underrun. It would be nice to have the source .ty clip that goes with your .mpg samples to most easily tell if that stutter is a relatively basic underrun/overrun error due to mplex or a problem introduced from your editing, or ???

I'll make it available for you shortly, so check your PMs. In regard to your comment about the problem being something I might have introduced by editing, I just wanted to re-state that this happens whether or not I make any cuts. Simply pulling apart the elementary streams and remuxing is enough to reproduce the issue.

Thanks for your attention on this - I'll end up owing you a drink or two on this.

bcc
05-22-2006, 07:32 PM
I'll make it available for you shortly, so check your PMs.Good, it'll be interesting to get to the bottom of this, and comparing against the source .ty should be the easiest way.
In regard to your comment about the problem being something I might have introduced by editing, I just wanted to re-state that this happens whether or not I make any cuts. Simply pulling apart the elementary streams and remuxing is enough to reproduce the issue.Ok, but it seems like you're also using tytool to convert .ty to .mpg, which may be restructuring the GOPs in the video stream. May be not - but I still lack the info to conclude whether it is the .ty->.mpg step or the remultiplexing that is introducing problems. I did try shrinking the GOP lengths to the the more "standard" size of 15, using videoredo on your .mpg, but that didn't help.
Thanks for your attention on this - I'll end up owing you a drink or two on this.Thanks, probably not realistic though. :)

phat_bastard
06-08-2006, 11:22 AM
Jumping to the appropriate thread...


ffmpeg is still not doing the right thing with timestamps in the above case.

Are you still seeing the PTS jumping backwards at the start of a GOP / progressive sequence, or is it something different?

Also, one thing I noticed in the sample stream I sent you was a sequence of progressive frames where the pulldown bit remained set even when it shouldn't have. I wonder how the decoder handles those cases.

i.e.:
[progressive frame, presentation order=1] [pulldown=1] (presented as 3 frames)
[progressive frame, presentation order=2] [pulldown=1] (presented as 3 frames) <<< ???
[progressive frame, presentation order=3] [pulldown=1] (presented as 3 frames)
[progressive frame, presentation order=4] [pulldown=0] (presented as 2 frames)

I guess it would make sense for the decoder to assume the pulldown bit should be off for frame 2 even though the picture coding extension says it's on. Regardless, D* uses some strange encoding methods.

bcc
06-16-2006, 10:38 PM
Are you still seeing the PTS jumping backwards at the start of a GOP / progressive sequence, or is it something different?Something different. ffmpeg avoids warping the PTS by leaving it out of the PES header of out of order frames. This seems to help the tivo quite a bit. But ffmpeg's compute_frame_duration() forgets to adjust the duration in the normal case to account for 3:2 pulldown. That looks like a misapplied patch to me (course I can't tell because none of it is commented or documented...). On the other hand it rounds off the result so this detail might not matter much in practice.
Also, one thing I noticed in the sample stream I sent you was a sequence of progressive frames where the pulldown bit remained set even when it shouldn't have. I wonder how the decoder handles those cases.

i.e.:
[progressive frame, presentation order=1] [pulldown=1] (presented as 3 frames)
[progressive frame, presentation order=2] [pulldown=1] (presented as 3 frames) <<< ???
[progressive frame, presentation order=3] [pulldown=1] (presented as 3 frames)
[progressive frame, presentation order=4] [pulldown=0] (presented as 2 frames)

I guess it would make sense for the decoder to assume the pulldown bit should be off for frame 2 even though the picture coding extension says it's on. Regardless, D* uses some strange encoding methods.As far as I know, the broadcom chip used in these pvrs works just fine when fed interlaced frames not in the typical 3-2-3-2 pulldown pattern. In fact it's one of directv's bit-saving tricks to use an atypical pattern. See the posts here from jdiner et. all. on this. I wouldn't go so far as to say they "shouldn't have" encoded things that way.

bcc
06-19-2006, 08:05 PM
ffmpeg avoids warping the PTS by leaving it out of the PES header of out of order frames. This seems to help the tivo quite a bit. But ffmpeg's compute_frame_duration() forgets to adjust the duration in the normal case to account for 3:2 pulldown. That looks like a misapplied patch to me (course I can't tell because none of it is commented or documented...). On the other hand it rounds off the result so this detail might not matter much in practice.A bit of an update...
Once I combined the audio stream with the video, it became obvious that ffmpeg's compute_frame_duration() needed my patch, and that the round-off error results in drift in practice. (The round-off was causing the video stream to play back at an artificially slow rate). So I tweaked ffmpeg's scaling of timestamps for the video stream, and that fixed the round-off. Now it looks like this problem stream plays muxes perfectly. Anyone want to play with some experimental ffmpeg source?

flagmaster
09-19-2006, 04:11 PM
Is there a way to do the DVD--->TY insertion via MFS_FTP without any xml?
I have tried twice and it worked up to about 500megs or so---then just completely crashed the upload task--I could not even ping my tivo once it crashed that task---but the tivo iteself was still running just fine.

Is there anyway to make a TMF file from a TY files made from the DVD in windows wihtout haveing to do it in linux?

Jamie
09-19-2006, 05:11 PM
Is there a way to do the DVD--->TY insertion via MFS_FTP without any xml?
I have tried twice and it worked up to about 500megs or so---then just completely crashed the upload task--I could not even ping my tivo once it crashed that task---but the tivo iteself was still running just fine.
If you lost the ability to ping your tivo, sounds like you have a network problem. What drivers are you using and on what hardware? I know some versions of the stock tivo drivers had "babble" problems and could freeze up under heavy load.


Is there anyway to make a TMF file from a TY files made from the DVD in windows wihtout haveing to do it in linux?I already pointed you to ty+2tmf.pl, and as described in its post, it can run on windows via either cygwin or ActiveState perl.

Neither of these questions directly relate to ty enabled mplex. As I mentioned in another thread, ty enabled mplex is a dead project. Current work is focused on ty format support in ffmpeg.

flagmaster
09-19-2006, 06:03 PM
Im running a standard linksys usb 200 driver set.
I have never had the network drop unitl I stated uploading ty or ty+ files.
I have successfully uploaded even those, even very large ones, so long as they where downloaded first from my tivo.

Im having trouble when uploading ty files I have made myself using the mplex program using dvd decrypted processed streams as the source.

The upload seems to die at random times. When the upload dies, my ping session also dies, but the tivo itself remains running just fine.

I was thinking if I take a ty file I just made from a dvd rip, and add a simple xml blob at the end of the file, and edit it with proper name, title, etc, then mfs_ftp would proabaly take it just fine as a tmf file.

Maybe your right, it may be the drivers are only effecetd durring high traffice uploading, but I moved over 2gigs in a tmf format without problems before.

insmod /lib/modules/usbcore.o
" " " /echi-hdc.o
" " " /ax8817x.o

note: I am running a monte'd kernal, from the S2 unscrable, still running that only because it gives me a slightly higher download speed for extraction. All of my current recordings are not scrabled at all.

---whats the easy way to maunaly add xml to the end of a straight ty file?
I got this in my cut/past clip board now....

(pulled from one of BCC's test pattern ty files).
----
<?xml version="1.1" tivoversion="3.1.5e-01-2-357"?>
<Object type="Recording" id="_top">
<StreamFileSize>13184</StreamFileSize>
<Duration>19</Duration>
<StartTime>0</StartTime>
<StopTime>19</StopTime>
<Title>PUT MY TITKE HERE</Title>
<Description>PUT MY DESCRIPTION HERE</Description>
<CallSign>tymplex</CallSign>
<Object>
######################################

BTUxNine
09-19-2006, 06:13 PM
mfs_ftp handles ty and ty+ (with xml at the end) identically UNTIL it reaches the xml, so that won't solve your problem

if it's dying at random points (not part boundaries) then it really is most likely the network setup... check that you have the latest unified mfs, too

If it's at part boundaries, you may want to try the patched version of mfs_ftp that uses the latest binary

Jamie
09-19-2006, 06:13 PM
I believe ty enabled mplex already adds xml at the end of the file. If a ty transfer is failing in the middle, it's before you are ever getting to the xml, so the presence or absence of xml is irrelevant. tmf is different in that the xml as at the start. That is one of the reasons it is more reliable.

You haven't said what tivo model and sofware version you are using. If it's a hr10-250 that is still running 3.1.5x, I'm pretty sure the stock usb2 drivers have the babble freeze problem.

flagmaster
09-19-2006, 10:42 PM
Yes, its an Hr10-250. Yes its the stock drivers.

Get this tho---I downloaded with mfs_ftp the HD TV file "big". Not only is the name big, the file is about 6gigs.

After it downloaded just fine, I started to upload it into the ty directory (not tmf, not ty+) and so far, its inserting and uploading just fine.

It says it will take anotehr 4 hours to complete the upload, but its arleady gotten fartehr the any other ty's I have tried to upload.

Could it be a corupted ty file casuing the crash?

I suspose I could run tytool on my ty file I made from the dvd and see if it can make a functional mpgs out of it.

Anyone know if videoredo can take an mpeg and break it into the ac3 and m2v files? I have not tried that yet.

Carey

BTUxNine
09-19-2006, 10:45 PM
Directory is completely irrelevant for mfs_ftp uploads... it uses the filename extension

nova1
09-19-2006, 11:05 PM
Several people have ripped VOBs from DVDs and used tymplex(mplex -f 10) to create TYs and insert them into the HR10-250. Those TY's are usually >5GBs. You should look into using the backported drivers. DVDDecryptor can demux VOBs into AC3 and M2V files.

flagmaster
09-20-2006, 08:08 AM
The big.ty file (6gigs) uploaded and inserted perfectly no errors.
This has to be the tyfile itself causing the trouble I would think...

Can you point me to a known good working copy of tymplex for windows that supports the hr-10-250?

Im using mplex_cc.exe

I have a "nova1" patch text file, but I dont know what to do with it. for now its just in the same dir as the exe. I have never compiled anything, so if I need to recompile this patch, I dont know how.

It would seem that any ty files that I "make" using this exe are the ones that crash out at random times during the upload---not durring part points.

Thanks again for all the support.

The syntax Im using is:

mplex_cc -f10 -o saw.ty audio.ac3 video.m2v



Carey

nova1
09-20-2006, 09:34 AM
mplex_cc.exe is the latest and is known to generate valid TY for upload to the HR10-250. This thread has some other versions as well that you can try. Can you include some output from the mplex_cc.exe run?

You can also try the ffmpeg tool here. (http://www.dealdatabase.com/forum/showthread.php?t=49935)
ffmpeg -i src.mpg -acodec copy -vcodec copy dest.ty

Have you tried using the backported drivers as suggested?

Also there is a support thread for ty enabled mplex here. (http://www.dealdatabase.com/forum/showthread.php?t=42006)

Jamie
09-20-2006, 12:55 PM
Another thread (http://www.dealdatabase.com/forum/showthread.php?p=264425#post264425) today reminded me that there is a bug in mfs_ftp.tcl that affects ty insertion. If you are having trouble with ty insertion, you might follow the link in that thread and try the change suggested there.

flagmaster
09-20-2006, 04:56 PM
Ok, I had an interesting success so far...

I took an MPG file that was generated with tytools. I put that thru videoredo, and saved as elemtry streams.

I had to rename the video stream to m2v.

Then I tried mplex_cc to make a ty file out of that, it did, and when I tried to upload it, I had the same old problem....it crashed out in the middle of the transfer.

So I made a fresh ty file using ffmpg, and when I uploaded that ty file, guess what??!! IT WORKED!

So now Im making a ty file from a dvd again, and once its done (soon) I will upload it and see what happens.

I will post the result soon. At this point, the problem looks like the generated ty file itself is the problem.

carey

flagmaster
09-20-2006, 06:46 PM
The upload of a DVD ty created by mpegff worked perfect!

In my eyes, this confrims the problem was with ty's created witth mplex_cc.

Do you all agree?

BTUxNine
09-20-2006, 07:47 PM
As was said before, this is a defunct project... ty-enabled ffmpeg replaces it because it handles such things more reliably. (in other words, people already know there are problems, but it's not worth trying to fix when there's a better alternative)

flagmaster
09-20-2006, 11:45 PM
I have not had 100% success rate with this fix, but its way better then what I had before with mplex_cc. (zero success).

On thing I just noticed, to try to speed up the transfer rate, I put both tunners onto channles I dont get---in my case 33 sat and 55 sat.

When I did this, it crashed my transfer.

Dose it make any sense that the transfer is more reliable if at least one, if not both, tuners are set to a real sat channle?

Transfer rates start at about 1megbyte per sec, but the ones that seem to acutally work, drop to less then 1/2 of that. Im at 481K per sec now.

Should I continue to consider new drivers?

Whats invovled in that, just replacing the files in /libs/modules with new ones? Whats the best drivers to use on the HR10-250 with my adapter?

(I have a netgear FA120---I had the USB200 but it broke.)

BTUxNine
09-20-2006, 11:49 PM
look in the files section for "backport"

Jamie
09-21-2006, 12:05 AM
look in the files section for "backport"That's a good guess, but for one reason or another, the files are posted in the development thread instead. Look here (http://www.dealdatabase.com/forum/showthread.php?t=44114) and follow the link in the first post to the latest version. See the README in the package for an overview and instructions.

If I was you, I'd install the backport drives, the mfs_ftp patch I mentioned earlier, and the latest mfs-utils (aka mfs_*).