Page 7 of 22 FirstFirst ... 5678917 ... LastLast
Results 91 to 105 of 326

Thread: hdemux: New tystream demuxer

  1. #91
    Join Date
    Nov 2002
    Posts
    1,076
    Wow, two hdemux processes. But both processes are reading from the same input pipe at the same time? Not sure how that could possibly work right - I think that's your problem. Each process is probably getting about 50% of the source chunks. The reason why hdemux has separate a/v file switches was so that you could theoretically output to 2 named pipes at the same time (not out of trying to make the UI cumbersome like some might think Personally I never did use it that way (read: untested). I actually at one point merged mplex&hdemux together into one program (with some c++ glue and memory buffering between them). That lets you go ty->mpg without extra intermediate disk or pipe I/O. Result was a bit messy and I didn't want to get into the mplex support+troubleshooting business. Not to mention mplex failing to be sufficiently portable (won't compile with visual c++ for example). But if you do want to get into that business maybe I could get you some code... Problem is I'm on the road the rest of the week.

  2. #92
    Join Date
    Feb 2004
    Posts
    155
    Thanks bcc, I got this sort of working with tees. Gotta hack on it this weekend and see what becomes of it.

    Code:
    hdemux -i /Users/john/.tivotool/tyfile1 -a /Users/john/.tivotool/pipe.m2a -v /dev/null &
    sleep 1
    hdemux -i /Users/john/.tivotool/tyfile2 -v /Users/john/.tivotool/pipe.m2v -a /dev/null &
    sleep 1
    tee ~/.tivotool/tyfile1 < ~/.tivotool/tyfile > ~/.tivotool/tyfile2 &
    sleep 1
    cat /Users/john/Desktop/ty\ Collection/Samurai\ Jack\ -\ IV.ty > ~/.tivotool/tyfile &
    sleep 1
    
    # This works like it should.. output the two streams..
    cat ~/.tivotool/pipe.m2v > test.m2v
    cat ~/.tivotool/pipe.m2a > test.m2a
    
    # Seems like this should work, two streams, valid command line
    #mplex -f 8 -o ~/test.mpg ~/.tivotool/pipe.m2v ~/.tivotool/pipe.m2a
    Thats where I'm at. Not really an hdemux issue, but this is the same behavior I got when I did something like

    Code:
    hdemux -i pipe.ty -a pipe.m2a -v pipe.m2v
    mplex -f 8 -o ~/test.mpg pipe.m2v pipe.m2a
    I would assume mplex is broke, but this line is used in the TivoServer program with success:
    Code:
    mplex -f 10 -r $bitplus -o $homed/pipe.tys $homed/pipe.m2v $homed/pipe.m2a &> $homed/mplex.log &

    Quote Originally Posted by bcc
    mplex support+troubleshooting business.
    No, thanks =) My goal here is to get vsplit out of Tivotool to HELP with all the support questions.
    Last edited by johnsolo; 10-07-2005 at 04:25 PM.
    TivoTool - Easy OSX Video Extraction

  3. #93
    Join Date
    Nov 2002
    Posts
    1,076
    Quote Originally Posted by johnsolo
    I would assume mplex is broke,
    Well broke probably isn't the right word for it... The a/v streams don't generate data at the same data rate, and so you're gonna get an underrun pretty readily if you try to read from them at a constant rate like mplex does initially. Here, underrun means the fifo gets blocked in pipe_wait. You can verify this with ps. Splitting the stream generation with tee and an extra hdemux doesn't get rid of the blocking problem - you still hit the same problem through back-pressure. If you're gonna do this with pipes, you do need to at least add buffering like you thought. Perhaps increasing the socket buffer sizes to 64k or 128k will be enough.

  4. #94
    Join Date
    Feb 2004
    Posts
    155
    Quote Originally Posted by bcc
    Perhaps increasing the socket buffer sizes to 64k or 128k will be enough.
    Alright, I understand about 75% of that. =) Lots of questions coming your way. First, what is the switch for ps to show if something is in pipe_wait?
    I am playing with buffering commands. Not sure what you mean by the socket buffer sizes. Do you mean inside the hdemux source?

    Do you know how I would seperate the audio and video stdout streams of hdemux? Leaving out -a and -v would send both streams to stdout right? How do I figure out which is which? Also, "hdemux -i input.ty > out.streams" doesn't seems to actually redirect for me. Any ideas?

    Finally, theres no way to get the total size of the stream in hdemux, is there? I imagine you are processing it as a stream, so you don't know where the end is. Anyways just checking (need max value for a deterministic progress bar =)
    TivoTool - Easy OSX Video Extraction

  5. #95
    Join Date
    Nov 2002
    Posts
    1,076
    Quote Originally Posted by johnsolo
    Alright, I understand about 75% of that. =) Lots of questions coming your way. First, what is the switch for ps to show if something is in pipe_wait?
    You can use ps axlw and look at the WCHAN column. 'Course it's truncated, so you might want something like ps axwo pid,wchan:12,comm to see the full pipe_wait name.
    Quote Originally Posted by johnsolo
    I am playing with buffering commands. Not sure what you mean by the socket buffer sizes. Do you mean inside the hdemux source?
    To be precise, I mean the socket buffer(s) for the named pipe. Those buffers are maintained by the kernel, but could be set via setsockopt(). I believe you could normally do this thru a script, if one was simply piping between commands. See mfs_ftp's use of fconfigure for an example, using 128k buffers. But this case is more complicated, where hdemux should really have 2 output pipes. In this case I don't know how you could do it thru a script. And so, yes, I'd do the setsockopt() from hdemux.
    Quote Originally Posted by johnsolo
    Do you know how I would seperate the audio and video stdout streams of hdemux? Leaving out -a and -v would send both streams to stdout right? How do I figure out which is which? Also, "hdemux -i input.ty > out.streams" doesn't seems to actually redirect for me. Any ideas?
    Well I guess if you really wanted to make the output redirectable, and you wanted to keep the a/v streams separate, one could be sent to stdout, and one to stderr. But that would be a strange UI.
    Quote Originally Posted by johnsolo
    Finally, theres no way to get the total size of the stream in hdemux, is there? I imagine you are processing it as a stream, so you don't know where the end is. Anyways just checking (need max value for a deterministic progress bar =)
    Of the input stream? hdemux can stat the input file and know the size that way. But last I knew you were streaming into hdemux so that wouldn't work. Otherwise hdemux could see the size of each segment from the master chunk record, but it wouldn't know how many segments are being streamed to it.

  6. #96
    Join Date
    Feb 2004
    Posts
    155
    Hi bcc, thanks for all your helpful advice. Just wanted to let you know I am using hdemux in TivoTool now as an optional demuxer for S2 users. I'll keep an eye out in this thread if you ever release that S1 code. If you would want to share that, I could try to patch and compile that in (for OSX at least..) and see how it goes with my S1 testers.
    TivoTool - Easy OSX Video Extraction

  7. #97
    Join Date
    Nov 2002
    Posts
    1,076

    hdemux with support for low-def tivos

    Here is a new release of hdemux. Changes include:
    1. Add support for S1 dtivo, S2 standalone tivo
    2. Change parsing of video frames to accommodate these older models which do not place PES frames adjacent to video sequence headers.
    3. Change read processing to allow input from blocking device such as a pipe or fifo.
    Last edited by bcc; 12-02-2005 at 02:46 PM. Reason: newer version below

  8. #98
    Join Date
    Feb 2004
    Posts
    155
    Huge update. great job bcc.

    Tested on a couple UK S1 streams and it works fine. (Also fine w/ S2SA and S2DTivo) Gonna play with named pipes soon.

    Looking forward to ditching vsplit and all the headaches it carries with it....
    TivoTool - Easy OSX Video Extraction

  9. #99
    Join Date
    Jul 2001
    Location
    WNC
    Posts
    72
    Just a quick note - I am using hdemux to demux music channels using s1 DTIVO 3.1.0b. The "timestamp discontinuity" stuff causes problems. the extracted audio only plays for 5-10 seconds before it skips forward about 5 minutes, plays for another 5 seconds, skips forward again...

    so I disabled the timestamp discontinuity whatever stuff, and OH YEAH BABY!
    perhaps this check could become optional in future versions of hdemux? since this check deals primarily with VTS, maybe make the -v flag optional and this check along with it?


    --- hdemux.c-orig 2005-11-17 08:39:29.000000000 -0500
    +++ hdemux.c 2005-11-17 08:39:54.000000000 -0500
    @@ -1323,7 +1323,7 @@
    }
    chunk_parse_rec(p, &type, &major, &time);
    if (!i) {
    - if (!time_validate(time, lasttime, vstream.ticks_per_chunk)) {
    + if (0) {
    if (errcnt++ < CONTINUITY_MAX_ERROR) {
    /*
    * Be silent about first discontinuity.

    BEFORE:
    D:\xtract>hdemux -i bpm-.ty -a test.m2a -v test.m2v
    Version 0.12
    Source is bpm-.ty
    MPEG2 audio at 48kHz, 576 bytes/sync frame
    Warning: Skipping chunk 4 due to timestamp discontinuity
    Warning: Attempting resync due to repeated time discontinuity
    Warning: Skipping chunk 68 due to timestamp discontinuity
    Warning: Attempting resync due to repeated time discontinuity
    Warning: Skipping chunk 132 due to timestamp discontinuity
    Warning: Attempting resync due to repeated time discontinuity
    Warning: Skipping chunk 196 due to timestamp discontinuity
    637466 audio stream bytes 0 video stream bytes

    THANKYOU!
    Last edited by khmann; 11-17-2005 at 10:22 AM.

  10. #100
    Join Date
    Nov 2002
    Posts
    1,076
    The timestamp continuity check has sometimes caused problems, on the other hand, it also has done a good job of fining bad chunks/discontinuities. Yes, a 'loose' switch that disables the check is probably in order.
    Quote Originally Posted by khmann
    since this check deals primarily with VTS
    Well the code is a general PTS timestamp check, but was presuming that your data rate included a video stream. The code probably should adjust the data rate if no video stream is found. Making -v/-a optional makes sense too, and the following accomplishes that
    Code:
    --- hdemux.c~   2005-11-06 17:04:08.000000000 -0800
    +++ hdemux.c    2005-11-17 10:48:28.000000000 -0800
    @@ -121,6 +121,9 @@
            printf("Writing %d to stream %s\n", size, stream->name);
         }
         ofd = stream->fd;
    +    if (!ofd) {
    +       return;
    +    }
         ret = write(ofd, buf, size);
         if (ret != size) {
            error("error on write: requested: %d got: %d", size, ret);

  11. #101
    Join Date
    Jul 2001
    Location
    WNC
    Posts
    72
    nice. I'm finding occasional glitches in the xm audio streams and am having problems after muxing with locally encoded video - seeking during playback, issues with DVD authoring apps, etc. I don't have any of these problems when using locally encoded audio.

    for PTS checking/correction you might find something interesting in "replex" @ http://www.metzlerbros.org/dvb/index.html

    For my application (combine extracted music with "timecode" video to make music DVDs so I don't have to transcode or buy mp3 players) a mplex mux > replex demux with PTS regeneration seems to give me an audio stream which behaves better, dunno why. I'm going to try to hack replex to operate on ES'...

    anyway, thanks again for hdemux. Apologies for hijacking the thread for a completely unrelated purpose :)
    Last edited by khmann; 11-22-2005 at 12:28 AM.

  12. #102
    Join Date
    Oct 2003
    Posts
    7

    Two Bug Fixes in a Patch

    While working with TivoTool under OS X, I encountered two issues detected by mplex in the ac3 audio streams:
    1. Periodically, an extra pair of nulls (two bytes of zero) would appear in the next to last byte of the AC3 frame, making it 2 bytes too long.

    I found this was the result of some code beginning at line 1158 that starts:
    if (astream.syncoffset && (astream.syncoffset < 4) && overlap)

    This special case caused the extra bytes to be written when it appears there is no legitimate special case. This code is commented out.

    2. After resolving this issue, a further error appeared in mplex indicating that no audio packet was found at a particular instant. This corresponded roughly to a message "Warning: Tossing AC3 audio frame with bad sync data."

    I also noted that as many as a quarter of the ac3 packs were marked as having recovered synchronization. This is clearly inappropriate for a stream the was visually flawless.

    Each of these 'recovered' cases involved a scan of the AC3 data to locate the sync bytes. Unfortunately, unlike most protocols, AC3 data does not exclude the sync pattern from appearing in valid data. This means that having found a sync word may not indicate that a valid packet start has been found.

    Because the sync recovery operation was being invoked far more often than it should have been, it would occasionally find a sync word in the AC3 data and conclude that a packet should be discarded.

    The sync recovery code was executed because the audio stream syncOffset value was only set in one of the two main cases. To maintain sync it needs to be set every time a frame is extracted.

    For whatever reason, neither of these paths seem to be exercised with satellite video, but of the air HD hits them immediately. I have seen no indication that these problems would be unique to either use with TivoTool or with an OS X compiled version.

    A patch file for hdemux.c in the 0.12 version is attached. Hopefully it is constructed correctly.

    john brisbin

  13. #103
    Join Date
    Nov 2002
    Posts
    1,076
    Actually no, that "fix" is not correct, as there is a legit case being covered by the condition you commented out. What is the actual problem you seeing with mplex, or is it just a message you get when you enable verbose diagnostics (-v 2)?
    Now, I tried to describe that case in the comment right there:
    Code:
     * Sometimes OTA PES payload overlaps with next PES
    * packet's start code prefix.
    In the problem case I've seen, the next PES header's start code is "sharing" two bytes of zeros with the payload from the AC3 frame. Sharing in the sense that thse 2 bytes are required to complete the 1536 byte ac3 frame, and later expected to not have already been consumed by my code that parses the start code. Your change would result in mplex bombing out on this stream due to the ac3 frame being truncated.
    Here's an example:
    Code:
    gdb /u3/tivo/tyconv/hdemux
    GNU gdb Red Hat Linux (6.3.0.0-1.21rh)
    Copyright 2004 Free Software Foundation, Inc.
    GDB is free software, covered by the GNU General Public License, and you are
    welcome to change it and/or distribute copies of it under certain conditions.
    Type "show copying" to see the conditions.
    There is absolutely no warranty for GDB.  Type "show warranty" for details.
    This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/libthread_db.so.1".
    
    (gdb) br hdemux.c:1169
    Breakpoint 1 at 0x804a538: file hdemux.c, line 1169.
    (gdb) run  -i oc-rager.ty -v o.m2v -a o.m2a
    Starting program: /u3/tivo/tyconv/hdemux -i oc-rager.ty -v o.m2v -a o.m2a
    Reading symbols from shared object read from target memory...done.
    Loaded system supplied DSO at 0x40000000
    Version 0.12
    Source is oc-rager.ty
    Video frame size is 1280x720  - high definition. Frame rate 59.940060
    Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
    Audio timestamp offset: 374.044444 ms
    AC3 audio at 48kHz, 1536 bytes/sync frame
    Recovered AC3 sync, offset 142, chunk 2
    Reported datarate 19384 Kbit/sec. (19000+384)
    Proceed with remuxer:
            mplex -f 8 -O 374ms -r 29076 o.m2a o.m2v -o <outfile>.mpg
    
    Breakpoint 1, parse_audio_frame (start_code=445, major=9, chunk=7, 
        headlen=0xbff4d41a, prepos=0xbff4d410, size=0xbff4d43c, 
        framesize=0xbff4d414) at hdemux.c:1169
    1169                    if (dbg(1)) {
    (gdb) p payoffset
    $1 = 122
    (gdb) p paybytes
    $2 = 1642
    (gdb) p *framesize
    $3 = 1536
    (gdb)  x/20b &astream.buf[122+1536-4]
    0x4001267e:     0xf1    0xb0    0x00    0x00    0x01    0xbd    0x05    0xba
    0x40012686:     0x80    0x80    0x05    0x2d    0xb4    0x17    0xa9    0x05
    0x4001268e:     0xa3    0xce    0x0b    0x77
    (gdb)
    I could probably handle your case by making the above special case check consider whether the subsequent data actually has a 000001bd start code, but I'd like to understand the case fully before doing that.

    As for case 2, yes that warning means that hdemux is in a world of hurt (maybe it should just assert), and you shouldn't see that in the middle of a healthy stream. Perhaps you're causing that situation thru your case 1 change, but I'd have to see your stream to be sure.
    Last edited by bcc; 12-01-2005 at 02:48 AM.

  14. #104
    Join Date
    Oct 2003
    Posts
    7
    Quote Originally Posted by bcc
    Actually no, that "fix" is not correct, as there is a legit case being covered by the condition you commented out. What is the actual problem you seeing with mplex, or is it just a message you get when you enable verbose diagnostics (-v 2)?
    Now, I tried to describe that case in the comment right there:
    Code:
     * Sometimes OTA PES payload overlaps with next PES
    * packet's start code prefix.
    In the problem case I've seen, the next PES header's start code is "sharing" two bytes of zeros with the payload from the AC3 frame. Sharing in the sense that thse 2 bytes are required to complete the 1536 byte ac3 frame, and later expected to not have already been consumed by my code that parses the start code. Your change would result in mplex bombing out on this stream due to the ac3 frame being truncated.
    That is a novel idea, but it is hard to justify coding to make a broken stream work and a legal one fail. Byte sharing is not legal (and I would argue does not happen in practice, even bad practice.)

    The case that I saw (many times) is one where the last two bytes are not 0 but are properly placed after the header information following the 0x000001bd marker (just as they are in the gdb example you provided). Two bytes are just as legitimate in this position as the times when 50 bytes overlap into the next packet. You can not assume that less than 4 is somehow magic.

    The fix you talk about would only work if the first data byte of the next packet were the first byte of the sync code. If it is not, you will write two additional bytes and the 0x700 byte frame becomes 0x702 which breaks any attempt to remux the data. In other words, this trick breaks any legal frame that extends into the next packet by anywhere from 1 to 3 bytes based on the value of astream.syncoffset.

    astream.syncoffset was just determined by verifying the location of the next sync byte a few lines above in function ac3_syncoffset. The data exists in the correct place, not sharing space with the start code. ac3_syncoffset found it there!

    The commented out lines just insert 1 to three bytes (2, in practice, because everything is even sized) of extra data in an otherwise perfectly good frame.

    In your gdb example, the now commented out code is trying to ignore the valid data bytes at offset 0xe and 0xf (0xa3 0xce) into the new pack. Instead a pair of zeros and the 0xa3 and 0xce get written to output and mplex rightly complains.

    If you throw away the idea of byte sharing instead of the bytes themselves, the special case disappears and everything works better.

    Quote Originally Posted by bcc
    As for case 2, yes that warning means that hdemux is in a world of hurt (maybe it should just assert), and you shouldn't see that in the middle of a healthy stream. Perhaps you're causing that situation thru your case 1 change, but I'd have to see your stream to be sure.
    To refute the assertion that the first fix might have caused this, the now commented out code only ran about once in twenty frames or only a fourth as often as the rescanning occurred in the same file. It could not be responsible for all the rescanning that occurs.

    As I explained in the first message, it only gets in a 'hurt' because it is not maintaining the astream.syncoffset except when the expression (paybytes > (*framesize +4)) is true. When that is false (about 1 in 5 times in the case I was looking at) the field value misinforms find_ac3_frame (which depends on it for the in sync case) and it does a byte by byte scan of the data to locate the 0xb77.

    This extra scanning is innocuous when the next instance of 0xb77 is the sync word, but eventually, with enough data, the scan picks up a 0xb77 in the audio data and throws the packet away when it cannot validate the AC3 configuration info that follows it.

    I would ask you to look at your code and explain how the function find_ac3_frame can work properly for the subsequent frame if the prior one does not set astream.syncoffset?

    All theory and argument aside, I am enjoying having OTA streams that remux without error.

    Thanks for making the tool available.

  15. #105
    Join Date
    Nov 2002
    Posts
    1,076
    Quote Originally Posted by MelvinPurvis
    That is a novel idea, but it is hard to justify coding to make a broken stream work and a legal one fail. Byte sharing is not legal
    Actually no, this is not an illegal case. It just sounds that way if you think of it as byte sharing. If you look at the way start code processing is defined in iso13818, you must merely have 23 zero bits, and a 1 bit before the start code. The spec doesn't say that you must insert 23 zero bits and a 1 bit before the next start code. In other words there is no requirement that those bits must not overlap the end of an access unit.
    But since I don't parse the stream in such a general bit stream fashion, I had to code a special case.
    Quote Originally Posted by MelvinPurvis
    (and I would argue does not happen in practice, even bad practice.)
    I guess I can't stop you, but I've already shown you an example stream proving otherwise. It's a real OTA case from fox's 'The OC' (no I don't watch that show I just got the stream sent to me I also have the same special case from 'jay leno'.

    Now, you've shown me some diffs that break the above cases. Even if you're unable to admit that the above cases are "legal" your changes violate the "be liberal in what you accept" rule, and thus break real world streams. You're supposedly fixing another case, but you have not made fully clear the circumstances for that case.

    I'll come up with a more general fix that'll probably cover your case. If you could provide a sample stream I could be sure. The existing code has worked fine for all streams reported thus far (a fairly wide variety IMO).

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •