Page 9 of 22 FirstFirst ... 789101119 ... LastLast
Results 121 to 135 of 326

Thread: hdemux: New tystream demuxer

  1. #121
    Join Date
    Nov 2004
    Location
    Gurnee, IL
    Posts
    2,384

    error: Cannot fit Video buffer

    OK. Had a number of files that exhibit the "Can't find next AC3 frame" problem in mplex, but after perusing this thread I'm gonna play with the audio files a bit and see what I can see. However, new problem, and this one is actually in hdemux itself:
    Code:
    Version 0.13
    Source is er.ty
    Video frame size is 1920x1080  - high definition. Frame rate 29.970030
    Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
    Audio timestamp offset: 461.400000 ms
    AC3 audio at 48kHz, 1536 bytes/sync frame
    Reported datarate 65384 Kbit/sec. (65000+384)
    Proceed with remuxer:
            mplex -f 8 -O 461ms -r 153701 er.m2a er.m2v -o <outfile>.mpg
    error: Cannot fit Video buffer, size 120704 offset 412740
    Any ideas? It's an OTA recording (of course)...the error comes up almost immediately.
    --
    Christopher D. Heer
    Quote Originally Posted by Oscar Wilde
    Perhaps, after all, America never has been discovered. I myself would say that it had merely been detected.

  2. #122
    Join Date
    Nov 2004
    Location
    Gurnee, IL
    Posts
    2,384
    Update on the audio issues.

    OK, so first I grabbed besweet and tried converting the audio to mp2 audio. That changed my error -- now I get a comment saying it needed to create a new file but no %d was present. So I added %d to the output filename and tried again, and I got three MPG files.

    Not liking that, I tried converting the audio to mp3. Same problem. So now I tried converting it to wav and then back to ac3. Back to the "can't find next AC3 frame" problem.

    So I downloaded ac3fix and ran the file through that. Same problem. So then I downloaded besplit and ran the file through that. Died with a different error (don't remember and didn't log it).

    So then I tried running through ac3fix and then besplit. Now I get
    Code:
    ++ WARN: [mplex] No seq. header starting new sequence after seq. end!
    **ERROR: [mplex] Sequence split detected 3 but no following sequence found...
    Augh! Anyone have any other ideas on how to fix a b0rked ac3 stream? I'm sure it's what bcc speculated: it's flipping between m2a and ac3 in the stream, or something similar.

    FWIW, I tried TyTool10R4...couple of bad dumps, but it keeps processing...until it just stops and says complete, far short of the actual file end.

    Note that hdemux did indeed spit out errors when demuxing:
    Code:
    Version 0.13
    Source is house.ty
    Video frame size is 1280x720  - high definition. Frame rate 59.940060
    Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
    Audio timestamp offset: 408.544444 ms
    AC3 audio at 48kHz, 1792 bytes/sync frame
    Recovered AC3 sync, offset 80, chunk 1
    Reported datarate 19448 Kbit/sec. (19000+448)
    Proceed with remuxer:
            mplex -f 8 -O 408ms -r 29172 house.m2a house.m2v -o <outfile>.mpg
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 1546 length 1840
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 1818 length 1840
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 2107 length 1840
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 5206 length 1840
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 5547 length 1840
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 6523 length 1840
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 8454 length 1840
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 9892 length 1656
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 10719 length 1656
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 11224 length 1656
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 11264 length 1656
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 11333 length 1656
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 11850 length 1840
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 12520 length 1840
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 13547 length 1840
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 13886 length 1840
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 17868 length 1840
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 19551 length 1840
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 21256 length 1840
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 21729 length 1840
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 22667 length 1656
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 22824 length 1472
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 23183 length 1472
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 23239 length 1656
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 23408 length 1656
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 23764 length 1656
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 24484 length 1840
    Warning: Tossing AC3 audio frame with bad sync data.  Chunk 26227 length 1840
    Warning: Skipping chunk 30384 due to timestamp discontinuity
    201142728 audio stream bytes 3763501022 video stream bytes
    What sucks is that OTA is the best quality HD out there...but the flakiness of the streams (esp. on audio) makes it such a PITA to deal with.
    --
    Christopher D. Heer
    Quote Originally Posted by Oscar Wilde
    Perhaps, after all, America never has been discovered. I myself would say that it had merely been detected.

  3. #123
    Join Date
    Nov 2004
    Location
    Gurnee, IL
    Posts
    2,384
    OK. I pulled every OTA HD show off my HR10-250 that was there. Not tons -- it is rerun season after all.

    Of the nine shows that I pulled off, I was successfully able to hdemux/mplex two of them. Of the seven that failed:
    • 2 failed during hdemux with "Cannot fit video buffer"
    • 2 errored during hdemux (repeated tossing audio frames with bad sync, as well as skipping chunk due to timestamp discontinuity) and failed during mplex (Can't find next AC3 frame, broken bitstream?)
    • 1 errored during hdemux (repeated tossing audio frames with bad sync, as well as skipping chunk due to timestamp discontinuity) and failed during mplex (no seq. header starting new sequence after seq. end! and sequence split detected 3 but no following sequence found)
    • 1 errored during hdemux (tossing audio frames w/bad sync, skipping chunks due to timestamp discont., AND attempting resync due to repeated time discontinuity) and failed during mplex (Can't find next AC3 frame, broken bitstream?)
    • 1 errored during hdemux (just skipping chunk due to timestamp discontinuity) and failed during mplex (Can't find next AC3 frame, broken bitstream?)
    I tried all sorts of things on the audio files, including running them through ac3fix, bbsplit, delaycut, mixtures of the above, and re-encoding to various other formats.

    Any ideas? Also, any suggestions for a different muxing tool?

    Just for a sanity check I'll run all of these through TyTool and post the results.

    EDIT: Attached are logs from my tests. They include the output of hdemux, the output of mplex (if I got that far), and a file listing to show the ending file sizes.
    Last edited by cheer; 06-16-2006 at 06:06 PM. Reason: Added attachment
    --
    Christopher D. Heer
    Quote Originally Posted by Oscar Wilde
    Perhaps, after all, America never has been discovered. I myself would say that it had merely been detected.

  4. #124
    Join Date
    Nov 2002
    Posts
    1,076
    Where to start? You seem to be reporting 3 actual errors, below:

    In general, "Can't find next AC3 frame" errors from mplex could be due to mplex issues, hdemux bugs, or problems with the actual stream. From your later messages, it seems like we can rule out the first scenario, as you're getting errors at the hdemux step. For a healthy ac3 stream, I'd expect to see the '0b 77' marker bytes repeating at a constant interval. You haven't sent me a stream, so I really can only guess what happened here. My guess would be that you truly have problems with your source stream, but that hdemux could do a lot better job in recovering from those problems. For example it should probably split the audio stream into segments if the stream changes between mpeg2 and ac3. If hdemux could assume a multiplexor that was able to handle such changes, this might not be necessary. But mplex is not such a multiplexor. You can work around this issue by pre-editing your .ty file with a chunk editor. Your "lost" demux log looks like a more complicated failure than that - but I can only guess from the log. Would be interesting to see that stream.

    Timestamp discontinuities:
    It is "expected" to see this from hdemux concerning chunks at the end of the file. Compare the # of chunks in the source vs the chunk # where the error is reported. Later versions of hdemux fix this so that you don't see this cosmetic warning. If you're seeing this from hdemux at some other point besides the end of the file, then you can expect problems with the demuxed streams as well. For those cases, I would verify with a .ty player such as mplayer whether or not there is a hiccup in the stream at the point that hdemux is claiming. Use tychopper or dd to splice to the problem section. Looks like most of your logs are from the cosmetic case, the exception being the two with the "attempting resync" messages.

    "Cannot fit video buffer":
    This can either be a symptom of an algorithm bug or a simple buffer sizing problem. I believe I did find the video buffer undersized since 0.13, so I probably simply need to make a new release. Again, lacking a sample, all I can give is my best guess.

    Quote Originally Posted by cheer
    Also, any suggestions for a different muxing tool?
    There's always ffmpeg, but your problems seem to be before you get to multiplexing.

  5. #125
    Join Date
    Nov 2004
    Location
    Gurnee, IL
    Posts
    2,384
    bcc, thanks for the detailed response.

    So far as the AC3 frame stuff...yeah, I'm sure it's the source streams that are hosed. You mention pre-editing my tystreams with a chunk editor...never done such a thing, but I'll give it a shot. Are you suggesting that I make edit points around the spots where the problems are and then process separately? I'll play around and see what I can find.

    The time seq. stuff...I'll fire up mplayer and see if I can spot stream errors, etc., near the spots hdemux identifies. If the other remarks (from the end of the file) are cosmetic, then I presume my mplex probs stem from something else (again, the fubar ac3 streams being a likely culprit).

    I can easily give you samples of the streams...in fact, PM me a mailing address and you'll have some DVD-Rs magically appear forthwith.

    Thanks for all your help!

    --chris

    P.S. Would it be of any use to see logs from something like ac3fix? Prolly not since by that point hdemux is already done, but...

    P.P.S. For laughs, I ran all the streams through TyTool10R4. That doesn't do me much good (because I need to be able to batch process these things), but...more information better than less, I suppose. Anyway, all but two appear to have spit out valid MPG files (I haven't watched 'em yet). The two that failed were House -- no errors (apart from Bad Dump messages, but I get those all the time on successful OTA HD muxes) and Lost. Lost actually caused TyTool to GPF (or whatever they call it now) -- dialog box offering to send a report to MS, yadda yadda yadda. The .txt file generated by TyTool is full of all sorts of issues -- OOB packets, unknown PES packets, damaged video frames, and finally (just before the crash) an audio type change.

    If those logs would be of any value, let me know; not sure if they mean anything to anyone but Josh.
    --
    Christopher D. Heer
    Quote Originally Posted by Oscar Wilde
    Perhaps, after all, America never has been discovered. I myself would say that it had merely been detected.

  6. #126
    Join Date
    Nov 2002
    Posts
    1,076
    Quote Originally Posted by cheer
    You mention pre-editing my tystreams with a chunk editor...never done such a thing, but I'll give it a shot. Are you suggesting that I make edit points around the spots where the problems are and then process separately? I'll play around and see what I can find.
    Since you're using OTA streams, they are likely fully of transitions in&out of HD, and varying AC3 audio rates due to commercial breaks and the like. Those transitions are likely screwing up hdemux and mplex both, as neither was designed for streams that change like that. Since you're probably going to edit out those transitions anyways, if you do so by pre-chopping your .ty, such that the resulting pieces are all one format, you could avoid the problem with hdemux&mplex. Personally I don't know of anything OTA that I'd want to archive so I haven't made hdemux handle this better. Yes, movies OTA are higher bit rate than directv's HD offerings but the commercial break interruptions make them unsuitable for archival IMO.

    By pre-chopping I'm thinking of tychopper (windows) or dd (unix)
    Quote Originally Posted by cheer
    P.S. Would it be of any use to see logs from something like ac3fix? Prolly not since by that point hdemux is already done, but...
    I'm not familar with ac3fix, it probably would be easier for me to just look at a sample .ty.
    Quote Originally Posted by cheer
    If those logs would be of any value, let me know; not sure if they mean anything to anyone but Josh.
    Right, he's got his own private debug messages that are only going to be fully meaningful to him. Lacking source, the rest of our hands are tied when tytool barfs.

  7. #127
    Join Date
    Nov 2004
    Location
    Gurnee, IL
    Posts
    2,384
    Quote Originally Posted by bcc
    Since you're using OTA streams, they are likely fully of transitions in&out of HD, and varying AC3 audio rates due to commercial breaks and the like. Those transitions are likely screwing up hdemux and mplex both, as neither was designed for streams that change like that. Since you're probably going to edit out those transitions anyways, if you do so by pre-chopping your .ty, such that the resulting pieces are all one format, you could avoid the problem with hdemux&mplex. Personally I don't know of anything OTA that I'd want to archive so I haven't made hdemux handle this better. Yes, movies OTA are higher bit rate than directv's HD offerings but the commercial break interruptions make them unsuitable for archival IMO.
    Agreed -- but in fact I'm not going to be either archiving or editing these.

    My son is a long-distance trucker. I'm using (or would like to use) EtiVo as an automated mechanism to allow him and his fiancee (she's on the truck too) to pull down their favorite shows. Most of them I just record on a non-HD Tivo, but some I do in HD because we watch them as well.

    EtiVo by default uses WMV (ugh) but bluewomble wrote a nifty addin that allows one to use Divx (or Xvid or...well, it just uses mencoder, so whatever). His addin runs vsplit prior to mencoder to turn the .ty into a .mpg. Vsplit doesn't handle HD streams at all, really, but santa8claws wrote a "vsplit.bat" file designed to call hdemux/mplex. This didn't work reliably for me, and it had the additional limitation of not using the -O -r values...so I wrote a wrapper program called fakevsplit. Fakevsplit runs your hdemux and redirects the output to a file, then parses the file, extracts the -O and -r values, and runs mplex. From there the resulting .mpg file is handed off to bluewomble's addin, which fires off mencoder to finish the job.

    This works remarkably well on all SD videos I've tried, and all of D*'s HD stuff.

    I should also point out, for the record, that the above success rate (2 out of 9) is atypical -- usually hdemux seems to do better for me than anything else (including TyTool). That it can easily be used in a batch fashion is the icing on the cake, here.
    By pre-chopping I'm thinking of tychopper (windows) or dd (unix)
    I'm not familar with ac3fix, it probably would be easier for me to just look at a sample .ty.
    Understood. I grabbed tychopper and will play with it today.
    Right, he's got his own private debug messages that are only going to be fully meaningful to him. Lacking source, the rest of our hands are tied when tytool barfs.
    Yep, that's what I figured. I'll play with tychopper and get some samples together for you.

    Thanks again!
    --
    Christopher D. Heer
    Quote Originally Posted by Oscar Wilde
    Perhaps, after all, America never has been discovered. I myself would say that it had merely been detected.

  8. #128
    Join Date
    Nov 2002
    Posts
    1,076
    Quote Originally Posted by cheer
    Agreed -- but in fact I'm not going to be either archiving or editing these.

    My son is a long-distance trucker. I'm using (or would like to use) EtiVo as an automated mechanism to allow him and his fiancee (she's on the truck too) to pull down their favorite shows. Most of them I just record on a non-HD Tivo, but some I do in HD because we watch them as well.
    If I was place-shifting HD tivo content, I'd probably just watch it via xvid compressed mpeg4 on an HTPC
    Quote Originally Posted by cheer
    I grabbed tychopper and will play with it today.
    Thanks again!
    Great, let me know how it works out.

  9. #129
    Join Date
    Nov 2004
    Location
    Gurnee, IL
    Posts
    2,384
    Quote Originally Posted by bcc
    If I was place-shifting HD tivo content, I'd probably just watch it via xvid compressed mpeg4 on an HTPC
    Well, yeah, that's sort of the idea, except replace HTPC with a laptop. But to get Xvid I need to get .MPG first. (At least I don't think I can go .ty -> Xvid...)
    Great, let me know how it works out.
    Father's Day intervened. And of course work has piled up on me. But tomorrow I should be able to chop 'em up.
    --
    Christopher D. Heer
    Quote Originally Posted by Oscar Wilde
    Perhaps, after all, America never has been discovered. I myself would say that it had merely been detected.

  10. #130
    Join Date
    Nov 2002
    Posts
    1,076
    Quote Originally Posted by cheer
    Well, yeah, that's sort of the idea, except replace HTPC with a laptop. But to get Xvid I need to get .MPG first. (At least I don't think I can go .ty -> Xvid...)
    Right; I had mpg->ty on my brain when I mentioned that, due to ffmpeg hacking.

  11. #131
    Join Date
    Nov 2002
    Posts
    1,076

    Version 0.15

    Here's a new release of hdemux. Changes include
    • Allow video sequences to be unbounded in length (fixes "Cannot fit Video buffer" error).
    • No more cosmetic "Skipping chunk .. due to timestamp discontinuity" errors at end of file
    • I've fixed a bug in ac3 audio packet processing when audio packet lengths oscillate, and don't match the frame length (Fox TV). (fixes "Tossing AC3 audio frame with bad sync data." errors).

    Binary release only for now.
    Last edited by bcc; 12-19-2006 at 01:55 AM.

  12. #132
    Join Date
    Nov 2002
    Posts
    1,076
    Quote Originally Posted by cheer
    Father's Day intervened. And of course work has piled up on me. But tomorrow I should be able to chop 'em up.
    I made a bunch more OTA recordings of my own, and I think I found the 3 main errors I mentioned. Should be fixed in the 0.15 image I just posted.

  13. #133
    Join Date
    Feb 2005
    Posts
    10
    Think you could see your way clear to posting source? I'm working on my own homebrew archiving system (on an OS I wouldn't expect any of you guys to support) and would like to take a stab at it, as I can't get tytools to work without barfing.

  14. #134
    Join Date
    Nov 2002
    Posts
    1,076
    Quote Originally Posted by lennier
    Think you could see your way clear to posting source? I'm working on my own homebrew archiving system (on an OS I wouldn't expect any of you guys to support) and would like to take a stab at it, as I can't get tytools to work without barfing.
    I'm not inclined to.
    I've integrated hdemux with my own multiplexer that I'm not ready to give out source for.
    Also hdemux hasn't really been working as an open source project in that it's simply meant extra support for me, with virtually no source contributions back. This makes it not worth the effort for me to maintain source releases.

  15. #135
    Join Date
    Oct 2001
    Posts
    36
    I'm trying to convert a 9GB ty file to mpg on windows xp. The hdemux goes without trouble, but the mplex seems to wrap around at the 2GB mark. The output file never gets bigger than 2GB and the beginning gets overwrriten.

    Any ideas?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •