PDA

View Full Version : tytompg: TY to mpg conversion in one step, alpha release



Pages : [1] 2

bcc
11-05-2006, 08:41 PM
Here's a beta release of tytompg. tytompg converts from TY/TY+/TMF to mpeg2 program streams in a single step. Ie it combines the TY demuxing operation with a loss-less re-multiplex operation. Inside it uses the mature hdemux code for demuxing, and a purpose-built muxer on the muxing side. For multiplexing, I'm preserving the timestamps from the original TY stream instead of trying to regenerate them, as was required with hdemux. This has some benefits and some drawbacks (mostly benefits I believe). Some benefits: Preserving a/v sync regardless of errors in the audio or video streams. For example, the current tivo bug that plagues much of the FOX OTA broadcasts with audio dropouts does not phase tytompg. tytompg's ability to pass-thru video with varying frame rates (these have historically been a deal-breaker for using the mplex muxer). mpeg2 editors such as videoredo work with tytompg a lot better than hdemux+ffmpeg, as tytompg includes timestamps at all video I frame boundaries (so long as the original recording included them), whereas ffmpeg would often leave out some of these timestamps.Some issues: Because tytompg does not try to re-generate the timestamps, this can cause problems for players and mpeg2 editors that assume that an mpeg2 program stream has no errors and/or is dvd compliant. For example, I've seen mplayer unable to properly recover from audio holes, but xine does so just fine. YMMV. I'm not at this time generating DVD navigation records, so players that are unable to perform trick-play-like operations without them (ie simplistic players like MyHD's) will have trouble. This really should only be a real concern for DVD output, in which case you'll need to post-process the mpeg anyways (to fix other issues including the video GOPs not being DVD compliant).

Typical usage:
tytompg -y <show>.ty
Output is to <show>.mpg by default. Some switches:
-y Allow overwrite of output file without confirmation, even if it already exists
-V Omit video stream from output file
-A Omit audio stream from output file
-s <number> Skip <number> chunks on input before processing. Note: if input file is a .tmf file, this option will not work correctly
-n <number> Process <number> chunks from input


Example:

% tytompg -y game7cut.ty
tytompg: Copyright (c) 2004-2007 B.C. <bcc24x7@gmail.com>
Multiplexer version 0.19, Demuxer version 0.26
Source is game7cut.ty
AC3 audio at 48kHz, 384 kbps, 1536 bytes/sync frame
Recovered AC3 sync, offset 134, chunk 0
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
Recovered AC3 sync, offset 134, chunk 0
Warning: temporal_reference order problem. Value: 3 group_start: 810 last: 836
audio: Warning: Decoder buffer empty, scr ahead by 81 (0.81) 0.0
... and encoder buffer full
audio: Warning: Dropping frame -- too old by 198662065 (662206.265) 7.32206
Done!
Stream audio: 1673 frames (62 seconds aprox.)
Stream video: 3737 frames, 7474 fields (62 seconds aprox.)
%


Windows 32, linux-x86, linux-x86-64, mac-x86 binaries attached.

Enjoy,

B.C. <bcc24x7@gmail.com>

bcc
11-05-2006, 08:42 PM
<Info about troubleshooting, diagnostics to go here.>

-d is the debugging switch. It has a lot of arguments, (some of which don't necessarily do anything right now). -dtr, for example, means 't' and 'r' options are enabled. The full list (subject to change):
t timestamps
r TY records
a audio records
v video records
b buffering
i i/o
m multiplexer buffering
p padding
f virtual buffer flow
q queueing
D packet dump
F full packet dump
R TY record details
T Tivo timestamps
Other useful command line options include '-s chunk-count' to skip chunk-count chunks before demuxing, and '-c chunk-count' to demux chunk-count chunks.

mr_zorg
11-05-2006, 09:44 PM
Windows & linux binaries attached.

This sounds awesome. I won't ask about the source, as we've already had that discussion over in your hdemux thread. However, I do want to reiterate my offer to compile an OS X binary for you, if you want. If you PM me the source, I will compile a universal binary (PPC & Intel) and send it back. I promise to delete the source immediately afterwards! :)

RavenStL
11-05-2006, 10:02 PM
Well I tried it on FOX OTA and got this


Multiplexer version 0.1
Demuxer version 0.19
Source is NLCSg6.ty
AC3 audio at 48kHz, 448 kbps, 1792 bytes/sync frame
Recovered AC3 sync, offset 110, chunk 1
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
Recovered AC3 sync, offset 110, chunk 1
audio: Warning: Decoder buffer empty, scr ahead by 2560 (8.160) 0.8
... and encoder buffer full
Assertion failed: FALSE, file demux.c, line 1259

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

Is that 8.160 seconds? Cause my dropouts are around 9 seconds long.

bcc
11-05-2006, 10:14 PM
Is that 8.160 seconds? Cause my dropouts are around 9 seconds long.The warning shows an audio underrun, which is what I'd expect you to see during those FOX recording glitches. Unfortunately you can't see the whole duration of the hole from that diagnostic message.

The assertion failure probably happened a bit later. I should be able to fix that one fairly easily.

bcc
11-05-2006, 10:15 PM
This sounds awesome. I won't ask about the source, as we've already had that discussion over in your hdemux thread. However, I do want to reiterate my offer to compile an OS X binary for you, if you want. If you PM me the source, I will compile a universal binary (PPC & Intel) and send it back. I promise to delete the source immediately afterwards! :)Easiest would be to just cross compile. Nobody responded to my cross compiler question...

bcc
11-05-2006, 10:32 PM
The assertion failure probably happened a bit later. I should be able to fix that one fairly easily.Try the 0.2 version. Unfortunately I don't have a test case to cover this, so you're the guinea pig (unless you want to send a clip).

RavenStL
11-05-2006, 10:56 PM
That got it going, but I accidentally deleted the file and have to re downlaod it from the HR10-250. Dont ask, I thought I was deleting somehting else............

I tried on another and got this though. FOX OTA still, differnt .ty


Multiplexer version 0.2
Demuxer version 0.19
Source is f:\WSG1.ty
AC3 audio at 48kHz, 384 kbps, 1536 bytes/sync frame
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
Assertion failed: stream->aucnt < stream->autotcnt, file mplex.c, line 405

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

bcc
11-05-2006, 11:11 PM
Assertion failed: stream->aucnt < stream->autotcnt, file mplex.c, line 405Hmm, I did pre-test this on a half a dozen different fox cases. Guess I need to download more fox recordings. I assume you got some good mpeg before this error hit.

RavenStL
11-05-2006, 11:32 PM
I got a good 6 gigs out of 24. 25% so far.

Another stream after 11.1 gigs of 20+


Multiplexer version 0.2
Demuxer version 0.19
Source is NLCSg7.ty
AC3 audio at 48kHz, 384 kbps, 1536 bytes/sync frame
Recovered AC3 sync, offset 34, chunk 1
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
Recovered AC3 sync, offset 34, chunk 1
audio: Warning: Decoder buffer empty, scr ahead by 877098 (2923.198) 0.2923
... and encoder buffer full
audio: Warning: Decoder buffer empty, scr ahead by 299636 (998.236) 0.998
... and encoder buffer full
audio: Warning: Decoder buffer empty, scr ahead by 740 (2.140) 0.2
... and encoder buffer full
audio: Warning: Decoder buffer empty, scr ahead by 1213454 (4044.254) 0.4044
... and encoder buffer full
Warning: Tossing AC3 audio frame with bad sync data at end of packet. Chunk 906
57 length 1450
Warning: Tossing AC3 audio frame with Rate out of bounds. Chunk 90657 length 29
48
Warning: Tossing AC3 audio frame with Rate out of bounds. Chunk 90657 length 27
48
Warning: Tossing AC3 audio frame with Rate out of bounds. Chunk 90657 length 28
32
Warning: Tossing AC3 audio frame with Rate out of bounds. Chunk 90657 length 29
16
Warning: Tossing AC3 audio frame with Rate out of bounds. Chunk 90657 length 28
16
Warning: Tossing AC3 audio frame with Rate out of bounds. Chunk 90657 length 29
00
Assertion failed: id != -1, file mplex.c, line 1542

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

mr_zorg
11-06-2006, 01:24 AM
Easiest would be to just cross compile. Nobody responded to my cross compiler question...
I know I didn't respond because I don't ever do it. I actually have a Mac. I'd love to tell you how to cross compile -- if I knew. :P

mr_zorg
11-06-2006, 02:14 AM
I know I didn't respond because I don't ever do it. I actually have a Mac. I'd love to tell you how to cross compile -- if I knew. :P
It just dawned on me what I can do though. How about if I set you up with your own SSH/SFTP account on my Mac Mini Core Duo server? Would that be agreeable? If so, let me know and I'll PM you the information you need. Of course, once logged on you may change the password. :D

bcc
11-06-2006, 02:23 AM
I know I didn't respond because I don't ever do it. I actually have a Mac. I'd love to tell you how to cross compile -- if I knew. :PRight, and it hasn't been a priority for me to go research the best way to support equipment I *don't* have. Perhaps the distcc approach but gcc would be preferable.

artships
11-06-2006, 04:34 AM
Wonderful tool, bcc. Tried it on an Ubuntu box, feeding it a 544x480 Sony Stand Alone (ancient, I know) tystream. Man is it fast! Alas, playback of the mpg was choppy in VLC, while playback of the original TY file was smooth. Listing:


Multiplexer version 0.2
Demuxer version 0.19
Source is The X-Files-Improbable.ty
Video frame size is 544x480 Frame rate 29.970030
Video bit rate is 5800000 bits/sec., 5800 Kbit/sec.
MPEG2 audio at 32kHz, 192 kbps, 864 bytes/sync frame
MPEG2 audio at 32kHz, 192 kbps, 864 bytes/sync frame
Done!
Stream audio: 100039 frames (2701 seconds aprox.)
Stream video: 107939 frames (3601 seconds aprox.)

Did I mention it was fast? Couldn't have lagged much behind cp for speed. And kept sync throughout, too. Quite impressive.

Cheezmo
11-06-2006, 10:15 AM
I'm getting lots of assertion failed in line 428 of mplex.c on my tests. So far they are all about 1Gb in, so I don't know a good way to send you a test case.

Cheezmo
11-06-2006, 10:38 AM
I got the same error message from an SD satellite .ty


That got it going, but I accidentally deleted the file and have to re downlaod it from the HR10-250. Dont ask, I thought I was deleting somehting else............

I tried on another and got this though. FOX OTA still, differnt .ty


Multiplexer version 0.2
Demuxer version 0.19
Source is f:\WSG1.ty
AC3 audio at 48kHz, 384 kbps, 1536 bytes/sync frame
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
Assertion failed: stream->aucnt < stream->autotcnt, file mplex.c, line 405

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

pdicamillo
11-06-2006, 07:30 PM
I tried running version 0.2 on a linux system with kernel 2.6.7, and got:


FATAL: kernel too old
Segmentation fault

If it doesn't actually require a newer kernel (which seems unlikely), maybe that restriction could be removed?

If I want separate MPEG audio and video files, how would the results produced by de-muxing the stream produced by tytompg compare to the files produced by hdemux? Would it make sense to add -a and -v options for people who need separate files sometimes? I'd like to have a way to get the "best" separate files when I need them (and the adjustment offset to keep them synchronized), taking advantage of everything you've figured out as you continue to refine you code.

RavenStL
11-06-2006, 09:26 PM
I tried to encode the product of tytompg (before the crash) and I am having trouble getting TMPGEnc to not lock up on the mpg file.

I got another one to load in TMPGEnc and I re-encoded it. It encoded and was finished, I tried to play it and it lost video about 3 minutes in. Only audio was then playing. If I fast forward or reverse, still no video. If I rewind to < 3 minutes, the video returns. This was to WMV

I will try and batch another encode tonight to dvd compliant mpeg2 and see if it does the same.

For the record, the ORIGINAL tytompg file plays FINE past the 3 minutes spot. There does not appear to be any kind of audio hole involved here.

But the product of tytompg is hard to process. TMPGEnc locks up on 2 of three I got and the third loses the video. VideoRedo wont process them at all. (Just got it d/l, maybe it wouldnt anyway??)

Also, HDEMUX ouputs of the SAME, orig .ty stream process ALMOST FINE. The only problem being the Audio Holes causes the audio to shift (Thats the almost). THey Encode fine too.

Thinkdiff
11-06-2006, 09:42 PM
Seems like a nice program. Too bad I cannot run it. I agree with the other poster, let somebody build an OS X version. If you're that obsessed with securing your sources that you cannot send it to one of us to test-compile, then I definitely could set up a SSH account on my box at home (a measily 400MHz G4, but will get the job done). If you're against that, then I don't know what to tell you. Can you post the link to the cross compile thread? Perhaps I could help, even though I too never cross compile.

Whatever you choose to do, good luck with your app.

bcc
11-06-2006, 10:25 PM
I got the same error message from an SD satellite .tyFor the "stream->aucnt < stream->autotcnt" assertion you two have seen:

There are some recordings (particularly fox again it seems) that are sometimes broadcasting surprisingly small(frequent) PES packets for the video stream. Well surprising for tytompg :) Fixing...

bcc
11-06-2006, 10:34 PM
Wonderful tool, bcc. Tried it on an Ubuntu box, feeding it a 544x480 Sony Stand Alone (ancient, I know) tystream. Man is it fast!Thanks, actually I half expected complaints about speed as I hadn't yet done anything to optimize the speed and I know it's doing some things that are cpu inefficient. But I did avoid putting a full abstraction layer between demuxing&muxing which would have added inefficiencies.

Alas, playback of the mpg was choppy in VLC, while playback of the original TY file was smooth. Hmm, not good. I haven't seen that with xine or windvd. I suspect by choppy you mean the video stutters but seems to play at the right rate. I could try dropping the DTS timestamps from the video stream, they may be the problem. I wound up having to fudge the video DTS timestamps as hr10-250 tivos with 3.1.5 are not generating this value correctly in TY streams (they are wrapping the # at 32 bits, which only takes a few minutes to occur).

bcc
11-06-2006, 10:39 PM
I'm getting lots of assertion failed in line 428 of mplex.c on my tests. So far they are all about 1Gb in, so I don't know a good way to send you a test case.
As with hdemux, tytompg supports the -s and -c options to skip chunks & limit the number of chunks processed, for your troubleshooting pleasure :)
Hopefully this will get fixed when I'm done with the fix for the more troublesome OTA recordings.

bcc
11-06-2006, 10:52 PM
I tried running version 0.2 on a linux system with kernel 2.6.7, and got:


FATAL: kernel too old
Segmentation fault

Ugh, I just got a newer glibc (2.5) on my machine and looks like it's enforcing this backwards-compatibility issue against my intent.
If it doesn't actually require a newer kernel (which seems unlikely), maybe that restriction could be removed?Yes, as soon as I figure out how:) I'm just doing straightforward file i/o syscalls, and I made sure to statically link, so I'm surprised there's an issue.

If I want separate MPEG audio and video files, how would the results produced
by de-muxing the stream produced by tytompg compare to the files produced by
hdemux?If you demux the mpeg2 into elementary streams, you should find them to be identical to the ones generated by hdemux. Except that the streams from tytompg will probably come up a few frames short (I'm still a little sloppy with my EOF processing).
Would it make sense to add -a and -v options for people who need
separate files sometimes? I'd like to have a way to get the "best" separate
files when I need them (and the adjustment offset to keep them synchronized),
taking advantage of everything you've figured out as you continue to refine
you code.Existing mpeg2 tools should be able to tell you the a/v sync offset from the .mpg I generate and extract the elementary streams. But yes, a demux only mode is probably in order. There are also some switches: -V which will generate the .mpg without a video stream, and -A which will generate the .mpg without the audio stream. However these new switches still result in mpeg2 program streams not elementary streams (the later lacking timestamps).

bcc
11-07-2006, 04:12 AM
Ugh, I just got a newer glibc (2.5) on my machine and looks like it's enforcing this backwards-compatibility issue against my intent.Looks to me like redhat configured glibc linked binaries to require linux 2.6.9 or greater as of fedora core 5. Seems like an unreasonable restriction to me but I still don't see an easy way to turn off the check.

bcc
11-07-2006, 04:33 AM
I've posted a new version (0.3). It should fix the "stream->aucnt < stream->autotcnt" assertion. It likely fixes the other assertions as well.

bcc
11-07-2006, 04:41 AM
Seems like a nice program. Too bad I cannot run it. I agree with the other poster, let somebody build an OS X version. If you're that obsessed with securing your sources that you cannot send it to one of us to test-compile, then I definitely could set up a SSH account on my box at home (a measily 400MHz G4, but will get the job done). If you're against that, then I don't know what to tell you. Can you post the link to the cross compile thread? Perhaps I could help, even though I too never cross compile.

Whatever you choose to do, good luck with your app.
You want free source code, free binaries and probably free support and you're calling me obsessed for not giving it to you? Think of me as a company. Would you call that company obsessed for not giving out their source code? I doubt it. Name calling is not how to get me to do something for you.

Thinkdiff
11-07-2006, 02:03 PM
You want free source code, free binaries and probably free support and you're calling me obsessed for not giving it to you? Think of me as a company. Would you call that company obsessed for not giving out their source code? I doubt it. Name calling is not how to get me to do something for you.

I'm not name calling at all. I just dont see your point in making an application and releasing it only for a certain number of people. If you're gonna release it and want people to give you feedback, then wouldnt you want the widest userbase possible?

I'll get along fine without your program, I already wrote a script that does TY to MPG on OS X in one step, I was just hoping I could help you bring your app to OS X so more people could use it (as my script is based heavily on my own setup on my computer). I dont want anything from you. I just wanted to help you. I guess you don't want/need my help, so that's fine.

For what it's worth, sourceforge has OS X machines to test compile on. But I guess that would require actually releasing source code. I guess i'll try to find your thread on cross compiling and see what your problem was.

mpauley
11-07-2006, 02:25 PM
bcc,

You guys have advanced by leaps and bounds since I was last active on here. This tool works perfect on both of my S2 SAs and is saving me major time demuxing and muxing back.

Thanks,

-Pauley

bcc
11-07-2006, 02:50 PM
I'm not name calling at all.Ok, I'll ignore the part where you called me obsessed. Please start your own thread if you want to keep arguing about lack of source code and your personal spin on that.

Note that I have repeatedly offered to compile for macs under my own terms.

Cheezmo
11-07-2006, 03:27 PM
My satellite test went through fine this time, now I'm getting an...

Assertion failed: code, file mplex.c, line 465

on an OTA sample file (6Gb, gets through about 800Mb).

[I perhaps spoke too soon, I'm seeing some bad audio sync problems for portions of the satellite clip, although they to sync back up eventually (several minutes off, then OK again).

AlphaWolf
11-07-2006, 05:21 PM
I've used OSX fairly extensively, and while I think it is a major POS (dunno why that 6% of all computer users seems to believe that they have the best in the world over the remaining 94% who do have a choice to use it if it is that much better, but whatever...) I'll point out that you can always download and run it on any intel/amd computer that supports SSE3. The development tools for compiling something on it are freely available after that point. I think their compilers make building a universal binary a one step process as well.

Just google osx86 torrent or something like that.

RavenStL
11-07-2006, 10:13 PM
OK, 0.3 has processed two of my FOX OTA files.

Still have the same question, what is it about these new mpg's from tytompg that is different from an Mplex mpg from an hdemux output? (WOW, the military would be proud with the Acronym usage)

My TMPGenc chokes on the 0.3 output (Lockup, no error, must kill the thread)

I have fired up Windows Media Encoder ver 9 and it appears to be chugging away just fine.

I will report back with the results.

Interestingly WME will allow you to

Preserve Timestamps
or
Make new ones.

Hmm, any reasons why you would want to make new ones? Is it possible, that the way tytompg works, its better to make new timestamps (Referencing the audio holes) Or will that bring my problem back?

If you cant answer that question, thats OK. Just curious if you have incite. I plan to do it again tomorrow with Make Timestamps to just see what happens. :)

Thanks BCC! :) heh, BCC for President, err Senator for Missouri! (Too many political ads of late for my noggin, it overflowed)

bcc
11-07-2006, 10:33 PM
I've used OSX fairly extensively, and while I think it is a major POS (dunno why that 6% of all computer users seems to believe that they have the best in the world over the remaining 94% who do have a choice to use it if it is that much better, but whatever...) I'll point out that you can always download and run it on any intel/amd computer that supports SSE3. The development tools for compiling something on it are freely available after that point. I think their compilers make building a universal binary a one step process as well.

Just google osx86 torrent or something like that.
Something that doesn't require pirating a hacked distribution of osx with funky hardware support would be nice. (Although osx under vmware/xen sounds pretty interesting...) If apple wants 3rd party applications to be built for their OS they would be best served by providing the build environment.
Yes the size of the osx user base vs the rest of the world adds some perspective doesn't it?

pdicamillo
11-07-2006, 11:11 PM
If apple wants 3rd party applications to be built for their OS they would be best served by providing the build environment.

Apple provides XCode, a completely free, high-quality build environment. It's the same one the Apple engineers themselves use. It's also gcc-based. However, you do need OS X to run it. Apple has never tried to provide even informal support for cross-compilation, and I really can't blame them. The situation with tytompg is just not common enough. Nearly all Mac developers are also Mac users, building GUI apps, and of the ones who are not, most either have open source code, or access to a Mac for building and testing. Also, you can get a Mac Mini for just $600, a machine which can easily be set up to triple boot OS X, Windows, and linux. From that point of view it's a great development machine.

Maybe a bunch of us should chip in to get bcc a Mac Mini :)

Thinkdiff
11-07-2006, 11:26 PM
Something that doesn't require pirating a hacked distribution of osx with funky hardware support would be nice. (Although osx under vmware/xen sounds pretty interesting...) If apple wants 3rd party applications to be built for their OS they would be best served by providing the build environment.
Yes the size of the osx user base vs the rest of the world adds some perspective doesn't it?

Hrmm. That raises a good point. You CAN run the full Darwin OS for free (just as any linux) on a PC. It's freely available from Apple.com. While it doesnt have the pretty GUI, it'd be great for a build environment. I can send you the related instructions and such if you want (better solution than the darwin-cross packages I was talking about in my PM).

burdellgp
11-08-2006, 01:04 AM
I'm just doing straightforward file i/o syscalls, and I made sure to statically link, so I'm surprised there's an issue.

Static linking to glibc is bad for a couple of reasons:

not all of glibc is available for static linking (although that shouldn't be a problem here)
it is probably a license violation: glibc is LGPL. Static linking makes the resulting binary a derived work, which means you must distribute source. I'm not sure why the GPL COPYING file is included in the rar file; if it is under the GPL (either because it is derived from other GPL code or because you created it all and want to use the GPL) then you must distribute the source.


The problem is probably because newer Fedora glibcs (IIRC FC5+) support only the new threading library, which in turn requires newer kernels (basically the old LinuxThreads system was broke and NPTL replaces it but required some kernel changes). I think even if you don't use the new threads some of that kicks in. I'm not sure how to build for older glibcs under newer systems.

AlphaWolf
11-08-2006, 01:49 AM
Apple provides XCode, a completely free, high-quality build environment. It's the same one the Apple engineers themselves use. It's also gcc-based. However, you do need OS X to run it. Apple has never tried to provide even informal support for cross-compilation, and I really can't blame them. The situation with tytompg is just not common enough.

Actually it is very common. Just about any linux program could be readily cross compiled for OSX with little to no modification. This is true for even programs that use x windows, as the OSX install CD includes an optional X11 server.

If you'll notice, many non-apple programs that come out these days and have a linux port usually run under x11 when they are ported to osx. Take matlab for example.

I think the biggest problem is that OSX really isn't as great as mac users make it out to be. It really is incredibly easy for commercial linux developers to make their software available for macintoshes, but they don't most of the time simply because there is little to no demand for it.

bcc
11-08-2006, 02:42 AM
OK, 0.3 has processed two of my FOX OTA files.

Still have the same question, what is it about these new mpg's from tytompg that is different from an Mplex mpg from an hdemux output? (WOW, the military would be proud with the Acronym usage)Right, all this arguing about macs has been getting in the way of the real problem at hand (achieving correctness with demuxing&muxing troublesome OTA recordings). If 2 of your fox recordings with holes in them processed OK then it sounds like we're a lot closer than ever before.

Anyways, hdemux generates mpeg2 video and ac3 or mpeg2 audio streams. These are called "elementary" streams in the spec, and they lack timing data. (No PTS, DTS or SCR). mplex adds back that timing data, by computing it up on its own, and interleaves the two streams at it's notion of the correct rate. It does some extra work to try and make video I frames available at the beginning of the mpeg2 packets. There are several problems with what mplex is doing in those steps on the elemental streams. 1) mplex, being rather dvd-centric, doesn't compute those timestamps correctly when it comes to some broadcasts that have varying field per frame rates. FOX is one of the broadcasters that employs this encoding technique. 2) mplex code newer than 1.6.2 actually doesn't even make the video I frames available at the right place, 3) There must be no holes in the a/v stream or else you'll lose a/v sync

tytompg preserves the timing data during the demux step, passing it on to the multiplexer. This saves the multipler from lots of issues: The a/v sync should start out perfect and stay that way; basically issues 1) and 3) go away. As far as I have seen, I pack mpeg2 packets just like mplex used to back in 1.6.2, with respect to positioning of the records. There may be some issues with timing. Because the hr10-250 tivo is wrapping the video DTS timestamp every few minutes, it is making straightforward processing based upon it a problem. Right now I'm throwing it out and using the PTS timestamps instead. This is not quite right but I would have expected the timing to be close enough for decoders (assuming they are doing virtual buffering per the specs). tmpgenc may not be nearly as forgiving about that as xine/windvd, it may be assuming the program stream was freshly encoded with perfect timestamps. I'll give it a try. It may also be that tmpgenc thinks I'm not conforming to the spec's virtual buffer model (allowing the streams to underflow). I hadn't finished my mpeg compliance tester before trying this project so I haven't been able to easily verify this :) I do try to keep the virtual buffers part full.

Short answer is that both tytompg and hdemux+mplex generate mpeg2 program streams as their end result.
Interestingly WME will allow you to

Preserve Timestamps
or
Make new ones.

Hmm, any reasons why you would want to make new ones?Depends upon how they make the new ones. Assuming they totally recompute the timestamps this could fix the above virtual buffer issue. It also stands a good chance of re-introducing problem #1 with mplex shown above.

Is it possible, that the way tytompg works, its better to make new timestamps (Referencing the audio holes) Or will that bring my problem back?I have code to regenerate the timestamps as well, but you'd have to take into account the original timestamps to measure the holes correctly. Not sure what algorithm one would want where you half use the original times and half make them up. Ok unless you're editing the streams. Otherwise I would think you'd either want the originals (because they are right or because the streams have holes), or regenerated ones (in the case where you don't have holes but the broadcaster screwed up a/v sync or somesuch).


Thanks BCC! :) heh, BCC for President, err Senator for Missouri! (Too many political ads of late for my noggin, it overflowed)Thanks, haven't even been there. Too many adds here too; I got political telemarketing spam at 6:30am even.

lsmod
11-08-2006, 03:08 AM
Actually it is very common. Just about any linux program could be readily cross compiled for OSX with little to no modification.

Crosscompiled? I don't think so. It's quite true that most linux software will compile and run on the mac with little if any modification, but that's not cross compilation. (compiling for one platform on another) I'm not aware of any cross compilation tools that target the mac from other platforms. (Xcode cross-compiles for the other processor architecture, but that doesn't help us here.)

Compiling under Darwin, perhaps running on on a VM would be worth a try, but really is the hard way to do it. I'd love to have bcc's new engine running on a mac, because hdemux 0.13 (the last version we have on the mac) doesn't seem to work with any of my HD streams.

My fastest machine is a mac, so I'd prefer to run tytompg there, and there's already a nice GUI in TivoTool.

Refurb minis are $520... time to pass the hat? If bcc's willing to support the platform, I'll put up the first $100.

bcc: PM me if you are interested.

AlphaWolf
11-08-2006, 03:27 AM
Well, right, not cross compiled, but thats what I meant.


Anyways, hdemux generates mpeg2 video and ac3 or mpeg2 audio streams. These are called "elementary" streams in the spec, and they lack timing data. (No PTS, DTS or SCR). mplex adds back that timing data, by computing it up on its own, and interleaves the two streams at it's notion of the correct rate. It does some extra work to try and make video I frames available at the beginning of the mpeg2 packets. There are several problems with what mplex is doing in those steps on the elemental streams. 1) mplex, being rather dvd-centric, doesn't compute those timestamps correctly when it comes to some broadcasts that have varying field per frame rates. FOX is one of the broadcasters that employs this encoding technique. 2) mplex code newer than 1.6.2 actually doesn't even make the video I frames available at the right place, 3) There must be no holes in the a/v stream or else you'll lose a/v sync

Would it be possible to insert blank video or audio frames in order to force the video streams remain sync in third party muxers?

pdicamillo
11-08-2006, 01:25 PM
I made sure to statically link, so I'm surprised there's an issue.

I'm wondering if static linking is what's causing the problem. Wouldn't dynamic linking get a version of glibc that works with my kernel? Would you be willing to do a build to test that?

While my 2.6.7 kernel should be updated, it was current just two years ago, and our production Red Hat machines at work still use kernel 2.4, so I suspect this is going to affect quite a few other people.

stealthdave
11-08-2006, 04:10 PM
Being a Mac convert (used to be a Linux guy), I decided to do some research into cross-compiling for Mac OS X on Linux and came across these links:

http://www.racoonfink.com/fink/darwin-cross/
http://myownlittleworld.com/miscellaneous/computers/darwin-cross-distcc.html
http://www.sable.mcgill.ca/~dbelan2/crossdev/crossdev-powerpc-i686.html

These are more about using the cross-compile environment with distcc to speed up compile times, but I imagine that it would be a big step forward in a stand-alone cross-compile environment as well. I would also look into using PearPC and/or QEMU with OpenDarwin (or Mac OSX) if the cross-compile environment turns out to be too complicated. QEMU is nice because it has built-in file sharing options (it will launch a custom smb daemon) so you can keep the source on your host system. I haven't used PearPC as much, so I don't know what facilities they have.

bcc
11-08-2006, 07:36 PM
Would it be possible to insert blank video or audio frames in order to force the video streams remain sync in third party muxers?Certainly.
But it's beyond the scope of what a demuxer should do IMO. Because you have to do all the work of evaluating the timestamps (like a multiplexer would) and encode new ac3 and mpeg2 frames (like an encoder would). tytompg is a lot closer to being up for the task than hdemux.

Cheezmo
11-08-2006, 08:18 PM
I tried a clip from The Tonight Show, OTA.

hdemux -> mplex: no problems.
tytompg: choppy audio, poor audio sync.

I can provide a clip if you would like.

bcc
11-08-2006, 09:01 PM
Static linking to glibc is bad for a couple of reasons:It didn't used to be. Certainly back in the old days of shared C libraries it did improve binary portability to statically link. For example SunOS. I see there is some new religion about this. Smells like binary compatibility bugs in glibc to me.


not all of glibc is available for static linking (although that shouldn't be a problem here)Right, non issue
it is probably a license violation: glibc is LGPL. Static linking makes the resulting binary a derived work,Being deemed a derived work violates my intent so I guess I won't be distributing this way any more. I've removed my image for now.

which means you must distribute source.Does NOT per paragraph 1 section 6 of the LGPL.

Wow, looks like glibc is not really an appropriate choice for the standard C library and needs to be replaced.
I'm not sure why the GPL COPYING file is included in the rar file;Because I intended to distribute the binaries with licensing terms and I was still using GPL terms. I'll replace this with some stricter terms that better apply.


The problem is probably because newer Fedora glibcs (IIRC FC5+) support only the new threading library, which in turn requires newer kernels (basically the old LinuxThreads system was broke and NPTL replaces it but required some kernel changes). I think even if you don't use the new threads some of that kicks in. I'm not sure how to build for older glibcs under newer systems.Yes, looks like the thread change must have been the motivating factor. I still don't understand how breaking backwards binary compatibility like this was considered acceptable, and I see no workarounds.

RavenStL
11-08-2006, 11:05 PM
OK, finished some re-encoding.

tytompg is doing an OK,,,, job so far. The audio appears to be in sync quite well. Referencing the House commercial you have a cut of, the MPG oupt from tytompg does the following.

House commercial starts, gets to hole. Audio drops out. Video remains and all looks good. Audio comes back and all is good. First Audio Hole past, GREAT!

Ok, now I continue watching PAST what you see in the cut. I come to the next Audio hole. The audio drops out but when it comes back, it POPS loudly with a kinda high frequency, 200 ms duration?? And the audio is VERY quiet. but as Windows Media Player continues playing the audio comes back to normal sound, gradually. Kinda ODD? I use AC3Filter for audio decoding.

Now to the re-encoding. The audio is out of sync, immediately. Maybe 200 - 300 ms??? (do I need to be supplied with an audio video offset or did tytompg virtually fix that and made it un-needed?)

As I watch, the encoded version, (WMV 9 cause only WME is working on the mpg at this time) out of sync slightly, I hit the first audio hole.

The House commercial drops out audio. No biggee. But oddly, the audio comes back and is in like SLOW MOTION. It sounds like the devil... :) (What did you hide in this ! :) ) Once the House commercial is over, the audio is kinda quiet again and quickly comes back.

Now here is what is weird, NO AUDIO HOLE here, but the audio essentially DISAPPEARS. But it SLOWLY from silence to normal sound, comes back over 60 seconds. It starts to sound good again and BAM, silent again. I start to hear it coming back and its back to normal volume again in 60 seconds again.

So there is DEFINATELY progress, but some quirks, especially when encoding.

Any Ideas? I could U/L you this 26 gig file, but it will take about 6 straight days to get it to you :)

AlphaWolf
11-09-2006, 12:09 AM
Certainly.
But it's beyond the scope of what a demuxer should do IMO. Because you have to do all the work of evaluating the timestamps (like a multiplexer would) and encode new ac3 and mpeg2 frames (like an encoder would). tytompg is a lot closer to being up for the task than hdemux.

Couldn't you do it without a complete re-encode, e.g. DCT space?

EDIT: Or is that what you were already hinting at?

Kurtis
11-09-2006, 01:42 PM
Being deemed a derived work violates my intent so I guess I won't be distributing this way any more. I've removed my image for now.

Close reading of the LGPL does allow redistribution of statically linked code to not be under the LGPL so long as the application provides a way to statically link in another version of the LGPL code.

This may sound obnoxious, but if you make a .a of your code and distribute that .a with the glibc static lib and a batch file that links the two to create your executable, you still fufill the LGPL and don't have to release the source to your code.

Painful, but it would allow you to repost the image, which I'd like since I haven't gotten to try it yet. :-)

bcc
11-09-2006, 04:01 PM
Couldn't you do it without a complete re-encode, e.g. DCT space?

EDIT: Or is that what you were already hinting at?

Yes, for the video stream you can do fixups on individual frames, perhaps with ripple effects to the entire GOP. My point wasn't that it can't be done, just that it would be architecturally bad (abstraction/layer violations, all that) for a demuxer to try and do it. The beauty of converting out of TY into standard mpg as quickly as possible is that we can then leverage existing code that already does this sort of thing. For example delaycut, videoredo,

'course it may turn out that some of the stream errors are too nasty for those existing programs to correct and so the multipler might still want to help out.

bcc
11-09-2006, 04:12 PM
This may sound obnoxious, but if you make a .a of your code and distribute that .a with the glibc static lib and a batch file that links the two to create your executable, you still fufill the LGPL and don't have to release the source to your code.
It is obnoxious and I don't plan to distribute like that.
I don't need to static link, it was only to try and help portability. If GNU licensing and glibc are bent on making binary portability extra difficult so be it.

I'll re-release, I'm now just trying to make the code a little less alpha quality before doing so.

burdellgp
11-09-2006, 10:27 PM
It didn't used to be. Certainly back in the old days of shared C libraries it did improve binary portability to statically link. For example SunOS. I see there is some new religion about this. Smells like binary compatibility bugs in glibc to me.


When the most complicated function most C libraries included was printf(), it wasn't a big deal. However now the C library is huge, providing all manner of network services, threading, Unicode, etc. The network services especially are a problem, as they don't want the whole thing directly linked into the C library (e.g. nobody wants a C library that always includes LDAP, NIS/YP, Kerberos, etc.), and people also like to replace those parts, so they are modular and always dynamically linked.

Also, the kernel interfaces change, and some of that is hidden by the C library. However, that means the C library "knows" about certain changes, and is often optimized to handle the new system only. When you statically link, you suddenly take a new C library "back in time" to where it never thought it would have to go.

In general, backwards compatibility (running a binary from FC1 on FC6 for example) should work, but building on a newer system and using on an older one is never guaranteed. This is true of many OSes (e.g. Solaris, Tru64, etc.), not just Linux or Fedora. You can set up a compatible build environment (where for example on a Fedora Core 6 system you could build for FC3 and newer), but I don't know that the packages are included in Fedora (you'd have to build it yourself).

bcc
11-09-2006, 10:52 PM
When the most complicated function most C libraries included was printf(), it wasn't a big deal. However now the C library is huge, providing all manner of network services, threading, Unicode, etc. The network services especially are a problem, as they don't want the whole thing directly linked into the C library (e.g. nobody wants a C library that always includes LDAP, NIS/YP, Kerberos, etc.), and people also like to replace those parts, so they are modular and always dynamically linked.

Also, the kernel interfaces change, and some of that is hidden by the C library. However, that means the C library "knows" about certain changes, and is often optimized to handle the new system only. When you statically link, you suddenly take a new C library "back in time" to where it never thought it would have to go.

In general, backwards compatibility (running a binary from FC1 on FC6 for example) should work, but building on a newer system and using on an older one is never guaranteed. This is true of many OSes (e.g. Solaris, Tru64, etc.), not just Linux or Fedora. You can set up a compatible build environment (where for example on a Fedora Core 6 system you could build for FC3 and newer), but I don't know that the packages are included in Fedora (you'd have to build it yourself).
The linker "knows" that I'm not using any functions creating dependencies on anything newer than glibc2.1, not using any of the stuff you mention, yet glibc is enforcing a 2.6.9 kernel requirement, which simply breaks binary compatibility.

% nm tytompg | grep 'GLIBC_2.[12345]'
U lseek64@@GLIBC_2.1
U open64@@GLIBC_2.1
%

bcc
11-09-2006, 10:55 PM
I tried a clip from The Tonight Show, OTA.

hdemux -> mplex: no problems.
tytompg: choppy audio, poor audio sync.

I can provide a clip if you would like.Choppy as seen by which ap? Thanks, but I don't think I need a sample in this case, I've got plenty to go on...

bcc
11-09-2006, 11:00 PM
Compiling under Darwin, perhaps running on on a VM would be worth a try, but really is the hard way to do it. I'd love to have bcc's new engine running on a mac, because hdemux 0.13 (the last version we have on the mac) doesn't seem to work with any of my HD streams.

My fastest machine is a mac, so I'd prefer to run tytompg there, and there's already a nice GUI in TivoTool.

Refurb minis are $520... time to pass the hat? If bcc's willing to support the platform, I'll put up the first $100.

bcc: PM me if you are interested.Darwin under vmware sounds sort of promising actually. Just would require some extra spare time doing sysadmin shores to setup. Why would it be the hard way? Would save me from having to deal with more hardware (which would include equivalent sysadmin chores plus extra monitor/kvm, etc). Thanks for the mac offer, but I don't think I'd like to go that route.

Cheezmo
11-09-2006, 11:02 PM
My old friend MPEG Streamclip (which uses Apple's MPEG decoder since I'm on the Mac). I know you have lots of issues to deal with at this point and it is probably best to focus on compatibility with the tools you know so I won't ask to you to reproduce things there, but if the clips I have problems with are of any value, I'll be glad to make them available to you.

bcc
11-09-2006, 11:12 PM
Ok, now I continue watching PAST what you see in the cut. I come to the next Audio hole. The audio drops out but when it comes back, it POPS loudly with a kinda high frequency, 200 ms duration?? And the audio is VERY quiet. but as Windows Media Player continues playing the audio comes back to normal sound, gradually. Kinda ODD? I use AC3Filter for audio decoding.At the audio discontinuities, the ac3 stream is malformed. One probably has to drop the bad ac3 frames from the stream to avoid audio glitches such as pops. The tivo's broadcom chip presumably does this during playback. I was leaving this for the mpeg2 stream fixer. I guess you didn't use one :)

Now to the re-encoding. The audio is out of sync, immediately. Maybe 200 - 300 ms??? (do I need to be supplied with an audio video offset or did tytompg virtually fix that and made it un-needed?)The .mpg includes the necessary timestamps for preserving a/v sync, so probably your encoding ap is busted in the sense that it's assuming both streams can be treated as if their timecodes start at 0 or somesuch. I have not tested this far yet.

The House commercial drops out audio. No biggee. But oddly, the audio comes back and is in like SLOW MOTION. It sounds like the devil... :) (What did you hide in this ! :) ) Once the House commercial is over, the audio is kinda quiet again and quickly comes back.Again this and the rest of your symptoms sounds like your aps are assuming they don't need to repair the ac3 stream. Looks like you'll need to run the .mpg through a cleanup step before feeding it to that encoder.

RavenStL
11-09-2006, 11:23 PM
Where do I find a stream fixer? It seems videoredo doesnt work on it. Encoders do strange things. Hmmm

burdellgp
11-10-2006, 12:21 AM
Darwin under vmware sounds sort of promising actually.

Since you run Linux, check out QEMU. It is in Fedora Extras, so just "yum install qemu" and then read /usr/share/doc/qemu-*/qemu-doc.html. Then you can do all kinds of crazy things (like build a Linux x86-64 binary maybe? :) ).

alldeadhomiez
11-11-2006, 02:52 PM
It is obnoxious and I don't plan to distribute like that.
I don't need to static link, it was only to try and help portability. If GNU licensing and glibc are bent on making binary portability extra difficult so be it.

You can create a dynamically linked binary that runs on all "LSB" compliant distributions:

http://www.intel.com/cd/ids/developer/asmo-na/eng/245604.htm?page=1

Basically this makes the symbol versioning problems go away.

drez
11-11-2006, 05:17 PM
can anyone PM me tytompg please?

upload to www.sendspace.com (no signup required) or another filesharing site

bcc
11-11-2006, 09:21 PM
can anyone PM me tytompg please?Well I had wanted to squash some more bugs before taking the trouble to cut a release again, but here you go. Consider this more of a developmental snapshot (less tested and so forth). I know the code still is not handling things right when the broadcaster resets or rolls the timestamps. But I'm now re-generating precise video DTS timestamps, which is a lot more accurate than setting DTS=PTS. The code plays a lot less loose with the buffer timings in general, which may make some editing tools happier - untested.

bcc
11-11-2006, 09:28 PM
Since you run Linux, check out QEMU. It is in Fedora Extras, so just "yum install qemu" and then read /usr/share/doc/qemu-*/qemu-doc.html. Then you can do all kinds of crazy things (like build a Linux x86-64 binary maybe? :) ).WIll probably switch my desktop to 64 bit in a couple months, at which point that'd be a lot more convenient. In the mean time I already have vmware for emulation. Probably not worth switching emulators unless it was to improve hardware-vt support.

Kurtis
11-12-2006, 12:56 AM
Well I had wanted to squash some more bugs before taking the trouble to cut a release again, but here you go.

Squash bugs, bcc. Most of us can wait.

I don't mind helping, either. Post the source. ;-)

bcc
11-12-2006, 04:14 AM
Where do I find a stream fixer? It seems videoredo doesnt work on it. Encoders do strange things. HmmmI'm not sure.
I tried videoredo, and it skipped over the audio hole and broke sync as a result, just like hdemux. Even in quickstream fix mode. So I guess one can't recommend that. mpeg_streamclip was able to preserve the hole, even after editing. The best mpeg fixers might turn out to only be those that work on transport streams. In which case it'd make more sense to go ty->ts->transport-stream-fixer->mpg. Someone with experience with the transport stream fixers feel free to chime in. You could experiment by going ty->mpg->ts->transport-stream-fixer->mpg.

For me ty->mpg->player is sufficient, and good players recover from stream errors.

bcc
11-12-2006, 04:19 AM
Squash bugs, bcc. Most of us can wait.

I don't mind helping, either. Post the source. ;-)If you are actually serious about helping, you could work on post-mpeg conversion stream repair. A stream repair library for ffmpeg would be nice. See my last note.

bcc
11-12-2006, 04:56 AM
You can create a dynamically linked binary that runs on all "LSB" compliant distributions:Thanks, it's promising that there are businesses worrying about toolchains for this. Looks like the right intent, but seem to come with too much extra baggage (like that run-time interpreter layer, compile time wrappers, rpm requirements). Would be nice to just link with a non-copyleft posix+ansi libc. Like used to be the default...

I've dynamically linked with glibc for now.
setenv LD_ASSUME_KERNEL 2.6.9might help backwards compat. for some. Not sure; I can't test it and haven't seen it completely documented.

94SupraTT
11-12-2006, 09:14 AM
Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Light>e:

E:\>tytompg -i Lost3_06.ty -o Lost3_06new.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.4, Demuxer version 0.19
Source is Lost3_06.ty
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 18900000 bits/sec., 18900 Kbit/sec.
AC3 audio at 48kHz, 192 kbps, 768 bytes/sync frame
PES packet overlaps start code at 3465216 by 2 (predicted 2)
PES packet overlaps start code at 7240704 by 2 (predicted 2)
Assertion failed: code, file mplex.c, line 490

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.


I'll have to read through this thread to see if there is a solution.

bato
11-12-2006, 12:39 PM
I tried videoredo, and it skipped over the audio hole and broke sync as a result, just like hdemux. Even in quickstream fix mode. So I guess one can't recommend that.
If you report that to Videoredo coders and provide a file so they can work on it, maybe they can find a way to make their product work with this file.

RavenStL
11-12-2006, 02:29 PM
tytompg ver 0.4 is read into TMPGEnc ok. I am encoding right now. THANKS!

I will see how the audio sync is. The output of 0.4 seems identical to 0.3 when playing in WMP10. so glad to see TMPGEnc reading it now! :)

94SupraTT
11-12-2006, 03:42 PM
Why is this happening?




E:\>tytompg -i Lost3_06.ty -o Lost3_06new.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.4, Demuxer version 0.19
Source is Lost3_06.ty
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 18900000 bits/sec., 18900 Kbit/sec.
AC3 audio at 48kHz, 192 kbps, 768 bytes/sync frame
PES packet overlaps start code at 3465216 by 2 (predicted 2)
PES packet overlaps start code at 7240704 by 2 (predicted 2)
Assertion failed: code, file mplex.c, line 490

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

RavenStL
11-12-2006, 05:35 PM
My WMV re-encode version is choppy. Will try a dvd-compliant re-encode tomorrow (takes forever)

Also videoredo still fails on 0.4 output.

bcc
11-12-2006, 10:06 PM
PES packet overlaps start code at 3465216 by 2 (predicted 2)
PES packet overlaps start code at 7240704 by 2 (predicted 2)
This is where some OTA broadcast is being recorded with 2 bytes missing from the payload. This seems to be an alignment related framing bug on tivo's part that I'm recovering from. Only happens with OTA broadcasts that are sloppy about PES packet alignment in the first place. Should be safe to ignore; I probably should move that back to a debug-only message.

You can see the details in the hdemux source.

Assertion failed: code, file mplex.c, line 490This is the same as cheezmo's: http://www.dealdatabase.com/forum/showpost.php?p=268871&postcount=30
I haven't figured that one out yet. Something wrong with the parsing of the video frames. I assume this happens when there are stream errors.

bcc
11-12-2006, 10:07 PM
If you report that to Videoredo coders and provide a file so they can work on it, maybe they can find a way to make their product work with this file.Yes, tho I'd rather finish making tytompg as spec compliant as feasible first. Someone else could do this too...

bcc
11-12-2006, 10:12 PM
My WMV re-encode version is choppy. Will try a dvd-compliant re-encode tomorrow (takes forever)

Also videoredo still fails on 0.4 output.You could just work with ~1 minute clips to test with first. Then you won't waste CPU on transcodes that aren't right. Probably a few hundred chunks, clippable with tytompg using -s and -c.

videoredo works on my tytompg output. It just fills in audio holes incorrectly.

bcc
11-12-2006, 11:37 PM
So I put on my end user hat and converted some content to DVD. Worked. For this test I was using streams without holes. Example workflow: TY->mpg with tytompg
Used DGindex from dgmpgdec to set cut points, create .d2v, calculate a/v sync for avisynth.
Wrote avisynth script (including Telecide, Selecteven, crop, and lanczos4resize to covert 720p to dvd frame rate&resolution)
Transcoded with tmpgenc and/or CCE (both worked without errors)
Audio & video now DVD compliant; author with your favorite program.

mpauley
11-13-2006, 08:30 AM
FYI - Comskip works with my S2SA mpeg streams to remove commercials. I then run the mpeg and edl file thru mencoder and have a nice commercial free file to work with.

wbelhaven
11-14-2006, 03:12 PM
So I put on my end user hat and converted some content to DVD. Worked. For this test I was using streams without holes. Example workflow: TY->mpg with tytompg
Used DGindex from dgmpgdec to set cut points, create .d2v, calculate a/v sync for avisynth.
Wrote avisynth script (including Telecide, Selecteven, crop, and lanczos4resize to covert 720p to dvd frame rate&resolution)
Transcoded with tmpgenc and/or CCE (both worked without errors)
Audio & video now DVD compliant; author with your favorite program.

Would you mind sharing your AVIsynth script(s)? And does it/do they work with both film1 source and video2 source?

Thanks,
WB

My personal lexicon, in case others use these terms differently:
1 film = 24 unique progressive frames per second, typically hard or soft telecined.
2 video = ~60 unique interlaced fields (1080i, 480i) or progressive frames (720p) per second.

cheer
11-14-2006, 03:52 PM
Well I'm not bcc, but I'd use something like this (again, 720p):
loadplugin("dgdecode.dll")
loadplugin("tivtc.dll")
loadplugin("bt709tobt601.dll")

mpeg2source("greysanatomy.d2v")
selecteven()
tdecimate()
crop(x,y,-x,-y)
lanczosresize(704,480)
bt709tobt601Open up in your favorite mpeg encoder and encode. Note that this assumes a film source. I honestly wouldn't know what to do with a 720p video (60 unique progressive frames/sec) -- haven't encountered such a beast yet. Also, I've been told that for sizing down it's actually a smidge better (and faster) to use bilinearresize instead of lanczosresize, but I don't see a difference.

(By the way, am I the only one who thinks its ridiculous to take 24frames/sec film and broadcast it as 60frames/sec 720p?)

For 1080i, I'd remove the selecteven and add a tfm() before the tdecimate, again assuming film source. For video, it'd be much simpler: take out the tfm and the tdecimate and leave it interlaced. You might have to do separatefields, then the bt709tobt601, then weave, then crop/resize. But I'm not positive on that.

For 480i...you're already there if it's 720x480i. No re-encoding needed. For the 480x480i that DirecTV puts out on SD channels, again don't re-encode; just use DVD-Lab to author.

bcc
11-14-2006, 06:19 PM
Would you mind sharing your AVIsynth script(s)? And does it/do they work with both film1 source and video2 source?Sure, I've started a new thread over here:
http://www.dealdatabase.com/forum/showthread.php?t=51835
I've posted examples of both kinds of sources.

bcc
11-14-2006, 06:47 PM
TY->mpg with tytompg
Used DGindex from dgmpgdec to set cut points, create .d2v, calculate a/v sync for avisynth.
Wrote avisynth script (including Telecide, Selecteven, crop, and lanczos4resize to covert 720p to dvd frame rate&resolution)
Transcoded with tmpgenc and/or CCE (both worked without errors)
...
For steps 2-4, it looks like tmpgenc xpress can combine this into 1 step. Could save one from a lot of extra trouble with avisynth. Not sure if it's better or worse when it comes to fixing pulldown and resolutions. tmpgenc xpress is able to take mpeg2 directly from tytompg which is promising.

Now for recordings with audio holes, steps 2-4 above break as avisynth bombs out when it hits the audio hole. It doesn't bomb if you omit the audio stream...
tmpgenc xpress is able to detect the audio hole without bombing, but it never recovers the audio after the hole.
So for recordings with stream errors, you still need something to do stream repair before step 2.

bcc
11-14-2006, 06:58 PM
Well I'm not bcc, but I'd use something like this (again, 720p):You (and many others) probably have more encoding experience than I. Is your script handling ac3 audio somehow?

(By the way, am I the only one who thinks its ridiculous to take 24frames/sec film and broadcast it as 60frames/sec 720p?)
I wonder if it's the norm for OTA broadcasters to have equipment to do "the right thing" in real time for this case? May still be too expensive for some. avisynth is certainly not real time :)

94SupraTT
11-14-2006, 09:00 PM
This is where some OTA broadcast is being recorded with 2 bytes missing from the payload. This seems to be an alignment related framing bug on tivo's part that I'm recovering from. Only happens with OTA broadcasts that are sloppy about PES packet alignment in the first place. Should be safe to ignore; I probably should move that back to a debug-only message.

You can see the details in the hdemux source.
This is the same as cheezmo's: http://www.dealdatabase.com/forum/showpost.php?p=268871&postcount=30
I haven't figured that one out yet. Something wrong with the parsing of the video frames. I assume this happens when there are stream errors.


Any new word on this issue?

bcc
11-14-2006, 09:14 PM
Any new word on this issue?Two separate issues being mentioned in that post. One cosmetic, concerning audio, and one assertion failure concerning video. Nothing to do on the first one. For the second, I thought I'd bump into the problem with a little more testing, but I have not. That is to say I haven't seen a stream that reproduces that failure. I may need a sample.

cheer
11-15-2006, 12:59 AM
You (and many others) probably have more encoding experience than I.Everything's relative. :) I feel like a babe in the woods when I stick my toe into the deep, cold waters of doom9.
Is your script handling ac3 audio somehow?No. If I'm going to DVD, then I'm using DVD-Lab so I just feed it the elementary streams. If I'm going to an MPEG-4 variant like Xvid, I'll do all the encoding and then mux with VirtualDubMod.

So far I've had no real sync issues to speak of, especially using hdemux 0.18/mplex. (Haven't had a chance to throw some streams at tytompg yet, but I will probably be able to tomorrow.)

I definitely recommend people use (or at least try) the BT709toBT601 (http://home.comcast.net/~trbarry/) filter. HD MPEG files are (always? usually? sometimes?) in the BT.709 colorspace, whereas SD MPEG/DVD/etc. are BT.601. Sometimes I find this makes no difference (at which point I assume the HD vids are already converted) but sometimes it makes a significant difference -- without it (in those cases) I can see chroma shifting.

Kurtis
11-15-2006, 05:12 PM
Downloaded tytompeg, and it's throwing a floating point exception on launch. Version 0.14 on Ubuntu 6.10.

Ran it through gdb, and it's getting the exception in dl_rtld_di_serinfo() in the ld library, but that's probably wrong because the backtrace says the top two stack frames are 0x1 and 0x0, so my guess is that your release build is compiled with -fomit-frame-pointer so stack crawls aren't possible.

Anyway, this kind of thing, actually, is why I was poking you for the source. I'm too busy with looking at TyShow to help implement features, but I could help squash little bugs like this one.

EDIT: I, of course, meant version 0.4.

bcc
11-15-2006, 05:31 PM
Downloaded tytompeg, and it's throwing a floating point exception on launch. Version 0.14 on Ubuntu 6.10.

Ran it through gdb, and it's getting the exception in dl_rtld_di_serinfo() in the ld library, but that's probably wrong because the backtrace says the top two stack frames are 0x1 and 0x0, so my guess is that your release build is compiled with -fomit-frame-pointer so stack crawls aren't possible.tytompg uses no such function. Looks like your OS is not able to dynamically link this right. I wonder whether ubuntu is using a compatible glibc. It's compiled -O3 which I believe implies omit-frame-pointer.


Anyway, this kind of thing, actually, is why I was poking you for the source. I'm too busy with looking at TyShow to help implement features, but I could help squash little bugs like this one.

EDIT: I, of course, meant version 0.4.It's a shame that windows binary portability is much better than linux. If you're using tyshow, you could just try the windows binary.

bcc
11-15-2006, 05:35 PM
I was poking you for the source.Yes. AGAIN. I believe I asked folks to take this source code axe grinding to another thread. I'm getting seriously tired of it.

bcc
11-15-2006, 06:30 PM
Downloaded tytompeg, and it's throwing a floating point exception on launch. Version 0.14 on Ubuntu 6.10.I figured it out. The gcc option --hash-style=sysv doesn't do anything (silently fails). I needed to have used -Wl,--hash-style=sysv instead...

Kurtis
11-16-2006, 09:59 AM
Yes. AGAIN. I believe I asked folks to take this source code axe grinding to another thread. I'm getting seriously tired of it.

Sorry, bcc. I don't have an ideological axe to grind; just used to poking at source.

I'll leave you alone.

chezpaul
11-16-2006, 11:48 PM
I know I'm coming in late but I would also greatly appreciate a OS X version for what it's worth... Just puting the word out... :)

Does this work with HD content ? I've not been able to get my HD video to quite sync with the audio yet.

For what it's worth... Are apple people smarter ? (http://news.com.com/2100-1040-943519.html)
But we all knew that right ? ;)

RavenStL
11-17-2006, 12:01 AM
For what it's worth... Are apple people smarter ? (http://news.com.com/2100-1040-943519.html)
But we all knew that right ? ;)

Yea, OK.... and the point brought by that was for what? "Its a MAC thing, you wouldnt understand" I guess....

I got one, Mensa test. First and only question. Do you own a MAC? :)

well anyways, I am having trouble getting a good encode. Apparently the video just STOPS encoding and only the audio finishes.

The clip was 3.5 hours. I had like 20 minutes of video, stopped for no obvious reason (no audio hole) and then only finished the audio. SO I had a 3.5 hour audio clip with 20 minutes of video.

THis was FOX OTA encoding with tmpgenc.

bcc
11-17-2006, 01:57 AM
well anyways, I am having trouble getting a good encode. Apparently the video just STOPS encoding and only the audio finishes.

The clip was 3.5 hours. I had like 20 minutes of video, stopped for no obvious reason (no audio hole) and then only finished the audio. SO I had a 3.5 hour audio clip with 20 minutes of video.

THis was FOX OTA encoding with tmpgenc.The sample that you sent me yesterday, that I was too busy to look at until today, shows the video is there. It's just that tmpgenc seems to have trouble processing it. As before, I believe this is a case where tmpgenc is being a lot less liberal than mpeg players are in accepting PTS values that it considers to be out of range. I believe I've fixed this since 0.4. Life beyond DDB has kept me from cutting a release yet.

I've run hours of OTA HD content thru avisynth+CCE without error...

bcc
11-17-2006, 02:33 AM
Note that this version will likely bomb out with lots of errors if the broadcaster resets their timestamps. May also just assert with:
video: Timestamp warping problem
tytompg: mplex.c:1546: update_streamtime: Assertion `0' failed.
Abortin such cases. I'm currently fixing those cases (remember this is alpha code), but folks might want to try this snapshot anyways.
Changes include: Prevent PTS from ever getting ahead of DTS (if our DTS calculation differs from the broadcaster's by roundoff for example) Improve DTS calculation (look forward instead of back to compute delta)

Thinkdiff
11-17-2006, 03:20 AM
For what it's worth... Are apple people smarter ? (http://news.com.com/2100-1040-943519.html)
But we all knew that right ? ;)

I'm sure that's gonna help :rolleyes:

AlphaWolf
11-17-2006, 03:45 AM
For what it's worth... Are apple people smarter ? (http://news.com.com/2100-1040-943519.html)
But we all knew that right ? ;)

Well, it isn't very obvious. I mean, you'd figure that the smarter person would do their research before buying a computer. And somebody who did their research would know that you can run pretty much anything and perform pretty much any task on a PC.

The smart person therefore would also know that with a mac on the other hand you have to frequently beg the authors of your favorite programs to cater to your needs, and the vast majority of the time you don't get what you want in spite of doing so, and therefore you still need a PC to perform your desired tasks anyways. Yet there's no need for a mac if you own a PC.

bcc
11-17-2006, 04:12 AM
I know I'm coming in late but I would also greatly appreciate a OS X version for what it's worth... Just puting the word out... :)

Does this work with HD content ? I've not been able to get my HD video to quite sync with the audio yet.

For what it's worth... Are apple people smarter ? (http://news.com.com/2100-1040-943519.html)
But we all knew that right ? ;)Being smug about ones desktop OS choice doesn't seem like "smart" behavior to me. That smugness is part of how the recent apple commercial has been backfiring to a good extent (tho it's a little entertaining I admit).

It's also a lousy way to ask for help

chezpaul
11-17-2006, 04:15 AM
Well, it isn't very obvious. I mean, you'd figure that the smarter person would do their research before buying a computer. And somebody who did their research would know that you can run pretty much anything and perform pretty much any task on a PC.

The smart person therefore would also know that with a mac on the other hand you have to frequently beg the authors of your favorite programs to cater to your needs, and the vast majority of the time you don't get what you want in spite of doing so, and therefore you still need a PC to perform your desired tasks anyways. Yet there's no need for a mac if you own a PC.

Yeah well where's the fun to getting all you want so easily ?:D

Back to topic please. This is most interesting. I'll just wait and see and hope.

I do all my ty. to avi convertion with ffmpeg but it just doesn't cut it for me. All my HD material just losses sync and quality somehow.
Then again I'm getting my Mac Intel next week so...

artships
11-17-2006, 11:18 AM
bcc,

Sound plays great while the video stops and starts (actually, more like stops; goes back; starts again). Fast-forwarding or moving the slider kills the audio completely. Still wicked fast! Hope it works on HD streams.

MPEG2 audio at 32kHz, 192 kbps, 864 bytes/sync frame
Video frame size is 544x480 Frame rate 29.970030
Video bit rate is 5800000 bits/sec., 5800 Kbit/sec.
MPEG2 audio at 32kHz, 192 kbps, 864 bytes/sync frame
Done!
Stream audio: 103371 frames (2791 seconds aprox.)
Stream video: 111533 frames, 223066 fields (3721 seconds aprox.)

millercentral
11-18-2006, 03:04 AM
Not sure if this is useful or not:

- Extracted a SD show (BSG "Heros") from my HR10-250 (6.3a)
- Ran it through tytompg 0.5, generated a mpg without error (damn fast)
- Opened the mpg in ProjectX and ran it through demux with default settings.

It worked, but generated a lengthly log file, which I've attached. Are any of these errors interesting to you? Let me know if you want some/all of the clips to debug.

Thanks for a great tool!!

RavenStL
11-18-2006, 12:51 PM
The sample that you sent me yesterday, that I was too busy to look at until today, shows the video is there. It's just that tmpgenc seems to have trouble processing it. As before, I believe this is a case where tmpgenc is being a lot less liberal than mpeg players are in accepting PTS values that it considers to be out of range. I believe I've fixed this since 0.4. Life beyond DDB has kept me from cutting a release yet.

I've run hours of OTA HD content thru avisynth+CCE without error...

Same problem, same spot. TMPGEnc has no video at the pizza hut commercial. 0.5 doesnt seem to help.

94SupraTT
11-18-2006, 04:37 PM
For the second, I thought I'd bump into the problem with a little more testing, but I have not. That is to say I haven't seen a stream that reproduces that failure. I may need a sample.


Alrighty. How big of a sample would you like? Let me know and I'll upload it to sendspace and post the link.

bcc
11-18-2006, 05:17 PM
bcc,

Sound plays great while the video stops and starts (actually, more like stops; goes back; starts again). Fast-forwarding or moving the slider kills the audio completely. Still wicked fast! Hope it works on HD streams.

MPEG2 audio at 32kHz, 192 kbps, 864 bytes/sync frame
Video frame size is 544x480 Frame rate 29.970030
Video bit rate is 5800000 bits/sec., 5800 Kbit/sec.
MPEG2 audio at 32kHz, 192 kbps, 864 bytes/sync frame
Done!
Stream audio: 103371 frames (2791 seconds aprox.)
Stream video: 111533 frames, 223066 fields (3721 seconds aprox.)
Ok, I've tried vlc now and I can see that it seems more touchy about low-def streams than xine or mplayer or windvd. (High def streams seem less likely to show any problem). Have you tried one of those players? Actually all I see is vlc showing a jump in the video roughly once every 5 seconds. Looks like it's playing the frames out of order at those points. Anyone know how to turn on diagnostic logging with vlc?

bcc
11-18-2006, 05:34 PM
Alrighty. How big of a sample would you like? Let me know and I'll upload it to sendspace and post the link.Just a couple seconds of video around the failure point would be fine, thanks. That'd be ~50 chunks if we're talking HD.

bcc
11-18-2006, 11:29 PM
Ok, I've tried vlc now and I can see that it seems more touchy about low-def streams than xine or mplayer or windvd.Looks like vlc was unhappy about the rate of timestamps. By accident I was often not outputting the PTS timestamps (and the computed DTS timestamps) for video P frames (which was putting the rate of timestamps out of spec). I put up version 0.6 with this fixed. vlc now works fine as far as I've tested.

94SupraTT
11-20-2006, 08:07 PM
Anyone encounter this yet?



E:\>tytompg -i Bear.ty -o Bear.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.4, Demuxer version 0.19
Source is Bear.ty
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
AC3 audio at 48kHz, 448 kbps, 1792 bytes/sync frame
video: Warning: Decoder buffer empty, scr ahead by 221266 (737.166) 0.737
... and encoder buffer full
While processing stream audio...
Assertion failed: !"Encoder buffer full", file mplex.c, line 466

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

bcc
11-20-2006, 08:37 PM
Anyone encounter this yet?
Not I. But you're two releases behind...

94SupraTT
11-20-2006, 08:57 PM
Not I. But you're two releases behind...

:o

Thanks for the heads up. I'll retry the file with the new release.

FS1
11-20-2006, 10:49 PM
All of my OTA extractions immediately fail with an assertion like this:

AC3 audio at 48kHz, 384 kbps, 1536 bytes/sync frame
Recovered AC3 sync, offset 112, chunk 4
tytompg: demux.c:1751: demux_prime: Assertion `shown_rate' failed.
Aborted
This happens using both the Linux and Windows binaries and with both the 0.4 and 0.6 releases. I've tried in on the local NBC, ABC, and PBS affiliates. The code works great on all the other HD streams I've tried (HDNet, Showtime, ESPNHD, CDUSA).

My HR10-250 is currently running 6.3a, but I have also tested on some older files extracted from 3.1.5f. All extractions were done using TyTool 10r4.

When it works, this is by far the best tystream processing tool I've used so I look forward its further development.

bcc
11-21-2006, 12:05 AM
All of my OTA extractions immediately fail with an assertion like this:

AC3 audio at 48kHz, 384 kbps, 1536 bytes/sync frame
Recovered AC3 sync, offset 112, chunk 4
tytompg: demux.c:1751: demux_prime: Assertion `shown_rate' failed.
Aborted
This happens using both the Linux and Windows binaries and with both the 0.4 and 0.6 releases. I've tried in on the local NBC, ABC, and PBS affiliates. The code works great on all the other HD streams I've tried (HDNet, Showtime, ESPNHD, CDUSA).

My HR10-250 is currently running 6.3a, but I have also tested on some older files extracted from 3.1.5f. All extractions were done using TyTool 10r4.

When it works, this is by far the best tystream processing tool I've used so I look forward its further development.
There should also be some output about the video stream, such as the following. In your example above, tytompg is not finding any parsable video in the first 64 or so chunks, which I've never seen.:
Multiplexer version 0.6, Demuxer version 0.19
Source is warhol_1.ty
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 12910000 bits/sec., 12910 Kbit/sec.
(This example is taken from OTA on an hr10-250 with 6.3a, PBS).
There may be a problem with your .ty files... Can you demux the .m2v with hdemux? (hdemux -i file.ty -v file.m2v)

FS1
11-21-2006, 12:24 AM
Full output:

tytompg -i Late\ Night\ With\ Conan\ O\'Brien--1.ty -o test.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.4, Demuxer version 0.19
Source is Late Night With Conan O'Brien--1.ty
AC3 audio at 48kHz, 384 kbps, 1536 bytes/sync frame
Recovered AC3 sync, offset 112, chunk 4
tytompg: demux.c:1741: demux_prime: Assertion `shown_rate' failed.
Aborted

Using hdemux:

hdemux -i Late\ Night\ With\ Conan\ O\'Brien--1.ty -v test.m2v
Version 0.18
Source is Late Night With Conan O'Brien--1.ty
AC3 audio at 48kHz, 1536 bytes/sync frame
Recovered AC3 sync, offset 112, chunk 4
Changing video frame rate to 59.940060 from 29.970030
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 80000000 bits/sec., 80000 Kbit/sec.
18307536 audio stream bytes 2662656 video stream bytes
Reported datarate 80384 Kbit/sec. (80000+384)
Proceed with remuxer:
mplex -f 8 -O -18851267ms -r 153701 (null) test.m2v -o <outfile>.mpg
OR
ffmpeg -acodec copy -vcodec copy -i test.m2v -itsoffset -18851.2677 -f ac3 -i (null) <outfile>.vob

The clip is not 720p (it's NBC), so I'm not sure what's up. The resulting test.m2v is only 2.5MB while the original tystream is 704MB.

For problem streams like this, the only tool I've found that works is the latest VLC on Windows. It will create a program stream that has only video in it. I can then demux an elementary video stream and use the audio extracted with hdemux to create something usable, but that's far from ideal.

bcc
11-21-2006, 12:50 AM
The clip is not 720p (it's NBC), so I'm not sure what's up. The resulting test.m2v is only 2.5MB while the original tystream is 704MB.

For problem streams like this, the only tool I've found that works is the latest VLC on Windows. It will create a program stream that has only video in it. I can then demux an elementary video stream and use the audio extracted with hdemux to create something usable, but that's far from ideal.Can you play the .ty with mplayer?
If hdemux says it's 720p then your .ty file has some 720p video frames in it. This may mean you're looking at garbage chunks not the recording you want. You can turn on debugging (-dr) with hdemux to see which chunks have valid video data in them. You may want to re-extract with something other than tytool, as tytool is known buggy with 6.3a.

FS1
11-21-2006, 02:12 AM
Can you play the .ty with mplayer?
If hdemux says it's 720p then your .ty file has some 720p video frames in it. This may mean you're looking at garbage chunks not the recording you want. You can turn on debugging (-dr) with hdemux to see which chunks have valid video data in them. You may want to re-extract with something other than tytool, as tytool is known buggy with 6.3a.
Don't have mplayer handy. My linux box is the file server that I extract all of this stuff to (having linux binaries is awesome), so no X. I'll look into it later, though.

I just extracted that stream again with mfs_ftp. Same results.

wget --no-passive-ftp ftp://10.1.0.50:3105/ty+/%7BLate%20Night%20With%20Conan%20O%27Brien%7D%7B2006-11-17%7D%7B%7D%7B11.37%20PM%20Fri%20Nov%2017,%202006%7D%7BWxxxDT%7D.ty+
--23:34:48-- ftp://10.1.0.50:3105/ty+/%7BLate%20Night%20With%20Conan%20O%27Brien%7D%7B2006-11-17%7D%7B%7D%7B11.37%20PM%20Fri%20Nov%2017,%202006%7D%7BWxxxDT%7D.ty+ => `{Late Night With Conan O'Brien}{2006-11-17}{}{11.37 PM Fri Nov 17, 2006}{WxxxDT}.ty+'
Connecting to 10.1.0.50:3105... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD /ty+ ... done.
==> PORT ... done. ==> RETR {Late Night With Conan O'Brien}{2006-11-17}{}{11.37 PM Fri Nov 17, 2006}{WxxxDT}.ty+ ... done.
7,675,583,151 3.70M/s

00:06:46 (3.82 MB/s) - `{Late Night With Conan O'Brien}{2006-11-17}{}{11.37 PM Fri Nov 17, 2006}{WxxxDT}.ty+' saved [7675583151]

tytompg -i \{Late\ Night\ With\ Conan\ O\'Brien\}\{2006-11-17\}\{\}\{11
.37\ PM\ Fri\ Nov\ 17\,\ 2006\}\{WxxxDT\}.ty+ -o test.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.4, Demuxer version 0.19
Source is {Late Night With Conan O'Brien}{2006-11-17}{}{11.37 PM Fri Nov 17, 2006}{WxxxDT}.ty+
AC3 audio at 48kHz, 384 kbps, 1536 bytes/sync frame
Recovered AC3 sync, offset 1296, chunk 1
tytompg: demux.c:1741: demux_prime: Assertion `shown_rate' failed.
Aborted
hdemux doesn't find the video at all this time:

hdemux -i \{Late\ Night\ With\ Conan\ O\'Brien\}\{2006-11-17\}\{\}\{11.37\ PM\ Fri\ Nov\ 17\,\ 2006\}\{WxxxDT\}.ty+ -v test.m2v
Version 0.18
Source is {Late Night With Conan O'Brien}{2006-11-17}{}{11.37 PM Fri Nov 17, 2006}{WxxxDT}.ty+
AC3 audio at 48kHz, 1536 bytes/sync frame
Recovered AC3 sync, offset 1296, chunk 1

In case it's helpful, the first part of the hdemux debug log is attached.

bcc
11-21-2006, 03:35 AM
Don't have mplayer handy. My linux box is the file server that I extract all of this stuff to (having linux binaries is awesome), so no X. I'll look into it later, though.You can run the player across the network on some other machine (via samba or nfs). Also I wonder what you get when you play the .m2v. I suspect it's leftover garbage that is part of an old recording.
In case it's helpful, the first part of the hdemux debug log is attached.Well it shows that there is plenty of video data throughout, including start of video sequences, so I don't see why hdemux hasn't been able to demux the video. I just tried some NBC OTA from an hr10-250 with 6.3a and it worked no problem. Maybe your transfers are not 8 bit clean? Just guessing; it's hard to tell what's wrong from here.

94SupraTT
11-21-2006, 08:29 AM
No dice. :(



E:\>tytompg -i Bear.ty -o Bear.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.6, Demuxer version 0.19
Source is Bear.ty
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
AC3 audio at 48kHz, 448 kbps, 1792 bytes/sync frame
video: Warning: Decoder buffer empty, scr ahead by 1142 (3.242) 0.3
... and encoder buffer full
While processing stream audio...
Assertion failed: !"Encoder buffer full", file mplex.c, line 471

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

FS1
11-22-2006, 01:04 AM
You can run the player across the network on some other machine (via samba or nfs). Also I wonder what you get when you play the .m2v. I suspect it's leftover garbage that is part of an old recording.
Kinda. The 2.5MB file that hdemux creates is actually a 720p clip of upconverted Scrubs. Don't know where that came from, since I don't know of any 720p channels that air Scrubs. It plays back just fine in Media Player Classic though.
Video: MPEG2 Video 1280x720 59.94fps 80000Kbps [Video] DGIndex is also OK with it.

mplayer on windows:
mplayer test.m2v
MPlayer 1.0rc1-3.4.2 (C) 2000-2006 MPlayer Team
CPU: AMD Athlon(tm) 64 Processor 3500+ (Family: 15, Model: 15, Stepping: 0)
CPUflags: MMX: 1 MMX2: 1 3DNow: 1 3DNow2: 1 SSE: 0 SSE2: 0
Compiled with runtime CPU detection.
c:/windows/fonts/arial.ttf doesn't look like a bitmap font description, ignoring.
Cannot load bitmap font: c:/windows/fonts/arial.ttf
Playing test.m2v.
H264-ES file format detected.
FPS not specified in the header or invalid, use the -fps option.
No stream found.
Exiting... (End of file)


Maybe your transfers are not 8 bit clean? Just guessing; it's hard to tell what's wrong from here.Wouldn't that bork all of them? So far it's only OTA that's weird. Plus VLC is still able to dump the video. You want a sample yet? :)

bcc
11-22-2006, 04:26 AM
Kinda. The 2.5MB file that hdemux creates is actually a 720p clip of upconverted Scrubs. Don't know where that came from, since I don't know of any 720p channels that air Scrubs.They must have been garbage chunks at the end of the file.
mplayer on windowsWhat I was getting at was whether you could play back the .ty file with mplayer (or tyshow). If not that would further suggest something special about your .ty. If yes then it'd point to something wrong (and probably simple) with hdemux's parsing of the TY records.
Wouldn't that bork all of them?Yes, you'd think.
So far it's only OTA that's weird. Plus VLC is still able to dump the video. You want a sample yet? :)I thought you wanted to make me play 20 questions. Yes please that'd be a lot more productive.:)

94SupraTT
11-23-2006, 01:26 AM
As with hdemux, tytompg supports the -s and -c options to skip chunks & limit the number of chunks processed, for your troubleshooting pleasure :)
Hopefully this will get fixed when I'm done with the fix for the more troublesome OTA recordings.

How do I calculate how many chunks equals how many minutes? I want to start in the middle of a ty.

RavenStL
11-23-2006, 08:03 PM
Chunks are fixed size, so trial and error is how I do it.

720P appears to be approx 1000 chunks per one min

Since 720P has the highest data rate, I would assume that is worse case?

FS1
11-23-2006, 11:00 PM
What I was getting at was whether you could play back the .ty file with mplayer (or tyshow).I'm out of town for Thanksgiving, but I'll try that when I get back. I think the answer is no, but I'll report if the result is otherwise.

I thought you wanted to make me play 20 questions. Yes please that'd be a lot more productive.:)My concern was that I was just doing something wrong. I wanted to make sure I wasn't creating unnecessary work with my "special" tystreams first. :)

Do 50MB chunks from a few files sound OK? It won't be till late Saturday anyway.

bcc
11-25-2006, 03:21 AM
Same problem, same spot. TMPGEnc has no video at the pizza hut commercial. 0.5 doesnt seem to help.I found the problem in this FOX recording (the clip with the pizza hut commercial). In the video stream, fox is sometimes putting mpeg2 video stream end codes between commercials. tmpgenc xpress is treating this as a sign to stop encoding the video stream (even tho there is plenty more valid video stream packets; go figure) I'll strip out these end codes. I've tested this somewhat and it makes tmpgenc happy.

bcc
11-25-2006, 03:24 AM
How do I calculate how many chunks equals how many minutes? I want to start in the middle of a ty.

File size in bytes/131072 = # of chunks in file
(Duration of clip/Duration of whole recording)*#chunks in file=aprox. # of chunks in clip.

bcc
11-25-2006, 03:33 AM
I'm out of town for Thanksgiving, but I'll try that when I get back. I think the answer is no, but I'll report if the result is otherwise.Well if the answer is no you can't play back the .ty via normal means then it suggests something iis unusual/wrong with your extracted .ty files. No hurry, I've had plenty other things going on as well :)
Do 50MB chunks from a few files sound OK? It won't be till late Saturday anyway.Yes that sounds like plenty

bcc
11-25-2006, 03:15 PM
I've posted a new version (link at head of this thread). Changes: Add careful modulus arithmetic for timestamps to handle timestamp wrap-around Drop mid-stream video sequence end codes to prevent false end of recording indicators

FS1
11-25-2006, 10:13 PM
Well if the answer is no you can't play back the .ty via normal means then it suggests something iis unusual/wrong with your extracted .ty files.50MB Clip (http://x/{Late Night With Conan O'Brien}{2006-11-17}{}{11.37 PM Fri Nov 17, 2006}{WxxxDT}-chunk.ty+)

In another bit of strangeness, mplayer will play the 50MB clip but not the 7.14GB original. My mplayer tests are all from windows, where I have all of this data mapped via Samba. I started to think that might be some of the problem, but I think it's just an mplayer bug. In any case, tytompeg 0.7 (the native linux build) still fails on the 50MB chunk so the original problem should still be evident in the sample.

FYI, I made the file with dd (which shouldn't matter).

dd if=file.ty+ of=file.ty bs=512k count=100

bcc
11-26-2006, 12:21 AM
fs1,
OK, I took a look and found the problem. In the video sequence header, I was looking at the wrong byte to check the bit for the non-intra quantiser matrix flag. That probably sounds greek, but it just means I've been mis-parsing these packets all along with hdemux, but only if you have this special extension. None of my test cases had this extension (including late night NBC broadcasts...)
I've fixed the parsing in 0.8 (now posted).
FYI, I made the file with dd (which shouldn't matter).

dd if=file.ty+ of=file.ty bs=512k count=100Right I always use dd as well (I just recommend tychopper for windows users).

FS1
11-26-2006, 12:44 AM
fs1,
OK, I took a look and found the problem. In the video sequence header, I was looking at the wrong byte to check the bit for the non-intra quantiser matrix flag.Sweet.:D
The new version works like a charm up to about 5.1GB in where it gives this:
tytompg -i {Late Night With Conan O'Brien}{2006-11-17}{}{11.37 PM Fri Nov 17, 2006}{WxxxDT}.ty+ -o conan6.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.8, Demuxer version 0.19
Source is {Late Night With Conan O'Brien}{2006-11-17}{}{11.37 PM Fri Nov 17, 2006}{WxxxDT}.ty+
AC3 audio at 48kHz, 384 kbps, 1536 bytes/sync frame
Recovered AC3 sync, offset 1296, chunk 1
Video frame size is 1920x1080 - high definition. Frame rate 29.970030
Video bit rate is 15000000 bits/sec., 15000 Kbit/sec.
Recovered AC3 sync, offset 1296, chunk 1
tytompg: mplex.c:1763: stream_update_parsed: Assertion `parsed_au->code != 0xff' failed.
Aborted
This is OTA of course, so I could just have a glitch in the tystream. If you think it might be a bug, I'll extract that chunk and post it also.

Thanks for your help! It's so great to have a fully (almost) working tool. Makes my life so much easier.

bcc
11-26-2006, 01:14 AM
tytompg: mplex.c:1763: stream_update_parsed: Assertion `parsed_au->code != 0xff' failed.
Aborted
[/CODE]This is OTA of course, so I could just have a glitch in the tystream. If you think it might be a bug, I'll extract that chunk and post it also.

Thanks for your help! It's so great to have a fully (almost) working tool. Makes my life so much easier.Cool. That one looks like another bug; a sample would help.

FS1
11-26-2006, 01:17 AM
Clip #2 - 25MB (http://x/CDUSA-chunk.ty)

Found a clip that works with hdemux, but fails with tytompg

tytompg -i CDUSA.ty -o CDUSA.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.9, Demuxer version 0.19
Source is CDUSA.ty
AC3 audio at 48kHz, 448 kbps, 1792 bytes/sync frame
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
While processing stream audio...
tytompg: mplex.c:471: stream_queue_au_buf: Assertion `!"Encoder buffer full"' failed.
Aborted

FS1
11-26-2006, 01:31 AM
Cool. That one looks like another bug; a sample would help.
Clip #3 - 25MB (http://204.27.135.172/conan-clip3.ty)

Sorry for the not-so-fast download speed.

TivoZA
11-26-2006, 02:26 AM
Hi guys, I'm having a problem converting a PAL ty file using tytompg. When playing back the converted file, the audio is fine but the video stutters. I was wondering if this somehow relates to the fact that PAL has 25.0 fps whereas NTSC has 29.970 fps?

I'm also getting hundreds of warnings stating "Decoder buffer empty".

tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.7, Demuxer version 0.19
Source is test.ty
Video frame size is 544x576 Frame rate 25.000000
Video bit rate is 5800000 bits/sec., 5800 Kbit/sec.
MPEG2 audio at 32kHz, 192 kbps, 864 bytes/sync frame
MPEG2 audio at 32kHz, 192 kbps, 864 bytes/sync frame
video: Warning: Decoder buffer empty, scr ahead by 1088872 (3629.172) 0.3629
video: Warning: Decoder buffer empty, scr ahead by 1091125 (3637.25) 0.3637
video: Warning: Decoder buffer empty, scr ahead by 1091125 (3637.25) 0.3637
video: Warning: Decoder buffer empty, scr ahead by 1091125 (3637.25) 0.3637
video: Warning: Decoder buffer empty, scr ahead by 1091125 (3637.25) 0.3637
video: Warning: Decoder buffer empty, scr ahead by 1091125 (3637.25) 0.3637
video: Warning: Decoder buffer empty, scr ahead by 1091125 (3637.25) 0.3637
...

bcc
11-26-2006, 08:13 PM
Clip #2 - 25MB (http://204.27.135.172/CDUSA-chunk.ty)

Found a clip that works with hdemux, but fails with tytompg

tytompg -i CDUSA.ty -o CDUSA.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.9, Demuxer version 0.19
Source is CDUSA.ty
AC3 audio at 48kHz, 448 kbps, 1792 bytes/sync frame
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
While processing stream audio...
tytompg: mplex.c:471: stream_queue_au_buf: Assertion `!"Encoder buffer full"' failed.
Aborted

This clip doesn't make much sense as it starts with about 25 seconds of audio only (no video) and then goes to normal a/v. Was there a still frame or a video glitch in the original recording? Looks like I should be handling this as a 25 second video hole.

bcc
11-26-2006, 08:22 PM
Clip #3 - 25MB (http://204.27.135.172/conan-clip3.ty)

Sorry for the not-so-fast download speed.Fixed. I've uploaded 0.9 with the fix. Basically I was sometimes mishandling the video frame type when the GOP is large enough to require splitting into pieces.

gyre
11-26-2006, 09:05 PM
If you find a GOP longer than 15 (PAL) do you break it into pieces? I'm currently using videoredo for this purpose.

Thanks for an awesome program. It even works on the crap that my S1 UK tivo spits out :)

-- gyre --

mchahn
11-26-2006, 09:48 PM
I just want to thank the author of TyToMpg. I have been trying to find some way to archive my HD videos from the HR10-250 and watch them from my PC. I have used TyTools, mencoder, mediaCoder, revideo, drdixv, etc., and everything had one problem or the other.

Now I use tytools to download a .ty file and TyToMpeg and I'm done. I switched back and forth from my PC to TIVO and the only difference was a bit of reduced color saturation probably due to my PC's video card. VLC can probably fix that.

External USB drives are so cheap now that I will archive the full-size files and just keep buying drives. The 800 gigabytes in my TiVo became too small very quickly.

tall1
11-26-2006, 10:11 PM
Let me echo mchahn's gratitude to the author(s) of TyToMpg. This is so dirt simple I don't miss the Multiplex mode in TyTool. It is so cool to have the Firefly eps on my HTPC. Thanks again.

FS1
11-26-2006, 10:19 PM
This clip doesn't make much sense as it starts with about 25 seconds of audio only (no video) and then goes to normal a/v. Was there a still frame or a video glitch in the original recording? Looks like I should be handling this as a 25 second video hole.That one isn't on the tivo anymore, so I can't play it and see what happens. I don't remember it being special in any way, though. A 25 second hole at the head is OK with me. That's why I pad my recordings. :)

Here's another problem clip, but I think this was caused by bad reception. Ideally tytompg should do its best and keep going?
Clip #4 - 15MB (http://x/lost-chunk.ty) does this:

tytompg -i lost-chunk.ty -o lost-chunk.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.9, Demuxer version 0.19
Source is lost-chunk.ty
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 80000000 bits/sec., 80000 Kbit/sec.
AC3 audio at 48kHz, 384 kbps, 1536 bytes/sync frame
Warning: temporal_reference order problem. Value: 2 group_start: 326 last: 329
video: Warning: Decoder buffer empty, scr ahead by 1614850 (5382.250) 0.5382
Warning: temporal_reference order problem. Value: 2 group_start: 402 last: 415
tytompg: mplex.c:502: stream_au_set_code: Assertion `code' failed.
Aborted

This next one isn't as obvious (to me):
Clip #5 - 15MB (http://x/office-chunk.ty)

tytompg -i office-chunk.ty -o office-chunk.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.9, Demuxer version 0.19
Source is office-chunk.ty
Video frame size is 1920x1080 - high definition. Frame rate 29.970030
Video bit rate is 15000000 bits/sec., 15000 Kbit/sec.
AC3 audio at 48kHz, 384 kbps, 1536 bytes/sync frame
Recovered AC3 sync, offset 1152, chunk 5
Recovered AC3 sync, offset 1152, chunk 5
tytompg: video.c:81: vid_frame_rate: Assertion `0' failed.
Aborted


I've had at least 20 other extractions work flawlessly with these last two versions, so you're definitely getting close. That's pretty good, since most of the tystreams I have hanging around are the ones that wouldn't work with other tools.

94SupraTT
11-26-2006, 11:09 PM
Got this while trying to do a Lost OTA Stream...




Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Light>e:

E:\>tytompg -i Lost3_06.ty -o lost306.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.9, Demuxer version 0.19
Source is Lost3_06.ty
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 18900000 bits/sec., 18900 Kbit/sec.
AC3 audio at 48kHz, 192 kbps, 768 bytes/sync frame
Warning: temporal_reference order problem. Value: 2 group_start: 8692 last: 869
9
Warning: temporal_reference order problem. Value: 2 group_start: 16635 last: 16
641
Warning: temporal_reference order problem. Value: 2 group_start: 16718 last: 16
724
Warning: temporal_reference order problem. Value: 11 group_start: 18128 last: 1
8157
Warning: temporal_reference order problem. Value: 5 group_start: 28380 last: 28
388
Assertion failed: code, file mplex.c, line 502

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

danpritts
11-27-2006, 03:12 AM
I just tried using tytompg to convert a file encoded on a series1 tivo.

used version 0.9 on linux.

No errors, but the video is really choppy when played in vlc, as if some of the intermediate frames are being lost. The original ty plays fine in vlc.

I'll PM you with a URL where you can see the files in question.

Also, to follow up on early threads regarding the tool's speed, I'm using an 800MHz Via C3 and it seemed plenty fast to me, as if it were limited by I/O speed. If it seems fast on THIS computer you don't have to worry.

Here is the output I got from the program.

Also, note that VLC semi-randomly chokes on the file; it will sometimes work fine and then other times audio doesn't work until I restart VLC. Not sure this is useful info or not.

Odd that the time reported for the video & audio is so far off - audio sync appears to be working.

/junk/clifford@nt650% time tytompg/tytompg -i \{Clifford\ the\ Big\ Red\ Dog\}\{2000-09-04\}\{\}\{09.00\ AM\ Wed\ Nov\ 01,\ 2006\}\{WTVS\}.ty -o test.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.9, Demuxer version 0.19
Source is {Clifford the Big Red Dog}{2000-09-04}{}{09.00 AM Wed Nov 01, 2006}{WTVS}.ty
Video frame size is 352x480 Frame rate 29.970030
Video bit rate is 1470000 bits/sec., 1470 Kbit/sec.
MPEG2 audio at 32kHz, 192 kbps, 864 bytes/sync frame
MPEG2 audio at 32kHz, 192 kbps, 864 bytes/sync frame
Done!
Stream audio: 50033 frames (1350 seconds aprox.)
Stream video: 53990 frames, 107980 fields (1801 seconds aprox.)
tytompg/tytompg -i -o test.mpg 25.96s user 11.50s system 75% cpu 49.398 total

danpritts
11-27-2006, 03:16 AM
I just tried using tytompg to convert a file encoded on a series1 tivo.

used version 0.9 on linux.

No errors, but the video is really choppy when played in vlc, as if some of the intermediate frames are being lost. The original ty plays fine in vlc.


Whoops, I see this was already reported early on, although i get lost in a bunch of source release flamage so i'm not sure if you thought you'd fixed it.

danpritts
11-27-2006, 03:20 AM
...and now that i test it with other players, mplayer looks basically OK although there are interlacing artifacts; quicktime player gives an error ("the movie could not be opened").

I'll skip posting the files unless you tell me toherwise.

gyre
11-27-2006, 04:37 AM
I've got a series 1 PAL clip that has the following properties:

* .ty file plays happily under vlc
* converted file's sound plays fine but video is jerky with vlc, wmp, etc.
* sound and video play fine after passing through videoredo.
* sound and video play fine after passing through tmpgenc dvd author.

No obligation to look at, but if you're bored, it can be found here (http://gyre.is-a-geek.org/gyre/skippy.ty) (32MB).

Thx.

-- gyre --

bcc
11-27-2006, 04:45 AM
Hi guys, I'm having a problem converting a PAL ty file using tytompg. When playing back the converted file, the audio is fine but the video stutters. I was wondering if this somehow relates to the fact that PAL has 25.0 fps whereas NTSC has 29.970 fps?I don't have the frame rate (or resolution) hard-coded anywhere. I also don't overload 25fps to mean unintialized like in ffmpeg. So I don't know why PAL would make a difference.

I'm also getting hundreds of warnings stating "Decoder buffer empty".Hmm, I did reproduce this by going from PAL .mpg to .ty with my ffmpeg then back to .mpg with tytompg. (Sick huh?) Will dig further. The final .mpg plays back fine however.

bcc
11-27-2006, 04:51 AM
If you find a GOP longer than 15 (PAL) do you break it into pieces? I'm currently using videoredo for this purpose.I purposely don't try to rewrite the GOP, instead I leave it in its original state. So it may for example be 4, 15, 45 frames. Whatever it was broadcasted with. Internally I have to chop up the GOP so that I can simply avoid unbounded buffer lengths.
Thanks for an awesome program. It even works on the crap that my S1 UK tivo spits out :)

-- gyre --Cool; interesting that you're not seeing the warnings TivoZA is.

gyre
11-27-2006, 05:16 AM
I'm seeing some warnings, but since the resulting file after being run through videoredo seems to be fine, I'm not too fussed. :)

I'm just praying that videoredo doesn't mess up the audio/video sync. On the couple of things I've tried it with, it didn't seem to. Small sample, I know.

-- gyre --

TivoZA
11-27-2006, 08:39 AM
Firstly a big THANK YOU to bcc for all the effort he has/is putting into developing and supporting this. It's a great little app, I hope we're not causing bcc too much of a headache :p


I'm seeing some warnings
...
Small sample, I know.When running tytompg on gyre's "skippy.ty (http://gyre.is-a-geek.org/gyre/skippy.ty)" file, I only get one "Decoder buffer empty" warning. Since "skippy.ty" is only 32MB and the original file I was testing was 7GB, this should explain the difference.

As for the stuttering, I still get that when running tytompg on the "skippy.ty" file. It's most noticeable during the channel logo and castle sequences. There is no stuttering when playing the "skippy.ty" via TyShow codec in Windows Media Player.

Not sure if this helps but when running the converted "skippy.mpg" through ffmpeg, I get the following errors.

ffmpeg -i skippy.mpg -vcodec copy -acodec copy skippy2.mpg
...
error, non monotone timestamps 8068 >= -2732
error, non monotone timestamps 8068 >= 868
error, non monotone timestamps 8068 >= 4468
error, non monotone timestamps 69275 >= 58476
error, non monotone timestamps 69275 >= 62076

ffmpeg -i skippy.mpg -vcodec mpeg2video -acodec copy skippy3.mpg
[mpeg2video @ 0x6a4458]end mismatch left=62880 1518B0= 845.6kbits/s
[mpeg2video @ 0x6a4458]00 motion_type at 0 11
[mpeg2video @ 0x6a4458]00 motion_type at 0 12
[mpeg2video @ 0x6a4458]00 motion_type at 0 13
[mpeg2video @ 0x6a4458]00 motion_type at 0 14
[mpeg2video @ 0x6a4458]00 motion_type at 0 15
[mpeg2video @ 0x6a4458]00 motion_type at 0 16
[mpeg2video @ 0x6a4458]00 motion_type at 0 17
[mpeg2video @ 0x6a4458]00 motion_type at 0 18
[mpeg2video @ 0x6a4458]00 motion_type at 5 19
[mpeg2video @ 0x6a4458]00 motion_type at 15 20
[mpeg2video @ 0x6a4458]00 motion_type at 1 22
[mpeg2video @ 0x6a4458]mb incr damaged
[mpeg2video @ 0x6a4458]00 motion_type at 0 23
[mpeg2video @ 0x6a4458]00 motion_type at 11 24
[mpeg2video @ 0x6a4458]00 motion_type at 0 25
[mpeg2video @ 0x6a4458]ac-tex damaged at 13 26
[mpeg2video @ 0x6a4458]00 motion_type at 0 27
[mpeg2video @ 0x6a4458]00 motion_type at 0 28
[mpeg2video @ 0x6a4458]00 motion_type at 1 29
[mpeg2video @ 0x6a4458]00 motion_type at 7 30
[mpeg2video @ 0x6a4458]00 motion_type at 6 31
[mpeg2video @ 0x6a4458]00 motion_type at 5 32
[mpeg2video @ 0x6a4458]00 motion_type at 0 33
[mpeg2video @ 0x6a4458]00 motion_type at 0 34
[mpeg2video @ 0x6a4458]00 motion_type at 1 35
[mpeg2video @ 0x6a4458]Warning MVs not available
[mpeg2video @ 0x6a4458]concealing 550 DC, 550 AC, 550 MV errors

FS1
11-27-2006, 10:08 AM
tytompg gets 240MB into a 2.71GB file and then does this:

Clip #6 - 15MB (http://x/ad-chunk.ty)

tytompg -i ad-chunk.ty -o ad-chunk.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.9, Demuxer version 0.19
Source is ad-chunk.ty
AC3 audio at 48kHz, 384 kbps, 1536 bytes/sync frame
Video frame size is 1280x1088 - high definition. Frame rate 29.970030
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
Warning: Skipping chunk 12 due to timestamp discontinuity
Warning: Skipping chunk 13 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity, at chunk 15
Warning: Skipping chunk 37 due to timestamp discontinuity
Warning: Skipping chunk 38 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity, at chunk 40
Warning: temporal_reference order problem. Value: 2 group_start: 54 last: 59
Warning: Skipping chunk 62 due to timestamp discontinuity
Warning: Skipping chunk 63 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity, at chunk 65
Warning: temporal_reference order problem. Value: 2 group_start: 107 last: 113
Warning: Skipping chunk 85 due to timestamp discontinuity
Warning: Skipping chunk 86 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity, at chunk 88
Warning: temporal_reference order problem. Value: 2 group_start: 158 last: 166
tytompg: mplex.c:1876: video_parse_gop: Assertion `stream->state_machine == PARSE_CODING_EXT' failed.
Aborted

This is from a series of 18 other similar extractions that worked just fine.

Also the resulting mpeg does not play correctly. There is some serious macroblocking that I guess is from dropped frames (moving macroblocks - like there was a bad I frame).

bcc
11-27-2006, 02:08 PM
Thanks everyone on the kind words. When I suddenly get more bug reports than the number of bugs I fixed it has to make me wonder if some regressions slipped in, but it's good to know that apparently it's just that testing is up.

bcc
11-27-2006, 02:12 PM
External USB drives are so cheap now that I will archive the full-size files and just keep buying drives. The 800 gigabytes in my TiVo became too small very quickly.In fact external SATA is is real cheap now too, at least if you assemble your own. ESATA is also faster, and avoids all the bridge board firmware issues of usb/firewire enclosures.

bcc
11-27-2006, 03:56 PM
That one isn't on the tivo anymore, so I can't play it and see what happens. I don't remember it being special in any way, though. A 25 second hole at the head is OK with me. That's why I pad my recordings. :)Well I inserted this one into my old hr10-250, and it plays with 25 seconds of audio and just a static tivo background screen, followed by ~10 seconds of cdusa audio+video.

Here's another problem clip, but I think this was caused by bad reception. Ideally tytompg should do its best and keep going?Well ideal might be to drop frames that are identifiably corrupt. In your clip4 case the video has a record with an illegal frame coding_type, probably due to reception issues. I don't need to assert here, and things look OK when I don't.

In your clip5 case, the video has a record with an illegal video frame rate. Same answer as for clip4 :)

bcc
11-27-2006, 04:05 PM
Got this while trying to do a Lost OTA Stream...That's the same as fs1's clip#4 above. Fix coming.

bcc
11-27-2006, 05:13 PM
tytompg gets 240MB into a 2.71GB file and then does this:In this case hdemux's sanity checker of tivo's timestamps was mis-firing. Looks like the low-data rate of that grainy letterboxed video tricked it. Fixed.
Also the resulting mpeg does not play correctly. There is some serious macroblocking that I guess is from dropped frames (moving macroblocks - like there was a bad I frame).The demuxer was dropping chunks trying to recover, which gives you dropped a/v frames.

bcc
11-27-2006, 05:39 PM
Whoops, I see this was already reported early on, Yes, I thought vlc was happy as of version 0.6, per:
http://www.dealdatabase.com/forum/showthread.php?p=269642&highlight=vlc#post269642

although i get lost in a bunch of source release flamage so i'm not sure if you thought you'd fixed it.That's largely why I tried to encourage folks to take it to a new thread

quicktime player gives an error ("the movie could not be opened"). Probably you have no mpeg2 codec installed for it. If folks want to test with quicktime or windows media player, any troubleshooting involves figuring out exactly what codec you're using at the moment.

drez
11-27-2006, 05:53 PM
I have some feature suggestions but first off, I have to thank you for all your hard work with ty's (hdemux, ty-ffmpeg, tytompg....) ... I really appreciate tytompg's speed and convenience.


Could you please have tytompg print the offset value when its done like hdemux use to print out?



I use the ms offset value hdemux gives for the Audio Skew Correction feature in VirtualDub-MPEG2 (http://fcchandler.home.comcast.net/stable/) (Audio > Interleave > Audio skew correction). Without that value, the video is out of sync in VirtualDub. It's easier to edit out commericials or save just one scene in xvid by opening the mpg's in VirtualDub-MPEG2.


Also, would it be possible to have tytompg accept drag-and-dropped files (multiple files, if possible? processed one at a time and if one fails, rename to .mpg.failed and continue onto the next file.... I might be asking for too much) and output the mpg's in the same folder as the .ty and with the same filename as the original (for example, fox.ty becomes fox.mpg)?

snoots
11-27-2006, 09:06 PM
I have been having quite a bit of trouble trying to get HD tyfiles to convert using the tool. Can anyone confirm they are using files from 6.3a? I am using tytool to download the files. I have been able to successfully convert only SD files, no HD ones so far. Another variable is I am running on Vista so that may be part of my trouble. Input welcome, Thanks

RavenStL
11-27-2006, 09:31 PM
I have FINALLY got a FULL FOX-OTA HD stream from 6.3 (NOT 6.3a) to complete and be able to be processed by TMPGEnc!!! THANK YOU BCC!

The audio drop outs are just that in the final product, audio ONLY dropouts (NICE!) THANKS!

Now I am going to do some more FOX OTA and see if other streams cause new errors.

I also am asking about the offset that HDEMUX use to supply. I had to add approximately a 200 ms audio offset when I Encoded my tytompg .mpg to a dvd compliant MPEG2. (HD to SD). So it does not appear that tytompg fixes the audio / video timing 100% (It could of been off more, I dunno) It does however maintain the 200 ms offset throughout, so a 200 ms offset is easy to add to TMPGEnc.

Thanks again!

94SupraTT
11-27-2006, 10:00 PM
I have FINALLY got a FULL FOX-OTA HD stream from 6.3 (NOT 6.3a) to complete and be able to be processed by TMPGEnc!!! THANK YOU BCC!

The audio drop outs are just that in the final product, audio ONLY dropouts (NICE!) THANKS!

Now I am going to do some more FOX OTA and see if other streams cause new errors.

I also am asking about the offset that HDEMUX use to supply. I had to add approximately a 200 ms audio offset when I Encoded my tytompg .mpg to a dvd compliant MPEG2. (HD to SD). So it does not appear that tytompg fixes the audio / video timing 100% (It could of been off more, I dunno) It does however maintain the 200 ms offset throughout, so a 200 ms offset is easy to add to TMPGEnc.

Thanks again!

I haven't been able to get a FOX OTA stream to work. I can verify the 200ms offset issue though. I have about a 180ms offset on a NBC OTA game that I did. Unfortunately I still can't get a FOX OTA HD football game to work. ESPN OTA and NBC OTA seems fine.

This is what I got with the last FOX OTA game.




Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Light>e:

E:\>tytompg -i Bear.ty -o BearsJets.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.9, Demuxer version 0.19
Source is Bear.ty
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
AC3 audio at 48kHz, 448 kbps, 1792 bytes/sync frame
video: Warning: Decoder buffer empty, scr ahead by 2712 (9.12) 0.9
... and encoder buffer full
While processing stream audio...
Assertion failed: !"Encoder buffer full", file mplex.c, line 478

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

E:\>

HuMan321
11-27-2006, 10:02 PM
Deleted until testing is complete

cheer
11-28-2006, 12:27 AM
I have been having quite a bit of trouble trying to get HD tyfiles to convert using the tool. Can anyone confirm they are using files from 6.3a? I am using tytool to download the files. I have been able to successfully convert only SD files, no HD ones so far. Another variable is I am running on Vista so that may be part of my trouble. Input welcome, Thanks
Confirmed -- I am successfully using this tool w/6.3a HD streams. But I'm not running Vista, for what it's worth, and I'm downloading via mfs_ftp (but that shouldn't matter).

bcc
11-28-2006, 01:21 AM
Here's a new version (attached to first post), which fixes several problem cases some of you hit: Don't assert for illegal video stream conditions: bad frame rate codes or bad coding_type
Improve tivo timestamp sanity checker
fix video stream underrun processing (video holes)

FS1
11-28-2006, 01:32 AM
I have some feature suggestions...

Also, would it be possible to have tytompg accept drag-and-dropped files (multiple files, if possible? processed one at a time and if one fails, rename to .mpg.failed and continue onto the next file.... I might be asking for too much) and output the mpg's in the same folder as the .ty and with the same filename as the original (for example, fox.ty becomes fox.mpg)?You could do this with the windows script host and some quick vbscript hacking. Prior to tytompg, that was how I used hdemux. I should be able to modify that code easily, so I'll try and cook something up later.

Really what I need to make myself is a bash script to do all the .ty files in a folder.

And while we're throwing out feature requests, how about patching 1088 streams to 1080? :D
http://www.highdefinition.ch/fix_1088_to_1080.html

bcc
11-28-2006, 01:43 AM
I have some feature suggestions but first off, I have to thank you for all your hard work with ty's (hdemux, ty-ffmpeg, tytompg....) ... I really appreciate tytompg's speed and convenience.You forgot tymplex and ty1to2 :) :)

Could you please have tytompg print the offset value when its done like hdemux use to print out? It already does if you add a -dI to the command line. It will add a little more diagnostic info to the output, including something like: "Multiplexor: Offset: 692 Low-def: FALSE", where the offset field is the number you're looking for. Units are milliseconds, and value may be positive or negative (same sense and units as -O option for mplex).

However, if you're finding a need for this with the .mpg, then I think there's something wrong with how you're doing the post-processing. The .mpg already includes the equivalent a/v offset information in the program stream packets I'm generating. For example, in the avisynth example I posted, dgindex is able to find the a/v offset from the .mpg, allowing perfect a/v sync without this extra info being manually specified.

Also, would it be possible to have tytompg accept drag-and-dropped files (multiple files, if possible? processed one at a time and if one fails, rename to .mpg.failed and continue onto the next file.... I might be asking for too much) and output the mpg's in the same folder as the .ty and with the same filename as the original (for example, fox.ty becomes fox.mpg)?Since I haven't implemented a GUI I'm not sure what you mean :) Probably you're only considering windows? Perhaps a windows batch script is in order for now.

bcc
11-28-2006, 01:49 AM
Odd that the time reported for the video & audio is so far off - audio sync appears to be working.
You mean the approximate times I report for the streams when processing is done? That was just a first blush estimate which doesn't take into account variations in the audio frame rate (such as when you go between the feature and some commercial). It also doesn't take into account variable video frame rates or even video pulldown. Reporting the stream times based upon the timestamps is likely better, on the other hand that'll fail to take into account audio or video holes.

bcc
11-28-2006, 01:54 AM
Not sure if this helps but when running the converted "skippy.mpg" through ffmpeg, I get the following errors.I haven't been able to download that yet, (so slow the connection times out after about 1MB).

As for the ffmpeg "non monotone" errors, I tried to explain that here:
http://www.dealdatabase.com/forum/showpost.php?p=269357&postcount=179
The workaround should apply here as well.

gyre
11-28-2006, 02:28 AM
I can upload skippy.ty anywhere of your choice. At the moment it is hosted on a web server on my very ropey adsl connection at home :)

-- gyre --

bcc
11-28-2006, 03:30 AM
I can upload skippy.ty anywhere of your choice. At the moment it is hosted on a web server on my very ropey adsl connection at home :)

-- gyre --Thanks, I was able to download enough of it after a couple tries. I can see vlc is stuttering the video a bit. Will look into it. (Audio seems fine). The whole thing plays smoothly with xine and powerdvd. windvd stutters the video like vlc...

drez
11-28-2006, 07:16 AM
Also, would it be possible to have tytompg accept drag-and-dropped files (multiple files, if possible? processed one at a time and if one fails, rename to .mpg.failed and continue onto the next file.... I might be asking for too much) and output the mpg's in the same folder as the .ty and with the same filename as the original (for example, fox.ty becomes fox.mpg)?
Since I haven't implemented a GUI I'm not sure what you mean :) Probably you're only considering windows? Perhaps a windows batch script is in order for now.


I was thinking of how this small Windows console program (28kb, try it) works: http://www.pc-tools.net/win32/md5sums/

It doesn't have a GUI either but you can drag multiple files into md5sums.exe in a Explorer window... Too bad there's no source for it (I'm sure the author could send you some helpful information on how he did that in an e-mail)

drez
11-28-2006, 07:58 AM
Could you please have tytompg print the offset value when its done like hdemux use to print out?



I use the ms offset value hdemux gives for the Audio Skew Correction feature in VirtualDub-MPEG2 (http://fcchandler.home.comcast.net/stable/) (Audio > Interleave > Audio skew correction). Without that value, the video is out of sync in VirtualDub. It's easier to edit out commericials or save just one scene in xvid by opening the mpg's in VirtualDub-MPEG2.



It already does if you add a -dI to the command line. It will add a little more diagnostic info to the output, including something like: "Multiplexor: Offset: 692 Low-def: FALSE", where the offset field is the number you're looking for. Units are milliseconds, and value may be positive or negative (same sense and units as -O option for mplex).

However, if you're finding a need for this with the .mpg, then I think there's something wrong with how you're doing the post-processing. The .mpg already includes the equivalent a/v offset information in the program stream packets I'm generating. For example, in the avisynth example I posted, dgindex is able to find the a/v offset from the .mpg, allowing perfect a/v sync without this extra info being manually specified.


The MPEG-2 support was tacked onto VirtualDub in VirtualDub-MPEG2 (it doesn't even recognize the mpg's 4:3 aspect ratio) so it's probably VirtualDub-MPEG2's fault...

Once I set that ms value, VirtualDub-MPEG2 works flawlessly. tytompg/hdemux give a positive offset value but I have to enter it as a negative value in VDub-MPEG2 for sync...

I'm pretty sure it's because I used an -s value to start processing in the middle of the .ty. When I play the .mpg in any player, theres a still frame for around a second and a half (same time as the offset) and then it catches up.

I find editing (removing scenes/commericials) much easier and more precise in VirtualDub than in any other program I've ever tried before.

Adding a deinterlace filter and resize filter (to 640x480, to force the aspect ratio in the output) is also very, very easy in VirtualDub.

94SupraTT
11-28-2006, 10:21 PM
New error...



E:\>tytompg -i Lost301.ty -o Lost301.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.10, Demuxer version 0.20
Source is Lost301.ty
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 18900000 bits/sec., 18900 Kbit/sec.
AC3 audio at 48kHz, 192 kbps, 768 bytes/sync frame
Warning: temporal_reference order problem. Value: 5 group_start: 20070 last: 20
093
Warning: Video sequence has illegal frame rate code: 15
Assertion failed: stream->state_machine == PARSE_SLICES_AND_HDR, file mplex.c, l
ine 1820

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

94SupraTT
11-28-2006, 11:14 PM
E:\>tychopper Lost301.ty 5200 5250 Lost5200to5250.ty
Copying 51 chunks from Lost301.ty to Lost5200to5250.ty
0 Chunks copied
E:\>tytompg -i Lost5200to5250.ty -o Lost5200to5250.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.10, Demuxer version 0.20
Source is Lost5200to5250.ty
AC3 audio at 48kHz, 192 kbps, 768 bytes/sync frame
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 18900000 bits/sec., 18900 Kbit/sec.
Warning: temporal_reference order problem. Value: 5 group_start: 120 last: 143
Warning: Video sequence has illegal frame rate code: 15
Assertion failed: stream->state_machine == PARSE_SLICES_AND_HDR, file mplex.c, l
ine 1820

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.



Here is 50 chunks. Failure point is between 5225 and 5250.

http://www.sendspace.com/file/xe5cxe

FS1
11-29-2006, 01:17 AM
This looks like it may be the same as above, but since I already have it cut up here it is. I found it late last night, I just didn't get around to posting.

Clip #7 - 15MB (http://x/lost2-chunk.ty)


tytompg -i lost2-chunk.ty -o lost2-chunk.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.10, Demuxer version 0.20
Source is lost2-chunk.ty
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 80000000 bits/sec., 80000 Kbit/sec.
AC3 audio at 48kHz, 384 kbps, 1536 bytes/sync frame
Warning: temporal_reference order problem. Value: 2 group_start: 142 last: 149
audio: Warning: Decoder buffer empty, scr ahead by 647220 (2157.120) 0.2157
Warning: temporal_reference order problem. Value: 14 group_start: 501 last: 518
tytompg: mplex.c:1820: video_parse_gop: Assertion `stream->state_machine == PARSE_SLICES_AND_HDR' failed.
Aborted

TivoZA
11-29-2006, 06:14 PM
It already does if you add a -dI to the command line. It will add a little more diagnostic info to the output, including something like: "Multiplexor: Offset: 692 Low-def: FALSE", where the offset field is the number you're looking for. Units are milliseconds, and value may be positive or negative (same sense and units as -O option for mplex).It might also be worth adding some progress information. Something similar to ffmpeg whereby the total frames are listed along with the other video information and the frames processed (i.e. progress) is continually updated on the last line.

bcc
11-29-2006, 08:14 PM
Ok, I've uploaded a new version. The only change with this new version is to allow video GOPs to be malformed. I spit out some warning messages for this case. This allows tytotmpg to keep chugging away on recordings that have reception glitches. The DTS timestamps I generate may be slightly off in any GOPs that are malformed. If you plan to re-author the resulting .mpg (not sure why you'd want to permanently archive recordings with errors like this but...), I'm almost certain you'll have problems if you don't run the .mpg thru some sort of fixer first.

This addresses the latest samples from fs1&94supratt. Do you guys have your DTV antennas in the basement or something? :)

94SupraTT
11-29-2006, 08:52 PM
Ok, I've uploaded a new version. The only change with this new version is to allow video GOPs to be malformed. I spit out some warning messages for this case. This allows tytotmpg to keep chugging away on recordings that have reception glitches. The DTS timestamps I generate may be slightly off in any GOPs that are malformed. If you plan to re-author the resulting .mpg (not sure why you'd want to permanently archive recordings with errors like this but...), I'm almost certain you'll have problems if you don't run the .mpg thru some sort of fixer first.

This addresses the latest samples from fs1&94supratt. Do you guys have your DTV antennas in the basement or something? :)


:D

I have an attic mount antenna. I'm locked in at 94% signal strength though.

FS1
11-30-2006, 12:03 AM
Ok, I've uploaded a new version. The only change with this new version is to allow video GOPs to be malformed. I spit out some warning messages for this case. This allows tytotmpg to keep chugging away on recordings that have reception glitches. The DTS timestamps I generate may be slightly off in any GOPs that are malformed. If you plan to re-author the resulting .mpg (not sure why you'd want to permanently archive recordings with errors like this but...), I'm almost certain you'll have problems if you don't run the .mpg thru some sort of fixer first.

This addresses the latest samples from fs1&94supratt. Do you guys have your DTV antennas in the basement or something? :)It's in the attic with at least 85% strength on all channels. :)

Both tytompg and videoredo seem to give warnings just as much with satellite stuff as with OTA (maybe not the malformed GOP error). I have some Arrested Development off HDNet that varies wildly. They all play fine on the tivo, but some give a ton of warnings and drop a lot of frames and some don't. I haven't seen a pattern. I'll test with 0.11 and see if there are still any good examples.

Oh, and a quick feature request: Can you check that the input and output files are different before you clobber the input file? Bash tab-completion and my stupidity both contributed to me hosing a few extractions. I have three nice zero-length files now. :(

mchahn
11-30-2006, 03:08 AM
I have been having quite a bit of trouble trying to get HD tyfiles to convert using the tool. Can anyone confirm they are using files from 6.3a? I am using tytool to download the files. I have been able to successfully convert only SD files, no HD ones so far. Another variable is I am running on Vista so that may be part of my trouble. Input welcome, Thanks

I'm using 6.3 with no problem. I download using tytools. I am only downloading HD movies. I only play on VLC.

mchahn
11-30-2006, 03:11 AM
Is there any chance you could add some kind of progress indicator? Some dots would be fine, percentage would be awesome.

When it sits there for a long time I have no way of telling if it is running. I know I should probably trust my computer more, but it hasn't really earned my trust. (No computer I've used over the last 40 years has <grin>).

Rowan
11-30-2006, 09:06 AM
When it sits there for a long time I have no way of telling if it is running.

Until BCC can add some program status you can at least see if the program/process is using CPU thus implying it is running (unless in endless loop).

Windows: startup the task manager and see if that process is using any of the CPU.

Linux: use top to see if the process is using any of the CPU.

mchahn
11-30-2006, 08:24 PM
I've remuxed around 100 files using TyToMpg and I have had only two files not work. The two failed files each spit out the error messages below a zillion times, although it only took a few minutes to finish. The resulting file was only a fraction of a gigabyte while the original files were over five gigs. The bad files had no video or audio when played:



video: Warning: Dropping frame -- too old by 713119103452 (2377063678.52) 26411.73678
video: Warning: Dropping frame -- too old by 713121355552 (2377071185.52) 26411.81185
video: Warning: Dropping frame -- too old by 713120454652 (2377068182.52) 26411.78182
video: Warning: Dropping frame -- too old by 713115950152 (2377053167.52) 26411.63167
Done!
Stream audio: 175581 frames (5618 seconds aprox.)
Stream video: 139385 frames, 336781 fields (4650 seconds aprox.)


The files were each HDTV movies from my hr10-250. I realize that 98% is a great result, but as every free-loading user I want 100% <grin>. Do you want me to send you a small sample of the file?

FS1
11-30-2006, 08:37 PM
I've remuxed around 100 files using TyToMpg and I have had only two files not work. The two failed files each spit out the error messages below a zillion times, although it only took a few minutes to finish. The resulting file was only a fraction of a gigabyte while the original files were over five gigs. The bad files had no video or audio when played:



The files were each HDTV movies from my hr10-250. I realize that 98% is a great result, but as every free-loading user I want 100% <grin>. Do you want me to send you a small sample of the file?Sounds like a good test case. From you description, I think I have two files with the same problem. One starts giving out warnings and dropping frames about halfway through, and the other does it from the start.

bcc
11-30-2006, 09:03 PM
The files were each HDTV movies from my hr10-250. I realize that 98% is a great result, but as every free-loading user I want 100% <grin>. Do you want me to send you a small sample of the file?Sure, I'd be interested to see what's going wrong. I may not get a chance to fix anything for about a week however (travel and such). The failures were from directv not OTA?

bcc
11-30-2006, 09:09 PM
Is there any chance you could add some kind of progress indicator? Some dots would be fine, percentage would be awesome.

When it sits there for a long time I have no way of telling if it is running. I know I should probably trust my computer more, but it hasn't really earned my trust. (No computer I've used over the last 40 years has <grin>).Been trying to get correctness down first, but yes I should be able to do something of the sort (and the output file protection as well). Recommendations for a good, free, portable (linux+windows+mac), non-copyleft gui library, preferably in C?

Avon
12-01-2006, 07:06 PM
Any thoughts as to how I can get around this error?

tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.11, Demuxer version 0.20
Source is in.ty
Video frame size is 480x480 Frame rate 29.970030
Video bit rate is 15000000 bits/sec., 15000 Kbit/sec.
MPEG2 audio at 48kHz, 160 kbps, 480 bytes/sync frame
MPEG2 audio at 48kHz, 160 kbps, 480 bytes/sync frame
Assertion failed: id != -1, file mplex.c, line 2103

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

bcc
12-01-2006, 08:03 PM
Any thoughts as to how I can get around this error?Looks like your audio stream probably has some erorrs in it. Do you hit the error right away or only after some of the content has been converted? Adding a '-da' to turn on audio debugging may help provide info. Can you demux the .ty with hdemux?

millercentral
12-01-2006, 08:51 PM
I've remuxed around 100 files using TyToMpg and I have had only two files not work. The two failed files each spit out the error messages below a zillion times, although it only took a few minutes to finish. The resulting file was only a fraction of a gigabyte while the original files were over five gigs. The bad files had no video or audio when played:



The files were each HDTV movies from my hr10-250. I realize that 98% is a great result, but as every free-loading user I want 100% <grin>. Do you want me to send you a small sample of the file?

I too have hit this error on DirectTV HD (not OTA) material. Same exact scenario, eveythings going fine for about half the file, then billions of "too old" error message. Let me know if you need another sample.

mchahn
12-01-2006, 08:58 PM
Sure, I'd be interested to see what's going wrong. I may not get a chance to fix anything for about a week however (travel and such). The failures were from directv not OTA?

The were all from satellite. I'm lucky enough to live in LA area so I don't need OTA.

I'll chop a small piece off that shows the problem (1 megabyte OK?). No rush though. It will take me a while to watch 98 movies.

Re: GUI. I don't know if it meets your specs or not, but I like wxWindows a lot.

mchahn
12-02-2006, 03:18 AM
I split off the first megabyte and it shows the same symptoms. Here is the file.

I noticed that the output of your program run on this piece did have good video. Hopefully the fact that it spit out a lot of the same error messages means you will be able to see the problem.

tlphipps
12-02-2006, 09:21 AM
Looks like your audio stream probably has some erorrs in it. Do you hit the error right away or only after some of the content has been converted? Adding a '-da' to turn on audio debugging may help provide info. Can you demux the .ty with hdemux?

I actually ran into this same problem (Assertion failed: id != -1, file mplex.c, line 2103) yesterday. Here's the end of the output:

Warning: temporal_reference order problem. Value: 1 group_start: 50019 last: 51
042
Recovered ac3 sync at 177 instead of 115, chunk 26465
ac3 syncword skew: Deleting 62
Assertion failed: id != -1, file mplex.c, line 2103

There are LOTS of the temporal_reference warnings as it processes. The output file is just slightly smaller than the original ty, but it doesn't seem to play (no audio OR video).

Also, I was able to demux the .ty with hdemux without any errors.

burdellgp
12-03-2006, 10:55 PM
I'm trying to convert a movie from TNT-HD. tytompg runs for a bit, then I get errors:

tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.11, Demuxer version 0.20
Source is tnthd-movie.ty
Warning: Tossing AC3 audio frame with missing syncword. Chunk 1 length 1809
AC3 audio at 48kHz, 448 kbps, 1792 bytes/sync frame
Video frame size is 1280x1088 - high definition. Frame rate 29.970030
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
Warning: Tossing AC3 audio frame with missing syncword. Chunk 1 length 1809
Warning: Warping clock by 4295828822 (14319429.122) 159.9429
Warning: Warping clock by 45495412 (151651.112) 1.61651
video: Warning: Decoder buffer empty, scr ahead by 19846814 (66156.14) 0.66156
Warning: Warping clock by 1186259198568 (3954197328.168) 43935.47328
video: Warning: Decoder buffer empty, scr ahead by 1186233553182 (3954111843.282) 43934.51843
... and encoder buffer full
audio: Warning: Dropping frame -- too old by 1186295225532 (3954317418.132) 43936.77418


After that, I get tons of the "Dropping frame" (both audio and video) messages. Eventually, I get:

video: Warning: Dropping frame -- too old by 1038509383632 (3461697945.132) 38463.27945
video: Warning: Dropping frame -- too old by 1038504879132 (3461682930.132) 38463.12930
Done!
Stream audio: 195463 frames (7297 seconds aprox.)
Stream video: 162260 frames, 374870 fields (5414 seconds aprox.)

The resulting mpeg file doesn't play all the way through.

shortkud
12-04-2006, 07:14 AM
This is great it handles a bunch of files that hdemux + mplex or ffmpeg would not do. I did have a problem with a bigger file though around 18gb. It converted but I could not play it in VLC,WMP, or MPlayer.

cheer
12-04-2006, 09:45 AM
Just a note to say that I am really having great success with tytompg -- it's handling two troublesome shows (ER and House) that hdemux/mplex rarely handled. It's worked well enough that I've integrated it into my EtiVo process that I use for my son the truck driver.

This latest version, in particular, seems to handle everything I throw at it. Way cool.

tlphipps
12-04-2006, 10:44 AM
It's worked well enough that I've integrated it into my EtiVo process that I use for my son the truck driver.

I'd love to see your etivo profile for this. I'm assuming you're still using Bluewomble's divx addin?

94SupraTT
12-04-2006, 12:40 PM
Just a note to say that I am really having great success with tytompg -- it's handling two troublesome shows (ER and House) that hdemux/mplex rarely handled. It's worked well enough that I've integrated it into my EtiVo process that I use for my son the truck driver.

This latest version, in particular, seems to handle everything I throw at it. Way cool.

I agree. I've been having problems with Lost and NFL Sunday Ticket games but this latest version of tytompg has handled both very well.

cheer
12-04-2006, 12:54 PM
I'd love to see your etivo profile for this. I'm assuming you're still using Bluewomble's divx addin?

Yes, and I've rewritten my dumb little fakevsplit. Testing it now with a batch of HD vids. If it works I'll clean it up and post it (along with my profiles) in the Divx add-in thread. But so far it's working great -- and not only is it more reliable, but it's faster since I don't have to do the demux/remux separately.

bcc
12-05-2006, 09:05 PM
I've remuxed around 100 files using TyToMpg and I have had only two files not work. The two failed files each spit out the error messages below a zillion times, although it only took a few minutes to finish.Ok, I'm back in town and took a look at this one. This recording has a bad PES header which is resulting in a corrupt PTS timestamp that throws everything off. I've done some extra sanity checking on the PES header which detects&recovers from this case. New version posted: 0.12

Other change in this new version: Included support for 64 bit linux, and also I threw in updated hdemux binaries (64 bit linux, 32 bit linux, 32 bit windows)

Nandy
12-06-2006, 07:34 PM
Please, forgive my ignorance. I am trying to see what is the advantage to use tytompg over the tytool. Looks like they do the same. Is that right? I have been looking for a ty to mp4 converter. Is this program goin to do something like that in the future?

Thanks!

PS- I used it and work great, just trying to see what will be the benefit. It is good to have anyway.

FS1
12-06-2006, 09:57 PM
Thanks for the x86-64 build.

I was down to only two extractions with problems. One of them is taken care of with 0.12, but the other one gets about halfway through a 6GB file and then drops the rest of the video. hdemux works fine.

Clip #7 - 15MB (http://x/lost3-chunk.ty)

tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.12, Demuxer version 0.20
Source is lost3-chunk.ty
AC3 audio at 48kHz, 384 kbps, 1536 bytes/sync frame
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 80000000 bits/sec., 80000 Kbit/sec.
Changing video frame rate to 29.970030 from 59.940060
Changing delay
Warning: Warping clock by 404193648646 (1347312162.46) 14970.12162
video: Warning: Decoder buffer empty, scr ahead by 404168003016 (1347226676.216) 14969.16676
audio: Warning: Dropping frame -- too old by 404178198066 (1347260660.66) 14969.50660
audio: Warning: Dropping frame -- too old by 404178198066 (1347260660.66) 14969.50660
audio: Warning: Dropping frame -- too old by 404178198066 (1347260660.66) 14969.50660
audio: Warning: Dropping frame -- too old by 404178198066 (1347260660.66) 14969.50660
audio: Warning: Dropping frame -- too old by 404174742066 (1347249140.66) 14969.39140
audio: Warning: Dropping frame -- too old by 404174742066 (1347249140.66) 14969.39140
audio: Warning: Dropping frame -- too old by 404174742066 (1347249140.66) 14969.39140
audio: Warning: Dropping frame -- too old by 404174742066 (1347249140.66) 14969.39140
audio: Warning: Dropping frame -- too old by 404171286066 (1347237620.66) 14969.27620
audio: Warning: Dropping frame -- too old by 404171286066 (1347237620.66) 14969.27620
Changing video frame rate to 59.940060 from 29.970030
Changing delay
audio: Warning: Dropping frame -- too old by 404171286066 (1347237620.66) 14969.27620
audio: Warning: Dropping frame -- too old by 404171286066 (1347237620.66) 14969.27620
<snip>


No comment on integrating the 1088i header patch? I realize it's not required functionality for a tystream->mpeg remuxer, but it's something that should be fixed eventually. A lot of encoders don't pad the bottom 8 lines with black (see ATSC A/53 page 28 (http://www.atsc.org/standards/a_53e-with-Amend-1-and-2.pdf)), so you get a gray bar at the bottom. Very annoying for playback on a 4:3 monitor. I wouldn't want the patching to be done by default, but just a switch to enable it. Failing that, piping the output into another program would work. I haven't tried it, but I've assumed tytompg doesn't output to stdout?

FS1
12-06-2006, 10:01 PM
Please, forgive my ignorance. I am trying to see what is the advantage to use tytompg over the tytool. Looks like they do the same. Is that right? tytompg doesn't crap out on 40% of my extractions like tytool does. In fact, I'm down to only one that doesn't work. :D

bcc
12-07-2006, 02:28 AM
Please, forgive my ignorance. I am trying to see what is the advantage to use tytompg over the tytool. Looks like they do the same. Is that right?I don't have an executive summary (it'd probably be too biased anyways). Here's a longer answer:

tytompg (and hdemux) were designed from the start to work with high-definition recordings, and OTA high-definition recordings. Tytool was written before tivo's supported high-definition. After hdemux came out, tytool was modified to try and support high-definition as well, and somewhat support OTA. However to support high definition, especially OTA high-definition requires more than just some bug fixes. For example in OTA high-definition TY streams, there are 3 layers of packet headers to deal with and alignment of the packets within each inner layer can, and is (thanks to sloppy broadcasters), arbitrary. The mapping of headers may also be 1-1, 1-many or many-1. One has to be quite a bit more sophisticated to correctly and efficiently handle buffering of these TY records.

tytool has a nice one-click-ish GUI, but from what I've seen that comes with assumptions that one's preferred end goal is a low-def DVD. Last I knew you also had to figure out which mode to run tytool in depending upon your content (chose the multiplexer and such). Kind of like using hdemux with mplex and ffmpeg. But with tytompg, the demuxing step is tightly coupled with the multiplexing, and the multiplexing was purpose-built to work for TY recordings, thus no permanent mine-field of multiplexer bugs to worry about (just my new ones :)

tytool also tries to be a kitchen sink GUI, building in the frame-accurate editing, low-def DVD knowledge, and some stream fixing. tytompg instead strives to be light weight and interoperate with 3rd party mpeg2 editors and stream fixers. I think this is a more desirable model. It allows correctness and completeness to be achieved more readily. If you prefer all-in-one GUIs you probably won't agree :)

I have been looking for a ty to mp4 converter. Is this program goin to do something like that in the future?Doubtful, but you should be able to easily feed the output from tytompg into an mpeg4 transcoding application.

Roger Dylan
12-07-2006, 02:51 AM
I don't have an executive summaryThat was one of the better executive summaries I've ever read. And I spend a significant portion of my life reading them.

bcc
12-07-2006, 04:02 AM
No comment on integrating the 1088i header patch? I realize it's not required functionality for a tystream->mpeg remuxer, but it's something that should be fixed eventually. A lot of encoders don't pad the bottom 8 lines with black (see ATSC A/53 page 28 (http://www.atsc.org/standards/a_53e-with-Amend-1-and-2.pdf)), so you get a gray bar at the bottom. Very annoying for playback on a 4:3 monitor. I wouldn't want the patching to be done by default, but just a switch to enable it. Failing that, piping the output into another program would work. I haven't tried it, but I've assumed tytompg doesn't output to stdout?I forgot to respond to that one before and the question got buried.
I don't really see this falling into the category of something that tytompg *should* fix. I'm trying to *not* have tytompg assume it knows how to best fix the endless problems streams might have. There are also other aps that will do this particular edit for you (avisynth filters, mplayer crop option, DVDpatcher(?). If your target is a media player or transcoder I think they should handle this for you. windvd for example handles this automatically. I hadn't particularly noticed as my displays are 768p and 1200p :)

I haven't tried either, but tytompg should be able to write to a pipe. It doesn't need to do random access I/O on the output file.

bcc
12-07-2006, 06:21 PM
I was down to only two extractions with problems. One of them is taken care of with 0.12, but the other one gets about halfway through a 6GB file and then drops the rest of the video. hdemux works fine.In this sample tytompg winds up processing a garbage chunk at the end of a chunk segment whose tivo timestamps look legit. I suspect you did not have a problem with this recording until you started chopping chunks? By chopping off the master chunk records, tytompg (and hdemux) lose a bit of context and garbage chunks at the end of segments can be misinterpreted as data chunks. (It's too bad the extraction code was never fixed to omit these chunks in the first place...)

burdellgp
12-07-2006, 11:45 PM
I'm trying to take the output of tytompg and down-convert it to DVD format, but I can't get synced audio and video. I am running on Linux, and I've been trying transcode and mencoder. I have missing video, because mplayer keeps saying it is switching framerate (I guess that is its way of handling sync issues).

I'd also like to edit out commercials, and was going to use transcode to do cuts, but then I have to come up with the list of cuts manually. Is there any better way to do this on Linux?

FS1
12-08-2006, 01:03 AM
In this sample tytompg winds up processing a garbage chunk at the end of a chunk segment whose tivo timestamps look legit. I suspect you did not have a problem with this recording until you started chopping chunks? By chopping off the master chunk records, tytompg (and hdemux) lose a bit of context and garbage chunks at the end of segments can be misinterpreted as data chunks. (It's too bad the extraction code was never fixed to omit these chunks in the first place...)Ah, but I did have a problem with the full file. That extraction is a 6GB episode of Lost. tytompg works fine until it gets to the part that I cut out for you. It does make it through the entire file, but once it gets caught on that glitch it never recovers.


-rw-r--r-- 1 root root 3715584000 Dec 6 19:11 {Lost}{2006-10-18}{Further Instructions}{07.00 PM Wed Oct 18, 2006}{xxxxDT}.mpg
-r-xr-xr-x 1 root root 6476013734 Oct 18 07:00 {Lost}{2006-10-18}{Further Instructions}{07.00 PM Wed Oct 18, 2006}{xxxxDT}.ty+

FS1
12-08-2006, 01:29 AM
I forgot to respond to that one before and the question got buried.
I don't really see this falling into the category of something that tytompg *should* fix. I'm trying to *not* have tytompg assume it knows how to best fix the endless problems streams might have. There are also other aps that will do this particular edit for you (avisynth filters, mplayer crop option, DVDpatcher(?). If your target is a media player or transcoder I think they should handle this for you. windvd for example handles this automatically. I hadn't particularly noticed as my displays are 768p and 1200p :)
So everyone just passes the buck until the very end and hopes the decoder treats the bottom 8 pixels as trash? :p
I guess it sorta is a decoder's job to work around encoder bugs, but decoders have certainly been hit or miss in my experience.

I haven't tried either, but tytompg should be able to write to a pipe. It doesn't need to do random access I/O on the output file.I certainly agree that it isn't a tystream remuxer's job to fix incorrect size headers. That's certainly more the function of something like ProjectX, VideoRedo, MPEG-VCR, etc. The problem is they don't implement it either. I was just asking in hope that it would be something trivial enough that you'd consider adding the option (despite it not being a core function). Certainly an avisynth crop(0,0,0,1080) or resize(x,y,0,0,0,-8) would fix the issue if I'm looking to re-encode, but with storage so cheap I'd rather just archive MPEG2. It's not like I'm asking you to resize GOPs and spit out a DVD compliant .vob (for SD extractions). That would be complete overkill. :)

94SupraTT
12-08-2006, 11:14 AM
Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Light>tytompg -i bearswk12.ty -o bearswk12.mpg
'tytompg' is not recognized as an internal or external command,
operable program or batch file.

C:\Documents and Settings\Light>e:

E:\>tytompg -i bearswk12.ty -o bearswk12.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.11, Demuxer version 0.20
Source is bearswk12.ty
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
AC3 audio at 48kHz, 448 kbps, 1792 bytes/sync frame
Recovered AC3 sync, offset 42, chunk 1
Recovered AC3 sync, offset 42, chunk 1
Garbage chunk at 4095: Claims 59656 TY records
Garbage chunk at 12287: Claims 51604 TY records
Warning: temporal_reference order problem. Value: 5 group_start: 216931 last: 2
16942
Warning: temporal_reference order problem. Value: 8 group_start: 217633 last: 2
17644
Warning: temporal_reference order problem. Value: 5 group_start: 217716 last: 2
17730
Warning: temporal_reference order problem. Value: 5 group_start: 229054 last: 2
29068
Warning: temporal_reference order problem. Value: 5 group_start: 255975 last: 2
55983
Warning: temporal_reference order problem. Value: 11 group_start: 257581 last:
257595
video: Warning: Decoder buffer empty, scr ahead by 1605506 (5351.206) 0.5351
Warning: temporal_reference order problem. Value: 5 group_start: 257947 last: 2
57961
Warning: temporal_reference order problem. Value: 5 group_start: 257947 last: 2
57961
Warning: temporal_reference order problem. Value: 11 group_start: 258143 last:
258157
Warning: temporal_reference order problem. Value: 5 group_start: 258749 last: 2
58760
Warning: temporal_reference order problem. Value: 8 group_start: 258788 last: 2
58802
Warning: temporal_reference order problem. Value: 11 group_start: 259322 last:
259336
Warning: temporal_reference order problem. Value: 5 group_start: 260230 last: 2
60238
video: Warning: Decoder buffer empty, scr ahead by 104166 (347.66) 0.347
video: Warning: Decoder buffer empty, scr ahead by 417464 (1391.164) 0.1391
Warning: temporal_reference order problem. Value: 11 group_start: 263134 last:
263148
Warning: Unexpected video frame coding_type: 5
Warning: temporal_reference order problem. Value: 5 group_start: 263134 last: 2
63145
Warning: temporal_reference order problem. Value: 8 group_start: 263134 last: 2
63145
video: Warning: Decoder buffer empty, scr ahead by 689972 (2299.272) 0.2299
Assertion failed: id != -1, file mplex.c, line 2103

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

E:\>

I'll have to give 0.12 a try on this stream.

bcc
12-08-2006, 02:24 PM
Ah, but I did have a problem with the full file. That extraction is a 6GB episode of Lost. tytompg works fine until it gets to the part that I cut out for you. It does make it through the entire file, but once it gets caught on that glitch it never recovers.Ok, it turns out I neglected to turn on my chunk segment length checking and so even with the master chunks present, the code was still getting faked out by this garbage chunk whose tivo timestamps look legit. I can fix both cases actually with some fancier sanity checking.

cheer
12-08-2006, 02:28 PM
I'm trying to take the output of tytompg and down-convert it to DVD format, but I can't get synced audio and video. I am running on Linux, and I've been trying transcode and mencoder. I have missing video, because mplayer keeps saying it is switching framerate (I guess that is its way of handling sync issues).

I'd also like to edit out commercials, and was going to use transcode to do cuts, but then I have to come up with the list of cuts manually. Is there any better way to do this on Linux?

Not sure about commercial editing, but for re-encoding you might try ffmpeg. Not sure if that will work better -- my limited experience with ffmpeg made me curl up in a fetal position and twitch at random intervals for a couple of hours.

Thing is, I'm concerned about what you say regarding frame-rate switching. Have you tried Xine or Totem? Do you see the same behavior? I don't think that's how mplayer handles sync issues at all -- I think something may be b0rked in your stream. (Perhaps the commercials are in a different frame rate than the main program? I bet that would confuse mencoder all to hell.)

cheer
12-08-2006, 02:36 PM
So everyone just passes the buck until the very end and hopes the decoder treats the bottom 8 pixels as trash? :p
I guess it sorta is a decoder's job to work around encoder bugs, but decoders have certainly been hit or miss in my experience.
I certainly agree that it isn't a tystream remuxer's job to fix incorrect size headers. That's certainly more the function of something like ProjectX, VideoRedo, MPEG-VCR, etc. The problem is they don't implement it either. I was just asking in hope that it would be something trivial enough that you'd consider adding the option (despite it not being a core function). Certainly an avisynth crop(0,0,0,1080) or resize(x,y,0,0,0,-8) would fix the issue if I'm looking to re-encode, but with storage so cheap I'd rather just archive MPEG2. It's not like I'm asking you to resize GOPs and spit out a DVD compliant .vob (for SD extractions). That would be complete overkill. :)
You don't need to re-encode to do a crop, you know.

Run it through DGindex and create a .d2v project file. DGIndex will actually notice the encoder error and prompt you if you want to fix it. You can do it there, or you can crop later.

Then create an avisynth script like so:
loadplugin("c:\blah\dgdecode.dll")
mpeg2source("c:\blahblah\videoname.d2v")(Add a crop if you want to.) Then you can play it in media player classic (or anything else that will play an AVI file). No re-encoding at all!

94SupraTT
12-08-2006, 02:43 PM
Doh




E:\>tytompg.exe -i bearswk12.ty -o bearswk12.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.12, Demuxer version 0.20
Source is bearswk12.ty
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
AC3 audio at 48kHz, 448 kbps, 1792 bytes/sync frame
Recovered AC3 sync, offset 42, chunk 1
Recovered AC3 sync, offset 42, chunk 1
Garbage chunk at 4095: Claims 59656 TY records
Garbage chunk at 12287: Claims 51604 TY records
Warning: temporal_reference order problem. Value: 5 group_start: 216931 last: 2
16942
Warning: temporal_reference order problem. Value: 8 group_start: 217633 last: 2
17644
Warning: temporal_reference order problem. Value: 5 group_start: 217716 last: 2
17730
Warning: temporal_reference order problem. Value: 5 group_start: 229054 last: 2
29068
Warning: temporal_reference order problem. Value: 5 group_start: 255975 last: 2
55983
Warning: temporal_reference order problem. Value: 11 group_start: 257581 last:
257595
video: Warning: Decoder buffer empty, scr ahead by 1605506 (5351.206) 0.5351
Warning: temporal_reference order problem. Value: 5 group_start: 257947 last: 2
57961
Warning: temporal_reference order problem. Value: 5 group_start: 257947 last: 2
57961
Warning: temporal_reference order problem. Value: 11 group_start: 258143 last:
258157
Warning: temporal_reference order problem. Value: 5 group_start: 258749 last: 2
58760
Warning: temporal_reference order problem. Value: 8 group_start: 258788 last: 2
58802
Warning: temporal_reference order problem. Value: 11 group_start: 259322 last:
259336
Warning: temporal_reference order problem. Value: 5 group_start: 260230 last: 2
60238
video: Warning: Decoder buffer empty, scr ahead by 104166 (347.66) 0.347
video: Warning: Decoder buffer empty, scr ahead by 417464 (1391.164) 0.1391
Warning: temporal_reference order problem. Value: 11 group_start: 263134 last:
263148
Warning: Unexpected video frame coding_type: 5
Warning: temporal_reference order problem. Value: 5 group_start: 263134 last: 2
63145
Warning: temporal_reference order problem. Value: 8 group_start: 263134 last: 2
63145
video: Warning: Decoder buffer empty, scr ahead by 689972 (2299.272) 0.2299
Assertion failed: id != -1, file mplex.c, line 2108

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

E:\>

0.12 didn't like it either.

bcc
12-08-2006, 02:44 PM
So everyone just passes the buck until the very end and hopes the decoder treats the bottom 8 pixels as trash? No, I just resist creeping futurism when the need for the feature hasn't even been shown. You say you just want to archive, in which case not modifying the mpeg2 is the most accurate representation of what was originally broadcast. Likely you don't really want to just archive, rather you want to play back or transcode the mpeg2. In both situations the software I've seen can and already does handle cropping of the video frame. For example avisynth, mplayer, windvd, vlc. Since 1088i should be equally prevalent with ordinary mpeg2 transport streams, any player that handles transport streams probably already handles this.

cheer
12-08-2006, 03:07 PM
No, I just resist creeping futurism when the need for the feature hasn't even been shown.
Not that my opinion matters any in this debate (bcc is the author after all), but FWIW my preference is that tytompg do as little as possible to alter the streams. Yes, I do want it to do some minimal "fixes" that are required to make the file usable (maintaining sync despite holes and the like), but beyond that I would prefer the stream is not altered. I'd rather be able to pick and choose my croppers, transcoders, filters, etc. separately. Of course it could always be implemented as an option, but since (as bcc points out) there are plenty of other ways to skin this cat, better that bcc spend his time on fixing new, wacky ways that OTA broadcasters find to b0rk streams. :)

burdellgp
12-08-2006, 05:15 PM
Not sure about commercial editing, but for re-encoding you might try ffmpeg.

I tried ffmpeg as well (forgot to list it). It seems to work okay on one video, but on another, all I get is audio in the output file. It also doesn't seem to offer a way to make cuts.


Thing is, I'm concerned about what you say regarding frame-rate switching. Have you tried Xine or Totem? Do you see the same behavior? I don't think that's how mplayer handles sync issues at all -- I think something may be b0rked in your stream. (Perhaps the commercials are in a different frame rate than the main program? I bet that would confuse mencoder all to hell.)

I have seen this behavior in mplayer with virtually everything I've ever extracted from a TiVo, including my old Series2 with TiVo2Go and the Windows method to decrypt transferred files. It doesn't change rate at any reasonable point like commercials, so I think it is just mplayer's way of handling something. The tytompg output plays okay except for the messages from mplayer.

bcc
12-08-2006, 06:02 PM
For PCs, the transcoding apps that seem to work the best are windows only, unfortunately. ffmpeg with xvid must get a lot of use, I'd try that if I was adamant on using linux for transcoding. For playback under linux, I find xine to be a lot more careful about stream presentation times (thus avoiding a/v sync drift, etc.) than mplayer. I think it does the right thing when the frame rate/resolution changes but I haven't tested too much. If only xine had a good non-linear stretch like powerdvd, then I could switch the HTPC to linux...

lsmod
12-08-2006, 08:41 PM
Thanks for the x86-64 binary, that lets me really use tytompg (only one windows laptop here, all the real hardware is Mac/Intel or Linux/AMD64) I ran I ran a big pile of .ty's that have been collecting dust for a while and most of them converted without incident. I did have a few that show the output that FS1 reported in #196 (http://www.dealdatabase.com/forum/showpost.php?p=270975&postcount=196) where the clock warps ahead and then you see tons (350k in one case?) of frame drop messages because they're too late by about the same amount we just warped forward. I haven't watched them all the way through, but these seem to play fine in VLC on the Mac.

Now, I'm seeing some:
tytompg: demux.c:1477: parse_audio_frame: Assertion `syncword == (0xfff8 | (0x2 << 1))' failed.

How can I help debug this?

-Zandr

bcc
12-09-2006, 02:05 AM
I've put up a new version. The new version will refuse to process garbage chunk(s) at the end of chunk segments even when those chunks have tivo timestamps that appear valid. This applies to both end of file and intra-segment situations.

This tests out fine for my mfs_ftp downloaded .ty files. I assume the algorithm isn't too aggressive for other download methods as well (like tytool) but I haven't tested.

I think this'll fix most of the remaining "warping clock"/"too old" situations.

94SupraTT
12-09-2006, 12:57 PM
I've put up a new version. The new version will refuse to process garbage chunk(s) at the end of chunk segments even when those chunks have tivo timestamps that appear valid. This applies to both end of file and intra-segment situations.

This tests out fine for my mfs_ftp downloaded .ty files. I assume the algorithm isn't too aggressive for other download methods as well (like tytool) but I haven't tested.

I think this'll fix most of the remaining "warping clock"/"too old" situations.

I'm still having issues with a stream.



Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Light>e:

E:\>tytompg -i bearswk12.ty -o bearswk12.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.13, Demuxer version 0.21
Source is bearswk12.ty
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
AC3 audio at 48kHz, 448 kbps, 1792 bytes/sync frame
Recovered AC3 sync, offset 42, chunk 1
Recovered AC3 sync, offset 42, chunk 1
Warning: temporal_reference order problem. Value: 5 group_start: 216931 last: 2
16942
Warning: temporal_reference order problem. Value: 8 group_start: 217633 last: 2
17644
Warning: temporal_reference order problem. Value: 5 group_start: 217716 last: 2
17730
Warning: temporal_reference order problem. Value: 5 group_start: 229054 last: 2
29068
Warning: temporal_reference order problem. Value: 5 group_start: 255975 last: 2
55983
Warning: temporal_reference order problem. Value: 11 group_start: 257581 last:
257595
video: Warning: Decoder buffer empty, scr ahead by 1605506 (5351.206) 0.5351
Warning: temporal_reference order problem. Value: 5 group_start: 257947 last: 2
57961
Warning: temporal_reference order problem. Value: 5 group_start: 257947 last: 2
57961
Warning: temporal_reference order problem. Value: 11 group_start: 258143 last:
258157
Warning: temporal_reference order problem. Value: 5 group_start: 258749 last: 2
58760
Warning: temporal_reference order problem. Value: 8 group_start: 258788 last: 2
58802
Warning: temporal_reference order problem. Value: 11 group_start: 259322 last:
259336
Warning: temporal_reference order problem. Value: 5 group_start: 260230 last: 2
60238
video: Warning: Decoder buffer empty, scr ahead by 104166 (347.66) 0.347
video: Warning: Decoder buffer empty, scr ahead by 417464 (1391.164) 0.1391
Warning: temporal_reference order problem. Value: 11 group_start: 263134 last:
263148
Warning: Unexpected video frame coding_type: 5
Warning: temporal_reference order problem. Value: 5 group_start: 263134 last: 2
63145
Warning: temporal_reference order problem. Value: 8 group_start: 263134 last: 2
63145
video: Warning: Decoder buffer empty, scr ahead by 689972 (2299.272) 0.2299
Assertion failed: id != -1, file mplex.c, line 2111

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

E:\>

Anyone else having a "temporal_reference order" problem. This is HD of the SAT.

bcc
12-09-2006, 03:28 PM
I'm still having issues with a stream.
video: Warning: Decoder buffer empty, scr ahead by 689972 (2299.272) 0.2299
Assertion failed: id != -1, file mplex.c, line 2111
Looks like a recording with too many reception problems to decode correctly. I can improve the recovery in any case. A sample could help or I could just guess as to the best recovery for streams with these kinds of errors.

bcc
12-09-2006, 03:41 PM
Thanks for the x86-64 binary, that lets me really use tytompg (only one windows laptop here, all the real hardware is Mac/Intel or Linux/AMY64)With 64 bit linux, you can run both 32&64 bit apps (and need to thanks to 3rd party apps like adobe reader&flash), so I'm curious why a lack of 64 bit binary was holding you back.
I ran I ran a big pile of .ty's that have been collecting dust for a while and most of them converted without incident. I did have a few that show the output that FS1 reported in #196 (http://www.dealdatabase.com/forum/showpost.php?p=270975&postcount=196) .The underlying bug should be fixed in 0.13, assuming you're having the same situation as FS1.

where the clock warps ahead and then you see tons (350k in one case?) of frame drop messages because they're too late by about the same amount we just warped forwardAfter warping the master clock forward it looks like I need to be ready to warp it back if things don't recover... 'course things are likely already broken at the point that the clock needs to be warped forward in the first place.
Now, I'm seeing some:
tytompg: demux.c:1477: parse_audio_frame: Assertion `syncword == (0xfff8 | (0x2 << 1))' failed.

How can I help debug this?That would be an mpeg2 audio stream that isn't parsing right. satellite reception problem? If you send me a sample I could take a look.

lsmod
12-09-2006, 06:12 PM
With 64 bit linux, you can run both 32&64 bit apps (and need to thanks to 3rd party apps like adobe reader&flash), so I'm curious why a lack of 64 bit binary was holding you back.

Well, that wasn't true when I first set up a 64bit machine, and I never looked back. Indeed the 32bit binaries run just fine.


The underlying bug should be fixed in 0.13, assuming you're having the same situation as FS1.

Testing now, but so far, so good.


That would be an mpeg2 audio stream that isn't parsing right. satellite reception problem? If you send me a sample I could take a look.

Quite likely. I'll retest with 0.13, and send a sample if the problem is still there.

94SupraTT
12-09-2006, 06:26 PM
Looks like a recording with too many reception problems to decode correctly. I can improve the recovery in any case. A sample could help or I could just guess as to the best recovery for streams with these kinds of errors.

Maybe it was stormy that day. I'll get you a sample. I don't remember having any reception issue with my dish but maybe I did. In any event, I'll post a sample of where it quit.

94SupraTT
12-09-2006, 07:20 PM
Sample.

sample (http://www.sendspace.com/file/4rih5r)

lsmod
12-10-2006, 07:10 AM
That would be an mpeg2 audio stream that isn't parsing right. satellite reception problem? If you send me a sample I could take a look.

Well, I suspect it was in a garbage chunk, because the same .ty's didn't show the problem in 0.13. Are you still interested in tracking it down?

If so, I'll run them with 0.12 and find a sample for you.

Thanks again, this is great.

-Zandr

RavenStL
12-10-2006, 01:21 PM
Well, I suspect it was in a garbage chunk, because the same .ty's didn't show the problem in 0.13. Are you still interested in tracking it down?

If so, I'll run them with 0.12 and find a sample for you.

Thanks again, this is great.

-Zandr

Fixing garbage chunks, now that would be cool! :) I assume its like getting gold from lead?

lsmod
12-10-2006, 01:39 PM
Fixing garbage chunks, now that would be cool! :) I assume its like getting gold from lead?

My understanding was that 0.13 had improvements to ignore garbage chunks rather than trying to process them. Since the problem I had with those streams went away with 0.13, I'm guessing that the bad MPEG audio data was in a garbage chunk, and 0.13 is now ignoring it.

Oh, and FWIW, I get a core dump when trying to process an encrypted .ty. Took me a minute to figure out what was going on. (not worth worrying about, obviously)

shortkud
12-11-2006, 04:29 AM
Is it just me or does 0.13 seem to be a little slower?

bcc
12-11-2006, 03:42 PM
Sample.

sample (http://www.sendspace.com/file/4rih5r)Ok, fixed. Your recording has both audio&video reception problems. With the back-to-back audio errors, including malformed PES packets (payload larger than what the header claims) the demuxer was eventually screwing up the audio framing.

If you'd tried hdemux+mplex, you would have seen it abort with "Can't find next AC3 frame" at the same point that tytompg was. Because the fix was in the demuxer side, hdemux now works in this case as well.

bcc
12-11-2006, 03:47 PM
Is it just me or does 0.13 seem to be a little slower?I just did a little a/b comparisons and didn't notice a difference.
time tytompg -i file -n count -o /dev/nullis handy for a/b comparisons under linux.
In 0.13 I started cross-architecture compiling for 32 bit linux. Perhaps there's an issue with that; the binaries are not identical to native builds.

bcc
12-11-2006, 03:54 PM
Well, I suspect it was in a garbage chunk, because the same .ty's didn't show the problem in 0.13. Are you still interested in tracking it down?No that's OK, it pretty much proves you were having a garbage chunk issue. Maybe if I run out of real world problems I'll make it recover from any arbitrary garbling of the streams...

94SupraTT
12-12-2006, 01:32 AM
Ok, fixed. Your recording has both audio&video reception problems. With the back-to-back audio errors, including malformed PES packets (payload larger than what the header claims) the demuxer was eventually screwing up the audio framing.

If you'd tried hdemux+mplex, you would have seen it abort with "Can't find next AC3 frame" at the same point that tytompg was. Because the fix was in the demuxer side, hdemux now works in this case as well.

It got further but still stopped. It was stopping at the 8gig mark now its stopping at the 10gig mark of a 27 gig file.



Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Light>e:

E:\>tytompg -i bearswk12.ty -o bearswk12.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.14, Demuxer version 0.22
Source is bearswk12.ty
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
AC3 audio at 48kHz, 448 kbps, 1792 bytes/sync frame
Recovered AC3 sync, offset 42, chunk 1
Recovered AC3 sync, offset 42, chunk 1
Warning: temporal_reference order problem. Value: 5 group_start: 216931 last: 2
16942
Warning: temporal_reference order problem. Value: 8 group_start: 217633 last: 2
17644
Warning: temporal_reference order problem. Value: 5 group_start: 217716 last: 2
17730
Warning: temporal_reference order problem. Value: 5 group_start: 229054 last: 2
29068
Warning: temporal_reference order problem. Value: 5 group_start: 255975 last: 2
55983
Warning: temporal_reference order problem. Value: 11 group_start: 257581 last:
257595
video: Warning: Decoder buffer empty, scr ahead by 1605506 (5351.206) 0.5351
Warning: temporal_reference order problem. Value: 5 group_start: 257947 last: 2
57961
Warning: temporal_reference order problem. Value: 5 group_start: 257947 last: 2
57961
Warning: temporal_reference order problem. Value: 11 group_start: 258143 last:
258157
Warning: temporal_reference order problem. Value: 5 group_start: 258749 last: 2
58760
Warning: temporal_reference order problem. Value: 8 group_start: 258788 last: 2
58802
Warning: temporal_reference order problem. Value: 11 group_start: 259322 last:
259336
Warning: temporal_reference order problem. Value: 5 group_start: 260230 last: 2
60238
video: Warning: Decoder buffer empty, scr ahead by 104166 (347.66) 0.347
video: Warning: Decoder buffer empty, scr ahead by 417464 (1391.164) 0.1391
Warning: temporal_reference order problem. Value: 11 group_start: 263134 last:
263148
Warning: Unexpected video frame coding_type: 5
Warning: temporal_reference order problem. Value: 5 group_start: 263134 last: 2
63145
Warning: temporal_reference order problem. Value: 8 group_start: 263134 last: 2
63145
video: Warning: Decoder buffer empty, scr ahead by 689972 (2299.272) 0.2299
video: Warning: Decoder buffer empty, scr ahead by 254386 (847.286) 0.847
Warning: temporal_reference order problem. Value: 5 group_start: 282748 last: 2
82756
Warning: Unexpected video frame coding_type: 0
Warning: Video GOP malformed. Expected picture coding extension
Warning: temporal_reference order problem. Value: 2 group_start: 296554 last: 2
96557
video: Warning: Decoder buffer empty, scr ahead by 167788 (559.88) 0.559
video: Warning: Decoder buffer empty, scr ahead by 387562 (1291.262) 0.1291
Warning: temporal_reference order problem. Value: 5 group_start: 322845 last: 3
22856
video: Warning: Decoder buffer empty, scr ahead by 186786 (622.186) 0.622
Warning: temporal_reference order problem. Value: 11 group_start: 324701 last:
324715
Warning: Unexpected video frame coding_type: 5
Warning: temporal_reference order problem. Value: 5 group_start: 324873 last: 3
24881
video: Warning: Decoder buffer empty, scr ahead by 253616 (845.116) 0.845
Warning: temporal_reference order problem. Value: 5 group_start: 327117 last: 3
27125
Assertion failed: id != -1, file mplex.c, line 2111

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

E:\>

danpritts
12-12-2006, 01:45 AM
You mean the approximate times I report for the streams when processing is done? That was just a first blush estimate which doesn't take into account variations in the audio frame rate (such as when you go between the feature and some commercial). It also doesn't take into account variable video frame rates or even video pulldown. Reporting the stream times based upon the timestamps is likely better, on the other hand that'll fail to take into account audio or video holes.

yeah, that's what i meant.

FWIW I'm using a series1 standalone so there shouldn't be any variations in the frame rates or or audio sample rate.

danpritts
12-12-2006, 02:24 AM
Yes, I thought vlc was happy as of version 0.6, per:
http://www.dealdatabase.com/forum/showthread.php?p=269642&highlight=vlc#post269642
That's largely why I tried to encourage folks to take it to a new thread
Probably you have no mpeg2 codec installed for it. If folks want to test with quicktime or windows media player, any troubleshooting involves figuring out exactly what codec you're using at the moment.

I just tried with v 14, and still have basically the same behavior I mentioned before; vlc has stuttery video, quicktime player pukes, and mplayer looks OK but with interlacing artifacts.

I'm quite sure that I have an mpeg2 codec installed for quicktime - it's a mac, and quicktime can play dvds and an mpeg file extracted from a downloaded vcd. The file has a .mpg extension (like the one from the vcd). Presumably some magic cookie or something isn't to quicktime's liking.

I have the Flip4Mac (windows media) codecs, the DiVX codecs, and the XVIDDelegate codec installed in quicktime in addition to the standard ones.

I just tried it out on my winxp machine. it's old and slow and it just can't keep up with the decoding, but it appears to have the same issue with vlc. windows media player looks better.

as a reminder, this is from a ty extracted from a standalone series 1 via mfs_ftp.

see http://www.deathstar.org/~danno/tytompg/test.mpg for an example file. (also the original ty there with a similar name)

danpritts
12-12-2006, 03:01 AM
I'm quite sure that I have an mpeg2 codec installed for quicktime - it's a mac, and quicktime can play dvds and an mpeg file extracted from a downloaded vcd. The file has a .mpg extension (like the one from the vcd). Presumably some magic cookie or something isn't to quicktime's liking.


So I'm a little confused now that I see that apple sells, for $20 extra, an MPEG2 codec for quicktime. I haven't purchased such a thing.

how the heck does it play DVDs without an MPEG2 codec?

bcc
12-12-2006, 08:52 PM
I just tried with v 14, and still have basically the same behavior I mentioned before; vlc has stuttery video, quicktime player pukes, and mplayer looks OK but with interlacing artifacts.I tried this sample and it works fine in xine, powerdvd, mplayer. windvd shows some slight stuttering; in vlc a bit more.
The good news is I finally got vlc to compile from source (they don't make it easy) and so I can figure out exactly what vlc doesn't like about this case. Same problem seems to apply to the PAL rate sample from a few weeks ago.
I have the Flip4Mac (windows media) codecs, the DiVX codecs, and the XVIDDelegate codec installed in quicktime in addition to the standard ones.
...
So I'm a little confused now that I see that apple sells, for $20 extra, an MPEG2 codec for quicktime. I haven't purchased such a thing.

how the heck does it play DVDs without an MPEG2 codec?
I have no experience with mac codecs or ways to test them. If you've found a working+free mpeg2 codec then no need to buy one from apple. Since dvd's are mpeg2 encoded, you almost certainly have an mpeg2 codec installed, who knows which one.

Will get to the bottom of what vlc doesn't like about this mpeg2.

bcc
12-13-2006, 09:01 PM
Will get to the bottom of what vlc doesn't like about this mpeg2.So I've figured out what is going on with the vlc video stuttering. For s1 and s2 standalone tivos, the streams are missing timestamps for the video P frames. My code was being permissive and allowing the PTS timestamp to remain unchanged until updated by the stream. vlc is instead requiring the PTS to be frame accurate if present.

So for frames that lack PTS timestamps I now omit them as well, and now everyone is happy (actually I calculate the missing PTS precisely for muxing purposes, but that's another story).

This fix applies to the PAL sample as well, as that was a standalone tivo. The PAL frame rate was not an issue.

danpritts
12-14-2006, 02:54 AM
I have no experience with mac codecs or ways to test them. If you've found a working+free mpeg2 codec then no need to buy one from apple. Since dvd's are mpeg2 encoded, you almost certainly have an mpeg2 codec installed, who knows which one.

I misspoke when i said "quicktime plays DVDs". The dvd player application that comes with the mac (which i presumed used quicktime as its backend, but who knows) plays DVDs.

I've been led to believe by others that it's able to play DVDs (and DVD disk images; maybe it's just if they are in VOB format) but not standard mpeg-2, unless you buy the extra codec pack.

I was really thrown for a loop when quicktime player played the mpeg file from a vcd, but i've since learned that those are mpeg-1.

I have been picking up knowledge about mpeg & codecs but i'm clearly at the point where i know enough to be dangerous but not enough to do anything useful. ;)

Thanks for figuring out the issue with vlc! I've never tried compiling it from source but i'm sure it's a nightmare.

alfonzotan
12-14-2006, 07:14 PM
Awesome. Latest version muxed a BSG file that I've been fighting with for literally months (okay, not 24/7 or anything) without a hitch. Thanks much.

gyre
12-14-2006, 07:23 PM
I've tried the latest version with my S1 UK PAL clip skippy.ty posted earlier.

The resulting mpeg had a problem in wmp... the video froze after a few seconds, although the sound continued.

With vlc, it seemed a lot less jumpy now.

-- gyre --

bcc
12-15-2006, 05:14 AM
Thanks for figuring out the issue with vlc! I assume this means you've got your old s1 recordings working fine thru vlc.
I don't know mac media players; I should think there must be a free mpeg2 codec for it like there is under windoze.
I've never tried compiling it from source but i'm sure it's a nightmare.Yes, it's almost as if they tried to make the dependencies prohibitive. They could make all the DRM/patented stuff dynamic modules. Then the core would be distributable by the linux distros.

bcc
12-15-2006, 05:21 AM
I've tried the latest version with my S1 UK PAL clip skippy.ty posted earlier.

The resulting mpeg had a problem in wmp... the video froze after a few seconds, although the sound continued.You gotta troubleshoot that deeper with graphedit or gspot to figure out what codec is having trouble.
With vlc, it seemed a lot less jumpy now.On closer inspection it looks like vlc is still having some minor gripes with the video, but it won't say exactly why and the details are buried in the libmpeg2 library. vlc has the same issue if you simply demux the .m2v with hdemux, so the issue is with the video stream itself not anything tytompg is doing by remuxing.
Meanwhile xine, mplayer, windvd and powerdvd play back your sample content without trouble.

alfonzotan
12-15-2006, 03:27 PM
Got a couple of recordings here (actually two recordings of the same episode) that've been giving me fits. No joy muxing with tytool or hdemux, and now no luck with tytompeg, either. Getting a long run of "Garbage chunk at XXXXX:" (consecutive numbers) "Claims XXXXX TY records" (non-consecutive numbers). After a whole lot of these, the MPEG file chops off at about 10% of the original(s).

Any ideas? Want a sample?

bcc
12-15-2006, 03:45 PM
Got a couple of recordings here (actually two recordings of the same episode) that've been giving me fits. No joy muxing with tytool or hdemux, and now no luck with tytompeg, either. Getting a long run of "Garbage chunk at XXXXX:" (consecutive numbers) "Claims XXXXX TY records" (non-consecutive numbers). After a whole lot of these, the MPEG file chops off at about 10% of the original(s).

Any ideas? Want a sample?Back-to-back "Claims XXXXX TY records" suggests you don't have a valid .ty. Scrambled recording? You're using a .tmf instead? Something went wrong with extraction?

Do you mean you're getting some valid mpeg out? This isn't a music channel recording is it?
Try running with the '-dr' switch for some debugging.

alfonzotan
12-15-2006, 04:35 PM
Back-to-back "Claims XXXXX TY records" suggests you don't have a valid .ty. Scrambled recording? You're using a .tmf instead? Something went wrong with extraction?

Do you mean you're getting some valid mpeg out? This isn't a music channel recording is it?
Try running with the '-dr' switch for some debugging.

Definitely not scrambled, my box had been hacked for months when they were recorded. Recording is from Universal HD, and I've tried everything under the sun to get these to work. The .ty's in this case were extracted using mfs_ftp, just a straight ftp copy. I am getting a ~550 MB MPEG on the other end. Using hdemux produces roughly the same result, looks like it craps out at the same point (which makes sense).

Looking at the MPG (which is fine, if way too short), it appears to drop out right at a commercial break. Hey, maybe I can isolate that chunk...

At any rate, I'll pull the .tmf of one of these files and try that. My clue light may be dim, but at least I'm getting some current through there now...

drez
12-15-2006, 08:02 PM
Also, would it be possible to have tytompg accept drag-and-dropped files (multiple files, if possible? processed one at a time and if one fails, rename to .mpg.failed and continue onto the next file.... I might be asking for too much) and output the mpg's in the same folder as the .ty and with the same filename as the original (for example, fox.ty becomes fox.mpg)?
Since I haven't implemented a GUI I'm not sure what you mean :) Probably you're only considering windows? Perhaps a windows batch script is in order for now.

Hey, I did some experimenting for my C course and learned that all you'd need is add for drag-n-drop support (.ty files dragged into tytompg.exe in a Windows Explorer window) would be add this to tytompg's accepted inputs:



tytompg "C:\path to file\in quotes\tvshow.ty"


(If there's no -i, assume the same path and filename ending in .mpg for the output file.
"C:\path to file\in quotes\tvshow.mpg" or if you wanted to do it the easy way: "C:\path to file\in quotes\tvshow.ty.mpg"... the easy way is fine.)

If tytompg is run that way, it should make a new variable set to 1, and at the end of the program if that variable is 1, it should do:


system("pause")


otherwise, the window will close by itself in XP. (want to keep the window open to read any errors that might pop up...)


I assume the way to accept multiple files would be with


tytompg "C:\path to file\in quotes\tvshow.ty" "C:\path to other file\in quotes\tvshow.ty" "C:\path to third file\in quotes\tvshow.ty"

and folders would be


tytompg "C:\path to file\in quotes\"


That would require scanning the folder though.... the multiple files and folders probably should be left to a batch file like you said.

Thanks again for writing tytompg. Adding this would make it so much easier for the newbies to use your great programs (I have absolutely no problem using tytompg how it is now, but people like my uncle are a little scared of the command line...)

94SupraTT
12-15-2006, 08:03 PM
E:\>tytompg -i bearswk12.ty -o bearswk12.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.14, Demuxer version 0.22
Source is bearswk12.ty
Video frame size is 1280x720 - high definition. Frame rate 59.940060
Video bit rate is 19000000 bits/sec., 19000 Kbit/sec.
AC3 audio at 48kHz, 448 kbps, 1792 bytes/sync frame
Recovered AC3 sync, offset 42, chunk 1
Recovered AC3 sync, offset 42, chunk 1
Warning: temporal_reference order problem. Value: 5 group_start: 216931 last: 2
16942
Warning: temporal_reference order problem. Value: 8 group_start: 217633 last: 2
17644
Warning: temporal_reference order problem. Value: 5 group_start: 217716 last: 2
17730
Warning: temporal_reference order problem. Value: 5 group_start: 229054 last: 2
29068
Warning: temporal_reference order problem. Value: 5 group_start: 255975 last: 2
55983
Warning: temporal_reference order problem. Value: 11 group_start: 257581 last:
257595
video: Warning: Decoder buffer empty, scr ahead by 1605506 (5351.206) 0.5351
Warning: temporal_reference order problem. Value: 5 group_start: 257947 last: 2
57961
Warning: temporal_reference order problem. Value: 5 group_start: 257947 last: 2
57961
Warning: temporal_reference order problem. Value: 11 group_start: 258143 last:
258157
Warning: temporal_reference order problem. Value: 5 group_start: 258749 last: 2
58760
Warning: temporal_reference order problem. Value: 8 group_start: 258788 last: 2
58802
Warning: temporal_reference order problem. Value: 11 group_start: 259322 last:
259336
Warning: temporal_reference order problem. Value: 5 group_start: 260230 last: 2
60238
video: Warning: Decoder buffer empty, scr ahead by 104166 (347.66) 0.347
video: Warning: Decoder buffer empty, scr ahead by 417464 (1391.164) 0.1391
Warning: temporal_reference order problem. Value: 11 group_start: 263134 last:
263148
Warning: Unexpected video frame coding_type: 5
Warning: temporal_reference order problem. Value: 5 group_start: 263134 last: 2
63145
Warning: temporal_reference order problem. Value: 8 group_start: 263134 last: 2
63145
video: Warning: Decoder buffer empty, scr ahead by 689972 (2299.272) 0.2299
video: Warning: Decoder buffer empty, scr ahead by 254386 (847.286) 0.847
Warning: temporal_reference order problem. Value: 5 group_start: 282748 last: 2
82756
Warning: Unexpected video frame coding_type: 0
Warning: Video GOP malformed. Expected picture coding extension
Warning: temporal_reference order problem. Value: 2 group_start: 296554 last: 2
96557
video: Warning: Decoder buffer empty, scr ahead by 167788 (559.88) 0.559
video: Warning: Decoder buffer empty, scr ahead by 387562 (1291.262) 0.1291
Warning: temporal_reference order problem. Value: 5 group_start: 322845 last: 3
22856
video: Warning: Decoder buffer empty, scr ahead by 186786 (622.186) 0.622
Warning: temporal_reference order problem. Value: 11 group_start: 324701 last:
324715
Warning: Unexpected video frame coding_type: 5
Warning: temporal_reference order problem. Value: 5 group_start: 324873 last: 3
24881
video: Warning: Decoder buffer empty, scr ahead by 253616 (845.116) 0.845
Warning: temporal_reference order problem. Value: 5 group_start: 327117 last: 3
27125
Assertion failed: id != -1, file mplex.c, line 2111

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.


Any luck with this?

alfonzotan
12-15-2006, 08:30 PM
Hmm. Getting a lot of these in verbose mode right before muxing comes to a crashing halt (sorry, no luck with a cut-and-paste, I'm a command line *****):

video: Warning: Dropping Frame -- too old by [ungodly long number] (slightly less ungodly long number) three-digit number with five-digit decimal

I recognize the spot where the MPEG ends in this chunk--it's where the MPEGs croaked when I tried to mux this sucker in Tytool several months back.

gyre
12-16-2006, 08:53 AM
graphedit says i'm using the intervideo codec for displaying the mpeg2 video.

gspot refuses to give me any information about the video codec on the resulting mpeg2 file.

if i run the mpeg2 through videoredo's quick fix, it plays fine through wpm etc.

i'm a n00b about codecs and things, so if you need more info to help debug, pls let me know.

thx.

-- gyre --

lsmod
12-16-2006, 05:27 PM
Another assertion failure, about 7GB into a HD movie from HDNM:



n6mod@mycroft:~/tystreams$ tytompg -dr -i 410404.ty -o 410404.ty.mpg
tytompg: Copyright (c) 2004-2006 B.C. <b24cc@yahoo.com>
Multiplexer version 0.15, Demuxer version 0.22
Source is 410404.ty
Skipping master chunk
Vid SEQ packet at index 1
Chunk 1 Record 0 Type f/20 4
Chunk 1 Record 1 Type 7/e0 4 Video/SEQ
Chunk 1 Record 2 Type 2/e0 112 Video/CONT
Video frame size is 1280x1088 - high definition. Frame rate 29.970030
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
Chunk 1 Record 3 Type 2/e0 4 Video/CONT
Chunk 1 Record 4 Type c/e0 8 Video/GOP_FRAME
Chunk 1 Record 5 Type f/20 4
Chunk 1 Record 6 Type 9/c0 1552 Audio/AC3
Audio timestamp offset: 595.500000 ms
Chunk 1 Record 7 Type 9/c0 1552 Audio/AC3
Warning: PES payload had 2 bytes end pad, chunk 1
Pes len 1550 headlen 14 paybytes 1536
AC3 audio at 48kHz, 384 kbps, 1536 bytes/sync frame
Chunk 1 Record 8 Type 9/c0 1552 Audio/AC3
Reported datarate 65384 Kbit/sec. (65000+384)

And we're off.... (BTW: are there other useful -d switches? -dr is the only one I've seen reference to)

50k+ blocks later:


Skipping junk payload chunk: 51032
Chunk 51032 Record 0 Type f/20 4
Chunk 51032 Record 1 Type 2/e0 47828 Video/CONT
Chunk 51032 Record 2 Type 2/e0 4 Video/CONT
Chunk 51032 Record 3 Type b/e0 4 Video/B_FRAME
Chunk 51032 Record 4 Type 2/e0 4 Video/CONT
tytompg: demux.c:283: parse_pes: Assertion `0' failed.
Aborted (core dumped)


50MB sample here (http://saratoga.milewski.org/410404_sample.ty).

dd if=410404.ty of=410404_sample.ty bs=131072 skip=50832 count=400

alfonzotan
12-16-2006, 05:38 PM
Okay, I pulled over the other copy of the show that had been giving me problems (no idea why I had the bright idea to record both showings that night, but I'm glad I did) in a .tmf tarball (the .ty had not been muxing from either recording), extracted the parts, and they all muxed fine with tytompeg. No clue why the full .ty file wouldn't work, or why the other recording is so spectacularly screwed.

But I'm happy. Thanks again for the help. bcc, if by some odd chance you'd like a copy of the spectacularly-screwed file, drop me a PM. Be glad to get it to you.

bcc
12-17-2006, 04:20 PM
Changes: The demuxer protects more against garbage chunks/scrambled recordings. More sanity checking when trying to re-establish ac3 sync after it's lost. Relax check against forbidden PTS_DTS flag in PES header (this is lsmod's failure case) Relax check against unparsable audio frames during remultiplexing

bcc
12-17-2006, 04:23 PM
Any luck with this?I never got a stream, but I've now changed things so that the code tries harder to re-establish ac3 sync when it's lost and allows more garbled data to get thru (like in your recording). Basically safeties are coming off. This should "fix" your case so long as we don't permanently lose audio sync. If we permanently lose audio sync you'll see endless error messages (since the code doesn't assert).