PDA

View Full Version : Video extraction from the new HD10-250



Pages : [1] 2 3 4 5

redstone
06-12-2004, 09:12 AM
Just testing the waters here.

Has anyone sucessfully extracted the High Def streams from the new HD Directv TIVO?

I am told that this is basically the same as the SD Directv TIVO with the addition of the High Definition capability.

Thanks

TechFarmer
06-12-2004, 11:15 AM
No joy. Look to this thread (http://www.dealdatabase.com/forum/showthread.php?t=35148).

aos
06-12-2004, 04:39 PM
Has anyone sucessfully extracted the High Def streams from the new HD Directv TIVO?


Yes.

-andrew

mrblack51
06-12-2004, 11:10 PM
No joy. Look to this thread (http://www.dealdatabase.com/forum/showthread.php?t=35148).

what does that thread have to do with extraction? the NoCSO hack works if you can get bash on the system, and disables encryption on all programs. niether jdiner's tools nor tystudio currently process the HD streams, but the standard streams work just fine. the hd streams (both OTA and DTV) are not encrypted with a properly applied NoCSO hack, that has been verified by a number of sources.

redstone
06-13-2004, 08:26 PM
AOS,

You answered yes but after searching this forum for half a day, I have found no one who has actually done it.

Found plenty of discussions regarding the topic but nothing specific.
Two were of merit:
One said that the Sleeper set of tools won't work because the kernel is different on the HD TIVO.
The other explained how to turn off encryption of the files.

A large numbr of discusssions were about how to handle the large size of HD data on DVD's but I am not interested in burning to a DVD.
I just need the HD streams on my PC so my PC HDTV card can play then back on my TV as well as dumping them to my D-VHS.

I have lots of tools for the PC to manipluate High Def streams to put them in a format that the PC HDTV card will understand.



I doubt you care but here is my motivation for pursuing this HD extraction:

I have been collecting good HD events (OTA or from SAT) for quite some time and I have a personal use library of around 300 HD movies,concerts,sports on D-VHS that I made with the DISH model 5000 before DISH changed the transmission format in February and made my 5000 a doorstop. I am now with Directv and bought an HD TIVO and this device is the only satellite receiver that has a chance at letting me regain the capability of archiving that I had last February.

The last straw was when I had to delete the great Kubrick film "A Clockwork Orange" which was redone in High Def in order to have room for my HD TIVO to record more stuff. I really wanted to be able to get that movie on D-VHS as my DVD of it sure looks lousy now.

I am determined to do this with the HD TIVO with the help of folks like you.


Can you point me to a more specific post where someone has actually extracted the HD streams from the HD TIVO?

Sure would appreciate it.

Joe

alldeadhomiez
06-13-2004, 10:04 PM
I have lots of tools for the PC to manipluate High Def streams to put them in a format that the PC HDTV card will understand.

Keeping in mind the fact that your tools probably can't mux HD tystreams yet...

To compromise your HD TiVo and extract streams, follow the same procedure as with a PROM modded SD TiVo running software 4.x/5.x. That is to say, socket/reflash the PROM, replace_initrd on the active kernel image, and start installing "stuff" on the drive. At the minimum you'll probably want mfs_ftp, nocso, and an rc.sysinit.author script that starts the network (I believe ax8817x and ehci drivers are stock on 3.1.5).

bcc
06-13-2004, 11:13 PM
Well you can extract HD tystreams today using mfs_ftp. And you can then archive them, which seems to be all redstone is asking for. I assume you can re-insert the HD tystreams with mfs_ftp but I haven't tried that yet.

You can also demux/playback HD tystreams using mplayer, tho I've had to address a few issues. Specifically http://www.dealdatabase.com/forum/showthread.php?t=35384 concerning OTA HD content. Sounds like redstone is concerned with premium content, which has played just fine for me (tho very limited testing).

Species8479
06-14-2004, 01:22 AM
Keeping in mind the fact that your tools probably can't mux HD tystreams yet...

Tools (http://www-personal.engin.umich.edu/~balazer/atsc/)

:cool:

AlphaWolf
06-14-2004, 12:41 PM
If it helps, I do have some hdtivo streams sitting on my hard disk right now. Granted nothing will play them, they are there.

bcc
06-14-2004, 02:04 PM
If it helps, I do have some hdtivo streams sitting on my hard disk right now. Granted nothing will play them, they are there.
So you haven't tried mplayer then? Works for me.

redstone
06-14-2004, 05:01 PM
That is to say, socket/reflash the PROM,


Thanks for all that replied to my post. I will just continue to monitor this forum as more of the HDTIVO's get into folks's hands to see if further progress is made in getting the HD streams off.

The above quote is a brick wall which I am not even considering climbing over

I have read enough in here about extraction of the video on Series 2 to see that re-flashing proms is not necessary.
Sleeper has some very nice tutorials that explain this as well as his complete set of SW (well, the compilation of work from folks like you)

I am 100% sure that in time, the same will be doable on the HDTIVO.

After all, who wants to erase some High Definition Gems from the TIVO never to be seen again?
Case in point:
Thanks to the HTPC world that I come from and my MYHD card, I have the Ravens Superbowl victory in High Def on D-VHS that I can just pop into my D-VHS deck,fire up the MYHD card and hit play.


BTW, for those interested in archiving satellite broadcast material to D-VHS (like me of course), you may find this link interesting. I either have my HDTIVO archiving for me by August or I am ordering the referenced setup.

http://www.avsforum.com/avs-vb/showthread.php?s=&threadid=409442

Thanks again.

Joe

redstone
06-14-2004, 05:04 PM
If it helps, I do have some hdtivo streams sitting on my hard disk right now. Granted nothing will play them, they are there.


Care to put a small (say 30 second) stream on an ftp site for some of us to try our reformating tools on?

Thanks,
Joe

alldeadhomiez
06-14-2004, 05:12 PM
The above quote is a brick wall which I am not even considering climbing over

Many community members offer this as a service for $50-100. It's pretty easy once you've had a little practice.


I have read enough in here about extraction of the video on Series 2 to see that re-flashing proms is not necessary.
Sleeper has some very nice tutorials that explain this as well as his complete set of SW (well, the compilation of work from folks like you)

I am 100% sure that in time, the same will be doable on the HDTIVO.

Don't hold your breath; the 2.4.4/3.1.U5/BASH_ENV hole was closed. A PROM hack is your only option on the HR10-250. This has nothing to do with video extraction.

Direct extraction (of encrypted streams) is still possible if you pull the drive.

bcc
06-14-2004, 06:06 PM
BTW, for those interested in archiving satellite broadcast material to D-VHS (like me of course), you may find this link interesting. I either have my HDTIVO archiving for me by August or I am ordering the referenced setup.
Strange - the idea of moding your hd-tivo STB is a "brick wall" but some other pre-released STB mod is worth waiting for?

redstone
06-14-2004, 08:31 PM
Strange - the idea of moding your hd-tivo STB is a "brick wall" but some other pre-released STB mod is worth waiting for?

The HD TIVO is $1000 and when I ultimately sell it, I want it pristine.
The STB needed for the referenced 5000-HD mod is less than $300.

mikemav
06-14-2004, 09:33 PM
I have managed to have one success with TyTools and HD content. Most of my extractions are from HBO or DiscoveryHD. They are all over 4GB and crash TyTools after a few seconds. I did record and extract the HD Net test pattern, which is 10 minutes long. The TY+ file was about 700 meg. With that, TyTools was able to VOB/Demux it to a VOB file, and I was thrilled to find out that the MyHD card (same one redstone was talking about) played it back under "file" format at full 1080i resolution. So there is hope. I suspect the 9-15 GB size of the typical movie extraction are choking TyTools, since I now know it can handle the HD resolution/format.

If anyone has any ideas or would like some screen shots or info from the TyTools crash, etc.. I would be glad to post them.

bcc
06-14-2004, 10:18 PM
If you extracted with mfs_ftp, you likely have the string "Priority set..." between chunks at the FSID boundaries (every 512MB most likely). Want to check? If so you may just need a copy of mfs_export that doesn't emit this debug output.

mikemav
06-14-2004, 11:30 PM
If you extracted with mfs_ftp, you likely have the string "Priority set..." between chunks at the FSID boundaries (every 512MB most likely). Want to check? If so you may just need a copy of mfs_export that doesn't emit this debug output.

Be glad to check. Forgive my ignorance, but how do I see if I have Priority set? Is there a command I can type in bash or something to check in the files or logs to be sure? Is it in settings.tcl or mfs_ftp.tcl? I see Insert Priority, but that is probably not it. If it is on, is there a way I can (reliably) change it in mfs_ftp so I do not need to install another program? Thanks for the tip and I'll be glad to let you know.

mikemav
06-14-2004, 11:32 PM
If you extracted with mfs_ftp, you likely have the string "Priority set..." between chunks at the FSID boundaries (every 512MB most likely). Want to check?

Is this what you are talking about:
proc setpri { prog_pri channel chan_pri } {
global info ; set p 1
if { [file exist $info(path)/setpri] } {
outd $p "set priorities: mfs_ftp $prog_pri % I/O chan $chan_pri %"
set prog_pid [pid $info(dc)] ; set chan_pid [pid $channel]
catch { exec $info(path)/setpri fifo $prog_pri $prog_pid }
catch { exec $info(path)/setpri fifo $chan_pri $chan_pid }
} else {
outd 0 "$info(path)/setpri not found, unable to adjust priority"
}
}

bcc
06-15-2004, 12:12 AM
Is this what you are talking about:}
Nope. On the extracted .ty, from a unix machine, try:
strings <filename>.ty | grep 'Priority set'
But if riley is right, then this is a red herring and you shouldn't see any matches.

rc3105
06-15-2004, 02:26 AM
well hrm... safest course is to extract as tmf then convert to ty (mfs_tarstream has been 100% reliable)

redstone
06-20-2004, 08:54 AM
I have managed to have one success with TyTools and HD content. Most of my extractions are from HBO or DiscoveryHD. They are all over 4GB and crash TyTools after a few seconds. If anyone has any ideas or would like some screen shots or info from the TyTools crash, etc.. I would be glad to post them.


I have found a forum member who is willing (for a price of course) to socket my prom and put a re-flashed one in.

Before I go down that route though, I still have a bunch of questions.

First one:

Has the problem that mikemav described been rectified?

No point in ruining my warranty if I can not get playable HD streams.

Second one:
I assume the video extraction is over an ethernet connection vs the same serial connection used for the bash prompt?

To get an ethernet connection, do I need a USB to ethernet adaptor or can I just set up a direct network between the 2 USB 2.0 connections (PC to TIVO)?

That would blow away a 100mbit network since the USB 2.0 spec is 480mbits/sec

If that does not work, if I can find a USB 2.0 to Gigabit ethernet adaptor, will that work or do I need a different device driver?
Since GIGE was designed to allow folks to seemlessly upgrade their networks from 100 mbit to GIGE, I would think that this is doable.

Thanks,
Joe

redstone
06-20-2004, 08:59 AM
Since the HD-TIVO has a few bugs in the SW (the HDNET one seems to be the only really critical one) , it is expected that they will download a new version of SW to fix these bugs.


If one has a modified prom, does that mean that any SW fixes that Directv downloads will be unavailable?

If I understand what the prom mod does, it loads a different kernel than the standard one so what happens to anmy SW upgrades from TIVO?

Joe

alldeadhomiez
06-20-2004, 09:15 AM
Has the problem that mikemav described been rectified?

Read the tytools thread. If you don't see it there, I'd guess your answer is "no."

Remember, if you insist on being an early adopter of a new platform, you need to expect that some things (especially muxing) might not work right for a while. If you're not willing to work out the problems yourself, you can't complain when somebody else doesn't fix them.


Since the HD-TIVO has a few bugs in the SW (the HDNET one seems to be the only really critical one) , it is expected that they will download a new version of SW to fix these bugs.


If one has a modified prom, does that mean that any SW fixes that Directv downloads will be unavailable?

If I understand what the prom mod does, it loads a different kernel than the standard one so what happens to anmy SW upgrades from TIVO?

This is a very common newbie question and not an appropriate topic for the extraction forum. Do a little reading on how software upgrades (and the security mechanisms) work, and you'll know your answer.

redstone
06-20-2004, 09:39 AM
If you're not willing to work out the problems yourself, you can't complain when somebody else doesn't fix them.

Odd comment.
Since when am I complaining? All I am looking for are pointers to some help.






This is a very common newbie question and not an appropriate topic for the extraction forum. Do a little reading on how software upgrades (and the security mechanisms) work, and you'll know your answer.

A simple "YES,NO or it is not a black/white answer" would have been sufficient. Is it that hard to type in Y,E,S or N,O?


Question for you as the moderator: I have spent hours reading over the posts in DDB and have found that the general attitude of this website is very unfriendly and often, downright rude.
Not every individual of course as some (like Sleeper,MIkemav,etc.) are very polite.

Why is that?

I spend a lot of time in the HTPC section of avsforum and it is the complete oposite - people go out of their way to help you and are very polite.
If someone is rude or hostile, they get chastised by the other members.
In here, it feels like I am posting in Usenet where one needs a flack jacket.

alldeadhomiez
06-20-2004, 10:46 AM
Odd comment.
Since when am I complaining? All I am looking for are pointers to some help.

You weren't complaining. I'm managing expectations since there's a good chance your requirements won't be met by our (all-volunteer, all-free) projects. I encourage you and anyone else in your situation to look elsewhere for a solution.

As for the rest of your rant, post it in General TiVo Discussions or the Sewer as it is not even remotely related to extraction.

bcc
06-20-2004, 06:17 PM
Has the problem that mikemav described been rectified?We didn't hear back from him as to whether or not he's actually seeing the 'Priority set' issue. If that is indeed his problem, I've posted a fix here:
http://www.dealdatabase.com/forum/showpost.php?p=169896&postcount=210
Riley's tmf workaround suggestion should work as well for any such 'Priority set' issue.
No point in ruining my warranty if I can not get playable HD streams.As I indicated earlier, you *can* playback the HD streams.

I assume the video extraction is over an ethernet connection vs the same serial connection used for the bash prompt?Either way - however you set it up. You can be a masochist and extract at just 115Kbit/sec. over the serial port if you want.
To get an ethernet connection, do I need a USB to ethernet adaptor or can I just set up a direct network between the 2 USB 2.0 connections (PC to TIVO)?A USB 2.0 to ethernet is easiest as the drivers are already built by tivo. If you want to hack drivers for IP over USB directly I suppose you could.
That would blow away a 100mbit network since the USB 2.0 spec is 480mbits/secAs if the tivo isn't already resource constrained outside the network layer.. 13Mbit/sec is the maxthroughput I've seen over a usb2.0 dongle, yet I've been able to get more like 80Mbit/sec. with the same dongle on a linux PC.

mikemav
06-20-2004, 09:44 PM
well hrm... safest course is to extract as tmf then convert to ty (mfs_tarstream has been 100% reliable)

Okay, dumb question, I know, but how do I do this? I searched the files section and did not see the app people talk about to convert tmf to ty. Please point me in the right direction. Thanks


Nope. On the extracted .ty, from a unix machine, try:
strings <filename>.ty | grep 'Priority set'
But if riley is right, then this is a red herring and you shouldn't see any matches.

Unfortunately I do not have access to a Unix machine. Is there a way to do this in Terminal or X11 on a Mac OS-X? I have that.

bcc
06-21-2004, 12:55 AM
Unfortunately I do not have access to a Unix machine. Is there a way to do this in Terminal or X11 on a Mac OS-X? I have that.And I don't have OS-X so I don't really know. Don't unix commads, including grep, work from the OSX "Terminal"? If so, you should be set, and also you could unpack the .tmf file with:
tar xf <filename>.tmf
cat part??.ty > show.tyOtherwise, I guess I can't help you.

SR712
06-22-2004, 08:46 PM
Open the file in a hex editor and search for the string "'Priority".

Jeff D
06-23-2004, 12:22 AM
Anyone who's got a HD ty file want to send me a copy? I'm willing to look at it and see if I can help figure some of this stuff out.

If you have a small (couple minute) sample clip please PM me.

redstone
06-23-2004, 08:53 AM
Anyone who's got a HD ty file want to send me a copy? I'm willing to look at it and see if I can help figure some of this stuff out.

If you have a small (couple minute) sample clip please PM me.

ALPHAWOLF on the first page of this thread says he does but did not respond when I asked him if I could get my hands on it. Perhaps you could approach him.


"If it helps, I do have some hdtivo streams sitting on my hard disk right now. Granted nothing will play them, they are there." Is what he said.


If you get your hands on a clip, please PM me also as I have a set of video maniplution programs and would like to see how those packets are constructed.

If you read back and look for Mikemav's post in this thread, you will see that the issue is NOT being able to play them but the issue is BIG streams.


BTW, anyone given any thought to what effect there will be WHEN the 5c flag starts showing up in the data for HD programming?
I think it does more than stick a word in sets of packets. I think it actually scrambles the data differently for copy once/copy never/copy always. Not sure about that yet though.


Thanks,
Joe

malfunct
06-23-2004, 09:32 AM
BTW, anyone given any thought to what effect there will be WHEN the 5c flag starts showing up in the data for HD programming?
I think it does more than stick a word in sets of packets. I think it actually scrambles the data differently for copy once/copy never/copy always. Not sure about that yet though.

You are talking about the broadcast flag correct? I was certain that the flag was a directive but it was up to the hardware to do the right thing. Its possible that we extract tivo streams before the tivo would worry about obeying the broadcast flag and so wouldn't be bothered by it. On the other hand I think by the letter of the law the tivo would be disallowed from even recording the flagged material in the first place. I guess the bottom line is I have no clue what happens but sounds interesting. I have to get a HDTV before any of it matters to me :)

mikemav
06-23-2004, 11:31 AM
ALPHAWOLF on the first page of this thread says he does but did not respond when I asked him if I could get my hands on it. Perhaps you could approach him.


"If it helps, I do have some hdtivo streams sitting on my hard disk right now. Granted nothing will play them, they are there." Is what he said.


If you get your hands on a clip, please PM me also as I have a set of video maniplution programs and would like to see how those packets are constructed.

If you read back and look for Mikemav's post in this thread, you will see that the issue is NOT being able to play them but the issue is BIG streams.


BTW, anyone given any thought to what effect there will be WHEN the 5c flag starts showing up in the data for HD programming?
I think it does more than stick a word in sets of packets. I think it actually scrambles the data differently for copy once/copy never/copy always. Not sure about that yet though.


Thanks,
Joe

I will try to record something smaller and extract it tonight. I may be able to post it online, or may need to send it out. I will post when I have something small enough to be practical. Then again, if it is really the size that is freaking out TyTools, then a smaller clip may not be a good test. I can confirm the 700MB HDNet test pattern did process in TyTools to a .VOB that played back in full 1080i on the MyHD PC card I have. Anything larger I have tried to convert has failed.

Then again, a smaller test clip would be a good place to start with for looking at other tools. I have not had the time to get Mencoder for OS-X properly working, or for that matter to play with the .tmf/priority issue.

If I extract, should I extract .tmf, or do you want .ty+ or. ty as well? (Redstone- FYI, I know this stuff is new to you. these are the formats mfs_ftp can extract to. The mfs_ftp thread goes into the difference if you want to know.)

malfunct
06-23-2004, 12:31 PM
I will try to record something smaller and extract it tonight. I may be able to post it online, or may need to send it out. I will post when I have something small enough to be practical. Then again, if it is really the size that is freaking out TyTools, then a smaller clip may not be a good test. I can confirm the 700MB HDNet test pattern did process in TyTools to a .VOB that played back in full 1080i on the MyHD PC card I have. Anything larger I have tried to convert has failed.

Then again, a smaller test clip would be a good place to start with for looking at other tools. I have not had the time to get Mencoder for OS-X properly working, or for that matter to play with the .tmf/priority issue.

If I extract, should I extract .tmf, or do you want .ty+ or. ty as well? (Redstone- FYI, I know this stuff is new to you. these are the formats mfs_ftp can extract to. The mfs_ftp thread goes into the difference if you want to know.)

What error do you get on files larger than 700MB?

.tmf should be fine, there are great tools available for turning that into a .ty. Making .ty+ is probably unecessary as the only programs I know that use its extra info also take .tmf which is better.

bcc
06-23-2004, 01:32 PM
Here is an interesting HD ty clip. Short by HD standards - 80MB. The audio is all AC3. I probably won't leave this up long so get it now...

It is an interesting clip because TyTool (9r14) is able to start to demux it, but dies with an exception after just a few seconds of content. The resulting truncated .m2a/.m2v play fine with elecard. I also tested this clip with tystudio's tydemux, and it fails to find anything to demux. I looked into tydemux, and it fails because it has a bunch of assumptions that are no longer holding true. 1. That records in chunks are small enough that multiple video B frames will fit in a chunk (no longer true as HD records are bigger). 2. That records of type 2c0 do not have PES headers (they often do). 3. That records of type 7e0/ae0/be0 should be long enough to include the PTS header (there are lots of such records whose total length is just 4). I tried to fix these, and am now able to successfully demux AC3 audio with tydemux, but there are still too many video packet errors.

BTW, this HD .ty plays back find with mplayer (well, except lip sync).

mikemav
06-23-2004, 02:20 PM
Here is an interesting HD ty clip. Short by HD standards - 80MB. The audio is all AC3. I probably won't leave this up long so get it now...
http://trillian.mit.edu/~cole/biker.ty
It is an interesting clip because TyTool (9r14) is able to start to demux it, but dies with an exception after just a few seconds of content. The resulting truncated .m2a/.m2v play fine with elecard. BTW, this HD .ty plays back find with mplayer (well, except lip sync).

That is the same problem I am having. But I was able to get the HDNet test pattern to work (but no others.) Everything else I have tried has resulted in the same outcome as described above from TyTools. I PM'd the TyTools developer about this a while back but got no response, BTW.

Can someone try to convert it to something standard with Mencoder (the encoder for Mplayer) ?

rung
06-23-2004, 09:02 PM
Do you know off the top of your head what record types "F20" and "E0F" are?

bcc
06-23-2004, 09:10 PM
Do you know off the top of your head what record types "F20" and "E0F" are?
No, but those are 2 of the 3 new record types I've seen in limited experimenting. F20 on SD or HD broadcasts via satellite, E0E in HD broadcasts from OTA, and E0F in HD broadcasts from satellite.

bcc
06-23-2004, 09:15 PM
Can someone try to convert it to something standard with Mencoder (the encoder for Mplayer) ?With what arguments? Why don't you try? I get an AVI file that appears unreadable by anything when I use the "copy" conversion. I get better results from the -dumpvideo option of mplayer (sometimes the resulting video file is playable by elecard).

lart2150
06-23-2004, 09:16 PM
the stock mplayer comes with mencoder that can transcode any file that mplayer can play. I have used the shell script on here to reencode a ty with out any patches but it does need a custom fiter that you will need to compile (might work under cygwin). if you can get that working you should be able to re encode your videos to mpeg/xvid/divx. I don't know if it will compile under cygwin but you will need to apply the patch so it can play hd files. another thing to check is if tyshow can play the stream. if so then any file that can encode avi files should be able to convert the ty stream but it will be a reencode not a stream copy like tystudio/tytools

rung
06-23-2004, 09:26 PM
No doubt that is Hi-def: Media player reports it is 1920 x 1088 (AR 16:9)! The future is bright! :cool:

mikemav
06-23-2004, 09:39 PM
With what arguments? Why don't you try? I get an AVI file that appears unreadable by anything when I use the "copy" conversion. I get better results from the -dumpvideo option of mplayer (sometimes the resulting video file is playable by elecard).

I'd be glad to try, but I can't get mencoder for OS-X to run and I do not have a Linux box. I have never used the program before. I would like to be able to convert to the same HD resolution VOB (as the one success I had w/ TyTools did) or a high res MPEG.

I just recorded some 1 minute clips from various HD broadcasts, both OTA and DSS. I will try to extract and see if I can post somewhere to further the experimenting data pool.

mikemav
06-23-2004, 10:49 PM
Open the file in a hex editor and search for the string "'Priority".

I opened the .tmf in Crimson Editor and string "Priority" is not found. Here is what shows up (in case it matters)

showing.xml
<Object type="Recording" id="_top">
<ApgStreamType>197</ApgStreamType>
<BitRate>0</BitRate>
<SubObject type="RecordingPart" id="Part">
<Begin>0</Begin>
<CommercialSkipOffset>0</CommercialSkipOffset>
<End>56055</End>
<File>128552</File>
</SubObject>
<SelectionType>3</SelectionType>
<SubObject type="Showing" id="Showing">
<Date>12593</Date>
<Duration>3600</Duration>
<Object type="Program" id="Program">
<SubObject type="ApgProgram" id="ApgProgram">
<Category>11403269</Category>
<Category>11403638</Category>
<Category>11337834</Category>
<Category>11272316</Category>
<Category>11403584</Category>
</SubObject>
<DescLanguage>English</DescLanguage>
<Description>A man works on 14 custom cars.</Description>
<EpisodeTitle>Dominator</EpisodeTitle>
<Genre>5</Genre>
<Genre>374</Genre>
<Genre>106</Genre>
<Genre>124</Genre>
<Genre>320</Genre>
<IsEpisode>1</IsEpisode>
<OriginalAirDate>12486</OriginalAirDate>
<RootServerId>A001f88a0</RootServerId>
<Object type="Series" id="Series">
<Genre>5</Genre>
<Genre>374</Genre>
<Genre>105</Genre>
<Genre>124</Genre>
<Genre>320</Genre>
<ServerId>ATSH626160</ServerId>
<ServerVersion>0</ServerVersion>
<ThumbData>268566912</ThumbData>
<Title>Rides</Title>
<TmsId>SH626160</TmsId>
</Object>
<ServerId>A00208044</ServerId>
<ServerVersion>1</ServerVersion>
<ShowType>5</ShowType>
<Title>Rides</Title>
<TmsId>EP6261600009</TmsId>
<TvRating>3</TvRating>
</Object>
<Reason>2</Reason>
<Object type="Station" id="Station">
<CallSign>DSHD</CallSign>
<Name>DSHD</Name>
<ServerId>AStation31046</ServerId>
<ServerVersion>23</ServerVersion>
<TmsId>31046</TmsId>
</Object>
<Time>3600</Time>
</SubObject>
<StartDate>12593</StartDate>
<StartTime>6098</StartTime>
<StopDate>12593</StopDate>
<StopTime>6154</StopTime>
<StreamFileSize>102400</StreamFileSize>
<TunedSignalType>6</TunedSignalType>
</Object>

mikemav
06-23-2004, 11:09 PM
One of my extractions tonight is a 66MB (~1 min) clip from HBOH. Got the .ty, the .tmf and the .ty+ from mfs_ftp. The .ty and the .ty+ both just converted to hi-def .VOB with TyTools9r14 using the VOB-MUX (New Format 1) option.

However, when I tried the same from a clip from Law and Order (OTA HD), I got an exception error. Details from the resulting error .txt file on Law and Order are below in case they mean anything. Also, I tried Rides (Discovery HD) and it too did not work, resulting in a exception error almost immediately. BTW, TyTools said Rides was Dolby Digital, and Law & Order was MPEG Layer II in the opening report. EDIT- it is Rides that is MPEGII audio and the NBC program was DD audio. Sorry. I have not tried pulling out the .ty from the .tmf yet since I am still transferring the files to my Mac, where I hope I can use the Terminal to do this. What is the Windows tool to convert .tmf to .ty? I searched but could not find it, only references to the program.

I will try to post some of these files for people to play with.

Here is the TyTools error .txt:

<?xml version="1.0" encoding="UTF-16"?>
<DATABASE>
<EXE NAME="TyTool9r14.exe" FILTER="GRABMI_FILTER_PRIVACY">
<MATCHING_FILE NAME="bmp2y4m.exe" SIZE="65536" CHECKSUM="0x7A46A652" MODULE_TYPE="WIN32" PE_CHECKSUM="0x0" LINKER_VERSION="0x0" LINK_DATE="05/27/2003 20:47:47" UPTO_LINK_DATE="05/27/2003 20:47:47" />
<MATCHING_FILE NAME="cygwin1.dll" SIZE="971618" CHECKSUM="0x89BD76EF" BIN_FILE_VERSION="1005.5.0.0" BIN_PRODUCT_VERSION="1005.5.0.0" PRODUCT_VERSION="1.5.5-cr-0x9b" FILE_DESCRIPTION="Cygwin® POSIX Emulation DLL" COMPANY_NAME="Red Hat" PRODUCT_NAME="Cygwin" FILE_VERSION="1.5.5-cr-0x9b" ORIGINAL_FILENAME="cygwin1.dll" INTERNAL_NAME="cygwin1.dll" LEGAL_COPYRIGHT="Copyright © Red Hat, Inc. 1996-2003" VERFILEDATEHI="0x0" VERFILEDATELO="0x0" VERFILEOS="0x4" VERFILETYPE="0x2" MODULE_TYPE="WIN32" PE_CHECKSUM="0xF69EE" LINKER_VERSION="0x10000" UPTO_BIN_FILE_VERSION="1005.5.0.0" UPTO_BIN_PRODUCT_VERSION="1005.5.0.0" LINK_DATE="09/20/2003 20:32:36" UPTO_LINK_DATE="09/20/2003 20:32:36" VER_LANGUAGE="English (United States) [0x409]" />
<MATCHING_FILE NAME="dvdauthor.exe" SIZE="82512" CHECKSUM="0x5A493D21" MODULE_TYPE="WIN32" PE_CHECKSUM="0x1DEB4" LINKER_VERSION="0x10000" LINK_DATE="09/22/2003 13:05:20" UPTO_LINK_DATE="09/22/2003 13:05:20" />
<MATCHING_FILE NAME="GopEditor.exe" SIZE="147456" CHECKSUM="0xC13D6BCA" MODULE_TYPE="WIN32" PE_CHECKSUM="0x0" LINKER_VERSION="0x0" LINK_DATE="04/24/2004 07:29:27" UPTO_LINK_DATE="04/24/2004 07:29:27" />
<MATCHING_FILE NAME="libmpeg2.dll" SIZE="98304" CHECKSUM="0x98458131" MODULE_TYPE="WIN32" PE_CHECKSUM="0x0" LINKER_VERSION="0x0" LINK_DATE="02/20/2004 21:57:35" UPTO_LINK_DATE="02/20/2004 21:57:35" />
<MATCHING_FILE NAME="mpeg2enc.exe" SIZE="126976" CHECKSUM="0xBEA50A20" MODULE_TYPE="WIN32" PE_CHECKSUM="0x26294" LINKER_VERSION="0x10000" LINK_DATE="02/21/2004 02:53:33" UPTO_LINK_DATE="02/21/2004 02:53:33" />
<MATCHING_FILE NAME="MuxMenu.exe" SIZE="61440" CHECKSUM="0x2E52A949" MODULE_TYPE="WIN32" PE_CHECKSUM="0x0" LINKER_VERSION="0x0" LINK_DATE="08/03/2003 20:18:00" UPTO_LINK_DATE="08/03/2003 20:18:00" />
<MATCHING_FILE NAME="TyFileSplit.exe" SIZE="45056" CHECKSUM="0x93DDCF9" MODULE_TYPE="WIN32" PE_CHECKSUM="0x0" LINKER_VERSION="0x0" LINK_DATE="12/24/2003 22:10:50" UPTO_LINK_DATE="12/24/2003 22:10:50" />
<MATCHING_FILE NAME="TyTool9r14.exe" SIZE="307200" CHECKSUM="0x99ADEF95" MODULE_TYPE="WIN32" PE_CHECKSUM="0x0" LINKER_VERSION="0x0" LINK_DATE="04/24/2004 07:28:50" UPTO_LINK_DATE="04/24/2004 07:28:50" />
<MATCHING_FILE NAME="TyTranscodeDLL.dll" SIZE="258048" CHECKSUM="0xE52A740D" MODULE_TYPE="WIN32" PE_CHECKSUM="0x0" LINKER_VERSION="0x0" LINK_DATE="06/24/2003 02:14:58" UPTO_LINK_DATE="06/24/2003 02:14:58" />
</EXE>
<EXE NAME="kernel32.dll" FILTER="GRABMI_FILTER_THISFILEONLY">
<MATCHING_FILE NAME="kernel32.dll" SIZE="930304" CHECKSUM="0xCBCCF8A9" BIN_FILE_VERSION="5.1.2600.1106" BIN_PRODUCT_VERSION="5.1.2600.1106" PRODUCT_VERSION="5.1.2600.1106" FILE_DESCRIPTION="Windows NT BASE API Client DLL" COMPANY_NAME="Microsoft Corporation" PRODUCT_NAME="Microsoft® Windows® Operating System" FILE_VERSION="5.1.2600.1106 (xpsp1.020828-1920)" ORIGINAL_FILENAME="kernel32" INTERNAL_NAME="kernel32" LEGAL_COPYRIGHT="© Microsoft Corporation. All rights reserved." VERFILEDATEHI="0x0" VERFILEDATELO="0x0" VERFILEOS="0x40004" VERFILETYPE="0x2" MODULE_TYPE="WIN32" PE_CHECKSUM="0xE7ED3" LINKER_VERSION="0x50001" UPTO_BIN_FILE_VERSION="5.1.2600.1106" UPTO_BIN_PRODUCT_VERSION="5.1.2600.1106" LINK_DATE="08/29/2002 10:40:40" UPTO_LINK_DATE="08/29/2002 10:40:40" VER_LANGUAGE="English (United States) [0x409]" />
</EXE>
</DATABASE>

bcc
06-24-2004, 12:12 AM
I opened the .tmf in Crimson Editor and string "Priority" is not found.The problem case involves longer recordings that span multiple FSIDs, and where you use .ty or .ty+ file types. So this result is expected.

mikemav
06-24-2004, 12:18 AM
The problem case involves longer recordings that span multiple FSIDs, and where you use .ty or .ty+ file types. So this result is expected.
Does that mean we have eliminated the Priority issue from the potential problem list, or are you saying that longer recordings (multiple FSID's) cannot be converted at all that you know of? The HDNet one I had success with was 10 minutes long, 688MB, so it may have spanned multiple FSIDs. I can check if you tell me where to look.

The three program clips I extracted are 66MB (HBOH, which is the one I got to convert to HD .VOB), and the two that failed are 100 and 150MB respectively. I do not have the web space to post any of them. I only have about 25MB free on my site. So at this time I cannot post any files for people to play with. I will keep working on them though.

bcc
06-24-2004, 12:23 AM
the stock mplayer comes with mencoder that can transcode any file that mplayer can play. I have used the shell script on here to reencode a ty with out any patches but it does need a custom fiter that you will need to compile (might work under cygwin). if you can get that working you should be able to re encode your videos to mpeg/xvid/divx. I don't know if it will compile under cygwin but you will need to apply the patch so it can play hd files. another thing to check is if tyshow can play the stream. if so then any file that can encode avi files should be able to convert the ty stream but it will be a reencode not a stream copy like tystudio/tytoolsBy shell script and filter I assume you mean the tivo2divx software? What patch are you referring to? I tried tivo2divx and the resulting avi plays back with mplayer, but the AC3 audio won't play back under windows with anything I tried (wm9, elecard, powerdvd).

bcc
06-24-2004, 12:35 AM
Does that mean we have eliminated the Priority issue from the potential problem list, or are you saying that longer recordings (multiple FSID's) cannot be converted at all that you know of? The HDNet one I had success with was 10 minutes long, 688MB, so it may have spanned multiple FSIDs. I can check if you tell me where to look. If you use tmf file types, we believe the problem doesn't occur. If you use ty/ty+, then the 'Priority set' string may pollute your recordings at the boundaries between FSIDs. This boundary is typically at each 512MB, but may occur more frequently (if your tivo disk was nearly full at recording time I believe). If your 688MB extraction was a ty/ty+ file, and does not include any 'Priority set' strings, then you can rule out this issue.

mikemav
06-24-2004, 08:16 AM
If you use tmf file types, we believe the problem doesn't occur. If you use ty/ty+, then the 'Priority set' string may pollute your recordings at the boundaries between FSIDs. This boundary is typically at each 512MB, but may occur more frequently (if your tivo disk was nearly full at recording time I believe). If your 688MB extraction was a ty/ty+ file, and does not include any 'Priority set' strings, then you can rule out this issue.

Now we may be getting somewhere. When I tried to open the HDNet test pattern ty+ in Crimson Editor, I did not see anything except one or two characters listed. Of course, "Priority" was not found. I have not used CE much, so I must have done something wrong. Anyway, I just opened the same fine in Console on the Mac OS-X, and this is what I got when I searched for "Priority"
===== Display starts at offset 721292187 within file of length 721423259. ====
===== Use File->Reload (Cmd-R) to display more. ====
˜"?Ÿ•v™¶ÖNp:? THE A BUNCH OF ASCII(?) CHARACTERS THAT WON'T PROPERLY COPY HERE.....THEN:
}}ûtPriority set...

Now since it found "Priority Set" in a file (over 512MB) that DID process to a playable .VOB in TyTools, what does that tell us? And yet most the other files I have will not process.

So how do I convert a .tmf I got from mfs_ftp to a .ty? I can try that too. Like I mentioned, I searched the files section but could not find the app to do this in Windows.

malfunct
06-24-2004, 12:38 PM
Now we may be getting somewhere. When I tried to open the HDNet test pattern ty+ in Crimson Editor, I did not see anything except one or two characters listed. Of course, "Priority" was not found. I have not used CE much, so I must have done something wrong. Anyway, I just opened the same fine in Console on the Mac OS-X, and this is what I got when I searched for "Priority"
===== Display starts at offset 721292187 within file of length 721423259. ====
===== Use File->Reload (Cmd-R) to display more. ====
˜"?Ÿ•v™¶ÖNp:? THE A BUNCH OF ASCII(?) CHARACTERS THAT WON'T PROPERLY COPY HERE.....THEN:
}}ûtPriority set...

Now since it found "Priority Set" in a file (over 512MB) that DID process to a playable .VOB in TyTools, what does that tell us? And yet most the other files I have will not process.

So how do I convert a .tmf I got from mfs_ftp to a .ty? I can try that too. Like I mentioned, I searched the files section but could not find the app to do this in Windows.

There is a conversion script, or you could do it the manual way and untar the file and concatenate the fsid's in order.

mikemav
06-24-2004, 12:53 PM
There is a conversion script, or you could do it the manual way and untar the file and concatenate the fsid's in order.

I read about TMFtoTy (for Windows) but have not been able to find it. Can anyone point me in the right direction, or elaborate on the conversion script? I will be glad to spend the time to try to track down the cause of the error, but being a newbie I need a little guidance too. Thanks

PS- I am also working a potential ftp site to upload these clips. Will post if I can.

malfunct
06-24-2004, 01:03 PM
I read about TMFtoTy (for Windows) but have not been able to find it. Can anyone point me in the right direction, or elaborate on the conversion script? I will be glad to spend the time to try to track down the cause of the error, but being a newbie I need a little guidance too. Thanks

PS- I am also working a potential ftp site to upload these clips. Will post if I can.

You might want to check alt.org for more info about the .tmf format. Its basically a tar of files, each equal to a single fsid of tystream plus one xml file to describe the other MFS info related to that file. To get to a .ty file you just untar and splice together (concatenate, use cat in linux or copy in windows) the fsid's in order.

lart2150
06-24-2004, 01:51 PM
By shell script and filter I assume you mean the tivo2divx software? What patch are you referring to? I tried tivo2divx and the resulting avi plays back with mplayer, but the AC3 audio won't play back under windows with anything I tried (wm9, elecard, powerdvd).

that's want I was talking about. do you have a ac3 direct show filter? you might want to try installing http://ac3filter.sourceforge.net/

you might also want to edit the script a little. remove
-vop pp=fd,scale=${SCALE}
and
-vf pp=fd,scale=${SCALE}

or edit what scale is.

bcc
06-24-2004, 05:24 PM
Now since it found "Priority Set" in a file (over 512MB) that DID process to a playable .VOB in TyTools, what does that tell us? And yet most the other files I have will not process. Great, so you've confirmed that your extraction tools have this problem. Now if you go back earlier in this thread, I posted a link to a fix:
http://www.dealdatabase.com/forum/showpost.php?p=170666&postcount=27
Install those binaries in your mfs_ftp directory, and you should no longer get 'Priority set' corruption, and so you won't need to use tmf file transfers to work around it.

As for TyTools, I'm not sure. Maybe it has error correction for this stream corruption, or maybe it stopped demuxing at 512MB and you didn't notice? Other tools, such as mplayer choke when they hit the "Priority set".

So how do I convert a .tmf I got from mfs_ftp to a .ty? I can try that too. Like I mentioned, I searched the files section but could not find the app to do this in Windows.Probably the best answer is don't bother as .tmf is not necessary if you install the above fix. But I did show you how to do this manually earlier in this thread:
http://www.dealdatabase.com/forum/showpost.php?p=170702&postcount=29
Also the tmf2ty windows program is linked to right at the top of the main mfs_ftp thread:
http://www.dealdatabase.com/forum/showpost.php?p=83880&postcount=1

bcc
06-27-2004, 07:57 PM
that's want I was talking about. do you have a ac3 direct show filter? you might want to try installing http://ac3filter.sourceforge.net/Thanks, the mencoder script saves a lot of trouble learning mencoder cryptic arguments. And idea how I tell for sure if I have an ac3 direct show filter? I have at least 3 ac3 decoders on my htpc (powerdvd, windvd, myhd), so I was hoping I didn't need another to playback this stream. In fact, the windvd ac3 decoder plays back the ac3 audio if I just demux it instead of trying to use mencoder.

lart2150
06-28-2004, 02:00 PM
you can use graphedit to see if windows is trying any of your ac3 filters. I think you can also make windows use one of the filters you have.

http://www.digital-digest.com/dvd/downloads/graphedit.html

redstone
06-28-2004, 05:09 PM
Please see:
http://www.dealdatabase.com/forum/showthread.php?p=171773#post171773

for a .ty clip from the HD-TIVO courtesy of mikemav that crashes tytool.

Thanks

lart2150
06-28-2004, 08:05 PM
is the audio ac3?

bcc
06-28-2004, 08:19 PM
Please see:
http://www.dealdatabase.com/forum/showthread.php?p=171773#post171773

for a .ty clip from the HD-TIVO courtesy of mikemav that crashes tytool....just like the biker.ty clip from a few days back. mplayer/mencoder can still successfully demux it however.

bcc
06-28-2004, 08:52 PM
you can use graphedit to see if windows is trying any of your ac3 filters. I think you can also make windows use one of the filters you have.Thanks, I can't figure out how to get the "AVI Splitter" box to link to a "CyberLink Audio Decoder" box with graphedit. Oh well, this is probably getting off topic and I'm thinking one should just remux to .mpg or a vob instead of trying to deal with an odd format .avi.

bcc
06-28-2004, 08:55 PM
is the audio ac3?Yes, tho TyTool will fail in a similar fashion if the audio is mpeg2. In fact, the ac3 case seems simpler in that it plays back with mplayer without requiring a patch.

mikemav
06-28-2004, 09:33 PM
I can now confirm that the fix mentioned above w/ new binaries for mfs_ftp have fixed the Priority Set issue. I extracted a short ty+ clip, then I updated the binaries, re-extracted, and searched both for the Priority string. #1 had it, the later one did not. Unfortunately, neither processed in TyTools, so I am not sure where we are there. I would really love someone to be able to look at the similar clip I had the other day that DID process in TyTools to see what is different about it. I have asked Redstone if he has the space to post that one as well, but I know ftp space is difficult.

redstone
06-28-2004, 11:07 PM
I have asked Redstone if he has the space to post that one as well, but I know ftp space is difficult.

It will be posted for download tomorrow.

I have a limit of 1.1 GB of disk space so I have to manage how many of these clips that mikemav is giving me.
They are all backup up to a DVD of course.

lart2150
06-29-2004, 12:25 AM
that ty file lawandorder.ty plays for me with tyshow... I need more cpu but I get video and audio.

bcc
06-29-2004, 01:09 AM
I was just able to demux a couple HD .ty and remux them with mjpeg tools to .mpg. The .mpg then played fine with MYHD at 1080i with dolby digital audio. These same .ty files would not demux with TyTool. I think I introduced some lip sync problems, but it's hard to tell as MYHD has lip sync issues anyways :)

mikemav
06-29-2004, 07:44 AM
I was just able to demux a couple HD .ty and remux them with mjpeg tools to .mpg. The .mpg then played fine with MYHD at 1080i with dolby digital audio. These same .ty files would not demux with TyTool. I think I introduced some lip sync problems, but it's hard to tell as MYHD has lip sync issues anyways :)

Could you describe the process and what tools you used? Any custom filters, program customizations, etc..? What settings? Sorry for the infant ?'s, but all I have ever used before is TyTools. Has anyone played w/ a Mac OS-X version? I see from the sourceforge web page that is can be complied for OS-X. I will read up some more. This is promising. Thanks.

bcc
06-29-2004, 03:38 PM
I wrote my own ty stream demux program, and then I'm feeding the result to mjpegtools' (http://mjpeg.sf.net) mplex program. Basically,
demux -i <show>.ty -v <show>.m2v -a <show>.m2a
mplex -r <data-rate> -O <delay> -f 8 <show>.m2a <show>.m2v -o <show>.mpgwhere,
<data-rate> is the data rate of the show. Currently I'm just setting this to what mplex is reporting as the "rough-guess multiplexed stream data rate", just to shut it up.
<delay> is the a/v timestamp delta, as reported from the demux stage
For example
demux -i biker.ty -v biker.m2v -a biker.m2a
Audio timestamp offset: 798.777778 ms
mplex -r 46325 -O 799ms -f 8 biker.m2a biker.m2v -o biker.mpg
My program is alpha quality (any errors in the ty stream and it'll probably blow up instead of correcting the a/v discontinuities). Could probably be spruced up&released if TyTools fixes for HD are not forthcoming.

redstone
06-29-2004, 06:26 PM
Mikemav has given me quite a number of extracted clips plus I have the biker.ty file that was posted.
Some process (like the biker.ty file) by tytools and others merely make tytools crash.

The biker.ty file,while processable by tytools, has the lip sync issue.

How well is your method doing with that audio issue?


Here is a clip from Mikemav that processes sucessfully by tytool that you may also find useful:

http://southriver.web.aplus.net/

bcc
06-30-2004, 03:21 AM
How well is your method doing with that audio issue?I wish I could say I have audio sync nailed, but I get confusing, inconsistent results. Actually, I always have seen off&on audio sync issues when watching content via an HTPC (with MYHD) or with mplayer. Maybe you can point me to something where the lip sync is rock solid? Wish there was an audio-sync test (like HDNET's test pattern) that I could use.

With that 'six feet under' clip, if I mux it to an .mpg with mplex, without using the -O argument to adjust the delay, then the audio is late (almost a second). When I add the -O argument, using the 775ms value reported by my demux program, then the audio sync is OK. By OK I mean that the sync doesn't look perfect but my brain can't place how it's off (early or late audio), as it's very close. At any rate, I get the same result when I just play the .ty file directly. The above tests are with mplayer, and I suspect mplayer isn't accounting for playback latency. Anyways, if I take the same .mpg and play it back with MYHD, the audio also looks OK. Same goes for windows media player9, using powerdvd's ac3 decoding and spdif. However, if powerdvd's ac3 decoder is set to 2-channel, the audio now comes in *early* (almost a second). The 2-channel audio is still going via spdif physically... Ok, so how can I tell when the audio sync issue is due to the content, the encoder, the decoder, or the playback device? It *seems* like mplex with -O is doing the right thing in this case.

Then there is biker.ty that always seems out of sync - except it plays back fine on the tivo. The clip is from the very start of the FSID, including the header chunk...

mikemav
06-30-2004, 06:02 AM
Now that some of you have access to short clips I pulled off, some of which process in TyTools (Six Feet Under) and some that do not, I wonder what is different about these clips? Can jdiner or anyone else familiar enough with these formats tell? I wish I was more of a programmer and could help more, but I will be glad to keep providing samples of whatever combo is needed to help figure this out. It also still strikes me as odd that I was able to get an over 512MB version of a .ty+ to process to .vob properly in TyTool (that HDNet program), and that all of my successes (while few and far between) were before fixing the Priority Set issue w/ new binaries in mfs_ftp. Unfortunately those new binaries do not lead to files that now process. The luck on that seems to be as random as before.

BTW, thanks Redstone for hosting the clips!

jdiner
06-30-2004, 10:52 AM
Alright. I have finally collected a few HDTivo streams. I think I already know what was wrong with TyTool and these things. Basically it all comes down to buffer sizes. I made some assumptions that are categorically wrong on the HDTivos now.

I will post more as I get more figured out.

--jdiner

mikemav
06-30-2004, 11:13 AM
Alright. I have finally collected a few HDTivo streams. I think I already know what was wrong with TyTool and these things. Basically it all comes down to buffer sizes. I made some assumptions that are categorically wrong on the HDTivos now.

I will post more as I get more figured out.

--jdiner

Thanks jdiner! Appreciate your help.

redstone
06-30-2004, 07:48 PM
Thanks jdiner! Appreciate your help.

Me too cause your sucess determines whether I send this to a member in here to ruin the warranty on my $1000 TIVO by having the prom socketed:)

I know a gentlemnan who is a well respected contributor to avsforum and is an expert on all things video. He is a retired engineer who is often asked by TV stations to figure out why their HDTV broadcasts are giving some STB's fits and hence complaints from the local viewers. He always can track it down to the equipment at the station being misconfigured and sending out non standard ATSC data.

He is NOT a software guy.

To the point:


I had him look at a couple of the .vob file's as to why the audio sync problem (I use the MYHD PC HdTV card) and here was his comment(SFU is six feet under):

"I uploaded SFU and ran the TP stream checker (checktp.exe). The TP checker will show you the PIDs contained within the program stream(s) and other problems like bad sync bytes.

Notice that with SFU and Biker streams HDTVtoMPEG2 can't ID and display the video/audio PIDs because they don't follow the normal ATSC standards for MPEG2 tables. MyHD is able to play the file because when it finds errors in the tables (PAT/PMT) it searches the TS for audio/video packets. Of coures searching the stream will disable some features because it has no ref to PES time stamps."

I really don't follow what he was saying but perhaps that info could help you out.

If you need either of these debug tools, please give a hollow and I will make them available since he did send them to me so I could run them.

Thanks for all your efforts

redstone
06-30-2004, 09:31 PM
Maybe you can point me to something where the lip sync is rock solid?


I am in the middle of mucking around with my video switcher so I can not play the RGBHV from my Radeon on my HDTV just now where I can really spot lip sync issues because the video is always a LOT darker on my PC Monitor.
Heck, I didn't spot the lip sync issue on my PC monitor with biker.ty until I processed it with tytool and played it through MYHD and saw it on my TV. Ouch!

However, it 'looks' like "law and order" has proper lip sync on my PC monitor when I play it with mplayer.

You may want to see for yourself.

Here is the file:
http://southriver.web.aplus.net/



If you want to try the file, I would wait until around 10:30 EDT to download because I am uploading it now and it is
150 MB (157,290,496 bytes) so it won't be done until around 10:30.

(I am on a cable modem and my uplink speed is limited to 128)

I think this is a good test file so I am going to leave it on for awhile so I hope you are saving the downloads as I am limited to 1.1 GB total on this ftp site so I delete stuff when Jdiner tells me he has them.

Heck, it is only $9.95/month for this hosting site:)

bcc
07-01-2004, 02:03 AM
He always can track it down to the equipment at the station being misconfigured and sending out non standard ATSC data.Hopefully directv will do better broadcasting HD national feeds than mom&pop stations are.:)
Notice that with SFU and Biker streams HDTVtoMPEG2 can't ID and display the video/audio PIDs because they don't follow the normal ATSC standards for MPEG2 tables.Is he looking at the original .ty streams or .vob files you made with TyTool? With the .ty streams, HDTVtoMPEG2 should fail as it won't know how to decode the Tivo chunk headers/record length information. I'm surprised that it works at all. I just tried it and it produces an output file that loses audio sync after a bit and has a lot of video artificating. And yes, in the .ty files the PES headers are sub-standard, but they do include PTS timestamps, which I'm processing. I hadn't thought of testing against HDTVtoMPEG2 - it is able to convert my .mpg back to a .ts without complaining, but some video artifacting is introduced then as well. Will look into it...
If you need either of these debug tools, please give a hollow and I will make them available since he did send them to me so I could run them.Cool, I'd be interested in trying that checktp program.
Heck, I didn't spot the lip sync issue on my PC monitor with biker.ty until I processed it with tytool and played it through MYHD and saw it on my TV. Ouch!Actually I checked that show again and the lip sync doesn't look great on my tivo. It really isn't worse with mplayer or when I demux&remux it (unless I leave out the -O audio delay adjustment).
However, it 'looks' like "law and order" has proper lip sync on my PC monitor when I play it with mplayer.When the curly haired guy starts talking at the beginning, his audio plays back before his lips finish moving - with mplayer playing back the .ty file. The audio *seems* to sync up better further on... I'm using
mplayer -framedrop -vo xvidix -vf crop=1589:1080 law.tywith a radeon 9600 and 3ghz...

redstone
07-01-2004, 10:30 AM
Is he looking at the original .ty streams or .vob files you made with TyTool? ...

VOB file of course. This fellows knows nada nor cares about the way the TIVO formats its data internally. He deals soley with open standards - ATSC,MPEG2,etc.

I merely asked him to take a look and tell me what he thought.

Re your other item,
Look here:
http://southriver.web.aplus.net/


for CheckTP.zip

redstone
07-10-2004, 08:15 PM
I have been following jdiner's tytool and see that he has made a number of changes that are being tested by some beta testers.

Question: Does anyone know if any of these changes being tested are in support of the HD-TIVO?

Thanks.

fixn278
07-10-2004, 08:28 PM
I have been following jdiner's tytool and see that he has made a number of changes that are being tested by some beta testers.

Question: Does anyone know if any of these changes being tested are in support of the HD-TIVO?

Thanks.

http://www.dealdatabase.com/forum/showpost.php?p=171966&postcount=72

As per an earlier post in this thread, he will let us know when he makes headway.

redstone
07-13-2004, 04:56 PM
Actually I checked that show again and the lip sync doesn't look great on my tivo. It really isn't worse with mplayer or when I demux&remux it (unless I leave out the -O audio delay adjustment).When the curly haired guy starts talking at the beginning, his audio plays back before his lips finish moving - with mplayer playing back the .ty file. The audio *seems* to sync up better further on... I'm using
mplayer -framedrop -vo xvidix -vf crop=1589:1080 law.tywith a radeon 9600 and 3ghz...

Since it looks like tytool may never be updated to process the .ty files from the HD-TIVO, I am looking at alternative ways of processing the .ty files.

First step (thanks to you cause I never heard of mplayer until this post), I have been experimenting with merely playing the various .ty files from the HD-TIVO.
I have been using mplayer on a system with a Radeon 9500 and a 2.8 GHz P4.
BTW, I found a GUI version that makes it a lot easier to use.

Question:
On the law and order clip, do you see what looks like Mpeg compression artifacts?

Example - look at the guy who has a striped shirt collar . The collar 'rings'.

I am wondering whether this is a CPU speed issue, a codec issue or what.

I am using the GUI version of the X86 mplayer and feeding "-framedrop " as the argument and my Radeon is feeding the video to my RP HDTV.

For those that want to see, the clip is here:
http://southriver.web.aplus.net/

I also have saved the Windows GUI version of Mplayer on my site but am not sure if posting it for download violates any OPen Source rules.

- JQ

bcc
07-13-2004, 08:20 PM
Since it looks like tytool may never be updated to process the .ty files from the HD-TIVO, I am looking at alternative ways of processing the .ty files.Have you tried tyshow? I haven't been able to get it to work with HD recordings (video stays paused at beginning, and can't manage to get the AC3 codecs to get used, even with graphedit). Just curious if anyone else has actually gotten it to work. You could of course archive your .ty+/tmf directly, and playback on your hd tivo...
On the law and order clip, do you see what looks like Mpeg compression artifacts? Example - look at the guy who has a striped shirt collar . The collar 'rings'.I see no digital artifacts. The stripes on his collar look solid to me. Slightly fuzzy, if you stare closely as his body bobs around, but that's it. Sounds like you may be causing the video artificating by dropping frames. Too bad his lips are pretty out of sync with his voice (audio comes in late) as he starts with "oh, silly me, I thought crucial meant". Seems that mplayer only syncs up the audio after lots of frames. Tyshow handled this a lot better by pausing the video at the beginning to let the audio catch up (when I played SD content).
I also have saved the Windows GUI version of Mplayer on my site but am not sure if posting it for download violates any OPen Source rules.Troubleshooting windoze codec issues gives me a headache, so I've been using linux mplayer. Is this gui version different than the windows versions at www.mplayerhq.hu?
PS: Testing with hdtvtompeg2 doesn't do any good, as hdtvtompeg2 assumes the source is an actual transport stream. When fed a .ty file, which isn't structured as transport stream packets, hdtvtompeg2 merely corrupts the stream with transport stream sync bytes, and silently fails to recognize anything.

moshmothma
07-14-2004, 08:32 AM
Hi, Redstone, I think Jdiner is actually looking at the HDTivo clips. I was hoping the next version had support but we'll see.
Also, could you check your Private messages. Thanks

Darwin23
07-14-2004, 02:04 PM
Since it looks like tytool may never be updated to process the .ty files from the HD-TIVO, I am looking at alternative ways of processing the .ty files.

First step (thanks to you cause I never heard of mplayer until this post), I have been experimenting with merely playing the various .ty files from the HD-TIVO.
I have been using mplayer on a system with a Radeon 9500 and a 2.8 GHz P4.
BTW, I found a GUI version that makes it a lot easier to use.

Question:
On the law and order clip, do you see what looks like Mpeg compression artifacts?

Example - look at the guy who has a striped shirt collar . The collar 'rings'.

I am wondering whether this is a CPU speed issue, a codec issue or what.

I am using the GUI version of the X86 mplayer and feeding "-framedrop " as the argument and my Radeon is feeding the video to my RP HDTV.

For those that want to see, the clip is here:
http://southriver.web.aplus.net/

I also have saved the Windows GUI version of Mplayer on my site but am not sure if posting it for download violates any OPen Source rules.

- JQ

mplayer should be able to "pipe" video to ffmpeg or mpeg2enc thru "null" video output

transcode can handle this, something like:

transcode -i your.ty -V -x mplayer,mplayer -y ffmpeg -F d -Z 1920x1080 --export_fps 30 --export_asr 2 -E 48000 -o output.mpg

Now you would be trancoding the .ty so it is not lossless, but setting the bit-rate of ffmpeg to 17000kb/s and HQ encoding two-pass seems to give HD qualty mpeg2 video@ 1920x1080.

I've not been able to process a HD.ty this way, but it SHOULD work once you set all the switches correct in the ARG.

I have been able to process some 720p and 1080i uncomepressed.avi files I have and encode them with ffmpeg to mpeg2@ 720p and 1080i 9,000kb/s(720p) - 17,000kb/s(1080i) with ac3 audio.

Made a few Hi-Res HQ DVD's(won't call them HD) they have menus and such ,but only VLC can handle them propperly.

Hi-Res HQ DVD my own stop gap mesure while waiting for HD-DVD's and Players.

bcc
07-14-2004, 02:05 PM
If you're bored, check out It is what I get when I demux your law&order clip and remux with mjpegtools' mplex. I think mplex is having trouble synthesizing correct timestamps (SCR values), but the lip sync looks good, even with MYHD.

jdiner
07-14-2004, 02:16 PM
Took a look at one of the first HD Tivo clips I was sent. People were claiming it was on. It is not. There is just no "people talking" where we can see the face and as a result it is late but appears to be correct.

I have found what I think is the reason for the A/V offset. But I need a good HD mpeg file to compare a few things with. Which means not from the tivo but actually a decent working Hi-Def MPEG-2 clip.

I don't need much of one. 8-10 meg would surfice. I really need to see what changed were made to the packaging to get things to work. If anyone has one post a link here or PM me directly with a link and I will download it.

If someone has one but no way to post it please PM me with a request and I will pass out some private FTP information.

--jdiner

bcc
07-14-2004, 02:35 PM
What would you consider decent? How about a MYHD transport stream converted to mpeg with hdtvtompeg2? I take it you don't want the mpeg2 packged as a transport stream? 8-10MB will only give you around 6 seconds of programming.

jdiner
07-14-2004, 03:17 PM
What would you consider decent? How about a MYHD transport stream converted to mpeg with hdtvtompeg2? I take it you don't want the mpeg2 packged as a transport stream? 8-10MB will only give you around 6 seconds of programming.

Decent means plays without errors or gaps or skips or pops in that small segment. I knew it would be short but that might be a bit short. How about 25MB then.

The key is that I want to see the packing of the data and not the data itself. Honestly the data itself doesn't matter for what I am looking at right now.

One of the problems I have seen is that things are much much larger and that is affecting the creation of the PACK and PES headers.

The rest of the mechanics are pretty much the same. Some of what I need to see might be in that hdtvtompeg2 program. Is the source available for that?

What I need is to know if the SCR rate has changed from one format to the other. It appears that things are going to have to have changed to some extent as the data is "too large" to fit properly. But if you shrink the offset in the SCR you in essence get more through in that measured time. I just need to see what should be used and I can pull that out of a stream by looking at the data or I can pull that value from the code. Either way works.

--jdiner

bcc
07-14-2004, 11:38 PM
One of the problems I have seen is that things are much much larger and that is affecting the creation of the PACK and PES headers.Exactly. Mjpegtools seems to be hardcoded to use 232K video buffers for DVD authoring, but that buffer doesn't necessarily fit a whole frame. Its demux program readily thinks there is a video stream underflow seemingly because of this. I had to hack its SCR vs DTS check just to get it to process the law&order clip. Probably demux should really be using the buffer size that the stream indicates, but it claims to be using a "standard conforming value"... Strangely, lying to demux about the data rate makes no difference in the byte position where it thinks there is an underflow... What do you think, are you using the same algorithm? Are you making use of the Tivo chunk timestamps from the chunk record headers? Anyways, here you go: http://trillian.mit.edu/~cole/janet.mpg Highlight from last superbowl:)
Some of what I need to see might be in that hdtvtompeg2 program. Is the source available for that?There is source at the author's web page http://www.midwinter.com/~bcooley/. I have newer source than what is posted there for some reason. It's OK to post according to the copyright, so here you go - copyright (C) Ben Cooley 2002
What I need is to know if the SCR rate has changed from one format to the other. It appears that things are going to have to have changed to some extent as the data is "too large" to fit properly. But if you shrink the offset in the SCR you in essence get more through in that measured time. I just need to see what should be used and I can pull that out of a stream by looking at the data or I can pull that value from the code. Either way works.Good idea; sounds like looking at the above mpg's SCR values would be easiest.

rung
07-15-2004, 04:11 AM
Anyways, here you go: http://trillian.mit.edu/~cole/janet.mpg Highlight from last superbowl:) Very funny! They could use that clip to sell HDTV's. "This is what you get in SD... and this is what you get with HDTV!"

jdiner
07-15-2004, 02:04 PM
Alright from janet clip I get no video, and the audio has 2 points of damage. Like I said I need one that actually is correct to be able to base what comes next off of. Any other sources or elements I can try?

--jdiner

bcc
07-15-2004, 05:25 PM
Alright from janet clip I get no video, and the audio has 2 points of damage. Like I said I need one that actually is correct to be able to base what comes next off of. Any other sources or elements I can try?It plays for me flawlessly with mplayer (unix, xvidix video driver, 3ghz processor). It plays for me under windows with windows media player (using elecard mpeg2 demultiplexor). On my windows pc, the audio has some drops because I only have a 2.4ghz processor. The audio gets better as I overclock, but it looks like I need about 3ghz to avoid all drops. I wonder if you've been able to get *any* 1920x1080 mpeg2 to decode in realtime with your setup?

bcc
07-15-2004, 05:26 PM
Very funny! They could use that clip to sell HDTV's. "This is what you get in SD... and this is what you get with HDTV!"
:) I actually would have used something less controversial but there was no content being aired at 1080i last night (PBS was off the air, and WB was transmitting with a broken audio stream).

jdiner
07-15-2004, 06:17 PM
It plays for me flawlessly with mplayer (unix, xvidix video driver, 3ghz processor). It plays for me under windows with windows media player (using elecard mpeg2 demultiplexor). On my windows pc, the audio has some drops because I only have a 2.4ghz processor. The audio gets better as I overclock, but it looks like I need about 3ghz to avoid all drops. I wonder if you've been able to get *any* 1920x1080 mpeg2 to decode in realtime with your setup?
Umm. My machine is a DUAL CPU, 4 gig ram, 450+ gig video raid.... I have plenty of horse power. It is not that it comes out wrong but rather that nothing was coming out for the video channel. I.e. my decompressor was telling me nothing was found. Strange that.

But I have not tried mplayer. I will have to try again with that player. I was user PowerDVD and WinDVD as they have worked well for me on just about everything in the past. Neither liked this new file.

--jdiner

jdiner
07-15-2004, 06:19 PM
Exactly. Mjpegtools seems to be hardcoded to use 232K video buffers for DVD authoring, but that buffer doesn't necessarily fit a whole frame. Its demux program readily thinks there is a video stream underflow seemingly because of this. I had to hack its SCR vs DTS check just to get it to process the law&order clip.
So do you, or anyone else here, know what the proper values are supposed to be? if I can just get certain values and guidelines it will be easy to finish up.

--jdiner

redstone
07-15-2004, 07:07 PM
Alright from janet clip I get no video, and the audio has 2 points of damage. Like I said I need one that actually is correct to be able to base what comes next off of. Any other sources or elements I can try?

--jdiner

Okay, at this point I am confused.

Are you looking for SD clips or High Def clips?

If High Def then you should have:

Law and Order.ty from OTA
Six feet Under.ty From HBOHD
rides.ty from DSCHD
HDNET test pattern.ty

As I put them on http://southriver.web.aplus.net/ a few at a time and then deleted them when you told me to cause I have a 1 GB limit.

At around 8:00 EDT, check http://southriver.web.aplus.net/ as I added another HD clip that was given to me.


bcc - would you mind playing the new one (IN&OUT) with your mplayer and let me know if the video is smooth and the audio is in sync as I am not seeing that.

Thanks

malfunct
07-15-2004, 07:41 PM
Okay, at this point I am confused.

Are you looking for SD clips or High Def clips?

If High Def then you should have:

Law and Order.ty from OTA
Six feet Under.ty From HBOHD
rides.ty from DSCHD
HDNET test pattern.ty

As I put them on http://southriver.web.aplus.net/ a few at a time and then deleted them when you told me to cause I have a 1 GB limit.

At around 8:00 EDT, check http://southriver.web.aplus.net/ as I added another HD clip that was given to me.


bcc - would you mind playing the new one (IN&OUT) with your mplayer and let me know if the video is smooth and the audio is in sync as I am not seeing that.

Thanks

He needs HD clips that have perfect sync to start out with. The ones you listed didn't seem to be adequate for him to test sync with. Mainly he needs to know if they changed the clock resolution or offsets or anything like that so that tytool can correctly contruct the mpeg2 pes from the tystream.

mikemav
07-15-2004, 08:49 PM
Redstone- perhaps you can take a short clip from a MyHD hi-def capture that plays well, and convert it to MPEG w/ HDTVtoMPEG, then post it? I will not have a chance myself this weekend, nor do I have the FTP space. If you cannot, I will when I get a chance and send it to you to post.

jdiner
07-15-2004, 09:47 PM
Ok. Allow me to try and be as clear as possible.

What I am looking for now IS NOT more HD-Tivo TyStream files. I have plenty at the moment for the stage I am in.

What I need from those that want to do this kind of thing is help in getting a proper. working MPEG Program Stream (PS) file that has the following properties:

1- it contains Hi-Def data.

2- it has perfect sync of audio and video (meaning the clock values are correct and the PTS values work.)

3- it plays in a standard working player so that I can later test what is coming out of my tools to match up the process.

4- if at all possible I would actually like a couple of these from various sources so that I can check a couple of other things and make sure there is no variance.

Now personally I have no source for, and no ideas about where to get Hi-DEF mpeg files.

I have little time left for the kind of poking around I had to do to get TyTool working in the first place with standard streams. Which means I need help and info before I can really move forward.

--jdiner

jdiner
07-15-2004, 09:48 PM
Redstone- perhaps you can take a short clip from a MyHD hi-def capture that plays well, and convert it to MPEG w/ HDTVtoMPEG, then post it? I will not have a chance myself this weekend, nor do I have the FTP space. If you cannot, I will when I get a chance and send it to you to post.
What exact format doe sthe MyHD program you mention save things in? The raw transport stream or is it something else.

--jdiner

bcc
07-16-2004, 03:32 AM
bcc - would you mind playing the new one (IN&OUT) with your mplayer and let me know if the video is smooth and the audio is in sync as I am not seeing that.For me, the video is fine for the first minute. The audio sync is worse than usual. The audio starts with "That's what this is all about" but the video seems to start with the man lipping just the last 2 or 3 syllables of it. For the next minute the video remains early by about 3 syllables (perhaps slightly closer sync than the beginning?). After 1 minute, the video glitches&pauses and I see some Rozanne sitcom played back in quadruplicate for a while.

When I demux&remux the content, my demuxer reports the video early by 737ms. Remuxing bombs out at the same 1 minute glitch point. The audio&video play back in sync (except the first sentence still seems off).

I think this proves that mplayer does a bad job of trying to sync ty AV streams.

bcc
07-16-2004, 03:38 AM
What exact format doe sthe MyHD program you mention save things in? The raw transport stream or is it something else.The MYHD HD capture card saves transport streams. That is why I first asked if you wanted a transport stream or a .mpg.
Redstone- perhaps you can take a short clip from a MyHD hi-def capture that plays well, and convert it to MPEG w/ HDTVtoMPEG, then post it?That is exactly what janet.mpg is (A clean MyHD clip converted to mpeg with hdtvtompeg2).

mikemav
07-16-2004, 07:09 AM
Can anyone confirm what a .trp is? I have a crosshatch and a color bars test pattern, each from the MyHD lead test engineer. Even if these are program streams instead of the usual .ts or .tp, I am not sure if they do us any good since they probably do not have audio. I will check into it and play them when I get a chance. Perhaps I can check with that engineer as well to see if he has any clips.

redstone
07-16-2004, 08:15 AM
The MYHD HD capture card saves transport streams. That is why I first asked if you wanted a transport stream or a .mpg.That is exactly what janet.mpg is (A clean MyHD clip converted to mpeg with hdtvtompeg2).

You ran Checktp on the original janet transport stream and it reported zero errors on the .tp file?



The only reasonably sized , 100% perfect MYHD transport stream file that I have on disk is from a Sports OTA HD broadcast but it switches to SD a few times during the clip as they switched between the HD camera and SD camera.

Do you want that want one? It still is ~640 MB

All the rest either have errors reported by checktp or they are 2GB files each.

Enclosed is a screen snapshot of the output from checktp on that good file.

mikemav
07-16-2004, 09:05 AM
You ran Checktp on the original janet transport stream and it reported zero errors on the .tp file?



The only reasonably sized , 100% perfect MYHD transport stream file that I have on disk is from a Sports OTA HD broadcast but it switches to SD a few times during the clip as they switched between the HD camera and SD camera.

Do you want that want one? It still is ~640 MB

All the rest either have errors reported by checktp or they are 2GB files each.

Enclosed is a screen snapshot of the output from checktp on that good file.

I think what we need to do is take one of these captured .ts, clip it so it is small and change the output to MPEG program stream via HDTVtoMPEG2. Sounds like this is exactly what was done to the Janet clip, but maybe another example would help. I will play around with it, as well as search to see if there are any known good MPEG HD clips already out there (that do not need converted from .ts)

Another question- jdiner- have you had a chance to look at the Six Feet Under clip and see what is different about it that allowed the current TyTools to process it but not the others? I also got a working HD .vob from a 688MB capture of the HDNet test pattern, so something must have been aligned with the moon and the stars on that run of TyTools w/ this file as well. Odd that two would work as-is and most would not, isn't it. I wonder if the answer lies in identifying the difference in the files that did work.

redstone
07-16-2004, 09:32 AM
I am uploading that Preakness .mpg file now.

It is an ATSC MPEG2 transport stream which was converted to a standard mpeg2 with hdtvtompeg2

From the HDTVtoMPEG2 Readme :
This program converts HDTV MPEG2 tansport stream files to standard MPEG2 video files playable (mostly) by media player, many software DVD player programs, and readable by video converter programs like FlaskMPEG and Vidomi.

ATSC MPEG2 transport stream files are the raw data sent to your Digital television from the broadcaster, and contain multiple mpeg video and audio streams which can be tuned by your digital tuner.
These streams differ from standard MPEG2 files in that they encapsulate the standard MPEG2 stream data in small 188 byte packets that are easy to transmit and decode.

HDTVtoMPEG2 simply processes the .TS transport stream files, extracting the data from each of these 188 byte packets, and reconstructing a standard MPEG2 file from them.








See:
http://southriver.web.aplus.net/

for the mpeg file

jdiner
07-16-2004, 12:13 PM
Can anyone confirm what a .trp is? I have a crosshatch and a color bars test pattern, each from the MyHD lead test engineer. Even if these are program streams instead of the usual .ts or .tp, I am not sure if they do us any good since they probably do not have audio. I will check into it and play them when I get a chance. Perhaps I can check with that engineer as well to see if he has any clips.
Ooohhh. Even without audio that would be perfect if it is a program stream.

I have in the past given long explanations of how mux'ing works. The key here is that a mux HD stream is still an MPEG-2 program stream. But there are certain key values that must change in order for it to work.

All I am interested in is those keys. While I would prefer one with audio one without would be at least 90% of what I am looking for. The rest could be constructed from that point if need be.

--jdiner

jdiner
07-16-2004, 12:16 PM
I am uploading that Preakness .mpg file now.

I am not at home at the moment. Can you leave that up for me for a few hours. I would like to get the data and try things out.

--jdiner

shigaloo
07-16-2004, 01:43 PM
jdiner - sent you a pm

bcc
07-16-2004, 03:00 PM
Can anyone confirm what a .trp is? I have a crosshatch and a color bars test pattern, each from the MyHD lead test engineer. Even if these are program streams instead of the usual .ts or .tp, I am not sure if they do us any good since they probably do not have audio. I will check into it and play them when I get a chance. Perhaps I can check with that engineer as well to see if he has any clips..trp is MYHD's naming scheme for transport stream files. Other programs normally use a .ts extension for this. Some of the test patterns at http://www.nabs.net/cwatson53/ have audio but I wouldn't think these are good samples of SCR values.

bcc
07-16-2004, 03:24 PM
You ran Checktp on the original janet transport stream and it reported zero errors on the .tp file?
The only reasonably sized , 100% perfect MYHD transport stream file that I have on disk is from a Sports OTA HD broadcast but it switches to SD a few times during the clip as they switched between the HD camera and SD camera.No, I didn't run checktp on janet first, but like I said, the audio&video plays back fine both before&after the mpg conversion. I don't know why you think it's hard to get a recording that checktp thinks is error free. I just ran checktp on a couple new ones and it found no errors. I'll upload a brand new 1080i sample from OTA a few minutes ago to prove it.

bcc
07-16-2004, 03:58 PM
Here you go. and the hdtvtompeg2 converted mpeg2, http://soap.mpg (25MB)
Checktp gives the transport stream its blessing. This is actually a good clip in that it shows a lot of close up head shots with speech.

bcc
07-16-2004, 05:26 PM
But I have not tried mplayer. I will have to try again with that player. I was user PowerDVD and WinDVD as they have worked well for me on just about everything in the past. Neither liked this new file.janet.mpg plays back fine for me with powerdvd&windvd. Seems like a codec problem. I wonder what graphedit says you're using for codecs? Even with powerdvd&windvd I'm probably using some elecard codecs, but I can't seem to get graphedit to connect to any remote graphs to confirm...

jdiner
07-16-2004, 05:57 PM
Gotta be some kind of a codec issue then. What version of PowerDVD where you testing with?

--jdiner

bcc
07-16-2004, 08:25 PM
Gotta be some kind of a codec issue then. What version of PowerDVD where you testing with?powerdvd 5, windvd 5. Did you try the soap.mpg clip?

redstone
07-17-2004, 09:28 AM
No, I didn't run checktp on janet first, but like I said, the audio&video plays back fine both before&after the mpg conversion. I don't know why you think it's hard to get a recording that checktp thinks is error free. I just ran checktp on a couple new ones and it found no errors. I'll upload a brand new 1080i sample from OTA a few minutes ago to prove it.

The reason I asked about running checktp is twofold:
1) jdiner says he needs a perfect set of data and checktp would ensure that we are giving him that.

2) While most of my recordings are now on D-VHS, the few (with the Preakness exception) that are still on my disk were from the DISH 5000 and they ALL have massive errors.
These errors never bother MYHD but it led me to think that a perfect stream is hard to get.

Like you, I get perfect audio and video playback on these transport streams even though lots of errors are reported.

See this for what I mean for a bad checktp yet it plays perfectly in MYHD:
(I only ran this on 11% of the data so I had to merge 2 snapshots) so you could see the begginning and end of the report.



BTW, those barsref,etc. files that mikemav mentioned were generated by a HD pattern generator in the lab so one can not get any better data for testing.

jdiner
07-18-2004, 12:33 AM
When I said I needed something "nearly perfect" I WAS NOT looking for a transport stream. I was looking for a program stream. They are 2 completely different styles of mux'ing.

A Transport Stream (TS) file is similar to what dish/directv/et al use... It allows for massively parallel streams of data etc... and sweet error recovery by goin for extremely small chunks of data. (Last I read it was around 188 byte per "pack".)

A program stream is the kind most of used from local media etc... meaning a hard drive, CD-Rom, DVD and so on. There is a limited view of a System Clock Reference in a TS file. It is absolutely crucial in a PS file because of the virtual padding idea of MPEG-2. So what I wanted to see was the mux-rate, the SCR delta from 1 PACK to the next, etc...

The reason I want/need to see that is that given a standard SCR diff as prescribed by a DVD, or a standard MPEG-2 PS file, there is a known gap between PACKs when it comes to the SCR. It is roughly what... 1.5ms or so. I started looking at some of the data and found that it was quickly getting off and doing so more and more. Meaning something like an SCR gap at .5 or even .25 rather than 1.5ms would in essence "give that much more space" in the file before getting off on the clock. But can DVD players do that? Can software players handle it? I don't know. And is there a prescribed value? Like the 1.5ms is the "standard" for DVD and being on that makes a major difference, I can't help but wonder what about the Hi-Def stuff. Is there a proper value to use there as well?

So that is what I am looking for and why.

--jdiner

redstone
07-18-2004, 08:08 AM
I wish I could help but I have no clue where to get a Program stream.

The transport streams that I have been referring to are the standard ATSC OTA streams that my digital OTA receivers (MYHD,Samsung T165,HDTIVO's RF antenna in) gets from the roof antenna.

All I know is this that I found:
"ATSC MPEG2 transport stream files are the raw data sent to your Digital television from the broadcaster, and contain multiple mpeg video and audio streams which can be tuned by your digital tuner.
These streams differ from standard MPEG2 files in that they encapsulate the standard MPEG2 stream data in small 188 byte packets that are easy to transmit and decode.

HDTVtoMPEG2 simply processes the .TS transport stream files, extracting the data from each of these 188 byte packets, and reconstructing a standard MPEG2 file from them."

mikemav
07-18-2004, 08:12 AM
The reason I want/need to see that is that given a standard SCR diff as prescribed by a DVD, or a standard MPEG-2 PS file, there is a known gap between PACKs when it comes to the SCR. It is roughly what... 1.5ms or so. I started looking at some of the data and found that it was quickly getting off and doing so more and more. Meaning something like an SCR gap at .5 or even .25 rather than 1.5ms would in essence "give that much more space" in the file before getting off on the clock. But can DVD players do that? Can software players handle it? I don't know.
--jdiner

I don't think anyone even has hopes that DVD players could handle HD content. They are resolution limited. The idea is to use hardware HD PCI cards & or software media players that can already handle the ATSC HD resolutions. For instance, when I did have success with the HDNet test pattern 688MB file in TyTools, I used the new muxer (in r14) and got a .vob. It played fine on the MyHD PC card. I have not tried it on some of the software media players, but I'm pretty sure it should work.

IMO, think trying to get HD extractions to convert to 720x480 DVD resolution for burning/ playback on stand-alone DVD players should be secondary (and may not be important at all.) The point of HD is to preserve the resolution, after all. I'm sure you know all this already; I'm just posting for the benefit of those just joining this discussion.

If you need any more examples (or want to see the HD .vob I got from TyTools that works), let me know.

redstone
07-18-2004, 08:16 AM
Give it until around 9:00 AM EDT to finish uploading but this http://southriver.web.aplus.net/

has the IN&OUT clip that mikemav gave me which used to bomb tytoolr14 but now tytoolr15 makes a VOB file that plays in windvd and myhd.


What is different between a ty+ and a ty file?



JQ

mikemav
07-18-2004, 09:48 AM
What is different between a ty+ and a ty file?



JQ
Ty+ (from my understanding) is just the .ty with additional XML info packaged in, for use in the TiVo (maybe thumbs info, actor info, etc..) From a TyTools perspective, they should process the same, correct?

jdiner
07-18-2004, 12:20 PM
I don't think anyone even has hopes that DVD players could handle HD content. They are resolution limited. The idea is to use hardware HD PCI cards & or software media players that can already handle the ATSC HD resolutions. For instance, when I did have success with the HDNet test pattern 688MB file in TyTools, I used the new muxer (in r14) and got a .vob. It played fine on the MyHD PC card. I have not tried it on some of the software media players, but I'm pretty sure it should work.

IMO, think trying to get HD extractions to convert to 720x480 DVD resolution for burning/ playback on stand-alone DVD players should be secondary (and may not be important at all.) The point of HD is to preserve the resolution, after all. I'm sure you know all this already; I'm just posting for the benefit of those just joining this discussion.

If you need any more examples (or want to see the HD .vob I got from TyTools that works), let me know.

Ok. That is good to know. To be honest I has assumed that DVD output was still what was wanted as someone PM'ed me asking about exactly that. I have no HiDef equipment so I have payed very little attention to it.

The problem with the SCR can be solved quite simply then. For the most part you can all just use the old format mux'er. It actually calculates and SCR such that an entire block of whatever speed fits in with no clock issues.

Many of the buffer size issues in TyTool have been resolved with this latest releas, 9r15, so someone should give it a try and see what is happening when using the old muxer format.

--jdiner

jdiner
07-18-2004, 12:22 PM
Give it until around 9:00 AM EDT to finish uploading but this http://southriver.web.aplus.net/

has the IN&OUT clip that mikemav gave me which used to bomb tytoolr14 but now tytoolr15 makes a VOB file that plays in windvd and myhd.


What is different between a ty+ and a ty file?
ty+ was something that was extracted via TyStudio but is supposed to still just be a plain TyStream file. .ty was extract by TyTool. tmf is the tar format that contains more information about a show and hence the ability to restore it after extraction.

--jdiner

redstone
07-18-2004, 05:51 PM
Many of the buffer size issues in TyTool have been resolved with this latest releas, 9r15, so someone should give it a try and see what is happening when using the old muxer format.

--jdiner

I tried it with that IN_AND_OUT clip and for the first few seconds, there is a lip sync problem and then for the remainder of the playback, I get NO sound at all.

With the new muxer format, I get sound throughout.
What is even wierder thoughm is that if I just let the sound come out in stereo through my PC speakers, the audio and video are in sync yet if I enable the AC3 SPDIF out so my Yamaha is doing the decoding, the audio is off by a few seconds.

rc3105
07-18-2004, 06:28 PM
ty+ has never had anything to do with tystudio. it's equivilant to tmf but the xml is appended to the end instead of tarpacked at the beginning.

basic ty utils assume the xml is junk and ignore it. enhanced apps can do some interesting things like generate dvd menu's, update a database, index a recording collection, etc.

ty from mfs_ftp is really ty+ and there's a tivoweb module to produce it as well

jdiner
07-18-2004, 07:11 PM
ty+ has never had anything to do with tystudio. it's equivilant to tmf but the xml is appended to the end instead of tarpacked at the beginning.

basic ty utils assume the xml is junk and ignore it. enhanced apps can do some interesting things like generate dvd menu's, update a database, index a recording collection, etc.

ty from mfs_ftp is really ty+ and there's a tivoweb module to produce it as well
Oh. Wow. Sorry. Someone else had told me it was a plain TyStream file. I have seen people talk about being able to use them directly with TyTool. But if there is the 1KB tar header at the start of the file before the actualy tivo data I am kind of amazed it works at all.

Interesting. Thanks for the info.

--jdiner

jdiner
07-18-2004, 07:14 PM
I tried it with that IN_AND_OUT clip and for the first few seconds, there is a lip sync problem and then for the remainder of the playback, I get NO sound at all.

With the new muxer format, I get sound throughout.
What is even wierder thoughm is that if I just let the sound come out in stereo through my PC speakers, the audio and video are in sync yet if I enable the AC3 SPDIF out so my Yamaha is doing the decoding, the audio is off by a few seconds.
Wow. That is officially whacked. Luckily I also have a nice Yamaha so I can at last try and see what you are seeing. I am only going to be able to put a bit of time into it over the next little while but hopefully something good will come from it.

I am surprised that the new muxer works better than the old one on thist stuff. The new muxer format is extremely regimented and desgined to be as close to the DVD spec as I could make it. The older format was serious free form and hence much more adaptable.

--jdiner

rc3105
07-18-2004, 07:33 PM
in a tmf the xml & each fsid has a 512 byte tarheader

th/showing.xml - th/part00.ty - th/part01.ty - th/part02.ty...


ty+ is a regular mfs_stream file, then you add 512 bytes of ###, the xml, then another 512 ###'s

part00 - part01 - part02 - ###XML###

the #xml# is rarely if ever exactly 128k, even if it were any sort of chunk verify routine would fail so it's discarded


ty+ enhanced utils merely skip to the end, look for blocks of ### and parse what's in between. makes for easy import into myth/freevo style apps

jdiner
07-18-2004, 07:47 PM
Ah I misunderstood. So the TMF is the tar packed which has been obvious for a long time, but the TY+ is just the Ty with the xml packed onto the end. Ok. Makes sense why it would work. You are right there is little chance that the next 128k chunk would try to be read right on that header. But then TyTool would check for it anyway and ignore it as a bad chunk as nothing would line up.

--jdiner

bcc
07-18-2004, 08:00 PM
I wish I could help but I have no clue where to get a Program stream.When you convert a .ts to a .mpg using hdtvtompeg2 you're converting a transport stream to a program stream.

bcc
07-18-2004, 08:03 PM
BTW, those barsref,etc. files that mikemav mentioned were generated by a HD pattern generator in the lab so one can not get any better data for testing.I don't think synthetic data with static video is a very good test case for HD mpeg muxing/demuxing. Certainly not a good stress test.

jdiner
07-18-2004, 08:21 PM
BCC is quite correct. What I was hoping for was one that was not a converted file. But since there seem to be none I will just have to make do with what we have.

--jdiner

mikemav
07-19-2004, 06:35 PM
Okay, I got 9r15, and it processed some files I had without crashing. At the beginning, I got lots of the "warning: serious damage,a single video record is larger than 1 meg, recovering." It does process the whole file without crashing. I tried several extracted files and they worked.
This issue now is that my .ty+ is 15GB (True Lies movie on HBO-HD), but the VOB it created is only 529MB. When I play it back in MyHD, it reports back that the VOB is 1920x1080. I have not had time to play the whole file back, so I do not know if the length is truncated, if there is an error 1/2 way through, or what. Any idea why TyTool would only make a 529MB file? It says it has finished. Reports:
DiffTime = 1113.016053 (1113016) == 18.550269 Minutes
total = 15900484777 (15163 MB)

Done with 'O:\TivoStreams\ty+\{True Lies}{1993-12-31}{}{05.30 PM Fri Jun 04, 2004}{HBOH}.ty+'...

Any ideas?

On second thought, is this that "priority set" issue? I had that fixed (gotta read back a few pages to remmeber how), but my .var got wiped, so I just reinstalled mfs_ftp. Also, these extractions on my PC may not have been from a time when I had this fixed. Perhaps I will try the .ty instead of .ty+.

redstone
07-19-2004, 07:05 PM
I don't think synthetic data with static video is a very good test case for HD mpeg muxing/demuxing. Certainly not a good stress test.


I can't win for trying:)

jdiner said several days ago that he wanted perfect program streams and that made sense so he did not have to deal with getting confused by any gunk in the data or missing packets,etc..
Hence, my insistence that we run checktp before running hdtvtompeg2 and not use any that had reported errors.

Now you want a stress test?

I realize that this release was not to address the HDTIVO but:

WHEN jdiner really starts digging into HDTIVO streams, shouldn't one first get a program stream (thanks for the answer about the difference) that converts properly first before one starts stress testing?

Yes,I know that he does not have an HDTIVO and will only work on this if it suits him but A) He already is taking a crack at it and B) You have no idea how popular this program will be as I have been getting a LOT of emails from folks on avsforum about extracting for archical purposes from the HDTVO.

Sorry for all the third person language. Just thought i would combine 2 posts into one.

JQ

jdiner
07-19-2004, 08:49 PM
"warning: serious damage,a single video record is larger than 1 meg, recovering." It does process the whole file without crashing. I tried several extracted files and they worked.
Wow! A 1meg frame. That is not right. it can't be. A standard DTivo video I-Frame is 60-70kbytes. A really large one from an SATivo at best quality is 120-150kbytes. 1 meg is well beyond the 4x it should roughly map out to be. But ok. That warning was intended to catch some errors in the way an HDVR2 is doing the packing.

It would appear that things are going to be a bit more work that I had figured on when I started looking at this. Kind of like when I finally had to break down and get a DTivo. It appears a few streams won't showcase enough of the issues but for now we will have to proceed that way as I don't have the money for an HDTivo, and all joking aside I would never ask one person to get me one. Last I heard on here they were more than $1k a piece.

What I can do start is start looking at why the packing gets off in terms of why the audio sync is having issues and so on. Perhaps gets some debugging into things and have those output files sent back to me to look at things.

Since going to DVD no longer seem to be the target it means I can get creative with the multiplexing to get the best results.


This issue now is that my .ty+ is 15GB (True Lies movie on HBO-HD), but the VOB it created is only 529MB. When I play it back in MyHD, it reports back that the VOB is 1920x1080. I have not had time to play the whole file back, so I do not know if the length is truncated, if there is an error 1/2 way through, or what. Any idea why TyTool would only make a 529MB file? It says it has finished. Reports:
DiffTime = 1113.016053 (1113016) == 18.550269 Minutes
total = 15900484777 (15163 MB)
Something went wrong. No idea what right now a multiple PTS reset and number of things could be there. There is no way a real file at 15gig could process down to .5gig. Something got lost/eaten along the way.

--jdiner

mikemav
07-19-2004, 09:08 PM
Something went wrong. No idea what right now a multiple PTS reset and number of things could be there. There is no way a real file at 15gig could process down to .5gig. Something got lost/eaten along the way.

--jdiner

I think it was the Priority set issue. Since both of the first two files I tried to process were original ty+ extractions from before I used the new mfs_ftp binaries that fixed that issue, and both stopped at around 500MB, that is probably it. I just processed a 1GB file from HD Net as a ty+ extraction made after running the new binaries, and the .vob is also 1Gb (only slightly smaller) Same errors reported along the way, but the file processed all the way through just the same. Seems to play in MyHD, but I do not have audio right now in me home theater since I am rewiring, so I cannot verify how the lip sync is. At least we are getting somewhere 9r15 has not crashed once on my tonight. Once I get my sound back I will have to play the HD VOBs and see how they sync.

Redstone, I extracted a 122MB clip tonight of some music on HD PBS. I have it as ty+ as well as the VOB TyTools 9r15 successfully processed. Let me know if you have the room to post either (or both.)

bcc
07-19-2004, 11:45 PM
Now you want a stress test?Me? I wouldn't want to spend much time testing with some MYHD transport stream that isn't representative of the kind of real-world OTA or satellite broadcasts we really want to demux. Yes, data that stresses the demux rate is part of what I thought would make a good test at this point. Getting simple HD data to demux at all has been done for weeks now. For example I even posted the law&order clip demuxed&remuxed. (I can't speak for jdiner and it wasn't too clear to me what kind of sample he wanted.)
WHEN jdiner really starts digging into HDTIVO streams, shouldn't one first get a program stream (thanks for the answer about the difference) that converts properly first before one starts stress testing?First, hd (and sd) tivo streams aren't exactly program streams or transport streams, as they are not 188 byte packets with transport stream sync bytes, and the PES packets are diced up with across proprietary chunk record headers, and... Oh never mind; in any case yes in general you start simple and add complexity when having trouble debugging.

jdiner
07-20-2004, 12:24 AM
I think it was the Priority set issue.
Oh yeah. That on was my bad. Sorry I never used that bin I built it for someone else.

Anyway, you know you can fix that right? Without re-downloading? You just need a top notch hex editor and a bit of understanding. Then you remove the priority string and save the changes and the results will process directly.

--jdiner

bcc
07-20-2004, 03:13 AM
Anyway, you know you can fix that right? Without re-downloading? You just need a top notch hex editor and a bit of understanding. Then you remove the priority string and save the changes and the results will process directly.
I also posted a program that would find&delete the offending strings for you in one of the mfs_ftp threads - but I'm not compiling it for mac-os :)

mikemav
07-20-2004, 07:34 AM
I also posted a program that would find&delete the offending strings for you in one of the mfs_ftp threads - but I'm not compiling it for mac-os :)

I'm not that bad.....I'm bilingual, just not trilingual! :) I will do a search. Is there a PC version, or just linux?

jdiner
07-20-2004, 12:51 PM
I also posted a program that would find&delete the offending strings for you in one of the mfs_ftp threads - but I'm not compiling it for mac-os :)
Now theres a good idea. The tool not the MAC part... :)

Never even thought of that. I just use a hex editor...

--jdiner

mikemav
07-20-2004, 03:53 PM
I found the thread bcc refers to, but it is source code (looks like it is in C.) I appreciate the fact that it is even available (and of course all the contributions to this forum from all), but I am not a programmer and have never complied anything myself. Sorry for the newbie question, but does anyone have a compiled version of this for PC, or can point me towards a PC hex editor that can fix the files? I use Crimson Editor now, but have not gotten really deep into it. Can it do what I need, and what would I need to do to fix the Priority Set issue assuming there is not a Windows version of bcc's tool?

I tried to search for this but did not find much. Sorry if this info is out there somewhere and I missed it.

jdiner
07-20-2004, 04:59 PM
I found the thread bcc refers to, but it is source code (looks like it is in C.) I appreciate the fact that it is even available (and of course all the contributions to this forum from all), but I am not a programmer and have never complied anything myself. Sorry for the newbie question, but does anyone have a compiled version of this for PC, or can point me towards a PC hex editor that can fix the files? I use Crimson Editor now, but have not gotten really deep into it. Can it do what I need, and what would I need to do to fix the Priority Set issue assuming there is not a Windows version of bcc's tool?
Find the exact post with the source file in it and I will compile it for you for the PC/Windows.

--jdiner

redstone
07-20-2004, 07:44 PM
Redstone, I extracted a 122MB clip tonight of some music on HD PBS. I have it as ty+ as well as the VOB TyTools 9r15 successfully processed. Let me know if you have the room to post either (or both.)

Ftp both of them into the /html directory.
Check your PM.

redstone
07-20-2004, 08:40 PM
Me? I wouldn't want to spend much time testing with some MYHD transport stream that isn't representative of the kind of real-world OTA or satellite broadcasts we really want to demux. Yes, data that stresses the demux rate is part of what I thought would make a good test at this point. Getting simple HD data to demux at all has been done for weeks now. For example I even posted the law&order clip demuxed&remuxed. .

My apologies. I just realized that we have two parallel projects going on in here.
Yours is to build a playable mpeg file with that demux and mux program and jdiners is to do the same with tytool.

That just dawned on me and hence my response to you - sorry.

Now I understand why you want stress test cases for your approach.
Do you have all the clips which mikemav gave me that I had been posting for download?

All I want is to be able to make an mpeg file that is playable by MYHD since it is connected to my HDTV/Receiver and also can record to/play from my D-VHS deck.





About that law.mpg that you posted. In mplayer and in Windows media player, it has perfect sync and yet when I play it with MYHD, the lip sync is WAY off.

Any idea why?

bcc
07-21-2004, 12:59 AM
Find the exact post with the source file in it and I will compile it for you for the PC/Windows.Here you go. I don't have a compiler set up under windows right now, otherwise I'd just do it.
http://www.dealdatabase.com/forum/showpost.php?p=172899&postcount=646

mikemav
07-21-2004, 07:52 AM
Ftp both of them into the /html directory.
Check your PM.

Both the 1080i .VOB and the .tmf used to create it w/ TyTools 9r15, VOB-Mux (old) are uploading now. I need to go to work, so I cannot babysit the upload, but the .tmf is going now and the VOB is next in the queue. Hopefully by later this morning they will be up there. Thanks for hosting Redstone! Hope getting these files out there will help this process. I am really curious how they play both in MyHD (my personal preference) and some of the software players. Like I mentioned, I can verify that files converted in 9r15 play in MyHD, but cannot check the lip sync, since I currently have no sound from my HTPC due to a wiring snafu.

jdiner
07-21-2004, 10:37 AM
Here you go. I don't have a compiler set up under windows right now, otherwise I'd just do it.
http://www.dealdatabase.com/forum/showpost.php?p=172899&postcount=646
Alright thanks. I have to run off to teach a class in just a second but I will get it recompiled for the PC users out there and then posted back up here it can be reposted at will into other threads past that...

--jdiner

shigaloo
07-21-2004, 12:35 PM
jdiner - still looking for program streams?

i can whip up a few (not converted transports streams) with mp2 2channel audio, ac3 might be a bit trickier...

let me know

jdiner
07-21-2004, 01:25 PM
jdiner - still looking for program streams?

i can whip up a few (not converted transports streams) with mp2 2channel audio, ac3 might be a bit trickier...

let me know
Oh yes definately. if you have access to a few that are not converted but proper streams from somewhere else I would really appriciate that. I don't have the kind of network bandwith needed to do it over the net unless they are pretty small.

--jdiner

jdiner
07-21-2004, 02:07 PM
Alright a dos compiled version. Since I have no files with that problem I could not test it but everything looks right.

The full project with changes, a getopt.cpp and .h files, etc... are all found in the release zip. Those looking just for the bin, head into the release directory and it is right there.

Hope it helps.
--jdiner

rc3105
07-21-2004, 02:51 PM
jdiner:

it's possible to insert ty without the xml from a tmf or ty+, just gotta fill in the blanks with tivoweb. since hd video most likely isn't going on dvd, how about an option to process hdty or hdty+ to mpg/vob that's reversable

hdty -> hdmpg + .diff

hdmpg + .diff -> ty


that way folks can archive / watch with regular tools, stuff it back in the hd-tivo or re-process with updated sw. .diff files in the VIDEO_TS folder are ignored and the only time they amount to much sizewise is when you edit out LOTS of commercials. restorability is well worth the wasted space


(btw - I've writtin some utils to manage/insert tmf/ty from tivo connected usb hd/dvd or network mounts like a pc server or xbox dvd drive. with the new double layer burners/disks even hd archiving is practical. haven't quite figured out yet what to call the project though)

shigaloo
07-21-2004, 02:57 PM
alright, basically they were originally from a transport stream but i have output the frame to uncompressed avi, brought the audio and video into an HD edit suite, output and then recompressed to mpeg2 program stream. Assuming the original was good and my work fixes any issues it should be dead on. The video quality *may* be somewhat compromised from a visual quality standpoint, but bitrates will be to spec (19.2 mb/s)

thats about the best i can do without a d5 master

mikemav
07-21-2004, 03:15 PM
jdiner:

it's possible to insert ty without the xml from a tmf or ty+, just gotta fill in the blanks with tivoweb. since hd video most likely isn't going on dvd, how about an option to process hdty or hdty+ to mpg/vob that's reversable

hdty -> hdmpg + .diff

hdmpg + .diff -> ty


that way folks can archive / watch with regular tools, stuff it back in the hd-tivo or re-process with updated sw. .diff files in the VIDEO_TS folder are ignored and the only time they amount to much sizewise is when you edit out LOTS of commercials. restorability is well worth the wasted space


(btw - I've writtin some utils to manage/insert tmf/ty from tivo connected usb hd/dvd or network mounts like a pc server or xbox dvd drive. with the new double layer burners/disks even hd archiving is practical. haven't quite figured out yet what to call the project though)

That is an awesome idea, and if I were more of a programmer I would be inclined to help. As it stands I would be glad to keep extracting anything anyone thinks is needed to test and with Redstone's help, I'm sure we could post it for ftp at some point.

I would also be glad to contribute to those developers' paypal accounts who take this on to thank and encourage the progress. :-)

rc3105
07-21-2004, 03:40 PM
groovy, lets see if that's the general concensus - TVArchive test baloon (http://www.dealdatabase.com/forum/showthread.php?p=174539#post174539)

who knows where we'd be by now if jdiner 'n I both the $$$ to spare for hd-tivos :D

redstone
07-21-2004, 04:25 PM
Mikemav uploaded the .tmf file and converted .vob file that jdiner requested.

It is called Dion Live and can be found here:
http://southriver.web.aplus.net/

mattdb
07-21-2004, 05:00 PM
Dion? Oh man what a test stream! I just couldn't lurk anymore..

Mattman

mikemav
07-21-2004, 05:06 PM
Dion? Oh man what a test stream! I just couldn't lurk anymore..

Mattman

You mean high definition Dion! Hey, at least it is not Dion Sanders. That would be even more obnoxious!
Sorry, it happened to be about the only true HD on at that time w/ easily seen lip sync. Send your complaints to Mark Cuban. His email is very public and he says he responds to all of them!

mikemav
07-21-2004, 09:49 PM
Alright a dos compiled version. Since I have no files with that problem I could not test it but everything looks right.

The full project with changes, a getopt.cpp and .h files, etc... are all found in the release zip. Those looking just for the bin, head into the release directory and it is right there.

Hope it helps.
--jdiner

Thanks for taking the time to compile this.
I see the .exe in the release directory, but it did not seem to do anything on my system. I see the command prompt pop up quickly when I run it, but then it goes away. No place where I tell it what extraction to process. It goes away too fast for me to see what if any error message there may be. Is there any configuration file that I need to edit or something? I am running XP pro and installed the entire extracted archive from the zip in a folder on C:\Program Files. Sorry if these are really dumb questions...I mostly use a Mac.

lart2150
07-21-2004, 09:59 PM
start-> run->cmd

cd in to the folder then run the program

bcc
07-21-2004, 10:03 PM
I am running XP pro and installed the entire extracted archive from the zip in a folder on C:\Program Files. Sorry if these are really dumb questions...I mostly use a Mac.Mouse start->accessories->command prompt
From DOS prompt:
cd c:\program files\whatever
programname -i <source-file>.ty -o <output-file>.ty
where <source-file> and <output-file> are the filenames you want to process.
The program will report a message for each extraneous string it removes from your tystream.

bcc
07-22-2004, 04:12 AM
All I want is to be able to make an mpeg file that is playable by MYHD since it is connected to my HDTV/Receiver and also can record to/play from my D-VHS deck.I have that mostly working. I've converted a feature length movie from tystream into mpeg and it plays back with myhd - pretty good lip sync. The ff/rew controls don't work so I guess myhd still thinks there's some problem. I get about 2 frames of pixelation at the end of the first FSID where there is a monster sized tivo audio record - I'm not sure why. Other FSID boundaries process OK. All HD samples I've seen process OK except leno where the audio track comes out at 2X speed (have to apply my mplayer fix these audio packets before demuxing them it looks like). I think I've tried all your samples except the HDNet test pattern.

About that law.mpg that you posted. In mplayer and in Windows media player, it has perfect sync and yet when I play it with MYHD, the lip sync is WAY off.Damn, I don't see this. Which part?
Any idea why?No...

mikemav
07-22-2004, 08:11 AM
Mouse start->accessories->command prompt
From DOS prompt:
cd c:\program files\whatever
programname -i <source-file>.ty -o <output-file>.ty
where <source-file> and <output-file> are the filenames you want to process.
The program will report a message for each extraneous string it removes from your tystream.

Thanks. I knew there had to be some command line prompts I needed to add. Should have known the -i/ -o, but I forgot.
I ran it and it reported "found turd at 1010, removing", then "all done!", but the output file is only 514MB, while the input file is 14.8GB (full HD movie)
Any idea what gives?

jdiner
07-22-2004, 12:42 PM
how about an option to process hdty or hdty+ to mpg/vob that's reversable

hdty -> hdmpg + .diff

hdmpg + .diff -> ty

that way folks can archive / watch with regular tools, stuff it back in the hd-tivo or re-process with updated sw. .diff files in the VIDEO_TS folder are ignored and the only time they amount to much sizewise is when you edit out LOTS of commercials. restorability is well worth the wasted space
Hummm. That is a very interesting idea. Doing so would have to include a couple of things. The extraction mechanism of TyTool would need an overhaul. To try and make things "faster" certain chunks are never sent which is turns out are required for re-insertion. (Since no one was doing that way back when it is really a matter of just changing needs...) But we would want/need the entire stream.

Then if we kept the entirety of the cut GOPS, both FAE and GOP cuts, so that we could restore to the original version... add a system for know where and when in a file a cut occured it could be stitched back together into the original file in terms of the content.

But that doesn't address the problem of the TyStream packing of the file. We had pieces and parts that are gone once you go to an MPEG. As far as I know no one has ever figured out a few of those pieces for being able to actually regenerate a TyStream from them.

Has that changed?

--jdiner

bcc
07-22-2004, 01:19 PM
Thanks. I knew there had to be some command line prompts I needed to add. Should have known the -i/ -o, but I forgot.
I ran it and it reported "found turd at 1010, removing", then "all done!", but the output file is only 514MB, while the input file is 14.8GB (full HD movie)
Any idea what gives?Doh, I flaked on a last-minute change to support ty+ as well as ty files. I've fixed the source zip (unfortunately I still don't have a compiler installed on my windows machine).

bcc
07-22-2004, 01:52 PM
Per request, I posted my muxer work over here:http://www.dealdatabase.com/forum/showthread.php?t=36322

mikemav
07-22-2004, 02:05 PM
Doh, I flaked on a last-minute change to support ty+ as well as ty files. I've fixed the source zip (unfortunately I still don't have a compiler installed on my windows machine).

jdiner- would you mind recompiling the revised zip of this program so I can use one one my ty+ files? I sure would appreciate it. I have a few HD movies I only have in ty+ that I now have deleted from my Tivo. Thanks

rung
07-22-2004, 05:44 PM
Hummm. That is a very interesting idea. Doing so would have to include a couple of things. The extraction mechanism of TyTool would need an overhaul. To try and make things "faster" certain chunks are never sent which is turns out are required for re-insertion. (Since no one was doing that way back when it is really a matter of just changing needs...) But we would want/need the entire stream.

Then if we kept the entirety of the cut GOPS, both FAE and GOP cuts, so that we could restore to the original version... add a system for know where and when in a file a cut occured it could be stitched back together into the original file in terms of the content.

But that doesn't address the problem of the TyStream packing of the file. We had pieces and parts that are gone once you go to an MPEG. As far as I know no one has ever figured out a few of those pieces for being able to actually regenerate a TyStream from them.

Has that changed?

--jdiner


The diff file would need to store the master chunks (1 per segment), the chunk headers and an offset/size map from ty chunk to MPEG. To recreate the ty, you would first stream the master chunk, then the first chunk header, then run through the chunk header to determine which video/audio record goes next. The offset map would tell you where exactly in the MPEG that particular audio/video data resides. Repeat for all records. Repeat for remaining chunks in the segment. Repeat for remaining segments in video.

If you are only doing GOP cuts, you don't need to save the data (see chunkedit). FAE cuts are a whole different ballgame, of course.

EDIT: I see in the MPEG2 spec there is a way to store user data into the mpeg stream. Do you think we could put the diff file inside the mpeg?

redstone
07-22-2004, 08:20 PM
See:
http://southriver.web.aplus.net/

for some new clips from the HDTIVO for testing.

1) Dion Live .ty clip in 1080i High Definition
2) #1 converted to a .VOB file with tytoolr15
3) Standard Defintion .ty clip - converts to vob file just fine with tytoolr15
4) IN and Out .ty clip in 1080i High Definition

bcc
07-22-2004, 10:24 PM
jdiner- would you mind recompiling the revised zip of this program so I can use one one my ty+ files? I sure would appreciate it. I have a few HD movies I only have in ty+ that I now have deleted from my Tivo. ThanksTry this (I finally got visual studio installed). This is my first compile with visual studio so I hope I did it right :)

jdiner
07-22-2004, 11:18 PM
jdiner- would you mind recompiling the revised zip of this program so I can use one one my ty+ files? I sure would appreciate it. I have a few HD movies I only have in ty+ that I now have deleted from my Tivo. Thanks
What updated zip? The only one I see is the one I posted and it has the binary inside of it...

--jdiner

jdiner
07-22-2004, 11:24 PM
The diff file would need to store the master chunks (1 per segment), the chunk headers and an offset/size map from ty chunk to MPEG. To recreate the ty, you would first stream the master chunk, then the first chunk header, then run through the chunk header to determine which video/audio record goes next. The offset map would tell you where exactly in the MPEG that particular audio/video data resides. Repeat for all records. Repeat for remaining chunks in the segment. Repeat for remaining segments in video.

If you are only doing GOP cuts, you don't need to save the data (see chunkedit). FAE cuts are a whole different ballgame, of course.

EDIT: I see in the MPEG2 spec there is a way to store user data into the mpeg stream. Do you think we could put the diff file inside the mpeg?
The master chunk is one of the ones that is never being sent down. At the time I liked the savings in speed etc...

As for the rest of it A chunk is 128kbytes with the various audio and video and "other" records found in it. Unpacked them is relatively easy. But redoing the packing would quickly become pretty complex and time consuming I fear across 10-15gig file like the movies people have been talking about.

But I agree with you. I fear that only by keeping copies of the headers could things be put back as so far I have no clue as to why TIVO generates the 1, 2 and "other" packets when and as it does. I don't know that anyone has figured it out.

But sadly the above is actually the easy part. It get complex when you have to factor in the errors repeated CHUNKs and other issues that can and do happen in DTivo streams. Given the nature of the HDTivos I can only assume they would do the same. This data is not reflected in the output and would be corner cases on the rest of it.

It is possible to do all of this. But before I commit to doing anything with regards to it I am going to have to go back over things and try to refresh my memory about the specific errors I am catching now and what I am doing about each one.

You can definately put user data in with other header data. But I have several players that choke and die on it. If the .diff files would be ignored I honestly think that would be the better idea. Then improper players/decoders would still be usable.

--jdiner

rc3105
07-23-2004, 04:26 AM
Hummm. That is a very interesting idea. Doing so would have to include a couple of things. The extraction mechanism of TyTool would need an overhaul. To try and make things "faster" certain chunks are never sent which is turns out are required for re-insertion. (Since no one was doing that way back when it is really a matter of just changing needs...) But we would want/need the entire stream.

Then if we kept the entirety of the cut GOPS, both FAE and GOP cuts, so that we could restore to the original version... add a system for know where and when in a file a cut occured it could be stitched back together into the original file in terms of the content.

But that doesn't address the problem of the TyStream packing of the file. We had pieces and parts that are gone once you go to an MPEG. As far as I know no one has ever figured out a few of those pieces for being able to actually regenerate a TyStream from them.

Has that changed?

--jdiner
so save the bits & pieces that are changed/discarded to the diff file with a header that specifies where they came from originally.

think of the way a tychunk header works, some records stand alone (like the cc info) while others refer to ranges later in the chunk. since the diff file will be created as the original is processed it only needs 3 record types. copied, discarded and changed

copied would specify byte ranges in the orignal ty and output mpg

discarded would contain the original data + byte range

changed can either be treated as discarded (for small chunks) or provide a binary diff to apply to the data written to the mpg file

to reconstruct the orignal ty you just parse the diff copying or calculating blocks as apropriate.


there's enough info avail to build ty from mpg or scratch, it's just scattered all over creation. anything missing can be worked out easily enough by using ele2pestriple on a s1 running 2.x sw. (well, allmost anything, still workin on generating ty that'll burn from an 810)


edit:

whoops, should catch up on the rest of the thread before posting. looks like it would be easy enough to add .diff generation to bcc's hdemux util, and rung's spot on with packing the diff data into a private stream. external should still be also be an option though - that way you could get much more content on a regular mpg dvd and make a companion disk of .diff files for re-insertion (the commercials contained in the .diff files can gather dust on a shelf)

rung, have you considered enhancing chunkedit to produce diff & adding the ability to recreate the original file?

rung
07-23-2004, 04:53 AM
rung, have you considered enhancing chunkedit to produce diff & adding the ability to recreate the original file?

No, that is a new idea. But I'm not sure if it would be necessary. If we all work together, I think we could put together a version of chunkedit and a demuxer that would allow for proper demuxing of an edited ty. The reverse process (mpg + diff -> ty) would produce the original edited ty. And edited ty's, as you know, play just fine. I assume that edited HD ty's would also play fine.

rc3105
07-23-2004, 05:20 AM
great theory, but edits don't seem to burn properly from an 810 :( (the other unit that produces HUGE recordings)

anybody tried playing chunkedited streams on a hd tivo yet?

rung
07-23-2004, 05:26 AM
great theory, but edits don't seem to burn properly from an 810 :( (the other unit that produces HUGE recordings)

Wow, I wonder why? I see chunkedited streams no different then a recording with a rain-fade or temporary cable drop. Can the 810 burn those types of streams okay?

rc3105
07-23-2004, 05:43 AM
810 has analog inputs - black/static from the cable box still gets encoded so there's no loss of continuity

rung
07-23-2004, 07:25 AM
810 has analog inputs - black/static from the cable box still gets encoded so there's no loss of continuity I think that when the internal tuner looses signal, encoding stops (blue screen appears). When the signal returns, encoding resumes. In theory, you could use this to edit commercials when you are recording. Does the 810 behave the same way with its internal tuner? Can it burn a show that has this type of discontinuity? Chunkedit is trying to mimick this type of drop-out.

rc3105
07-23-2004, 09:37 AM
tuner??? GASP! sorta defeats the purpose of having a box with s-video in & progressive out ;) after shuffling cables & re-running guided setup to try that it looks like they won't burn either

haven't poked through the chunkedit routines, but I would think since the packets are so much larger you'd need to take that into consideration for proper hd playback???

rung
07-23-2004, 06:11 PM
tuner??? GASP! sorta defeats the purpose of having a box with s-video in & progressive out ;) after shuffling cables & re-running guided setup to try that it looks like they won't burn either

haven't poked through the chunkedit routines, but I would think since the packets are so much larger you'd need to take that into consideration for proper hd playback???

Bummer - looks like a design defect in the 810. :D

The larger packets in the HD streams shouldn't matter - they will just get split-up across chunks (Tivo handles that now with the generic video/audio record types). Since the cuts only happen at sequence headers, I think chunkediting should still work.

redstone
07-25-2004, 06:45 PM
I started this thread to explore the possibility of getting playable High Definition files (playable via such things as Windows Media Player,WinDVD,MYHD PC HDTV card,etc.).

This thread seems to have gone off on a different tangent but thanks to bcc , we(actaully he) have been working on making mpeg2 files from the HDTIVO.

Thanks to his efforts, we have an alpha version that does work on the 6 or 7 test HDTIVO clips that mikemav has provided to me and which I hosted on an ftp site.
It does need more work obviously but is exactly what we HDTIVO folks need to make archives of the HD material that are playable with any mpeg2 compliant program.

Please see the following thread for tools that convert HDTIVO .ty files to mpeg2 compliant files. Please note that this is solely for High Definition streams from the HDTIVO.
http://www.dealdatabase.com/forum/showthread.php?t=36322

I am playing them with MYHD as that is connected to my HDTV as well as my Audio Recceiver for my Home Theater setup.

My thanks to bcc.

Moderator - could you please lock this thread as it has served it's purpose?

-JQ

jdiner
07-27-2004, 10:51 PM
Well I figured out why the audio and video are getting off. I have a new version of TyTool and VSplit in the works, might be done tonight if all goes well, that will properly process, edit, etc... HDTivo files.

--jdiner

jdiner
07-28-2004, 12:47 AM
Wow. The output/data in the janet file is bad. Wow. Multiple PES packets have exactly the same SCR. This is quite against the specs. Hummm. it does seem to work... sort of...

--jdiner

jdiner
07-28-2004, 12:59 AM
Alright. Well I found the numbers that I needed. While the output I am making now plays well I want to try and make sure that the buffers are never overfilled and that is going to take a bit of time. If anyone wants to give what I have built a shot, then post a request and I will put up the current dev bin here on DealDB.

--jdiner

bcc
07-28-2004, 01:06 AM
Wow. The output/data in the janet file is bad. Wow. Multiple PES packets have exactly the same SCR. This is quite against the specs. Hummm. it does seem to work... sort of...--jdinerSame SCR across packets would be hdtvtompeg2's doing. I had noticed that but wasn't sure if it was actually against the specs. hdtvotompeg2 output might not be a good test case...

mikemav
07-28-2004, 08:01 AM
Alright. Well I found the numbers that I needed. While the output I am making now plays well I want to try and make sure that the buffers are never overfilled and that is going to take a bit of time. If anyone wants to give what I have built a shot, then post a request and I will put up the current dev bin here on DealDB.

--jdiner

If it will work on ty+ files, I have many HD ones I would be glad to try. I do not have MPEG program streams (like the Janet clip) to test, but for the pure HD Tivo stuff, I would be glad to give it a shot. I have a MyHD card I can use for playback once converted. Does it convert to both HD VOB and HD MPEG (our choice?) Several people have said they think MPEG would be a more portable format. I always used VOB for MyHD ripped DVD playback, since it then follows the chapter markers of the original DVD. I guess for HD it makes no difference since there are no pre-defined chapters anyway.

Be glad to give it a shot if you post it.

-Mike

jdiner
07-28-2004, 10:37 AM
If it will work on ty+ files, I have many HD ones I would be glad to try.
It will work on ty, ty+, tmf, etc... It is after all a part of TyTool. So what worked before now works with HDTivo streams as well. The biggest differences are just the sheer number of records per 128k chunk.

A standard DTivo stream has 60+ records/chunk on average. An HDTivo stream seems to have 4-8 max. The video is just that much larger.

Then there are some new "place holder" records of type 0x20 with a flag of 0xf. Don't know what they are supposed to indicate yet but they are always 4 bytes with a sequence in hex of: 20 C0 00 00


I have a MyHD card I can use for playback once converted. Does it convert to both HD VOB and HD MPEG (our choice?) Several people have said they think MPEG would be a more portable format. I always used VOB for MyHD ripped DVD playback, since it then follows the chapter markers of the original DVD. I guess for HD it makes no difference since there are no pre-defined chapters anyway.
A VOB is just an MPEG with a few tiny extras. Anything reading only MPG and not VOB is just poorly written. But whatever... Yes just like before you can pick VOB, MPG, new format VOB, new format MPG etc...

However in testing the new format muxer has NO HOPE of lining things up correctly at the moment. The problem is that, as I thought, the SCR gap is too large. And the very first video Frame gets behind the SCR by at least 100ms. I.e. the decoder does not even have it all before it should have been displayed and then replaced by the next frame. So at the moment the new muxer is right out.

The old muxer is much much more flexible by design. And it can be used as it calculates the SCR gap rather than forcing it to match what a DVD wants. So the old format MPG and VOB will actually work.

I have converted and played the 3 HDTivo TyStream files I have. Actually I have 1 TMF, 1 TY+ and 1 TY file.

--jdiner

jdiner
07-28-2004, 10:45 AM
Alright in looking at the output from some of the current conversion tools out there I thought I would just pass on a bit more information.

The HDtoMPEG2 tool that some are using appears to have some serious issues. I mentioned it did not playback well for me. The problem appears to be that the SCR values are not incrememented within a video frame. The SCR is the base system clock reference. Everything is hung off of this value. I.e. there is a seperate clock for video and audio. These clocks are then offset just a bit. The SCR get started and as a PTS value comes by we have things decoded and ready to present etc...

By definition from the MPEG-2 specs this SCR value marches by from PACK to PACK. The SCR clock is updated via a mechanism indicating that a particular byte is found at a particular time based on a simple formula. With this tool we get a proper SCR for the first PACK that is part of the now MUCH MUCH larger video records. But a PACK is only 2048 bytes. So you MUST have more than 1. Each successive PACK is supposed to have a new SCR at a regular standard location, coupled with virtual padding this can get a bit complex but not too bad.

Some something like this:
Video P1 - 0
Video p2 - 100
Video P3 - 200
Video p3 - 300

...

Video P1 - 10,000
Video P2 - 10,100
Video P3 - 10,200
Video P4 - 10,300

What this tool is doing is generating the following:
Video P1 - 0
Video p2 - 0
Video P3 - 0
Video p3 - 0

...

Video P1 - 10,000
Video P2 - 10,000
Video P3 - 10,000
Video P4 - 10,000

Since the SCR decoder clock is supposed to be reconciled with the value from the PACK header it is in essence supposed to toss the current data at that same SCR. It appears to work for some decoders but not all of the values are right...

--jdiner

malfunct
07-28-2004, 12:50 PM
Wow. The output/data in the janet file is bad. Wow. Multiple PES packets have exactly the same SCR. This is quite against the specs. Hummm. it does seem to work... sort of...

--jdiner

I think there are some decoders that figure out the speed they should pull the data and then decide to ignore the clockings, probably especially in the case where the clocking is bizzare or just plain wrong.

redstone
07-28-2004, 06:07 PM
I have converted and played the 3 HDTivo TyStream files I have. Actually I have 1 TMF, 1 TY+ and 1 TY file.

--jdiner

I have these following HDTIVO TyStream files from mikemav:

1)Six Feet Under
2)HDNET test pattern
3)Biker
4) In and Out
5) Law and Order
6) Rides
7) Dion Live

I thought you had gotten all of these. Do you want me to put any of these back on my ftp site?

I am going to test these now with you new version and report back.

Oh yeah - Thanks a lot!

JQ

redstone
07-28-2004, 06:12 PM
:cool:
Alright. Well I found the numbers that I needed. While the output I am making now plays well I want to try and make sure that the buffers are never overfilled and that is going to take a bit of time. If anyone wants to give what I have built a shot, then post a request and I will put up the current dev bin here on DealDB.

--jdiner


Could you please post it and I can run it through those 7 files that I have.

jdiner
07-29-2004, 12:15 PM
I have these following HDTIVO TyStream files from mikemav:

1)Six Feet Under
2)HDNET test pattern
3)Biker
4) In and Out
5) Law and Order
6) Rides
7) Dion Live

I have tested with:
Six Feed Under
Law and Order
Dion Live

I might have gotten these other ones but I can't find them now. Rather than send them to me again, the ones I could not find, I will just post the binary. (I was out of town for work all day yesterday and could not make the post until now as a result...)

--jdiner

jdiner
07-29-2004, 12:21 PM
Here is the first test copy. There are still a few warnings etc... For now don't sweat them. The key is the playability of things with sync at this point.

--jdiner

redstone
07-29-2004, 07:37 PM
Thanks for posting the new program.

I am not having the best of luck though.

No matter which option I choose, law and order comes out as a 6 MB file

Rides has bad pixelation and no lyp sync and after a few seconds, NO audio at all.


Dion works but there is a fairly large dropout in it as well as six feet under.

Not sure what option I chose but the HDnet test Pattern file caused tytool to go into an infinite loop.


Maybe I am just picking the wrong options
I take it from your comments above that you want us NOT to use the new format multiplex or VOB option?
Does it matter if I choose tystream or multiplex mode?

jdiner
07-30-2004, 10:45 AM
Wow. That is not good. I processed Six Feet, Law and Order, and Dion and everything is fine here. I am using the PowerDVD player with them. I have no drop outs no sync problems etc... I will reprocess with each of the options and see what I can come up with. It shouldn't really matter except for the fill audio gaps option which should be off.

--jdiner

mikemav
07-31-2004, 05:33 PM
I converted a full 1 hr episode of Rides (6.7GB) with the new beta program (multiplex files tool) and it converted without error and played back fine. It had some drop outs that look similar to when you have iffy reception on OTA HD (pixelizations, 1 or 2 second drop outs), but it recovered and the audio sync seems fine. Interestingly, at the end of the file (last 5 minutes or so) these drop outs came more often. I cannot be sure if these were conversion errors or a problem with the MyHD or my PC at the time. I am converting a full movie ty+ now (True Lies) and will report how that plays back as well. I may try vob option too to see if that is better or worse w/ MyHD.

jdiner
07-31-2004, 06:46 PM
Alright. I finally had the time to give things a really thorough going over. I have been able to find several things that I missed before that had a major impact on the system.

I have fixed most of these and I am right now looking at the last one I found. Once this one is right things should be ready for some real testing.

Some of the assumption that "have always been true" are no longer there or are missing. This should finally put things back as needed for a working output.

--jdiner

redstone
07-31-2004, 06:47 PM
Just so you don't think I am losing my mind, I am uploading the Rides test file mikemav gave me weeks ago.

I am uploading the .ty file as well as the .mpeg file created by HDtool.

I would appreciate it if someone could download the .ty file and convert it with HDtool and compare your results with the posted .mpg file which has almost no audio.

Another data point. mikemav uloaded a 500 MB file of the Bismark and it converted fine.

Seems like with me, HDtool is hit or miss.

Here is the site with both files. They won't be finished uploading for another hour (at least)

http://southriver.web.aplus.net/

jdiner
07-31-2004, 07:12 PM
Alright here is test #2 of HD support for TyTool.

Please let me know how it works for you. I have only 1 issue I can't explain right now. I don't know what is causing it. There is a slight pop in the video of the Six Feet under clip that is going around. It happend right when some man picks up the child from the table. I haven't been able to figure out if it is an audio clock issue, a missing video frame or what. It is kind of wierd. Everything in the mux looks right but there is still a gap. It is possible that the machine I was doing the testing on is just not powerfull enough to render everything.

--jdiner

jdiner
07-31-2004, 07:32 PM
Where on earth I they getting these streams from? There are a number of I-Frames in this file that are larger than a meg a piece. That is rediculous IMHO. A standard MPEG I-frame is about 85k normally. I improved the buffer sizes yet again to allow for dealing with ever larger frames.

No wonder the rest of the non-HD channels suck so much these days. If every HD channel is using bandwidth like this then it is over for the rest of them. There is just nothing left bandwidth wise.

The ****y playback is still there. It appears to be quite simply just that things are still way way to far apart. Meaning I am going to have to build the HD style mux'er I was talking about.

I begin to see why the hdtompeg2 program just used the repeating 0's. It is probably the best bet as then no matter how cracked in the head the HD streams are things line will still line up. I might just use the same technique as it seemed to work for most people who were posting before.

--jdiner

redstone
07-31-2004, 08:16 PM
Looking MUCH better.

I have no clue what happened to my law and order clip as it only produces a 6 mb file with your tool but here is the status of what I see/hear when these new mpegs are played with MYHD.

Note, I only used the multiplex option ie. generate mpegs for this test.

I have these following HDTIVO TyStream files from mikemav:

1)Six Feet Under - Audio/Vido glitch when the guy picks up the baby, otherwise perfect.

2)HDNET test pattern - @ ~ 3:46 minute mark, it used to have a major dropout but it is now working fine.
Near the begginning, there are a few places where the clock 'hops', otherwise perfect.

Cool, My HDTV has really good high def resolution. It is an 8.5 (I can almost read line 9) resolution on a 1 to 10 scale according to the HDnet resolution pattern:)

3)Biker - Perfect

4) In and Out - one split second pixelation glitch a few seconds into the clip but otherwise perfect.

5) Law and Order - My copy must have gotten corrupyed as I only get a 6 MB clip

6) Rides - It is a mess. See the mpg and ty file on http://southriver.web.aplus.net/

7) Dion Live - Dropout after about 10 seconds into the song. Pixelation for a second or 2 later when waving to the crowd, otherwise perfect

8) Cameron's Bismark - new 550 MB file form mikemav. There are 8 places evenly spaced ie. not near each other in time , where the video breaks up for a split second ie. pixelation.
My longest test clip and except for those 8 spots, it is perfect.


Oh yeah, I take back my comment 'Seems like with me, HDtool is hit or miss.' I said above. It is a HIT!


Thanks for the great progress.

Looking GOOD!!

mikemav
07-31-2004, 09:27 PM
Looking MUCH better.

I have no clue what happened to my law and order clip as it only produces a 6 mb file with your tool but here is the status of what I see/hear when these new mpegs are played with MYHD.

Note, I only used the multiplex option ie. generate mpegs for this test.

I have these following HDTIVO TyStream files from mikemav:

1)Six Feet Under - Audio/Vido glitch when the guy picks up the baby, otherwise perfect.

2)HDNET test pattern - @ ~ 3:46 minute mark, it used to have a major dropout but it is now working fine.
Near the begginning, there are a few places where the clock 'hops', otherwise perfect.

Cool, My HDTV has really good high def resolution. It is an 8.5 (I can almost read line 9) resolution on a 1 to 10 scale according to the HDnet resolution pattern:)

3)Biker - Perfect

4) In and Out - one split second pixelation glitch a few seconds into the clip but otherwise perfect.

5) Law and Order - My copy must have gotten corrupyed as I only get a 6 MB clip

6) Rides - It is a mess. See the mpg and ty file on http://southriver.web.aplus.net/

7) Dion Live - Dropout after about 10 seconds into the song. Pixelation for a second or 2 later when waving to the crowd, otherwise perfect

8) Cameron's Bismark - new 550 MB file form mikemav. There are 8 places evenly spaced ie. not near each other in time , where the video breaks up for a split second ie. pixelation.
My longest test clip and except for those 8 spots, it is perfect.


Oh yeah, I take back my comment 'Seems like with me, HDtool is hit or miss.' I said above. It is a HIT!


Thanks for the great progress.

Looking GOOD!!
I agree. Great work jdiner. I concur with the slight dropouts only. I had 7 or 8 on the Cameron clip as well (maybe in the same spots that redstone is seeing.) I also tried vob and it was the same. Redstone, do you see these slight pixelization glitches if you process the clip using the linux software from bcc? I sure like the simplicity of the TyTools method, but I am curious about these glitches. I am going to try playing them with a software media player to see if it looks any different.

mikemav
07-31-2004, 10:03 PM
I tried playing the James Cameron Bizmark file in PowerDVD as well, and also saw pixelization dropouts every minute or so. Not sure if there were as many as on MyHD, but it was pretty close.

VOB made no difference, except my Windows install knows I want to associated VOB's with MyHD application. I know it is OT, but how do I get Windows XP to release Media Player 9 as the default MPEG player? Even with I do an "open with.." and check "always use this application for this file type", and select MyHD, next time I got to play, it is WMP again! Those guys from Redmond, WA are evil!

rc3105
07-31-2004, 11:14 PM
not to get off topic, but yall might want to keep an eye on this

http://www.dealdatabase.com/forum/showthread.php?t=36610

jdiner
08-01-2004, 01:25 AM
Looking MUCH better.
Good. Looks like I closed a few of the major bugs out as I was hoping it would.


1)Six Feet Under - Audio/Vido glitch when the guy picks up the baby, otherwise perfect.
Yeah. This one I can't figure out yet either. I have to take a bit of time and dig through the output to make sure nothing is getting lost and then go back to the MUX'ing to get a better handle on what it is.

So far I have tested the timestamps and they are all correct...


7) Dion Live - Dropout after about 10 seconds into the song. Pixelation for a second or 2 later when waving to the crowd, otherwise perfect
This one I can explain. There is a giant 5+ second hole in the audio and video data within the file. I can provide timestamps if someone wants to see what I am talking about. But there is actual missing data there so that one will be getting no better.


What's up next for HD work:

1- Put in an option to try the same mux'ing pattern other tools have used. A way to get things to stack up on 1 SCR so that the mux'ing will line up the audio and the video better. Should hopefully solve a few of the glitching issues.

2- Dig back into the 6feet clip and see what is causing the glitching there in the video as the baby gets picked up. There is just no clear reason yet for the problem. Hopefully this is also some of what is going on elsewhere so we can kill a few bird at the same time.

--jdiner

jdiner
08-01-2004, 01:46 AM
For the curious about why I complain about the extra data rate for the HD stuff, check out the clip I posted in the main TyTool thread. I record that about a week ago from the local Utah WB channel.

It is a slightly less than 1 meg file.

--jdiner

redstone
08-01-2004, 11:31 AM
6) Rides - It is a mess. See the mpg and ty file on http://southriver.web.aplus.net/


Any idea what is going on with this one?

It converts and plays quite well with hdemux/mplex so I know the data is good.
Eg. When the guy is holding that U shaped pipe and is laughing, the whole segment before,during and after is fine. When I convert with HDtytool, it is all pixelated and no audio at all.


Please note: I am not trying to do a 'shoot out' of these 2 tools but rather I try both when I notice a glitch to see if the problem is the data or the tool.

redstone
08-01-2004, 11:35 AM
I am converting a full movie ty+ now (True Lies) and will report how that plays back as well. I may try vob option too to see if that is better or worse w/ MyHD.

How did True Lies come out?

jdiner
08-01-2004, 11:41 AM
Not yet. I need to dig in and see what I can see. But I was tired last night and went to bed.

--jdiner

mikemav
08-01-2004, 11:52 AM
How did True Lies come out? The usual pixelization drop outs every now and then, otherwise fine.

jdiner
08-01-2004, 09:17 PM
Oh wow. Well I just figured out what was wrong with the recently posted rides TyStream. My goodness. The audio and video are completely off from each other and a result group/block together causing the wierd playback pattern.

I have yet to figure out why it is doing that but it is also a DD audio stream and it is possible that that has something to do with it.

More to come. Just have to figure out a bit more about what is going on before I try to put in a fix and get it posted, but at least the effect in the output is extremely clear.

--jdiner

jdiner
08-01-2004, 09:18 PM
The usual pixelization drop outs every now and then, otherwise fine.
Hummm. Not sure what is causing that. But let me ask this, is it what is seen in the 6FeetUnder clip that was posted? That little drop-out or is it something else that you are seeing? I just need to make sure I have something that causes that effect.

--jdiner

jdiner
08-01-2004, 09:38 PM
Alright. The problem with the rides clip is that the stream type detector failed. It thought it was a DTivo stream and not an HDTivo stream. You can see that detection as part of the the initial text output from the processing engine.

Something I had thought was pretty much a garantee was not actually working. I hate that. Going to keep looking for something that will be a bit more regular for the auto-detection. At least part of what is wrong in the rides clip is that it appears to have come from the middle of a file, but either way the detector is off and needs to be fixed... When forced into the right type it processes correctly.

--jdiner

jdiner
08-02-2004, 02:25 PM
The problem with the mis-diagnosis has been solved. At least from what I can tell. I have to get back home to get access to a few things to make the finishing touches. I will post a version once I am back.

Also dug a bit into the other occasional ****y playback going on. Nothing conclusive yet but given that other tools handle it it makes me think that I just have something going wrong. I.e. it is fixable I just have to find it.

--jdiner

redstone
08-02-2004, 06:28 PM
not to get off topic, but yall might want to keep an eye on this

http://www.dealdatabase.com/forum/showthread.php?t=36610


I have no clue as to how many HDTIVO owners are reading this thread but I wanted to make sure folks spotted rileys post.

With HDTytool getting real close to a solid beta state as well as the Linux only hedmux/mplex tool, one has to decide whether to pay $100 to have someone re-solder their Tivo to socket/add the modded prom or pay whatever towards the SW method being discussed in the above thread.


This is not OT because right now, there is only 1 way to even think about extraction from the HDTIVO and now we 'may' have a potential, less risky alternative.


Besides, I started this thread so it can not be OT:)

mikemav
08-02-2004, 08:38 PM
Hummm. Not sure what is causing that. But let me ask this, is it what is seen in the 6FeetUnder clip that was posted? That little drop-out or is it something else that you are seeing? I just need to make sure I have something that causes that effect.

--jdiner

Yep, this looks like the same error I am seeing on the other files. For kicks, I reprocessed with HD TyTool v2 and it still had the glitch, but you probably knew that.

jdiner
08-02-2004, 11:30 PM
Sweet! The problems with that latest Rides ty is fixed.

Also the proper auto-detection of being an HD stream is in there. Hopefully it will work correctly from here on out. Sadly the only thing to do about that next is to really increase initial buffer sizes. The only way I can see it failing now is things just being way to far apart physically in the TyStream source file. But going for an even larger initial buffer is unappealing as it is already pretty large.

Before I make the next release though I want to dig into the problem with the 6 feet under for a bit and see what I can see. If I can get a fix in place that will make a proper test #3 I think.

--jdiner

jdiner
08-03-2004, 12:34 AM
Ooohhh nice. Found a major bug that poped up in the Ty splitting engine. i.e. only shows when you are actually splitting into seperate files. Ack. Wonder how many others of those might have crept in. I use an int called type all over the place. Well I also use one that is part of the main class that determines the type of the writing engine in use, VOB/MPG/Splitting/Key etc... I forgot to declare it in one spot and it used the main type rather than a local one.

Luckily it only took about 2 hours to find. :(

--jdiner

jdiner
08-03-2004, 12:36 AM
Oh and the reason for looking for/at that bug. I wanted to output with no mux'ing and see if that skip in the 6feet clip still happened. I.e. was it data or mux'ing that was the culprit. It was the data. With nothing but the raw stream there is still a jump. Meaning I have figureout the how and the why.

Someone posted here awhile back and pointed out that another tool did not have that poping problem. What tool was that? Where did you get it? I just wanted to look at the raw output data and see how they compared. They should be identical but right now they will not be.

--jdiner

jdiner
08-03-2004, 01:12 AM
Woah... The output from hdemux is just plain wrong. That is wierd...

--jdiner

jdiner
08-03-2004, 02:24 AM
Well crap. The search for what is wrong in the occasional glitching is far from simple. In looking over the code everythings is correct. But it is obviously not so... This one could take awhile folks.

--jdiner

jdiner
08-03-2004, 09:23 PM
Alright. I got it. The fix was simple once I tracked it down.

Other than the missing data in the middle fo the dion clip I now get proper creation from each of the sample HDTivo files that I have.

So here it is, HD Tivo Test #3. Just the TyTool EXE once again.

Give it a try and let me know what results you get.

--jdiner

jdiner
08-04-2004, 12:16 AM
Well I guess people weren't waiting with baited breath for this particular release. Ah well. Hopefully it will solve problems for some.

--jdiner

rc3105
08-04-2004, 12:44 AM
wait till 50 people install the sw exploit tomorrow & start extracting hd content

runaway! :p

jdiner
08-04-2004, 03:18 AM
Wow. Someone is seriously way to smart... :)

That eff hack sounds pretty stinking cool.

--jdiner

redstone
08-04-2004, 08:28 AM
From Version 2:

Note, I only used the multiplex option ie. generate mpegs for this test.

I have these following HDTIVO TyStream files from mikemav:

1)Six Feet Under - Audio/Vido glitch when the guy picks up the baby, otherwise perfect.


4) In and Out - one split second pixelation glitch a few seconds into the clip but otherwise perfect.

6) Rides - It is a mess. See the mpg and ty file on http://southriver.web.aplus.net/

7) Dion Live - Dropout after about 10 seconds into the song. Pixelation for a second or 2 later when waving to the crowd, otherwise perfect

8) Cameron's Bismark - new 550 MB file form mikemav. There are 8 places evenly spaced ie. not near each other in time , where the video breaks up for a split second ie. pixelation.
My longest test clip and except for those 8 spots, it is perfect.



Thanks for a great piece of SW!!

Here are the NEW results:
(the perfect ones are still perfect so I cut them out of this post)


1)Six Feet Under - Perfect - The Audio/Vido glitch when the guy picks up the baby is gone.


4) In and Out - Perfect - the one split second pixelation glitch a few seconds into the clip is gone.

6) Rides - 100% flawless!!


7) Dion Live - Dropout after about 10 seconds into the song still there as you noted but the Pixelation for a second or 2 later when waving to the crowd is gone.
I guess that dropout is the HDTIVO fault as I have seen mine occasionally 'blip' - rare but it does happen

8) Cameron's Bismark - new 550 MB file form mikemav. My longest test clip and it is now perfect.

Thanks for all your hard work.

mattdb
08-04-2004, 09:38 AM
Well I guess people weren't waiting with baited breath for this particular release. Ah well. Hopefully it will solve problems for some.

--jdiner

I have been waiting to get into my HDTiVo to do this. Now with the new hack I will tonight.
This board is going to see some serious action over the next couple of days.

Thanks for the hard work, again!

Matt

mikemav
08-04-2004, 10:21 AM
jdiner, thanks again. I did not have net access last night, and am at work now, so I will try it out tonight. Since redstone's files all came from me though, I'm sure it will work for me as well. Awesome work. BTW, does it work on .tmf, or is .ty+ or .ty my best best?
Now I only need to decide if I have the room to keep both a ty+ version and an mpeg version of the files. At 6-9GB/hr, that is not cheap to keep double copies. Is there any hope of getting a working version of Riley's suggestion (app that reverses the process, to go from mpeg-ty?) so I can play these files on the TiVo again some day but still only keep one version? With the new interest in HD hacking that is certain to come with the new software exploit, this would generate a lot of interest.....

Also, does anyone know of a good PC app that will take the now clean HD mpegs and downres to 720x480/VOB so we can make a lower res DVD should we ever want to share a program with out non-HD brethren? Could TyTools do that as an option, or do I need another app. That was not the goal of this update, but now that the HD part is solved, it would be a nice option in the toolkit if the need ever comes up.

jdiner
08-04-2004, 02:07 PM
mikemav: So how have you been extracting then? Did you have your box socketed with a different prom then?

That would seem both expensive and a pain.

Glad that someone figured out the new stuff.

--jdiner

mikemav
08-04-2004, 02:16 PM
mikemav: So how have you been extracting then? Did you have your box socketed with a different prom then?

That would seem both expensive and a pain.

Glad that someone figured out the new stuff.

--jdiner

Yep, up until this week PROM modding was the only known way to hack the HD Tivo. Sleeper did a great job modding mine and several others. Well worth the price since I would not get near that $1000 box with an iron myself! Anyway, now maybe the software exploit will bring many more into this club soon.

Any thoughts on reversing HDTyTool so we can get mpeg back to ty+ if we need to re-insert?

jdiner
08-04-2004, 02:20 PM
I have thought about it but don't have many specific plans on any front at the moment. There are a few things on the list and I will figure out what I want to do next sometime soon. (At the moment I figure I will try to see what is going wrong for the UK users...)

The main delay in other areas has been getting the splitting to actually work at all. There were a few things that just didn't make sense and I had to go back to the beginning and revisit each of the assumptions in use to find the problem.

--jdiner

jdiner
08-04-2004, 02:24 PM
BTW, I mentioned that there was a problem in the hdemux program when I downloaded the cygwin compiled bin and ran it on the rides test that was posted about here.

I meant no disrespect to bcc the author of that tool. I was frustrated by a number of things at the time and fear it might have spilled over in my wording.

The problem was simple. It started with the 00 00 01 b3 sequence header and that is 100% correct. But later in the data I found tons of 00 00 01 e0 headers which are the PES header of the data stream. These are not supposed to be present in an Elementary Stream (ES). But even there things played which is what had me so frustrated. I was filtering that part correctly and still getting playback errors. :(

--jdiner

moshmothma
08-04-2004, 05:30 PM
Jdiner, thanks for the awesome work!! Now how about getting a the VOB muxer to work with HD files? I'd love to be able to author HD - dvds (I know I must play them from a pc). What I would like are chapters and menus. Also, what about FAE support? Thanks

redstone
08-04-2004, 08:15 PM
Now I only need to decide if I have the room to keep both a ty+ version and an mpeg version of the files. At 6-9GB/hr, that is not cheap to keep double copies. Is there any hope of getting a working version of Riley's suggestion (app that reverses the process, to go from mpeg-ty?) so I can play these files on the TiVo again some day but still only keep one version? With the new interest in HD hacking that is certain to come with the new software exploit, this would generate a lot of interest.....

.

I am trying to find a program that will make .ts files which work. If you have a .ts file, you can archive it to a D-VHS and tapes are cheap.

I have tried 2 programs on the 'older' versions of the mpegs made with jdiner's tool but they do not work properly.

I am told that the resultant .ts files are non-standard but I wonder if it really lies with the inout mpeg files? I need to revisit the whole mpeg to .ts with this newest version.

Meanwhile, once I get the SW exploit working and archiving, I will be backing up the .ty files on my DDS4 drive until I lick this .ts file conversion issue.




On a related thought,
The longest .ty file that you gave me is 550MB and it converts REALLY fast.

I am getting 500 MB converted in just under 30 seconds.
If my math is right, a 12 GB (1.5 hour HD movie) would take 12 minutes to convert from .ty to .mpeg2. That is nothing.

This is on a 2.8 Ghz P4 and a SATA RAID 0.
Until I crack the conversion to .ts issue, I can wait 12 minutes before I want to watch a movie.


How long does your system take to run HDTytool on a 12 GB file?

If it is not really long, perhaps you can just keep the .ty files on your disks and convert when you want to watch the program?

jdiner
08-04-2004, 08:49 PM
I have tried 2 programs on the 'older' versions of the mpegs made with jdiner's tool but they do not work properly.
Re-process them with the newer versions of the tools back into an MPEG or a VOB. It is aware of the older versions of output and what is generate now is more generally liked.


If it is not really long, perhaps you can just keep the .ty files on your disks and convert when you want to watch the program?
Mux'ing to a TS file is a relatively simple process. There was even a "to TS" mux'er release some time ago here on DealDB.

--jdiner

jdiner
08-04-2004, 08:49 PM
Jdiner, thanks for the awesome work!! Now how about getting a the VOB muxer to work with HD files? I'd love to be able to author HD - dvds (I know I must play them from a pc). What I would like are chapters and menus. Also, what about FAE support? Thanks
Have you even tried to FAE an HDTivo file?

--jdiner

moshmothma
08-04-2004, 09:11 PM
Jdiner, hmmm, I could be mistaken - I am trying to use your tool with non-TIVO HD files. I thought I saw the tools should work on these as well. Maybe I could send some samples....

jdiner
08-04-2004, 09:19 PM
Jdiner, hmmm, I could be mistaken - I am trying to use your tool with non-TIVO HD files. I thought I saw the tools should work on these as well. Maybe I could send some samples....
It should work. But there are a ton of formats for MPEG and not all of them are supported. There is NO mpeg-1 support, but that isn't what is being discussed here. There is newly expanded support for MPEG-2 Program Stream files (PS). There is similarly NO support for Transport Stream files.

I have a number of HD PS files that various people have sent me. FAE and other editing works perfectly with these files. You can make key files, you can cut as needed, you then process to new output files.

A transport stream file is what a number of people here sent me as a sample. They are extremely different in the packing being used from the PS files. No transport stream demux'ing is found in TyTool.

--jdiner

jdiner
08-04-2004, 09:29 PM
Just to be clear for one and all...

TyTool operates in 3 distinct stages.

1- Split the input file into properly maintained Elementary streams.
2- Cut and re-compress as needed for FAE etc...
3- Generate output into the desired form (splitting, mux'ing, etc...)

So using a TyStream file, TY+, MPEG, VOB, etc... are just get cut up into the correct ES pieces timestamps are calculated where needed etc...

Meaning that to TyTool an HD TyStream is just a slightly different format of the same old thing. Now that I have identified the differences, at least the ones I have seen so far, there is going to be very little that will cause issues.

In terms of cutting HiDef data is just larger. It is no different than the MPEG-2 data we got before. Same headers, same layout, same everything just wider and taller. So the GopEditor still works, the compression for FAE takes greater than 4x as long due to the size of the data but it still works, etc...

For output we run into some problems. The data is too large to work with the "standard" mux'ing format. So the new muxer which tries to be as DVD perfect as possible will never work. You get off on the PTS vs SCR before the 3 frame goes by. It is literally that quick.

The old format muxer which is still there was designed to work with darn near everything as the input. So it takes the HiDef right in stride calculates the proper SCR value, Non-standard as all get out, and uses this to package the data. As a result you get proper playback of HD data.

Now all of this has been explained before and fairly recently too boot. So to answer the recent question... how about a VOB muxer? Already there, just use the non-beta menu options (these are the same old ones we have always used...)

Will it work on FAE for HD streams? Yes.

Will it recompress correctly at cuts? Yes.

The whole point of getting TyTool working for HiDef data was to get everything else as well. All of the editing, and error correction etc...

--jdiner

mikemav
08-04-2004, 10:48 PM
Just to be clear for one and all...

So to answer the recent question... how about a VOB muxer? Already there, just use the non-beta menu options (these are the same old ones we have always used...)


--jdiner

How about 720x480 VOB? Is there a way to convert the HD into this format for playback on stand-alone players (realizing of course the loss in resolution)? I do not see this happening often, but it may be nice on occation. Or is that dependent on the new muxer, which will not work with the HD streams? I know you have explained the new muxer does not have a chance...I am just making sure there is not an option for the old VOB-muxer to change resolution to DVD standard. Sorry if I am being clear as mud.

jdiner
08-04-2004, 10:50 PM
How about 720x480 VOB? Is there a way to convert the HD into this format for playback on stand-alone players (realizing of course the loss in resolution)? I do not see this happening often, but it may be nice on occation. Or is that dependent on the new muxer, which will not work with the HD streams? I know you have explained the new muxer does not have a chance...I am just making sure there is not an option for the old VOB-muxer to change resolution to DVD standard. Sorry if I am being clear as mud.
Sure, it can be resized just NOT with TyTool. Use something like tmpgenc to do a resize and recompress all in 1 pass. (Or for better overall support/filtering etc... use VirtualDub-MPEG2 and get really nice output frame served to tmpgenc...) You are going to have to recompress every frame of the entire file to get that to work. There are not tricks that can be played. Resolution in MPEG files is pretty much absolute.

--jdiner

moshmothma
08-05-2004, 12:47 AM
Just to be clear for one and all...
The whole point of getting TyTool working for HiDef data was to get everything else as well. All of the editing, and error correction etc...
--jdiner

You are awesome! I am really psyched about these fixes. I will definetly try out FAE tomorrow. Problem number one though - using the old vob-muxer, none of the HD files I mux have audio. They start out with 5.1 ac3s which I can play in the mpg before muxing. Afterwards, the video plays back with no sound. Interestingly, if I try and demux the resulting vob, my app doesn't detect an audio track.

Thanks again for the wonderful work.

jdiner
08-05-2004, 01:07 AM
You are awesome! I am really psyched about these fixes. I will definetly try out FAE tomorrow. Problem number one though - using the old vob-muxer, none of the HD files I mux have audio. They start out with 5.1 ac3s which I can play in the mpg before muxing. Afterwards, the video plays back with no sound. Interestingly, if I try and demux the resulting vob, my app doesn't detect an audio track.
The VOB/MPG import system does not support AC3 audio yet. I didn't have and couldn't find any such files at the time. So the code for the audio just does nothing. Which would be why there is no audio track once the mux'ing is done.

--jdiner

moshmothma
08-05-2004, 09:10 AM
Ok, cool - lmk if you want some samples. Thanks again

jdiner
08-05-2004, 11:17 AM
Oh it is easy to make AC-3 mpegs I just start with an AC-3 based TyStream file and make an MPEG/VOB from it. I wanted to get things working before going that far though. Makes little sense to have dozens of options that don't work well so I stuck with the core features.

--jdiner

moshmothma
08-05-2004, 11:29 AM
You are quickly becoming my hero! That would be all we needed. Authoring HD has been a long time goal of ours. So far Tytool is the only thing that can reliably make VOBs from some tricky HD streams. AC3 support is awesome! Now if we can convince you to support PCM/DTS..... :)
Thanks again!

jdiner
08-05-2004, 05:28 PM
Alright TyTool 9r16 has been released. Features are a few minimal list of new things for most of the users. But for this group should be pretty good news.

1- A few audio fixes are now in place.

2- The big items in it are support for Dolby/AC-3 MPEG/VOB files as source files.

3- Also includes the final changes for using HDTivo streams. Removed a bit of debugging code that was still present etc...

4- Vsplit has been updated to match.

You will find it in the normal spot on page one of the main TyTool thread.

--jdiner

moshmothma
08-05-2004, 09:40 PM
Ok, that's it - you have forced me to send you a Paypal! Thanks so much - this tool is fantastic. Now as a paying customer :) can I please make the following requests? A man of your talent should be able to add these things in no time flat!

1. PCM/DTS support for TyTool (at least the ability to mux those audio formats)
2. Ability to mux streams based on user input - so you choose an m2v and an ac3 and then mux. Someway to offset the audio would make this muxer perfect.
3. More support for DVD authoring (actually, I keep getting an error when trying to do this - vob is unused or something like that).

Thanks again for your time.

malfunct
08-05-2004, 10:37 PM
Ok, that's it - you have forced me to send you a Paypal! Thanks so much - this tool is fantastic. Now as a paying customer :) can I please make the following requests? A man of your talent should be able to add these things in no time flat!

1. PCM/DTS support for TyTool (at least the ability to mux those audio formats)
2. Ability to mux streams based on user input - so you choose an m2v and an ac3 and then mux. Someway to offset the audio would make this muxer perfect.
3. More support for DVD authoring (actually, I keep getting an error when trying to do this - vob is unused or something like that).

Thanks again for your time.

2) Since tytool only processes tystreams and mpeg2 program streams or vobs all of which have the audio embedded you don't get to choose an external audio track. I don't see a need for it within the realm of what the tool is for.

3) The "unused vob" error is just that. You aren't assigning all of your selected vobs to a menu item and its warning you of this. Assign them all and the error will go away. If you want more dvd support learn how to use DVD author manually, it can do almost anything with a dvd (from what I've read) if you just learn how. It can take the vob's from a vob-mux as input so just vob-mux and then build whatever you want with dvd author.

moshmothma
08-05-2004, 10:57 PM
2) Since tytool only processes tystreams and mpeg2 program streams or vobs all of which have the audio embedded you don't get to choose an external audio track. I don't see a need for it within the realm of what the tool is for..

Aww, you may not be as serious about audio as some. HD movies are presented in ac3 format at 384kbps. Although, the video is mainly superiour to DVD the audio is inferior. Most DVDs have ac3 encoded at 448kbps. Secondly, many movies are broadcast in 2.0 rather than 5.1. Quite often there is a nice 6 channel DVD soundtrack we could mux with our HD TIVO video. Thirdly, many HD movies could be muxed with DTS audio present on the DVDS (the HD version of The Two Towers extended edition would go quite nicely with the 6.1 DTS audio from the DVD). This would be a killer feature for TyTool and may not be that difficult to implement. It really looks like the worst part is dealt with (cleaning up the video)


2)The "unused vob" error is just that. You aren't assigning all of your selected vobs to a menu item and its warning you of this. Assign them all and the error will go away. If you want more dvd support learn how to use DVD author manually, it can do almost anything with a dvd (from what I've read) if you just learn how. It can take the vob's from a vob-mux as input so just vob-mux and then build whatever you want with dvd author.

Actually, I've only been using one VOB and I've been picking it from the drop-down. I'll dig into it a bit more. More support really means a more intuitive interface but to each his own. Thanks

jdiner
08-06-2004, 12:43 PM
Actually, I've only been using one VOB and I've been picking it from the drop-down. I'll dig into it a bit more. More support really means a more intuitive interface but to each his own. Thanks
You need to hit either the enter key or the "set item" button to "set the changes in stone". They can be altered later but that allows for changes you want abandoned.

I am sorry you find the GUI non-intuitive. I do. But then I wrote it so that should be a given.

It has expanded enough that I have begun to think the whole window needs an overhaul. But until that happens you get what you have.

--jdiner