PDA

View Full Version : Video extraction from the new HD10-250



Pages : 1 2 [3] 4 5

snoots
10-30-2004, 05:42 PM
I did a search looking for this and can't locate it. Do you have a link to the executable ?

redstone
10-30-2004, 07:26 PM
Back to our regularly scheduled HDtytools discussion :)


Speaking of which, when is jdiner going to address some of the issues we have identified with HDtytool and when is it going to be merged back into the official tytool program?

I have gotten bit many times by running the wrong program when I forgot whether the prog to be converted was High def or SD.

Unless I missed a post in this thread, the most recent version of HDtytool that I have is from August.

rudolpht
10-31-2004, 12:58 PM
I did a search looking for this and can't locate it. Do you have a link to the executable ?

Try post 330 in this thread.

snoots
10-31-2004, 06:41 PM
Thanks, I searched for the .exe not the zip 'DOH"

desplaines
11-02-2004, 12:31 AM
WinDVD will play TS files. Try it, they just may not show up so try all files or just change the extension. They look very good on WinDVD actually.
I have not been able to get WinDVD 6 to play TS files created by VLC from HD Tivo files. I tried .old MPEG', 'old VOB', and 'new VOB' (all renamed to '.ts'). Are your files also from an HD Tivo, or are they MyHD or other 'more standard' .ts files?

rudolpht
11-02-2004, 02:00 AM
I have not been able to get WinDVD 6 to play TS files created by VLC from HD Tivo files. I tried .old MPEG', 'old VOB', and 'new VOB' (all renamed to '.ts'). Are your files also from an HD Tivo, or are they MyHD or other 'more standard' .ts files?

I just played the vob of ROTK in WinDVD6 with a Vob extension (I used old VOB offload). I could not play a TS file like VLC or MyHD so I stand corrected. Why not play the intermediate VOB?

Tim

rudolpht
11-02-2004, 02:01 AM
TheaterTek played the TS file correctly.

Tim

ViperX2
11-03-2004, 12:26 AM
Any progress on getting sound to work with OTA HD streams? I know this is the #1 HD issue for jdiner.


I made a .vob file with HDTytool Version 4 of a segment from the Olympics and I get no sound from the VOB file.

This was an OTA recording on the HDTIVO.

I tried New format VOB and old format mpg with the same results


Mplayer plays the .ty file and does have sound.

I am uploading a .ty clip here:
http://southriver.web.aplus.net/

so you can replicate.

Thanks

Rowan
11-03-2004, 08:51 AM
Redstone,

The link to the LOTR ty files is not longer valid (at least for me). Is there any way I could get a copy of it to take a look at the audio?

Thanks,

Rowan

ViperX2
11-03-2004, 07:31 PM
Josh,

I know you said you probably have to pretty much redesign the conversion process to demux OTA ty files in order to get mpeg output with audio.

It would be really cool if you could add the ability to convert from a TY directly to an ATSC compliant TS (transport stream) so that people with hardware decoder cards and DVHS decks have fewer loss inducing conversions to do.

jdiner
11-03-2004, 08:09 PM
Speaking of which, when is jdiner going to address some of the issues we have identified with HDtytool and when is it going to be merged back into the official tytool program?

I have gotten bit many times by running the wrong program when I forgot whether the prog to be converted was High def or SD.

Unless I missed a post in this thread, the most recent version of HDtytool that I have is from August.
What exactly has been identified? I asked about what had been found and received no clear answers or even lists of issues.

I put a bunch of time in the GopEditor portion to try and get it out the door but was unable to do so before I ran out of time. I have had several business trips in the last month and more to come. As a result I am running low on free time of any kind.

Further please keep in mind that just because an issue has been located it not always "all of the issue" just in locating it. Some of what I am fighting with on things that are OTA are a real mess and take time to get stability out of them much less a true fix.

--jdiner

jdiner
11-03-2004, 08:10 PM
Josh,

I know you said you probably have to pretty much redesign the conversion process to demux OTA ty files in order to get mpeg output with audio.

It would be really cool if you could add the ability to convert from a TY directly to an ATSC compliant TS (transport stream) so that people with hardware decoder cards and DVHS decks have fewer loss inducing conversions to do.
This one has already been answered.

--jdiner

desplaines
11-04-2004, 01:48 AM
I just played the vob of ROTK in WinDVD6 with a Vob extension (I used old VOB offload). I could not play a TS file like VLC or MyHD so I stand corrected. Why not play the intermediate VOB?

Tim
Yes, I can play VOB files OK. The value of using TS files would be that it is the most 'universal' file format in the HD world currently.

ViperX2
11-04-2004, 02:12 AM
I'd like to use HDTyTool to edit out commercials from a ty file and leave it in ty format. I made the the cuts using the GOP Editor but I don't see a way to make HDTyTool process the video and save it off in ty format. It always converts it to mpeg (with no audio for an OTA stream).

Is there something I'm missing or another tool you guys can point me to? I'd like to continue use the GOPEditor if possible.

On another note:
Since I got no response from Josh regarding the OTA audio fix, I assume he's still working on it.

I don't want to sound ungrateful, but just wanted to see if the issue had been resolved or if the wait continues.

You guys rock.

malfunct
11-04-2004, 02:14 AM
I'd like to use HDTyTool to edit out commercials from a ty file and leave it in ty format. I made the the cuts using the GOP Editor but I don't see a way to make HDTyTool process the video and save it off in ty format. It always converts it to mpeg (with no audio for an OTA stream).

Is there something I'm missing or another tool you guys can point me to? I'd like to continue use the GOPEditor if possible.

On another note:
Since I got no response from Josh regarding the OTA audio fix, I assume he's still working on it.

I don't want to sound ungrateful, but just wanted to see if the issue had been resolved or if the wait continues.

You guys rock.

Tytool is not capable of outputting .ty files.

ViperX2
11-04-2004, 03:37 PM
Josh,

Wouldn't it be trivial to add the ability to HDTyTool to write out a ty file with cuts?

redstone
11-04-2004, 07:56 PM
Redstone,

The link to the LOTR ty files is not longer valid (at least for me). Is there any way I could get a copy of it to take a look at the audio?

Thanks,

Rowan

That link I posted is an ftp site of mine and it has a 1.1 GByte limit.

I uploaded 1 Gbytes worth of data from ROTK for jdiner to use there but no one was hitting it so I deleted it because I had other stuff I wanted that space for.

If that data is still desired, I can kick off an upload tonight.

Post back and let me know.

redstone
11-04-2004, 08:16 PM
What exactly has been identified? I asked about what had been found and received no clear answers or even lists of issues.


--jdiner

1) OTA recordings
2) Do we use NEW or 'OLD' VOB/Mpg Format option?
3) One of those in #2 (forget which now) produces files which don't play in WinDVD or is it PowerDVD
4) The MYHD issue: New format option produces files where using FF kills the audio and OLD format option produces files where their is no audio. Something like that.

These are all clearly addressed and listed in this thread.

Regarding my forgetting exactly which is which in #2 through #4.
I am simply archiving to dds4 tape the High def .ty files of the few keepers waiting for updates to HDtytool so it will produce MYHD playable files without having to post process through VLC so I have not used HDtytool in quite a long time.

redstone
11-04-2004, 08:19 PM
Yes, I can play VOB files OK. The value of using TS files would be that it is the most 'universal' file format in the HD world currently.

That is why the author of Dvhstool is working on converting mpg or vob files to TS files.
He has the required $2500 SDK but I am not sure of his status with the project.

jdiner
11-04-2004, 11:24 PM
Josh,

Wouldn't it be trivial to add the ability to HDTyTool to write out a ty file with cuts?
No. Not even close to trivial. Please see any one of the hundreds of posts on the subject by myself and others. The get a compiler and try it yourself until the reasons become clear and the reality of the situation sinks in.

--jdiner

jdiner
11-05-2004, 12:28 AM
That link I posted is an ftp site of mine and it has a 1.1 GByte limit.

I uploaded 1 Gbytes worth of data from ROTK for jdiner to use there but no one was hitting it so I deleted it because I had other stuff I wanted that space for.

Oh I missed the message. Don't even know what the data was for. I.e. what was hoped to be accomplished by my getting it.

--jdiner

jdiner
11-05-2004, 12:37 AM
1) OTA recordings
The sheer variety is the problem right now. I am working on the splitting engine to get a proper parse of the input data but so far it is still just a work in progress.


2) Do we use NEW or 'OLD' VOB/Mpg Format option?
I answered this one a couple of times. The old format is the one to use. The new format uses the DVD spec which fixes the mux rate to a very LOW data rate for HiDef data. You will get behind in the SCR/PACK relationship before you hit the second I-Frame in the stream. OLD is the only viable option right now.


3) One of those in #2 (forget which now) produces files which don't play in WinDVD or is it PowerDVD
My guess would be the NEW format given the coupling between data rate and mux pattern.


4) The MYHD issue: New format option produces files where using FF kills the audio and OLD format option produces files where their is no audio. Something like that.
This is part of #1 from the code standpoint. All will be well once I finally get things being recognized right. The problem is that from a standard def tivo, stand alone or DTivo, you get 1 audio record per audio block. The code was designed to use only that as that was all there ever was.

The MPEG/VOB importer uses a much more complex algorithm to split up the multiple pieces that can occur in that format.

So while I have the code for the actual split written and working there are a few considerable inconsistancies between the format, my own lack of free time, and other issues between me and the finish line. And that is the audio portion. The video wierdness is where most of the work right now is taking place.


These are all clearly addressed and listed in this thread.
Most likely they are. In and around requests for other features, arguments about hardware and other options for recording/playback/etc... I don't have time to read thing with the care I would like. There have been weeks recently where I can come up with about an hour to look at or work on anything. I can easily spend that hour here on the forum and get nothing done or I can do what I can on other fronts.

Also keep in mind that the 2 main issues here, better/working OTA support and better output on the mux, are the 2 things I have listed several times as being at the top of my HiDef todo list. It is not like they have been forgotten.

Also everyone should remember that I don't even own a HiDef system of any flavour. People would either laugh or cry at my own personal HT setup. So I am doing this to try and help a relatively new and small block of users that feel the same desperation the rest of us did a couple of years ago when they could extact but not process things from the original DTivos. I also have my own personal list of wants that I put off to work on this stuff. So please cut me some slack. I am doing what I can.

--jdiner

jdiner
11-05-2004, 12:41 AM
That is why the author of Dvhstool is working on converting mpg or vob files to TS files.
He has the required $2500 SDK but I am not sure of his status with the project.
It will be interesting to see what happens there. I have a TS output system about 50% done. There are some elements that appear to be requried that I don't have implemented yet but I know how to mux files. I had no idea there were mass market playback devices, even if high end, that used them.

If he gets there first then it is done and I can quit worrying about it and will just let it go. If not then I will get it in there somewhere in the next couple of releases.

The reason I started on it at all was that Transport Stream (TS) mux'es are quite a bit more flexible for the data rate issues. They CAN NOT be used on DVDs, from what I understand they never will be able to be used that way. But these JVC devices people talk about will work, and playback is possible with the PowerDVD application and others. So rather than trying yet again with the PS mux I figured it was easier just to put the TS mux in place.

--jdiner

redstone
11-05-2004, 07:27 AM
I answered this one a couple of times. The old format is the one to use. OLD is the only viable option right now.

:
:
:
:
My guess would be the NEW format given the coupling between data rate and mux pattern.



So please cut me some slack. I am doing what I can.

--jdiner

I know you are doing what you can and it is appreciated (I think I have proven that to you via paypal). I am just keeping HDtytool in the limelight so it continues to progress.


Your answer's above show exactly what I said in my earlier post about issues that need to be addressed.

You yourself just said to sometimes use OLD format and sometimes use NEW format. I consider a 'hit or miss' approach to be an issue.

I am aware that you ran into an issue with audio on OTA material. For those of us with High Def tuner cards in our PC's or STB's that support the JVC DVHS, that is a low priority item for us since these tuner cards record OTA material.

We have a large selection of tools to put these on various storage medium as well as postprocessing since High Def OTA has been around for awhile.
On the other hand, You are the one breaking new ground because you are dealing with High Def from a DVR.


Keep in mind that my comments are meant to be constructive NOT confrontational and I support your efforts 100%.
With Tystudio a dead project, you have the ONLY tool that we HDTIVO owner's have for making playable High Def extractions.


Final question that was not answered (at least that I could find). When are you going to merge this code into the main branch of Tytoolr5vXX so we have one program to use for SD and High def?

I get lots of emails/PM's from folks asking for help with errors when processing High Def Material with Tytool and I always end up telling them to hunt down HDtytool.

After all, the Tytool thread always stays at the top of this bbs whereas this HR10-250 thread 'slides' down the page.

Thanks

redstone
11-05-2004, 07:48 AM
It will be interesting to see what happens there. I have a TS output system about 50% done.


If he gets there first then it is done and I can quit worrying about it and will just let it go. If not then I will get it in there somewhere in the next couple of releases.

--jdiner


Glad to hear that you are looking into this as well in case he gets stuck.

I will keep the other fellow's efforts/status up todate in this thread because it is germaine to this thread and I am sure it is of interest to HR10-250 owners.

jdiner
11-05-2004, 05:56 PM
Final question that was not answered (at least that I could find). When are you going to merge this code into the main branch of Tytoolr5vXX so we have one program to use for SD and High def?
The next release will be both together in 1 application in a single release.

--jdiner

jdiner
11-06-2004, 04:33 AM
Alright. The major issue for the next stage of the release has been completed. All of the resize/rendering issues have been fixed for these insane resolutions everone wants to use.

Now I have to fix the FAE part and that portion should be ready. That part does not address the OTA issues, this is just the editing side of things for HiDef users.

--jdiner

jdiner
11-06-2004, 05:37 PM
More up to date information. Have the GopEditor loading the data for FAE editing now. Again still not part of the OTA issues but the editing is working now as far as things have been working in TyTool to date.

I am going to spend some more time on the OTA problems for a bit before making the release to try and get it all out at the same time.

--jdiner

patriot
11-06-2004, 09:00 PM
read message below

patriot
11-06-2004, 09:00 PM
More up to date information. Have the GopEditor loading the data for FAE editing now. Again still not part of the OTA issues but the editing is working now as far as things have been working in TyTool to date.

I am going to spend some more time on the OTA problems for a bit before making the release to try and get it all out at the same time.

--jdiner

Tytools has been working out great! Cant wait to get my WB and UPN shows that are OTA going too!

Thx jdiner for your hard work and making Tytools workout for us!

I only wish Direc-tv was as good about letting us have the Sat channels that are HD, or picked up WB and UPN on their line of networks carried on Sat.

rudolpht
11-06-2004, 10:20 PM
I was wondering if the array or whatever the issue for "get parts" that did dropped off the last few files on very long recordings, e.g., LOTR: Return of the King.

From myself also, your work is much appreciated. Thanks!

Tim

jdiner
11-07-2004, 03:44 AM
I was wondering if the array or whatever the issue for "get parts" that did dropped off the last few files on very long recordings, e.g., LOTR: Return of the King.

From myself also, your work is much appreciated. Thanks!
No it is on the list. Just not very high. With actual functional bugs with no work arounds taking a higher priority. While get parts is a pain in comparison you can still get the entire show.

--jdiner

jdiner
11-13-2004, 04:28 AM
Well it took longer than I planned but I finally get the speed up I wanted when resizing the image for the GopEditor. It was so slow before on my machines as to be unusable.

Funny how thinking about things the normal way can often make things slower than needed.

So now all I have to do is get the OTA issues finished and it will be ready. I plan to make a release before I get back to that. As I have no idea when I am going to have it finished.

So that for those getting DTV source streams they can test the editing processes with HiDef streams.

Those having no luck with OTA source will still have no luck with OTA sources in this release as none of the dev code will be in this release.

--jdiner

mikemav
11-13-2004, 10:23 AM
Well it took longer than I planned but I finally get the speed up I wanted when resizing the image for the GopEditor. It was so slow before on my machines as to be unusable.

Funny how thinking about things the normal way can often make things slower than needed.

So now all I have to do is get the OTA issues finished and it will be ready. I plan to make a release before I get back to that. As I have no idea when I am going to have it finished.

So that for those getting DTV source streams they can test the editing processes with HiDef streams.

Those having no luck with OTA source will still have no luck with OTA sources in this release as none of the dev code will be in this release.

--jdiner
Jdiner- thanks for the continued development on this. I have not used your tool before for much other than HD archiving (via conversion to MPEG or HD VOB) Will the resize you mention give us the option of converting HD programs to DVD resolution (ideally anamorphic, if that is possible, for max res) for playback on set top DVDs?

jdiner
11-13-2004, 03:52 PM
Will the resize you mention give us the option of converting HD programs to DVD resolution (ideally anamorphic, if that is possible, for max res) for playback on set top DVDs?
No. What it is for is the playback during the editing process. Some of the test streams I have have extreme resolutions in them.

One clip I have is from "Law & Order" and while it has probably the most amazing picture quality I have ever seen it is 1920x1088. I run my monitors at 1600x1200 which means that even on my setup I can't see the whole window.

So the resize is used to force things down to a more usable setting. There are 3 options: no resize, 1/2 resize, and 1/4 resize. Note that all are downsizing not upsizing of the video data.

No resize mode is the fastest mode as no further work is done altering the data the frame is decoded and displayed on the screen.

1/2 mode is well exactly that you cut the displayed image exactly in half. For most of the HiDef streams I have this gives a nice ouput and keeps the window within reason.

For those running on older equipment or using a TV as the display mechanism this will still be too much. So I hadded the 1/4 mode. Even for the Law & Order clips which is the largest res one I have, the results are smaller than the original 480x480 that you get from an S1/S2 DTivo. Makes it usable on a smaller format display. (Within reason my best guess is that trying to do this on a cell phone still won't work... :)

What you seem to be asking about is transcoding which on the face of it is a simple piece of work:

decode the stream -> alter the data to match what you want -> re-encode the altered stream

Now TyTool after a fashion does the first one. For visual editing purposes GopEditor does the second. And TyTool does an extremely limited portion of the 3rd 1 GOP at a time around FAE cuts.

So while things are somewhat "close" to being able to do transcoding I have no intention right now of closing that gap and actually adding it. I have thought about adding such an engine so that people could seemlessly make AVIs or DIVx or ... this way but there are already other tools out there that do this. I know from this thread that rc3105 does it so he can watch stuff on his XBOX. So while there is something else that already does it I will focus on things that still need to be done instead.

--jdiner

mikemav
11-13-2004, 04:09 PM
Thanks for the explanation. Since I had not used the editor, now this makes sense.

One more quick question or feature request: a lot of new boxes are coming out the offer possible playback of HD programs. I am referring to set-top possible HTPC replacements, such as a networked DVD player or media player. The new Sigma chip offers decoding up to and including HD resolutions. An example of a device using this is the new I-O Data AveLink networked DVD. Other HD media players include the Roku viewer. Anyway, the reason I bring this up is, they all have one big limitation when it comes to files originally sourced from an HD TiVo: a 2GB file size limitation.

When I use a MyHD card to capture .ts files from OTA, there is a setting to keep file size @ "X" (I use 1900MB for reasons shown above.) Therefore, larger shows are recorded as Law_and_Order_01.ts, Law_and_Order_02.ts, etc.. For playback, it knows to play sequential #s in titles back.

Since the TiVo cannot do this w/ recordings, I find myself adding a step to redstone's suggested archiving method. My process is: mfs_ftp or Tytool to extract to ty+/ process old VOB method/ run VOB through VLC to get .ts/ run .ts through HDTVtoMPEG2 program to convert single large .ts file into multiple 1.9GB 001.ts, 002.ts, etc.. files.

If TyTool could specify a max output size, and auto-number sequential outputs above such a size, it would be a huge time saver. I know you do not create transport streams, but evidence shows with these linux based set top boxes, program streams may be better anyway. So if we could get .MPEG or .VOB files at a max size, sequentially numbered in TyTool, it would be wonderful. Not sure how tough that is to code, though.

Thanks again for all your support. Have a nice weekend all.

toyotafan
11-13-2004, 04:45 PM
I just got my hdtivo hacked and have successfully transferred a ty file to my pc. When I process it with HDTytool using VOB-Mux File I get the error message below.

Ready...

Mux'ing (with cuts): C:\Tivo\{ Recording}{}{1970-01-01}{}{11.28 AM Sat Nov 13, 2004}{HDN}.ty
Sorry... Failed to get the first 10 initial chunks...
Have to have at least that many to start the analysis phase...
DiffTime = 0.000000 (0) == 0.000000 Minutes
total = 1310720 (1 MB)
Done with 'C:\Tivo\{ Recording}{}{1970-01-01}{}{11.28 AM Sat Nov 13, 2004}{HDN}.ty'...

toyotafan
11-13-2004, 05:16 PM
Never mind, encryption is still turned on.

malfunct
11-13-2004, 09:28 PM
I just got my hdtivo hacked and have successfully transferred a ty file to my pc. When I process it with HDTytool using VOB-Mux File I get the error message below.

Ready...

Mux'ing (with cuts): C:\Tivo\{ Recording}{}{1970-01-01}{}{11.28 AM Sat Nov 13, 2004}{HDN}.ty
Sorry... Failed to get the first 10 initial chunks...
Have to have at least that many to start the analysis phase...
DiffTime = 0.000000 (0) == 0.000000 Minutes
total = 1310720 (1 MB)
Done with 'C:\Tivo\{ Recording}{}{1970-01-01}{}{11.28 AM Sat Nov 13, 2004}{HDN}.ty'...

Sounds like the file is still scrambled.

jdiner
11-15-2004, 05:46 PM
Well we have some HiDef issues... :(

I thought I had things licked and I do sort of. But for FAE editing the recompression tool, mjpegtools mpeg2enc, is refusing to re-process the data at even the smallest HD resolution I have, 1280x720 much less work at the largets which is 1920x1088.

So are any of you doing re-encoding without changing the size and if so how? I might just need to plug in the ability to use another tool. I do believe that tmpgenc will re-encode at that resolution and someone a long time ago mentioned seeing/using/heard that tmpgenc could be used as a second stage encoder like what I am using now.

Anyone have any further informtion on this. Until I can get re-encoding working for HiDef resolutions I can't get any further on that front.

--jdiner

toyotafan
11-15-2004, 05:49 PM
Yep, for some reason the hack for disabling encryption didn't take the first time. Got it fixed now and am able to play back files.

redstone
11-15-2004, 07:55 PM
Where do you stand on the NEW vs 'OLD' VOB/Mpg Format option?

jdiner
11-15-2004, 09:25 PM
Where do you stand on the NEW vs 'OLD' VOB/Mpg Format option?
If this question is directed at me, the same place as where I did before. Only the old will work.

What I have been working on is getting editing/FAE to work for these streams.

--jdiner

ronnythunder
11-15-2004, 09:50 PM
But for FAE editing the recompression tool, mjpegtools mpeg2enc, is refusing to re-process the data at even the smallest HD resolution I have, 1280x720 much less work at the largets which is 1920x1088.josh, what kind of errors are you getting from mpeg2enc? perhaps one of us could help you by debugging?

ronny

jdiner
11-15-2004, 10:29 PM
josh, what kind of errors are you getting from mpeg2enc? perhaps one of us could help you by debugging?
I get this:

**ERROR: [mpeg2enc] Horizontal size is greater than permitted in specified Level

It just won't take anything larger than 720.

I could hack into the source but that is never a great idea with no knowledge of internal buffer sizes etc... it could just be begging for trouble.

I searched their mailing list and so far a few people asked about it back in 2003. But nothing seems to have been done about it.

--jdiner

joeblough
11-16-2004, 06:39 PM
have you tried mencoder? i use this all the time to transcode 1080i and 720p to mpeg4. i dont know how well it works transcoding mpeg2 to mpeg2 though.

i believe that mencoder/mplayer uses the ffmpeg library (which i think is the same thing that mjpegtools is based on) to decode mpeg2. i get console warning about the size exceeding the MP@ML1 (i think that's what it's called) profile, but it is just a warning and the output is fine. so i wonder if you can somehow just disable this in mpeg2enc.


Well we have some HiDef issues... :(

I thought I had things licked and I do sort of. But for FAE editing the recompression tool, mjpegtools mpeg2enc, is refusing to re-process the data at even the smallest HD resolution I have, 1280x720 much less work at the largets which is 1920x1088.

So are any of you doing re-encoding without changing the size and if so how? I might just need to plug in the ability to use another tool. I do believe that tmpgenc will re-encode at that resolution and someone a long time ago mentioned seeing/using/heard that tmpgenc could be used as a second stage encoder like what I am using now.

Anyone have any further informtion on this. Until I can get re-encoding working for HiDef resolutions I can't get any further on that front.

--jdiner

jdiner
11-18-2004, 04:37 AM
Alright for those not following the TyTool main thread I managed to get the 1.6.2 version of mpeg2enc compiled. It is a touch newer than what has been in use with a few new features.

It now works to FAE cut HiDef streams as it can recompress the needed sections.

The problem is that when I tried to compile it is just plain screwing up. When I turn off MMX/SSE so that there are no simd enhancements it compiles. But doing so, turning things off that is, it makes things run a touch slower. Luckily when cutting not that much is recompressed but any and all speed is pretty much the goal.

Now others have obviously figured out how to make it work as it works in the version that we are currently using with TyTool for SD streams. However I can't make it compile and I have gone as far towards trying to make it as I plan too... So if anyone here wants a challenge then please feel free. If anyone here wants to search around or ask on the various mjpegtools forums and relates sites for a Win32 version that has MMX/SSE in it then please feel free. If we can find one I would love to make it the official version for use with TyTool. If not then you are all just going to have to put up with what I could build when cutting/editing.

--jdiner

desplaines
11-19-2004, 04:46 AM
jdiner-

Thank you for your persistence with the HD-related problems. I will try your new editing tools, and let you know if I find anything that will be of use to you.

toyotafan
11-20-2004, 01:09 PM
jdiner, thanks for all of your efforts on developing HDTytool!

I'm having a problem with my local NBC and UPN OTA HD channels. No problem extracting and processing the files but there is no audio when I play them back. The audio track listed during processing (VOB-Mux files) in hdtytool is mpeg layer2. Any ideas?

Mux'ing (with cuts): D:\HiDef\The Tonight Show With Jay Leno-.ty
Detected Tivo Type: HDTivo
Detected Audio Stream Type: MPEG Layer II
Final standardAudioSize = 6160
Final standardFrameLength = 0
Final standardAudioDiff = 6480 or 00:00:00.072
First Video PTS: 08:41:35.341

Steve

gottahavit
11-20-2004, 01:48 PM
Great tool thanks.

I am having one issue with extraction that I cannot find reference to here and I am curious if anyone else is having this issue and whether there is a solution.

When I extract(either tserver or mfs_ftp), I then run HDtytool to make the keys files and I edit the file to .mpg. When I go to playback that mpeg file in media player it reports the length as considerably longer than it actually is. This also presents itself in all the other mpeg editing tools(tmpgenc, xmpeg) as it reports there are more frames than there are. Now everything plays back perfectly in synch but when it reach the last actualy frame it just replays the last frame over and over until it get to waht it thinks is the end. If this were the end of things I wouldn't care, but when I go to convert to IFO using tytools it start spitting audio out of range errors when it gets to that point in the audio processing and creates a DVD that is all messed up. Using Tmpgenc DVD author I am able to make a DVD that works because somewhere along th eline it removes the erroneous information. I will also point out that all the other tools actually make the mpg files large enough to fit the information that is actually not there so there is a lot of wasted space.

Note: This acutally seems to have nothing to do with the cuting process as this happens even if I go directly from the original TY file to mpg.

The audio is AC3 5.1( I have not tried a file without 5.1)

any help is appreciated as I don't like using tmpeg DVD author and it is very difficult to judge the final file size as it stands now.

jdiner
11-20-2004, 06:02 PM
I'm having a problem with my local NBC and UPN OTA HD channels. No problem extracting and processing the files but there is no audio when I play them back. The audio track listed during processing (VOB-Mux files) in hdtytool is mpeg layer2. Any ideas?
As has been stated a bunch recently there is no real support for OTA streams yet. They are a nightmare to try and deal with. More to come on that front but what is out there now is a 10% solution at best.

--jdiner

jdiner
11-20-2004, 06:04 PM
I am having one issue with extraction that I cannot find reference to here and I am curious if anyone else is having this issue and whether there is a solution.
Sounds like you are using the "new format" muxer and not the old format muxer.

--jdiner

gottahavit
11-20-2004, 06:11 PM
Sounds like you are using the "new format" muxer and not the old format muxer.

--jdiner
I think I tried both, but I can try agian. Is this an issue with the new muxer?

gottahavit
11-20-2004, 06:30 PM
I think I tried both, but I can try agian. Is this an issue with the new muxer?

ok, you were right, don't have this problem with old muxer. Thanks for the quick reply.

ok so now I'm able to turn my HD programs with 5.1 into DVDs with stereo with no issues. I would however love to be able to preserve the 5.1. I can extract the ac3 from the click then downrez in tmpgenc, however whenever I remux the acs with tmpgenc tools the audio is off. Any way to adjust the audio synch like you can with vdub and divx conversions?

jdiner
11-21-2004, 03:28 AM
ok, you were right, don't have this problem with old muxer. Thanks for the quick reply.
Yeah. Re-read the post I made in this and the TyTool threads about the difference between the 2 muxers and how they work.


ok so now I'm able to turn my HD programs with 5.1 into DVDs with stereo with no issues. I would however love to be able to preserve the 5.1. I can extract the ac3 from the click then downrez in tmpgenc, however whenever I remux the acs with tmpgenc tools the audio is off. Any way to adjust the audio synch like you can with vdub and divx conversions?
Run a search. Ignoring the hidef nature of what you are trying to do this has been covered in depth.

--jdiner

redstone
11-22-2004, 04:24 PM
Been corresponding with a fellow who has the AVeLLink DVD player.
It can be found here:
$249 http://www.iodata.com

It may be another way of playing the VOB files from HDtytool from a networked drive and not need Powerstrip,Trancoders and never quite right custom T&R's. Plus may elimnate the need for all the post VOB processing that MYHD users have to do.


I asked him a bunch of questions and also asked him to play a 1 Gbyte clip of a VOB file I made with HDtytool which is here:http://southriver.web.aplus.net/

The VOB file had NO post processing beyond the initial .ty to .vob with HDtytool.

----------------------------------------------------------------------
Q/A:


1) Will the Avelink play VOB or mpg files in full resolution like the MYHD card will for non-copyrigthed files?
OR - does it require transport stream files?

It will play vob files. I have tried a few straight dvd vobs and they seem to work fine.

2) Do you have any audio/video issues.ie. dropouts when playing over the network?

Not so far.
The aspect ratio controls don't seem to work right, and it looks like it is overscanning too much on the left and right of the HD Picture. Kinda buggy product right now. Support says a new firmware is due in Dec.



3) If I plug in an external USB disk to the avelink, will it play the movies which are stored on that?

It is supposed to support that but I haven't tried it.

4) How much does it cost?

$249 http://www.iodata.com

5) Where did you get it?

see above

6) Assuming the Avelink can do #1 Do you have a wideband connection where I could have you download a 1 Gbyte High Def sample VOB file made form the HDTIVO to try out on the avelink?

Yes and here are the Playback results of ROTK:


It seems to play fine. The fast forward has 1x, 2x, 3x, and 4x and works but is ****yy, and when you hit play the audio sometimes stutters (1x and 2x seem to work flawlessly, where 3x and 4x cause stuttering of the audio on play). If you pause and then play the audio is fine. Normal playback works great though.


---------------------------------------------------------------------------

To use MYHD on files made from the TIVO we currently have to do this to fix the permanent loss of audio when using FF or RW:
.ty --->HDtytool--->.vob ---->VLC --->.tp ---> mpeg2repair (sometimes)--->tp which is DUMB.

I may be buying this DVD player as a Xmas present to myself.

mikemav
11-22-2004, 08:09 PM
Redstone-
I have been following the exact same thread on the I-O, and I was going to try to reach out to one of the early adopters this weekend. I too want to trash my HTPC one day. Missing for me on this are internet radio and playback of .IFO files (so ripped DVD movies can be played back with menus.)

Glad you have contacted him. I am also looking closely at the KISS DP-600, which uses the same chipset but also may software server support for internet radio & Mac iTunes syncing. It is due out the end of the month my industry sources tell me. If I can get one of those, I may go that way. Support of TiVo should be the same, since it is the same Sigma chip.

Also, one nice thing is that I-O has posted (just today) a timeline and map of future development on this product. They even are working on an Linux version of their server. Seems like (despite the horrible english) they are interested in what the US market has to say. I wonder if they would be able to add playback of .ty files? After all, VLC and M-player can play them native. Wouldn't that be great?

redstone
11-22-2004, 09:28 PM
[QUOTE=mikemav]Redstone-
I have been following the exact same thread on the I-O, and I was going to try to reach out to one of the early adopters this weekend. ?[/QUOTE


I was not aware that there was a thread devoted to it. I spotted a post by a guy in the R5000 thread about the DVD player and PM'd him.

What is the thread url?

Thanks

redstone
11-22-2004, 09:33 PM
IN case folks have not spotted the work being done on network speed, there is a set of USB device drivers that have been optimized as well as code changes to the tools that use them.

I am now getting around 3.15 Mbytes/second upload using the new tserver with the TIVO idle (tuned to channels 888 and 889).

Just got it all working so have not had a chance to benchmark yet when the HDTIVO is more busy.

It is an encouraging start since it sure beats 1.5 Mbytes/second under the same conditions.


More later

jdiner
11-23-2004, 12:11 AM
IN case folks have not spotted the work being done on network speed, there is a set of USB device drivers that have been optimized as well as code changes to the tools that use them.

I am now getting around 3.15 Mbytes/second upload using the new tserver with the TIVO idle (tuned to channels 888 and 889).

I have not seen these. Where are these drivers and the updated tserver located at? I would like to see what they did and see about getting the changes incorporated into the major new release that is coming.

--jdiner

ronnythunder
11-23-2004, 01:05 AM
here's the link to the files; there's also a support thread. http://www.dealdatabase.com/forum/showthread.php?t=39487

it's a pretty complete reworking of tridge's stuff and the stuff that's crept in following. got rid of some double buffering, unneeded moves etc.

ronny

lsmod
11-23-2004, 04:19 AM
Been corresponding with a fellow who has the AVeLLink DVD player.
...
The VOB file had NO post processing beyond the initial .ty to .vob with HDtytool.


Oh yeah!

As soon as the toy fund has recovered from the HDTiVo itself, this is on the list. The biggest problem I was facing was that HTPCs aren't normally "wife-compatible", but this would do nicely.

Keep us posted.

mikemav
11-23-2004, 10:41 AM
[QUOTE=mikemav]Redstone-
I have been following the exact same thread on the I-O, and I was going to try to reach out to one of the early adopters this weekend. ?[/QUOTE


I was not aware that there was a thread devoted to it. I spotted a post by a guy in the R5000 thread about the DVD player and PM'd him.

What is the thread url?

Thanks
It's up to 28 pages now, with several people in the US and abroad that have the different versions of this networked DVD player. Link is here (http://www.avsforum.com/avs-vb/showthread.php?s=&threadid=459759&perpage=20&pagenumber=1) I am waiting for the KISS DP-600 (at least to see if it truly comes out the end of this month) since the US version of the I-O has no DVI. They say a DVI version, like the Japan released one, will be to the US in early 2005.

jdiner
11-23-2004, 03:52 PM
here's the link to the files; there's also a support thread. http://www.dealdatabase.com/forum/showthread.php?t=39487

it's a pretty complete reworking of tridge's stuff and the stuff that's crept in following. got rid of some double buffering, unneeded moves etc.
I am glad someone beat me to it. I was planning to go over the code with a fine tooth comb this long weekend. Now I don't have to worry about it. :)

--jdiner

jdiner
12-01-2004, 07:37 PM
I posted this in the main TyTool thread but since it was done for those here I thought I would post it again here...

------------------------------------

Alright preferences anyone?

I was asked by the HDTivo group to alter the main display so that it showed the correct size. The formated display in use would turn a show over 9,999meg into 0 meg, then 1 meg and so on.

So I did it. But in looking at it it is hard to tell what would be best. We have 2 options right now. Alter anything over 1024meg into a gig which means many shows on a DTivo and most on an SATivo would change, as would most on the HDTivo.

However, and perhaps I have merely gotten used to it, I kind of prefer the meg count we currently have. Where you see 840, 950, 1286 as the first 3 shows I have on my DTivo right now.

Which means I have test another option. Put the roll over at 9,999. Meaning anything past what can be display currently rolls over.

Any thoughts?

--jdiner

redstone
12-02-2004, 05:08 PM
Today, you use 4 digits for the size so for example, 5362m is no problem to see that a show is 5.36 Gbytes.

A four hour High Def recording could theoretically be 32 Gbytes so
why can't you just add another digit to the display to keep the units in megs:

EG.123456m would be 123.4 Gbytes

and like you do now, leave out leading zero

jdiner
12-02-2004, 06:50 PM
A four hour High Def recording could theoretically be 32 Gbytes so why can't you just add another digit to the display to keep the units in megs:
In the end this is what I did.

--jdiner

jdiner
12-02-2004, 06:57 PM
Ok... Calling all cars.

I need peoples help if I am going to get the OTA problems fixed. I can only do so much with what I have gotten so far. I ran into the same problem with DTivos and in the end people contributed and I was able to purchase one. But with a price tag of over $1k I think getting those kinds of contributions isn't going to happen.

So what I need is a plethora of OTA sample files. Different channels, different shows, different regions/cities/states, etc...

I thought I had the problems licked and then I went onto a different newer clip I got from someone directly. It shows errors in a new way. So I have to start over a bit on a few things in my design to better support what is going on. Before I do I want to get the best understanding of where the flexs in the format are occuring then get a proper design done and get a truly working solution in place.

So what this means. I need DVDs full of portions/segments of shows from as many people as will burn one, and hopefully that is at least several people, with the OTA data in their area.

Once I can see what everyone is considering legal to use I should be able to beat this problem once and for all.

So sound off those that are willing to burn a DVD filled with OTA clips and I will send out my address for people to mail them too.

--jdiner

snoots
12-02-2004, 07:20 PM
Be glad to help out, please let me know what you need, where to send it and which process you want used to create the disks.

mikemav
12-02-2004, 09:30 PM
Me too, lmk how. I can get HD stuff to fill up a DVD-R. I can try 720p (Fox or ABC) and 1080i (other networks), as well as up-converted "fake" HD (such as the local news & many 1/2 shows in prime time.) While these are technically 1080i or 720p, they are 4:3 up-converted at the station in a 16:9 window, and not 5.1 audio. Just tell me how long you want the clips to be each, and if you want ty+, ty, or tmf.

jdiner
12-02-2004, 10:33 PM
Ok. A public answer to the first part.

I need as many roughly 200meg pieces as I can get. I know that this is small for HD stuff but I don't need large files I need to see as many ways as things have been packed together as possible.

As for how to package them to send just data files on a DVD. As I would like to get as many as possible from as many different sources as each person has. Meaning specifically if you have 10 HiDef channels and 4 are OTA please send me segments from all 10 channels but focus on the OTA to say about 75% or so.

The more formatting issues I can find the more robust this whole thing can be made.

--jdiner

jdiner
12-02-2004, 10:36 PM
Me too, lmk how. I can get HD stuff to fill up a DVD-R. I can try 720p (Fox or ABC) and 1080i (other networks), as well as up-converted "fake" HD (such as the local news & many 1/2 shows in prime time.) While these are technically 1080i or 720p, they are 4:3 up-converted at the station in a 16:9 window, and not 5.1 audio. Just tell me how long you want the clips to be each, and if you want ty+, ty, or tmf.
All of the above please.

200 meg chunks of TyStream files would be the thing I would want. You can use the TyFileSplit tool from the recent TyTool releases, or from older archives here on DealDB.

I mean it, some of all of the above would be great.

Thanks,
--jdiner

toyotafan
12-02-2004, 10:45 PM
I'd be happy to send a bunch of samples. Shoot me an address to send the DVD to.

Steve

jdiner
12-03-2004, 02:35 AM
Alright the latest and greatest features including all of the current ready to be released HD stuff is now in TyTool 9r18 in the new 9r18 thread.

TyTool9r18 (http://www.dealdatabase.com/forum/showthread.php?t=39740)

It includes the ability to FAE edit HiDef streams. Please read up carefully and then give it a shot.

--jdiner

desplaines
12-03-2004, 05:58 AM
I am also happy to send some samples-can I just download 'one segment' using TyTool for each show/channel I try?

redstone
12-03-2004, 08:45 AM
Let me know where to send the samples. I can give you data from all the channels from both Washington,DC and Baltimore

mikemav
12-03-2004, 11:23 AM
I am also happy to send some samples-can I just download 'one segment' using TyTool for each show/channel I try?

The way I did this in the past when I first got samples to put up on redstone's server was to record live TV on an HD station, and stop recording after about 60-90 seconds. I believe (IIRC) that 60 seconds was about 100MB on most stations. So perhaps you want to try a 1-2 minute recording at first. When you reach your count, stop recoding, then extract.

One question jdiner- I use mfs-ftp to extract usually. Is that okay for your tests, and would ty+ files be okay? (which is the usual method I personally use.)

jdiner
12-03-2004, 04:21 PM
I am also happy to send some samples-can I just download 'one segment' using TyTool for each show/channel I try?
Yes that would work. It will probably be more data than I need from each one but it will work.

If you want to chop it down to the 200meg size use TyFileSplit and a chunk count of 1600.

--jdiner

jdiner
12-03-2004, 04:23 PM
One question jdiner- I use mfs-ftp to extract usually. Is that okay for your tests, and would ty+ files be okay? (which is the usual method I personally use.)
That will be fine.

--jdiner

jdiner
12-03-2004, 04:42 PM
Look it wasn;t supposed to be anywhere near this hard.

I have answered the same questions many times now.

I need small segments where I can look at the internals of the file format for the OTA sources. Why? OTA is a feaking nightmare! Almost every stream I is different in some way. I want to be able to get a good cross section of the differences.

The specifics one last time:

1- I need at right around 200meg in each file to see how things are being packed together. How you get it is up to you, get a single segment/record for 2 minutes, whatever.

2- Each OTA channel in each area can be different so I want at least one from each that you can receive in your area.

3- TMF files are just a pain to deal with. So I want them in the standard format. Which means Ty or Ty+. Plain and simple.... Even if you use mfs_ftp you can still get these formats of files.

4- In the end I will have you mail me the DVD directly. I am not however going to put my own address here. Meaning you have to PM me for the address. I have kind of been watching to see when people posted here asking where to send it but I don't remember who I had sent it to and who I had not. So PM me and we can go from there.

That is it...

Now why specifically I am trying to do this. Here is a brief list of the oddities I have seen so far:

1- Just I-Frames. Makes the files huge but is extremely quick to compress as there is no motion search. In MPEG-2 compression motion search accounts for about 80% of the work.

2- Just I-Frames and P-Frames. Again faster than a full recompression but still a bit larger.

3- I-Frames P-Frames and more than 2 B-Frames in between. The standard format is IPBBPBBPBBPBBIBB and so on. I have seen one that would go something more like: IPBPBBBPBBPBPBBBBBPBPB and so on.

4- I have seen an entire GOP plugged together as a single HUGE, and I do mean HUGE, 4 meg video record.

5- In terms of audio I have seen 1 record per audio block. This is the standard tivo TY format.

6- 2 audio records per block, meaning the PTS has to be recalculated for the middle interior one and is not being done so. (The current problem with OTA recordings...)

7- I have seen a wierd mixed mode where some records are 1 long and others are up to 2 seconds of audio records long.

and so on. Once I can figure out where the variations are present I can rebuild the state machines in such a way that it will handle all of these cases.

So make the small records, put them onto a simple data formated DVD, and mail it to me...

--jdiner

moshmothma
12-06-2004, 11:24 PM
Jdiner - I am happy to send you lots of OTA files. I will PM you for addy.

Redstone - could you please contact me when you get a chance? My beloved Ravens are killing me! Thanks mimel1@hotmail.com

samhammer
12-07-2004, 12:49 AM
jdiner,

Have you gotten any clips or commitments from Denver staions?

Maybe you could periodicaly post where you have have received samples from.

George

jdiner
12-07-2004, 01:07 AM
Have you gotten any clips or commitments from Denver staions?

Maybe you could periodicaly post where you have have received samples from.
So far I have actually gotten anything. A few have been promised and I have sent my address to them but I have not gotten where they are from anyone yet.

If you want to speak for Denver feel free. PM me with a request for my address and I will send it to you in kind.

--jdiner

rung
12-07-2004, 05:58 AM
Now why specifically I am trying to do this. Here is a brief list of the oddities I have seen so far:

1- Just I-Frames. Makes the files huge but is extremely quick to compress as there is no motion search. In MPEG-2 compression motion search accounts for about 80% of the work.

2- Just I-Frames and P-Frames. Again faster than a full recompression but still a bit larger.

3- I-Frames P-Frames and more than 2 B-Frames in between. The standard format is IPBBPBBPBBPBBIBB and so on. I have seen one that would go something more like: IPBPBBBPBBPBPBBBBBPBPB and so on.

4- I have seen an entire GOP plugged together as a single HUGE, and I do mean HUGE, 4 meg video record.

5- In terms of audio I have seen 1 record per audio block. This is the standard tivo TY format.

6- 2 audio records per block, meaning the PTS has to be recalculated for the middle interior one and is not being done so. (The current problem with OTA recordings...)

7- I have seen a wierd mixed mode where some records are 1 long and others are up to 2 seconds of audio records long.



What kind of morons are setting up these HD Encoders anyway? Kind of reminds me of 1,000,000 monkeys trying to write Shakespeare.

Regards,
Rung

mikemav
12-07-2004, 09:53 AM
So far I have actually gotten anything. A few have been promised and I have sent my address to them but I have not gotten where they are from anyone yet.

If you want to speak for Denver feel free. PM me with a request for my address and I will send it to you in kind.

--jdiner

I hope to be able to get mine extracted, burned & mailed this weekend. Last weekend I had too many family obligations. I'd be glad to let you know when they are on the way. Thanks again, and sorry for the delay.

jdiner
12-07-2004, 02:55 PM
What kind of morons are setting up these HD Encoders anyway? Kind of reminds me of 1,000,000 monkeys trying to write Shakespeare.
Can't explain it but my thoughts exactly.

Now obvisously it is possible to be flexible enough to handle it, as an HDTivo will play it. But... I counted on things being the same when parsing TyStreams as they always had been in the past. Now, not so much.

--jdiner

toyotafan
12-08-2004, 03:53 AM
So far I have actually gotten anything. A few have been promised and I have sent my address to them but I have not gotten where they are from anyone yet.
--jdiner

I mailed a DVD out to you today from Seattle. Also made a donation.

Steve

jdiner
12-11-2004, 12:10 AM
Alright. So far I have gotten 1 DVD from, I believe, Snoots.

I have yet to receive any others.

Could anyone that has actually mailed out a DVD/CD with files on it please sound off. I would like to know how soon I can expect more so that I can decide what I want to work on next. I have a several things planned.

I need a good cross section before I can begin the next stage of HiDef development. But I don't want to be mid-way into something else that is large just to get what I need in the mail.

--jdiner

snoots
12-11-2004, 01:14 AM
yep , you got mine.
Snoots

samhammer
12-11-2004, 09:17 AM
I'll be collecting samples from Denver stations this weekend. (already have two but need to get HD from three others)

Will mail out on Monday.

I will pm you for your address

George

drewh
12-11-2004, 03:11 PM
I'll be sending you a collection of HD snippets from the SF Bay Area. I have many collected already, but will try to fill in all the other OTA stations I receive.

Please return my PM with a postal address.

Thanks so much!

Drew

jdiner
12-11-2004, 04:09 PM
Alright I replied with the address to the PM's I had waiting. Good to know I will be getting more sample files...

--jdiner

THardie
12-11-2004, 04:12 PM
Alright I replied with the address to the PM's I had waiting. Good to know I will be getting more sample files...

--jdiner
I didn't get your PM - I sent you a request last night...

Terry

jdiner
12-11-2004, 04:43 PM
I didn't get your PM - I sent you a request last night...
Yeah. After 2am. I was not at the computer until just now. I sent the PM less than 5 minutes before I posted in this thread. Should you still not have it in a few minutes, I don't know how long it takes the system to deliver a PM, I will resend it.

--jdiner

THardie
12-12-2004, 11:58 PM
Yeah. After 2am. I was not at the computer until just now. I sent the PM less than 5 minutes before I posted in this thread. Should you still not have it in a few minutes, I don't know how long it takes the system to deliver a PM, I will resend it.

--jdiner
Nope. Still don't have it. Can you resend, please?

tPeter42
12-13-2004, 04:05 AM
Just got my Tivo hacked this weekend. I will grab some samples when I have time this week from the Los Angeles area. Can you PM your address please?

Thanks,
Sean.

jdiner
12-13-2004, 01:37 PM
Just got my Tivo hacked this weekend. I will grab some samples when I have time this week from the Los Angeles area. Can you PM your address please?
Make sure things are extracting right, that scrambling is turned off etc... And then hit me with a PM when you have the disk ready.

--jdiner

jdiner
12-13-2004, 08:50 PM
Alright a second disk showed up in the mail. This one from Seattle. As more trickle in I will report what I discover about the nature of the streams.

--jdiner

samhammer
12-14-2004, 11:35 AM
I now have samples from the Denver stations. I was going to copy to DVD strickly as data via Nero InCD. You should be able to copy them off if you are running winXP.

Would you rather it come differently?

George

jdiner
12-14-2004, 05:14 PM
I now have samples from the Denver stations. I was going to copy to DVD strickly as data via Nero InCD. You should be able to copy them off if you are running winXP.

Would you rather it come differently?
That should be fine. As long as it is burned as a plain old data disk I should be able to get the TyStreams off and use the data.

--jdiner

lsmod
12-14-2004, 07:32 PM
plain old vanilla 733/64 softmod 1.1 xbox. boot a basic gentoo kernel with networking (no gui) then invoke a mplayer build with combined nvidia drivers ( essentially treating mplayer as though it were a mpeg decoder card or chip )

I'll probably get an AVeLLink2 eventually, but since I already have the Xbox....

Any further wisdom to someone who's about to try this?

Are you running 1080i or 720p out of the Xbox?

Thanks!

moshmothma
12-14-2004, 11:21 PM
I will be sending a disc out this weekend. Thanks for the hard work!!

mikemav
12-15-2004, 11:08 AM
I am currently having network problems & I yanked the HD TiVo network to enable my new Avel LinkPlayer network DVD. I will get a bigger router this weekend & have the TiVo back online, so I can make a disc to send out. Thanks

snoots
12-15-2004, 01:27 PM
I am hoping the 1st of the year advanced server "might" play .ty files over the network, since it supports "transcoding and all files playable by Windows Media Player" If the server side will use the installed "tyshow" filters it may work on the player. I have been using it for a while and playing back tivo files that have been converted to mpeg and of course the HD demo files in Divx and WMV9

mikemav
12-15-2004, 01:36 PM
I am hoping the 1st of the year advanced server "might" play .ty files over the network, since it supports "transcoding and all files playable by Windows Media Player" If the server side will use the installed "tyshow" filters it may work on the player. I have been using it for a while and playing back tivo files that have been converted to mpeg and of course the HD demo files in Divx and WMV9

Good info snoots. I did not know Media Player could play .ty files. I knew VLC & Mplayer did, but do you have to do something special to get .ty to play in WMP9 or 10?

drewh
12-15-2004, 01:42 PM
Josh,

After sending you a disc of OTA TY files I realized one file has a resolution change in it. One of the local stations broadcasts a "Currently Off-Air" message in SD and the TiVo started recording early (actually the station started late) and caught some of that before a resolution change to HD. There are some other degenerate problems with the beginning of this file - like the SD portion includes no audio.

If nothing else, it seem reasonable to expect to be able to edit out this SD section using tytool.

Unfortunately this resolution change did not make it into the randomly chosen snippet I provided you. Can I provide this to you in some electronic form? If not, and if you want to see it, I can mail it to you.

Has anyone else seen anything like this?

drewh

tPeter42
12-15-2004, 02:02 PM
I'm intentionally trying to get clips with trasitions from commercial to show or vice versa. I figure those spots make things more interesting and Tytool needs to be able those situations.

snoots
12-15-2004, 02:41 PM
http://sourceforge.net/projects/tyshow/

allows playback of local ty files using directshow filters. also works with tivoweb plus to allow playback from the web interface

jdiner
12-15-2004, 03:01 PM
I have gotten 2 more disks. One for Dallas/Ft Worth and one from Boise. That makes a total of 4 now. While I would still like more I am going to start in on the evaluation process of these now and see what all I can see.

--jdiner

snoots
12-15-2004, 03:09 PM
I'm keeping my fingers and toes crossed !

jdiner
12-15-2004, 04:21 PM
After sending you a disc of OTA TY files I realized one file has a resolution change in it. One of the local stations broadcasts a "Currently Off-Air" message in SD and the TiVo started recording early (actually the station started late) and caught some of that before a resolution change to HD. There are some other degenerate problems with the beginning of this file - like the SD portion includes no audio.
I would like to see this change over. If you could provide it to me I would appriciate it. I can download 1 such file. I have the bandwidth for that if you can put it up somewhere for me to download from.

I had an FTP server but someone hacked it the other day, and I have not had a chance to repair it yet.

As for resolution changes, TyTool handles that. Missing audio, it handles that as well. What is wrong/missing is the proper splitting of HD streams in the first place coupled with proper, or perhaps it would be more correct to say improper, output on the backend. Right now it is generating things that are correct for the PC/DVD players and incorrect for the needs of HiDef. All of this should be fixed sometime soon.

--jdiner

samhammer
12-15-2004, 04:34 PM
Disk with Denver stations is in the mail today.

George

snoots
12-15-2004, 05:15 PM
I sent in a number of clips that are "tagged" with local SD news promo's just prior to the "network HD" programs so we should get a pretty decent cross reference of sd/hd and even audio changes.

jdiner
12-18-2004, 03:21 PM
Alright. I got 2 more disks in the mail. Making for a total of 4 now. I plan to get started this weekend with the look into what variations are present. Hopefully more data will come in soon. The better the cross-section the better this is all going to be.

--jdiner

jdiner
12-19-2004, 09:04 PM
This has already proved to be interesting...

In looking at the OTA streams I have gotten I have found framing error after framing error.

When you watch OTA channels on an HDTivo are there are errors in the audio or video or does it play cleanly when played on the box directly? What I am seeing in some cases doesn't always looking 100% recoverable. Which would seem to indicate either damaged playback or held-over frames etc...

Any comments from the OTA users amoung us?

--jdiner

redstone
12-19-2004, 09:39 PM
When you watch OTA channels on an HDTivo are there are errors in the audio or video or does it play cleanly when played on the box directly?

Speaking just for myself, whether Live or on Playback, OTA is perfect except you may hit a few momentary spots of audio or video dropout/pixelation due to various bad signal issues.
In my case it is caused by multipath.
eg. In a hour show, I may get 3 or 4 times where I get a few seconds of pixelation

Would not expect you to see that on the short 300 Mbyte samples.

tPeter42
12-19-2004, 09:41 PM
I have the same expreience as redstone. The only blips I've seen I believe are due to signal loss, i.e. large pixelation.

snoots
12-19-2004, 10:00 PM
I am only running an indoor "rabbit" ear style antenna, I live only about 4 miles from the majority of the antennas. I do get occasional drops in audio and pixelation but not too offten unless the weather is bad. Overall it is mostly stable.

jdiner
12-19-2004, 11:44 PM
Hummm. I better take another look at this then.

--jdiner

gwaldock
12-20-2004, 11:29 AM
Thank you for your continued effort on this important issue, Josh.

My Hughes HR10-250 has no problem of any kind with playback of OTA shows, but I have a large antenna with clear line-of-sight to the transmitter (signal @ 95%). It is unfortunate, but your wonderful "tyTool" simply does not recognize any OTA audio whatsoever. However, the HD picture that "tyTool" produces is nearly flawless!! :)

Thanks again and Happy Holidays!
gwaldock

Hughes HR10-250 - networked
Sony SAT T-60 - Cache-Card networked, 120GB

drewh
12-20-2004, 03:05 PM
Josh,

I rarely get through a whole episode of anything in High-Def without some kind of serious (visible or audible) error. Occasionally when the error is in an 'I' frame it can last much longer than a single frame with it sitting for (or getting smeared across) an entire GOP. Well, you know what it looks like.

This is the norm for me - both for HD off-air and also for HD off-satallite. I don't know if my situation is uncommonly bad, but it is bad. At first I thought that it was just that there are so many more bits in an HD recording that there is just more chance for an error, but now I'm not so sure of that. An average hour of DTV or OTA HD is about 4-5GB. It will commonly have one or two (noticable) errors in it. 4-5GB of SD is eight or nine hours which I can commonly record with no (noticable) errors. I can't remember that last time I saw or heard an error on SD. I can only remember a couple HD recordings with no noticable errors.

These are some of the things that I think might be causing the problems:

1) My OTA errors might be caused by honestly bad reception. I'm located in a valley and there are some line-of-sight obstructions between me and the transmitter. The first antenna I bought wouldn't receive ANYTHING (but that was a stupid unresearched omni-directional). I now have a large high gain highly directional antenna aimed at the single transmit tower in my area.

2) Most if not all of the HD content is on a different satallite than the SD content isn't it? Maybe my dish alignment is a little off for one satallite but not the other?

3) In an effort to reduce noise and power my expansion drive is 5400RPM drive. When you do the math it should have more than enough throughput for multiple streams, but maybe there is a bottleneck here (maybe in seek time?) that is causing data drops? (I was surprized when I saw that the base drive in the HR 10-250 is a WD2500 7200RPM drive.)

4) There are some errors that I am starting to feel are introduced intentionally, or are maybe not intentional, but a really problem with the TiVo hardware. An episode of the wire will have three or four frames that are missing. When watching the show it looks like a frame of mostly white noise during playback (on my system which is HDMI (DVI) out to a Sony VPL-HS20 projector). When I do a still on the offending frame it looks black. I recall reading a post a while back where someone claimed that sometimes the decryption key doesn't come back from the smartcard in time for a frame and that the frame is garbage. Is that true? Maybe that is this problem? (Which I can only recall seeing on DTV HBOH.)

In the past I've suggested that I could really use a utillity to automatically merge multiple streams into one higher-quality stream. (I didn't notice until recently that you replied to that post - thanks.) A lot of people including you suggested that it wasn't really possible since some re-encoding would be required at the GOP boundries. I guess I wasn't really clear, but I was really looking for an automated editing system to fix these kinds of errors, which at the time I thought were 'normal'. I could use an MPEG editor to fix them by hand (from multiple copies of a show), but there is no way I could keep up - so I thought "automate it".

Anyway, I don't know if my situation is much worse than normal, or completely normal. Any comments or suggestions would be welcome.

Thanks,

Drew

jdiner
12-20-2004, 09:14 PM
Actually I think it was your files, drewh, that I was looking at. So there may be some correlation to the frequency of the errors and the source material... :)

--jdiner

samhammer
12-20-2004, 10:50 PM
When I play back OTA on my HR10-250, the playback seems flawless. I see no blips at all.
I only tried to to replay a muxed file from OTA on my computer once. Picture was flawless --- no video at all.

EDITED - I mean no audio at all.

George m

jdiner
12-21-2004, 12:16 AM
For the record. The audio problems are the major thing I am trying to solve with all of this. It is not an unknown but the cause of the all of the recent effort. I will get it fixed. I just needed the source files and the time. I now have some of both so hopefully...

--jdiner

jdiner
12-23-2004, 02:11 AM
Wow... Well I have found the first problem with OTA recordings.

For some reason it is mis-detecting things as LayerII audio, and when I look at the data it is obviously Dolby...

--jdiner

jdiner
12-23-2004, 02:12 AM
Ok. Just a quick report. I got a number of disks and now have 69 seperate files to test with. Many of which crash the current TyTool setup so there is a ton of work to do. But I do believe I have enough files to get more than started here.

--jdiner

jdiner
12-23-2004, 02:23 PM
Alright. Well. One of the HD Clips I have gotten has a brand new audio scheme in it. I have it fixed now. Every single audio record is the last one in each chunk and it always split into the next chunk. Just wierd crap going on with that one. But it works now. :)

--jdiner

jdiner
12-23-2004, 07:59 PM
Well I appear to be the only one around on this fine snowy day. But I just had to say "HA! Who's the man?" :)

I solved the HiDef audio and video problems.

I have been testing with a ton of the clips I have gotten and have fixed a few buffer issues but look for a new test release before Christmas.

My Christmas present I guess to the HD community.

--jdiner

tPeter42
12-23-2004, 08:20 PM
Snow, what snow? It's 62 and sunny (I mean smoggy) here in L.A. ;) I'll pay for that tomorrow when I fly through O'Hare.

You are indeed the man. That was quick. Thanks for all the hard work!

Sean.

snoots
12-23-2004, 08:30 PM
Josh,

Great work ! You will indeed be making my Christmas much happier !

I hope you and your family have a great Christmas.

Snoots

gwaldock
12-23-2004, 08:48 PM
Check your paypal, Josh. You're not the only one who can play "Santa Claus" :p

Merry Christmas!

samhammer
12-24-2004, 06:59 PM
There's something in your stocking from PayPal!!!

Thanks for you work on the OTA recordings

George M

rudolpht
12-24-2004, 11:30 PM
I solved the HiDef audio and video problems.
--jdiner

Now that is a Christmas present. Great work. Merry Christmas,
Tim

jdiner
12-25-2004, 03:46 AM
It is almost ready to go. I need to rebuild in non-debug mode and remove the excessive debug prints.

I have not forgotten. But it isn't Christmas morning yet, at least not here, so I still have time. :)

It is coming. And for me it is working perfectly... :)

--jdiner

toyotafan
12-25-2004, 10:10 PM
Oustanding! Thanks and Merry Christmas.

Steve

jdiner
12-26-2004, 03:30 AM
First by way of explanation...

I am sorry I didn't make the post I had intended to make. I ran into a couple of problems that took much more time than I had thought to find and fix. About 12 hours or so counting now.

First I ran into issues with my machine. I have an Asus motherboard machine an AX something... It had a better sound card on it than I have really ever had. However in the case of dolby digital I had issues I have never experienced before. I told it I had 2 speakers. It still always does 6 speaker surround sound. But I didn't know this. So I could never hear the voices.

(BTW if anyone has enough experience with these motherboards to know how I can turn this off and get it to work the way the rest of my machines do I would be forever grateful to know how I make it act like sound cards always have... To be specific I have an ASUS with an NFORCE APU / SoundStorm audio system in it. I can't make it play stop acting as if there are 6 speakers...)

This is also the only machine I have now that will play the HiDef clips. Once I jerry-rigged a few things I managed to get another machine to play these things, sort of. I have to almost minimize the video window to make it work.

Which lead to discovery #2, I found some massive audio skew that could be present in OTA Dolby clips. By massive I am talking about +1 to +3 seconds. It was extremely noticable.

While I have been unable to solve #1 on my own. I have solved #2.

Which led me into testing the audio with even greater care. I have found that there is a completely, until just a short time ago, undiscovered sub-family in the OTA realm. That is doing audio in the wierdest way possible. I am working it over, but it isn't ready yet. The main vein of OTA clips now works extremely well. This odd group I just discovered still has some issues.

Running into all of these bugs turned it into a really merry Christmas for me. Talk about frustrating. But... I am closing in on a most of the issues.

I wanted people to see what I was talking about. So I have been racing towards the mainstream release. But I just haven't made it yet. So I am going to be releasing a quick "look see" version that works with about 90% or more of the clips I have received. I am still working on the other clips. I know what is wrong I just have to find a reasonable way to deal with this sub-category. Once I do OTA should be pretty much in the bag.

I wanted to get this out, as mentioned, in a ready to use form today, but I just couldn't make it. Sorry.

--jdiner

jdiner
12-26-2004, 03:39 AM
In this release is just a new version of TyTool. It is the pre-release mentioned in the previous post. It works on about 90% of the streams I have received. I know what is wrong with the rest I just have some work to do to make it work. I wanted people to see the progress and start testing things a bit.

If you happen to have a stream that is in the category that doesn't work it will crash TyTool almost immediately on the start of processing. Don't panic, just know that it isn't in this version, but will be in the next. If it works, check the sync, check well... whatever you feel like checking and we will go from there.

--jdiner

jdiner
12-26-2004, 04:08 AM
I was just thinking about it and I thought I would provide a quick summary of what the sub-family consists of...

The main form of TyStreams both SD and HD are 1 audio header and 1 audio record. This 1 to 1 match is always present on SD Tivo stream of any flavour.

With the OTA records I am getting from people the main case is 1 audio header and 4 audio records. The pattern looks like this:



> 0 = * 0 0 1 BD 18 8* pts = 3414993701 (10:32:24.374)

> 14 = * B 77 1B 44 1C 40*
> 1550 = * B 77 DB 88 1C 40*
> 3086 = * B 77 E1 35 1C 40*
> 4622 = * B 77 66 2E 1C 40*found = 4
audio record: size 6160

> 2 = * 0 0 1 BD 18 8* pts = 3415005221 (10:32:24.502)

> 16 = * B 77 D0 9D 1C 40*
> 1552 = * B 77 CA 8A 1C 40*
> 3088 = * B 77 E8 FA 1C 40*
> 4624 = * B 77 DA 7B 1C 40*found = 4
. audio record: size 6156


Always 4 records, always the same size, etc... This is the case that works. The key here is that you get complete audio records, they start as they should and things line up. All in all not a bad change once I figured out what was going wrong. (There were several problems here including that thanks to the nature of this it was being detected as LayerII audio and not Dolby from the very start...)


Now from certain other channels I am start to see a pattern that is nowhere near as clean and this is what needs to be fixed. The new pattern is:



> 0 = * 0 0 1 BD 16 FA* pts = 439742775 (01:21:26.030)

> 640 = * B 77 B0 48 1C 40*
> 2176 = * B 77 83 40 1C 40*
> 3712 = * B 77 1D BF 1C 40*
> 5248 = * B 77 AB 1F 1C 40*found = 4
audio record: size 5888

> 0 = * 0 0 1 BD 16 FA* pts = 439754295 (01:21:26.158)

> 912 = * B 77 E6 6E 1C 40*
> 2448 = * B 77 BF 25 1C 40*
> 3984 = * B 77 BC 71 1C 40*
> 5520 = * B 77 93 0 1C 40*found = 4
audio record: size 5888

Not the sliding locations. What causes this? Incomplete audio records. In the first 1 above there 640-14 bytes of a previous audio record in there. Then there are several complete ones and then finally there is a partial at the end, 5888 - 5248 = 640 as the remainder of a record size of 1536. 1536 BTW is the standard Dolby audio size.

Now for a variation on a theme I have just seen:



> 0 = * 0 0 1 BD 16 FA* pts = 3902431567 (12:02:40.350)

> 112 = * B 77 F1 9 14 40*
> 880 = * B 77 CB 3B 14 40*
> 1648 = * B 77 2D D1 14 40*
> 2416 = * B 77 C6 78 14 40*
> 3184 = * B 77 83 9D 14 40*
> 3952 = * B 77 93 10 14 40*
> 4720 = * B 77 44 75 14 40*
> 5488 = * B 77 DA E6 14 40*found = 8
audio record: size 5888

Same incomplete lead in packets, but now the record size is 768. Which is the first time I have seen that changed on anything. Kind of sad to keep running into changing standards.

Like I said I know what the difference is I just have to add a stage to the process of it that throws the first "junk" portion away and then keeps things from 1 record to the next. Not hard just a bit of time because it is a completely new process that hasn't been needed before.

--jdiner

gwaldock
12-26-2004, 02:24 PM
Happy Holidays Josh-

Thanks for sharing your "beta". I have only used it on a few UPN shows, but I can report REAL PROGRESS here! Not only do the HD clips have audio (for the first time) but said audio seems to be in sync and without noticable anomolies!

Outstanding work! :) :) :)

ronnythunder
12-26-2004, 07:30 PM
In this release is just a new version of TyTool. hey josh, you probably haven't received my dvd yet, but i went ahead and tried both this version and the 9r18 "stock" version on the clips i sent you.

both versions worked on all of the abc, cbs, nbc and upn clips that i sent. well, "worked" insofar as i could create key files and edit them.

neither version worked on either of the fox clips or the pbs clip.

however, interestingly, the "older" version (9r18) successfully processed the wb clips, but the hd-pre version could *not* process these; it crashed immediately upon trying to create a key file for "crossroads" and got into a cpu loop trying to do the other ones. so, it seems that there is a slight regression in the "hd-pre" version at the moment.

just thought i'd let you know; i did read where you were making other changes, so no worries.

ronny

snoots
12-26-2004, 08:21 PM
I also was able to get working OTA audio in the ty file using tyshow, However trying to make a key file to process to mpeg dies immediatly in the new tools.

Step in the right direction :)

samhammer
12-27-2004, 01:05 AM
Joshua,

I thought I would post some of my results. These are from Denver stations.

First, I cannot edit any key files with R19 - either OTA or satelite. It will create them OK, but when I try to edit them tt starts and immediately quits saying it is done.

CBS:
R18 Good Video - No Audio
R19 Good Video - Good Audio

NBc:
R18 Good Video - No Audio
R19 Good Video - Good Audio

WB:
R18 Good Video - No Audio
R19 Immediately crashes and closes down.


You're certainly making progress.

George M

jdiner
12-27-2004, 01:51 AM
I will be getting the next version out soon. Spent the day with Family, being the Christmas Holiday and all so nothing more was done today. I will be making the release as soon as I can get it ready.

So no ideas on the stupid AC'97 drivers for this ASUS motherboard then. Darn. I want to just switch it back to the older style of operation.

--jdiner

jdiner
12-27-2004, 02:00 AM
so, it seems that there is a slight regression in the "hd-pre" version at the moment.
No. Just some partial code. Like I said. When it hits things that aren't as expected it is just plain gone... More to do there.

--jdiner

jdiner
12-28-2004, 06:00 AM
Alright.

All of the major problems that I have identified have been resolved. The various forms of the OTA audio and video structures are now being processed.

Not saying I am done yet, just that in the main things are working perfectly. I still have a few clips that cause odd problems. One that appears to detect as Dolby and isn't and one that starts off fine and then slows as it eats tons of memory. But these are pretty standard issues...

Having said this, I can't believe the amount of CPU power HD takes. My 2600+ is always skipping a bit and makes it terrible to try and verify A/V sync. My 2800+, the fastest machine I have, is still having audio issues and since I can't hear the main center channel, i.e. the voices, I can't verify things on that machine.

What a pain.

But as far as I can tell Sync is right on and things are running along fine. I am about to start processing a larger cross section of the files that I have. I will post results as I go and if this is a major step forward I will post a second pre-release binary containing the next round of major OTA/HiDef changes.

--jdiner

jdiner
12-28-2004, 06:04 AM
Ugh... Just found another odd sub-case. This is getting old...

EDIT: What I found was a new one where each audio record has less than 1 full audio packet. This is a first. Always had at least one until now. Not sure where this one is from but...

--jdiner

mikemav
12-28-2004, 09:50 AM
Having said this, I can't believe the amount of CPU power HD takes. My 2600+ is always skipping a bit and makes it terrible to try and verify A/V sync. My 2800+, the fastest machine I have, is still having audio issues and since I can't hear the main center channel, i.e. the voices, I can't verify things on that machine.

What a pain.



--jdiner

First off, thanks for all the hard work Josh! We really appreciate it. I just now got my network problems fixed & the TiVO back online, but at this point it sounds like you have a good cross-section of data to test. Let me know if you still want some DC stations' OTA files now that I am back online.

Regarding the horsepower required for HD, that is one of the reasons you will find most people serously trying to play these files back are using hardware-based HD decoders. The most popular had been the MyHD PCI card, but a few of us are trying out the I-O Data AveLink networked DVD player w/ the latest Sigma chipset that supports up to 1080i files. My tests so far show this $250 DVD player/network player can do all the MyHD can do except OTA tuning (which I use my TiVo for now anyway.) It can also play XVid, DiVX HD, and WM9-HD and is a stand-alone box instead of a computer. Perhaps one day you might want to pick up one of these for testing (or a MyHD card), especially if enough PayPals come in. :)

snoots
12-28-2004, 11:27 AM
IO-DATA tech support is helping me out on playing ty files on the link player. So far I can get audio only on some of the ty files, some come up and say encrypted mpeg file ?? I think we are close to getting it to play correctly using the tyshow directshow filters :) This would be wonderfull, pull tys of tivo no more processing required if you just want to watch ppvs or non editted shows.

mikemav
12-28-2004, 12:34 PM
Josh-
Now that TyTool is almost complete in handling all HD content (thanks again!), what are your thoughts on processing these files in a way to convert them to lower res playable on a DVD player? I know someone mentioned mencoder before, but I have not messed with it much. I also have tried ffmpegx on my Mac, which seems to work ok. Would it be difficult to add this feature to a future TyTool? Certainly most things would be worth keeping as true HD, but I could see a convert to DVD option being useful for some content, especially if we could make anamorphic widescreen DVDs to preserve the most PQ possible on DVD.

malfunct
12-28-2004, 05:21 PM
Josh-
Now that TyTool is almost complete in handling all HD content (thanks again!), what are your thoughts on processing these files in a way to convert them to lower res playable on a DVD player? I know someone mentioned mencoder before, but I have not messed with it much. I also have tried ffmpegx on my Mac, which seems to work ok. Would it be difficult to add this feature to a future TyTool? Certainly most things would be worth keeping as true HD, but I could see a convert to DVD option being useful for some content, especially if we could make anamorphic widescreen DVDs to preserve the most PQ possible on DVD.

TyTool is not for doing transcoding. If you want transcoding you should use a different tool.

jdiner
12-28-2004, 08:00 PM
Oh man. Every idea I have slams into a dead end at the moment.

I am trying to figure out a way to catch and deal with these stupid packets that have less than 1 audio record. This is beyond frustrating. I think I am going to have to take some of this back to the drawing board.

Much of what I have done should be usable but this is an unexpected twist on the theme.

--jdiner

rung
12-28-2004, 09:15 PM
Oh man. Every idea I have slams into a dead end at the moment.

I am trying to figure out a way to catch and deal with these stupid packets that have less than 1 audio record. This is beyond frustrating. I think I am going to have to take some of this back to the drawing board.

Much of what I have done should be usable but this is an unexpected twist on the theme.

--jdiner

Random access file? Write the whole audio record out with zeros in the location you don't have yet. When (if?) you get the rest of the record, you back up and or-in the new data. If you get a new PES header, just continue on at the end of the file and leave the zeros in the file.

Just a thought.

Regards,
Rung

jdiner
12-28-2004, 11:03 PM
Random access file? Write the whole audio record out with zeros in the location you don't have yet. When (if?) you get the rest of the record, you back up and or-in the new data. If you get a new PES header, just continue on at the end of the file and leave the zeros in the file.

Not a bad thought. Not far from what I came up with.

The problem is simple, the solution not so much. First some background:

In a standard SD TyStream you get 1 audio header followed by 1 audio record it continues in this pattern from start to finish. The audio record always begins right after the header (either 00 00 01 C0 or 00 00 01 BD).

The problems with it are sometimes DirecTV is 1 or 2 bytes short or long and so on, but always close. An occasional record way way out of position timewise etc... An easy enough format to deal with for the most part.


The problems with OTA streams are much much more numerous. I figure the largest reason is that things are compressed by the local station in whatever way they want and things just get worse form there.

In just the first 10 of the test clips I ahve from people I have seen so many whacked out things it isn't funny.

The real issue right now is that sometimes I get an incomplete packet with skew. Meaning the Presentation Unit doesn't actually start for up to 1500 bytes into the record. Couple that with the odd occurance of the supposedly protected header pattern (0b 77) and things get wild. I am already triple buffering the data to get around the other problems it is clear that a completely different process is needed.

I haven't given up yet, so keep watching the thread.

--jdiner

Jeff D
12-29-2004, 01:20 AM
The problems with it are sometimes DirecTV is 1 or 2 bytes short or long and so on, but always close. An occasional record way way out of position timewise etc... An easy enough format to deal with for the most part.
Is the work around to ignore the size fields and get what you can until you get the next 00 00 01?



The problems with OTA streams are much much more numerous. I figure the largest reason is that things are compressed by the local station in whatever way they want and things just get worse form there.

Just FYI, Fox doesn't allow the locals to do this for any of the network stuff. Or at least that's what I'm told from folks in the know.


Couple that with the odd occurance of the supposedly protected header pattern (0b 77) and things get wild. I am already triple buffering the data to get around the other problems it is clear that a completely different process is needed.
B0 77 or 0B 77? You are talking mpeg header, right? 0B was valid I thought...

Can you PM me a location for some samples? I'd love to see what you are seeing.

jdiner
12-29-2004, 02:21 AM
Is the work around to ignore the size fields and get what you can until you get the next 00 00 01?
This is pretty much what the code used to do back before OTA. Now the problem is that you don't have a 00 00 01 BD between each record. You have 1 and then "a partial" or more records before the next one.

So I was locking onto the "0B 77" that starts the next actual presentation unit. But these are now appearing within the audio data itself. According the AC3 spec this is not supposed to happen so go figure. So I extended it to lock onto the size as well (0b 77 <CRC1 byte1> <CRC1 byte2> <Size byte1> <Size byte2>) but this was non-deterministic because I have seen over 5 different sizes of audio data now. 48k @ 80, 160, 338, 448 and so on. Plus you can't just "remember" these as audio streams can change mid-stream. You have to adapt on the fly... So I extended even further to look at the next several header fields. The AC3 ID field which should always be 8. Then the "which audio stream is this" which should always be 0 for "main" and onto the "which channels do we have". At first this looked consistent at 8 0 2. But then I found wildly divergent results from other streams. 6 0 7, 8 0 5, 5 1 1, and so on.

So then I got creative and looked at what the results should be as they line up. A bitrate that matches etc... And in the very first test file I got a false positive entry in the 3rd chunk.

So.... I moved on. I ran out of clever ideas, or at least those that seemed that way at first, and now I am trying a much more brute force method. As a result I have split it off from the audio code of the rest of the system. I.E. OTA streams will all be processed this way and everything else will use the same old code with a few updates. (As the old code will be a touch faster and is well debugged at this point.)


Just FYI, Fox doesn't allow the locals to do this for any of the network stuff. Or at least that's what I'm told from folks in the know.
I am willing to believe that some don't. and some areas don't. But what I have seen stream wise doesn't bear that out. A CSI NY ep from Denver had a totally different set of stream data rates than one from Boise and so on. Now I know that is not FOX my point is that it isn't universal. Which means I can't count on it unless I want to go major networks only. They do have the best internal file formatting of any seen yet. (BTW, PBS and other channels have the absolute worst.)


B0 77 or 0B 77? You are talking mpeg header, right? 0B was valid I thought...
0B 77 is the 16 bit sequence that is the Sync Word for Dolby audio of all forms. 2 channel, 5.1 and so on. Trust me I have the docs right here and I have been doing Dolby audio processing for a very long time now. :)


Can you PM me a location for some samples? I'd love to see what you are seeing.
My networking link doesn't upload anywhere near fast enough for me to share out 200meg files. I get less than 12K/sec upstream but 160k/s down. So I don't complain.

--jdiner

Jeff D
12-29-2004, 03:46 AM
Sorry, josh. My point on the networks should have been more to the point of...
Fox is the only network that seems to have a "standardized" system in place. All the network supplied data is just as FOX has created it. They are the only network doing this AFAIK.

All the other networks and channels are still in the learning process. I've delt with system engineers from FOX, ABC, NBC and CBS. Most of the guys I deal with are heads of engineering for the stations they work for. So... these are the guys who know best what's going on. A lot of times they admit that there's a lot of stuff with DTV they don't know or understand yet. Just getting the encoder configured correctly alone takes a lot of work, these boxes are pretty complex because of all they support. This means it's no surprise that you may be seeing things that may be just plain wrong. This is still the ramp-up time.

redstone
12-29-2004, 02:42 PM
The problems with OTA streams are much much more numerous. I figure the largest reason is that things are compressed by the local station in whatever way they want and things just get worse form there.

In just the first 10 of the test clips I ahve from people I have seen so many whacked out things it isn't funny.

--jdiner

These are interesting observations.

Makes one wonder how Directv or TIVO went about testing this device before it was released because you don't read about it having trouble with OTA stations around the country.

I know the MYHD card had problems with a few non standard ATSC transmissions and it did take samples from a handfull of folks to make it's SW robust.

Also, since the HDTIVO plays back OTA just fine, you would expect it to be doing a lot of error correction during playback but it does not have a very high powered processor. Does the HDTIVO do this decoding/processing in an ASIC?

mikemav
12-29-2004, 03:17 PM
Josh-
With your HD Tivo Pre-release, which format should we be using, and which are you using for testing:Multiplex/Split-Multiplex,VOB-Mux, or new format 1 (VOB-Mux) or 2 (Multiplex)? Since the UI looks the same as the other versions, I was not sure which process you were tweaking for the HD files.

atscguru
12-29-2004, 04:21 PM
Also, since the HDTIVO plays back OTA just fine, you would expect it to be doing a lot of error correction during playback but it does not have a very high powered processor. Does the HDTIVO do this decoding/processing in an ASIC?

redstone (aka hdtivojoe on Yahoo, aka Joe Q on avsforum),

It is nothing short of puzzling that you ask so many questions on this forum, after making so many demeaning statements (http://www.avsforum.com/avs-vb/showthread.php?postid=4185279#post4185279) about forum members (http://groups.yahoo.com/group/hdtivo/message/3) and even about jdiner (http://www.avsforum.com/avs-vb/showthread.php?postid=4614191#post4614191), a guy who is quite frankly going WAY above and beyond the call of duty to help your ungrateful a$$.

Nevertheless, I will indulge your curiosity, as you obviously have no idea how these devices work. Perhaps you can take some pointers from me on how to act like a decent human being instead of a spoiled child.

Set top boxes (STB) such as your HD Tivo all use hardware mpeg-2 decoders because it is far cheaper to use specialized hardware than general purpose microprocessors for this task. This model uses the Broadcom bcm7037, a COTS part, for decoding. Since the product brief is not readily available, you can look at the bcm7038 (http://www.broadcom.com/collateral/pb/7038-PB00-R.pdf) information. On the HD Tivo, the cpu is separate, not on-chip as it is on the bcm7038. Also, most of the non mpeg related components are separate, like the ethernet and ide controllers.

Obviously, a commodity mpeg2 decoder will not directly handle proprietary formats such as the stored Tivo program streams. For this reason it is important to take into account the possibility of differences between the mpeg ps produced by hacker mux tools, and the pes data that is actually fed to the decoder hardware.

snoots
12-29-2004, 04:21 PM
Josh,

I don't know if it matters or helps but I called the local CBS station and found out that they are using the Harris Flexicoder for MPEG encoding. The manual is available here. Might be a waste of your time but thought it might be helpful.
http://www.broadcast.harris.com/product_portfolio/product_details.asp?sku=WWWFLEXICODER

Jeff D
12-29-2004, 10:28 PM
Josh,

I don't know if it matters or helps but I called the local CBS station and found out that they are using the Harris Flexicoder for MPEG encoding. The manual is available here. Might be a waste of your time but thought it might be helpful.
http://www.broadcast.harris.com/product_portfolio/product_details.asp?sku=WWWFLEXICODER


FWIW: The Harris Flexicoder is one of the most popular encoders and used by a majority of the stations. The problem is that thet many know how to configure it. I'm actually impressed you were able to find someone who would tell you this! That's normally a pretty tough answer to get. =?



Also, most of the non mpeg related components are separate, like the ethernet and ide controllers.[/url]

Just a little flame protection... The 7038 can handle most other stuff. It has controllers for USB, SATA, Ethernet, MODEM and other devices. Unless the designers decided to not use these internal controllers I'd guess they are used. The programmers would still have control of the stream and could do the encryption and ty processing.

jdiner
12-30-2004, 02:28 AM
Josh-
With your HD Tivo Pre-release, which format should we be using, and which are you using for testing:Multiplex/Split-Multiplex,VOB-Mux, or new format 1 (VOB-Mux) or 2 (Multiplex)? Since the UI looks the same as the other versions, I was not sure which process you were tweaking for the HD files.
The old format mux and VOb are what I am using. As posted before the new mux forces things into DVD compatiblity modes which is no where near the rates needed for HiDef streams.

--jdiner

jdiner
12-30-2004, 02:32 AM
The big problem I am having versus a standard Tivo device is that I in effect reverse engineering a file format that they have full documentation for.

Also it is 10 times easier to pull things apart than it is to put them back together. Finding and keeping enough information to do so properly with a format that seems so flexible is well interesting.

Also they get the data is some TS form and save it in a TyStream form. Certain things have "always" been one way with a Tivo and now that way has dramatically changed. That is what I am facing.

So call it my fault, I built for what was there not every possible contingency.

--jdiner

jdiner
12-30-2004, 02:40 AM
Well I didn't bother going through the docs on the encoder or decoders. most of the time they are about the hardware and have nothing to do with the algorithms etc...

But I came up with a pretty slick design. The interface will basically be 1- pass in a timestamp in one call and then pass in the elementary stream to another. Then thing will be read from that insert buffer into another internal buffer. Basically I am trying to think abot it the way I think the chip keeps everything in line. That is infact I think a bit of a better idea than what I was doing before. Honestly I think the overall idea has merit.

I will be recoding taking a look at this and should know soon whether or not it will work.

--jdiner

bcc
12-30-2004, 03:08 AM
Couple that with the odd occurance of the supposedly protected header pattern (0b 77) and things get wild. I am already triple buffering the data to get around the other problems it is clear that a completely different process is needed.
--jdinerThe 0b77 start code is allowed to appear in the audio payload per section g.0.4 of iso13818-1. So you have to protect against this "emulation" of the start code by knowing what the payload length should be. I fixed this last month with hdemux's demuxing of the audio streams. You might want to take a look at what I did.

jdiner
12-30-2004, 02:56 PM
The 0b77 start code is allowed to appear in the audio payload per section g.0.4 of iso13818-1. So you have to protect against this "emulation" of the start code by knowing what the payload length should be. I fixed this last month with hdemux's demuxing of the audio streams. You might want to take a look at what I did.
g.0.4 is talking about the 00 00 01 start code emulation, especially as it refers to video start codes versus audio and data start codes.

According to timestamps etc... in the my doc files the AC3 audio documents supercede the 13818 files I have by quite a bit. More than a year infact.

But then I am certain my docs are not the latest versions and not entirely correct. They specify that the bsid header field is always always always supposed to be 8. I have found references online that indicate that less than 8 is used to indicate a sub-set of the AC3 spec. However I can't find anything that talks about what each "level" defines.

I came to the same conclusion. Positionally is the only way to go. never had to do that before thanks to the way that DTV streams are organized but that just doesn't hold true anymore.

--jdiner

mikemav
01-01-2005, 01:45 AM
Any idea what would cause a problem with only a certain non-OTA show? I have tried several times to extract & process Dog Day Afternoon from HD Net (high def, w/ DD audio.) Let me say off the bat I know the problem is not the Priority Set issue, even though the symptoms indicate that is the cause. I cannot get TyTool HD-Pre to output a file on this one larger than about 1/2 GB. Every time it processes without error, but the 11GB .ty only results in a 519MB .vob. I know it is not Priority Set since I have updatred the binaries on my mfs_ftp, and even for kicks ran the Unpriority.exe on the .ty to be sure. I even deleted the show off the tivo, recorded another airing of it, and tried again. Same result. Ohter files extracted & processed in the same way the last few days work fine.

When it is processing the file, it gets to around that 500MB mark and the speed noticably increases. No error reports, but it starts really zipping at 500MB & never writes a file larger than that 519MB range. Also, when finished, it reports:DiffTime = 527.047025 (527047) == 8.784118 Minutes
total = 8516409900 (8121 MB)

Any ideas? The .ty is 7.92GB, so I cannot even burn it on a DVD & send it out for jdiner to look at. Unless I try to split the single file to two DVDs w/ some kind of backup program.

jdiner
01-01-2005, 12:57 PM
Does TyTool eat memory when this happens? Does it get really slow?

I can't think what would cause what you are seeing here. Might be a second PTS reset problem. I dunno.

--jdiner

snoots
01-01-2005, 01:06 PM
I looked to see if it is going to be on again but did not find it, if you see an upcoming time, I can record it on mine and see if it does the same thing. I don't know if that will help or not though.

mikemav
01-01-2005, 02:43 PM
Does TyTool eat memory when this happens? Does it get really slow?

I can't think what would cause what you are seeing here. Might be a second PTS reset problem. I dunno.

--jdiner

No, it just gets really fast & starts flying through them. This is the same behavior I saw when I had some files that had the bad priority set strings and it would not get to the second FSID. However, in this case I verified I am using the corrected binaries & I ran it though unpriority.exe just to be safe.

Looks like the show's run on HD Net is over too. Also strange is, having the same thought that it may have been a bad recording or something, I recorded it again & saw the same result.

I did not play it back on the TiVo yet. I will check to see if there are any problems w/ the recording on that side.

lsmod
01-01-2005, 03:46 PM
Any ideas? The .ty is 7.92GB, so I cannot even burn it on a DVD & send it out for jdiner to look at. Unless I try to split the single file to two DVDs w/ some kind of backup program.

WinRAR will split files into DVD-R sized chunks and reassemble them easily.

I actually have Dog Day Afternoon sitting on my DTivo now, I'll extract it and see what happens.

-Z

EDIT: I just extracted, edited, and multiplexed it with no problem. This was the 12/27 2:45pm showing.

jdiner
01-01-2005, 04:34 PM
No, it just gets really fast & starts flying through them. This is the same behavior I saw when I had some files that had the bad priority set strings and it would not get to the second FSID. However, in this case I verified I am using the corrected binaries & I ran it though unpriority.exe just to be safe.
Yeah that is what happens when it can't even lock onto the headers for the tivo records. Ummm. Put it into debug level 2 and then sit back for a long long time and check the file when it is done. Level 2 dumps a freaking ton of information into the debug .txt file. It will tell you if it can't figure out what is going on. There is on other case I can think of where something might be going wrong. A bad download and the old lost byte bug.

Make sure you still have the stream on the tivo just in case that is the problem. But no one has complained of that issue in a long long time so I thought I had it fixed...

--jdiner

lsmod
01-01-2005, 05:03 PM
Does TyTool eat memory when this happens? Does it get really slow?

This reminds me. I've occasionally noticed that things will chew memory after an FAE, until the next cut. I say "things", because the memory doesn't show up in the task manager as belonging to TyTool, yet it is only freed if you quit TyTool (or another cut comes before the system swaps to death.)

I've got a situation now that does it. It's just two cuts, GAE at the beginning of the file to FAE, then an FAE to GAE at the end. The encoding for the first FAE happens, (I see the DOS box) and TyTool keeps processing. But the output .mpg file doesn't grow. In fact, it sits at 2k, which makes me think the re-encoded GOP at the cut didn't even get written. System memory usage grows at about the same rate as the looping total from TyTool, but nothing makes it to disk.

The situation that creates this at the moment is a hand-edited .cut file, I'm going to regenerate the .key and cut again with GopEditor to make sure I haven't hosed anything in the cut file.

-Z

PS: On that hand-edited cut file... Is it legal to comment out cuts? How picky is TyTool about whitespace in the cut file?

jdiner
01-01-2005, 05:14 PM
This reminds me. I've occasionally noticed that things will chew memory after an FAE, until the next cut. I say "things", because the memory doesn't show up in the task manager as belonging to TyTool, yet it is only freed if you quit TyTool (or another cut comes before the system swaps to death.)
The video compressor that is in use to do the cutting takes an extremely large amount of memory. It will spike up almost immediately and won't release it until it is done with compressing the new replacement GOP.


The situation that creates this at the moment is a hand-edited .cut file, I'm going to regenerate the .key and cut again with GopEditor to make sure I haven't hosed anything in the cut file.
Never never never do that. It is position dependant down to the last millisecond in that code for FAE processing. If you altered a cut timestamp you are causing things to get lost.

I hate to say it but you can't abuse that tool in that way and expect things to work. TyTool processes GopEditor cut files, not manually created ones. Can you manually make a FAE cut that will work? Yes. Is it easy? No. Should you? Never.


On that hand-edited cut file... Is it legal to comment out cuts? How picky is TyTool about whitespace in the cut file?
Yes. You can comment cut's. I do so regularly when testing things etc... It is fairly white-space tolerant and extremely data specific.

--jdiner

lsmod
01-01-2005, 06:02 PM
The video compressor that is in use to do the cutting takes an extremely large amount of memory. It will spike up almost immediately and won't release it until it is done with compressing the new replacement GOP.

That's not what's going on here. I do see that memory spike, but that's nothing compared to what happens next. I should have been clearer.

I have a simple cut file:



# This cut file created by the TyTool GOP Editor...

00:00:00.000 00:05:03.855 -1 21
00:31:03.713 13:15:21.858 18 -1
# Chapters!
# Done


It chugs along for about 90MB until it gets to the end of the first cut, and does an encoding run. DOS box appears in the task bar, DOS box disappears. Nothing is written to disk beyond a 2k file with the right name. Processing continues, nothing is written to disk, memory usage climbs, everything slows down as the system swaps to death. Memory usage is climbing at the same rate as the looping total in TyTool. I usually have to kill it by about 1.3GB used, as there's only 384M physical memory in that box. The task manager tells me that TyTool is only using about 80-150MB of memory, but the excess memory usage gets freed if I quit TyTool.



Never never never do that. It is position dependant down to the last millisecond in that code for FAE processing. If you altered a cut timestamp you are causing things to get lost.

I hate to say it but you can't abuse that tool in that way and expect things to work. TyTool processes GopEditor cut files, not manually created ones. Can you manually make a FAE cut that will work? Yes. Is it easy? No. Should you? Never.


Two comments at this point: One, this happens with an unaltered file from GopEditor. I've just reproduced that.

Two, yes, I understand the issue with the contents. I was attempting a workaround for the fact that Split Multiplex doesn't do FAE. So I created a .cut file in GopEditor, commented out all but the first cut, and then created an new cut that started with the beginning of the second cut, and ended at the "EOF" timestamp. The idea was that I could make the right cut files to manually create the segments faster in vi than in GopEditor, once I got the right values.

Like this:



00:00:00.000 00:05:03.855 -1 21
00:31:03.713 13:15:21.858 18 -1
#00:31:03.713 00:32:03.923 18 9
#00:58:34.128 13:15:21.858 -1 -1


What debug information can I collect to figure out what's going on? I have a .txt at Verbose Level 2 if that helps. I'd paste relevant bits into this message, but I really have no idea what to grep for. :D

-Z

Edit: gooder grammer :eek:

Jeff D
01-02-2005, 04:49 AM
Two, yes, I understand the issue with the contents. I was attempting a workaround for the fact that Split Multiplex doesn't do FAE. So I created a .cut file in GopEditor, commented out all but the first cut, and then created an new cut that started with the beginning of the second cut, and ended at the "EOF" timestamp. The idea was that I could make the right cut files to manually create the segments faster in vi than in GopEditor, once I got the right values.

Like this:



00:00:00.000 00:05:03.855 -1 21
00:31:03.713 13:15:21.858 18 -1
#00:31:03.713 00:32:03.923 18 9
#00:58:34.128 13:15:21.858 -1 -1


Well, it's a good thing I read this, I was about to do the same thing to workaround the "no FAE split-mux" problem. I would have suspected what you did to work, just taking previous edit points... live and learn.

lsmod
01-02-2005, 02:14 PM
Well, it's a good thing I read this, I was about to do the same thing to workaround the "no FAE split-mux" problem. I would have suspected what you did to work, just taking previous edit points... live and learn.

I have no reason to believe that it won't. The problem with the memory use happens with an unmolested .cut file from GOPEditor, untouched by human hands. (The first file in my last post is an unmodified file.)

I'll try editing .cuts on another file and see how it goes, I just haven't had a chance to try that.

What's odd is that I've converted this file before, and just went back to make a split version to fit on DVD-Rs.

-Z

EDIT:
It works fine. jdiner is right, that you must precisely match the timestamps, down to the millisecond, or TyTool won't find the GOP.

But, if you're careful, you can rearrange the cuts and TyTool will process them correctly.

As to my memory leak issue, I'm not entirely sure it's software. It seems to be in one of two classes of problems that get fixed when I reinstall (OK, restore from the .zip) TyTool and its DLLs. The other problem is that the MPEG2DLL can't be found. I'll start watching this with md5sums on all the binaries to see if something is corrupting the software. This is old, crufty hardware, so anything is possible.

drewh
01-03-2005, 02:45 AM
First of all, thanks much to you Josh. I've been playing with r18 a little and my OTA recordings seem to be fine. I've not tried all of them, but the ones I have tried are doing well. Thanks! Great, great work.

And on a minor subject change...

Has anyone had any luck moving tytool output, most specifically HD, to other tool chains? I've just started trying to use Dr. Divx to transcode to smaller file sizes. I don't want to sacrafice picture quality, but I think I should be able to cut these (HD) files in 1/2 or 1/4 (So the Dr. Divx people claim) without much noticable detriment - except that Dr. Divx is gagging on these files. Once again, haven't (yet) put too much time into it, but I thought I'd solicite pointers before I do.

Anyone? Thanks in advance, and to Josh, much thanks after the fact.

Drew

jdiner
01-03-2005, 03:03 AM
Been working on the changes for some of the new OTA bugs. Wow. Thats all I can see.

I keep running into bad streams. Which is good in one way but hard on what I was trying to do.

Found that audio holes are not uncommon. They aren't large but they are present.

This stream:

KTVT-CBS.ty

from someone here has multiples of these holes. Working things over now to better handles holes. Right now if there are no holes all of the known OTA audio formats work. Kind of forgot about the holes when I was working on this algorithm and it came back to bite me a bit.

Will keep you all posted as I work this over...

--jdiner

TheSaint
01-03-2005, 07:25 PM
This stream:

KTVT-CBS.ty

from someone here has multiples of these holes. Working things over now to better handles holes. Right now if there are no holes all of the known OTA audio formats work. Kind of forgot about the holes when I was working on this algorithm and it came back to bite me a bit.

Will keep you all posted as I work this over...

--jdiner

I went back and reviewed this clip on the Tivo and the only noticeable audio holes are at the beginning. There doesn't seem to be any that are long enough to detect when played. There is another strange occurence that I want to point out in case it is significant. At frame 129 is a scene change and the Tivo stops playing on frame 128. It never shows the rest of the frames on the Tivo, but the file contains the additional frames. Is this where you are finding the audio holes per chance?

lsmod
01-04-2005, 03:26 PM
Well, I've been working on this trying to isolate the problem.

The problem occurs when doing a multiplex or vob-mux, with cuts, of an HD file. I think I've seen it in the distant past with SD files, but none of the .tys I have archived seem to do it.

If the first cut starts at 0:00:00.000, and ends with an FAE, then sometimes after the encoding of the FAE completes, memory use starts climbing, and (almost) nothing gets written to the .mpg or .vob file. The memory consumed matches the progress of the "looping total" and is only freed when you exit TyTool. (Not when processing completes, and you can process other files without freeing the memory)

Changing the first cut to a GAE (change the field number to -1) solves the problem.

This doesn't happen with every stream, but I've seen it on both DSHD and UHD streams so far, and I'm working my way through some archived stuff so I expect to find more.

Josh, what can I do to help debug this one?

-Z

jdiner
01-04-2005, 07:56 PM
Please please please keep in mind that the beta code is just that. Wildly beta. It was to test out the main idea and I suppose to show some progress to one and all...

I don't doubt there were memory leaks in it. There were things in there that were just plain wrong. I have come a long way since then and there is a bit more to go. I will test for memory issues when I get ready for the big release. I hate memory leaks...

--jdiner

lsmod
01-04-2005, 08:36 PM
Please please please keep in mind that the beta code is just that. Wildly beta.

No doubt, and I understand completely. BTW, these are not OTA streams, and both 9r18 and HD-pre show the same behavior.

I see it cutting HD almost every time, so I'm happy to help debug when you get there.

-Z

jdiner
01-05-2005, 03:55 AM
Hummm. Sounds like there is something damaged in that clip. Are you saying that it happens in tons of clips for you?

I don't get that. I processed 5 today, I average about 5 a day from different sources. I have NEVER seen what you describe.

Basic breakdown:

2 buffy episodes from FX.
1 Star Trek from Sci FI (the original series)
1 movie (trying to fill out my collection etc...)
1 misc (Smallville, CSI, CSI: Miami, etc... Just whatever is on that day) These come from many sources WB, FOX, UPN, CBS, NBC, ABC, etc...

So I pulling from all over the place. I never see a freeze and in 9r18 I never see the memory climbing either.

Wonder what on earth is going on. Are you still editing key files manually?

--jdiner

lsmod
01-05-2005, 09:17 PM
Hummm. Sounds like there is something damaged in that clip. Are you saying that it happens in tons of clips for you?


Yes, happens in almost every Hi-Def clip I have. (I have vague memories of seeing it with SD long ago, but none of the SD stuff I've done lately has the problem.) These are all HD, mostly from DSHD and UHD, but a couple of movies as well. Nothing is OTA, it's all from D*.


So I pulling from all over the place. I never see a freeze and in 9r18 I never see the memory climbing either.

It's not a freeze, it's that nothing ends up in the file. The counters all chug along normally, though things slow down as the system swaps to death.



Wonder what on earth is going on. Are you still editing key files manually?


This happens with cut files straight out of GopEditor, no editing.

OH! It's possible that I'm an ***** here: Is the mpeg2enc that's in the 9r18 distribution the right one for cutting HD, or do I still need to swap in the one you distributed earlier?

EDIT: Yes, that was it. Not sure why I thought the new MPEG2ENC.EXE was part of the distribution now. :o

jdiner
01-06-2005, 02:02 PM
Alright. The latest variant of the OTA code is now processing things much much more carefully. So far everything that is working in the main. A few small issues are cropping up. One stream, a football game, appears to have audio holes extremely regularly. Always exactly the same size etc...

--jdiner

jdiner
01-06-2005, 08:08 PM
Well the new HiDef audio code is performing well...

This clip: KDFW-FOX.ty

I just found some odd errors. In a very regular fashion the PTS for the Dolby audio is off by 1 in either direction.

Here is the list of them that I saw:



@@@ off by 1
@@@ predicted bPTS = 08:38:45.474
@@@ next real PTS = 08:38:45.474

@@@ off by -1
@@@ predicted bPTS = 08:38:47.394
@@@ next real PTS = 08:38:47.394

@@@ off by 1
@@@ predicted bPTS = 08:39:01.762
@@@ next real PTS = 08:39:01.762

@@@ off by -1
@@@ predicted bPTS = 08:39:03.650
@@@ next real PTS = 08:39:03.650

@@@ off by 1
@@@ predicted bPTS = 08:39:18.018
@@@ next real PTS = 08:39:18.018

@@@ off by -1
@@@ predicted bPTS = 08:39:20.002
@@@ next real PTS = 08:39:20.002

@@@ off by 1
@@@ predicted bPTS = 08:39:34.274
@@@ next real PTS = 08:39:34.274

@@@ off by -1
@@@ predicted bPTS = 08:39:36.258
@@@ next real PTS = 08:39:36.258

@@@ off by 1
@@@ predicted bPTS = 08:39:50.562
@@@ next real PTS = 08:39:50.562

@@@ off by -1
@@@ predicted bPTS = 08:39:52.610
@@@ next real PTS = 08:39:52.610

@@@ off by 1
@@@ predicted bPTS = 08:40:06.818
@@@ next real PTS = 08:40:06.818

@@@ off by -1
@@@ predicted bPTS = 08:40:08.866
@@@ next real PTS = 08:40:08.866

@@@ off by 1
@@@ predicted bPTS = 08:40:23.106
@@@ next real PTS = 08:40:23.106

@@@ off by -1
@@@ predicted bPTS = 08:40:25.122
@@@ next real PTS = 08:40:25.122

@@@ off by 1
@@@ predicted bPTS = 08:40:39.394
@@@ next real PTS = 08:40:39.394

@@@ off by -1
@@@ predicted bPTS = 08:40:41.474
@@@ next real PTS = 08:40:41.474


What is interesting here is that this is the first stream I have seen this in.

--jdiner

jdiner
01-07-2005, 12:15 AM
Can anyone claim the stream:

KERA-PBS.ty

For whoever that one belongs and whatever station put it out, together you win the worst stream of the year award.

I have been going through it trying to figure out why it behaves so poorly.

I have found audio records 12 byte short. I have found them various sizes up to 8 bytes to long. And here is one that is 160 bytes too long. When playing just the audio when split out through PowerDVD I get constant noice and garbage.

I have made it so this kind of stream doesn't freeze or crash TyTool but do to the content it is going to be pretty unusable.

--jdiner

jdiner
01-07-2005, 03:22 AM
Alright. I think that has it.

I have just run all 70+ test clips that I have through the system. Sync looks good. Most thing process correctly. Some streams have holes and worse which cause glitches but for the most part it all looks good.

There are 3 streams that play back jerky and 2 that play back with no audio.

But in looking at the packing and what not they are all correct. Everything is framed right, the audio is there, the PTS/DTS timestamps are present and correct the PTS is staying aligned with the SCR values, so I can't for the life of me figure out what could be going wrong.

Sadly I think for now the soon to be released version with the new HD code in it is going to be about as good as it gets for awhile. I need something that will play these streams back properly. Right now I have reduce the display size by to at least 25% of normal for it not to be "skippy" during playback as it is. makes testing sync a royal pain.

And a new machine for this purpose is a ways out for me. Especially given that I don't even have an HD Tivo. I wish there was a tool out there that would point out problems in the stream/packing/etc... I have written my own to do this and they have worked well in the past but like I said. They report these problem clips as 100%. So I missed something that is an issue with HD streams for some reason.

Just a bit more regression testing to go before the release but it is too late to continue tonight.

--jdiner

TheSaint
01-07-2005, 10:32 AM
Can anyone claim the stream:

KERA-PBS.ty

For whoever that one belongs and whatever station put it out, together you win the worst stream of the year award.

I have been going through it trying to figure out why it behaves so poorly.

I have found audio records 12 byte short. I have found them various sizes up to 8 bytes to long. And here is one that is 160 bytes too long. When playing just the audio when split out through PowerDVD I get constant noice and garbage.

I have made it so this kind of stream doesn't freeze or crash TyTool but do to the content it is going to be pretty unusable.

--jdiner

Yep, that's mine. After all, it's PBS :)
I don't know if they are consistently bad, maybe I need to send you several different snippets so you can determine if they suck all the time or just this once. I'll burn you a DVD and send it to you.

jdiner
01-07-2005, 01:21 PM
I got another KERA clip fom someone else. Don't know if it is the same region or what but it is also bad. Really bad but not as bad as the first one. Scary.

What other players are people here using to play the MPEG-2 HD output? I want to try some others and see if I can figure out anything about the HiDef playback issues.

Like i said last night the streams appear to be correct but the results aren't good. And they look just like some of the other results.

I figured out the audio playback issues. Those where no audios is heard. They appear to be using bitrates that I had never seen before and it just appears that the player I am using doesn't like the bitrat/freq combination.

--jdiner

Rowan
01-07-2005, 01:30 PM
KERA is the PBS station from Dallas. They are the only ones that I get a weak signal from (less than 90) all the other channels are > 90. So I would say that may be part of the problem. FYI: I did not send the samples to you - still have not hacked my HD unit.

Rowan

jdiner
01-07-2005, 01:37 PM
Ok. Good to know.

I took multiple samples from the same area from different people. The more I had the better chance of getting proper code in place. A good thing to. I made everything from the first 2 posters work (denver and SF Bay area) and then nothing from boise would work. Seattle was not bad and Dallas was the worst. Anyway now things work really nicely.

However I am beginning to think that I should just release this thing and let those with installation for playing them take the next round of testing rather than trying to do it all myself. I don't like to release software I don't think is right but I am not certain what is wrong with these few clips...

--jdiner

snoots
01-07-2005, 02:19 PM
We can break anything, and then tell you we did not do anything different than "when it used to work" !

mattdb
01-07-2005, 02:31 PM
Bring it on! I have a ton HD ty files in the waiting. Would be happy to burn some of these to DVD for you if have any way to watch them.

The AVEL Linkplayer2 likes the ones I have been throwing at it so far.

Matt

jdiner
01-07-2005, 02:32 PM
Ugh. This sucks. I just tried ZoomPLayer to discover it uses the DirectShow filter approach meaning same old codecs. So same old problems.

So then I tried WinDVD6. Plays the broken streams perfectly and the working stream in a very broken fashion.

Which tells me nothing.

Way back when there was a program people were using to mux things that was doing something I thought odd at the time. Each SCR counted up only by 1 for each PACK following the very first one. What program was that? Can anyone point me back to it? I don't seem to have it anymore and I given than I am getting all of the data I am beginning to think that I need to revisit the actual output to solve this last problem.

--jdiner

snoots
01-07-2005, 02:34 PM
mattdb

are you playing them as MPEG or ty ? I have been unable to get the advanced server to playback tys properly, I just get audio and then only sometimes.

jdiner
01-07-2005, 02:52 PM
Alright. You guys can test it. here it is...

The 2nd pre-release. This contains 100% all new OTA processing code. Works on everything I have thrown at it now but not everything works 100% for some reason.

Out of 85 test shows I had 2 play with no audio, appears to be a wierd bitrate/freq combination that just won't play on anything, and 3 that playback in a skippy popping way. Annoying but...

So give it a whirl and see what results you get.

Installation notes:

Unpack the new TyTool*.exe and copy it into the real TyTool directory/location. Then run it from that main directory. That simple.

--jdiner

jdiner
01-07-2005, 02:54 PM
What no downloads yet? It has been posted for 2 whole minutes? :)

--jdiner

mattdb
01-07-2005, 02:59 PM
mattdb

are you playing them as MPEG or ty ? I have been unable to get the advanced server to playback tys properly, I just get audio and then only sometimes.

I have been using TYTOOL to kill some of the stuff and split the ty's so that they fit on a single layer dvd.

Not trying to play native ty files. Return of the King takes 5 DVD's.

mattdb
01-07-2005, 03:00 PM
What no downloads yet? It has been posted for 2 whole minutes? :)

--jdiner

Sorry I am at work. Gotta work to pay to play. ;-)


EDIT: Just logged on to house PC and downloaded it. ;-)

jdiner
01-07-2005, 04:06 PM
Yeah. Work takes an amazing amount of time doesn't it.

--jdiner

justinkwaugh
01-07-2005, 04:49 PM
This is probably the wrong thread for these questions, though they are about extracted hd from my hr10-250. Here they are...

I've extracted a few things in HD content from my tivo, directv broadcast stuff.. not OTA. Tytool did a fine job of editing / converting to mpeg 2. The videos play fine in wmp. Various tools report the videos as 1280x1088 resolution at 29.976 fps. I have been trying to use nero recode to convert to Mp4 AVC w/ multichannel AAC sound in preparation for the hd-dvd era. The transcoding works without error, and the video is beautiful, however the audio is severly out of sync from the video, and appears to be quite a bit longer than the video. If I look at the original extracted mpg in virtual dub it tells me that my video stream is 1:14:38.574 in length, and the audio is 1:31:30.144 in length. Now for the actual questions....

1. How does 1280x1088 end up looking 16:9 on playback? Why do the extra 8 pixels exist on the vertical res... those lines are garbagy on playback.

2. How does the audio sync up correctly in the extracted mpeg 2 version during playback if it is reported as being 17 min longer?

3. Anyone know of the best way to resolve the audio length problem I'm having when I transcode? Should I change the length of the audio using something like AC3Machine before the transcode?

I appreciate any suggestions or edification you guys can give.

jdiner
01-07-2005, 06:13 PM
The transcoding engine is ignoring the TFF/RFF flags and the odd way that DTV uses them. As a result more that DTV uses them, which is extensive the further and further off the audio is going to be by the end. Should get noticably worse over playback time.

As for the fix, a proper transcoding engine is what is required. But TyTool isn't one.

--jdiner

lsmod
01-07-2005, 06:39 PM
1. How does 1280x1088 end up looking 16:9 on playback? Why do the extra 8 pixels exist on the vertical res... those lines are garbagy on playback.



MPEG2 resolution and aspect ratio are totally unrelated. Pixels aren't square. The MPEG headers contain an aspect ratio, and the decoder stretches the pixels it has to fit.

DTV does most of the HD stuff in 1280x1088, just as they do SD stuff at 480x480. In the SD world, that's called 2/3 res, and their HD trick is the same. DVDs do the same thing: Most movies are 16:9 and 720x480. The player knows (because you told it) if you have a 16:9 or 4:3 TV, and scales/letterboxes accordingly.

As to the extra 8 lines, MPEG2 resolutions must be a multiple of 16 pixels., so you get 1088, not 1080. It has been reported that using DVDPatcher to change the headers will work, but I've never bothered. That will be in the overscan on most displays anyway.

-Z

tPeter42
01-07-2005, 06:49 PM
As for the fix, a proper transcoding engine is what is required. But TyTool isn't one.

--jdiner

Are you saying that no HD mpg created from a .ty file using TyTool can be transcoded successfully using some other transcoding software, or am I misunderstanding? In other words, might Dr. Divx, tmpgenc, or some other transcoder work even if nero recode doesn't?

Sean.

jdiner
01-07-2005, 06:58 PM
Are you saying that no HD mpg created from a .ty file using TyTool can be transcoded successfully using some other transcoding software, or am I misunderstanding? In other words, might Dr. Divx, tmpgenc, or some other transcoder work even if nero recode doesn't?
No. It has NOTHING to do with TyTool.

Run a search, for posts not threads, on the various posts I made about the TFF and RFF flags. They control how the fields are used to recreate the actual playback images.

DTV uses them in wildly non-standard ways.

If the transcoding engine is aware of these flags and uses them properly then all is fine. If not, and most of them historically ignore them entirely, you get garbage as output.

--jdiner

jdiner
01-07-2005, 07:21 PM
Ok. Look one more time on transcoding and how DTV messes things up. :)

I didn't want to rewrite it but perhaps something specifically about this can be used as a pointer later when the question arises again.

First a few definitions:

Picture = An MPEG-2 structure that contains either 1 progressive image or 2 interlaced fields. Everything from DTV is interlaces at the SD level. Everything I have seen from DTV, and that is not much, is interlaced at the HD level as well, but I am nowhere near as certain at the HD level.

TFF == Top Field first.

RFF == Repeat first field.

Now the "standard" final display mechanism is this, decode the picture (i.e. both fields) and display 1 field for a duration of 1/(29.97*2) and then display the second field. That is it.

However using the 2 flags above you can change that behaviour. And DTV does to try and save space and it is honestly a major savings in data space.

The TFF is used to pick the order of the field at each individual picture header. Almost all "standard" MPEGs use TFF = 1, i.e. Top Field first, and it never ever changes. In a DTV stream is flops almost every GOP. So a TFF of 0 means show the Bottom field first and then top. Still 2 fields, same playing time etc..., but shown in the altered order.

The RFF is used to extend the play time of the picture by 1 fields duration. So TFF = 1 and RFF = 1 means show the top field first and then the bottom and then the TOP field again before moving on.

The effect we go from the standard of:

[T B] with a duration of 1/29.97 total

to a new pattern of:

[T B T] with a duration of 1.5/29.97 total.


Now this might seem familiar to some. The telecine (often movie) reduction is a fixed pattern of these:

[T B T]

Right next to each other in a prescribed order gives you 24fps rather than 29.97fps.

[B]However, AND THIS IS KEY, this is NOT what DTV did. They use the above repeating fields anytime it was "close" to the same. Why? Save the bandwidth of sending an extra field through the sat. So you do not get the regular pattern shown first, you don't even get the telecine pattern, you get something completely random. Further you don't get the signal'ed frame rate of 24 you still get 29.97 in a DTV stream. Because technically it is...

The only answer is that the decoder has to be aware and use these flags even in the "normal" stream speed of 29.97.

Here is why in a simple picture form:

Standard:

[T B][T B][T B][T B][T B][T B][T B][T B][T B][T B]

10 pictures 2 fields each total playing time 10/29.97 or .334 seconds.

DTV:

[T B T][T B T][B T B][T B T][B T B][T B T]

7 pictures, 3 fields each, total playing time (7*1.5)/29.97 or .35 seconds.

Note that the extra play time is from that extra Top field I that is in there.

Now if you ignore the flags what you appear to see is:

[T B][B T][T B][B T][T B][B T][T B]

7 pictures, just like before, 2 fields each and a total playing time of .23 seconds.

Now the audio is running along as if "tied" to the video and so it is expecting the above to take .334 seconds to display. Thanks to the ignored flags you get .23 seconds and in the first GOP you are 10ms off. Not visible yet but 1/4 the way to a visible skewing. The second GOP makes it worse and so on through the stream

By the time you are at the end of a 1hour show you would be WAY off.

[b]Bottom line time:
So the what does it take to get sync when transcoding? And mechanism that is aware that even in 29.97 FPS you have to pay attention too and use the RFF and TFF flags.

--jdiner

justinkwaugh
01-07-2005, 08:13 PM
Thanks for the super informative post. Now to find a solution that will work for me (aside from petitioning nero to fix their stuff, which I will do in another forum).

Does anyone know of an avisynth plugin that will do the appropriate inverse telecine on the DTV stream? Seems that a dgindex project using field mode none run through an avisynth plugin that knows the correct thing to do would end up serving the frames to the encoder correctly. Worst case I suppose I could try to write one, but I would prefer to find an existing product.

mikemav
01-07-2005, 09:47 PM
MPEG2 resolution and aspect ratio are totally unrelated. Pixels aren't square. The MPEG headers contain an aspect ratio, and the decoder stretches the pixels it has to fit.

DTV does most of the HD stuff in 1280x1088, just as they do SD stuff at 480x480. In the SD world, that's called 2/3 res, and their HD trick is the same. DVDs do the same thing: Most movies are 16:9 and 720x480. The player knows (because you told it) if you have a 16:9 or 4:3 TV, and scales/letterboxes accordingly.

As to the extra 8 lines, MPEG2 resolutions must be a multiple of 16 pixels., so you get 1088, not 1080. It has been reported that using DVDPatcher to change the headers will work, but I've never bothered. That will be in the overscan on most displays anyway.

-Z

When you play these files back on an I-O Data LinkPlayer DVD/network HD box, it shows a small strip of gray at the bottom of the screen- the extra 8 pixels.

JustDan
01-07-2005, 11:06 PM
Using 9R19=Pre2 I'm seeing behaviour similar to a post in the TyTool9R18 thread. When making a key file or muxing a file, TyTool is posting about
a stream of errors about what appears to be audio errors.
!!! BAD
!!! sample rate code == 48 (0)
!!! frame size code == 0x21 33
!!! Bit Rate == 512
!!! bsid == 1
!!! bsmod == 5
!!! acmod == 5
. !!! BAD
!!! sample rate code == 44.1 (1)
!!! frame size code == 0x3A 58
!!! Bit Rate == 951628302
!!! bsid == 0
!!! bsmod == 4
!!! acmod == 2
. !!! BAD
!!! sample rate code == 44.1 (1)
!!! frame size code == 0x6 6
!!! Bit Rate == 56
!!! bsid == 13
!!! bsmod == 5
!!! acmod == 7
.. !!! BAD
!!! sample rate code == 48 (0)
!!! frame size code == 0x2E 46
!!! Bit Rate == -1255287978
!!! bsid == 26
!!! bsmod == 0
!!! acmod == 0

After awhile (300Mb in on Showtime program, 800Mb in another) the progress screen "takes off", dumping data to the screen, but not progressing any further in the file. The data is in hex and I'm trying to capture any
errors just before that, but abort has not been fast enough for me to
scroll back far enough.

TyTool-HD-Pre and earlier process these files fine. It appears to only be DD that is impacted.

snoots
01-07-2005, 11:07 PM
Got the following during key processing and again when converting to mpg and it is playing back now with good audio. I will look through and see if something shows up at the point of the message.


.........45100.........45200.........45300.........45400... !!! BAD
!!! sample rate code == 44.1 (1)
!!! frame size code == 0x9 9
!!! Bit Rate == 64
!!! bsid == 18
!!! bsmod == 7
!!! acmod == 0
......45500

.........45600.........45700.........45800.........45900.........46000

snoots
01-07-2005, 11:24 PM
Sorry for not posting version, latest prerelease. Watching the show now on my Avel Linkplayer and it is playing back fine with good audio.

lsmod
01-08-2005, 12:22 AM
When you play these files back on an I-O Data LinkPlayer DVD/network HD box, it shows a small strip of gray at the bottom of the screen- the extra 8 pixels.

What are you using for a display device? This means things are set up for (less than?) zero overscan.

-Z

jdiner
01-08-2005, 12:29 AM
When making a key file or muxing a file, TyTool is posting about a stream of errors about what appears to be audio errors.
!!! BAD
!!! sample rate code == 48 (0)
!!! frame size code == 0x21 33
!!! Bit Rate == 512
!!! bsid == 1
!!! bsmod == 5
!!! acmod == 5

I forgot to remove this one. Don't sweat these. They mean nothing. And I mean nothing.

When I was first tracking the 0b 77 in the data I was checking the results of the decode for that "header". This is the report that it saw one that isn't actually a header. Nothing more. It was informational at the time and has nothing to do with the new audio code or the output. I will be removing it ASAP.


After awhile (300Mb in on Showtime program, 800Mb in another) the progress screen "takes off", dumping data to the screen, but not progressing any further in the file. The data is in hex and I'm trying to capture any
errors just before that, but abort has not been fast enough for me to
scroll back far enough.

TyTool-HD-Pre and earlier process these files fine. It appears to only be DD that is impacted.
Humm. That dump takes place when the audio is "off" internally. I see it when there is too little data or too much that is supposed to be 1 single record. But it recovers here. Even in the worst streams it re-locks onto the data all goes smoothly from there.

The reason it is still there is there should be a .txt file that contains all of it in the same place as the .vob output file. If there is really a problem I will need a copy of that output file. ZIP'ed or RAR'ed at the strongest compression levels please.

--jdiner

jdiner
01-08-2005, 12:30 AM
Sorry for not posting version, latest prerelease. Watching the show now on my Avel Linkplayer and it is playing back fine with good audio.
It should. That is informationally only. It should have been removed but I was still using it right up until I made the release.

Don't worry about this one.

--jdiner

jdiner
01-08-2005, 12:57 AM
I just lost a nice long post about a few of the other messages you might see. That sucks. No great desire to retype all of that at this point so fior now let me just say that the 3 line report that start with @@@ is all about the internally predicted PTS versus what is extracted from the source stream. It indicated where the code thinks things should be versus where they are. It is odd but things aren't always correct in there streams, I ahve seen 10ms changes mid-stream but using them the sync always stays on. (Almost like the audio encoding process had to be restarted...)

Anyway there are a couple of new messages in there. Most are informational/warnings. The errors are clearly marked. But then !!! BAD was supposed to just be a warning so perhaps not as clear as I thought...

--jdiner

JustDan
01-08-2005, 04:31 AM
I re-ran the tests, and indeed it does continue to process the stream. Normally is see about 20MB/s and when it hits the suspect area is drops
closer to 20KB/s. So after a minute of watching the data scroll by I thought it was hung.

PM me with the best way to get a sample to you. I normally could burn a DVD, but a bad BIOS upgrade has disabled my DVD drive. The first instance in this stream is at about 385MB. I've aborted the mux after letting it process 15MB past the error point. RAR'ed with best compression with a copy of the text file is about 355MB. I can put that on a private FTP site if the size is small enough.

Dan

joeblough
01-08-2005, 02:55 PM
i would suggest taking a look at mplayer. it is open source and it has played pretty much anything i've ever thrown at it. not only mpeg4 but all manner of 1080i and 720p captured OTA from bay area TV stations.

www.mplayerhq.hu




What other players are people here using to play the MPEG-2 HD output? I want to try some others and see if I can figure out anything about the HiDef playback issues.

--jdiner

jdiner
01-08-2005, 04:02 PM
PM me with the best way to get a sample to you. I normally could burn a DVD, but a bad BIOS upgrade has disabled my DVD drive. The first instance in this stream is at about 385MB. I've aborted the mux after letting it process 15MB past the error point. RAR'ed with best compression with a copy of the text file is about 355MB. I can put that on a private FTP site if the size is small enough.
The zip'ed .txt file should be small enough to post here. Until I know if there is any different between what is there and what I already have there is no pressing need to get the TyStream file itself.

--jdiner

JustDan
01-08-2005, 07:01 PM
Here's the text file.

jdiner
01-09-2005, 01:25 AM
Wow. Well that stream sucks in one way or another. I think I will need a copy of it. Check your PMbox.

--jdiner

JustDan
01-09-2005, 02:08 AM
I assume that you would prefer the .ty stream over any mux'ed output.

I need to get access to another burner, but hope to get it out early next week.

I'm curious if anyone else is seeing this with Showtime-HD streams. I've
been testing each of the Pre releases. HD-Pre and 9R18 both process this
same stream without error and the resulting mpg does not have any audio issues.

I'm trying a HBO-HD stream now to see if it does the same thing.

Dan

jdiner
01-09-2005, 02:30 AM
HD-Pre and 9R18 both process this same stream without error and the resulting mpg does not have any audio issues.
Believe it or not I changed things between these 2 versions. Pretty seriously.

Something terribly wrong is going on. But I have yet to figure out what. The audio lengths make me think that something just isn't going right but I have no idea what at this point. Right now the data just appears to be off. But I can't for the life of me figure out how or why yet.

--jdiner

lsmod
01-09-2005, 03:10 AM
I'm curious if anyone else is seeing this with Showtime-HD streams. I've
been testing each of the Pre releases. HD-Pre and 9R18 both process this
same stream without error and the resulting mpg does not have any audio issues.


Same here. I've processed a ton of files today, and have gone back to HD-pre, because HD-Pre2 has started spraying hex dumps on every stream I've processed. Mostly HDN, HDNM, DSHD streams today, but a couple of HBOH as well.

I wouldn't worry about it, except that one file had been blasting byte for 4 hours when I killed it.

No audio problems in the finished product, from HD-Pre.

-Z

JustDan
01-09-2005, 04:16 AM
I just tried a HBO-HD of X-Men 2 and it does the same thing. Up until I tried it every pass had been on Showtime material.

I am actually glad to see lsmod's report.

I noticed someting in the txt files. I've been following the TyTool threads and
recall a couple days ago a discussion about the DD packet flags, forgive me that I don't understand all that was posted, but I see that the last "Frame Size Code" posted to the log before the hex dumps is consistently 0xB 11.
This section of the log seems to always preceed the hex:
!!! sample rate code == 48 (0)
!!! frame size code == 0xB 11
!!! Bit Rate == 80
!!! bsid == 14
!!! bsmod == 7

The !!! acmod value has been different though. Is there a chance that this might provide a clue.

lsmod- Can you look at the txt files from your processing and see if this pattern exists in them?

lsmod
01-09-2005, 01:57 PM
lsmod- Can you look at the txt files from your processing and see if this pattern exists in them?

Actually, I turned off .txt files before I switched back to HD-Pre to see if it made a performance difference.

But if I read Josh's last post on the subject correctly, those messages aren't terribly useful, and just get printed every time something that might be a header comes along and isn't. (every occurance of 0x0b77 in the file, or some such thing)

If I'm wrong, and those messages are useful, I'd be happy to post a text file.

-Z

jdiner
01-10-2005, 12:29 AM
The !!! acmod value has been different though. Is there a chance that this might provide a clue.
No. As mentioned before it means nothing.

--jdiner

jdiner
01-10-2005, 12:52 AM
I have been going through the streams I have looking for causes for problems etc...

What I am learning here isn't good news.

There are several data sequences I use in TyTool to lock onto things. These values are fuzzy at best but taken all together and in context they let the code draw conclusions about what is going on.

In the DTV streams things remain consistent as they always pretty much where. They are infact more regular now than ever before. Which I suppose is good news.

However in the OTA streams things that should apply don't. So many things are off it isn't funny. What follows are few quick data blocks showing what I am talking about:



ParseChunk: : Number of Records: 13 ( d 0 2 80)
ParseChunk: : # 1 vid ( 14160): 3 75 2 e0 2b 17 5c 10 0 0 0 1f d0 16 ae c3
ParseChunk: : # 2 vid ( 176): 0 b 7 e0 2b 17 93 60 0 0 0 1f ed eb a1 56
Vid PTS: (592363256) 01:49:41.813
ParseChunk: : # 3 vid ( 4): 0 0 42 e0 2a ad d0 74 0 0 0 1f ed eb a1 56
ParseChunk: : # 4 vid ( 8): 0 0 8c e0 2a ad d0 78 0 0 0 1f ed eb a1 56
ParseChunk: : # 8 aud ( 1552): 0 61 9 c0 2a e1 aa 80 0 0 0 1f ed eb a1 56
Aud PTS: (592309748) 01:49:41.219
ParseChunk: : # 9 aud ( 1548): 0 60 c9 c0 2a e1 b0 90 0 0 0 1f ed eb a1 56
Aud PTS: (592312628) 01:49:41.251
ParseChunk: : # 10 aud ( 1552): 0 61 9 c0 2a e1 b6 9c 0 0 0 1f ed eb a1 56
Aud PTS: (592315508) 01:49:41.283
ParseChunk: : # 11 vid ( 4): 0 0 48 e0 2a ad d0 80 0 0 0 1f ed eb a1 56
ParseChunk: : # 12 vid ( 111856): 1b 4f 2 e0 2b 17 94 1c 0 0 0 1f ed eb a1 56

ParseChunk: : Number of Records: 3 ( 3 0 ff ff)
ParseChunk: : # 1 vid ( 114756): 1c 4 42 e0 2b 19 49 c 0 0 0 1f ed eb a1 56
ParseChunk: : Number of Records: 4 ( 4 0 ff ff)
ParseChunk: : # 1 vid ( 61576): f 8 82 e0 2b 20 66 18 0 0 0 1f ed eb a1 56
ParseChunk: : # 2 vid ( 45216): b a b e0 2b 21 56 a0 0 0 0 1f ed eb a1 56
Vid PTS: (520159124) 01:36:19.545
ParseChunk: : # 3 vid ( 24212): 5 e9 4b e0 2b 22 7 40 0 0 0 1f ed eb a1 56
Vid PTS: (520162127) 01:36:19.579

Found an OOB packet... The Video Diff is: 00:13:22.268
BBB the PTS was bad, but the new SEQ check lines up|!?
Nope... Not in sequence... Skipping it...


ParseChunk: : Number of Records: 15 ( f 0 ff ff)
ParseChunk: : # 1 vid ( 21036): 5 22 c2 e0 2b 22 65 d4 0 0 0 1f ed eb a1 56
ParseChunk: : # 9 aud ( 1548): 0 60 c9 c0 2a e2 3 2c 0 0 0 1f ed eb a1 56
Aud PTS: (520122672) 01:36:19.140


The above shows for the most part as in sequence as things are running along. In the first chunk shown the timestamps are at 1:49:41, 1 hour 49 minutes and 42 seconds. This one ends with the start of a P-Frame.

In the second chunk you just get more video data.

In the third chunk you get a new video record that is a B-Frame. But the PTS for it is at 1:36:19... Or 13 minutes and 22 second earlier. This gets flag'ed as a PTS reset but what is in there when you actually watch it is the start of that very same show.

Which makes no sense at all. And leads me to a few very unsatisfactory conclusions... Either the extraction mechanism is busted, or somehow the tivo saw that much data twice. But if it saw it twice then you should se it on the TV when watching from the tivo.

Now it gets worse, According to the 2 main stream data this is both not what should have been next (but it is just a touch in the future) and is exactly what should have been next.

Like I said I see this only on OTA streams. But things this badly malformed... I dunno what to do with them. I don't have any idea what the receiver actually does with them. Does it just dump everything until we hit more data tha we actually want to see or ???

I don't know that much of anything can be done with them honestly. At least not by me.

Continued...

--jdiner

jdiner
01-10-2005, 12:53 AM
So here is another stream showing the same general behaviour:



ParseChunk: : # 11 vid ( 52936): c ec 8a e0 2b 86 e9 30 0 0 0 2a e2 1c b3 50
Vid PTS: (134081503) 00:24:49.794

ParseChunk: : Number of Records: 4 ( 4 0 ff ff)
ParseChunk: : # 1 vid ( 8056): 1 f7 82 e0 2b 87 b7 f8 0 0 0 2a e2 1c b3 50
ParseChunk: : # 2 vid ( 40832): 9 f8 b e0 2b 87 d7 70 0 0 0 2a e2 1c b3 50
Vid PTS: (134075497) 00:24:49.727

ParseChunk: : Number of Records: 14 ( e 0 ff ff)
ParseChunk: : # 1 vid ( 18740): 4 93 42 e0 2b 89 19 4c 0 0 0 2c 2a 43 1f a1
ParseChunk: : # 8 aud ( 1552): 0 61 9 c0 2b 41 8 3c 0 0 0 2c 2a 43 1f a1
Aud PTS: (87841679) 00:16:16.018

Found an OOB packet... The Video Diff is: 00:08:33.412
BBB the PTS was bad, but the new SEQ check lines up|!?
In checking dolby errors we need more than 1 audio packet!
Nope... Not in sequence... Skipping it...

From 24 minutes back to 16 minutes. Get marked as a PTS reset etc... But Once again when watching it is appears to be part of the same stream just older.

Both of these are immediately followed by the hex dump of the audio because they are off once things are glued back together. Why? Because the 2 pieces on the sides of the change don't belong back to back.

In neither case do I have the rest of the file up to the point where it might catch back up to see what might really be going on there. And this is happening at the start. But what if it happens in the middle or at the end of a stream? Does it even occur there or this only found right near the start?

I am so frustrated with this that I am ready to scream. I just don't get how anything can play this kinds of crap streams.

--jdiner

jdiner
01-10-2005, 12:58 AM
Kind of like before with the DTivo, the only honest hope I think I have with this getting an HDTivo of my own. Something that I can just pull and pull from. Which isn't feasible. My budget doesn't have $1k in it for a new Tivo. What it would let me check is the extraction engine etc... See if things are as they should be etc...

But even then it isn't really a complete solution. In checking things on the web I find that I get no OTA stations here. There are some in SLC and I could put the tivo down there to start grabbing things, but what if by some hook or crook the SLC stations aren't doing this kind of crap?!!?

I am at a loss as to what to say or do next folks. I tried. I really did. But I have run out of options at this point.

3 of the streams where I found these kinds of resets in the blocks are:

KXAS-NBC.ty
WFAA-ABC.ty
Without a Trace-Malone v. Malone.ty.txt

I don't know who sent them to me anymore. Things just kind of got all thrown into 1 directory. If the people that sent these too me will check to see what it looks like on the tivo and perhaps be willing to send me much much more of the streams like hopefully the entire thing then I might be able to get a bit further, but even then it might just be more of the same and then I am back at square 1.

--jdiner

jdiner
01-10-2005, 01:02 AM
As for what lsmod is seeing. I have no clue. The hex dump shows where a problem occurs specifically. if it continueing to dump then it is possible things are really off or that things are continually erroring out for some reason. I don't know and I can't tell just from a description.

If JustDan or lsmod want to send me the entire bad stream I will take a look at it but from some what I have seen written on here a complete stream could be a ton of DVDs. Don't know if it is worth it. if you want to throw it onto a smaller/older HD and mail me that I will ship the HD back to you once I get it, other than that I can't help much more.

Like I said I don't know that there is much more I can do. In trying to get OTA working looks like I messed up the DTV support which is probably more a main case for people anyway which I don't like to see happening. Not at all.

So if anyone has any ideas for me or wants to loan me an HDTivo with tons of things already recorded on it, then sound off. Otherwise, I think I hear the fat lady warming up.

--jdiner

lsmod
01-10-2005, 01:52 AM
So if anyone has any ideas for me or wants to loan me an HDTivo with tons of things already recorded on it, then sound off. Otherwise, I think I hear the fat lady warming up.

--jdiner

I've actually had a 100GB disk full of streams sitting on my desk for a couple of weeks now. The problem is that I always remember to ship it out to you *after* I've left the house without it. :D

So, given the recent fun, I'm going to march out to the office, verify that the problem with Pre2 happens on (at least some of) those streams, and the put the disk in the car so it will actually get shipped tomorrow.

-Z

PS: I've got a couple of multiple-reset SD streams for you too.

jdiner
01-10-2005, 02:26 AM
In taking a deeper look at the data. It isn't a bad extraction that is causing that repeat in the data. In looking at it is actually in there twice. Can't explain the why on that one but that is how it is.

So at least we can rule that part out. Hummm. Food for thought anyway.

--jdiner

ronnythunder
01-10-2005, 03:35 AM
josh, these streams where there seems to be vid from earlier in the show... are you sure it's vid from the exact same show? i know this is far fetched, but is it possible this could be leftover data in the same mfs part from an old episode that just happened to be from the same show? you know, like some stuff from "the trouble with the tribbles" in with "balance of terror"?

the only other things that come to mind are: (1) trouble with tyfilesplit, or (2) the show was >2gb or 4gb, extracted with mfs_ftp, and the client or server wrapped around at 2**31 or 2**32 yadda yadda.

i know, i'm really reaching...

ronny

jdiner
01-10-2005, 04:25 PM
josh, these streams where there seems to be vid from earlier in the show... are you sure it's vid from the exact same show? i know this is far fetched, but is it possible this could be leftover data in the same mfs part from an old episode that just happened to be from the same show? you know, like some stuff from "the trouble with the tribbles" in with "balance of terror"?

the only other things that come to mind are: (1) trouble with tyfilesplit, or (2) the show was >2gb or 4gb, extracted with mfs_ftp, and the client or server wrapped around at 2**31 or 2**32 yadda yadda.
In checking things the data inside the stream says it is part of that stream in each case, and in checking the actual show on my DTivo, only had the Without a trace episode, it is indeed the start of the same exact show.

Yeah I am grasping at straws too. I have seen this only 1 other time. In a stream that was from one of these SD Sat cards.

--jdiner

dtle
01-13-2005, 12:21 PM
3 of the streams where I found these kinds of resets in the blocks are:

KXAS-NBC.ty
WFAA-ABC.ty
Without a Trace-Malone v. Malone.ty.txt

--jdiner

As per our exchange of PM, I was the person who gave you the KXAS-NBC and WFAA-ABC files.

One thing that I don't know if you are aware of is that these two stations multicasts on different substations. The NBC station has a radar weather map, and the ABC station has both a radar weather map and a all-news channel. I believe that the DTivo records the whole stream (all substations) onto the hard drive, even though only one substation is picked.

Could you be seeing the data for these substations, especially the radar weather map, where it's mostly static?

mikemav
01-13-2005, 12:50 PM
Josh-
I am still having the issue with some .ty files only outputting about 500MB of data. Recall that a few pages pack we discussed that after the 500MB mark, the application speeds up dramatically and writes nothing else to the .vob file. As I said, it is not the priority set issue since I have the corrected mfs_ftp S2 binaries that fix this, AND I have run the files in question through unpriority.exe just in case. You suggested putting the app in verbose or logging mode and posting the results. Forgive my ignorance, but where do I do that? I want to try again and this is driving me nuts. Seems to happen mostly (or only?) on HDNet programs. If the logs do not tell us anything, I will try to burn two DVDs with the 8GB .ty split to send to you. Also a thought- should I redownload the latest Tytool and the cygwyn file? I had been using 9r17, and then augmented those with the pre-release HD app. I wonder if some other components of the r19 application are missing, older versions, or needed with the HD app that r17 did not have?

jdiner
01-13-2005, 02:47 PM
As per our exchange of PM, I was the person who gave you the KXAS-NBC and WFAA-ABC files.

One thing that I don't know if you are aware of is that these two stations multicasts on different substations. The NBC station has a radar weather map, and the ABC station has both a radar weather map and a all-news channel. I believe that the DTivo records the whole stream (all substations) onto the hard drive, even though only one substation is picked.

Could you be seeing the data for these substations, especially the radar weather map, where it's mostly static?
No. What I am seeing, as I mentioned, is the same show with a restart. You get several minutes into the show, but it appears to have started partway into the show, and then is followed by the very start of that stream.

I am definately not seeing radar or weather reports or whatever... It is the same channel and apparently the same show. Although in the case the of the football segment it resets into a commercial break so there is no way to tell. In the case of the Without a Trace episode it is reseting the very start of the that episode.

But it was an idea worth looking into.

--jdiner

bgrubb1
01-18-2005, 12:21 AM
Joshua
You asked earlier what others are using for HD playback. I am attempting to use a Roku Photobridge (formerly called a HD1000). They have new beta firmware that plays ripped dvd's flawlessly. Quality looks great. I am having problems with files extracted off my hd10-250. I have tried all possible encoding schemes in the HD pre-release. original VOB still works the best but I have a problem with the Roku. When I play files from my hd10-250, they play, but eventually crash the network on the Roku in a manner that appears to be a memory leak( both ota and non). System intermittently just freezes then no network / reboot time. You mentioned a while back that you were working on a new format (not vob new/old - mpg new/old) that was appropriate for hd content. I suspect that is not present and may be my issue. Things are better, but still fail if I run the file through DVDAuthorGUI Can you point me to anything that can help me take the issue to the next stage ?? Roku will probably help, but I need to provide more than "it don't work"

Thanks in advance
..Barry

tmb
01-22-2005, 07:31 AM
I have found that if I enable the Tridimension feature in WinDVD 6 for JUST high def vobs made with tytool, that WinDVD crashes.

It works fine with Standard Definition vobs made with tytool.

I obviously do not have access to any other source of high def VOBS to try out so I was wondering if jdiner cuould try this out and see if 1) he could duplicate and 2) see if this is a bug in Windvd for high def material.

If it is a bug in Windvd , this could be reported to Intervideo for a fix.

Not to sound nit picky but that Tridimension feature makes for a a very nice looking,smooth picture for fast moving scenes.

Thank You

jdiner
01-25-2005, 05:37 PM
I have found that if I enable the Tridimension feature in WinDVD 6 for JUST high def vobs made with tytool, that WinDVD crashes.

I obviously do not have access to any other source of high def VOBS to try out so I was wondering if jdiner cuould try this out and see if 1) he could duplicate and 2) see if this is a bug in Windvd for high def material.

If it is a bug in Windvd , this could be reported to Intervideo for a fix.

Not to sound nit picky but that Tridimension feature makes for a a very nice looking,smooth picture for fast moving scenes.
I am sorry. But I have neither the time nor the inclination to debug WinDVD. It may or may not have problems but I can't put the kind of time required into trying to figure out what went wrong. You are on your own to find other materials and make whatever recommendations to that company that you wish to make.

--jdiner

tmb
01-28-2005, 07:09 PM
I am sorry. But I have neither the time nor the inclination to debug WinDVD. It may or may not have problems but I can't put the kind of time required into trying to figure out what went wrong. You are on your own to find other materials and make whatever recommendations to that company that you wish to make.

--jdiner

My request was simply to playback a High Def vob made with tytool in windvd with that option selected and see if it crashed.

It was my way of trying to politely point out to you that it looks like there is a bug in tytool - NOT ask you to debug Windvd.

In reading over various threads, I see that you are using Windvd for testing the output of tytool so this test would take all of 5 minutes for you to do.

jdiner
01-29-2005, 10:32 PM
My request was simply to playback a High Def vob made with tytool in windvd with that option selected and see if it crashed.
Tried it. For some reason I can't enable it.


It was my way of trying to politely point out to you that it looks like there is a bug in tytool - NOT ask you to debug Windvd.
Do you know what the feature actually does? Besides making it look better? It is a motion comp and colorization algorithm. Know what that means? Buffer space. Tons of it. it is trying to keep in memory enough video frame information to detect and deal with field ordering issues, high motion scenes, etc... so that it can compensate. Why is it crashing? Best guess... They have a buffer overflow error when the source material is just to large. if you check the intervideo website they talk about how much more memory is needed to play things back at various sizes and settings.

from their website:
128MB RAM; For Trimension function, 256MB RAM is recommended; For WMV HD playback, 384MB RAM is minimum request; 512MB RAM is recommended

Look at it. 2 times as much memory for the use of trimension on the same SD source. Trust me I have run into it over and over, memory management for HD is extremely hard to predict.


In reading over various threads, I see that you are using Windvd for testing the output of tytool so this test would take all of 5 minutes for you to do.
No. I mentioned I have it and have used it. I use PowerDVD 99.9999% of the time. That is the one I recommend to people.

--jdiner

tmb
01-30-2005, 07:43 PM
Tried it. For some reason I can't enable it.
--jdiner

Well, I apreciate your looking into this anyway.

They have made this version overly complicated in that enabling some of the post processing algorithms will not allow one to choose some others. Tri-dimension falls into that category.

Since you can not enable that feature, I'll just live without it.

I would just use PowerDVD but WinDVD has the great feature of being able to program in different Jump Forward and Backward time's for bouncing over commercials instead of having to use the Fast Forward.


Thanks for your time.

jdiner
01-31-2005, 12:08 AM
They have made this version overly complicated in that enabling some of the post processing algorithms will not allow one to choose some others. Tri-dimension falls into that category.
If you can figure out what I have to turn off I am willing to try it. But my best guess is that it is a buffer management issue. Especially since it works with SD stream but not HD stream. The audio between the 2 size wise is almost identical. It is the video stream, where TyTool does almost no processing at all, that is causing problems. And between SD and HD there are no video differences at all right now. Some to come with a custom mux'er but right now there is no difference.


I would just use PowerDVD but WinDVD has the great feature of being able to program in different Jump Forward and Backward time's for bouncing over commercials instead of having to use the Fast Forward.
Definately a nice feature. Haven't found that one yet. But like I said I almost never use it.

--jdiner

jdiner
01-31-2005, 12:12 AM
Oh man, I gotta by a better video card for this test machine. :)

I just ran some tests because I was curious. At the resolution that WinDVD and PowerDVD open up at it obscures most of the screen at 1024x768, and regardless of source material, it is almost always the same size. No matter what is being decoded from mpeg-1 music videos to mpeg-2 HD streams it takes 100% of the CPU. When I minimize it to 1/4 that res it takes 11%. My guess, my video card sucks and it is taking that much CPU just to push it to the screen.

So any recommendations from the HiDef crowd on a card that is known to work and not cost a bundle? I guess I would be looking from something to replace the $8 card I had laying around and just stuck in there. So anyone?

--jdiner

malfunct
01-31-2005, 02:35 AM
Oh man, I gotta by a better video card for this test machine. :)

I just ran some tests because I was curious. At the resolution that WinDVD and PowerDVD open up at it obscures most of the screen at 1024x768, and regardless of source material, it is almost always the same size. No matter what is being decoded from mpeg-1 music videos to mpeg-2 HD streams it takes 100% of the CPU. When I minimize it to 1/4 that res it takes 11%. My guess, my video card sucks and it is taking that much CPU just to push it to the screen.

So any recommendations from the HiDef crowd on a card that is known to work and not cost a bundle? I guess I would be looking from something to replace the $8 card I had laying around and just stuck in there. So anyone?

--jdiner

I'm fairly certain a few of my friends playing with media center use the ATI Radeon 9600xt with no problems. Its not a bundle and has a lot of modern features.

snoots
01-31-2005, 08:31 PM
Josh read your pm from me.

desplaines
01-31-2005, 09:35 PM
So any recommendations from the HiDef crowd on a card that is known to work and not cost a bundle? I guess I would be looking from something to replace the $8 card I had laying around and just stuck in there. So anyone?
For the best combination of price (US$100 or less), quietness (no fan on the board), and performance (including DX9 hardware ability), I recommend any of the "Radeon 9600 non-Pro" boards.