DTV files are getting bigger

[cipher]

Has anyone noticed the average file size for a show has gotten bigger? Some of the shows that I've been saving for a while are 40% bigger than an episode I pulled off last year. I've got an Hughes Series 1 DTV Tivo, running 3.5b-01-1-031. I just pulled off some one hour episodes that are 1.5GB. Usually one hour is like 900MB. The shows are the same size on the Tivo, so it's not an extraction problem. Did DTV change something?

[jt1134]

I haven't noticed larger file sizes, however, I was pleasantly surprised the other day when I extracted a few HD recordings that showed they were at full 1920x1080 resolution when I muxed them. Well, 1920x1088 anyways.

[Fugg]

Whatzit??!?!?:eek:

First it was DirecTV/TiVo to update DTivos (http://phx.corporate-ir.net/phoenix.zhtml?c=127160&p=irol-newsArticle&ID=1034357&highlight=)...

Then after years of video degradation, they're now using a higher sd bitrate?

Now I hear they're transmitting hd at an actual hd resolution??!?!?!?

Geeze!!! Has anyone noticed any flying pigs lately?

;)

[ronnythunder]

Now I hear they're transmitting hd at an actual hd resolution??!?!?!?do you have a link for this? it's something i care about since i got my hdtv in may :)

ronny

[dburckh]

I heard they used to broadcast at full 1080 HD resolution. Then they went to HD Lite at 1280x1088. I think that prompted a law suit. Maybe they lost, or just gave in.

[jt1134]

Here's some snippets from my TySuite J log, for anyone curious. Both recorded from Directv PPV. Mux'ing with tytompg produces the same results.
19:42:51.078 Refreshing from hdtivo
19:42:52.375 Starting dserver on hdtivo
19:42:57.656 Got 21 titles, 124303 MB from hdtivo
19:42:57.687 Refresh Complete
19:43:57.343 Queued C:\cygwin\mux\Casino Royale.mpg
19:43:57.375 Started C:\cygwin\mux\Casino Royale.mpg
19:43:58.843 Stream: Video Size: 1920x1088 Aspect: 16x9 fps: 29.97003 Audio: AC3 Delay: 0:00:00.003

19:50:57.187 Refreshing from hdtivo
19:50:58.218 Starting dserver on hdtivo
19:51:03.250 Got 21 titles, 124303 MB from hdtivo
19:51:03.281 Refresh Complete
19:51:27.890 Queued C:\cygwin\mux\Norbit.mpg
19:51:27.906 Started C:\cygwin\mux\Norbit.mpg
19:51:29.468 Stream: Video Size: 1920x1088 Aspect: 16x9 fps: 29.97003 Audio: AC3 Delay: 0:00:00.022
These were both mpeg2 recordings (obviously), and were also recorded off the 101 satellite, since I only have a basic dish.

[ronnythunder]

i see a mixed bag; from recordings within the last week:

firefly in hd on universal hd: 1280x1088
to wong foo on uhd: 1280x1088
diamond road on discovery hd theater: 1920x1088
ocean of fear on dhd: 1920x1088
...

i don't have anything from any other channels, but the recordings above and a few others seem to support that discovery hd is full res most of the time, but uhd is still hd lite most of the time.

still, this is progress. :)

ronny

[phat_bastard]

Has anyone noticed the average file size for a show has gotten bigger?.... Did DTV change something?

You won't get much discussion regarding SD stuff, but I have in the past noticed dramatic changes in recording size on certain channels. MTV / MTV2 are some that I've noticed this on in the past, where a half-hour program (same program, even same episode) will be 600mb one day, and 1.2gb another.

DTV and other broadcasters do this from time to time (ramp up compression, decrease bitrate) in order to free up bandwidth for sporting events and other prime time programming.

[cipher]

Thanks phat b, I'll have to see if these episodes shrink down the next time they're run. I had a hard time deciding where to post my question. I figured the only people that really notice how much space a show takes up would be other extraction people. Now I feel like my hacked SD tivo is so five minutes ago. It's so hard to keep up with the cool kids.

[ciper]

Do any of the satellite based tivos also have an analog tuner?

It's so hard to keep up with the cool kids.
Nice name

[jt1134]

Do any of the satellite based tivos also have an analog tuner?

Nope. Just satellite tuners.

[phat_bastard]

Now I feel like my hacked SD tivo is so five minutes ago.

More like two years ago, but join the crowd... I'd love to have an HR10-250 but this late in the game it seems silly to invest in something that will be so short lived. That and I'd really miss MRV and tivoserver. :(

[HDTVFanAtic]

70 - HBO-HD
http://img488.imageshack.us/img488/9133/70hbohdeast1280x1088iqm4.th.jpg (http://img488.imageshack.us/my.php?image=70hbohdeast1280x1088iqm4.jpg)

71 - SHO-HD
http://img172.imageshack.us/img172/145/71showtimehdeast1280x10ed7.th.jpg (http://img172.imageshack.us/my.php?image=71showtimehdeast1280x10ed7.jpg)

72 - ESPN2-HD
http://img466.imageshack.us/img466/5661/72espn2hd720x1280psh2.th.jpg (http://img466.imageshack.us/my.php?image=72espn2hd720x1280psh2.jpg)

73 - ESPN-HD
http://img466.imageshack.us/img466/9289/73espnhd720x1280pca2.th.jpg (http://img466.imageshack.us/my.php?image=73espnhd720x1280pca2.jpg)

74 - UHD
http://img172.imageshack.us/img172/7388/74uhd1280x1088iek3.th.jpg (http://img172.imageshack.us/my.php?image=74uhd1280x1088iek3.jpg)

75 - TNT-HD
http://img172.imageshack.us/img172/5428/75tnthd1280x1088iib5.th.jpg (http://img172.imageshack.us/my.php?image=75tnthd1280x1088iib5.jpg)

76- DHDT
http://img466.imageshack.us/img466/2824/76discoveryhdt1920x1088cu4.th.jpg (http://img466.imageshack.us/my.php?image=76discoveryhdt1920x1088cu4.jpg)

78 - HDNET MOVIES
http://img172.imageshack.us/img172/6919/78hdnetmovies1280x1088itk1.th.jpg (http://img172.imageshack.us/my.php?image=78hdnetmovies1280x1088itk1.jpg)

79 - HDNET
http://img488.imageshack.us/img488/4949/79hdnet1280x1088iyu1.th.jpg (http://img488.imageshack.us/my.php?image=79hdnet1280x1088iyu1.jpg)

[HDTVFanAtic]

80 - KCBS-DT
http://img488.imageshack.us/img488/5092/80kcbsdt1280x1088iok7.th.jpg (http://img488.imageshack.us/my.php?image=80kcbsdt1280x1088iok7.jpg)

81 - WCBS-DT
http://img264.imageshack.us/img264/6900/81wcbsdt1280x1088iqy2.th.jpg (http://img264.imageshack.us/my.php?image=81wcbsdt1280x1088iqy2.jpg)

82 - KNBC-DT
http://img466.imageshack.us/img466/972/82knbcdt1280x1088ihj3.th.jpg (http://img466.imageshack.us/my.php?image=82knbcdt1280x1088ihj3.jpg)

83 - WNBC-DT
http://img264.imageshack.us/img264/7813/83wnbcdt1280x1088ive3.th.jpg (http://img264.imageshack.us/my.php?image=83wnbcdt1280x1088ive3.jpg)

86 - WABC-DT
http://img264.imageshack.us/img264/4994/86wabcdt1280x1080ptw9.th.jpg (http://img264.imageshack.us/my.php?image=86wabcdt1280x1080ptw9.jpg)

87 - KABC-DT
http://img488.imageshack.us/img488/1676/87kabcdt1280x1080pud1.th.jpg (http://img488.imageshack.us/my.php?image=87kabcdt1280x1080pud1.jpg)

88 - WYNY-DT
http://img488.imageshack.us/img488/7969/88wnywdt1280x720ptf9.th.jpg (http://img488.imageshack.us/my.php?image=88wnywdt1280x720ptf9.jpg)

89 - KTTV-DT
http://img172.imageshack.us/img172/9872/89kttvdt1280x720pyw1.th.jpg (http://img172.imageshack.us/my.php?image=89kttvdt1280x720pyw1.jpg)

100 - PPV Previews and Most other SD Channels
http://img488.imageshack.us/img488/1964/100ppvpreviews480x480idi6.th.jpg (http://img488.imageshack.us/my.php?image=100ppvpreviews480x480idi6.jpg)

[HDTVFanAtic]

730 - BHD1
http://img466.imageshack.us/img466/2275/730bhd11920x1088iax6.th.jpg (http://img466.imageshack.us/my.php?image=730bhd11920x1088iax6.jpg)

731 - BHD2 (Yes, they appear to be converting 1280x720p to 1920x1080i)
http://img466.imageshack.us/img466/7402/731bhd21920x1088ifw0.th.jpg (http://img466.imageshack.us/my.php?image=731bhd21920x1088ifw0.jpg)

732 - Baseball Mix - the only SD Channel I've observed at 540x480
http://img172.imageshack.us/img172/1126/732bmlx540x480ijpgjd2.th.jpg (http://img172.imageshack.us/my.php?image=732bmlx540x480ijpgjd2.jpg)

733 - Strike Zone
http://img172.imageshack.us/img172/9128/733bsrz480x480icy6.th.jpg (http://img172.imageshack.us/my.php?image=733bsrz480x480icy6.jpg)

http://img172.imageshack.us/img172/9126/735480x480idc8.th.jpg (http://img172.imageshack.us/my.php?image=735480x480idc8.jpg)

[HDTVFanAtic]

Pretty sad state of affairs when Directv is blowing away Dish on DiscoveryHD Theater

1920x1080i on DirecTV
1440x1080i on Dish

13.25 Mbps Video Bitrate on DirecTV
8.22 Mbps Video Bitrate on Dish

I-Frame 61,648 bytes on DirecTV
I-Frame 40,028 bytes on Dish

Both flunk putting 2 B frames back to back - but Directv still has 50% more Video Bitrate and higher resolution than Dish.

http://img507.imageshack.us/img507/3030/directvdhdtig0.jpg (http://imageshack.us)

http://img103.imageshack.us/img103/2344/dishdhdtyv9.jpg (http://imageshack.us)

[dburckh]

I understand PES streams pretty well, but I'm pretty weak on MPEG2 encoding. What's wrong with putting 2 B frames back to back? Is that a spec. violation, or is it just because it puts a lot of stress on the decoder?

[HDTVFanAtic]

I understand PES streams pretty well, but I'm pretty weak on MPEG2 encoding. What's wrong with putting 2 B frames back to back? Is that a spec. violation, or is it just because it puts a lot of stress on the decoder?

If you had GOP of variable length that analyzed the picture and only used what was needed at the time, it wouldn't be bad at all.

However, when you use 15 Frames per GOP - or, god forbid, 30 frames per GOP (as Directv has used in the past on other HD-LITE channels) with only 1 I-Frame in a set length GOP, it will look like crap with any type of motion.

Remember that you cannot use B Frames as a basis for another frame, where P Frames can be used as the basis for another frame.

B frames throw away part of your bitrate as no other frames can be based on them. And the more B frames are in use, obviously the further each P frame is from the others, which makes the P frame quality worse.

So then as the dog chases it's tail...and you carry this out to it's natural conclusion, since the B Frames are based on the surrounding P frames, the B frames will end up looking worse because there are more B frames in between the P frames.

It would be better to use IBPBPBPBPBPBP etc. instead of IBBPBBPBBPBBPBBP.

[dburckh]

Cool. I did not know that B frames could not be referenced by other frames. I do know that DTV normally uses a GOP size of 30ish with doubled up B frames. Here's an example of a 29 frame GOP. I was using it to try to figure out why the PS3 won't play my extracted Tivo video.

Thanks for info!

FYI:

It appears the PS3 chokes on the when frame rate varies at end of the GOP. The sequence header states a frame rate of 30000/1001 fps, but there are only 29 fps. DTV adds two 1.5 frame intervals in the last few millis on the GOP to make the timing "right". I've seen it drop down as low as 24 fps in a sequence that's supposed to be 30000/1001 fps.

I only put in the last few frame types (note the 2 B's in a row), but everything else is mapped.

[HDTVFanAtic]

Cool. I did not know that B frames could not be referenced by other frames. I do know that DTV normally uses a GOP size of 30ish with doubled up B frames. Here's an example of a 29 frame GOP. I was using it to try to figure out why the PS3 won't play my extracted Tivo video.

Thanks for info!

FYI:

It appears the PS3 chokes on the when frame rate varies at end of the GOP. The sequence header states a frame rate of 30000/1001 fps, but there are only 29 fps. DTV adds two 1.5 frame intervals in the last few millis on the GOP to make the timing "right". I've seen it drop down as low as 24 fps in a sequence that's supposed to be 30000/1001 fps.

I only put in the last few frame types (note the 2 B's in a row), but everything else is mapped.

I've never seen a 29 frame GOP on D* -which channel? For 24 frames I suspect you aren't seeing the telecine flag, so only 24 frames are transmitted.

[dburckh]

This was USA.

DTV uses a variable frame rate in most streams. Even though the sequence says it's 30000/1001, it's usually between 24-29.97 fps.

If you use the analyze.cmd file in TySuiteJ, it will dump any PES stream info including the sequence, GOP and picture information. One of the things it dumps is the temporal reference, so doing a max in your head should give you the GOP size. It's pretty verbose, so you'll probably want to redirect it to a file.

[HDTVFanAtic]

This was USA.

DTV uses a variable frame rate in most streams. Even though the sequence says it's 30000/1001, it's usually between 24-29.97 fps.

If you use the analyze.cmd file in TySuiteJ, it will dump any PES stream info including the sequence, GOP and picture information. One of the things it dumps is the temporal reference, so doing a max in your head should give you the GOP size. It's pretty verbose, so you'll probably want to redirect it to a file.

I really don't look at the SD streams, so could be.

The HD Streams have been 30 frames per GOP everytime I have checked unless its telecined.

[FredThompson]

Variable FRAME rate? Methinks you are confusing field repetition with VFR. DTV does sometimes use telecine for movie channels.

[dburckh]

I did a little quick reading. I don't think this is field repetition. The the sequence in the spreadsheet attached above the frame rate is declared as 30000/1001. The sequence lasts 1.001 secs and there are 29 pictures. Shouldn't there be 30?

If I look around, I know I can find examples that are less than that and aren't declared as 24 fps. Unfortunately, documenting this is a PITA.

Here's what I'm doing to come up with this SpreadSheet. See if you can explain what I'm doing wrong.

TySuiteJ creates a new Video packet (0x00 00 E0 00) for each Picture (0x00 00 01 00). On the HR10-250 these all happen to have a PTS. I record that PTS and the temporal reference in the spreadsheet attached.

Where I'm coming up with variable frame rate is that the PTSes are not evenly spaced. It appears that DTV has stretched the pictures to fit 29 where there should be 30. Most PTSes have the correct 3003 clock ticks (aka 1001/30000s or 33ms) between them, but a few have 4504 clock ticks between them. Further, I've seen plenty of examples where there are only 1502 clock ticks between them. If this was telecined, shouldn't the pictures be evenly spaced at 3003 clock ticks?

Am I missing something here?

I'll look as some DVD video after work and verify or disprove my "evenly spaced" assumption.

[FredThompson]

I don't know about HD, I'm only using SD. I guess it's possible as a way to save bandwidth but it's a playback nightmare from a coding/design standpoint.

[dburckh]

First, thank you for confirming that I'm not insane. :)

The PS3 Media Center is completely choking on it. I tried to aligning the picture's PTS with their temporal reference. That fixed the odd flashing and jump backs, but left black holes in the video. Next I'm going to try to align the temporal references with the PTS, but that's more complicated. Hopefully the PS3 will just interpret this as dropped video and I'll be good to go.

This was SD recorded off of USA. I see this all the time though.

[FredThompson]

Whenever possible, use TyTool for your editing. Josh put a lot of work into very low level aspects of how the field-accurate splicing works and getting the streams to work in hardware players. It works at the byte level, for lack of a long explanation. I've worked on thousands of SD DTiVo streams over the past few years using TyTool. A very small percentage have hiccups and we've traced those to corruption in the stream itself. IIRC, a few of those were backwards-referencing pointers in the stream which are very rare. The time counter can also have multiple resets. Josh said he solved that but it's not in the latest public release. He seems to have "disappeared" again. I know he had changed jobs and moved and maybeo something like that has happened again.

[dburckh]

Unfortunately, TyTool doesn't work well with my HR10-250. I think it does still work with some (???) SD streams. I can give it a shot.

Primarily I'm trying to get video Tivo video to play on my PS3. Unless Josh is completely recoding the video stream, I doubt these issues will be addressed. I've tried TyToMpg and it has the same issues I (TySuiteJ) do.

I have been able to get video to play be recoding the video through ffmpeg. Unfortunately this is lossy and bloats the size.

[bcc]

Unfortunately, TyTool doesn't work well with my HR10-250. I think it does still work with some (???) SD streams. I can give it a shot.

Primarily I'm trying to get video Tivo video to play on my PS3. Unless Josh is completely recoding the video stream, I doubt these issues will be addressed. I've tried TyToMpg and it has the same issues I (TySuiteJ) do.
I have no outstanding bug reports for tytompg, actually. When you say *it* has an issue I assume you really mean the PS3 has an issue with DTV/OTA fox recordings? What troubleshooting has been performed? Have you narrowed this down to the likely problem (the ps3 not handling video streams with variable number of fields per frame)?

[dburckh]

The "it" I was referring to was TyTool and HR10-250s. I have had little success with the HR10-250 and direct TyTool Mpeg conversion. Due to this, I've never tried to edit video in TyTool. This is a known issue.

The video issue is totally a PS3 issue. It has issues with most DTV MPG streams. It won't play any streams muxed with VideoRedo or TyToMpeg.

DTV streams muxed (or remuxed) with TySuiteJ or ffmpeg will play, but they "jump" and flash green when the field rate varies from the frame rate.

FYI: I've muxed with TyToMpeg then remuxed with ffmpeg and the PS3 will play them, but they jump like the TySuiteJ ones.

[bcc]

I think it's well known that tytool has all along had problems with HD streams, and even more so with recent tivo software releases (now that tivo changed their timestamp processing).

You did say "I've tried TyToMPG and it has the same issues". On that I have to step in and say tytompg is by design passing along the video stream unchanged. If the PS3 is having problems with the fields per frame within the GOPs it is not a tytompg issue. Videoredo on the other hand was designed to rebuild the GOPs to be more compatible with programs that presume the DVD subset of the mpeg2 spec. I think it'd be feasible for an application such as videoredo to fix the fields per frame to be DVD-compliant without re-transcoding the stream. Another videoredo-like application could do it if videoredo won't. On the other hand it might just be that the PS3 is not worth using as an HD media player. Ie not worth fixing.
FYI: I've muxed with TyToMpeg then remuxed with ffmpeg and the PS3 will play them, but they jump like the TySuiteJ ones.That sounds like a symptom of ffmpeg rebuilding the timestamps such that they are incorrect. See the old discussion I had on this issue that I fixed for tymplex/ty-ffmpeg (phat_bastard's sample).

[dburckh]

Ok, by your post you missing the point. Let me summarize:

PS3 won't play anything muxed by VideoRedo or TyToMpeg. HD, SD doesn't matter. TyToMpeg is fine with other players, as is VideoRedo. There is no defect that I'm aware of. This is not a commentary on it or an attack on either tool in anyway. This is a statement of fact.

TySuiteJ, Project X DVB, ffmpeg are the only things that mux video that will play on the PS3, but they have issues. Both HD and SD are somewhat playable. I believe the issues are DTV varying from the declared frame rate. This appears to be a PS3 issue not a TyToMpeg issue or TyTool issue.

The only thing that remains to for me to try is TyTool, which doesn't work on (some) SD or HD from the HR10-250.

What I'm trying to do is tweak the PTS and temporal references to work around the PS3 issues so I can use it as a media center. Does that make sense?

Anyway, this thread is on it's way to Cuba (I'll take the blame for that). :) I'll start another thread.

[bcc]

Ok, by your post you missing the point. Let me summarize:Ok, your previous posts made points that muddied that waters and I think I clarified things (and pointed you to where I resolved similar sounding problems in the past).What I'm trying to do is tweak the PTS and temporal references to work around the PS3 issues so I can use it as a media center. Does that make sense?If the PS3 can't handle the mpeg2 compliant (but different than DVD) rate of fields per frame, then I don't think fudging the timestamps will fix things. I'd expect such a change to trigger underrun/overruns in the virtual buffer. Depending upon how the PS3 handles such under/over runs, best case would likely be to get the PS3 to play the content with jerky frames (like the tymplex case with FOX OTA before I fixed it). I am just speculating because I don't have a PS3 to see what is going wrong.