PDA

View Full Version : Took a quick look at a Series2 TyStream...



jdiner
11-12-2002, 03:43 PM
As expected by many and feared by others Tivo has changed the format of the TyStream. But only slightly. I did not dig to deep but here is what I found out:

1- The biggest difference appears to be that there is now no longer any 16 byte Tivo Video header blocks. Instead the buried PES header starts the begining of each video frame.

This is very similar to what happened to audio in the DTivo with respect to the SA Tivo style.

2- There are now even more of the "who know what they are for" packet types. Many of the fixes I have employed to get other things working right will no longer work as a result of what I observed in the data itself.

3- Most of what I have seen in there means that splitting and mux'ing will actually be quite a bit easier. As some of the wierdness that is so hard to counteract is gone.

Anyway, just a couple of thoughts for those that are interested.

--jdiner

olaf_sc
11-12-2002, 04:40 PM
Hello,

Nice findings,


Originally posted by jdiner
As expected by many and feared by others Tivo has changed the format of the TyStream. But only slightly. I did not dig to deep but here is what I found out:

1- The biggest difference appears to be that there is now no longer any 16 byte Tivo Video header blocks. Instead the buried PES header starts the begining of each video frame.

This is very similar to what happened to audio in the DTivo with respect to the SA Tivo style.

Is the video PES header embeded in such way that you don't know when it start or is it so that it will always start at the first byte in the record.

If it doesn't start at the first byte I guess it will be very simiar to audio extracting from a DTivo.

If it does start on the first byte (byte 0) in the record it will be even easier than extracting audio from a DTivo since there is no need to search for the start byte of the PES header.



2- There are now even more of the "who know what they are for" packet types. Many of the fixes I have employed to get other things working right will no longer work as a result of what I observed in the data itself.


I'm qurious about those unknow packets - I mean there is a quite good list at the alt.org forum listing the packet types. So far I have only encounterd unknows packet types in Chunks that are way out of sync (OOB packet as you call them).

If those new unknown packets are present in valid chunks whouldn't it be very possible that they are for new features that Tivo will introduce. E.g. maybe it's upcomming things for e.g HDTV recording and tivo need those special packtes to store information needed for e.g. decoding.

Well, just my 0.02 cent thoughts about it - ordered a Series 2 so I will have in a week or two - will start my probe into the issue then.

Cheers Olaf

PS: Trying to give constructive thoughts and remarks

keith721
11-12-2002, 05:53 PM
. . wants to know:

how'd ya get a Series2 ty stream off the box? was it the recent Series2 partition hacks noted at TC.com, removing the drive from the TiVo S2, or did someone else do the hacks, and provide the stream?

thanks!
;)

AlphaWolf
11-12-2002, 07:44 PM
You can probably easily extract it from a PC using mfs_export. I used to do it with s1 tivos that way before I got a turbonet.

The question is though, what, if any, measures have they taken to scramble the video?

jdiner
11-13-2002, 01:03 AM
Originally posted by keith721
. . wants to know:

how'd ya get a Series2 ty stream off the box? was it the recent Series2 partition hacks noted at TC.com, removing the drive from the TiVo S2, or did someone else do the hacks, and provide the stream?

thanks!
;)

Someone else got it and contacted me through email. Don't know if they are even on the forum.

They ftp'ed it to me.

But what he told me was that he was using the BASH_ENV in the BOOT_PARAMs to get things to run differently. But he did point out that Tivo has apparantly already closed that hole for the Series 2 in the next version of that OS... I have no more information than that.

--jdiner

jdiner
11-13-2002, 01:10 AM
Originally posted by olaf_sc
Is the video PES header embeded in such way that you don't know when it start or is it so that it will always start at the first byte in the record.


In the 1 single short clip I looked at it was the first byte everytime. Which is why I said it would be easier. That cycling 1-3 byte lead-in for the audio on the Dtivo is a pain to work around when trying to do byte accurate count across frames.

I can't garantee it will stay that way but it looks good in this one.



I'm qurious about those unknow packets - I mean there is a quite good list at the alt.org forum listing the packet types. So far I have only encounterd unknows packet types in Chunks that are way out of sync (OOB packet as you call them).

Ummm. Not sure what is on alt.org. But there are a number of packet types. 0xc0 being layerII audio, 0xe0 being video, and if you have the wrong beta of a DTivo something seeming random for Dolby audio, if not then dolby audio in the TyStream is indicated by an 0xc0 as well.

Then there is 1, 2, 3, 4, 5 and 6. All of which are different things. Some are closed captioning, some are the "press thumbs up to record" that shows on the screen at various times.

But in the 16 byte Tivo header for each reacord in a chunk, there are a myriad of flags. And on the series 2 these flags appear to be radically different. Or at the various least they are is a ton of new flags and that changes how you have to deal with them.

The largest impact from this will probably be the UK tivo as that machines makes more extensive use of them than either of the US tivo models but the series2 is a whole new ballgame.

--jdiner

olaf_sc
11-13-2002, 03:13 AM
Originally posted by jdiner
In the 1 single short clip I looked at it was the first byte everytime. Which is why I said it would be easier. ]

Yes, it sure will - I guess it's like the DTivo Video PES header then.



That cycling 1-3 byte lead-in for the audio on the Dtivo is a pain to work around when trying to do byte accurate count across frames.


Yes, it sure is especially when you have audio pes that has only 4 bytes and the next part of the header is in the next chunk.



I can't garantee it will stay that way but it looks good in this one.


Nope who can but on the other hand why not have a generic code searching for the start of the PES header (or header in general). This algo could easily be shared both for Audio and Video headers.



Ummm. Not sure what is on alt.org. But there are a number of packet types. 0xc0 being layerII audio, 0xe0 being video, and if you have the wrong beta of a DTivo something seeming random for Dolby audio, if not then dolby audio in the TyStream is indicated by an 0xc0 as well.


Well I guess you know but for all other folks on the list:
The record header starts with:
YY YY YX XX

Where AB is a byte - here YY YY Y is the size of the record and X XX is the type audio, video, etc.

The list of types is then (hex):

2c0 == Audio Data
3c0 == MPEG PES header - start
4c0 == Audio Data (SA only??)
9c0 == AC3 PES header - start

2e0 == Video Data
6e0 == Video PES header
7e0 == Video SEQ header
8e0 == Video I frame/field
ae0 == Video P frame/field
be0 == Video B frame/field
ce0 == GOP header
e01 == CC data (closed captions)
e02 == EE ?? Extened Data
e05 == UK teletext Data

So far I have only bumped into one unknown type and it's 000 and it's in OOB or as I call it OOS (out of sync) chunks that are junk/garbage.

the e02 is something I don't really know what it is but maybe you can enlighten us.



Then there is 1, 2, 3, 4, 5 and 6. All of which are different things. Some are closed captioning, some are the "press thumbs up to record" that shows on the screen at various times.


I guess you mean:
1 == e01 == CC data
2 == EE ?? Extened Data do you know what this really is?
5 == e05 == UK teletext Data

but what is 3,4 and 6?



But in the 16 byte Tivo header for each reacord in a chunk, there are a myriad of flags. And on the series 2 these flags appear to be radically different. Or at the various least they are is a ton of new flags and that changes how you have to deal with them.


As far as I know it's only SA Tivo that uses this for PTS times times - (have not yet got my hands on a SA stream - beside it's quite well documented at this site (http://www.30below.com/~zmerch/tivo/tivoheaderdata.cfm))

If I got it right so far with at least DTivo this is how the header is:
XX XX XY YY ZZ ZZ ZZ ZZ TT TT TT TT TT TT TT TT

where X is size, Y is type, Z is unknown and T is a timestamp note the T time stamp is not a PES PTS.

Hence the question is what the Z is there is also a lot of Z in the SA docs in the above mentioned site.

So what you are saying is that the 2 series has implemented yet another set of flags. Which you yet don't know what they mean.



The largest impact from this will probably be the UK tivo as that machines makes more extensive use of them than either of the US tivo models but the series2 is a whole new ballgame.


It sure will if we don't know what they represent - although we don't need to know everything but we need to know the relevant once to be able to extract the data. Although it would be nice to e.g. extract the Closed Caption so we can use that data in a DVD.

Cheers Olaf

PS: Ahh could you exract any Video and if so was it not encrypted.

jdiner
11-13-2002, 03:21 PM
Oh man. I told roger that his Tivo header info was wrong. Crud. I was going to start working with him to get it straightened out. But I got busy and stopped using the groups at yahoo...

Some of the assumption on that page are way off. Which explains some of the things I saw when I looked at your code.

I don't know if you read the old old messages from me but I started from scratch to try and avoid picking those things up.

I will have to find the time to put something together to go through it all for people. Many of the flags and other things are not right.

Hummm....

--jdiner

Ok. No trying to be purposefully vague here. Let me be clear about that. There is just a TON of information I have collected and it would take forever to post it all up 1 thought at a time.

I am trying to think of the best way to get started on all of this. That is what the Hummm.... was all about.

olaf_sc
11-13-2002, 05:45 PM
Well, my aproach has been just to use the type/size information and then look at the real PES time stamps in the video/audio stream it self.

Thats why I don't look at the rest of the header since it not yet known - the 64 bit TT is obviously some sort of stamp since it is increasing at regular interwall - however it's not formed as a PES time stamp at all.

Cheers Olaf


Originally posted by jdiner
Oh man. I told roger that his Tivo header info was wrong. Crud. I was going to start working with him to get it straightened out. But I got busy and stopped using the groups at yahoo...

Some of the assumption on that page are way off. Which explains some of the things I saw when I looked at your code.

I don't know if you read the old old messages from me but I started from scratch to try and avoid picking those things up.

I will have to find the time to put something together to go through it all for people. Many of the flags and other things are not right.

Hummm....

--jdiner

Ok. No trying to be purposefully vague here. Let me be clear about that. There is just a TON of information I have collected and it would take forever to post it all up 1 thought at a time.

I am trying to think of the best way to get started on all of this. That is what the Hummm.... was all about.

jdiner
11-13-2002, 05:55 PM
Many have thought it is a timestamp. And after a fashion it is. Part of it is that is. With extra at the start and the end.

But it has no granularity to it at all. It skips in full seconds at times and in some cases, no idea why, it skips but 1:30. All I can figure from what I have observered is that it could be used for a very rapid fast-foward or rewind position check in the file. Just jump headers until you see what you want...

It is useless in my opinion in terms of using it. The granularity is just too large.

--jdiner

olaf_sc
11-13-2002, 09:29 PM
Originally posted by jdiner
Many have thought it is a timestamp. And after a fashion it is. Part of it is that is. With extra at the start and the end.

But it has no granularity to it at all. It skips in full seconds at times and in some cases, no idea why, it skips but 1:30. All I can figure from what I have observered is that it could be used for a very rapid fast-foward or rewind position check in the file. Just jump headers until you see what you want...

It is useless in my opinion in terms of using it. The granularity is just too large.

--jdiner

Yes, that was kind of my thought too - i.e. that it might be used for ff and rw.

Cheers Olaf

PS: Did you get my mail with answers in regards to how to make a real DVD compliant VOB? Just checking since we had some bad experience with mail dropps.

jdiner
11-17-2002, 01:10 AM
I got the email. I am just now getting back to things.

AS for your questions on the TyStream ask away.

--jdiner

chrsturbo
10-08-2003, 03:39 PM
This is almost on topic. I am trying to fix tystudio to use Series 2 streams and it is mostly breaking in the indexing. It seems it can't find the 0x6e0 Video PES record that is present in a series 1 ty. It finds the 0x8e0 Iframes but never gets the PES so it bombs the chunk as unindexable. Is it a matter of simply looking for the PES in the record differantly or does the scan algo have to be redone?
Apologies for being long on question and short on understanding of the TY.
-Chris
BTW If anyone needs series 2 streams I got tystudio's tyserver running on my DSR7000 and I can make it available for you.