Page 1 of 11 123 ... LastLast
Results 1 to 15 of 151

Thread: ExtractStream and PlayStream failing on every sector in a DTivo

  1. #1
    Join Date
    Dec 2001
    Location
    Lafayette, California
    Posts
    47

    35 ExtractStream and PlayStream failing on every sector in a DTivo

    I am using a 3-week old Hughes DTivo box, running Tivo 2.5.1, bash enabled on the serial port, TivoNet and TivoWeb installed, no space upgrade.

    I have tried both ExtractStream (Nov 9, 2001 version from http://home.earthlink.net/~garyw90/TivoApp.html) and PlayStream 0.1b (Jan 14, 2001) and they both complain about not finding the FSID in the sector(s).

    I have confirmed that I am using the correct FSID (119309) three ways. I tried:
    1) Browsing the MFS using TivoWeb (MFS -> Recording -> Now Showing -> pick the show from the list of shows -> find the “Part” line and click on its link -> find the “File” line and read the FSID from it.
    2) Manually repeating #1 using tivosh, mls /Recording/Complete, mls /Recording/NowShowing, dumpobj to find the “Part” definition, and then dumpobj to find the “File” definition (takes a few tries because my DTivo reboots after each dumpobj).
    3) Running the TivoApp “ShowList.tcl” utility, reading the FSID from the <Part> line.
    4) Running TivoApp on my PC and watching it telnet over to the DTivo to run ShowList.tcl, grab the FSID, and watching it attempt ‘ExtractStream –s 119309’.

    In all 4 cases, the FSID came out as 119309. Some examples of the fail case:
    cd /var/hack/bin
    ./ExtractStream 119309
    Attempting to locate tyStream with fsid 0x1d20d...
    Sector FSID passed (0x1d20d) failed to match the FSID value in the sector (0x0).
    Trying next block
    Sector FSID passed (0x1d20d) failed to match the FSID value in the sector (0x0).
    Trying next block
    Sector FSID passed (0x1d20d) failed to match the FSID value in the sector (0x0).
    Trying next block
    Sector FSID passed (0x1d20d) failed to match the FSID value in the sector (0x4df
    ). Trying next block
    Sector FSID passed (0x1d20d) failed to match the FSID value in the sector (0x4df
    ). Trying next block
    ...etc...

    cd /var/hack/playstream
    ./PlayStream 119309
    Attempting to locate tyStream with fsid 119309...
    Sector FSID passed (1d20d) failed to match the FSID value in the sector (0).
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    00 00 00 00 00 00 00 00 00 00 00 00 91 23 1e bc .............#..
    a9 dd 19 85 00 00 00 00 00 00 00 00 00 00 00 00 ................
    ...etc...

    Now, I have searched several online forums and have not yet seen anyone stuck at this point. I have read that ExtractStream only works 5% of the time on DTivos, and that in some cases it will only Extract the beginning portion of a show because it runs into sectors that don’t match the FSID. I specifically have delayed any sort of space upgrade on this DTivo because I understand there is a concern that ExtractStream has trouble with expanded drives and/or secondary drives.

    If this is an unsolved problem I'd be happy to dig into it deeper to see if I can fix this, but I first wanted to see if I am just using an old version or using these tools incorrectly.

    Any suggestions?
    Thanks
    Eric

  2. #2
    Join Date
    Dec 2001
    Posts
    18
    You are correct. There are presently no solutions to this problem. Most recent theory that I recall is that since the DTivo records the actual satellite feed and is capable of recording the 5.1 audio stream, this file format differs from that on the standalone, specifically that the mpeg2 frames are variable-sized, and extractstream isn't capable of handling variable sized frames, thus sector id's aren't uniform-sized and extractstream's code not built to handle this fact.

    or, i could be completely off my chump.

    people were waiting to dig into this until the 2.5 problems were fixed. they're fixed now, so it'd be a great time to tackle this issue too i'll be first in line to help test anything you come up with!

  3. #3
    Join Date
    Dec 2001
    Location
    Lafayette, California
    Posts
    47

    28 How Hard Can It Be? :)

    I was afraid that no one had this working on the DTivo boxes (maybe I shouldn't sell my standalone Tivo on EBay after all...). I really want to make this work, but to get a sense of how deep this problem is, here are my worst fears:

    1) The FSID does not really point to the a/v streams. There is some other info hidden in the same structures that points to the a/v streams.

    2) There is no simple formula to convert the FSID into a sector #. For example, simple or hard encryption could be used.

    (We could dismiss #1 and 2 if we could disassemble the playback code or wipe out the sector that the FSID points to and see if the DTivo can still play back that show.)

    3) The a/v streams are encrypted in some way. (Once we can examine a stream, it won't be hard to see if it is valid MPEG or some encrypted mess.)

    4) The a/v streams are not MPEG2. In fact, I thought I had read that D* does not use MPEG2, they use some custom version of MPEG1. If that is true, we should still be able to identify the a/v streams and find the frames, but then we might need some sort of MPEG1->2 conversion before these streams were playable on a PC. I guess reading the CS22 spec closely would help here.

    If anyone can dismiss any or all of these issues, then great, we are on our way! Otherwise, this is going to take a serious effort.

    I've heard other people that share your concern that because variable bit rate (VBR) encoding is used on the MPEG data, the FSID or sector sequencing is weird. I'm not sure that I agree. My understanding is that VBR would just mean that the I-frames, B-frames, and P-frames are different sizes. But there would still be a valid MPEG1 or 2 structure to the data, allowing you to find the start of an I/B/P frame, and to find the start of the next. I'm not sure that there is a relationship between frames and sectors. I would assume that there is one layer of code (or an ASIC with access to the drive) in the Tivo that knows how to extract a byte stream from a sequence of sectors, and another layer (the MPEG chip set) which knows how to find I/B/P frames in such a byte stream.

    CAN ANYONE SHED ANY LIGHT ON THESE QUESTIONS?

    Eric

  4. #4
    Join Date
    Dec 2001
    Posts
    18
    eric:

    i'm fairly convinced that encryption is not to blame here, as is evident by the occasional successful extraction of data (i get about 10% success rate here).

  5. #5
    Join Date
    Sep 2001
    Posts
    889
    FSID (I've always assumed FileSystemID) probably a pointer to a database location/address)

    there IS no simple calculation converting fsid to sector, mfs is a database and as such has no discrete element size.

    the stream is modified mpeg1 400x400 iirc, as it is not created by tivo sw, it does not contain all of the specific clues that the SA tivo has involving what to do with the data.

    if you're just interested in capturing the mpeg stream, why not replace the /dev/mpgXX files with pointers to your target files?

  6. #6
    Join Date
    Oct 2001
    Posts
    36
    Originally posted by BubbaJ
    the stream is modified mpeg1 400x400 iirc, as it is not created by tivo sw,
    If its mpeg1 why do they get more hours per GB than the mpeg2 streams on
    the SA? I know the DTV VBR is supposed to be more agressive than the SA
    VBR, but mpeg2 is inherently better compression than mpeg1.

  7. #7
    Join Date
    Dec 2001
    Posts
    37
    It's probably that DirecTivo is recording a DSS transport stream. This transport stream was created before the MPEG2 TS stream spec was finalized. It is 147 bytes instead of 188, doesn't have a 0x47 sync byte at 188 byte intervals, has a 3 byte link header, 127 bytes of payload, and a 17 byte error correcting suffix.

    The specs to the MPEG2 transport stream (used in the U.S. ATSC over the air broadcast spec, the DVB transmission spec for over the air broadcasting in Europe and elsewhere, and for Dish network) uses a 188 byte packet in which the elementary streams of a normal MPEG2 file are multiplexed. Two other types of packets come along every 1 and 4 milliseconds which give you the program info and other data. To convert a TS stream to a normal MPEG file, you have to grab the 184 bytes of payload out of the transport stream packets and put them into a normal MPEG file (not too hard).

    I haven't seen a DirecTivo block, but you could probably find out if it is an MPEG elementary stream

    The DSS stream format that DirecTV uses is 147 bytes, and doesn't really look much like an MPEG 2 TS packet. However, it's similar enough to be able to decode and put back together into an MPEG file. All channels in a given transponder are multiplexed in one stream, and each is given it's own id which should be at the beginning of the packet. I would assume that DirecTivo demuxes the packets before recording the program (or it would take up a huge amount of space), but I'm not sure it demuxes the DSS packets back into the mpeg2 PES elementary streams.

    To demux a transport stream, it's pretty easy. You just go along looking at the link headers of all the packets until you find one that is marked as a 'random access' start packet. (You obviously have to know the program id of the packets you're looking for though.. you have to decode the program info packets to find the packet id map).

    That packet will be the start of a video frame or an audio slice. The video and audio packets will start with system headers which describe the video/audio formats, and then just have the data in the payload parts of the packets. The last transport packet will be padded. Once you extract the payload's fromt the 188 (or 147 byte for DSS) payload packets, you have a raw PES MPEG stream of video and audio data. You can then remerge the streams back together and add a pack and a program header to get a standard MPEG2 file, or you can use some sort of remuxing program to do this.

    DirectTV uses a preliminary version of MPEG2 for it's normal channels (I don't think it uses B frames, though I'm not sure). It uses MPEG1 Layer 2 for audio streams. For HD it uses standard MPEG2 and AC3. I would assume that HD channels are encoded the same as normal channels, though I don't know if the DirecTivo can be made to record them. I would be very interested to know if the DirecTivo files are encrypted or not. You can tell if they are encrypted by looking for the MPEG2 PES block headers and see if they have a 00 00 01 format.

    You will probably want to take a look at these links.

    http://www.coolstf.com/mpeg/#dss

    http://fiddle.visc.vt.edu/courses/ee...u_scoglio.html

    http://swpat.ffii.org/vreji/pikta/tx.../306/desc.html

    It would be very interesting if DirecTV HD were to be extractable from a DirecTivo.

  8. #8
    Join Date
    Jun 2001
    Location
    Dallas
    Posts
    588
    Okay great, now it seems we have someone who knows what they're talking about here. At least when it comes to mpeg streams =]

    I'm pretty sure we can make the DTivo record from one of the HDTV streams. You can manually tune to that channel and I think it also may be possible to manually record. Need to investigate that further.

    As far as getting the data to a point where we can use it, I've read a few posts on ptvupgrade where people with DTivo's are able to extract the video with extractstream, but have audio sync issues. I think the first step to de-mux ing the streams correctly and then remuxing is to get raw dump of the tystream. So someone that is able to dump streams with extractstream (mine won't) needs to do a raw dump. Once we get the raw dump, I can host it somewhere so that it can be downloaded and worked on.
    Information wants to be free....

  9. #9
    Join Date
    Dec 2001
    Posts
    11
    I don't have the process working, but have found some interesting information that might be useful.

    Check out the following post in the ExtractStream Group:
    http://groups.yahoo.com/group/ExtractStream/message/270

    Basically, you extract a raw tyStream with ExtractStream, tweak ConvertStream to ignore the checksum, and use the new ConvertStream to split the tyStream into an m2a & m2v files. This gets you a playable video file, but audio file is messed up.

    Here is some information on MPEG audio headers:
    http://groups.google.com/groups?hl=e...ni-mb.si#link3

    frame size = 144 * bit rate / sample freq + pad bit

    The DTivo audio frames are different than the SA's. According the ExtractStream source, the audio frame headers for the SA TiVo start with 0xFFFDA8 (frame size 0x360).

    Looking at a tyStream from a DTivo, the first chunk contained headers starting with 0xFFFD94 (frame size 0x1E0, 48K, 160kbps), but many of the other chunks have audio headers starting with 0xFFFDA9 (frame size 0x240, 48K, 192kbps) with an occasional 0xFFFD94. There is also a PES header at the start of each audio record starting with 0x0001C0XXXXYYYYYYYYYY (where XXXX is the frame length and Y’s appear to be counting). To make the audio file playable in WinAmp, the PES headers need to be stripped.


    Mithrandir
    Last edited by mithrandir; 01-02-2002 at 02:10 AM.

  10. #10
    Join Date
    Jun 2001
    Location
    Dallas
    Posts
    588
    I'm pretty sure that the PES headers are the key to getting the audio to sync correctly.
    Information wants to be free....

  11. #11
    Join Date
    Oct 2001
    Posts
    36
    Originally posted by mithrandir
    I don't have the process working, but have found some interesting information that might be useful.

    Check out the following post in the ExtractStream Group:
    http://groups.yahoo.com/group/ExtractStream/message/270

    Mithrandir
    The yahoo message is from June, which would be prior to 2.5. I didn't see
    anything recent in that group on the Direct Tivo, but my search may have been
    flawed.

    Forgetting about audio, has anyone extracted a useable video stream recorded on 2.5?

  12. #12
    Join Date
    Nov 2001
    Posts
    17
    I'm really suprised that no one has mentioned this yet.

    Post-2.5 recordings on the DirecTivo are encrypted. Old recordings are plaintext, but new recordings are ciphertext. This applies to all boxes with "MediaSwitch2" or later. I would bet that any new rev of the standalone box will also include this video-stream encryption.

    So the PROM hack and the initrd hack together with extractstream won't get you anywhere on a v2.5 DirecTivo. Unless, of course, you break the encryption ...

    Besides, extractstream doesn't create an MPEG2 stream from the recorded data. What you really want is a program to preserve the PTS information on the audio and video segments while creating a proper MPEG2 system layer.

    To correct a few other comments in this thread:

    The DSS transport format versus the MPEG2 transport format is irrelevant to getting video from the Tivo. What you see on the disk (e.g., extractstream) comes from the STi5505 which has already repackaged the data and formatted it for "trick-play".

    There are lots of B-frames in the DSS MPEG2 stream (>60%). In fact, there are suprisingly few I frames (about 1/second). Of course, it depends on the source. But, boy, those B-frames are small.

    There really isn't that much difference between MPEG1 and MPEG2 with respect to encoding NTSC. MPEG1 is progressive only, MPEG2 allows for interlace, higher bit-rates and higher resolution. But at 480 x 480, 4:3, 29.97 fps, MPEG1 is good enough. Don't confuse MPEG1 with the particular choice of MPEG1 parameters which yields VCD -- which is a low-res format designed to fit one hour of video on a CD. The stream on the DirectTivo disk is pure MPEG2 main profile at main level, with system header extensions, picture coding extensions, 4:2:0 chroma, interlacing, and all.
    Last edited by PPCHacker; 01-01-2002 at 06:09 PM.

  13. #13
    Join Date
    Nov 2001
    Posts
    117

    29

    This is one of the best threads on the subject so far. It looks like if the previous post is correct extractstream would work for the DTivo users. Maybe there is a post on this but I'm wondering if it's time to write an app that can "snif" the data flying out of the video/audio out ports. At that point everything is back in the open. The only con to this is it might be just as easy to attatch a PC with a Video capture device and grap the program. Have we come full circle yet ?

    -----------
    Do under other as you would have then do unto you. You never know if you'll have the change to get the upper hand again like this again.
    Did I do that...

  14. #14
    Join Date
    Nov 2001
    Posts
    2

    Smile

    I have been a DirecTV subscriber since day 1...literally. the original boxes and transmissions were MPEG-1...a few months later, they switched over to MPEG-2 through a field-upgrade that arrived from the satellites, and everything went over to MPEG-2, so they said.

    As for the current extractstream problems, remember that FSID by FSID, there are successes when running ExtractStream. this would strongly suggest that the content data is MPEG-2. It seems to me that it's a simple inability to resolve the fs locations of the FSIDs....

    a buddy of mine and i are of the mind that extract stream may be unaware of additional fs mounts or partitions and simply fails to resolve those FSIDs which map to unknown or unaccounted for partitions.

    From a biz standpoint, since TiVo, the company, is small and resource-constrained, it seems that they'd be best served by getting any data into a single format across the board that their hardware could decode and play. ...also, the fact that DirecTiVos hit the market fairly quickly, corroborates the single-content-format argument.

  15. #15
    Join Date
    Jun 2001
    Location
    Dallas
    Posts
    588
    Originally posted by PPCHacker
    Post-2.5 recordings on the DirecTivo are encrypted. Old recordings are plaintext, but new recordings are ciphertext. This applies to all boxes with "MediaSwitch2" or later. I would bet that any new rev of the standalone box will also include this video-stream encryption.
    [/B]
    Well you definatly seem to know what your talking about, but I don't think that the actual streams could be encrypted on the fly by the DTivo. Not enough horsepower and no dedicated encryption chip capable of doing it.

    Could you explain further on how you cam to the conclusion that the streams were encrypted? Any supporting evidence that we could look at to help shed more light on the issue so that a solution can be worked out?
    Information wants to be free....

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •