Packet dump needed of MRV transfer

[mikesown]

Can someone with MRV(two Series3/HD, not series2 for the moment) please post or email me a packet dump of the first few seconds(including initialization) of an MRV transfer? I aim to develop an MRV emulator for the PC so that transfers will go much faster than TTG. If you want to email me the dump, email 1337mail [atatatat] gmail [dotdotdot] com.

Thanks,
Mike

[moyekj]

Some packet sniffing prompted by you revealed the following:
MRV transfers can be initiated from a PC very similar to TTG transfer but with the addition of video format specification at the end. For example:
http://192.168.1.101/download/Grey's%20Anatomy.TiVo?Container=%2FNowPlaying&id=1169277&Format=video%2Fx-tivo-raw-tts

Note the format specification at the end: &Format=video%2Fx-tivo-raw-tts
(%20=space %2F=/)

While this is all well and good and avoids having the Tivo have to re-mux and re-encrypt and hence speeds up transfers to MRV speeds, the question is what can be done with the resulting encrypted transport stream file on the PC?

[bcc]

Some packet sniffing prompted by you revealed the following:
MRV transfers can be initiated from a PC very similar to TTG transfer but with the addition of video format specification at the end. For example:
http://192.168.1.101/download/Grey's%20Anatomy.TiVo?Container=%2FNowPlaying&id=1169277&Format=video%2Fx-tivo-raw-tts

Note the format specification at the end: &Format=video%2Fx-tivo-raw-tts
(%20=space %2F=/)

While this is all well and good and avoids having the Tivo have to re-mux and re-encrypt and hence speeds up transfers to MRV speeds, the question is what can be done with the resulting encrypted transport stream file on the PC?Wow, lacking a working MRV setup I had tried to guess the HTML post command that would get x-tivo-raw-tts but hadn't quite manged. See here:
http://dealdatabase.com/forum/showpost.php?p=289628&postcount=46

Now, I took a look at what is extracted when one modifies the HTML POST as you found, and the resulting file is *not* encrypted! In fact it looks a lot like the raw mpeg2 transport stream content that s3tots processes. If you attempt playback with xine you'll see that it can almost play the video&audio streams as-is.

So the idea here:

http://dealdatabase.com/forum/showpost.php?p=290025&postcount=58

was on the money...
Guess it's time to see if s3tots can be made to clean up this new file format.

[bcc]

Now, I took a look at what is extracted when one modifies the HTML POST as you found, and the resulting file is *not* encrypted! In fact it looks a lot like the raw mpeg2 transport stream content that s3tots processes. If you attempt playback with xine you'll see that it can almost play the video&audio streams as-is.Note that I have the no encryption patch in my tivoapp. Maybe someone else could verify whether or not the recordings are scrambled without the tivoapp patch.

[moyekj]

Note that I have the no encryption patch in my tivoapp. Maybe someone else could verify whether or not the recordings are scrambled without the tivoapp patch. Mine definitely seems encrypted extracted from a non-hacked S3. I have not found any app that can recognize it - Womble, VLC, TSReaderLite, etc. all do not recognize the file. Direct Show Dump doesn't know what to do with it. Tivo Decoder UI does something with it but crashes about 1/4 of the way through and the resulting file it spits out is not anything recognizable either (probably because it expects program stream instead of transport stream).

Did you verify that the modified transfer is indeed about 2x faster than normal TTG transfer? (Not much point pursuing this too much if the speedup is not significant enough).

[bcc]

Mine definitely seems encrypted extracted from a non-hacked S3. I have not found any app that can recognize it - Womble, VLC, TSReaderLite, etc. all do not recognize the file. Direct Show Dump doesn't know what to do with it. Tivo Decoder UI does something with it but crashes about 1/4 of the way through and the resulting file it spits out is not anything recognizable either (probably because it expects program stream instead of transport stream).vlc doesn't read mine either. You need to test with a player that is really permissive about the mpeg streams (it'll search for the headers when they are not where they should be). Have you tried xine or mplayer?Did you verify that the modified transfer is indeed about 2x faster than normal TTG transfer? (Not much point pursuing this too much if the speedup is not significant enough).
No I haven't checked. A point would be that with this method, one could bypass use of mfs_ftp/tserver&mfs-utils and extract with stock software on their tivo (without any pesky DRM added to the recording).

[moyekj]

Just tried mplayer - no dice. "Cannot seek backward in linear streams! Seek failed Exiting... (End of file)" I'm very sure I have an encrypted file. I've dealt a lot with mpeg2 transport streams and in it's current form this file is not it.

[bcc]

Ok, so I benchmarked regular TTG vs TTG with x-tivo-raw-tts. With regular TTG:
.tivo file of 620864088 bytes, 405 seconds to extract. That's 11Mbit/sec. Transfer rate was pretty constant during the download as well.

With x-tivo-raw-tts:
ts file of 647653868 bytes, 142 seconds. That's 35Mbit/sec. Ditto the constant transfer rate.

So we finally have some performance results that quantify just the overhead of adding the Qualcomm cipher to the recordings. Wow, that's a big performance hit.

[moyekj]

Ok, so I benchmarked regular TTG vs TTG with x-tivo-raw-tts. With regular TTG:
.tivo file of 620864088 bytes, 405 seconds to extract. That's 11Mbit/sec. Transfer rate was pretty constant during the download as well.

With x-tivo-raw-tts:
ts file of 647653868 bytes, 142 seconds. That's 35Mbit/sec. Ditto the constant transfer rate.

So we finally have some performance results that quantify just the overhead of adding the Qualcomm cipher to the recordings. Wow, that's a big performance hit. That's very good info. If you tune both tuners of your S3 to channels you don't receive for the duration of the transfer you can speed it up even more to about 44Mbit/sec or so. I would assume you can monitor Tivo resources in use (perfmeter or "top" style monitoring) and can determine that the transfer is still CPU limited?

So looks like very little chance of this benefiting users with unhacked Tivos? Personally I don't have the skills necessary for the PROM hack and wouldn't feel comfortable letting someone else do the hack either. Any other way?

[bcc]

That's very good info. If you tune both tuners of your S3 to channels you don't receive for the duration of the transfer you can speed it up even more to about 44Mbit/sec or so. I would assume you can monitor Tivo resources in use (perfmeter or "top" style monitoring) and can determine that the transfer is still CPU limited?My benchmark was with an s3 with both tuners tuned to channels I don't receive, and the system was in standby. Net connection was via built-in ethernet, stock MTU, stock kernel, and gig-e switch to a gig-e host.
So looks like very little chance of this benefiting users with unhacked Tivos? Personally I don't have the skills necessary for the PROM hack and wouldn't feel comfortable letting someone else do the hack either. Any other way?Well I haven't verified that TTG leaves the recording scrambled with x-tivo-raw-tts without a tivoapp change. What I see are headers before every ~128K of payload so you might just be having problems with your players being confused by those headers.

If it is in fact true that extractions remain scrambled, the weak link might be to figure out how to get the equivalent of unscramble.o working on a PC. You can feed the scrambler content via TTCB so it might not require getting a secret out of the crypto chip.

[bcc]

If it is in fact true that extractions remain scrambled, Ok, I looked & agree they're scrambled when tivoapp isn't patched (except for the header info).

[Jamie]

...
So we finally have some performance results that quantify just the overhead of adding the Qualcomm cipher to the recordings. Wow, that's a big performance hit.Is it really just the qualcomm cipher? Isn't there a remux involved in a non-raw TTG transfer that is being skipped with raw?

[AlphaWolf]

Just tried mplayer - no dice. "Cannot seek backward in linear streams! Seek failed Exiting... (End of file)" I'm very sure I have an encrypted file. I've dealt a lot with mpeg2 transport streams and in it's current form this file is not it.

Open the downloaded file with a hex editor and look at the first two bytes. If they aren't the same as a normal unencrypted stream, then it is definitely encrypted. I don't know if these newer streams have a different magic number, otherwise I would tell you what it is.

[Jamie]

Open the downloaded file with a hex editor and look at the first two bytes. If they aren't the same as a normal unencrypted stream, then it is definitely encrypted. I don't know if these newer streams have a different magic number, otherwise I would tell you what it is.The "magic" number is the same (see drmcheck.tcl). It might not be the first two bytes in the data stream, depending on what other metadata is being sent with the raw ty data.

This could still be useful on a hacked tivo. It's a way of extracting without need the vagaries of mfs_ftp. It also could be useful on an unhacked tivo if we could at least send it back to the tivo to play it. Of course, I assume the copy protection bits restrict what you can extract via http, while you have no so restrictions with mfs_ftp.

[bcc]

Is it really just the qualcomm cipher? Isn't there a remux involved in a non-raw TTG transfer that is being skipped with raw?Oh right, there would be the mpeg2 transport stream to program stream conversion as well. But that should be computationally fairly easy - more like reformatting the streams than re-muxing. The program stream is already muxed at the correct rate and with all the timestamps. For reference, notice how quickly videoredo can perform a ts->ps rewrite.

[Jamie]

Oh right, there would be the mpeg2 transport stream to program stream conversion as well. But that should be computationally fairly easy - more like reformatting the streams than re-muxing. The program stream is already muxed at the correct rate and with all the timestamps. For reference, notice how quickly videoredo can perform a ts->ps rewrite.That's what I've always thought (that the DRM processing is the bottleneck), but the TCF crowd (supported by TiVoPony, I think) have always disputed that and insisted that the reformat/remuxing was the issue with slow TTG transfers.

[bcc]

That's what I've always thought (that the DRM processing is the bottleneck), but the TCF crowd (supported by TiVoPony, I think) have always disputed that and insisted that the reformat/remuxing was the issue with slow TTG transfers.I guess I wouldn't put it past tivo to implement the conversion in such a way that it does cause a performance impact :) But I also wouldn't give TCF users so much credit. If someone over there mentions reformatting overheard they probably don't know whether they mean Qualcomm cipher and/or transport stream conversion.

Speaking of TCF, anyone notice that their forum says .tivo conversion discussion is not allowed, yet talk of programs performing such conversion is widespread?
Apparently it's ok to talk about videoredo, and tivodecode manager performing the very same conversion. It seems like tivo is happy to let these programs fill in the functionality gaps left by their own tivoserver. Sanctioned DRM violations?

[bcc]

So in the packet tracing of MRV, does MRV actually use the &Format=video%2Fx-tivo-raw-tts format spec? If so, this suggests that the two boxes are sharing scrambling information. That in turn would suggest that the scrambling keys/seed/whatever are transmitted between the MRV boxes, and it may be easy to sniff that information.

[Jamie]

So in the packet tracing of MRV, does MRV actually use the &Format=video%2Fx-tivo-raw-tts format spec? If so, this suggests that the two boxes are sharing scrambling information. That in turn would suggest that the scrambling keys/seed/whatever are transmitted between the MRV boxes, and it may be easy to sniff that information.There was an fcc filling from tivo that goes over how the "old style" MRV key exchange works. They called it "TiVoGuard", at the time. I think it was this (http://fjallfoss.fcc.gov/prod/ecfs/retrieve.cgi?native_or_pdf=pdf&id_document=6516214597) one. I don't think it was trivially snoopable. Could be different now though, as MRV has obviously been significantly changed.

[moyekj]

So in the packet tracing of MRV, does MRV actually use the &Format=video%2Fx-tivo-raw-tts format spec? If so, this suggests that the two boxes are sharing scrambling information. That in turn would suggest that the scrambling keys/seed/whatever are transmitted between the MRV boxes, and it may be easy to sniff that information. Yes, this is how I found out about the that format spec - by putting my Tivos and PC on the same hub, starting packet sniffer, and then initiating an MRV transfer. I have the packet dump of that experiment saved. You can find it here:
http://replayguide.sourceforge.net/mrvTransfer.pcap

[AlphaWolf]

Of course, I assume the copy protection bits restrict what you can extract via http, while you have no so restrictions with mfs_ftp.

But what is stopping us from neutering such bits?

[Jamie]

But what is stopping us from neutering such bits?Well, the context here was on an unhacked tivo, so I'd guess I'd say the tivo PROM. On a hacked tivo, of course, you can zap that stuff, at least on unencrypted recordings. If I remember right, you can't play back encrypted recordings at all if you modify the Drm data structures. They are "signed" and you'd probably have to find a way to bypass the signature checks to get encrypted recordings to play.

[AlphaWolf]

Well, the context here was on an unhacked tivo, so I'd guess I'd say the tivo PROM. On a hacked tivo, of course, you can zap that stuff.

Ah, I misinterpreted that.

[bcc]

Yes, this is how I found out about the that format spec - by putting my Tivos and PC on the same hub, starting packet sniffer, and then initiating an MRV transfer. I have the packet dump of that experiment saved. You can find it here:
http://replayguide.sourceforge.net/mrvTransfer.pcap
So then the scrambling algorithm is two-way and can be descrambled on a box other than the one that did the scrambling.

So have you found the scrambling info being negotated between the tivos (and what the scrambling algorithm is)? If tivo was careful, it'd be encrypted with the MRV certificate's key, but who knows, perhaps the algorithm only requires the box's service number.

[moyekj]

So then the scrambling algorithm is two-way and can be descrambled on a box other than the one that did the scrambling.

So have you found the scrambling info being negotated between the tivos (and what the scrambling algorithm is)? If tivo was careful, it'd be encrypted with the MRV certificate's key, but who knows, perhaps the algorithm only requires the box's service number. If you look at the Wireshark (http://sourceforge.net/project/showfiles.php?group_id=255) (ethereal) dump I gave in above post (http://replayguide.sourceforge.net/mrvTransfer.pcap) there is some authentication information being exchanged between boxes prior to the actual transfer taking place. If it's any help interpreting the dump:
PC = 192.168.1.100
Bedroom S3 = 192.168.1.107
LivingRoom S3 = 192.168.1.101
Sequence followed once packet sniffing enabled:
1. Start packet sniffer on PC (with Wireshark)
2. From Bedroom S3 browse Now Playing List of LivingRoom S3
3. Scroll down and select Grey's Anatomy and push into it's description
4. Initiate a transfer
5. Turn off packet sniffing after about 1 minute into the transfer

[bcc]

If you look at the Wireshark (ethereal) dump I gave in above post there is some authentication information being exchanged between boxes prior to the actual transfer taking place. If it's any help interpreting the above dump:
PC = 192.168.1.100
Bedroom S3 = 192.168.1.107
LivingRoom S3 = 192.168.1.101
Sequence followed once packet sniffing enabled:
1. Start packet sniffer on PC (with Wireshark (http://sourceforge.net/project/showfiles.php?group_id=255))
2. From Bedroom S3 browse Now Playing List of LivingRoom S3
3. Scroll down and select Grey's Anatomy and push into it's description
4. Initiate a transfer
5. Turn off packet sniffing after about 1 minute into the transfer

Thanks, yes, I can see those 2 boxes communicating, the request for Grey's Anatomy, etc.. The session would involve authenticating using the MAK. But my questions were lower level than that. Personally I don't plan on figuring out their scrambling algorithm. But you might, if you're serious about extracting your streams without DRM and without a prom mod.

[moyekj]

Thanks, yes, I can see those 2 boxes communicating, the request for Grey's Anatomy, etc.. The session would involve authenticating using the MAK. But my questions were lower level than that. Personally I don't plan on figuring out their scrambling algorithm. But you might, if you're serious about extracting your streams without DRM and without a prom mod. I don't have a heck of a lot of motivation to do it either, but perhaps the OP does (if he ever shows up in this thread). After all we are only talking about a ~2x speedup over regular TTG transfers - if it was 5-10x I would be more motivated.

Do you know the actual mechanics taking place for MRV? Is the originating S3 unencrypting and then re-encrypting the stream prior to transfer? You mentioned about a 35Mbps ceiling on transfers with the video format flag but did you determine what is limiting the speed? Is CPU still the bottleneck? It would be good (if you haven't already) to closely monitor resources on the Tivo ("top" or similar util) during a transfer to see if CPU is maxing out.

[bcc]

I don't have a heck of a lot of motivation to do it either, but perhaps the OP does (if he ever shows up in this thread). After all we are only talking about a ~2x speedup over regular TTG transfers - if it was 5-10x I would be more motivated.Well it was just over 3X in my test without any performance tuning.

Do you know the actual mechanics taking place for MRV? Is the originating S3 unencrypting and then re-encrypting the stream prior to transfer? You mentioned about a 35Mbps ceiling on transfers with the video format flag but did you determine what is limiting the speed? Is CPU still the bottleneck? It would be good (if you haven't already) to closely monitor resources on the Tivo ("top" or similar util) during a transfer to see if CPU is maxing out.The stream transmitted on the wire for s3<->s3 is apparently the same as what is on disk. When the file on disk is unscrambled, the transmitted copy is unscrambled as well. When the original is scrambled (as is on an unmoded system) the version on the wire is scrambled exactly like it was on the source machine's disk.

When I mentioned 35Mbps that was an average, sustained rate, not a peak rate.
I didn't check if it was CPU, disk, bus or what that was the bottleneck. Someone would have to start paying me before I'd do a thorough performance analysis.
However, folks especially jamie have looked into that more closely in the performance derby thread.

[mikesown]

I don't have a heck of a lot of motivation to do it either, but perhaps the OP does (if he ever shows up in this thread). After all we are only talking about a ~2x speedup over regular TTG transfers - if it was 5-10x I would be more motivated.

Do you know the actual mechanics taking place for MRV? Is the originating S3 unencrypting and then re-encrypting the stream prior to transfer? You mentioned about a 35Mbps ceiling on transfers with the video format flag but did you determine what is limiting the speed? Is CPU still the bottleneck? It would be good (if you haven't already) to closely monitor resources on the Tivo ("top" or similar util) during a transfer to see if CPU is maxing out.

I'm around here... lurking ;). My main motivation is indeed the transfer time. Another motivation is being able to transfer HD in realtime on a THD unit(TTG is too slow with the THD). Due to this transfer increase, I think it would be possible(and cool) to have a player like VLC support streaming directly, so we could watch the file as it downloads(without stopping and starting it).

For the MRV mechanism, I'll be honest in saying that this information is just what I've heard lurking around here and TCF, but my impression is that the THD has a hardware encryption/decryption mechanism which the file is fed through to decrypt it. I would speculate that the file is transfered natively, and the same hardware is used to perform the decryption on the second box. I would also speculate that the main limiting factor now(when using the new native stream method) is the kernel's drivers for the ethernet controller. I remember someone posting benchmarks of .ty transfers using mfs_ftp where using the native kernel was about 30mbit, whereas using a modified kernel, with external adapter supporting jumbo frames, the transfer went up to 75mbit. There's not much we can do about this, but if Tivo opts to upgrade the drivers, the speed would likely see a big increase.

[lrhorer]

I aim to develop an MRV emulator for the PC so that transfers will go much faster than TTG.
Are you still planning to move ahead with this? I am certainly interested. I don't really care that the content can't be played on a PC. I just want to be able to move it off the TiVo and on to the server to free up space, and then to watch it real-time on the TiVo when I want. TTCB is not fast enough to handle real-time transfers if the bit rate is greater than roughly 15 or 16 Mbps. Even with 12 Mbps video, I find one needs to buffer at least 30 seconds or so or risk a pause during the show.

[lrhorer]

I don't have a heck of a lot of motivation to do it either, but perhaps the OP does (if he ever shows up in this thread). After all we are only talking about a ~2x speedup over regular TTG transfers - if it was 5-10x I would be more motivated.
Well, for my purposes, 2x is a lot. In fact, 20% is significant. The thing is, TTCB is not quite fast enough to transfer high bandwidth HD real time. Unless one buffers about 4 - 5 minutes per hour of program, one will wind up facing automated pauses during playback for any program which exceeds about 15 or 16 Mbps.