Cachecard network burst mode

[jafa]

Hi all,

I have been playing around with a proof of concept.

The cachecard ethernet interface can theoretically go really fast but it is held back by the software achitecture on the TiVo.

With my TiVo I typically see approx 1MB/s (8Mbps) harddrive to network speeds.

I have just completed a proof of concept for a burst mode - clocked up a sustained harddrive to network speed of 5.6MB/s (45Mbps) on the same TiVo.

Edit: New test... transfered 95780380 bytes in 12.352 seconds = 62Mbps = 7.75MB/s

Nick

[JJBliss]

Feel free to discuss it further here if you like.

Replicatable on Series 2 units or is it a hardware modification to your cachecards?

[jafa]

It is an update to the cachecard drivers to make use of the bursting features of the cachecard. It links the IDE cache interface to the ethernet interface.

The API is an ioctl - you specify the sector range and the destination PC and it will burst the data out the network interface.

The communication is over UDP - as long as the PC can handle the data rate there should be low loss over an internal network. Lost data can be re-requested.

A second conventional TCP channel should be used to manage the transfer.

45Mbps was achieved with a 64k block size and 4 threads.

Nick

[schottjy]

Brilliant.. So close to buying a CacheCard now

[Jamie]

The communication is over UDP - as long as the PC can handle the data rate there should be low loss over an internal network. Lost data can be re-requested.

Sounds interesting, but unless I'm missing something, all the user mode software (both client and server) is going to have to be replaced to use UDP.

For what it's worth, at least one user is reporting >3MB/sec with the existing cachecard drivers and the latest tserver, which has internal performance improvements that reduce CPU load. Still a far cry from 45Mb/sec but much better than 8Mb/sec.

[jafa]

New test...

Transfered 95780380 bytes in 12.352 seconds = 62Mbps = 7.75MB/s

Still tweeking

[sanderton]

What software are you using at the PC end? Is the sector range representing part file stored in MFS, if not, how might you get it?

[rbiro]

The communication is over UDP - as long as the PC can handle the data rate there should be low loss over an internal network. Lost data can be re-requested.

A second conventional TCP channel should be used to manage the transfer.

45Mbps was achieved with a 64k block size and 4 threads.

Nick

So would that mean that the core hacking software

mfs_ftp
tserver
vserver

would have to be modified to take advantage of the significant throughput improvements?

What about the PC side software?

[laserfan]

New test...

Transfered 95780380 bytes in 12.352 seconds = 62Mbps = 7.75MB/s

Still tweeking

Your post made me sufficiently weak-in-the-knees to order a cachecard from 9thtee.

Now you gots to deliver! Presumably your tweaks will work w/o new cachecard firmware (latest 2.2 I think)?

[jafa]

Yep - it works with any cachecard... I will have test program to download and try in the next two weeks.

[jafa]

Ok, test program...

Edit - updated installer link is in post below.

There are a number of tweakable parameters... in the next few days I will convert them to be command line parameters so you can play with them.

BTW XP SP2 has serious network performance problems. If it is transfering data but getting overflow/timeout messages, this usually indicates that your PC can't keep up.

I am seeing approx 50Mbps with this test program. I have seen >60Mbps but it will take a bit more software work to get right.

Installation on tivo - see readme.txt.

When you run burst.exe it will give an example for transfering the root partition - this is a good test.

Nick

[laserfan]

So many people are having so much trouble with so many programs with XP SP2--I'm glad I have a slow Internet connection. I'm not even tempted to upgrade (from SP1 + hotfixes).

Anyway DL'ing the zip now and looking forward to playing with what, a 10x improvement? Too early for higher math this day-after.

My Tivonet card works great and I have all the time-in-the-world, but I really like this hardware hacking stuff. Thanks!

[jafa]

Edit - updated installer link is in post below

[jafa]

Native driver installer:
http://www.silicondust.com/nic_insta...o_20041226.zip

Installation instructions:
http://www.silicondust.com/forum/viewtopic.php?t=340

Once installed, reboot, telnet in and run:
burstd

PC program::
http://www.silicondust.com/burst_20041226.zip

Speed test (backup root filesystem):
burst //192.168.0.200/0:7:0-262144 root7.bak

[rbiro]

Here are my results:

PC: P4, 2.6 GHz, WinXP SP2
TiVo: Sony SVR-2000 (SA1). (The TiVo was in standby mode and not recording.)

Using burst_test_20041226.zip

Network topology is:
PC to Linksys 5 port switch to Linksys 4 port router to Cachecard

Results:
transfered 134217728 bytes in 36.984000 seconds = 29.032604 Mbps
transfered 134217728 bytes in 33.172000 seconds = 32.368920 Mbps
transfered 134217728 bytes in 37.547000 seconds = 28.597273 Mbps

So I'm not getting 60-75 Mbps that Jafa is, but it is at least 3x faster than my best FTP...

Output from 1st transfer:

C:\burst_test_20041226\pc>burst //192.168.1.23/0:7:0-262144 root7.bak
sending request for 0:7:00000000-00008000
sending request for 0:7:00006ef2-00006ef5
rx overflow or timeout
sending request for 0:7:00002536-00002564
rx overflow or timeout
sending request for 0:7:00008000-00010000
sending request for 0:7:0000bf25-0000bf31
rx overflow or timeout
sending request for 0:7:0000b480-0000b483
rx overflow or timeout
sending request for 0:7:0000b3aa-0000b3cd
rx overflow or timeout
sending request for 0:7:00010000-00018000
sending request for 0:7:00014ec4-00014edf
rx overflow or timeout
sending request for 0:7:00018000-00020000
rx overflow or timeout
sending request for 0:7:0001fa1c-0001fa28
rx overflow or timeout
sending request for 0:7:0001aa69-0001aa7e
rx overflow or timeout
sending request for 0:7:0001a925-0001a956
rx overflow or timeout
sending request for 0:7:00020000-00028000
sending request for 0:7:00020c99-00020ca3
rx overflow or timeout
sending request for 0:7:00028000-00030000
sending request for 0:7:0002e827-0002e82e
rx overflow or timeout
sending request for 0:7:000295fa-00029603
rx overflow or timeout
sending request for 0:7:000294de-000294f0
rx overflow or timeout
sending request for 0:7:00030000-00038000
sending request for 0:7:0003707a-000370cd
rx overflow or timeout
sending request for 0:7:000318b6-000318cb
rx overflow or timeout
sending request for 0:7:00038000-00040000
sending request for 0:7:0003f444-0003f480
rx overflow or timeout
sending request for 0:7:0003c4ef-0003c4fb
rx overflow or timeout
sending request for 0:7:0003a6db-0003a700
rx overflow or timeout
sending request for 0:7:0003a352-0003a378
rx overflow or timeout
sending request for 0:7:0003a200-0003a20c
rx overflow or timeout
sending request for 0:7:0003a10b-0003a10f
rx overflow or timeout
sending request for 0:7:00039a3e-00039a53
rx overflow or timeout
sending request for 0:7:00039908-00039934
rx overflow or timeout
transfered 134217728 bytes in 36.984000 seconds = 29.032604 Mbps