USB2 backport from 2.4.27 to 2.4.4/2.4.18/2.4.20

[Jamie]

*************************************************************************************
The latest package is here.
Previous versions are: here, here, here, here, here and here.
Support thread: here.
*************************************************************************************

As I mentioned in this thread, I've had some success backporting the linux 2.4.27 usb subsystem to 2.4.18 for TiVo software 4.0.

Here's the source as a patch to the kernel tree in TiVo-4.0-linux.2.4.tar.gz. The patch can be applied with "patch -p1 <usb2-2.4.18.patch" from within the linux-2.4 directory.

The patch is big since I moved the whole drivers/usb subtree in from 2.4.27. I barely got it in under the 1MB attachment limit. If you want to see my changes, it's probably easiest to apply the patch, then diff the drivers/usb subtree against a clean linux-2.4.27 source tree.

Caveats:

I compiled with a gcc3.0 tool chain.

The only thing I tested was the usbnet module with a linksys usb200m.

I didn't configure all usb devices, so the devices I didn't configure may not even compile if turned on.

I don't claim to have a full understanding of urb's, sk_buff's, pci_set_mwi, or a variety of other things in the code I changed, so I may have botched things. All I can say is, works for me with the usb200m.

Improvements would be welcome.

[Edit 2004/10/08: Please continue reading the thread. Later posts contain newer versions.]

[TheWickedPriest]

So, what improvements does this give?

[Jamie]

Originally Posted by TheWickedPriest

So, what improvements does this give?

You should probably go read the referenced support thread to understand the motivation for this backport.

Summary: Usb2 at "hi-speed" is faster than usb1.1 (480Mb/sec verses 12Mb/sec). The MuscleNerd compiled usb2 modules available here wedged my network when MRV was used to transfer programs. It didn't just hang my tivo, it hung the whole network. That's what drove me to look for an alternative. Others experienced similar problems, though some people are using the MN modules without problem. I can't explain why they work for some, but not for others. Different switches? Riley described "network oddities using various switches" here.

The 2.4.27 drivers also support additional devices. For example, the usbnet driver is supposed to support IP over usb2 with host-to-host USB cables (no intermediate ethernet). This might be faster than using an ethernet connection. I haven't tried it. The cables are cheap ($15 or so at newegg), so I might pick one up and give it a shot next time they are in stock. In practice, I see that netcat is using 100% of the CPU during my performance tests. So there might not be a lot of room for improvement unless the driver overhead can be reduced, e.g. with a larger MTU.

[alldeadhomiez]

Changes from your code to 20041006 code (see later posts for information on the newer revisions attached to this post):

First, I added kernel 2.4.20 support, to address the USB babble issues seen on 5.1.1b (and maybe fix some of the network lockup issues on the HD units). The babble issue is identical on 4.x and 5.x, but on my SD-H400 it was particularly frequent using the stock 2.4.20 USB and ax8817x drivers.

I also changed the structs in usb.h so that (depending on the kernel you're building for) the USB core and host controller drivers use the same ABI as what the TiVo-supplied drivers expect. This allows us to use existing, working drivers, particularly the closed-source Atmel WLAN drivers, with your 2.4.27 USB subsystem backport. In my testing, using the new 2.4.27 USB subsystem with any version of the ax8817x driver still produces babble problems, but switching to usbnet resolves the problem.

In this archive, usbcore and ehci-hcd have been merged into a single module, which detects whether your box supports USB 2.0 based on the BORD code. Idea being, you can install these new modules as follows:

- back up /lib/modules to somewhere safe
- delete ax8817x.o, usbcore.o, usb-ohci.o, and (if it exists) ehci-hcd.o
- copy in the new modules: usbcore.o, usb-ohci.o, and usbnet.o
- symlink usbnet.o to ax8817x.o

Using this method, no scripts will need to be changed to gain the benefit of the new modules on 4.x or 5.x, and you can test different chipsets with little or no driver reconfiguration.

---

Note to heroes: these drivers are experimental! Do not include them in any installation tool, and do not link directly to the attachments (link to the thread instead).

Latest version: 20050104
Last known good version: 20041024
usbsrc = source code, usbobj = binary objects

[Jamie]

Originally Posted by alldeadhomiez

I made a few changes to the code you posted.
...

Great. Thanks! I'll give them a try and will use this as a base for any future motifications.

I'm trying to get ALI M5632 based host-to-host support in usbnet working for someone (backported from 2.6.8) but the debugging cycle is a bit long since I don't have the hardware. I'm also waiting for feedback from someone trying out the usb-storage.o module.

[bcc]

Have you been able to see any performance improvements out of the new driver with ax8817x based ethernet dongles or in host-to-host mode?

[Jamie]

Originally Posted by bcc

Have you been able to see any performance improvements out of the new driver with ax8817x based ethernet dongles or in host-to-host mode?

See the support thread for what little performance data has been reported. Summary: the drivers in the original post did about 3MB/sec on a quiescient system on a netcat </dev/zero >/devnull test. I haven't tried the ADH modified drivers yet. I don't have a usb2 host-to-host cable, and I haven't heard from anyone that has one working yet.

[alldeadhomiez]

usbnet, pc->tivo, kernel 2.4.20, Xterasys generic ax8817x (0x7b8/0x420a)
  134217728 bytes transferred in 34.687189 seconds (3869375 bytes/sec)
  134217728 bytes transferred in 34.625633 seconds (3876253 bytes/sec)
  134217728 bytes transferred in 34.836329 seconds (3852809 bytes/sec)

usbnet, tivo->pc, kernel 2.4.20, Xterasys generic ax8817x (0x7b8/0x420a)
  134217728 bytes transferred in 62.285 seconds (2154896 bytes/sec)
  134217728 bytes transferred in 62.269 seconds (2155450 bytes/sec)
  134217728 bytes transferred in 64.578 seconds (2078382 bytes/sec)

ax8817x, pc->tivo, kernel 2.4.20, Xterasys generic ax8817x (0x7b8/0x420a)
  134217728 bytes transferred in 33.659399 seconds (3987526 bytes/sec)
  134217728 bytes transferred in 33.507143 seconds (4005645 bytes/sec)
  134217728 bytes transferred in 33.906486 seconds (3958468 bytes/sec)

ax8817x, tivo->pc, kernel 2.4.20, Xterasys generic ax8817x (0x7b8/0x420a)
  134217728 bytes transferred in 65.580 seconds (2046626 bytes/sec)
  134217728 bytes transferred in 64.858 seconds (2069409 bytes/sec)
  134217728 bytes transferred in 63.798 seconds (2103792 bytes/sec)

[bcc]

Darn, no better than what I've seen with the old driver for tivo->pc transfers. That's what I was afraid of...

[Jamie]

Originally Posted by bcc

Darn, no better than what I've seen with the old driver for tivo->pc transfers. That's what I was afraid of...

I've been wondering if adapting the scatter/gather style transfers from the usb-storage module would help reduce CPU load and increase performance. I've been waiting for some performance data from someone trying the usb-storage driver to see if it does any better than the eth drivers on the tivo. Not sure I have the time and energy to attack such a project, but it's an idea.

[Jamie]

Here's some more data.

I cross compiled netperf. This is a generally accepted tool for network performance benchmarking.

Bottom line: you gain about 20% in throughput by running a kernel that has CONFIG_NET_FILTER and CONFIG_FILTER turned off. I guess even with empty chains, there is still some extra code each packet has to go through. {edit: turning CONFIG_FILTER breaks dhcp, and doesn't affect TCP performance anyway. Turn only CONFIG_NET_FILTER off for the performance gain.}

One thing that's strange: netperf shows better performance sending from the tivo than receiving. This is the reverse of what we all see with dd|netcat style tests.

------------------------------------------------------------------------
ADH drivers; tivo 4.0.1b standard kernel

pc->tivo
 % netperf -t TCP_STREAM -H tivo-bedroom -l 60
TCP STREAM TEST to tivo-bedroom
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 43689  16384  16384    60.01      18.76   

tivo->pc
bash-2.02# netperf -t TCP_STREAM -H 192.168.1.100 -l 60
TCP STREAM TEST to 192.168.1.100
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 87380  16384  16384    60.00      25.57          

------------------------------------------------------------------------
ADH drivers; 2.4.18 kernel without netfilter

pc->tivo
 % netperf -t TCP_STREAM -H tivo-bedroom -l 60
TCP STREAM TEST to tivo-bedroom
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 43689  16384  16384    60.01      22.24   

tivo->pc
bash-2.02# netperf -t TCP_STREAM -H 192.168.1.100  -l 60
TCP STREAM TEST to 192.168.1.100
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 87380  16384  16384    60.00      31.09

[alldeadhomiez]

I have attached a new revision above (20041007) which includes 2.4.4 support and extra drivers included in the old 2.4.4 USB package. This package is intended to supercede the 2.4.20->2.4.4 backport in that thread.

For 2.4.4 users, this package offers the following improvements over the package I posted last year:
- fixes unaligned access nags
- more recent, less buggy driver and core versions
- replaces ax8817x.o with usbnet.o
- supports more devices by adding "newly" discovered device IDs
- cleaner build system
- usbdevfs/usbfs now works correctly
- supports the 2.4.4 ABI, so the core works with many other 2.4.4 drivers supplied by TiVo or third parties

For 2.4.18 and 2.4.20 users, the benefits over the 20041006 release are:
- adds support for pegasus, kaweth, rtl8150, and at76c503 adapters (in case the TiVo supplied drivers do not work)
- slight improvements in logging initialization status of the integrated EHCI driver

I have run several tests against the modules included in this package:

Tests on 2004/10/07 drivers:

 PASS: 2.4.20, usbfs works
 PASS: 2.4.20, kaweth, 3com 3c19250 (0x506/0x3e8)
   notes: assertion failures on device removal
 PASS: 2.4.20, usbnet, Xterasys ax8817x adapter (0x7b8/0x420a)
 PASS: 2.4.20, pegasus, Linksys USB100TX (0x66b/0x2203)
 PASS: 2.4.20, TiVo pegasus, Linksys USB100TX (0x66b/0x2203)
 FAIL: 2.4.20, TiVo kaweth, 3com 3c19250 (0x506/0x3e8)
 FAIL: 2.4.20, TiVo kaweth w/TiVo usbcore, 3com 3c19250 (0x506/0x3e8)
   notes: unaligned accesses, no network connectivity, system lockup
 PASS: 2.4.20, at76c503-rfmd, Belkin F5D6050 (0xd5c/0xa002)

 PASS: 2.4.18, usbnet, Xterasys ax8817x adapter (0x7b8/0x420a)

 PASS: 2.4.4, usbfs works
 PASS: 2.4.4, usbnet, Xterasys ax8817x adapter (0x7b8/0x420a)
 PASS: 2.4.4, kaweth, 3com 3c19250 (0x506/0x3e8)
   notes: assertion failures on device removal
 PASS: 2.4.4, pegasus, Linksys USB100TX (0x66b/0x2203)
 FAIL: 2.4.4, TiVo pegasus, Linksys USB100TX (0x66b/0x2203)
 FAIL: 2.4.4, TiVo pegasus w/TiVo usbcore, Linksys USB100TX (0x66b/0x2203)
 PASS: 2.4.4, at76c503-rfmd, Belkin F5D6050 (0xd5c/0xa002)

Unless noted as "TiVo," all drivers and usbcore/usb-ohci/(ehci-hcd) are from
this package.

MRV was not tested, but so far the usbnet driver has appeared to fix the
babble issue associated with EHCI 2.4.20 + ax8817x.o.

Future work:
- Merge in the linux-wlan-ng stuff
- Add other useful/updated USB drivers: printer, scanner, usb-storage, serial, etc.
- Update to the latest at76c503 drivers

[Woody Allen]

Originally Posted by Jamie

I'm trying to get ALI M5632 based host-to-host support in usbnet working for someone (backported from 2.6.8)

Here are the results. All credit goes to Jamie!

tivo:/usr/local/test$ ./test3
100000+0 records in
100000+0 records out

real 0m24.640s
user 0m1.190s
sys 0m19.630s
tivo:/usr/local/test$
tivo:/usr/local/test$ cat test3
#!/bin/sh
time dd if=/dev/zero bs=1024 count=100000 | netcat 192.168.1.1 10000 -w 1

tivo:/usr/local/test$ /usr/local/busybox/ifconfig
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:4 errors:0 dropped:0 overruns:0 frame:0
TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:312 (312.0 B) TX bytes:312 (312.0 B)

usb0 Link encap:Ethernet HWaddr 362:9F:6B:BE:AB
inet addr:192.168.1.2 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:15000 Metric:1
RX packets:22486 errors:0 dropped:0 overruns:0 frame:0
TX packets:33867 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1169304 (1.1 MiB) TX bytes:332020734 (316.6 MiB)

tivo(/dev/zero)->pc(/dev/null) 4 MBS
pc(/dev/zero)->tivo(/dev/null) 6.2 MBS

These results are still very experimental in that they have not been tested for side effects and actual file transfers

Increasing the MTU to greater than 15000 causes the tivo to become unstable - hangs and/or kevent messages.

[Jamie]

Originally Posted by Woody Allen

[b]Here are the results....

Thanks for doing the testing and reporting results.

I believe the dropped kevents are due to EVENT_RX_MEMORY. Basically the driver is unable to allocate kernel memory, causing it to defer some of its work to a kevent. Turns out it can queue up only one of these defered kevents. If it tries a second time while the first is pending, it gets this warning.

The driver is currently allocating 60 transmit and receive buffers internally. That's probably too much for the poor TiVo when the MTU is large. 2*60*15000 works out to about 2MB. It should be possible to take the MTU into account when deciding how much queueing to do. Larger MTUs should require less queueing, so the number should be smaller. I tweaked with the queueing in the backport to adopt the larger queues from the 2.6.8 code rather than what the 2.4.27 code original had (which, as I recall, was a queue length of 1). Preliminary testing indicated that the larger queues helped when the MTU was the ethernet standard of 1500. But it did tickle this kevent warning occasionally.

I'm not sure that the txqueuelen has any effect, but if it means txqueuelen*MTU bytes are being allocated, 2000 is way too large.

I'll post netperf. I'm not sure that the dd|netcat measurement approach is capturing just the network performance. There are some extra data copies involved, and the context switching between processes when you run a dd|netcat pipeline.

Ultimately what we care about though is MRV and mfs_ftp transfer performance, so some final end-to-end performance numbers would be useful too once we get the network parameters tuned. I was hoping the host-to-host cables could break 10MB/sec.

[alldeadhomiez]

Release 20041013 has been posted (above), with the following changes:

2004/10/12 Jamie

 - Added ALI M5632 support to usbnet

2004/10/11 alldeadhomiez

 - Added usb-storage.o, scanner.o, and printer.o from 2.4.27
 - Changed directory layout (back)
 - Added precompiled SCSI modules for all three kernels
 - Added new ifconfig binary per EvilJack

2004/10/08 alldeadhomiez

 - Added ehci-dummy.o (keeps a "real" ehci-hcd module from loading, when
   renamed)

Please acquaint yourself with the contents of the README file before you proceed.