fix for libc's infamous "Illegal Instruction" error

[maxwells_daemon]

Hi,

I have found a fix to the problem which causes an "Illegal Instruction" error on Series 1 TiVos (at least with UK software version 2.5.5). The fix allows built-in and newly compiled TiVo programs to perform a number of functions (notably DNS and passwd lookups) that previously had to be hacked around.

The fix and other programs mentioned here can be found on my TiVo software page.

Without the fix, programs that don't have an explicit workaround (eg. with libtivohack) crash. To use a couple of random examples

TiVo ~ > route.tivo   
route_info, afname=inet, options=33
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
Illegal Instruction
TiVo ~ > id
uid=0Illegal Instruction

("route" is part of the standard TiVo installation, though it's renamed to route.tivo by the TurboNET upgrade, and "id" is from tivobin.tgz).

With the fixed shared libraries in /var/hack/lib, these programs work correctly (assuming some reasonable settings in /etc/passwd and /etc/hosts).

TiVo ~ > export LD_LIBRARY_PATH=/var/hack/lib
TiVo ~ > route.tivo
route_info, afname=inet, options=33
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.1.0     *               255.255.255.0   U     32767  0        0 eth1
default         my.router       0.0.0.0         UG    0      0        0 eth1
TiVo ~ > id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon)

OK, those are trivial examples because more useful commands have been made to work after a fashion by linking to libtivohack. But with the fixed libc, it is now possible to build fully functional Unix tools, with little or no source-code modification necessary. I have built GNU InetUtils, procps, and TinyLogin (nearly) out of the box (OK, I modified InetUtils because I found some bugs that weren't TiVo specific, and I modified procps because of the floating-point error I reported earlier).

TiVo ~ > ping www.cern.ch
PING webr2.cern.ch (137.138.28.230): 56 data bytes
64 bytes from 137.138.28.230: icmp_seq=0 ttl=112 time=170.268 ms
64 bytes from 137.138.28.230: icmp_seq=1 ttl=112 time=161.671 ms
--- webr2.cern.ch ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 161.671/165.969/170.268/4.299 ms

TiVo ~ > rcp maxwell@mypc:mfs_bin.ppc-20041213a.tar.bz2 .
TiVo ~ > /var/hack/sbin/inetd

With inetd running, the the TiVo can run various daemons with full login capability, eg.

mypc ~ > rsh tivo uname -a
Linux tims-tivo 2.1.24-TiVo-2.5 #14 Wed Oct 8 12:06:25 MDT 2003 ppc unknown
mypc ~ > telnet tivo
Trying 192.168.1.150...
Connected to tivo.
Escape character is '^]'.

Linux 2.1.24-TiVo-2.5 (tims-tivo) (ttyp1)

tims-tivo login: root
Password: 
TiVo ~ >

and so on.

The libc error occurred whenever a program tried to execute any of the lookup functions implemented via the C Library's Name Service Switch (NSS) scheme (passwd, group, hosts, protocols, services, etc). This bug is present in both the TiVo 2.5.5 standard libc and in other rebuilt libraries (such as from tivoutils). I'd be interested to know whether it is also present in later TiVo PowerPC software versions (3.0+).

The fix was very simple: for some unaccountable reason, nss_lookup_function's body (in libc/nss/nsswitch.c) had been removed in TiVo's official TiVo-2.5.x-toolchain.tar.gz. Without this, a random function pointer was returned, so when this was executed, it's not surprising that an Illegal Instruction ensued. I wonder whether this was random stupidity on a TiVo programmer's part, or some clumsy (but effective ) plan to make hacking more difficult. Anyway I restored the missing code, and you can download the recompiled libraries from my web page.

Tim.

[rbiro]

Using my Sony SVR-2000 (SA, 3.0), it works like a charm.

[maxwells_daemon]

Thanks rbiro, that's good to know. Which packages did you try?

I'd also be interested to confirm whether the TiVo 3.0 software had the libc bug (I haven't seen any mention of it being fixed, but I can't see the same error in the 3.0 sources). Did you previously see the "illegal instruction" error? Did you have a previously installed libc, or were you using the one that came with the TiVo?

Thanks,
Tim.

[rbiro]

I've installed everything and tested out the simple stuff - telnet, ftp, dns resolution, etc.

I haven't done the more involved daemon stuff.

[rbiro]

I hadn't touched this in a while, and I wanted to copy it to my other tivo's - but now I find my DNS doesn't work and the lib directory doesn't stay exported.

In my standard startup script I have the following lines:

export PATH=$PATH:/var/hack:/var/hack/bin:/var/mfs_ftp

export LD_LIBRARY_PATH=/var/hack/lib:/lib:/lib/modules

But after a reboot to ensure it takes, my system acts as though I didn't export the LD_LIBRARY_PATH variable. I do an id and get the uid=0Illegal Instruction.
I manually re-type the export and the id works correctly.
Even more telling on a login is when I try to use telnet, it will tell me it can't find a library - of course it is there and after re-exporting the environment variable again it works.

I've tried exporting it from my /.profile as well, but it just won't take.

Then there is the DNS lookup. Again, I swear earlier this was working, but not anymore. My resolv.conf is in /etc. All nameservers listed there are valid. From the tivo, I can ping all of them by IP address.

(none):/var/hack$id
uid=0Illegal Instruction
(none):/var/hack$export LD_LIBRARY_PATH=/var/hack/lib:/lib:/lib/modules
(none):/var/hack$id
uid=0 gid=0
(none):/var/hack$ping 67.19.53.190
PING 67.19.53.190 (67.19.53.190): 56 data bytes
64 bytes from 67.19.53.190: icmp_seq=0 ttl=47 time=79.564 ms
64 bytes from 67.19.53.190: icmp_seq=1 ttl=47 time=69.508 ms
64 bytes from 67.19.53.190: icmp_seq=2 ttl=47 time=62.084 ms
64 bytes from 67.19.53.190: icmp_seq=3 ttl=47 time=57.143 ms

--- 67.19.53.190 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max/stddev = 57.143/67.075/79.564 ms
(none):/var/hack$ping dns1.odsol.com
ping: cannot resolve dns1.odsol.com: Unknown server error

[rbiro]

I was concentrating on the library path issue and I overlooked my secondary binary path issue.
Of course ping comes in the standard /bin directory.
If I have my path append /var/hack/bin to the end, it will always run the 1st ping it finds. I switch my export statement to

export PATH=/var/hack/bin:/var/hack:/var/mfs_ftp:$PATH

and the binary path issue is gone.

Since I couldn't figure out what was wrong with my library path issue, I merely copied the contents of my /var/hack/lib directory to /lib

[maxwells_daemon]

I'm afraid environment variables like PATH and LD_LIBRARY_PATH are local to the process you set them in (also in subprocesses if you use "export" as you did), so if you reboot (or even just log out and in again), they will be lost.

The usual way to save those settings is to put the commands to set them in your .profile startup script. The only problem there is to ensure that the .profile is executed - and the right one. .profile is only executed from a login shell, and it is found in the user's home directory (presumably the user is "root"). Depending on how you log into your TiVo, that may not be set up the way you expect. How do you log into your TiVo?

I'm using inetd to log in via telnet or rsh using the setting I gave on my web page. That does a full canonical login, so it picks up the home directory from the root account's entry in the /etc/passwd file (I have this set to /var/hack) and runs .profile from there.

If you are using tnlited, then I'm not sure where it reads the .profile from by default (or indeed whether it does it at all). When I used tnlited (before I got inetd working), I used the following

HOME=/var/hack /sbin/tnlited 23 /bin/bash -login &

in my /etc/rc.d/rc.sysinit file. That ensured a full login shell (-login) with the .profile read from /var/hack.

Does this help?

Anyway, putting the libraries in /lib should be OK too and, as you found, does simplify things. You should be aware that this means that the recompiled libc.so will also be used by all dynamically-linked executables, not just ones that you run from the shell. I don't know whether any of the TiVo standard tasks are dynamically-linked (I see that myworld is statically linked), and even if they are there shouldn't be any difference (my libc.so was rebuilt from the same sources, without that one bug). Nevertheless, I preferred not to touch anything I didn't need to, which is why I messed around with LD_LIBRARY_PATH.

Tim.

[rbiro]

Yeah, today I moved the binary and library paths to my /.profile and that takes care of it.
I guess I need to find an original copy of the contents of /lib to restore that, just to be safe again.

[maxwells_daemon]

I don't have TiVo software version 3.0, so can't be sure, but I guess the only thing that'll have been overwritten is libc.so.

[rbiro]

Yeah, and I got that from another backup