PDA

View Full Version : Porting the Kaffe java runtime to the tivo



Jamie
09-16-2007, 02:45 PM
... As far as I know, there is no Java version that runs on MIPS and I doubt the Tivo could hardware could support Java (it's kind of fat).I asked this question recently too and was point to kaffe (http://www.kaffe.org). It's been ported to many platforms (http://www.kaffe.org/ports.shtml), including linux on mips, and includes a JIT compiler. If you can run a JRE on your cell phone, it seems like a tivo could handle it too.

dburckh
09-16-2007, 04:53 PM
I asked this question recently too and was point to kaffe (http://www.kaffe.org). It's been ported to many platforms (http://www.kaffe.org/ports.shtml), including linux on mips, and includes a JIT compiler. If you can run a JRE on your cell phone, it seems like a tivo could handle it too.

That rocks. I'll give it a shot. Anybody tried this?

This would be very cool as I could turn the Tivo into a UPNP server. I'm not overly concerned about CPU, but I have some concerns about memory, especially with HD streams. Version 3 should be leaner and meaner, so it maybe a possibility.

Update:

Ok. I have an HR10-250. I downloaded the MIPS version of Kaffe kaffe-1.0.5-6.mips.rpm (The site went down shortly after that). Extracted the cpio.gz file. unzip and did a cpio -id into /var/java. Did a symbolic link to /var/java/usr to /usr (seems to want to run out of /usr).

I type java -version and I now get:

/usr/bin/kaffe: /usr/lib/kaffe/bin/Kaffe: cannot execute binary file
/usr/bin/kaffe: /usr/lib/kaffe/bin/Kaffe: Exec format error

Any ideas?

Jamie
09-16-2007, 06:10 PM
Any ideas?I would expect you'd need to build from source with a cross compiler rather than using pre-built binaries. If you have debian-mips on your tivo, you might be able to use debian-mips binary packages, but I'd be surprised if a random linux-mips rpm would work out of the box on a tivo.

dburckh
09-16-2007, 06:28 PM
How involved do you think something like that is?

I haven't compiled C for about 10 years and the complier was already setup. I have Windows XP Home and Ubuntu. I would prefer Cygwin if possible, but I'm willing to fumble around in Unix if it saves me some headaches.

Jamie
09-16-2007, 07:00 PM
How involved do you think something like that is?

I haven't compiled C for about 10 years and the complier was already setup. I have Windows XP Home and Ubuntu. I would prefer Cygwin if possible, but I'm willing to fumble around in Unix if it saves me some headaches.I wouldn't expect it to be completely trivial, but at least we know it has been compiled for linux-mips before, so it shouldn't be too big a bear.

You can build a cross compilation tool chain in cygwin using the scripts I posted here (http://www.dealdatabase.com/forum/showthread.php?t=46361). They might need to be tweaked a little. I know, at least, that the host in the url used to fetch sources from tivo.com is now dynamic.tivo.com instead of www.tivo.com.

I would give it a shot, but I've got a new TiVoHD to play with instead :)

dburckh
09-16-2007, 11:01 PM
Damn. I was hoping you would. :)

Anyway, have fun with your TiVoHD. I'm a little envious. Here's hoping DTV comes out with an H264 Tivo.

Ok, I'll start playing with this in a little bit. I suspect it's going to take some tweaking.

Jamie
09-18-2007, 12:01 AM
I managed to build a mips-TiVo-linux version of kaffe-1.1.7. I did it on a fedora box. It had to be installed native first, then cross compiled using these configure options:

% CC=mips-TiVo-linux-gcc ./configure --disable-sound --disable-alsa --prefix=/usr/local/tivo --host=mips-TiVo-linux --without-esd --without-alsa --disable-native-awt --disable-gtk-peer
% make
% sudo make install
This populated /usr/local/tivo, which I moved to the tivo.

I also had to put libgcc_s.so.1 from the toolchain build into /lib on the tivo.

Result on the tivo:

bash-2.02# /usr/local/tivo/bin/kaffe -version
java full version "kaffe-1.4.2"

kaffe VM "1.1.7"

Copyright (c) 1996-2005 Kaffe.org project contributors (please see
the source code for a full list of contributors). All rights reserved.
Portions Copyright (c) 1996-2002 Transvirtual Technologies, Inc.

The Kaffe virtual machine is free software, licensed under the terms of
the GNU General Public License. Kaffe.org is a an independent, free software
community project, not directly affiliated with Transvirtual Technologies,
Inc. Kaffe is a Trademark of Transvirtual Technologies, Inc. Kaffe comes
with ABSOLUTELY NO WARRANTY.

Engine: Just-in-time v3 Version: 1.1.7 Java Version: 1.4
Heap defaults: minimum size: 5 MB, maximum size: unlimited
Stack default size: 64 KBI didn't actually try to run any java code.

It's too big to post here, but I uploaded to mediafire. kaffe.tbz (http://www.mediafire.com/?5wzynoxtmik) is a bzip2 compressed tar ball to be unpacked in /usr/local on the tivo. libgcc_s.so.1.bz (http://www.mediafire.com/?dhtxoxjbz9g) needs to be bunziped and moved to /lib.

[Followup - unfortunately I'm getting an illegal instruction exception when I try to run even the most basic java class.

Second followup: seems to be a JIT problem. Works (albiet slowly) if I force JIT off with --with-engine=intrp when configuring the crosscompile. I'll update the link above to use the intrp verses jit version]

dburckh
09-18-2007, 01:41 AM
Thank you very much!

Java runs painfully slow without the JIT. Performance was increased 10x when Sun released HotSpot (their JIT).

It looks like they only support 1.4. What version did you compile with? That may be your JIT problem.

Another update:

It looks like a lot of time is consumed on JVM startup. I can count to 10000 in 93ms (that still sucks mind you), but it takes like 2-5 seconds to actually run.

Do you have the JIT somewhere still? I'd like to play with that too.

I've attached a little class the counts up to what ever you pass in. It's compiled with a 1.2 target VM, so it should work with most anything.

./java Test 10000

It gotta get to bed or my wife will kill me. :)

Jamie
09-18-2007, 02:13 AM
Thank you very much!

Java runs painfully slow without the JIT. Performance was increased 10x when Sun released HotSpot (their JIT).

It looks like they only support 1.4. What version did you compile with? That may be your JIT problem.I compiled with a gcc-3.3.4 mips-TiVo-linux cross compiler. Google gets some hits on similar problems other people have had with the Kaffe JIT compiler on mips.

dburckh
09-18-2007, 02:19 AM
I compiled with a gcc-3.3.4 mips-TiVo-linux cross compiler. Google gets some hits on similar problems other people have had with the Kaffe JIT compiler on mips.

I was talking about what Java version, or did you compile on the Tivo?

Jamie
09-18-2007, 02:44 AM
I was talking about what Java version, or did you compile on the Tivo?I'm not sure what you are getting at, but no Sun JDK is used in building Kaffe. It does use Jikes to compile Java to byte code.

[Sorry, I'm dense. The test program was byte-compiled on the fedora box using the Kaffe javac, which uses Jikes. I don't think it is the problem. The class file runs fine within Kaffe on the fedora box. I don't have Jikes on the tivo, so I can't compile there.]

dburckh
09-18-2007, 10:43 AM
I'm not sure what you are getting at, but no Sun JDK is used in building Kaffe. It does use Jikes to compile Java to byte code.

[Sorry, I'm dense. The test program was byte-compiled on the fedora box using the Kaffe javac, which uses Jikes. I don't think it is the problem. The class file runs fine within Kaffe on the fedora box. I don't have Jikes on the tivo, so I can't compile there.]

Ok, you are clearly far from dense. I tried to explain this to our head of development the other day and his eyes glazed over.

What you probably know already is that Java is run interpretively. What happens is the code is converted to byte codes. Bytecodes are pre-parsed instructions that are easily executed by the Java VM. Most 4 GLs work the same way (MS VB, Flash, PL SQL).

Each level of VM supports/is optimized for a certain set of instructions. What you probably don't know is that most Java version compile for VM's that are a revision or two older. For example, a 1.4 compiler by default compiles for VM version 1.2 for compatibility reasons.

The JIT (just in time) compiler compiles the byte code into machine code (via C usually). This allows Java to be portable but be nearly as fast as C.

What I was guessing was that the jikes compiler was compiling for a newer version of the VM than the JIT compiler supported (fully). Assuming it worked at one time on MIPS, you should be able to back the compiler target off to get a working version. Unfortunately, we (I) might have to recompile the classes that come with it too.

So, all that being said, do you still have the JIT compiled version?

Jamie
09-18-2007, 11:39 AM
...
What I was guessing was that the jikes compiler was compiling for a newer version of the VM than the JIT compiler supported (fully). Assuming it worked at one time on MIPS, you should be able to back the compiler target off to get a working version. Unfortunately, we (I) might have to recompile the classes that come with it too.I'm fairly certain the class files are fine. I used the kaffe javac to compile them. The same class files run fine on the x86 build of the same kaffe source tree. The man page for jikes says it defaults to a target VM of 1.4.2, which seems to match the Kaffee VM. Correct me if I am wrong, but a decent VM should notice if it is being given class files compiled for a newer VM release, right? I would expect at least a warning.


So, all that being said, do you still have the JIT compiled version?It's easy to build it again. Next step, I think, is to run through the Kaffe JIT regression tests to see if anything pops out there. They test the JIT engine at a lower level than even a simple "Hello World" java test. It's a bit of a pain to run it for the cross compiled case though, since we don't have a full development environment on the tivo.

Another thing we could do is to look at the debian-mips kaffe package to see iif JIT works there. If it does, there may be able to get some clues by looking at how that version was built. I don't think I have a debian environment on any of my tivos, but I could look at recreating that again (e.g. via this (http://www.dealdatabase.com/forum/showthread.php?t=20662)).

It looks exactly like the problem here (http://www.kaffe.org/pipermail/kaffe/2004-October/099974.html), but that thread never lead to any resolution. One possibility someone suggested to me is that the cache handling needs to be adjusted for the broadcom MIPS core. Self modifying code needs to be careful to flush the instruction and data caches after poking code into memory, and it is possible that isn't being done quite right for the Broadcom MIPS processors.

dburckh
09-18-2007, 01:30 PM
I'm fairly certain the class files are fine. I used the kaffe javac to compile them. The same class files run fine on the x86 build of the same kaffe source tree. The man page for jikes says it defaults to a target VM of 1.4.2, which seems to match the Kaffee VM. Correct me if I am wrong, but a decent VM should notice if it is being given class files compiled for a newer VM release, right? I would expect at least a warning.

Yeah it SHOULD. I'm AssUMing that the MIPS JIT didn't get updated for 1.4. That's why I attached the 1.2 target class file to test with.


It's easy to build it again. Next step, I think, is to run through the Kaffe JIT regression tests to see if anything pops out there. They test the JIT engine at a lower level than even a simple "Hello World" java test. It's a bit of a pain to run it for the cross compiled case though, since we don't have a full development environment on the tivo.

Another thing we could do is to look at the debian-mips kaffe package to see iif JIT works there. If it does, there may be able to get some clues by looking at how that version was built. I don't think I have a debian environment on any of my tivos, but I could look at recreating that again (e.g. via this (http://www.dealdatabase.com/forum/showthread.php?t=20662)).


I'd like to play with it a while. We might not need the JIT or I may be able to work around the existing JIT crash. You've done so much already just getting the non-jit version working.



It looks exactly like the problem here (http://www.kaffe.org/pipermail/kaffe/2004-October/099974.html), but that thread never lead to any resolution. One possibility someone suggested to me is that the cache handling needs to be adjusted for the broadcom MIPS core. Self modifying code needs to be careful to flush the instruction and data caches after poking code into memory, and it is possible that isn't being done quite right for the Broadcom MIPS processors.

Yeah, what you said. :) If it's easy to do, that would be great, but like I said you've done so much already.

The bigger problem with turning the Tivo into a UPNP media server (which I think would be cool) is parsing the SOAP XML. Parsing XML is not trivial on a "real" box, much less a Tivo. Further, I'm using an XML library that isn't available in 1.4.

I'm going to try to bring up the TySuiteJ web server first and see how it performs. It's a fairly trivial piece of the app. If that doesn't work, the UPNP server is right out.

Jamie
09-18-2007, 05:46 PM
Here's (http://www.mediafire.com/?6pxmsatvyot) the (broken) tarball with the mips-TiVo-linux Kaffe version compiled with JIT support.

I also attached a transcript from a gdb session trying to track down the problem running with your Test.class. The first signal is expected -- the code that tries to determine the stack boundaries purposely causes an seg fault and catches the signal. Gdb is just catching it first. The second signal (Illegal instruction) is the fault. I disassembled some code around the program counter and return address.

I'm not convinced the PC is where it should be. There was some talk in the earlier bug report I linked to about the possibility of a bad trampoline jump when jumping to the JIT generated code.

dburckh
09-18-2007, 07:06 PM
It's blowing up compiling String. I don't think I can avoid constructing a String.

It is weird, there are 2 implementations of String. A general one and a VM specific one. I can try to swap out the versions and see what happens.

Jamie
09-18-2007, 08:54 PM
It's blowing up compiling String. I don't think I can avoid constructing a String.

It is weird, there are 2 implementations of String. A general one and a VM specific one. I can try to swap out the versions and see what happens.It looks to me like it is blowing up on the first JIT method call. It's the Object init (constructor) method. Switching String classes is unlikely to change that.

dburckh
09-19-2007, 02:11 PM
I'm in right in the middle of my V3 muxer, so I want to get that done before I start playing with running TySuiteJ on the Tivo. I'm trying to make it as efficient as possible it give it a shot at running on the Tivo. Ultimately, the ability to mux Ty files in realtime is going to be the limiting factor on getting anything to work. I can alway rebuild the XML parser if I have to.

Hopefully by the end of the weekend I can report back on whether this is realistic.

This thread has gone to Cuba, so I would like to repost it under general development with your permission Jamie. I was thinking we could rar the files so we could post them on this forum.

dburckh
09-20-2007, 02:07 PM
I wrote a little tester to simulate remuxing on the Tivo. It doesn't look promising. I can demux Ty at about 100kBps. Unfortunately SD video averages 300kBps. Probably with spikes higher than that. Without the JIT it's looks unlikely that I'll be able to serve up mpg.

There still could be other uses like an HME server on the Tivo.

Jamie
09-20-2007, 02:31 PM
I wrote a little tester to simulate remuxing on the Tivo. It doesn't look promising. I can demux Ty at about 100kBps. Unfortunately SD video averages 300kBps. Probably with spikes higher than that. Without the JIT it's looks unlikely that I'll be able to serve up mpg.

There still could be other uses like an HME server on the Tivo.It sounds like it may be within reach if we can get JIT to work.

Is there any floating point math in your demuxer? Some tivo processors have an FPU and others don't. It depends on the model. For example, the NEC cpu in the Series2 has an FPU. The broadcom processors in the Series2.5 and TiVoHD do not, while the processor in the Series3 does. In any case, it may be advisable to avoid floating point math, if possible.

dburckh
09-20-2007, 02:59 PM
It's all integer/long math. The thing that killing me is that fact that I have to search for start codes (0x000001XX). I can makes some assumptions like word alignment and other things that could double speed, but I'm still not there. Other than that, I would have to "trust" the ty stream to report every video packet, which I'm pretty certain is an invalid assumption.

Jamie
09-20-2007, 03:05 PM
I know you aren't keen on JNI, since it breaks the "run anywhere" promise of Java, but in the absence of JIT, it might make sense on the tivo to code some of the key inner loops in C and call them from a JNI interface.

dburckh
09-20-2007, 03:11 PM
Yeah, C would rip this stuff up. I haven't done JNI, but I'll look into it. I'm dealing primary with byte arrays. Hopefully they are a 1 to 1.

Update:

It looks like bytes are 1:1 with chars.

Let me get the V3 muxer finished (this weekend???) and I'll see where the hot spots are. No sense in optimizing before we have all the information.

bcc
09-20-2007, 05:38 PM
Yeah, C would rip this stuff up. I haven't done JNI, but I'll look into it. I'm dealing primary with byte arrays. Hopefully they are a 1 to 1.tytompg could be built to run on the tivo directly, which would get you that result with less effort/better performance than java . Would basically be trivial to do assuming one is ok with existing mfs tools piping the recording into tytompg. I still think it's preferable to offload such processing to a general purpose box. Can't UPNP allow for pre-processing of the stream at the receiver (to convert to mpeg)?

dburckh
09-20-2007, 05:52 PM
It should be possible. We would have to get tytompg to stream to stdout or preferably socket. Since we are on unix, a fifo would probably be easiest and require the least work.

When I run tytompg locally it uses 64mb. I think that's all the Tivo's got. Anyway to get the footprint down?

bcc
09-20-2007, 06:05 PM
It should be possible. We would have to get tytompg to stream to stdout or preferably socket. Since we are on unix, a fifo would probably be easiest and require the least work.tytompg already does that (already can stream to a fifo).

When I run tytompg locally it uses 64mb. I think that's all the Tivo's got. Anyway to get the footprint down?Hmm, I didn't try to be miserly about memory usage and am pre-allocating for HD video rates. If one wants to presume low-def you can basically divide by 4. I could also tune this better I'm sure.

dburckh
09-20-2007, 06:22 PM
The 64mb was an HD number. I don't recall what the SD footprint was. Even in C, I think HD muxing is going to be hard.

I had a thought, if we are going to use tytompg, it may be more practical to just port the UPNP code to TCL. I found some TCL XML libraries. TCL is pretty slow, but so is Java without the JIT. Further, it would run on a Series 1. I'll look at the UPNP code tonight and see what the effort would be.

Update:

Short lived idea. No UDP sockets native to TCL.

bcc
09-20-2007, 06:38 PM
The 64mb was an HD number. I don't recall what the SD footprint was. Even in C, I think HD muxing is going to be hard.HD muxing is hard on under-powered CPUs but has been a solved problem with tytompg for quite some time now. Still not clear to me how UPNP requires the conversion to happen before the content is offloaded from the underpowered tivo.
I had a thought, if we are going to use tytompg, it may be more practical to just port the UPNP code to TCL.Lost me on how that's practical. tytompg is written in C, already has good performance, and doesn't introduce any TCL dependency.

dburckh
09-20-2007, 06:55 PM
HD muxing is hard on under-powered CPUs but has been a solved problem with tytompg for quite some time now. Still not clear to me how UPNP requires the conversion to happen before the content is offloaded from the underpowered tivo.

No standalone media player that I know of recognizes the ty stream format and I doubt most PC media players do (ShowTime, PowerDVD) based ones do. Vista w/TyShow might.


Lost me on how that's practical. tytompg is written in C, already has good performance, and doesn't introduce any TCL dependency.

There a lot to UPNP than just streaming media. The major components are:
1. The SSPD service broadcaster/responder (this requires UDP)
2. The EventServer
3. The DirectoryServer
4. The Http server that finally sends out the stream.

If you are interested: http://upnp.org/

bcc
09-20-2007, 07:01 PM
No standalone media player that I know of recognizes the ty stream format and I doubt most PC media players do (ShowTime, PowerDVD) based ones do. Vista w/TyShow might.And so why not have a master UPNP server do the conversion and serve up the content in standard mpeg2, leaving the tivo as some sort of dumb upnp slave that just serves ty?

dburckh
09-20-2007, 07:09 PM
You have just described exactly what the TySuiteJ UPNP server does. :)

The downside is that you need to have a dedicated box. We were trying to get it to all run on the Tivo.

The original theory was since it's all Java we should be able to run it on the Tivo, but due to a lack of JIT support it just wasn't fast enough.

bcc
09-20-2007, 07:37 PM
You have just described exactly what the TySuiteJ UPNP server does. :)

The downside is that you need to have a dedicated box. We were trying to get it to all run on the Tivo.Oh ok. Kind of like trying to overload your mail server as your file server, sql server, and so on :) My hr10-250s used to be maxed out navigating menus while recording HD.

dburckh
09-22-2007, 09:26 PM
Well, I muxed freshgear_cut.ty on my Tivo today. :)

It worked, but it took 229 seconds for a 48 second clip. Further, this clip isn't even full SD. I'm still working on it and I'm going to profile it for hot spots, but it's not real encouraging.

Even less encouraging is that it takes 250ms on my PC, 344 with the JIT off. The PC time may all be mostly I/O.

Any progress on compiling TyToMpeg for Tivo?

bcc
09-22-2007, 09:39 PM
Well, I muxed freshgear_cut.ty on my Tivo today. :)

It worked, but it took 229 seconds for a 48 second clip. Further, this clip isn't even full SD. I'm still working on it and I'm going to profile it for hot spots, but it's not real encouraging.

Even less encouraging is that it takes 250ms on my PC, 344 with the JIT off. The PC time may all be mostly I/O.

Any progress on compiling TyToMpeg for Tivo?I didn't know you if you were serious. It compiles trivially. Attached. It took 6 seconds to transcode freshgear_cut.ty from the ext2 filesystem on my s3 (not via the socket):

s3:/tmp$ date; ./tytompg freshgear_cut.ty ; date
Sun Sep 23 00:35:52 UTC 2007
tytompg: Copyright (c) 2004-2007 B.C. <bcc24x7@gmail.com>
Multiplexer version 0.19, Demuxer version 0.27
Source is freshgear_cut.ty
MPEG2 audio at 32kHz, 192 kbps, 864 bytes/sync frame
Video frame size is 352x480 Frame rate 29.970030
Video bit rate is 1470000 bits/sec., 1470 Kbit/sec.
MPEG2 audio at 32kHz, 192 kbps, 864 bytes/sync frame
Done!
Stream audio: 1351 frames (36 seconds aprox.)
Stream video: 1462 frames, 2924 fields (48 seconds aprox.)
Sun Sep 23 00:35:59 UTC 2007

drez
09-22-2007, 09:59 PM
bash-2.02# date; tytompg /var/rap.ty /var/rap.mpg; date
Sun Sep 23 00:49:59 UTC 2007
tytompg: Copyright (c) 2004-2007 B.C. <bcc24x7@gmail.com>
Multiplexer version 0.19, Demuxer version 0.27
Source is /var/rap.ty ; Destination is /var/rap.mpg
Video frame size is 480x480 Frame rate 29.970030
Video bit rate is 15000000 bits/sec., 15000 Kbit/sec.
MPEG2 audio at 48kHz, 192 kbps, 576 bytes/sync frame
MPEG2 audio at 48kHz, 192 kbps, 576 bytes/sync frame
Done!
Stream audio: 4969 frames (59 seconds aprox.)
Stream video: 3552 frames, 7154 fields (118 seconds aprox.)
Sun Sep 23 00:50:24 UTC 2007
bash-2.02#


Took 25 seconds for a 2min full SD clip with dead tuners (channels 0 and 1) on a S2 - 6.2a.



bash-2.02# date; tytompg /var/rap.ty /var/rap.mpg; date
Sun Sep 23 00:53:24 UTC 2007
tytompg: Copyright (c) 2004-2007 B.C. <bcc24x7@gmail.com>
Multiplexer version 0.19, Demuxer version 0.27
Source is /var/rap.ty ; Destination is /var/rap.mpg
Video frame size is 480x480 Frame rate 29.970030
Video bit rate is 15000000 bits/sec., 15000 Kbit/sec.
MPEG2 audio at 48kHz, 192 kbps, 576 bytes/sync frame
MPEG2 audio at 48kHz, 192 kbps, 576 bytes/sync frame
Done!
Stream audio: 4969 frames (59 seconds aprox.)
Stream video: 3552 frames, 7154 fields (118 seconds aprox.)
Sun Sep 23 00:53:56 UTC 2007
bash-2.02#



Took 32 seconds with 2 busy tuners.



bash-2.02# date; setpri fifo 10 $$; tytompg /var/rap.ty /var/rap.mpg; date
Sun Sep 23 00:56:09 UTC 2007
tytompg: Copyright (c) 2004-2007 B.C. <bcc24x7@gmail.com>
Multiplexer version 0.19, Demuxer version 0.27
Source is /var/rap.ty ; Destination is /var/rap.mpg
Video frame size is 480x480 Frame rate 29.970030
Video bit rate is 15000000 bits/sec., 15000 Kbit/sec.
MPEG2 audio at 48kHz, 192 kbps, 576 bytes/sync frame
MPEG2 audio at 48kHz, 192 kbps, 576 bytes/sync frame
Done!
Stream audio: 4969 frames (59 seconds aprox.)
Stream video: 3552 frames, 7154 fields (118 seconds aprox.)
Sun Sep 23 00:56:33 UTC 2007
bash-2.02#



Took 24 seconds with high priority and 2 busy tuners.



bash-2.02# date; setpri fifo 10 $$; tytompg /var/rap.ty /var/rap.mpg; date
Sun Sep 23 01:03:14 UTC 2007
tytompg: Copyright (c) 2004-2007 B.C. <bcc24x7@gmail.com>
Multiplexer version 0.19, Demuxer version 0.27
Source is /var/rap.ty ; Destination is /var/rap.mpg
Video frame size is 480x480 Frame rate 29.970030
Video bit rate is 15000000 bits/sec., 15000 Kbit/sec.
MPEG2 audio at 48kHz, 192 kbps, 576 bytes/sync frame
MPEG2 audio at 48kHz, 192 kbps, 576 bytes/sync frame
Done!
Stream audio: 4969 frames (59 seconds aprox.)
Stream video: 3552 frames, 7154 fields (118 seconds aprox.)
Sun Sep 23 01:03:36 UTC 2007
bash-2.02#


Took 24 seconds with high priority and dead tuners.





Vs. 2 seconds on my PC. Still, 4x real-time on the Tivo is pretty good to me. Might have a shot at HD muxing.



Kinda related question: would it be possible to add stdin support to tytompg? then I could maybe pipe a wget .ty download from mfs_ftp to tytompg and just have a .mpg?

Thanks again bcc. All your work is truly appreciated.

dburckh
09-22-2007, 10:50 PM
If nothing else, we (I) may be able to mod mfs_ftp to return an mpg stream!

drez
09-22-2007, 11:02 PM
If nothing else, we (I) may be able to mod mfs_ftp to return an mpg stream!

That'd be great too. :)

But that would have a transfer rate of ~1-1.5MB/s.

I'd still prefer to mux on my PC as I download because I can download at 4MB/s (sustained, with peaks at 4.5MB/s) with a custom kernel and "setpri fifo 10 $$", even with busy tuners. That's without gig-e.

dburckh
09-22-2007, 11:24 PM
bcc,

I tried to read from a fifo and I got this.



bash-2.02# ./tytompg -i tyin -o tyout
tytompg: Copyright (c) 2004-2007 B.C. <bcc24x7@gmail.com>
Multiplexer version 0.19, Demuxer version 0.27
Source is tyin ; Destination is tyout
MPEG2 audio at 32kHz, 192 kbps, 864 bytes/sync frame
Video frame size is 352x480 Frame rate 29.970030
Video bit rate is 1470000 bits/sec., 1470 Kbit/sec.
error: lseek failed
System error string: Illegal seek

Any ideas? I'm assuming you are seeking backwards, which probably won't work on a pipe. Is it possible to read from/write to a pipe. I'm looking for something like:

tytompg -i - > something.mpg < something.ty

Eventually the file would be replaced by a socket.

drez
09-22-2007, 11:31 PM
http://dealdatabase.com/forum/showthread.php?p=276450#post276450

(I think it might be time to start that interoperability thread. bcc: pipes do work in DOS.)

dburckh
09-22-2007, 11:41 PM
http://dealdatabase.com/forum/showthread.php?p=276450#post276450

(I think it might be time to start that interoperability thread. bcc: pipes do work in DOS.)

Pipes don't seem to work on the Tivo either. I don't think this is a DOS issue.

Update:

Arg. I need to read before I type

bcc
09-22-2007, 11:56 PM
bcc,

I tried to read from a fifo and I got this.Used to work with hdemux. I see I broke reading from a fifo with tytompg. Let me think about how to fix.
Is it possible to read from/write to a pipe.Writing still works.

bcc
09-23-2007, 12:01 AM
If nothing else, we (I) may be able to mod mfs_ftp to return an mpg stream!Before you go to far with that thought you should realize there is no need to mux mpeg on the s3s/Tivo HDs, and so I don't think such an architectural change is worth it.
We are probably over due to replace mfs_ftp with a native C file transfer daemon that hooks into TCL&mfs only where necessary.

bcc
09-23-2007, 12:11 AM
http://dealdatabase.com/forum/showthread.php?p=276450#post276450

(I think it might be time to start that interoperability thread. bcc: pipes do work in DOS.)Ya, I was wrong about DOS pipes. In my defense, when I first played with dos I could only find 1 command for which "|more" actually worked. It seemed to be implemented as a special case not as a general OS facility like under unix. Guess it's a general facility that lots of dos commands choose to ignore. Or something like that.
I'd still prefer to mux on my PC as I downloadAgree, and I think it's a better model too.

dburckh
09-23-2007, 12:17 AM
The PC is a much better machine, but by modding mfs_ftp, we would be able to remove TyTool, TySuiteJ from the equation. We (HR10-250 owners) have a one click solution. If this existed when I started, I probably wouldn't have written TySuiteJ.

Anyway. Time to watch the end of The Hunter for Red October.

drez
09-23-2007, 12:27 AM
Yeah, I still think it'd be worth adding to mfs_ftp. I think many people would use it.

But a wget | tytompg pipe would be a really great time/space saver.

bcc
09-23-2007, 12:31 AM
The PC is a much better machine, but by modding mfs_ftp, we would be able to remove TyTool, TySuiteJ from the equation. We (HR10-250 owners) have a one click solution. If this existed when I started, I probably wouldn't have written TySuiteJ.Wow, that's a lot of effort so save perhaps 1 click (1 click in ftp gui client, 1 click to drag result to a tytompg.cmd). If I fix the tytompg pipe input problem this can be turned into 1 click with a little gui work. I recall offering support of that a long time ago.

Given the problems folks have with mfs_ftp, I kind of think having the transfer as a separate step is preferable anyways.

bcc
09-23-2007, 03:24 AM
Used to work with hdemux. I see I broke reading from a fifo with tytompg. Let me think about how to fix.Writing still works.Ok, here's a fix. To re-multiplex, I basically have to do 2 passes over the beginning of the TY file. The first pass is to figure out the audio & video data rates. For HD, this can require processing a lot of chunks (50 even), before enough of both streams have been seen. For the second pass, I was re-seeking to the beginning. Now I dynamically buffer up the chunk(s) that get 2 passes.

dburckh
09-23-2007, 11:16 AM
Wow, that's a lot of effort so save perhaps 1 click (1 click in ftp gui client, 1 click to drag result to a tytompg.cmd). If I fix the tytompg pipe input problem this can be turned into 1 click with a little gui work. I recall offering support of that a long time ago.

Some of this is my industries mentality. This provides a thin client to the Tivo (IE or FireFox). It also always you to access Tivo shows with an out of the box PC. No need to reinstall a bunch of tools when you upgrade your box.



Given the problems folks have with mfs_ftp, I kind of think having the transfer as a separate step is preferable anyways.

I don't think there's anything we can do about that except to educate and simplify. Lots of people have problems setting up TyTool and TySuiteJ too.

Maybe a prebundled series 2 mfs_ftp with mfs_uberexport, mfs_import, tytompg (and whatever else)?

dburckh
09-23-2007, 02:24 PM
Here's a version of mfs_ftp (1.2.9p) with mpg support. Overlay it on an existing install. It requires the following:

mfs_ftp is/was already working
a writeable file system
you've done a mkfifo fifo in the mfs_ftp directory
you have mfs_uberexport in the mfs_ftp directory
you have tytompg MIPS in the mfs_ftp directory (thanks bcc)



ftp> open 192.168.2.202 3105
Connected to 192.168.2.202.
220 Mfs_Ftp ver 1.2.9p - {sock22} from "192.168.2.103:1538"
User (192.168.2.202:(none)):
331 User name okay, need password.
Password:
230 Running in TiVo Mode.
ftp> get "{South Park}{2000-12-06}{Fat Camp}{06.30 PM Wed Aug 29, 2007}{COM}.mpg"
200 PORT command successful.
150 About to open data connection.
226 Transfer complete.
ftp: 426711040 bytes received in 590.66Seconds 722.44Kbytes/sec.


The transfer speeds seem to be limited by mux speed.

Is there an newer mfs_ftp??? I had to fix a problem with large NowShowings

This is pretty sweat.:

mplayer ftp://192.168.2.202:3105/"{South Park}{2000-12-06}{Fat Camp}{06.30 PM Wed Aug 29, 2007}{COM}.mpg"

VLC doesn't seem to like mfs_ftp.

drez
09-23-2007, 05:02 PM
Wow, that was quick. Works as expected.

Yeah, there's a newer mfs_ftp and riley doesn't like people posting mfs_ftp.tcl.

You need to do three patches from a virgin mfs_ftp.tcl:

mfs_ftp.20070121.patch.tar.gz (http://www.dealdatabase.com/forum/attachment.php?attachmentid=6047&d=1170041056)
mfs_ftp-20070207-pasv-fix.patch.gz (http://www.dealdatabase.com/forum/attachment.php?attachmentid=6113&d=1172073748)
mfs_ftp-20070717b-ya-pasv-fix.patch.gz (http://www.dealdatabase.com/forum/attachment.php?attachmentid=6356&d=1184693896)


Busybox's patch doesn't work well so you'll have to use the patch here (http://www.dealdatabase.com/forum/attachment.php?attachmentid=4103&d=1101913110). What I do is upload patch to /, and then use "/patch".

(for anybody who stumbles into this thread in the future: )

Upload patch.gz to /


cd /
gzip -d patch.gz

Apply the 20070121, 20070207, 20070717b-ya-pasv-fix patches:

cd mfs_ftp
/patch < mfs_ftp.20070121.patch
/patch < mfs_ftp-20070207-pasv-fix.patch
/patch < mfs_ftp-20070717b-ya-pasv-fix.patch


The patches must be applied in the correct order, and you must start with an original 1.29p mfs_ftp.tcl


I'd still leave it up until you,me, or somebody else makes a patch.



Update: Since you never heard of the patches, I'm assuming you also never heard of chrised's p1.tcl and p2.tcl for mfs_ftp.

p1.tcl adds folders support so shows get grouped into their respective folder when re-uploaded or fxp'd.
http://www.dealdatabase.com/forum/showthread.php?p=246780#post246780

p2.tcl fixes mfs_ftp crashing when you have more than 200 recordings.
http://www.dealdatabase.com/forum/showthread.php?p=253232#post253232
(although chrised guesses that it might be slower, in my experience, it's the same load time.)


All you have to do to install these is put them in your mfs_ftp.tcl folder and restart mfs_ftp.

dburckh
09-23-2007, 05:33 PM
Yeah, I figured out the p2.tcl patch on my own. :)

I'll ask Riley about posting. If he wants me to pull it, I will.

At this point it's more of a proof of concept. I'd have to clean the code up a little before I would officially post it.

drez
09-23-2007, 10:27 PM
I've made some rollup patches that include dburckh's mpg output support (read what you have to do here (http://dealdatabase.com/forum/showthread.php?p=288021#post288021)) and integrate chrised's p2.tcl into mfs_ftp.tcl (since most people don't seem to know about it.) I thought about integrating chrised's p1.tcl (http://www.dealdatabase.com/forum/showthread.php?p=246780#post246780)that adds folders support but I've read that it has issues with 8.3. If you use 6.x, I highly recommend it.

The virgin patch also includes the 20070121 (http://www.dealdatabase.com/forum/showthread.php?p=273667&postcount=1152), 20070207 (http://www.dealdatabase.com/forum/showthread.php?p=276419#post276419), 20070717b-ya-pasv-fix (http://www.dealdatabase.com/forum/showthread.php?p=284467#post284467) patches. (I've included a virgin mfs_ftp.tcl in that .tgz to simplify things.)

If you're already patched up to 20070717b-ya-pasv-fix, you can use mfs_ftp-20070923-beta1-drez-from20070717.gz.

This one works with busybox patch in my tests. If there are any issues, let me know.

Thanks to dburckh for modding mfs_ftp to output mpg, bcc for making tytompg and compiling a mips version for us, Jamie for all his mfs_ftp patches and mfs-utils work, jerrymc for the original patch, and chrised for his excellent p1.tcl and p2.tcl patches.