MovieLoader: HME meets MFS_FTP

[wundernaut]

At 330 programs, it's possible that it could indeed take 30 seconds or more to return a listing. Have you verified this by connecting to your mfs_ftp server on your TiVo via a standard ftp client program, and done a directory ("ls -l" or similar) listing to see how long it takes? If you don't know how, just ftp to your TiVo's IP address at port 3105.

If that's what is going on, I still don't know how to increase the timeout. A quick Google of "java.net.SocketTimeoutException: Read timed out" didn't reveal an easy way to fix the timeout. Note there are two types of timeouts you may be dealing with. One is a possible timeout of the overall directory listing. Another is a timeout due to no data being returned by the ftp server in the middle of the directory listing. Unfortunately, the way mfs_ftp is written it builds a list internally of the entire directory, and then sends it to the client in one long burst. As a result, it may send no data at all to the client for a couple of minutes, which causes the client to think the server has timed out.

It's likely this latter issue is the cause of your problems. From my cursory search, it doesn't look like there's an easy way to increase the timeout in the .properties file. If you know java (I don't), supposedly the MovieLoader.jar file contains the java source code and you can increase the timeout there with the proper call to whatever handles the ftp client stuff. The Google search I did before gave some clues you might want to look into.

The alternative is to attack it from the mfs_ftp side. However, there's no easy way to get mfs_ftp to send data to the client as it reads the directory; this behavior is deeply embedded into the tcl code. A long time ago, somebody did post a patch to mfs_ftp that reportedly did send data to the client as it was generated. I've never tried it and I don't know if it works, and you'll have to go through the main mfs_ftp support thread to find it.

Neither MovieLoader nor mfs_ftp.tcl are being actively supported currently as far as I can tell. The simplest fix may be to go through your 330 programs, and archive off what you don't immediately need to a disk somewhere so the directory listing takes less than 30 seconds on your TiVo. If this archive disk is on the PC running MovieLoader, the beauty of MovieLoader is that you can easily move any program you want to view from your archive disk back to your TiVo. If you have a NAS that supports a ftp server with a passive mode, you might even be able to get MovieLoader to transfer programs between it and your TiVo, though that's a hit or miss deal.

[jkozee]

Here's a patched version of 0.17 to allow for filenames that include the FSID as an additional item.

[Roger Dylan]

Here's a patched version of 0.17 to allow for filenames that include the FSID as an additional item.

I am seeing no parsing errors.

I am seing some other errors on MovieLoader startup that I didn't notice before, but they may have been so buried in the parsing errors that I just didn't see them. Or, coincidentally my Java (on a Mac) updated today and that might be it, something new. No harm from these errors noted as yet.

java.io.IOException: reading from an output buffer
at org.apache.tools.tar.TarBuffer.readRecord(Unknown Source)
at org.apache.tools.tar.TarInputStream.getNextEntry(Unknown Source)
at FtpMediaFile.update(Unknown Source)
at FtpMediaFolder.update(Unknown Source)
at VirtualMediaFolder.update(Unknown Source)
at CacheUpdater.perform(Unknown Source)
at MovieLoader$MovieLoaderFactory.init(Unknown Source)
at com.tivo.hme.sdk.Factory.initFactory(Factory.java:628)
at com.tivo.hme.sdk.Application.getAppFactory(Application.java:1495)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.tivo.hme.host.sample.Main.createFactory(Main.java:390)
at com.tivo.hme.host.sample.Main.<init>(Main.java:151)
at com.tivo.hme.host.sample.Main.<init>(Main.java:60)
at com.tivo.hme.host.sample.Main.main(Main.java:534)
at MovieLoader.main(Unknown Source)

Thanks.

[Smee]

jkozee,

Thanks. I was able to set up my environment last night, but you beat me to the change. :-) I will say this, using tyftpd instead of mfs_ftp, has resulted in far less issues since I started using it. I'll let you know if the date parsing issues go away.

Thanks..

Smee

[Smee]

I am seeing no parsing errors.
I am seing some other errors on MovieLoader startup that I didn't notice before, but they may have been so buried in the parsing errors that I just didn't see them. Or, coincidentally my Java (on a Mac) updated today and that might be it, something new. No harm from these errors noted as yet.

I just started up MovieLoader that jkozee uploaded and am no longer seeing the parse errors I was getting with tyftpd.

Take care...

Smee

[Smee]

On another subject, I have MovieLoader installed on a WinXP machine which has jre7 installed. Once MovieLoader finishes caching all of my TiVos for programming, the menus start acting all weird. As I scroll the listings, items will appear and disappear below the selection. Also, I go to the transfer queue and open an item that is or has been transferred to see the ftp log for that item and then hit channel down on the remote, the log will become garbled. Like if someone took a comb and combed the screen. The log of coarse becomes unreadable. I also went ahead and tried Movie loader with jre6 and jre1.5.0_06 but the effect was the same.

I looked everywhere but don't see this issue reported by anyone else. Does anyone have any ideas or does what I describe even make sense?

Thanks...

Smee

[RandC]

Also, I go to the transfer queue and open an item that is or has been transferred to see the ftp log for that item and then hit channel down on the remote, the log will become garbled. Like if someone took a comb and combed the screen. The log of coarse becomes unreadable.

I looked everywhere but don't see this issue reported by anyone else. Does anyone have any ideas or does what I describe even make sense?

Thanks...

Smee

Smee, I have seen this issue before on my machines. Are you running one of the caller ID programs? or are you calling any text to screen command while booting up reporting rc.sysint.author progress?

[Smee]

Smee, I have seen this issue before on my machines. Are you running one of the caller ID programs? or are you calling any text to screen command while booting up reporting rc.sysint.author progress?

As a matter of fact I am. I just found this post which talked about it. Found it by accident too. I run yac on all my TiVos & PCs. There doesn't seem to be a solution out there either that I found other quit using yac or elseed. I was thinking that trying the -Z switch on yac, but it says its an invalid switch even though it shows up in -h. Surprised this issue hasn't come up more often. Do you know of any solutions?

Thanks..

Smee

[RandC]

Do you know of any solutions?

Thanks..

Smee

none other than not using yac unfortunately

[jkozee]

Here's a patched version of 0.17 to allow for filenames that include the FSID as an additional item and adds tmf+ file handling.

[crashHD]

does this version do tmf+ only, or is there a way to tell it whether you want it to use tmf or tmf+?

[jkozee]

As described, this version "adds tmf+ file handling". This means that when it encounters a tmf+ file type, it will recognize it as a movie and know how to extract out the xml data to build a description for the show. The .properties file decides what path to follow on the ftp server so using tyftpd -p 3105:
FriendlyName=ftp://tivo_ip:3105/tmf/ will serve tmf only
FriendlyName=ftp://tivo_ip:3105/tmf+/ will serve tmf+ only
FriendlyName=ftp://tivo_ip:3105/ will serve tmf and tmf+ (ty directory is attempted but will not work on tyftpd because it does not support REST, so it cannot get the xml file from it)

You could also do this and control what directories are listed in tyftpd by using the -m or -M flags:
FriendlyName=ftp://tivo_ip:3105/

[djl]

I know this was a while back, but you might want to try NCID with the ncid-fly module. You need to install rsyslog as well, but it will then watch for HME activity and will not write to the screen if any is detected. The output is a lot nicer than elseed, as well.

As a matter of fact I am. I just found this post which talked about it. Found it by accident too. I run yac on all my TiVos & PCs. There doesn't seem to be a solution out there either that I found other quit using yac or elseed. I was thinking that trying the -Z switch on yac, but it says its an invalid switch even though it shows up in -h. Surprised this issue hasn't come up more often. Do you know of any solutions?

Thanks..

Smee

[darrin75]

I know this is old thread but why doesnt movieloader use some tivoserver tools such as ffmpeg for on the fly conversion of files such as wma,avi, etc. Surpised someone has not done this already.

[wundernaut]

I know this is old thread but why doesnt movieloader use some tivoserver tools such as ffmpeg for on the fly conversion of files such as wma,avi, etc. Surpised someone has not done this already.

First off, the creator of Movieloader, gfb107, no longer supports it after spending many years developing it and then moving on to other things. It just wasn't a priority for him to add conversion. Nobody else has picked up the code and it hasn't seen active development for years. It's still quite useful, though.

Mostly Movieloader is for moving tivo files around. I use it mainly for managing moving files between two different TiVos, so conversion on the fly wouldn't help me. Certainly if you move files to a disk on your PC conversion has its uses, but when I need to do that I just use tytompg-1.1. It's fast and it has minimal loss of quality (basically it just removes the extra stuff TiVo added to mpeg-2). While mpeg-2 isn't the most compact video codec, storage is cheap these days and most devices still play mpeg-2.