Mfs_Ftp: extract, archive, restore & transfer recordings

[rfc]

I am having troble re-inserting tmfs from one series 2 box to another. Variously , going from A to B, it truncates the files at 256mb and keeps inserting unviewable (32 second) duration files in the "now Playing" list or 512k files going the other way. It also disconnects suddenly (as seen in the log below) I can seem to extract fine.

Here is log from AceFTP, but if someone can point me to an appropriate LOG where I can get more information to troubleshoot this, it would be appreciated.

TIA

TYPE I
200 Type set to I
PASV
227 Entering Passive Mode (192,168,1,102,12,32).
STOR {Battleplan}{2005-01-25}{Assault From the Sea}{04.00 PM Wed Jun 28, 2006}{MILT}.tmf
150 Opening BINARY mode data connection for "{{Battleplan}{2005-01-25}{Assault From the Sea}{04.00 PM Wed Jun 28, 2006}{MILT}.tmf}"
Disconnected

[helpdeskdan]

The problem I was having before seems to be occuring again. I checked my ifconfigs to see if it was a layer1/2 problem. No errors either side.

I have noticed that my tivo seems a bit touchy. When mfs_ftp crashes, and I kill the process, the whole tivo crashes & reboots. This is new. I wonder if one of the HD's is going - it is an original 14 hour. Any ideas?

Second question: I was trying to think of other ways to transfer the programs. I rarely do uploads, so downloading alone would be fine.

Many thanks,
-Dan

[helpdeskdan]

Hmmm.... it does similar things if I try to download via vstream-client. Crap, this must be a TIVO problem as it is not limited to mfs_ftp. Dang. I guess this would be the wrong forum; now where do I ask for help?

[ercdvs]

it seems that I needed to give it a LONG time before my directories were created.. To clarify the process, as is helpful every 30 pages ::

1) I am using ffmpeg to take a .avi and sucessfully create a .ty+ file (verified xml is at the end and I can play it via WMP w/ tyshow)
2) I transfer the .ty+ file to the /ty+ directory via mfs_ftp AS --- Show.ty+
3) when its on the tivo, I need to rename it to the {xx}{yy}{zzz} notation ????

I uploaded a poor quality .ty+ to the /ty directory, and it will not play via the tivo. if I attempt to delete it via the tivo, the tivo reboots.

I assume that mfs_ftp can delete this (or rather, the ftp program can?)

[Turbo3]

I updated one of my DTivo Series 1 boxes to the LBA48 Kernel and two 250GB drives then installed the standard hacks (ftp, telnet, tivowebplus, tserver, mfs_ftp, ...). Everything worked find with mfs_ftp when I initially tested it by copying a show directly from it to another Series 1 using SmartFTP. I come back a month or two later after Tivo suggestions has filled it up with 540 shows and now starting mfs_ftp results in a reboot several minutes after it has started.

I originally installed the 1.2.9p-Jamie patched version (two patch files) of mfs_ftp and included the p2.tcl patch which people have added to fix the memory problem with NLST trying to store all the directory info into one variable. Here is the start of the log file.

07:51:06:AM - sourcing settings
07:51:06:AM - sourcing p2
07:51:06:AM - versioncheck:
07:51:06:AM - init: 1.2.9p-Jamie - TiVo_OS 3.5-01-1-031

I changed dbl to 5 and it shows that the problem is with prune_cache running out of memory. I don't know tcl but it appears it is trying to load temp_list with all the file names in the cache directory which in this case was 1080 entries. I will add some debug messages to verify that this line is causing the problem.

catch { set temp_list [glob $info(path)/cache/*] }

I decided to do a test of just commenting out the call to prune_cache in init_procs to see what would happen. This fixed the auto reboot I got shortly after starting mfs_ftp.

However, when I started SmartFTP and tried to get the directory listing for the tmf directory it again rebooted the Tivo. Here is the end of the log file.

01:03:31:AM - parseline:
"LIST -aL"
01:03:31:AM - NLIST: "LIST -aL" ("" == LIST command)
01:03:31:AM - 150 Opening ASCII mode data connection for file list.
01:03:31:AM - list_type is "LIST"
01:03:31:AM - update_rec_fsids: forced 1
Dumping mempool to /tmp/BlockFailure.204

To view the blocks, run:
$TIVO_ROOT/devbin/poolview.tcl <app-with-symbols> /tmp/BlockFailure.204

The behavior of SmartFTP would make me think that the p2.tcl code did not replace the NLST code since I don't see any directory entries showing up in the SmartFTP window. But the start of the log file clearly shows p2 was processed.

07:51:06:AM - sourcing settings
07:51:06:AM - sourcing p2
07:51:06:AM - versioncheck:
07:51:06:AM - init: 1.2.9p-Jamie - TiVo_OS 3.5-01-1-031
07:51:06:AM - tivo version "3.x" mfspath set to /Recording/NowShowingByClassic
07:51:06:AM - get_tzoffset:

I will need to add more debug messages tonight to see what is going on with NLST.

I have done a few searches and did not see the problem with the prune_cache code failing, only the problem with NLST failing with p2.tcl being the fix.

I need a little guidance in how to fix prune_cache to work with large numbers of shows. The cache_xml_2_disk seems to have a "force" option to delete entries. Can we just delete the entire cache directory and start fresh each time mfs_ftp is run and would that eliminate the need for the pruch_cache call? I know next to nothing about tcl and the details of each command in mfs_ftp. I can see generally what is happening but a little help understanding the intricacies would be much appreciated. Thanks.

Ok, just reviewed what I wrote and now see that the last failure may not have been in the NLST procedure but in update_rec_fsids. I will have to have a look at that routine tonight and add more debug messages.

Turbo3

[ercdvs]

could someone give an overview of the process of which scripts are called?

I realize some of my trouble could have been a large file via a wireless connection, but I sucessfully transfered a 850mb .ty+ file to the .ty+ directory.

However, the filesize of the resulting renamed file is 0 bytes, but the xml data, and the info reported via the tivo shows the proper file size and time..

In looking at the mps files via tivoweb, I see that the particular show is under /database/orphan/xxxxxxxx

rather then what looks to be /server/xxxxxxxxx

What process actually renames and correctly registers this fiel under the proper mps database? It seems I have qute a few files under /database/orphan ... should I be worried ?

[Narf54321]

I need a little guidance in how to fix prune_cache to work with large numbers of shows. The cache_xml_2_disk seems to have a "force" option to delete entries. Can we just delete the entire cache directory and start fresh each time mfs_ftp is run and would that eliminate the need for the pruch_cache call? I know next to nothing about tcl and the details of each command in mfs_ftp.

You can shutdown mfs_ftp, manually clear out the cache directory, and start up mfs_ftp again to rebuild it.

Keep in mind that even the NLST patch creator, chrised, commented that it would only work to around 300 or so listings. One fix he made in March, 2006 (in the link above) is to try to send back one filename at a time instead of the whole shot. I don't know what assumptions the script makes about Series-1 or Series-2 units.

There was also an older thread with sanderton about SmartFTP timeouts while waiting for mfs_ftp.

[Turbo3]

Version 0.02 of the ChrisEd's patch changed the way NLST worked so it would handle an unlimited number of shows by sending back directory entries one at a time. This patch can never have a memory problem and permanently fixes the original problem ChrisEd was trying to solve.

But there are other parts of mfs_ftp that get called and also have problems with large numbers of shows and running out of memory. I have identified two. The prune_cache and now update_rec_fsids also need some reworking to handle large numbers of shows.

There is nothing wrong with ChirsEd's patch and it does as much as it can to get mfs_ftp working with large numbers of shows. We just need some additional changes in other sections of mfs_ftp.

Any usage of lappend or append could potentially overflow memory if it involves a variable on a per show bases. Problem areas must be reworked with an ChrisEd like solution of doing the operation on a per show basis if possible or coming up with some other way of working with smaller chucks of the show list.

The problems I am running into are actually before you ever get to the ChrisEd patches but the problem looks the same in that you run out of memory and Tivo reboots. So if you don't set dbl=5 and look at the logs you might think the ChrisEd patch has hit some other limit which it has not. This is the mistake I first made in thinking the p2.tcl code was never loaded.

Right now I need to understand the working of ForeachMfsFile and RetryTransaction commands before I can come up with a fix for the update_rec_fsids procedure.

Turbo3

[Turbo3]

Turns out the fix for the memory problem with large numbers of shows is to just tell the tcl interpreter (tivosh a.k.a. tivoapp) to allocate more storage for the memory pool. This is done with the TIVOSH_POOLSIZE environment variable. It must be set before starting mfs_ftp.tcl. I use this two line script in a file called doit.

export TIVOSH_POOLSIZE=2097152
/var/mfs_ftp/mfs_ftp.tcl

Then I just type ./doit in the /var/mfs_ftp directory or you could fully qualify the path and type it from any where.

I get a full directory listing of 538 files in just 27 seconds using the p2.tcl patch and SmartFTP.

The default pool0 size seems to be 1458176 bytes (0x164000) which is just a little too small for the number of files I have. I started with 3megs which is too much and decided on 2 megs (2097152 = 0x200000 bytes). This leaves me with around 40000 bytes to spare.

I added some code into mfs_ftp.tcl to log the memory statistics each time NLST is called so I can easily spot any future problems.

So the idea that there is some type of limit on variable size is not really correct. There is a fixed amount of storage allocated when a tcl program is started and the tcl variables must fit in the amount or the amount must be increased.

You probably don't need the p2.tcl patch with this increase to the available memory. I also get the feeling it runs faster with more memory based on the 27 seconds for 538 files which seems really fast.

I was surprised at the amount of information there is on all the Tivo specific tcl commands but it turns out I did not need to use any of it except the "pool pool0 size" and "pool pool0 dump" commands.

Turbo3

[ciper]

I am experiencing a new problem that I need help with. MFS_FTP is crashing when I try to view the contents of a directory. I ftp to tivo:3105 and the root shows fine. I then click on TY and it crashes.


Havn't visited the forum in a while, glad to see others have the same problem as I did regarding large numbers of recordings Ive always resorted to deleting recordings until the program wouldnt crash. I will try the new enviroment variable.

Edit: Is it possible to have the application crash and not reboot the Tivo? It takes aproximately 4 minutes for my Tivo to boot so its a pain in the ass while troubleshooting.

[cheer]

No -- any program that has its hooks in mfs like that will likely trigger a reboot if it crashes.

Your best bet is to crank up the logging level and post a log.

[whackit]

I'm having a problem with inserts stalling. My log reads the following

bgerror invoked with error

" error reading "sock24": connection timed out "

I am using fetch on the mac (recommended client) and passive mode. Any suggestions?

[labbie48]

Having the following problem using tmf2ty after extraction with mfs_ftp. All parts are processed except the last part which fails. This happens on both a HDVR2 and a HR10-250. Both have encryption disabled. The following is the error I receive.

E:\Ty>tmf2ty Jailbait.tmf
00:00 - entering init
00:00 - filename is "Jailbait.tmf"
00:00 - file root is "Jailbait"
00:00 - file tail is "Jailbait.tmf"
00:00 - root tail is "Jailbait"
input filename is "Jailbait.tmf"
00:00 - entering getxml
00:00 - entering parsexml
00:00 - listing info parsed from xml
00:00 - Title is "The Shield"
00:00 - EpisodeTitle is "Jailbait"
00:00 - Description is "The Strike Team figures out its next step concerning Int
ernal Affairs' investigation of Lem; Tina goes under cover to bust a sex traffic
king ring; Claudette reopens one of Dutch's cases."
00:00 - Date is "13173"
00:00 - Time is "10800"
00:00 - OriginalAirDate is "13172"
00:00 - CallSign is "FX"
00:00 - Name is "FX"
00:00 - StartDate is "13173"
00:00 - StartTime is "10798"
00:00 - Duration is "3840"
00:00 - entering setinfo
00:00 - Date & Time is "01/24/2006 10:00PM"
00:00 - the recording actually started at "01-24-2006 09.59 PM"
00:00 - and the original air date is "01-23-2006"
00:00 - notme is ": {"} < >"
00:00 - entering build_text_from_xml
00:00 - entering uptmf2ty
00:00 - fname4 is "part"
00:00 - creating "c:/extracts/The Shield/{Jailbait}{01-23-2006}{FX}"
00:00 - starting segment part00.ty
..........100 ..........200 ..........300 ..........400 ..........500
..........600 ..........700 ..........800 ..........900 ..........1000
..........1100 ..........1200 ..........1300 ..........1400 ..........1500
..........1600 ..........1700 ..........1800 ..........1900 ..........2000
..........2100 ..........2200 ..........2300 ..........2400 ..........2500
..........2600 ..........2700 ..........2800 ..........2900 ..........3000
..........3100 ..........3200 ..........3300 ..........3400 ..........3500
..........3600 ..........3700 ..........3800 ..........3900 ..........4000
..........4100

00:10 - fname4 is "part"
00:10 - creating "c:/extracts/The Shield/{Jailbait}{01-23-2006}{FX}"
00:10 - starting segment part01.ty
..........4200 ..........4300 ..........4400 ..........4500 ..........4600
..........4700 ..........4800 ..........4900 ..........5000 ..........5100
..........5200 ..........5300 ..........5400 ..........5500 ..........5600
..........5700 ..........5800 ..........5900 ..........6000 ..........6100
..........6200 ..........6300 ..........6400 ..........6500 ..........6600
..........6700 ..........6800 ..........6900 ..........7000 ..........7100
..........7200 ..........7300 ..........7400 ..........7500 ..........7600
.........

00:33 - end of file or invalid segment - leaving tmf2fsid
00:33 - entering weightgain
00:33 - entering cleanup

Anyone run into this before or am I doing something wrong? I've tried Filezilla and Ftp Voyager on an XP box. Not using passive mode for transfer so at this point I'm stumped.

[cheer]

I assume the reason you're doing this is because you're archiving the .tmf files? (Otherwise you'd just download in .ty format.)

I've encountered errors with tmf2ty, but I didn't really bother troubleshooting. I just renamed the .tmf to .tar, extracted all the part files, and used copy /b to combine them.

[labbie48]

I've done that and it works but I can't understand why this format doesn't work. I've used:

/B C:\part00.ty+C:\part01.ty+C:\part02.ty c:\combined.ty

and it works fine but tmf2ty just craps out on the last part and I can't figure this out. Must be the engineer in me, if it doesn't work I like to know why. Oh well, back to the drawing boards....