Mfs_Ftp: extract, archive, restore & transfer recordings

[tas3086]

Where did "Using new MFS_Import options..." in the log come from? That's not in any version of the mfs_ftp.tcl source I can find, and I believe those changes are causing problems. You don't want the verbose option with mfs_import unless you redirect output to a file. Otherwise the stderr output from mfs_import may confuse mfs_ftp.

The message was added to the mfs_ftp file so that I do not forget that I modifed the code to turn on the verbose output, and redirect the output to a file.

Here's a current mfs_import output (file attached).

I'm suspecting a tcp problem as mfs_import apparently is still trying to get data, and the client had timed out (timeout set to 180 seconds).

Should I try a different set of usb drivers? Any suggestions?

Again, an unmodified version of mfs-Ftp seems to freeze in the exact same way. The older data_in method never fails !

[Jamie]

The message was added to the mfs_ftp file so that I do not forget that I modifed the code to turn on the verbose output, and redirect the output to a file.
...

How was this tmf file generated? Is it a capture from mfs_ftp, or generated with other tools?

Do a "tar tv your_file.tmf" on your PC (e.g. in cygwin). Is the third part exactly 536870912 bytes long? What does "ls -l your_file.tmf" show for the length? If the file length is shorter than the combined lengths in the tar header, you'll get this kind of behavior where mfs_import is trying to read more data from the socket than what the client is sending. I think, though, that it will see the socket connect closed when the client finishes, and exit with an error rather than just hang forever.

It is possible that unreliable network drivers are an issue here. I've lost track of what hw/sw you are running, but I'm pretty sure that that the stock 3.1.5 usb drivers have been reported to be unreliable.

If you think the problem is loaded related, you can play around with the priority and rate control options in mfs_ftp. These are mfs_ftp options, but in the end they get passed down to the -p and -r options of mfs_import:

        -p <priority>       Priority  0: ts; 1-99 RT FIFO
...
        -r <ms>             rate control (throttle)
                              -'ve  : no delay (default)
                              0     : sched_yield() between chunks
                              +'ve  : # of ms to delay between chunks

Higher priority and/or a delay between chuncks might help work around the problem, if it is flakey usb drivers under load.

[tas3086]

How was this tmf file generated? Is it a capture from mfs_ftp, or generated with other tools?

Do a "tar tv your_file.tmf" on your PC (e.g. in cygwin). Is the third part exactly 536870912 bytes long? What does "ls -l your_file.tmf" show for the length? If the file length is shorter than the combined lengths in the tar header, you'll get this kind of behavior where mfs_import is trying to read more data from the socket than what the client is sending. I think, though, that it will see the socket connect closed when the client finishes, and exit with an error rather than just hang forever.

File has 8 parts. First 7 are 536870912 part 8 is:503316480
total file length 4,685,873kb long enough to get past part 3.

Originally created with mfs_ftp from 3.x system.
h10-250 6.3d system with .20 kernel and stock usb drivers.
Should I try the backport drivers?

Upload speeds mfs_import vs old data_in seem to be the same slow 600KB.
have tried upping the priority to 80. seems to help but still freezes here and there. Old data-In method is fine. Adding additional rate delay just makes things a lot slower.

Maybe I will do an IP trace and see exactly what is happening when it does freeze.

Thanks for all your help.. More later.

Just noticed, I'm running with rate 0. I'll try the rate -1 and see what happens

Update: throttle rate of -1 seems to have solved the problem. mfs_ftp Upload completed successfully. Will try one more time.

[twostop]

I had mfs_ftp working fine, but today I get the error shown below when trying to read the contents of any of the folders (e.g. "ty+"). This is on a HDVR2 running 6.2 (patches applied).

03:11:06:PM - readlinefromsocket: "sock22"
03:11:06:PM - echo to verify: "CWD /ty+"
03:11:06:PM - parseline:
"CWD /ty+"
03:11:06:PM - 250 Directory change successful.
03:11:06:PM - readlinefromsocket: "sock22"
03:11:06:PM - echo to verify: "PWD"
03:11:06:PM - parseline:
"PWD"
03:11:06:PM - PWD:
03:11:06:PM - 257 "/ty+" is current directory.
03:11:06:PM - readlinefromsocket: "sock22"
03:11:06:PM - echo to verify: "TYPE A"
03:11:06:PM - parseline:
"TYPE A"
03:11:06:PM - 200 Type set to I
03:11:06:PM - readlinefromsocket: "sock22"
03:11:06:PM - echo to verify: "PORT 192,168,1,10,4,58"
03:11:06:PM - parseline:
"PORT 192,168,1,10,4,58"
03:11:06:PM - PORT 192,168,1,10,4,58
03:11:06:PM - data channel "sock23" to 192.168.1.10:1082
03:11:06:PM - 200 PORT command successful.
03:11:06:PM - readlinefromsocket: "sock22"
03:11:06:PM - echo to verify: "LIST -aL"
03:11:06:PM - parseline:
"LIST -aL"
03:11:06:PM - NLIST: "LIST -aL" ("" == LIST command)
03:11:06:PM - 150 Opening ASCII mode data connection for file list.
03:11:06:PM - list_type is "LIST"
03:11:06:PM - update_rec_fsids: forced 1
bgerror invoked with error

 " can't scan path (0x70009)
 "

errorInfo:
can't scan path (0x70009)

    while executing
"mfs scan $dirName -start $prefix -count 50"
    ("uplevel" body line 1)
    invoked from within
"uplevel $body"
    invoked from within
"transaction {uplevel $body}"
    (procedure "RetryTransaction" line 5)
    invoked from within
"RetryTransaction {
        if { [catch {mfs scan $dirName -start $prefix -count 50} batch] } {
            global errorCode errorInfo
            if {..."
    (procedure "ForeachMfsFile" line 9)
    invoked from within
"ForeachMfsFile fsid name type "$info(mfspath)" "" {
  RetryTransaction { set rec [db $db openid $fsid] ; set State [dbobj $rec get State] }
  if { [ls..."
    (procedure "update_rec_fsids" line 5)
    invoked from within
"update_rec_fsids 1"
    (procedure "NLST" line 10)
    invoked from within
"NLST   $line   $fsock "
    ("LIST" arm line 1)
    invoked from within
"switch $cmd {
  USER  { USER  $fsock  $line }
  PASS  { PASS  $fsock  $line }
  CWD   { CWD   $args   $fsock }
  CDUP  { CDUP  $fsock }
  DELE  { DELE  $fsock $line..."
    (procedure "parseline" line 6)
    invoked from within
"parseline $line $sock "
    (procedure "readlinefromsocket" line 10)
    invoked from within
"readlinefromsocket sock22"

re-initializing mfs_ftp

close the current ftp connection and simply open another

"core dump" :p

info(version): 1.2.9p-patchlevel-20070717b
info(tswv): 6.2-01-2-151
info(dbl): 5
info(ithrottle): 2
info(insert_priority): 10
info(multithreaded): 0
info(saveuntil): suggestion
info(name_detail): 5
info(bjuggle): 0
info(active): 0
info(ac_interval): 1800
info(gatewayip): 127.0.0.1
info(gatewayport): 3105


03:11:07:PM - abortcheck: "bgerror"
03:11:07:PM - reset_control_socket:
catch close lastsock val ""
03:11:07:PM - setup_control_socket:
03:11:07:PM - "192,168,1,26:3105" ready for connections

Update: I found this patch: http://www.dealdatabase.com/forum/sh...d.php?p=245152

But now I get the following fatal error:

Dumping mempool to /tmp/BlockFailure.6357
Tmk Assertion Failure:
    DumpArenaAndBlocksAndDie, line 1472 ()
Tmk Fatal Error: Thread tivosh <6357> strayed!
Paste the following into a shell to get a backtrace...

bt -t /tvbin/tivoapp <<END_OF_BT
  read 0x2aaa8000 /lib/ld.so.1
  read 0x2ab04000 /lib/libutil.so.1
  read 0x2ab48000 /lib/libdl.so.2
  read 0x2ab8c000 /lib/libpthread.so.0
  read 0x2abe8000 /lib/libm.so.6
  read 0x2acb0000 /lib/libc.so.6
  read 0x2ae4c000 /lib/libnss_files.so.2
  read 0x2ae9c000 /lib/libnss_dns.so.2
  read 0x2aee4000 /lib/libresolv.so.2
  0x013b3e3c 0x013b3c90 0x013b3f84 0x00f58234 0x00f9312c 0x00fc579c 0x00fc5234
  0x00ff4184 0x00ff5c40 0x00fb9144 0x00f8eb18 0x00fecc98 0x00fb7a28 0x00f8eb18
  0x00fed644 0x00fb7a28 0x00f8eb18 0x00fed644 0x00fb7a28 0x00f8eb18 0x00fed644
  0x00fb7a28 0x00f8eb18 0x00f9fd3c 0x00fb7a28 0x00f8eb18 0x00fed644 0x00fb7a28
  0x00f8eb18 0x00fed644 0x00fb7a28 0x00f8eb18 0x00f8e770 0x00f91280 0x00fd37d0
  0x00fd30a0 0x0100127c 0x00fe31c8 0x00fe3714 0x00fb672c 0x00f8da78 0x00fb7a28
  0x00f8eb18 0x00fd99ec 0x00fdd23c 0x00f58150 0x00612c48 0x00403090 0x2acc13fc
END_OF_BT

Tmk Fatal Error: Thread tivosh <6357>: assertion failure

To view the blocks, run:
   $TIVO_ROOT/devbin/poolview.tcl <app-with-symbols> /tmp/BlockFailure.6357

In the UI that comes up, find your block by address (see above)
This will help you identify the type and ownership of the blocks.

[tas3086]

I had mfs_ftp working fine, but today I get the error shown below when trying to read the contents of any of the folders (e.g. "ty+"). This is on a HDVR2 running 6.2 (patches applied).

I've had some problems with smart ftp. Try filezilla at http://filezilla.sourceforge.net/ and see if that helps. Be sure that you are going to port 3105.

[tas3086]

I found a reference to some nice mfs_ftp installation documentation here: http://www.dealdatabase.com/forum/sh...5&postcount=13

Please note that I believe that the following needs to be added to the settings file. I did not see it documented.:

set info(seriescreate) 1 ;# Allow Series to be created? 0 = disabled, 1/others = enabled.

and I seem to have good success using:

set info(ithrottle) -1

[Barefooter]

Have a question regarding insertion on a 6.3D HR10. I am pulling off the shows from a 6.2 SD D*Tivo and insterting into the HR10 ( Using tmf format). The entire process works great. But folders on the HR10 won't group the shows anymore. It's like the keys to the show are lost and the Tivo doesn't know they are related. Any suggestions ?

[Soapm]

followed all the instructions from http://www.dvrpedia.com/MFS_FTP and believe I have this ready to run. This set of instructions confuses me, can someone please clarify?

Shutting Down MFS_FTP

To shutdown MFS_FTP, FTP to the TiVo (port 3105) and download the "shutdown" file.
Or, you can telnet into port 3105 and type in exit
If you kill a tivosh script that accesses the mfs db it forces a reboot. mfs_ftp has a ping_pong "heartbeat" that shuts down the program if /var/mfs_ftp/abort.txt exists. The default interval is 15 mins. Abortcheck is also called whenever a new control channel is opened. So, create a file named /var/mfs_ftp/abort.txt, wait 15 minutes, mfs_ftp will exit, then delete the file abort.txt.

Perhaps this will be more clear when I crank it up and perhaps not.

Does mfs_ftp shut down after 15 minutes of inactivity?
Will my Tivo reboot each time it shuts down?
Can I just leave it running?
What if the Tivo reboots, does that hurt anything if I don't have abort.txt?

Can I get the *****t version???

[jt1134]

folders on the HR10 won't group the shows anymore. It's like the keys to the show are lost and the Tivo doesn't know they are related. Any suggestions ?

http://dealdatabase.com/forum/showthread.php?t=51734

Does mfs_ftp shut down after 15 minutes of inactivity?
Will my Tivo reboot each time it shuts down?
Can I just leave it running?
What if the Tivo reboots, does that hurt anything if I don't have abort.txt?

Mfs_ftp doesn't shut down every 15 minutes. If you start it up, it'll run until you kill it. Creating abort.txt is just one way of shutting it down, doesn't matter if that's ever there or not. I usually just download shutdown.txt.

[PortlandPaw]

hackman automates that process and makes life a little easier.

[Soapm]

hackman automates that process and makes life a little easier.

I was hoping to hear that. Hackman is my friend. So will it appear automatically in Hackman when I start it up or should I delete the config file and let it look again?

[Soapm]

I have mfs_ftp working. However, I can't get it to start automatically when the Tivo starts because the drive has to be RW to start it. I also can turn the drive back to RO while it is running. Is this normal behavior? Should I just put the mount RW command in the author file?

[Jamie]

I have mfs_ftp working. However, I can't get it to start automatically when the Tivo starts because the drive has to be RW to start it. I also can turn the drive back to RO while it is running. Is this normal behavior? Should I just put the mount RW command in the author file?

Put the cache directory, and the port3105.log file somewhere on /var, and symbolicly link them to the root file system directory where you have mfs_ftp installed.

Either that, or just install it on var and be prepared to refresh it if var should get cleared.

[Soapm]

Put the cache directory, and the port3105.log file somewhere on /var, and symbolicly link them to the root file system directory where you have mfs_ftp installed.

Either that, or just install it on var and be prepared to refresh it if var should get cleared.

How do I undo an existing symlink? Where do they store?

I had created a link from the root to /var/mfs_ftp so I think I should undo that one first.

[jt1134]

A symlink is stored where ever you made it. Delete it in the usually manner. If you have your mfs_ftp directory in root, then you could do this:

mkdir -p /var/mfs_ftp/cache
touch /var/mfs_ftp/port.3105.log
ln -sf /var/mfs_ftp/cache /mfs_ftp/cache
ln -sf /var/mfs_ftp/port.3105.log /mfs_ftp/port.3105.log