Mfs_Ftp: extract, archive, restore & transfer recordings

[Numbski]

This is driving me totally bonkers.

I can install mfs_ftp just fine. I use the default settings, no problem.

I'm on MacOS X, on the command line. I've tried numerous iterations on this, but invariably, I get the same response from the server:

421 Service not available, remote server has closed connection.

If I look at the logs, I get numerous reasons for this, and all end in a core dump.

Here's my typical attempt to download a show:

ftp ftp://172.16.30.11:3105
220 Mfs_Ftp ver 1.2.9p - {sock32} from "172.16.30.5:53300"
331 User name okay, need password.
230 Running in TiVo Mode.
Remote system type is UNIX.
200 Type set to I
ftp> cd tmf
250 Directory change successful.
ftp> passive
Passive mode: off; fallback to active mode: off.
ftp> 27121.tmf
local: 27121.tmf remote: 27121.tmf
502 Command not implemented "EPRT |1|172.16.30.5|53302|"
421 Service not available, remote server has closed connection.


I've been beating my head against the perverbial wall for days with this. I'm a unix administrator, work with this stuff all day. I've read the readme, read any docs I can find. Something just doesn't seem right. :\

----

Okay, so I figured out that EPRT is part of the extended port command. using epsv4 turns this off. Now it's complaining that the size command isn't implemented:

502 Command not implemented "SIZE 27121.tmf"
421 Service not available, remote server has closed connection.

Here's the log

10:43:59:PM - 230 Running in TiVo Mode.
10:43:59:PM - 215 UNIX
10:43:59:PM - 502 Command not implemented "FEAT"
10:43:59:PM - 257 "/" is current directory.
10:43:59:PM - 200 Type set to I
10:44:06:PM - 250 Directory change successful.
10:44:06:PM - 257 "/tmf" is current directory.
10:44:13:PM - 502 Command not implemented "SIZE 27121.tmf"
10:44:13:PM - 200 PORT command successful.
10:44:13:PM - 150 About to open data connection.
bgerror invoked with error 
 
 " broken pipechild process exited abnormally " 

re-initializing mfs_ftp

close the current ftp connection and simply open another

"core dump" :p

info(version): 1.2.9p
info(tswv): 3.1.1c-01-2-321
info(dbl): 0
info(ithrottle): 2
info(insert_priority): 10
info(multithreaded): 0
info(saveuntil): suggestion
info(name_detail): 6
info(bjuggle): 0
info(active): 0
info(ac_interval): 1800
info(gatewayip): 127.0.0.1
info(gatewayport): 3105

[rc3105]

people, I'm not a mind reader, neither are the other folks that might be willing to help

"lots of problems" is just about useless for troubleshooting purposes


capture a level 5 log, zip and attach it to the post with your question

*a log from the client side would also be VERY usefull

sounds like you're on the way to figuring it out, a tivo side level 5 log should fill in the missing pieces

[Masnark]

sounds like you're on the way to figuring it out, a tivo side level 5 log should fill in the missing pieces

....

I set the settings.tcl and the settings at the top of MfsFtp.tcl to be:

set info(dbl) 5

That's what the FAQ said to do (afai understand). Where else should I be setting log level 5 at? The log looks pretty verbose, but I guess I still missed something.

[Numbski]

Here's the level 5 log

12:03:39:AM - setup_control_socket:
12:03:39:AM - "127,0,0,1:3105" ready for connections
12:25:54:AM - newcsocket: {sock22} "172.16.30.5:54132"
12:25:54:AM - abortcheck: "newcsocket"
12:25:54:AM - 220 Mfs_Ftp ver 1.2.9p - {sock22} from "172.16.30.5:54132"
12:26:00:AM - serverip "172.16.30.11"
12:26:00:AM - readlinefromsocket: "sock22"
12:26:00:AM - echo to verify: "USER anonymous"
12:26:00:AM - parseline:
"USER anonymous"
12:26:00:AM - USER: "anonymous"
12:26:00:AM - 331 User name okay, need password.
12:26:00:AM - readlinefromsocket: "sock22"
12:26:00:AM - echo to verify: "PASS numbski@"
12:26:00:AM - parseline:
"PASS numbski@"
12:26:00:AM - PASS: "numbski@"
12:26:00:AM - 230 Running in TiVo Mode.
12:26:00:AM - readlinefromsocket: "sock22"
12:26:00:AM - echo to verify: "SYST"
12:26:00:AM - parseline:
"SYST"
12:26:00:AM - 215 UNIX
12:26:00:AM - readlinefromsocket: "sock22"
12:26:00:AM - echo to verify: "FEAT"
12:26:00:AM - parseline:
"FEAT"
12:26:00:AM - 502 Command not implemented "FEAT"
12:26:00:AM - readlinefromsocket: "sock22"
12:26:00:AM - echo to verify: "PWD"
12:26:00:AM - parseline:
"PWD"
12:26:00:AM - PWD:
12:26:00:AM - 257 "/" is current directory.
12:26:00:AM - readlinefromsocket: "sock22"
12:26:00:AM - echo to verify: "TYPE I"
12:26:00:AM - parseline:
"TYPE I"
12:26:00:AM - 200 Type set to I
12:26:15:AM - readlinefromsocket: "sock22"
12:26:15:AM - echo to verify: "CWD tmf"
12:26:15:AM - parseline:
"CWD tmf"
12:26:15:AM - 250 Directory change successful.
12:26:15:AM - readlinefromsocket: "sock22"
12:26:16:AM - echo to verify: "PWD"
12:26:16:AM - parseline:
"PWD"
12:26:16:AM - PWD:
12:26:16:AM - 257 "/tmf" is current directory.
12:26:21:AM - readlinefromsocket: "sock22"
12:26:21:AM - echo to verify: "TYPE I"
12:26:21:AM - parseline:
"TYPE I"
12:26:21:AM - 200 Type set to I
12:26:28:AM - readlinefromsocket: "sock22"
12:26:28:AM - echo to verify: "SIZE 27121.tmf"
12:26:28:AM - parseline:
"SIZE 27121.tmf"
12:26:28:AM - 502 Command not implemented "SIZE 27121.tmf"
12:26:28:AM - readlinefromsocket: "sock22"
12:26:28:AM - echo to verify: "PORT 172,16,30,5,211,117"
12:26:28:AM - parseline:
"PORT 172,16,30,5,211,117"
12:26:28:AM - PORT 172,16,30,5,211,117
12:26:28:AM - data channel "sock23" to 172.16.30.5:54133
12:26:28:AM - 200 PORT command successful.
12:26:28:AM - readlinefromsocket: "sock22"
12:26:28:AM - echo to verify: "RETR 27121.tmf"
12:26:28:AM - parseline:
"RETR 27121.tmf"
12:26:28:AM - RETR:
"27121.tmf"
12:26:28:AM - fsid_from_filename:
"27121.tmf"
12:26:28:AM - searching $info(rec_filenames) for 
"27121"
12:26:28:AM - searching fsid list for "27121"
12:26:28:AM - found filename in fsid list
12:26:28:AM - recfsid "27121" filetype "tmf"
12:26:28:AM - 150 About to open data connection.
12:26:28:AM - rec_info_from_db: recobj{27121}
12:26:28:AM - Dragonball GT 503316480 k
12:26:28:AM - cache_xml_2_disk: recobj{27121}  force cache = "0"
12:26:28:AM - allready cached 27121.xml
12:26:28:AM - calc_tertiary_info: recobj{27121}
12:26:29:AM - build_parts: recobj{27121}
12:26:29:AM - building new lists
12:26:29:AM - 
orig list: {41102}
new list:  {41102}
12:26:29:AM - get_xml: recobj{27121}
12:26:29:AM - reading xml from disk cache
12:26:29:AM - stream_tmf: recobj{27121}
12:26:29:AM - set priorities: mfs_ftp 1 % I/O chan 1 %
bgerror invoked with error 
 
 " broken pipechild process exited abnormally " 

re-initializing mfs_ftp

close the current ftp connection and simply open another

"core dump" :p

info(version): 1.2.9p
info(tswv): 3.1.1c-01-2-321
info(dbl): 5
info(ithrottle): 2
info(insert_priority): 10
info(multithreaded): 0
info(saveuntil): suggestion
info(name_detail): 6
info(bjuggle): 0
info(active): 0
info(ac_interval): 1800
info(gatewayip): 127.0.0.1
info(gatewayport): 3105

Here's the client side commands:

220 Mfs_Ftp ver 1.2.9p - {sock22} from "172.16.30.5:54132"
331 User name okay, need password.
230 Running in TiVo Mode.
Remote system type is UNIX.
200 Type set to I
ftp> epsv4
EPSV/EPRT on IPv4 off.
ftp> cd tmf
250 Directory change successful.
ftp> bin
200 Type set to I
ftp> get 27121.tmf
local: 27121.tmf remote: 27121.tmf
200 PORT command successful.
150 About to open data connection.
421 Service not available, remote server has closed connection.
ftp: No control connection for command.

[Numbski]

Okay, I've had a moment of complete idiocy.

When I first installed MFS_FTP, I didn't follow the correct procedure (I'm thinking somehow I didn't get it from the dealdatabase boards).

That "broken pipechild" error is due to a bad binary, or in my case, an S1 binary. Anyone getting this error should refer to this post for accurate instructions for installation.

Sorry for wasting everyone's time.

[jasdjensen]

I have a Hughes HDVR2 DirecTivo

I have installed mfs_ftp 1.2.9 with the provided instructions.. I've overwritten the install with the s2bins.tar. It seems to work fine with tyExtract, Etivo, and FTP clients but I only get about 30 minutes of video out of a 1 hour ty file.

If I try to download other files, it ends up being anywhere from 23 to 32 minutes. I have a feeling others have experienced this.

For my HDVR2 is there someone who can tell me what I did wrong? or give me a step by step instruction with the files I need?

I'm not the greatest at unix commands so I may need basic instruction.


By the way TyTools works fine.. They download fully and are complete. That's what makes me think I have a problem.

Thanks in advance for your help. I tried to find a solution on my own by searching the forums but couldnt find anything, so I'm sorry if it's been answered a hundred times.

I just couldnt find it.

Thanks.

Jason

[Numbski]

There were more steps than just putting the s2bins in there, if you look at my previous post. I don't know if that's related or not, but there's a chance.

Also, a tip I've learned on searching is to search by post rather than by thread. Works wonders. YMMV.

As I've seen on other threads about searching:

Build a man a fire, keep him warm for the night. Set a man on fire, keep him warm for the rest of his life. :P

[jasdjensen]

What am I missing? I looked through your posts.. but I didnt see anything that relates to my problem

[Numbski]

That "broken pipechild" error is due to a bad binary, or in my case, an S1 binary. Anyone getting this error should refer to this post for accurate instructions for installation.

Sorry for wasting everyone's time.

and then all the stuff I did to my tivo.

http://www.numbski.net/hacks/tivo

[jasdjensen]

After installing mfs_ftp 1.2.9
installing the s2bins from your link above
manually copying the mfs_stream via ftp over the one in the directory
I get this message: Any ideas?



tivo:/var/mfs_ftp$ mfs_ftp.tcl
tivo:/var/mfs_ftp$
tivo:/var/mfs_ftp$ couldn't execute "./mfs_stream": permission denied
while executing
"open "|$info(path)/mfs_stream -s $fsid 2>/dev/null " r"
(procedure "mfs_stream_size" line 3)
invoked from within
"mfs_stream_size $part"
("foreach" body line 2)
invoked from within
"foreach part $info($fsid,parts) {
if { [catch { mfs streamsize $part } p_s ] } { set psize [mfs_stream_size $pa
rt] } else { set psize [expr [lindex..."
("uplevel" body line 9)
invoked from within
"uplevel $body"
invoked from within
"transaction {uplevel $body}"
(procedure "RetryTransaction" line 5)
invoked from within
"RetryTransaction {
set rec [db $db openid $fsid]
foreach item "State Showing Part" { set info($fsid,$item) [ntrim [dbobj $rec g
et $item]] }
fore..."
(procedure "rec_info_from_db" line 5)
invoked from within
"rec_info_from_db $fsid"
(procedure "update_rec_fsids" line 17)
invoked from within
"update_rec_fsids 1"
(procedure "init_procs" line 11)
invoked from within
"init_procs"
(file "./mfs_ftp.tcl" line 1534)

tivo:/var/mfs_ftp$

[WRabbit8]

After installing mfs_ftp 1.2.9
installing the s2bins from your link above
manually copying the mfs_stream via ftp over the one in the directory
I get this message: Any ideas?



tivo:/var/mfs_ftp$ mfs_ftp.tcl
tivo:/var/mfs_ftp$
tivo:/var/mfs_ftp$ couldn't execute "./mfs_stream": permission denied

chmod 755 mfs_stream

[montyshaw]

Hi,

I don't know if this is the right place for this question, so please feel free to move/edit this post.

I have 2 series 1 US SA Tivos, and I am writing a little utility to run on my pc to move shows recorded on one Tivo to the other. I have a question about mfs_ftp's implementation of the NLST ftp command. I am not a tcl programmer, so this might be a newbie tcl question.

I have searched thru the forum, and although I have found some references to NLST, I didn't see an answer to this (at least not that I understood). One thing that I did read about is a column number or width that is needed. Again, that didn't make sense to me either (which means nothing) because from my reading of the tcl code, I didn't see any reference to it.

From looking at mfs_ftp.tcl, it appears that the a regular expression can be sent as a parameter to the NLST command.

If I want to see all recordings for 'Lost', would I send a parameter of '{Lost}' or 'Lost'?

Assuming the bracket are sent, if I want to see all recording that start with Lost, do I send '{Lost*', '{Lost.*'? If I want to find all recordings with 'Lost' anywhere in the title, do I use '*Lost*', or '.*Lost.*'?

Are the regular expressions 'ls'-like, where '*' means match zero or more of any character, or like normal regular expressions where '*' means match zero or more of the preceeding character (or atom), i.e. '.*'

Finally, what's up with the column width/number parameter? Do I need to pass that as well?

Thanks
]Monty[

[dlang]

the regular expression matching in the current version is broken (it returns more items then should match the regex)

to answer your question you would need to put wildcards at the beginning and end of the string you want to match in almost every case. remember that everything after the date field that you see when you do a list is part of the name.

I'm running a modified version of 1.2.5 as a result. I'll try to remember to send you a copy of it when I get home tonight (it's still under the GPL unlike the current version). 1.2.5 runs into trouble when inserting shows, but is great for retreiving shows.

[montyshaw]

Hi,

It would be great to see what your 1.2.5 version looks like, so thanks for the offer.

I found a patch here that supposedly fixes the NLST regex problems. I guess I'll give it a try as well. It looks from the comments in the code of the patch that the NLST parameter in an unpatched mfs_ftp should be a regular expression, not an 'ls-like' one, which makes sense. It looks like the first part of the patch is for allowing an 'ls-like' pattern and converting it into a normal regular expression, i.e. you could send '*.t*', instead of '.*\.t.*'. The second part of the patch appends the extension to the name for the regex match. Based on that, I think the NLST implementation in mfs_ftp is against the name only (which is fine for my purposes) and takes a normal regular expression (again that is fine for my purposes).

One of the reasons I wanted to use NLST was to avoid the long time it takes to get a list back from LIST when I'm only looking for a few shows.

I guess I should have said in my first post that all of my testing is using 1.2.9P

Thanks
]Monty[

[dlang]

unfortunantly I don't think the nlst will help you much, the time taken is spent walking through the data on the tivo, not in transfering it to your machine.

I use nlst heavily becouse I use command-line ftp clients to access the tivo and make extensive use of mget and mdel to manage shows and ls with wildcards to show what matches a regex to make sure the mget/mdel is safe to use.

one other advantage of 1.2.5 is that it doesn't re-generate the file list every time it's requested. it re-generates it when the list changes (I believe it uses the same hooks that update the onscreen list) this has one bug in that it doesn't always catch when a show finishes recording and so sometimes a show will show as recording long after it has finished (making any other change to the now-showing list clears this up)

if I had the time I would try to fix the insert problem and not bother with 1.2.9p ever, but I haven't had the time to do this.