PDA

View Full Version : How do I get mget to work on mfs_ftp ?



tiver
10-12-2008, 04:47 PM
This shouldn't be difficult:


ftp>
ftp>
ftp> cd ts
250 Directory change successful.
ftp>
ftp> bin
200 Type set to I
ftp> prompt
Interactive mode off.
ftp> hash
Hash mark printing on (1024 bytes/hash mark).
ftp>
ftp> mget *
Command not implemented "EPSV"
ftp>
ftp>

(and also disconnects me there)


Ok, fine, I will toggle EPSV off:

ftp>
ftp>
ftp> cd ts
250 Directory change successful.
ftp>
ftp> epsv
EPSV/EPRT on IPv4 off.
ftp> bin
200 Type set to I
ftp> hash
Hash mark printing on (1024 bytes/hash mark).
ftp> prompt
Interactive mode off.
ftp>
ftp>
ftp> mget *
ftp>


and ... disconnected.

Let's try one more time with a more specific mask:


ftp>
ftp>
ftp> cd ts
250 Directory change successful.
ftp>
ftp> epsv
EPSV/EPRT on IPv4 off.
ftp> bin
200 Type set to I
ftp> hash
Hash mark printing on (1024 bytes/hash mark).
ftp> prompt
Interactive mode off.
ftp>
ftp> mget {*
ftp>
ftp>
ftp>


and ... disconnected again.

So, how does ftp work in tivo-land ? Perhaps we should call it something besides ftp if it isn't going to function like actual ftp ?

lrhorer
10-12-2008, 07:18 PM
It works just fine, for me. Are you sure you have write permissions for the system from which you are starting the ftp session? Also, I presume you are starting the ftp session from a PC, or was this an attempt to mget files to the TiVo from elsewhere?

tiver
10-13-2008, 01:14 PM
Yes, I am on a unix pc and I run command line ftp to log into the tivo.

Once logged in over ftp to the tivo, I am attempting to "mget *".

I do indeed have write permissions for the local destination and it is mounted read-write.

I can successfully "get", so all is well structurally ... I just can't "mget".

Jamie
10-13-2008, 03:26 PM
You might want to look at the server side log (port.3105.log) to see why you are getting disconnected. It could be the PASV bug, for example, if you aren't patched for it.

tiver
10-13-2008, 05:06 PM
jamie:

If it is the "PASV bug" is there a workaround for it ?

tiver
10-13-2008, 05:08 PM
ok here are the logs - definitely an error of some kind:


08:38:30:PM - 226 Transfer complete.
08:38:40:PM - 227 Entering Passive Mode (192,168,0,11,12,32).
08:38:40:PM - 150 Opening ASCII mode data connection for file list.
08:38:42:PM - updating cached recording info
.....................................................................

.....................................................................

bgerror invoked with error

" couldn't compile regular expression pattern: quantifier operand invalid "

errorInfo:
couldn't compile regular expression pattern: quantifier operand invalid
while executing
"regexp -nocase $matchstr $info($fsid,rec_filename)"
(procedure "NLST" line 19)
invoked from within
"NLST $line $fsock "
("NLST" arm line 1)
invoked from within
"switch $cmd {
USER { USER $fsock $line }
PASS { PASS $fsock $line }
CWD { CWD $args $fsock }
CDUP { CDUP $fsock }
DELE { DELE $fsock $line..."
(procedure "parseline" line 6)
invoked from within
"parseline $line $sock "
(procedure "readlinefromsocket" line 10)
invoked from within
"readlinefromsocket sock24"

re-initializing mfs_ftp

close the current ftp connection and simply open another

"core dump" :p

info(version): 1.2.9p-patchlevel-20080512
info(tswv): 9.4-01-2-648
info(dbl): 0
info(ithrottle): 2
info(insert_priority): 10
info(multithreaded): 0
info(saveuntil): suggestion
info(name_detail): 5
info(bjuggle): 0
info(active): 0
info(ac_interval): 1800
info(gatewayip): 127.0.0.1
info(gatewayport): 3105

Jamie
10-13-2008, 06:48 PM
The PASV bug is fixed in patches that have been posted. You version says "patchlevel-20080512", so you probably have the PASV fixes.

Looks to me like it is treating the "{*" wildcard string you are passing in as a regular expression, where "{" has a special meaning and would need to be escaped.

tiver
10-13-2008, 11:29 PM
ok, but it also fails with:

mget *
mget *M* (matches all files because they all say AM or PM)
mget *ts
mget *.ts

Here is error output with a plain old "mget *"


08:27:54:PM - 226 Transfer complete.
08:27:57:PM - 200 Type set to I
08:28:02:PM - 200 Type set to I
08:28:02:PM - 227 Entering Passive Mode (192,168,0,11,12,32).
08:28:02:PM - 150 Opening ASCII mode data connection for file list.
08:28:07:PM - updating cached recording info
..................................................................

..................................................................

bgerror invoked with error

" couldn't compile regular expression pattern: quantifier operand invalid "

errorInfo:
couldn't compile regular expression pattern: quantifier operand invalid
while executing
"regexp -nocase $matchstr $info($fsid,rec_filename)"
(procedure "NLST" line 19)
invoked from within
"NLST $line $fsock "
("NLST" arm line 1)
invoked from within
"switch $cmd {
USER { USER $fsock $line }
PASS { PASS $fsock $line }
CWD { CWD $args $fsock }
CDUP { CDUP $fsock }
DELE { DELE $fsock $line..."
(procedure "parseline" line 6)
invoked from within
"parseline $line $sock "
(procedure "readlinefromsocket" line 10)
invoked from within
"readlinefromsocket sock24"

re-initializing mfs_ftp

close the current ftp connection and simply open another

"core dump" :p

info(version): 1.2.9p-patchlevel-20080512
info(tswv): 9.4-01-2-648
info(dbl): 0
info(ithrottle): 2
info(insert_priority): 10
info(multithreaded): 0
info(saveuntil): suggestion
info(name_detail): 5
info(bjuggle): 0
info(active): 0
info(ac_interval): 1800
info(gatewayip): 127.0.0.1
info(gatewayport): 3105



Comments ? Additional information - I am runnnig with that s3tots code that presents .ts files directly to mfs_ftp - so you can download real .ts files and not convert them after the fact. I wonder if that is breaking things ?

tiver
10-23-2008, 10:23 AM
ok then. craptastic.

Jamie
10-23-2008, 12:47 PM
mfs_ftp is unsupported. If you want a supported tool, you need to look elsewhere.

I can't reproduce this error. If you want it fixed, you may need to roll up your sleeves and do some TCL debugging. It looks like it is complaining about the regular expression in $matchstr. Perhaps you can print it out before the failure so we can see what it is?

tiver
10-31-2008, 12:20 AM
ok, here is the entire log from a full ftp session with attempted mget:


info(gatewayip): 127.0.0.1
info(gatewayport): 3105


catch close lastsock val "can not find channel named "sock24""
04:51:47:PM - updating cached recording info
...................................................................

...................................................................

^M
^M
09:18:10:PM - 220 Mfs_Ftp ver 1.2.9p-patchlevel-20080512 - {sock24} from "192.168.0.10:50668"
09:18:14:PM - 331 User name okay, need password.
09:18:14:PM - 230 Running in TiVo Mode.
09:18:14:PM - 215 UNIX
09:18:14:PM - 502 Command not implemented "FEAT"
09:18:14:PM - 257 "/" is current directory.
09:18:16:PM - 250 Directory change successful.
09:18:16:PM - 257 "/ts" is current directory.
09:18:21:PM - 502 Command not implemented "EPSV"
09:18:21:PM - 227 Entering Passive Mode (192,168,0,11,12,32).
09:18:21:PM - 150 Opening ASCII mode data connection for file list.
09:18:23:PM - updating cached recording info
...................................................................

...................................................................

09:18:34:PM - 226 Transfer complete.
09:18:39:PM - 200 Type set to I
09:18:49:PM - 200 Type set to I
09:18:49:PM - 227 Entering Passive Mode (192,168,0,11,12,32).
09:18:49:PM - 150 Opening ASCII mode data connection for file list.
09:18:50:PM - updating cached recording info
...................................................................

...................................................................

bgerror invoked with error

" couldn't compile regular expression pattern: quantifier operand invalid "

errorInfo:
couldn't compile regular expression pattern: quantifier operand invalid
while executing
"regexp -nocase $matchstr $info($fsid,rec_filename)"
(procedure "NLST" line 19)
invoked from within
"NLST $line $fsock "
("NLST" arm line 1)
invoked from within
"switch $cmd {
USER { USER $fsock $line }
PASS { PASS $fsock $line }
CWD { CWD $args $fsock }
CDUP { CDUP $fsock }
DELE { DELE $fsock $line..."
(procedure "parseline" line 6)
invoked from within
"parseline $line $sock "
(procedure "readlinefromsocket" line 10)
invoked from within
"readlinefromsocket sock24"

re-initializing mfs_ftp

close the current ftp connection and simply open another

"core dump" :p

info(version): 1.2.9p-patchlevel-20080512
info(tswv): 9.4-01-2-648
info(dbl): 0
info(ithrottle): 2
info(insert_priority): 10
info(multithreaded): 0
info(saveuntil): suggestion
info(name_detail): 5
info(bjuggle): 0
info(active): 0
info(ac_interval): 1800
info(gatewayip): 127.0.0.1
info(gatewayport): 3105


catch close lastsock val ""



that is the entire log. On the client side, it looked like this:


#ftp 192.168.0.11 3105
Connected to 192.168.0.11.
220 Mfs_Ftp ver 1.2.9p-patchlevel-20080512 - {sock24} from "192.168.0.10:55638"
Name (192.168.0.11:root):
331 User name okay, need password.
Password:
230 Running in TiVo Mode.
Remote system type is UNIX.
ftp> cd ts
250 Directory change successful.
ftp> bin
200 Type set to I
ftp> hash
Hash mark printing on (1024 bytes/hash mark).
ftp> prompt
Interactive mode off.
ftp> mget *
Command not implemented "EPSV"
ftp>
ftp>
(disconnected at this point)


So, does this full log give any hints ?

Many thanks.

Jamie
10-31-2008, 12:25 AM
So, does this full log give any hints ?Not really. It might give more if you crank the debug level "info(dbl)" up to 5.

What's the value of $matchstr at the point of the failure?

tiver
10-31-2008, 10:18 AM
WHere are you setting this debuglevel for mfs_ftp ?

I was going to just change syslog.conf, but that does not seem to be where that debug level is set ...

thanks.

Jamie
10-31-2008, 10:49 AM
I believe it comes from settings.tcl in the mfs_ftp directory.

tiver
11-03-2008, 11:54 AM
Ok.

Unfortunately, you can't "mount -uw /" on linux like you can on FreeBSD to change a read-only filesystem to read-write on the fly.

So I am having trouble editing settings.tcl because my '/' is read-only.

I am thinking of killing mfs_ftp and restarting it on the command line with an alternate config file that I create in /var ... any tips on how to do this ?

killall mfs_ftp

then ...

??

tiver
11-03-2008, 12:41 PM
nevermind, I got it.

Ok, here is the level 5 debug output:



10:36:50:AM - readlinefromsocket: "sock24"
10:36:50:AM - echo to verify: "TYPE I"
10:36:50:AM - parseline:
"TYPE I"
10:36:50:AM - 200 Type set to I
10:40:01:AM - readlinefromsocket: "sock24"
10:40:01:AM - echo to verify: "TYPE A"
10:40:01:AM - parseline:
"TYPE A"
10:40:01:AM - 200 Type set to I
10:40:01:AM - readlinefromsocket: "sock24"
10:40:01:AM - echo to verify: "EPSV"
10:40:01:AM - parseline:
"EPSV"
10:40:01:AM - 502 Command not implemented "EPSV"
10:40:01:AM - readlinefromsocket: "sock24"
10:40:01:AM - echo to verify: "PASV"
10:40:01:AM - parseline:
"PASV"
10:40:01:AM - PASV: closing info(dc) just in case
10:40:01:AM - catch close info(dc) "can't read "info(dc)": no such element in array"
10:40:01:AM - closing pdcs just in case
10:40:01:AM - errval "can't read "pdcs": no such variable"
10:40:01:AM - attempting to setup callback
10:40:01:AM - passive data channel "192,168,0,11:3104"
10:40:01:AM - 227 Entering Passive Mode (192,168,0,11,12,32).
10:40:02:AM - newpdc: open data channel sock26 - connect from "192.168.0.10:49788"
10:40:02:AM - closing server data socket "sock25" to new connections
10:40:02:AM - errval ""
10:40:02:AM - passive data channel info(dc) set to sock26
10:40:02:AM - readlinefromsocket: "sock24"
10:40:02:AM - echo to verify: "NLST *"
10:40:02:AM - parseline:
"NLST *"
10:40:02:AM - NLIST: "NLST *" ("" == LIST command)
10:40:02:AM - 150 Opening ASCII mode data connection for file list.
10:40:02:AM - list_type is "NLST"
10:40:02:AM - update_rec_fsids: forced 1
10:40:04:AM - updating cached recording info
...................................................................

10:40:05:AM - build_rec_filenames:
...................................................................

bgerror invoked with error

" couldn't compile regular expression pattern: quantifier operand invalid "

errorInfo:
couldn't compile regular expression pattern: quantifier operand invalid
while executing
"regexp -nocase $matchstr $info($fsid,rec_filename)"
(procedure "NLST" line 19)
invoked from within
"NLST $line $fsock "
("NLST" arm line 1)
invoked from within
"switch $cmd {
USER { USER $fsock $line }
PASS { PASS $fsock $line }
CWD { CWD $args $fsock }
CDUP { CDUP $fsock }
DELE { DELE $fsock $line..."
(procedure "parseline" line 6)
invoked from within
"parseline $line $sock "
(procedure "readlinefromsocket" line 10)
invoked from within
"readlinefromsocket sock24"

re-initializing mfs_ftp

close the current ftp connection and simply open another

"core dump" :p

info(version): 1.2.9p-patchlevel-20080512
info(tswv): 9.4-01-2-648
info(dbl): 5
info(ithrottle): 2
info(insert_priority): 10
info(multithreaded): 0
info(saveuntil): suggestion
info(name_detail): 5
info(bjuggle): 0
info(active): 0
info(ac_interval): 1800
info(gatewayip): 127.0.0.1
info(gatewayport): 3105


10:40:05:AM - abortcheck: "bgerror"
10:40:05:AM - reset_control_socket:
catch close lastsock val ""
10:40:05:AM - setup_control_socket:
10:40:05:AM - "192,168,0,11:3105" ready for connections


So that's that. Any good info there ? :)

phdeez
11-03-2008, 12:43 PM
...Unfortunately, you can't "mount -uw /" on linux like you can on FreeBSD to change a read-only filesystem to read-write on the fly.

mount -o remount,rw /
mount -o remount,ro / To return back to a RO root


...
I am thinking of killing mfs_ftp and restarting it on the command line with an alternate config file that I create in /var ... any tips on how to do this ?

killall mfs_ftp...

Easier to reboot system?

phdeez
11-03-2008, 02:55 PM
1) Is this a TivoHD?
1a) If so, are you using the 64bit mfs utils?
2) Disable PASV
3) Try using a different FTP client... I was using SecureFX and couldn't get anything working... using a different client now and it works like a charm. [Filezilla set to *1* connection]

mike_s
11-03-2008, 08:30 PM
..........

tiver
11-06-2008, 11:04 AM
1) Is this a TivoHD?
1a) If so, are you using the 64bit mfs utils?
2) Disable PASV
3) Try using a different FTP client... I was using SecureFX and couldn't get anything working... using a different client now and it works like a charm. [Filezilla set to *1* connection]

This is an s3 tivo.

The problem persists even if I disable PASV. That was the first thing I tried, of course :)

The ftp client I am using is:

/usr/bin/ftp

The standard FreeBSD userland ftp client. There is nothing more stock or standard than this.


# file mfs_streams
bash: file: command not found


How would you check the binaries for 32 vs. 64-bit ? I don't seem to have the 'file' command (!)

Thanks.

Jamie
11-06-2008, 03:10 PM
I tracked this down. If you want to use the NLST ftp command with a pattern, it has to be a legal TCL regular expression. For example, ".*' rather than just a bare "*". In other words, NLST is expecting a regular expression, not a glob pattern. Here's a reference on tcl regular expressions: link (http://tmml.sourceforge.net/doc/tcl/re_syntax.html).

I'm sure it is possible to rewrite mfs_ftp.tcl to fix this. I'm not feeling terribly inspired to do it myself. If someone else wants to do it, I'd look at using string match (http://tmml.sourceforge.net/doc/tcl/string.html) instead of regexp (http://tmml.sourceforge.net/doc/tcl/regexp.html).

tiver
11-07-2008, 12:11 PM
Thank you very much - I ran:

ftp> mget .*

and it worked perfectly. I am not requesting that this be "fixed" - as long as I can mget in my CLI ftp program, I am happy.

Thanks again.

philhu
02-13-2009, 06:19 PM
Sure you can

Use the /remount switch