Mfs_Ftp: extract, archive, restore & transfer recordings

[cheer]

Remember that ftp generally uses TWO TCP connections -- one for control and one for data. 3105 is the control and (I'm guessing, haven't ever checked) 3104 is the data.

Hackman in TiVoWeb says that mfs_ftp is running, if I telnet into TiVo and run ps -ax I can see the ms_ftp process but when I run tivoftp.bat then the whole thing hangs at the mget m* and I have to kill it

Which tells me that it's your data connection that's not working.

ftp> cd asx  
250 Directory change successful.  
ftp> prompt off  
Interactive mode Off .  
ftp> lcd c:\tivo\asx_temp  
Local directory now C:\tivo\asx_temp.  
ftp> mget m*  

Connection closed by remote host.  
ftp>

So the mget is the first "data" flow you attempt. This is a clue.

and if I ftp into port 3105 manually and try something simple like "ls" then the connection gets closed

Same thing...the actual data retrieved by "ls" is sent via the data connection.

So this smells to me like a network(ish) issue. Are you on the local LAN with your Tivo or are you connecting from somewhere else via router/firewall? What is the OS you are running on the PC? Check to make sure you don't have some kind of firewall blocking traffic to/from that port (Windows SP2 firewall, ZoneAlarm...even Norton Antivirus now comes with "worm protection" that blocks all sorts of ports).

Post some more details...this should be easy to fix.

--chris

[CarlWalters]

Hi,

my TiVo has a CacheCard which is plugged into a Netgear WGE101 wireless ethernet adapter which then connects wirelessly into the Netgear WGR614 Cable Wireless Router in the study which my PC running Windows XP SP2 and my Cable Modem also plug into. I am also running ZoneAlarm with AntiVirus (not the free version) but have never had to set up anything specifically on ZoneAlarm for TiVo or ftp or SmartFTp as far as I can remember. I had to set up port 443 forwarding on the NetGear WGR614 so that I could run orenosp and get a secure external TiVoWeb connection - but that's probably not related at all.

any more clues here?

[hanzacra]

Been having an odd problem with one show I'm trying to TiVo-to-Tivo copy with FXP.

It's failing to copy, and when I look in the logs the name of the xml in the tmf is coming out wrong:

08:57:39:PM - tarpart is "wing.xml"

The four-byte offset is buggering everying else up too, so the copy fails.

I can FXP other shows fine, and this is the first one I've had an issue with.

Any ideas?

Was this ever addressed? I've searched around and haven't found an answer. Here is what I am seeing when transferring a show using SNP & MFS_FTP:

Source Tivo:

12:11:01:AM - RETR:
"/tmf/{Monster Garage}{2005-04-25}{Mobile Funhouse}{12.00 AM Tue Apr 26, 2005}{DSC}.tmf"
12:11:01:AM - fsid_from_filename:
"{Monster Garage}{2005-04-25}{Mobile Funhouse}{12.00 AM Tue Apr 26, 2005}{DSC}.tmf"
12:11:01:AM - searching $info(rec_filenames) for 
"{Monster Garage}{2005-04-25}{Mobile Funhouse}{12.00 AM Tue Apr 26, 2005}{DSC}"
12:11:01:AM - filename matches fsid "1701901"
12:11:01:AM - recfsid "1701901" filetype "tmf"
12:11:01:AM - rec_info_from_db: recobj{1701901}
12:11:02:AM - Monster Garage 1426063360 k
12:11:02:AM - cache_xml_2_disk: recobj{1701901}  force cache = "0"
12:11:02:AM - allready cached 1701901.xml
12:11:02:AM - calc_tertiary_info: recobj{1701901}
12:11:02:AM - 150 Partsizes: {PS 536870912 536870912 352321536}
12:11:13:AM - build_parts: recobj{1701901}
12:11:13:AM - building new lists
12:11:13:AM - 
orig list: {1739638 1739645 1739652}
new list:  {1739638 1739645 1739652}
12:11:13:AM - get_xml: recobj{1701901}
12:11:13:AM - reading xml from disk cache
12:11:13:AM - stream_tmf: recobj{1701901}
12:11:13:AM - set priorities: mfs_ftp 1 % I/O chan 1 %
bgerror invoked with error 
 
 " child killed: write on pipe with no readers "

Target Tivo:

12:11:15:AM - AddPart: "219770" - "536870912" bytes
12:11:15:AM - AddPart stream allocate errval 
"219774"
12:11:15:AM - AddPart: retryev
""
12:11:15:AM - added part "219774" of "512" meg
12:11:16:AM - Priority set...
importing fsid 219774 of size 536870912 from -
Done

Exiting with error code 0
12:11:16:AM - AddPart: "219770" - "352321536" bytes
12:11:16:AM - AddPart stream allocate errval 
"219775"
12:11:16:AM - AddPart: retryev
""
12:11:16:AM - added part "219775" of "336" meg
12:11:17:AM - Priority set...
importing fsid 219775 of size 352321536 from -
Done

Exiting with error code 0
12:11:17:AM - Partlist: 219773 219774 219775
12:11:17:AM - 150 Opening BINARY mode data connection for "{{Monster Garage}{2005-04-25}{Mobile Funhouse}{12.00 AM Tue Apr 26, 2005}{DSC}.tmf}"
12:11:17:AM - parse_tmf:
12:11:17:AM - xml_from_tmf:
12:11:17:AM - tarpart is "wing.xml"
bgerror invoked with error 
 
 " syntax error in expression "00004443" "

I've transferred a couple of 30- and 60-minute shows without problem. Any ideas?

[sanderton]

Hi,

my TiVo has a CacheCard which is plugged into a Netgear WGE101 wireless ethernet adapter which then connects wirelessly into the Netgear WGR614 Cable Wireless Router in the study which my PC running Windows XP SP2 and my Cable Modem also plug into. I am also running ZoneAlarm with AntiVirus (not the free version) but have never had to set up anything specifically on ZoneAlarm for TiVo or ftp or SmartFTp as far as I can remember. I had to set up port 443 forwarding on the NetGear WGR614 so that I could run orenosp and get a secure external TiVoWeb connection - but that's probably not related at all.

any more clues here?

can you download the files with SmartFTP or similar rather than your script?

[compwiz312]

OK, I am trying to upload a freshly created ty file (converted AVI parsed through mplex) onto my Tivo and simply put it does not work.

I initiate the mfs_ftp transfer and it quits immedietly. Upon examining the log (attached and quoted), I can see that it is a "broken pipe" error.

See log below:

05:08:03:AM - 227 Entering Passive Mode (192,168,10,109,12,32).
05:08:04:AM - 150 Opening BINARY mode data connection for "{ncis.ty}"
bgerror invoked with error 
 
 " error writing "file24": broken pipe " 

re-initializing mfs_ftp

close the current ftp connection and simply open another

"core dump" :p

info(version): 1.2.9p-Jamie
info(tswv): 4.0.1b-01-2-240
info(dbl): 0
info(ithrottle): 2
info(insert_priority): 10
info(multithreaded): 0
info(saveuntil): suggestion
info(name_detail): 5
info(bjuggle): 0
info(active): 0
info(ac_interval): 1800
info(gatewayip): 127.0.0.1
info(gatewayport): 3105


catch close lastsock val ""

I searched around on the boards and corrupt binaries were suggested, so I deleted all my mfs_* binaries and replaced them with Jamie's newest (I didn't forget to simlink mfs_stdinsert to mfs_import if that's what you're about to think) and tried again.

Same thing.

I then applied jamie's patch, tried again and the same thing.

Now truly frustrated, I edited my rc.sysinit.author (attached to post) file to comment out all startup programs except for telnet, completely deleted the mfs_ftp directory and reinstalled from the start (redownloaded from first post, deleted S1 binaries, replaced with jamie's and applied his patch) and still nothing.

Some information on my Tivo:

• Tivo model - 140
• Version - 4.0.1b-01-2-240 (generated directly from slices)
• Superpatch installed
• 80GB hard drive (92 hours)

Any insight anyone could provide to me is greatly appreciated.

Thanks,
Justin

P.S. I can't spell all that great, so please excuse all spelling errors

[compwiz312]

As per Jamie's request, I kicked the debug level to 5 and created a small (15MB) version of the stream. It exhibits the same issue.

You can find the stream on my FTP (temporarily) at ftp://compwiz312.dyndns.org/Test.ty

Here is the revelant portion of the log (the whole thing is attached below):

07:02:27:AM - echo to verify: "STOR Test.ty"
07:02:27:AM - parseline:
"STOR Test.ty"
07:02:27:AM - STOR:
"STOR Test.ty"
07:02:27:AM - make_blank_rec: "Test.ty"
07:02:27:AM - created blank recording {18656}
07:02:27:AM - rec_info_from_db: recobj{18656}
07:02:27:AM - Test.ty  k
07:02:27:AM - cache_xml_2_disk: recobj{18656}  force cache = "0"
07:02:27:AM - recording in progress, don't cache 18656.xml yet
07:02:27:AM - calc_tertiary_info: recobj{18656}
07:02:27:AM - 150 Opening BINARY mode data connection for "{Test.ty}"
07:02:27:AM - parse_ty+:
07:02:27:AM - ty2fsid: fsid{18656}
07:02:27:AM - header descriptor is "⌡Fz╜"
07:02:27:AM - tchl is "00000000" == "0" chunks
07:02:27:AM - AddPart: "18656" - "0" bytes
07:02:27:AM - AddPart stream allocate errval
"18659"
07:02:28:AM - AddPart: retryev
""
07:02:28:AM - added part "18659" of "0" meg
07:02:28:AM - mfs_import on first part
bgerror invoked with error

 " error writing "file24": broken pipe "

re-initializing mfs_ftp

close the current ftp connection and simply open another

"core dump" :p

info(version): 1.2.9p-Jamie
info(tswv): 4.0.1b-01-2-240
info(dbl): 5
info(ithrottle): 2
info(insert_priority): 10
info(multithreaded): 0
info(saveuntil): suggestion
info(name_detail): 5
info(bjuggle): 0
info(active): 0
info(ac_interval): 1800
info(gatewayip): 127.0.0.1
info(gatewayport): 3105


07:02:29:AM - abortcheck: "bgerror"
07:02:29:AM - reset_control_socket:
catch close lastsock val ""
07:02:29:AM - setup_control_socket:
07:02:29:AM - "192,168,10,109:3105" ready for connections

[sanderton]

The Part file it's creating to stream the data into is of size 0Mb - I can't remember for sure if that's normal, but I don't think it is. IIRC it parses the header to get the file size, and your seems to be returning gibberish.

[Jamie]

The Part file it's creating to stream the data into is of size 0Mb - I can't remember for sure if that's normal, but I don't think it is. IIRC it parses the header to get the file size, and your seems to be returning gibberish.

It's definitely a bad stream. The master chunk at the start of the file is:

 % hexdump -C compwiz-Test.ty  | head -4
00000000  f5 46 7a bd 00 00 00 00  00 00 00 00 00 00 00 00  |.Fz.............|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00020000  2d 00 00 80 00 06 c7 e0  00 00 00 00 00 00 00 00  |-...............|

The magic number looks right (0xf5467abd), but the rest of the master chunk header is all zeros. That's not right. The stream size information should be there starting at offset 0x0c.

Compwiz312: This is either a problem with the tools you are using to create the ty, or your use of them.

[adam1115]

Sorry if this is a stupid question, I've searched and can't seem to figure this out..

I download a TY recording (for example a 2 hour movie). It is about 2 Gig, I download the entire file (shows the same size in windows as it did on the TiVo. The thing is, it's only about half of the movie. I can play the entire thing on the TiVo. I'm playing the TY file with TyShow using Windows Media player and have tried to different computers. I'm downloading using MFS_FTP and WIndows FTP client.

[compwiz312]

Sorry if this is a stupid question, I've searched and can't seem to figure this out..

I download a TY recording (for example a 2 hour movie). It is about 2 Gig, I download the entire file (shows the same size in windows as it did on the TiVo. The thing is, it's only about half of the movie. I can play the entire thing on the TiVo. I'm playing the TY file with TyShow using Windows Media player and have tried to different computers. I'm downloading using MFS_FTP and WIndows FTP client.

Try using mplayer and see if it works with the whole movie. It seems to handle bad streams better than anything else (it worked fine with my botched ty from my previous post).

If it plays in mplayer, you might want to convert it to mpg using TyTools and then play it in Windows Media Player.

[adam1115]

Try using mplayer and see if it works with the whole movie. It seems to handle bad streams better than anything else (it worked fine with my botched ty from my previous post).

If it plays in mplayer, you might want to convert it to mpg using TyTools and then play it in Windows Media Player.

I have converted to MPG using TyTools, and get the same thing.

If I play with Mplayer, when it gets to where it ends in Windows Media Player it hangs, and Mplayer says:

Building audio filter chain for 48000Hz/2ch/16bit -> 48000Hz/2ch/16bit...
Starting playback...
A: 288.9 V: 290.7 A-V: -1.774 ct: 0.104 100/ 97 0% 0% 0.0% 0 0 28%
demux_mpg: 3:2 TELECINE detected, enabling inverse telecine fx. FPS changed to
3.976!
@@@ libmpeg2 returned from sig11... (bad file?) @@@ 0% 0% 0.0% 0 0 1%%

[compwiz312]

It may have gotten corrupted downloading through MFS-FTP. Try downloading it with TyTools directly and see if you get the same problem. They transfer shows differently so sometimes a show thats corrupted via one method works via the other.

[CarlWalters]

can you download the files with SmartFTP or similar rather than your script?

Just tried that and got

SmartFTP v1.1.984.15
Resolving host name 192.168.0.100...
Connecting to (192.168.0.100) -> IP: 192.168.0.100 PORT: 3105
Connected to (192.168.0.100) -> Time = 1312ms
Socket connected waiting for login sequence.
220 Mfs_Ftp ver 1.2.9p - {sock30} from "192.168.0.2:3388"
USER anonymous
331 User name okay, need password.
PASS (hidden)
230 Running in TiVo Mode.
SYST
215 UNIX
TYPE I
200 Type set to I
REST 0
350 Restart okay, awaiting file request.
PWD
257 "/" is current directory.
Compression disabled for private IP addresses.
PASV
Connection closed.

[adam1115]

It may have gotten corrupted downloading through MFS-FTP. Try downloading it with TyTools directly and see if you get the same problem. They transfer shows differently so sometimes a show thats corrupted via one method works via the other.

Interesting. Downloaded the same show FIVE times with Windows FTP all with the exact same results.

Works fine with Tserver / tytools. Not sure why I was hell bent on using MFS_FTP, but am up and running on tytools.

Thanks!

[sanderton]

Sorrt, didn't read last post.