mfs_ftp: easy extract / insert via ftp protocols
 

here's a little tivo side ftp server for moving recordings from tivo -> pc, pc -> tivo or tivo <-> tivo with a regular browser or ftp client. video can be moved between different versions of tivo hardware / software and streamviewed on a recieving tivo

mfs_ftp.tar.zip - current version of the complete program dir including s1 support binaries
(attached to this post)

installation:

shutdown any previous version of mfs_ftp running in the tivo

unzip the mfs_ftp.tar.zip to mfs_ftp.tar on your pc

open mfs_ftp.tar & extract readme.txt - READ IT!

transfer mfs_ftp.tar to /var in the tivo

backup the old mfs_ftp dir: "mv /var/mfs_ftp /var/mfs_ftp.bak"

from the "/var " prompt unpack with

"tar -xvf mfs_ftp.tar" or "cpio -i -H tar -d -F mfs_ftp.tar"

start from bash "/var/mfs_ftp/mfs_ftp.tcl" or load from a startup script


mfs_ftp 1.2.9 MAY NOT BE REDISTRIBUTED. PERIOD! (read the license and terms of use)

sadly many are offering mfs_ftp installed on units for sale, since the legality of the license is unlikely to deter the unscrupulous here's the deal. if you want to include mfs_ftp on a tivo your selling (presumably as an inducement to buy) send your contact info with a $40 paypal to rc3105@hotmail.com I'll provide a transferrable license tied to the tivo itself & permission to install / distribute mfs_ftp on that particular tivo. this provides a legit means for techs to install sw for the clueless & specified damages that can be recovered (in court, etc) from those distributing in violation of copyright

my paypal is rc3105@hotmail.com and donations are allways appreciated. mfs_ftp is FREE for personal non-commercial use if you install it yourself, so enjoy ;)


Usefull things to have / know about

THIS THREAD IS THE FAQ! (and moderated to stay that way)

please read readme.txt & this ENTIRE thread BEFORE posting a mfs_ftp related question - off topic posts, obsolete info & chitchat more than a month or so old gets nuked

please ask any questions re: installation / usage in this support thread instead of cluttering the forum starting new ones


ALL required S1 support files are included in the zip

here's links to usefull alternate threads

MFS_FTP on Series 2 Units contains S2 support binaries - substitute s2 binaries after installation but before starting mfs_ftp

Mfs_Ftp: suggestions, feedback, etc

mfs ftp issues in the newbie forum

tmf / ty+ tivoweb extraction module: extract tmf or ty+ via tivoweb

tmf2ty_cfg.exe is a windows binary that extracts ty from tmf. these ty are insertable with mfs_ftp & compatable with all ty utilities. if savepath.txt is in the directory with tmf2ty_cfg.exe (or in the dir with the tmf under some versions of XP) the first line of the file is used as the output path. these are tcl paths so use forward slashes - "x:/extracts/" "d:/my_recordings/" "v:/video_files/" etc

Commonly asked questions (and answers)

Nice one Guys, this is the best TVIO app i have got. Can you please pass thanks onto all the clever people who made this possible.

Ok, I'm having a "Blonde" moment, I;ve copied the 3 files to /mfs-ftp, made them exce with chmod. but when i do ./mfs_ftp.tcl i get "No Such File or Directory" ?

I have missed somthing here, i can run the app by doing /tvbin/tivosh msf_ftp.tcl, but it stop when i try to download as it cant run the other file.

Please call me stupid as i'm sure it a very silly mistake that i have made.

Regards

Paul

set your ftp client for binary mode whenever transferring via tivoftpd.

a tmf is just a tarfile. winrar & some other utils can only see files in the first 2 gig of a tar. winzip works fine

many ftp clients will only download the first 2 gig even if you have fat32 (4gig limit) or hpfs, ntfs, ext2/3 (no limit) please make sure your ftp client isn't broken

So after doing some testing with the tool, results were as I expected. I can ftp a show off my standalone and ftp it to my DirecTivo and it works beautifully (although for a half hour show, the Tivo thinks it's an hour, however the green in the progress bar stops at 30 mins like it should). Also as I expected I cannot ftp stuff off of my DirecTivo and get it to play on my standalone. I had suspected this would be the case from earlier work I was doing with insertion. Overall, things look great.

What I'd love to be able to have is a tool that I could give an AVI, MPEG, etc along with some details about it, and could then covert that to the TMF format for insertion onto the Tivo. i.e. Download a movie trailer from the net then insert it on the Tivo.

Keep up the great work...

Derek aka ][ronMan aka DSmooth

TMF specs
 

ok, couple of people have asked, so here's the info.

tmf stands for tivo media format. the NAME was AlphaWolf's idea :-p the original code to generate it was done by tivodvlpr & mbm.

edit: sorry, how did I forget to mention dtype? duh@me...

a .tmf file is a regular tar file that contains the following, in this order.

showing.xml
part00.ty
part01.ty
...
part04.ty


showing.xm is a xml file with the info from the nowshowing database stored as regular text. title, episode, description, everything. the .ty parts each represend an individual fsid as it's stored in the tivo.

pretty simple really. the xml is a bit hairy, but easy enough to parse out the title / description etc and it contains all the info the tivo stores about a recording.

--
Riley

Correction... With the 1.1.1 version I am also able to ftp a show off of my DirecTivo and ftp it onto my series 1 standalone. This is really starting to rock!

Derek aka ][ronMan aka DSmooth

See my last post... This is working OK with 1.1.1 and greater... :)

Derek aka ][ronMan aka DSmooth

Sure enough, as you were posting this I had just tried turning off passive mode, and that fixed the problem. My guess is that's what the majority of the client-specific problems are (I can't be sure of course, but it seems likely).

Would it be possible to add a short "how-to" to the readme.txt file? IMO, it should include the following:[list=1][*]command-line server configuration[*]turn passive mode OFF[*]filenames (i.e., command-line clients need only use the FSID listed as the filename to get)[*]importing vs. exporting[*]deleting programs (and how to prevent it (?) )[*]other[/list=1]
Might save a lot of repetitive questions. :) Btw, fantastic work!!! This blows away my old export-via-nfs method! (For some reason nfs is really slow on my Tivo, even with rsize=8192&wsize=8192.)

- Stealth Dave

Re: Advice... things not working
 

alanjudy,
This is my exact setup, and as you see from my post above, with versions 1.1.1 and beyond it works in bot directions...

Derek aka ][ronMan aka DSmooth

SA2dtivo - dtivo2SA - working!
 

Well; I'm xfering a .tmf originally from my dtivo to my SA. It's the same one that created my 0Byte file, undeletable listing. I found out that my SA is MUCH slower to respond and transfer to. Both tivos are 3.1 - I just didn't WAIT long enough for it to allocate space .. It is a 2hour movie, so it appears it;s going to take some time. I'll report back on results.

here is a dump of the transfer sofar...

echo to verify: PORT 192,168,1,49,7,156
ip is 192.168.1.49 port is 1948
200 PORT command successful.
echo to verify: STOR Hackers .tmf
150 Opening BINARY mode data connection for "Hackers"
entering dumpxml
segment is showing.xml
leaving dumpxml
entering parsexml
title is "Hackers" - episode "unknown episode"
leaving parsexml
attempting to create recording - success
attempting to set recordingfsid - success
attempting to set bitrate & such - success
attempting to create a dummy station - success
attempting to create a showing - success - trying to set showing info - success
trying to create the program & set the info - success
attempting to showing set program - success
created Hackers : unknown episode : fsid# 2842851
entering tmf2fsid
starting segment part00.ty
entering AddPart
trying to open recordingfsid
success
createing recording ? success
trying to add part - success
trying to mfs streamfill allocate space
success - created newfsid 2842854
trying to set file to fsidtwo - success
leaving AddPart

the initial space allocation can take bit, especially if the db is busy & it has to retry a few times.

I'm working on that issue

set your ftp client timeout to at least a minute, 2 ought to be worst case.

--
Riley

Re: SA2dtivo - dtivo2SA - working!
 

It works perfectly! Plays and looks as good if not BETTER than anything ever recorded on my SA, well, there you have it!

the down side is xfer speed. it took about 5hours to xfer a 1h45m movie to my SA. What is surprising is how fast it downloaded from my dtivo, less than 25mins.

I noticed the time difference the listing info reported a 1hr recording yet the green progbar reported correctly at 1:45.

Great Job RC!

here is the ftp sessionlog info:

sending Hackers .tmf as Hackers .tmf (1 of 1)
PORT 192,168,1,49,7,156
200 PORT command successful.
STOR Hackers .tmf
150 Opening BINARY mode data connection for "Hackers"
Transmitted 1920999936 bytes in 17739.4 secs, (1.04 Mbps), transfer succeeded
226 File transfer complete
PWD
257 "/" is current directory.
TYPE A
200 Type set to A.
PORT 192,168,1,49,8,109
200 PORT command successful.
LIST
150 Opening data connection for file list.
Received 1068 bytes in 0.1 secs, (100.00 Kbps), transfer succeeded
226 Transfer complete.
QUIT
shutting down & exiting


**** please note the last entry ****

on my SA it shuts the server down at the end of a session, on my dtivo it just returns a 221 call 'server closing control connection' the former is abit bit of a disadavantage as I have to go in and manually restart it for each session. On my dtivo it starts on bootup, and runs along side my tivoftpd only on port 24.

using the same client WS_FTP Pro, and both my tivos are running 3.1, and mfs_ftp 1.1.2

ok heres an explanation of the file formats

.ty is a plain binary dump of the individual tystream parts combined into one file. This is the legacy method of extracting tystreams, and its my personal recommendation that nobody dumps/extracts/archives to this format anymore.

.tmf is a tar archive of the above, only a few additions. Each part of the tystream is stored as an individual file inside of this tar archive. On top of that, there is an XML dump of the recordings guide data. This provides a lot of the flexability needed for insertion, and can also be used to help automate dvd authoring.

That said, converting between the two is simple. To convert from .tmf to .ty, you can just untar the archive (winrar does this), discard the showing.xml file, and merge each of the .ty parts into one file. To go from .ty to .tmf, you would need to break apart the tystream parts at their boundaries and into individual files, and generate the xml data by manualy filling in the title information, description, genre, etc etc etc, as this is not stored in a tystream dump.

There is one exception to this however. Any tystreams that were extracted using jdiners tytool will have issues with the master chunk. These streams may be converted to .tmf, but they probably won't be able to be reinserted into a tivo properly, because they were not extracted properly to begin with. This is an issue with the way tserver exports tystreams.

mfs_ftp uses the same xml dumping routine (tcl) and the same mfs_tarstream (binary) as tivodvlpr's tivoweb tarserver module, the idea being that files produced are identical and interchangable with the (eventual) tivoweb insert module

tivoweb mfsstream produces regular .ty files.

mfs_ftp does not need tivoweb running.

Problem decompressing large .tmf file? Nope it was winrar.
 

For what its worth,

I just downloaded a 5.59 GB (6,008,352,768 bytes on disk) to my ntfs formatted drive. When I opened it up with winrar, it (winrar) said the file was corrupt. I got a similar error with winzip.

So I tried a couple of other windows based decompressors and they worked just fine. (Specifically 7-Zip File Manager from source forge (free) and WinTAR-SCSI (trial)).

Z

ok, i'm a dummy. i've got this installed and have successfully pulled off a .tmf, and i want to make a dvd.

the tmf is a 30 minute show, and is in two parts in the tmf. so, i have two .tys that i need to concat and what - vsplit and mux? how do i put the two ty back together?

sorry if this has been covered; please feel free to point me to a faq

ronny

Well with the windows command line it goes like this:

copy /B C:\part00.ty+C:\part01.ty+C:\part02.ty c:\combined.ty

Just as an FYI: I am running TiVoWeb, mfs_ftp and ftpd simultaneously with out problems. I just started mfs_ftp on a different port. I have extracted 6 shows today without any problems.

FTP Client for Mac OS X?
 

https://www.panic.com/transmit seems to work good for me.

I had *exactly* the same problem. As soon as you turn of the multiple connection option in FileZilla it all starts working again....
( at least one place was under Queue, Use Multiple Connections )

Rgds,

R.

explanation-why mfs-ftp wont run.
 

I've noticed a number of posts about why the mfs_ftp.tcl script doesn't run for some people. I think I can clarify the confusion.

First, what's really going on:
A text file on a dos system has a <cr><nl> at the end of each line. Unix expects just a nl.
Looking at the 1st line in mfs_ftp_1.2.3.tcl, we see a line that is supposed to start the tcl interpreter:
#!/tvbin/tivosh
This invokes a program called /tvbin/tivosh. So
far this is 1st grade stuff. Now for the fly in the ointment...
If the file is a DOS format file, the <cr> before the <nl> becomes part of the filename, and looks like this:
#!/tvbin/tivosh<cr>
Yes, in UNIX a <cr> *can* be part of a filename. Of course the tivo can't find the tivosh<cr> program, and, well, end of story.

The dos2unix command probably strips the <cr> (I dont use this utility myself), or Unix purists might use the command
"tr -d '\r' <mfs_ftp_1.2.3.tcl >mfs_ftp_1.2.3_unix.tcl"
to do the same thing. In either case, this will remove the <cr> and fix the problem.
----------------------------------
why does the command "/tvbin/tivosh ./mfs_ftp_1.2.3.tcl" work? because TCL doesnt
need that filename, and doesn't (usually) care
about the extra <cr> in the file.
----------------------------------
Why is this happening? I looked at the unziped files on my unix system, and find that the mfs_ftp_1.2.2.tcl and mfs_ftp_1.2.3.tcl files are in 'DOS' format. In order to work the <cr> needs to be stripped.. somehow...

IF you use FTP from a *DOS* system up to the tivo
(unix) system in *ASCII* mode, the format translation is done for you , and things should work out of the box. If you use *BINARY* mode,
OR come from a UNIX system, FTP believes no translation is needed... and the extra <cr> is left intact (booo).

If you are running unix, or have the 'od' utility on your Tivo, the following command will show the line terminators and you can see it for yourself:
od -c mfs_ftp_1.2.3.tcl | less
-------
Conclusion: Just run the dos2unix command, or the tr command to remove the 'cr', and enjoy the use of the program!:cool:

Re: Linux FTP client?
 

gtkfxp works

http://gtkfxp.sourceforge.net/

I haven't downloaded or tried this yet, in fact, I just found it, but I have a question...

With this, and a corresponding client/server app sitting on another tivo, how hard would it be to have Tivo A pull a recording from Tivo B, then start playing that recording once it has buffered enough data, ala HMO for Series 1...

I think the real challange would be doing that from the Tivo interface w/o having to actually log into either box.

Thanks...
Doc

Success Report
 

Hi,

Just wanted to report in another successful installation of this program. I really love the feature, you did a great job.

Just to let you know a little about my experiences:

I am using 1.2.4 on a SA Series 1 (S-2000) v3.0. Turbonet card using the standard 3.0 drivers. (Should I change these?)

I have to start it using tivosh mfs_ftp.tcl, and I edited the script to call tivosh directly also.

I am getting 1.2 to 1.5 MB/s downloads using FileZilla 2.1a, but uploads are between 200 and 300 kB/s. I haven't gone through the debug suggestions a few pages back yet.

A few times I seem to have locked up the program, either from aborted transfers, or unknown causes. Sometimes using the abort.txt file fixed it. Sometimes I was able to telnet to 3105 and issue the exit command. Other times, however it wouldn't accept new connections and never responded and I ended up killing the process (and causing a TiVo reboot).

One file which NowShowing reports as 5GB shows up on the tmf directory as being only 1GB. When downloading it actually is splitting itself into 2 pieces. The first 1GB file gets then conflicts with the remaining 4GB file at the last moment. Download fails with a "Critical transfer error" message. I need to kill mfs using the abort.txt file to recover.

All other files seem to report the correct size. I don't know if this is a FileZilla issue or the mfs_ftp package. Let me know how I could trace down better debug info.

Thanks again for the great package.

Wactuary

phrend,

I was one of the few experiencing the critical transfer errors. I have switched clients to WS-FTP and am having NO problems whatsoever. However, if I try to use filezilla, I still get that error.

I haven't had time to really debug this situation, but apparently it's a client issue, NOT necessarily an mfs_ftp error.

Hope you have luck fixing your problem.

Just wanted to be one of the folk that says thanks for all the work on this. It works great! mfs_ftp is the icing on the cake for the tivo network!

Reinsertion Question
 

I have not yet attempted to re-insert a program, but I have a question regarding the process. Say I download a tmf file, extract the ty files from it, and then use cat to join the ty files into one big ty file.

If I replace the multiple ty files in the tmf with the large ty file I created and then re-insert the tmf file, will this work? I guess my question is whether or not the re-inserted tmf needs to have the same number and size ty files as the original tmf.

Thanks.

I don't think it will work if a pice is larger then 512M I don't know if it will work with multiple pieces smaller then that (it's designed around multiple 512M pieces plus one smaller piece at the end)

Can you put a cat'd ty back into a tmf and reinsert?

No. (unless you segment it into 512mb chunks and generate the required xml data for Now Showing to display).

Does it need the same sized segments to reinsert?

Yes.

500k/sec typical with hub?
 

I'm running a stock Philips 20hr SA (no hd upgrades) with a turbo net card - anyone else got this setup?

I'm getting 500k/sec - which is fine, but I'm seeing others get 6x that speed.

Is my hub the limiting factor (compared to a switch), or the hard drive?

Riley - this is a truly great app - takes me 11 minutes for a half hour show at basic quality - takes about a minute in tytool 6r3 to convert it to a m2v which I can watch through media player.:)

Time to start taking tv shows to work on the ipod to watch at lunchtime :D :D

Re: 500k/sec typical with hub?
 

kangolo,

I have a 14 hr Philips SA with a TiVonet card and Im getting 800k/sec ... with a Turbonet on my other TiVo, I get about 1.5M/sec.

Mickey

Share your shows at LookMa-NoHands.Com!

950k/sec on another machine .

So,
600k/sec from AMD K6 400 running XP with a realtek card
950k/sec from a P2 400 running XP with a 3com card.

Both with loads of ram, and maxtor 40gb's as the data drive, os on a seperate one

Time to replace those crappy realteks

Have you got ftp set to use passive (PASV) connections?

Do you have a firewall?

Commands and listings are sent over a seperate TCP port to data - so one may be being blocked while the command conncetion works ok

That sounds exactly like what happens if you have a DirectTivo with out the noscramble module/patch. That's not the problem is it?

noscramble
 

Bingo Doc!

Take a look here.
http://www.dealdatabase.com/forum/sh...ght=noscramble

Riley - thanks so much for your great programs and all your help, it's really appreciated. People on this board really do rock!

This is the readout from my telnet into my tivo box:

/var/hack/mfsftp# ./mfs_ftp.tcl 2000
00:00:00 - entering init_mfs_ftp
00:00:00 - entering background
/var/hack/mfsftp# 00:00:00 - entering init_mfs_ftp
00:00:00 - entering background
00:00:00 - ping - mfs_ftp running on 127,0,0,1 : 2000
00:00:00 - mfs_ftp version 1.2.5.1 - checking tivo system sw
00:00:01 - version 3.1.0-01-1-011
00:00:01 - setting mfspath to /Recording/NowShowingByClassic
00:00:01 - insert throttle is 0, == 0 ms
00:00:01 - extract throttle is 0, == 0 ms
00:00:01 - entering reset_mfs_ftp
00:00:01 - port 2000 control socket initialized - ready for connections
00:00:01 - entering buil_dir_LIST
00:00:01 - entering update_rec_fsids
00:00:01 - getting current rec_fsids list
00:00:02 - rec_fsids doesn't match cached_rec_fsids, deleting cached info
00:00:02 - entering build_rec_list
00:00:02 - from mfs "/Recording/NowShowingByClassic"
........................................
00:00:15 - build_rec_LIST complete

****

as you can see -- here I selected to use port 2000... the mfs_ftp is using the 127.0.0.1 loopback address [dhcp] is this my problem? Should I specify the Tivo's IP address [192.168.1.101] manually and if so, what's the syntax for that?

ports below 1024 are reserved for unix services. 21 is the port assigned to ftp. if you use a port other than 21 it should be > 1024. the default ftp data port is portnum-1. so for control port 21, 20 is used for data. for 3105, 3104 is used for data

passive mode & custom port usage is in the help/readme for your particular ftp client

the default ip is set to 127.0.0.1 on startup, the first time a ftp client connects it auto-detects the tivo & client ips

When moving a stream from a DTiVo to a Standalone, a couple of inconsistancies were found.

The mfs utilities were all on an NFS mount on both machines. 1.2.5.1 was used in the test.

Rating not set. (PG-13)
Advisory not set. (AC, AL)
Classifications missing (movie)
Genre missing (action/adventure)
Attributes missing like CC
Save until not set. (until I delete)
Time was off by 1 hour. (4:30 displayed instead of 5:30)

The tzos was not set correctly. It was set for central and I'm in eastern. But that should not have changed anything.

Normally I would have fixed this myself, but finding the sources which were used has been a real task. (One which I haven't completed yet... pointers/source code would be helpful)

The only real pain is the standalone is using a tivonet card (5+ hours to inserta 2 hour show).

not all info is restored. never has been, I may or may not chase down the details. there are differences in the info between 1.x-3.x sw. the "universal" info (and what I look at) is restored. nearly everything is included in the xml and available for archiving / catalogging like tmf2ty does. (side note, the few attributes that aren't extracted were left out to maintain compatability with tmf 1.0 as implemented by mbm/tivodvlpr)

the compiled binaries only do the actual extract/insert of video data. unless you really know the layout of mfs don't mess with those. using improperly recompiled versions can trash mfs and nuke your tivo.

the dbinfo is restored in the mfs_ftp tivosh script. it's easy to modify with joe and the info you need to restore the missing attributes is in tivoweb

ftp Client for Mac
 

Let me preface this by saying that 90% of the people I've helped, either on Wintels or Macs, are making basic mistakes and not reading carefully. Nothing to do with any defects in mfs_ftp or the ftp Clients.

But undeniably there are problems beyond that, that are complicating the situation, and Mac ftp Clients that work with mfs_ftp are rare indeed. Some people are doing everything else right and that is their only problem. But many are compounding a misunderstanding of the basics with ALSO using a non-friendly ftp Client on top of that; real trouble.

Fetch 4.03 works pretty well with mfs_ftp. Navigating the subdirectories is a minor nuisance, you have to do it manually. Also I occasionally have trouble with the first digit of a filename (fsid) getting clipped off and I haven't figured out how to enter a filename or a complete pathname manually for a download, in a way that gets accepted. But all-in-all Fetch 4.03 is a good solution.

There's a little gem called SimpleFTP (version 2.01) that works _very_ nicely with mfs_ftp. I like it a lot. Doesn't chop off that first digit. But it aborts file downloads at 2 GB! Hello!?

At least one poster here has said Transmit works. Not for me. It has the redundant subdirectory problem that many ftp Clients have with mfs_ftp.

I've tried more than a dozen other Mac ftp Clients, all NG. Sometimes the redundant subdirectories, to infinity. Some simply can't make any sense of the host's directory structure/syntax at all, some seem to see everything OK up until download time. Some clip off the first digit of all the subdirectories. BTW many Wintel ftp Clients have the same problems.

Riley seems to be one of those wonderful practical, pragmatic geniuses who have a good grasp of currently available tools and technology and a knack for putting them together to provide useful products. But brevity of coding and compliance with standards and protocols (ftp in this case) may be beyond his attention span. If this were a commercial product I was developing I'd have an anal analyst following him around and cleaning up after him. I'd pay them both well, but Riley quite a bit more.

As it is, for a free product he's hacking together for us, I guess as labor of love, I'm very grateful for it and perfectly willing to grope around for an ftp Client that can cope with what he's giving us.

Kudos and thanks to Riley!