![]() |
mfs_ftp: easy extract / insert via ftp protocols
here's a little tivo side ftp server for moving recordings from tivo -> pc, pc -> tivo or tivo <-> tivo with a regular browser or ftp client. video can be moved between different versions of tivo hardware / software and streamviewed on a recieving tivo
mfs_ftp.tar.zip - current version of the complete program dir including s1 support binaries (attached to this post) installation: shutdown any previous version of mfs_ftp running in the tivo unzip the mfs_ftp.tar.zip to mfs_ftp.tar on your pc open mfs_ftp.tar & extract readme.txt - READ IT! transfer mfs_ftp.tar to /var in the tivo backup the old mfs_ftp dir: "mv /var/mfs_ftp /var/mfs_ftp.bak" from the "/var " prompt unpack with "tar -xvf mfs_ftp.tar" or "cpio -i -H tar -d -F mfs_ftp.tar" start from bash "/var/mfs_ftp/mfs_ftp.tcl" or load from a startup script mfs_ftp 1.2.9 MAY NOT BE REDISTRIBUTED. PERIOD! (read the license and terms of use) sadly many are offering mfs_ftp installed on units for sale, since the legality of the license is unlikely to deter the unscrupulous here's the deal. if you want to include mfs_ftp on a tivo your selling (presumably as an inducement to buy) send your contact info with a $40 paypal to rc3105@hotmail.com I'll provide a transferrable license tied to the tivo itself & permission to install / distribute mfs_ftp on that particular tivo. this provides a legit means for techs to install sw for the clueless & specified damages that can be recovered (in court, etc) from those distributing in violation of copyright my paypal is rc3105@hotmail.com and donations are allways appreciated. mfs_ftp is FREE for personal non-commercial use if you install it yourself, so enjoy ;) Usefull things to have / know about THIS THREAD IS THE FAQ! (and moderated to stay that way) please read readme.txt & this ENTIRE thread BEFORE posting a mfs_ftp related question - off topic posts, obsolete info & chitchat more than a month or so old gets nuked please ask any questions re: installation / usage in this support thread instead of cluttering the forum starting new ones ALL required S1 support files are included in the zip here's links to usefull alternate threads MFS_FTP on Series 2 Units contains S2 support binaries - substitute s2 binaries after installation but before starting mfs_ftp Mfs_Ftp: suggestions, feedback, etc mfs ftp issues in the newbie forum tmf / ty+ tivoweb extraction module: extract tmf or ty+ via tivoweb tmf2ty_cfg.exe is a windows binary that extracts ty from tmf. these ty are insertable with mfs_ftp & compatable with all ty utilities. if savepath.txt is in the directory with tmf2ty_cfg.exe (or in the dir with the tmf under some versions of XP) the first line of the file is used as the output path. these are tcl paths so use forward slashes - "x:/extracts/" "d:/my_recordings/" "v:/video_files/" etc Commonly asked questions (and answers) |
Nice one Guys, this is the best TVIO app i have got. Can you please pass thanks onto all the clever people who made this possible.
Ok, I'm having a "Blonde" moment, I;ve copied the 3 files to /mfs-ftp, made them exce with chmod. but when i do ./mfs_ftp.tcl i get "No Such File or Directory" ? I have missed somthing here, i can run the app by doing /tvbin/tivosh msf_ftp.tcl, but it stop when i try to download as it cant run the other file. Please call me stupid as i'm sure it a very silly mistake that i have made. Regards Paul |
set your ftp client for binary mode whenever transferring via tivoftpd.
a tmf is just a tarfile. winrar & some other utils can only see files in the first 2 gig of a tar. winzip works fine many ftp clients will only download the first 2 gig even if you have fat32 (4gig limit) or hpfs, ntfs, ext2/3 (no limit) please make sure your ftp client isn't broken |
So after doing some testing with the tool, results were as I expected. I can ftp a show off my standalone and ftp it to my DirecTivo and it works beautifully (although for a half hour show, the Tivo thinks it's an hour, however the green in the progress bar stops at 30 mins like it should). Also as I expected I cannot ftp stuff off of my DirecTivo and get it to play on my standalone. I had suspected this would be the case from earlier work I was doing with insertion. Overall, things look great.
What I'd love to be able to have is a tool that I could give an AVI, MPEG, etc along with some details about it, and could then covert that to the TMF format for insertion onto the Tivo. i.e. Download a movie trailer from the net then insert it on the Tivo. Keep up the great work... Derek aka ][ronMan aka DSmooth |
TMF specs
ok, couple of people have asked, so here's the info.
tmf stands for tivo media format. the NAME was AlphaWolf's idea :-p the original code to generate it was done by tivodvlpr & mbm. edit: sorry, how did I forget to mention dtype? duh@me... a .tmf file is a regular tar file that contains the following, in this order. showing.xml part00.ty part01.ty ... part04.ty showing.xm is a xml file with the info from the nowshowing database stored as regular text. title, episode, description, everything. the .ty parts each represend an individual fsid as it's stored in the tivo. pretty simple really. the xml is a bit hairy, but easy enough to parse out the title / description etc and it contains all the info the tivo stores about a recording. -- Riley |
Quote:
Derek aka ][ronMan aka DSmooth |
Quote:
Derek aka ][ronMan aka DSmooth |
Quote:
Would it be possible to add a short "how-to" to the readme.txt file? IMO, it should include the following:[list=1][*]command-line server configuration[*]turn passive mode OFF[*]filenames (i.e., command-line clients need only use the FSID listed as the filename to get)[*]importing vs. exporting[*]deleting programs (and how to prevent it (?) )[*]other[/list=1] Might save a lot of repetitive questions. :) Btw, fantastic work!!! This blows away my old export-via-nfs method! (For some reason nfs is really slow on my Tivo, even with rsize=8192&wsize=8192.) - Stealth Dave |
Re: Advice... things not working
Quote:
This is my exact setup, and as you see from my post above, with versions 1.1.1 and beyond it works in bot directions... Derek aka ][ronMan aka DSmooth |
SA2dtivo - dtivo2SA - working!
Well; I'm xfering a .tmf originally from my dtivo to my SA. It's the same one that created my 0Byte file, undeletable listing. I found out that my SA is MUCH slower to respond and transfer to. Both tivos are 3.1 - I just didn't WAIT long enough for it to allocate space .. It is a 2hour movie, so it appears it;s going to take some time. I'll report back on results.
here is a dump of the transfer sofar... echo to verify: PORT 192,168,1,49,7,156 ip is 192.168.1.49 port is 1948 200 PORT command successful. echo to verify: STOR Hackers .tmf 150 Opening BINARY mode data connection for "Hackers" entering dumpxml segment is showing.xml leaving dumpxml entering parsexml title is "Hackers" - episode "unknown episode" leaving parsexml attempting to create recording - success attempting to set recordingfsid - success attempting to set bitrate & such - success attempting to create a dummy station - success attempting to create a showing - success - trying to set showing info - success trying to create the program & set the info - success attempting to showing set program - success created Hackers : unknown episode : fsid# 2842851 entering tmf2fsid starting segment part00.ty entering AddPart trying to open recordingfsid success createing recording ? success trying to add part - success trying to mfs streamfill allocate space success - created newfsid 2842854 trying to set file to fsidtwo - success leaving AddPart |
the initial space allocation can take bit, especially if the db is busy & it has to retry a few times.
I'm working on that issue set your ftp client timeout to at least a minute, 2 ought to be worst case. -- Riley |
Re: SA2dtivo - dtivo2SA - working!
Quote:
the down side is xfer speed. it took about 5hours to xfer a 1h45m movie to my SA. What is surprising is how fast it downloaded from my dtivo, less than 25mins. I noticed the time difference the listing info reported a 1hr recording yet the green progbar reported correctly at 1:45. Great Job RC! here is the ftp sessionlog info: sending Hackers .tmf as Hackers .tmf (1 of 1) PORT 192,168,1,49,7,156 200 PORT command successful. STOR Hackers .tmf 150 Opening BINARY mode data connection for "Hackers" Transmitted 1920999936 bytes in 17739.4 secs, (1.04 Mbps), transfer succeeded 226 File transfer complete PWD 257 "/" is current directory. TYPE A 200 Type set to A. PORT 192,168,1,49,8,109 200 PORT command successful. LIST 150 Opening data connection for file list. Received 1068 bytes in 0.1 secs, (100.00 Kbps), transfer succeeded 226 Transfer complete. QUIT shutting down & exiting **** please note the last entry **** on my SA it shuts the server down at the end of a session, on my dtivo it just returns a 221 call 'server closing control connection' the former is abit bit of a disadavantage as I have to go in and manually restart it for each session. On my dtivo it starts on bootup, and runs along side my tivoftpd only on port 24. using the same client WS_FTP Pro, and both my tivos are running 3.1, and mfs_ftp 1.1.2 |
ok heres an explanation of the file formats
.ty is a plain binary dump of the individual tystream parts combined into one file. This is the legacy method of extracting tystreams, and its my personal recommendation that nobody dumps/extracts/archives to this format anymore. .tmf is a tar archive of the above, only a few additions. Each part of the tystream is stored as an individual file inside of this tar archive. On top of that, there is an XML dump of the recordings guide data. This provides a lot of the flexability needed for insertion, and can also be used to help automate dvd authoring. That said, converting between the two is simple. To convert from .tmf to .ty, you can just untar the archive (winrar does this), discard the showing.xml file, and merge each of the .ty parts into one file. To go from .ty to .tmf, you would need to break apart the tystream parts at their boundaries and into individual files, and generate the xml data by manualy filling in the title information, description, genre, etc etc etc, as this is not stored in a tystream dump. There is one exception to this however. Any tystreams that were extracted using jdiners tytool will have issues with the master chunk. These streams may be converted to .tmf, but they probably won't be able to be reinserted into a tivo properly, because they were not extracted properly to begin with. This is an issue with the way tserver exports tystreams. |
Quote:
tivoweb mfsstream produces regular .ty files. mfs_ftp does not need tivoweb running. |
Problem decompressing large .tmf file? Nope it was winrar.
For what its worth,
I just downloaded a 5.59 GB (6,008,352,768 bytes on disk) to my ntfs formatted drive. When I opened it up with winrar, it (winrar) said the file was corrupt. I got a similar error with winzip. So I tried a couple of other windows based decompressors and they worked just fine. (Specifically 7-Zip File Manager from source forge (free) and WinTAR-SCSI (trial)). Z |
ok, i'm a dummy. i've got this installed and have successfully pulled off a .tmf, and i want to make a dvd.
the tmf is a 30 minute show, and is in two parts in the tmf. so, i have two .tys that i need to concat and what - vsplit and mux? how do i put the two ty back together? sorry if this has been covered; please feel free to point me to a faq ronny |
Well with the windows command line it goes like this:
copy /B C:\part00.ty+C:\part01.ty+C:\part02.ty c:\combined.ty |
Just as an FYI: I am running TiVoWeb, mfs_ftp and ftpd simultaneously with out problems. I just started mfs_ftp on a different port. I have extracted 6 shows today without any problems.
|
FTP Client for Mac OS X?
https://www.panic.com/transmit seems to work good for me.
|
Quote:
( at least one place was under Queue, Use Multiple Connections ) Rgds, R. |
explanation-why mfs-ftp wont run.
I've noticed a number of posts about why the mfs_ftp.tcl script doesn't run for some people. I think I can clarify the confusion.
First, what's really going on: A text file on a dos system has a <cr><nl> at the end of each line. Unix expects just a nl. Looking at the 1st line in mfs_ftp_1.2.3.tcl, we see a line that is supposed to start the tcl interpreter: #!/tvbin/tivosh This invokes a program called /tvbin/tivosh. So far this is 1st grade stuff. Now for the fly in the ointment... If the file is a DOS format file, the <cr> before the <nl> becomes part of the filename, and looks like this: #!/tvbin/tivosh<cr> Yes, in UNIX a <cr> *can* be part of a filename. Of course the tivo can't find the tivosh<cr> program, and, well, end of story. The dos2unix command probably strips the <cr> (I dont use this utility myself), or Unix purists might use the command "tr -d '\r' <mfs_ftp_1.2.3.tcl >mfs_ftp_1.2.3_unix.tcl" to do the same thing. In either case, this will remove the <cr> and fix the problem. ---------------------------------- why does the command "/tvbin/tivosh ./mfs_ftp_1.2.3.tcl" work? because TCL doesnt need that filename, and doesn't (usually) care about the extra <cr> in the file. ---------------------------------- Why is this happening? I looked at the unziped files on my unix system, and find that the mfs_ftp_1.2.2.tcl and mfs_ftp_1.2.3.tcl files are in 'DOS' format. In order to work the <cr> needs to be stripped.. somehow... IF you use FTP from a *DOS* system up to the tivo (unix) system in *ASCII* mode, the format translation is done for you , and things should work out of the box. If you use *BINARY* mode, OR come from a UNIX system, FTP believes no translation is needed... and the extra <cr> is left intact (booo). If you are running unix, or have the 'od' utility on your Tivo, the following command will show the line terminators and you can see it for yourself: od -c mfs_ftp_1.2.3.tcl | less ------- Conclusion: Just run the dos2unix command, or the tr command to remove the 'cr', and enjoy the use of the program!:cool: |
Re: Linux FTP client?
Quote:
http://gtkfxp.sourceforge.net/ |
I haven't downloaded or tried this yet, in fact, I just found it, but I have a question...
With this, and a corresponding client/server app sitting on another tivo, how hard would it be to have Tivo A pull a recording from Tivo B, then start playing that recording once it has buffered enough data, ala HMO for Series 1... I think the real challange would be doing that from the Tivo interface w/o having to actually log into either box. Thanks... Doc |
Success Report
Hi,
Just wanted to report in another successful installation of this program. I really love the feature, you did a great job. Just to let you know a little about my experiences: I am using 1.2.4 on a SA Series 1 (S-2000) v3.0. Turbonet card using the standard 3.0 drivers. (Should I change these?) I have to start it using tivosh mfs_ftp.tcl, and I edited the script to call tivosh directly also. I am getting 1.2 to 1.5 MB/s downloads using FileZilla 2.1a, but uploads are between 200 and 300 kB/s. I haven't gone through the debug suggestions a few pages back yet. A few times I seem to have locked up the program, either from aborted transfers, or unknown causes. Sometimes using the abort.txt file fixed it. Sometimes I was able to telnet to 3105 and issue the exit command. Other times, however it wouldn't accept new connections and never responded and I ended up killing the process (and causing a TiVo reboot). One file which NowShowing reports as 5GB shows up on the tmf directory as being only 1GB. When downloading it actually is splitting itself into 2 pieces. The first 1GB file gets then conflicts with the remaining 4GB file at the last moment. Download fails with a "Critical transfer error" message. I need to kill mfs using the abort.txt file to recover. All other files seem to report the correct size. I don't know if this is a FileZilla issue or the mfs_ftp package. Let me know how I could trace down better debug info. Thanks again for the great package. Wactuary |
phrend,
I was one of the few experiencing the critical transfer errors. I have switched clients to WS-FTP and am having NO problems whatsoever. However, if I try to use filezilla, I still get that error. I haven't had time to really debug this situation, but apparently it's a client issue, NOT necessarily an mfs_ftp error. Hope you have luck fixing your problem. |
Just wanted to be one of the folk that says thanks for all the work on this. It works great! mfs_ftp is the icing on the cake for the tivo network!
|
Reinsertion Question
I have not yet attempted to re-insert a program, but I have a question regarding the process. Say I download a tmf file, extract the ty files from it, and then use cat to join the ty files into one big ty file.
If I replace the multiple ty files in the tmf with the large ty file I created and then re-insert the tmf file, will this work? I guess my question is whether or not the re-inserted tmf needs to have the same number and size ty files as the original tmf. Thanks. |
I don't think it will work if a pice is larger then 512M I don't know if it will work with multiple pieces smaller then that (it's designed around multiple 512M pieces plus one smaller piece at the end)
|
Can you put a cat'd ty back into a tmf and reinsert?
No. (unless you segment it into 512mb chunks and generate the required xml data for Now Showing to display). Does it need the same sized segments to reinsert? Yes. |
500k/sec typical with hub?
I'm running a stock Philips 20hr SA (no hd upgrades) with a turbo net card - anyone else got this setup?
I'm getting 500k/sec - which is fine, but I'm seeing others get 6x that speed. Is my hub the limiting factor (compared to a switch), or the hard drive? Riley - this is a truly great app - takes me 11 minutes for a half hour show at basic quality - takes about a minute in tytool 6r3 to convert it to a m2v which I can watch through media player.:) Time to start taking tv shows to work on the ipod to watch at lunchtime :D :D |
Re: 500k/sec typical with hub?
Quote:
I have a 14 hr Philips SA with a TiVonet card and Im getting 800k/sec ... with a Turbonet on my other TiVo, I get about 1.5M/sec. Mickey Share your shows at LookMa-NoHands.Com! |
950k/sec on another machine .
So, 600k/sec from AMD K6 400 running XP with a realtek card 950k/sec from a P2 400 running XP with a 3com card. Both with loads of ram, and maxtor 40gb's as the data drive, os on a seperate one Time to replace those crappy realteks |
Quote:
Do you have a firewall? Commands and listings are sent over a seperate TCP port to data - so one may be being blocked while the command conncetion works ok |
Quote:
|
noscramble
|
Riley - thanks so much for your great programs and all your help, it's really appreciated. People on this board really do rock!
This is the readout from my telnet into my tivo box: /var/hack/mfsftp# ./mfs_ftp.tcl 2000 00:00:00 - entering init_mfs_ftp 00:00:00 - entering background /var/hack/mfsftp# 00:00:00 - entering init_mfs_ftp 00:00:00 - entering background 00:00:00 - ping - mfs_ftp running on 127,0,0,1 : 2000 00:00:00 - mfs_ftp version 1.2.5.1 - checking tivo system sw 00:00:01 - version 3.1.0-01-1-011 00:00:01 - setting mfspath to /Recording/NowShowingByClassic 00:00:01 - insert throttle is 0, == 0 ms 00:00:01 - extract throttle is 0, == 0 ms 00:00:01 - entering reset_mfs_ftp 00:00:01 - port 2000 control socket initialized - ready for connections 00:00:01 - entering buil_dir_LIST 00:00:01 - entering update_rec_fsids 00:00:01 - getting current rec_fsids list 00:00:02 - rec_fsids doesn't match cached_rec_fsids, deleting cached info 00:00:02 - entering build_rec_list 00:00:02 - from mfs "/Recording/NowShowingByClassic" ........................................ 00:00:15 - build_rec_LIST complete **** as you can see -- here I selected to use port 2000... the mfs_ftp is using the 127.0.0.1 loopback address [dhcp] is this my problem? Should I specify the Tivo's IP address [192.168.1.101] manually and if so, what's the syntax for that? |
ports below 1024 are reserved for unix services. 21 is the port assigned to ftp. if you use a port other than 21 it should be > 1024. the default ftp data port is portnum-1. so for control port 21, 20 is used for data. for 3105, 3104 is used for data
passive mode & custom port usage is in the help/readme for your particular ftp client the default ip is set to 127.0.0.1 on startup, the first time a ftp client connects it auto-detects the tivo & client ips |
When moving a stream from a DTiVo to a Standalone, a couple of inconsistancies were found.
The mfs utilities were all on an NFS mount on both machines. 1.2.5.1 was used in the test. Rating not set. (PG-13) Advisory not set. (AC, AL) Classifications missing (movie) Genre missing (action/adventure) Attributes missing like CC Save until not set. (until I delete) Time was off by 1 hour. (4:30 displayed instead of 5:30) The tzos was not set correctly. It was set for central and I'm in eastern. But that should not have changed anything. Normally I would have fixed this myself, but finding the sources which were used has been a real task. (One which I haven't completed yet... pointers/source code would be helpful) The only real pain is the standalone is using a tivonet card (5+ hours to inserta 2 hour show). |
not all info is restored. never has been, I may or may not chase down the details. there are differences in the info between 1.x-3.x sw. the "universal" info (and what I look at) is restored. nearly everything is included in the xml and available for archiving / catalogging like tmf2ty does. (side note, the few attributes that aren't extracted were left out to maintain compatability with tmf 1.0 as implemented by mbm/tivodvlpr)
the compiled binaries only do the actual extract/insert of video data. unless you really know the layout of mfs don't mess with those. using improperly recompiled versions can trash mfs and nuke your tivo. the dbinfo is restored in the mfs_ftp tivosh script. it's easy to modify with joe and the info you need to restore the missing attributes is in tivoweb |
ftp Client for Mac
Quote:
But undeniably there are problems beyond that, that are complicating the situation, and Mac ftp Clients that work with mfs_ftp are rare indeed. Some people are doing everything else right and that is their only problem. But many are compounding a misunderstanding of the basics with ALSO using a non-friendly ftp Client on top of that; real trouble. Fetch 4.03 works pretty well with mfs_ftp. Navigating the subdirectories is a minor nuisance, you have to do it manually. Also I occasionally have trouble with the first digit of a filename (fsid) getting clipped off and I haven't figured out how to enter a filename or a complete pathname manually for a download, in a way that gets accepted. But all-in-all Fetch 4.03 is a good solution. There's a little gem called SimpleFTP (version 2.01) that works _very_ nicely with mfs_ftp. I like it a lot. Doesn't chop off that first digit. But it aborts file downloads at 2 GB! Hello!? At least one poster here has said Transmit works. Not for me. It has the redundant subdirectory problem that many ftp Clients have with mfs_ftp. I've tried more than a dozen other Mac ftp Clients, all NG. Sometimes the redundant subdirectories, to infinity. Some simply can't make any sense of the host's directory structure/syntax at all, some seem to see everything OK up until download time. Some clip off the first digit of all the subdirectories. BTW many Wintel ftp Clients have the same problems. Riley seems to be one of those wonderful practical, pragmatic geniuses who have a good grasp of currently available tools and technology and a knack for putting them together to provide useful products. But brevity of coding and compliance with standards and protocols (ftp in this case) may be beyond his attention span. If this were a commercial product I was developing I'd have an anal analyst following him around and cleaning up after him. I'd pay them both well, but Riley quite a bit more. As it is, for a free product he's hacking together for us, I guess as labor of love, I'm very grateful for it and perfectly willing to grope around for an ftp Client that can cope with what he's giving us. Kudos and thanks to Riley! |
| All times are GMT -4. The time now is 06:29 PM. |
Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2010, Jelsoft Enterprises Ltd.
Copyright 2000-2008 © dealdatabase.com.