PDA

View Full Version : Mfs_Ftp: extract, archive, restore & transfer recordings



Pages : [1] 2 3 4 5 6

rc3105
02-14-2003, 10:35 PM
here's a little tivo side ftp server for moving recordings from tivo -> pc, pc -> tivo or tivo <-> tivo with a regular browser or ftp client. video can be moved between different versions of tivo hardware / software and streamviewed on a recieving tivo

mfs_ftp.tar.zip (http://www.dealdatabase.com/forum/attachment.php?attachmentid=2339) - current version of the complete program dir including s1 support binaries
(attached to this post (http://www.dealdatabase.com/forum/showthread.php?s=&postid=128729&perpage=1))

installation:

shutdown any previous version of mfs_ftp running in the tivo

unzip the mfs_ftp.tar.zip to mfs_ftp.tar on your pc

open mfs_ftp.tar & extract readme.txt - READ IT!

transfer mfs_ftp.tar to /var in the tivo

backup the old mfs_ftp dir: "mv /var/mfs_ftp /var/mfs_ftp.bak"

from the "/var " prompt unpack with

"tar -xvf mfs_ftp.tar" or "cpio -i -H tar -d -F mfs_ftp.tar"

start from bash "/var/mfs_ftp/mfs_ftp.tcl" or load from a startup script


mfs_ftp 1.2.9 MAY NOT BE REDISTRIBUTED. PERIOD! (read the license and terms of use)

sadly many are offering mfs_ftp installed on units for sale, since the legality of the license is unlikely to deter the unscrupulous here's the deal. if you want to include mfs_ftp on a tivo your selling (presumably as an inducement to buy) send your contact info with a $40 paypal to rc3105@hotmail.com I'll provide a transferrable license tied to the tivo itself & permission to install / distribute mfs_ftp on that particular tivo. this provides a legit means for techs to install sw for the clueless & specified damages that can be recovered (in court, etc) from those distributing in violation of copyright

my paypal is rc3105@hotmail.com and donations are allways appreciated. mfs_ftp is FREE for personal non-commercial use if you install it yourself, so enjoy ;)


Usefull things to have / know about

THIS THREAD IS THE FAQ (http://www.dealdatabase.com/forum/printthread.php?s=&threadid=21915&perpage=1000)! (and moderated to stay that way)

please read readme.txt & this ENTIRE thread BEFORE posting a mfs_ftp related question - off topic posts, obsolete info & chitchat more than a month or so old gets nuked

please ask any questions re: installation / usage in this support thread instead of cluttering the forum starting new ones


ALL required S1 support files are included in the zip

here's links to usefull alternate threads

MFS_FTP on Series 2 Units (http://www.dealdatabase.com/forum/showthread.php?s=&threadid=29459) contains S2 support binaries - substitute s2 binaries after installation but before starting mfs_ftp

Mfs_Ftp: suggestions, feedback, etc (http://www.dealdatabase.com/forum/showthread.php?t=34048)

mfs ftp issues (http://www.dealdatabase.com/forum/showthread.php?t=32568) in the newbie forum

tmf / ty+ tivoweb extraction module: (http://www.dealdatabase.com/forum/showthread.php?t=32049) extract tmf or ty+ via tivoweb

tmf2ty_cfg.exe (http://alt.org/forum/index.php?t=getfile&id=96&rid=24) is a windows binary that extracts ty from tmf. these ty are insertable with mfs_ftp & compatable with all ty utilities. if savepath.txt is in the directory with tmf2ty_cfg.exe (or in the dir with the tmf under some versions of XP) the first line of the file is used as the output path. these are tcl paths so use forward slashes - "x:/extracts/" "d:/my_recordings/" "v:/video_files/" etc

Commonly asked questions (and answers) (http://www.dealdatabase.com/forum/showthread.php?p=141112#post141112)

wheelbarrow
02-17-2003, 10:39 AM
Nice one Guys, this is the best TVIO app i have got. Can you please pass thanks onto all the clever people who made this possible.

Ok, I'm having a "Blonde" moment, I;ve copied the 3 files to /mfs-ftp, made them exce with chmod. but when i do ./mfs_ftp.tcl i get "No Such File or Directory" ?

I have missed somthing here, i can run the app by doing /tvbin/tivosh msf_ftp.tcl, but it stop when i try to download as it cant run the other file.

Please call me stupid as i'm sure it a very silly mistake that i have made.

Regards

Paul

rc3105
02-17-2003, 10:58 AM
set your ftp client for binary mode whenever transferring via tivoftpd.

a tmf is just a tarfile. winrar & some other utils can only see files in the first 2 gig of a tar. winzip works fine

many ftp clients will only download the first 2 gig even if you have fat32 (4gig limit) or hpfs, ntfs, ext2/3 (no limit) please make sure your ftp client isn't broken

][ronMan
02-18-2003, 08:38 PM
So after doing some testing with the tool, results were as I expected. I can ftp a show off my standalone and ftp it to my DirecTivo and it works beautifully (although for a half hour show, the Tivo thinks it's an hour, however the green in the progress bar stops at 30 mins like it should). Also as I expected I cannot ftp stuff off of my DirecTivo and get it to play on my standalone. I had suspected this would be the case from earlier work I was doing with insertion. Overall, things look great.

What I'd love to be able to have is a tool that I could give an AVI, MPEG, etc along with some details about it, and could then covert that to the TMF format for insertion onto the Tivo. i.e. Download a movie trailer from the net then insert it on the Tivo.

Keep up the great work...

Derek aka ][ronMan aka DSmooth

rc3105
02-18-2003, 11:29 PM
ok, couple of people have asked, so here's the info.

tmf stands for tivo media format. the NAME was AlphaWolf's idea :-p the original code to generate it was done by tivodvlpr & mbm.

edit: sorry, how did I forget to mention dtype? duh@me...

a .tmf file is a regular tar file that contains the following, in this order.

showing.xml
part00.ty
part01.ty
...
part04.ty


showing.xm is a xml file with the info from the nowshowing database stored as regular text. title, episode, description, everything. the .ty parts each represend an individual fsid as it's stored in the tivo.

pretty simple really. the xml is a bit hairy, but easy enough to parse out the title / description etc and it contains all the info the tivo stores about a recording.

--
Riley

][ronMan
02-19-2003, 08:24 PM
Originally posted by ][ronMan
So after doing some testing with the tool, results were as I expected. I can ftp a show off my standalone and ftp it to my DirecTivo and it works beautifully (although for a half hour show, the Tivo thinks it's an hour, however the green in the progress bar stops at 30 mins like it should). Also as I expected I cannot ftp stuff off of my DirecTivo and get it to play on my standalone.

Correction... With the 1.1.1 version I am also able to ftp a show off of my DirecTivo and ftp it onto my series 1 standalone. This is really starting to rock!

Derek aka ][ronMan aka DSmooth

][ronMan
02-19-2003, 08:27 PM
Originally posted by AlphaWolf
DT wouldn't work on an SA because of the fact that its very non-standard. Again, doing this would involve being able to generate tystream archives.

Also another comment on the binary optimizations: It would be better to do it anyways, b/c you could further minimize the chance of pausing or any other errors.

See my last post... This is working OK with 1.1.1 and greater... :)

Derek aka ][ronMan aka DSmooth

stealthdave
02-19-2003, 08:36 PM
Originally posted by rc3105
PASV is most likely the problem it's still a bit iffy. there it was trying to connect to itself. use regular mode instead.
Sure enough, as you were posting this I had just tried turning off passive mode, and that fixed the problem. My guess is that's what the majority of the client-specific problems are (I can't be sure of course, but it seems likely).

Would it be possible to add a short "how-to" to the readme.txt file? IMO, it should include the following:
[list=1]
command-line server configuration
turn passive mode OFF
filenames (i.e., command-line clients need only use the FSID listed as the filename to get)
importing vs. exporting
deleting programs (and how to prevent it (?) )
other
[/list=1]
Might save a lot of repetitive questions. :) Btw, fantastic work!!! This blows away my old export-via-nfs method! (For some reason nfs is really slow on my Tivo, even with rsize=8192&wsize=8192.)

- Stealth Dave

][ronMan
02-19-2003, 08:39 PM
Originally posted by alanjudy
The other question is, while I have seen that someone has gone from SA to DTivo, has any sucessfully gone from DTivo (2.5.2) to SA (3.0.1)

Thanks

=alanjudy

alanjudy,
This is my exact setup, and as you see from my post above, with versions 1.1.1 and beyond it works in bot directions...

Derek aka ][ronMan aka DSmooth

Hi8
02-19-2003, 09:34 PM
Well; I'm xfering a .tmf originally from my dtivo to my SA. It's the same one that created my 0Byte file, undeletable listing. I found out that my SA is MUCH slower to respond and transfer to. Both tivos are 3.1 - I just didn't WAIT long enough for it to allocate space .. It is a 2hour movie, so it appears it;s going to take some time. I'll report back on results.

here is a dump of the transfer sofar...

echo to verify: PORT 192,168,1,49,7,156
ip is 192.168.1.49 port is 1948
200 PORT command successful.
echo to verify: STOR Hackers .tmf
150 Opening BINARY mode data connection for "Hackers"
entering dumpxml
segment is showing.xml
leaving dumpxml
entering parsexml
title is "Hackers" - episode "unknown episode"
leaving parsexml
attempting to create recording - success
attempting to set recordingfsid - success
attempting to set bitrate & such - success
attempting to create a dummy station - success
attempting to create a showing - success - trying to set showing info - success
trying to create the program & set the info - success
attempting to showing set program - success
created Hackers : unknown episode : fsid# 2842851
entering tmf2fsid
starting segment part00.ty
entering AddPart
trying to open recordingfsid
success
createing recording ? success
trying to add part - success
trying to mfs streamfill allocate space
success - created newfsid 2842854
trying to set file to fsidtwo - success
leaving AddPart

rc3105
02-19-2003, 10:31 PM
the initial space allocation can take bit, especially if the db is busy & it has to retry a few times.

I'm working on that issue

set your ftp client timeout to at least a minute, 2 ought to be worst case.

--
Riley

Hi8
02-20-2003, 02:29 AM
Originally posted by Hi8
I'll report back on results.



It works perfectly! Plays and looks as good if not BETTER than anything ever recorded on my SA, well, there you have it!

the down side is xfer speed. it took about 5hours to xfer a 1h45m movie to my SA. What is surprising is how fast it downloaded from my dtivo, less than 25mins.

I noticed the time difference the listing info reported a 1hr recording yet the green progbar reported correctly at 1:45.

Great Job RC!

here is the ftp sessionlog info:

sending Hackers .tmf as Hackers .tmf (1 of 1)
PORT 192,168,1,49,7,156
200 PORT command successful.
STOR Hackers .tmf
150 Opening BINARY mode data connection for "Hackers"
Transmitted 1920999936 bytes in 17739.4 secs, (1.04 Mbps), transfer succeeded
226 File transfer complete
PWD
257 "/" is current directory.
TYPE A
200 Type set to A.
PORT 192,168,1,49,8,109
200 PORT command successful.
LIST
150 Opening data connection for file list.
Received 1068 bytes in 0.1 secs, (100.00 Kbps), transfer succeeded
226 Transfer complete.
QUIT
shutting down & exiting


**** please note the last entry ****

on my SA it shuts the server down at the end of a session, on my dtivo it just returns a 221 call 'server closing control connection' the former is abit bit of a disadavantage as I have to go in and manually restart it for each session. On my dtivo it starts on bootup, and runs along side my tivoftpd only on port 24.

using the same client WS_FTP Pro, and both my tivos are running 3.1, and mfs_ftp 1.1.2

AlphaWolf
02-21-2003, 03:45 PM
ok heres an explanation of the file formats

.ty is a plain binary dump of the individual tystream parts combined into one file. This is the legacy method of extracting tystreams, and its my personal recommendation that nobody dumps/extracts/archives to this format anymore.

.tmf is a tar archive of the above, only a few additions. Each part of the tystream is stored as an individual file inside of this tar archive. On top of that, there is an XML dump of the recordings guide data. This provides a lot of the flexability needed for insertion, and can also be used to help automate dvd authoring.

That said, converting between the two is simple. To convert from .tmf to .ty, you can just untar the archive (winrar does this), discard the showing.xml file, and merge each of the .ty parts into one file. To go from .ty to .tmf, you would need to break apart the tystream parts at their boundaries and into individual files, and generate the xml data by manualy filling in the title information, description, genre, etc etc etc, as this is not stored in a tystream dump.

There is one exception to this however. Any tystreams that were extracted using jdiners tytool will have issues with the master chunk. These streams may be converted to .tmf, but they probably won't be able to be reinserted into a tivo properly, because they were not extracted properly to begin with. This is an issue with the way tserver exports tystreams.

rc3105
02-21-2003, 11:01 PM
Originally posted by laserfan
I'm still confused about why/how tar_server (tivoweb module) is used in all this. Why is this needed? It's mentioned in your comments/notes RC but there's no discussion of what module is used for what. Is Tivoweb supposed to be running when you do an ftp transfer to-or-from?
Sorry, but I can't sort out how this thing works.
mfs_ftp uses the same xml dumping routine (tcl) and the same mfs_tarstream (binary) as tivodvlpr's tivoweb tarserver module, the idea being that files produced are identical and interchangable with the (eventual) tivoweb insert module

tivoweb mfsstream produces regular .ty files.

mfs_ftp does not need tivoweb running.

zobetron
02-22-2003, 03:03 AM
For what its worth,

I just downloaded a 5.59 GB (6,008,352,768 bytes on disk) to my ntfs formatted drive. When I opened it up with winrar, it (winrar) said the file was corrupt. I got a similar error with winzip.

So I tried a couple of other windows based decompressors and they worked just fine. (Specifically 7-Zip File Manager from source forge (free) and WinTAR-SCSI (trial)).

Z

ronnythunder
02-23-2003, 01:16 AM
ok, i'm a dummy. i've got this installed and have successfully pulled off a .tmf, and i want to make a dvd.

the tmf is a 30 minute show, and is in two parts in the tmf. so, i have two .tys that i need to concat and what - vsplit and mux? how do i put the two ty back together?

sorry if this has been covered; please feel free to point me to a faq

ronny

zobetron
02-23-2003, 02:11 AM
Well with the windows command line it goes like this:

copy /B C:\part00.ty+C:\part01.ty+C:\part02.ty c:\combined.ty

bfdhe
02-23-2003, 04:42 PM
Just as an FYI: I am running TiVoWeb, mfs_ftp and ftpd simultaneously with out problems. I just started mfs_ftp on a different port. I have extracted 6 shows today without any problems.

geowar
02-27-2003, 01:07 PM
https://www.panic.com/transmit seems to work good for me.

Spod
02-28-2003, 01:12 PM
Originally posted by mini__me
Strange my problem is slightly different to yours as I can get them to run from 1.2.1 and I can browse around with the phoenix and shutdown working fine and I'm able to see shows however I get this critical error when I try to download and upload just hangs as if waiting for something that never happens.

Are you running the .tcl through dos2unix just in case? I am doing this to every file now and have had no problems of file not found anymore.

I had *exactly* the same problem. As soon as you turn of the multiple connection option in FileZilla it all starts working again....
( at least one place was under Queue, Use Multiple Connections )

Rgds,

R.

majictek
02-28-2003, 11:30 PM
I've noticed a number of posts about why the mfs_ftp.tcl script doesn't run for some people. I think I can clarify the confusion.

First, what's really going on:
A text file on a dos system has a <cr><nl> at the end of each line. Unix expects just a nl.
Looking at the 1st line in mfs_ftp_1.2.3.tcl, we see a line that is supposed to start the tcl interpreter:
#!/tvbin/tivosh
This invokes a program called /tvbin/tivosh. So
far this is 1st grade stuff. Now for the fly in the ointment...
If the file is a DOS format file, the <cr> before the <nl> becomes part of the filename, and looks like this:
#!/tvbin/tivosh<cr>
Yes, in UNIX a <cr> *can* be part of a filename. Of course the tivo can't find the tivosh<cr> program, and, well, end of story.

The dos2unix command probably strips the <cr> (I dont use this utility myself), or Unix purists might use the command
"tr -d '\r' <mfs_ftp_1.2.3.tcl >mfs_ftp_1.2.3_unix.tcl"
to do the same thing. In either case, this will remove the <cr> and fix the problem.
----------------------------------
why does the command "/tvbin/tivosh ./mfs_ftp_1.2.3.tcl" work? because TCL doesnt
need that filename, and doesn't (usually) care
about the extra <cr> in the file.
----------------------------------
Why is this happening? I looked at the unziped files on my unix system, and find that the mfs_ftp_1.2.2.tcl and mfs_ftp_1.2.3.tcl files are in 'DOS' format. In order to work the <cr> needs to be stripped.. somehow...

IF you use FTP from a *DOS* system up to the tivo
(unix) system in *ASCII* mode, the format translation is done for you , and things should work out of the box. If you use *BINARY* mode,
OR come from a UNIX system, FTP believes no translation is needed... and the extra <cr> is left intact (booo).

If you are running unix, or have the 'od' utility on your Tivo, the following command will show the line terminators and you can see it for yourself:
od -c mfs_ftp_1.2.3.tcl | less
-------
Conclusion: Just run the dos2unix command, or the tr command to remove the 'cr', and enjoy the use of the program!:cool:

kmt
03-03-2003, 10:31 AM
Originally posted by jodell
[B]Has anyone found a suitable Linux GUI FTP client that supports FXP?

gtkfxp works

http://gtkfxp.sourceforge.net/

TheD0CT0R
03-07-2003, 11:17 AM
I haven't downloaded or tried this yet, in fact, I just found it, but I have a question...

With this, and a corresponding client/server app sitting on another tivo, how hard would it be to have Tivo A pull a recording from Tivo B, then start playing that recording once it has buffered enough data, ala HMO for Series 1...

I think the real challange would be doing that from the Tivo interface w/o having to actually log into either box.

Thanks...
Doc

wactuary
03-10-2003, 12:28 AM
Hi,

Just wanted to report in another successful installation of this program. I really love the feature, you did a great job.

Just to let you know a little about my experiences:

I am using 1.2.4 on a SA Series 1 (S-2000) v3.0. Turbonet card using the standard 3.0 drivers. (Should I change these?)

I have to start it using tivosh mfs_ftp.tcl, and I edited the script to call tivosh directly also.

I am getting 1.2 to 1.5 MB/s downloads using FileZilla 2.1a, but uploads are between 200 and 300 kB/s. I haven't gone through the debug suggestions a few pages back yet.

A few times I seem to have locked up the program, either from aborted transfers, or unknown causes. Sometimes using the abort.txt file fixed it. Sometimes I was able to telnet to 3105 and issue the exit command. Other times, however it wouldn't accept new connections and never responded and I ended up killing the process (and causing a TiVo reboot).

One file which NowShowing reports as 5GB shows up on the tmf directory as being only 1GB. When downloading it actually is splitting itself into 2 pieces. The first 1GB file gets then conflicts with the remaining 4GB file at the last moment. Download fails with a "Critical transfer error" message. I need to kill mfs using the abort.txt file to recover.

All other files seem to report the correct size. I don't know if this is a FileZilla issue or the mfs_ftp package. Let me know how I could trace down better debug info.

Thanks again for the great package.

Wactuary

tlphipps
03-11-2003, 12:37 AM
phrend,

I was one of the few experiencing the critical transfer errors. I have switched clients to WS-FTP and am having NO problems whatsoever. However, if I try to use filezilla, I still get that error.

I haven't had time to really debug this situation, but apparently it's a client issue, NOT necessarily an mfs_ftp error.

Hope you have luck fixing your problem.

nfg2u
03-13-2003, 11:07 PM
Just wanted to be one of the folk that says thanks for all the work on this. It works great! mfs_ftp is the icing on the cake for the tivo network!

lonaman
03-20-2003, 11:30 PM
I have not yet attempted to re-insert a program, but I have a question regarding the process. Say I download a tmf file, extract the ty files from it, and then use cat to join the ty files into one big ty file.

If I replace the multiple ty files in the tmf with the large ty file I created and then re-insert the tmf file, will this work? I guess my question is whether or not the re-inserted tmf needs to have the same number and size ty files as the original tmf.

Thanks.

dlang
03-21-2003, 01:35 AM
I don't think it will work if a pice is larger then 512M I don't know if it will work with multiple pieces smaller then that (it's designed around multiple 512M pieces plus one smaller piece at the end)

Martlet
03-21-2003, 01:52 AM
Can you put a cat'd ty back into a tmf and reinsert?

No. (unless you segment it into 512mb chunks and generate the required xml data for Now Showing to display).

Does it need the same sized segments to reinsert?

Yes.

kangolo
03-22-2003, 08:42 PM
I'm running a stock Philips 20hr SA (no hd upgrades) with a turbo net card - anyone else got this setup?

I'm getting 500k/sec - which is fine, but I'm seeing others get 6x that speed.

Is my hub the limiting factor (compared to a switch), or the hard drive?

Riley - this is a truly great app - takes me 11 minutes for a half hour show at basic quality - takes about a minute in tytool 6r3 to convert it to a m2v which I can watch through media player.:)

Time to start taking tv shows to work on the ipod to watch at lunchtime :D :D

mpauley
03-22-2003, 08:50 PM
Originally posted by kangolo
I'm running a stock Philips 20hr SA (no hd upgrades) with a turbo net card - anyone else got this setup?

I'm getting 500k/sec - which is fine, but I'm seeing others get 6x that speed.

Is my hub the limiting factor (compared to a switch), or the hard drive?

Riley - this is a truly great app - takes me 11 minutes for a half hour show at basic quality - takes about a minute in tytool 6r3 to convert it to a m2v which I can watch through media player.:)

Time to start taking tv shows to work on the ipod to watch at lunchtime :D :D

kangolo,

I have a 14 hr Philips SA with a TiVonet card and Im getting 800k/sec ... with a Turbonet on my other TiVo, I get about 1.5M/sec.

Mickey

Share your shows at LookMa-NoHands.Com (http://www.LookMa-NoHands.Com)!

kangolo
03-22-2003, 09:35 PM
950k/sec on another machine .

So,
600k/sec from AMD K6 400 running XP with a realtek card
950k/sec from a P2 400 running XP with a 3com card.

Both with loads of ram, and maxtor 40gb's as the data drive, os on a seperate one

Time to replace those crappy realteks

kangolo
03-24-2003, 04:01 PM
Originally posted by Neutronflux
Hey Riley -- I am having trouble with tystudio extraction -- and I'm dying to get mfsftp running. I've tried several programs, all with the same result. [smart ftp, filezilla, etc]

I can login to the tivo and it lists all directories and recorded shows [FAST] -- however, when I try to transfer over a recording to my pc, the time counter just rises and eventually times out, creating a 0kb file. Has anyone else seen this? I would love to get this program running so I can extract files for use with tystudio.

Thanks!
Kev

Have you got ftp set to use passive (PASV) connections?

Do you have a firewall?

Commands and listings are sent over a seperate TCP port to data - so one may be being blocked while the command conncetion works ok

FreydNot
03-24-2003, 11:20 PM
Originally posted by Neutronflux
I can login to the tivo and it lists all directories and recorded shows [FAST] -- however, when I try to transfer over a recording to my pc, the time counter just rises and eventually times out, creating a 0kb file. Has anyone else seen this? I would love to get this program running so I can extract files for use with tystudio.

That sounds exactly like what happens if you have a DirectTivo with out the noscramble module/patch. That's not the problem is it?

osetivo
03-24-2003, 11:32 PM
Bingo Doc!

Take a look here.
http://www.dealdatabase.com/forum/showthread.php?s=&threadid=19918&highlight=noscramble

Neutronflux
03-26-2003, 03:06 AM
Riley - thanks so much for your great programs and all your help, it's really appreciated. People on this board really do rock!

This is the readout from my telnet into my tivo box:

/var/hack/mfsftp# ./mfs_ftp.tcl 2000
00:00:00 - entering init_mfs_ftp
00:00:00 - entering background
/var/hack/mfsftp# 00:00:00 - entering init_mfs_ftp
00:00:00 - entering background
00:00:00 - ping - mfs_ftp running on 127,0,0,1 : 2000
00:00:00 - mfs_ftp version 1.2.5.1 - checking tivo system sw
00:00:01 - version 3.1.0-01-1-011
00:00:01 - setting mfspath to /Recording/NowShowingByClassic
00:00:01 - insert throttle is 0, == 0 ms
00:00:01 - extract throttle is 0, == 0 ms
00:00:01 - entering reset_mfs_ftp
00:00:01 - port 2000 control socket initialized - ready for connections
00:00:01 - entering buil_dir_LIST
00:00:01 - entering update_rec_fsids
00:00:01 - getting current rec_fsids list
00:00:02 - rec_fsids doesn't match cached_rec_fsids, deleting cached info
00:00:02 - entering build_rec_list
00:00:02 - from mfs "/Recording/NowShowingByClassic"
........................................
00:00:15 - build_rec_LIST complete

****

as you can see -- here I selected to use port 2000... the mfs_ftp is using the 127.0.0.1 loopback address [dhcp] is this my problem? Should I specify the Tivo's IP address [192.168.1.101] manually and if so, what's the syntax for that?

rc3105
03-26-2003, 03:45 AM
ports below 1024 are reserved for unix services. 21 is the port assigned to ftp. if you use a port other than 21 it should be > 1024. the default ftp data port is portnum-1. so for control port 21, 20 is used for data. for 3105, 3104 is used for data

passive mode & custom port usage is in the help/readme for your particular ftp client

the default ip is set to 127.0.0.1 on startup, the first time a ftp client connects it auto-detects the tivo & client ips

n4zmz
04-03-2003, 02:39 PM
When moving a stream from a DTiVo to a Standalone, a couple of inconsistancies were found.

The mfs utilities were all on an NFS mount on both machines. 1.2.5.1 was used in the test.

Rating not set. (PG-13)
Advisory not set. (AC, AL)
Classifications missing (movie)
Genre missing (action/adventure)
Attributes missing like CC
Save until not set. (until I delete)
Time was off by 1 hour. (4:30 displayed instead of 5:30)

The tzos was not set correctly. It was set for central and I'm in eastern. But that should not have changed anything.

Normally I would have fixed this myself, but finding the sources which were used has been a real task. (One which I haven't completed yet... pointers/source code would be helpful)

The only real pain is the standalone is using a tivonet card (5+ hours to inserta 2 hour show).

rc3105
04-03-2003, 04:31 PM
not all info is restored. never has been, I may or may not chase down the details. there are differences in the info between 1.x-3.x sw. the "universal" info (and what I look at) is restored. nearly everything is included in the xml and available for archiving / catalogging like tmf2ty does. (side note, the few attributes that aren't extracted were left out to maintain compatability with tmf 1.0 as implemented by mbm/tivodvlpr)

the compiled binaries only do the actual extract/insert of video data. unless you really know the layout of mfs don't mess with those. using improperly recompiled versions can trash mfs and nuke your tivo.

the dbinfo is restored in the mfs_ftp tivosh script. it's easy to modify with joe and the info you need to restore the missing attributes is in tivoweb

Will
04-08-2003, 07:43 AM
god, I am having a bitch of a time finding an ftp proggie that plays nice with this on os x....

Let me preface this by saying that 90% of the people I've helped, either on Wintels or Macs, are making basic mistakes and not reading carefully. Nothing to do with any defects in mfs_ftp or the ftp Clients.

But undeniably there are problems beyond that, that are complicating the situation, and Mac ftp Clients that work with mfs_ftp are rare indeed. Some people are doing everything else right and that is their only problem. But many are compounding a misunderstanding of the basics with ALSO using a non-friendly ftp Client on top of that; real trouble.

Fetch 4.03 works pretty well with mfs_ftp. Navigating the subdirectories is a minor nuisance, you have to do it manually. Also I occasionally have trouble with the first digit of a filename (fsid) getting clipped off and I haven't figured out how to enter a filename or a complete pathname manually for a download, in a way that gets accepted. But all-in-all Fetch 4.03 is a good solution.

There's a little gem called SimpleFTP (version 2.01) that works _very_ nicely with mfs_ftp. I like it a lot. Doesn't chop off that first digit. But it aborts file downloads at 2 GB! Hello!?

At least one poster here has said Transmit works. Not for me. It has the redundant subdirectory problem that many ftp Clients have with mfs_ftp.

I've tried more than a dozen other Mac ftp Clients, all NG. Sometimes the redundant subdirectories, to infinity. Some simply can't make any sense of the host's directory structure/syntax at all, some seem to see everything OK up until download time. Some clip off the first digit of all the subdirectories. BTW many Wintel ftp Clients have the same problems.

Riley seems to be one of those wonderful practical, pragmatic geniuses who have a good grasp of currently available tools and technology and a knack for putting them together to provide useful products. But brevity of coding and compliance with standards and protocols (ftp in this case) may be beyond his attention span. If this were a commercial product I was developing I'd have an anal analyst following him around and cleaning up after him. I'd pay them both well, but Riley quite a bit more.

As it is, for a free product he's hacking together for us, I guess as labor of love, I'm very grateful for it and perfectly willing to grope around for an ftp Client that can cope with what he's giving us.

Kudos and thanks to Riley!

rc3105
04-08-2003, 08:30 AM
LOL - if this were a commercial project I'd fire MYSELF for such sloppy coding! :p

1.2.5.1 has been tested here with command line ftp from OS-X 10.2, fetch 3.0 from the jaguar 9.x classic environment and 7.6-8.6 installed on 6100/7500/8500/9500 & beige g3's.


--
Riley

TivoDvlpr
04-08-2003, 06:37 PM
Well Rc, I see you did it. You realized the dream I had for tivo extraction so-so-long ago! And you even posted a reply to that response duly noting the idea!

(see post:http://www.dealdatabase.com/forum/showthread.php?s=&postid=52182#post52182)

Cheers to you buddy!

Sorry I haven't been around, I'll be back in the summertime when I have more time available and my daughter is not in school.

later!

rc3105
04-08-2003, 10:05 PM
folks, for those of you that don't know tivodvlpr (with help from mbm) is the author of mfs_import, which is what mfs_stdinsert is based on, which is the cornerstone of mfs_ftp's insert ability.


I suspect his original idea was more of a single coherent binary (as is my eventual goal) the current version is really just a proof of concept test that got loose! :D

--
Riley

stumbleshoe
04-10-2003, 03:18 PM
Thanks Riley - this is great stuff. Do you take Paypal ?

I've been extracting like mad from my SA 1 w/ 3.0 - planning to setup my g/f SA soon so we can exchange stuff.

Any suggestions for setting up some form of password protection or should I look into setting up an FTP proxy ?

rc3105
04-10-2003, 10:39 PM
Originally posted by stumbleshoe
Thanks Riley - this is great stuff. Do you take Paypal ?

I've been extracting like mad from my SA 1 w/ 3.0 - planning to setup my g/f SA soon so we can exchange stuff.

Any suggestions for setting up some form of password protection or should I look into setting up an FTP proxy ?
yep, can never have enough toys ;)

the best way to move shows around it to go from tivo-pc, then pc-pc (ftp, p2p, e-mail attachment, sneakernet, carrier pigeon, whatever) then pc-tivo

password or access control depends on the method you use. you can use tivo-tivo directly across the net, but since there's no resume implemented yet it's just asking for a lot of partial transfers. (giving your tivo full net access is a bad idea anyway)

zodiacal
05-04-2003, 02:17 AM
the dox for mfs-ftp say that the .tmf are for "streaming from one tivo to another tivo / pc". does that mean i can stream .tmf from my tivo to my pc? or does that mean that i *use* a pc to stream from one tivo to another tivo. i want to watch my tivo shows on my windows pc with out downloading them first.

mikertx
05-04-2003, 05:06 PM
What I really want to do is store my collection
of tmf's on my RAID-5 linux fileserver and
stream them back to the tivo. I want them
to appear in the "Now Showing" list on the
tivo and stream back when selected from
the tivo menu.

Seems to me that the crucial missing piece
is whatever it takes to initiate the transfer
from the Tivo side when the show is selected.

I'm betting that Riley has this working now :)

Thoughts,
--
mikertx

rc3105
05-05-2003, 05:08 PM
Re: streaming FROM pc

all you need is a tivo osd ftp client for browing a remote server (mfs_ftp or regular ftp either) and initiating the server->tivo transfer

anybody feel like whipping one up? (mine is SO ugly... :p )

Will
05-14-2003, 02:47 PM
Originally posted by rd001
which Windoze FTP clients are working... this thread is so huge and so varied it's pretty hard to find things in here.
Kind of a moving target. In an appropriate distribution of labor here, I say keep the project manager moving forward, and we'll all pitch in on some of the administrative details like keeping track and sorting through the thread for info. Not unreasonable to ask of users who are getting an _extremely_ useful piece of custom software!


Originally posted by rd001
to make sure that I have your latest binaries.
Always in the first message, right?


Originally posted by rd001
Any progress on better compatibility for all FTP clients?
Again, why take time away from time spent on functionality to deal with something that's easy enough for the users to deal with on a trial & error basis? That's the kind of detail you clean up after the whole thing is done.


Originally posted by rd001
especially the Mac clients.
Well that's easy enough. Fetch works nearly perfectly. It screws up a filename now & then (drops digits from the number*) so use the OSX command-line ftp client to pick those files up. Do a directory, edit-copy the filename, edit-paste for your source and destination.

* edit: for anyone interested, Fetch's problem is clearly related to the cycle of the moon in that programs recorded early in the month can't be downloaded while those mid/late month download fine. Now some might say it's simply a question of field length and position, that when the total length of the date field includes only one digit for the day (e.g. 4/1/03) the first digit of the filename shifts to the left and gets clipped off, while with a date of 4/11/03 the filename shifts over to the right by one digit (where it belongs) and the filename begins where expected and everything is fine. I prefer superstition: it's the moon. Among other things it prevents me from worrying about what's going to happen when we get into the late Fall with 2 digit months.

geowar
05-14-2003, 03:51 PM
I've found transmit to be the best:

<http://www.panic.com/transmit/>

--
Enjoy,
George Warner,
Schizophrenic Optimization Scientist
Apple Developer Technical Support (DTS)

Will
05-14-2003, 05:25 PM
Originally posted by geowar
I've found transmit to be the best

I never had any luck with Transmit and mfs_ftp. Gives me a valid-looking list of files, but the download starts going into an endless sequence of empty subdirectories and then Transmit quits after a few seconds.

Could you describe what you do to make it work for you?

geowar
05-14-2003, 05:59 PM
Other than run it on port 3105 (because I already run tivoftpd on port 21 (23?) I don't do anything special. I didn't do any extensive testing thou since I normally use vserver and the mfs_streams, mfs_extract, etc. tools.

Looks like I can see the /Recordings/NowShowingByClassic listings ok.

MyKroFt
05-26-2003, 01:55 PM
Getright is a web download manager - not a straight FTP client. Granted - yes it can get ftp files from off the web. But it does the standard %20 converting thingie for spaces in the filenames. I dont think MFS_FTP understands that.

I have been using FlashFXP very sucessfully. From extracting .ty files to inserting a .tmf from one tivo to another.....

L8r
Myk



Originally posted by zodiacal
how come getright (a 3rd party download manager application)can't download ty files from mfs-ftp server? it seems like it doesn't understand that there are spaces in the file names and tries to download "ty\FSID" and doesn't download the whole file name. it just errors out. any ideas?

John Silver
05-28-2003, 02:11 PM
Riley,
Thank you for a great product!

All,
I installed mfs_ftp recently on my two DTivo's (T-60 and a Hughes). Both machines have xPlusz 3.1 over 25xtreme, both are legal subs with locals, etc. xPlusz telnet, web, and ftp work fine. I'm using static IPs.

I am using SmartFTP and can download *.tmf files from either machine to the other (FXP works fine), or to any PC without problems. Transfer rate is between 1 and 1.2 gig Tivo <-> Tivo, Tivo <-> PC, and there are no errors. Am I correct in assuming that the only file needed to be copied is the tmf?

I have been waiting for the transfer to complete before trying to play it back. When I go to the destination Tivo to playback the new mfs entry, it shows up fine in Now Playing, gives a good synopsis, and when selected, prompts me to delete, and I get the following message:

"The Recorder was not able to record this program because there was no video signal on the channel... etc."

I searched everywhere in the forum, and the only reference I could find was related to scramble/no scramble incompatibilities. Both machines are definitely using the default scramble.

Anyone have any ideas on what I'm doing wrong?

SR712
05-28-2003, 04:16 PM
I don't believe that you can exchange streams of scrambled video between TiVos... Hence the word scrambled. The common denominator, is that both systems must be running NoScramble, and the stream must be likewise.

John Silver
05-31-2003, 08:52 AM
SR, right you are. I unscrambled both machines, recorded, and copied. Worked like a million. Thank you.

willieb
06-01-2003, 08:41 PM
Riley,
Question. Make that 2 questions. I have my setup working really well with mfs-ftp but I have changed file paths etc because I don't like writing files to my c drive (small partition) and every time I upgrade it means changing all that again plus refiddling with smartftp (for some reason it never works right out of the box for me).

The new mfs_ftp sounds like a good improvement if I am reading you right, but I want to check this before I upgrade. I think I am reading that the new tmf2ty extracts the ty file from the tmf and then adds the xml data from the tmf to the end of the ty file. This allows tystudio/tytool to process the ty file directly and just exit when they reach the "crap" that is the xml data at the end of the file. Also, the ty's with the xml data appended to the end can now be reinserted to the Tivo with mfs_ftp 1.2.6. Sooooo.... does the new mfs_ftp output these ty files directly, thus saving me the step of converting my tmf to a ty before processing? Also, should I just convert my library of tmfs to this ty format, delete the tmfs and, if mfs_ftp can do this, only extract in the new ty+ format. Or is there something better about the tmf format that will be taken advantage of in the future. Thanks

Willie

rc3105
06-02-2003, 01:46 AM
yep, tmf2ty just extracts the parts & appends the xml data to the end, HOWEVER, it also appends 512 #'s before and after the xml text making it possible to easily identify/locate/retrieve the xml. if you just cat the xml onto the end of a ty w/o the #'s it WILL NOT be restored on insert by 1.2.6b

tystudio/tytool/mplayer/vlc all handle extra junk at the end of a file automatically so mfs_ftp actually produces ty+ when you extract ty. recordings are available with both suffixes for sorting convenience if you extract ty with other utils

thanks to cwingert's ty enabled mplayer & mythtivo we can watch ty+ recordings w/o any conversion.

extract, archive, & restore as ty+ works nicely. tmf allways be more usefull for tivo->tivo or pc->tivo streamviewing (think hmo) because the xml is stored at the beginning of the file instead of the tail end

bjarne
06-02-2003, 10:49 PM
First, this is great stuff, I have been using it for a while with really good results. Thanks RC3105.

Now for my question: In the past I have tried with success to insert tmf files, and also transfer directly from one of my DTiVos to the other, not that I needed to, but it was fun !!!

When I did the insertion, the show would show up in Now Playing after transferring a little bit, and I could start watching almost immediately. When I do the insertion with the new ty format, it does not show up untill the whole file has been transferred. Is this a limitation of the new format, where the xml data is at the end?

It is really not a problem for me, just wanted to understand it.

It really was cool, though, to start watching the show while it was still being transferred.

Bjarne

SirTrini
06-02-2003, 11:08 PM
I let it stream for about a minute then go to the now showing... it shows up as a "yellow" item.

RC3105 this program is awesome and is the only program I use for extraction. I love the idea of having one program on the Tivo performing multiple functions. Thanks for creating this!!!

I do have a couple of questions for you though...

1. are you planning to incorporate some more of the show information into the insert? i.e. the show description or is this for cross compatibility reasons?

2. I had mentioned a while back about streaming programs and when you "catch up" to the green bar the Tivo asks you to delete the program. i.e. while watching the show (say... after a minute into the streaming) the show will report itself as 10 minutes when it is really 30... the duration of the show does not change to reflect what is really on the Tivo at the current time. Is this something you still think you can fix?

3. I had my friend directly connect his Tivo to his DSL modem in an attempt for me to download a ty file from him.... this did not work... we got a timeout from the server.... When I ask him to go into FTP from his lan MFS_FTP works.... is there a timeout value set somewhere?

Thanks for your time.... I appreciate all you hard work!

Mike

rc3105
06-03-2003, 05:03 AM
Originally posted by bjarne
Now for my question: In the past I have tried with success to insert tmf files, and also transfer directly from one of my DTiVos to the other, not that I needed to, but it was fun !!!

When I did the insertion, the show would show up in Now Playing after transferring a little bit, and I could start watching almost immediately. When I do the insertion with the new ty format, it does not show up untill the whole file has been transferred. Is this a limitation of the new format, where the xml data is at the end? Bjarne

if you watch the nowshowing list as you insert the recording will appear with the name of the file actually being uploaded, then when it reaches the xml at the end it will change to the correct info. you can still start watching (streamview) as before ;)

tards
06-03-2003, 08:39 PM
OK, I think I got this installed and I can run it. I also got smart_ftp and can ftp to the tivo. Now what? Can somebody post some "newbie" or "for dummies" instructions on USING this program?

Example questions:
1) where are the program files (tmfs?) located on the Tivo (I probably should know this but I don't).
2) What's all this command line ftp stuff? Can't I just drag/drop?
3) how to download and re-upload using smart-ftp...in layman's terms.

I've been able to install and run every DTivo app I've tried without asking any questions. But this one gives me some problems. And how do I stop the program?

SirTrini
06-03-2003, 10:24 PM
In MFS_FTP if you were to use an FTP prog, then all you would do is "click" on the Shutdown folder and that woudl stop MFS_FTP.

In DOS, I would assume if you were to change dir's to "Shutdown" then that would stop the program.

I think in the simpliest of terms, the shutdown folder is "like" to a function that excutes the shutdown.

You can drag and drop, I would suggest download SmartFTP and use that as it seems to work the best.

The TMF's are an "older" format that I think RC3105 will stop using. TY's now include the show info so that is all that is needed. Once again:

1. Launch SmartFTP
2. FTP into your Tivo on the port that you setup MFS_FTP on (newer releases has this as 3105 as a default) - anonymous logon is fine
3. you'll get a list of folders double click on the TY folder
4. you'll see a list of shows drag it to your HD and it will start to download
5. to shutdown MFS_FTP double click on the folder "shutdown" - I don't do this I just disconnect from the Tivo. I want to be able to reconnect without having to restart the program.

I hope this helps!

tards
06-04-2003, 06:03 AM
Whoah, it works! I was not using port 3105 in my ftp. Now it works great!! Thanks...this is a NICE app!!

johnmacd
06-04-2003, 08:59 AM
OK since I am a newbie this may sound like a dumb question.

The TMF's are an "older" format that I think RC3105 will stop using. TY's now include the show info so that is all that is needed.
Wha is the difference between TY and TMF? I think there were a few other folders that I saw as well (I'm not where I can access my Tivo right to to look), what are those formats? I was using the TMF's and it worked great.

John

SirTrini
06-04-2003, 10:40 AM
TMF's work great for insertation back into the Tivo, but other conversion apps: TYStudio/TYTool etc... don't use this format yet (not sure if they will either).

To get a TMF into a TY you would decompress it with a program that support RAR, join the pieces back together and discard the XML information. RC3105 did develop a program to do that in one step and it can be found on the main page to this thread.

RC3105 updated the original TY format to include the TMF info, so now TY's can be inserted back into the Tivo, and run through regular conversion programs. IMHO it's the best of both worlds and allows the most flexability.

TYX is a developing format - I don't know much about it, the other folders TXT is a text file of the show info, XML is an XML file of the show info, shuthdown is to stop the server process and I think phoenix, is shutdown and restart (not 100% sure).

rc3105
06-04-2003, 01:40 PM
Originally posted by SirTrini
To get a TMF into a TY you would decompress it with a program that support RAR, join the pieces back together and discard the XML information. RC3105 did develop a program to do that in one step and it can be found on the main page to this thread.
don't discard the xml!!! append it to the end of the tyfile so it can be restored!!! :D

tmf is a tarball with the the recording parts encapsulated. building tmf's from ty or ty+ is easy enough, but why not just insert a ty+? (easier to build with cat)

the real advantage tmf has over ty+ is that the xml is at the beginning of the file instead of the end. this allows the info to be restored before the insert starts. I was hesitant to release ty insert at all since there are about a zillion corrupted tytool extracts out there & I REALLY didn't want to waste any more time on support questions because of someone else's error.

considering cwingert's view on parsing tmf's in mplayer,


Originally posted by cwingert
...Parsing this in the context of chunks (at least for my application) is somewhat of a nightmare.

that anybody who hasn't heard me mention some tytool extracts are corrupt by now is brain dead, and that I really have no desire to crawl in & add tmf support to mythtivo, I figured it was time time to release ty insert.

once tivoweb adds insert capability it would be trivial to add ty support, but I can certanly understand not wanting to open that can of worms

osetivo
06-04-2003, 02:00 PM
NM:

EDIT: Being a happy camper I haven't been following MFS_FTP closely. Figured I'd better catch up and see what's going on.


Originally posted by rc3105
recordings can now be extracted, archived, restored & watched as ty+ w/o any drawbacks so go ahead & convert your tmf library to ty+ (I am)[/B]

OK, Riley, I know you're not converting 1 file at a time. Do your progs do Batch processing? Your TyDeTar that I use doesn't. (I could just be stupid)

I tried tmf2ty once but IIRC I had 2 episodes with the same name, Seinfeld: The Trip, without any Part 1 or Part 2) and it overwrote the first so I stuck with tydetar which throws in the FSID. Has tmf2ty take over from a developement standpoint?

I've got 200GB+ of TMFs. "IF" you get the time, how about a bone?

TIA

johnmacd
06-04-2003, 02:16 PM
To get a TMF into a TY you would decompress it with a program that support RAR, join the pieces back together

So lets see if I understand this correctly. Since you can get a TY out of a TMF, one could simply store the TMF on your PC and then later get a TY out of it for use in other programs (like TyTools). There's no need to duplicate TY's and TMF's that are stored on a PC.


its more than likely that when tivoweb adds extraction/reinsertion features, TMF is the only file format it will support.

When this module comes out its gonna kick butt.


John

rc3105
06-04-2003, 09:36 PM
osetivo:

tydetar was just for extracting the ty prior to conversion to mpeg. tydetar doesn't append the xml info to the end so those ty's aren't really going to work for insertion. :(

the dos command

FOR %a IN (*.tmf) DO tmf2ty_cfg.exe %a

will sequentially launch tmf2ty_cfg.exe for every tmf in the current directory.

if you want to call it from a batch file use

FOR %%a IN (*.tmf) DO tmf2ty_cfg.exe %%a



Originally posted by johnmacd
...There's no need to duplicate TY's and TMF's that are stored on a PC...

right, BUT, tmf2ty tyfiles are equivilant to tmf's in that they contain all the same info. they can be converted back to tmf. once you convert the tmf to ty+ you can just delete the tmf.

the point is that mythtivo / mplayer / tystudio / tytool can't read tmf's, so if you want to be able to watch / convert a file allready on the pc w/o the nuisance of an extra conversion step just store it as a ty+

load the notebook hd up with ty+ to watch on the plane. give the kids a dvd of ty+ to watch on the xbox... whatever :)

deebo
06-06-2003, 07:53 PM
I am able to extract/import .tmf files succesfully, However I have read that .tmf's aren't the best way to archive shows anymore. Should you instead archive the shows in the ty folder, or the tyx folder? Are ty files the same as ty+ files?
-David

rc3105
06-06-2003, 11:21 PM
for PURE archiving, p2p transfer or streamview tmf is the best format. (easier to repair a corrupted file or access the xml) if you never ever use mplayer or convert an extracted recording to mpg stick with tmf.

mplayer, tystudio & tytool can't read tmf files. if you use those utilities then extract files as ty

ty from mfs_ftp is really ty+. the difference between regular mfsstream ty & ty+ is that in ty+ the xml data is appended to the end. (the xml stores the actor list, original air date, all the extended info)

ty+ contains all the same info as a tmf but you don't have to convert the file before you can use mplayer / tystudio / tytool / ty2vob or whatever

use whichever format best suits what you'll be doing

rc3105
06-07-2003, 07:38 AM
the fifo in /var/mfs_ftp is only used for resume on extracts

the cache directory contains 2 types of files, xml & pts. a .pts file is just a text file listing each fsid in the recording. the xml is the xml text that's included in a ty or tmf.

there's a few reasons for the cache dir:

1) by caching the xml data to an ext2 partition /dev/hda9 (var) the number of mfs db accesses is drastically reduced and things run smoother all around

2) when a recording is inserted not all the xml data is restored to the db (yet) the entire xml from an insert is saved to the cache dir, then on extract it's retrieved from there. this allows you to insert a tmf & extract an identical tmf (maintaining the actor list 'n so forth) if you put a 2.5 ty/tmf into a 3.1 box it plays, then when you extract it again you get your original 2.5 ty/tmf back

3) with the xml & pts info on an easily accessable partition you can automate the process of extracting recordings by connecting the hd to a pc. this is usefull for retrieving recordings from a drive that's gone GSOD or doing REALLY fast drive to drive extracts. see this thread (http://www.dealdatabase.com/forum/showthread.php?s=&threadid=23192) for a howto

larray
06-12-2003, 10:07 PM
Originally posted by wkozun
Am I doing something wrong? Is this how things work? Is there any way to change it so that I can see progress.

Nope, your doing it right.

The reason you don't get status is because the Tivo is actually sending it, not SmartFTP. Once the process is initiated, everything moves between the Tivos and not through SmartFTP. (I believe)

blahman
06-15-2003, 10:24 PM
Originally posted by Stephen
Ok I will admit I did not read all 51 pages of this thread but DID read the read me's and instructions.. I did get the program running properly a quick word to those of you that are running regular FTP CHANGE THE FTP PORT!!!! or else it will look like a regular ftp client

General Question I didnt see in the readme.. Which files Do I need to download? I see the files listed in ty ty+ xml txt tmf ect.... If I want to import them back into the tivo Im guessing ty and xml?

To re-import later, either the TMF or the TY+ will work. Both have the data that is stored in the XML embedded within them.

TY+ is compatible with the TY file editting tools as it appendeds the XML data to the end of the file so as not to interfere with the editors.

stumbleshoe
06-18-2003, 02:16 PM
Originally posted by stumbleshoe

OK, I'll try that tomorrow - just got home from work and I'm beat!
Thanks!

OK, now I'm really stumped. Turned on the extra debugging, re-tried the insert with the same tmf and it worked. No problems whatsoever.

So, does this sound like a timing issue of some sort ? I.e. dugging slows things down enough to keep it working ? I had previously tried throttling the SmartFTP transfer, but even at an extremely low setting the transfer would still fail.

Other than the need to rotate the log file, any harm in leaving this on ? :)

rc3105
06-19-2003, 12:38 AM
debugging doesn't change functions or timing, just logs what happened, when, & with what params. it's not quite upto black box specs yet but that's the general idea ;)


there are 3 types of errors

retrytransaction - the db was busy so it retries until it works

fsstream create failed - there wasn't any free space in mfs

tmk assert - usually ran out of memory or unknown. causes a reboot


1 is normal, don't sweat it. if you get type 2 or 3 & it's repeatable I'd like to see a copy of a level 3 log along with info about sys config. (zip & attach to a post here or e-mail is good)


cmd line ftp issues

the filename is

{Stargate SG-1}{2000-02-25}{Maternal Instinct}{07-14-2003 07.00 PM}{SCFI}.tmf

not

06-18-03 09:00PM 2235564032 {Stargate SG-1}{2000-02-25}{Maternal Instinct}{07-14-2003 07.00 PM}{SCFI}.tmf

so

get "{Stargate SG-1}{2000-02-25}{Maternal Instinct}{07-14-2003 07.00 PM}{SCFI}.tmf"

works here with cmd line ftp from win98 / RH 7.3 / OS-X

TroyCarpenter
06-19-2003, 07:01 AM
Oh my goodness. I just realized that after 15 years of using ftp and the internet tools, I'm an *****! :o

Thank you for letting me show my ignorance to the entire group. I don't know what I was thinking. Yes I do, I thought that the timestamp information was part of the filename. Duh.

Can I claim that it was late and just didn't have my senses around me (I've seen others use that excuse before :) )

Again, thanks for the pointer...I just wish I had a real challenge for you...then again, these types of problems are a lot less work.

Oh, and if noboby else has mentioned this yet...this is a great tool!

deebo
06-19-2003, 04:06 PM
I just installed 1.2.8a and it works great here with IE6 and smart FTP. I am able to drag and drop to and from my tivo using IE.

Being able to just drag and drop from IE will make things MUCH simpler !

Thanks again Riley this is a GREAT program that just keeps on getting better!!!!!!

osetivo
06-25-2003, 12:52 PM
Originally posted by rc3105
there were some varients of versions < 1.6 [SIC 1.2.6] posted to the thread a while back. as long as you extract/insert in tmf they should work fine. Sorry, I know, that the whole TMF/TY/TY+/TYx file thing has been explained to death, and I REALLY try not to ask repetitive and/or stupid question, but....... (did you see that coming?) I want to make sure I'm OK. (from an extraction/insertion stand point) :-)

I have been extracting TMFs with the various versions of mfs_ftp.tcl, until just recently. I now extract in TY which is really TY+, i.e. it has the xml stuff attached to the end, right?

I have now converted all these old TMFs to TY with a recent version (?) of tmf2ty, which turns my TMFs into TYs (really TY+). So that my entire PC archive is now TYs (TY+) from various sources and versions. (though, there is no way for me to tell what's what anymore.) I have now deleted all of my old TMFs

I always use the latest mfs_ftp.tcl. Is it ok to reinsert ANY/ALL of my archived TYs, regardless of their origin, directly as a TY file (really a TY+) via mfs_ftp.tcl, or do I need to insert pre 1.2.6 tmfs as tmfs? or stated differently, will a TY (TY+) recently converted from a pre-1.2.6 tmf be OK.

The reason for asking is that I had a bad insertion the other day.
I know that historically my TMF extraction streams were OK, because I would regularly convert TMF to TY, via TYDETAR, to make DVDs (I kept the TMF so I didn't loose the xml for insertion), but I don't do many insertions, so when it went wrong, I started wondering if I properly understood the insertion stuff and tmf2ty.

Details of bad TY insertion:
I inserted a kid's show from a TY. I DO KNOW that the TY was created by tmf2ty, what version of TMF (1.2.5,6,7,8), I don't know. On playback "This Old House" jumped in for a few minutes in the middle of the show and then back to the kids show. (Not a pretty sight from a 3 year olds perspective) I don't think that TiVo has ever recorded "This Old House".

I checked, and the TY on the PC DOES have the interuption, so it's most likely that either the ORIGINAL TiVo recording was bad or it was the original extraction. I have never seen anything like this ON the TiVo. Have any of the old mfs_ftp extractions ever exihibited this problem.

Do you see any problems, or is this just an isolated recording problem?

Thanks

rc3105
06-25-2003, 06:42 PM
a tmf is a tmf is a tmf. doesn't matter what version it was extracted with it'll insert just fine.

the minimum fsid "block" allocation size in a recording on the tivo is often a bit larger than the space used in the fsid. it's normal to get a few chunks of junk in a mfsstream or mfs_tarstream (tmf) extraction. these extra chunks are faithfully extracted & restored when using tmf's. since tivoapp put them there & uses the header chunk to determine where the real end of the fsid is it isn't a problem on playback.

never caused problems here, and I did LOTS of testing prior to releasing ty+ insertion because...

ty+ also extracts these extra chunks, but on restore it ONLY restores the chunks that are actually part of the recording. the junk chunks are discarded & you end up with different junk in the "cracks" between fsids (if the cracks exist at all, usually it's less than a meg at the tail end of the last fsid)

tmf -> ty (ty+) via tmf2ty are identical to current ty extracts & include these junk chunks if present. that junk is discarded on insert so you should never see it.


what was the original question again... :D


I would guess that that was just a random thing. I've never seen anything like that here. I can only speculate w/o looking at the file. if I had some ftp space I'd say upload it, if you've got space to host it please pm me a link.

the tmf format isn't changing, feel free to use it for long-term archiving to +/- r media w/o any worries.

the ty (ty+) format may get a few minor tweaks yet, but tmf's can be built from tmf2ty converted files or mfs_ftp ty extracts so it's not a problem either. if it does get tweaked that will be accomodated in the insert routines (allready is actually) so ty+ is ALSO good for archive to +/- r media w/o any worries.


--
Riley

mavrcksd
06-25-2003, 11:56 PM
I have a question:

I did a manual record, and well just used the first channel in my lineup (ch 6). I have Direct TV feeding the tivo on video/a/v inputs.

On the DTV receiver, I just choose another source (input 1) via the A/V matrix built in. The input 1 is my VHS.

I dubbed a movie into the Tivo using A manual record, and it took the information about that lineup channel and put it in the database - all well and good.

However, How can I modify this information to make it really what I want it to be while it lives in the Tivo?

?????

Do I extract it, change it, and then re-import it? But the XML data is part of the file.. How can It be changed?

rc3105
06-26-2003, 06:14 AM
Originally posted by mavrcksd
I have a question:

However, How can I modify this information to make it really what I want it to be while it lives in the Tivo?

?????

Do I extract it, change it, and then re-import it? But the XML data is part of the file.. How can It be changed?
easiest method is this:

shutdown mfs_ftp

delete the xml files in /var/mfs_ftp/cache via bash or tivoftpd

edit the recording information to whatever you want with tivoweb

re-start mfs_ftp


the cached xml file will be rebuilt from the new info as set by tivoweb. on extract that'll be what's included in the tmf or ty

SR712
06-30-2003, 07:29 PM
Originally posted by rbiro
I would assume that opening up 2 ftp sessions on a PC and "dragging" a file from 1 session to another is still taking the long route through the PC.


Not really. If the FTP Client supports FXP, than it is not in the loop after the transfer initiates. SmartFTP does support multiple ftp sessions, and FXP.

tivowatcher
07-05-2003, 11:29 PM
OK, I am completely stumped.

You'd think after hacking a bunch of TIVOs and successfully installing various tools (such as MFS_FTP) I could get this to work better. :)

What steps are you taking to *avoid* a passive transfer? I have 3 different clients here and they all max out around 288Kbs on tivo-to-tivo transfers.

Just tell the client you are using (such as SmartFTP 1.0) and the settings you use. I just can't figure this one out.

FYI:

I am using SVR-2000s with the latest 3.x software.
I have Turbonet cards from 9thtee.
Cabling and switch are both good.
Downloading to my PC rocks - over 1,000Kbs.
MFS_FTP is excellent and worked like a charm right "out of the box"

Thoughts?

glider
07-09-2003, 10:23 AM
Riley, has anything besides adding the extra stuff at the end of the .ty+ file changed between the two versions ?

I only watch the .ty files using (the modified, latest) mplayer, and it seems like the files from the 1.2.8a version have more random segments in them than files extracted using the older version (which only had ty, right ?).

Seems to happen about once per half hour of recorded material - very annoying! Seems to be random material from other shows, but kind of hard to tell since its usually only for a couple of milliseconds..

I saw one other post that also seemed to have noticed something similar, has anyone else noticed the same behavior ?

rc3105
07-09-2003, 10:54 AM
ty from mfs_ftp has allways had the xml appended to the end as do tmf2ty and tmf2mpg conversions. (tydetar tmf->ty conversions don't include the xml)

1.2.5.8 & 1.2.8a use the same mfs_stream procedure as tivoweb-mfsstream to extract ty. extracts from the same tivo with both mfs_ftp versions should be byte identical

the junk in the cracks on fsid boundries is normal but generally not a problem. it'd be interesting to know your drive config & the mfstool options used to partition the drive(s)

mfs_ftp & tmf2ty don't filter out the junk so that a ty is a 1:1 copy of the data from the tivo

tmf restores are currently 1:1 while ty inserts are automatically filtered. one of these days I'll post a filter utility and add that as an extract option


--
Riley

glider
07-09-2003, 02:36 PM
the junk in the cracks on fsid boundries is normal but generally not a problem. it'd be interesting to know your drive config & the mfstool options used to partition the drive(s)

Sorry, its been so long I simply don't remember the options I used when creating my 2nd drive... mfstool didn't know how to backup data though (if that helps), which was added in later versions of mfstool.
The 1st drive came with the my tivo, and after blessing the 2nd drive they got joined by tivo - so I guess it might depend on which drive is used when saving that determines how much 'junk' is extracted ?

But yes - a filter utility (either built into mfs_ftp, or standalone) would be very handy!
Or I guess if mplayer the ty-enabled mplayer is further enhanced to avoid the junk ?

osetivo
07-09-2003, 08:18 PM
Riley:

Decided it was time to redo the TiVo drive. In the Hinsdale how to the resore command is mfsrestore -s 127 -xzpi /yada yada

Is the "P" in the switch "-xzpi" the -p switch you referred to earlier to set the partitions to their factory settings?

Thanks

Will
07-10-2003, 01:25 AM
Originally posted by rc3105
the junk in the cracks on fsid boundries is normal but generally not a problem.

mfs_ftp & tmf2ty don't filter out the junk so that a ty is a 1:1 copy of the data from the tivo

tmf restores are currently 1:1 while ty inserts are automatically filtered. one of these days I'll post a filter utility and add that as an extract option

Am I understanding the above correctly: tmf files that have been downloaded from a DTivo running mfs_ftp as a host have the junk at the boundaries? But you will at some point post a filter utility to clean existing files, and a future version of mfs_ftp will have the option to filter at the original download?

If my understanding is correct then the filtering of files previously downloaded to a Macintosh would have to be copied to a Wintel box to run the filter utility? If you could at some point discuss the logic of how such a filter would work, I could probably put a Mac filter utility together and I'd be glad to post it here.

rc3105
07-10-2003, 01:54 AM
the header chunk of each fsid tells you how many chunks of video it contains in bytes 12-15

from the mfstools 2 readme

-r scale
This option allows you to reduce the amount of RAM TiVo uses by
increasing the block size for the media storage created with the -x
option. The acceptable values are 0 to 4, corresponding to 1, 2, 4,
8 and 16 megabytes. The larger the value, the less RAM will be used
(and the faster menus will respond) but a (small) amount of storage
will be wasted by some recordings. At the extreme small end (-r 0)
some PVRs with a large amount of storage may not be able to perform
some tasks, such as self repair. (A.K.A. Green Screen) The default
is 2 (or 4 megabytes). Any TiVo created partitions are created with
a scale of 0 (or 1 megabyte). It is best to leave this option alone.


if you have a mfstools restore on a large drive or adjusted the mimimum mfs allocation size with the -r option then sometimes the fsid is actually larger than the size required for the chunks it contains. the space between the last chunk and the end of the fsid is the junk that falls through the cracks. it's just whatever happened to be there when the fsid was allocated. 98% of the time it only happens at the end of the last fsid.

since tcl is required for tivo db calls and runs on every flavor of os/hardware I tinker with I write tivo utils as tcl scripts. (it's also easy to link in platform specific c modules as functions for speed) if needed they can be "compiled" into an executable with a runtime lib like freewrap as I did with tmf2ty & tmf2mpg. porting is unnecesary for the mac since X includes tcl. the import filter algo is in the ty2fsid proc in mfs_ftp that starts around line 500 and in tmf2ty but disabled by default


--
Riley

osetivo
07-10-2003, 10:49 AM
Originally posted by rc3105
RE: AW's teaser. if you extract encrypted recordings as tmf they CAN be restored to the same unit. (slightly easier than archiving mfstools backups that include recordings)By "same unit", do you mean the same harddrive as well, or could you d/l and reload scrambled tmf on a fresh HD install? Can you do it with TY+?

rc3105
07-10-2003, 11:22 AM
AW has a bee in his bonnet and noodged me 'n mbm to update our insert routines... (if only I were that young and energetic again!) ;)

so far scrambled recs seem to be tied to the crypto chip. they're working on 3.1 and I'm focusing on 2.x. I'll be moving a cyrpto between boards this evening to pin down whether it's tied to the chip or the drive image. my $$$'s on the chip

scrambled ty+ should insert ok, but tmf is better for archiving scrambles as the "junk between the cracks" is preserved and that may be a factor in restoring scrambles


--
Riley

AlphaWolf
07-10-2003, 12:09 PM
Originally posted by rc3105

so far scrambled recs seem to be tied to the crypto chip. they're working on 3.1 and I'm focusing on 2.x. I'll be moving a cyrpto between boards this evening to pin down whether it's tied to the chip or the drive image. my $$$'s on the chip


Are you still trying to get oprah to play? :D

BTW, when you say you are moving the chip, are you actualy talking about physicaly moving the chip? or cloning the serial number?

rc3105
07-13-2003, 10:53 PM
simply cloning the sernum doesn't work so I'll be physically desoldering the !@#$@#$ surface mount crypto chip. gotta replace the tuners / volt reg & who knows what else on those boards anyway. should be fun in the root canal sense of the word

Originally posted by needo
How exactly does this work? I have a scrambled stream on my TiVo 4.0. and tried to download the .ty (Same error as before.) I then downloaded the .tmf without any problems, but it still choked on the tmf2ty conversion. And I ended up with an invalid .ty.

What am I missing?
scrambled recs can't be converted to mpeg on the pc (yet). they can be backed up, restored and played back on the EXACT same tivo, or they can be decrypted into regular ty in the tivo, then extracted & converted to mpg. mfs_stream can't read scrambles so to archive w/o decrypting you have to use mfs_tarstream to extract as tmf
speed issues - the biggie
here I thought it was something complex & it was just the !@#$% ideturbo module, figures. thanks mbm!

to check your tivo boot parms issue this command from bash

bootpage -q /dev/hda

you should get roughly this

IP address: 192.168.0.2
Primary boot partition: 3
Alternate boot partition: 6
Hostname: unnamed
Net boot kernel name: linux.px
Boot parameters: root=/dev/hda4 runideturbo=false
MAC address: bd:ca:43:47:75:90
RF channel: 0
Standby: 109
1

if the "Boot paramters:" line DOES NOT include "runideturbo=false" then you need to add it

note whatever your root parms are. "root=/dev/hda4" or "root=/dev/hda7" and update them like this

bootpage -P "your_current_root_parms runideturbo=false" /dev/hda

reboot, try an insert. should be MUCH faster :cool:

tivowatcher
07-14-2003, 01:37 AM
Oh yeah - this is nice!

Here are some transfer examples once the runideturbo=false parameter is added:

TIVO -> TIVO: 416 KBs
PC -> TIVO: 409 KBs

Seems like if I let it queue up about 5-10 mins, I can watch the show "real time" as it streams from the source.

Congrats rc3105 you've got a winner!

P.S. This is using 2 SVR-2000's - which are Series 1 standalones.

Will
07-14-2003, 02:56 AM
Originally posted by tivowatcher
PC -> TIVO: 409 KBs

Seems like if I let it queue up about 5-10 mins, I can watch the show "real time" as it streams from the source.

Even at that speed you really should be able to start watching after a minute or so (though my experience is all on DTivos).

mini__me
07-14-2003, 10:11 AM
WOW, that is sooo much faster I am now having the problem of inserts running so fast they are affecting normal use :D

I have extract running around 1,500k while recording at best quality and inserts around 1,000k again while recording at best however this severely impedes on normal use I'll have to get experimenting on throttles.

LOVING IT :D :D :D

osetivo
07-14-2003, 10:24 AM
Originally posted by mini__me
WOW, that is sooo much faster I am now having the problem of inserts running so fast they are affecting normal use :D

I have extract running around 1,500k while recording at best quality and inserts around 1,000k again while recording at best however this severely impedes on normal use I'll have to get experimenting on throttles.

LOVING IT :D :D :D

1000KB Wheew! glad to hear it. I saw that 400KB insert speed mentioned earlier, so I didn't want to get my hopes up.

Already did the bootpage cmd (from work), just can't wait to get home to test upload speeds. :D

As always thanks Riley (and of course mbm)

cojonesdetoro
07-14-2003, 10:37 AM
I'm getting about 550-650KBS uploads and am more than satisfied with the Tivo's performace. Recordings and menus all look fine so far. That's a good enough speed for me. It seems like high quality recordings go in at 'real time' now. That is, 1 hour of video uploads in about an hour.

osetivo
07-14-2003, 08:51 PM
Originally posted by osetivo
1000KB Wheew! glad to hear it. I saw that 400KB insert speed mentioned earlier, so I didn't want to get my hopes up.

Already did the bootpage cmd (from work), just can't wait to get home to test upload speeds. :D

As always thanks Riley (and of course mbm)


Much better, With both tuners on empty channels I'm now getting ~650KB/s up from 250. Geesh 1,000 KB/s, I'm still jealous.

embeem
07-16-2003, 07:30 PM
The ideturbo module is intended to put the drives in "streaming media" mode; a few tests suggest that drive access actually slows down while ideturbo is loaded, adding runideturo=false to the boot params will prevent the ideturbo module from loading:


bash-2.02# bootpage -p /dev/hda
root=/dev/hda4
bash-2.02# bootpage -P "root=/dev/hda4 runideturbo=false" /dev/hda
Updated boot page on /dev/hda

Playing around with hdparm can lead to further performance gains.

Also, the version of mfs_import distributed with mfs_ftp is rather inefficient; a newer version is avaiable here: http://alt.org/forum/index.php?t=msg&th=55&start=20

rc3105
07-16-2003, 11:38 PM
mfs_ftp is a basic ftp implmentation. some os's and proxy's can mount it as a remote filesystem, other's can't. try & few & see what works for you

setpri is there to prevent inserts from eating so many cpu cycles that menu's get sluggish. get it from alt.org, put it in /var/mfs_ftp/ & make it executable or don't worry about it. it's there to slow things down. the log just lets you know whether it was found/used

mbm's optomized mfs_import is a bit faster :) BUT it has to be renamed to mfs_stdinsert (or mfs_ftp edited appropriatly) and it only works with mfs_ftp versions > 1.2.8f it doesn't affect transfer speeds so much as lowering the cpu load for a given transfer rate (which lets you insert faster w/o impacting system performance)

if you add runideturbo=false to your boot parms you have to reboot the tivo before that takes effect. check to see if ideturbo is loaded with "cat /proc/modules" if it's not listed you're good

hdparms can make a big difference, this is safe for stock quantums "hdparm -c1 -d1 -m8 -S0 /dev/hda /dev/hdb" try it from bash before adding to rc.sysinit or rc.remote-login

increasing info(ithrottle) on line 21 slows inserts down. values can be 0 to 10

increasing info(insert_priority) accelerates inserts if setpri is available. values are 1 (lowest) to 99 (highest) 5 works nicely here

ithrottle & insert_priority will interact differently depending on your hd / network / running-hacks / sw-version / viewing-behavior / karma. the worst case scenario is a dtivo/tivonet/Q30 recording 2 shows & rewinding a third while inserting. even under those conditions I'm getting 500k/sec+ here. a standby dtivo/turbonet should run 1,200k/sec+ easily

don't try to watch scrambled recordings untill import is complete. depending on your noscramble module/settings it could cause a reboot

test your network / hdparms / insert speed & tweak your settings with the tivo in standby. once you find the peak rate then try a worst case scenario & re-adjust. dtivo style partition layout can make a big difference on slow drives like the stock quantums (see previous posts re: mfstools restore with the -p option)

AVD
07-17-2003, 10:02 PM
Originally posted by rc3105

check your hdparms with hdparm -i /dev/hda and pay attention to MaxMultiSect. if you have dual drives you also need to do this for /dev/hdb--
Riley

I have a quantum and my maxmultisect=16, should I go with 8 or 16?

Do I have to reboot?

Thanks

Keep up the good work, thanks for working on this!


/dev/hda:
Model=QUANTUM FIREBALLlct15 30, FwRev=A01.0F00, SerialNo=314015418411
Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs }
RawCHS=16383/16/63, TrkSize=32256, SectSize=21298, ECCbytes=4
BuffType=3(DualPortCache), BuffSize=418kB, MaxMultSect=16, MultSect=off
DblWordIO=no, maxPIO=2(fast), DMA=yes, maxDMA=2(fast)
CurCHS=16383/16/63, CurSects=-66060037, LBA=yes, LBAsects=58633344
tDMA={min:120,rec:120}, DMA modes: mword0 mword1 *mword2
IORDY=on/off, tPIO={min:120,w/IORDY:120}, PIO modes: mode3 mode4


bash-2.02# hdparm -m8 /dev/hda

/dev/hda:
setting multcount to 8
multcount = 8 (on)
bash-2.02# hdparm -i /dev/hda

/dev/hda:
Model=QUANTUM FIREBALLlct15 30, FwRev=A01.0F00, SerialNo=314015418411
Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs }
RawCHS=16383/16/63, TrkSize=32256, SectSize=21298, ECCbytes=4
BuffType=3(DualPortCache), BuffSize=418kB, MaxMultSect=16, MultSect=8
DblWordIO=no, maxPIO=2(fast), DMA=yes, maxDMA=2(fast)
CurCHS=16383/16/63, CurSects=-66060037, LBA=yes, LBAsects=58633344
tDMA={min:120,rec:120}, DMA modes: mword0 mword1 *mword2
IORDY=on/off, tPIO={min:120,w/IORDY:120}, PIO modes: mode3 mode4

rc3105
07-17-2003, 10:37 PM
AVD:

if the drive MaxMultiCount is 16 then use that. the point was not to set multicount to more than the reported MaxMultiCount for the drive. the hdparm settings are lost when you power-down or reboot the tivo. once you figure out what options you're going to use add them to a startup script somewhere

needo
07-23-2003, 02:59 PM
Is there a quality difference using mfs_ftp over tyserver? Or is a ty is a ty is a ty is a ty?

rc3105
07-23-2003, 03:08 PM
quality wise a ty is a ty is a ty, however, tyserver / tystudio's sole purpose is conversion to mpg. it doesn't extract the header chunk for each fsid part (indexing info) or the xml. tyserver tyfiles can't be reinserted w/o a LOT of hassle


--
Riley

cojonesdetoro
08-05-2003, 03:59 PM
I found a CLI ftp/fxp client for linux here:

https://sourceforge.net/projects/quftp/

has anyone been able to get this one to work? I'm still trying to use the man page but there are no examples using fxp.

cojonesdetoro
08-05-2003, 10:17 PM
Originally posted by cojonesdetoro
Does anyone know of any different settings to get smartftp/fxp to work? I'm trying to transfer between TIVOS and he last thing that happens is the message "ABOUT TO OPEN DATA CONNECTION" shows on the sending Tivo's window. After that, smartftp just hangs waiting and nothing is ever transfered. nI also have to restart the tivo's to get mfs_ftp to work aain.

I am bewildered. I left it running in hopes that it was actually working even though it gave no transfer statistics. It turns out that it was working because the recordin was transfererd. However, now it works completely as expected. It shows th stats as it copies.

rc3105
08-05-2003, 11:20 PM
smart-ftp's funny that way, sometimes it can evesdrop on fxp transfers & sometimes it can't. once the transfer is initiated the tivos will complete it automatically. you can close the fxp client, shutdown the pc, makes no difference


--
Riley

Stephen
08-06-2003, 10:16 AM
Originally posted by Stephen
Suggestion dont just copy the files over my unit is now stuck in a green screen loop.... Oh well I guess it was time to upgradde to 3.1 anyways

K here is the deal I had upgraded to 3.1 with no real luck so I decided to start from scratch again... I started back with surgons 2.5 image from a bootable CD rom.. did the Serail port upgrade to xupgrade back to 2.5 and reinstalled what hacks I wanted. with the SAME equpiment and the SAME route that I have taken before I am now getting about 350k upload speeds... for those of you that are getting crappy upload speeds. think of it this way.. Just like a windows box needs to be rebuilt after (what 2 weeks of playing with it) your Tivo sometimes might need to be rebuilt... Take a hour or 2 and do it. I had done so many things to my box Im sure I had screwed it up somewheres. By redoing the box I proved my point that I had and didnt know it!!

SirTrini
08-06-2003, 04:40 PM
It's been my experience that deleting either a .tmf, .ty, or .ty+ will all result in the show getting deleted.

MFS_FTP is really just giving you a "3 or 4 virtual" representations of the same ONE show.

When you choose a specifc format to extract, MFS_FTP takes the native show and "wraps" it up the way you've requested.

cojonesdetoro
08-18-2003, 10:27 PM
Has anyone else encountered a 4GB limit on dowloadable tmf recordings ? I got a recording that is a 3 hour movie at high quality. it shows up as 116MB on the file listing from mfs_ftp. The transfer dies with a timeout at 4095MB. My guess is that the actual recording is 4096+116MB in size. I tried the 'a' and the newest 'f' version.


BTW, I know watcha thinkin but the destination file is going to an NTFS filesystem.

cojonesdetoro
08-18-2003, 11:06 PM
BTW, I know watcha thinkin but the destination file is going to an NTFS filesystem. [/B]

Duh, ME AM ST00PID.. Drive C was NTFS but.. ah... not drive D:... the uh... destination drive, that is... heh.. nevermind.

<excuses>
But it does show up as 116MB on the file listing... I guess that kinda threw me off.
</excuses>

cojonesdetoro
08-19-2003, 11:44 PM
It seems that recordings that I insert with mfs_ftp seem to make tivoweb 1.9.4 barf. I can view recordings that were recorded on the tivo in the 'now playing' list of tivoweb but selecting mfs_ftp inserted entries results in this error:

INTERNAL SERVER ERROR
--cut here--
action_series '/' ''
expected integer but got ""
while executing
"db $db openid $objectid"
("uplevel" body line 2)
invoked from within
"uplevel $body"
invoked from within
"transaction {uplevel $body}"
(procedure "RetryTransaction" line 5)
invoked from within
"RetryTransaction {
set series [db $db openid $objectid]
set title [strim [dbobj $series get Title]]
set score [dbobj $series get ThumbData]
se..."
(procedure "::action_series" line 15)
invoked from within
"::action_$action $chan $part $env"
("eval" body line 1)
invoked from within
"eval {::action_$action $chan $part $env}"
--cut here--

rc3105
08-20-2003, 01:01 AM
the xml from an inserted tmf is cached in /var/mfs_ftp/cache/ not all attributes are restored to mfs. some tivoweb modules expect those "missing" attributes & barf when they're not found

jjk999
08-20-2003, 02:22 AM
Hello,
Can anyone out there help me with a quick question. About 30 mins ago I was switching off the ideturbo proc on each of my S/A tivo's. During the 2nd tivo edit I accidentally typed in:

bootpage -P "roo=/dev/hda4 runideturbo=false" /dev/hda

instead of:

bootpage -P "root=/dev/hda4 runideturbo=false" /dev/hda

notice the type-o in that I left out the "t" in root

my question is, how harmful might this be if I reboot the tivo (my guess is not harmful) and is there a way to remove that line from the bootpage, I have seen no way in any of the threads for removing from the bootpage, just adding to it.

thanks for your help!
John

p.s. these threads are awesome, over the past 5 days, Ive upgraded a 14 hr philips tivo to a 160 gig capacity (no subscription) installed mecessary binaries and scripts onto the 160 gig and my fiancee's sub'd Philips S/A 20 gigA+120 gigB Tivo installed mfs_ftp onto both tivos and already transferred 10 gigs of programming from her tivo to mine! great threads, keep up the good discussions

rc3105
08-20-2003, 03:03 AM
when you issue that command it overwrites the old bootpage parms with the new ones. just re-issue the command with the correct syntax. w/o a correct root statement it probably won't reboot at all

gobsmack
08-21-2003, 09:24 PM
Originally posted by jjk999
my question is, how harmful might this be if I reboot the tivo

Ummm, well, you remember the blackout that hit the east coast last week?

Some guy did exactly what you did -- mistyped a parameter in their TiVo bash-- and poof, down went the power grid.

Where do you live -- I may just buy a generator...

rbird
08-22-2003, 08:10 AM
Originally posted by rc3105
the xml from an inserted tmf is cached in /var/mfs_ftp/cache/ not all attributes are restored to mfs. some tivoweb modules expect those "missing" attributes & barf when they're not found I've been meaning to ask about this...WHY doesn't mfs_ftp retain/restore all the data? Even the original air date is missing upon insertion (one of the most important pieces of data - it's even in the PC filename!). This doesn't seem like it would be difficult...maybe in some future version a config file could allow us to choose what attributes are important?

And on a related note, why are all inserted programs showing marked as best quality, even if they aren't? It seems to be screwing up the space allocation routines ("Save Until..." doesn't work right with inserted shows present - you can't save shows longer even if there is plenty of space available and little or nothing upcoming). It's like the Tivo thinks inserted shows are bigger than they really are.

Bob

rc3105
08-22-2003, 11:54 AM
Originally posted by rc3105
....original idea was more of a single coherent binary (as is my eventual goal) the current version is really just a proof of concept test that got loose! :D


<mfs_ftp sourcecode>

switch -glob $attr {
ActualShowing -
BitRate -

</mfs_ftp sourcecode>

the tmf/xml data doesn't contain the bitrate info so it can't be restored as anything other than a best guess. on dtivos (what I mostly have/develop on) the quality is allways best since there's no other option so that's the default

it's easy enough to include/restore the bitrate info but that would produce nonstandard tmf files. tmf might get a revision when the tivoweb or tivoweb-ng import module is released but don't hold your breath


<mfs_ftp sourcecode>

set progitems "Title EpisodeTitle OriginalAirDate"
...
foreach item $progitems { set info($fsid,$item) [strim [dbobj $info($fsid,Program) get $item]] }

</mfs_ftp sourcecode>

the original airdate should be restored


mbm's import.tcl restores all the xml info, not a big priority here though (silly me, watching the video instead of reading the credits! :p )


--
Riley

sanderton
08-31-2003, 12:00 PM
Can anyone shed any light on this:

I have two S1 TiVos.

Both have the same configuration:

UK 2.5.5 software
mfs_ftp, TyStudio, TiVoWeb loaded at boot (in that order)
Running version f
embeem's mfs_import
ideturbo = false
hdparms set as suggested

The only difference between the two is that one is a factory 40G drive, the other a 80 + 30 upgrade. They are both connected to the same switch, and both give perfectly fine download speeds.

On the factory 40, my problem is that inserts run too fast - with the default settings they run at 1,150K and that locks up the machine. (The 600k mentioned above was with version a and the normal mfs_stdinsert)

On the other one they run too slow - 350K or so, too slow to watch the stream as it copies over.

I'd settle for an average!

Any ideas anyone?

rc3105
08-31-2003, 04:53 PM
re-read the posts that relate to adjusting info(ithrottle) and the priority that mfs_stdinsert or mfs_import runs at.

make sure you have a working copy of mbm's setpri in /bin

eliminate other variables while testing configuration changes. nosignal the tuner, disconnect the pc from the net, etc

(725k/sec is nearly 3 gig / hour??? is that right??? < 1.5 gig/hr on my sa here at 720*480...)

burriko
09-01-2003, 06:27 PM
riley,

i was just messing with the latest mplayer and noticed that it now has ftp support. since the main branch of mplayer doesn't contain the code to stream from vserver, but does contain the code to play ty streams, i though it would be interesting to try and get it to stream from mfs-ftp.
i've got it playing ty streams from an ftp server on my pc, and it works brilliantly. however mfs-ftp doesn't support the SIZE command that mplayer seems to require before it can play a file over ftp. Any chance of adding it some time? :)
obviously to scan to various points in the stream it would require REST to resume, but it doesn't seem to need that to play the stream from start to finish. i'm not even sure it would be possible to implement REST in mfs-ftp?

if mplayer could stream from mfs-ftp then cygwin wouldn't be necessary anymore as the latest mplayer is available as a mingw32 binary. which may also get around the 4GB limit that mplayer compiled under cygwin has.

anyway, just throwing the idea out there. feel free to ignore it as it's hardly a high priority. :D

tlphipps
09-03-2003, 10:50 PM
I've been watching this thread and using mfs_ftp from the beginning with my single SA HDR312 S1 unit. I'm now contemplating purchasing an HDVR2 unit (only $99!!).

Can I transfer shows between these two?? I've read various (contradicting) posts on this issue.

rc3105
09-03-2003, 11:33 PM
it's only moving recordings from S1 dtivo to S2 dtivo that causes playback without audio

TRILIGHT
09-04-2003, 11:42 PM
Hey Riley! I finally got a chance to try out mfs_ftp today. I just wanted to say "kudos" on such an inventive tool! I connected via setting up a "Network Place" to the TiVo's IP in Windows XP and "drag and drop" works great. I only get around 1MB/s compared to TyTool's 1.5MB/s but it's still very respectable! I'm impressed! :)

FreydNot
09-07-2003, 12:59 AM
Originally posted by mavrcksd
Riley:

When I do a cat of the port.3105.log ... it lists through the % and the 'k' done... however then my bash prompt comes right along and drops on top of it, so I cannot see what is on the left side of the screen.

Mav

cat will list the contents of the log file and exit. What you want is

tail -f port.3105.log

The tail command 'cats' just the end of the file requested. The -f option tells tail when it hits the end of the file, don't exit, just wait for more data to be written and print that out too.

IpEvery211where
09-07-2003, 03:30 PM
Good work!!

How do I clean up the "now showing" index and maybe? empty allocations? When attempting to delete shows with the remote or TiVoWeb, it just BONGS or gives an error. This is caused by TiVo-TiVo FXP transfers failing early in the transfer.

I thought MBM had something, I couldn't find it.

Thanks in advance!

(is it as easy as choosing "delete" with the FTP client?


edit yep, just delete the bogus recording via the ftp client

FredThompson
09-10-2003, 05:58 AM
How stable is extraction using mfs_ftp while a DTiVo is recording from one tuner? From both?

rc3105
09-10-2003, 08:27 AM
if you have setpri in /bin it'll adjust the extraction priority to the lowest level possible. even without setpri it won't interfere with dual recordings. (menu response might get sluggish but that's all)

quantum_3
09-10-2003, 10:32 PM
ok...i have been able to set the mfs_ftp on both my dtivo's with no problem. i have read the "read me" and it says...

How to transfer a show directly from one tivo to another:

this is how it works with w98/smartftp 1.0. I'm sure other platforms / clients
differ but figuring them out it up to you. this is just what I generally use.

open a connection to each tivo & drag a .tmf from one remote file list window
to the other.

you can hit play as soon as the recording appears in the recieving tivo
nowshowing list (about 10-15 seconds) and watch it as it streams across.

transfers between tivonet/dtivos here generally run about 900k/sec. ymmv.
the speed of the pc running smart-ftp makes NO difference since fxp transfers
are DIRECT and the pc just coordinates traffic


the stupid question:
Where are the .tmf File located on the Dtivo please??? (are they in var or /var/hack?)

genepool
09-11-2003, 01:07 AM
When mfs_ftp is running and you connect to the port 3105, you will see several directories including ty and tmf. The specific file types are in those respective directories.

quantum_3
09-11-2003, 07:34 PM
now all is installed and i'm now using port 3105 (thanks for the info) and i have installed smartFTP v1.0. i have pulled up 2 boxs (one for each Dtivo) and have both of them pointed to the TMF folder. I try to pull one TMF file from Dtivo A to Dtivo B and it looks like i can drop the file there but when i release the mouse button nothing happens. i have tried with passive mode on and off and from B to A and A to B. same thing. The engine is running i just need to know how to pull the gear out of "park". Thanks for your tolerence with me.

Q_3

larray
09-11-2003, 11:20 PM
quantum:

Have you checked on Tivo's A and B to see if you have an inserted program. If you just drag from one box to the other without using the right click queue feature, you won't get any statistics or even a signal that it's done. (I actually don't get any stats on the FXP even when it's queued, but I can see when it moves to the next item and clears the current one out.)

You should have a program inserting in the background even though it doesn't look like its doing anything in SmartFTP.

Poohbear
09-13-2003, 01:02 PM
I'm having problems getting mfsstream to download .TY files over 2 gb (using OS X Safari and Internet Explorer) within TivoWeb.

I'd like to install MFS_FTP to get around my browser's download limitation. I may have missed it in the mfs_ftp thread, but does MFS_FTP conflict or interfere with my existing FTP server (tivoftpd?) already on my DTivo?

How do I differentiate between the two servers for 1) downloading .TY files (mfs_ftp) and 2) uploading applications to the DTivo (tivoftpd)?

Thanks

sanderton
09-13-2003, 01:24 PM
No, they get along just fine. You connect on a different port.

Poohbear
09-13-2003, 01:30 PM
Excellent. How do I tell my FTP client which server to connect to?

If I'm using Fetch or Terminal, do I have to designate different port addresses for each server?

sanderton
09-13-2003, 03:50 PM
The port number will be on a screen with the connection information in whatever software you use. It defaults to 21, so you've probably never changed it. 21 will get you the TiVo FTP; 3105 will get mfs_ftp.

Poohbear
09-13-2003, 04:23 PM
That's exactly what I was looking for! Thanks so much.

Martlet
09-14-2003, 02:12 PM
In Fetch 4.0.3 the "New Connection" window has a "Non-Standard Port Number" box for you to enter 3105.



In Terminal:

ftp -Aa tivo 3105

will get you where you need to be. But the latest MFS_FTP doesn't seem to be as command line friendly as the old ones, now that it has the {}'s around everything.



And don't forget to create a .telnetrc in your home directory with the following line in it:

tivo mode line

for when you want to telnet into your tivo.



I'm assuming you added the name 'tivo' to your NetInfo Manager as:

/
machines
tivo

With the Properties : Values of...
ip_address : ###.###.###.###
name : tivo



Mac OS X doesn't check the hosts file like other flavors of un*x, so you have to do it in NetInfo.

gart
09-22-2003, 08:30 PM
All the usual disclaimer stuff. It works well for me, for the most part. I think overall, the mwstate error and the 2gb file limit on the appliance is going to be an issue this tv-season for me as I'm recording more shows in Best.

I'm working on another perl script that will automatically upload the files to the other tivo, but there's something funky going on with the timeout sequence that's stalling the transfer. I'll eventually get it to work, when it bugs me enough.

--Pat

--Start --
#!/usr/bin/perl

use Net::FTP;

####
## User Configuration
####
# Directory where ShowSubDir Lives
my $Progdir = '/home/gart/';
# Sub-Directory of Progdir that will hold the shows
my $ShowSubDir = 'shows/';
# Net::FTP Debug level (set at least 2 to see anything useful)
my $Verbose = "0";
# the Tivo's IP address/Hostname
my $TivoIP = 'tivo2';
# The port mfs_ftp listens on (3105 is default)
my $TivoPort = "3105";
# The show size in bytes(?) that you want to download (default 2gb)
# This is due to an old kernel on the appliance I'm using to transfer my shows
my $MaxSize = "2000000000";

####
## End user configuration
####
my $FullDir = $Progdir.$ShowSubDir;

my $ftpdebug = 0;
if( $Verbose > 1 ) { $ftpdebug = $Verbose - 1; }
# Build the connection, change dir, get a listing
my $ftp = Net::FTP->new($TivoIP, Debug=>$ftpdebug, Port=> $TivoPort);
$ftp->cwd("tmf");
$ftp->type("I");
$test = $ftp->pwd();
@temp = $ftp->dir();
# If there are no shows available, exit.
if ($#temp eq 0) { $ftp->quit; exit; };
$tc = $#temp + 1;
print "Found $tc shows to transfer\n";
# Start processing the listing
for $i (0..$#temp) {
@junk = split(/\s+/,$temp[$i]);
$filesize = $junk[4];
for $o (0..7) {
shift @junk;
}
$filename="";
for $o (0..$#junk) {
if ( $filename eq "" ) {
$filename = $junk[$o];
$ofilename = $junk[$o];
} else {
$filename = $filename."-".$junk[$o];
$ofilename = $ofilename." ".$junk[$o];
}
}
$Start = time();
# Test against Maxsize skip if too big
if ($filesize >= $MaxSize) { print " Skipped - too big\n"; next; };
# Remove all the funky characters
$newname = $filename;
$newname =~ s/ /-/g;
$newname =~ s/{//g;
$newname =~ s/}/-/g;
$newname =~ s/:/-/g;
$newname =~ s/,//g;
print "Starting Transfer for $newname\n";
$ffilename = $FullDir.$newname;
# Pull down the ftp file
$ftp->get($ofilename,$ffilename);
# This doesn't really work, but I want to make it work some day
sprintf("Transferred %d sec\n", time() - $Start);
}
$ftp->quit;

---END ---

scottf
09-23-2003, 04:33 PM
Thanks for help. Didn't realize winzip would do the trick. I'll read over and watch the threads. Do hope the progress continues as I see this to be more benefical then burning dvd's for archive.

Thanks again, learn something every day!:)

newlooper
09-25-2003, 07:42 AM
Just adding a little tidbit:)

With Winzip, make sure you go to Configuration > MISC > and disable the TAR file smart CR\LF option. This can really cause some headaches.:p

salva
09-27-2003, 10:36 AM
Hi, been uning mfs_ftp for some days, and I'm more than happy with it.

I've a couple of questions.

I'm running mfs_ftp from a telnet sesion, but if I kill the sesion, the server will kill.

Is there anyway to run the server from say rc.sysinit and get it working from there when tivo starts ?

I'm using a uk tivo with software 2.5.5 , should I pay attention to the Dtivo 2.5.2 mwstate error & fix ?

Tks for your help

Salva

newlooper
09-27-2003, 10:58 AM
Is there anyway to run the server from say rc.sysinit and get it working from there when tivo starts ?


Sure!:D There is a way to handle your situation.

if you start mfs_ftp like this from its directory:

./mfs_ftp > /dev/null &

It will background the process and you can kill telnet without worries.

You can do this from telnet or from the rc.sysinit like this:

/pathtomfs_ftp/mfs_ftp > /dev/null &


Ain't it fun :)

rc3105
09-30-2003, 07:19 PM
Originally posted by mavrcksd
00:00:00 - mfs_ftp version 1.2.5.1 - checking tivo system sw
00:00:00 - entering reset_mfs_ftp
00:00:00 - port 3150 control socket initialized - ready for connections

there was never an official version that defaulted to port 3150. the one you have was corrupted somehow / somewho. the current version is allways linked to in the first post

mikey
10-01-2003, 08:46 AM
Below is a copy of error messages in Terra Term when I tried to install mfs_ftp. Can someone please tell me what I'm doing wrong? This is on a GXCEBOT S1 at 3.01 via xtreme & Xpluz, 2 120 gb harddrives and turbonet.

Thanks in Advance,
Mikey

/var# tar -zxvf /var/mfs_ftp.1.2.9.tgz
mfs_ftp/
mfs_ftp/fifo
mfs_ftp/edit_title.tcl
mfs_ftp/mfs_export
mfs_ftp/mfs_ftp.tcl
mfs_ftp/mfs_stdinsert
mfs_ftp/mfs_stream
mfs_ftp/mfs_tarstream
mfs_ftp/ftpf.sh
mfs_ftp/cache/
mfs_ftp/tzoffset.tcl
mfs_ftp/mkfifo
mfs_ftp/setpri
mfs_ftp/log
tar: mfs_ftp/log: Cannot lchown to uid 0 gid 0: Function not implemented
mfs_ftp/abort_toggle.sh
mfs_ftp/ftpf21.sh
mfs_ftp/seasonpass/
mfs_ftp/readme.txt
mfs_ftp/ftp
tar: mfs_ftp/ftp: Cannot lchown to uid 0 gid 0: Function not implemented
mfs_ftp/drdos/
mfs_ftp/abort.doc
mfs_ftp/nuke_DC_key.tcl
mfs_ftp/set_DC_key.tcl
mfs_ftp/dump_DC_key.tcl
tar: Error exit delayed from previous errors

rc3105
10-01-2003, 10:16 PM
the tar lchown errors are normal, should work fine

cojonesdetoro
10-02-2003, 05:40 PM
I wrote a quick-n-dirty bash script that runs on a third linux machine and copies files between tivos. It could also work on a Windows/Cygwin box.


I'm posting it to spark discussion but it may not work for someone who doesn't know this stuff well. It needs a dos2unix and curl binaries. I had to get the latest curl which supports rate limiting but the dos2unix was the one included in my RH7.2 (probably same binary since forever).

Right now it lets you select one recording to transfer. I tried this with smartftp and FXP but I never get it to work consistently. I also couldn't figure out how to limit the rate. My Tivo gets really slow at anything over 200k get or put. The disadvantage to this is you need a machine with several gigabytes of free 'swing space'.

I figure it needs a whole lot more error checking (check drive space,failures,etc) and a change such that it lets you select multiple tasks and then do them all sequentially in a background batch.

If I get time I'll enhance it but if anyone else wants to, feel free.


BTW: I did some last minute changes w/out testing so the syntax may not be totally right but the curl commands are right on.


ABTW: I toyed with the idea of starting the 'get' in the background and sleeping 10 minutes and start the 'put' . It would allow a faster transfer because of the overlap but there's no way to be sure it gets the whole file.


#!/bin/sh

# bash function that gets a file listing from the tivo
function getlisting()
{
TIVO=$1

if [ "${TIVO}" != "" ]
then
curl ftp://$TIVO:3105/tmf/ 2>/dev/null > listing.$TIVO || echo error getting listing
else
echo usage: $0 tivoaddress
fi

IFS="
"

# I start at 100 so all columns the same. 1000+ recordings will break this
COUNT=100
mv -f listing.$TIVO templist.tmp
for LINE in $( cat templist.tmp )
do
LINE=${LINE:28:197}
echo "${COUNT}: ${LINE}"
COUNT=$(( ${COUNT} + 1 ))
done > listing.${TIVO}

dos2unix listing.${TIVO}
}



# this assumes you have two tivos that resolve to tivo and tivo2
# IP addresses should also work here
echo "1 - tivo to tivo2"
echo "2 - tivo2 to tivo "
read -n1 usrin


case $usrin in
1)
FROM=tivo
TO=tivo2
LISTING=listing.tivo

;;
2)
FROM=tivo2
TO=tivo
LISTING=listing.tivo2
;;
*)
exit
;;
esac

# get listing from source tivo
getlisting ${FROM}


# I use the 'less' file browser so you can make note
# of the recrding number
less ${LISTING}

echo
echo
echo
echo Enter Number
echo

# enter the number of the recording to transfer
read NUMBER

# parse the name
name="$( grep "^${NUMBER}:" ${LISTING} | cut -c33- )"
echo "+$name+"

# get the file
curl --limit-rate 200k --disable-epsv --remote-name --verbose --globoff "ftp://${FROM}:3105/tmf/${name}"
# put the file
curl -T "${name}" --limit-rate 200k --disable-epsv --verbose --globoff "ftp://${TO}:3105/tmf"

# delete the file from local drive
rm -f "${name}"

Wes
10-03-2003, 10:26 AM
I tried to read through 20+ pages of the skicky thread and it's almost half way. That's quite a bit to read up there.

Anyway, so far I still haven't find why my upload is soooo slow yet other than setting the priority for the Mfs_Ftp higher.

I have a newly cleared DTivo1 and it has nothing in the to-do and nothing at all. I've tried to upload shows into this one and another DTivo that has tons of shows in there. The download speed is great in the 1-2Mbit range but my upload is only like 100-300K range into either DTivo.

Since I just started trying Mfs_Ftp and installed the latest Mfs_Ftp into both DTivo just last night, I'm wondering if there is something obvious that people have experienced may cause this extreme slow upload problem.

TIA.

osetivo
10-03-2003, 11:02 AM
Wes:

a couple of things that I did (all from Riley's suggestions), you may have to do a bit more research for specifics, but I get 1MB inserts (WinXP SP1, Linksys BEFSX41 router)

EDIT: Active vs Passive FTP (big difference)

1. Boot parameters: root=/dev/hda4 runideturbo=false
2. checked my Maxtor HD with "hdparm -i /dev/hda"
Found MaxMultSect=16 and MultSect=off
rc.sysinit "hdparm -m16 /dev/hda"
Now the MultSect=16
3. used mbm's mfs_import (think it's at alt.org) and renamed it as mfs_stdinsert (EDIT: I assume this is now part of DL package, I haven't updated mfs_ftp for the last couple of versions)

4. Installed "setpri"
5. Set ithrottle at 0
6. Set insert_priority at 10


Originally posted by rc3105

speed issues - the biggie

here I thought it was something complex & it was just the !@#$% ideturbo module, figures. thanks mbm!

to check your tivo boot parms issue this command from bash

bootpage -q /dev/hda

you should get roughly this


IP address: 192.168.0.2
Primary boot partition: 3
Alternate boot partition: 6
Hostname: unnamed
Net boot kernel name: linux.px
Boot parameters: root=/dev/hda4 runideturbo=false
MAC address: bd:ca:43:47:75:90
RF channel: 0
Standby: 109
1


if the "Boot paramters:" line DOES NOT include "runideturbo=false" then you need to add it

note whatever your root parms are. "root=/dev/hda4" or "root=/dev/hda7"

and update your parms by issuing this command

bootpage -P "your_current_root_parms runideturbo=false" /dev/hda

on my boxes it's

bootpage -P "root=/dev/hda4 runideturbo=false" /dev/hda

reboot, try an insert. should be MUCH faster :cool: [/B]

Wes
10-03-2003, 08:02 PM
osetivo, thanks for the tips. I've made all of the changes except setting the priority for ithrottle and insert_priority. I'm getting about 500MB/s when i have the unit in standby mode. How do you use the setpri to set those priority ?

osetivo
10-03-2003, 09:49 PM
Originally posted by Wes
osetivo, thanks for the tips. I've made all of the changes except setting the priority for ithrottle and insert_priority. I'm getting about 500MB/s when i have the unit in standby mode. How do you use the setpri to set those priority ? They are in first couple of lines in mfs_ftp.tcl. (FYI im still using version 1.2.8f)

To be honest, I made the suggested changes to my TiVo back in July and went from 200 to 600-700KB inserts, and sort of settled on that as the best I could do, which was fine since I rarely insert TYs. However, the other day I inserted a file and it was flying (1MB/sec) I don't recall any other efforts (Since July) to increase speeds nor have I done any other changes to the Tivo since then, so I speculate the latest speed bump was probably something on the WinXP/PC side of the equation.

I did recently reinstall the OS and got the latest version of SmartFTP, and may have changed firmware of the router, other than that I really don't know how I got it from 600 to 1MB, sorry.

Will
10-04-2003, 03:20 AM
Originally posted by osetivo
the other day I inserted a file and it was flying (1MB/sec). I don't recall any other efforts to increase speeds

An increase from your normal 0.7 to 1MB is not totally out of the range of random variation. I'm normally around 0.82 and peak a little above 1MB from time to time. Without changing anything else I can go to a line command ftp client and vary from 0.8-1.7 MB. There are _lots_ of variables in this equation! Seemingly random. And they mislead you into seeing patterns that frequently aren't really there.

osetivo
10-04-2003, 10:13 AM
Originally posted by Will
An increase from your normal 0.7 to 1MB is not totally out of the range of random variation. I'm normally around 0.82 and peak a little above 1MB from time to time. Without changing anything else I can go to a line command ftp client and vary from 0.8-1.7 MB. There are _lots_ of variables in this equation! Seemingly random. And they mislead you into seeing patterns that frequently aren't really there.

my 1MB was average sustained for the entire insert via SmartFTP

mikey
10-10-2003, 11:07 AM
One more question, for now :)

How do I rename a show with mfs_ftp?
Thanks
Mikey

rc3105
10-10-2003, 11:15 AM
you don't. use tivoweb before extracting the recording

edit: to answer the ? in the next post

really doesn't matter except for 1 thing. if the last line of your rc.sysinit is "exec < /dev/ttyS3 >& /dev/ttyS3; bash --login" without a trailing "&" then it needs to loaded before that

mikey
10-10-2003, 05:51 PM
Thanks Riley.
I tried changing the name using tivoweb and after updating some tivoweb modules it does change the name, BUT on a gxcebotd runnng software version 3.01 xtreme via xupgrade, when you sort the NowPlaying list alphabetically the show is still sorted by its original name:( no biggie. I was just wongering if there was another way using mfs_ftp.

Thanks,
Mikey

P.S.
Where in rc.sysinit would you insert hdparm -m16 /dev/hda ?
the beginng, middle, end, or just above/ below something else...
Not really wanting to screw up my tivo on something simple

Thanks again.

cojonesdetoro
10-10-2003, 07:20 PM
I know you must hear this a lot but I just can't help myself. This is by far the most rockin' tivo hack ever! It is freekin' AWSOME!!!!

It was because of mfs_ftp that I got another Tivo from Ebay and slapped a NIC and a coupla 129GB drives in it. Anyway.. that's all I had to say.. thanks.

Dank
10-12-2003, 04:16 PM
Is it possible to have this program also have MPEG available? (Sorry if this has already been asked, this thead is gigantic...) It seems that TyStudio just tags some header info onto the TY file to convert to mpeg; could this do that as well dynamically? Would be great to get rid of one extra step.

cojonesdetoro
10-12-2003, 07:05 PM
Originally posted by Dank
Would be great to get rid of one extra step.

There are other Tivo hacks that already do that pretty well. The strength of mfs_ftp is that you can use a regular ftp client to extract a recording as well as INSERT a recording. Right now, it's the only way to move recordings between Tivos. The fact that is uses FTP means that you can easily develop your own front end.

I believe tystudio can go direct to MPG in one step but I'm not sure that tystudio is still actively developed.

What would be sweeter is if we had a process whereby we go straight to a ready-to-burn DVD directory structure. Though, We're pretty close to that already with tytool. I think there's a 5 step process that can have you burn an already extracted recording in less than 1 hour.

Dank
10-12-2003, 07:58 PM
Originally posted by cojonesdetoro
There are other Tivo hacks that already do that pretty well. The strength of mfs_ftp is that you can use a regular ftp client to extract a recording as well as INSERT a recording. That is definitely a stength of this, but in my mind (and my usage) the strength of this program is that it brings the tivo closer to being a true internet multimedia device. By using FTP instead of TyTool or TyStudio, just about any computer that can access the network my tivo is on can grab the files with no additional software necessary. But since most computers cannot play TY files, they need additional software installed to either convert it to MPEG (using TySyudio or TyTool) or to install MPlayer or the DirectShow plugin in order to watch the video.

If the files could just be grabbed by FTP and then watched via any random DVD player program, with no extra steps required, it would be awesome.

cojonesdetoro
10-12-2003, 08:50 PM
Originally posted by Dank
If the files could just be grabbed by FTP and then watched via any random DVD player program, with no extra steps required, it would be awesome.

The closest you can get right now is to watch a direct stream from mplayer. Mplayer will play a file without even copying it. I think it will stream and play right from the Tivo. I haven't read the details but I believe they have it working. I think it still doesn't support random access (FF/Rew/skip,etc ), it just plays the stream as it gets it.

There is an mplayer version on the Xbox that supposedly works in this manner.

Dank
10-12-2003, 09:21 PM
Originally posted by cojonesdetoro
The closest you can get right now is to watch a direct stream from mplayer. Mplayer will play a file without even copying it. I think it will stream and play right from the Tivo. Yes, there is such an mplayer. I use it. But it is nothing close to what I am asking for for this software...

Why are you so against having the mpeg header put on the front of the file? Do you not see the benefit of being able to use generic, non-tivo, software to be able to extract and watch the tivo? Why are you against this seemingly simple feature??

rc3105
10-12-2003, 11:20 PM
many many MANY moons ago I looked into direct mpeg / vob extraction. (before this was released actually) works great, 'bout 700k/sec. the bottleneck being the tivo doesn't have a lot of horsepower & ty->mpg is more than just adding header info

2 main problems

most folks wanting mpg plan to burn dvd or vcd - exporting mpg directly opens a pandora's box of support / format questions (even as mpg tivo video is still kinda wacky)

mplayer, XBMP, XBMC & mythtivo allready play ty and there's a ms mediaplayer directshow filter in the works. why burden the tivo with the extra load when there are plenty of ty enabled apps???


--
Riley

cojonesdetoro
10-13-2003, 03:27 PM
Originally posted by Dank
Why are you against this seemingly simple feature??

"seemingly" is the operative word. I'm not trying to belittle your idea but I have followed the development of a lot of these projects. I am no developer myself but have worked around them enough to know that what seems simple can sometimes require a great deal of effort to materialize.

MFS_FTP was designed to be an extraction and insertion tool but not a conversion tool. There are already other hacks to do that well so there's no point in "re-inventing the weel". It would be better to create a script that can FTP a file down to a PC from a mfs_ftp tivo server and then use tytool to convert it to mpg all in one motion. That can be done in a batch file or written in VB.

BTW, the best procedure I have found for getting a recording to a DVD is here:

http://www.dealdatabase.com/forum/attachment.php?s=&postid=102993

Written by DDB user Just4Fun

It works for all recordings that are not "broken" (there are some recordings that have problems whereby the Tivo can still play them but nothing else will play or convert them properly).

ABTW, the Tivo does not record dvd compliant mpg streams but there are hacks for that too. Without these hacks, the Tivo makes mpg streams that will play properly on a lot of new DVD players but some older ones are screwey.

Dank
10-13-2003, 05:53 PM
Originally posted by cojonesdetoro
"seemingly" is the operative word. Yes, and since the developer has said it isn't actually as simple as I thought, then I understand why it isn't in product. Oh well, it would have been nice.
I'm not trying to belittle your idea but I have followed the development of a lot of these projects. I am no developer myself but have worked around them enough to know that what seems simple can sometimes require a great deal of effort to materialize.

MFS_FTP was designed to be an extraction and insertion tool but not a conversion tool. There are already other hacks to do that well so there's no point in "re-inventing the weel". It would be better to create a script that can FTP a file down to a PC from a mfs_ftp tivo server and then use tytool to convert it to mpg all in one motion. That can be done in a batch file or written in VB. Actually, you are trying to belittle my idea... You are saying there is no use for it, because there is other software for extraction, when the whole point of my idea is the need for no tivocentric software on the PC. So saying "other conversion tools do this" is 100% misguided, because my whole goal is making a system that needs no clientside conversion tools. That means no VB scripts, no TyTool batches, nothing other than a standard FTP client and standard DVD software.

I know how to write batches, I know how to convert the FTP'd TY to MPEG, I know how to write software to automate that process. I am a programmer by profession. But if I have to write clientside software or scripts, then it isn't what I'm looking for here. Makes sense? I understand that YOU don't see the need in this, and YOU want mfs_ftp to just move files between your tivos, but that doesn't mean other people can't use the tool for other purposes.

But since rc3105 says its too much drain on the TiVo, then I guess it can't be done. I'll accept a technical rationale, but you can't convince me that nobody wants the feature that I want, because I know at least one person who wants it...

rc3105
10-13-2003, 07:57 PM
it's perfectly doable, 700k/sec is still better than realtime for most recordings. I just don't have any compelling reason to release / support it ;)

for the do-it-yourselfer, 1.2.9f, line 508 is

exec $info(path)/mfs_stream -s $fsid >@$info(dc) 2>/dev/null

port typrocess from TyStudio to the tivo and change line 508 to something like this

exec $info(path)/mfs_stream -s $fsid 2>/dev/null | typrocess -pipe -mplex { -V -f 3 } -demux >@$info(dc) 2>/dev/null

don't think that syntax is quite right but you get the generall idea, pipe from mfs_stream into typrocess & then from typrocess out the ftp data channel


--
Riley

cojonesdetoro
10-13-2003, 09:34 PM
Originally posted by Dank
I know how to write batches, I know how to convert the FTP'd TY to MPEG, I know how to write software to automate


Well... good for you. I didn't write anothing about it not being useful. I merely stated that mfs_ftp was not designed with that in mind whereas other tools are more evolved along those lines. I think that's a fair statement. I'm sorry that you found my response insulting. I wasn't trying to "talk down" to you but I thought I was dealing with a newb. So, seriously, I apologize if I ruffled any feathers.

Riley seems to have given you a pretty good answer. Piping it the way he shows looks like it would work. You'd just have to make sure you name the file with an mpg extension.. but I'm sure you already knew that. If you manage the CPU usage, the Tivo may even be usable while doing all this work. Let us know how it goes.

BTW, if you have an FTP client that can output to <stdout> like wget or curl you can pipe it right at the client side without having to have an intermediary file. That would be easier than porting typprocess to the Tivo.

maybe something like:

wget ftp://tivo:3105/ty/recording -o - | typrocess ... > mpegfile

(not sure of syntax here, of course)

I use a technique like this to transfer recordings between tivos using my linux box as an intermediary with a command like this:


curl --options get tivo1 | cur --options put tivo2


the bytes come into the linux box from tivo1 and make a hairpin turn (through a buffer) to tivo2. Fun stuff. It... mostly works. :-/
I've been able to automate the batch transfer of many recordings without the need of intermediate disk space. It still craps out once in a while and I haven't figured out why.


This should even work under cygwin/windows


AsI stated in earlier threads. FXP would be even better but I could never get it working right. This technique let's me watch the transfer and detect an error which can trigger an automated response (restart mfs_ftp, retry copy, delete partial file, etc.)

milhouse
10-14-2003, 10:34 AM
Would there happen to be any work underway to get mfs_ftp to work with ftpfs? I would like to mount my tivo mfs_ftp as a drive on my linux box that I am using for an ftp server. It would allow for simpler sharing of ty files over the internet with the security of my linux Proftp server.

If there is any other way to add security to MFS FTP, I would appreciate ant tips from the pros.

Milhouse.

Generiq
10-14-2003, 11:36 AM
I think more securtiy for mfs_ftp was in rc3105's to do's that he listed earlier in the thread.

cojonesdetoro
10-14-2003, 03:28 PM
Originally posted by milhouse
sharing of ty files over the internet with the security of my linux Proftp server.


Also remember that mfs_ftp is a single threaded app. You have to make sure that two people don't try to get a recording at once. Most FTP servers are tunable in this way so I'm sure ProFTP can accomodate this.

I like the idea of sharing directly from the Tivo. THe alternative is to copy the tmf files to your PC and using that edonkey program some guys are using to share shows. It creates a lot of storage demands. The advantage of Edonkey/PC is that more than one recording can be shared at once and the recording can be dloaded from multiple sources if more than one person is sharing the same tmf file.

Dank
10-14-2003, 07:31 PM
Originally posted by CptKirk
I just tried to install v129f on the Tivo and got the following errors.
...
tar: Error exit delayed from previous errors
Don't worry about those errors... It installed. Everyone gets those.
Just try it out and verify that its working; if it isn't then ask again.

Hi8
10-15-2003, 12:20 PM
Originally posted by rc3105

if savepath.txt is in the directory with tmf2ty_cfg.exe the first line of the file is used as the output path. these are tcl paths so use forward slashes like so

x:/extracts/ - d:/my_recordings/ - v:/video_files/



RC;

having troublegetting path to change to my d:/extracts/ line specified in the savepath.txt that resides in the same directory as the .exe


it says it can't create the path or doesn't exist. Yet it does create it on c: however I don't have enough room for the file to be extracted there, and it fails about 3/4 the way through.

it just seems to ignore my savepath.txt file.

mavrcksd
10-15-2003, 11:06 PM
Originally posted by Hi8
RC;

having troublegetting path to change to my d:/extracts/ line specified in the savepath.txt that resides in the same directory as the .exe


it says it can't create the path or doesn't exist. Yet it does create it on c: however I don't have enough room for the file to be extracted there, and it fails about 3/4 the way through.

it just seems to ignore my savepath.txt file.

I had this issue as well.

put the savepath.txt file in the same folder that you have the TMF file in, NOT where the EXE file is.

So for all my TMF folders, I have the same savepath.txt file there.

It will create a folder with the name of the show under the path specified in your savepath.txt file

my savepath.txt file is

e:/tivo/video/ty

I created it on the tivo with JOE and then ftp'd it down to my computer, because i didn't have a copy of dos2unix.exe handy.

HTH

nsysblh
10-16-2003, 02:48 PM
I had the same sluggish upload performance that others had reported earlier in this thread. It was taking 40 min to insert a 30 min show.

The fix was: (As previously mentioned)

runideturbo=false

For example: (Make sure of the parms, and the reboot)

bash-2.02# bootpage -p /dev/hda
root=/dev/hda7
bash-2.02# bootpage -P 'root=/dev/hda7 runideturbo=false' /dev/hda
bash-2.02# bootpage -p /dev/hda
root=/dev/hda7 runideturbo=false
bash-2.02# reboot

Now it can insert a 30 min show in 10 min.

cojonesdetoro
10-16-2003, 05:23 PM
I have found that lftp (http://lftp.yar.ru/) may be the best CLI FTP client for linux to use with MFS FTP. It uses FXP which facilitates direct tivo-to-tivo transfers. It also has nice scripting and bandwidth throttling features.

It also has a workaround that stymies other FXP clients. Since MFS_FTP is single threaded, I think you can't "talk" to it while it's doing an FXP transfer. This means you can't get stats to see how fast the transfer is going. LFTP has an option to turn off stat checking. You also have the option of doin tivo-to-tivo transfer using the LFTP client as a hairpin (no local storage of files needed). It seems to transfer much more slowly this way but you can monitor progress.

EDIT: of course, you can still tail -f /var/mfs_ftp/port.3105.log to view progress so there's not real reason not to use fxp

.


Here's a sample script of the syntax I've used:


script listing:

open tivo2 -p 3105
set ftp:use-stat false
cd tmf
get "{Beer, Hot Chix and Baby Oil}{1970-01-01}{}{11.35 PM Tue Sep 09, 2003}{PPVP}.tmf" -o ftp://tivo:3105/tmf/
set cmd:move-background false
wait all
quit

called with:

lftp -f tempftpscript.txt


Note: The two tivos I have resolve to 'tivo' and 'tivo2'. Substitute your names or IP addresses for those in the script.


EDIT: lftp is also available for cygwin so it may be the best CLI FTP client for Windows and MFS_FTP as well.

http://cygwin.com/cgi-bin2/package-cat.cgi?file=lftp/lftp-2.6.6-1&grep=lftp

(I haven't tried it.)

cojonesdetoro
10-17-2003, 11:36 AM
Originally posted by philhu
Does the newest mfs_ftp work with Series 1 standalones with 16meg? But loaded with 2 120g HD's?

I've tried a few versions. It crashes the tivo while trying to enumerate the list of programs......

I have an unsubbed SA1 16MB and it's pretty stable. In fact, I've seen more stability issues with tivoweb. Maybe you've got a corrupt DB? Riley suggested that mfs_check program earlier. He says it's mostly harmless but I found that it shuts down one of the main tivo programs and you have to reboot after using it.

Edit: Did you find any messages in /var/mfs_ftp/port3105.log?

misato
10-17-2003, 10:23 PM
Just to add a little perspective to the "why can't mfs_ftp generate clean mpeg" question, I've been using mfs_ftp since 1.2.5 (and in fact am still using that version) and was one of the alpha tresters for tystudio. I think some here would not believe the amount of work that went into tystudio to get it to the point of being usable (and I was just testing it, not developing it), and it was (maybe still is) imperfect when I lost track of the project.

The version I have (and I think it's been pretty much static since Olaf went travelling), does a pretty good job of converting .TY files to MPEG, using the resources of a regular PC, but still doesn't (didn't) handle AV sync correctly in cuts.

So for material requiring editing (eg removing commercials), I still do an analog capture and edit and re-encode the resulting AVI files.

For material requiring only start and end trimming (eg uninterrupted movies), here's my procedure:

Extract from tivo using mfs_tools v1.2.5

Convert the TMF file to TY using a version of tmf.exe that Riley whipped up for me way back. It puts the result into X:\extract (hardcoded path), but subst lets me manufacture an X: drive if my PC doesn't already have one.

Convert to MPEG with TyStudio 0.5.0. Trim only the start and end. Bounce the end point 8 clicks beyond where I want the actual cut and I'll ususally get all the program material. (It has trouble stopping where I tell it, so the extra margin is necessary.)

remux the MPG file with TMpegEnc. This seems to reduce problems down the line.

Simple demux, again with TMpegEnc

Pull the audio into my general editor, look at the audio waveform by eyeball. Usually requires 2X or 3X amplification to achieve anything approaching correct recording levels. Apparently the tivo amps up audio more than my DVD playback card and TV together can. Maybe this is a way to shave bandwidth?

Recompress the amplified audio to MPA (time consuming, but it works. Leave it at 48 kHz to avoid any further degradation from resampling.

Remux the video with the new MPA file.

Done!

Sounds like a lot of handling, and it is, but I can do it faster than real time end to end... if I watch 1 movie while I work on the next, I can keep the TV running continuously as long as I can stay awake. And I'm using 1.5 gHz class PCs.

Comment:

I imagine the current tools are in some way better than the ones I've been using, or they wouldn't still be in flux. But I stopped uograding when things became workable and reliable. First rule of software: if it's not too badly broken, don't fix it any more.

2nd Comment: Can y'all imagine the CPU drain on the poor little tivo if it had to do all that work itself, to spare the much bigger/faster PCs the job of post processing?

3rd Comment:

Many thanks to:

Surgeon for 25Xtreme (and to all the people who made the pieces that went into it),

Kraven for his update (and those who made the bits that went into *it*),

Riley for mfs_ftp

Olaf and the gang for TyStudio (I hope the bugs eventually get worked out, but I use it almost every day anyway)

I love my Tivo, but I love it 10 times as much with tivoweb, the extraction tools, and mfs_ftp.

Riley's my HERO! (He, if we can single out any one person, made extraction work for me. After all, for me, it's not the tinkering that's fun, but getting results.)

rung
10-18-2003, 03:17 PM
Originally posted by misato

Convert the TMF file to TY using a version of tmf.exe that Riley whipped up for me way back.
You may want to consider extracting a "ty+" file with mfs_ftp instead of a tmf file. The "ty+" filetype is re-insertable like a tmf but can also be directly used by tystudio without an additional conversion step.

misato
10-18-2003, 08:30 PM
That's a thought. The TMF.EXE step is by far the fastest of the bunch, so there's not a lot of speed motivation to try upgrading, but it's something to try when I get the itch.

cojonesdetoro
10-18-2003, 11:59 PM
Originally posted by rung
The "ty+" filetype is re-insertable like a tmf but can also be directly used by tystudio without an additional conversion step.
So there is a benefit to Ty+ file extraction over tmf.

Is there any benefit to tmf extraction?

Can tmf files be considered 'deprecated'?

I imagine that support would have to be continued because there are archived recordings around that need to be inserted.

rc3105
10-19-2003, 12:48 AM
the ty & ty+ from mfs_ftp and tmf2ty are exactly the same thing

they're insertable and can be processed into mpg or played by any current utility

tmf are only required for archiving scrambled recordings and some of mfs_ftp's optional features (check the readme)

mikey
10-20-2003, 03:44 PM
Before I try it I thought I would ask.

If I record a ppv or HBO or Showtime ect. movie on my dtivo and insert it onto a friends dtivo that did not buy the ppv and/or does not subscribe to the movie channels are we asking for DTV/tivo legal trouble?

Just wondering,
thanks,
mikey

dasatelliteguy
10-20-2003, 05:03 PM
Hi. First off, just like to say wow! I love this little gem you whipped up Riley! I had some shows that I stored about a year ago on my hd and I have been dumping them back onto my DTivo T60 all day now. I love it! BTW, they were extracted with TivoApp if that helps anyone out.

I've been reading through this thread and I see a few people who have reported insertion bottlenecks. I am another one I guess. I can't really make out what has worked for others though. On the extraction side, I get about 1900 KB/s. I'm very satisfied with that of course. On the insertion side though, I only get about 250 KB/s if I'm lucky. I've tried changing the insertion priority from 10 to 3 and ithrottle from 1 to 4 in both the settings.tcl and mfs_ftp.tcl files. That didn't seem to do a darn thing. Maybe even slowed me down a touch.

Anyway, did I miss something? Did I adjust these settings in the wrong direction? What can I do to bump up my insertion throughput? I even tried putting the machine into standby mode which made little to no change. Any suggestions. I'm willing to try anything.

Thanks.


edit: the most likely problem is the ideturbo module is loaded. check the readme & earlier in the thread where that's discussed - Riley

cojonesdetoro
10-20-2003, 07:17 PM
Originally posted by mikey
Before I try it I thought I would ask.

If I record a ppv or HBO or Showtime ect. movie on my dtivo and insert it onto a friends dtivo that did not buy the ppv and/or does not subscribe to the movie channels are we asking for DTV/tivo legal trouble?

Just wondering,
thanks,
mikey

I think it falls under "fair use" like giving a friend a video tape. As long as you don't circumvent encryption then you do not violate DMCA ( in USA). "fair use" says you can give a friend a video tape but you can't distribute the tape widely and show it to a large audience.

Sharing the shows ala napster/gnutella might more likely be illegal but that doesn't seem to be stopping some people.

My .02$, though. Someone will correct me with great pomposity, I'm sure. ;^>

cojonesdetoro
10-20-2003, 11:10 PM
Connecting to 192.168.1.101
Connected to 192.168.1.101 Port 3104
220 Mfs_Ftp ver 1.2.9F - {sock23} from "192.168.1.100:2989"
[/B]

Okay... I just noticed something else. First let me explain quickly how FTP works (forgive me if you already know this stuff)

FTP needs to open two connections, a control connection and a data connaction. The control connection will always orginate from the client to some fixed port on the server. The data connection will either originate from the client, if in passive mode, or the server, if in active mode.

In the case of mfs_ftp, 3105 is the control channel but, somehow, you're using 3104 as the control channel. That might explain why passive mode is choosing 3103. Did you make a change in mfs_ftp to listen on 3104?

Make sure your forwarding

port 3105 on the outside to tivo port 3105 on the inside
port 3104 on the outside to tivo port 3104 on the inside


Oh, yeah, remember... Pokemon.. ;-/

AVD
10-20-2003, 11:27 PM
Originally posted by rc3105
tmf are only required for archiving scrambled recordings and some of mfs_ftp's optional features (check the readme)

when inserting a large file, the tmf file will stream, the ty file will get stuck after some time until the transfer is complete and the xml information that is at the end is processed

cojonesdetoro
10-20-2003, 11:47 PM
Originally posted by cojonesdetoro
Okay... I just noticed something else. First let me explain quickly how FTP works (forgive me if you already know this stuff)


<slap to the forehead>

Y'know... I just looked at the readme and it has a whole lot of valuable information in there. Yes, active FTP is supported and its the prefered method because it's faster. So try getting the router/firewall working with active. Opening port 3105 should be enough if you do active transfers.

rc3105
10-21-2003, 03:32 AM
Originally posted by mikey
Before I try it I thought I would ask.

If I record a ppv or HBO or Showtime ect. movie on my dtivo and insert it onto a friends dtivo that did not buy the ppv and/or does not subscribe to the movie channels are we asking for DTV/tivo legal trouble?

Just wondering,
thanks,
mikey
well, yes & no. the new recording doesn't have the same type of "roots" in the mfs db as a regular recording. as far as dtv/tivo knows you just recorded something manually & renamed it with tivoweb

this is partly lazy & partly on purpose. personally I DO NOT want the tivo to skip recoding something because it was inserted once upon a time & went into the recording history :)


edit:

an inserted ppv won't be logged to the access card or phoned in by the dtv side of the unit - however - the tivo logs will contain the original ppv info (name, date, time, channel) & upload to the mothership unless you run something like logsnull. it's unlikely tivo / dtv compare notes that closely but better safe than sorry. if you insert janet-superbowl & slowmo the halftime show 50 times tivo will know :eek:


one other thing

the terms of use explicitly prohibit any commercial or illegal use

you can record home videos, security camera footage, product demos or anything you hold copyright to & transfer / share it (I actually do this btw) but please don't distribute anything illegally - hollywood will freak, tivo will have to respond (again) and it'll just make things difficult for everybody

cojonesdetoro
10-21-2003, 07:13 PM
Originally posted by Karpa
I can run in active mode inside my firewall, but outside I have to run passive. I assume that I would have to open a port on my remote computer to get the back route for active. Is this correct?


Is the remote computer behind a firewall? If it is, the firewall needs to know how to handle active FTP. Active FTP means that the Tivo will be originating a TCP connection back to the remote computer whe you try to copy or do dir listing. Look at the firewall documentation to see if it supports active FTP. If it doesn;t then either use the PC outside the firewall or you're stuck with passive.

If you must use passive then make sure you forward port 3105 on the outside-of-the tivo router to port 3105 on the tivo. Do the same for 3104.

There's not much more you can do beyond this here. If it's not working then keep trying. What you showed me before showed that you were forwarding 3105 to 3104 on the tivo.

dialanothernumb
10-22-2003, 07:33 AM
First of all thanks Riley for a superb utility!:)

With tivo -> tivo transfers, using smartftp, I'm getting an active outgoing connection but passive incoming connection, so overall transfer speed is about 263 KB/s. Both tivos have port 3105 open and are within my LAN and behind my NAT/Router (as is the PC running smartftp). mfs_ftp is of course running on both tivos having opened up 3105 by default.

Is this the limit or can tivo->tivo run active connx?

Thanks

TriscuiT
10-22-2003, 09:28 AM
dialanothernumb,

From reading about FXP (http://www.smartftp.com/support/kb/index.php?id=14) on the SmartFTP (http://www.smartftp.com) it appears that FXP can only use passive mode transfers. Or, perhaps SmartFTP only supports passive transfers using FXP. Either way, it seems we are limited to passive.


To everyone/anyone:

I tried using MFS_FTP for the first time last night. I got the latest versions from the first post in this thread and installed/ran it on two DSR6000s. I then got the latest SmartFTP (v1.0 build 979) and installed/ran it on my laptop. I initiated an FXP transfer of an NFL Sunday Ticket game (so it is better than 4 hours of recording) from one box to the other. I left it running over night.

This morning I checked the destination DTivo and found that it was showing the new "Now Playing" entry as still recording. Checking the MFS_FTP logs I found a "broken pipe" error. I don't have the full error message with me (had to hurry in to the office). From what I have read here, I understand that I should be able to delete the failed recording through the MFS_FTP program. Is there anything in particular I should look at regarding the "broken pipe" error message?

rc3105
10-22-2003, 07:02 PM
Originally posted by dialanothernumb
First of all thanks Riley for a superb utility!:)

With tivo -> tivo transfers, using smartftp, I'm getting an active outgoing connection but passive incoming connection, so overall transfer speed is about 263 KB/s. Both tivos have port 3105 open and are within my LAN and behind my NAT/Router (as is the PC running smartftp). mfs_ftp is of course running on both tivos having opened up 3105 by default.

Is this the limit or can tivo->tivo run active connx?

Thanks
a ftp transfer is between server & client tcp sockets. when the server is active then the ftp client is passive. when the transfer is server - server one server must act as a client & run passive

passive mode extracts don't suffer from the tivo tcp bug as much as inserts do. if the sending tivo happens to run passive & the recieving tivo runs active then fxp transfers run pretty much full speed, otherwise you see the passive mode insert slowdown :eek:

philhu
10-23-2003, 11:39 AM
rc3105-

I've asked in here a few times now. It seems to be ignored.

Does the latest version fix the problem with a large series 1 tivo with ALOT of shows and 16 meg crashing as mfs_ftp tries to enumerate the files in the tivo?

We spoke about this a while back and you said it was due to the way you get the names and I should try an older version and you'd look at it for a future version.

Any fixes found?

rc3105
10-23-2003, 03:16 PM
try it & see, I dunno. it works fine here on a hdr112 with 300+ recordings. the tivo itself is so slow as to be unusable with that many shows, (2 mins to go from livetv to nowplaying) but insert & extract works fine

tivomaster
10-25-2003, 03:07 PM
RC3105... I want to start off by thanking you for MFS_FTP...

I just upgraded from 1.2.8g to 1.2.9F and everthing seems to be working. I can transfer shows but I noticed a couple of things that seem different.

1. When I initially sign on it seems like it takes considerably longer between entering the username and prompting for the password. 1.2.8g used to prompt almost immediately. .9F takes about 15 seconds to prompt for password. Is this normal?

2. When I start up mfs_ftp it displays: catch close lastsock val "can't read "info(lastsock)": no such element in array". Is this nomal?

Again, Thanks for the great tool.

rc3105
10-26-2003, 04:29 AM
yes & yes. F builds the cache before enabling the control socket so login can take a bit longer. the failed catch close on startup is normal - verbose logging on new functions/features makes troubleshooting from posted logs easier :)

wkozun
10-26-2003, 06:46 PM
I thought I read somewhere in this thread that you can now extract and insert ty+ files. I tried to do this and the file seems to insert ok except the show info is incorrect - it uses the filename (i.e. 123456 {Seinfeld...). Is that supposed to happen?

What format should I use to extract if I want to reinsert and also play files directly from my PC? Are there players that can play tmf files?

Is there some documentation available for mfs_ftp beyond what is in the readme.txt file. It doesn't go into too much detail on this issue.

Can anyone point me in the correct direction to troubleshoot slow insertions? When inserting I am lucky to get 160k/sec (according to the log). I am using the network drivers that came with version 3.0 of the TiVo software and I don't believe I have made any changes to the mfs_ftp.tcl file. I have tried looking through this thread but it is like looking for a needle in a haystack.

eastwind
10-26-2003, 07:38 PM
Tmf files include the ty files and so can be used for extraction/insertion. If you want to play the file on you PC you can extract the ty from the tmf with a program called tmf2ty (link found in the initial post of this thread (tmf2ty_cfg.exe I think).

Then you can play the files in WMP (with the appropriate direct-show filters), or use another toolset to make a mpg file. Since you want to deal with ty+ files I'm guessing you already know what to do once you get a ty file. IIRC, TYTool8r3 will let you work directly with the tmf.

ew

wkozun
10-26-2003, 10:20 PM
[i]

Then you can play the files in WMP (with the appropriate direct-show filters), or use another toolset to make a mpg file. Since you want to deal with ty+ files I'm guessing you already know what to do once you get a ty file. IIRC, TYTool8r3 will let you work directly with the tmf.

ew [/B]

How dows TyTools work with tmf files. Do you first have to download the tmf to your PC or can TyTools work with the tmf directly from your TiVo.

The ideal situation would be if someone wrote a player to play tmf files since right now if I want to play a file from PC and perhaps reinsert some day I will need it in both tmf and ty (or mpetg) format, will I not?

rc3105
10-26-2003, 11:48 PM
tmf & ty from mfs_ftp both insert fine

mplayer, TyStudio and the windows directshow filter need ty

X-box Media Player and X-box Media Center both play ty, XMBP will also play tmf. (if you use your xbox as a dvd player all you need to do is put the tmf or ty on dvd-r)

there's really not any reason to archive as tmf - except perhaps that ocassionally the xml isn't restored when inserting ty or ty+. works fine here, I can't track down why it doesn't work for some unless they post level 3 logs

mikey
10-30-2003, 12:32 AM
I have been trying to speed up my insertion, currently averaging 750 KB/s, and extraction, currently about 1600 KB/s, speeds and read this earlier post by Riley:

“hdparms can make a big difference, this is safe for stock quantums “hdparm -c1 -d1 -m8 -S0 /dev/hda /dev/hdb" try it from bash before adding to rc.sysinit or rc.remote-login”

-m8 is for MultiSect, -d is for DMA and can be set to a maximum of 2 for a series 1 dtivo, I think, but can someone please tell me what are the appropriate values are for –c, and –S from the example below, assuming it shows the needed information, and if I should set the DMA? I also noticed that the original quantum had TrkSize=32256, SectSize=21298 and the my Maxtor is showing 0 for both. Is this correct? I don’t think I have changed this but..



/var/hack# hdparm -i /dev/hda

/dev/hda:
Model=Maxtor 4R120L0, FwRev=RAMB1TU0, SerialNo=M0000000000000000000
Config={ Fixed }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=57
BuffType=3(DualPortCache), BuffSize=2048kB, MaxMultSect=16, MultSect=16
DblWordIO=no, maxPIO=2(fast), DMA=yes, maxDMA=0(slow)
CurCHS=16383/16/63, CurSects=-66060037, LBA=yes, LBAsects=240121728
tDMA={min:120,rec:120}, DMA modes: mword0 mword1 *mword2
IORDY=on/off, tPIO={min:120,w/IORDY:120}, PIO modes: mode3 mode4

Riley thanks for the great tool, all or your hard work, and answering all of these questions,
Mikey

rc3105
10-30-2003, 01:50 PM
add these lines to rc.sysinit, rc.sysinit.author or wherever you like in a startup script

hdparm -c 1 -S 0 -m 8 -M 1 -d 1 /dev/hda /dev/hdb
hdparm -c 1 -S 0 -m 16 -M 1 -d 1 /dev/hda /dev/hdb

disabling ideturbo (via the boot parms or simply deleting the file) and the hdparm settings -c 1 & -m 8 make the biggest differences


cmd explainations:

the first line sets the multicount to 8, supported by nearly every drive you're likely to see. the second line bump's that to 16, supported by 95% of all drives. if you have one of the unfortunate 5% hdparm returns an error & multicount remains at 8

/dev/hda /dev/hdb applies changes to drive A and B if available

-c (0/1) is for 16/32 bit transfers. 32 is of course MUCH more effecient

-S is the standby timeout. 0 == never. generally not an issue, but never hurts to be thorough

-m get/set multiple sector count, varies by drive, generally 8 or 16

-d (0/1) disable/enable dma. set to 1 (enable) of course.

(there's another command to change the dma mode, but don't change it unless you're feeling adventureous, have everything backed up & know exactly what the command is all about. *nix guru's only)

-M drive streaming-media mode (0/1) - generally doesn't make any difference unless you're running 10,000 rpm AV drives

Shr00m
10-30-2003, 05:31 PM
Does the version of mfs_ftp effect the way a .ty file gets extracted to your computers hard drive?

In other words, as long as I have the .ty file on my computers hard drive, will it be exactly the same no matter what version of mfs_ftp im using?

Thanks.

rkshack
10-30-2003, 11:47 PM
I just downloaded the mfs_ftp file. After I install it in to the var directory how do I use it. Is there an FAQ that goes over how to use it? Also I have upgraded hard drives on a tivo but no other hacking. Where do I get the files etc to do the modifications? If I download the files to my pc how do I convert them to mpeg or something for dvd burning. Also Can I send video from one tivo to another without using my pc (from a menu on the tivo) if the tivos are networked. How do I set up streaming so I can watch one show from another tivo.

dialanothernumb
10-31-2003, 04:57 AM
No one will give you all this in one post.

There is a pretty good search function on this site. I would use it if I were you.

When your questions become far more specific, you'll get more specific answers.

Read thru this thread in particular if you want to find out more about mfs_ftp.

In the compressed file you downloaded from the first post there should be a readme.... my suggestion is that you read it...

rc3105
10-31-2003, 08:15 AM
Originally posted by Shr00m
Does the version of mfs_ftp effect the way a .ty file gets extracted to your computers hard drive?

In other words, as long as I have the .ty file on my computers hard drive, will it be exactly the same no matter what version of mfs_ftp im using?
short answer - every version of mfs_ftp has produced identical tmf, ty & ty+ files (so far)

long answer - the embedded xml has a version number and there's a RFC thread on the forum where tmf was developed. if/when the formats evolve the specifics will be available there

edit: the embedded xml has evolved slightly but the ty will allways be a 1:1

TriscuiT
10-31-2003, 10:51 AM
Is there an FAQ that goes over how to use it?

To my knowledge there is no FAQ for MFS_FTP. As dialanothernumb pointed out, there is a README in the file you have downloaded. Read that and then through this thread. I know it is a long thread but there is a tremendous amount of useful information in here.


Also I have upgraded hard drives on a tivo but no other hacking.

Do you have a NIC in your TiVo? Certainly that would qualify as a "hack" of sorts. If you do not have a NIC in it then MFS_FTP is rather pointless.


Where do I get the files etc to do the modifications?

You can find pretty much everything you will need at TivoStuff (http://www.tivostuff.com). There is a download section and a HOW-TO section (among other things).


If I download the files to my pc how do I convert them to mpeg or something for dvd burning.

You can start reading here (If I download the files to my pc how do I convert them to mpeg or something for dvd burning.).



Also Can I send video from one tivo to another without using my pc (from a menu on the tivo) if the tivos are networked.

No. You can use an FXP transfer from one TiVo to another, but it will need to be initiated by your PC. Look a little way back in this thread for some comments I made on SmartFTP and FXP.


How do I set up streaming so I can watch one show from another tivo.

Not at the moment. You can check here (http://www.dealdatabase.com/forum/showthread.php?s=&threadid=25139) for a status of the project.

scottf
11-03-2003, 01:39 PM
Trying to uninstall mfs_ftp. I've telneted into Tivo and managed to delete all but can't delete a few files in /var/mfs_ftp/cache. Can anyone help?

Thanks

cojonesdetoro
11-03-2003, 03:30 PM
Originally posted by scottf
Trying to uninstall mfs_ftp. I've telneted into Tivo and managed to delete all but can't delete a few files in /var/mfs_ftp/cache. Can anyone help?
Thanks

What error are you getting?

try this "rm -rf /var/mfs_ftp"


NOTE: THIS IS A DANGEROUS COMMAND!! Make sure there is at least one reply that agrees with me before trying this and make sure you read it twice then read it again before pressing enter.

scottf
11-03-2003, 07:49 PM
Was getting error couldn't be removed with FTP client. Used the rm command and it worked great, thanks :)

SteveJenkins
11-03-2003, 09:26 PM
I haven't needed it until this TV season, but I just installed MFS_FTP on my 2 Sony SVR-2000 SAs without any problems. Killer app. My wife thinks I'm a hero for being able to pull recordings from the downstairs TiVo to the upstairs one. Thanks to Riley and all others who contributed.

This probably doesn't qualify as an MFS_FTP bug, but I figure I'd mention it any way in case anyone else has encountered this and has a possible solution.

I mfs_ftp'd an episode of Friends from the Theater TiVo to the Living Room one (tivo2 to tivo1), and it shows up just fine in the Now Showing list in TiVo Web. However, clicking on the item in the TiVo Web interface shows this error:

INTERNAL SERVER ERROR
--cut here--
action_series '/' ''
expected integer but got ""
while executing
"db $db openid $objectid"
("uplevel" body line 2)
invoked from within
"uplevel $body"
invoked from within
"transaction {uplevel $body}"
(procedure "RetryTransaction" line 5)
invoked from within
"RetryTransaction {
set series [db $db openid $objectid]
set title [strim [dbobj $series get Title]]
set score [dbobj $series get ThumbData]
se..."
(procedure "::action_series" line 15)
invoked from within
"::action_$action $chan $part $env"
("eval" body line 1)
invoked from within
"eval {::action_$action $chan $part $env}"
--cut here--

I'm assuming this has something to do with the fact that some info must be missing when mfs_ftp-ing vs. recording natively on the machine. Anyone got any ways to hack around this?

cojonesdetoro
11-03-2003, 09:34 PM
Originally posted by SteveJenkins

snip
the TiVo Web interface shows this error:

INTERNAL SERVER ERROR
--cut here--
snip
I'm assuming this has something to do with the fact that some info must be missing when mfs_ftp-ing vs. recording natively on the machine. Anyone got any ways to hack around this?

It's a known problem. Tivoweb is probably trying to read some stuff that MFS_FTP doesn't insert. The easy workaround is to click the episode title (to the right) instead of the series title. You can do everything you need to do there (rename, change 'save until', etc.) The other possibility is to edit the tivoweb tcl code to make it mfs_ftp friendly. It probably would involve just commenting some stuff out.

quantum_3
11-03-2003, 10:56 PM
please help,

i tried to put this on a dsr6000 dtivo and this is what it did.


/var# tar -zxvf /var/mfs_ftp.1.2.9G.tgz
mfs_ftp/
mfs_ftp/fifo
mfs_ftp/edit_title.tcl
mfs_ftp/mfs_export
mfs_ftp/mfs_ftp.tcl
mfs_ftp/mfs_stdinsert
mfs_ftp/mfs_stream
mfs_ftp/mfs_tarstream
mfs_ftp/ftpf.sh
mfs_ftp/cache/
mfs_ftp/tzoffset.tcl
mfs_ftp/mkfifo
mfs_ftp/setpri
mfs_ftp/log
tar: mfs_ftp/log: Cannot lchown to uid 0 gid 0: Function not implemented
mfs_ftp/abort_toggle.sh
mfs_ftp/ftpf21.sh
mfs_ftp/seasonpass/
mfs_ftp/readme.txt
mfs_ftp/ftp
tar: mfs_ftp/ftp: Cannot lchown to uid 0 gid 0: Function not implemented
mfs_ftp/abort.doc
mfs_ftp/nuke_DC_key.tcl
mfs_ftp/set_DC_key.tcl
mfs_ftp/dump_DC_key.tcl
mfs_ftp/settings.tcl
tar: Error exit delayed from previous errors
/var#






so what did i do wrong please? thanks

Q_3

SteveJenkins
11-03-2003, 11:06 PM
Nothing is wrong. You can ignore those errors. They are just minor complaints about the tarball wanting to but not being able to change ownership of some files (in this case, the symlinks for log and ftp). I saw the same errors on my TiVos when initially installing, and it works fine.

sanderton
11-04-2003, 02:03 PM
Originally posted by SteveJenkins
This probably doesn't qualify as an MFS_FTP bug, but I figure I'd mention it any way in case anyone else has encountered this and has a possible solution.

I mfs_ftp'd an episode of Friends from the Theater TiVo to the Living Room one (tivo2 to tivo1), and it shows up just fine in the Now Showing list in TiVo Web. However, clicking on the item in the TiVo Web interface shows this error:

INTERNAL SERVER ERROR
--cut here--
action_series '/' ''
expected integer but got ""
while executing
"db $db openid $objectid"


This is a tricky one to deal with for mfs_ftp, as the series id is specific to the TiVo it was recorded on.

To work the code would need to tranfer the TmsId (which is universal), and then look up the equivalent series id on the destination TiVo.

As far as I can see looking in the mfs_ftp code, the TmsId is not sent in the XML?

quantum_3
11-04-2003, 08:08 PM
great...now i am up and rolling.
I have opened up cute ftp pro and set it for the downstairs dtivo using port 3105 telling the program it's unix, binary transfer, and to use port instead of passive. no problems i'm now in the tmf folder and want to stream the latest ep. of enterprise from the downstairs to the upstairs dtivo. i click on a new connection and to the same thing for the upstairs. no problem... i now have 2 tabs in ftp pro, both with active connections to both dtivos.
now the question is this.

will someone please tell me how to pull a .tmf file from one tab in ftp pro and stream it to the other tab (tivo) without haveing to download the whole ep. to my harddrive 1st??

just call me slow, :)
Q_3


ps..thanks SteveJenkins for letting me know all is well with the install.

SteveJenkins
11-04-2003, 08:38 PM
I'm using FTP Voyager, and all I did was drag from one session's remote window to the other, which initiates FXP. From what I've heard, CuteFTP will do the same.

Funny - Enterprise was the second show I transferred from downstairs to upstairs :)

rc3105
11-05-2003, 12:59 AM
Originally posted by sanderton
As far as I can see looking in the mfs_ftp code, the TmsId is not sent in the XML?
if the tmsid exists in the recording info it'll be included in the xml. the tmf spec is well over a year old now and basically just for archiving. Tivo Media Format - RFC & utilities (http://alt.org/forum/index.php?t=msg&th=55&start=0&rid=24) is the thread to discuss extending the tmf/xml spec

rc3105
11-16-2003, 04:32 AM
this is just a parking spot for files, see the first post (http://www.dealdatabase.com/forum/showthread.php?s=&threadid=21915&perpage=1) for installation instructions

bcc's mfs_stream & mfs_export binarys (http://www.dealdatabase.com/forum/showpost.php?p=169896&postcount=210)

(200,669@04:07:20:26pm)


-----------------------

mfs_ftp itself is not GPL, but the support binaries mfs_stream, mfs_tarstream, mfs_export, mfs_import & mfs_stdinsert are

it has been brought to my attention that the source & apropriate GPL notices are not included with these binaries. I aplogize for that oversight and will be updating things to correct that as time permits. (on deadline for a few things right this minute, please be patient)

in the meantime, the source can be found by searching here or on alt.org/forum

rung
11-16-2003, 03:13 PM
Maybe there's another use for the "txt" files, but I thought these were for getting human readable information about the show. "xml" files I assumed were to be used for automation but there is xml in the "txt" file as well, so I am confused. What about a txt files that looks more like the detail screen when you press "enter" on your Tivo (like the attached)?

keithc
11-17-2003, 07:08 AM
I recently upgraded my UK tivo from 2x100gb to 1x300gb. mfsbackup wouldn't divorce the drives so I had to start with a new image. No problem, I thought, I use mfs_ftp to grab the shows I want to keep.

Everything I wanted extracted in tmf format except for 2 very old streams which just caused mfs_ftp to crash. I was able to extract them as .ty but after insertion into upgraded machine, tivo just said there was an internal error when trying to play them.

By now, I'd put the old disks into my Linux machine and having hacked the vplay code to work with two disks, mfs_dumpobj showed that there was only one fsid for each stream even though the tys extracted were ~4gb. I thought each chunk was limited to 512mb. Is this a corruption in the MFS?

tystudio says it's unable to open the ty. Fastforwarding in mplayer seems to show the streams as intact.

Any suggestions for how I can salvage these and get them back onto tivo?

rc3105
11-17-2003, 08:34 AM
largest size of a fsid is 512 meg, if they came out around 4 gig each then something's definatly wrong. not a clue what though :confused:

edit: were you by any chance extracting to a filesystem with a 4 gig filesize limit like fat32?

keithc
11-17-2003, 10:27 AM
Streams were extracted to ext3 and using mplayer I can skip to the end which looks like the end of the films.

I think they were recorded when the software was at 1.5.x. There wasn't a format change from 1.x to 2.x was there?

rc3105
11-17-2003, 04:04 PM
actually there was.

I've got a phillips at 1.3, I'll see if i can duplicate your problem

maule
11-17-2003, 04:08 PM
can i still use ftp to transfer programs and other files in addition to .ty stuff with this? i know it's a stupid question, but i had to ask.

rc3105
11-17-2003, 04:48 PM
use tivoftpd (running on port 21) to transfer regular files, same as allways

use mfs_ftp (on port 3105) to transfer recordings

Wes
11-17-2003, 08:09 PM
Just want to say thanks for such great program and keep improving it.

After using the 1.2.9B for quite a few months, suddently, it has been crashing my DTivo until I removed it from my startup. No idea what happened to it. Just upgraded to 1.2.9H and all is back to normal again. :)

rc3105
11-18-2003, 01:32 AM
probably those wacky taken episodes. H should have that solved

SirTrini
11-18-2003, 11:01 AM
I'm not sure where I lost this function, but has anyone else noticed this... when I first setup MFS_FTP, the shutdown and phoenix "commands" would show up as folders and if you double clicked on them they would perform their function.

In the last 4 or 5 iterations of MFS_FTP, those commands now show up as txt files and when I double click on them nothing happens.

Thoughts or suggestions on how to get around this?

Thanks!

Dunkwho
11-18-2003, 12:33 PM
I'm getting intermittent transfer speeds switching around regularly (regular intermittencey ... nice).

if I extract a ty using tytool i typically get just under 500kBps for the duration of the transfer. I've got dumeter showing a graphical display of the transfer ... bit of a wibbly top of the graph but pretty much constant.

when i fire up an mfs_ftp tmf transfer with smartftp on my winxp pro (sp1) box i get a nice 600-700kBps extraction for a few seconds ... then the graph drops to 350kBps ... then down to 200kBps and quickly back up to 600Kbps ... and so on (looks like a load of churchs scrolling past my dumeter display ! :) ). net effect is my transfer runs at around 350kBps average ... but possibly more importantly usually dies after a minute or two. smartftp sits waiting, dumeter shows no traffic, then smartftp shows a message that the connection timed out and was closed by the server (tivo). I've tried switching passive off & on without any joy.

the full connection path is UK SA1 tivo with a turbonet to a D-Link 900+ wireless access point, D-Link 520+ PCI card in my PC ... i get consistant tytool speed, wonder what could be mocking my mfs_ftp stuff :(

did i forget ... thanks for a fantastic tool !!!

Duncan

SteveT
11-18-2003, 02:12 PM
Originally posted by SirTrini
In the last 4 or 5 iterations of MFS_FTP, those commands now show up as txt files and when I double click on them nothing happens.

I use SmartFTP to my HDVR2 running 1.2.9H, and double-clicking on the shutdown.txt shuts down the mfs_ftp server-side program (daemon?) as it should.
I see a message in the "log" window, but my "connection" window doesn't visibly change. This is identical (IIRC) with the way it worked with the shutdown "folder".

When you say "nothing" happens, are you saying you're still able to download shows, etc.?

BAHitman
11-19-2003, 12:43 AM
In regards to the shutdown and phoenix beeing a text file. Some FTP programs won't automatically do anything with the files like they will a folder. for example, in SmartFTP, I have to right-click the text file--say shutdown.txt, and click View. That will take care of it...

TriscuiT
11-19-2003, 08:01 PM
for example, in SmartFTP, I have to right-click the text file--say shutdown.txt, and click View. That will take care of it...

I only have to double-left-click on "shutdown.txt" in SmartFTP to shut down MFS_FTP. Perhaps it is dependent on how you have the "double-left-click" configured in SmartFTP?

TriscuiT
11-19-2003, 08:05 PM
I am currently transfering an NFL game from one DTivo to another DTivo via FXP (it will be some time before it is finished). The source unit is running 1.2.9G and the destination unit is running 1.2.9H. (I didn't feel like upgrading the source machine, and the destination machine is a new install). I received the following warning/error:



07:02:43:PM - 150 Opening BINARY mode data connection for "{{Chargers @ Broncos
San Diego Chargers at Denver Broncos}{1970-01-01}{San Diego Chargers at Denver B
roncos}{04.00 PM Sun Nov 16, 2003}{NFL}.tmf}"
importing fsid 57084 of size 536870912
<174>Nov 20 00:07:34 TmkTclEvent::QEvent[2030]: Event Q reached max size, events
have been lost
100% 512 meg 840k/sec

inserted 512 meg at 840k/sec
importing fsid 57088 of size 536870912
21% 112 meg 877k/sec


I've seen this error before but the transfer seemed to be ok. But I thought I would ask specifically, what does "TmkTclEvent::QEvent[2030]: Event Q reached max size, events have been lost" mean?

rc3105
11-20-2003, 12:30 AM
the tcl event loop can trigger procedures whenever a variable changes, new socket is opened, channel becomes readable/writeable, etc

the import & export procedures don't return control to the event loop durning transfers, events aren't processed & eventually you get "<174>Nov 20 00:07:34 TmkTclEvent::QEvent[2030]: Event Q reached max size, events have been lost" or some such

nothing to worry 'bout - only affects mfs_ftp which isn't multithreaded yet anyway

TriscuiT
11-20-2003, 12:56 AM
Roger that! Muchas Gracias!

sanderton
11-21-2003, 11:54 AM
haven't seen this mentioned, so I though't I'd point it out.

mfs_ftp doesn't like dealing with programmes where a power outage or reboot has split a programme into two identically named files. If you try to copy both segments, you get the first one twice!

Renaming one in TiVoWeb and restarting (so the cache is bebuilt) solves.

rc3105
11-21-2003, 03:57 PM
if you use filename option 1 the recobj fsid becomes part of the name. allows for easy identification of split recording segments or copies of a recording inserted multiple times ( handy if you're experimenting with editing or descrambling ;) )

Zanzibar
11-21-2003, 04:28 PM
Originally posted by rc3105
ty & tmf from mfs_ftp contain the same xml info, in tmf it's at the beginning of the archive, in ty it's at the tail end


Is there an easy way to convert from ty+ to tmf? I ask because I notice that Xbox media player will play .ty and .tmf, but not .ty+. I have downloaded many recordings as ty+, and was wondering how to make them play in XBMP.

I guess if mfs_ftp downloads .ty+ and .ty identically, then I could theoretically just rename the recordings from .ty+ to .ty. My question in doing this is, will I lose the ability to re-insert the recordings with all pertinent XML info at a later time?

Thanks :)

rc3105
11-21-2003, 05:21 PM
simply rename the ty+ to ty or (MUCH easier) add ty+ to the list of recognized video extensions in xbmp's config.xml

mfs_ftp will insert ty from mfs_stream, mfs_export, the tivoweb mfsstream module or vplay and mplayer's play-to-disk option (without embedded xml the recording will default to the name of the tyfile)

wkozun
11-23-2003, 01:05 PM
I recorded a show that stopped recording at 6:00. At 6:10 I went to copy the show to my other TiVo using mfs_ftp, but under all of the directories(tmf, ty, ty+, etc.) in my FTP client (SmartFTP) it showed that the show was still recording and that the file size was 512MB - although it should have been about 800M. I tried phoenixing mfs_ftp but that didn't change anything. Looking at Now Playing it did not indicate that the show was still recording. Finally at about 6:15 - a full 15 minutes after the recording stopped was the directory listing shown correctly.

Has anyone else seen this?

rc3105
11-23-2003, 07:59 PM
my lil bro used to get that all the time, turns out he had directory caching enabled in his ftp client ;)

danhi
11-23-2003, 10:25 PM
First let me apologize for posting this twice, once at the top level and once here in the thread. I meant to just post it in this thread, but screwed up. Oh well, here it is again:

I'm running a S1 with two Maxtor 80 gig hard drives. I've read through all 29 pages of this thread and have tried the following:

ithrottle = 0
insert_priority = 50
most current copy of mfs_stdinsert

[None of these seemed to make any difference]

runideturbo=false [this helped some]

hdparms -m16 /dev/hda [this helped the most]
[I tried -c1 -S0, but that didn't seem to make a difference]

The listing for my drive is:
/dev/hda:
Model=Maxtor 98196H8, FwRev=ZAH814Y0, SerialNo=V803N7XC
Config={ Fixed }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=57
BuffType=3(DualPortCache), BuffSize=2048kB, MaxMultSect=16, MultSect=16
DblWordIO=no, maxPIO=2(fast), DMA=yes, maxDMA=0(slow)
CurCHS=16383/16/63, CurSects=-66060037, LBA=yes, LBAsects=160086528
tDMA={min:120,rec:120}, DMA modes: mword0 mword1 *mword2
IORDY=on/off, tPIO={min:120,w/IORDY:120}, PIO modes: mode3 mode4

I get 1.5 MB/sec upload that's prett consistent, but on download I average less than 600 KB/sec and it's spiky, running mainly around 600KB, but occasionally spiking up to 850KB and down to 350KB.

I think I've tried everything that's been mentioned in this thread. If there's anything else left to try, I'd appreciate the tip.

rc3105
11-23-2003, 11:13 PM
the ithrottle & insert_priority settings are in the first few lines of mfs_ftp.tcl, they're ALSO in the file settings.tcl

the defaults in mfs_ftp will be overridden by settings.tcl if it's present. so make sure that's where you're adjusting them and that you phoenix or shutdow/restart mfs_ftp so they take effect

dlang
11-27-2003, 05:23 AM
I just attempted to upgrade to 1.2.9j (from the ancient 1.2.5x or thereabouts) but after every transfer the control connection was terminated, it looks like mfs_ftp restarts (if it's a fast command I can try to reconnect and get a 'nothing listening on that port' error)

I see the following message in the logs

catch close lastsock val "can't read "info(lastsock)": no such element in array"
12:21:05:AM - 226 Transfer complete.
12:21:05:AM - 200 shutting down the server NOW from "ac_intv_bail"

rc3105
11-27-2003, 06:30 AM
you have multithreading enabled with "set info(multithreaded) 1" in mfs_ftp.tcl or settings.tcl (k is the current ver btw)

when a transfer starts it closes the control socket to new clients & spawns a copy on the same port to service new requests - when the transfer finishes the old copy terminates & closes the control connection

that's usefull for multiple transfers or browsing/housekeeping during a transfer

dlang
11-27-2003, 07:32 AM
I am still getting the following message on startup

catch close lastsock val "can't read "info(lastsock)": no such element in array"

it no longer produces this message between fetches, thanks.