Mfs_Ftp: extract, archive, restore & transfer recordings

[stumbleshoe]

Originally posted by stumbleshoe

OK, I'll try that tomorrow - just got home from work and I'm beat!
Thanks!

OK, now I'm really stumped. Turned on the extra debugging, re-tried the insert with the same tmf and it worked. No problems whatsoever.

So, does this sound like a timing issue of some sort ? I.e. dugging slows things down enough to keep it working ? I had previously tried throttling the SmartFTP transfer, but even at an extremely low setting the transfer would still fail.

Other than the need to rotate the log file, any harm in leaving this on ?

[rc3105]

debugging doesn't change functions or timing, just logs what happened, when, & with what params. it's not quite upto black box specs yet but that's the general idea


there are 3 types of errors

retrytransaction - the db was busy so it retries until it works

fsstream create failed - there wasn't any free space in mfs

tmk assert - usually ran out of memory or unknown. causes a reboot


1 is normal, don't sweat it. if you get type 2 or 3 & it's repeatable I'd like to see a copy of a level 3 log along with info about sys config. (zip & attach to a post here or e-mail is good)


cmd line ftp issues

the filename is

{Stargate SG-1}{2000-02-25}{Maternal Instinct}{07-14-2003 07.00 PM}{SCFI}.tmf

not

06-18-03 09:00PM 2235564032 {Stargate SG-1}{2000-02-25}{Maternal Instinct}{07-14-2003 07.00 PM}{SCFI}.tmf

so

get "{Stargate SG-1}{2000-02-25}{Maternal Instinct}{07-14-2003 07.00 PM}{SCFI}.tmf"

works here with cmd line ftp from win98 / RH 7.3 / OS-X

[TroyCarpenter]

Oh my goodness. I just realized that after 15 years of using ftp and the internet tools, I'm an *****!

Thank you for letting me show my ignorance to the entire group. I don't know what I was thinking. Yes I do, I thought that the timestamp information was part of the filename. Duh.

Can I claim that it was late and just didn't have my senses around me (I've seen others use that excuse before )

Again, thanks for the pointer...I just wish I had a real challenge for you...then again, these types of problems are a lot less work.

Oh, and if noboby else has mentioned this yet...this is a great tool!

[deebo]

I just installed 1.2.8a and it works great here with IE6 and smart FTP. I am able to drag and drop to and from my tivo using IE.

Being able to just drag and drop from IE will make things MUCH simpler !

Thanks again Riley this is a GREAT program that just keeps on getting better!!!!!!

[osetivo]

Originally posted by rc3105
there were some varients of versions < 1.6 [SIC 1.2.6] posted to the thread a while back. as long as you extract/insert in tmf they should work fine.

Sorry, I know, that the whole TMF/TY/TY+/TYx file thing has been explained to death, and I REALLY try not to ask repetitive and/or stupid question, but....... (did you see that coming?) I want to make sure I'm OK. (from an extraction/insertion stand point) :-)

I have been extracting TMFs with the various versions of mfs_ftp.tcl, until just recently. I now extract in TY which is really TY+, i.e. it has the xml stuff attached to the end, right?

I have now converted all these old TMFs to TY with a recent version (?) of tmf2ty, which turns my TMFs into TYs (really TY+). So that my entire PC archive is now TYs (TY+) from various sources and versions. (though, there is no way for me to tell what's what anymore.) I have now deleted all of my old TMFs

I always use the latest mfs_ftp.tcl. Is it ok to reinsert ANY/ALL of my archived TYs, regardless of their origin, directly as a TY file (really a TY+) via mfs_ftp.tcl, or do I need to insert pre 1.2.6 tmfs as tmfs? or stated differently, will a TY (TY+) recently converted from a pre-1.2.6 tmf be OK.

The reason for asking is that I had a bad insertion the other day.
I know that historically my TMF extraction streams were OK, because I would regularly convert TMF to TY, via TYDETAR, to make DVDs (I kept the TMF so I didn't loose the xml for insertion), but I don't do many insertions, so when it went wrong, I started wondering if I properly understood the insertion stuff and tmf2ty.

Details of bad TY insertion:
I inserted a kid's show from a TY. I DO KNOW that the TY was created by tmf2ty, what version of TMF (1.2.5,6,7,8), I don't know. On playback "This Old House" jumped in for a few minutes in the middle of the show and then back to the kids show. (Not a pretty sight from a 3 year olds perspective) I don't think that TiVo has ever recorded "This Old House".

I checked, and the TY on the PC DOES have the interuption, so it's most likely that either the ORIGINAL TiVo recording was bad or it was the original extraction. I have never seen anything like this ON the TiVo. Have any of the old mfs_ftp extractions ever exihibited this problem.

Do you see any problems, or is this just an isolated recording problem?

Thanks

[rc3105]

a tmf is a tmf is a tmf. doesn't matter what version it was extracted with it'll insert just fine.

the minimum fsid "block" allocation size in a recording on the tivo is often a bit larger than the space used in the fsid. it's normal to get a few chunks of junk in a mfsstream or mfs_tarstream (tmf) extraction. these extra chunks are faithfully extracted & restored when using tmf's. since tivoapp put them there & uses the header chunk to determine where the real end of the fsid is it isn't a problem on playback.

never caused problems here, and I did LOTS of testing prior to releasing ty+ insertion because...

ty+ also extracts these extra chunks, but on restore it ONLY restores the chunks that are actually part of the recording. the junk chunks are discarded & you end up with different junk in the "cracks" between fsids (if the cracks exist at all, usually it's less than a meg at the tail end of the last fsid)

tmf -> ty (ty+) via tmf2ty are identical to current ty extracts & include these junk chunks if present. that junk is discarded on insert so you should never see it.


what was the original question again...


I would guess that that was just a random thing. I've never seen anything like that here. I can only speculate w/o looking at the file. if I had some ftp space I'd say upload it, if you've got space to host it please pm me a link.

the tmf format isn't changing, feel free to use it for long-term archiving to +/- r media w/o any worries.

the ty (ty+) format may get a few minor tweaks yet, but tmf's can be built from tmf2ty converted files or mfs_ftp ty extracts so it's not a problem either. if it does get tweaked that will be accomodated in the insert routines (allready is actually) so ty+ is ALSO good for archive to +/- r media w/o any worries.


--
Riley

[mavrcksd]

I have a question:

I did a manual record, and well just used the first channel in my lineup (ch 6). I have Direct TV feeding the tivo on video/a/v inputs.

On the DTV receiver, I just choose another source (input 1) via the A/V matrix built in. The input 1 is my VHS.

I dubbed a movie into the Tivo using A manual record, and it took the information about that lineup channel and put it in the database - all well and good.

However, How can I modify this information to make it really what I want it to be while it lives in the Tivo?

?????

Do I extract it, change it, and then re-import it? But the XML data is part of the file.. How can It be changed?

[rc3105]

Originally posted by mavrcksd
I have a question:

However, How can I modify this information to make it really what I want it to be while it lives in the Tivo?

?????

Do I extract it, change it, and then re-import it? But the XML data is part of the file.. How can It be changed?

easiest method is this:

shutdown mfs_ftp

delete the xml files in /var/mfs_ftp/cache via bash or tivoftpd

edit the recording information to whatever you want with tivoweb

re-start mfs_ftp


the cached xml file will be rebuilt from the new info as set by tivoweb. on extract that'll be what's included in the tmf or ty

[SR712]

Originally posted by rbiro
I would assume that opening up 2 ftp sessions on a PC and "dragging" a file from 1 session to another is still taking the long route through the PC.

Not really. If the FTP Client supports FXP, than it is not in the loop after the transfer initiates. SmartFTP does support multiple ftp sessions, and FXP.

[tivowatcher]

OK, I am completely stumped.

You'd think after hacking a bunch of TIVOs and successfully installing various tools (such as MFS_FTP) I could get this to work better.

What steps are you taking to *avoid* a passive transfer? I have 3 different clients here and they all max out around 288Kbs on tivo-to-tivo transfers.

Just tell the client you are using (such as SmartFTP 1.0) and the settings you use. I just can't figure this one out.

FYI:

I am using SVR-2000s with the latest 3.x software.
I have Turbonet cards from 9thtee.
Cabling and switch are both good.
Downloading to my PC rocks - over 1,000Kbs.
MFS_FTP is excellent and worked like a charm right "out of the box"

Thoughts?

[glider]

Riley, has anything besides adding the extra stuff at the end of the .ty+ file changed between the two versions ?

I only watch the .ty files using (the modified, latest) mplayer, and it seems like the files from the 1.2.8a version have more random segments in them than files extracted using the older version (which only had ty, right ?).

Seems to happen about once per half hour of recorded material - very annoying! Seems to be random material from other shows, but kind of hard to tell since its usually only for a couple of milliseconds..

I saw one other post that also seemed to have noticed something similar, has anyone else noticed the same behavior ?

[rc3105]

ty from mfs_ftp has allways had the xml appended to the end as do tmf2ty and tmf2mpg conversions. (tydetar tmf->ty conversions don't include the xml)

1.2.5.8 & 1.2.8a use the same mfs_stream procedure as tivoweb-mfsstream to extract ty. extracts from the same tivo with both mfs_ftp versions should be byte identical

the junk in the cracks on fsid boundries is normal but generally not a problem. it'd be interesting to know your drive config & the mfstool options used to partition the drive(s)

mfs_ftp & tmf2ty don't filter out the junk so that a ty is a 1:1 copy of the data from the tivo

tmf restores are currently 1:1 while ty inserts are automatically filtered. one of these days I'll post a filter utility and add that as an extract option


--
Riley

[glider]

the junk in the cracks on fsid boundries is normal but generally not a problem. it'd be interesting to know your drive config & the mfstool options used to partition the drive(s)

Sorry, its been so long I simply don't remember the options I used when creating my 2nd drive... mfstool didn't know how to backup data though (if that helps), which was added in later versions of mfstool.
The 1st drive came with the my tivo, and after blessing the 2nd drive they got joined by tivo - so I guess it might depend on which drive is used when saving that determines how much 'junk' is extracted ?

But yes - a filter utility (either built into mfs_ftp, or standalone) would be very handy!
Or I guess if mplayer the ty-enabled mplayer is further enhanced to avoid the junk ?

[osetivo]

Riley:

Decided it was time to redo the TiVo drive. In the Hinsdale how to the resore command is mfsrestore -s 127 -xzpi /yada yada

Is the "P" in the switch "-xzpi" the -p switch you referred to earlier to set the partitions to their factory settings?

Thanks

[Will]

Originally posted by rc3105
the junk in the cracks on fsid boundries is normal but generally not a problem.

mfs_ftp & tmf2ty don't filter out the junk so that a ty is a 1:1 copy of the data from the tivo

tmf restores are currently 1:1 while ty inserts are automatically filtered. one of these days I'll post a filter utility and add that as an extract option

Am I understanding the above correctly: tmf files that have been downloaded from a DTivo running mfs_ftp as a host have the junk at the boundaries? But you will at some point post a filter utility to clean existing files, and a future version of mfs_ftp will have the option to filter at the original download?

If my understanding is correct then the filtering of files previously downloaded to a Macintosh would have to be copied to a Wintel box to run the filter utility? If you could at some point discuss the logic of how such a filter would work, I could probably put a Mac filter utility together and I'd be glad to post it here.