PDA

View Full Version : TyExtract - Windows command line MFS_FTP batch extraction utility



JohnSorTivo
05-18-2004, 11:28 AM
Attached is a Windows command line MFS_FTP batch extraction utility that I wrote in order to automate the process of extracting multiple shows from your Tivo/Tivos via a single process, and I thought that I would share it with the group.

The utility is intended to be run from the Windows Scheduler, so that on a specified interval, it will automatically run and extract programs that contain your specified "keyword(s)" in their filenames, which have been recorded within a specific day range.

In this way, rather than having to connect to my Tivo to see what has been recorded, select the files, and then wait for them to download before I can then begin editing them, I will have my shows waiting for me, so that I can begin editing right away.

I built this using the WinInet libraries, and as such, your PC will require Internet Explorer 4.0 or greater, since the required DLLs accompany this application when installing, and obvioulsy, since TyExtract is an MFS_FTP command line utility, you need to be running the MFS_FTP server on your Tivo.

The attached zip file contains the executable, tyExtract.exe, a README.TXT file with detailed usage instructions, an example .bat file setup (example.bat), and an example of the resultant messages that get generated when the utility is run (tyExtract.log).

Version History:

tyExtract v1.0.0 - New Release
tyExtract v1.1.0 - Coding optimizations (no feature changes)
tyExtract v1.2.0 - Automatic logging via -g flag, * for -s flag for downloading ALL recordings within a range.
tyExtract v1.2.1 - Coding optimization (no feature changes)
tyExtract v1.2.2 - Output command line arguments to log file, Fix day range bug, Add -x flag to auto-delete recordings after extraction.
tyExtract v1.2.3 - Coding optimizations (no feature changes)
tyExtract v1.3.0 - Added ability to point to a to a text file, via the '-s' flag, containing a list of show name 'keywords' to extract upon.
tyExtract v1.4.0 - Format output filename using '-n' switch, skip extraction if file already exists in local directory, include 'shutdown.txt' to shutdown MFS_FTP server upon completion.

osetivo
05-18-2004, 11:46 AM
Can you tweak it to allow downloading by "thumbs up" rating in addition to the date.

JohnSorTivo
05-18-2004, 12:11 PM
Can you tweak it to allow downloading by "thumbs up" rating in addition to the date.
I'm limited to the information that is supplied to me via the MFS_FTP server, in terms of the 'show name' and 'file create date'. If MFS_FTP could be modified to include, as part of the 'show name' string, a thumbs rating value, then it could be done. However, this is not currently part of the standard 'show name' string that does get generated.

As basic 'show name' string looks something like:

{Seinfeld}{1996-10-17}{The Package}{11.00 PM Mon May 17, 2004}{CH32}.ty

So you can extract, using the "-s" flag on any of these sub-strings, but since "thumbs" is not a part of this string, there would be no way to know.

osetivo
05-18-2004, 03:27 PM
I've asked Riley about accessing the Thumbs Up in the mfs_ftp thread.

Hi8
05-18-2004, 09:42 PM
I've asked Riley about accessing the Thumbs Up in the mfs_ftp thread.

this is a great application! thanks..

would you consider allowing wildcards .. as in -s * so that it would transfer anything matching by date range switch. This way it would get everything, recorded.

can't wait to try it out!

JohnSorTivo
05-18-2004, 09:46 PM
this is a great application! thanks..

would you consider allowing wildcards .. as in -s * so that it would transfer anything matching by date range switch. This way it would get everything, recorded.

can't wait to try it out!

Great idea. In fact, I've already started working on it (wildcards) for the next version. Just makes sense.

In addition, I've begun integrating logging directly into the application, so that when passing another flag, '-g', you can enable screen messages to be written to a log file automatically.

Let me know how it works for you.

JohnSorTivo
05-19-2004, 12:05 PM
this is a great application! thanks..

would you consider allowing wildcards .. as in -s * so that it would transfer anything matching by date range switch. This way it would get everything, recorded.

can't wait to try it out!

Just posted a new version (v1.2) in the .zip file above, that has added wildcard support, i.e. '-s *', so that you can download ALL recordings made within the date range specified via the '-d' flag, regardless of their show name.

Also added automatic logging using the '-g' flag, so that screen messages will be automatically written/appended to tyExtract.log. This file will be located in the same directory that you have specified for your recordings to be saved, using the '-l' switch.

Hi8
05-19-2004, 08:07 PM
Just posted a new version (v1.2) in the .zip file above, that has added wildcard support, i.e. '-s *', so that you can download ALL recordings made within the date range specified via the '-d' flag, regardless of their show name.

Also added automatic logging using the '-g' flag, so that screen messages will be automatically written/appended to tyExtract.log. This file will be located in
the same directory that you have specified for your recordings to be saved, using the '-l' switch.


having trouble figuring out the 'day:day' switch. It seems to be off by 3 days... tried a few times with 0:0 0:6 finally after 3:3 it seemed to think that that was recordings for today. ??

here's a copy of the console results as the log doesn't show the commandline.

G:\_downloads\_TIVO\TyExtract\tyExtract.12>tyExtract.exe -h 192.168.1.111 -s * -
l d:\_tivo -d 3:3 -v -g
Running in verbose mode.

Attempting to establish connection on 05/19/04 at 19:46:35...
Connection established with '192.168.1.111' on port 3105

Matching Show: The Tonight Show With Jay Leno-2004.04.29-No Title.ty
Recorded on: Thu, April 29
Request range: Wed, May 19 to Wed, May 19
Sorry, recording does not fall within range

Matching Show: The Tonight Show With Jay Leno-2004.04.30-No Title.ty
Recorded on: Fri, April 30
Request range: Wed, May 19 to Wed, May 19
Sorry, recording does not fall within range



Matching Show: The Pathfinder-.15 AM Wed May 19, 2004-No Title.ty
Recorded on: Wed, May 19
Request range: Wed, May 19 to Wed, May 19
Show is within requested range
Downloading 2.1 GB

Hi8
05-19-2004, 08:57 PM
also tried to get the "tmf" instead of the "ty" and it still search for the "ty" files...



G:\_downloads\_TIVO\TyExtract\tyExtract.12>tyExtract.exe -t tmf -h 192.168.1.111 -s SHO -l d:\_tivo -d 5 -v -g
Running in verbose mode.

Attempting to establish connection on 05/19/04 at 20:23:27...
Connection established with '192.168.1.111' on port 3105

Matching Show: The Tonight Show With Jay Leno-2004.04.29-No Title.ty
Recorded on: Thu, April 29
Request range: Mon, May 17 to Mon, May 17
Sorry, recording does not fall within range

Matching Show: The Ellen DeGeneres Show-5.17-No Title.ty
Recorded on: Mon, May 17
Request range: Mon, May 17 to Mon, May 17
Show is within requested range
Downloading 888.0 MB



I also tried to get recordings with SHO hoping to get recordings from Showtime .. the tmf files have {SHO} in the file name can I put that complete string in? including the {} characters? are there limits or "special" character restrictions?

JohnSorTivo
05-19-2004, 10:32 PM
having trouble figuring out the 'day:day' switch. It seems to be off by 3 days... tried a few times with 0:0 0:6 finally after 3:3 it seemed to think that that was recordings for today. ??

here's a copy of the console results as the log doesn't show the commandline.

G:\_downloads\_TIVO\TyExtract\tyExtract.12>tyExtract.exe -h 192.168.1.111 -s * -
l d:\_tivo -d 3:3 -v -g
Running in verbose mode.

Attempting to establish connection on 05/19/04 at 19:46:35...
Connection established with '192.168.1.111' on port 3105

Matching Show: The Tonight Show With Jay Leno-2004.04.29-No Title.ty
Recorded on: Thu, April 29
Request range: Wed, May 19 to Wed, May 19
Sorry, recording does not fall within range

Matching Show: The Tonight Show With Jay Leno-2004.04.30-No Title.ty
Recorded on: Fri, April 30
Request range: Wed, May 19 to Wed, May 19
Sorry, recording does not fall within range



Matching Show: The Pathfinder-.15 AM Wed May 19, 2004-No Title.ty
Recorded on: Wed, May 19
Request range: Wed, May 19 to Wed, May 19
Show is within requested range
Downloading 2.1 GB



Thanks for the feedback.

Are the "Recorded On" dates displaying correctly for your shows, and just not the request range?

If so, what Timezone are you in? Normalizing the Tivo filesystem time with your current localtime can be tricky, and I am doing some timezone offsets, etc, in an attempt to compare apples to apples. Also, since I am using your system time, make sure your date and time on your PC is accurate.

I'll get to the bottom of it once I get a bit more feedback. Interesting that it appears to be off by exactly three days. You are correct in that 0:0 should be equivalent to today. If you don't enter anything for the '-d' flag, which is equivalent to 0:0, does the "Request Range" actually show as 3 days into the future?

JohnSorTivo
05-19-2004, 10:39 PM
also tried to get the "tmf" instead of the "ty" and it still search for the "ty" files...

That's just a bonehead move on my part. It actually IS downloading the "tmf" file. In attempting to "clean" the file name to something more condensed, I inadvertently hardcoded the extension to .ty, regardless of the "type" specified. The actual file is right, just not the extension ;-(

I'm going to remove the "cleaning" of the filenames all together for now, in order to get a revised version up ASAP. This will correct the extension naming error, and will write the output file exactly as it is provided by the MFS_FTP server. While verbose, at least it will be correct.

As far as what you can pass in via the "-s", show name flag, yes you can include the open and close braces, etc, in order to get more finite. You can essentially include any and everything that is actually part of the entire show name string, to the point that if you entered it exactly as it appears in your FTP client, you would end up making a request to download a single file. Just make sure to surround your name in double quotes if it includes any spaces, i.e:

-s "{Everybody Loves Raymond}"

Hi8
05-19-2004, 10:44 PM
Thanks for the feedback.

Are the "Recorded On" dates displaying correctly for your shows, and just not the request range?

If so, what Timezone are you in? Normalizing the Tivo filesystem time with your current localtime can be tricky, and I am doing some timezone offsets, etc, in an attempt to compare apples to apples. Also, since I am using your system time, make sure your date and time on your PC is accurate.

I'll get to the bottom of it once I get a bit more feedback. Interesting that it appears to be off by exactly three days. You are correct in that 0:0 should be equivalent to today. If you don't enter anything for the '-d' flag, which is equivalent to 0:0, does the "Request Range" actually show as 3 days into the future?


I'm in EST (GMT -5:00) and my PC is in sync via timeserver -

and yes the recorded on time is acurate, if you notice that the last recordind matches and is downloading ...


and here is a dump I just made .. with -d 0:0 show the future dates...


Running in verbose mode.

Attempting to establish connection on 05/19/04 at 22:33:17...
Connection established with '192.168.1.111' on port 3105
Matching Show: The Pathfinder-.15 AM Wed May 19, 2004-No Title.ty

Recorded on: Wed, May 19
Request range: Sat, May 22 to Sat, May 22
Sorry, recording does not fall within range

Matching Show: The Ellen DeGeneres Show-5.19-No Title.ty
Recorded on: Wed, May 19
Request range: Sat, May 22 to Sat, May 22
Sorry, recording does not fall within range

*** No shows with '*' found in your Now Playing List,
recorded between Sat, May 22 and Sat, May 22. ***

JohnSorTivo
05-19-2004, 10:56 PM
Okay. I'll have to dig into my time calculations to see what is happening. I've only to date used the program on my machine, which is CST, and haven't had any problems, so I'll have start doing some debugging.

In the meantime, I've posted another update, v1.2.1, to fix the file naming convention error. Please give it a try with the various extensions for the "-t" flag and let me know if you now get what you expect.

Again, the resultant filenames will be much larger, as they should be a direct pass-through of what MFS_FTP provides, rather than me trying to "pretty them up".

Hi8
05-19-2004, 10:58 PM
I have an S1 DTIVO mfs_ftp running on port 24 I set the -p 24 but would never get any recordings, even thought the log sdaid it connected. Tried serveral times and dates.. nothing. I then went in and killed MFS_FTP and restarted on the default port 3105, everything worked after that, still had the 3 day issue but it transfered those files that matched. That was the only change I made, I took out the -p 24 and restarted mfs_ftp on default ---

JohnSorTivo
05-20-2004, 10:16 AM
I have an S1 DTIVO mfs_ftp running on port 24 I set the -p 24 but would never get any recordings, even thought the log sdaid it connected. Tried serveral times and dates.. nothing. I then went in and killed MFS_FTP and restarted on the default port 3105, everything worked after that, still had the 3 day issue but it transfered those files that matched. That was the only change I made, I took out the -p 24 and restarted mfs_ftp on default ---

Check the contents of your port.24.log in your MFS_FTP directory for errors. I've got the port stuff working pretty well. I shutdown my MFS_FTP server and started it back up on port 24 to see what happens, and it fails as well, and I see the following in my log:



catch close lastsock val ""
01:51:08:AM - 220 Mfs_Ftp ver 1.2.9p - {sock22} from "192.168.1.2:4170"
01:51:28:AM - 331 User name okay, need password.
01:51:28:AM - 230 Running in TiVo Mode.
01:51:28:AM - 250 Directory change successful.
01:51:28:AM - 200 Type set to I
bgerror invoked with error

" couldn't open socket: address already in use "

re-initializing mfs_ftp

close the current ftp connection and simply open another


You can see that tyExtract is able to connect, but then is kicked out by the server with a socket issue. The same exact thing happens to me when attempting to connect using FlashFXP and smartFTP on port 24, so it appears to be an error while attempting to use this specific port (24).

Do you see the same errors in your log file?

I shutdown and ran on various other random ports (high numbers), such as 2000, 2222, 3333, and all worked perfectly. Try starting up on some of these ports, and let me know your findings.

Other items: I'm going to dig into the code today to see if I can determine what is going on with the "day range" being messed up for you. I've even reset my computer to your timezone, and still can't get the error to repeat.

Also, I've added to the top of the log file writing, the command line arguments, so that, for debugging purposes, it will first show the commands you passed in, followed by the resultant output. So in the next release, hopefully, in addition to the day range fix, this little change will also be incorporated.

Correct me if I am wrong, but the "day range" issue seems to be the last major outstanding issue, provided my last release resolved your issue with improper extension naming (always .ty), which I believe it should have.

Thanks for your help / feedback.

Hi8
05-20-2004, 10:44 AM
Check the contents of your port.24.log in your MFS_FTP directory for errors. I've got the port stuff working pretty well. I shutdown my MFS_FTP server and started it back up on port 24 to see what happens, and it fails as well, and I see the following in my log:



catch close lastsock val ""
01:51:08:AM - 220 Mfs_Ftp ver 1.2.9p - {sock22} from "192.168.1.2:4170"
01:51:28:AM - 331 User name okay, need password.
01:51:28:AM - 230 Running in TiVo Mode.
01:51:28:AM - 250 Directory change successful.
01:51:28:AM - 200 Type set to I
bgerror invoked with error

" couldn't open socket: address already in use "

re-initializing mfs_ftp

close the current ftp connection and simply open another


You can see that tyExtract is able to connect, but then is kicked out by the server with a socket issue. The same exact thing happens to me when attempting to connect using FlashFXP and smartFTP on port 24, so it appears to be an error while attempting to use this specific port (24).

Do you see the same errors in your log file?

I shutdown and ran on various other random ports (high numbers), such as 2000, 2222, 3333, and all worked perfectly. Try starting up on some of these ports, and let me know your findings.

Other items: I'm going to dig into the code today to see if I can determine what is going on with the "day range" being messed up for you. I've even reset my computer to your timezone, and still can't get the error to repeat.

Also, I've added to the top of the log file writing, the command line arguments, so that, for debugging purposes, it will first show the commands you passed in, followed by the resultant output. So in the next release, hopefully, in addition to the day range fix, this little change will also be incorporated.

Correct me if I am wrong, but the "day range" issue seems to be the last major outstanding issue, provided my last release resolved your issue with improper extension naming (always .ty), which I believe it should have.

Thanks for your help / feedback.

I'm just going to run mfs_ftp on port 3105... WAY BACK when .. it used to default on the FTP port hence a conflict, I never changed the startup rc.sysinit.author.

As far as port 24, I've been using it flawlessly with FlashFXP - for at least 5-6months.. ? like I said it's a non-issue, I just thought I would provide the feedback.

The day range is the biggest/only issue as far as I can see, I did download ver 1.2.1 with the -t[TYPE] switch fixed and that worked fine , I tried "ty , tmf, ty+" .

However on the day range... there seems to be something else there ... as I can't seem to do anything other than 3:3 I tried alot of different values, and it always reports that there were no recordings found, as opposed to checking each one in turn and saying no match. I will do somemore tests with the fix to the log file to report commandline switches, so you can see what I mean.

sanderton
05-20-2004, 11:00 AM
A guess on the port 24 thing, but have a vague recollection that mfs_ftp opens the PASV data port on a number one less than the control port. That would be port 23, which would already be open if you were telnetting to TiVo at the time, but not if you were just FTPing...

JohnSorTivo
05-20-2004, 11:17 AM
A guess on the port 24 thing, but have a vague recollection that mfs_ftp opens the data port on a number one less than the control port. That would be port 23, which would already be open if you were telnetting to TiVo at the time, but not if you were just FTPing...

Could be talking cr@p though.

Ahh! That makes sense. I was telneted in at the same time I was running tyExtract, which would explain the socket error.

So, it would appear that if you re-ran tyExtract with the -p 24 flag, and made sure everything was closed (telnet session), it would probably be successful.

Hi8
05-20-2004, 11:50 AM
Ahh! That makes sense. I was telneted in at the same time I was running tyExtract, whick would explain the socket error.

So, it would appear that if you re-ran tyExtract with the -p 24 flag, and made sure everything was closed (telnet session), it would probably be successful.

Ah.. I was telnet'd in at the time as well... as I said I will end up changing the port it starts on at some point.

I now have to fix my mfsstream for my S2 - as anytime I extract to burn to DVD-R I have been using tytool as a client/server method, which works fine. But now with the ability to auto-extract daily using TyExtract / MFS_FTP I will be using that from now on.

However I only seem to get the first FSID 512meg chunk - there was threads about that, and the need to get the correct mfsstream binary for the S2 because the one that Reily includes is flawed. ???

I downloaded what I was told was the right one, and replaced it, but still get incomplete recordings. What is strange is that the file size always seems correct.

I was suspisous between the two files, the on that comes packed with MFS_FTP is about 500-600k and the one that is suppose to fix the problem is only 100-200k I would have thought it was a minor patch, but for some reason the file sizes are completly different.

Anyway Thanks for you work on this! I was hoping jdiner would have it included in tytool by now, as he said that was on his list of features he intended to include.

The one thing I would like to see added is a switch to delete the recording on TiVo after it is transfered if possible.

JohnSorTivo
05-20-2004, 11:58 AM
The one thing I would like to see added is a switch to delete the recording on TiVo after it is transfered if possible.
Good call. I'll look to incorporate this. Shouldn't be too difficult.

I'm going to put together a quick program to print out some time value settings from your computer. If you could run it, and output the contents to a file, that would be great. It will help me to understand what may be happening. I'll PM you once I have it ready. Is this okay?

Hi8
05-20-2004, 12:01 PM
Good call. I'll look to incorporate this. Shouldn't be too difficult.

I'm going to put together a quick program to print out some time value settings from your computer. If you could run it, and output the contents to a file, that would be great. It will help me to understand what may be happening. I'll PM you once I have it ready. Is this okay?


sure ..
thanks ..

Hi8
05-20-2004, 12:08 PM
Good call. I'll look to incorporate this. Shouldn't be too difficult.

I'm going to put together a quick program to print out some time value settings from your computer. If you could run it, and output the contents to a file, that would be great. It will help me to understand what may be happening. I'll PM you once I have it ready. Is this okay?

I do recall when I was a beta tester for another TiVo app, (TivoMovies ) one of the problems was time/date (zone) and a modified version of a script that was included I think with MFS_FTP tzoffset.tcl was involved. It played hell around the daylight savings time switch a few weeks back. It was a major issue trying to reconcile the realtime on the PC side application and the TiVo.

JohnSorTivo
05-22-2004, 02:25 PM
Just posted a new version (v1.2.3) in the .zip file attached to the top of this thread.

Special thanks to Hi8 for his assistance in testing some of the code on his system, and for running some debugging scripts in order to optimize the code.

Included in this version is a fix for the 'day range' bug that may have improperly calculated the requested day range for downloading recordings, passed in via the '-d' flag. This bug would only have surfaced when run on certain computers within specific timezones.

Added the '-x' flag option, which, when included, will automatically delete any recordings from your Tivo, if, and only if, they have been successfully extracted.

Hi8
05-22-2004, 08:08 PM
Just posted a new version (v1.2.3) in the .zip file attached to the top of this thread.

Special thanks to Hi8 for his assistance in testing some of the code on his system, and for running some debugging scripts in order to optimize the code.

Included in this version is a fix for the 'day range' bug that may have improperly calculated the requested day range for downloading recordings, passed in via the '-d' flag. This bug would only have surfaced when run on certain computers within specific timezones.

Added the '-x' flag option, which, when included, will automatically delete any recordings from your Tivo, if, and only if, they have been successfully extracted.


John;

thanks for all your hard work on this, it is truely one of the BEST utilities I have now, paired up with MFS_FTP it really Rocks!

I have successfully test on :

2 sony svr-2000's S1 SA
1 sony sat T-60 S1 DTIVO
1 hughes sd-dvr-40 S2 DTIVO

the only thing I noticed was that I had to disable sanderton's P1.tcl script, on my S1's where it was running; other than that it's PERFECT!

JohnSorTivo
05-24-2004, 11:46 AM
John;
thanks for all your hard work on this, it is truely one of the BEST utilities I have now, paired up with MFS_FTP it really Rocks!!
Thanks for the feedback. I'm glad you are able to benefit from this little utility as much as I have.

FYI, another small feature I'm working on (added over the weekend and am testing before posting), is the ability to point to an external file with multiple show names keywords listed within the file for extracting, rather than passing a single show name keyword in via the '-s' flag.

In this way, you won't have to run the utility each time, for each show name keyword you wish to automatically extract for, but rather, the program will automatically loop through your list.

You can then point to the file by still using the '-s' flag, by using a special character (!) at the start of the string, which will instruct the program to look for the filename that follows the !, ie. '-s !shows.txt'.

Hi8
05-24-2004, 12:08 PM
Thanks for the feedback. I'm glad you are able to benefit from this little utility as much as I have.

FYI, another small feature I'm working on (added over the weekend and am testing before posting), is the ability to point to an external file with multiple show names keywords listed within the file for extracting, rather than passing a single show name keyword in via the '-s' flag.

In this way, you won't have to run the utility each time, for each show name keyword you wish to automatically extract for, but rather, the program will automatically loop through your list.

You can then point to the file by still using the '-s' flag, by using a special character (!) at the start of the string, which will instruct the program to look for the filename that follows the !, ie. '-s !shows.txt'.

I was hoping for a multi-string solution!

I like to use the channel label as my string.... ie {HBO {SHO {TMC

this way it should grab all my premium channel movies/shows I recorded.

EXCELLENT!

JohnSorTivo
05-25-2004, 02:49 PM
Just posted a new version (v1.3.0) in the .zip file attached to the top of this thread.

Added to this version is the ability to create and point to a custom text file containing a list of show name keywords to extract upon. In this manner, you won't have to run individual instances of tyExtract for each show name keyword you wish to extract upon, but rather, just point tyExtract to your text file which contains all of your keywords at once.

Create and name your text file anything you wish, such as 'shows.txt', and, to instruct tyExtract to use your external list, use the '-s' flag, and append an exclamation to the front, '-s !shows.txt'. The exclamation point tells tyExtract not to treat your input as a 'keyword', but rather, to look for a external keyword list file.

tyExtract will look for this file in the same directory that you have specified for your shows to be extracted to, using the '-l' flag.

Your text file should simply contain, on an indivudial line, each show name keyword that should qualify as a match. Unlike providing input via the command line for the '-s' flag, do not surround multiple keywords on a single line with quotes. They are not needed when using an external list / file.

Hi8
05-25-2004, 06:23 PM
Just posted a new version (v1.3.0) in the .zip file attached to the top of this thread.

Added to this version is the ability to create and point to a custom text file containing a list of show name keywords to extract upon. In this manner, you won't have to run individual instances of tyExtract for each show name keyword you wish to extract upon, but rather, just point tyExtract to your text file which contains all of your keywords at once.

Create and name your text file anything you wish, such as 'shows.txt', and, to instruct tyExtract to use your external list, use the '-s' flag, and append an exclamation to the front, '-s !shows.txt'. The exclamation point tells tyExtract not to treat your input as a 'keyword', but rather, to look for a external keyword list file.

tyExtract will look for this file in the same directory that you have specified for your shows to be extracted to, using the '-l' flag.

Your text file should simply contain, on an indivudial line, each show name keyword that should qualify as a match. Unlike providing input via the command line for the '-s' flag, do not surround multiple keywords on a single line with quotes. They are not needed when using an external list / file.


John;

so the syntax for this file is all on the same line just separated by a space? not a keyword per line...

ie...

{SHO {HBO {TMC {WVIT {CNN

that would be a valid file?

JohnSorTivo
05-25-2004, 07:24 PM
John;

so the syntax for this file is all on the same line just separated by a space? not a keyword per line...

ie...

{SHO {HBO {TMC {WVIT {CNN

that would be a valid file?
Sorry if my instructions were confusing. Each entry should be placed on a separate line, i.e:

{SHO
{HBO
{TMC
{WVIT
{CNN

What I was referring to in my instructions is when you have a single show, with multiple keywords, place it on a single line, without quotes, i.e:

Everybody Loves Raymond
The Screen Savers

Etc..etc., and not like:

"Everybody Loves Raymond"
"The Screen Savers"

A combination of your list and my list would look like:

{SHO
{HBO
{TMC
{WVIT
{CNN
Everybody Loves Raymond
The Screen Savers

Hope this is clear ;-)

JohnSorTivo
05-28-2004, 10:02 AM
After using tyExtract, if any of you have any more ideas / suggestions, etc, for enhancements, let me know. I'd be glad to consider them for implementation.

Hi8
05-29-2004, 06:54 AM
After using tyExtract, if any of you have any more ideas / suggestions, etc, for enhancements, let me know. I'd be glad to consider them for implementation.

John;

I'd love to see the ability to select naming verbose-ness via commandline switch.

for example..
{Chris Isaak Show}{2001-06-11}{Lost & Found}{11.00 PM Fri May 28, 2004}{SHO}.ty

say using a "-n 13" would return 'Chris Isaak Show-Lost & Found.ty'

the 13 referring to the field position so if you entered "-n 132" it would return 'Chris Isaak Show-Lost & Found-2001-06-11.ty'

obviously using any naming flag would strip the {} braces and insert a hyphen between fields instead.

I know you did a form of renaming in prior releases, I was just thinking it would be nice to have a personal choice depending on purpose.


because of the now 'fixed' XBMC .ty streaming I now have the need to push most if not all of my tyextracts to a specific folder that is shared to my xbox and is pushed directly to my xbox HD via another autobatched FTP scheduled task.


Thanks for a GREAT APP!

grahamk
05-31-2004, 02:37 PM
Thanks for the program - I have played with it a bit and it is going to be very useful. I really appreciate your work.

Since you asked for enhancement ideas I have a couple. The one that can probably be implemented would be an option that checks to see if a file already exists in the output path, and if it does it skips downloading it.

The other would be to automatically download anything that is marked "Save Until I Delete", but from previous posts in the thread it doesn't look like mfs_ftp gives you that information.

Thanks again for the program - very cool tool. :-)

Graham

rward765
06-05-2004, 03:16 AM
John,

Here's what I am trying to do. I use Microsoft scheduler running a modified batch file to pull the shows off each day that I want. The purpose of this batch file is to connect to the tivo via ssh (running lsh on tivo), start mfs_ftp, then download the shows that I want to capture, and then stop mfs_ftp. I would like to make TyExtract download shutdown.txt at the end of my shows.txt file. Is this possible?

I know, why not pipe mfs_ftp through an lsh tunnel - performance drag on tivo was too great and the transfer rate slowed significantly (the tivo's network connection is wireless). Thanks for the great app!

rward

JohnSorTivo
06-07-2004, 03:21 PM
Thanks all for your feedback. I apologize for my delay in responding. Have been on vacation the past week...



I'd love to see the ability to select naming verbose-ness via commandline switch.

for example..
{Chris Isaak Show}{2001-06-11}{Lost & Found}{11.00 PM Fri May 28, 2004}{SHO}.ty

say using a "-n 13" would return 'Chris Isaak Show-Lost & Found.ty'

the 13 referring to the field position so if you entered "-n 132" it would return 'Chris Isaak Show-Lost & Found-2001-06-11.ty'

obviously using any naming flag would strip the {} braces and insert a hyphen between fields instead.

I know you did a form of renaming in prior releases, I was just thinking it would be nice to have a personal choice depending on purpose.

Yes, as you have mentioned, I had a "fixed / forced" version of this functionality in a previous version, that automatically used fields 1 & 3 for renaming the output file, but what I found was that many times, field 3 (episode title) would be blank, which I would fill in with 'No Title'. The end result would then be multiple episodes for the same
show all being assigned the same filename, and thus over-writing each other after each consecutive download. I therefore stripped this back out. I can consider looking at this again, but this would require maintaining some sort of array, to compare the filename I am attempting to download, to what was previously just downloaded, to make sure I increment a counter or something, in order generate a "unique" filename. Not only is there potential, in this case, to assign a recording the same filename for something I just downloaded, but at the same time, to something that already exists in your download directory as well.

Another option is that I could look at allowing you to pick a few fields, and then ALWAYS append the 'date' field, or something other field/value, to maintain uniqueness.



Since you asked for enhancement ideas I have a couple. The one that can probably be implemented would be an option that checks to see if a file already exists in the output path, and if it does it skips downloading it.

Good idea and easily implementable. This way, if you have multiple Tivos downloading shows, if a show has already been downloaded from the first Tivo, no reason to download again from the second Tivo.



The other would be to automatically download anything that is marked "Save Until I Delete", but from previous posts in the thread it doesn't look like mfs_ftp gives you that information.

Yes, as you have mentioned, since this information is not part of the data provided by the MFS_FTP server, there would be no way for me to know this, though it would be nice to have.



I would like to make TyExtract download shutdown.txt at the end of my shows.txt file. Is this possible?

Yes, this can be easily done. Currently, based on the '-t' flag you pass, I know which directory to change to, i.e. /ty or /tmf, etc, for retreiving recordings. What I can do is, if I see 'shutdown.txt' in your 'shows.txt' file, I can back up to the root directory and request the file, which will then shut down your MFS_FTP server. I'll also write it so that 'shutdown.txt' can be requested directly via the '-s' flag, rather than having to include it in your externally supplied list file, for those that don't use an external file as input. This would require executing tyExtract multiple times, one to connect and get the recording specified in the '-s' flag, then upon completion, tyExtract would have to re-connect, to request the 'shutdown.txt' file, so it won't be as efficient...

Thanks again for the feedback, and as soon as I get a chance, I'll start working on your ideas.

Hi8
06-07-2004, 05:08 PM
Another option is that I could look at allowing you to pick a few fields, and then ALWAYS append the 'date' field, or something other field/value, to maintain uniqueness.


John;

it would be nice to at lease have an option to truncate the file name abit.. like to the fields that are AWAYS there . Perhaps Show Title, and Date & strip {}braces?


This way, it would save

Chris Isaak Show-2001-07-02.ty

from ...

{Chris Isaak Show}{2001-07-02}{Behind the Isaak}{11.00 PM Sat Jun 05, 2004}{SHO}.ty

I find that my XBOX hates LONG filenames as well as squirely characters...

JohnSorTivo
06-09-2004, 06:00 PM
It would be nice to at lease have an option to truncate the file name abit.. like to the fields that are AWAYS there . Perhaps Show Title, and Date & strip {}braces?

This way, it would save

Chris Isaak Show-2001-07-02.ty

from ...

{Chris Isaak Show}{2001-07-02}{Behind the Isaak}{11.00 PM Sat Jun 05, 2004}{SHO}.ty


I hear you, and understand where you are coming from. I've actually got all of the new features implemented and am just testing before I post it. Hopefully tomorrow sometime.

The new features include:

1. Prevent downloading a recording when the same recording (filename) already exists in the output directory.
2. Include 'shutdown.txt' anywhere in your "shows" input file, or directly in the command line "-s" switch, which will cause MFS_FTP to shutdown upon completion.
3. Renaming recording / filename ouput, using "-n" switch, by including any combination of integers 1-5, i.e. -n 12, -n 123, -n 534, etc. Where each integer represents the "fieldname" value within the original recording name that should be included as part of the new output filename. Also, any braces "{}" will be replaced with dashes.

I will also include in the README that this feature should be used AT YOUR OWN RISK, as if you choose the wrong fields, all recordings could be given the same filename, and thus only the first will get downloaded. Only the first will get downloaded as a result of new feature #1 (see above). ALWAYS including "2" as part of your "-n" switch should help to ensure uniqueness and prevent this problem. The "-n" switch is optional, and therefore, passing nothing will simply output the filename exactly as provided by the MFS_FTP server.

Therefore, in your example, passing in "-n 12" will give you the file format output you are looking for, i.e:

{Chris Isaak Show}{2001-07-02}{Behind the Isaak}{11.00 PM Sat Jun 05, 2004}{SHO}.ty

Will become:

Chris Isaak Show-2001-07-02.ty

The other point to keep in mind, that my be tricky, or appear confusing, is that by re-naming a recording, it may not be immediately obvious as to why it was downloaded / qualified as a match, since the matching "keyword" may no longer be present in the name of the file saved, as a result of using the "-n" switch. However, if you review the logging file, you will be able to see what the "original" filename was, and thus, why it was downloaded.

Stay tuned...Hopefully I'll have it up shortly.

Hi8
06-10-2004, 06:25 AM
Stay tuned...Hopefully I'll have it up shortly.

WOW! that's Great! more than I hoped for... I can say it has been quietly working away everynight TyExtracting like crazy .... who needs to put it a Bigger HD in your Tivo?

Thanks!

JohnSorTivo
06-10-2004, 02:24 PM
Just posted a new version (v1.4.0) in the .zip file attached to the top of this thread. New features as described in my previous post, with details in the README.txt file.

Enjoy.

rward765
06-10-2004, 04:40 PM
It works perfectly. Nice job!

Do you think I should post my instructions for starting mfs_ftp via a putty command line in this thread? Along with the files I used for LSH on the Tivo?

rward765

JohnSorTivo
06-10-2004, 05:17 PM
It works perfectly. Nice job!

Do you think I should post my instructions for starting mfs_ftp via a putty command line in this thread? Along with the files I used for LSH on the Tivo?


You know, it probably wouldn't hurt for others to see how you've setup your .bat file, etc. Since tyExtract can now shut down the MFS_FTP server upon completion, it would be nice if the entire extraction "process" could start up, extract, and shutdown.

Adding start-up was something I was considering as well, but this is a much greater challenge, in terms of the protocols for establishing a connection to the Tivo, in order to accomplish, directly within tyExtract.

rward765
06-11-2004, 01:08 AM
Okay. For those of you who would like to have a way to securely start up the mfs_ftp server, then run tyExtract to get your shows and shutdown mfs_ftp on the tivo here you are. Before I get into that, first some background. My DTivo (RCA 40)is connected to my wireless network via a Netgear USB 802.11b adapter with all the usual hacks. Because of the inherent weakness in wireless security, I want to have as few ports open on the tivo as possible. My biggest issue of course was with telnet and Tivoweb. While both great tools, running them on a wireless tivo is not a good idea.

So my thoughts turned to tunnels via ssh. I read here (http://www.dealdatabase.com/forum/showthread.php?t=27950) that lsh was the way to go on the Tivo. I followed the instructions to install lsh and setup public/private key authentication. At that point, I setup a putty profile named mfs via ssh to my tivo, using my key, and the remote command "/bin/tivosh /var/mfs_ftp/mfs_ftp.tcl". I considered tunneling mfs_ftp via ssh but the additional overhead could bog down the tivo and certainly would make my wireless transfers even slower.

So my daily 3am batch file issues the following command:

putty -load mfs

then 2 minutes later:

tyExtract.exe -h tivo -s !shows.txt -t ty+ -l y:\ -p 60XXX -d 1:1 -v -g

Now that tyExtract works with shutdown.txt, I know that mfs_ftp will be down once my shows have been transfered. Also, I use a non standard port in the upper range for mfs_ftp - lots of times port scanner tools are used to scan default ports only so a wardriver might not catch that upper port even when its open.

I suppose if I ever wire my tivo, I would still use ssh. Unless I am using one of the hacks that I need such as mfs_ftp or tivoweb, the only open port on the tivo is 22. I just login and start whatever hack I need. Also I use sftp to transfer files rather than ftp which keeps another port locked down. It's much more secure for me.

rward765

Hi8
06-11-2004, 06:09 AM
John;

new version works GREAT... everything thing as advertised. The only think I did notice was that you are replacing the hyphens in the DATE field with periods??? I not sure yet if my XBOX FS will like that or not. But will keep you posted.

example:
"The Tonight Show With Jay Leno-2004.06.10.ty"

Thanks again for your response on adding these changes!

JohnSorTivo
06-11-2004, 10:14 AM
The only think I did notice was that you are replacing the hyphens in the DATE field with periods??? I not sure yet if my XBOX FS will like that or not. But will keep you posted.

example:
"The Tonight Show With Jay Leno-2004.06.10.ty"


You know, with so many things being updated, I forgot that I was actually doing this! The reason I swapped out the dashes with periods in field 2, was that, since I am using the dash "-" as a field delimiter, I didn't want there to be confusion between what delimited a field, and what was "within" a field. I.E. if I kept the dashes in field 2, it would appear as:

"The Tonight Show With Jay Leno-2004-06-10.ty"

Which I thought looked confusing when looking at the file name, especially if using many fields with the "-n" flag.

Let me know if it causes any problems.

malfunct
06-11-2004, 01:11 PM
You know, with so many things being updated, I forgot that I was actually doing this! The reason I swapped out the dashes with periods in field 2, was that, since I am using the dash "-" as a field delimiter, I didn't want there to be confusion between what delimited a field, and what was "within" a field.

The underscore (_) might be a safer character to use.

JohnSorTivo
06-11-2004, 09:33 PM
The underscore (_) might be a safer character to use.
I could certainly change to using this, but it would seem to be six of one, a half dozen of the other. The period is already used elsewhere in the recording name, prior to the file extension, specifically in field 4, for the "time" value.

JohnSorTivo
06-14-2004, 12:22 PM
The only think I did notice was that you are replacing the hyphens in the DATE field with periods??? I not sure yet if my XBOX FS will like that or not. But will keep you posted.

Just wondering if the periods have created a problem or not...

Thanks.

Hi8
06-18-2004, 05:34 PM
Just wondering if the periods have created a problem or not...

Thanks.

sorry for the delay ... off on a fishing trip up north .. BIG BASS! & Northeren Pike.

Still haven't tried it on my XBOX, but an underscore sounds safe to me... perhaps I'm just an old DOS guy, periods being a filename.ext separtator when the ole' UNIX guys see it as just another character.

I know that XBMC looks at extensions to determine proper codec for play, but perhaps they just look at the last 3-4 characters...??? I will let you know, later .. still have to unpack!

Hi8
09-09-2004, 02:15 PM
I Just wanted to POST a message to 'bump' this thread...

I still can't believe this app is not a popular topic of discussion! I NEVER use TyTool to extract my streams anymore... I have a batch file setup on a PC on my network, that extracts every new recording each night off 3 TiVos while I'm Sleeping!

I just have to go in from time to time and cull the list to those that I wan to keep for archiving - This has been something I thought jdiner was going to build into TyTool at some point, and I was waiting with baited breath.

Anyway GREAT Application!! Try it You'll Like It!

JohnSorTivo
09-11-2004, 02:54 PM
I Just wanted to POST a message to 'bump' this thread...

I still can't believe this app is not a popular topic of discussion! I NEVER use TyTool to extract my streams anymore... I have a batch file setup on a PC on my network, that extracts every new recording each night off 3 TiVos while I'm Sleeping!

I just have to go in from time to time and cull the list to those that I wan to keep for archiving - This has been something I thought jdiner was going to build into TyTool at some point, and I was waiting with baited breath.

Anyway GREAT Application!! Try it You'll Like It!
Glad it's working well for you. I think the best attribute of the program is that you setup / configure it once, and it just keeps on working, without any further interaction required, except, as you mention, the occasional update of your extraction list.

ericchile
10-06-2004, 03:37 PM
First let me say awesome work. I watch my late shows at work during off time, and instead of waking up 30 minutes early to extract them this has allowed me to sleep that time. (or be early to work).

But I have a problem. My half an hour shows work fine. But all my hour plus shows only get about 20 minutes. I have dsr7000 running 4.0. Tytool extraction works fine. I tried converting the ty files to mpeg to see if it was WMP that was just not reading the end of the file right, but the new mpeg only has ~ 20 min also.

Anyone experanced such?

Update: Replaced the binaries again for mfs_ftp and it appears the problem is gone. Thanks

JohnSorTivo
10-06-2004, 03:57 PM
But I have a problem. My half an hour shows work fine. But all my hour plus shows only get about 20 minutes. I have dsr7000 running 4.0. Tytool extraction works fine. I tried converting the ty files to mpeg to see if it was WMP that was just not reading the end of the file right, but the new mpeg only has ~ 20 min also.

Anyone experanced such?

This sounds more like a general overall problem with your version of MFS_FTP, as compared to the extraction client. You can confirm this by connecting to MFS_FTP using a standard FTP client and attempting to download an hour long show, to see if it too only brings down 20 minutes worth, which, it seems, should be the case.

If so, posting your question in the MFS_FTP forum will probably result in some better answers.

Also, I'd check your mfs_ftp logs on your tivo to see if they provide any clues as to what might be happening, port.3105.log, in your mfs_ftp directory.

Radams58
10-14-2004, 04:04 PM
Heck if I'd known about this I wouldn't have written my own app that does the same :)

Very nice job and cleaner than my own app.

I do have a couple questions for you.

In my app I've added a couple extra features you may want to include. (Not that your app isn't great, just extended usage)

XML File that contains paths for Exported Shows / Binary Files / Conversion settings.

Much like yours mine goes out and grabs shows from MFS_FTP based on show titles in an xml file. The XML file also stores the mux settings for vsplit and the conversion settings necessary for mencoder (a nifty command line converter) . Then when it grabs the show, it muxes it out to a vob and runs it through mencoder, deletes the temp files and puts the .avi file in a specified directory.

I'm so lazy I use this to archive everything onto my storage server and then play it back with the Xbox.

Let me know if you have any interest in it.

Rob

Hi8
10-15-2004, 01:11 PM
Heck if I'd known about this I wouldn't have written my own app that does the same :)

Very nice job and cleaner than my own app.

I do have a couple questions for you.

In my app I've added a couple extra features you may want to include. (Not that your app isn't great, just extended usage)

XML File that contains paths for Exported Shows / Binary Files / Conversion settings.

Much like yours mine goes out and grabs shows from MFS_FTP based on show titles in an xml file. The XML file also stores the mux settings for vsplit and the conversion settings necessary for mencoder (a nifty command line converter) . Then when it grabs the show, it muxes it out to a vob and runs it through mencoder, deletes the temp files and puts the .avi file in a specified directory.

I'm so lazy I use this to archive everything onto my storage server and then play it back with the Xbox.

Let me know if you have any interest in it.

Rob

I'd love to try it... I would use it for a different purpose than I use TyExtract - I like the .avi final format.

I have an modded XBOX installed as my CarPC - and like to move some of the comedy and music videos over to it from time to time. divx is a nice compact format that displays perfectly on my Liliput LCD screen.

tlphipps
10-16-2004, 12:33 AM
Heck if I'd known about this I wouldn't have written my own app that does the same :)

Very nice job and cleaner than my own app.

I do have a couple questions for you.

In my app I've added a couple extra features you may want to include. (Not that your app isn't great, just extended usage)

XML File that contains paths for Exported Shows / Binary Files / Conversion settings.

Much like yours mine goes out and grabs shows from MFS_FTP based on show titles in an xml file. The XML file also stores the mux settings for vsplit and the conversion settings necessary for mencoder (a nifty command line converter) . Then when it grabs the show, it muxes it out to a vob and runs it through mencoder, deletes the temp files and puts the .avi file in a specified directory.

I'm so lazy I use this to archive everything onto my storage server and then play it back with the Xbox.

Let me know if you have any interest in it.

Rob

I'd be very interested in this as well. My ultimate goal is to have a simple solution for transferring shows to my Treo600 Smartphone. The only trick would be getting these AVI files into a 160x160 resolution as that's the screen size on my phone.

Radams58
10-16-2004, 12:43 AM
I will post my ugly source code, config files and db files...

it's not polished but it works.

On a side note, it's going to be a tommorrow as my var hack directory bit the big one last night and my rc.sysinit.author file was poorly configured.

one re-image later....

anyway I shall post tomorrow

as for the 160x160, I don't think that mencoder cares what res you want in the end

Rob

JohnSorTivo
10-16-2004, 03:12 PM
Very nice job and cleaner than my own app.

I do have a couple questions for you.

In my app I've added a couple extra features you may want to include. (Not that your app isn't great, just extended usage)

XML File that contains paths for Exported Shows / Binary Files / Conversion settings.

Much like yours mine goes out and grabs shows from MFS_FTP based on show titles in an xml file. The XML file also stores the mux settings for vsplit and the conversion settings necessary for mencoder (a nifty command line converter) . Then when it grabs the show, it muxes it out to a vob and runs it through mencoder, deletes the temp files and puts the .avi file in a specified directory.

I'm so lazy I use this to archive everything onto my storage server and then play it back with the Xbox.

Let me know if you have any interest in it.

Rob
Thanks for the complement.

If anyone is interested, I can certainly take a look at your process and adapt TyExtract to convert using the apps described. I don't do any conversion, so I don't know alot about vsplit & mencoder, but it shouldn't be too difficult to incorpate the necessary system calls to each of these executables once the extraction is complete.

Is your process multi-threaded, or is it linear, in that a file is extracted, muxed, and converted, before another file is extracted and so on?

Given a large list of files to extract, I would envision such a process taking quite some time to complete.

John

Radams58
10-17-2004, 02:28 AM
It's linear via batch...

that is, it makes up a list of all the things it needs to do for each steap

1)Get list of shows
2)Compare against master lsit
3)Build list of shows to export
4)Export All Shows
5)Mux All Shows
6)Convert all shows


Anyway you sllice it, it does take some time to run :)

and on top of it I'm getting choppy results from mencoder, so I just swapped the conversion to VitrualDubMod.... which means I need to have the system build a script for each conversion job.

Not horrible, but interesting.

I think the way I'm heading it's going to be too involved for a casual user.

I'll have more after my Chiefs whip the snot out of the Jags.

Rob

ddsdave
10-27-2004, 11:05 AM
For the past year, I've been on a quest to archive every movie that my two DTivo's record. I actually watch Directv's programming on two Ultimate TV's which "she who must be obeyed" does not want to retire and transition to Tivoland. The two Series 2 Tivo's were hacked with the sleeper ISO and prior to August 8th were using TyTools to "harvest" movies from the two DTivo's to multiple networked PC's. I then eliminate duplicates, compare sizes for potential transmission problems and then write two to three movies on a DVD in .ty format. This approach allows me to "on demand", re-insert a movie into one of the DTivo's or write a DVD playable on a standard DVD player using TyTools.

In August, I implemented TyExtract as an XP scheduled task--my only complaint was the temporary files it "left behind", so I also scheduled a "Clean" task to run every four hours and that solved the problem. At that point I had about 1,500 movies on 600 DVD's.

I left the -t parm to default to .ty files, but looking back, I never had the need to re-insert or use TyTools to generate a playable DVD.

In late September, I purchased and hacked a Stand Alone Series 2 Tivo and was planning to populate it with movies for my October Vacation—when I attempted to re-insert in the Ty+ folder, I found only the first 512MB were available for play on the Tivo. When I tried to create a standard DVD using TyTools, the GOP Editor crashed. I went through a several week “panic” rebuilding systems, and checking the forum which led me to believe I had some “bad” MFS-FTP code. I finally took a .ty file prior to August 8th and it inserted perfectly and worked perfectly on Tytools. I was able to “load-up” one of the DTivo’s with movies, ship it out, and it worked perfectly during my vacation.

I’m now back to the “real world” and am about to delete 350 movies that appear to be the right size, but apparently are not really in the .ty format—any thoughts would be appreciated?

ddsdave

JohnSorTivo
10-27-2004, 11:45 AM
In August, I implemented TyExtract as an XP scheduled task--my only complaint was the temporary files it "left behind", so I also scheduled a "Clean" task to run every four hours and that solved the problem.
What temporary files are you referring to? The only output TyExtract creates, by default, is the actual recording, unless you tell it to create a log file, via the -g flag.


I left the -t parm to default to .ty files, but looking back, I never had the need to re-insert or use TyTools to generate a playable DVD.

In late September, I purchased and hacked a Stand Alone Series 2 Tivo and was planning to populate it with movies for my October Vacation—when I attempted to re-insert in the Ty+ folder, I found only the first 512MB were available for play on the Tivo.
By leaving the -t flag blank, the files being extracted are .ty, not .ty+, so it's not clear to me why you would be re-inserting them into the .ty+ folder. Also, as you may already know, the best "format" for re-insertion is .tmf or .ty+, as compared to .ty, as the XML recording "info" is retained.


I finally took a .ty file prior to August 8th and it inserted perfectly and worked perfectly on Tytools. I was able to “load-up” one of the DTivo’s with movies, ship it out, and it worked perfectly during my vacation.
Have you tried extracting a .ty file directly through a standard FTP client, post August 8th, and then re-inserting to see what happens? This will help determine where the problem exists, i.e. on the mfs_ftp server versus the mfs_ftp client side.

Again, though, what is most glaring to me is extracting a .ty file and re-inserting into the .ty+ folder.

I have been extracting and re-inserting .tmf files from one tivo to another, using tyExtract, for quite some time, so I'm not sure what is going on in your setup.



I’m now back to the “real world” and am about to delete 350 movies that appear to be the right size, but apparently are not really in the .ty format—any thoughts would be appreciated?
As for the difference between a .ty file extracted using TyTool versus MFS_FTP, and the potential implications, this I cannot answer, and such a question would have to be directed to Jdiner & Riley.

Thanks.

John

rc3105
10-27-2004, 01:29 PM
I’m now back to the “real world” and am about to delete 350 movies that appear to be the right size, but apparently are not really in the .ty format—any thoughts would be appreciated?

ddsdave
sounds like maybe you have a bad mfs_stream (sleeper's ISO had several broken utils)

don't delete those 350 movies, do a lil reading in the MFS_FTP on Series 2 Units (http://www.dealdatabase.com/forum/showthread.php?s=&threadid=29459) thread for the correct binary and utils to fix bad extracts (a log message is added to the video file. it's not corrupted or anything, just needs the text removed)

*if you're still stuck in a week or two send me a pm

Karma
12-12-2004, 09:06 PM
Doing a temporary file cleanup on c: gets rid of these files. Not exactly sure where/what they are. Files seem to be the same size as the .ty files, but are not found when doing a search (even for hidden files). Running XP SP2. I noticed this when downloading to D: and my C: drive started filing up too.

JohnSorTivo
12-13-2004, 01:08 PM
Doing a temporary file cleanup on c: gets rid of these files. Not exactly sure where/what they are. Files seem to be the same size as the .ty files, but are not found when doing a search (even for hidden files). Running XP SP2. I noticed this when downloading to D: and my C: drive started filing up too.tyExtract does not create any temporary files anywhere on your system. The only files tyExtract opens for writing are when you specify logging, via the -g switch, and the recording itself, both written directly to the output directory specified via the -l switch.

mmoore99
01-10-2005, 03:27 AM
I am attempting to use tyExtract on my three dtivos (2-hdvr2's, 1-sdvr80). Everything is working fine on the two hdvr2's. However, when I attempt to use tyExtract on the sdvr80 I get the following messages in the log:
tyExtract.exe -h tivo.study -s 24 -l F:\My Videos - F\Tivo Video Files\tyExtract -d 1 -v -g

Running in verbose mode.

Attempting to establish connection on 01/09/05 at 22:43:49...
Connection established with 'tivo.study' on port 3105


*** Sorry, Unable to locate any recordings on your Tivo ***All three units are running the same software, including the same version of mfs_ftp. Also, on the unit that tyExtract is having the problem with, I am able to connect with mfs_ftp via SmartFTP and upload/download recordings to the PC as well as from tivo to tivo. Any help would be appreciated.

JohnSorTivo
01-10-2005, 12:07 PM
What does the corresponding content of your 'port.3105.log' file say in your mfs_ftp directory as well?

The block of code that will return such a message usally occurs when there are 0 recordings present on your Tivo OR when changing to the directory specified, in this case your /ty directory, for retreiving a list of files (recordings) in this directory, and a timeout occurs while tyExtract waits for the first filename in your directory to be returned, so that it can then begin looping through your list, looking for matches.

Is the Tivo you are having trouble with on the same network segment (hub/switch) as the rest of your units? Do you notice any latency at all when connecting through MFS_FTP to this Tivo as compared to your 2 others units?

mrwalker66
01-10-2005, 01:03 PM
same problem...it might be timing out, the tivo is 800 miles away, i can see in the port3105 log that it is using the local ip on the network there, 192.168.10.3 that could also be the problem. maybe i need to open up another port on the router..i dont know..
bat file
tyExtract.exe -h mommativo -s !shows.txt -l g:\tytool9r18 -v -d 0:6 -g -x
tyextract log
tyExtract.exe -h mommativo -s !shows.txt -l g:\tytool9r18 -v -d 0 -g -x
Running in verbose mode.
Attempting to establish connection on 01/10/05 at 11:47:13...
Connection established with 'mommativo' on port 3105
*** Sorry, Unable to locate any recordings on your Tivo ***
port3105.log
04:27:45:PM - sourcing settings
10:27:47:AM - updating cached recording info
catch close lastsock val "can't read "info(lastsock)": no such element in array"
10:49:15:AM - 220 Mfs_Ftp ver 1.2.9p - {sock18} from "####:1434"
10:49:15:AM - 331 User name okay, need password.
10:49:16:AM - 230 Running in TiVo Mode.
10:49:16:AM - 250 Directory change successful.
10:49:16:AM - 200 Type set to I
10:49:16:AM - 227 Entering Passive Mode (192,168,10,3,12,32).
10:49:37:AM - 502 Command not implemented ""

mmoore99
01-10-2005, 02:38 PM
[QUOTE]I am attempting to use tyExtract on my three dtivos (2-hdvr2's, 1-sdvr80). Everything is working fine on the two hdvr2's. However, when I attempt to use tyExtract on the sdvr80 I get the following messages in the log:
tyExtract.exe -h tivo.study -s 24 -l F:\My Videos - F\Tivo Video Files\tyExtract -d 1 -v -g

Running in verbose mode.

Attempting to establish connection on 01/09/05 at 22:43:49...
Connection established with 'tivo.study' on port 3105


*** Sorry, Unable to locate any recordings on your Tivo ***Further update...I scheduled an unattended tyExtract to take place at 1:30am on one of the units that had been working properly. However, that attempt failed with the same error message as above which had been occurring on the other unit. This morning I am able to manually execute tyExtract successfully on all three units. So the problem is intermittent. I turned up the logging level for mfs_ftp and will provide the log info the next time tyExtract fails.

JohnSorTivo
01-10-2005, 02:39 PM
Okay. Working under the auspices that a timeout of some fashion is the likely culprit, I've compiled a test version to try, where I have attempted to increase some timeout flags for the send and receive of data between the client and server. These flags can be a bit flaky sometimes and may not be of any benefit, but they're worth a try.

I've also added some additional error message outputs within the specfic block of code where the problems appears to be occuring. The output is strictly to screen, so if the timeout increase does not resolve the problem, pay attention to the screen output for any error messages and/or values that are written and let me know what they are.

Rather than attach a debugging binary to this thread, if any of you want to PM me with an email address to send a copy to, I will forward it over.

JohnSorTivo
01-10-2005, 02:42 PM
Further update...I scheduled an unattended tyExtract to take place at 1:30am on one of the units that had been working properly. However, that attempt failed with the same error message as above which had been occurring on the other unit. This morning I am able to manually execute tyExtract successfully on all three units. So the problem is intermittent. I turned up the logging level for mfs_ftp and will provide the log info the next time tyExtract fails.Curious, when you use the automated approach, before connecting to MFS_FTP, are you also scheduling the startup of your mfs_ftp server? What I'm wondering is if you are attempting to connect too quickly while mfs_ftp is still starting up / has not settled down, and as such, it is not responding with the list of files in your directory.

mrwalker66
01-10-2005, 03:44 PM
no same problem, works fine on my local network...
07:20:04:PM - sourcing settings
01:20:06:PM - updating cached recording info
catch close lastsock val "can't read "info(lastsock)": no such element in array"
01:28:30:PM - 220 Mfs_Ftp ver 1.2.9p - {sock18} from "hidden:2758"
01:28:30:PM - 331 User name okay, need password.
01:28:30:PM - 230 Running in TiVo Mode.
01:28:30:PM - 250 Directory change successful.
01:28:30:PM - 200 Type set to I
01:28:30:PM - 227 Entering Passive Mode (192,168,10,3,12,32).
01:28:51:PM - 502 Command not implemented ""

tyExtractbeta.exe -h mommativo -s tv8 -l g:\tytool9r18 -v -d 0 -g -x -t tmf
Running in verbose mode.
Attempting to establish connection on 01/10/05 at 14:09:19...
Connection established with 'mommativo' on port 3105
*** Sorry, Unable to locate any recordings on your Tivo ***

JohnSorTivo
01-10-2005, 03:48 PM
tyExtractbeta.exe -h mommativo -s tv8 -l g:\tytool9r18 -v -d 0 -g -x -t tmf
Running in verbose mode.
Attempting to establish connection on 01/10/05 at 14:09:19...
Connection established with 'mommativo' on port 3105
*** Sorry, Unable to locate any recordings on your Tivo ***What's the additional ouput to the screen after the last line?:

*** Sorry, Unable to locate any recordings on your Tivo ***

I am not writing to the log file, but there should be some sort of error message and/or value printed to the screen.

mrwalker66
01-10-2005, 03:52 PM
FindFirst error: error 12002.. i think that's a timeout...

JohnSorTivo
01-10-2005, 03:57 PM
FindFirst error: error 12002Yes indeed. Error 12002 is a standard winsock Internet Timeout Error. The MFS_FTP server is not responding to tyExtract with the file list within the specified timeout period. Let me see if there are some additional flags I can set to circumvent / wait longer / or re-try a set number of times before bailing out all together.

mrwalker66
01-10-2005, 04:05 PM
yep, it's a long way away...
Reply from #.#.#.#: bytes=32 time=69ms TTL=52
Reply from #.#.#.#: bytes=32 time=64ms TTL=52
Reply from #.#.#.#: bytes=32 time=63ms TTL=52
Reply from #.#.#.#: bytes=32 time=63ms TTL=52

JohnSorTivo
01-10-2005, 04:47 PM
yep, it's a long way away...
Reply from #.#.#.#: bytes=32 time=69ms TTL=52
Reply from #.#.#.#: bytes=32 time=64ms TTL=52
Reply from #.#.#.#: bytes=32 time=63ms TTL=52
Reply from #.#.#.#: bytes=32 time=63ms TTL=52I would suspect that in your case, i.e. attempting to access from across the Web, that the same sort of error occurs using any basic FTP client. To confirm, I'm curious what happens if you exit to the command prompt and type:

ftp
open tivoaddress 3105
cd /ty
ls

And see if the 'ls' command hangs.

mrwalker66
01-10-2005, 05:08 PM
yes everytime. i use flashxp to extract. i got this to work by opening up port 3104 also. i just can't seem to get filezilla to work, i can't figure out what port , other that 3105, that it is trying to access..imagine my surprise after the long trip there, to get back and not be able to get a directory listing..using tyextract will be nice to automate the 5hr download per 1/2 hour show..

ftp> open mommativo 3105
Connected to mommativo.
220 Mfs_Ftp ver 1.2.9p - {sock18} from "#.#.#.#:1607"
User (mommativo:(none)):
331 User name okay, need password.
Password:
230 Running in TiVo Mode.
ftp> cd /ty
250 Directory change successful.
ftp> ls
Connection closed by remote host.

JohnSorTivo
01-10-2005, 05:17 PM
ftp> ls
Connection closed by remote host.As suspected. Unfortunately, I'm 100% at the mercy of the server I am connecting to, and it seems, for whatever reason, when not within the same network, no matter how high I try to set my timeout thresholds, a listing of the directory content will not return. As you have found, being within the same network is fine, in most cases (barring any other issues), but as soon as you jump out to the Internet, these problems arise.

I'll continue to look at this, but there may not be much I can do...

mmoore99
01-10-2005, 05:26 PM
Curious, when you use the automated approach, before connecting to MFS_FTP, are you also scheduling the startup of your mfs_ftp server? What I'm wondering is if you are attempting to connect too quickly while mfs_ftp is still starting up / has not settled down, and as such, it is not responding with the list of files in your directory.No, I am not scheduling the startup of mfs_ftp, it is already running on the tivo.

I installed the debug version that you sent me and for the last two hours have been running a script every 10 minutes that executes tyExtract against all three units. So far no error conditions have occurred. I'll keep you posted.

mrwalker66
01-10-2005, 05:28 PM
thanks for your hard work...you've gone above and beyond...the whole fun of this is trying to solve the problems you run into...thanks

JohnSorTivo
01-10-2005, 05:40 PM
No, I am not scheduling the startup of mfs_ftp, it is already running on the tivo.

I installed the debug version that you sent me and for the last two hours have been running a script every 10 minutes that executes tyExtract against all three units. So far no error conditions have occurred. I'll keep you posted.Thanks, please do keep me posted. I bumped all timeout settings in the debug version you are running to 60 seconds, which, when connecting internally should prevent any issues, if the problem is indeed related to timeouts, which it certainly appears is the case at first glance.

If you are scripting this, waiting to see if a similar error occurs, you may want to redirect ALL output to your own file, since I didn't modify the code to output the debug error messages to the tyextract.log file, it only outputs the pertinent error code, such as what mrwalker66 saw, to the screen. So you might what your execution string to appear as follows:

tyExtract.exe -h tivo.study -s 24 -l F:\My Videos - F\Tivo Video Files\tyExtract -d 1 -v -g > mylog.txt

JohnSorTivo
01-10-2005, 05:43 PM
...the whole fun of this is trying to solve the problems you run into...thanksI agree :)

Hopefully, we'll get to the bottom of this one way or another...

JohnSorTivo
01-12-2005, 11:49 AM
same problem...it might be timing out, the tivo is 800 miles awayOkay, I've been doing some extensive testing on this and have confirmed what the root cause of the problem is as it relates to using tyExtract, or various other FTP clients for that matter, when attempting to connect to the mfs_ftp server from across the Web. I apologize in advance for the dissertation that follows. In addition, the results of my testing are based on my network setup, so YMMV.

First a little background information:

tyExtract connects to the mfs_ftp server in PASSIVE mode, which is required in order to successfully connect and initiate transfers. One of the key differences between FTP ACTIVE and PASSIVE mode is "who" specifies which "data channel" will be used for transfer. In active mode, the data channel used is specified by the FTP client connecting to the server. This data channel includes the FTP client's IP to use for communication, which, when connecting from across the Web is an external, routable IP. However, in PASSIVE mode, the data channel is supplied by the FTP server, in this case MFS_FTP. The problem is, when connecting to an MFS_FTP server running on your local network, behind NAT, the IP supplied as part of the data channel is internal / non-routable. Thus, when any further communcation between the client (tyExtract) and the server (mfs_ftp) is attempted, using this data channel, what do you get? Bingo, TIMEOUTS, and no amount of increases in timeout thresholds will resolve this issue, since again, you are attempting to communicate on a non-routable data channel.

So, this begs the question, how/why am I able to connect to a standard FTP server within my network (behind NAT), say tivoftpd, remotely, in passive mode, and successfully transfer data back and forth. Glad you asked. Standard FTP servers run on PORT 21, and lo-and-behold, many routers using NAT (including mine), know to translate the local (non-routable) IP supplied by your FTP server in PASSIVE mode to your router's external IP address, when this communcation occurs on PORT 21, and thus, no issues.

However, mfs_ftp runs on a non-standard port, 3105 by default, and the router is not setup / configured to recognize this port for NAT translation. Want to test this out? If you're running tivoftpd on your Tivo, kill it, and startup mfs_ftp on your Tivo on PORT 21, now, use tyExtract from across the Web and attempt to connect and extract a show. Works nicely huh! Albeit slow as molasses.

So now you say, but John, I'm able to use certain FTP clients from across the Web and can connect to the MFS_FTP server on port 3105 just fine. How can this be? Well, certain FTP clients, such as wfs_ftp PRO are smart enough to detect that when in PASSIVE mode, if the FTP server specifies a data channel that includes a non-routable IP, to translate this IP on the client side to your machine's external IP address, in much the same way your router does so automatically for passive requests on port 21. Some FTP clients aren't smart enough to do this translation, and this is why you will find that some work, and others don't for this purpose.

Unfortunatley, tyExtract was built using the wininet API, and these libraries don't provide the opportunity for me to analyze the data channel and adapt accordingly. Thus, there are 2 solutions to this problem.

The first solution, as referenced above, is to run mfs_ftp on port 21 if you want to be able to use tyExtract to connect remotely from the Web, as the NAT translation will automatically occur in most cases (router dependent).

Or, you can make a small, temporary tweak to your mfs_ftp.tcl file, so that when connecting in passive mode, the mfs_ftp server provides, for the data channel, your network's external IP address, rather than the internal address, essentially providing your own manual NAT translation that would have occured otherwise at the router level, when running on port 21, or at the ftp client level for some software. To manually translate, determine what the external IP of your network is, and adjust the following within mfs_ftp.tcl:


set option1 [lindex $args 1] ; set ipaddr $info(localip)

To:


set option1 [lindex $args 1] ; set ipaddr XXX.XXX.XXX.XXX

Where XXX.XXX.XXX.XXX is your external IP address.

Either method provides a workaround to this problem.

Hope this helps.

John

mrwalker66
01-13-2005, 10:10 PM
man what a read....thank you john for your help...will let you know if it works after my next 5 hour download finishes...whew...

xjphil
02-17-2005, 06:27 PM
I read earlier about a poster having probs with the recorded on date. My recordings are showing 2 years and 5 (almost 6) months before they ACTUALLY happened. Currently recording Aug 2002.

I didn't see if there was a reason / solution for this. Thanks, great app, if I can get it to work.

phil

bdeweese
02-18-2005, 12:58 PM
Great little hack John! I am encountering a couple of issues. I have the HR10-250 and most of my recordings are HD - typically moveis which are 8 - 10gb in size. I have been having a problem where the first movie will download fine, but then apparently due to some code in mfs_ftp and its large file size an error will follow and prevent any additional recordings. I have made changes to my mfs_ftp per an earlier post in this thread and have yet to determine if that has fixed that issue or not. However, I notice that whenever it goes to fetch the file it reports back the size its getting as 4gb, even though it is much larger than that. Is this normal, or is there something I can or should do that would allow it to show a more accurate figure. Even though it says 4gb, if the download is successful it will be for the whole file size of 8 - 10gb.

Thanks! Bob

EDIT: Just checked my latest extraction request and it has successfully downloaded the 1st movie (8gb) and proceeded to the next. The patch to mfs_ftp mentioned earlier in this thread seems to have made the diff. Just still curious about the 4 gb message, however.

bdeweese
02-20-2005, 01:20 PM
Another thought - I am not a coder so no idea how easy or difficult some of these feature requests may be. But, would it be feasable to specify multiple output directories based on certain criteria? For example, it would be nice to be able to specify d:\extracts for movies A-L, e:\extracts for movies M - S, and so on. This would make it easier to use along side the media server hack for reinsertion.

If this isn't possible - no worries. Still an excellent hack.

Bob

jarob10
04-28-2005, 04:35 AM
As another feature request, would it be feasible to provide a synch option between TIVO and a nominated folder?

brianDamge
07-30-2005, 09:25 PM
I am trying to extract my kids' shows nightly. A number of Blue's Clues episode titles end with a question mark. Although tyExtract is able to match on the date/title, it fails when attempting to extract episodes that include a question mark in the title.

SteveT
09-27-2005, 08:24 PM
This is a great app that I use regularly. However, there is one enhancement that I'd love to see, and that is a switch to NOT include the channel name when checking for an already-downloaded copy of a show.

Some of my kids' shows are available from more than one channel, and I keep getting duplicates on my PC. I hadn't noticed it until my kids started using tivoserver to select the shows and pointed out the duplicates.

In my case, I'd like to just use the title, episode description and original air date to check, leaving channel name out of the decision.

Thanks.

JohnSorTivo
09-28-2005, 11:57 AM
This is a great app that I use regularly. However, there is one enhancement that I'd love to see, and that is a switch to NOT include the channel name when checking for an already-downloaded copy of a show.

Some of my kids' shows are available from more than one channel, and I keep getting duplicates on my PC. I hadn't noticed it until my kids started using tivoserver to select the shows and pointed out the duplicates.

In my case, I'd like to just use the title, episode description and original air date to check, leaving channel name out of the decision.

Thanks.Glad you've been getting good use of this utility.

I can certainly take a look at this, but a quicker / immediate solution from your side would be to modify mfs_ftp's settings.tcl file, so that the channel name is simply not included as the downloaded file name is the first place, by setting:

set info(name_detail) 5

To a lower setting, such as 0, 1, or 2.

In this manner, duplicates won't be downloaded, since the show names won't be unique between channels.

Or, you can accomplish the same with tyExtract by using the -n flag, and not including the channel name field as part of the output filename.

SteveT
09-28-2005, 12:25 PM
....Or, you can accomplish the same with tyExtract by using the -n flag, and not including the channel name field as part of the output filename.OK, great. I'll give "-n 123" a try.
Am I correct that I'll have to rename my previously-downloaded files to the new naming convention?

JohnSorTivo
09-28-2005, 12:37 PM
OK, great. I'll give "-n 123" a try.
Am I correct that I'll have to rename my previously-downloaded files to the new naming convention?That is correct.

amphion
09-26-2006, 09:09 PM
TyExtract works great!

But...
I've noticed that it is leaving temp files in my Temporary Internet Files.
Anyone else notice this?

Thanks,
Amphion

deutsch1
04-08-2007, 11:44 PM
I have two dsr704's running 6.2a with mfs_ftp. I would like have a evergreen backup of my tivo recordings on a external 500gb drive connected to my pc. Is there any way set it up so I will have shows that have been deleted on my tivo automatically deleted on my archive usb drive? I know I can do the opposite, but my goal is to have my backup'ed recordings match what is currently in the now playing list without user intervention. So any shows that were previously transferred to the backup drive that are no longer on the tivo would be automatically deleted.

michaeljc70
05-08-2007, 07:41 PM
1) Since this thread is so old and there are no replies, is there a newer batch program for getting files of Driectivo?

2) This program works great for me. I am having a problem with "24" though. It only accepts one keyword so anything recorded with the minute, hour, day, etc of "24" is also downloaded. It would be nice if you could use wildcards specify that the keyword is only in the title of the show. Am I missing something?

ghuff
11-21-2010, 01:49 PM
I am attempting to use tyExtract on my three dtivos (2-hdvr2's, 1-sdvr80). Everything is working fine on the two hdvr2's. However, when I attempt to use tyExtract on the sdvr80 I get the following messages in the log:
tyExtract.exe -h tivo.study -s 24 -l F:\My Videos - F\Tivo Video Files\tyExtract -d 1 -v -g

Running in verbose mode.

Attempting to establish connection on 01/09/05 at 22:43:49...
Connection established with 'tivo.study' on port 3105


*** Sorry, Unable to locate any recordings on your Tivo ***All three units are running the same software, including the same version of mfs_ftp. Also, on the unit that tyExtract is having the problem with, I am able to connect with mfs_ftp via SmartFTP and upload/download recordings to the PC as well as from tivo to tivo. Any help would be appreciated.

I have just found this utility and believe it would work great for me. I get the same problem Mmoore99 had (unable to locate recordings). However, I am on a LAN.

I suspect I am just hitting timeout issues as it takes well over a minute to get a TMF directory listing with a FTP client and tyExtract seems to give up and throw this error before then.

Is it possible to get the debug versions with extended timeouts that you developed to diagnose his problem so I can see if longer timeouts fixes mine?