Page 1 of 7 123 ... LastLast
Results 1 to 15 of 95

Thread: TyExtract - Windows command line MFS_FTP batch extraction utility

  1. #1
    Join Date
    Feb 2004
    Location
    Chicago
    Posts
    877

    TyExtract - Windows command line MFS_FTP batch extraction utility

    Attached is a Windows command line MFS_FTP batch extraction utility that I wrote in order to automate the process of extracting multiple shows from your Tivo/Tivos via a single process, and I thought that I would share it with the group.

    The utility is intended to be run from the Windows Scheduler, so that on a specified interval, it will automatically run and extract programs that contain your specified "keyword(s)" in their filenames, which have been recorded within a specific day range.

    In this way, rather than having to connect to my Tivo to see what has been recorded, select the files, and then wait for them to download before I can then begin editing them, I will have my shows waiting for me, so that I can begin editing right away.

    I built this using the WinInet libraries, and as such, your PC will require Internet Explorer 4.0 or greater, since the required DLLs accompany this application when installing, and obvioulsy, since TyExtract is an MFS_FTP command line utility, you need to be running the MFS_FTP server on your Tivo.

    The attached zip file contains the executable, tyExtract.exe, a README.TXT file with detailed usage instructions, an example .bat file setup (example.bat), and an example of the resultant messages that get generated when the utility is run (tyExtract.log).

    Version History:

    tyExtract v1.0.0 - New Release
    tyExtract v1.1.0 - Coding optimizations (no feature changes)
    tyExtract v1.2.0 - Automatic logging via -g flag, * for -s flag for downloading ALL recordings within a range.
    tyExtract v1.2.1 - Coding optimization (no feature changes)
    tyExtract v1.2.2 - Output command line arguments to log file, Fix day range bug, Add -x flag to auto-delete recordings after extraction.
    tyExtract v1.2.3 - Coding optimizations (no feature changes)
    tyExtract v1.3.0 - Added ability to point to a to a text file, via the '-s' flag, containing a list of show name 'keywords' to extract upon.
    tyExtract v1.4.0 - Format output filename using '-n' switch, skip extraction if file already exists in local directory, include 'shutdown.txt' to shutdown MFS_FTP server upon completion.
    Attached Files Attached Files
    Last edited by JohnSorTivo; 06-16-2004 at 11:46 AM.
    1 HR10-250, upgraded to 570 SD hours, hacked, 6.3b.
    1 HDVR2, upgraded to 206 hours, hacked, 6.2.
    1 HDVR2, upgraded to 168 hours, hacked, 6.2.
    tyExtract - Automated batch extraction utility
    YacMon - YAC Server log monitor for new call(s) notification via email/text message

  2. #2
    Join Date
    Nov 2002
    Posts
    221
    Can you tweak it to allow downloading by "thumbs up" rating in addition to the date.
    Scott

    SAT-T60 Ver 3.5

  3. #3
    Join Date
    Feb 2004
    Location
    Chicago
    Posts
    877
    Quote Originally Posted by osetivo
    Can you tweak it to allow downloading by "thumbs up" rating in addition to the date.
    I'm limited to the information that is supplied to me via the MFS_FTP server, in terms of the 'show name' and 'file create date'. If MFS_FTP could be modified to include, as part of the 'show name' string, a thumbs rating value, then it could be done. However, this is not currently part of the standard 'show name' string that does get generated.

    As basic 'show name' string looks something like:

    {Seinfeld}{1996-10-17}{The Package}{11.00 PM Mon May 17, 2004}{CH32}.ty

    So you can extract, using the "-s" flag on any of these sub-strings, but since "thumbs" is not a part of this string, there would be no way to know.
    1 HR10-250, upgraded to 570 SD hours, hacked, 6.3b.
    1 HDVR2, upgraded to 206 hours, hacked, 6.2.
    1 HDVR2, upgraded to 168 hours, hacked, 6.2.
    tyExtract - Automated batch extraction utility
    YacMon - YAC Server log monitor for new call(s) notification via email/text message

  4. #4
    Join Date
    Nov 2002
    Posts
    221
    I've asked Riley about accessing the Thumbs Up in the mfs_ftp thread.
    Scott

    SAT-T60 Ver 3.5

  5. #5
    Join Date
    May 2002
    Location
    West Hartford CT
    Posts
    620
    Quote Originally Posted by osetivo
    I've asked Riley about accessing the Thumbs Up in the mfs_ftp thread.
    this is a great application! thanks..

    would you consider allowing wildcards .. as in -s * so that it would transfer anything matching by date range switch. This way it would get everything, recorded.

    can't wait to try it out!
    ~Hi8

    (4) Hughes SD DVR40
    (1) Hughes HR10-250
    (2) xbmc XBOX X2 & xbit

  6. #6
    Join Date
    Feb 2004
    Location
    Chicago
    Posts
    877
    Quote Originally Posted by Hi8
    this is a great application! thanks..

    would you consider allowing wildcards .. as in -s * so that it would transfer anything matching by date range switch. This way it would get everything, recorded.

    can't wait to try it out!
    Great idea. In fact, I've already started working on it (wildcards) for the next version. Just makes sense.

    In addition, I've begun integrating logging directly into the application, so that when passing another flag, '-g', you can enable screen messages to be written to a log file automatically.

    Let me know how it works for you.
    Last edited by JohnSorTivo; 05-18-2004 at 09:50 PM.
    1 HR10-250, upgraded to 570 SD hours, hacked, 6.3b.
    1 HDVR2, upgraded to 206 hours, hacked, 6.2.
    1 HDVR2, upgraded to 168 hours, hacked, 6.2.
    tyExtract - Automated batch extraction utility
    YacMon - YAC Server log monitor for new call(s) notification via email/text message

  7. #7
    Join Date
    Feb 2004
    Location
    Chicago
    Posts
    877
    Quote Originally Posted by Hi8
    this is a great application! thanks..

    would you consider allowing wildcards .. as in -s * so that it would transfer anything matching by date range switch. This way it would get everything, recorded.

    can't wait to try it out!
    Just posted a new version (v1.2) in the .zip file above, that has added wildcard support, i.e. '-s *', so that you can download ALL recordings made within the date range specified via the '-d' flag, regardless of their show name.

    Also added automatic logging using the '-g' flag, so that screen messages will be automatically written/appended to tyExtract.log. This file will be located in the same directory that you have specified for your recordings to be saved, using the '-l' switch.
    Last edited by JohnSorTivo; 12-11-2004 at 10:55 PM.
    1 HR10-250, upgraded to 570 SD hours, hacked, 6.3b.
    1 HDVR2, upgraded to 206 hours, hacked, 6.2.
    1 HDVR2, upgraded to 168 hours, hacked, 6.2.
    tyExtract - Automated batch extraction utility
    YacMon - YAC Server log monitor for new call(s) notification via email/text message

  8. #8
    Join Date
    May 2002
    Location
    West Hartford CT
    Posts
    620
    Quote Originally Posted by JohnSorTivo
    Just posted a new version (v1.2) in the .zip file above, that has added wildcard support, i.e. '-s *', so that you can download ALL recordings made within the date range specified via the '-d' flag, regardless of their show name.

    Also added automatic logging using the '-g' flag, so that screen messages will be automatically written/appended to tyExtract.log. This file will be located in
    the same directory that you have specified for your recordings to be saved, using the '-l' switch.

    having trouble figuring out the 'day:day' switch. It seems to be off by 3 days... tried a few times with 0:0 0:6 finally after 3:3 it seemed to think that that was recordings for today. ??

    here's a copy of the console results as the log doesn't show the commandline.
    Code:
    G:\_downloads\_TIVO\TyExtract\tyExtract.12>tyExtract.exe -h 192.168.1.111 -s * -
    l d:\_tivo -d 3:3 -v -g
    Running in verbose mode.
    
    Attempting to establish connection on 05/19/04 at 19:46:35...
    Connection established with '192.168.1.111' on port 3105
    
    Matching Show: The Tonight Show With Jay Leno-2004.04.29-No Title.ty
    Recorded on: Thu, April 29
    Request range: Wed, May 19 to Wed, May 19
    Sorry, recording does not fall within range
    
    Matching Show: The Tonight Show With Jay Leno-2004.04.30-No Title.ty
    Recorded on: Fri, April 30
    Request range: Wed, May 19 to Wed, May 19
    Sorry, recording does not fall within range
    
    
    
    Matching Show: The Pathfinder-.15 AM Wed May 19, 2004-No Title.ty
    Recorded on: Wed, May 19
    Request range: Wed, May 19 to Wed, May 19
    Show is within requested range
    Downloading  2.1 GB
    ~Hi8

    (4) Hughes SD DVR40
    (1) Hughes HR10-250
    (2) xbmc XBOX X2 & xbit

  9. #9
    Join Date
    May 2002
    Location
    West Hartford CT
    Posts
    620

    other ommandline switch problems

    also tried to get the "tmf" instead of the "ty" and it still search for the "ty" files...

    Code:
    G:\_downloads\_TIVO\TyExtract\tyExtract.12>tyExtract.exe -t tmf -h 192.168.1.111 -s SHO -l d:\_tivo -d 5 -v -g
    Running in verbose mode.
    
    Attempting to establish connection on 05/19/04 at 20:23:27...
    Connection established with '192.168.1.111' on port 3105
    
    Matching Show: The Tonight Show With Jay Leno-2004.04.29-No Title.ty
    Recorded on: Thu, April 29
    Request range: Mon, May 17 to Mon, May 17
    Sorry, recording does not fall within range
    
    Matching Show: The Ellen DeGeneres Show-5.17-No Title.ty
    Recorded on: Mon, May 17
    Request range: Mon, May 17 to Mon, May 17
    Show is within requested range
    Downloading 888.0 MB
    I also tried to get recordings with SHO hoping to get recordings from Showtime .. the tmf files have {SHO} in the file name can I put that complete string in? including the {} characters? are there limits or "special" character restrictions?
    ~Hi8

    (4) Hughes SD DVR40
    (1) Hughes HR10-250
    (2) xbmc XBOX X2 & xbit

  10. #10
    Join Date
    Feb 2004
    Location
    Chicago
    Posts
    877
    Quote Originally Posted by Hi8
    having trouble figuring out the 'day:day' switch. It seems to be off by 3 days... tried a few times with 0:0 0:6 finally after 3:3 it seemed to think that that was recordings for today. ??

    here's a copy of the console results as the log doesn't show the commandline.
    Code:
    G:\_downloads\_TIVO\TyExtract\tyExtract.12>tyExtract.exe -h 192.168.1.111 -s * -
    l d:\_tivo -d 3:3 -v -g
    Running in verbose mode.
    
    Attempting to establish connection on 05/19/04 at 19:46:35...
    Connection established with '192.168.1.111' on port 3105
    
    Matching Show: The Tonight Show With Jay Leno-2004.04.29-No Title.ty
    Recorded on: Thu, April 29
    Request range: Wed, May 19 to Wed, May 19
    Sorry, recording does not fall within range
    
    Matching Show: The Tonight Show With Jay Leno-2004.04.30-No Title.ty
    Recorded on: Fri, April 30
    Request range: Wed, May 19 to Wed, May 19
    Sorry, recording does not fall within range
    
    
    
    Matching Show: The Pathfinder-.15 AM Wed May 19, 2004-No Title.ty
    Recorded on: Wed, May 19
    Request range: Wed, May 19 to Wed, May 19
    Show is within requested range
    Downloading  2.1 GB
    Thanks for the feedback.

    Are the "Recorded On" dates displaying correctly for your shows, and just not the request range?

    If so, what Timezone are you in? Normalizing the Tivo filesystem time with your current localtime can be tricky, and I am doing some timezone offsets, etc, in an attempt to compare apples to apples. Also, since I am using your system time, make sure your date and time on your PC is accurate.

    I'll get to the bottom of it once I get a bit more feedback. Interesting that it appears to be off by exactly three days. You are correct in that 0:0 should be equivalent to today. If you don't enter anything for the '-d' flag, which is equivalent to 0:0, does the "Request Range" actually show as 3 days into the future?
    1 HR10-250, upgraded to 570 SD hours, hacked, 6.3b.
    1 HDVR2, upgraded to 206 hours, hacked, 6.2.
    1 HDVR2, upgraded to 168 hours, hacked, 6.2.
    tyExtract - Automated batch extraction utility
    YacMon - YAC Server log monitor for new call(s) notification via email/text message

  11. #11
    Join Date
    Feb 2004
    Location
    Chicago
    Posts
    877
    Quote Originally Posted by Hi8
    also tried to get the "tmf" instead of the "ty" and it still search for the "ty" files...
    That's just a bonehead move on my part. It actually IS downloading the "tmf" file. In attempting to "clean" the file name to something more condensed, I inadvertently hardcoded the extension to .ty, regardless of the "type" specified. The actual file is right, just not the extension ;-(

    I'm going to remove the "cleaning" of the filenames all together for now, in order to get a revised version up ASAP. This will correct the extension naming error, and will write the output file exactly as it is provided by the MFS_FTP server. While verbose, at least it will be correct.

    As far as what you can pass in via the "-s", show name flag, yes you can include the open and close braces, etc, in order to get more finite. You can essentially include any and everything that is actually part of the entire show name string, to the point that if you entered it exactly as it appears in your FTP client, you would end up making a request to download a single file. Just make sure to surround your name in double quotes if it includes any spaces, i.e:

    -s "{Everybody Loves Raymond}"
    Last edited by JohnSorTivo; 05-22-2004 at 03:19 PM.
    1 HR10-250, upgraded to 570 SD hours, hacked, 6.3b.
    1 HDVR2, upgraded to 206 hours, hacked, 6.2.
    1 HDVR2, upgraded to 168 hours, hacked, 6.2.
    tyExtract - Automated batch extraction utility
    YacMon - YAC Server log monitor for new call(s) notification via email/text message

  12. #12
    Join Date
    May 2002
    Location
    West Hartford CT
    Posts
    620
    Quote Originally Posted by JohnSorTivo
    Thanks for the feedback.

    Are the "Recorded On" dates displaying correctly for your shows, and just not the request range?

    If so, what Timezone are you in? Normalizing the Tivo filesystem time with your current localtime can be tricky, and I am doing some timezone offsets, etc, in an attempt to compare apples to apples. Also, since I am using your system time, make sure your date and time on your PC is accurate.

    I'll get to the bottom of it once I get a bit more feedback. Interesting that it appears to be off by exactly three days. You are correct in that 0:0 should be equivalent to today. If you don't enter anything for the '-d' flag, which is equivalent to 0:0, does the "Request Range" actually show as 3 days into the future?

    I'm in EST (GMT -5:00) and my PC is in sync via timeserver -

    and yes the recorded on time is acurate, if you notice that the last recordind matches and is downloading ...


    and here is a dump I just made .. with -d 0:0 show the future dates...

    Code:
    Running in verbose mode.
    
    Attempting to establish connection on 05/19/04 at 22:33:17...
    Connection established with '192.168.1.111' on port 3105
    Matching Show: The Pathfinder-.15 AM Wed May 19, 2004-No Title.ty
    
    Recorded on: Wed, May 19
    Request range: Sat, May 22 to Sat, May 22
    Sorry, recording does not fall within range
    
    Matching Show: The Ellen DeGeneres Show-5.19-No Title.ty
    Recorded on: Wed, May 19
    Request range: Sat, May 22 to Sat, May 22
    Sorry, recording does not fall within range
    
    *** No shows with '*' found in your Now Playing List,
    recorded between Sat, May 22 and Sat, May 22. ***
    ~Hi8

    (4) Hughes SD DVR40
    (1) Hughes HR10-250
    (2) xbmc XBOX X2 & xbit

  13. #13
    Join Date
    Feb 2004
    Location
    Chicago
    Posts
    877
    Okay. I'll have to dig into my time calculations to see what is happening. I've only to date used the program on my machine, which is CST, and haven't had any problems, so I'll have start doing some debugging.

    In the meantime, I've posted another update, v1.2.1, to fix the file naming convention error. Please give it a try with the various extensions for the "-t" flag and let me know if you now get what you expect.

    Again, the resultant filenames will be much larger, as they should be a direct pass-through of what MFS_FTP provides, rather than me trying to "pretty them up".
    1 HR10-250, upgraded to 570 SD hours, hacked, 6.3b.
    1 HDVR2, upgraded to 206 hours, hacked, 6.2.
    1 HDVR2, upgraded to 168 hours, hacked, 6.2.
    tyExtract - Automated batch extraction utility
    YacMon - YAC Server log monitor for new call(s) notification via email/text message

  14. #14
    Join Date
    May 2002
    Location
    West Hartford CT
    Posts
    620

    PORT switch also not working...

    I have an S1 DTIVO mfs_ftp running on port 24 I set the -p 24 but would never get any recordings, even thought the log sdaid it connected. Tried serveral times and dates.. nothing. I then went in and killed MFS_FTP and restarted on the default port 3105, everything worked after that, still had the 3 day issue but it transfered those files that matched. That was the only change I made, I took out the -p 24 and restarted mfs_ftp on default ---
    ~Hi8

    (4) Hughes SD DVR40
    (1) Hughes HR10-250
    (2) xbmc XBOX X2 & xbit

  15. #15
    Join Date
    Feb 2004
    Location
    Chicago
    Posts
    877
    Quote Originally Posted by Hi8
    I have an S1 DTIVO mfs_ftp running on port 24 I set the -p 24 but would never get any recordings, even thought the log sdaid it connected. Tried serveral times and dates.. nothing. I then went in and killed MFS_FTP and restarted on the default port 3105, everything worked after that, still had the 3 day issue but it transfered those files that matched. That was the only change I made, I took out the -p 24 and restarted mfs_ftp on default ---
    Check the contents of your port.24.log in your MFS_FTP directory for errors. I've got the port stuff working pretty well. I shutdown my MFS_FTP server and started it back up on port 24 to see what happens, and it fails as well, and I see the following in my log:

    Code:
    catch close lastsock val ""
    01:51:08:AM - 220 Mfs_Ftp ver 1.2.9p - {sock22} from "192.168.1.2:4170"
    01:51:28:AM - 331 User name okay, need password.
    01:51:28:AM - 230 Running in TiVo Mode.
    01:51:28:AM - 250 Directory change successful.
    01:51:28:AM - 200 Type set to I
    bgerror invoked with error
    
     " couldn't open socket: address already in use "
    
    re-initializing mfs_ftp
    
    close the current ftp connection and simply open another
    You can see that tyExtract is able to connect, but then is kicked out by the server with a socket issue. The same exact thing happens to me when attempting to connect using FlashFXP and smartFTP on port 24, so it appears to be an error while attempting to use this specific port (24).

    Do you see the same errors in your log file?

    I shutdown and ran on various other random ports (high numbers), such as 2000, 2222, 3333, and all worked perfectly. Try starting up on some of these ports, and let me know your findings.

    Other items: I'm going to dig into the code today to see if I can determine what is going on with the "day range" being messed up for you. I've even reset my computer to your timezone, and still can't get the error to repeat.

    Also, I've added to the top of the log file writing, the command line arguments, so that, for debugging purposes, it will first show the commands you passed in, followed by the resultant output. So in the next release, hopefully, in addition to the day range fix, this little change will also be incorporated.

    Correct me if I am wrong, but the "day range" issue seems to be the last major outstanding issue, provided my last release resolved your issue with improper extension naming (always .ty), which I believe it should have.

    Thanks for your help / feedback.
    1 HR10-250, upgraded to 570 SD hours, hacked, 6.3b.
    1 HDVR2, upgraded to 206 hours, hacked, 6.2.
    1 HDVR2, upgraded to 168 hours, hacked, 6.2.
    tyExtract - Automated batch extraction utility
    YacMon - YAC Server log monitor for new call(s) notification via email/text message

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •