Page 1 of 10 123 ... LastLast
Results 1 to 15 of 144

Thread: tyftpd (mfs_ftp.tcl alternative)

  1. #1
    Join Date
    Mar 2005
    Posts
    233

    tyftpd (mfs_ftp.tcl alternative)

    This is an alternative to mfs_ftp.tcl that began as a replacement written in C and based on oftpd. After adding uploading ability it made more sense to optimize the mfs_* calls and keep the ftp code in tcl. Along the way I added another file type (tmf+) which is meant to solve the bookmark juggle problem in mfs_ftp by sending the master block of the ty file at the beginning of the upload. Anyhow, this alternative should be fairly stable and fixes some flaws I found when trying to use mfs_ftp.tcl.
    New Version: 1.9
    Code:
    ################################################################################
    # tyFTPd - Copyright (C) 2012 jkozee                                           #
    # This program is distributed WITHOUT ANY WARRANTY; without even the implied   #
    # warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.             #
    #                                                                              #
    # TERMS OF USE: personal (noncommercial) fair-use backup & restore             #
    # This software may not be used for copyright infringment or any illegal       #
    # purpose, bundled with any product, sold, distributed or redistributed        #
    # under any circumstances on any media without express written consent.        #
    ################################################################################
    # History: 1.9 20130113 Fixed missing quote as noted by tas3086.               #
    # History: 1.8 20120729 Catch and log TransactionRetry attempts.               #
    # History: 1.7 20120717 Fixed a bug related to not clearing CleanupStart when  #
    #                       doing multiple STOR in a single client connection.     #
    # History: 1.6 20120715 Changed the routines that call AddPart to use 64-bit   #
    #                       values by using wide().  This may break compatibility  #
    #                       with older units, however it allows for parts larger   #
    #                       than 2GB to be created.  Disable it by using           #
    #                       "set GlobalData(fUseWide) 0"                           #
    # History: 1.5 20120710 Fixed a bug that would crash when a TmsID exists in    #
    #                       the GuideIndex, but the object no longer exist int the #
    #                       mfs database.                                          #
    # History: 1.4 20120709 Added real station and npk support.                    #
    #                       Updated tyftpd station to be reusable.                 #
    #                       Fixed a bug that would not handle maxSTOR correctly.   #
    # History: 1.3 20120627 Fixed errors in UpdateParts.                           #
    #                       Fixed handling of tmf+ file uploads.  Using this file  #
    #                       type allows for initial allocation of all parts, which #
    #                       means the playbar is accurate almost immediately.      #
    #                       Added -M flag to only list the tmf+ directory.         #
    #                       Added code to handle incomplete STOR operations        #
    #                       better.  Now it removes unused parts and nulls out     #
    #                       portions of the final part to hide the junk data that  #
    #                       may exist due to using streamcreate vs streamfill.     #
    # History: 1.2 20120610 Added -p and -c items to usage text.                   #
    #                       Added TYPE for better EOL handling (FlashFXP).         #
    #                       Now using unified mfs32/64 code for mfs_ftp_helper.    #
    #                       Added -m flag to only list the tmf directory.          #
    #                       Added daemonize code unless loging to the screen.      #
    #                       Added code to set Parts Begin/End attribute set.       #
    #                       WARNING: tmf+ type handling suspect - DO NOT USE       #
    # History: 1.1 20111118 Fixed changed list EOLs to CRLF to support FlashFXP    #
    #                       Added [-p #] flag to change the listening port         #
    #                       Added [-c] to cleanup incomplete recordings on startup #
    #                       The default behavior is to not cleanup so startup is   #
    #                       much quicker.  The -c flag is only needed if tyFTPd    #
    #                       was aborted/died during upload.                        #
    # History: 1.0 20110815 Initial release.                                       #
    #                                                                              #
    # Purpose:  This script is an alternative to mfs_ftp.tcl.  It began as an      #
    #           experiment inspired by mfs_ftp.tcl and written in C based on oftpd #
    #           and linked to the mfs_utils library to see if I could improve      #
    #           download speeds.  Eventually I added upload abilities which        #
    #           required a tcl based server to handle the mfs objects. At that     #
    #           point I decided to just rewrite it all in tcl.  This code has a    #
    #           advantages over mfs_ftp.tcl.                                       #
    #           1) All of the required mfs_* functions are contained in a singe    #
    #              executable and is the only other required file (included).      #
    #           2) Faster directory listings without caching.                      #
    #           3) The directory listings are sent as they are being gathered      #
    #              instead of leaving the client waiting while they are gathered   #
    #           4) More complete mfs object creation including series creation to  #
    #              preserve folders on upload.                                     #
    #           5) Better error handling, especially when uploads are aborted they #
    #              are marked as completed and the recording status is stopped.    #
    #           6) Forced the FSID to be part of the filename to assure that the   #
    #              correct file is downloaded/deleted from the client.             #
    #                                                                              #
    # Usage:   -c       Cleanup incomplete recordings from prior instance          #
    #          -d       Include debug log information                              #
    #          -l       Log to /var/log/tyFTPd.log                                 #
    #          -m       Only list tmf directory                                    #
    #          -M       Only list tmf+ directory                                   #
    #          -p       Port to listen on (Default is 5013)                        #
    #          -s       Log to the Screen                                          #
    #          -v       Inlcude verbose log information                            #
    #          -x       Limit the reported filesize to 2GB for poorly written      #
    #                   clients (LeechFTP)                                         #
    #                                                                              #
    # Important: Make sure that you  update mfs_ftp_helper with each new release.  #
    #            It is now using the unified mfs-utils 32/64 version.              #
    #                                                                              #
    # Notes:   Portions of this code based on, borrowed from, or inspired by       #
    #          other projects found on DDB, including TWP, mfs_ftp, etc.  I only   #
    #          tested this on S3/11.0k and S2.5/7.2.2, probably needs tweaked for  #
    #          DTiVos.                                                             #
    #                                                                              #
    # MovieLoader: If you are using this with MovieLoader make sure you are using  #
    #              MovieLoader.v0.17c.zip or later.  You should change the port    #
    #              for tyftp by using the -p 3105.                                 #
    #                                                                              #
    ################################################################################
    Attached Files Attached Files
    Last edited by jkozee; 01-13-2013 at 05:25 PM. Reason: New Version

  2. #2
    Join Date
    Apr 2005
    Location
    Providence, RI
    Posts
    466
    I gave this a quick run, and it ran error-free on my Tivo HD. Really nice, and the file listing comes up much faster than mfs-ftp. Any thoughts of adding TS 'extraction', as in this thread?
    More software at http://davidlauria.com/software.
    The lost alt-rock masterpiece from the '90s, Range of Motion's "Soft Buzz of Silence", is now available on iTunes!

  3. #3
    Join Date
    Jan 2002
    Location
    Sonoran Desert
    Posts
    2,823
    I am somewhat curious, how processor intensive is it to run s3tots? Would it be possible to s3tots the stream in realtime while maintaining maximum transfer speed?

    I've been intending on looking into it myself but never put any serious effort into it.
    Before PMing me: Iím not your personal tech support. If you have a question, ask in public so I don't have to repeat if somebody else asks. If you want images or slices, use emule. I will ignore all support PMs.

    Sponsor a vegetarian! I have taken the pledge, how about you?

  4. #4
    Join Date
    Jul 2004
    Location
    California
    Posts
    298
    Hey, great tool! VERY VERY fast! I am getting 2.5MB/s transfer rate. The the file list populates very quickly as well. Rock solid so far. Well done!!

  5. #5
    Join Date
    Aug 2004
    Posts
    4,085
    Quote Originally Posted by AlphaWolf View Post
    I am somewhat curious, how processor intensive is it to run s3tots? Would it be possible to s3tots the stream in realtime while maintaining maximum transfer speed?

    I've been intending on looking into it myself but never put any serious effort into it.
    Have you looked at this thread?

  6. #6
    Join Date
    Mar 2010
    Posts
    10
    Did not notice the port changed at the first try Now I can connect to the server on my THD. But there is no show listed while the original mfs_ftp correctly shows two. Any idea what is wrong? Thanks for your help.
    Last edited by myasus; 09-27-2011 at 01:51 PM.

  7. #7
    Join Date
    Mar 2005
    Posts
    233
    Sorry, I probably should have pointed out that this listens on port 5013 not 3105, and can only be changed by editing the code as I did not add a port option to the arg list.

    To be honest I haven't looked at the code in quite a while, so I don't have any ideas off the top of my head. The show list generation is based on code from the unified mfs_* tools. Do the shows appear in the output from NowShowing from that package?

    The only thing I encountered in my testing was ftp clients that would not list shows that exceeded 2GiB (LeechFTP). Try running this command:

    mfs_ftp_helper -F LIST -A /tmf

    If they appear in this list then invoke the script with the -x flag to prevent it from reporting sizes over 2GiB.
    Last edited by jkozee; 09-27-2011 at 11:52 PM.

  8. #8
    Join Date
    Mar 2010
    Posts
    10
    Quote Originally Posted by jkozee View Post
    Do the shows appear in the output from NowShowing from that package?
    Yes. And regular mfs_ftp can list correctly.

    Quote Originally Posted by jkozee View Post
    The only thing I encountered in my testing was ftp clients that would not list shows that exceeded 2GiB (LeechFTP).
    Try running this command:
    mfs_ftp_helper -F LIST -A /tmf
    If they appear in this list then invoke the script with the -x flag to prevent it from reporting sizes over 2GiB.
    Not likely since regular mfs_ftp can list my 2 shows (both ~4G) correctly. "mfs_ftp_helper -F LIST -A /tmf" worked correctly, even with "-x 2GiB" option.

    Might because the patches I applied to mfs_ftp.tcl? Following the instruction on dvrpedia, I applied these:
    mfs_ftp.20070121.patch.tar.gz
    mfs_ftp-20070207-pasv-fix.patch.gz
    mfs_ftp-20070717b-ya-pasv-fix.patch.gz

  9. #9
    Join Date
    Mar 2005
    Posts
    233
    Wierd. The script uses mfs_ftp_helper to produce the directory listing. Did you rename mfs_ftp_helper.mips64 to mfs_ftp_helper and place it in the same directory as tyftpd.tcl. What do you see in /var/log/tyftpd.log when you run the script with logging turned on "tyftpd.tcl -l -v -d &"

  10. #10
    Join Date
    Mar 2010
    Posts
    10
    Quote Originally Posted by jkozee View Post
    Did you rename mfs_ftp_helper.mips64 to mfs_ftp_helper and place it in the same directory as tyftpd.tcl.
    I did. The log shows:

    Code:
    09/29/11 23:39:41 - (2559) >>> 220 tyFTPd 1.0 FTP server ready.
    09/29/11 23:39:41 - Now serving 192.168.1.200 port 2559
    09/29/11 23:39:41 - (2559) <<< USER anonymous
    09/29/11 23:39:41 - (2559) >>> 331 Please specify the password.
    09/29/11 23:39:41 - (2559) <<< PASS flashfxp-user@flashfxp.com
    09/29/11 23:39:41 - (2559) >>> 230 Login successful.
    09/29/11 23:39:41 - (2559) <<< SYST 
    09/29/11 23:39:41 - (2559) >>> 215 UNIX Type: L8
    09/29/11 23:39:41 - (2559) <<< REST 100
    09/29/11 23:39:41 - (2559) >>> 350 Restart okay, awaiting file retrieval request.
    09/29/11 23:39:41 - (2559) <<< REST 0
    09/29/11 23:39:41 - (2559) >>> 350 Restart okay, awaiting file retrieval request.
    09/29/11 23:39:41 - (2559) <<< PWD 
    09/29/11 23:39:41 - (2559) >>> 257 "/"
    09/29/11 23:39:43 - (2559) <<< TYPE A
    09/29/11 23:39:43 - (2559) >>> 200 I only have TiVo mode.
    09/29/11 23:39:43 - (2559) <<< PASV 
    09/29/11 23:39:43 - Updating the server ip address from 127,0,0,1...
    09/29/11 23:39:43 - Entering Passive mode for ip=192,168,1,201 port=34286
    09/29/11 23:39:43 - (2559) >>> 227 Entering Passive Mode (192,168,1,201,133,238).
    09/29/11 23:39:43 - In Pasv Accept with {sock17} {sock21} {192.168.1.200} {2560}
    09/29/11 23:39:43 - (2559) <<< LIST -al
    09/29/11 23:39:43 - Data Socket Connected
    09/29/11 23:39:43 - Suspending timeout for port 2559
    09/29/11 23:39:43 - (2559) >>> 150 Opening data channel
    09/29/11 23:39:43 - (2559) >>> 226 List transfer complete.
    09/29/11 23:39:43 - Restarting timeout for port 2559
    Last edited by myasus; 09-29-2011 at 07:43 PM.

  11. #11
    Join Date
    Mar 2005
    Posts
    233
    What client are you using? I think it might be the flag being sent with the LIST command. You can try removing the $flag from the call to mfs_ftp_helper in the proc LIST.

  12. #12
    Join Date
    Mar 2010
    Posts
    10
    Quote Originally Posted by jkozee View Post
    What client are you using? I think it might be the flag being sent with the LIST command. You can try removing the $flag from the call to mfs_ftp_helper in the proc LIST.
    I use flashfxp. It sends "LIST -al" command. Seems mfs_ftp.tcl has no problem with that.

  13. #13
    Join Date
    Mar 2010
    Posts
    10
    Just tried to use "LIST" only in flashfxp, still nothing listed.

  14. #14
    Join Date
    Mar 2005
    Posts
    233
    Not sure. Maybe it doesn't like the format that mfs_ftp_helper is listing. If you would still like to test tyftpd you can try another client. I'm not sure what platform your using, but I tested with LeechFTP and filezilla, which are both free. If I get time I may try debugging with flasfxp. Is it a free client?

  15. #15
    Join Date
    Mar 2010
    Posts
    10
    Quote Originally Posted by jkozee View Post
    If you would still like to test tyftpd you can try another client. I'm not sure what platform your using, but I tested with LeechFTP and filezilla, which are both free. If I get time I may try debugging with flasfxp. Is it a free client?
    Just tested with LeechFTP, it can list dirs, but not the shows. The logs are here:
    Code:
    ~ Connecting...
    ~ Connected to 192.168.1.201, waiting for response...
    < 220 tyFTPd 1.0 FTP server ready.
    > USER anonymous
    < 331 Please specify the password.
    > PASS *****
    < 230 Login successful.
    > REST 1
    < 350 Restart okay, awaiting file retrieval request.
    > REST 0
    < 350 Restart okay, awaiting file retrieval request.
    > SYST
    < 215 UNIX Type: L8
    > PWD
    < 257 "/"
    ~ Login completed.
    > PORT 192,168,1,200,9,136
    < 200 PORT OK
    > TYPE A
    < 200 I only have TiVo mode.
    > LIST
    < 150 Opening data channel
    < 226 List transfer complete.
    > CWD ty+
    < 250 Directory successfully changed.
    > PWD
    < 257 "/ty+"
    > PORT 192,168,1,200,9,140
    < 200 PORT OK
    > TYPE A
    < 200 I only have TiVo mode.
    > LIST
    < 150 Opening data channel
    < LISTERROR: -rwxr-xr-x   1 0        0     004320133120 Sep 24  2011 {SportsCenter}{2011-09-24}{}{09.00 PM Sat Sep 24, 2011}{ESPNWHD}{237550}.ty+
    < LISTERROR: -rwxr-xr-x   1 0        0     003785359360 Sep 21  2011 {CBS Evening News With Scott Pelley}{2011-09-21}{}{05.30 PM Wed Sep 21, 2011}{WCBSDT}{229266}.ty+
    < 226 List transfer complete.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •