Page 75 of 99 FirstFirst ... 2565737475767785 ... LastLast
Results 1,111 to 1,125 of 1476

Thread: Mfs_Ftp: extract, archive, restore & transfer recordings

  1. #1111
    Join Date
    Nov 2002
    Posts
    480

    Trouble inserting

    I am having troble re-inserting tmfs from one series 2 box to another. Variously , going from A to B, it truncates the files at 256mb and keeps inserting unviewable (32 second) duration files in the "now Playing" list or 512k files going the other way. It also disconnects suddenly (as seen in the log below) I can seem to extract fine.

    Here is log from AceFTP, but if someone can point me to an appropriate LOG where I can get more information to troubleshoot this, it would be appreciated.

    TIA

    TYPE I
    200 Type set to I
    PASV
    227 Entering Passive Mode (192,168,1,102,12,32).
    STOR {Battleplan}{2005-01-25}{Assault From the Sea}{04.00 PM Wed Jun 28, 2006}{MILT}.tmf
    150 Opening BINARY mode data connection for "{{Battleplan}{2005-01-25}{Assault From the Sea}{04.00 PM Wed Jun 28, 2006}{MILT}.tmf}"
    Disconnected

  2. #1112
    Join Date
    Dec 2005
    Posts
    42
    The problem I was having before seems to be occuring again. I checked my ifconfigs to see if it was a layer1/2 problem. No errors either side.

    I have noticed that my tivo seems a bit touchy. When mfs_ftp crashes, and I kill the process, the whole tivo crashes & reboots. This is new. I wonder if one of the HD's is going - it is an original 14 hour. Any ideas?

    Second question: I was trying to think of other ways to transfer the programs. I rarely do uploads, so downloading alone would be fine.

    Many thanks,
    -Dan

  3. #1113
    Join Date
    Dec 2005
    Posts
    42
    Hmmm.... it does similar things if I try to download via vstream-client. Crap, this must be a TIVO problem as it is not limited to mfs_ftp. Dang. I guess this would be the wrong forum; now where do I ask for help?

  4. #1114
    Join Date
    Aug 2006
    Posts
    65
    it seems that I needed to give it a LONG time before my directories were created.. To clarify the process, as is helpful every 30 pages ::

    1) I am using ffmpeg to take a .avi and sucessfully create a .ty+ file (verified xml is at the end and I can play it via WMP w/ tyshow)
    2) I transfer the .ty+ file to the /ty+ directory via mfs_ftp AS --- Show.ty+
    3) when its on the tivo, I need to rename it to the {xx}{yy}{zzz} notation ????

    I uploaded a poor quality .ty+ to the /ty directory, and it will not play via the tivo. if I attempt to delete it via the tivo, the tivo reboots.

    I assume that mfs_ftp can delete this (or rather, the ftp program can?)
    Last edited by ercdvs; 11-01-2006 at 01:22 PM. Reason: stupidity

  5. #1115
    Join Date
    Dec 2004
    Posts
    23

    DTivo Series 1 LBA48 500+ show reboot problem

    I updated one of my DTivo Series 1 boxes to the LBA48 Kernel and two 250GB drives then installed the standard hacks (ftp, telnet, tivowebplus, tserver, mfs_ftp, ...). Everything worked find with mfs_ftp when I initially tested it by copying a show directly from it to another Series 1 using SmartFTP. I come back a month or two later after Tivo suggestions has filled it up with 540 shows and now starting mfs_ftp results in a reboot several minutes after it has started.

    I originally installed the 1.2.9p-Jamie patched version (two patch files) of mfs_ftp and included the p2.tcl patch which people have added to fix the memory problem with NLST trying to store all the directory info into one variable. Here is the start of the log file.

    07:51:06:AM - sourcing settings
    07:51:06:AM - sourcing p2
    07:51:06:AM - versioncheck:
    07:51:06:AM - init: 1.2.9p-Jamie - TiVo_OS 3.5-01-1-031

    I changed dbl to 5 and it shows that the problem is with prune_cache running out of memory. I don't know tcl but it appears it is trying to load temp_list with all the file names in the cache directory which in this case was 1080 entries. I will add some debug messages to verify that this line is causing the problem.

    catch { set temp_list [glob $info(path)/cache/*] }

    I decided to do a test of just commenting out the call to prune_cache in init_procs to see what would happen. This fixed the auto reboot I got shortly after starting mfs_ftp.

    However, when I started SmartFTP and tried to get the directory listing for the tmf directory it again rebooted the Tivo. Here is the end of the log file.

    01:03:31:AM - parseline:
    "LIST -aL"
    01:03:31:AM - NLIST: "LIST -aL" ("" == LIST command)
    01:03:31:AM - 150 Opening ASCII mode data connection for file list.
    01:03:31:AM - list_type is "LIST"
    01:03:31:AM - update_rec_fsids: forced 1
    Dumping mempool to /tmp/BlockFailure.204

    To view the blocks, run:
    $TIVO_ROOT/devbin/poolview.tcl <app-with-symbols> /tmp/BlockFailure.204

    The behavior of SmartFTP would make me think that the p2.tcl code did not replace the NLST code since I don't see any directory entries showing up in the SmartFTP window. But the start of the log file clearly shows p2 was processed.

    07:51:06:AM - sourcing settings
    07:51:06:AM - sourcing p2
    07:51:06:AM - versioncheck:
    07:51:06:AM - init: 1.2.9p-Jamie - TiVo_OS 3.5-01-1-031
    07:51:06:AM - tivo version "3.x" mfspath set to /Recording/NowShowingByClassic
    07:51:06:AM - get_tzoffset:

    I will need to add more debug messages tonight to see what is going on with NLST.

    I have done a few searches and did not see the problem with the prune_cache code failing, only the problem with NLST failing with p2.tcl being the fix.

    I need a little guidance in how to fix prune_cache to work with large numbers of shows. The cache_xml_2_disk seems to have a "force" option to delete entries. Can we just delete the entire cache directory and start fresh each time mfs_ftp is run and would that eliminate the need for the pruch_cache call? I know next to nothing about tcl and the details of each command in mfs_ftp. I can see generally what is happening but a little help understanding the intricacies would be much appreciated. Thanks.

    Ok, just reviewed what I wrote and now see that the last failure may not have been in the NLST procedure but in update_rec_fsids. I will have to have a look at that routine tonight and add more debug messages.

    Turbo3

  6. #1116
    Join Date
    Aug 2006
    Posts
    65
    could someone give an overview of the process of which scripts are called?

    I realize some of my trouble could have been a large file via a wireless connection, but I sucessfully transfered a 850mb .ty+ file to the .ty+ directory.

    However, the filesize of the resulting renamed file is 0 bytes, but the xml data, and the info reported via the tivo shows the proper file size and time..

    In looking at the mps files via tivoweb, I see that the particular show is under /database/orphan/xxxxxxxx

    rather then what looks to be /server/xxxxxxxxx

    What process actually renames and correctly registers this fiel under the proper mps database? It seems I have qute a few files under /database/orphan ... should I be worried ?

  7. #1117
    Join Date
    Jan 2005
    Location
    Narnia
    Posts
    1,266
    Quote Originally Posted by Turbo3
    I need a little guidance in how to fix prune_cache to work with large numbers of shows. The cache_xml_2_disk seems to have a "force" option to delete entries. Can we just delete the entire cache directory and start fresh each time mfs_ftp is run and would that eliminate the need for the pruch_cache call? I know next to nothing about tcl and the details of each command in mfs_ftp.
    You can shutdown mfs_ftp, manually clear out the cache directory, and start up mfs_ftp again to rebuild it.

    Keep in mind that even the NLST patch creator, chrised, commented that it would only work to around 300 or so listings. One fix he made in March, 2006 (in the link above) is to try to send back one filename at a time instead of the whole shot. I don't know what assumptions the script makes about Series-1 or Series-2 units.

    There was also an older thread with sanderton about SmartFTP timeouts while waiting for mfs_ftp.
    Last edited by Narf54321; 11-01-2006 at 10:48 PM.

  8. #1118
    Join Date
    Dec 2004
    Posts
    23
    Version 0.02 of the ChrisEd's patch changed the way NLST worked so it would handle an unlimited number of shows by sending back directory entries one at a time. This patch can never have a memory problem and permanently fixes the original problem ChrisEd was trying to solve.

    But there are other parts of mfs_ftp that get called and also have problems with large numbers of shows and running out of memory. I have identified two. The prune_cache and now update_rec_fsids also need some reworking to handle large numbers of shows.

    There is nothing wrong with ChirsEd's patch and it does as much as it can to get mfs_ftp working with large numbers of shows. We just need some additional changes in other sections of mfs_ftp.

    Any usage of lappend or append could potentially overflow memory if it involves a variable on a per show bases. Problem areas must be reworked with an ChrisEd like solution of doing the operation on a per show basis if possible or coming up with some other way of working with smaller chucks of the show list.

    The problems I am running into are actually before you ever get to the ChrisEd patches but the problem looks the same in that you run out of memory and Tivo reboots. So if you don't set dbl=5 and look at the logs you might think the ChrisEd patch has hit some other limit which it has not. This is the mistake I first made in thinking the p2.tcl code was never loaded.

    Right now I need to understand the working of ForeachMfsFile and RetryTransaction commands before I can come up with a fix for the update_rec_fsids procedure.

    Turbo3

  9. #1119
    Join Date
    Dec 2004
    Posts
    23

    Problem solved - mfs_ftp can handle 500+ shows with no changes!!!

    Turns out the fix for the memory problem with large numbers of shows is to just tell the tcl interpreter (tivosh a.k.a. tivoapp) to allocate more storage for the memory pool. This is done with the TIVOSH_POOLSIZE environment variable. It must be set before starting mfs_ftp.tcl. I use this two line script in a file called doit.

    export TIVOSH_POOLSIZE=2097152
    /var/mfs_ftp/mfs_ftp.tcl

    Then I just type ./doit in the /var/mfs_ftp directory or you could fully qualify the path and type it from any where.

    I get a full directory listing of 538 files in just 27 seconds using the p2.tcl patch and SmartFTP.

    The default pool0 size seems to be 1458176 bytes (0x164000) which is just a little too small for the number of files I have. I started with 3megs which is too much and decided on 2 megs (2097152 = 0x200000 bytes). This leaves me with around 40000 bytes to spare.

    I added some code into mfs_ftp.tcl to log the memory statistics each time NLST is called so I can easily spot any future problems.

    So the idea that there is some type of limit on variable size is not really correct. There is a fixed amount of storage allocated when a tcl program is started and the tcl variables must fit in the amount or the amount must be increased.

    You probably don't need the p2.tcl patch with this increase to the available memory. I also get the feeling it runs faster with more memory based on the 27 seconds for 538 files which seems really fast.

    I was surprised at the amount of information there is on all the Tivo specific tcl commands but it turns out I did not need to use any of it except the "pool pool0 size" and "pool pool0 dump" commands.

    Turbo3

  10. #1120
    Join Date
    Jul 2005
    Posts
    507
    I am experiencing a new problem that I need help with. MFS_FTP is crashing when I try to view the contents of a directory. I ftp to tivo:3105 and the root shows fine. I then click on TY and it crashes.


    Havn't visited the forum in a while, glad to see others have the same problem as I did regarding large numbers of recordings Ive always resorted to deleting recordings until the program wouldnt crash. I will try the new enviroment variable.

    Edit: Is it possible to have the application crash and not reboot the Tivo? It takes aproximately 4 minutes for my Tivo to boot so its a pain in the ass while troubleshooting.
    Last edited by ciper; 11-14-2006 at 03:33 AM.

  11. #1121
    Join Date
    Nov 2004
    Location
    Gurnee, IL
    Posts
    2,385
    No -- any program that has its hooks in mfs like that will likely trigger a reboot if it crashes.

    Your best bet is to crank up the logging level and post a log.
    --
    Christopher D. Heer
    Quote Originally Posted by Oscar Wilde
    Perhaps, after all, America never has been discovered. I myself would say that it had merely been detected.

  12. #1122
    Join Date
    Oct 2002
    Posts
    63
    I'm having a problem with inserts stalling. My log reads the following

    bgerror invoked with error

    " error reading "sock24": connection timed out "
    I am using fetch on the mac (recommended client) and passive mode. Any suggestions?
    Sat T-60 130 HRS 3.5
    Tivoweb Plus, MFS FTP, Cachecard, Elseed
    HR10-250 80 hrs, killhdinitrd, MFS FTP, Tivoweb Plus

  13. #1123
    Join Date
    Feb 2005
    Location
    Bergen County
    Posts
    1,307
    Having the following problem using tmf2ty after extraction with mfs_ftp. All parts are processed except the last part which fails. This happens on both a HDVR2 and a HR10-250. Both have encryption disabled. The following is the error I receive.

    E:\Ty>tmf2ty Jailbait.tmf
    00:00 - entering init
    00:00 - filename is "Jailbait.tmf"
    00:00 - file root is "Jailbait"
    00:00 - file tail is "Jailbait.tmf"
    00:00 - root tail is "Jailbait"
    input filename is "Jailbait.tmf"
    00:00 - entering getxml
    00:00 - entering parsexml
    00:00 - listing info parsed from xml
    00:00 - Title is "The Shield"
    00:00 - EpisodeTitle is "Jailbait"
    00:00 - Description is "The Strike Team figures out its next step concerning Int
    ernal Affairs' investigation of Lem; Tina goes under cover to bust a sex traffic
    king ring; Claudette reopens one of Dutch's cases."
    00:00 - Date is "13173"
    00:00 - Time is "10800"
    00:00 - OriginalAirDate is "13172"
    00:00 - CallSign is "FX"
    00:00 - Name is "FX"
    00:00 - StartDate is "13173"
    00:00 - StartTime is "10798"
    00:00 - Duration is "3840"
    00:00 - entering setinfo
    00:00 - Date & Time is "01/24/2006 10:00PM"
    00:00 - the recording actually started at "01-24-2006 09.59 PM"
    00:00 - and the original air date is "01-23-2006"
    00:00 - notme is ": {"} < >"
    00:00 - entering build_text_from_xml
    00:00 - entering uptmf2ty
    00:00 - fname4 is "part"
    00:00 - creating "c:/extracts/The Shield/{Jailbait}{01-23-2006}{FX}"
    00:00 - starting segment part00.ty
    ..........100 ..........200 ..........300 ..........400 ..........500
    ..........600 ..........700 ..........800 ..........900 ..........1000
    ..........1100 ..........1200 ..........1300 ..........1400 ..........1500
    ..........1600 ..........1700 ..........1800 ..........1900 ..........2000
    ..........2100 ..........2200 ..........2300 ..........2400 ..........2500
    ..........2600 ..........2700 ..........2800 ..........2900 ..........3000
    ..........3100 ..........3200 ..........3300 ..........3400 ..........3500
    ..........3600 ..........3700 ..........3800 ..........3900 ..........4000
    ..........4100

    00:10 - fname4 is "part"
    00:10 - creating "c:/extracts/The Shield/{Jailbait}{01-23-2006}{FX}"
    00:10 - starting segment part01.ty
    ..........4200 ..........4300 ..........4400 ..........4500 ..........4600
    ..........4700 ..........4800 ..........4900 ..........5000 ..........5100
    ..........5200 ..........5300 ..........5400 ..........5500 ..........5600
    ..........5700 ..........5800 ..........5900 ..........6000 ..........6100
    ..........6200 ..........6300 ..........6400 ..........6500 ..........6600
    ..........6700 ..........6800 ..........6900 ..........7000 ..........7100
    ..........7200 ..........7300 ..........7400 ..........7500 ..........7600
    .........

    00:33 - end of file or invalid segment - leaving tmf2fsid
    00:33 - entering weightgain
    00:33 - entering cleanup

    Anyone run into this before or am I doing something wrong? I've tried Filezilla and Ftp Voyager on an XP box. Not using passive mode for transfer so at this point I'm stumped.
    Life should NOT be a journey to the grave with the intention of arriving safely in an attractive and well preserved body, but rather to skid in sideways - a cigar in one hand - a Martini in the other - body thoroughly used up, totally worn out, and screaming - WOO HOO! What a ride!

    1 HDVR2, 500 Gb, hacked, 6.3e
    1 R10 Prom Mod 500 Gb 6.3e
    1 HR20-100 Stock
    1 HR21-100 Stock

  14. #1124
    Join Date
    Nov 2004
    Location
    Gurnee, IL
    Posts
    2,385
    I assume the reason you're doing this is because you're archiving the .tmf files? (Otherwise you'd just download in .ty format.)

    I've encountered errors with tmf2ty, but I didn't really bother troubleshooting. I just renamed the .tmf to .tar, extracted all the part files, and used copy /b to combine them.
    --
    Christopher D. Heer
    Quote Originally Posted by Oscar Wilde
    Perhaps, after all, America never has been discovered. I myself would say that it had merely been detected.

  15. #1125
    Join Date
    Feb 2005
    Location
    Bergen County
    Posts
    1,307
    I've done that and it works but I can't understand why this format doesn't work. I've used:

    /B C:\part00.ty+C:\part01.ty+C:\part02.ty c:\combined.ty

    and it works fine but tmf2ty just craps out on the last part and I can't figure this out. Must be the engineer in me, if it doesn't work I like to know why. Oh well, back to the drawing boards....
    Life should NOT be a journey to the grave with the intention of arriving safely in an attractive and well preserved body, but rather to skid in sideways - a cigar in one hand - a Martini in the other - body thoroughly used up, totally worn out, and screaming - WOO HOO! What a ride!

    1 HDVR2, 500 Gb, hacked, 6.3e
    1 R10 Prom Mod 500 Gb 6.3e
    1 HR20-100 Stock
    1 HR21-100 Stock

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •