Page 1 of 2 12 LastLast
Results 1 to 15 of 16

Thread: Replacing mangled MFS objects?

  1. #1
    Join Date
    Sep 2005
    Location
    Annapolis, MD
    Posts
    31

    Replacing mangled MFS objects?

    Ok, maybe an odd/stupid question:

    Any ideas about how to restore the contents of tyDb /SwSystem/ACTIVE? ...or opinions on the utter futility of such a move?

    I have twin dtivos (SD-DVR40) built from the same v6.4 InstantCake, full network connectivity, and a rudimentary knowledge of navigating MFS and its objects.

    Note this is not an emergency of any kind. I'm getting my recordings off this thing now and I'm prepared to flatline and restore. But I'm TiVo-curious. I'd like to learn something here. So if someone who knows this stuff better than I do has a few minutes to offer tips/pointers... Or if not, no sweat. Next post please.

    If you're still with me, a few things come to mind right off:

    1. Perhaps mfs_dumpobj to get objects from the good dtivo, but what can import those objects on the damaged dtivo? I do not yet know such things.

    2. What's the chances there are all manner of unique values (FSID's?) in these copied objects that aren't going to "line up" with objects on a different dtivo, and that by copying to the bad dtivo, I'm just going to bork the thing even further?

    3. There could be other MFS damage I haven't found yet. So if it's going to take countless hours, or I break it worse, I'm prepared to re-InstantCake this thing as soon as my backup is done (sometime tomorrow). I'm mostly curious if this kind of MFS repair is remotely an option. (Could save me having to restore 272GB of recordings...)

    For anyone interested, here's the full saga (beer optional, but recommended):

    Came inside, cold, tired and soggy Sunday evening after 5 hours of shoveling snow, only to find a very upset and confused TiVo. We'd had a power outage earlier, so thinking that's related. It's basically locked up. Even telnetting in and using the command line was very very slow. Managed an mfsassert -please, only to put the poor thing into a GSOD loop. Nice.

    So, switched to serial terminal to watch what's going on. fsfix runs, hits an UncorrectableError on the disk, says it completed with 2 warnings and no errors, then reboots. fsfix runs again, same results, reboots again. Lather, rinse, repeat.

    I'm feeling a bit lazy (see shoveling, above) and don't want to pull the drive. But how to stop a GSOD loop? Well, I'll tell ya one way to do it...

    I've got maybe a minute between first getting a serial console shell prompt and the inevitable reboot. So leisurely poking around and fiddling with things, not a viable option. I had a full backup of the ext2 filesystems from this TiVo, located on another system. So I walked the startup process, starting with inittab, and working into rc.sysinit and beyond. Turns out it's not that hard. Typed up the following in notepad and flung it through the serial terminal:

    Code:
    mount -o remount,rw /
    cd /etc/rc.d/
    mv StageD_PreMfs StageD_PreMfs.bak
    mv StageE_PreApplication StageE_PreApplication.bak
    mv StageF_ApplicationLaunch StageF_ApplicationLaunch.bak
    mv StageG_PostApplication StageG_PostApplication.bak
    (Disclaimer: I'm far from a TiVo expert. There's probably an easier/better way, but that worked for me and I still had a serial console shell prompt when the smoke cleared. Check where your serial shell gets invoked -- disable the wrong startup scripts and you won't have a shell prompt!)

    After one more reboot, the cycle of madness stopped and I could take my time poking around.

    First I played with fsfix and mfscheck options, but it seems clear that these tools don't get "intimate" with the hard drive -- they can do nothing about clearing hard errors, remapping, managing a defect list, etc. Really need to pull this drive and run some hard core diags... dang it. Mutter, cuss, pull drive...

    Ran Spinrite first. It made the handful of bad sectors go away, but couldn't recover the data from those sectors. After that, running DLDiag (WD 500gb Caviar Blue) said the drive was fine, no errors. (So I'm thinking this was not an actual hard media defect.)

    Now back into the TiVo (full startup still disabled) and more fun with fsfix and mfscheck. I started with fsfix -uncollide -rehash -salvage. That went much better than previous. I don't know how good (or bad) an idea running mfscheck -fix is, but I did, and it spewed scary stuff across the screen for about an hour. At least this time it completed without core dumping!

    Then I put all the startup stuff back that I disabled above, crossed my fingers and rebooted. After a few more spontaneous crashes, GSOD's, reboots... spitting and sputtering, it finally came to life! LiveTV is working, all recordings accounted for. Yay!!

    Except...

    It wont record anything. Can't view the ToDo list, grid or program info. Get "DVR Service Unavailable" and/or "Please Activate Service" messages. Grrr...

    At this point, I figure it's a salvage operation. Get the recordings off and re-InstantCake this thing, then put my custom stuff back on, then import the backed up recordings.

    Then I notice that TivoWeb and mfs_ftp aren't running. Tried to run both by hand, get pretty much the same error, which traces back to both programs trying to get the TiVo software version (Name[16]=6.4a-01-2-151) from MFS /SwSystem/ACTIVE.

    I managed to get mfs_ftp working by hard-coding the SW version instead of querying the missing /SwSystem/ACTIVE tyDb and I'm now 184gb into extracting 272gb of recordings.

    With a backup fully underway, I'm now wondering about the possibility of restoring this thing to full working condition by replacing the missing objects.

    Poking at MFS, I see right off:

    Code:
    LivingRoom:/var/mfs_ftp $ mfs_ls /
    dir: fsid=1 count=38
          fsid   type     name
          -----------------------------------
        670852   tyDir    Anchor
        674563   tyDb     ApgBoot
        670879   tyDir    ApgChannel
        670520   tyDir    AreaCode
        670624   tyDir    AuxInfo
        670711   tyDir    CaptureRequest
        670846   tyDir    Clips
        670432   tyDir    Component
        670521   tyDir    CorrelationIndexPart
        670463   tyDir    DataSet
        670464   tyDir    DataSetVersion
        670434   tyDir    Database
        670430   tyDir    DiskUsed
        673288   tyDir    GuideIndexV3
        674269   tyDir    GuideIndexV3.temp
        670791   tyDir    LeadGeneration
        670485   tyDir    LogoGroup
        670714   tyDir    MenuItem
        670794   tyDir    Package
        670458   tyDir    Person
        670444   tyDir    Preference
        670437   tyDir    Recording
        670428   tyDir    Resource
        670494   tyDir    Rubbish
        670431   tyDir    Schedule
        670509   tyDir    SeasonPass
        670427   tyDir    Server
          5995   tyDb     Setup
        670792   tyDir    Showcase
        671243   tyDir    ShowcaseIndex
        672673   tyDir    ShowcaseIndex.temp
        670839   tyDir    SpigotMap
        670516   tyDir    State
        670856   tyDir    StationTms
        670480   tyDir    SwModule
        670462   tyDir    Table
        670488   tyDir    TuikRes
        670880   tyDir    Uri
    Diff'ing against the sister dtivo, what's missing there (at least)is /Genre, /MessageItem, /SwSystem and /Theme.

    On the good dtivo:

    /Genre is empty. Non-critical.

    /MessageItem looks to be non-critical and has one entry. (System messages?)

    /SwSystem -- this contains two tyDb entries (same FSID) and would seem to be very critical. Seems to me this all comes down to replacing (or reconstructing) the tyDb object /SwSystem/ACTIVE, if that's even feasable. The mfs_dumpobj tool shows an awful lot of stuff in there, including possibly system-unique FSID's and MFS paths. (This is where I'm starting to feel a little stupid for even asking...)

    /Theme appears to contain Wishlist entries, presumably non-critical.

    So is there a hope of restoring the critical missing stuff and learning something in the process, or should I really just clobber and start fresh?

    Thanks in advance for any clues.

    -cw-
    ObBragline: 3 x Hughes SD-DVR40, 100GB, 250GB & 500GB. TivoWebPlus, mfs_ftp, MovieLoader, NCID, etc. But the best part of all is just having a bash prompt on a home appliance!

  2. #2
    Join Date
    Jun 2006
    Location
    Dougal County
    Posts
    1,007
    I guess you could try reloading the swsytem slice (see the files section for a link to 6.4a slices) and try an emergency reinstall (see /tvbin/installSw.itcl)

    I've nuked just about everything from MFS before (besides /Resource /Component and /SwSystem) with no ill-effects

  3. #3
    Join Date
    Aug 2004
    Posts
    4,075
    The other thing you could try is the mfs from scratch" approach. Copy all the resources from the good tivo and load them into the bad one. Not sure it would work, but if you are curious, it's an interesting path to go down.

  4. #4
    Join Date
    Sep 2005
    Location
    Annapolis, MD
    Posts
    31
    Thanks guys. I'm gonna try the more surgical "emergency reinstall" first. If that bombs, then either "mfs from scratch" or wimp out and re-instantcake.

    ...and a little update: Extraction was chugging along nicely last night, just as I said above. Then around 7-8PM, the thing reboots. And reboots again, and reboots again, etc.

    /var/log/tverr is full of pages and pages of stuff like this with every subsequent boot attempt:

    Code:
    Feb 10 03:14:26 (none) TvCylonDocument.C[180]: Failed to load document named "/s
    w_resource/lib/tvresource/RuntimeCeDifference_unpaid.brf"   Error 0x30001 (0x300
    01)   TvUrlResource BaseUrl is "shmem://TvShmemd"  Check to be sure:   If you're
     using TV_URL_RESOURCE_BASE that it points to a     valid directory   If you're
    on a load-all system that your document's ism is listed     in sw/data/common/ge
    n/import-processes/import-processes.list   If you've added/changed this Document
     in your tree check that     your
    Feb 10 03:14:26 (none) TvCylonDocument.C[180]: Failed to load document named "/s
    w_resource/lib/tvresource/RuntimeNetworkDifference_disabled.brf"   Error 0x30001
     (0x30001)   TvUrlResource BaseUrl is "shmem://TvShmemd"  Check to be sure:   If
     you're using TV_URL_RESOURCE_BASE that it points to a     valid directory   If
    you're on a load-all system that your document's ism is listed     in sw/data/co
    mmon/gen/import-processes/import-processes.list   If you've added/changed this D
    ocument in your tree check that
    Feb 10 03:14:26 (none) TvCylonDocument.C[180]: Failed to load document named "/s
    w_locale/lib/tvlocaleplugin/LocaleInfo.brf"   Error 0x30001 (0x30001)   TvUrlRes
    ource BaseUrl is "shmem://TvShmemd"  Check to be sure:   If you're using TV_URL_
    RESOURCE_BASE that it points to a     valid directory   If you're on a load-all
    system that your document's ism is listed     in sw/data/common/gen/import-proce
    sses/import-processes.list   If you've added/changed this Document in your tree
    check that     your content is in
    Etc...

    Is that bad or something? Is there a button on the remote I can push to fix it? 8-o

    Seriously, looks like the crash this time had something to do with this:

    Code:
    Feb 10 03:13:01 (none) TmkAssertionFailure[264]: :  (reorderIndexes, line 2502 (
    ))
    Feb 10 03:13:01 (none) Activity TvRecorderActivity[264]: Tmk Fatal Error: Activi
    ty TvRecorderActivity <264> strayed! (block timestamp 8875389644893)
    So once again, the puzzler of how to get out of the reboot loop and continue with extraction. (I can be a stubborn SOB...)

    In short, it went like this:

    Code:
    mount -o remount,rw /
    cd /etc/rc.d/
    mv StageD_PreMfs StageD_PreMfs.bak
    mv StageE_PreApplication StageE_PreApplication.bak
    mv StageF_ApplicationLaunch StageF_ApplicationLaunch.bak
    mv StageG_PostApplication StageG_PostApplication.bak
     
    cd /lib/modules
    insmod usbcore.o
    insmod usbnet.o
    insmod router.o
     
    #ifconfig eth0 192.168.100.7 broadcast 192.168.255.255 netmask 255.255.0.0
    /sbin/dhclient
     
    /tvbin/mfsd &
     
    /tvbin/tivosh /var/mfs_ftp/mfs_ftp.tcl &
    That allowed me to complete the extraction -- something that actually amazed me.

    So now it's all about MFS repair...

    -cw-
    ObBragline: 3 x Hughes SD-DVR40, 100GB, 250GB & 500GB. TivoWebPlus, mfs_ftp, MovieLoader, NCID, etc. But the best part of all is just having a bash prompt on a home appliance!

  5. #5
    Join Date
    Jun 2006
    Location
    Dougal County
    Posts
    1,007
    Quote Originally Posted by cwilkins View Post
    Is that bad or something? Is there a button on the remote I can push to fix it? 8-o
    looks pretty bad to me ; no magic button either :P

    I've only seen errors like when I had screwed with my boot params and accidentally set my root partition sw to a different version than what was in mfs. since those brf's are referenced by the 6.4a tivoapp, I doubt that's the case. looks like some good ole mfs corruption

    the brf's should be contained within the appropriate swsystem slice, so it should be theoretically possible to fix it

    my gameplan would be something along the lines of :

    create /test.conf with the following :
    Code:
    #!/bin/bash
    export TIVO_ROOT=
    export MFS_DEVICE=/dev/hda10
    export PATH=/bin:/sbin:/tvbin:/tivo-bin
    export TERM=xterm
    setsid bash --login -i < /dev/ttyS2 >& /dev/ttyS2
    setsid bash --login -i < /dev/ttyS2 >& /dev/ttyS2 &
    <insert networking stuff here if wanted>
    that'll startup serial bash immediately and not background it, in case you need to make immediate emergency repairs/changes to the root fs. then you can exit the shell, and another will spawn & bootup will continue

    stopping the boot as your doing at StageD is a good way to get access to mfs without loading tivoapp completely. from there you can load the appropriate slices into mfs, and run installSw.itcl to reinstall them

    there could be damage elsewhere in mfs, if so, you can also nuke just about everything through tivosh as I mentioned before, and attempt the sw reinstall

    if necessary, you could also apply the tivoapp patch I posted in this thread to bypass Guided Setup

  6. #6
    Join Date
    Jun 2006
    Location
    Dougal County
    Posts
    1,007
    hmmmm....

    now that I think about it, you may be able to fix everything by simply reloading the swsystem slice

    /SwSystem contains pointers to all kinds of objects including the brf files, and since you still have /TuikRes, you probably still actually have the brf files in mfs

  7. #7
    Join Date
    Sep 2005
    Location
    Annapolis, MD
    Posts
    31
    Yup, you're right -- it pretty much came down to reloading the swsystem slice.

    ...and it appears to be working now! One little bump in the plan is the DC- area snow storm. I get the thing booted finally and it's just sitting at "Acquiring Satellite... 0%" So explaining to the impatient wife -- "Honey, the good news is, I think the TiVo is fixed. The bad news is..." So, outside on a stepladder in a snow storm with the 12 ft piece of PVC pipe banging on the dish and the roof trying to knock the snow off (which of course, falls on me). Yay, signal again! Sometimes I feel like the geek version of Indiana Jones.

    What do DirecTV customers further north do? You know, where they get real snow? Heated dishes?

    Anyway, here's the recovery in a little more detail...

    First, the steps I already detailed to get the TiVo up with just networking and mfsd.

    Next, after failed efforts to D/L the slices from RapidShare (too busy), I found a version-specific "getslice" script on DVRUpgrade's site, picked that apart and figured out where to download the correct slices, also from their site. Sweet! (Not sure if it's ok or useful to post the actual URL's).

    Downloaded/unpacked slices.

    Code:
    $ ls -l
    total 18728
    -rwxr-xr-x  1 cwilkins cwilkins     1014 Sep 17  2008 dbload
    -rw-r--r--  1 cwilkins cwilkins 13169435 Sep 17  2008 GZcore-127004584-2.slice
    -rw-r--r--  1 cwilkins cwilkins  1354664 Sep 17  2008 GZhpk-Series2-127004588-2.slice
    -rw-r--r--  1 cwilkins cwilkins   922661 Sep 17  2008 GZkernel-Series2-127004586-2.slice
    -rw-r--r--  1 cwilkins cwilkins  3645079 Sep 17  2008 swsystem-127008031-2.slice
    -rw-r--r--  1 cwilkins cwilkins    38951 Sep 17  2008 utils-127004582-2.slice
    Ran dbload against just the swsystem slice. It appeared to work, but there was still no /SwSystem directory in MFS. That caused much head-scratching and trying to figure out how to create a stupid tyDir, but ultimately I just dbload-ed all the slices, rechecked /SwSystem and now that directory exists, and I have a tyDb named 6.4a-01-2-151 underneath!

    Lastly, there was the puzzle of how to recreate /SwSystem/ACTIVE... jt1134 had mentioned installSw.itcl, so I started poking around in there and ultimately found a juicy TCL fragement to steal and hack up for the purpose:

    Code:
    #!/bin/bash
    #
    # With any luck, this sets /SwSystem/ACTIVE properly
    #
    # (Pilfered from /tvlib/tcl/tv/SwSystem.itcl)
    #
    #\
    TIVO_ROOT=${TIVO_ROOT:-} MFS_DEVICE=${MFS_DEVICE:-/dev/hda10} DBLOAD_HANDCRAFT=true exec /tvbin/tivosh "$0" "$@"
    
    set name "6.4a-01-2-151"
    
    set db [dbopen]
    
    # First, deactivate any previously active software system
    #RetryTransaction {
    #    catch {
    #        set oldActive [db $db open /SwSystem/ACTIVE]
    #        dbobj $oldActive remove Active
    #    }
    #}
    
    # Now, activate the desired one.
    RetryTransaction {
        set sysHandle [db $db open "/SwSystem/$name"]
        dbobj $sysHandle set Active 1
    }
    
    dbclose $db
    Worked like a charm!

    Code:
    $ mfs_ls /SwSystem
    dir: fsid=682637 count=2
          fsid   type     name
          -----------------------------------
        682628   tyDb     6.4a-01-2-151
        682628   tyDb     ACTIVE
    After that, it was re-enable all the startup scripts, cross fingers, and reboot. And aside from snow on the dish, it's booted up nicely and been a good little dtivo! (So far, of course. Watching the logs...)

    Thanks again to jt1134 and Jamie and DDB. I cannot even imagine how hard (impossible?) efforts like this would be without helpful tips and tools from the folks here, and endless site searches via Google.

    -cw-
    ObBragline: 3 x Hughes SD-DVR40, 100GB, 250GB & 500GB. TivoWebPlus, mfs_ftp, MovieLoader, NCID, etc. But the best part of all is just having a bash prompt on a home appliance!

  8. #8
    Join Date
    Sep 2005
    Location
    Annapolis, MD
    Posts
    31
    ...just a quick followup, five days later and all is well.

    Oh, but I have discovered one important tidbit: If you have your dtivo recording from both tuners, importing a recording via MovieLoader/mfs_ftp, serving pages via TivoWeb and navigating the NowPlayinglist all at the same time, it just might give up and reboot.

    After much research, the solution to this problem would appear to be "don't do that."

    ...unless someone has been able to use setpri or other tools to more or less bulletproof against this? (Give tivoapp and friends top priority.) I'm already doing a setpri ts 0 $$ at the top of my rc.sysinit.author.

    One problem with that might be a similar problem we see with some of our systems at work. That is, Unix is good at allocating CPU as needed and you can twiddle priorities there so that nothing important gets completely starved for CPU, but I/O is another story. There is no simple way to manage I/O to ensure specific processes get I/O priority. And an IDE drive (no NCQ, etc.) just makes it worse. On a busy drive, a process can block for many seconds waiting for its turn to read/write.

    So no matter how "nice" you can make mfs_ftp and such, they can still starve tivoapp for I/O bandwidth and past a certain point, the box will give up and reboot. (I've noted that TiVos are so very good at rebooting!)

    That about right?

    -cw-
    ObBragline: 3 x Hughes SD-DVR40, 100GB, 250GB & 500GB. TivoWebPlus, mfs_ftp, MovieLoader, NCID, etc. But the best part of all is just having a bash prompt on a home appliance!

  9. #9
    Join Date
    Aug 2004
    Posts
    4,075
    Make sure mfs_ftp and/or tivoweb are running with the ts scheduler too. They may be altering their own priorities, or those of their children. I'm fairly certain mfs_ftp does this by default. I'm less sure about tivoweb.

    mfs_ftp, when run with the standard patches, also has a parameter that can be tweaked to insert usleeps between processing each chunk. This can help if it is dominating I/O. This is really down in mfs_uberexport. mfs_ftp just passes the flag on down.

    If I remember right, the tivo disk driver actually does have a mechanism to reserve bandwidth for specific processes.

  10. #10
    Join Date
    Jan 2004
    Location
    Noo Hampsha
    Posts
    767
    Quote Originally Posted by cwilkins View Post
    What do DirecTV customers further north do? You know, where they get real snow? Heated dishes?
    That's one option. I'll comment that the LNB coated in ice is much more of an issue for me than snow on the dish. Some people use dish covers, others swear by spraying the dish with Pam. (I've got a bottle of stuff meant to spray on snow shovels and snowblowers to make the snow slide off. It's some sort of wax-based product. Have always meant to try it on my dish but never got around to it.)

    My solution was to mount the dish within broom's reach of a second-floor window. Most of the time I don't need this - even in heavy snow storms. It's the sticky snow/ice that gives me headaches.
    Steve

  11. #11
    Join Date
    Jul 2001
    Posts
    19
    I have the same situation as CWilkins. I have a TCD 24000a that was running 7.x software and the applications partition got hosed. Does anyone have a lead on the slice files for the 7.x software that you could send me? I would like to try reloading the applications partitions like Cwilkins did.

    Unfortunately I cannot PM StanSimmons over at tivocommunity and that thread is now closed.

  12. #12
    Join Date
    Jul 2001
    Posts
    19

    does anyone know how to decrypt a ".slice.bnd" file

    I've made some progress. I've realized I can grab my own slice files for the software upgrade as they appear in /var/packages. The most important file (for me) is the swsystem-xxxxx.slice.bnd file. and it's corresponding SHA file (key). I can grab these files but I don't know how to apply the key to decrypt the ".bnd" file.

    Can someone please point me towards what will allow me to decrypt this file.

    Thanks

  13. #13
    Join Date
    Jun 2003
    Posts
    611
    The file /tvlib/tcl/tv/unbundler.tcl should clue you in on how to do it. I refer to it every time I have to decrypt a swsystem slice.

    -psxboy
    TCD652160 TivoHD
    1TB
    11.0n.J1-01-2-652

  14. #14
    Join Date
    Jul 2001
    Posts
    19
    Thanks,

    I will look at it tonight!

  15. #15
    Join Date
    Jul 2001
    Posts
    19

    unbundler.tcl

    Alright, I think I'm beginning to get it. Please let me know if I'm on the right track...

    unbundler.tcl must run whenever files are downloaded from the mothership. From the looks of it, it scrutinizes all the files by type and extracts them appropriately. So the easiest way to extract/decrypt my own slice & .bnd files would be to restore to an eairler backup on a spare hacked disk and then edit unbundler.tcl so that as it does its normal job it also writes its output to a directory of mine as well. I believe I would just need to edit the code here to include my directroy as well:

    putlog "Session unlocked"
    foreach file [glob *$Inc::TS_CRYPT_EXT] {
    set basefile [file rootname $file]
    putlog "Decrypting $file to $basefile"
    eval exec $Inc::TS_DECRYPT_PROG $sessionkey < $file > $basefile
    file delete $file
    Then I just force a call and wait for the output in my directory.

    Since I don't do tcl (until now) I will have to look at this tcl code a little more before I can make my mods.

    Thanks for the shove in the right direction.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •