Page 1 of 3 123 LastLast
Results 1 to 15 of 44

Thread: Building MFS from scratch on an S2

  1. #1
    Join Date
    Jan 2002
    Posts
    1,777

    Building MFS from scratch on an S2

    A long time ago, d18c7db (evil_alex) posted instructions on how to use the fsmake and load-all.tcl utilities bundled with SA software v1.3 to construct an arbitrarily small (or large) MFS partition from scratch.

    Times have changed, the TiVo software distribution has changed, and exporting/importing tystreams is no longer the hard part of the process. I am posting some utilities that may allow an intermediate or advanced S2 user to experiment with this himself.

    This is for development purposes only; I am not attempting to provide a complete solution. No warranty, no assurance that this will work at all, yadda yadda. WORKSFORME.

    The prerequisities for this project were:

    1) A method of extracting and re-importing the following parts of the MFS hierarchy:

    /Resource/LoopSet
    /Resource/VideoClip (seems to be contained entirely within the LoopSet)
    /Genre
    /Component

    I have modified mbm's dumpobj-tcl script to do this by traversing the filesystem and invoking mfs_export when appropriate.

    If you notice any parts of the hierarchy I am missing, let me know.

    2) Slice files for the software version you are trying to install. They can be captured during a phone call or possibly from a satellite download.

    3) An image of a blank MFS filesystem, freshly initialized with fsmake. Included.

    4) A copy of a working TiVo root filesystem and kernel image. 3.1u5 makes the job easier, and since you aren't starting myworld there's a good chance it will work on the newer DTiVo boards.

    5) Another drive or another S2 TiVo, in working order, which has been compromised with BASH_ENV or a PROM hack, from which you can extract the needed resources.

    6) A tar binary for the TiVo.

    The basic steps I took were:

    0) Read the original d18c7db document several times. It is included in the attachment.

    1) Extract the resources from a working TiVo setup.

    Boot the working TiVo and run resource_dump.tcl. It will create a tarball called resources.tgz which contains everything needed to reconstruct the necessary resources. Copy this tarball to your PC or something.

    2a) Create a blank MFS filesystem.

    I have done this and included it in the archive. I followed the d18c7db instructions up to "fsmake" on my S1, then messed with mfstools until I got the byte order right in the backup.

    2b) mfsrestore the blank MFS filesystem to an empty drive

    Use -x if you wish, or expand it later. It will fit on very small drives, which was the intent of this project. It will also fit on the stock 40GB drive and allow you to make larger root/swap partitions for hacking purposes, at the expense of recording space. I don't know of any reason why it wouldn't work with, say, a 200GB drive, if you are equipped for LBA48. This has not been tested.

    3) Populate the non-MFS partitions.

    This backup image ONLY contains hda10 and hda11; all other partitions are blank. Therefore you must populate hda6/hda7 with the kernel/root, mkswap on hda8, and write a valid bootpage to hda. Format /var while you're at it, since rc.sysinit isn't going to be running quite yet. Although my configuration was significantly more complex, you should be able to get away with a basic BASH_ENV hack, as long as you do not allow the virgin rc.sysinit to start myworld (don't fork bash into the background).

    mke2fs -O none /dev/hda7
    mke2fs -O none /dev/hda9
    mkswap -v0 /dev/hda8
    dd if=31u5kern.img of=/dev/hda6
    bootpage ... blah blah

    4) Start up the TiVo and get MFS running.

    Power up the TiVo. Get MFS running. Something like:

    export TIVO_ROOT=""
    export MFS_DEVICE=/dev/hda10
    mount -n /dev/hda9 /var
    /tvbin/switcherstart -m

    Other environment variables might be necessary. Maybe not. You'll find out.

    5) Check to make sure MFS is okay.

    Start up tivosh and mls /. Ensure that MFS is empty and accessible. If you're paranoid you might want to run fsfix - see rc.sysinit for the syntax.

    6) Install the resources.

    Transfer resources.tgz to this unit, unpack, and install:

    cd /tmp
    tar zxvf resources.tgz
    export TIVOSH_POOLSIZE=16777216
    cd work
    tivosh resource_create.tcl

    You may safely ignore any warnings about the poolsize being a few hundred bytes too big.

    There's probably a way to use cpio to unpack the archive if you don't have tar. I don't remember it offhand.

    <split here to make the forum software happy>

  2. #2
    Join Date
    Jan 2002
    Posts
    1,777
    7) Insert the TiVo software into MFS.

    Transfer the slices to this drive. The slices I used for 3.1.1b were:

    GZbin-16099016-1.slice GZetc-16099018-1.slice GZkernel-16099020-1.slice GZlib-16099022-1.slice GZprom-16099024-1.slice GZsbin-16099026-1.slice GZtvbin-16099028-1.slice GZtvlib-16099030-1.slice swsystem-16099034-1.slice utils-16099032-1.slice

    gunzip if necessary.

    Use the included dbload.tcl script to load these slices into MFS:

    cd /tmp
    work/dbload.tcl *.slice

    I got several errDbNotFound errors which did not happen on my normal 3.1U5 box, however they did not cause any problems (yet?).

    9) Start installation.

    Use /tvbin/installSw.tcl <version> to install the new software. My normal (clumsy) procedure for this is to copy it to /tmp, comment out the reboot line, and then fix up my bootpage after it runs so that the new software does not lock me out. Do whatever best suits your method of compromise. Also, make sure the software doesn't misdetect your TiVo root filesystem and clobber the root that you are running from. If it does... have a backup handy. Since most of the process executes from the utils slice, you will need to unpack that slice if you want to step through it by hand or change things (like commenting out bootpage references).

    The version number can be found by doing an mls on /SwSystem after the slices have been loaded. In my case it was 3.1.1b-02-2-151. If /SwSystem is empty you are probably missing some really important resources.

    10) Reboot or switcherstart -k and restart the TiVo software.

    Obviously you will need to reboot if you are installing a software version which uses an incompatible kernel, for instance if you are upgrading from 3.1 to 4.0. At this point the process should be complete. If you're using monte, BASH_ENV, etc. do any necessary fixups now.

    I am currenting running with a 1GB MFS media region, on a very small drive. My status screen confirms that my SW version is 3.1.1b and my recording capacity is 0 hours.

    Nagging questions which I haven't figured out yet:

    1) How do you adjust the "used" tystream size? mfs_stdinsert does not do this so I need to use mfsbackup -T to correctly back up the inserted streams. Edit: this appears to be part of the mfs inode structure, but nobody knows of a safe way to change it.

    2) Why do I get an error 0xffff when I try to delete MFS objects? This has got to be something obvious I'm overlooking. Edit: dbobj $obj markasrubbish, or MfsRubbishTree.

    3) How do I tell the difference between a tyfile and a tystream object? Edit: mfs streamsize

    4) How do I initialize a tcl array, which may or may not exist, to zero elements? set then unset is very ugly.

    5) Are there going to be problems in the long term by using the v1.3 fsmake binary to generate an MFS filesystem that a 3.x or 4.x box uses?

    6) Is there anything else I'm leaving out or installing incorrectly that will impact the setup's stability?

    Any useful feedback would be appreciated.

    Everything in the archive worked for me at one point but should be considered untested at the moment.

    Edit: This has been tested successfully with 3.1.1b and 4.0 so far. Hint: do not run a 0-hour box; it will not work for long. Use mfsadd or something to expand it.

    Edit 2: I have run into some trouble when the number of "used" chunks in the loopset stream is too large (i.e. does not match the actual length of the loopset data). This seems to happen most often on 4.x and 5.x. If you are seeing weird tverr log entries or reboots associated with the rendering of loopsets, this may be causing your problem.

    Edit 3: Scroll down to Jamie's post to grab the latest versions of the resource_dump stuff.
    Last edited by alldeadhomiez; 07-23-2005 at 02:50 PM.

  3. #3
    Join Date
    Mar 2002
    Posts
    1,335
    Quote Originally Posted by alldeadhomiez
    1) How do you adjust the "used" tystream size? mfs_stdinsert does not do this so I need to use mfsbackup -T to correctly back up the inserted streams. Edit: this appears to be part of the mfs inode structure, but nobody knows of a safe way to change it.
    change line 5 of support.tcl from

    set fsid [mfs streamcreate 131072 $y]

    to

    set fsid [mfs streamfill 131072 $y]

    and used size will equal allocated size


    cool util, makes the process SO much easier
    ---
    Give a man a fish and he will eat for a day. Teach a man to fish and he will sit in a boat all day and drink beer

  4. #4
    Join Date
    Mar 2002
    Posts
    99
    alldeadhomiez - your pm folder is full.

    It is hard to reproduce your work without fsmake and the slices. Care to share?
    -- Dennis

    100 Hr Series 2 (development) Lifetime
    120 Hr Series 2 DTiVo (mine) Zipper-ed
    120 Hr Series 2 DTiVo (wife's) Zipper-ed
    120 Hr Series 2 DVD-RW (wife's) unhacked

  5. #5
    Join Date
    Jan 2002
    Posts
    1,777
    Quote Originally Posted by n4zmz
    alldeadhomiez - your pm folder is full.

    It is hard to reproduce your work without fsmake and the slices. Care to share?
    fsmake came from TiVo software v1.3. You don't need it because I included a blank mfsbackup image in the archive - just byteswap accordingly for S1 or S2.

    The slices are the standard TiVo software slices which you can get from the usual sources. Names and such will be listed in your tclient.log.

  6. #6
    Join Date
    Jan 2002
    Location
    Sonoran Desert
    Posts
    2,829
    you know what would be trick, is to create stream files (e.g. tivosh's mfs streamcreate) in MFS without the use of tivosh.
    Before PMing me: Iím not your personal tech support. If you have a question, ask in public so I don't have to repeat if somebody else asks. If you want images or slices, use emule. I will ignore all support PMs.

    Sponsor a vegetarian! I have taken the pledge, how about you?

  7. #7
    Join Date
    Jan 2002
    Posts
    1,777
    Quote Originally Posted by AlphaWolf
    you know what would be trick, is to create stream files (e.g. tivosh's mfs streamcreate) in MFS without the use of tivosh.
    I believe the following items would be necessary or helpful in pulling this off:

    descriptions of some of the undocumented "fill" fields in the mfs inode
    information on the structure of the zone map data
    information on how buddy maps work
    information on what the first 1000-odd sectors in mfs, between the superblock and the inode zonemap, are used for

    It seems like your intent would be to completely remove tcl from the process, so we will also have to figure out what is involved in creating and linking tyDb objects into the database, indexing them, and adding dirents. With any luck, fsfix and mfscheck will noisily correct our mistakes and tell us what we are looking for.

    It would appear that several of the "old school" TiVo hackers have access to some of this information but have not published it.

  8. #8
    Join Date
    Jan 2002
    Location
    Sonoran Desert
    Posts
    2,829
    Quote Originally Posted by alldeadhomiez
    I believe the following items would be necessary or helpful in pulling this off:

    descriptions of some of the undocumented "fill" fields in the mfs inode
    information on the structure of the zone map data
    information on how buddy maps work
    information on what the first 1000-odd sectors in mfs, between the superblock and the inode zonemap, are used for

    It seems like your intent would be to completely remove tcl from the process, so we will also have to figure out what is involved in creating and linking tyDb objects into the database, indexing them, and adding dirents. With any luck, fsfix and mfscheck will noisily correct our mistakes and tell us what we are looking for.

    It would appear that several of the "old school" TiVo hackers have access to some of this information but have not published it.
    If anybody could do it, tridge could...but he hasn't been around since the pre-ddb days

    But yeah, tivosh gets on my nerves.
    Before PMing me: Iím not your personal tech support. If you have a question, ask in public so I don't have to repeat if somebody else asks. If you want images or slices, use emule. I will ignore all support PMs.

    Sponsor a vegetarian! I have taken the pledge, how about you?

  9. #9
    Join Date
    Sep 2001
    Posts
    458
    I'm trying to move some loopsets from one box to another, I can't seem to get it to work. Everything looks like it should be working. It just doesn't create the LoopSet tyDb for anything to be inserted into. Here is a snippet of the resourse_create.tcl file and what it returns. Any ideas what could be going wrong?

    Code:
     #!/tvbin/tivosh
    source "support.tcl"
    set db [dbopen]
    puts "Creating object: /Resource/LoopSet/GreenBasic_3"
    if [catch {
      RetryTransaction {
        set LS_0 [db $db create LoopSet]
        set id_LS_0 [list [dbobj $LS_0 fsid] [dbobj $LS_0 subobjid]]
        dbobj $LS_0 set ServerVersion 2
        dbobj $LS_0 set Name {GreenBasic_3}
          set LSC_0 [db $db createsub LoopSetClip $LS_0]
          set id_LSC_0 [list [dbobj $LSC_0 fsid] [dbobj $LSC_0 subobjid]]
          dbobj $LSC_0 set Entrance 1000
          dbobj $LSC_0 set Exit 1001
            set VC_0 [db $db create VideoClip]
            set id_VC_0 [list [dbobj $VC_0 fsid] [dbobj $VC_0 subobjid]]
            dbobj $VC_0 set ServerVersion 2
            set fsid [tycreate "GreenBasicLoop.mpv"]
            lappend inserts "$fsid GreenBasicLoop.mpv"
            dbobj $VC_0 set File $fsid
            dbobj $VC_0 set Name {GreenBasicLoop.mpv}
            dbobj $VC_0 set ServerId {13245010}
          dbobj $LSC_0 set VideoClip $VC_0
        dbobj $LS_0 set State $LSC_0
          set LSC_1 [db $db createsub LoopSetClip $LS_0]
          set id_LSC_1 [list [dbobj $LSC_1 fsid] [dbobj $LSC_1 subobjid]]
          dbobj $LSC_1 set Entrance 5
          dbobj $LSC_1 set Exit 1000
            set VC_1 [db $db create VideoClip]
            set id_VC_1 [list [dbobj $VC_1 fsid] [dbobj $VC_1 subobjid]]
            dbobj $VC_1 set ServerVersion 2
            set fsid [tycreate "GreenBasicDrawOn.mpv"]
            lappend inserts "$fsid GreenBasicDrawOn.mpv"
            dbobj $VC_1 set File $fsid
            dbobj $VC_1 set Name {GreenBasicDrawOn.mpv}
            dbobj $VC_1 set ServerId {13245012}
          dbobj $LSC_1 set VideoClip $VC_1
        dbobj $LS_0 set Trans $LSC_1
        dbobj $LS_0 set ServerId {13840307}
        dbobj $LS_0 set Port 5
      }
    }] {
      puts "Creation failed"
    }
    dbclose $db
    tyinsert $inserts
    puts "Finished."
    [/var/dump/work]: tivosh test.tcl
    Creating object: /Resource/LoopSet/GreenBasic_3
    creating stream of size 8: fsid 398925
    creating stream of size 8: fsid 398927
    Creation failed
    inserting stream GreenBasicLoop.mpv into fsid 398925
    ERROR: Didn't find fsid=398925!
    while executing
    "exec ./mfs_stdinsert $fsid $fname"
    (procedure "tyinsert" line 6)
    invoked from within
    "tyinsert $inserts"
    (file "test.tcl" line 46)

  10. #10
    Join Date
    Oct 2003
    Location
    Australia
    Posts
    7

    freshly_initialized_mfs_s2.mfs size

    Hi,

    Love the toolkit. Great work! This has worked perfectly for us doing series 1 (and even a bit of series 2) stuff in Australia. I've been trying to do an image with a 128MB partition 2 (for dedicated hacks).

    I've created the mfs with fsmake, but I can't get the resulting mfstools backup down anywhere near the size of yours. all partitions dd'd with /dev/zero except mfs.

    Any pointers?

    Cheers,

    Ed

  11. #11
    Join Date
    Jan 2002
    Posts
    1,777
    Quote Originally Posted by ozTiVoEd
    Love the toolkit. Great work! This has worked perfectly for us doing series 1 (and even a bit of series 2) stuff in Australia. I've been trying to do an image with a 128MB partition 2 (for dedicated hacks).

    I've created the mfs with fsmake, but I can't get the resulting mfstools backup down anywhere near the size of yours. all partitions dd'd with /dev/zero except mfs.
    Try zeroing the entire drive (or all mfs partitions) before you run fsmake. mfstools is probably backing up leftover "junk" that fsmake did not erase; as you can see, the uncompressed image size is still very large despite that fact that it is an empty filesystem.

  12. #12
    Join Date
    Nov 2002
    Location
    New York City, NY, USA
    Posts
    296
    ADH, can't thank you enough for your method. Unfortunetly when trying to run the resource dump script on my 4.1b SA unit, I get the following errors:

    Code:
    invalid attribute: 0x2f0013
        while executing
    "dbobj $obj get $attr"
        (procedure "dump" line 43)
        invoked from within
    "dump $f $obj "{no parent}" 10 "    ""
        ("uplevel" body line 19)
        invoked from within
    "uplevel $body"
        invoked from within
    "transaction {uplevel $body}"
        (procedure "RetryTransaction" line 5)
        invoked from within
    "RetryTransaction {
        set obj [db $db open $x]
        set fsid [dbobj $obj fsid]
        set subid [dbobj $obj subobjid]
    
        # this check is redundant but ..."
        ("foreach" body line 2)
        invoked from within
    "foreach x $list {
      RetryTransaction {
        set obj [db $db open $x]
        set fsid [dbobj $obj fsid]
        set subid [dbobj $obj subobjid]
    
        # this che..."
        (file "resource_dump.tcl" line 155)
    Any insight you can provide would be highly helpful.

    Thanks in advance,
    Justin

  13. #13
    Join Date
    Sep 2001
    Location
    West of Bermuda
    Posts
    1,017
    ok, here are some additional notes from my experience with this. i have a 0 hour hdvr2 running 3.1.1c (i've got all of the 3.1.1e slices except tvlib, grrrrrr) that seems to come up fine, though i haven't tried any live buffering or anything with it (doesn't even have sat cables attached). one of the cool things about this config (if it isn't obvious) is that it only uses one mfs pair. of course, as adh said, a 0 hour tivo isn't going to be good for much, so you'll have to make one expansion, but you get make that expansion as you wish. for example, if you had a 160 gb drive, you could load a 0 hour backup, expand, and have a 160gb tivo that only uses two pairs. you could then, at a later date, get a 400gb drive, and expand to a 400gb a drive without running out of partitions.

    based on adh's step numbers, here are the extra things that i did (and/or should have done):

    1) for some reason, my mfs_export was bitching with crc errors when extracting the loopsets. after much consternation, i realized that tcl will bomb if an exec'd process puts *any* output on stderr, even if the return code is zero. to work around this, i nulled the first byte of the "crc mismatch" text in the mfs_export binary. the resources.tgz was created ok after that, and the loopsets looked fine.

    3) i assume adh just overlooked this when trying to "dumb down" his config for this post, but... you gotta have the contents of a root fs after the mke2fs in order for anything to happen on the tivo! i just used the u5fs.img that most folks have laying around from the original two partition monte days. i renamed the stock rc.sysinit and put a brief one based on adh's step 4, adding the bash execution:
    Code:
    /bin/bash </dev/ttyS2 &>/dev/ttyS2 &
    at the end. i also put:
    Code:
    export PATH=/sbin:/bin:/tvbin:.:/etc/rc.d
    at the beginning.

    also, this is a good time to mount the /var partition and place the slice files and resources.tgz onto it. you should put the tar binary in /bin of your chosen root partition as well. since you have /var mounted, create the following dirs under it: packages, run and tmp. you'll need those later, trust me.

    i used a killhdinitrd'd u5 kernel, nothing fancy

    4) after a false start with the missing /var/run and /var/tmp dirs, the eventswitcher started ok. however, i think, in retrospect, i should have set the date and time on the box here; a lot of stuff in mfs has the 1970 date, but it seems ok.

    9) after the install script completed (don't forget to remove/comment the reboot statement!), i used dd to copy the killhdinitrd'd kernel from the current to the new kernel partition, then appended the bash statement onto the rc.sysinit in the new root.

    i also installed the fakecall.tcl script to get it past the guided setup. i rebooted, came to the gs screen, ran the script, rebooted again, and it came up to directv central.

    at this point, i'll take a mfstools backup and probably give it to, ahem, someone who has a lot of backup images so that it can be tried by others. of course, thanks to the lovely (not) license with killhdinitrd, any image that gets onto a public server will be kernel-less or have a stock u5 kernel that will blow away the rc.sysinit changes upon boot.

    many thanks to adh and the others who toiled before him to bring this together. the process is really not difficult, and between adh's instructions and my notes, it's darned near turnkey.

    ronny

  14. #14
    Join Date
    Mar 2002
    Posts
    1,335
    Quote Originally Posted by ronnythunder
    of course, thanks to the lovely (not) license with killhdinitrd...
    feel free to donate your own exploit to the ebay leeches

    really, how hard is it to type one additional dd command?

  15. #15
    Join Date
    Jan 2002
    Posts
    1,777
    Quote Originally Posted by ronnythunder
    i assume adh just overlooked this when trying to "dumb down" his config for this post, but... you gotta have the contents of a root fs after the mke2fs in order for anything to happen on the tivo!
    That was included in "blah blah."

    Back in the BASH_ENV days, several extra steps were needed, which were well-documented elsewhere on the forum.

    at this point, i'll take a mfstools backup and probably give it to, ahem, someone who has a lot of backup images so that it can be tried by others. of course, thanks to the lovely (not) license with killhdinitrd, any image that gets onto a public server will be kernel-less or have a stock u5 kernel that will blow away the rc.sysinit changes upon boot.
    The rc.sysinit changes shouldn't really be necessary, if you have already completed the process. At this point you can restore a virgin 3.1.x root filesystem and kernel image, and distribute the 0-hour image.

    BTW: I'm not sure if I mentioned it, but on 4.0+ you will want to use known good slices to insert the loopsets, or you will need to find a way to adjust the "used blocks" field in the inode.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •