Results 1 to 14 of 14

Thread: MFS kernel module

  1. #1
    Join Date
    Sep 2003
    Location
    earth
    Posts
    44

    MFS kernel module

    Well, initial progress has a 2.4.x kernel module coded ... (this originated here...)

    I can successfully process the various headers and generate proper inodes ...
    Code:
    tivo-hack:/usr/src/linux/fs/mfs# cp /usr/src/linux/fs/mfs/mfs.o /lib/modules/2.4.27.tivo2/kernel/fs/mfs/; insmod mfs tivo_hda=hdc; mount /dev/hdc10 /mnt/test/
    Using /lib/modules/2.4.27.tivo2/kernel/fs/mfs/mfs.o
    tivo-hack:/usr/src/linux/fs/mfs# ls -al /mnt/test/
    total 0
    dr--------  0 root root     1460 2005-12-12 04:47 Anchor
    -r--------  0 root root    10056 2005-12-11 10:43 ApgBoot
    dr--------  0 root root       48 2005-12-11 10:44 ApgNetwork
    dr--------  0 root root       16 2005-12-05 17:02 AreaCode
    dr--------  0 root root       32 2005-12-05 18:25 AuxInfo
    dr--------  0 root root        4 2005-12-06 22:33 AvalanchePP
    -r--------  0 root root 10485760 2004-09-22 09:35 cachefile0.trp
    -r--------  0 root root 10485760 2004-09-22 09:35 cachefile1.trp
    dr--------  0 root root       52 2005-12-05 17:03 CaptureRequest
    dr--------  0 root root       44 2005-12-08 01:06 Clips
    dr--------  0 root root       40 2004-10-30 17:21 Component
    dr--------  0 root root       56 2004-10-30 17:44 Config
    dr--------  0 root root       16 2005-12-06 02:31 CorrelationIndexPart
    dr--------  0 root root      244 2005-12-12 05:34 Database
    dr--------  0 root root      932 2005-12-12 02:31 DataSet
    dr--------  0 root root     2476 2004-10-30 17:20 Genre
    dr--------  0 root root      372 2005-12-12 12:49 GuideIndexV2
    dr--------  0 root root        4 2005-12-12 12:49 GuideIndexV2.temp
    dr--------  0 root root     1096 2005-12-12 02:31 LeadGeneration
    dr--------  0 root root     1092 2005-12-11 20:16 LinkTag
    dr--------  0 root root       52 2004-10-30 17:26 LogoGroup
    dr--------  0 root root       20 2005-12-06 22:35 MenuItem
    dr--------  0 root root     1252 2005-12-12 02:31 Package
    dr--------  0 root root       32 2005-12-06 08:39 Person
    dr--------  0 root root       12 2005-12-05 17:55 Preference
    dr--------  0 root root      284 2005-12-08 01:01 Recording
    dr--------  0 root root       60 2004-10-30 17:14 Resource
    dr--------  0 root root      364 2005-12-12 07:50 Rubbish
    dr--------  0 root root     2116 2005-12-12 09:06 Schedule
    dr--------  0 root root       76 2005-12-05 17:56 SeasonPass
    dr--------  0 root root     5420 2005-12-12 09:11 Server
    -r--------  0 root root    44828 2005-12-12 16:30 Setup
    dr--------  0 root root      388 2005-12-12 02:31 Showcase
    dr--------  0 root root      216 2005-12-12 12:29 ShowcaseIndex
    dr--------  0 root root        4 2005-12-12 12:29 ShowcaseIndex.temp
    dr--------  0 root root       36 2005-12-05 16:39 Star
    dr--------  0 root root      300 2005-12-11 10:43 State
    dr--------  0 root root       28 2005-12-05 16:24 StationTms
    dr--------  0 root root      112 2005-12-09 00:00 SwModule
    dr--------  0 root root       44 2005-12-08 03:04 SwSystem
    dr--------  0 root root       52 2004-10-30 17:20 Table
    dr--------  0 root root       84 2005-12-06 21:09 Theme
    dr--------  0 root root        4 2005-12-12 16:29 tmp
    dr--------  0 root root       28 2004-10-30 17:37 User
    I also implemented what I think is "proper" file access ... (unverified correctness at this point)
    Code:
    tivo-hack:/usr/src/linux/fs/mfs# hexdump /mnt/test/ApgBoot
    0000000 0000 2e01 0000 4827 2400 2400 4f00 0000
    0000010 0000 0000 0000 0b00 1000 0800 0000 0000
    0000020 1180 0c00 0100 4e1b 0000 0c00 2000 2000
    0000030 4e00 0000 0000 0000 0000 0d00 1100 0800
    0000040 0500 787c 1000 0800 0000 0062 2000 2000
    0000050 4e00 0000 0000 0000 0000 0e00 1100 0800
    0000060 0500 2a82 1000 0800 0000 0142 2000 2000
    0000070 4e00 0000 0000 0000 0000 0f00 1100 0800
    0000080 0500 dc87 1000 0800 0000 0262 2000 2000
    0000090 4e00 0000 0000 0000 0000 1000 1100 0800
    00000a0 0500 8e8d 1000 0800 0000 0342 2000 2000
    00000b0 4e00 0000 0000 0000 0000 1100 1100 0800
    00000c0 0500 4093 1000 0800 0000 0462 2000 2000
    00000d0 4e00 0000 0000 0000 0000 1200 1100 0800
    00000e0 0500 f298 1000 0800 0000 0542 2000 2000
    00000f0 4e00 0000 0000 0000 0000 1300 1100 0800
    0000100 0500 a49e 1000 0800 0000 0662 2000 2000
    <snipped>
    performance is actually VERY good (using: ls -Ral, hexdump of a large "file", etc) ...

    At this point, I'm thinking I need some volunteers for code review and/or testing ... this means someone who understands MFS to a level that can verify that what I've got is sound, and also perhaps benchmark the use of this fs module versus pure user level tools ... cwingert?, jamie?, other? ... PM please if interested. I have a bit of cleanup/commenting todo before I can provide a patch, but sure would appreciate a look over.

    Current Status:
    Dev done against a single S2 drive (aka no swapping details)
    Written to a v2.4 kernel

    ToDo:
    Fixup listing bugs (e.g. "total 0" ...)
    Test against a set of S1 drives (probably using bswap kernel option)
    Port to v2.6 kernel (possibly submitting to kernel.org as well)
    Implement "write" support" (actually might not be too bad)
    Avoid memcpy during inode and directory access (should be easy and bump performance)
    "MFSck" ability? (aka use of "backup" sectors to "repair"?)
    <lotsa more>

    My end goal is to provide kernel level access to the FS data (i.e. be fast due to kernel block caching ...), and thereby allow user level tools (aka ls, ncftp, nbd, etc) to be used in place of "special" apps that currently exist.
    Last edited by DeadSock; 12-27-2005 at 06:06 PM.
    Give a man a link and you answer his question of the day. Teach a man to "search" and he can answer his own damn question.

  2. #2
    Join Date
    Feb 2005
    Posts
    41
    Hi,
    I think making a fs module makes sense to use it with other services and am very impressed with what you've done so far! I'm just surprised that other developers hadn't pursued this approach such a long time ago. Just a question though, would you still need separate tools to read/write the databases, records, and resources within the files? Anyway, good work!
    -- S8C

  3. #3
    Join Date
    Mar 2002
    Posts
    1,335
    Quote Originally Posted by santa8claws
    Hi,
    I'm just surprised that other developers hadn't pursued this approach such a long time ago.
    yah, like 'bout 4 years ago
    Quote Originally Posted by santa8claws
    Hi,
    would you still need separate tools to read/write the databases, records, and resources within the files?
    reading's easy, writing is dangerous if dependencies aren't updated correctly

  4. #4
    Join Date
    Sep 2003
    Location
    earth
    Posts
    44
    Quote Originally Posted by santa8claws
    Just a question though, would you still need separate tools to read/write the databases, records, and resources within the files?-- S8C
    Currently it treats all tyFile, tyStream, tyDb types as S_IFREG ...
    and like any filesystem, tools have to exist for the data (aka vi for text files, gimp for jpegs, etc).

    Obviously tools that already know how to deal with raw files just work (simple things like strings, hexdump, etc as well as more sophisticated tools like samba, nfs, nbd, etc)

    The point is the low level gore about reading (AND writing) from/to the disk is handled by the kernel (including the page/buffer caching it provides).
    Give a man a link and you answer his question of the day. Teach a man to "search" and he can answer his own damn question.

  5. #5
    Join Date
    Aug 2004
    Posts
    4,075
    Quote Originally Posted by DeadSock
    At this point, I'm thinking I need some volunteers for code review and/or testing ... this means someone who understands MFS to a level that can verify that what I've got is sound, and also perhaps benchmark the use of this fs module versus pure user level tools ... cwingert?, jamie?, other? ... PM please if interested. I have a bit of cleanup/commenting todo before I can provide a patch, but sure would appreciate a look over.
    I'd be happy to review code and try things out. I'm no linux files system expert, but I know the basics of VFS, etc. It's nice to see someone with an idea actually willing to try to implement it.

    I must admit that I'm still sceptical that this offers many advantages over the current software architecture. I doubt there is going to be a lot of interest in rewriting the user level apps to go through a file system interface rather than the current mfs library. The apps still need to groke tydb's, tystream's, brf's, etc, which IMHO, is the hard part.

    If you can get write support working, that would offer something new. I suspect this is not as easy as you think, but maybe you've dug deeper into the block/inode allocation management data structures than I have. Writing on a live tivo is also problematic unless you know how to synchronize access with the tivo processes to avoid transient inconsistent file system states.

  6. #6
    Join Date
    Sep 2003
    Location
    earth
    Posts
    44
    Quote Originally Posted by Jamie
    I'd be happy to review code and try things out. I'm no linux files system expert, but I know the basics of VFS, etc. It's nice to see someone with an idea actually willing to try to implement it.
    yea ... that is sorta novel

    I must admit that I'm still sceptical that this offers many advantages over the current software architecture. I doubt there is going to be a lot of interest in rewriting the user level apps to go through a file system interface rather than the current mfs library. The apps still need to groke tydb's, tystream's, brf's, etc, which IMHO, is the hard part.
    the primary advantage I can see is reusing the tools that already exist for dealing with filesystems. Re-doing "tivo specific" tools is a downside, but they should also be much easier (since they don't need to deal with inodes, page caching, etc).

    If you can get write support working, that would offer something new. I suspect this is not as easy as you think, but maybe you've dug deeper into the block/inode allocation management data structures than I have.
    It doesn't look "easy", but it's just bookeeping, so not impossible. Might need to add an ioctl interface inorder for "creation" of some inode types (aka creating a TyDB vs. a TyStream vs. a TyFile).

    Writing on a live tivo is also problematic unless you know how to synchronize access with the tivo processes to avoid transient inconsistent file system states.
    I agree, this may be problematic. Then again, if the tivo kernel is using the linux page cache, I may find that pages/buffers are marked "dirty" when tivoapp manipulates them ... definetly needs investigation before I'd try write support on a runnning tivo.
    Give a man a link and you answer his question of the day. Teach a man to "search" and he can answer his own damn question.

  7. #7
    Join Date
    Aug 2004
    Posts
    4,075
    Quote Originally Posted by DeadSock
    the primary advantage I can see is reusing the tools that already exist for dealing with filesystems. Re-doing "tivo specific" tools is a downside, but they should also be much easier (since they don't need to deal with inodes, page caching, etc).
    You can accomplish the same goal with a library. It doesn't have to be a kernel interface. For example, who cares if the call to read an MFS file is read() or mfs_read()?

    The buffer cache is actually a bad thing for streaming apps, where the data is only read once. It adds extra data copies. I'm not certain (I'd have to go through some strace dumps and read through the tivo ide driver), but I think that MFS access from the tivo software uses ioctl's that bypass the kernel buffer cache, for the most part. The buffer cache might be helpful for tydir traversal and tydb access, as those do tend to get re-read a lot.
    Last edited by Jamie; 12-28-2005 at 05:35 PM.

  8. #8
    Join Date
    Sep 2003
    Location
    earth
    Posts
    44
    Quote Originally Posted by Jamie
    You can accomplish the same goal with a library. It doesn't have to be a kernel interface. For example, who cares if the call to read an MFS file is read() or mfs_read()?

    The buffer cache is actually a bad thing for streaming apps, where the data is only read once. It adds extra data copies. I'm not certain (I'd have to go through some strace dumps and read through the tivo ide driver), but I think that MFS access from the tivo software uses ioctl's that bypass the kernel buffer cache, for the most part. The buffer cache might be helpful for tydir traversal and tydb access, as those do tend to get re-read a lot.
    I agree ... but ...

    To me, the "library" *should* be a fs module ... and user mode apps get the caching bonuses when they deal with tydir/tydb stuff. In looking at the code I have (mfstools and mfs-utils), there is little "library" shared between those toolsets. (not to mention the multitude of "tools" not implemented that come standard with most linux package sets).

    I've already given thought to having "special" ops sets for tystreams (to bypass page caching) ... perhaps make a tystream *be* a S_IFBLK, and then have the ops set for that bypass the std page/buffer caches ...
    Give a man a link and you answer his question of the day. Teach a man to "search" and he can answer his own damn question.

  9. #9
    Join Date
    Sep 2003
    Location
    earth
    Posts
    44
    Not to pik a fight ... but to clarify some points ...

    Quote Originally Posted by Jamie
    For example, who cares if the call to read an MFS file is read() or mfs_read()?
    LOTS of applications ... for example *any* existing linux app that deals with files. They know about read() ...

    I think that MFS access from the tivo software uses ioctl's that bypass the kernel buffer cache, for the most part.
    My perusal has me thinkin the same ... I might try building a custom tivo kernel that implements a "tivo_lock()/tivo_unlock()" where they do this. I'd be interested to see the overhead that would impose (without any external contention). Perhaps then apps that are async IO capable could behave nicely on a running tivo (and not interfere with tivoapp).
    Give a man a link and you answer his question of the day. Teach a man to "search" and he can answer his own damn question.

  10. #10
    Join Date
    Aug 2004
    Posts
    4,075
    Quote Originally Posted by DeadSock
    Not to pik a fight ... but to clarify some points ...


    LOTS of applications ... for example *any* existing linux app that deals with files. They know about read() ...
    I'm thinking about apps that actually do something useful with the data in MFS. Not hexdump, but things that grok the tydb schema, brf files, tystreams, etc. For these apps, IMHO, it does not matter whether you use a standard kernel interface like open()/read()/write() or a library interface. Perhaps our disagreement is simply that I don't see much usefulness in generic (non-tivo-smart) apps accessing MFS. I'll be happy to be proved wrong!

  11. #11
    Join Date
    Sep 2003
    Location
    earth
    Posts
    44
    Quote Originally Posted by Jamie
    I'm thinking about apps that actually do something useful with the data in MFS. Not hexdump, but things that grok the tydb schema, brf files, tystreams, etc. For these apps, IMHO, it does not matter whether you use a standard kernel interface like open()/read()/write() or a library interface. Perhaps our disagreement is simply that I don't see much usefulness in generic (non-tivo-smart) apps accessing MFS. I'll be happy to be proved wrong!
    ftp, ncftp, samba, nfs, nbd, cp, rcp ... using these "non-tivo-smart" apps can then open up the ability for "tivo-smart" apps then to work "outside the box".

    perl, python, etc might be useful for manipulating tyDB stuff ...

    just some off the top thoughts.

    (BTW, you should have a PM).
    Give a man a link and you answer his question of the day. Teach a man to "search" and he can answer his own damn question.

  12. #12
    Join Date
    Aug 2004
    Posts
    4,075
    Quote Originally Posted by DeadSock
    ftp, ncftp, samba, nfs, nbd, cp, rcp ... using these "non-tivo-smart" apps can then open up the ability for "tivo-smart" apps then to work "outside the box".
    You can do that with a library too. mfs-utils does exactly that through vserver.

    I've beaten this dead horse enough. I'm just being a curmudgeon. I commend you for doing something new, even if rc3105 says it's been done before.
    (BTW, you should have a PM).
    Thanks, I got it.

  13. #13
    Join Date
    Sep 2003
    Location
    earth
    Posts
    44
    Quote Originally Posted by Jamie
    You can do that with a library too. mfs-utils does exactly that through vserver.
    perl code can read/write to the fs with vserver? ... didn't realize it provided that ...
    Give a man a link and you answer his question of the day. Teach a man to "search" and he can answer his own damn question.

  14. #14
    Join Date
    Aug 2004
    Posts
    4,075
    Quote Originally Posted by DeadSock
    perl code can read/write to the fs with vserver? ... didn't realize it provided that ...
    Yes, perl code can talk through a socket to vserver.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •