MFS kernel module

[DeadSock]

Well, initial progress has a 2.4.x kernel module coded ... (this originated here... (http://www.dealdatabase.com/forum/showthread.php?p=244128))

I can successfully process the various headers and generate proper inodes ...

tivo-hack:/usr/src/linux/fs/mfs# cp /usr/src/linux/fs/mfs/mfs.o /lib/modules/2.4.27.tivo2/kernel/fs/mfs/; insmod mfs tivo_hda=hdc; mount /dev/hdc10 /mnt/test/
Using /lib/modules/2.4.27.tivo2/kernel/fs/mfs/mfs.o
tivo-hack:/usr/src/linux/fs/mfs# ls -al /mnt/test/
total 0
dr-------- 0 root root 1460 2005-12-12 04:47 Anchor
-r-------- 0 root root 10056 2005-12-11 10:43 ApgBoot
dr-------- 0 root root 48 2005-12-11 10:44 ApgNetwork
dr-------- 0 root root 16 2005-12-05 17:02 AreaCode
dr-------- 0 root root 32 2005-12-05 18:25 AuxInfo
dr-------- 0 root root 4 2005-12-06 22:33 AvalanchePP
-r-------- 0 root root 10485760 2004-09-22 09:35 cachefile0.trp
-r-------- 0 root root 10485760 2004-09-22 09:35 cachefile1.trp
dr-------- 0 root root 52 2005-12-05 17:03 CaptureRequest
dr-------- 0 root root 44 2005-12-08 01:06 Clips
dr-------- 0 root root 40 2004-10-30 17:21 Component
dr-------- 0 root root 56 2004-10-30 17:44 Config
dr-------- 0 root root 16 2005-12-06 02:31 CorrelationIndexPart
dr-------- 0 root root 244 2005-12-12 05:34 Database
dr-------- 0 root root 932 2005-12-12 02:31 DataSet
dr-------- 0 root root 2476 2004-10-30 17:20 Genre
dr-------- 0 root root 372 2005-12-12 12:49 GuideIndexV2
dr-------- 0 root root 4 2005-12-12 12:49 GuideIndexV2.temp
dr-------- 0 root root 1096 2005-12-12 02:31 LeadGeneration
dr-------- 0 root root 1092 2005-12-11 20:16 LinkTag
dr-------- 0 root root 52 2004-10-30 17:26 LogoGroup
dr-------- 0 root root 20 2005-12-06 22:35 MenuItem
dr-------- 0 root root 1252 2005-12-12 02:31 Package
dr-------- 0 root root 32 2005-12-06 08:39 Person
dr-------- 0 root root 12 2005-12-05 17:55 Preference
dr-------- 0 root root 284 2005-12-08 01:01 Recording
dr-------- 0 root root 60 2004-10-30 17:14 Resource
dr-------- 0 root root 364 2005-12-12 07:50 Rubbish
dr-------- 0 root root 2116 2005-12-12 09:06 Schedule
dr-------- 0 root root 76 2005-12-05 17:56 SeasonPass
dr-------- 0 root root 5420 2005-12-12 09:11 Server
-r-------- 0 root root 44828 2005-12-12 16:30 Setup
dr-------- 0 root root 388 2005-12-12 02:31 Showcase
dr-------- 0 root root 216 2005-12-12 12:29 ShowcaseIndex
dr-------- 0 root root 4 2005-12-12 12:29 ShowcaseIndex.temp
dr-------- 0 root root 36 2005-12-05 16:39 Star
dr-------- 0 root root 300 2005-12-11 10:43 State
dr-------- 0 root root 28 2005-12-05 16:24 StationTms
dr-------- 0 root root 112 2005-12-09 00:00 SwModule
dr-------- 0 root root 44 2005-12-08 03:04 SwSystem
dr-------- 0 root root 52 2004-10-30 17:20 Table
dr-------- 0 root root 84 2005-12-06 21:09 Theme
dr-------- 0 root root 4 2005-12-12 16:29 tmp
dr-------- 0 root root 28 2004-10-30 17:37 User


I also implemented what I think is "proper" file access ... (unverified correctness at this point)

tivo-hack:/usr/src/linux/fs/mfs# hexdump /mnt/test/ApgBoot
0000000 0000 2e01 0000 4827 2400 2400 4f00 0000
0000010 0000 0000 0000 0b00 1000 0800 0000 0000
0000020 1180 0c00 0100 4e1b 0000 0c00 2000 2000
0000030 4e00 0000 0000 0000 0000 0d00 1100 0800
0000040 0500 787c 1000 0800 0000 0062 2000 2000
0000050 4e00 0000 0000 0000 0000 0e00 1100 0800
0000060 0500 2a82 1000 0800 0000 0142 2000 2000
0000070 4e00 0000 0000 0000 0000 0f00 1100 0800
0000080 0500 dc87 1000 0800 0000 0262 2000 2000
0000090 4e00 0000 0000 0000 0000 1000 1100 0800
00000a0 0500 8e8d 1000 0800 0000 0342 2000 2000
00000b0 4e00 0000 0000 0000 0000 1100 1100 0800
00000c0 0500 4093 1000 0800 0000 0462 2000 2000
00000d0 4e00 0000 0000 0000 0000 1200 1100 0800
00000e0 0500 f298 1000 0800 0000 0542 2000 2000
00000f0 4e00 0000 0000 0000 0000 1300 1100 0800
0000100 0500 a49e 1000 0800 0000 0662 2000 2000
<snipped>


performance is actually VERY good (using: ls -Ral, hexdump of a large "file", etc) ...

At this point, I'm thinking I need some volunteers for code review and/or testing ... this means someone who understands MFS to a level that can verify that what I've got is sound, and also perhaps benchmark the use of this fs module versus pure user level tools ... cwingert?, jamie?, other? ... PM please if interested. I have a bit of cleanup/commenting todo before I can provide a patch, but sure would appreciate a look over.

Current Status:
Dev done against a single S2 drive (aka no swapping details)
Written to a v2.4 kernel

ToDo:
Fixup listing bugs (e.g. "total 0" ...)
Test against a set of S1 drives (probably using bswap kernel option)
Port to v2.6 kernel (possibly submitting to kernel.org as well)
Implement "write" support" (actually might not be too bad)
Avoid memcpy during inode and directory access (should be easy and bump performance)
"MFSck" ability? (aka use of "backup" sectors to "repair"?)
<lotsa more>

My end goal is to provide kernel level access to the FS data (i.e. be fast due to kernel block caching ...), and thereby allow user level tools (aka ls, ncftp, nbd, etc) to be used in place of "special" apps that currently exist.

[santa8claws]

Hi,
I think making a fs module makes sense to use it with other services and am very impressed with what you've done so far! I'm just surprised that other developers hadn't pursued this approach such a long time ago. Just a question though, would you still need separate tools to read/write the databases, records, and resources within the files? Anyway, good work!
-- S8C

[rc3105]

Hi,
I'm just surprised that other developers hadn't pursued this approach such a long time ago.

yah, like 'bout 4 years ago (http://www.dealdatabase.com/forum/showthread.php?p=209849&highlight=module+kernel#post209849) ;) Hi,
would you still need separate tools to read/write the databases, records, and resources within the files?
reading's easy, writing is dangerous if dependencies aren't updated correctly

[DeadSock]

Just a question though, would you still need separate tools to read/write the databases, records, and resources within the files?-- S8C

Currently it treats all tyFile, tyStream, tyDb types as S_IFREG ...
and like any filesystem, tools have to exist for the data (aka vi for text files, gimp for jpegs, etc).

Obviously tools that already know how to deal with raw files just work (simple things like strings, hexdump, etc as well as more sophisticated tools like samba, nfs, nbd, etc)

The point is the low level gore about reading (AND writing) from/to the disk is handled by the kernel (including the page/buffer caching it provides).

[Jamie]

At this point, I'm thinking I need some volunteers for code review and/or testing ... this means someone who understands MFS to a level that can verify that what I've got is sound, and also perhaps benchmark the use of this fs module versus pure user level tools ... cwingert?, jamie?, other? ... PM please if interested. I have a bit of cleanup/commenting todo before I can provide a patch, but sure would appreciate a look over.I'd be happy to review code and try things out. I'm no linux files system expert, but I know the basics of VFS, etc. It's nice to see someone with an idea actually willing to try to implement it.

I must admit that I'm still sceptical that this offers many advantages over the current software architecture. I doubt there is going to be a lot of interest in rewriting the user level apps to go through a file system interface rather than the current mfs library. The apps still need to groke tydb's, tystream's, brf's, etc, which IMHO, is the hard part.

If you can get write support working, that would offer something new. I suspect this is not as easy as you think, but maybe you've dug deeper into the block/inode allocation management data structures than I have. Writing on a live tivo is also problematic unless you know how to synchronize access with the tivo processes to avoid transient inconsistent file system states.

[DeadSock]

I'd be happy to review code and try things out. I'm no linux files system expert, but I know the basics of VFS, etc. It's nice to see someone with an idea actually willing to try to implement it.
yea ... that is sorta novel ;)


I must admit that I'm still sceptical that this offers many advantages over the current software architecture. I doubt there is going to be a lot of interest in rewriting the user level apps to go through a file system interface rather than the current mfs library. The apps still need to groke tydb's, tystream's, brf's, etc, which IMHO, is the hard part.
the primary advantage I can see is reusing the tools that already exist for dealing with filesystems. Re-doing "tivo specific" tools is a downside, but they should also be much easier (since they don't need to deal with inodes, page caching, etc).


If you can get write support working, that would offer something new. I suspect this is not as easy as you think, but maybe you've dug deeper into the block/inode allocation management data structures than I have.
It doesn't look "easy", but it's just bookeeping, so not impossible. Might need to add an ioctl interface inorder for "creation" of some inode types (aka creating a TyDB vs. a TyStream vs. a TyFile).

Writing on a live tivo is also problematic unless you know how to synchronize access with the tivo processes to avoid transient inconsistent file system states.
I agree, this may be problematic. Then again, if the tivo kernel is using the linux page cache, I may find that pages/buffers are marked "dirty" when tivoapp manipulates them ... definetly needs investigation before I'd try write support on a runnning tivo.

[Jamie]

the primary advantage I can see is reusing the tools that already exist for dealing with filesystems. Re-doing "tivo specific" tools is a downside, but they should also be much easier (since they don't need to deal with inodes, page caching, etc).You can accomplish the same goal with a library. It doesn't have to be a kernel interface. For example, who cares if the call to read an MFS file is read() or mfs_read()?

The buffer cache is actually a bad thing for streaming apps, where the data is only read once. It adds extra data copies. I'm not certain (I'd have to go through some strace dumps and read through the tivo ide driver), but I think that MFS access from the tivo software uses ioctl's that bypass the kernel buffer cache, for the most part. The buffer cache might be helpful for tydir traversal and tydb access, as those do tend to get re-read a lot.

[DeadSock]

You can accomplish the same goal with a library. It doesn't have to be a kernel interface. For example, who cares if the call to read an MFS file is read() or mfs_read()?

The buffer cache is actually a bad thing for streaming apps, where the data is only read once. It adds extra data copies. I'm not certain (I'd have to go through some strace dumps and read through the tivo ide driver), but I think that MFS access from the tivo software uses ioctl's that bypass the kernel buffer cache, for the most part. The buffer cache might be helpful for tydir traversal and tydb access, as those do tend to get re-read a lot.

I agree ... but ...

To me, the "library" *should* be a fs module ... and user mode apps get the caching bonuses when they deal with tydir/tydb stuff. In looking at the code I have (mfstools and mfs-utils), there is little "library" shared between those toolsets. (not to mention the multitude of "tools" not implemented that come standard with most linux package sets).

I've already given thought to having "special" ops sets for tystreams (to bypass page caching) ... perhaps make a tystream *be* a S_IFBLK, and then have the ops set for that bypass the std page/buffer caches ...

[DeadSock]

Not to pik a fight ... but to clarify some points ...

For example, who cares if the call to read an MFS file is read() or mfs_read()?
LOTS of applications ... for example *any* existing linux app that deals with files. They know about read() ...

I think that MFS access from the tivo software uses ioctl's that bypass the kernel buffer cache, for the most part.
My perusal has me thinkin the same ... I might try building a custom tivo kernel that implements a "tivo_lock()/tivo_unlock()" where they do this. I'd be interested to see the overhead that would impose (without any external contention). Perhaps then apps that are async IO capable could behave nicely on a running tivo (and not interfere with tivoapp).

[Jamie]

Not to pik a fight ... but to clarify some points ...


LOTS of applications ... for example *any* existing linux app that deals with files. They know about read() ...I'm thinking about apps that actually do something useful with the data in MFS. Not hexdump, but things that grok the tydb schema, brf files, tystreams, etc. For these apps, IMHO, it does not matter whether you use a standard kernel interface like open()/read()/write() or a library interface. Perhaps our disagreement is simply that I don't see much usefulness in generic (non-tivo-smart) apps accessing MFS. I'll be happy to be proved wrong!

[DeadSock]

I'm thinking about apps that actually do something useful with the data in MFS. Not hexdump, but things that grok the tydb schema, brf files, tystreams, etc. For these apps, IMHO, it does not matter whether you use a standard kernel interface like open()/read()/write() or a library interface. Perhaps our disagreement is simply that I don't see much usefulness in generic (non-tivo-smart) apps accessing MFS. I'll be happy to be proved wrong!

ftp, ncftp, samba, nfs, nbd, cp, rcp ... using these "non-tivo-smart" apps can then open up the ability for "tivo-smart" apps then to work "outside the box".

perl, python, etc might be useful for manipulating tyDB stuff ...

just some off the top thoughts.

(BTW, you should have a PM).

[Jamie]

ftp, ncftp, samba, nfs, nbd, cp, rcp ... using these "non-tivo-smart" apps can then open up the ability for "tivo-smart" apps then to work "outside the box".You can do that with a library too. mfs-utils does exactly that through vserver.

I've beaten this dead horse enough. I'm just being a curmudgeon. I commend you for doing something new, even if rc3105 says it's been done before.
(BTW, you should have a PM).Thanks, I got it.

[DeadSock]

You can do that with a library too. mfs-utils does exactly that through vserver.
perl code can read/write to the fs with vserver? ... didn't realize it provided that ...

[Jamie]

perl code can read/write to the fs with vserver? ... didn't realize it provided that ...Yes, perl code can talk through a socket to vserver.