vserver commands

[dburckh]

I'm considering dumping dserver.tcl. I think I have enough of an understanding of the MFS file system to do everything by fsid and just parse the results.

So, I started looking at vserver. The command structure code references a blk. Is a blk the same as a fsid? If not, how (if at all) can you get the fsid from a blk and vica versa?

Thanks.

[Jamie]

I'm considering dumping dserver.tcl. I think I have enough of an understanding of the MFS file system to do everything by fsid and just parse the results.

So, I started looking at vserver. The command structure code references a blk. Is a blk the same as a fsid? If not, how (if at all) can you get the fsid from a blk and vica versa?

Thanks.vserver provides a low level (aka block or sector level) access to MFS. If you want to operate at that level, you need to parse all the file system data structures yourself to figure out mfs dir and db objects. You can use the C code in mfs-utils to do that. In fact, all the mfs-utils programs can be used remotely connecting to a remote vserver. Just set the environment variable MFS_DEVLIST to ":your-tivo-ip-or-dns-name".

If I remember right, you are working on a project written in Java. It might be possible to wrap a Java JNI API around the mfs-utils library. The API isn't particularly clean or well documented though. And mfs-utils is GPL'd, which might have implications on licensing.

[dburckh]

I've looked at the C code to parse the db obj structures and directories. It looks fairly straight forward. It'll be pretty easy to replicate in Java. The only concern I have is the schema changing, which I could still handle.

Doing JNI defeats the platform independence of Java, so I'd only do that as a last resort.

That being said, vserver sounds too low level for what I want. Primarily, I want to just dump files by FSID. I could use tserver assuming get TYSTRM (sp) will dump db objs.

I think what I really need is a server version of mfs_uberexport.

Here are what I think my ideal requirements would be:
1. Simple setup (e.g. third party dependencies)
2. A socket based interface.
3. The ability to dump multiple FSIDs to said socket.
4. The ability to support multiple threads
5. The ability to dump a FSID from a seek point (live extraction)

I believe tserver meets 1-3 (with some limitations), 4 with some advanced setup.

I could meet 1-5 with a simplified version of dserver.tcl that basically wraps mfs_uberexport. At least then, I could removed my dependency on NowShowing.

[Jamie]

I've looked at the C code to parse the db obj structures and directories. It looks fairly straight forward. It'll be pretty easy to replicate in Java. The only concern I have is the schema changing, which I could still handle.

Doing JNI defeats the platform independence of Java, so I'd only do that as a last resort.

That being said, vserver sounds too low level for what I want. Primarily, I want to just dump files by FSID. I could use tserver assuming get TYSTRM (sp) will dump db objs.

I think what I really need is a server version of mfs_uberexport.

Here are what I think my ideal requirements would be:
1. Simple setup (e.g. third party dependencies)
2. A socket based interface.
3. The ability to dump multiple FSIDs to said socket.
4. The ability to support multiple threads
5. The ability to dump a FSID from a seek point (live extraction)

I believe tserver meets 1-3 (with some limitations), 4 with some advanced setup.

I could meet 1-5 with a simplified version of dserver.tcl that basically wraps mfs_uberexport. At least then, I could removed my dependency on NowShowing.It sounds to me like a custom server is your best bet, either in tcl using the mfs-utils tools for the heavy lifting, or in C using the mfs-utils libraries to deal with the MFS FS details.

[dburckh]

Ok, I have this working (for the most part). I can extract/parse directory entries and Mfs DB Objects. Extracting the /Recording/NowShowingByClassic objects takes about 10 seconds as compared to NowShowing which takes 17 seconds. I still have to get the Showings and the Channels, but I should be in the ballpark with a lot more information.

I ran into a weird thing. It appears that the Mfs DB Object attributes need to be word aligned. I found that with empty strings, the attribute length was 0x05 (eltype, attribType, size (x02), null), but the next attribute didn't start until 0x08. There is a 3 byte discrepancy. The three bytes appears to be garbage and I've seen 2 byte changes if the size is something like 0x06. I changed the code to skip to the next word after the end of the string and everything works now.

I didn't see mfs_utils (mfs_dumpobj.c) handling this anywhere (my C is not great), but it appears to work. Is this a C thing or did I miss something?

[Jamie]

...
I didn't see mfs_utils (mfs_dumpobj.c) handling this anywhere (my C is not great), but it appears to work. Is this a C thing or did I miss something?Not my code originally, but I see ret = (attr->len+3)&~3 in object.c line 22 which does the skipping forward to the next word boundary, I think. It sounds like you are doing the same thing.

Be aware that much of the MFS internals changed for the TiVoHD, where sector offsets became 64 bits instead of 32.

[dburckh]

Yeah, I can see how I missed that. I've never seen a bitwise NOT before. Interesting way of doing that, but probably very efficient. Looks like Java supports it as well. When in Rome...

Be aware that much of the MFS internals changed for the TiVoHD, where sector offsets became 64 bits instead of 32.
I'm doing batch extraction on the fsid level with mfs_uberexport, so I don't think I need to worry about sectors.

[spaceman1013]

Extracting the /Recording/NowShowingByClassic objects takes about 10 seconds as compared to NowShowing which takes 17 seconds. I still have to get the Showings and the Channels, but I should be in the ballpark with a lot more information.Sounds like you have written a Java client that communicates with vserver for your now showing list. Will you be releasing this code so that others can use it in their projects?

Also can your client obtain show description? Does vserver even provide that functionality?

I wanted to write a utility on the PC that would like TyTools grab the now showing list but in addition grab each shows description as well. I was going to just try to telnet in from my client and run NowShowing script to get all the FSIDs and then use the mfs_uberexport -RX option on each FSID to get show details. My solution soounds like a hack compared to how you implemented yours.

It sounds like you are currently using the NowShowing program to get your listing via a dserver that you have written (I found your project TySuiteJ). I do not know much about tcl programming but it looks like a single tcl script can run in the background as a server - is that a correct understanding? And it seems you communicate with dserver via a socket (56178)

I would like to use your dserver in my project if thats okay. I was surprised you did not write your server in java since linux can run java. Or was dserver based off some existing code you reused and thus quicker to just get it done?

I noticed this in your dserver code:

} elseif {$command == {xShowing}} {
set fsid [read $outSocket 8]
nowShowingXml $fsid $outSocket
}

Is there a way to invoke this from CLI interface?

[dburckh]

Sounds like you have written a Java client that communicates with vserver for your now showing list.
vserver ended up being too low level for what I wanted. I've wrapped mfs_uberexport and I'm parsing the Ty DB Objects to get what I need.
Will you be releasing this code so that others can use it in their projects?
Every version of TySuiteJ has the source in the jar file. You have my permission to use it.

Also can your client obtain show description? Does vserver even provide that functionality?

Not in version 2. Version 3.0 (In progress) will be able to get pretty much everything about the title including the description. I can send you Alpha code if you are interested.

It sounds like you are currently using the NowShowing program to get your listing via a dserver that you have written (I found your project TySuiteJ). I do not know much about tcl programming but it looks like a single tcl script can run in the background as a server - is that a correct understanding? And it seems you communicate with dserver via a socket (56178)

Correct

I would like to use your dserver in my project if thats okay. I was surprised you did not write your server in java since linux can run java. Or was dserver based off some existing code you reused and thus quicker to just get it done?

dserver is/was based loosely off of mfs_ftp and WebTivo. TCL comes as part of the Tivo, so using it reduces the number of things required to get TySuiteJ working. As far as I know, there is no Java version that runs on MIPS and I doubt the Tivo could hardware could support Java (it's kind of fat).

I noticed this in your dserver code:

} elseif {$command == {xShowing}} {
set fsid [read $outSocket 8]
nowShowingXml $fsid $outSocket
}

Is there a way to invoke this from CLI interface?

Not currently. I was actually planning or removing this method call in version 3.

Depending on your programming skills you can open a socket to 56178 and send in "xShowing99999999". Where 9999999 is the fsid. It must be 8 characters long and padded with spaces.

I am considering dumping the NowShowing in xmltv format (with description, duration, original air date, ...) in version 3. Further, I think I could dump the entire program guide in xmltv format.