Database - increase performance?

[Meklos]

Is the guide data and/or now playing database located on a separate partition? If so, could it be relocated to a dedicated drive? Maybe even a flash disk (http://pcmag.shopping.com/xPC-Simple_Tech_Flash_Drive_512_MB_STI_FLD35_512) for those with more money to blow?

[cojonesdetoro]

There's a product called the 'cache card' sold by www.9thtee.com that works along these lines. IIRC, the database is not more than 500MB in size. A cachecard equipped with a 500MB stick will cache the entire DB. I have mine with a 256MB stick and it still offers a substantial increase in performance.

BTW, the cachecard is for S1 only. It's also an ethernet card.

[Meklos]

There's a product called the 'cache card' sold by www.9thtee.com that works along these lines. IIRC, the database is not more than 500MB in size. A cachecard equipped with a 500MB stick will cache the entire DB. I have mine with a 256MB stick and it still offers a substantial increase in performance.

BTW, the cachecard is for S1 only. It's also an ethernet card.

Yeah, I know about the cachecard, and that's what got me to thinking about this subject.

If the database is on it's own partition(s), and if those could be placed elsewhere (not sure if they're mounted or if their location is hardcoded somewhere) then maybe a performance increase could be gained.

I'm thinking of people like myself, with a single 120GB in my S2 DTivo. I'll probably not go larger than that, especially with the ability to move things to DVD, with the upcoming advancements in mfsftp servers on the PC, and all the interesting things we'll be able to do. Right now, 120GB full on my S2 is tolerable speed-wise. If I could drop a 1GB flash drive on the IDE bus, map the partition(s) there instead of on the 120, then maybe I could get a performance increase. Maybe I could even get a performance increase by just moving it to a dedicated regular hard drive.

That's assuming, of course, that part of the slowdown is in accessing the data itself. I know the cachecard uses RAM, and it's located on a system bus with a high data rate, so maybe moving the partitions wouldn't gain much.

That's why I need the advice of someone who knows the inner workings of the Tivo much better than me.

[RKone]

Flash drives are actually much slower than hard drives.

I've been working a method to speed up the s2 guide myself. If it works, I'll release the info.

[BTUx9]

Part of the slowdown from MFS is disk access, but part is the locking mechanism, and things like the cache card don't help that much. If tivo would just perform intelligent caching on the Now Showing List, the worst (IMHO) delays would be greatly improved. While the re-prioritizing delay is one of the longest, I don't consider it one of the worst, because it happens SO seldom, and can be accomplished painlessly thru tivoweb+.

edit:
It's a shame that there's no cachecard for S2. I got dual-120's full with recordings (w/ suggestions) and my NP list is almost always up in 5 sec or (usually) less.
I guess the locking mechanism is improved more by cache card than I thought. My Bad.

[Meklos]

I'm more concerned with the Now Showing list speed than anything. What good are big drives with lots of recording time when the NS list takes 2 minutes to pull up?

[cojonesdetoro]

It's a shame that there's no cachecard for S2. I got dual-120's full with recordings (w/ suggestions) and my NP list is almost always up in 5 sec or (usually) less.

[BTUx9]

I'm more concerned with the Now Showing list speed than anything. What good are big drives with lots of recording time when the NS list takes 2 minutes to pull up?
I agree 150%. If only they'd add folders without breaking events, we'd be golden. Well... at least it'd be a step in the right direction.

[Meklos]

Does the software on the HDTivo have folders? I know it's 3.1.5 or something like that.


I have to wonder what will happen to that box if someone doesn't record HD but does record 250GB of SD. Will it come to a crawl like ours, or does it have more memory - or something?

[cojonesdetoro]

Now that I remember, the S2 has 32MB RAM and a much faster processor. I didn't think it had the slow NP problems, even with much larger drives.

is this isolated or are there other people with the same problem?

Perhaps the HD Tivo takes it a step further. Maybe it can be considered an S3?

[BTUx9]

Now that I remember, the S2 has 32MB RAM and a much faster processor. I didn't think it had the slow NP problems, even with much larger drives.

is this isolated or are there other people with the same problem?

Perhaps the HD Tivo takes it a step further. Maybe it can be considered an S3?
the S2 dtivo is as slow, or SLOWER than s1, when it comes to the NSL.

[malfunct]

the S2 dtivo is as slow, or SLOWER than s1, when it comes to the NSL.

There is a fix for the issue (I thought) in the 4.x series, one of the benifits that hopefully will roll onto the dtivo side of things.

[BTUx9]

There is a fix for the issue (I thought) in the 4.x series, one of the benifits that hopefully will roll onto the dtivo side of things.
Folders do speed things up, but with vwait and events broken in 4.x, it's a tossup. vwait can be dealt with, but nobody has come forward with a workaround for 4.x events, yet.

[Meklos]

OK I've looked through some scripts that pull the Now Showing list, and someone please tell me that the scripts are really kludgy, and that the tivoapp really does something different each time it displays the Now Showing list.

Let's face it... the NS list doesn't change that often. The events which change it are well defined and could be 'captured' easily. MFS is a journaled file system, right? If you wanted to brute force it, either have an app which reads the transaction log or which gets a pipe of the realtime calls or something... so that it can watch for an event which qualifies for a NS list rebuild.

That's the point I'm getting to. We know the list doesn't change that much, so why on earth is it 'rebuilt' each time it's called? If that's the case, that's *****ic.

You don't create a webpage like the following:

[reference: Oxford English unabridged.txt]

<head>
03a7 72c1 78 3f8 9a2

....



You don't create a webpage where each word on the page must be looked up in a separate database, thus slowing the crap out of the access... but what does the NS list appear to do?


(ShowStatus) (ShowTitleID) (ShowDate) etc etc... for each stupid line, necessitating several database queries for each line of the NS list.


NO WONDER it's so freaking slow!



Wouldn't it be smarter to create an app which wrote a file which in ONE reading of the file could display the current NS list... and then trigger the update of that app (and thus the list) whenever a 'worthy' event occured? Heck, if you really wanted to go belt-and-suspenders, write the display app such that the rebuild app could notify it when a new version was available so it could be reloaded.


So this all begs the question: How hard would it be to implement an app like this, replacing the NS list functions of tivoapp with this pre-built-file type of approach? How hard would it be to hack the tivoapp to call this file for display instead of going through all the stupid slow database lookups?

I still can't get over this. If it works like that, it amazes me. Databases are a great thing, but if the data doesn't change that much, don't beat the database to death over something that could be an index or flat file.

[BTUx9]

I still can't get over this. If it works like that, it amazes me.
I'm afraid it does seem to work like that. It can tell if the list is "valid" or not, but it seems to consider watching a show (bookmark gets modified) and recording (stop time) to invalidate the current list, and makes no attempt to only load those items that have been modified. I'm not 100% sure this is the case, but it fits all the facts.

Maybe it's possible to monitor which fsids are involved in the NSL, and only cache those writes/reads... something similar to the driver the cachecard uses, but much more focused. It wouldn't have to be 100%, so a periodic checking of the NSL could update those areas that should be cached. Shouldn't require that much memory to accomplish... the s2 machines should certainly have enough, and the s1's have the cachecard.

[Meklos]

I'm afraid it does seem to work like that. It can tell if the list is "valid" or not, but it seems to consider watching a show (bookmark gets modified) and recording (stop time) to invalidate the current list, and makes no attempt to only load those items that have been modified. I'm not 100% sure this is the case, but it fits all the facts.

Maybe it's possible to monitor which fsids are involved in the NSL, and only cache those writes/reads... something similar to the driver the cachecard uses, but much more focused. It wouldn't have to be 100%, so a periodic checking of the NSL could update those areas that should be cached. Shouldn't require that much memory to accomplish... the s2 machines should certainly have enough, and the s1's have the cachecard.


Wouldn't even have to cache it to RAM. Just have a 'current-version' file that it gets redirected to.

[RKone]

Would it be feasible to create a modified kernel that automatically caches the Now Playing data?

[Meklos]

I'm sitting here wondering how the driver for the cachecard works. Does it sit between tivoapp and the mfs database(s)? If so, does it intelligently cache the items needed to speed things up?


If it does, it wouldn't be too hard to alter that driver to, instead of reading stuff from RAM, to write the appropriate entries relating to the NS list (or Season Passes, or whatever) to a flat file and to load that flat file whenever the NS list is called - instead of calling 5,000 database queries. Then the driver would also have to make sure the flat file was kept up to date. But if it sat 'between' tivoapp and the databases, then it could watch the whole data stream and decide which entries are needed to update.


Alternatively, the tivoapp itself might be able to be rewritten slightly to cache to a flat file.

[BTUx9]

Wouldn't even have to cache it to RAM. Just have a 'current-version' file that it gets redirected to.
I don't think a flat file would help that much because the tivo would still be accessing it in a scattered manner -- it would only help if you could somehow modify the routine that fetches the current NSL. (good luck!)

I'm sitting here wondering how the driver for the cachecard works. Does it sit between tivoapp and the mfs database(s)? If so, does it intelligently cache the items needed to speed things up?
I think the caching is of all accesses to MFS application partitions. I don't think the driver interprets the data at all.

Alternatively, the tivoapp itself might be able to be rewritten slightly to cache to a flat file.
Now THAT would be non-trivial, if not pert near impossible.

[cojonesdetoro]

Now THAT would be non-trivial, if not pert near impossible.

Not impossible but definitely improbable. Two things could happen:

Tivo could release source (when pigs fly)

Someone could reverse engineer the binary. A big job that would invite a lot of trouble if released. Anyone who has the skills and can accomplish this will likely keep it to themselves or just within a small circle of trusted contacts.

[Meklos]

I think the caching is of all accesses to MFS application partitions. I don't think the driver interprets the data at all.

Hmm... at best, we would have to make the driver more intelligent and only cache the database entries directly related to the Now Showing list.


I wonder if it would be possible, or even simpler, to do this to speed up the grid guide?

[malfunct]

Hmm... at best, we would have to make the driver more intelligent and only cache the database entries directly related to the Now Showing list.


I wonder if it would be possible, or even simpler, to do this to speed up the grid guide?

Whether we cache only Nowplaying entries or the entire MFS database wouldn't change the speedup provided by the cache. You can either make each access quick or you can make accesses less often. Unfortunately the latter is not under our control so the cachecard does a good job of the former.

[Meklos]

Whether we cache only Nowplaying entries or the entire MFS database wouldn't change the speedup provided by the cache. You can either make each access quick or you can make accesses less often. Unfortunately the latter is not under our control so the cachecard does a good job of the former.

I'm thinking of S2 users, like myself, who can't get a cachecard.

In theory, if the driver holds the entries related to NP in memory and delivers them without an actual call-to-mfs-on-disk, then the time spent building the NP list could be lessened.

[malfunct]

I'm thinking of S2 users, like myself, who can't get a cachecard.

In theory, if the driver holds the entries related to NP in memory and delivers them without an actual call-to-mfs-on-disk, then the time spent building the NP list could be lessened.

True, though I don't know what memory use looks like currently, I'd hate to eat up memory thats being used for other important tasks. BTW your scheme is basically a memory resident cache instead of an offline cache like the cache card.

[Meklos]

True, though I don't know what memory use looks like currently, I'd hate to eat up memory thats being used for other important tasks. BTW your scheme is basically a memory resident cache instead of an offline cache like the cache card.

Not necessarily. It wouldn't have to all be in memory all the time. Maybe a routine that recognized when the NP list is being called by tivoapp. When it sees this, it pulls all the relevant soon-to-be-accessed mfs database entries from a flat file and has them all ready to be nearly instantly delivered. Once the NP list is generated, the memory is cleared.


Just thinking out loud more than anything. The way the tivoapp builds the list is really inefficient. It's something that is user interactive, that doesn't change that often, that the events which cause it to change can be recognized easily.

It'd be like running a large text database without an pre-built index. Each time someone ran a search, let's rebuild the index, or worse - just roam through the database manually looking for the search terms.

Maybe the discussion will prompt someone with better programming skills than me to some "Eureka" moment. There's GOTTA be a better way than the tivoapp does it now.