PDA

View Full Version : folders for inserted and non-series shows



jbuehl
11-28-2004, 01:59 PM
I'm starting a thread in this group to discuss development of ways to group shows in folders that have been inserted with mfs_ftp or otherwise wouldn't be grouped because they aren't part of a TV series. There has been some previoius discussion in these threads:

http://www.dealdatabase.com/forum/showthread.php?t=39523
http://www.dealdatabase.com/forum/showthread.php?t=36208
http://www.dealdatabase.com/forum/showthread.php?t=37413

When a show is recorded, the recording object gets linked to a series object that is created from the guide data. An inserted show must be linked to a series object in order to be grouped. The problem is that the series object can't be easily found without a long search of the database.

For shows that there isn't a series object for in the database, the problem is worse. Creating a new series object is hazardous to the database, according to Riley. I'm sure that's true, but I would be interested in hearing from anyone who has tried it and what problems they encountered.

jbuehl
11-28-2004, 02:10 PM
Creating a link to a series in mfs_ftp seems straightforward enough, assuming one knows the fsid of the series object. The series can be identified by the TmsId that is contained in the xml of the .tmf being imported. Searching through the database for an object containing a specific TmsId can take many minutes.

I have written a C program that pulls out all the TmsId -> fsid relationships of series objects and stores them in an index file. This file can be very quickly (< 1 sec) be searched to return the fsid that corresponds to a TmsId. The idea would be to run this indexing program once a day or so to keep it up to date. I need to compile this for mips and I will make it available to anyone who wants to try it.

I am currently figuring out how to modify tcl (can anyone suggest a good tutorial?) so I can modify mfs_ftp to use it.

alldeadhomiez
11-28-2004, 03:12 PM
I have written a C program that pulls out all the TmsId -> fsid relationships of series objects and stores them in an index file. This file can be very quickly (< 1 sec) be searched to return the fsid that corresponds to a TmsId. The idea would be to run this indexing program once a day or so to keep it up to date. I need to compile this for mips and I will make it available to anyone who wants to try it.

It's built in... look at /GuideIndexV2/Tms.index:


# hexdump Tms.index | grep "0005 5730"
006db00: 5348 0000 0000 0000 0005 5730 0000 656f
# mfs_dumpobj `printf "%d" 0x656f`
Program 25967/11 PRIMARY {
ServerVersion=112
ColorCode=4
DescLanguage=English
Description=The television magazine series covers a broad spectrum of modern life.
Genre=106 180 1003 1007 1010
NetworkSource=cbs
OriginalAirDate=-464
Series=25966/-1
ShowType=5
SourceType=1
Title=60 Minutes
TmsId=SH0000350000
ServerId=19061
Version=1
IndexPath=/Server/19061
}

Since this index is ordered by TmsId, you can use a binary search so that your lookup time is bounded by O(lg n). You should be able to get very quick lookups by extending the vplay code (instead of using FromMfs to yank the index on each lookup). Obviously, there are cache/race problems which need to be taken into consideration if you use this method, so maybe FromMfs can be retained as a fallback if the lookup fails.

It is also interesting to note that the TmsId for a given series or program is consistent across headends, including the DirecTV (APG) service.

The two character alphanumeric prefix appears to denote the following object types:


00 = Station
EP = Program (TV)
MV = Program (movie)
SH = Series
SP = Program (sports)

jbuehl
11-28-2004, 04:12 PM
It's built in... look at /GuideIndexV2/Tms.index:


# hexdump Tms.index | grep "0005 5730"
006db00: 5348 0000 0000 0000 0005 5730 0000 656f

"0005 5730" is hex of the TmsId 350000 which references the program, but how does that help to find the series object? The problem is, given program object in fsid 25967 with the Series field blank, what is the fsid (25966) of the series object that has TmsId SH000035.

What is the format of the entries in Tms.index? Also, what is FromMfs? I'm not familiar with that. The program I wrote is based on the tridge/vplay code.


It is also interesting to note that the TmsId for a given series or program is consistent across headends, including the DirecTV (APG) service.

That shouldn't be a surprise because Tribune Media Services is the guide data aggregator for Tivo, DirecTV, and most cable companies.

jbuehl
11-28-2004, 08:58 PM
Here's an interim way to find a series fsid given the TmsId. It searches /Server and builds a temporary index file in /var/tmp that can be searched.

mfs_tmsid is the program that builds the series index. It takes about 15 minutes to run on a HDVR2.



bash-2.02# ./makeSeriesIndex
54974 entries found in /Server
7926 TmsIDs found

The tmsid.idx file contains fsid, TmsId, and title of all the series objects found. The title isn't really necessary, so it can eventually be deleted to reduce the file size.



bash-2.02# head /var/tmp/tmsid.idx
297444 SH000001 {Paid Programming}
305387 SH000021 {20/20}
342688 SH000030 {3-2-1 Classroom Contact}
312648 SH000035 {60 Minutes}
312763 SH000051 {A Different World}
314463 SH000066 {A Pup Named Scooby-Doo}
340472 SH000115 {The Addams Family}
344545 SH000150 {Against All Odds: Inside Statistics}


Finding a particular series given the TmsId is fast. The shell script getSeriesFsid does it:



# ./getSeriesFsid SH000115
340472

This should be able to be called from a tcl script.

Note: There are some duplicate entries for the same TmsId. This needs to be taken into account.



# grep SH476707 /var/tmp/tmsid.idx
305262 SH476707 {What Not to Wear}
349357 SH476707 {What Not to Wear}

jbuehl
11-29-2004, 01:32 PM
"0005 5730" is hex of the TmsId 350000 which references the program, but how does that help to find the series object? The problem is, given program object in fsid 25967 with the Series field blank, what is the fsid (25966) of the series object that has TmsId SH000035.

What is the format of the entries in Tms.index? Also, what is FromMfs? I'm not familiar with that. The program I wrote is based on the tridge/vplay code.
OK, now I get it. Your color coding was just a bit too subtle for me, and I took your example too literally :)

And I found the post that describes the layout of the tms index. Do you suppose it would be worthwhile adding it to mfs.h?

chrised
11-30-2004, 05:48 AM
I've been working on this for the last week. My objective is to decrypt, migrate, and reorganize all my recordings from my old 3.1.1x systems to new 4.x systems (all DTivo). In this process, I did not want to lose any information for the Tivo UI or TivoWebPlus. The current mfs_ftp makes a minimal effort at this. If you compare it to MRV, you'll see MRV preserves all attributes of the recordings. With this updated version, you won't be able to tell the difference!

My starting point for mfs_ftp.tcl is based on Jamie's modifications from this message: http://www.dealdatabase.com/forum/showthread.php?t=39487
The instructions for that install ask for certain changes to the supporting binaries. Those must be in place for this code.


My testing has been all DTivo, transfers from 3.1.1x and 4.x to 4.x. Strictly TMF format! (the others might be broken).

For folder support, it is critical that all shows be linked to the correct series. This must be done before the recording is saved, or you have to figure out how to correct the recording's indexing. (ick)

There's been lots of discussion about using the TmsID. What's been overlooked is that Tivo assigns each TmsId a ServerId. Using ServerId, it isn't that difficult to find the series object once you figure out the data pattern.

This code is also setup to CREATE series objects that look just like Tivo's objects. It appears to be slightly risky. Testing will prove whether that is true.

It seems that TMS assigns SeriesID's to almost every show, even those you wouldn't expect. It is difficult to find shows without them. And, surprisingly even shows that haven't aired in many months seem to get their series information sent down to new tivos. My only way to test creating a new series was to temporarily hardcode it to a fake TMSId. I do recommend that you allow a new Tivo to gather guide data for a day before transfering shows. This way it should reduce the chance of conflict between tivo's series objects and my objects (which I still think will be one in the same based on the indexing).

This code will use the same concept to link to the real station record instead of a fake one. This allows additional information such as channel number and logo to display where appropriate. Clearly, if you are transfering from SA to DTivo the channel ID's are unlikely to match (they might, but I don't have an SA to try with).

I spent alot of time making sure that every attribute is supported. This means every piece of text on the program info screen. A list of added attributes includes:

Bits EpisodeNum AltEpisodeNum IsEpisode
MovieRunTime MovieYear MpaaRating
ShowType TvRating
ApgProgram (Odd thing that controls CC, Black and White, Series, etc)
Advisory (AC, AL, etc)
Genre (Action/Adventure, Drama, etc)
Actor Director Writer GuestStar Host ExecProducer Producer
Premier

Some of these attributes I've never seen. I added them based on the TivoWebPlus code.

There is a correction for expiration date in here too.

I wouldn't be surprised if this code is not compatible with older Tivo software under version 3.1.1.

This code is barely considered BETA.
USE AT YOUR OWN RISK, HAVE A BACKUP, AND BE PREPARED TO FIX YOUR OWN PROBLEMS.

Also, beware, I've done everything that Riley has warned us is dangerous. Riley understands the Tivo much better than I do. He might be right about this.

This code is being posted here to allow others to do further testing, to review the code, and to let the brave (foolish?) try it out. Ultimately, it would be nice if it could become adopted into the next official release of mfs_ftp.

Good luck!

Edit 11/30: WARNING: It is now confirmed that ServerId varies between SA and DTivo. That renders my solution only valid for DTivo to DTivo transfers. If you attempt a cross platform transfer, shows will not be grouped into the appropriate series. They may be grouped under the original series on the "foreign" platform. This will result in potentially two series objects with the same TmsId. The results of that may cause problems.

Volunteers to help fix this would be appreciated. (I've got what I need for now, so I don't have alot of motivation to spend more time on it)


-----
attachment removed - please post patches / modules as per the license

chrised
11-30-2004, 05:49 AM
Originally I posted the entire modified mfs_ftp.tcl file. Since that apparently is forbidden, it looks like I have two choices. Post a diff patch (painful to me and the users to apply), or implement a plug-in module/patch. I've opted for the plugin patch (I am pleasantly surprised how easy that is to do)

12/1/04 Beta 1.1 features:
- fix uninitialized variables affecting SA1 > Dtivo2 transfer
- should support series linking for SA to SA transfers.
- does NOT support series linking for cross-platform yet.
- disabled creating series objects. Want to see results of requested test before considering re-enabling http://www.dealdatabase.com/forum/showpost.php?p=196929&postcount=23

12/3/04 Beta 1.2 features:
- supports station matching on SA.
- will not support cross-platform station matching. At this time, there isn't good enough reasons to justify building that feature.
- supports cross-platform series matching. This was done by incorporating some code from tivowebplus. I did end up fixing a bug in that code. This code can lookup a series by TmsId in about a second. Can't complain about that performance.

12/6/04 Beta 1.3 features:
- Bug fix for TMS search returning invalid object id. (object no longer in MFS, but is in index)
- User configurable Series creation

12/7/04 Beta 1.4 features:
- Bug fix for Series Lookup. Had originally assumed all DTivo series ServerId started with AT. Found one with AR, and it failed. New approach tries both DTivo and SA ServerId formats before trying the TMS search.

12/8/04 Beta 1.5 features:
- Bug fix for error during series create when no ServerId assigned to series


With the plugin, it no longer matters whether you used Jamie's patches / changes first. So, its gotten even easier to install. Make sure you have an operating version of mfs_ftp 1.2.9p installed. Copy the p1.tcl file into the mfs_ftp directory. Restart mfs_ftp.

As of Beta 1.3+, you need to add a line to settings.tcl or you will get an error:

set info(seriescreate) 0 ;# Allow Series to be created? 0 = disabled, 1/others = enabled.

At this time, I feel its safe to enable seriescreate when you are transferring on the same platform (ie DTivo to DTivo, or SA to SA). Using this for cross-platform transfers may have undesireable long term results.

Beta 1.1 removed.
Beta 1.2 removed: p1.tcl (15.8 KB, 12 views)
Beta 1.3 removed: p1.tcl (16.1 KB, 3 views)
Beta 1.4 removed: p1.tcl (16.5 KB, 5 views)
Beta 1.5 is current version

Beta 1.7 located here http://www.dealdatabase.com/forum/showthread.php?p=246780#post246780

philhu
11-30-2004, 09:16 AM
This has been done. I think Riley has code for it. It is/was designed to be added as a called module to mfs_ftp. I tested a beta of it and it worked fine.

What I did was 'hijack' seriesIDs of series I would never record for non-series stuff. And his code searched the db for series id's for items xferred into the tivo, attaching the item to the series in its group.

I've been using it for months.

jbuehl
11-30-2004, 11:47 AM
This has been done. I think Riley has code for it. It is/was designed to be added as a called module to mfs_ftp. I tested a beta of it and it worked fine.

What I did was 'hijack' seriesIDs of series I would never record for non-series stuff. And his code searched the db for series id's for items xferred into the tivo, attaching the item to the series in its group.

I've been using it for months.
Really? That's interesting. I saw a couple of sthread that you were involved with back in August, but they seemed to go nowhere and Riley didn't sound too interested. In recent posts on this subject he didn't mention it.

chrised
11-30-2004, 12:54 PM
This has been done. I think Riley has code for it. It is/was designed to be added as a called module to mfs_ftp. I tested a beta of it and it worked fine.

What I did was 'hijack' seriesIDs of series I would never record for non-series stuff. And his code searched the db for series id's for items xferred into the tivo, attaching the item to the series in its group.

I've been using it for months.

Its always nice spending 20+ hours developing something someone else has already done. Or has it been done? I don't see it publicly available. Enough people are asking for it. I privately asked for it, and got nothing.

Therefore, in my mind it hasn't been done.

Riley's messages (that I've found) never indicated having a version that did this during the inserts. All I could find was reference to a utility to fix them "Afterwards". And that utility is also not being shared.

I needed this now. And I decided I would share it with the group as a contribution for all the neat stuff I've been using. Thats supposed to be the idea here, right?

alldeadhomiez
11-30-2004, 01:04 PM
There's been lots of discussion about using the TmsID. What's been overlooked is that Tivo assigns each TmsId a ServerId. Using ServerId, it isn't that difficult to find the series object once you figure out the data pattern.

Last time I looked, the TmsId (and not the ServerId) was consistent across SA and combo units for a given Series.

philhu
11-30-2004, 01:06 PM
chrised-

Your solution is very good, and complete.

What Riley wrote was an addon program that could be called by mfs_ftp. It could run for hours (literally), looking for shows in a deep mfs scan

Your method of sending the data across AND building series id record objects is, well, fantastic.

I do not think Riley ever finished his code, and definitely never incorporated it in an mfs_ftp release.

I will play with your code tonight, since it is more complete. It looks PERFECT for what people have been looking for, at least, for me! :)

And FYI, TMSID IS consistent always. It is from the Tribune data stored by Zap2it, and gets stored blindly on the tivo. A show has a generic tmsid or an Episode id. The former is a generic for a show without a definite episode description, and they start with 'SH'. An episode id is a description of an episode of a show, and they start with 'EP'. There is also 'MV' or movie or 'SP' for sports show.

I use it in my Parsexmlguide_dd program to read datadirect zap2it outputs into Canadian tivos. See www.tivocanada.com for details

jbuehl
11-30-2004, 01:20 PM
For folder support, it is critical that all shows be linked to the correct series. This must be done before the recording is saved, or you have to figure out how to correct the recording's indexing. (ick)

There's been lots of discussion about using the TmsID. What's been overlooked is that Tivo assigns each TmsId a ServerId. Using ServerId, it isn't that difficult to find the series object once you figure out the data pattern.
What do you mean by "linked to the correct series"? Assuming xml for the program has the series TmsId, don't you perform the linkage? Or are you saying that it's easier to link it while inserting, rather than trying to correct it after the fact?



This code will use the same concept to link to the real station record instead of a fake one. This allows additional information such as channel number and logo to display where appropriate. Clearly, if you are transfering from SA to DTivo the channel ID's are unlikely to match (they might, but I don't have an SA to try with).
I have an SA which is connected to a DirecTV box, so the channels will be the same. What really matters, though, is the station TmsId, which should be the same for all satellite, cable, and broadcast channels. I can configure my SA to get guide data for my local cable system and test this theory.

jbuehl
11-30-2004, 01:28 PM
Last time I looked, the TmsId (and not the ServerId) was consistent across SA and combo units for a given Series.
I think chrised figured out the naming scheme for the ServerIDs which is based on the TmsId:

#example path /Server/ATSH225421:4:0:0
set seriespath "/Server/$info(SeriesServerId):4:$info(SeriesServerVersion):0"
outd 3 "series path: $seriespath"
if {[catch {RetryTransaction {mfs find "$seriespath"}} l] != 0} {
outd 3 "series path not found. discarding series property"
set seriespath ""
}

If it works reliably and is fast enough, it makes searching Tms.index unnecessary.

chrised
11-30-2004, 01:29 PM
Last time I looked, the TmsId (and not the ServerId) was consistent across SA and combo units for a given Series.

I hope you're wrong. Otherwise my solution will fail when you go cross platform. Can someone out there with both do some investigation and let me know for sure one way or another?

Its my understanding that Tivo uses the ServerId as a global unique identifier. Each series has three identifying elements: object id (unique to each database), TmsId (assigned by Tribune Media Services), and ServerId (assigned by Tivo "central").



chrised-

Your method of sending the data across AND building series id record objects is, well, fantastic.

I will play with your code tonight, since it is more complete. It looks PERFECT for what people have been looking for, at least, for me! :)



Btw, I changed nothing in the "sending" code. In fact, my testing involved using sending units with the normal mfs_ftp code. Everything that was needed was already in the xml. All I did is consume nearly all the xml, and introduce it into the database objects.

alldeadhomiez
11-30-2004, 01:42 PM
I think chrised figured out the naming scheme for the ServerIDs which is based on the TmsId:

The guide data does not seem to be indexed this way on my SA (sw 5.4-01-2-595):


% dumpobj 5841
Series 5841/11 {
Episodic = 1
Genre = 1014 88
IndexPath = /Server/17028
ServerId = 17028
ServerVersion = 148
ShortTitle = {Stargate SG-1}
ThumbData = Predicted 0 0
Title = {Stargate SG-1}
TmsId = SH225421
Version = 3
}
% dumpobj /Server/ATSH225421:4:0:0
can't open object (errDbNotFound)


Its my understanding that Tivo uses the ServerId as a global unique identifier. Each series has three identifying elements: object id (unique to each database), TmsId (assigned by Tribune Media Services), and ServerId (assigned by Tivo "central").

Sounds about right. But the problem we face here is that TiVo probably does not issue ServerId's to DirecTV for the APG. It looks as though they have worked around this by allowing the DirecTV APG system to create special ServerId's in the "AT" namespace, based on the TmsId. The end result is that the two guide data sources have little in common other than the TmsId.

Jamie
11-30-2004, 01:44 PM
I hope you're wrong. Otherwise my solution will fail when you go cross platform. Can someone out there with both do some investigation and let me know for sure one way or another?

I added a recursive dump feature to mfs_dumpobj that might be useful for this investigation. We can accumulate dumps of /Server from various hardware and software versions. I could easily write a little perl script to check whether the ServerId <-> TmsId mapping is unique once I have the dumps.

chrised
11-30-2004, 01:53 PM
The guide data does not seem to be indexed this way on my SA (sw 5.4-01-2-595):


% dumpobj 5841
Series 5841/11 {
Episodic = 1
Genre = 1014 88
IndexPath = /Server/17028
ServerId = 17028
ServerVersion = 148
ShortTitle = {Stargate SG-1}
ThumbData = Predicted 0 0
Title = {Stargate SG-1}
TmsId = SH225421
Version = 3
}
% dumpobj /Server/ATSH225421:4:0:0
can't open object (errDbNotFound)

Ahh crap. I'll add a warning to my "release" page. This code is DTivo only until someone with a mixed environment modifies it to another approach.

philhu
11-30-2004, 01:55 PM
Although the tmsID's are compatible across platforms, since it is tms based, ther seriesids are not!

I have a S2 SA (v4.01b) and a S1 SA (v3.x). The tmsid's DO match, but serverid's do not for the same show recorded off the same channel on both machines.

I think the Tivo creates the server id on the fly, pointing it to the tms id.

On Canadian Tivos, we store/use TMSIDs from zap2it, but we build seriesID, programID from a list we keep and assign from.

jbuehl
11-30-2004, 02:06 PM
Ahh crap. I'll add a warning to my "release" page. This code is DTivo only until someone with a mixed environment modifies it to another approach.
I have the C program that will search Tms.index almost done. You should be able to call it from your script to get the series fsid. I expect to have it done tonight. In the meantime, if someone wants to try what I described in post #5, I'll send you that program. It should work, it's just suboptimal.

alldeadhomiez
11-30-2004, 02:11 PM
I think the Tivo creates the server id on the fly, pointing it to the tms id.

The local machine should never create a ServerId locally in the "main" namespace. The "AT" stuff is handled differently, so these ServerId's will never conflict with the numeric ones coming from the mothership.

As it so happens, the same Series object (with the same ServerId) appears to be sent down to all SAs, regardless of lineup:


Series/1/17028/148 {
TmsId: {SH225421}
Title: {Stargate SG-1}
Genre: 88
Genre: 1014
Episodic: 1
ShortTitle: {Stargate SG-1}
}

chrised
11-30-2004, 03:03 PM
I'd like someone to use MRV to transfer a show cross-platform.

Once its to the other side, use Tivowebplus MFS browse, go to the program/series object, and post the details of the series here. (Make sure you mention the source and destination used in this test).

I am dying to know if MRV has the ability to cross-reference series by TmsId, or if it simply duplicates the "foreign" series object.

If it duplicates the foreign object, we then know that it is safe to do that.

I've already got a good idea on what I need to change in the code to make it work properly on SA units. Its just the cross-platform transfers that look really tricky.

It would also be nice to see the details on the station object.

jbuehl
11-30-2004, 03:14 PM
I have the C program that will search Tms.index almost done. You should be able to call it from your script to get the series fsid. I expect to have it done tonight. In the meantime, if someone wants to try what I described in post #5, I'll send you that program. It should work, it's just suboptimal.
I attached the interim program and scripts to post #5.

jbuehl
12-01-2004, 02:58 AM
I have tried a few transfers to a S2 D-Tivo and it is mostly working.

Shows that were extracted from a S2 get placed into existing folders. They are linked to the correct series and station objects. I haven't yet tried importing 2 or more shows that would result in creation of a new folder.

The bad news is that shows that were extracted from a S1 SA cause a crash:


05:55:52:AM - 220 Mfs_Ftp ver 1.2.9p-Jamie-ChrisEd:V1 - {sock24} from "192.168.1.12:33264"
05:55:53:AM - 331 User name okay, need password.
05:55:53:AM - 230 Running in TiVo Mode.
05:55:53:AM - 215 UNIX
05:55:53:AM - 200 Type set to I
05:55:53:AM - 257 "/" is current directory.
05:55:53:AM - 200 PORT command successful.
05:55:53:AM - 150 Opening ASCII mode data connection for file list.
05:55:53:AM - 226 Transfer complete.
05:56:07:AM - 200 PORT command successful.
05:56:09:AM - 150 Opening BINARY mode data connection for "{SouthPark-SimpsonsAlreadyDidIt-20020630.tmf}"
bgerror invoked with error

" can't read "stationpath": no such variable "

re-initializing mfs_ftp

Here is a snippet of the xml:


<Object type="Station" id="Station">
<Affiliation>Satellite</Affiliation>
<CallSign>COMEDYP</CallSign>
<Name>Comedy Central (Pacific)</Name>
<ServerVersion>76</ServerVersion>
<TmsId>10150</TmsId>
</Object>

It looks like the Station object doesn't contain a ServerId tag which is causing the code in setrecinfo that initializes stationpath to be skipped. This same show imports successfully using mfs_ftp with Jamie's mods.

I'll continue testing.

chrised
12-01-2004, 05:06 AM
The bad news is that shows that were extracted from a S1 SA cause a crash:


" can't read "stationpath": no such variable "


Here is a snippet of the xml:


<Object type="Station" id="Station">
<Affiliation>Satellite</Affiliation>
<CallSign>COMEDYP</CallSign>
<Name>Comedy Central (Pacific)</Name>
<ServerVersion>76</ServerVersion>
<TmsId>10150</TmsId>
</Object>

It looks like the Station object doesn't contain a ServerId tag which is causing the code in setrecinfo that initializes stationpath to be skipped. This same show imports successfully using mfs_ftp with Jamie's mods.

I'll continue testing.

I've posted Beta 1.1 http://www.dealdatabase.com/forum/showpost.php?p=196853

That should fix the error.

I'd like to fix the station linking for SA tivos. To do so, I need someone to send me an objdump of a station (the xml above has some properties removed that I need to see).

rc3105
12-01-2004, 06:09 AM
OK, Riley has spoken. Apparently I'm a moron for posting the entire modified mfs_ftp.tcl file. I've decided further comments on the matter won't serve any purpose, so I'll hold my tongue (keyboard in this case).

lot of threads lately that touch on respecting dev's wishes, especially when they're spelled out in a license...


Looks like I have two choices. Post a diff patch (painful to me and the users to apply), or implement a plug-in module/patch.

posting alpha/beta stuff as diff patches isn't really a bad thing, it tends to prevent the truly clueless from hurting themselves


I've opted for the plugin patch (I am pleasantly surprised how easy that is to do)

that's what the module sourcing scheme's there for. look how well it's worked for tivoweb ;)


Its always nice spending 20+ hours developing something someone else has already done. Or has it been done? I don't see it publicly available.
there was a tivoweb module to change/manage grouping on tcf back when 4.x was first released. it ocassionally bjorked the scheduler & season passes. by the time the was an s2 with a folder capable sw version here I'd lost site of it (pun intended)


Enough people are asking for it. I privately asked for it, and got nothing.

other things on the todo list besides tutor MFS 101 - see alphawolf (http://www.dealdatabase.com/forum/member.php?userid=1793)'s sig re public / private discussion

*no you don't need to establish links while creating objects, it's perfectly feasable to sync them up later


Riley's messages (that I've found) never indicated having a version that did this during the inserts. All I could find was reference to a utility to fix them "Afterwards". And that utility is also not being shared.

lotta toys in that category for various reasons - in this case mostly lack of time to test all the possible permutations and side effects


I needed this now. And I decided I would share it with the group as a contribution for all the neat stuff I've been using. Thats supposed to be the idea here, right?
very cool. nice to see another person on the dev side of the producer/consumer fence

Jamie
12-01-2004, 09:58 AM
I'd like to fix the station linking for SA tivos. To do so, I need someone to send me an objdump of a station (the xml above has some properties removed that I need to see).
Here's a dump of one station from 4.0.1b and 5.1.1b, both S2SA's. The only difference between the two is the FSID and the path in /StationTms (since it includes the hex encoded FSID).

jbuehl
12-02-2004, 03:13 PM
What I did was 'hijack' seriesIDs of series I would never record for non-series stuff.
I had thought of doing that too. Presumably you change the Title of the series object to be something else. Does the title stay changed, or does the Tivo software periodically refresh it
with the real title?

alldeadhomiez
12-02-2004, 03:22 PM
I had thought of doing that too. Presumably you change the Title of the series object to be something else. Does the title stay changed, or does the Tivo software periodically refresh it
with the real title?

If it comes from a slice (as SA guide data does), the object will be overwritten when the ServerVersion increments. Since the ServerVersion starts at 1, you can tell how many times a given object has changed at the headend by looking at the current value. You can also look at the Version attribute to see how many times it has been changed on the local machine.

I'm not sure how versioning of APG objects works, but I would expect it to be similar so as to reduce load on the IRDs.

philhu
12-02-2004, 03:55 PM
I had thought of doing that too. Presumably you change the Title of the series object to be something else. Does the title stay changed, or does the Tivo software periodically refresh it
with the real title?


Hmmm...good point. That might explain why some of my objects got put back, after months, to the old names. When that show came in the listings again, with the same ID's, it rechanged the series id names.

With canadian listings, although we use the Zap2it/tribune tmsid, we make our own seriesid, programid, etc.

So, I started going into the seriesID database on the machine creating slices and changing the id's there, that get used to create the slices, in essence hijacking the name and the ID for a show I want to group. Then take the old show name out completely. The first new episode of one of these shows, we create a new series id for it.

chrised
12-03-2004, 03:24 AM
With cross-platform series linking! http://www.dealdatabase.com/forum/showpost.php?p=196853

Please give it a try, and leave some feedback on your results. I'm particularly interested in hearing how it works on SA to SA and SA to/from DTivo transfers.


About the series lookup by TmsId.

I had started writing my own and was about half done. I ran into some issues with converting the 10 digit decimal number to binary (overflows a longint), and in trying to solve that issue I stumbled upon code in TivoWebPlus which did exactly what I was looking for. After testing that code (and fixing a bug in their code), I was able to successfully integrate it into my patch. The search performance I measured was less than a second per lookup, so I don't see any need to try to optimize this code further.

Assuming we get some good feedback on this version, we may be ready to make it a release.

One open issue yet is whether to allow it to create a missing series. I still have that code turned off. From what I can tell on DTivo, it appears that all series information is constantly updated. (Btw, someone asked about ServerVersion on DTivo series objects. I have never found any values other than zero, so I don't think they even use that field).

In particular, I feel that inserting series objects during cross platform transfers is risky. If I use the original server id, it will be the wrong format for the destination. If we make one up, there are risks that later when Tivo sends the real one out for the series it will not match, and then you may have two folders for the same series.

hxmiller
12-03-2004, 09:48 AM
I extracted several shows using the unscramble kernel from 3.1.1c to a PC. I've inserted these shows onto a 4x Dtivo.

Extracting and re-inserting the already inserted shows to/from the 4.x machine didn't seem to do the trick.

I'm assuming I need to go back to the original 3.1.1c drive and re-extract/unscramble to shows for the add-on to mfs_ftp to work.

chedlin
12-03-2004, 10:35 AM
I extracted several shows using the unscramble kernel from 3.1.1c to a PC. I've inserted these shows onto a 4x Dtivo.

Extracting and re-inserting the already inserted shows to/from the 4.x machine didn't seem to do the trick.

I'm assuming I need to go back to the original 3.1.1c drive and re-extract/unscramble to shows for the add-on to mfs_ftp to work.

This surprised me, as Riley has stated that the origional XML is restored when an inserted recording is extracted. I will compaire the xml on a few shows tonight.

I truely hope this works, I have a large archive of Stargate SG1 that was driving my wife nuts on 3.1.1c because it cluttered the Now Playing list (and slowed things down). I had to put them on my old S1 SA (not connected to any program source) so I could return the 200GB HD I borowed from a friend during the upgrade to store my recordings. I didn't have that much space anywhere else. So if the xml data doesn't get preserved like I thought, I will be up the creek.

rc3105
12-03-2004, 11:37 AM
the xml from a tmf/ty+ is stored to /var/mfs_ftp/cache/recfsid#.xml during insert. if that cached xml is available during extract it's used instead of what's in mfs. if var was rebuilt you might lose some details when the xml is generated from the limited info that was restored to mfs


*on a lighter note, I have XML for nearly every stargate episode and archived gateworld pages with more detailed info if you need it :D

hxmiller
12-03-2004, 11:57 AM
I extracted the inserted show then I immediately reinserted onto the same Tivo. Think clearing the cache before the insert would change the results? I looked at the showing.xml of the extracted show using the normal mfs and the p1 mfs and they are the same. No series tags in the document.

I'm going add some debugging to the log to the add-in to follow the flow.

chrised
12-03-2004, 12:15 PM
I extracted the inserted show then I immediately reinserted onto the same Tivo. Think clearing the cache before the insert would change the results? I looked at the showing.xml of the extracted show using the normal mfs and the p1 mfs and they are the same. No series tags in the document.

I'm going add some debugging to the log to the add-in to follow the flow.

When working with shows that were previously inserted, most of the xml attributes were lost during the previous insert. This patch has no way to recover those attributes. It is easy to see the difference in the xml. Download the xml from the source and destination systems (from the original transfer). Open them up, and you'll see the destination xml is more than half gone.

Riley suggested that if you could get the original xml and place that in the cache before the "new" extraction, it would use that xml and not the real mfs attributes, and then the subsequent insert should work.

jbuehl
12-04-2004, 01:38 AM
One open issue yet is whether to allow it to create a missing series. I still have that code turned off. From what I can tell on DTivo, it appears that all series information is constantly updated. (Btw, someone asked about ServerVersion on DTivo series objects. I have never found any values other than zero, so I don't think they even use that field).

In particular, I feel that inserting series objects during cross platform transfers is risky. If I use the original server id, it will be the wrong format for the destination. If we make one up, there are risks that later when Tivo sends the real one out for the series it will not match, and then you may have two folders for the same series.
I tried manually creating a new series object for some existing shows. I used a bogus TmsId SH900000 that didn't already exist and isn't likely to in the near future. I set the ServerId to ATSH900000. For the existing shows I set the Program/TmsId to SH9000000000 and Program/Series to the new fsid.

The shows immediately showed up in a folder with the name that matched the Title of the new Series. Wierd stuff happened when I tried to play the shows, however. I played a few seconds and then after going back to "Now Playing" they disappeared. tivoweb didn't list them as deleted.

After a reboot, they were there again and are now in the new bogus folder and can be played and they don't vanish. Recordings/TmsId also lists them under the new TmsId. A MRV transfer to another Tivo results in a non-grouped show.

I don't know what changed in the reboot to fix things. I didn't capture the previous state with Jamie's dumpobj so I'll have to repeat the experiment.
-----
Update: The new series appears in Tms.index. As far as I can tell it's now a legitimate series as far as the tivo is concerned. I don't know when this index gets rebuilt. Presumably when there is series information in the guide data it would be. An SA gets it by making a call, but DTV guide data is probably being continuously trickled in every transport stream. Creating a new series properly would need to involve triggering the index rebuild.

chrised
12-06-2004, 10:31 PM
I've transferred about 100 shows DTivo 3.1.1x to DTivo 4.x. I encountered a couple of shows that gave me this error:


12:10:09:AM - 150 Opening BINARY mode data connection for "{{Rudolph the Red-Nosed Reindeer}{1964-12-06}{}{07.00 PM Wed Dec 01, 2004}{MN4}.tmf}"
bgerror invoked with error

" can't open object (errDbNotFound)
"

I traced this down to an interesting situation. This show used the TMS search to lookup the series. The TMS search returned an object ID that doesn't exist in MFS. I manually verified the TMS index data, and found that the data returned by the search was valid.

Therefore, it is my conclusion that series objects may get purged from MFS, but not from the index.

To compensate for the error, I've added a verification process. Once that code was installed, my error went away.

At the same time, I added a series create attribute to settings.tcl to make it easier to turn the feature on and off. For my few problem shows, I allowed it to create the missing series.

It will be interesting to monitor the index and see if and when the new series is added.

Anyhow, Beta 1.3 is posted http://www.dealdatabase.com/forum/showpost.php?p=196853

Edit: The new series was updated in the index and the old (orphaned) series was removed. My new series was created at 6:45, I checked at 8:11 and it wasn't there yet. The next time I checked was 12:42 and this time it was there. If anyone knows anything about this "update" process, I'd love to hear more about it.

chedlin
12-07-2004, 02:03 AM
I have not had much success with Beta 1.3. I didn't have a chance to test with earlier versions. When trying to insert recordings my TiVo reboots.

When I start a transfer the show will appear in now playing right away, in the correct group (an existing series, Stargate SG-1). On the first attempt I tried to play it while it was copying and couldn't. mfs_ftp crashed but the TiVo didn't imediately reboot. I did have to reboot it to start up mfs_ftp again. Later I transfered the show to see if it would work without the patch.

The log file didn't survive the reboot, so I can't provide more information at this time. I'll look into it later (my wife has run out of patience).

Charlie

jbuehl
12-07-2004, 02:37 AM
1.3 is working great for me. I had been reliably inserting shows into existing series on a S2 D-tivo that were extracted from both another S2 D-tivo and a S1 SA using 1.2.

I edited the xml of a couple of S1 originated shows to change the series TmsId and the episode TmsId to a non-existent TmsId. I also changed both occurrences of the Title.


<Object type="Program" id="Program">
...
<IsEpisode>1</IsEpisode>
<NetworkSource>fox</NetworkSource>
<OriginalAirDate>12554</OriginalAirDate>
<Object type="Series" id="Series">
<Episodic>1</Episodic>
<Genre>4</Genre>
<Genre>91</Genre>
<Genre>1000</Genre>
<Genre>1002</Genre>
<ServerId>16642</ServerId>
<ServerVersion>520</ServerVersion>
<ThumbData>268566912</ThumbData>
<Title>The Zimpsons</Title>
<TmsId>SH990000</TmsId>
</Object>
<ServerId>24055378</ServerId>
<ServerVersion>8</ServerVersion>
<ShowType>5</ShowType>
<SourceType>2</SourceType>
<Title>The Zimpsons</Title>
<TmsId>EP9900000356</TmsId>
</Object>

Both shows inserted fine and were grouped under the fake series. According to the mfs_ftp log, the new series object was created during insertion of the first show, and it was found via the Series ServerID the second time. The new TmsId does appear in the index.

Everything looks right, the shows play, and nothing crashed. Nice work, Chris!

chedlin
12-09-2004, 01:03 AM
Well, it may have been the vwait tivoapp patch. I backed it out, and have things working. I had a failed recording that was causing tivoweb to crash and the tivo to reboot if I tried to play or delete it. After removing the recording from tivosh, I am reluctent to do much further testing for fear I might not be able to fix things.

Thanks for the great patch, and of course thanks to Riley for the app that made this possible to begin with.

Charlie

chrised
12-09-2004, 01:41 AM
I ran into a new error tonight. I was transfering a couple of movies recorded back in March.

The error:

07:50:20:PM - SeriesGenre :: 105
bgerror invoked with error

" commit failed (0x70003)

The cause:

Series 78573/10 {
Version = 21
ServerVersion = 0
Episodic = 0
Genre = 1 369 257 105
ThumbData = 268566912
Title = {A Bug's Life}
TmsId = MV065381
IndexPath = /Database/Orphan78573
}

This movie, which was never transferred before, apparently had its ServerId removed from the series and consequently became an Orphan. Apparently due to some "purging" process that must have occurred sometime.

My code didn't look to see if the ServerId was missing, and that caused the error. Assigning a blank ServerId was fatal, leaving it unassigned would have worked. My fix was to "reject" creating the series if the ServerId was missing. I figured this was the safest action.

The alternative would have been to create it without the ServerId, but unless it made it into the TMS Index it wouldn't have been useful anyway.

jbuehl
12-09-2004, 03:12 AM
I just transferred some movies. The Series in all my cases have a ServerId.


Series 355585/10 {
Episodic = 0
Genre = 1 371 32 34 37 111 116 105 375
IndexPath = /Server/AR001f5f2b:4:0:0
ServerId = AR001f5f2b
ServerVersion = 0
ThumbData = Predicted 1 149
Title = {Lost in Translation}
TmsId = MV140381
Version = 14
}

I did notice that on the tivo that recorded it, the Program has a RootServerId attribute which contains the same number 001f5f2b as the Series TmsId. This attribute is not present in the tivo that the movie was transferred to.


Program 325840/10 {
Actor = Murray|Bill Johansson|Scarlett Ribisi|Giovanni Faris|Anna Hayashi|Fumihir o Lambert|Catherine
Advisory = 10 1 3
ApgProgram = 325840/11
DescLanguage = English
Description = {A middle-aged actor (Bill Murray) falls for a young woman (Scarlett Joha nsson) in Tokyo.}
Director = Coppola|Sofia
Genre = 1 371 113 32 34 37 111 116 105 375
IndexPath = /Server/A00229600:3:1:0
IsEpisode = 1
MovieRunTime = 102
MovieYear = 2003
MpaaRating = 4
OriginalAirDate = 12052
RootServerId = A001f5f2b
Series = 355585/-1
ServerId = A00229600
ServerVersion = 1
Title = {Lost in Translation}
TmsId = MV1403810000
Version = 2
}

I wonder if that can be used to reconstruct the ServerId of the Series. I have seen that a ServerId seems to be required when creating a Series.

chrised
12-09-2004, 03:29 AM
I did notice that on the tivo that recorded it, the Program has a RootServerId attribute which contains the same number 001f5f2b as the Series TmsId. This attribute is not present in the tivo that the movie was transferred to.

I didn't program it to include RootServerId. It certainly is possible to add.



I wonder if that can be used to reconstruct the ServerId of the Series. I have seen that a ServerId seems to be required when creating a Series.

Its an interesting thought. From some spot checking, I didn't see consistencies. Of three programs with orphaned series, two had rootserverid and one didn't.

I'd be curious to know if anyone else understands the RootServerId's purpose, and whether this looks like a good idea.

jbuehl
12-09-2004, 01:47 PM
I didn't program it to include RootServerId. It certainly is possible to add.

Its an interesting thought. From some spot checking, I didn't see consistencies. Of three programs with orphaned series, two had rootserverid and one didn't.

I'd be curious to know if anyone else understands the RootServerId's purpose, and whether this looks like a good idea.
I see that some of my movies have RootServerIds and some do not. Looking through the /Server objects, the ones whose names begin with "A" seem to follow this pattern:

A<hex number> - guide related?
AR<hex number> - guide related?
ASER<name> - sports events
AStation<decimal number> - stations
ATMV<decimal number> - movie TmsId
ATSH<decimal number> - series TmsId
ATSP<decimal number> - sports TmsId

My guess would be that the A<hex number> ones are where the movie Series object starts out when it is in the guide. Once it gets recorded, it (may) get renamed to a ATMV
ServerId. If this is true, you could probably get away with creating the movie Series with the ATMV name.

philhu
12-10-2004, 12:57 PM
I see that some of my movies have RootServerIds and some do not. Looking through the /Server objects, the ones whose names begin with "A" seem to follow this pattern:

A<hex number> - guide related?
AR<hex number> - guide related?
ASER<name> - sports events
AStation<decimal number> - stations
ATMV<decimal number> - movie TmsId
ATSH<decimal number> - series TmsId
ATSP<decimal number> - sports TmsId

My guess would be that the A<hex number> ones are where the movie Series object starts out when it is in the guide. Once it gets recorded, it (may) get renamed to a ATMV
ServerId. If this is true, you could probably get away with creating the movie Series with the ATMV name.

The SH id's are for Series, generic descriptions (Shows), there is another called EP, which is series Episode listings. The former used for generic description of a show when an episode is not listed.


These come directly from tribune/zap2it

blueman2
12-29-2004, 05:11 PM
For what it is worth, I ended up using this patch in order to download 200 Gigs of shows to my PC, then re-upload them to my HDVR2 after a drive change. Worked perfectly. All folders are there just like before. I did have to wait about 3 hours after installing a freshly upgraded 4.01b system before things settled down enough to do the uploads from PC back to the Tivo. I think it needs to get some info from the satellite such as series IDs and such. Some uploads would not work at first. But by waiting a couple hours, everything then went great.

VERY nice job with this patch!!!!

makinrain
08-19-2005, 02:20 PM
1.3 is working great for me. ...

I edited the xml of a couple of S1 originated shows to change the series TmsId and the episode TmsId to a non-existent TmsId. I also changed both occurrences of the Title. Both shows inserted fine and were grouped under the fake series. According to the mfs_ftp log, the new series object was created during insertion of the first show, and it was found via the Series ServerID the second time. The new TmsId does appear in the index.

Have you tried this with 6.2 yet ?

Also, I'm thinking this might be useful in fixing the generic episode problem. Could extract the tmf, update the xml with fake tmsid's and new series, and episode descriptions.

Have you found any problems with the tmsid's that you chose ?

Callindril
11-21-2005, 01:16 AM
Trying to upload some home movies into my DSR-708, and while the shows are uploading properly, they arent being put into a new folder...

I used the above posted XML example of using a Bogus tmsid (990000), but the folder isnt created.

Here is the XML Im using for one of the files...the others are the same with just the description and the episode number (EP990000000x) incremented by one for each file...The descpriptions and names are all correct, just no folder info created for the new bogus folder...


<?xml version="1.1" tivoversion="3.1.1c-01-2-301"?>
<Object type="Recording" id="_top">
<BitRate>0</BitRate>
<SelectionType>3</SelectionType>
<SubObject type="Showing" id="Showing">
<Date>13002</Date>
<Duration>7200</Duration>
<Object type="Program" id="Program">
<SubObject type="ApgProgram" id="ApgProgram">
<Category>11403648</Category>
<Category>11272297</Category>
<Category>11272316</Category>
</SubObject>
<DescLanguage>English</DescLanguage>
<Description>Andie playing in the snow outside Grandma's house, December 2004</Description>
<EpisodeTitle>Andie - Snow 2004</EpisodeTitle>
<Genre>384</Genre>
<Genre>105</Genre>
<Genre>124</Genre>
<IsEpisode>1</IsEpisode>
<RootServerId>A000322f8</RootServerId>
<Object type="Series" id="Series">
<Genre>384</Genre>
<Genre>105</Genre>
<Genre>124</Genre>
<ServerId>ATSH016872</ServerId>
<ServerVersion>0</ServerVersion>
<ThumbData>268632831</ThumbData>
<Title>Andie Goodies</Title>
<TmsId>SH990000</TmsId>
</Object>
<ServerId>A00211d3c</ServerId>
<ServerVersion>2</ServerVersion>
<ShowType>5</ShowType>
<Title>Andie Goodies</Title>
<TmsId>EP9900000002</TmsId>
<TvRating>4</TvRating>
</Object>
<Object type="Station" id="Station">
<CallSign>SCFI</CallSign>
<Name>SCFI</Name>
<ServerId>AStation11097</ServerId>
<ServerVersion>61</ServerVersion>
<TmsId>11097</TmsId>
</Object>
<Time>28800</Time>
</SubObject>
<StartDate>13008</StartDate>
<StartPadding>60</StartPadding>
<StartTime>28738</StartTime>
<StopDate>13008</StopDate>
<StopTime>32700</StopTime>
<StreamFileSize>958464</StreamFileSize>
<SubPriority>8</SubPriority>
<UsedBy>1</UsedBy>
</Object>

DarkHelmet
01-17-2006, 01:29 PM
The tms_to_fsid procedure has a 3.x - 5.x specific pathname encoded in it. /GuideIndexV2 became /GuideindexV3 in 6.x, according to tivowebplus.

I made the following change to p1.tcl:


proc tms_to_fsid {tmsid} {
# global guideindexdir
# global db
global version
if {$version >= 6} { set guideindexdir "/GuideIndexV3" } else { set guideindexdir "/GuideIndexV2" }


This is to replace the hard coded
set guideindexdir "/GuideIndexV2"
line.

Otherwise when inserting to 6.2, you get errDbNotFound aborts from tms_to_fsid, regardless of the info(seriescreate) setting.

Other than that, it seems to be working on 6.2. With the usual mfs_ftp edits of course. (fix tzoffset.tcl, fix the event lines by commenting them out, etc)

Time for 1.6 perhaps?

chrised
01-18-2006, 01:03 AM
I wasn't exactly ready to share this yet, but here it is anyway.

Working good with 6.2. I've used it to transfer 400+ shows from my old V4 systems to new drives / 6.2 systems.

Didn't get around to making it backwards compatible with 4.

This includes other enhancements, particularly setting the program server id. This prevents Tivo from re-recording the episode if its broadcast again.

drez
01-18-2006, 01:27 AM
cant wait to try it out, thanks for your work chrised

brendag4
01-19-2006, 05:29 AM
Other than that, it seems to be working on 6.2. With the usual mfs_ftp edits of course. (fix tzoffset.tcl, fix the event lines by commenting them out, etc)

Where do I find out about the edits that need to be made? The post after this one has a link to the .tcl file, but it does not contain info on what needs to be changed.

I had thought this was the official post to download the file from, but it is only up to 1.5...

http://www.dealdatabase.com/forum/showpost.php?p=196853

It contains info about a change to be made, but it doesn't seem to be the same change you are referring to.

I have tried to find posts about the program but maybe I am missing some because the program just seems to be called "beta".. so that makes it hard to know how to search for it. This is the only thread I know of besides the one I mentioned above.

thanks

Brenda

brendag4
01-19-2006, 05:40 AM
Didn't get around to making it backwards compatible with 4.
Does this mean it won't work on a Series 1 with v 3.x?

I am using Beta v1.5 (just found out about v1.6.2) on a Series 1 Philips. It seems to be working fine.

Thanks for the great app! I didn't really get it for the folder thing, I got it so that stuff like actors would show on the TV screen.

I noticed that some info shows on the TV screen but does not show up on TivoWebPlus 1.2.1. I don't know if this is from TWP or Beta 1.5. The items are "TV Rating, Type, and Bits". I would try 1.6.2 to see if it is fixed there but it sounds like it might not be for my Tivo. (and of course it might be TWP dropping the info)

thanks,

Brenda

chrised
01-20-2006, 03:07 AM
I took pity, and added the code to make the GuideIndex version specific, so hopefully this version will work on all systems.

To recap on what changed since 1.5:
- Now restoring ProgramServerID and Version. This keeps Tivo from re-recording the same episode when re-broadcasted.
- As a consequence of that, added code to see if program already existed in MFS, and link to it when possible. (Note: This code may not be SA friendly, as in might not locate a match when it should.)
- Added code to improve linking to existing station items (Keeps those pretty logos intact)
- GuideIndex supports versions 6+ which use GuideIndexV3 path (Thanks DarkHelmet for the code snippet, you made that easy for me)

Personally, I've only tested this stuff on DTivo 6.2

angra
02-20-2007, 09:12 AM
Just stumbled across this thread after a few hours of work trying to do the same thing. Mainly I was trying to figure out if the tms_to_fsid functions I have on my local drive were derived from somewhere else (they're not, but I see that TWP, and others, have updated and fixed them a bit). Some of the "tricky" stuff above had to be written for the original SP/WL backup programs that became a module long ago.

I was going down the same technical path with parsing and reconstruction methods, so this saves me the hassle, and I wanted to thank the authors in the thread.

PS I had to make a minor change to work on a version 8.X box. ApgProgram is apparently not part of the "program" object any more, so I added a version check to skip the appropriate database entries in that case.