folders for inserted and non-series shows

[jbuehl]

I'm starting a thread in this group to discuss development of ways to group shows in folders that have been inserted with mfs_ftp or otherwise wouldn't be grouped because they aren't part of a TV series. There has been some previoius discussion in these threads:

http://www.dealdatabase.com/forum/sh...ad.php?t=39523
http://www.dealdatabase.com/forum/sh...ad.php?t=36208
http://www.dealdatabase.com/forum/sh...ad.php?t=37413

When a show is recorded, the recording object gets linked to a series object that is created from the guide data. An inserted show must be linked to a series object in order to be grouped. The problem is that the series object can't be easily found without a long search of the database.

For shows that there isn't a series object for in the database, the problem is worse. Creating a new series object is hazardous to the database, according to Riley. I'm sure that's true, but I would be interested in hearing from anyone who has tried it and what problems they encountered.

[jbuehl]

Creating a link to a series in mfs_ftp seems straightforward enough, assuming one knows the fsid of the series object. The series can be identified by the TmsId that is contained in the xml of the .tmf being imported. Searching through the database for an object containing a specific TmsId can take many minutes.

I have written a C program that pulls out all the TmsId -> fsid relationships of series objects and stores them in an index file. This file can be very quickly (< 1 sec) be searched to return the fsid that corresponds to a TmsId. The idea would be to run this indexing program once a day or so to keep it up to date. I need to compile this for mips and I will make it available to anyone who wants to try it.

I am currently figuring out how to modify tcl (can anyone suggest a good tutorial?) so I can modify mfs_ftp to use it.

[alldeadhomiez]

I have written a C program that pulls out all the TmsId -> fsid relationships of series objects and stores them in an index file. This file can be very quickly (< 1 sec) be searched to return the fsid that corresponds to a TmsId. The idea would be to run this indexing program once a day or so to keep it up to date. I need to compile this for mips and I will make it available to anyone who wants to try it.

It's built in... look at /GuideIndexV2/Tms.index:

# hexdump Tms.index | grep "0005 5730"
006db00: 5348 0000 0000 0000 0005 5730 0000 656f
# mfs_dumpobj `printf "%d" 0x656f`
Program 25967/11 PRIMARY {
        ServerVersion=112 
        ColorCode=4 
        DescLanguage=English 
        Description=The television magazine series covers a broad spectrum of modern life. 
        Genre=106 180 1003 1007 1010 
        NetworkSource=cbs 
        OriginalAirDate=-464 
        Series=25966/-1 
        ShowType=5 
        SourceType=1 
        Title=60 Minutes 
        TmsId=SH0000350000 
        ServerId=19061 
        Version=1 
        IndexPath=/Server/19061 
}

Since this index is ordered by TmsId, you can use a binary search so that your lookup time is bounded by O(lg n). You should be able to get very quick lookups by extending the vplay code (instead of using FromMfs to yank the index on each lookup). Obviously, there are cache/race problems which need to be taken into consideration if you use this method, so maybe FromMfs can be retained as a fallback if the lookup fails.

It is also interesting to note that the TmsId for a given series or program is consistent across headends, including the DirecTV (APG) service.

The two character alphanumeric prefix appears to denote the following object types:

00 = Station
EP = Program (TV)
MV = Program (movie)
SH = Series
SP = Program (sports)

[jbuehl]

It's built in... look at /GuideIndexV2/Tms.index:

# hexdump Tms.index | grep "0005 5730"
006db00: 5348 0000 0000 0000 0005 5730 0000 656f

"0005 5730" is hex of the TmsId 350000 which references the program, but how does that help to find the series object? The problem is, given program object in fsid 25967 with the Series field blank, what is the fsid (25966) of the series object that has TmsId SH000035.

What is the format of the entries in Tms.index? Also, what is FromMfs? I'm not familiar with that. The program I wrote is based on the tridge/vplay code.

It is also interesting to note that the TmsId for a given series or program is consistent across headends, including the DirecTV (APG) service.

That shouldn't be a surprise because Tribune Media Services is the guide data aggregator for Tivo, DirecTV, and most cable companies.

[jbuehl]

Here's an interim way to find a series fsid given the TmsId. It searches /Server and builds a temporary index file in /var/tmp that can be searched.

mfs_tmsid is the program that builds the series index. It takes about 15 minutes to run on a HDVR2.

bash-2.02# ./makeSeriesIndex
54974 entries found in /Server
7926 TmsIDs found

The tmsid.idx file contains fsid, TmsId, and title of all the series objects found. The title isn't really necessary, so it can eventually be deleted to reduce the file size.

bash-2.02# head /var/tmp/tmsid.idx
297444 SH000001 {Paid Programming}
305387 SH000021 {20/20}
342688 SH000030 {3-2-1 Classroom Contact}
312648 SH000035 {60 Minutes}
312763 SH000051 {A Different World}
314463 SH000066 {A Pup Named Scooby-Doo}
340472 SH000115 {The Addams Family}
344545 SH000150 {Against All Odds: Inside Statistics}

Finding a particular series given the TmsId is fast. The shell script getSeriesFsid does it:

# ./getSeriesFsid SH000115
340472

This should be able to be called from a tcl script.

Note: There are some duplicate entries for the same TmsId. This needs to be taken into account.

# grep SH476707 /var/tmp/tmsid.idx
305262 SH476707 {What Not to Wear}
349357 SH476707 {What Not to Wear}

[jbuehl]

"0005 5730" is hex of the TmsId 350000 which references the program, but how does that help to find the series object? The problem is, given program object in fsid 25967 with the Series field blank, what is the fsid (25966) of the series object that has TmsId SH000035.

What is the format of the entries in Tms.index? Also, what is FromMfs? I'm not familiar with that. The program I wrote is based on the tridge/vplay code.

OK, now I get it. Your color coding was just a bit too subtle for me, and I took your example too literally

And I found the post that describes the layout of the tms index. Do you suppose it would be worthwhile adding it to mfs.h?

[chrised]

I've been working on this for the last week. My objective is to decrypt, migrate, and reorganize all my recordings from my old 3.1.1x systems to new 4.x systems (all DTivo). In this process, I did not want to lose any information for the Tivo UI or TivoWebPlus. The current mfs_ftp makes a minimal effort at this. If you compare it to MRV, you'll see MRV preserves all attributes of the recordings. With this updated version, you won't be able to tell the difference!

My starting point for mfs_ftp.tcl is based on Jamie's modifications from this message: http://www.dealdatabase.com/forum/sh...ad.php?t=39487
The instructions for that install ask for certain changes to the supporting binaries. Those must be in place for this code.


My testing has been all DTivo, transfers from 3.1.1x and 4.x to 4.x. Strictly TMF format! (the others might be broken).

For folder support, it is critical that all shows be linked to the correct series. This must be done before the recording is saved, or you have to figure out how to correct the recording's indexing. (ick)

There's been lots of discussion about using the TmsID. What's been overlooked is that Tivo assigns each TmsId a ServerId. Using ServerId, it isn't that difficult to find the series object once you figure out the data pattern.

This code is also setup to CREATE series objects that look just like Tivo's objects. It appears to be slightly risky. Testing will prove whether that is true.

It seems that TMS assigns SeriesID's to almost every show, even those you wouldn't expect. It is difficult to find shows without them. And, surprisingly even shows that haven't aired in many months seem to get their series information sent down to new tivos. My only way to test creating a new series was to temporarily hardcode it to a fake TMSId. I do recommend that you allow a new Tivo to gather guide data for a day before transfering shows. This way it should reduce the chance of conflict between tivo's series objects and my objects (which I still think will be one in the same based on the indexing).

This code will use the same concept to link to the real station record instead of a fake one. This allows additional information such as channel number and logo to display where appropriate. Clearly, if you are transfering from SA to DTivo the channel ID's are unlikely to match (they might, but I don't have an SA to try with).

I spent alot of time making sure that every attribute is supported. This means every piece of text on the program info screen. A list of added attributes includes:

   Bits EpisodeNum AltEpisodeNum IsEpisode 
   MovieRunTime MovieYear MpaaRating 
   ShowType TvRating 
   ApgProgram (Odd thing that controls CC, Black and White, Series, etc)
   Advisory (AC, AL, etc)
   Genre (Action/Adventure, Drama, etc)
   Actor Director Writer GuestStar Host ExecProducer Producer
   Premier

Some of these attributes I've never seen. I added them based on the TivoWebPlus code.

There is a correction for expiration date in here too.

I wouldn't be surprised if this code is not compatible with older Tivo software under version 3.1.1.

This code is barely considered BETA.
USE AT YOUR OWN RISK, HAVE A BACKUP, AND BE PREPARED TO FIX YOUR OWN PROBLEMS.

Also, beware, I've done everything that Riley has warned us is dangerous. Riley understands the Tivo much better than I do. He might be right about this.

This code is being posted here to allow others to do further testing, to review the code, and to let the brave (foolish?) try it out. Ultimately, it would be nice if it could become adopted into the next official release of mfs_ftp.

Good luck!

Edit 11/30: WARNING: It is now confirmed that ServerId varies between SA and DTivo. That renders my solution only valid for DTivo to DTivo transfers. If you attempt a cross platform transfer, shows will not be grouped into the appropriate series. They may be grouped under the original series on the "foreign" platform. This will result in potentially two series objects with the same TmsId. The results of that may cause problems.

Volunteers to help fix this would be appreciated. (I've got what I need for now, so I don't have alot of motivation to spend more time on it)


-----
attachment removed - please post patches / modules as per the license

[chrised]

Originally I posted the entire modified mfs_ftp.tcl file. Since that apparently is forbidden, it looks like I have two choices. Post a diff patch (painful to me and the users to apply), or implement a plug-in module/patch. I've opted for the plugin patch (I am pleasantly surprised how easy that is to do)

12/1/04 Beta 1.1 features:
- fix uninitialized variables affecting SA1 > Dtivo2 transfer
- should support series linking for SA to SA transfers.
- does NOT support series linking for cross-platform yet.
- disabled creating series objects. Want to see results of requested test before considering re-enabling http://www.dealdatabase.com/forum/sh...9&postcount=23

12/3/04 Beta 1.2 features:
- supports station matching on SA.
- will not support cross-platform station matching. At this time, there isn't good enough reasons to justify building that feature.
- supports cross-platform series matching. This was done by incorporating some code from tivowebplus. I did end up fixing a bug in that code. This code can lookup a series by TmsId in about a second. Can't complain about that performance.

12/6/04 Beta 1.3 features:
- Bug fix for TMS search returning invalid object id. (object no longer in MFS, but is in index)
- User configurable Series creation

12/7/04 Beta 1.4 features:
- Bug fix for Series Lookup. Had originally assumed all DTivo series ServerId started with AT. Found one with AR, and it failed. New approach tries both DTivo and SA ServerId formats before trying the TMS search.

12/8/04 Beta 1.5 features:
- Bug fix for error during series create when no ServerId assigned to series


With the plugin, it no longer matters whether you used Jamie's patches / changes first. So, its gotten even easier to install. Make sure you have an operating version of mfs_ftp 1.2.9p installed. Copy the p1.tcl file into the mfs_ftp directory. Restart mfs_ftp.

As of Beta 1.3+, you need to add a line to settings.tcl or you will get an error:

set info(seriescreate) 0                        ;# Allow Series to be created? 0 = disabled, 1/others = enabled.

At this time, I feel its safe to enable seriescreate when you are transferring on the same platform (ie DTivo to DTivo, or SA to SA). Using this for cross-platform transfers may have undesireable long term results.

Beta 1.1 removed.
Beta 1.2 removed: p1.tcl (15.8 KB, 12 views)
Beta 1.3 removed: p1.tcl (16.1 KB, 3 views)
Beta 1.4 removed: p1.tcl (16.5 KB, 5 views)
Beta 1.5 is current version

Beta 1.7 located here http://www.dealdatabase.com/forum/sh...780#post246780

[philhu]

This has been done. I think Riley has code for it. It is/was designed to be added as a called module to mfs_ftp. I tested a beta of it and it worked fine.

What I did was 'hijack' seriesIDs of series I would never record for non-series stuff. And his code searched the db for series id's for items xferred into the tivo, attaching the item to the series in its group.

I've been using it for months.

[jbuehl]

This has been done. I think Riley has code for it. It is/was designed to be added as a called module to mfs_ftp. I tested a beta of it and it worked fine.

What I did was 'hijack' seriesIDs of series I would never record for non-series stuff. And his code searched the db for series id's for items xferred into the tivo, attaching the item to the series in its group.

I've been using it for months.

Really? That's interesting. I saw a couple of sthread that you were involved with back in August, but they seemed to go nowhere and Riley didn't sound too interested. In recent posts on this subject he didn't mention it.

[chrised]

This has been done. I think Riley has code for it. It is/was designed to be added as a called module to mfs_ftp. I tested a beta of it and it worked fine.

What I did was 'hijack' seriesIDs of series I would never record for non-series stuff. And his code searched the db for series id's for items xferred into the tivo, attaching the item to the series in its group.

I've been using it for months.

Its always nice spending 20+ hours developing something someone else has already done. Or has it been done? I don't see it publicly available. Enough people are asking for it. I privately asked for it, and got nothing.

Therefore, in my mind it hasn't been done.

Riley's messages (that I've found) never indicated having a version that did this during the inserts. All I could find was reference to a utility to fix them "Afterwards". And that utility is also not being shared.

I needed this now. And I decided I would share it with the group as a contribution for all the neat stuff I've been using. Thats supposed to be the idea here, right?

[alldeadhomiez]

There's been lots of discussion about using the TmsID. What's been overlooked is that Tivo assigns each TmsId a ServerId. Using ServerId, it isn't that difficult to find the series object once you figure out the data pattern.

Last time I looked, the TmsId (and not the ServerId) was consistent across SA and combo units for a given Series.

[philhu]

chrised-

Your solution is very good, and complete.

What Riley wrote was an addon program that could be called by mfs_ftp. It could run for hours (literally), looking for shows in a deep mfs scan

Your method of sending the data across AND building series id record objects is, well, fantastic.

I do not think Riley ever finished his code, and definitely never incorporated it in an mfs_ftp release.

I will play with your code tonight, since it is more complete. It looks PERFECT for what people have been looking for, at least, for me!

And FYI, TMSID IS consistent always. It is from the Tribune data stored by Zap2it, and gets stored blindly on the tivo. A show has a generic tmsid or an Episode id. The former is a generic for a show without a definite episode description, and they start with 'SH'. An episode id is a description of an episode of a show, and they start with 'EP'. There is also 'MV' or movie or 'SP' for sports show.

I use it in my Parsexmlguide_dd program to read datadirect zap2it outputs into Canadian tivos. See www.tivocanada.com for details

[jbuehl]

For folder support, it is critical that all shows be linked to the correct series. This must be done before the recording is saved, or you have to figure out how to correct the recording's indexing. (ick)

There's been lots of discussion about using the TmsID. What's been overlooked is that Tivo assigns each TmsId a ServerId. Using ServerId, it isn't that difficult to find the series object once you figure out the data pattern.

What do you mean by "linked to the correct series"? Assuming xml for the program has the series TmsId, don't you perform the linkage? Or are you saying that it's easier to link it while inserting, rather than trying to correct it after the fact?

This code will use the same concept to link to the real station record instead of a fake one. This allows additional information such as channel number and logo to display where appropriate. Clearly, if you are transfering from SA to DTivo the channel ID's are unlikely to match (they might, but I don't have an SA to try with).

I have an SA which is connected to a DirecTV box, so the channels will be the same. What really matters, though, is the station TmsId, which should be the same for all satellite, cable, and broadcast channels. I can configure my SA to get guide data for my local cable system and test this theory.

[jbuehl]

Last time I looked, the TmsId (and not the ServerId) was consistent across SA and combo units for a given Series.

I think chrised figured out the naming scheme for the ServerIDs which is based on the TmsId:

  #example path  /Server/ATSH225421:4:0:0
  set seriespath "/Server/$info(SeriesServerId):4:$info(SeriesServerVersion):0"
  outd 3 "series path: $seriespath"
  if {[catch {RetryTransaction {mfs find "$seriespath"}} l] != 0} {
     outd 3 "series path not found.  discarding series property"
     set seriespath ""
     }

If it works reliably and is fast enough, it makes searching Tms.index unnecessary.