PDA

View Full Version : best file format for archiving recordings on your PC



ChromePlanet
04-04-2004, 02:31 AM
I'm curious how people are archiving their recordings on the PC, both TiVo extracted content and DVD ripped content. Hard drives are big enough to be feasible archive storage at less than $1 per GB and dropping. What file format do you prefer for archiving content, keeping in mind you may need to burn anything to DVD at any time, play it on your PC at any time, and re-edit as needed without re-encoding anything.

lgkahn
04-04-2004, 10:22 AM
you left out .tmf the one format I prefer if I loose a hard drive on a tivo and want to push shows back to a tivo or want to spool back at any time to replay...

ChromePlanet
04-04-2004, 02:26 PM
you left out .tmf the one format I prefer if I loose a hard drive on a tivo and want to push shows back to a tivo or want to spool back at any time to replay...

My bad. I'm still learning a lot. What's different about TMF vs. TY? Which programs use it?

AlphaWolf
04-04-2004, 03:41 PM
My bad. I'm still learning a lot. What's different about TMF vs. TY? Which programs use it?

About the only two softwares I can think of that don't support it are TyStudio and jdiners stuff. TyStudio just hasn't been updated in a long time. I think jdiner doesn't add it b/c he doesn't like open source stuff.

cojonesdetoro
04-04-2004, 04:07 PM
you left out .tmf the one format I prefer if I loose a hard drive on a tivo and want to push shows back to a tivo or want to spool back at any time to replay...

ty files can be reinserted with mfs_ftp. If they're extracted with mfs_ftp then they can also have the xml data re-inserted as well.

edit: I believe Riley gave some good info about differences in the mfs_ftp thread. a TMF file is just a tar file with the extracted chunks of a recording as well as an xml file with program description data. The ty files extracted with mfs_ftp employ a little trick whereby the xml data is inserted in a 'junk' section at the end of a file. Most apps that read ty files ignore the data at the tail end of the file. A byproduct of this is that the xml data is not applied until the recording has almost finished insertion whereas a tmf file will have correct program data very soon after an insertion starts.


About the only two softwares I can think of that don't support it are TyStudio and jdiners stuff. TyStudio just hasn't been updated in a long time. I think jdiner doesn't add it b/c he doesn't like open source stuff.

I believe that jdiner's stuff has supported tmf files for some time now. I don't have any firsthand experince with tytools and tmf but it is mentioned in the realease notes. Either way, conversion from tmf to ty is pretty quick. The bigger issue might be storage 'swng space' while converting.

ChromePlanet
04-04-2004, 08:47 PM
For maximum flexibility, what would be the downside to collecting in MPG/AC3 with generic MPG files, which can play in most software players and can be imported in most editors and can also be imported into most DVD authors? None of the other formats can make that universal compatiblity claim can they?

lart2150
04-04-2004, 10:29 PM
I think tyshow can play tmf files but I can't rember if he ever added it

rc3105
04-05-2004, 12:26 AM
ChromePlanet:

the downside to archiving in mpg / divx4 / xvid (all standard formats) is that you can't put that back in the tivo yet (really needa finish that project, yanno?)

while there are lots of cool hardware / software dvd / media players out there, the TiVo interface is hard to beat

mac, pc, xbox & dvd capable tivos (including s2's with external usb dvd drives) can play back insertable tmf/ty+ from disk

the only reason to archive in mpg/vob (same thing) is playback on plain vanilla standalone dvd players that don't support anything else

edit: ty+ & tmf added to the poll ;)

dlang
04-05-2004, 01:15 AM
when archiving to hard drives remember that drives WILL fail and so make sure that you store things to a RAID array

I am speaking from experiance as I just had a (full) 250G drive die on my, that's a LOT of shows to loose at once.

as a result I went out and built a 1TB array so it won't ever happen to me again :)

eastwind
04-05-2004, 02:08 AM
For maximum flexibility, what would be the downside to collecting in MPG/AC3 with generic MPG files, which can play in most software players and can be imported in most editors and can also be imported into most DVD authors? None of the other formats can make that universal compatiblity claim can they?
Are you talking about transcoding the whole stream? If so, that would be the downside. jAlong with not being able to re-insert it on the TiVo. If not transcoding, you probably won't have the universal compatibilty you might think when you try to import into at least some DVD authors.

ew

wkozun
04-06-2004, 11:03 PM
For maximum flexibility, what would be the downside to collecting in MPG/AC3 with generic MPG files, which can play in most software players and can be imported in most editors and can also be imported into most DVD authors? None of the other formats can make that universal compatiblity claim can they?

One downside is size! If you reencode to DivX or WMV you can reduce the file size - often by as much as 75% without causing much, if any, degradation in quality.

The problem is that standalone DVD players typically can't play these file types.

ChromePlanet
04-07-2004, 12:46 AM
ChromePlanet:

the downside to archiving in mpg / divx4 / xvid (all standard formats) is that you can't put that back in the tivo yet (really needa finish that project, yanno?)

while there are lots of cool hardware / software dvd / media players out there, the TiVo interface is hard to beat

mac, pc, xbox & dvd capable tivos (including s2's with external usb dvd drives) can play back insertable tmf/ty+ from disk

the only reason to archive in mpg/vob (same thing) is playback on plain vanilla standalone dvd players that don't support anything else

edit: ty+ & tmf added to the poll ;)

My philosophy on this is looking forward into the future, trying to consider where software and hardware are going so that the files I save now will be usable down the road.

Where I believe we're headed is a living room device that can play both secure (scrambled/rights-managed files like encrypted DVD, WMV, AAC, WMA, copy-protect-flagged HDTV, scrambled TY) and unlocked content (MP3, WAV, CD audio, MPG, AVI/MP4). For maximum flexibility, we hackers will convert all we can to an unlocked format so we can burn, edit, share, and transfer under fair use.

That said, it would be wise to choose one popular high-quality format that is both flexible and universal with staying power.

Size should not be an issue. We have 1TB drives now. Prices drop every day. Even a dual layer 9GB DVD will seem way too small very soon. Archive on the hard drive and you can burn DVDs at any time as you need them. Blu-ray is around the corner for HD-DVD, but why mess with a bunch of discs? A modded Xbox can store lotsa movies on its drive and play them nicely on your TV or from the hard drive on your network. The TiVo of tomorrow will as well, though it probably won't come from TiVo.

Eventually, hard drives will be smaller and lighter than a single DVD.

I'm not concerned about not being able to put shows back into a TiVo, because a TiVo, as it is now, is a limited device. A modded Xbox is a far more useful tool. And it's only a matter of time before someone creates an open-source TiVo killer box that runs on Linux and does the same thing only with open files. Don't get me wrong. I love TiVo. But it's like Sony's Magic Gate system for MP3s... it's hard to breathe.

I use a Wiebetech Firewire-800 dual-bay removable drive system. I can fit tons of movies and shows on a 320GB drive. When I travel, I transfer what I want to a laptop over the network. And I can stream to a modded Xbox for watching content on TV. I can even stream to a Sony PS2 running BroadQ software.

So from where I see it, archiving content to generic MPEG-2 files can't be beat. But I'm open to other ideas.

ChromePlanet
04-07-2004, 12:51 AM
Are you talking about transcoding the whole stream? If so, that would be the downside. jAlong with not being able to re-insert it on the TiVo. If not transcoding, you probably won't have the universal compatibilty you might think when you try to import into at least some DVD authors.

ew

No, I feel ill about transcoding anything. Quality is paramount. Just found out about the incompatiblity issues with extracted TY to MPG/VOB and editors. Perhaps this can be fixed without transcoding. Although I was able to load an MPG file created with TyTool into Adobe Premiere, so exactly where will things go wrong?

ChromePlanet
04-07-2004, 12:54 AM
One downside is size! If you reencode to DivX or WMV you can reduce the file size - often by as much as 75% without causing much, if any, degradation in quality.

The problem is that standalone DVD players typically can't play these file types.

I hate lossy compression. I ripped 1200 CDs into WAV files just to maintain full quality.

We're dipping below $1 per GB. That's cheap enough to avoid compression past the original MPEG-2 we get off the TiVos.

AlphaWolf
04-07-2004, 12:55 AM
So from where I see it, archiving content to generic MPEG-2 files can't be beat. But I'm open to other ideas.

The main reason you don't want to do this at this point in time, is that no tystream conversion software does a proper job of this yet. Jdiners stuff artificially syncs the audio, and some codecs/devices will not properly sync or seek as a result in some cases, and in most cases you can't make any changes to these streams with any third party software without re-encoding. Tydemux doesn't have any of these shortcomings, however, it can screw up on some streams as it was never completed and some bugs remain.

If you ask me, the best thing to do with it is keep it in tmf format, because tmf is as true to the original as you will get. Namely, all of the stream info is preserved, and the parts aren't all concatonated together, so if the stream is abnormal or corrupt in any way, its more likely that the software can compensate. When the software advances enough to be able to make the resultant streams more kosher, you'll have more options in the future by preserving the tmf.

lgkahn
04-07-2004, 06:09 PM
At the risk of repeating myself....
I agree with alphawolf...

tmf format is the way to go and I archive all my movies in that format..


jdiner's stuff does work in that format I have imprted at least 50 movies directly into tytools and created .vdo's/vobs from the .tmfs directly no issues.

this way you can extract the mpg ii files at any time if you like...

you can put the files back on any tivo box at any time if you like...

and you can burn dvds from the movies at any time if you like.. in addition you can recombine the movies with other movies to burn different dvds at any time...

I just put my second hard drive in my .tmf storage box since I just filled up the first 160 gig drive... got a 200 gig seagate for 99 bucks after rebate... not bad....

only problem is that since these files are already compressed, dlt backup doesn't do too good backing them up... 4 dlt iv tapes just to back up the 150 or so gig (remainder is os overhead and tivo images) I already have... bummer... but don't want to loose those movies.. that was the whole point of hacking the boxes... I lost 2 hard drives previously...

ChromePlanet
04-08-2004, 02:29 PM
At the risk of repeating myself....
I agree with alphawolf...

tmf format is the way to go and I archive all my movies in that format..


jdiner's stuff does work in that format I have imprted at least 50 movies directly into tytools and created .vdo's/vobs from the .tmfs directly no issues.

this way you can extract the mpg ii files at any time if you like...

you can put the files back on any tivo box at any time if you like...

and you can burn dvds from the movies at any time if you like.. in addition you can recombine the movies with other movies to burn different dvds at any time...

I just put my second hard drive in my .tmf storage box since I just filled up the first 160 gig drive... got a 200 gig seagate for 99 bucks after rebate... not bad....

only problem is that since these files are already compressed, dlt backup doesn't do too good backing them up... 4 dlt iv tapes just to back up the 150 or so gig (remainder is os overhead and tivo images) I already have... bummer... but don't want to loose those movies.. that was the whole point of hacking the boxes... I lost 2 hard drives previously...

I see your point. But why use tape? Hard drives are cheap enough and random access. Do you have a removable drive system? They work great. At .50 per GB (200GB for $99) you write to 2 drives mirrored in a removable drive system and store the duplicate set at a family member's or friend's house. You don't have to fumble with backing up and managing tapes.

Can you give more details about your procedure? Which settings do you use in each program?

wkozun
04-08-2004, 02:50 PM
jdiner's stuff does work in that format I have imprted at least 50 movies directly into tytools and created .vdo's/vobs from the .tmfs directly no issues.

Was this fixed recently as I know that the 8rx series of TyTools did not correctly handle tmfs? You had to use teh tmf2ty tool. From what I remember Josh said he would try to fix this, but it wasn't at the top of his list.

lgkahn
04-08-2004, 09:13 PM
I use tapes because I already have a dlt drive to back up the 5 pcs I have at home and already have a closet full of dlt tapes so no big deal and they are safer than hard drives in the long run..


my procedures are to use ws_ftp32. to pull the .tmf files of the tivo via mfs_ftp
set to the media store pc on standard prt 3105

in .tmf format

once I have the files on my media store pc running windows 2000

I pull them over to one of the two faster pc's I burn dvds with usually the dual xeon 2.4 ghz running nt 4.0 with 1gig...
with scsi raid... I find it is much faster to do this on good hardware...

then I simply run tytools both version 11, 12 and no2 13 have had no problems importing .tmfs directly

I simply select the .tmf files and choose file/"vob mux new format"
and select the .tmf files
it processes them into vob files

then if I want a .bmp file for the menu I select on of the .tmf files again with tytools version 13 and use file/"make key file" and choose the .tmf
I let it go about 100 meg through and select abort.

then I open gopeditor and choose the key file tytools created fast forward to a suitable frame and select some bitmap from the frame... now you can delete the .key file we are done with it, we can also delete the .tmf files from the local computer we are done with those also..

next step is to create the dvd itself

I create a subdirectory for the dvd
then in tytools again I select file/"create ifo files"
I chose the vob's I created earlier and select process
I then select the menu names for each of the movies I want and the font...
I put a few spaces in front of each so they don't get cut of on the left and left justfy them... I also select the .bmp we created earlier and the directory we created and a title if I want...
then select "make ifo" and away it goes...


when it is done I simply run nero

select new /dvd-video drag the video_ts folder it created in our new directory into the video_ts directory in nero give it a label and select book-type dvd-rom checkbox and burn the dvd onto a dvd+r ....

never had one bad disk...

ChromePlanet
04-10-2004, 03:32 PM
I take it TMF files can't be edited directly so when you archive in this format you have to save all the commercials?

Griff1324
04-10-2004, 07:42 PM
I have just started "archiving" my extracted videos. Most of them are television shows and I want to edit out the commercials. I extract the ty file using tytools 9r13. I then use GOP editor to go in and edit the commercials using cuts at specific frames. I then save the video as a .vob and I am then done.

I am not concerned with putting the videos back onto the DTivo which is in my bedroom connected to a 27" television. What I do is then stream the videos wirelessly from my pc to my xbox in the living room which is connect to a 65" Mitsub. HDTV. Xbox Media Player can handle the .VOB files so that isn't a concern. Plus I can burn a DVD anytime since I already have the file in the .VOB format.

I am not aware of any downside to keeping VOB files, but I would be interested in hearing other's comments.

later,

Chris

lgkahn
04-10-2004, 10:31 PM
I am not archiving tv shows so yes If I was archiving thos they would not be in that format... or I would archive them with commercials if I wanted to put them back on the tivo to watch later.. .I am only archiving movies with no commericials from ppv or hbo, showing , cinemax, stars etc. etc.

dlang
04-10-2004, 10:36 PM
the only downside of keeping the VOB file (or any other format) instead of the tmf/ty+ file is that when jdiner makes improvents in his program (or someone else makes something better) you won't be able to go back and generate improved versions.

every other format looses data about the show (even ty+ looses a little bit of info, namely what show data was in which part when it was stored on the tivo)

you may consider the current version good enough and not care about it (in which case more power to you) but to future-proof yourself the only way is to keep the origional around

an example of this happening in the last several months is when jdiner introduced frame accurate cuts so that the commercials could be cut more precisely. anyone who had thrown out their ty or tmf files was out of luck, those that had them could re-create their trimmed shows.

remember that jdiner does NOT have a copy of the DVD specs (they cost ~$10k as far as anyone can tell) so everything he is doing is being done by trial and error and blind testing, there is no way of knowing if the output meets the specs or if it just happens to work in a particular player

Zak0
04-11-2004, 10:35 AM
I'm about evenly split between archiving to DVD-R using the IFO-generation in TyTools and transcoding to XviD. I only have an 80 gig drive to keep video online, so size is important and the AVI files are lots smaller. I have an Xbox, so putting the shows back on the Tivo is not a concern for me. When the drive gets full, I archive some XviD files to DVD-R and continue about my business...

-Zak

ChromePlanet
04-12-2004, 03:45 AM
remember that jdiner does NOT have a copy of the DVD specs (they cost ~$10k as far as anyone can tell) so everything he is doing is being done by trial and error and blind testing, there is no way of knowing if the output meets the specs or if it just happens to work in a particular player

Interesting. And no one on these forums could get a copy of them to share? Aren't they public domain?

I take it he doesn't have a license for TyTool's Dolby Digital encoding then. That definitely isn't public domain. Does that mean transcoding audio to Dolby Digital could be taken out of the software if Dolby wanted to make an issue out of it?

dlang
04-12-2004, 10:52 PM
the DVD specs are very definantly not public domain, they are tightly protected trade secrets controlled by the DVD parent orginization.

at $10k to get a copy it's not surprising that nobody is willing to risk their business by sharing their copy with Josh (you've seen these folks in action over the DeCSS fiasco, do you really think they would allow the specs of how that works to be distributed??)

as for the Dolby issue, my understanding (which could be completely wrong) is that Dolby doesn't work via trade secrets, the patent the stuff and charge a license fee to use it in players. since Josh isn't writing a player he has no problem.

ChromePlanet
04-29-2004, 07:55 PM
Assuming then that TMF is the best way to archive on the PC at this point, what would be a practical way for playing these files, considering after reading here for hours I found no software that plays TMF directly?

And what would be an easy way to extract a TY from a TMF, edit it, and put it back in the TMF?

ChromePlanet
04-29-2004, 08:04 PM
Assuming then that TMF is the best way to archive on the PC at this point, what would be a practical way for playing these files, considering after reading here for hours I found no software that plays TMF directly?

And what would be an easy way to extract a TY from a TMF, edit it, and put it back in the TMF?

TMF files are just TAR files which contain a TY and an XML file. Associate TMF files in Windows with WinTar (or your favorite TAR management program). Then when you click on it, you will see the files it contains, which include the TY file. Click on the TY file (after associating it with Windows Media Player) and you'll be able to play it without actually extracting it (although WinRAR is actually extracting it for you) provided you have the TyShow codec installed for Windows Media Player.

To edit the TY, extract it from the TMF and use TyStudio or TyTool and then put it back in the TMF using WinRAR.

Next time read a little more before posting. I'm busy.

sanderton
04-30-2004, 06:16 AM
TMF files are just TAR files which contain a TY and an XML file. Associate TMF files in Windows with WinTar (or your favorite TAR management program). Then when you click on it, you will see the files it contains, which include the TY file. Click on the TY file (after associating it with Windows Media Player) and you'll be able to play it without actually extracting it (although WinRAR is actually extracting it for you) provided you have the TyShow codec installed for Windows Media Player.

To edit the TY, extract it from the TMF and use TyStudio or TyTool and then put it back in the TMF using WinRAR.

Next time read a little more before posting. I'm busy.

Longer shows can contain more than 1 .ty Part though. And you can't edit and put back I'm afraid.

JohnSorTivo
04-30-2004, 11:36 AM
The other great thing about having a hacked xbox with XBMC is that it can stream .ty files directly without any manipulation / alternation etc, either archived on your PC, or directly from the Tivo.

As such, I archive in this format (.ty). If I want to edit out commericals and convert to .vob for DVD creation I can. If I want to re-insert I can. While I know .tmf is optimal for re-insertion, I believe the only thing you really loose when re-inserting a .ty file versus a .tmf file is some of this show "meta-data" available via the 'Info' screen, which is a fine compromise for me.

psxboy
04-30-2004, 01:40 PM
the DVD specs are very definantly not public domain, they are tightly protected trade secrets controlled by the DVD parent orginization.

at $10k to get a copy it's not surprising that nobody is willing to risk their business by sharing their copy with Josh (you've seen these folks in action over the DeCSS fiasco, do you really think they would allow the specs of how that works to be distributed??)
Actually, it's just the CSS (Content Scramble System) specs that are not in the public domain. This is what the DeCSS case was all about. The DVD specs are most certainly readily available (and free) from a bunch of different sources. And its basically just an extension of the MPEG2 specs anyway, dealing with the specifics of the VTS logical format, navigation control, etc.

That said, the DVD specs only deal with playback mostly. The encoding process is left open-ended which gives the encoder-makers lots of leeway when designing their products. As long as the output fits the specs, they can do the encoding any way they want. This is why there is so much variation between the output from different encoders, and what makes it so difficult to adapt a proprietary format (like Tivo's) to a standard format that will play on most players.

-psxboy