PDA

View Full Version : Any experiences converting HDTivo files to MP4 variants?



alfonzotan
05-10-2006, 02:31 PM
Toying with the idea of transcoding extracted/edited files to an MP4 variant (preferably H.264, but I'm open to any playable codec) to reduce file footprints. Anybody else tried that yet?

I've got a Nero Recode 2 conversion running right now, I'll report back on the results.

cheer
05-10-2006, 03:00 PM
Haven't done much with that myself, since I don't really have a device (other than a PC) that will play an HD MP4 file.

Having said that, it should be a piece of cake; not really any different than encoding an SD file. Now personally I wouldn't use Nero Recode; I'd use avisynth to frameserve up to Vdub, but then I'm a control freak.

alfonzotan
05-10-2006, 03:17 PM
Haven't done much with that myself, since I don't really have a device (other than a PC) that will play an HD MP4 file.

Having said that, it should be a piece of cake; not really any different than encoding an SD file. Now personally I wouldn't use Nero Recode; I'd use avisynth to frameserve up to Vdub, but then I'm a control freak.

Hell, I didn't understand more than six words of that last sentence. :)

I'm going to put together a media server for my house, with a computer (probably a dual-core Intel Mac Mini) as the playback device at the TV. It'd be nice to have a library of HD content stored away without having to invest in double-digit terabytes of hard drives; plus (IIRC) the MPEG4 codecs have less demanding playback overhead compared to MPEG2.

cheer
05-10-2006, 03:41 PM
Hell, I didn't understand more than six words of that last sentence. :)
A year ago, I wouldn't have either. Go hang around doom9 for a while and your head will explode. But some quickie definitions:

Frameserving basically means taking a video file and serving it up, frame-by-frame (or field-by-field) to another application. Why on Earth would you want to do such a thing? Well, because most frameserving tools are scriptable/programmable and let you do all sorts of powerful things.

Avisynth is a frameserving app. You create an avisynth script (just a text file with a .avs extension), and within the script you can specify filters, resizing, and all sorts of good stuff. To any other Windows application, opening a .avs file looks just like opening a .avi file -- in fact, you can play .avs scripts in Windows Media Player.

To illustrate this, here's a very basic avisynth script:


AviSource('c:\vids\frog.avi')
LanczosResize(720,480)
This just resizes frog.avi to 720x480. So if frog.avi is 320x200, I can open frog.avs (my script file) with WMP and I'll see a 720x480 video. Obviously I can do a lot more, including deinterlace or inverse telecine, sharpen or smooth the video, remove logos, etc. Takes a while to get the hang of it, but once you do you are almost overwhelmed with the possibilities. (Which brings me to an important point...you can't create video information. So don't go overboard with processing unless you have a good reason to.)

Meanwhile, VirtualDub is a useful tool for converting to another compression format, editing, cropping, etc. (Of course, you can crop and edit in avisynth too, but VDub is useful because it's a GUI app and you can at least figure out what your crop/edit numbers are.) So I create my avisynth script, then open it in virtualdub, choose my destination codec, and save.

I apologize; that wasn't quick at all.

I'm going to put together a media server for my house, with a computer (probably a dual-core Intel Mac Mini) as the playback device at the TV. It'd be nice to have a library of HD content stored away without having to invest in double-digit terabytes of hard drives; plus (IIRC) the MPEG4 codecs have less demanding playback overhead compared to MPEG2.
I'd think the reverse...I would think decoding MPEG4 would take more horsepower than MPEG2. But I could be mistaken.

As for the rest...agreed. You might also look at Xvid or some other alternative...lots of semi-automated or fully-automated tools that will get you there. AutoGK typically does a nice job, will preserve DD5.1, etc.

alfonzotan
05-10-2006, 03:56 PM
Thanks, I'll give all that a look.

I've read elsewhere that you can encode an HD MPEG4 at half or less the bitrate MPEG2 would require, but I haven't personally tried it yet or seen it proven. Might be an "internet legend," for all I know.

texmex
05-10-2006, 04:32 PM
I've been playing with Dr. DivX 2.0 using the DivX-HD profile. I've squashed a few movies which I can play on the HDTV via the Avel LP2 - they look very good. I'm encoding to 1280x720 in Insane mode. I'm still playing with bitrates - using a formula I found based on resolution and framerate (most movies are 24fps, tv shows 30fps), I end up using bitrates between 3500kbps and 6000kpbs. That's generally a filesize reduction of 1/2 or better. So, no, it's not an internet legend - you can achieve good quality in MPEG4 at half the bitrate of MPEG2

alfonzotan
05-11-2006, 07:30 AM
Thanks.

My initial experiment with Nero Recode was a failure. Output file was useless. Apple's tools take waaaay too long, but do produce very nice video output. I'll start messing around with Divx stuff next.

davidh
05-15-2006, 07:47 PM
I am looking to do this also, but getting poor results. I am using Tytools to bring the files over, then use Tytools to create the mpeg. I tried using AutoGK but had poor results (with both HD and Non-HD files). Picture glitches every few seconds and the file is bigger than the orig. mpeg.
I am just looking to archive HD and Non-HD files on a computer and have a means of playing them back. HD files don't play (created through Tytools) and I have yet to find a easy way of creating them and have them be good quality.

texmex
05-16-2006, 11:36 AM
I've managed to archive most of this season of Lost in HD using TyTool, VideoRedo and DrDivx2. I've also had success with several HD movies from HBO and Showtime. Here's my workflow:
- TyTool: extract in ty mode
- TyTool: "Multiplex File"
- VideoReDo Plus: QuickStreamFix (not always necessary, but I do it anyway)
- VideoReDo Plus: Edit out commercials (fix audio sync if necessary)
- VideoReDo Plus: Save Video As (save as mpeg)
- VideoReDo Plus: QuickStreamFix again (sometimes this helps, sometimes it doesn't)
- DrDivX2 OSS: Settings vary, but I always use High Definition profile, 1280x720 (unless it's a letterbox movie), AC3 audio, Insane Quality, 2-pass, H.263 Optimized. Unless I'm trying to squeeze a certain number of episodes on a disc, I use the following formula for bitrate: VRes x HRes x FPS x .22 / 1024

For example, movies from HBO and Showtime are 1280x1088, and most are 24fps (well, 23.9xxx; cut a small clip using VideoRedo and let DrDivx analyze it - it'll tell you the fps - film sources are generally 24fps, television/video sources 30fps). I re-encode to 1280x720, so the bitrate to achieve max quality for a movie is: 1280 x 720 x 24 x .22 / 1024 = 4752. DirecTV MPEG2 bitrates for HBO and Showtime HD hover around 10mbps, so you can cut the size in half using DivX and still maintain very high quality. Note that you'll usually want to crop 8 pixels from the bottom of DirecTV 1080i material - their encoders add 8 gray pixels to the bottom of the 1280x1088 image. As far as encoding times go, on my AMD Sempron 3100+ OC'd to 2.25Ghz it takes about 3.5 times the length of the show; so a 1 hour show would take about 3.5 hours. For some reason my Intel P4 3.2 w/ HT takes about 3 times that long - goodbye Intel! Please note: DrDivX2 is beta software at the moment - they're still working out the bugs. Set your expectations accordingly...

I don't re-encode any SD material using DivX - the DirecTV MPEG2 encoders are compressing SD material down to about 2mbps. In order to retain the image quality, DivX would need about the same 2mbps (based on formula above). Now, if my target was a portable device then I would certainly use DivX to squash the video.

Now, having said all of that, I do still have problem streams. I've had mostly good luck with ABC HD (OTA), ESPN/ESPN2, and most DirecTV 1080i. I don't think I've ever been successful trying NBC HD OTA. If TyTool bombs on a stream, I'll usually try HDemux and have had mixed results. If possible, I try to re-record the show (possible with movies, not possible with sports).

For those of you having problems, try VideoRedo and the "QuickStreamFix". It's been well worth the $50 for me, but you can try it for free and judge for yourself.

davidh
05-17-2006, 01:49 PM
texmex, you state you use TyTool to Multiplex the file. My understanding is TyTool doesn't multiplex an HD file correctly (waiting for an updated version) at least on large HD files like two hour movies. If it does work on your Lost HD (1 hour HD file) doesn't Multiplex create a mpeg file (just want to make sure I understand what is going on here).
I am going to try VideoRedo. Thanks for the help.

cheer
05-17-2006, 02:14 PM
Normally TyTool muxes an HD MPEG just fine. Some streams give it problems, but overall it seems to work fine for me.

texmex
05-17-2006, 02:23 PM
TyTool does work for many HD streams, just not all. And as you stated, longer streams are less likely to work. But I've had success using TyTool on the entire season of Lost in HD and about 25 HD movies from HBO, Showtime and HDNet.

As far as the workflow goes - every stream that DirecTV beams into our HR10-250 is an mpeg2 stream**. Tivo takes that mpeg2 stream and "wraps" it in a ty file. This ty file is storing extra information that primarily makes for better video scanning. What TyTool does is extracts the original mpeg2 information from the ty file. Generally, TyTool is very good at this, but due to the nature of DirecTV mpeg2 streams, there can be issues. Anyway, VideoRedo has a function called QuickStreamFix that sometimes helps to "clean up" the issues in TyTool mplexed streams. VideoRedo is also a very slick editor for both standard and HD mpeg2 streams. Back to the workflow:
- Extract from Tivo: TY file
- TyTool-Multiplex: "converts" TY into MPEG2
- VideoRedo: QSF and edit MPEG2, save as MPEG2
- DrDivx (or any other divx/xvid tool): convert MPEG2 to AVI (which is MPEG4)

** DirecTV is now encoding HD locals using mpeg4. Sometime next year or the year after, national HD channels will also be converted to mpeg4. However, there currently isn't any way to record those mpeg4 streams, and there will NEVER be a DirecTV mpeg4 Tivo. DVR yes, coming soon - but it won't be a Tivo.

alfonzotan
05-19-2006, 01:37 PM
Thanks for the tips. I've tried a few different methods with varying success. I'll give DrDivx a shot next. One consistent factor in MP4 conversions, already noted: it's real, real slow. even on fast machines (i.e., a dual processor G5 tower).

A tip: Be sure to check the "fill AC3 gaps" (I may have the verbiage slightly off, but you'll know it when you see it) in the TyTool audio preferences for all HD Tivo extractions/saves. Ought to be set by default IYAM, or at least the setting ought to be persistent once checked. That's fixed all my audio sync issues to date, with the exceptions being shows that weren't in sync during the original recording (all too often, unfortunately).

alfonzotan
05-20-2006, 01:25 PM
Okay, here's a cross-platform method that worked well for me. Biggest downside is the re-encoding time, which is roughly forever on the "insane" quality setting:

1. Windows: Extract file with TyTool in .ty mode, being sure to set "Fill AC3 audio holes" in Options > Preferences > Audio first.
2. Windows: Edit with VideoReDo. I haven't bothered with QuickStream runs, and haven't seen any problems. Save as .mpeg.
3. Windows: Split the edited .mpeg file with TyTool to produce .m2v and .m2a files. The .m2v file may be discarded.
4. Transfer the edited .mpeg and .m2a file to a Macintosh (dual G5 tower in my case).
5. Mac: Change the extension of the .m2a audio file to .ac3.
6. Mac: Using the "Custom" option and "Advanced Settngs" in DivX Converter, set compression options to your taste. Here's what I used:

- High Def Profile
- 1280 x 720
- Crop bottom 8 pixels (select the "up" arrow 8 times on the bottom crop selector)
- 4000 kbps (the max setting)
- Performace: Insane Quality (fair warning: this slows encoding to a crawl)
- Audio: N/A; DivX Converter doesn't like the audio in the .mpegs for some reason, and doesn't record any (fixed later in procedure).

Encoding a five-minute clip of "Battlestar Galactica" took several hours with these settings. "Insane Quality" might not be practical for regular use. Returning to the procedure:

7. Macintosh: Using ffmpegX, mux the .divx file with the .ac3 audio file, save as "AVI + audio." The resulting file will play in VLC with very high quality.

To answer the inevitable question, "Why not just use DrDivx," either my Windows box doesn't like DrDivx, or DrDivx doesn't like my HD files. I haven't been able to get an encode to run with that program yet. I'll keep trying as DrDivx is upgraded; as noted above, it is a beta version.

Next up: comparison runs with faster encode settings. Thanks to all for the suggestions to date.

alfonzotan
05-20-2006, 02:25 PM
Oh, I nearly forgot. The whole point of the exercise is to reduce HD file sizes. My original MPEG2 file was 445MB, the final muxed Divx file is 142MB, and it looks great (ought to, for as long as it took to encode). Not bad at all.

alfonzotan
05-21-2006, 10:52 AM
Okay, quick update. I ran the same HD clip (445.7MB, MPEG2) with the "Balanced" Divx encode setting, single pass encode. Took about 40 minutes to encode a 5 minute file, finished file size (142.2MB) was very slightly larger than the 142MB "Insane Quality" 2-pass file. I then took both Divx files, loaded them on a minimum-spec Mac Mini (Intel Core Solo, 256MB RAM), and plugged it into a 50" Panasonic plasma. If the two recoded files ("Insane quality" vs. "Balanced") have any visual differences, I couldn't see them.

Both Divx files appeared to be a bit washed out in the dark scenes compared to the original HD Tivo recording--but as I'm typing this, I recall that the Tivo is plugged into a calibrated HDMI input, and the Mac was plugged into a factory-settings VGA port. Going through HDMI and/or calibrating the VGA input on the TV could well clean up the "grey blacks" and somewhat grainy backgrounds I noticed. Bright or outdoor scenes were indistinguishable from the Tivo original.

I don't actually expect anybody else to use my screwy multi-computer method, but for what it's worth, my experience indicates that the Divx quality settings can safely be put on "Balanced" at a minimum and still retain excellent picture quality. The upside is that you get a roughly 4:1 reduction in file size. The downside is, encoding is ~10X the real-time length of the recording on my last-year's-hardware (still much better than 140X, which is about how long the "insane" setting took--12 hours of encoding for a five-minute file).

alfonzotan
05-27-2006, 11:35 AM
Small update. I finally got Dr. Divx working on the PC side (hint: be sure the AC3 filter is loaded) and it does a very nice job. I'm still muxing in the original AC3 file after the Divx conversion with ffmpeg. Getting roughly a 4:1 file compression rate, and quality is excellent.

alfonzotan
06-06-2006, 07:04 AM
Anybody else having problems doing a Windows Dr. Divx encode of HD files longer than about 1:51 (one hour, 51 minutes)? I've done two different movies now that suddenly end at about the 1:51 mark; both are about 2:20 in actual length. The MPEG2 original files seem to be complete, and will play all the way through. File sizes for the Divx versions (which look fantastic) are just over 5 gigs. Shorter files (i.e. TV show episodes) re-encode completely.

Any ideas?

cheer
06-06-2006, 08:00 AM
Can't say as I've not used Dr. Divx. Have you tried an alternative like AutoGK? Just to rule out software...

alfonzotan
06-06-2006, 09:04 AM
Can't say as I've not used Dr. Divx. Have you tried an alternative like AutoGK? Just to rule out software...

Not yet (just found this late last night), but I'll try AGK next. I'd thought it was an encoding hiccup or some kind of file transfer weirdness until the second movie did the exact same thing. I'll also upgrade to the latest version of Dr. Divx, they're up to Release Candidate status now.

davidh
06-06-2006, 03:21 PM
alfonzotan, I have been going down much the same path as you. I have been doing the following with good results.

1. Bring the file over with TyTools
2. Create an mpeg2 file with TyTools
3. Run the mpeg2 file through AutoGK with the following parameters that rbreding suggested http://dealdatabase.com/forum/showpost.php?p=256054&postcount=22
4. End up with a much smaller xvid file, and quality looks great.

I have just started using this sequence so I only have a short history, but I have done it on very large (8 gig and over) files and it seems to work.

Post back how your efforts are going.

alfonzotan
06-06-2006, 06:46 PM
Thanks. I've got a 100% quality encode running now, and it's not going to be finished anytime soon. I'll report back when it's done. I'll probably run another one after that at a lower setting for comparison.

One question, if anybody knows, why the 1040 width setting? Why not the original resolution, minus the 8 lines at the bottom?

cheer
06-06-2006, 07:08 PM
No reason you have to scale it to 1040 -- in fact, my preference is to never change the resolution unless I have to. But sometimes it's a question of your playback device. My laptop sure as heck can't do 1920x1080. :) Some dedicated media players have resolution limitations, etc.

alfonzotan
06-06-2006, 07:33 PM
No reason you have to scale it to 1040 -- in fact, my preference is to never change the resolution unless I have to. But sometimes it's a question of your playback device. My laptop sure as heck can't do 1920x1080. :) Some dedicated media players have resolution limitations, etc.

Gotcha. In my case, I'll be sticking with the source resolution.

alfonzotan
06-07-2006, 05:14 PM
Hmm. Well, here are the results from my AutoGK Xvid encode of a certain Episode that runs 2 hours 20 minutes.

The Good:

1. The picture is great. Black blacks, brilliant colors, synched audio.
2. File compression is slightly less than 2:1 (~9GB to ~5GB). Not great, but okay.

The Bad:

1. S-L-O-W encoding. Nearly 24 hours--but to be fair, I set it for maximum quality, which may have been overkill.
2. Despite settings to crop the bottom 8 pixels, they're there, and they're grey (but it's entirely possible that I didn't set up the encode correctly; this is only the second time I've used AutoGK).

The Ugly:

1. The movie aspect is 2.35:1. Instead of presenting it with top-and-bottom letterbox, the encode mashes everything up to the top of the window. Weird, but again probably a result of my inexperience with the program.
2. Quicktime Player does not like this file, and sees it as only 53 minutes long for some reason. VLC, on the other hand, plays the whole thing. Probably Quicktime's fault.

Thoughts?

cheer
06-07-2006, 06:16 PM
Compressing to Divx/Xvid/MP4 is never fast, and HD video will, yes, take forever. Not worth the space savings IMO, but that's me.

Not sure I understand about the aspect ratio. Are you saying AutoGK is distorting it? AutoGK will attempt to crop the letterboxing off, so in theory you should end up with a file that is approx. 2.35:1. That's by default; if you've been playing with the settings, it might not be so intelligent.

Quicktime player blows. Use media player classic.

alfonzotan
06-07-2006, 06:55 PM
No, the image isn't distorted, bad choice of words on my part. The image is at the top of the viewing window, with a double-sized letterbox black bar across the bottom.

AutoGK is about half the compression speed of Dr. Divx on the same hardware (disclaimer: settings may be the reason). Overnight encoding is fine with me, but a day and a night is pushing it.

cheer
06-07-2006, 09:05 PM
No, the image isn't distorted, bad choice of words on my part. The image is at the top of the viewing window, with a double-sized letterbox black bar across the bottom.
That doesn't make any sense...that's just weird. Must be something odd about the stream causing AutoGK to mis-detect things. You might want to consider using Gordian Knot, which gives you more manual control.

AutoGK is about half the compression speed of Dr. Divx on the same hardware (disclaimer: settings may be the reason).
Has to be...after all, it's the same codec, right? To be honest, I don't use the one-button-type encoders -- if I really do want to convert to Divx/Xvid/whatever I use Avisynth to frameserve into Vdub so I can have total control.

alfonzotan
06-08-2006, 07:21 AM
Has to be...after all, it's the same codec, right? To be honest, I don't use the one-button-type encoders -- if I really do want to convert to Divx/Xvid/whatever I use Avisynth to frameserve into Vdub so I can have total control.

Not exactly, Divx codec for Dr. Divx, Xvid for AGK. How much difference that should make in encoding time, I don't have a clue. I have heard from others that as a rule AutoGK is on the slow side compared to other encoders. Still, given that I did the encoding on a POS white box PC (1.1 GHz Athlon without a hell of a lot of memory), I probably don't have a lot of room to gripe.

cheer
06-08-2006, 04:19 PM
Ah OK, fair enough. You might take a stab at Gordian Knot, which can use Divx, Xvid, or others...

alfonzotan
06-11-2006, 10:08 AM
Okay, I think I've found my "sweet spot" for HD MP4 re-encodes. Process goes like this:

1. Extract with TyTools as MPEG, be sure to set "Fill AC3 Holes" in the audio prefs.
2. Edit with VideoReDo, save as MPEG.
3. Encode with Dr. Divx using these settings: High Definition, Balanced quality, crop bottom 8 pixels, progressive source, 4000 kbps bit rate, 1-pass, AC3 audio (should be a pass-through).

Output file is roughly half the size of the original file, and the picture quality is excellent. Encode time is "overnight" for a long movie, and the files are playing back nicely on every software player I've tried. I think this is going to work for me. Now, to start building that media server...

tlphipps
06-11-2006, 10:39 AM
Encode time is "overnight" for a long movie, and the files are playing back nicely on every software player I've tried. I think this is going to work for me. Now, to start building that media server...

Speaking of media servers, have you tried streaming any of the files you've produced back to a TiVo through Tivoserver? I'd love to archive some of my HD stuff, but only if I can stream it through tivoserver back to my SD TiVos.

alfonzotan
06-11-2006, 10:42 AM
Speaking of media servers, have you tried streaming any of the files you've produced back to a TiVo through Tivoserver? I'd love to archive some of my HD stuff, but only if I can stream it through tivoserver back to my SD TiVos.

No, I've never really played around with that. I'm planning to use a computer (Mac Mini) as the playback device, not a Tivo.

tlphipps
06-11-2006, 10:36 PM
No, I've never really played around with that. I'm planning to use a computer (Mac Mini) as the playback device, not a Tivo.

If anyone has any test clips (less than 200MB or so) that they wouldn't mind sharing I can provide an FTP server for upload and I'll be happy to test streaming back to my SD TiVo with tivoserver.

rbreding
06-14-2006, 11:06 PM
If anyone has any test clips (less than 200MB or so) that they wouldn't mind sharing I can provide an FTP server for upload and I'll be happy to test streaming back to my SD TiVo with tivoserver.


I don't have any small test clips but I have tried what you have asked.

The quality is awesome, however your transcode time to get it back from AVI to the SD Tivo can be looooooonnnnnnnnnnnnggggggggg.

tlphipps
06-14-2006, 11:09 PM
I don't have any small test clips but I have tried what you have asked.

The quality is awesome, however your transcode time to get it back from AVI to the SD Tivo can be looooooonnnnnnnnnnnnggggggggg.


Awesome! Thanks for the update. I figured the return trip would take awhile, but I'm just glad to know it CAN be done.

jkrell
06-28-2006, 05:57 PM
I want to be able to play files in XBMC and PSP/Video iPod, and I'd like very much not to have to have custom files depending on the intended playback device. In other words, I'd like to be able to encode a show in H.264 that I will then play through the network on XBMC (both on 4:3 TVs and 16:9 TVs), as well as on my PSP and/or Video iPod.

Has anyone here used SUPER? I have only had time to experiment with it, and it seems promising. I also have Nero Recode 2, but my first "test" failed miserably (and after about 24 hours spent encoding).

Is anyone aware of tools that run on Linux? Ideally, I'd have my dedicated Linux server run the encoder, thus freeing up my Windows machine for day-to-day use.

cheer
06-28-2006, 06:50 PM
On Linux, mencoder will do you just fine. Look here (http://www.mplayerhq.hu/DOCS/HTML/en/video-codecs.html#codec-x264-encode).