ffmpeg with MPEG to TY+ encoding support

[bcc]

Ok, I've posted a new version of the source patches, version 1.3. Changes:

• Completely skip generation of master chunks if writing to a pipe/fifo (for tivoserver)
• New switch -x to include template XML.

Someone let me know if this new XML support does what is wanted. I coded this differently than before in that I'm simply merging the template XML with the dynamically generated XML. The dynamically generated XML is appended to the old, with the assumption that the later values will supersede the template values during insertion. This keeps ffmpeg from having to use libxml.

[shstevens]

i will give that a try tonight (currently encoding some stuff with the old version of ffmpeg and i don't want to run two vesions at once and freak windows out, heh). any chance you could put a windows binary together? unfortunately i have never gotten into compiling (i can build a mean network though :^)

if anyone is an ffmpeg guru and can help me out, i would love to figure out the right command line arguments to downconvert the bitrate. currently i have had to express all of the video attributes (aspect, resolution, bitrate, and frame rate) to get it to properly pull the video out. i love the idea of not doing anything to a stream unless i have to though (in this case down converting the bitrate is something i consider i have to do)...

thanks,
shawn

[phat_bastard]

When a recording is xferred using MRV, as opposed to mfs_ftp, that information is NOT sent... the tivo regenerates it. If you try to xfer with both tivoserver and mfs_ftp, and the trick play is different, it could be problems with that bitmap.

Strange, of the handful of short sample streams I've tested before and after transferring with MRV, the first N bytes of the (first and only) master chunk match exactly. After byte N the synthetic ty master chunk contains all nulls, so I'd assume the regenerated master chunk (after tivoserver > mrv > tivo) is just uninitialized random bits. There are more than [nchunks / 8] bytes after the header, so there must be something else here, but I don't think it's relevant.

This is such a minor issue that it seems foolish to even look at it. I'm suprised no-one else is noticing it, but then my OCD may be worse than my shrink has led me to believe.

BCC: Thanks for adding the XML stuff btw, very nice. I'll try to test it out and report back if no-one beats me to it.

[shstevens]

thanks for the 1.3 binary post update. trying the xml now :^)

shawn

[bcc]

thanks for the 1.3 binary post update. trying the xml now :^)

Welcome. I wasn't awake enough last night to post a notice.

[shstevens]

Not sure if your chunk issues are related to this, but the output seems a little strange when it is generating the master chunks.

Multiplexing Audio and Video into TY format...
FFmpeg version SVN-r, Copyright (c) 2000-2004 Fabrice Bellard
configuration: --enable-gpl --enable-a52 --disable-v4l --disable-dv1394 --disable-ffplay --disable-ffserver
libavutil version: 49.0.0
libavcodec version: 51.9.0
libavformat version: 50.4.0
built on Jun 29 2006 01:28:36, gcc: 3.4.4 (cygming special) (gdc 0.12, using dmd 0.125)
could not open '-y'
Input #0, mpeg, from 'd:\vobs\Slapshot.VOB':
Duration: 02:03:00.1, start: 0.280633, bitrate: 7802 kb/s
Stream #0.0[0x1e0]: Video: mpeg2video, yuv420p, 720x480, 9800 kb/s, 29.97 fps(r)
Stream #0.1[0x80]: Audio: ac3, 48000 Hz, stereo, 192 kb/s
Output #0, ty, to 'd:\tys\Slapshot.ty':
Stream #0.0: Video: mpeg2video, yuv420p, 720x480, q=2-31, 2500 kb/s, 29.97 fps(c)
Stream #0.1: Audio: ac3, 48000 Hz, stereo, 192 kb/s
Stream mapping:
Stream #0.0 -> #0.0
Stream #0.1 -> #0.1
Press [q] to stop encoding
Generating master chunk for chunks 0:4095379.5 bitrate= 0.0kbits/s
Generating master chunk for chunks 4096:8191
Generating master chunk for chunks 8192:12287
Generating master chunk for chunks 12288:16383
Generating master chunk for chunks 16384:19081
Successfully wrote new XML
frame=221204 q=2.0 Lsize= 0kB time=7380.8 bitrate= 0.0kbits/s
video:1777972kB audio:172988kB global headers:0kB muxing overhead -100.000000%
'el' is not recognized as an internal or external command,
operable program or batch file.

D:\vobs>

not sure what the:
el' is not recognized as an internal or external command,

is, but it could be a typo in my batchfile....

Just thought the part on creating master chunks looked odd, and it mentioned chunks so I thought i would post it :^)

Shawn

[bcc]

The batch file error would be from your batch script.
I also note you got a "could not open '-y'" error, so your command line must be messed up.
The "chunks 0:4095379.5" part is because ffmpeg and my libraries are cross talking with their diagnostic output. Looks like I should make my libraries use stderr to appease ffmpeg.

[shstevens]

yes, i do have some commandline issues. i forgot the XML file name so -y was the next arguement and it thought that was my xml file.

wasn't sure on the chunk stuff. looked odd, so i posted it. you know much better what is really going on though, heh.

working it out now... have to go get a sammich though....

shawn

[phat_bastard]

No luck with xml yet - but I may be doing something stupid. The dynamic xml seems to get inserted correctly, then...

      6 [main] ffmpeg 4004 _cygtls::handle_exceptions: Exception: STATUS_ACCESS_VIOLATION
 216806 [main] ffmpeg 4004 open_stackdumpfile: Dumping stack trace to ffmpeg.exe.stackdump

Here's the xml I was feeding it:

<?xml version="1.1" tivoversion="6.2-01-2-151"?>
 <Object type="Recording" id="_top">
  <SubObject type="Showing" id="Showing">
  <Object type="Program" id="Program">
  <DescLanguage>English</DescLanguage>
  <Description>Description</Description>
  <Title>Title</Title>
  </Object>
  </SubObject>
  <StartTime>0</StartTime>
  <StopTime>0</StopTime>
  <StreamFileSize>0</StreamFileSize>
 </Object>

Is it my xml?

[bcc]

Is it my xml?

Yes, your XML needs to be bounded by 512 # marker bytes. If you could see the ffmpeg output before the crash you should have seen a hint, such as:

Generating master chunk for chunks 0:27
Found XML but rewrite failed, appending instead...
Successfully wrote new XML

Not sure why this makes ffmpeg crash, but it's crashing after your file is complete (but without the XML template you want)

[BTUxNine]

re: appended xml... I'm afraid just appending the fields may not work well with tivoserver... certain fields (actor, genre, etc.) have multiple entries, so it's set up NOT to supercede any of the data... it aggregates it

re: logging... if those messages are being sent to stdout, then it'd certainly help if they were changed over to stderr (or the ffmpeg logging) because stdout messages would break piping to stdout

[phat_bastard]

bcc: I attempted to apply the 1.3 diffs to the current SVN and I'm running into some errors. Would you mind verifying that they merge against the current snapshot? I'm not really familiar with tweaking diff files, but it looks like there might be a few lines miffed in this latest one.

I'm wanting to look at the xml append routines and possibly mod them to:
1) use one or the other but not both (internal or external xml)
2) hopefully insert only the showing duration and file size tags if using external xml
3) generate the delimiter chars

[bcc]

re: appended xml... I'm afraid just appending the fields may not work well with tivoserver... certain fields (actor, genre, etc.) have multiple entries, so it's set up NOT to supercede any of the data... it aggregates it

The objects dynamically updated with my code are simple objects that are not comprised of lists. The user supplied template comes first, then these dynamically changed objects. So it wouldn't be hard to have the objects get superseded in the order they are listed, as with mfs_ftp.
Yes it would be cleaner to just #define HAVE_XML_LIB and include my xml.c, and libxml. Then the objects will be overwritten instead. ffmpeg could be compiled for tivoserver this way.

Or maybe ffmpeg shouldn't need to deal with XML at all if it's being used with tivoserver.

re: logging... if those messages are being sent to stdout, then it'd certainly help if they were changed over to stderr (or the ffmpeg logging) because stdout messages would break piping to stdout

Hmm, non-error informative message to stderr doesn't make much sense to me. And ffmpeg is actually already inconsistent about this - in ffmpeg's usage output, half the output goes to stdout, and half to stderr.
Personally I don't want to make my log messages directly reference ffmpeg's log facility, as the libraries are useful without ffmpeg, but it'd be OK for dbgprint() to hook in to ffmpeg I suppose. For now I've simply made dbgprint use stderr.

[bcc]

bcc: I attempted to apply the 1.3 diffs to the current SVN and I'm running into some errors. Would you mind verifying that they merge against the current snapshot? I'm not really familiar with tweaking diff files, but it looks like there might be a few lines miffed in this latest one.

I checked and it looks like they made some cosmetic changes that make 1 of the hunks fail to apply. I would suggest using an older SVN revision. I'm currently on 5510. So you could just use '-r 5510' during checkout.

I'm wanting to look at the xml append routines and possibly mod them to:
1) use one or the other but not both (internal or external xml)
2) hopefully insert only the showing duration and file size tags if using external xml
3) generate the delimiter chars

1. You get just auto-generated XML if you leave off the new -x argument
2. ? Show duration & file size are already handled in the auto-generated XML
3. Already handled in the auto-generated XML

I've also got some new code that handles a user supplied template with or without the '#' marker bytes.

[phat_bastard]

I checked and it looks like they made some cosmetic changes that make 1 of the hunks fail to apply. I would suggest using an older SVN revision. I'm currently on 5510. So you could just use '-r 5510' during checkout.

Thanks, will try it out.

1. You get just auto-generated XML if you leave off the new -x argument
2. ? Show duration & file size are already handled in the auto-generated XML
3. Already handled in the auto-generated XML

I've also got some new code that handles a user supplied template with or without the '#' marker bytes.

How about this idea: Since, for all intents and purposes the only thing most of us are trying to accomplish with external xml is to populate Description, MpaaRating, and Title, why not just pull those three fields from a file instead of messing around with the entire tempate and the additional overhead of dealing with xml? I've never been a big fan of it to start with, so I'm sure a good old fashioned config file with it's [variable]=[value][\n] layout would be just fine for everyone, and should be relatively simple to implement.