PDA

View Full Version : S1 -> S2: Analyzing the audio problem...



AlphaWolf
02-06-2004, 03:38 PM
I had some spare time today, so I recorded some sample audio only (easiest to compare) tystreams from an S1 and an S2 and did some playing around with them.

I am not much of an mpeg expert, but I took a peek peeked at the stream with a hex editor. The first thing I really noticed is that on S2 audio only streams, the master chunk is all but blank. All it contains is the regular tystream part header to indicate chunk sizes, part size, etc. The S1 stream has that, plus its loaded with crap that means nothing to me.

So far as the chunk records go, everything is pretty much as you'd expect (nothing but 3C0 headers, or MPEG Layer 2 audio, except in the S1 streams, where the record always begins with a 2C0 header.)

I was hoping to see an obvious change that would be necessary to make these streams compatible with S2 units, but its obviously way beyond me.

This one is definitely for our tystream experts to look at, so, I have made some sample streams available here (http://members.cox.net/alphawolf_/audiostreams.rar). (it's only a 5 meg file) The S1 stream wa recorded from 3.1.0b, and the S2 stream was recorded from 3.1.1b.

captain_video
02-06-2004, 06:41 PM
I've been playing around with mfs_ftp lately and one of the things I've been trying to do is transfer some shows from my S1 DTivo to my S2 DTivos. Aside from some scrambling issues (noscramble.o on the S1 vs. tivoapp patches on the S2's; csoscout.tcl fixed some of them) the transferred shows that eventually played back on the S2 units had no sound. The video was fine but they were dead quiet. The strange thing is that when I attempted to play them back while still in the process of being transferred, they played back with no scrambling issues and with full sound. The scrambling and audio problems didn't occur until the transfer was completed. How's that for weird?

AlphaWolf
02-06-2004, 07:42 PM
There are two possible explanations for this.

One explanation would be that what you thought was your inserted stream really wasn't, and instead it was the remains of a stream that was previously recorded in that exact location of your tivos hard disk. When you use mfs streamcreate to produce a blank tystream (and subsequently fill it) this can easily happen, and your audio would work while your tivo was playing that.

Another explanation would be that the S2 dtivos really are fully compatible with S1 streams, only theres an attribute in MFS which when set breaks this compatibility when the stream plays. I am currently investigating this possibility...

EDIT: tried a bunch of settings and nothing worked. It's probable that your tivo was playing something other than the actual stream you tried to insert.

EDIT 2: Yep, just confirmed that is the cause. I started an insert of an S1 stream, and right after the insertion began, I tried to play the stream, it played the first bits of video, but then after that ran out, the timeshift arrow moved to the very end of the bar, and started playing audio from some other previously existing tystream.

captain_video
02-06-2004, 09:17 PM
Interesting. I didn't try playing the entire clip as only the first FSID was still in the process of transfer [the progress bar indicated a full one-hour length for the video but was indicating a partial recording of about 30 minutes (green bar indicating the actual recording) for a tmf file of 1.09GB]. The audio definitely matched the clip I was transferring because the voices were in perfect sync. This happened twice with the same clip.

The first time I hadn't deleted the *.xml files from the cache directory so I deleted the files and the cache and transferred them again from scratch after a reboot of each DTivo with the exact same results. I only tried the audio at the beginning of the clip and it played both times, but only while the transfer was in progress. When it completed the video wouldn't play at all. A quick run of ciphercheck.tcl indicated the CSOs were still set (i.e. a No in the 1st column and a Yes in the 2nd) for the clips I transferred. Oddly, one of the three clips still refused to play after running csoscout.tcl. None of the clips had any audio after nuking the CSOs.

jdiner
02-06-2004, 09:50 PM
Ok. There are number of crucial differences in the formats of the 2 types, S1 vs. S2. I have to deal with them when splitting and what not.

Without a clear picture of what it is you are trying to do I am not sure what you know or looking to find out.

But if you can fill me in a bit I would be happy to explain what I have figured out between the 2 formats. The nice thing is that the differences at the level I deal with them on are subtle yet are all packagaing of the data. I.e. no specific data changes between the 2 formats.

--jdiner

AlphaWolf
02-06-2004, 10:31 PM
The goal here is to get S1 dtivo streams (non AC3) to play in an S2 dtivo. The video plays alright, but there is no audio at all. S2 streams play fine on an S1 dtivo however.

Disconnect
04-03-2004, 12:32 PM
The goal here is to get S1 dtivo streams (non AC3) to play in an S2 dtivo. The video plays alright, but there is no audio at all. S2 streams play fine on an S1 dtivo however.

Non-AC3? Does that mean that AC3 streams will play? (Also, if its really an issue with the mfs info being set after the insert completes, would it be fixable in the future - ie insert stuff now, and later alter mfs to whatever the 'correct' info is..)

Tomorrow is my big s1->s2 dtivo conversion, so I'll experiment a bit and get back to you. (But I'm not a tcl or mfs guy so any guidance/specific tests/etc would be very welcome..)

Disconnect
04-04-2004, 06:47 PM
So far, I've got a 2.0-analog/2.0-digital/5.1 digital stream that plays fine and a standard stream that doesn't.

Tested:
- Show off HBO (Starts at analog 2.0, then digital 2.0, then digital 5.1)
* Played fine during insert, still plays fine
* analog/2.0/5.1 all output correctly on digital out
- Generic stream (cartoon)
* There is no sound on analog or digital - digital out says pcm/48000 but doesn't make a peep :(

I'm going to extract something (analogish) recorded on it and see what I can find out, but I'm not an mpeg guy so suggestions are greatly welcome..

Also, I can confirm that it is possible during insert to 'jump' into old data that hasn't been overwritten yet.

AVD
04-06-2004, 07:01 PM
isn't this thread a good candidate for the development forum, as it is analysis for a future fix and not really mfs_ftp support

[edit] that was fast :eek:

Disconnect
04-07-2004, 08:43 PM
OK. Shut down tivoapp, catted the mp2 to /dev/mp2 and it played. So the hardware is more than capable of playing the s1 stereo streams. Bonus, it also played (both analog and digital) when catted to /dev/ac3. So if we can convince tivoapp that these streams are ac3, it should Just Work.

So...umm... anyone know how to do that? :confused:

alldeadhomiez
04-09-2004, 06:05 PM
VALID audio data from a D2 stream, seen at /dev/mpeg0a on a D2:

aud_write 00 00 01 c0 02 4a 80 80 07 21 04 13 f0 f5 ff ff ff fd a4 64 66 33
33 44 55 44 55 55 44 44 54 6d ...

aud_write 00 00 01 c0 02 4a 80 80 07 21 04 15 01 d5 ff ff ff fd a4 64 55 22
22 44 55 44 44 55 55 55 44 b1 ...

aud_write 00 00 01 c0 02 4a 80 80 07 21 04 15 12 b5 ff ff ff fd a4 64 66 33
33 33 44 44 34 44 44 34 34 92 ...

aud_write 00 00 01 c0 02 4a 80 80 07 21 04 15 23 95 ff ff ff fd a4 64 77 33
33 33 43 33 43 43 43 43 43 92 ...

SILENT audio data inserted from a D1, as seen at /dev/mpeg0a on a D2:

aud_write 00 00 00 01 c0 02 45 21 02 f1 49 cf ff fd a4 64 66 44 44 66 56 55
45 34 34 22 11 6d b2 49 00 00 ...

aud_write 00 00 00 00 01 c0 02 45 21 02 f3 cd ef ff fd a4 64 66 33 33 66 67
66 56 55 34 23 22 48 92 48 00 ...

aud_write 00 00 00 01 c0 02 45 21 02 f3 de cf ff fd a4 64 66 33 33 66 66 66
66 66 34 33 22 28 92 49 00 00 ...

aud_write 00 00 01 c0 02 45 21 02 f3 ef af ff fd a4 64 56 33 33 66 77 77 66
56 33 34 22 24 92 48 00 00 01 ...

Removing the padding looks tempting, but I tried it and it didn't work. It may or may not be necessary, but it's definitely not sufficient. The next step would be to parse this data and figure out what other differences exist.

AlphaWolf
04-09-2004, 07:06 PM
OK. Shut down tivoapp, catted the mp2 to /dev/mp2 and it played. So the hardware is more than capable of playing the s1 stereo streams. Bonus, it also played (both analog and digital) when catted to /dev/ac3. So if we can convince tivoapp that these streams are ac3, it should Just Work.

So...umm... anyone know how to do that? :confused:

*shrug* change the PES start header from either 3C0 (mpeg audio layer 1) or 4C0 (mpeg audio layer 2) to 9C0 (AC3 audio)?

Disconnect
04-13-2004, 11:04 AM
Well yeah...

But I'm having issues getting it to parse on x86 (not 110% certain I'm reading it properly either.) Next step is to pull out the mplayer-tivo parser and see if I did a stupid.

In theory, if I get time (hopefully later this week) I'll set up a convert-audio that will just stream the ty from one file to another, converting audio tags along the way.

The better fix is still to patch tivoapp, but....

alldeadhomiez
04-13-2004, 11:09 PM
I wrote some code that modernizes the MPEG-1 PES header found on D2 MPEG-2 streams:


int
write_audio(int fd, uint8_t *buf, size_t count)
{
int d, c, ret;
uint8_t buf2[BUFSIZE];
uint16_t pes_header_size;

if((count > (BUFSIZE - 3)) || (count < 12))
return(mywrite(fd, buf, count));

d = -1;
for(c = 0 ; c < 8 ; ++c)
{
if((buf[c] == 0x00) &&
(buf[c + 1] == 0x00) &&
(buf[c + 2] == 0x01) &&
((buf[c + 3] & 0xc0) == 0xc0))
{
d = c;
break;
}
}

if(d == -1)
return(mywrite(fd, buf, count));

/* 0x21 means we're missing the PES header flags (broken S1 stream) */
if((buf[d + 6] & 0x21) != 0x21)
return(mywrite(fd, buf, count));

pes_header_size = buf[d + 5] | (buf[d + 4] << 8);

pes_header_size += 3;
memcpy(buf2, &buf[d], 4); /* prefix code: 00 00 01
* stream id: c0 */

buf2[4] = pes_header_size >> 8; /* 2 bytes: pkt size */
buf2[5] = pes_header_size & 0xff;
/* flags: 80 80
* remaining hdr len: 05
*/
buf2[6] = 0x84; buf2[7] = 0x80; buf2[8] = 0x05;

/* 5 bytes: PTS timestamp */
memcpy(&buf2[9], &buf[d + 6], 5);

/* PES packet data follows */
memcpy(&buf2[14], &buf[11 + d], count - 11 - d);

fprintf(stderr, "real_write ");
for(c = 0 ; c < 20 ; ++c)
{
fprintf(stderr, "%02x ", (int)(unsigned char)buf2[c]);
}
fprintf(stderr, "\n");

ret = mywrite(fd, buf2, count + 3 - d);
if(ret > 0)
ret = count;
return(ret);
}

By intercepting myworld's writes to mpeg0a and rewriting the PES header prior to output, I was able to get sound from inserted D1 streams which were previously silent. However, the audio is either very fast or partially cut off - I only hear pieces of it every few seconds. Those pieces seem to be at least somewhat in sync with the video when they start.

Possibilities include:

- wrong bitrate
- other missing/broken parameters elsewhere in the tystream
- I'm adding a PES header that is not appropriate for the payload?
- buggy code
- something to do with syncing up the video?

Furthermore I have not gotten vplay to play audio reliably on the D2. I've logged a few ioctls that myworld uses to initialize the device, but most of the time I just get silence:


at startup (only?):

aud_ioctl: 00000456 10102844
aud_ioctl: 000003f0 00000000
aud_ioctl: 000007eb 00000001


aud_ioctl: 000007eb 00000000
aud_ioctl: 00000461 7d1ff8b0 00 00 00 00 00 00 00 01 ff ff ff ff 00 00 00 02 00 00 00 00 00 00 00 41 00 00 00 00 00 00 00 00
aud_ioctl: 000003ef 00000000
aud_ioctl: 00000426 00000000

prior to playing audio stream:

aud_ioctl: 000007eb 00000001
aud_ioctl: 000003f0 0000942c
aud_ioctl: 000007eb 00000000
aud_ioctl: 000003ef 00000000

on occasion, during streams:

aud_ioctl: 00001f46 7d1ff8e8 10 08 2c f0 00 00 00 0b 10 10 27 10 7d 1f f9 88 10 08 2c f0 01 34 ce 58 10 08 2c f0 10 08 2c f0

It's possible that a mixer or some other integrated device needs to be dealt with to properly play sound. I have not looked into this yet. Having a working vplay binary would speed up the testing cycle, as rebooting the box takes several minutes.

Can anyone shed any light on either of these issues?

Many of those ioctls are not documented in (cwingert's?) osdtxt.c, and may be new in the Series 2.

smoke_007
04-27-2004, 05:58 PM
Has anyone tried yet or know if upgrading a S1, to the latest Tivo Software version, will get the audio to play with the video when shows are transfered to a S2 Tivo?

My DSR6000 is still at 2.5.2. I'm going to try installing xtreme upgrade 3.1.0 on a spare HD and throwing it in the DSR6000. I'll let it record a little bit and try transfering that to my new DSR704. If that still doesn't work, I'm debating pluggin in the DSR6000 to the phone and try to get it to update to the latest version it will pull.

(Doing that just brings up more questions in my mind though. Is that okay to do? Will the tivo people come and get me for having hacks on the unit? :eek: I'm guessing that even if it does update, it'll probably break some of the hacks.)

So, in the hopes of saving me a bunch of time that I really don't have anyways... :D Does someone know if a newwer tivo OS on a S1 will make any difference?

I'd really like to transfer off a bunch of the shows onto my new tivo. Also, I was hoping to be able to do the streaming from one tivo to the other... Guess I might have to pick up another DSR704 for $99 if no one can figure this out... =\

Thank you guys a TON!
With out all your hacks and mods, tinkering with my tivo wouldn't be as fun... :D

Smoke_007

BTUx9
04-27-2004, 06:04 PM
Has anyone tried yet or know if upgrading a S1, to the latest Tivo Software version, will get the audio to play with the video when shows are transfered to a S2 Tivo?A software upgrade is unlikely to fix the problem.

a) It's the format the stream is written to disk, and they're unlikely to change it, especially when the ONLY advantage would be for hackers :)

b) I doubt there will be many upgrades for s1 dtivos, and the latest version does not produce compatible streams

Tiros
04-28-2004, 01:24 PM
It's possible that a mixer or some other integrated device needs to be dealt with to properly play sound. I have not looked into this yet.


I wonder if this has something to do with the fact that S2 Tivos can emit the system sounds through the digital output, and S1 Tivos can not. There must be some new hardware (mixer?) in the S2 to accomplish this. I doubt the stream controls the mux, but the S2 stream may have some markers or something in it to allow digital audio insertion.

Just trying to provoke some thought here, I really don't know why it wont work.

BTUx9
04-28-2004, 02:16 PM
I wonder if this has something to do with the fact that S2 Tivos can emit the system sounds through the digital output, and S1 Tivos can not.Since when? When playing a Dolby Digital stream, my S2's optical out does NOT emit system sounds. Even if it did, I very much doubt they'd change the stream format for THAT (in fact, I'm pretty sure the DD is the only audio from an s1 that DOES play properly on an s2 dtivo).

Tiros
04-28-2004, 03:12 PM
Since when? When playing a Dolby Digital stream, my S2's optical out does NOT emit system sounds. Even if it did, I very much doubt they'd change the stream format for THAT (in fact, I'm pretty sure the DD is the only audio from an s1 that DOES play properly on an s2 dtivo).

Uh.....Since always.
I never said it plays 'em during DD playback. Not sure what happens during a DD stream, but my S2 has always played the sys sounds out the digital.

The stream IS different. The audio hardware is different. So what is the basis for your doubt? Do you have any technical reason, other information, or even an idea?

jterwelp
05-10-2004, 08:44 PM
Does this problem only occur with DTivo or would it happen with SA S1 -> SA S2 as well? I've tried searching the general forum but keeping getting DTivo-related posts. Thanks!

Jeff

BTUx9
05-10-2004, 10:27 PM
Does this problem only occur with DTivo or would it happen with SA S1 -> SA S2 as well?
AFAIK, the only incompatibility is trying to play s1 dtivo streams on an s2 dtivo... all other combinations supposedly work.

bcc
06-18-2004, 08:49 PM
I've somewhat successfully transcoded an S1 .ty file into an S2 readable format, and played it back on my hd-tivo. I say somewhat because somehow I've introduced some video artificating - a few bad blocks of pixels every 5 seconds or so. This is probably due to how I'm expanding the stream... A.D.H. was on the right track, but missed some details: you have to update the audio packet length in two places for each audio packet that you're rewriting. You have to dynamically adjust the padding in order to preserve 4 byte alignment (presumably this new requirement is MIPS processor induced and is the whole reason for the compatibility issue).

In my test stream, expanding the audio packets (to fix alignment) causes the packets in the chunk to exceed the chunk size. In this case I insert a new chunk to handle the leftover packets. Probably one should be completely rebuilding the chunks so as to not double the stream size as I am.

alldeadhomiez
06-18-2004, 10:49 PM
I've somewhat successfully transcoded an S1 .ty file into an S2 readable format, and played it back on my hd-tivo. I say somewhat because somehow I've introduced some video artificating - a few bad blocks of pixels every 5 seconds or so. This is probably due to how I'm expanding the stream... A.D.H. was on the right track, but missed some details: you have to update the audio packet length in two places for each audio packet that you're rewriting. You have to dynamically adjust the padding in order to preserve 4 byte alignment (presumably this new requirement is MIPS processor induced and is the whole reason for the compatibility issue).

In my test stream, expanding the audio packets (to fix alignment) causes the packets in the chunk to exceed the chunk size. In this case I insert a new chunk to handle the leftover packets. Probably one should be completely rebuilding the chunks so as to not double the stream size as I am.

Are you referring to the length field in the chunk header?

Note that I was not producing a modified tystream; I was intercepting device writes between tivoapp and the kernel. There's a high likelihood that I was not doing it in a way that caused the start address of the modified writes to be 4-byte aligned, which may have been a fatal mistake.

bcc
06-19-2004, 12:12 AM
Are you referring to the length field in the chunk header?
There were actually 3 length fields for me to deal with. The one in the chunk header, the one in the audio packet itself, representing the audio packet length, and the one in the PES header counting the # of pad bytes I dynamically add. I'm not sure if I got the names of these length fields right, I'm lacking a proper spec :)


Note that I was not producing a modified tystream; I was intercepting device writes between tivoapp and the kernel. There's a high likelihood that I was not doing it in a way that caused the start address of the modified writes to be 4-byte aligned, which may have been a fatal mistake.Oh ok. If what you're intercepting is the data starting with the "00 00 01 c0" then you still have the 2 other length fields to deal with. My machine was nice enough to log alignment errors to /var/log/tverr when I got them wrong. For example::
Jun 18 20:25:13 (none) TmkTransform::Trace[195]: Get Validate (buffer 0) 179 segments failed: PES buffer segment 57 length is not word aligned (593).Trying to recover.Doing the S1->S2 transcoding offline is handy as my desktop is a lot faster than the tivo and mplayer has so far been able to reproduce any playback-ability issues (other than the alignment).

bcc
06-19-2004, 03:32 AM
Just an update on the definitions for those length fields... The 'audio packet length' field is really the PES packet length field of the PES header. The third length field I mentioned is called the 'PES header data length'. It should be set to the length of the PTS data plus any stuff bytes you have (or are adding to provide alignment).

bcc
06-20-2004, 09:52 PM
I fixed a bug in my splitting of chunks, and now I get flawless audio&video playback of S1 streams on my HD-Tivo. I've only tested with MPEG audio, not AC3...
Anyways, here's the source in case you want to see exactly how I did the padding and length field processing (or transcode some S1 recordings!).
[Update: use newer version posted below instead]

bcc
11-29-2004, 01:52 AM
Here is a windows binary, and some additional info I provided in a private discussion:

The idea was that the conversion would be a 1 time thing as you decommission your old S1 tivo. So I didn't bother to streamline anything. The code was actually really meant as a proof of concept - I was hoping someone else would run with it if there was ongoing interest. For example AllDeadHomiez was suggesting that he intended to hack the tivo audio driver to support S1 backwards compatibility.

The code is really just a first cut - the converted ty file will be almost 2X the size (tho this is still not large by HD standards). Also I didn't bother to rebuild the index at the start of the ty file so tivo trick plays, such as skipping, may not work great.

It should not be hard to more tightly pack the converted ty file, and fix the trick play support, if that were important.

For me, I was pleasantly surprised to see that my old S1 recordings actually look better when played back on the HD tivo.

[Update: use newer version posted below instead]

bcc
12-12-2004, 08:35 PM
I've re-worked ty1to2 so that it no longer needs to double the size of the converted recordings. To achieve this I re-buffer the audio stream, and rebuild the data chunks of the ty stream. I also rebuild the master chunk records for each sequence of chunks.

Synthesizing master chunk records can be generally useful for adding master chunks to arbitrary ty stream clips (or for the holy-grail of importing .mpg to a tivo). I've tested the library's ability to do the former, and that worked for me as well.

Here's the new version with windows, linux binaries, and source. Old version had a dumb bug where I was sometimes dropping data by screwing up the chunk record count. Oh, and I've again only tested this with mpeg2 audio records not ac3 (I don't seem to have any series 1 ac3 recordings left).

EXAMPLE:

(Note that the below warning is expected for full length recordings, as
there are usually some garbage chunks at the end of the recording. During the
conversion, such garbage chunks are thrown out.)
% ty1to2 -i hou.ty -o h.ty
Warning: Skipping chunk 6674 due to timestamp discontinuity
Rebuilding master chunks
Generating master chunk for chunks 0:4095
Generating master chunk for chunks 4096:6927
% ls -l *.ty
-rw-r--r-- 1 root root 876613789 Dec 12 14:02 hou.ty
-rw-r--r-- 1 root root 908075322 Dec 12 14:12 h.ty
%

rung
12-13-2004, 04:53 PM
I've re-worked ty1to2 so that it no longer needs to double the size of the converted recordings. To achieve this I re-buffer the audio stream, and rebuild the data chunks of the ty stream. I also rebuild the master chunk records for each sequence of chunks.

Synthesizing master chunk records can be generally useful for adding master chunks to arbitrary ty stream clips (or for the holy-grail of importing .mpg to a tivo). I've tested the library's ability to do the former, and that worked for me as well.

Here's the new version with windows, linux binaries, and source. Old version had a dumb bug where I was sometimes dropping data by screwing up the chunk record count. Oh, and I've again only tested this with mpeg2 audio records not ac3 (I don't seem to have any series 1 ac3 recordings left).

EXAMPLE:

(Note that the below warning is expected for full length recordings, as
there are usually some garbage chunks at the end of the recording. During the
conversion, such garbage chunks are thrown out.)
% ty1to2 -i hou.ty -o h.ty
Warning: Skipping chunk 6674 due to timestamp discontinuity
Rebuilding master chunks
Generating master chunk for chunks 0:4095
Generating master chunk for chunks 4096:6927
% ls -l *.ty
-rw-r--r-- 1 root root 876613789 Dec 12 14:02 hou.ty
-rw-r--r-- 1 root root 908075322 Dec 12 14:12 h.ty
%


When you delete chunks, are you adjusting the sequence bitmap in the master chunk as well? If you don't you will loose trick play.

Regards,
Rung

bcc
12-13-2004, 05:42 PM
When you delete chunks, are you adjusting the sequence bitmap in the master chunk as well? If you don't you will loose trick play.

Regards,
RungYes, I'm adjusting the video sequence bitmap (and timestamps). In fact I'm completely rebuilding the bitmap not just shuffling its contents. I am also adjusting the per chunk video sequence record number, as the video sequence records are moving within the chunks.

This is why I was trying to suggest that the code in master.c can be of general use. I've used it to create master chunks for high-def ty stream clips .

rung
12-13-2004, 06:03 PM
Yes, I'm adjusting the video sequence bitmap (and timestamps). In fact I'm completely rebuilding the bitmap not just shuffling its contents. I am also adjusting the per chunk video sequence record number, as the video sequence records are moving within the chunks.

This is why I was trying to suggest that the code in master.c can be of general use. I've used it to create master chunks for high-def ty stream clips .

Wow, that's quite a piece of work! Congrads. First time I tried to do the bitmaps I had a heck of time until I realized that I had the bits in the bitmap bytes in the wrong order (MSB -> LSB reversed). Have you had any luck predicting the stream identifiers and offsets based on the mpeg data? Toughest thing I see is that there seems to be no easy way to predict the size of the picture records. I guess you would just have to linearly scan for mpeg escape codes.

Regards,
Rung

bcc
12-13-2004, 07:30 PM
Wow, that's quite a piece of work! Congrads. First time I tried to do the bitmaps I had a heck of time until I realized that I had the bits in the bitmap bytes in the wrong order (MSB -> LSB reversed). Have you had any luck predicting the stream identifiers and offsets based on the mpeg data? Toughest thing I see is that there seems to be no easy way to predict the size of the picture records. I guess you would just have to linearly scan for mpeg escape codes.
Thanks, I probably wasted too much time on it :) Yes tystream bit ordering is a pain since it uses both bit orderings (and there seems to be no fully standard portable routines to handle bit/byte order).

In this code, I'm digging into the partial PES header to get the length for the audio records. I have to since I'm increasing the PES packet size, but it also lets me find the start of the next PES packet. In this code I'm assuming 1 audio record per PES packet, which I think is true for series 1 ty streams. In my hdemux program, I dig down a layer and process the audio packets to find out their size. This is necessary because for HD tystreams, there may be multiple audio frames per PES packet.

You can't safely do a linear search for audio start codes, as the spec allows the start code to appear in the payload. (I actually have a tystream that demonstrates this). For video, I believe you can search if necessary. In ty1to2, I don't have to dip into the video records at all. In the hdemux code, I do search the video payload at the start of the recording.

bcc
12-13-2004, 07:48 PM
I guess I didn't understand the question about predicting stream identifiers? You mean the first nibble of the ty chunk record type? And predicting for the purpose of going mpg->ty? I should think you could avoid the continuation records more than what is found in typical streams. Hopefully there isn't a requirement for their frequency. I bet you could eliminate their use if you don't care about packing the chunk completely. For the rest of the types, I think they simply mirror the type of data being packed.

rung
12-13-2004, 09:17 PM
I guess I didn't understand the question about predicting stream identifiers? You mean the first nibble of the ty chunk record type? And predicting for the purpose of going mpg->ty? I should think you could avoid the continuation records more than what is found in typical streams. Hopefully there isn't a requirement for their frequency. I bet you could eliminate their use if you don't care about packing the chunk completely. For the rest of the types, I think they simply mirror the type of data being packed.

For 3.0 tys (the only type I am familiar with) there are at least four stream identifiers (two for audio and two for video) in a field in the chunk header. These are separate from the record type identifier in the field. For each stream identifier there is a running offset that (somewhat) follows the size of the record. I would assume that these would have to be generated properly for a ty to work. If I recall correctly, the audio offsets are completely understandable, it was the video ones that get wacky. I have concluded that this wackiness is associated with the 32bit boundry issue that you have ran into with your issue. ty's seem to use a read/bitwise and/write technique to keep the video stream pure (no padding) but also only using 32 bit transfers on 32 bit boudries. It's the only explanation I can come up with from what I see when I analyze the data in ty's.

Regards,
Rung

bcc
12-13-2004, 09:55 PM
For 3.0 tys (the only type I am familiar with) there are at least four stream identifiers (two for audio and two for video) in a field in the chunk header.Still working on your terminology... So when you say 'stream identifiers' you are referring to what is named the tivo_stream_buffer_id here (http://dvd-create.sourceforge.net/tystudio/tystream.shtml)? Theoretically I should have updated the corresponding tivo_stream_buffer_offset fields when I changed the audio record lengths, but I didn't, as I wasn't totally sure of their semantics. Note that I heard *no* audio defects on playback (on an hr10-250). Since I'm sometimes folding audio records together, and just using the existing tivo_stream_buffer_id/offset values, it should have been necessary to update these if the tivo was really relying upon them. Maybe they only come into play during error correction (data missing from the stream).

rung
12-14-2004, 05:31 AM
Still working on your terminology... So when you say 'stream identifiers' you are referring to what is named the tivo_stream_buffer_id here (http://dvd-create.sourceforge.net/tystudio/tystream.shtml)? Theoretically I should have updated the corresponding tivo_stream_buffer_offset fields when I changed the audio record lengths, but I didn't, as I wasn't totally sure of their semantics. Note that I heard *no* audio defects on playback (on an hr10-250). Since I'm sometimes folding audio records together, and just using the existing tivo_stream_buffer_id/offset values, it should have been necessary to update these if the tivo was really relying upon them. Maybe they only come into play during error correction (data missing from the stream).

That's good news. Taking this off-line - see your PM.

Thanks,
Rung

bcc
01-10-2005, 01:14 AM
Ok, I've put up a new version of ty1to2. The previous version was forgetting to flush out the last partial chunk of rewritten records - that has been fixed. I've also added a little bit more error processing. Particularly, rung's idea to check the source ty stream to protect against being fed a stream that already has full audio PES headers. New version is in my old post:
http://www.dealdatabase.com/forum/showpost.php?p=199120&postcount=28

newbie
01-10-2005, 01:35 PM
This version works great, thanks.




Ok, I've put up a new version of ty1to2. The previous version was forgetting to flush out the last partial chunk of rewritten records - that has been fixed. I've also added a little bit more error processing. Particularly, rung's idea to check the source ty stream to protect against being fed a stream that already has full audio PES headers. New version is in my old post:
http://www.dealdatabase.com/forum/showpost.php?p=199120&postcount=28

DarkHelmet
01-11-2005, 01:42 AM
I had to fix ty1to2 to make it work on my 64 bit system. I changed all the u_short and getshort/putshort to explict uint16_t's and all the u_long/getlong/putlong to explicit uint32_t's (and renamed getlong to get32 etc).

And all the printf("%llx", tivo_tstamp_t)'s should be changed to printf("%llx", (long long)tivo_tstamp_t); uint64_t (what tivo_tstamp_t is on a 64 bit system) is a plain long, not a long long. Doing the cast is the least painful fix.

I was in the process of trying to bulk transfer all my S1 dtivo recordings to a new S2 dtivo (SD-DVR80 running 4.0.1b). I hit this:

../ty1to2/ty1to2 -i part00.ty-1 -o part00.ty-2
Warning: Source ty stream already has complete PES header (no need to convert)
Rebuilding master chunks
Generating master chunk for chunks 0:4095
Generating master chunk for chunks 4096:4097
Assertion failed: (i), function generate_gop, file master.c, line 173.
Abort trap (core dumped)

The recording was an AC3/DD recording from ages ago... Showtime usually switch audio formats several times during the shows - this one likely has DD2, DD5.1 and plain mpeg audio. I'm not sure yet whether the warning is relevant for a small part, or for the entire stream. Something doesn't quite look right.. From reading the code, it looks like it is going to be an invalid conversion... (wild guess. maybe it needs to look back in the previous chunks to find the last valid stamp?)

edit: or change 'assert(i)' to 'assert(i >= 0); because gop_stamp[0] looks like it should be valid. It is the first entry in the array.

bcc
01-11-2005, 03:58 AM
I had to fix ty1to2 to make it work on my 64 bit system. I changed all the u_short and getshort/putshort to explict uint16_t's and all the u_long/getlong/putlong to explicit uint32_t's (and renamed getlong to get32 etc).
Looks like you're compiling for a different platform than the ones I tried. With microsoft's (supposedly) 64 bit aware compiler the u_shorts were already unsigned __int16's per win32/stdint.h. Still not clear to me what definition will be totally portable without more complicated OS specific type defining.

And all the printf("%llx", tivo_tstamp_t)'s should be changed to printf("%llx", (long long)tivo_tstamp_t);Dunno why you think that. I actually started out using long long for 64 bit timestamps and had to change it in order to shut up microsoft 64 bit compiler warnings.

uint64_t (what tivo_tstamp_t is on a 64 bit system) is a plain long, not a long long. Doing the cast is the least painful fix.And since long has a different meaning on 32 bit only development environments, best to stay away from such an ambiguous data type... Maybe %qu instead of %ll would be portable?

I was in the process of trying to bulk transfer all my S1 dtivo recordings to a new S2 dtivo (SD-DVR80 running 4.0.1b). I hit this:

../ty1to2/ty1to2 -i part00.ty-1 -o part00.ty-2
Warning: Source ty stream already has complete PES header (no need to convert)
Rebuilding master chunks
Generating master chunk for chunks 0:4095
Generating master chunk for chunks 4096:4097
Assertion failed: (i), function generate_gop, file master.c, line 173.
Abort trap (core dumped)

The recording was an AC3/DD recording from ages ago... Showtime usually switch audio formats several times during the shows - this one likely has DD2, DD5.1 and plain mpeg audio. I'm not sure yet whether the warning is relevant for a small part, or for the entire stream. Something doesn't quite look right.. From reading the code, it looks like it is going to be an invalid conversion... (wild guess. maybe it needs to look back in the previous chunks to find the last valid stamp?)

edit: or change 'assert(i)' to 'assert(i >= 0); because gop_stamp[0] looks like it should be valid. It is the first entry in the array.
Oops, you're right, i = 0 should be considered valid. Will fix. Can't simply change the assert to i >=0, as i is unsigned - have to fix the for loop as well... As for the warning, it only indicates that you've hit at least 1 audio record that has a full PES header. Other audio records that lack the full PES header would still be rewritten. I assume the AC3 S1 dtivo audio records don't need rewriting (I didn't have any ac3 streams to test with).

DarkHelmet
01-11-2005, 05:48 AM
My platform is gcc on amd64/x86_64 (FreeBSD, but linux is the same).

Microsoft has 'long' as 32 bits on this platform, but the unix world generally uses 'long' = 64 bits on 64 bit hardware (alpha, ultrasparc, etc). I only changed u_short for completeness.

My recommendation is to use explicit sized types. I know of no platforms that have u_short not being 16 bits, but since you had get64 and put64, and I needed the explicit get/put32 versions, it made sense to hit them all at once.

O_BINARY doesn't exist here.. that's more of a dos/windows thing, it would make sense to inclusively enable it for M$ platforms only.

tivo_tstamp_t t;
printf("%llx", (long long)t);
.. should work everywhere and not cause any problems, even windows. You've defined tivo_stamp_t to be uint64_t - which I do not suggest changing. Just don't assume that uint64_t is implemented by 'long long'. The cast won't change the code, just fix the warnings.

Anyway, the patch I used: http://people.freebsd.org/~peter/ty1to2.diff

DarkHelmet
01-11-2005, 05:55 AM
BTW; for what it is worth:
printf("SEQ at %d:%d Time: %llx\n", chunkcnt, seq, time);
produces:
cc -c -g -Wall -Wuninitialized -O1 master.c
master.c: In function `parse_chunk':
master.c:118: warning: long long unsigned int format, tivo_tstamp_t arg (arg 4)

defs.h defines it as:
typedef uint64_t tivo_tstamp_t; /* Tivo timestamps are 64 bits */

I'm not saying to change the definition. uint64_t is actually ideal. Just don't try and printf it as though uint64_t were a long long. It might not, 'long long' could be a 128 bit thing.

PS: I apologize heading off topic. I'll shut up now.

DarkHelmet
01-12-2005, 12:57 PM
I found another bug. There is an integer overflow in the lseek() calls in master.c for streams larger than 2GB long. All of these:
lseek(ofd, masteridx*CHUNK_SIZE, SEEK_SET);
need to be changed like this:
lseek(ofd, (off_t)masteridx*CHUNK_SIZE, SEEK_SET);

off_t is a 64 bit size. masteridx etc are unsigned integers (4GB wraparound), but this one:
lseek(ofd, (pos+1)*CHUNK_SIZE, SEEK_SET);
uses pos which is a signed integer giving a wraparound at the 2GB mark. ((off_t)pos+1)*CHUNK_SIZE does the trick.

Adding the casts fixes the master regeneration that stops prematrely at 16384.

For the bystanders, an integer multiplied by an integer is still an integer. In C, if you expect a result too large to fit in an integer, you have to promote one part to the larger size first, because C won't do the promotion for you.

bcc: I found it more convenient to print the tivo timestamps with printf("%.9f", (double)time / TIVO_TIMESCALE), instead of in hex. That way you can easily see the second values and yet not be likely to lose enough significant digits to matter.

Also, I added some trivial code to generate the XML code fragments to use for updating the showing.xml files that I'm reinserting into my S2 dtivo. I had a few cases where I rolled over from 4 to 5 part files as a result of the expansion and having the wrong timestamps on there bothered me. (The tivo seemed to manage, but funny things happened on the program details screen).

I'm sorely tempted to try and hack it into being able to produce 512MB partNN files directly, and avoid the seperate join/split stages when processing the guts of a .tmf file. It has taken quite a while to grind though 120 gigs of data for many passes..

bcc
01-13-2005, 02:34 AM
I found another bug. There is an integer overflow in the lseek() calls in master.c for streams larger than 2GB long.Damn, thanks for finding. Methinks lseek() makes it too easy to hang yourself with type promotion in this case .
bcc: I found it more convenient to print the tivo timestamps with printf("%.9f", (double)time / TIVO_TIMESCALE)That is a subset of my timestamp pretty print routine - tivo_show_time(). But anyways, I still would rather have a more general fix as I also have debug code that prints out 64 bit PTS/DTS timestamps.
Also, I added some trivial code to generate the XML code fragments to use for updating the showing.xml files that I'm reinserting into my S2 dtivo. I had a few cases where I rolled over from 4 to 5 part files as a result of the expansion and having the wrong timestamps on there bothered me. (The tivo seemed to manage, but funny things happened on the program details screen).Sounds handy. I guess ideally the the xml would be rewritten during the conversion which would require parsing the old xml as well. Could get messy.
I'm sorely tempted to try and hack it into being able to produce 512MB partNN files directly, and avoid the seperate join/split stages when processing the guts of a .tmf file. It has taken quite a while to grind though 120 gigs of data for many passes..If you extract/insert as .ty+ instead of tmf you wouldn't have to deal with part splitting&merging. .ty+ is the case I designed for.
I've reworked&integrated the fixes - thanks. Looks like it'ss time for another release.

DarkHelmet
01-13-2005, 11:53 AM
That is a subset of my timestamp pretty print routine - tivo_show_time(). But anyways, I still would rather have a more general fix as I also have debug code that prints out 64 bit PTS/DTS timestamps.
Yeah, I only mentioned it because I found it convenient while trying to figure out how to recreate the begin/end markers in the xml.

Sounds handy. I guess ideally the the xml would be rewritten during the conversion which would require parsing the old xml as well. Could get messy.
I just did it the easy way.. tagged and dumped the output, and used a few lines of perl to automagically replace the entries in the split showing.xml files.

If you extract/insert as .ty+ instead of tmf you wouldn't have to deal with part splitting&merging. .ty+ is the case I designed for.
Hmm. I was using .tmf so that I could work on the showing.xml file as well. It might be my imagination, but correcting the begin/end values seemed to make the forward/reverse jumps smoother. Without doing it, I saw some cases where chunks of the show were missing from the play progress bar, or the info about the show length on the program detail page was missing - eg: it said the time was "0:00 (partial)" even though it had material to play for an hour. Having said that, the tivo doesn't seem to care that much if Begin/End values are right though. It seems more important that they are present for the program detail page to be complete.

I took the begin/end values from the gop timestamps. However... in cases where a live buffer was converted into a recording, the gop timestamps were not zero based.. so I had to correct for that.

FWIW, I was using .tmf inserts on a 4.01b dtivo (sd-dvr80 rid box) using mfs_ftp with the p1.tcl plugin that recreates the folder structures. Besides the downloads and uploads being glacially slow, things have been working well so far. I even learned a few things in the process. :-)

bcc
01-16-2005, 03:42 PM
I've updated ty1to2 to fix the 2/4GB file size bug, and the assertion failure. I think I addressed 64 bit portability as well. The proposed long long hack doesn't work with microsoft's compiler so I had to do a more complex fix. Updated version is here: http://www.dealdatabase.com/forum/showpost.php?p=199120&postcount=28

In case you're curious, the microsoft compiler error was:
Microsoft (R) Program Maintenance Utility Version 7.00.9466
Copyright (C) Microsoft Corporation. All rights reserved.

cl -c -O1 -Iwin32 -o ty1to2.obj ty1to2.c
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 13.00.9466 for 80x86
Copyright (C) Microsoft Corporation 1984-2001. All rights reserved.

ty1to2.c
ty1to2.c(408) : error C2632: 'long' followed by 'long' is illegal
ty1to2.c(409) : error C2632: 'long' followed by 'long' is illegal
NMAKE : fatal error U1077: 'cl' : return code '0x2'
Stop.

tx413
01-18-2005, 02:21 PM
bcc: I'm looking at the code in master.c, where you are parsing the chunk header information. I think there's a problem with how you calculate the number of records. On line 89 you have (for 16-bit record count):

recs = (buf[1] >> 8) | buf[0];

I think for any 8-bit number, if you shift it right 8 bits, you get 0, right? Shouldn't that be:

recs = (buf[1] << 8) | buf[0];

Of course you could use + instead of |...

This should only affect those rare cases where a chunk contains more than 256 records.

bcc
01-18-2005, 02:49 PM
Right, I flubbed the record count read in master.c. Got it right in ty1to2.c. Should have used common code :) Will fix. Now this doesn't matter in practice because when I generate the new chunks I'm limiting the record count to 8 bits via:
#define CHUNK_REC_INDEX_MAX 254 /* Max # records we'll write per chunk */

bcc
02-15-2005, 09:00 PM
Here's a new version of ty1to2. This version updates the StreamFileSize tag in the XML associated with the recording. I'm not sure, but I think this is the only XML tag that needs to be updated as the recording size changes length. Anybody want to tell me for sure?

Same usage as before. Actually I should probably add that the recommended file format is .ty+. Using .ty+ format instead of tmf enables you to convert the entire recording at once instead of per recording segment. If you try to convert the recording separately for each segment you'll wind up with twice as many segments as would otherwise be necessary. If you use .ty+ format, ty1to2 will pack the segments for you. For windows users, the program is now dependent upon 3 dll libraries to do the XML processing. See xml-win-lib.zip for those, and install in the same dir as the ty1to2.exe program.

Newer version here:
http://www.dealdatabase.com/forum/showpost.php?p=233975&postcount=65
Important for files >2GB...

rung
02-16-2005, 04:21 AM
For windows users, the program is now dependent upon 3 dll libraries to do the XML processing. rc3105 write a couple of lines in tcl to do all the xml processing in mfs_ftp. He gave permission to copy that section to allow interoperabilty with other applications. Shouldn't be very hard to convert to c.

bcc
02-16-2005, 12:54 PM
I didn't want to write special purpose C XML parsing code, so I just handed off the job to the nearest standard library. Less work for me that way. Should be OK as long as the API doesn't get in the way. Verdict is still out on that :) Other than the tivoversion tag (which the library won't parse), the library parses and dumps identical XML as mfs_ftp in my tests. If someone has a pointer to a better library...

Now about those <Begin> <End> tags in the "RecordingPart" subobject. I'm thinking those tags just mark which data to use the encryption keys on. And so, the values are probably irrelevent with zeroed out encryption keys. But since I often create an additional FSID segment, do I need to add an additional dummy "RecordingPart" subobject for the new segment, in order to make insertion work right?

rc3105
02-16-2005, 01:15 PM
the xml is sanity checked to ensure that there's a begin/commercialskipoffset/end set for each part. if they don't match the begin/commercialskipoffset/end fields aren't set. for unscrambled recordings that's not really an issue

ty without #xml# will parse/insert properly, it just gets default data and a best-guess name based on uploaded filename

DarkHelmet
02-16-2005, 07:56 PM
I found that the incorrect (or missing) Begin/End markers would mainly just mess up the information on the NowPlaying list. Deleting them entirely (I was using .tmf) would cause the program to still play ok, but it would show a size of 0:00 on the program detail page.

When I first tried this using earlier versions (and not-updated Begin/End tags), I had strange effects with skipping forwards/backwards and to the 15-minute markers. Along the way towards updating the tags, I realized I was doing something else stupid. It might have been the other things that messed me up, or the 2GB bug or something else.

What I did when I bulk converted my recordordings from S1 -> S2, was to simply create fresh recordingpart records. In my case, updating the XML was the easy part. The bulk of the time was spent uploading/downloading (several days worth), so I didn't want to take the chance. Using correct recordingparts only took a few minutes, and I didn't want to have to re-upload because I found something later on that cared.

I cheated though and used a perl script to update it, not a real XML parser etc. If I updated the begin/end parts and did *not* add new parts when the expansion had caused an overflow into a new 512MB part, the nowplaying list data was wrong.

bcc
02-16-2005, 09:29 PM
the xml is sanity checked to ensure that there's a begin/commercialskipoffset/end set for each part. if they don't match the begin/commercialskipoffset/end fields aren't set. for unscrambled recordings that's not really an issueThanks. In mfs_ftp 1.2.9p, I just noticed that set_rec_csos is only called for tmf not .ty/.ty+. So it looks like that whole section of the XML doesn't matter for insertion, as inserting .ty+ has worked fine for me without it, including trick-play support :)
ty without #xml# will parse/insert properly, it just gets default data and a best-guess name based on uploaded filenameReminds me - those best guesses aren't so great, as such recordings are left with bad expiration dates. And so ones tverr fills up with messages like:

Feb 17 01:20:08 (none) Scheduler[206]: Scheduled recording 875687 expires before it starts

I've been using the fix_expdate.tcl script to fix these up when I insert .ty without xml...

bcc
02-16-2005, 10:05 PM
I found that the incorrect (or missing) Begin/End markers would mainly just mess up the information on the NowPlaying list. Deleting them entirely (I was using .tmf) would cause the program to still play ok, but it would show a size of 0:00 on the program detail page.Hmm, I can't reproduce those problems when I insert as .ty (and so the begin,end, and cso values aren't used). I can make the recording length go to 0:00 if I chop the recording before insertion such that the recording length no longer matches the <Duration> tag or <StopTime> minus <StartTime>.

If I updated the begin/end parts and did *not* add new parts when the expansion had caused an overflow into a new 512MB part, the nowplaying list data was wrong.Hmm, if the number of segments mismatch, set_rec_csos doesn't use any of 'em, as riley indicated. In that case the behavior should be like inserting as .ty, so I'm confused how your nowplaying data was wrong.

I'm gonna just have to claim this all works if you use .ty+ :)

nova1
02-21-2005, 01:24 PM
Here's a version of copy_rec() that can handle large records(>128KB)and generate continuation records to pack a chunk. Useful for working with HiDef TY files(hdemux ty -> a,v -> ty).

jmaziarz
08-10-2005, 03:45 PM
I am trying to convert programs from my S1 DTivo Sony Sat-T60 to prepare for my upgrade to a S2 DTivo Philips DSR708. The file I am trying to convert is a PPV program recorded in DD.

For reference here is the system/kernel version info for the S1:

Software Version: 3.1.0c2-01-1-011
Kernel Version: 2.1.24-TiVo-2.5

Here is the output of the first file I tried to convert using ty1to2 version 1.1 (latest):



D:\Videos\FromS1Tivo>ty1to2 -i "d:\videos\FromS1Tivo\program1.ty+"
-o "d:\Videos\ToS2Tivo\program1.ty"
Warning: Source ty stream already has complete PES header (no need to convert)
Warning: Skipping chunk 17570 due to timestamp discontinuity
Rebuilding master chunks
Generating master chunk for chunks 0:4095
Generating master chunk for chunks 4096:8191
Generating master chunk for chunks 8192:12287
Generating master chunk for chunks 12288:16383
Generating master chunk for chunks 16384:16384
Assertion failed: i >= 0, file master.c, line 159

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.


Thanks for your hard work and efforts. Any help would be greatly appreciated. :)

bcc
08-14-2005, 02:17 AM
Assertion failed: i >= 0, file master.c, line 159Ah, that assertion failure with master.c was fixed since the last ty1to2 build. Just needs to be recompiled with a fresh copy. Now in this case, ty1to2 (master.c) got faked out because you had a segment that was not long enough to contain any video sequences. Your file was probably converted fine, with a bad chunk right at the end. However, this part:
Warning: Source ty stream already has complete PES header (no need to convert)Leads me to believe that your source file wasn't even a valid s1 dtivo recording. Perhaps you already had converted it once.

bndt12
08-23-2005, 08:39 PM
I am trying to convert programs from my S1 DTivo Sony Sat-T60 to prepare for my upgrade to a S2 DTivo Philips DSR708. The file I am trying to convert is a PPV program recorded in DD.

For reference here is the system/kernel version info for the S1:

Software Version: 3.1.0c2-01-1-011
Kernel Version: 2.1.24-TiVo-2.5

Here is the output of the first file I tried to convert using ty1to2 version 1.1 (latest):



D:\Videos\FromS1Tivo>ty1to2 -i "d:\videos\FromS1Tivo\program1.ty+"
-o "d:\Videos\ToS2Tivo\program1.ty"
Warning: Source ty stream already has complete PES header (no need to convert)
Warning: Skipping chunk 17570 due to timestamp discontinuity
Rebuilding master chunks
Generating master chunk for chunks 0:4095
Generating master chunk for chunks 4096:8191
Generating master chunk for chunks 8192:12287
Generating master chunk for chunks 12288:16383
Generating master chunk for chunks 16384:16384
Assertion failed: i >= 0, file master.c, line 159

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.


Thanks for your hard work and efforts. Any help would be greatly appreciated. :)

You do not need to run DD recordings through this tool. As long as the recording is decrypted you can transfer it directly to the S2.

I know because I am doing the same upgrade, T60 to DSR708.

bndt12
08-23-2005, 08:48 PM
In an effort to get full compatability with the Folders option in 6.2 you need to use TMF files. I have a ton of Enterprise's and need them to group themselves but using the ty+ files if fails(missing XML data).

When I try to run the TMF through..

F:\ty1to2\Proc>ty1to2 -i "{Star Trek Enterprise}{2005-02-25}{Divergence}{08.00 P
M Fri Feb 25, 2005}{ME30}.tmf" -o "{Star Trek Enterprise}{2005-02-25}{Divergence
}{08.00 PM Fri Feb 25, 2005}{ME30}_a.tmf"
Warning: Skipping chunk 4 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity
Warning: Skipping chunk 68 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity

F:\ty1to2\Proc>

The above generates a 640KB file.

From what I have read it would appear that TMF's should process but not as efficiently as the TY+ files but .....

Please help!

bcc
08-23-2005, 10:25 PM
ty1to2 doesn't support tmf, only ty. Over here: http://www.dealdatabase.com/forum/showthread.php?t=44588 a ty->tmf conversion program was posted. There's a tmf2ty program as well. Someone also stated that tivoserver would be supporting .ty+ and I suspect that's what you're trying to interoperate with.

For ty1to2 I tried to keep it a simple tool, and ty+ is simpler to format and parse.

bndt12
08-23-2005, 11:27 PM
Cool, great tool! Where there's a will there's a way...

BTW - I found the fix for teh folder issues regarding some modules for TWP. Works like a charm even with the TY+ files. Come to find out even TMF's inserted have the same issues:

http://www.dealdatabase.com/forum/showthread.php?t=36208

One question...should I ever see any video quality degradation using ty1to2? I have been moving some old stuff and they appear grainy and this is the only trancoding I have done on them?

Again, thanks for ty1to2!

bcc
08-25-2005, 02:57 PM
One question...should I ever see any video quality degradation using ty1to2? I have been moving some old stuff and they appear grainy and this is the only trancoding I have done on them?
No, the video stream is copied, not modified or transcoded, by ty1to2.

bndt12
08-25-2005, 03:12 PM
No, the video stream is copied, not modified or transcoded, by ty1to2.

Thanks, that is what I thought.

bcc
09-06-2005, 08:11 PM
Here's a new version of ty1to2. Changes:
Fixes 2GB file size issue
Fix assertion failure in master.c
cosmetic gcc 4.0 changes

bcc
09-07-2005, 02:50 PM
Here's a new version of ty1to2. Changes:
Fixes 2GB file size issue
Fix assertion failure in master.c
cosmetic gcc 4.0 changes
First round was buggy - hopefully it's fully ironed out with the new 1.4 image, above.

bcc
09-26-2005, 02:21 PM
In case anyone still cares :)
Fixed a bug where there would occasionally be an audio glitch from runt sized (4 byte) tivo audio records (about 1 per half hour of recording). Reworked the XML processing code so that we update more of the XML: <StreamFileSize>, <Duration>, <StartTime>, <StopTime>, <StopDate>. Added ability to strip out XDS data from recording (Extended Data Service) - on by default.

DarkHelmet
01-15-2006, 09:25 PM
What I originally did, to elaborate on my earlier posts in this thread:
* extract tmf from tivo
* extract part??.ty files and showing.xml files from .tmf
* cat part??.ty > part.ty
* ran my hacked version of ty1to2 on part.ty to create part2.ty and new RecordingPart records alongside in part2.ty.txt
* ran a series of perl scripts to manually replace the RecordingPart records in the showing.xml file with the new ones
* split -b512m to convert the part2.ty file back into part??.ty files
* create new tmf from fixed showing.xml file and new part??.ty files
* insert new .tmf file

This obviously was a lot of work. But it did deal with the case when the number of parts increased from 3 to 4 or the like.

Now, my wife's s1 dtivo started flaking out, so I couldn't put off revisiting this for much longer.

This time, instead of using my hacked ty1to2 that generated the updated RecordingPart info and all, I did this:

* extract .ty+ from tivo
* ran new xml-aware ty1to2 on old.ty to create new.ty
* insert new.ty to tivo.

The problem was that I ended up with 'now playing' list showing 0 minutes (partial), even though the progress bar when playing the show was correct.

I read some more and saw that mfs_ftp handles the RecordingPart offsets differently for .ty+ and .tmf files. ie: it ignores them or something.

So, I grabbed ty+2tmf and converted new.ty to new.tmf. I inserted new.tmf and voila! it worked with the correct sizes on the nowplaying list. The same information in .tmf works while in .ty+ format it has the cosmetic glitch.

Anyway, this seems to have drastically cut down the manual hacking and data crunching of multi-gigabyte files. My computer is much happier with me.

So now, I'm doing this:
* extract old.ty+
* ty1to2 -i old.ty+ -o new.ty+
* ty+2tmf new.ty+ new.tmf
* insert new.tmf

And so far I haven't been able to find any problems at all, cosmetic or otherwise.

BTW: I use ChrisEd's p1.tcl plugin for mfs_ftp 1.2.9p that correctly restores folders, and attributes etc. I strongly recommend this plugin for a complete tivo migration! http://www.dealdatabase.com/forum/showthread.php?p=196853#8

Now all I have to figure out is how to delete the failed insertion attempts (due to forgetting to comment out the event lines in mfs_ftp.tcl) - hopefully they'll go away after a reboot.

bcc
01-15-2006, 10:36 PM
The problem was that I ended up with 'now playing' list showing 0 minutes (partial), even though the progress bar when playing the show was correct.That's strange I do not get that behavior on my 3.1.5 tivo with 1.2.9p mfs_ftp and the newer mfs_utils. The now playing entries show the expected length. Since you mention folders, you probably have a newer tivo model and there is something more that needs to be set that hasn't been made clear to me. Sounds like the kind of problems darrin75 was having with newer tivo code, never resolved as far as I know.
I read some more and saw that mfs_ftp handles the RecordingPart offsets differently for .ty+ and .tmf files. ie: it ignores them or something.In the mfs_ftp tcl code, there is just 1 reference to RecordingPart, and that is in the AddPart subroutine that is used by both the .ty+ and .tmf handling. AddPart is not even adding Begin/End objects to the RecordingPart object, so I don't know what offsets could be of issue.

DarkHelmet
01-16-2006, 01:59 AM
That's strange I do not get that behavior on my 3.1.5 tivo with 1.2.9p mfs_ftp and the newer mfs_utils. The now playing entries show the expected length. Since you mention folders, you probably have a newer tivo model and there is something more that needs to be set that hasn't been made clear to me. Sounds like the kind of problems darrin75 was having with newer tivo code, never resolved as far as I know.In the mfs_ftp tcl code, there is just 1 reference to RecordingPart, and that is in the AddPart subroutine that is used by both the .ty+ and .tmf handling. AddPart is not even adding Begin/End objects to the RecordingPart object, so I don't know what offsets could be of issue.

Bizzare indeed. I was thinking of your comment a page or two back about set_rec_csos only being called from parse_tmf, and not parse_ty+. set_rec_csos sets the Begin and End records. I remember that I found that if I delete only the those from the showing.xml file, and insert the modified one instead, then it too shows a 0 length program in the program detail page.

Anyway, it really doesn't matter because it is cosmetic. I thought I'd mention it in case anybody else ran into the problem and the solution I found that worked for me would at least be written down. Not that enough people actually read or search anyway, but at least I tried. :)

I previously (many months ago) inserted into 4.01b on a RID tivo. Today I'm inserting into 6.2 on the same hardware. I don't recall ever attempting to insert onto 3.1.0 or 3.1.1.

I wonder if the 4.01+ builds depend on the Begin/End records somehow? Whatever the case, set_rec_csos seems to matter somehow on my boxes. Fortunately it is a trivial (and quick) workaround to run ty+2tmf.pl and insert that instead.

I'm also really pleased that your current ty1to2 works out of the box on my 64 bit machines. :) It crashes if I turn on debug due to printf format mismatches, but I can live with that.

bcc
01-30-2006, 05:09 PM
Bizzare indeed. I was thinking of your comment a page or two back about set_rec_csos only being called from parse_tmf, and not parse_ty+. set_rec_csos sets the Begin and End records. I remember that I found that if I delete only the those from the showing.xml file, and insert the modified one instead, then it too shows a 0 length program in the program detail page.Oops, forgot about that detail (hey, it was almost a year ago). Well, I'm pretty confused now. Lacking a tivo that runs anything newer than 3.1.5, I can't troubleshoot this myself. If this is fixable just thru XML, and someone can figure out exactly how newer boxes differ in their XML requirements, then I'd be happy to fix.
Otherwise it's simply looking like mfs_ftp's .ty+ insertion support has issues. I have found other issues with .ty+ insertion over here:
http://www.dealdatabase.com/forum/showpost.php?p=247282&postcount=1023

I'm also really pleased that your current ty1to2 works out of the box on my 64 bit machines. :) It crashes if I turn on debug due to printf format mismatches, but I can live with that.Glad it's working smoothly for you. I did just fix some corner case issues with EOF processing of .ty files. Maybe if someone figures out the XML details it'd be worth cutting a new release.

happypup6969
04-16-2007, 09:30 PM
Hi bcc,
Don't know if you're monitoring this board any more, or if you're still maintaining ty1to2, but I'm having a problem that I haven't seen described. I'm trying to convert at TY+ file (ty1to2v1.5), about 480MB (an episode of South Park) from an S1 DTivo running 3.5 (downloaded with patched mfs_ftp 1.2.9b). I get the following output, which seems to barf on every 64 chunks. It aborts at the end with no other error messages. Any ideas what I'm doing wrong? Thanks in advance for your help.

C:\Download Files\TiVo\mfs_ftp\ty1to2-1.5>ty1to2 -i sp-dat.ty+ -o sp-dat1.ty+
Warning: Skipping chunk 2 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity, at chunk 64
Discontinuity set to 54882553:04 (e0834979726c975c)
Warning: Skipping chunk 66 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity, at chunk 128
Discontinuity set to 47518300:03 (9ec6b6873eb6ad23)
Warning: Skipping chunk 130 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity, at chunk 192
Discontinuity set to 43377374:43 (5fb93b8c9ccf6685)
Warning: Skipping chunk 194 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity, at chunk 256
Discontinuity set to 49920586:00 (a0c6ca8254cce7af)
Warning: Skipping chunk 258 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity, at chunk 320
Discontinuity set to 46677081:24 (62789b7e6d77497a)
Warning: Skipping chunk 322 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity, at chunk 384
Discontinuity set to 48084911:12 (9f3f7e518b3ea3c0)
Warning: Skipping chunk 386 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity, at chunk 448
Discontinuity set to 42849754:17 (5f48c3783fdff595)
Warning: Skipping chunk 450 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity, at chunk 512
Discontinuity set to 50448205:39 (a137428bb461c965)
Warning: Skipping chunk 514 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity, at chunk 576
Discontinuity set to 44353741:03 (60895b8ca3ed6e31)
Warning: Skipping chunk 578 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity, at chunk 640
Discontinuity set to 47441192:56 (9eb646d1616f4a7a)
Warning: Skipping chunk 642 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity, at chunk 704
Discontinuity set to 43454778:48 (5fc9bb773fc0dd06)
Warning: Skipping chunk 706 due to timestamp discontinuity
Warning: Attempting resync due to repeated time discontinuity, at chunk 768
Discontinuity set to 47591087:02 (9ed63a7dad7bd741)

...aborts...

bcc
04-16-2007, 10:35 PM
The program is finding discontinuities with *all* your chunks and reporting on it every 64. I bet your source stream is scrambled.

happypup6969
04-16-2007, 11:46 PM
Doh, what a putz... I patched tivoapp to disable encryption, but apparently I haven't recorded much since I did. I must be downloading files from before I installed the patch. I downloaded a file that I just recorded and voila, worked like a champ! Converted and inserted up to my HD10-250 and played (with audio) with no problems. Wow, thanks for the great app, and thanks for the quick response.

samtheman
07-02-2007, 10:02 PM
So, I grabbed ty+2tmf and converted new.ty to new.tmf. I inserted new.tmf and voila! it worked with the correct sizes on the nowplaying list. The same information in .tmf works while in .ty+ format it has the cosmetic glitch.

Anyway, this seems to have drastically cut down the manual hacking and data crunching of multi-gigabyte files. My computer is much happier with me.

So now, I'm doing this:
* extract old.ty+
* ty1to2 -i old.ty+ -o new.ty+
* ty+2tmf new.ty+ new.tmf
* insert new.tmf


Hi, I am trying to move about 500hrs of recordings from a S1 to S2 machine, but have not been successful in getting the audio part to work. Wanted to try DarkHelmet's method but cannot locate the ty+2tmf files, a search turns up nothing and if you search for just ty2tmf you get the old programs that don't work on the ty+ files. Can anyone point me to the files or send me a copy via a pm.

Thanks.

jerrymc
07-02-2007, 10:55 PM
I had some trouble finding this myself, but finally found it. Search for "tmf conversion" in thread titles only and then it's on the second page of thread "Ty---> Tmf Conversion Discussion". Searching for ty+2tmf doesn't hit because of the "+" character I think.

The link is http://www.dealdatabase.com/forum/showthread.php?t=44588&highlight=conversion&page=2 (here)

-Jerry

bangheader
10-18-2007, 12:30 PM
Appreciate the development of this ...
Wish I had the time to do this kind of development ...

Here is my small contribution:
Put this .bat file in your "SendTo" folder, (I named it 122.bat) which will allow you to right click on an S1 ty, and "SendTo" the bat file to do the conversion to S2.

-----------cut------------
ty1to2 -i %1 -o %1.ty
-----------cut------------

It adds a .ty extension, so if you don't rename, it will end up with a ".ty.ty". I just shorten the name, and leave off the extension. When you insert it the full name is restored anyway.

Thanks Again


Here's a new version of ty1to2. Changes:
Fixes 2GB file size issue
Fix assertion failure in master.c
cosmetic gcc 4.0 changes

bcc
02-11-2008, 06:01 AM
Per request, here's a new build that adds support for s1 standalone systems. I only have 1 test stream to test against, so hopefully this is right. I didn't even realize s1 standalone recordings required a conversion step to play back on newer tivos.

(The xml-win-lib is for windows users only, and is the same as what I've posted previously.)

ciper
02-13-2008, 04:15 PM
If you like I can provide S1 streams. Tell me what to record and its yours.

bcc
02-14-2008, 02:16 PM
Turns out the case I was looking at with s1 sa recordings was just user error (audio was unexpectedly recorded in mono). From what I've seen s1 sa audio streams don't need to be run through ty1to2 before inserting onto an s2. Technically speaking, this is because the audio streams already have the necessary PES header that ty1to2 otherwise adds.
In any case, the newer code is cleaned up a bit and ty1to2 will now correctly process s1 sa recordings (even when unnecessary :)

I don't think I need any more samples, thanks.