tytompg: OSX (mac) specific support issues [Archive] - DealDatabase Forum - Deals, Freebies, and TiVo & DirecTivo Hacking

mr_zorg

02-19-2007, 12:12 AM

Show me how to do it with apple's darwin 8.0.1 build environment, and then I probably could do it. On the other hand, I've found no benefit from building 64 bit linux versions over 32 bit - the 32 bit version runs fine on 64 bit machines and I've noticed no performance differences either.
I just wanted to back up to the original poster that a 64 bit binary probably wouldn't make a difference. The demux/remux process is typically IO bound, not CPU bound. It's not like it's transcoding... Even on my 32 bit machine, it only sits at about 30% CPU utilization, while IO is maxed out.

Jetstream

02-19-2007, 05:09 PM

mr_zorg

02-19-2007, 05:33 PM

I am swithcing over to Mac. I've learned how to extract and convert the .tmf files to ty and then to mpeg on a pc.

So far I'm successful at extracting the .tmf files to my Mac.

What program is used on a Mac to use tmf2ty and tytompg?

Do Macs have the equivilant of a command prompt?
Welcome in to the fold. Yes, the Mac has a *nix style command prompt running the bash shell. It is called Terminal and can be found in the /Applications/Utilities folder. This is what you'll use to run tytompg.

As far as I know, hdemux and tytompg only work with .ty or .ty+ files. If you want to extract with the metadata, you can use ty+ instead of tmf format. It's just a ty file with the xml appended at the end.

I haven't used any tools on the mac to convert tmf into ty, but you can do it youself at the command line. A tmf file is nothing more than a tar file with a non-standard extension. Use "tar xvf myfile.tmf" to extract it. You'll get a series of sequentially numbered ty files and an xml metadata file (sorry, I don't remember the exact name, but you should see what I mean). You can then combine those ty files back together by "cat"ing them together, like so: "cat part001.ty part002.ty part003.ty ... > myfile.ty" Viola, you now have a ty file from the tmf file.

Now that we have the awesome hdemux and tytompg natively for mac (thanks bcc!), there's really nothing you can't do. Also, be sure to check out the ever-so-handy MPEG Streamclip app (http://www.squared5.com/) for editing and converting to other formats (like iPod)...

Enjoy!

P.S. BCC, you mentioned you were looking at creating a gui? What would be totally killer (in my book) is if you and the MPEG Streamclip guys could work together and get native ty support in that app. That would blow my mind. ;)

Jetstream

02-20-2007, 11:43 AM

Thanks Mr. Zorg. Before posting my question I did learn how to use terminal to extract the .tmf from the HR10-250. But I didn't know Terminal was used like the command prompt in Windows. I thought it was for ftp and so on.

Thanks much!

Jetstream

02-20-2007, 01:07 PM

OK I've successfully converted the .tmf to .ty, and I downloaded and untarred the tytompg for Mac.

I get "command not found" when I input "tytompg -i NeilYoung.ty -o NeilYoung.mpg".

Can I assume this procedure is the same for both Windows and Mac?

In other words, I have the .ty file and the tytompg script in the Movies folder. I use terminal to navigate to the Movies folder and execute this script.

drez

02-20-2007, 01:50 PM

you need a ./ before the command to so bash knows the program is in the same folder...

so try

./tytompg -i NeilYoung.ty -o NeilYoung.mpg

bcc

02-20-2007, 02:25 PM

you need a ./ before the command to so bash knows the program is in the same folder...

so try

./tytompg -i NeilYoung.ty -o NeilYoung.mpg
BTW, simply ./tytompg NeilYoung.ty
is now sufficient syntax.

Jetstream

02-20-2007, 02:43 PM

Thanks so much guys, working with the HR10-250 files was my biggest concern about switching to Mac.

mr_zorg

02-20-2007, 03:05 PM

But I didn't know Terminal was used like the command prompt in Windows.
Ack, don't say that! It's oh so much more... Take a look at some bash shell programming tutorial to see just how much you can do with it. :)

mr_zorg

02-20-2007, 03:11 PM

Ack, don't say that! It's oh so much more... Take a look at some bash shell programming tutorial to see just how much you can do with it. :)
Just by way of totally overkill example, try this command line, replacing YOUR_TY_HERE with the name of your ty file. It will extract out the resolution of the file. Of course, tytompg tells you that too, but this is just an example of some of the more sophisticated stuff you can do with a bash script:
eval "`xxd -s 10485760 -l 1048576 \"YOUR_TY_HERE\" | grep \"0000 01b3 .... ....\" | sed -e \"s/.*0000 01b3 $..$$..$ $..$...*/let \\\"x = (0x\\1 << 4) + ((0x\\2 \\& 0xf0) \\>\\> 4)\\\"; let \\\"y = ((0x\\2 \\& 0xf) \\<\\< 8) \\+ 0x\\3\\\"; echo \\\"\\\${x}x\\\${y}\\\"/\" | head -n 1`"

:eek:

bcc

02-20-2007, 03:49 PM

P.S. BCC, you mentioned you were looking at creating a gui? What would be totally killer (in my book) is if you and the MPEG Streamclip guys could work together and get native ty support in that app. That would blow my mind. ;)I don't have much bandwidth to help others on something like that, but I did design tytompg so that it should hook into a GUI pretty easily. GUI developers can pipe the output from tytompg into their programs, all they have to do is agree to the attribution-based license.
I have no need for editing commercials, so I haven't really used mpeg streamclip, but from my limited testing, it seems inferior to videoredo under windows.

mr_zorg

02-20-2007, 04:37 PM

I have no need for editing commercials, so I haven't really used mpeg streamclip, but from my limited testing, it seems inferior to videoredo under windows.
That could be, I haven't tried videoredo since they don't have a Mac version. :(

transeau

10-04-2007, 06:41 PM

-bash: /usr/bin/tytompg: cannot execute binary file

Running a Mac Pro w/8gb on 10.5

no chance of a build for 10.5?

bcc

10-05-2007, 08:00 PM

Is there a Power PC version of this for MacOS X?

Could run it on a pc but would rather do it on a Mac.

ThanksNo, nobody could tell me how to make one under darwin with Apple's build tools. See here http://www.dealdatabase.com/forum/showpost.php?p=275855&postcount=300, and subsequent posts in that thread (and this one).

bcc

10-11-2007, 01:39 PM

-bash: /usr/bin/tytompg: cannot execute binary file

Running a Mac Pro w/8gb on 10.5

no chance of a build for 10.5?Can't mac pros with 10.5 run mac x86 executables?% file mac-x86/tytompg
mac-x86/tytompg: Mach-O executable i386
%

mr_zorg

10-11-2007, 04:50 PM

Can't mac pros with 10.5 run mac x86 executables?
The OS X compatibility layer only works in one direction. An Intel machine can run a PPC binary, but not the other way around.

EDIT: Duh. He said Mac Pro, which is Intel... So, maybe I don't know what's going on. If I may be permitted to speculate some more, perhaps something about the download/unzip process it became un-executable? Try running a chmod +x on it...

bcc

10-12-2007, 04:41 AM

The OS X compatibility layer only works in one direction. An Intel machine can run a PPC binary, but not the other way around.

EDIT: Duh. He said Mac Pro, which is Intel... So, maybe I don't know what's going on. If I may be permitted to speculate some more, perhaps something about the download/unzip process it became un-executable? Try running a chmod +x on it...The zip file has execute bit information for unix, so if the file was unpacked without the execute bit I think there is something wrong with the user's unpacking procedure% zipinfo tytompg-0.19.zip
Archive: tytompg-0.19.zip 272215 bytes 15 files
drwxr-xr-x 2.3 unx 0 bx stor 11-Jun-07 14:21 windows-x86/
-rwxr--r-- 2.3 unx 110592 bx defN 11-Jun-07 15:02 windows-x86/hdemux.exe
-rwxr--r-- 2.3 unx 135168 bx defN 11-Jun-07 15:02 windows-x86/tytompg.exe
drwxr-xr-x 2.3 unx 0 bx stor 11-Jun-07 14:15 linux-x86/
-rwxr-xr-x 2.3 unx 52588 bx defN 11-Jun-07 14:58 linux-x86/tytompg
-rwxr-xr-x 2.3 unx 28516 bx defN 11-Jun-07 14:58 linux-x86/hdemux
drwxr-xr-x 2.3 unx 0 bx stor 11-Jun-07 14:14 linux-x86-64/
-rwxr-xr-x 2.3 unx 53088 bx defN 11-Jun-07 14:58 linux-x86-64/tytompg
-rwxr-xr-x 2.3 unx 31984 bx defN 11-Jun-07 14:58 linux-x86-64/hdemux
drwxr-xr-x 2.3 unx 0 bx stor 11-Jun-07 14:45 mac-x86/
-rwxr-xr-x 2.3 unx 67312 bx defN 11-Jun-07 15:02 mac-x86/tytompg
-rwxr-xr-x 2.3 unx 38596 bx defN 11-Jun-07 15:02 mac-x86/hdemux
-rw-r--r-- 2.3 unx 3247 tx defN 11-Jun-07 14:46 tytompg.txt
-rw-rw-r-- 2.3 unx 4544 tx defN 11-Jun-07 14:55 hdemux.txt
-rw-r--r-- 2.3 unx 2031 tx defN 11-Jun-07 14:56 LICENSE.txt
15 files, 527666 bytes uncompressed, 270101 bytes compressed: 48.8%
%

mr_zorg

10-12-2007, 05:04 PM

The zip file has execute bit information for unix, so if the file was unpacked without the execute bit I think there is something wrong with the user's unpacking procedure.
Agreed... If that's it. :)

ForrestB

10-13-2007, 06:19 PM

I suggest waiting until OSX 10.5 is actually released

danpritts

10-15-2007, 03:20 AM

No, nobody could tell me how to make one under darwin with Apple's build tools. See here http://www.dealdatabase.com/forum/showpost.php?p=275855&postcount=300, and subsequent posts in that thread (and this one).

This works for me:

gcc -isysroot /Developer/SDKs/MacOSX10.4u.sdk -arch i386 -arch ppc boof.c

Presumably the -isysroot piece is the necessary magic.

This is mentioned in passing on this page:
http://developer.apple.com/documentation/Porting/Conceptual/PortingUnix/compiling/chapter_4_section_3.html#//apple_ref/doc/uid/TP40002850-BAJCFEBA

but you really have to dig for it. Note that the resulting binary will be 10.4 only.

bcc

10-15-2007, 04:42 AM

This works for me:

gcc -isysroot /Developer/SDKs/MacOSX10.4u.sdk -arch i386 -arch ppc boof.c

Presumably the -isysroot piece is the necessary magic.

This is mentioned in passing on this page:
http://developer.apple.com/documentation/Porting/Conceptual/PortingUnix/compiling/chapter_4_section_3.html#//apple_ref/doc/uid/TP40002850-BAJCFEBA

but you really have to dig for it.Already been over that web page a few times and saw the reference to the seemingly unpublished SDK. That is what this post was about:
http://www.dealdatabase.com/forum/showpost.php?p=276110&postcount=315
If you know where the SDK library is freely published that'll work with Apple's published darwin tools, that'd be great, otherwise we're still going in circles. My 8.0.1 system install includes no /Developer/SDKs directory.
Note that the resulting binary will be 10.4 only.So apple is switching to a non-backwards compatible executable file format in 10.5?

danpritts

10-26-2007, 11:46 AM

Already been over that web page a few times and saw the reference to the seemingly unpublished SDK. That is what this post was about:
http://www.dealdatabase.com/forum/showpost.php?p=276110&postcount=315
If you know where the SDK library is freely published that'll work with Apple's published darwin tools, that'd be great, otherwise we're still going in circles. My 8.0.1 system install includes no /Developer/SDKs directory.
So apple is switching to a non-backwards compatible executable file format in 10.5?

Sorry, I missed the whole thread, I had just read the single post you had linked to before.

You can download XCode from here:

http://developer.apple.com/tools/download/

You will need to join the Apple Developer Connection to do the download but it is free to join.

It is a huge download, nearly a gig. However, it includes the stuff you are looking for, in the subdirectory:

MacOSX10.4.Universal.pkg

This is apple's installer package format, which probably doesn't have a click-installer in darwin, but the actual files are in a pax archive that you should be able to extract directly.

If you would like, I can provide you with this directory out of band so you do not have to download the entire gi-normous compiler kit. email DANNO me AT directly UMICH if EDU you want me to do so. This piece is about 60 megs.

I've never tried darwin so i cannot say for sure that this will work but it's worth a shot.

Regarding whether apple is switching to a new binary format in 10.5, I don't know, but they did from 10.3->10.4. I'd be shocked if they broke backward compatibility with old binaries, though. My warning about "10.4 only" was because a binary you produce using my instructions will not work on 10.3. THere's some rigamarole you can go through to produce such a thing but I wouldn't bother.

Note that I have no vested interest in this myself, I used linux to do this when i did it, and I no longer own a tivo (although thinking about buying an HD). But I'm happy to help if i can.

bcc

10-26-2007, 03:55 PM

Sorry, I missed the whole thread, I had just read the single post you had linked to before.

You can download XCode from here:

http://developer.apple.com/tools/download/
Thanks, I had surmised that it's the xcode package that provides the missing pieces. So I downloaded the 900M .dmg file, and yes the instructions that say just click on it are of no help with opendarwin. I manged to mount the .dmg with disktool (thru vmware mounted as if it were an .iso). The files showed up in /Volumes/Xcode Tools/, but none of them were readable (scrambled garbage results even with the .pdf). I could not unpack the .pax archives either. Perhaps vmware can't connect .dmg files this way.

I also tried encapsulating the .dmg in a .iso, connecting that .iso under vmware, then on darwin:
mount_cd9660 /dev/disk1s0 /mnt1
/usr/libexec/vndevice attach /dev/vn0 /mnt1/XCODE_2_.DMG
but then
mount -t hfs /dev/vn0 /mnt3
fails with a permission denied error.

Note that I have no vested interest in this myself, I used linux to do this when i did it, and I no longer own a tivo (although thinking about buying an HD). But I'm happy to help if i can.Thankfully none of this is necessary for s3 tivos, as my s3tots is distributed with source (no re-multiplexing required). So everyone can build it themselves.

mr_zorg

10-29-2007, 04:38 PM

It appears that when I upgraded to 10.5 Leopard TyToMpg stopped working. Anyone else have the same problem?
There was a report of this earlier in this thread. I haven't yet tried it... BCC doesn't have an actual Mac, but he was compiling on Open Darwin. It's possible something changed/broke in 10.5, but until the Open Darwin version of 10.5 comes out I doubt he'll be able to do anything about it. Once I get Leopard (soon) installed I'll play with it myself and see if there's anything to do about it in the mean time.

bcc

10-29-2007, 04:59 PM

There was a report of this earlier in this thread. I haven't yet tried it... BCC doesn't have an actual Mac, but he was compiling on Open Darwin. It's possible something changed/broke in 10.5, but until the Open Darwin version of 10.5 comes out I doubt he'll be able to do anything about it. Once I get Leopard (soon) installed I'll play with it myself and see if there's anything to do about it in the mean time.An actual error message might help. I see that the opendarwin project no longer exists, is there some replacement in the works for 10.5? Otherwise it seems like it's no longer feasible, by design, for non-mac owners to support macs. (The good news is you could simply upgrade to an s3 tivo and use s3tots, for which I've provided source).

I compiled tytompg with the default cc, which was gcc-3.3. I could have compiled with gcc-4.0 if that would make any difference.

Oh, and as for xcode, I put the .dmg on a usb hard drive, and that didn't work either - same problem with the scrambled files. Looks like the .dmg is encrypted or compressed and the algorithm is proprietary and not supported by any public tools.

cognac

10-30-2007, 08:03 AM

I'm using a MacBook Pro 2.16 GHZ Intel Core Duo with 10.5, when I run tytompg I get:

-bash: /bin/tytompg: cannot execute binary file

It had been working fine under 10.4

Console log:

10/30/07 3:55:39 AM login[7026] DEAD_PROCESS: 7026 ttys002

bcc

10-30-2007, 07:27 PM

-bash: /bin/tytompg: cannot execute binary file

It had been working fine under 10.4

Console log:

10/30/07 3:55:39 AM login[7026] DEAD_PROCESS: 7026 ttys002I'm thinking 10.5 busted binary compatibility with anything built using the opendarwin version of the build tools. You could perhaps do some more digging, maybeotool -h tytompg vs otool output from a current executable would shed some light.

I tried building a relocatable version of tytompg (with ld -r & strip) so that you can link it yourself. Unfortunately the opendarwin version of the build tools seem unable to strip private symbols without making the .o unusable.

Basically it doesn't look like it's practical for me to support tytompg on macs anymore.

mr_zorg

10-30-2007, 09:33 PM

I'm thinking 10.5 busted binary compatibility with anything built using the opendarwin version of the build tools. You could perhaps do some more digging, maybeotool -h tytompg vs otool output from a current executable would shed some light.
...
Basically it doesn't look like it's practical for me to support tytompg on macs anymore.
I'll do some more technical digging once my copy of 10.5 arrives and will report back here on what I find. It would be a shame if it broke tytompg...

P.S. The S3 upgrade path is tempting, if only I could find an IPTV -> CableCard converter so I could switch from D* to my local IPTV provider instead of Comcast... :)

jasch

10-30-2007, 10:09 PM

I can confirm, that the same executable works with 10.4, and doesn't woth 10.5

bcc

10-30-2007, 10:20 PM

I'll do some more technical digging once my copy of 10.5 arrives and will report back here on what I find. It would be a shame if it broke tytompg...If it's something simple I'm missing, I'll fix. (Doesn't look like it). If I have to do something like construct my own darwin 10.5 system from scratch, no thanks. P.S. The S3 upgrade path is tempting, if only I could find an IPTV -> CableCard converter so I could switch from D* to my local IPTV provider instead of Comcast... :)D* orphaned tivo a long time ago so that loyalty doesn't make much sense to me.

bcc

10-30-2007, 10:29 PM

I can confirm, that the same executable works with 10.4, and doesn't woth 10.5Do tell if you know *why* 10.5 doesn't think it's a valid executable anymore.

mr_zorg

10-31-2007, 12:25 AM

If it's something simple I'm missing, I'll fix. (Doesn't look like it). If I have to do something like construct my own darwin 10.5 system from scratch, no thanks. D* orphaned tivo a long time ago so that loyalty doesn't make much sense to me.
Understood regarding your efforts for tytompg... It may be something simple, though I'm kinda with you in thinking it may not be.

As for loyalty to D*, it's not that as it is a distaste for over-compressed cable. Leaving Dish and Surewest IPTV as my only options. Dish doesn't do TiVo, and Surewest IPTV doesn't do HD TiVo (no cablecard support). So I'm kinda stuck with what I've got (my HR10-250) until I figure it out. I'm all ears for potential solutions... I'd love to see TiVo come out with an IPTV unit, but that doesn't seem likely anytime soon.

bcc

10-31-2007, 01:31 AM

As for loyalty to D*, it's not that as it is a distaste for over-compressed cable. Don't know why you'd expect comcast to be more compressed than d*. My experience has been the opposite, and others have posted their tsreader results showing similar results. s3tots reports overall average bitrate so I could show specific examples.

comcast offers locals in HD, no extra package to buy, with d* locals were SD only, and cost extra... With d*, 1080i was almost always 1280x1080, with comcast, usually 1920x1080... Now I'm way off topic.

mr_zorg

10-31-2007, 04:44 PM

comcast offers locals in HD, no extra package to buy, with d* locals were SD only, and cost extra... With d*, 1080i was almost always 1280x1080, with comcast, usually 1920x1080... Now I'm way off topic.
Hmm. Well maybe that's not such a bad way to go then. I'd been told by some people who seemed to be in the know that Comcast was overcompressed, perhaps that's not the case. Anyhoo, yes, we've strayed a bit haven't we... OK, no more from me on this subject. :)

mr_zorg

11-02-2007, 03:52 AM

Do tell if you know *why* 10.5 doesn't think it's a valid executable anymore.
OK, I've figured some things out. I'm not a gcc expert though, so I don't know quite what to make of it. I've attached a zip file with some files that accompany my descriptions if you'd like to take a look.

1) Sure enough, tytompg does not run under 10.5. Nor does my older copy of hdemux I compiled from when the source was still available.

2) Given that I still had that hdemux source, I decided to try recompiling it under 10.5. That works, so my examples focus on that.

3) There are differences in the otool -h output between my hdemux compiled on 10.4 and the one compiled in 10.5 (see zip).

4) The hdemux make process outputs a few more warnings/errors under 10.5 than it does under 10.4 (see zip). Perhaps these are the source of the problems, but the 10.5 linker compensates for it somehow.

5) The resulting binaries are also in the zip.

It'd be great if you, or anyone else, can make something out of this -- great.

bcc

11-02-2007, 03:54 PM

Looks like the "cannot execute binary file" error is a misnomer and the problem is likely with the shared libraries. With otool -L I can see that the 2 versions use different versions of libSystem. Apparently these libraries are neither backwards nor forwards compatible. I assume this is an opendarwin thing, as there are only a few programs, like python and this: http://netwinsite.com/cgi/dnewsweb.cgi?cmd=article&group=netwin.surgemail&item=22295&utag= one that report this problem with 10.5.

I cannot even get opendarwin to statically link anything as a workaround. Looks like I need a new 10.5 compatible build environment for this to possibly work.% otool -L hdemux.10.4
hdemux.10.4:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.0.0)
% otool -L hdemux.10.5
hdemux.10.5:
/usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 111.0.0)
% ./hdemux.10.5
sh: ./hdemux.10.5: Bad executable (or shared library)
%

mr_zorg

11-02-2007, 04:51 PM

mr_zorg

11-03-2007, 01:53 AM

Though you did give me an idea. It might hose my system, but I will try copying the old libSystem.B.dylib to Leopard to see what that does. :)
Uh, yeah. That was NOT a good idea. The moment I renamed the old one all of my CLI and many of the GUI items stopped working. Doh. Thank goodness for the Mac's firewire target disk mode... I was able to rename it back to normal that way. :p

bcc

11-03-2007, 06:02 AM

Uh, yeah. That was NOT a good idea. The moment I renamed the old one all of my CLI and many of the GUI items stopped working. Doh. Thank goodness for the Mac's firewire target disk mode... I was able to rename it back to normal that way. :pGotta understand how osx handles shared library versioning (I assume osx *can*).
Maybe one can set DYLIB_LIBRARY_PATH to a compat directory, put the old 10.4 libraries there and then invoke 10.4 programs. Not sure if this works for the libsystem library.

mr_zorg

11-03-2007, 07:00 PM

Gotta understand how osx handles shared library versioning (I assume osx *can*).
Maybe one can set DYLIB_LIBRARY_PATH to a compat directory, put the old 10.4 libraries there and then invoke 10.4 programs. Not sure if this works for the libsystem library.
I assume it can too... I'll do some more research and see what I can figure out. This kinda thing is new territory for me, but not out of my realm of ability. :)

Thanks for your patience and assistance in catering to use Mac users... We do appreciate it!

mr_zorg

11-08-2007, 12:42 AM

Hmm. I'm not sure the library versioning is it. Turns out versioning on OS X is very simple. You just have multiple versions of the library names like so: <library_name>.<version_num>.dylib and the loader takes care of the rest. I tried that, still got the same issue. I even tried taking a full copy of /usr/lib from a 10.4 system into a test directory on 10.5, then used DYLD_LIBRARY_PATH to set that directory. Using DYLD_PRINT_LIBRARIES=TRUE I could see that it was in fact loading from that directory when running things like "ls", etc. Most of the system commands failed with a "Bus Error" (whatever that means) in this configuration. I tried running both tytompg and hdemux and still had the same problem. Most interestingly, is it did not print out any libraries at all -- it immediately failed. It's like it's not even *trying* to run it. Furthermore, I have other binaries I compiled under 10.4 that run just fine on 10.5. Go figure. It's certainly an interesting challenge, I'll keep trying to figure it out as time permits.

bcc

11-08-2007, 01:03 AM

tytompg's dependencies on the C library are simple - largely limited to a few basic posix functions like read() and lseek() that should easily resolve into system calls. It should be easy for someone familiar with the osx dynamic loader to determine what is going wrong here. Recommend you just ask on one of those osx enthusiast forums such as insanelymac.

jcabello

11-11-2007, 11:46 PM

Could someone replicate what mr_zorg did in post #36 but for the tytompg.

Thanks a lot, help appreciated...

mr_zorg

11-12-2007, 12:12 AM

Could someone replicate what mr_zorg did in post #36 but for the tytompg.
Unfortunately, no. That's the point of this endeavor. Tytompg has proprietary code in it which is not open source (as do later versions of hdemux). We're trying to figure out what the difference is so BCC can hopefully compile us a version that works on 10.5. He's not running a Mac, but has graciously compiled us a binary using Open Darwin -- for which there is no 10.5 equivalent (and it looks like there won't be).

bcc

11-12-2007, 12:42 AM

Since I don't have a MAC, here's the best I've been able to figure out.
I compiled a version of tytompg on a vmware+hackintosh system. 10.4 release. Enclosed. I note that the binary doesn't run on my opendarwin system so that seems promising :)
It was a real pain to do this much as the hackintosh system often hangs, but in the interest of solving the mystery...

PS: I also unpacked xcode 2.4.1 under opendarwin but the MacOSX10.4u doesn't appear to be sufficient to make fat binaries. (I get a linker error about crt1.o being missing.) Perhaps someone knows which packages are necessary to install manually and any tricks necessary for opendarwin.

mr_zorg

11-12-2007, 02:27 AM

danpritts

11-15-2007, 01:28 AM

PS: I also unpacked xcode 2.4.1 under opendarwin but the MacOSX10.4u doesn't appear to be sufficient to make fat binaries. (I get a linker error about crt1.o being missing.) Perhaps someone knows which packages are necessary to install manually and any tricks necessary for opendarwin.

FWIW on my mac it's in the SDK

~@mx-5% locate crt1.o
/Developer/SDKs/MacOSX10.3.9.sdk/usr/lib/crt1.o
/Developer/SDKs/MacOSX10.3.9.sdk/usr/lib/gcrt1.o
/Developer/SDKs/MacOSX10.4u.sdk/usr/lib/crt1.o
/Developer/SDKs/MacOSX10.4u.sdk/usr/lib/gcrt1.o
/usr/lib/crt1.o
/usr/lib/gcrt1.o

surmise that you somehow got the SDK extracted from the dmg file.

If opendarwin has "hdiutil" it can apparently convert the dmg to an iso.

but again i don't care myself, i now have a series3, woo hoo. It felt good to send the comcrap dvr back with the cable guy.

bcc

11-15-2007, 07:39 PM

FWIW on my mac it's in the SDK

~@mx-5% locate crt1.o
/Developer/SDKs/MacOSX10.3.9.sdk/usr/lib/crt1.o
/Developer/SDKs/MacOSX10.3.9.sdk/usr/lib/gcrt1.o
/Developer/SDKs/MacOSX10.4u.sdk/usr/lib/crt1.o
/Developer/SDKs/MacOSX10.4u.sdk/usr/lib/gcrt1.o
/usr/lib/crt1.o
/usr/lib/gcrt1.oYes my opendarwin system has the 10.4u version of those thanks to the xcode distribution. Nevertheless I was getting an error that crt1.o was missing. Unfortunately I don't remember the exact command I typed at the time. Here's a more specific example of how xcode still fails for me:[darwin:~/test] bcc% cat test2.c

#include <stdio.h>
#include <assert.h>
main()
{
printf("hello world!\n");
assert(0);
}
[darwin:~/test] bcc% make

cc -c "-isysroot /Developer/SDKs/MacOSX10.4u.sdk" -arch ppc -arch i386 test2.c
cc "-Wl,-syslibroot,/Developer/SDKs/MacOSX10.4u.sdk" -arch ppc -arch i386 -o test2 test2.o
ld: for architecture ppc
ld: warning /usr/lib/gcc/darwin/3.3/crt2.o cputype (7, architecture i386) does not match cputype (18) for specified -arch flag: ppc (file not loaded)
ld: warning /usr/lib/gcc/darwin/3.3/libgcc.a archive's cputype (7, architecture i386) does not match cputype (18) for specified -arch flag: ppc (can't load from it)
ld: Undefined symbols:
___eprintf
[darwin:~/test] The above links if I leave out the ppc architecture, or if I leave off the assert. But tytompg uses assert so I need that to work as well. Even when I get hello world to link, the above warnings prevail and the resulting executable does not run on opendarwin (crashes with illegal instruction). Does not seem like xcode provides a usable development environment under opendarwin.

surmise that you somehow got the SDK extracted from the dmg file.

If opendarwin has "hdiutil" it can apparently convert the dmg to an iso.I was only able to unpack the .dmg file under vmware+hackintosh. I then manually unpacked the MacOSX10.4.Universal.pkg. Who knows if manually unpacking is kosher or not anyways. I have to wonder whether this package depends upon one of the others and or whether I need to have performed any of the actions from Resources/postflight, or the like.
but again i don't care myself, i now have a series3, woo hoo. It felt good to send the comcrap dvr back with the cable guy.It felt good to mothball my hr10-250s as well.

bcc

11-15-2007, 07:45 PM

If opendarwin has "hdiutil" it can apparently convert the dmg to an iso.Forgot to mention - no it doesn't have it. I think the xcode .dmg files are encrypted, magiciso can't extract from them either.

bcc

11-15-2007, 07:50 PM

Sorry, still no dice on 10.5... Same issue. Thanks for trying! i find it funny how you keep saying there's only so much you can do, but yet you keep trying. Admit it, you like the challenge! :)
Yes, I like challenges, but...
Figuring out in more detail how to work around OSX's dynamic library versioning problems is something I'd rather defer to others. It should be easier & more rewarding for them. Most *nix distributions learned years ago to number their shared libraries to provide some basic versioning, apparently OSX or at least opendarwin hasn't even reached that point yet.
I'm sorely tempted to just pack up my Intel Mac Mini and give it to you for all your efforts! But then I'd have to get a new one. :(Thanks, if OSX didn't require buying overpriced proprietary hardware I'd get it myself.

bjdraw

11-17-2007, 03:05 PM

Thanks for all your efforts on this.

I'd be willing to allow you ssh or vnc access to my intel mac running 10.5 if that'd help.

bcc

11-18-2007, 03:13 PM

Thanks for all your efforts on this.

I'd be willing to allow you ssh or vnc access to my intel mac running 10.5 if that'd help.Thanks for the offer, but no thanks. I don't want to upload the full source.

bjdraw

11-18-2007, 04:07 PM

No problem, I understand.
Even with OSX's Secure Empty Trash feature, you never know.

mr_zorg

11-23-2007, 03:01 AM

Hey, I finally found something. The key difference between the hdemux.10.4 and hdemux.10.5 (previously uploaded), aside from older code, is revealed with otool -l <file>:

The non-working version has this:

Load command 9
cmd LC_UNIXTHREAD
cmdsize 80
flavor 4294967295 (unknown)
count 16
state:
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0000001f 00000000 00001e58 00000017 0000001f 0000001f 00000000 00000000

While the older source I compiled myself has this:

Load command 8
cmd LC_UNIXTHREAD
cmdsize 80
flavor i386_THREAD_STATE
count i386_THREAD_STATE_COUNT
eax 0x00000000 ebx 0x00000000 ecx 0x00000000 edx 0x00000000
edi 0x00000000 esi 0x00000000 ebp 0x00000000 esp 0x00000000
ss 0x00000000 eflags 0x00000000 eip 0x00002270 cs 0x00000000
ds 0x00000000 es 0x00000000 fs 0x00000000 gs 0x00000000

I don't know (yet) what command line flags affect this key load command, but I did find a work around. It probably isn't the best or safest thing to do, but it did work on both the official hdemux and tytompg releases and got them running under 10.5. Go to the command line and use these commands (substituting <file> for the binary you want to "fix". DO NOT RUN THIS ON ANYTHING OTHER THAN HDEMUX AND TYTOMPG, I CAN'T SAY HOW IT WILL WORK ON ANYTHING ELSE!!!

cp <file> <file>.backup

xxd <file> | sed -e "s/ffff ffff 1000/0100 0000 1000/" | xxd -r - <file>

chmod +x <file>

mr_zorg

11-23-2007, 09:08 PM

It has been suggested to me that obscuring the thread state is often used as an anti-disassembler technique used to try and protect some binaries. I don't know if that's the case here, but that certainly could explain why 10.5 rejects this binary. Leopard introduces a number of new security enhancements, including things like library address randomization, so it seems likely they view this form of obfuscation as something potentially used by malware to cloak itself and therefore just refuse to run such binaries. Makes sense. (Please note, I am NOT saying these binaries are malware, I know they're not.)

I've also been told that my proposed solution is "safe" to do. If you encounter this with other binaries and wish to try my technique, first verify the situation with otool -l <file>, then hex edit the *first* FFFF FFFF to 0100 0000. I want to reiterate my sample command does not do this in a generic fashion, it is specific to these two binaries (hdemux and tytompg).

bradn

11-26-2007, 08:30 PM

DO NOT RUN THIS ON ANYTHING OTHER THAN HDEMUX AND TYTOMPG, I CAN'T SAY HOW IT WILL WORK ON ANYTHING ELSE!!!

cp <file> <file>.backup

xxd <file> | sed -e "s/ffff ffff 1000/0100 0000 1000/" | xxd -r - <file>

chmod +x <file>

EDIT: confirmed both tytompg and hdemux now work on a 10.5.1 equipped mbpro.

THANKS MR ZORG!! amazing.

EDIT: added the binaries I patched per MR ZORG's instructions.

I'll contrib $25 to BCC's intel mac mini fund if he agrees to keep mac support going and add a ppc version (i run a mac mini ppc as a media server). anyone else join me?

bcc

11-27-2007, 04:19 AM

Nice find, I don't know what that otool difference really means. But...

You indicated you are having trouble with forwards compatibility for the hdemux binary you yourself compiled under 10.4. It does not make sense that you would then be considering that the root cause is some sort of code obfuscation on anybody else's part. Any obfuscation would have to be happening automagically by the 10.4 build tools. You've seen the source and the build commands and had complete control over it - you can see there was no obfuscation involved; the build process is quite simple in fact.

The otool -l output you show for the 2 versions of hdemux you built does not match what I see under opendarwin. That further suggests to me pretty fundamental build tool/shared library compatibility issues with Apple's build tools.

For the record I've never performed any code obfuscation in the tools I've provided to users. No phoning home, backdoors, timebombs, worms, viruses, or anything of the sort. That's more than can be said for software which folks are actually purchasing from vendors including tivo :)

Here's what otool -l reports for me on your hdemux binaries under opendarwin:% otool -l hdemux.10.4|tail
nhints 32
Load command 9
cmd LC_UNIXTHREAD
cmdsize 80
flavor i386_THREAD_STATE
count i386_THREAD_STATE_COUNT
eax 0x00000000 ebx 0x00000000 ecx 0x00000000 edx 0x00000000
edi 0x00000000 esi 0x00000000 ebp 0x00000000 esp 0x00000000
ss 0x0000001f eflags 0x00000000 eip 0x00001e58 cs 0x00000017
ds 0x0000001f es 0x0000001f fs 0x00000000 gs 0x00000000
% otool -l hdemux.10.5 | tail -22
Load command 8
cmd LC_UNIXTHREAD
cmdsize 80
flavor 1 (unknown)
count 16
state:
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00002270 00000000 00000000 00000000 00000000 00000000
Load command 9
cmd LC_LOAD_DYLIB
cmdsize 52
name /usr/lib/libgcc_s.1.dylib (offset 24)
time stamp 2 Wed Dec 31 16:00:02 1969
current version 1.0.0
compatibility version 1.0.0
Load command 10
cmd LC_LOAD_DYLIB
cmdsize 52
name /usr/lib/libSystem.B.dylib (offset 24)
time stamp 2 Wed Dec 31 16:00:02 1969
current version 111.0.0
compatibility version 1.0.0
% Since you have all the pieces: 10.4, 10.5 and source with which you can reproduce the problem, maybe you can figure out how to get the 10.4 vintage compilers to produce a binary that doesn't require binary patching. I'd be happy to pick up a fix to use under opendarwin.

mr_zorg

11-29-2007, 02:03 AM

You indicated you are having trouble with forwards compatibility for the hdemux binary you yourself compiled under 10.4.

I was mistaken in that, I thought the 10.4 binary I was comparing to was one I had compiled, but it was, in fact 0.25 that you provided. The one I compiled under 10.5 was 0.13. Alas, I no longer have 10.4 installed with which to re-test and see what that version does.

It does not make sense that you would then be considering that the root cause is some sort of code obfuscation on anybody else's part. ...snip... For the record I've never performed any code obfuscation in the tools I've provided to users. No phoning home, backdoors, timebombs, worms, viruses, or anything of the sort.

Didn't mean to imply, in any way shape or form that you did. And code obfuscation, at any rate, isn't per-se a bad thing. Given your prior experiences with people misappropriating your code, I wouldn't blame you if you *were* trying to obfuscate it. :)

Since you have all the pieces: 10.4, 10.5 and source with which you can reproduce the problem, maybe you can figure out how to get the 10.4 vintage compilers to produce a binary that doesn't require binary patching. I'd be happy to pick up a fix to use under opendarwin.

Sadly, as mentioned above, my original comparison was invalid, and I don't have 10.4 anymore. So I think my experimenting is over. At least we have a "workaround". Just out of curiosity, what version of GCC are you using under OpenDarwin? If it's 3.3, does it have 4.01 and can you try that?

P.S. I just pulled the trigger on an S3 box, so this will be a moot point for me soon anyway. :o

rla

12-08-2007, 11:10 AM

bcc

12-08-2007, 02:34 PM

Argh.. I applied the patch above, ran tytompg on an HD ty file on a mac mini 10.5.1. Ran without incident.

$ ./Converters/tytompg -y test.ty
tytompg: Copyright (c) 2004-2007 B.C. <bcc24x7@gmail.com>
Multiplexer version 0.19, Demuxer version 0.26
Source is test.ty
Video frame size is 1280x1088 - high definition. Frame rate 29.970030
Video bit rate is 65000000 bits/sec., 65000 Kbit/sec.
AC3 audio at 48kHz, 384 kbps, 1536 bytes/sync frame
Done!
Stream audio: 253166 frames (8101 seconds aprox.)
Stream video: 208767 frames, 485563 fields (6965 seconds aprox.)

output has no audio. The audio is in the ty file though. The above output suggests that ac3 audio is in the .mpg as well. Perhaps you don't have the right codecs for quicktime to play ac3 encoded in an mpeg. Lacking a sample, that'd be my best guess. You could confirm whether or not this is the case by testing whether audio playback from a .vob file that includes ac3 works. You could also verify with utilities such as vobedit whether or not the audio stream is missing. And you could use the -V and or -A options to tytompg to omit the video or audio stream, respectively, from the .mpg, to narrow things down.

shox

12-19-2007, 04:07 AM

I have been following this thread as I also have a keen interest in a binary for 10.5. What is the final state as of now? Do we take one of the previous binaries and hack it up to work? Or can you post your working 10.5 bin?

Thanks in advance!

bradn

12-19-2007, 11:27 AM

Mr Zorg's patch instructions are pretty clear, but I added the tytompg and hdemux binaries I patched to post 31. Please post and let everyone know it worked.

http://www.dealdatabase.com/forum/showpost.php?p=291126 (http://www.dealdatabase.com/forum/showpost.php?p=291126&postcount=31)

EDIT: attachment now includes 10.5.1 compatible v21 tytompg and v27 hdemux per mr_zorg post 64 on this thread.

mr_zorg

12-27-2007, 03:02 PM

I just tried my technique on the new tytompg 0.21 binaries bcc just released, it works for them too.

bcc

12-27-2007, 04:11 PM

I just tried my technique on the new tytompg 0.21 binaries bcc just released, it works for them too.Maybe you could fix the opendarwin build tools?

bradn

12-27-2007, 04:41 PM

I just tried my technique on the new tytompg 0.21 binaries bcc just released, it works for them too.

i updated the attachment on the earlier thread post listed below to include this new version with edited headers per mr_zorg's instructions

http://www.dealdatabase.com/forum/showpost.php?p=291126

mr_zorg

12-27-2007, 11:09 PM

Maybe you could fix the opendarwin build tools?
I'll see if I can block out some time to try and see what's going on there... Might be a little bit before I can get to it though. It would be nice to fully understand what's going on. Can you PM me the specifics of which version of what you're using and where you got them from? Thanks!

mr_zorg

12-30-2007, 07:23 PM

I've made a little progress. Turns out that the darwin iso you're using (8.0.1, which, sadly is the latest available) is from right around the Intel transition and has a known bug in it that uses the wrong flavor code for LC_UNIXTHREAD. That's what's causing this behavior. Looks like 10.4 had some workaround code for the bug to accept the bad flavor, but in 10.5 they must have removed it, figuring people should be using a newer xcode. The good news is that based on my research, my earlier fix for manually changing the thread state does indeed seem a benign fix. You could always do it yourself before distributing the binaries...

To fix this at compile time, I see three possible solutions:

1) Somehow locate or roll your own distro for the latest Darwin version. Doesn't sound like fun to me.

2) Somehow upgrade xcode to 2.4 (source here: http://www.opensource.apple.com/darwinsource/DevToolsAug2006/). I tried downloading and compiling from source, but the Makefiles seem to rely on "strip" which isn't in 8.0.1, and I've been unable to bootstrap the upgrade so far...

3) Somehow tell the linker to use i386_NEW_THREAD_STATE (1) for the LC_UNIXTHREAD load command instead of i386_THREAD_STATE (-1). I haven't yet been able to figure out how to do this.

Does anyone out there know how to do one of these three?

bcc

12-30-2007, 07:34 PM

2) Somehow upgrade xcode to 2.4 (source here: http://www.opensource.apple.com/darwinsource/DevToolsAug2006/). I tried downloading and compiling from source, but the Makefiles seem to rely on "strip" which isn't in 8.0.1, and I've been unable to bootstrap the upgrade so far...Hmm? It shows up in /usr/bin/strip after you install the opendarwin 8.0.1 binaries.% what /usr/bin/strip
/usr/bin/strip
PROGRAM:cctools_misc PROJECT:cctools-576 DEVELOPER:root BUILT:Wed Mar 30 19:03:47 PST 2005
%

mr_zorg

12-30-2007, 10:00 PM

Hmm? It shows up in /usr/bin/strip after you install the opendarwin 8.0.1 binaries.
Huh. Maybe something in the build process blew it away and failed to replace it... I'll try #2 again and see if I can figure it out.

mr_zorg

01-02-2008, 02:57 AM

bcc

01-02-2008, 03:36 AM

I'm just not having much luck. Looking around, it seems only Apple knows how to compile this stuff... I also tried manually extracting the Xcode 2.4 binaries out of the dmg files, but they don't run right on that older version of Darwin. :(Yes, now you're getting to the root of the problem I think. Apple needs to support developers for developers to be able to support Apple. Instead it seems they've just done the bare minimum to avoid violating the GPL.

Now, I don't know which package you were trying to build, but if the only hang-up was with strip, I should think you could just comment out its use of strip. It's doubtful that the package really needs to strip something to make the build succeed.

Do you know which package is supposed to be setting the LC_UNIXTHREAD bits? Apple's published gcc changes don't seem to.

mr_zorg

01-02-2008, 04:39 PM

Yes, now you're getting to the root of the problem I think. Apple needs to support developers for developers to be able to support Apple. Instead it seems they've just done the bare minimum to avoid violating the GPL.

Now, I don't know which package you were trying to build, but if the only hang-up was with strip, I should think you could just comment out its use of strip. It's doubtful that the package really needs to strip something to make the build succeed.

Do you know which package is supposed to be setting the LC_UNIXTHREAD bits? Apple's published gcc changes don't seem to.
No, it's more than just strip, I got past that. One makes use of a utility called seg_hack, which I can't find. Another one simply spits out a "#error Please port this!" message. Others don't even have a Makefile. :rolleyes:

I don't know enough about the mechanics of compiling and linking to know where LC_UNIXTHREAD gets set, and haven't figured that out yet. The one thing I *can* say, is that this is a "simple" case of the i386_THREAD_STATE having a bad value of -1 instead of the correct (positive) 1. Thus, my fix of editing the binaries is technically correct, and should achieve the same results -- even though it's less elegant. :)

I'll post back if I figure anything else out, but I'm not hopeful...