PDA

View Full Version : Mips disassembler v0.1



tmesis
09-15-2003, 06:46 PM
This is a post-processor for objdump output from MIPS ELF binaries to make it readable by mere mortals. I used it on tivoapp.

# tmesis MIPS linux disassembler 2003-09-15, v.0.1 #
# 1. Finds procedure entry points. #
# 2. Resolves string references. #
# 3. Resolves syscalls. #
# 4. Resolves calls to dynamically linked prcoedures. #
# 5. Resolves addresses obtained from GOT. #

To run it:
-- Make sure that unistd.h is present in the same directory (needed for resolving syscalls).
-- Make sure that mips-TiVo-linux-objdump is present on your path.

Run

./mips.dasm.pl <executable>

The results will be in <executable>.S. Enjoy.

ronnythunder
09-15-2003, 08:47 PM
Possible reference to string: "TivoApp: Ack! I don't support program %s"cute... :)

ronny

alldeadhomiez
09-15-2003, 10:09 PM
Nice work, tmesis.

So, now that we are examining tivoapp, it would be helpful to start looking for "entry points" that help us understand what various parts of the code do. Obviously we already have the strings that are built into tivoapp, but there are numerous other "external" events we can observe which will help explain different areas of the tivoapp codebase.

Here are my initial thoughts:

Strings in MFS. Tivo stores a lot of informational and error messages in MFS; they are indexed by number. We can devise a way to correlate each of these strings with the code that pulls it up if we can find the function(s) that are used for this purpose.
Incoming and outgoing EventSwitcher events. This will point us to tivoapp's handlers for these events, and potentially show us where its own events are generated.
Incoming and outgoing oslink messages. These will describe tivoapp's interaction with the tuner subsystem on combo boxes. This will allow us to understand the filtering of APG data, and possibly help us solve issues related to locals reception and "28% problems." This is something I am working on from the other end.
The tivosh interface. This is certain to be really nasty but it might be the easiest way to start exploring Tivo's C MFS interface.


What else are we interested in learning?

mrblack51
09-16-2003, 03:42 PM
please keep this topic in the clear by not discussing dssapp hacks. as with the btl on the s1, the file is strictly dtv related, and as such, is not an appropriate topic for this board. tivoapp is fair game, as are many of the other binaries on the unit.

Fletch319
09-16-2003, 04:24 PM
is this all done on the tivo itself or on a linux workstation with the executable in question just copied over?

Also.. if it's not run on the tivo itself but it's a perl script can it be run in a windows environment?

ronnythunder
09-16-2003, 04:43 PM
fletch, it's run on a linux box, not on the tivo. you also have to have the objdump program built for mips on the machine you're running on, so you'd have to do cygwin or something for that.

ronny

cali
09-16-2003, 05:25 PM
Originally posted by alldeadhomiez



Incoming and outgoing oslink messages. These will describe tivoapp's interaction with the tuner subsystem on combo boxes. This will allow us to understand the filtering of APG data, and possibly help us solve issues related to locals reception and "28% problems." This is something I am working on from the other end.


Looking forward to seeing what you guys have from this end.:)

tmesis
09-16-2003, 09:25 PM
Here is a new version 0.2.

Improvements:
-- Can use symbolic function names from a file <executable name>.proc. A small file tivoapp.proc is included for tivoapp 3.1.
-- Recognizes a common function call pattern: lui/addu/lw/jalr.
-- Works on executables containing debug info.

Enjoy.

alldeadhomiez
09-17-2003, 11:10 AM
Here's a thought to improve readability:

The gcc MIPS backend has an annoying habit of taking small blocks of code (such as "then" and "else" clauses) and moving them to the end of a function. At the end of the block it jumps back into the middle of the function. This often causes the flow to skip all over the place and makes things hard to read.

If we could develop an algorithm that rearranges these blocks and inverts the conditional branches, we could make it a bit easier to follow the program flow. It would be nice if each function ended with jr $ra instead of a dozen tiny blocks of code that the compiler deemed unlikely to execute.

Anyone want to give it a shot?

tmesis
09-18-2003, 06:44 PM
This is a bugfix release:

1. Fixed infinite loop when a symbol table is missing.
2. Fixed LALJ pattern handling when the first instruction is "lui t9,0x1".
3. Major refactoring of code to improve readability: 4 modes (default, syscall, LAJ, LALJ).

Future plans:
-- Create crossreference for procedure calls, jumps.
-- Visually separate jump branches, procedure returns.
-- Output html hyperlinks to click-and-jump to an address; will not be useful for huge disassemblies, such as tivoapp though.

tmesis
09-19-2003, 01:52 AM
This is a helper utility that scans the annotated dump from MIPS disassembler and prints out the tree of procedure calls:



0x00D05464
0x00D51BB8
0x00D51844
0x00D4FE98
0x00D7BCD0
0x00DA1420, size_t strlen(const char *s);
0x00DA1630
0x00DAA538
0x00DB0D50
0x00DB0D28

etc.

Run it as follows:

calltree.pl <annotated dump file> <start address>

MuscleNerd
09-19-2003, 05:10 PM
Nifty scripts. If even just a few people put a coordinated effort into this, you could come up with quite an impressive tivoapp.proc list.

tmesis
09-19-2003, 06:16 PM
How would you suggest going about mapping procedures?

I can find C runtime library functions by matching tivoapp code to my test executable compiled with symbols, but that's about it.

Beyond that, it's just trying to understand the logic and guessing what each procedure does, isn't it?

MuscleNerd
09-19-2003, 06:21 PM
Originally posted by tmesis
How would you suggest going about mapping procedures? ... it's just trying to understand the logic and guessing what each procedure does, isn't it?
Yep, pretty much. Obviously the strings help out there...especially those error message strings (the ones that identify the procedure names). Also, code spatial locality often holds; related procedures are often found back-to-back, just before or after the constructor and destructor code for each class.

tmesis
09-25-2003, 12:33 AM
No major changes in mips.dasm.pl.

-- New script xrefs.pl, finds all procedures that call each given procedure and puts that info in its header. Run like this:

./xrefs.pl <file created by mips.dasm.pl> <output file name>

-- More than 1000 procedure addresses in tivoapp.proc (tivoapp 3.1).

Enjoy!

opticalcarrier
09-25-2003, 03:37 PM
Originally posted by tmesis
-- Make sure that mips-TiVo-linux-objdump is present on your path.


can i run this on a normal bsd machine? or can we only run this on the series2 system itself? If i can run it on my home bsd system, where do i get the mips-TiVo-linux-objdump file from?

Thanks,
OC

David Bought
09-25-2003, 03:43 PM
Originally posted by opticalcarrier
can i run this on a normal bsd machine? or can we only run this on the series2 system itself? If i can run it on my home bsd system, where do i get the mips-TiVo-linux-objdump file from?

Thanks,
OC

tivoutils.sourceforge.net

You might be able to use the precompiled toolchain if you are running the Linux compatibility module on an x86. I haven't tried because BSD is dying anyway.

rpl
09-27-2003, 01:57 AM
To run it:
-- Make sure that unistd.h is present in the same directory (needed for resolving syscalls).
-- Make sure that mips-TiVo-linux-objdump is present on your path.

Run

./mips.dasm.pl <executable>
I loaded up RedHat 8. Put a copy of mips-TiVo-linux-objdump file from tivoutils.sourceforge.net in the same directory as mips.dasm.pl and when I try to run the disassembler I get as far as Reading constants ...

What else do I need on my linux box to run this disassembler?

tmesis
09-27-2003, 12:36 PM
That's where it scans the dump file for .rodata section and then tries to load it from the executable (this is where all strings are located).

Could you please look at the file <your executable>.dump and send me the first 60 lines or so. I suspect it cannot find .rodata section because objdump failed.

tmesis
09-27-2003, 12:39 PM
And by the way, I don't think objdump will run alone by itself. Just extract the whole tivo utils archive into /usr/local/mips-tivo directory and set the path properly.

rpl
09-27-2003, 10:40 PM
You were correct. I extracted the entire archive and set the PATH. Took care of the problem. Thanks

ronnythunder
10-15-2003, 12:32 AM
sorry to bring up an old thread, but can anyone comment on how the tivoapp.proc file was created? i'm trying to find the "oncatchup" 30 second skip defaults-to-on patch location for 3.1.1b. i've used the mips.dasm and xrefs, but neither seems to do the trick.

thanks in advance,

ronny

tmesis
10-15-2003, 02:31 AM
can anyone comment on how the tivoapp.proc file was created?

Since I did it, I can assure you that it is very incomplete. I used two methods:

1. Looking at strings included in executable.
2. Compiling a test executable calling some standard C library functions with debug info, and then trying to find matching functions in tivoapp disassembly.


i'm trying to find the "oncatchup" 30 second skip defaults-to-on patch location for 3.1.1b. i've used the mips.dasm and xrefs, but neither seems to do the trick.

One approach would be to find the procedure that is patched in the old tivoapp (3.1) disassembly and then to look for similar procedure in the new tivoapp (3.1.1b) disassembly.

drnull
05-07-2004, 11:45 AM
Just thought I would bump this thread back up, because it's a great thread for those of us wanting to get some idea of what tivoapp is doing. Also, though, wanted to post a very slight change to mips.dasm.pl. I wanted comments in the .S code at the tivoapp patch locations, so I modified the script to differentiate between a prototype line like <addr> <short name> <prototype> and a patch location line like <addr> <patch>. Just added 4 or 5 lines of code, no biggie. Plus, I thought I would post the 3.1.1c version of the tivoapp.proc file. I've made a few additions based on scanning error messages, plus, all entries that were in the 3.1 .proc file have been updated to reflect their locations in 3.1.1c

<stupid file type checking....WHY?>

drnull
05-07-2004, 01:44 PM
One more ever so slight change. If you have used mips.dasm.pl, you've noticed that there are some instructions that objdump did not disassemble, but instead left as 0x5462fff9 or whatever. That's because objdump wasn't interpreting the object type correctly for some reason. If you pass a -mmips:4000 in when mips.dasm.pl calls objdump, then all commands will be disassembled correctly.

We are working on a mips R5432 (pretty sure it supports up through MIPS IV), but objdump defaults to just MIPS I. Some instructions (like "branch ... likely" and such) weren't added until MIPS II, so they stay un-disassembled (assembled?).

Course, then again, maybe most everybody who uses this stuff knows this already. I didn't till today.

drnull
05-19-2004, 06:06 PM
So I've used mips.dasm.pl quite a bit now, and really enjoy it. I've made a few changes, and I hope tmesis won't mind me releasing a new version of his perl script.

Basically, v0.4 of mips.dasm recognized strings/values that were referred to by the lw-addiu pattern:
lw reg,offset(gp)
addiu reg,reg,offset

But there were a few other patterns that are used. lw-addiu-lw shows up a few times.
lw reg1,offset(gp)
addiu reg1,reg1,offset
lw reg2,offset(reg1) ## where reg2 could be the same as reg1

Also, he recognized jumps that were done using the lui-addu-lw-t9 pattern:
lui t9,upper
addu t9,t9,gp
lw t9,offset(t9)
jalr t9

But strings/values can be loaded using the same pattern. So I added support for these and the successive load by generalizing the lui-addu-lw pattern.
lui reg1,upper
addu reg1,reg1,gp
lw reg2,offset(reg1) ## where reg2 could be the same as reg1
## and optionally, memory indirect mode
lw reg3,offset(reg2) ## where reg3 could be the same as reg2

I didn't let it go any further than that, although it probably does a few times. Plus, there were some cases where the last lw in the lui-addu-lw-lw pattern is followed by a few more lw's from different offsets of reg2. I didn't catch those either.

I have changed it so that the "Possible reference to string..." lines now are preceded by the type of reference:
LA = lw-addiu (22588 uses)
LAL = lw-addiu-lw (93 uses)
LUAL = lui-addu-lw (444 uses)
LUALL = lui-addu-lw-lw (717 uses)

So, as you can see, these are not the most popular methods, but they ARE used, and I found some interesting strings that way.

Also, the comments at the end of the line that say what the value of the register is (they looked like # reg = value) have been changed. They now read
# MODE reg = [address] = value
where MODE is one of (LA,LAL,LUAL,LUALL) and the [address] part is left out if in LA mode (obviously).

You can use the below readmem.c program passing in the pid of myworld and the [address] value to see if the value is still the same at runtime as it is in the file definition.


readmem.c

I also made a really simple c program that uses ptrace to latch onto tivoapp so that I could then take a peek at any memory location. This is useful for when you know what the memory location a variable will point to is, but you don't have a clue what is there based on the source, or you just want to verify that the value is the same (useful for addresses that are not in the constant section).

http://www.secretmango.com/Jimb/Whitepapers/ptrace/ptrace.html was extremely helpful in developing readmem.c

tmesis
05-19-2004, 06:29 PM
So I've used mips.dasm.pl quite a bit now, and really enjoy it. I've made a few changes, and I hope tmesis won't mind me releasing a new version of his perl script.


I don't mind at all; I am happy to see somebody continuing to work on it.

drnull
05-19-2004, 06:38 PM
Do you (anybody else can chime in here too) still use this or have you switched over to IDA. I found that your script produced much more useful asm code than trying to use IDA. Granted, branching and xrefs are really, really pretty in IDA... :-)

mrblack51
05-19-2004, 11:03 PM
Do you (anybody else can chime in here too) still use this or have you switched over to IDA. I found that your script produced much more useful asm code than trying to use IDA. Granted, branching and xrefs are really, really pretty in IDA... :-)

I tend to use both...IDA for exploring, the dump output for locating things and such. IDA has the ability to use IDC scripts, which could be used to take the annotated dump, and merge things like string refs into the IDA setup as comments or whatnot. That would be a very useful tool.

drnull
05-31-2004, 01:12 AM
Hmm, what's new in this version? Oh, lots of fun stuff. :-)

Positives:
- branch analyzing: find branch points so we know where code sections can be entered, thus decomposing the code into cohesive blocks.
- entry points are marked with a ] after the address so you can easily see where blocks start
- code prettification. I'm not fond of the tabs in the assembly code. So I lined things up to my liking.
- branch table loading. Explained below.
- SwSystem resource loading. Explained below.
- register tracking. Explained below.

Negatives:
It now takes a _REALLY_ long time to run. Like 15 minutes. If you like the new functionality, great. If not, comment out the track_registers call and it will run, oh, 10x faster? :-) Seriously. Maybe I just coded that very poorly. If somebody can speed it up, PLEASE share.

So, back to the positives.

- Branch Table Loading
I was often seeing code like the following:

sltiu v0,s5,9
beqzl v0,0x00401cd4
lw a0,164(s3)
sll v0,s5,0x2
lw v1,-32728(gp)
nop
addiu v1,v1,-9912
addu v1,v1,v0
lw v1,0(v1)
addu v1,v1,gp
jr v1

Which was annoying, because it's not a function call (hence jr instead of jalr) so it's some sort of branching. Also, there was this weird "possible string reference" annotated in there (the string was just junk characters). Well, after looking at it a bit, I found that the "string reference" was a reference to an array of DWords. Those DWords (when added to gp in the second to last statement above) were locations in the code near this jr. We index into the array using some register (in this case, s5, which is multiplied by 4 and added to the base of the array to get the jump location). So anyways, from this code, we know that s5 can go from 0-8, and we know where the array is, so I decided to print out the jump table listing like so:


# SBS Possible Branch Table:
# s5: Address (s5's value from the sltiu line)
# 0: 0x00401728
# 1: 0x0040191c
# 2: 0x0040194c
# 3: 0x0040199c
# 4: 0x004019ec
# 5: 0x00401a3c
# 6: 0x00401b08
# 7: 0x00401bcc
# 8: 0x00401c84

We recognize 338 of these branch tables.


- SwSystem resource loading.
I was interested in annotating the functions that use the resource id's so that I could get the resource strings into the disassembly also. So, to do that, I found the function that uses those resource id's, and using the tracked registers, I load the correct resource out of a .res file. For this to work, you need to include the def line for the getResource function in your .proc file. My line looks like
0x00a34954 getResource getResource(?, int resID)
for tivoapp311c.

So, you need a .res file for this. Generate it using a script (only slightly modified) that I found over on alt.org from embeem called resource.tcl. Run it on tivo, redirect to a .res file. Copy that file back to the mips.dasm.pl directory.
I'm including my tivoapp311c.res file. (obviously, this needs to be the same basename as your .proc file and your binary)

- Register tracking.
This uses the block of code idea that branch analyzing gave us. Basically, we can more or less keep track of where code is entered, so we can flush registers at that point. Otherwise, we can keep track of register values so that the constants that make up 25% of code (I've heard that somewheres, I think) can be kept track of. The primary motivation behind this was so I could do SwSystem resource loading, as the a1 register was loaded in two steps: lui followed (sometimes distantly) by a ori, and I got tired of searching for where it was set. Granted, we still miss some of these getresource calls, as they often happen right after a branch, but, hey, we pick some up! 1407 Resource references to be exact.

Don't think I'm missing anything... Hope you enjoy it. If it borks for you, let me know, maybe I forgot to include something.

mrblack51
05-31-2004, 01:18 AM
in a future release, you might go about seeing if you can recognize macro instructions, and then use those to better identify the use of constants. specific examples are an lui followed by an ori, which is equivilent to the macro instruction li with a 32 bit value. also, be aware that there are delay slots associated with mips, so some re-ordering can happen. specificaly, in the case of the li macro instruction, i have seen it in the following form quite often:

lui $x, 0xAAAA
<inst>
ori $x, $x, 0xBBBB

that would be equivilent to:

li $x 0xAAAABBB
<inst>

basically, there are some references that will be missed if that is overlooked

drnull
05-31-2004, 01:50 AM
in a future release, you might go about seeing if you can recognize macro instructions, and then use those to better identify the use of constants. specific examples are an lui followed by an ori, which is equivilent to the macro instruction li with a 32 bit value. also, be aware that there are delay slots associated with mips, so some re-ordering can happen.

I thought delay slots only applied after jumps & branches (with a very special case on branch-likely statements)? Since I am breaking the code down into branch-less segments with a single entry point, I shouldn't have to worry about delay slots.

Also, I am recognizing the lui followed by ori by the nature of what I am doing. I am tracking the register values, and basically running a brain-dead simulator over that branch-less segment of the code. So after the lui $x,0xAAAA, I set my value of $x to be 0xAAAA0000, then whatever happens next could modify that. Macro instructions were exactly what I had in mind as something this register tracking would help me, uh, track. :-)

drnull
05-31-2004, 01:55 AM
And I meant to say above, kudos to tmesis for the design of this script. Using a state machine for his pattern recognition has made it very easy for me to modify and add states. Great design!

mrblack51
05-31-2004, 03:51 AM
I thought delay slots only applied after jumps & branches (with a very special case on branch-likely statements)? Since I am breaking the code down into branch-less segments with a single entry point, I shouldn't have to worry about delay slots.

Also, I am recognizing the lui followed by ori by the nature of what I am doing. I am tracking the register values, and basically running a brain-dead simulator over that branch-less segment of the code. So after the lui $x,0xAAAA, I set my value of $x to be 0xAAAA0000, then whatever happens next could modify that. Macro instructions were exactly what I had in mind as something this register tracking would help me, uh, track. :-)

remember that instructions like load and store can have delays as well. because of that, all kinds of re-ordering can occur (from before, from after, from target). not sure how much of a delay cache misses can cause either...probably should look that up. in any event...thats a -O3 compiler for ya

ronnythunder
05-31-2004, 11:30 PM
just tried it on my "messin' with tivo" pc. it's a 100% text based machine, no graphics whatsoever, 128mb ram and 512mb of swap. when trying to do a 311c tivoapp, i get the following messages:
tmesis MIPS linux disassembler 2004-05-30 v0.4.2

Disassembling .../usr/local/mips-tivo/bin/objdump: tivoapp311c: no symbols
/usr/local/mips-tivo/bin/objdump: tivoapp311c: not a dynamic object

Reading constants ... 615936 bytes
Reading sections ...
a few minutes later, the process gets killed by the kernel because it's using all of the ram *and* swap space. how much ram/swap does it need? :)

ronny

drnull
06-01-2004, 12:32 AM
tmesis MIPS linux disassembler 2004-05-30 v0.4.2

Disassembling .../usr/local/mips-tivo/bin/objdump: tivoapp311c: no symbols
/usr/local/mips-tivo/bin/objdump: tivoapp311c: not a dynamic object

Reading constants ... 615936 bytes
Reading sections ...
a few minutes later, the process gets killed by the kernel because it's using all of the ram *and* swap space. how much ram/swap does it need? :)
Yeah, the not a dynamic object error is just because of the -T flag passed to objdump. It is looking for the dynamic symbol table, but it don't exist. Just a warning. No biggie.

The memory usage shouldn't be THAT bad. I use ~ 80 meg.

The problem is the no symbols error. I don't know why you're getting that. What does your objdump -v show? I show
GNU objdump 2.14.90.0.7 20031029

Maybe we're on different versions?

That would cause the error, though. I'm looking for the start of the message that says SYMBOL TABLE: no symbols to tell me that I'm at the end of the section.

Try changing line 138 from
while (!/^SYMBOL/o) {
to
while (!/^Disassembly/o) {

Should fix that problem. Maybe your objdump doesn't put in the "no symbols" sections for some reason?

ronnythunder
06-01-2004, 01:08 AM
mins is 2.13 (just 2.13, no subversions). i did try the change and it's running now. will let you know. thanks!

ronny

Xybyre
06-04-2004, 11:38 PM
I ran into the same problem (using objdump 2.13), which drnull's fix took care of.

One other problem I ran into was when I added the tivoapp.proc getResource() line. I found out after the lengthy disassembly that tivoapp.proc lines need to be delimited by tabs instead of spaces. Hopefully, this will save others some time.

I just wanted to give a big "Thanks!" to tmesis (for a great tool) and drnull (for making it even better). Those resource strings are soooo helpful!

---Xybyre

drnull
06-07-2004, 01:51 PM
Thanks to jeboo for pointing out that I had broken xrefs.pl. calltree.pl was also broken because of my tab->space conversions.

Here's two patches that should fix them.

Xybyre
06-17-2004, 05:55 PM
On my machine, the branch table entries are all 0xffffffff. After some investigation, I discovered that $_GP + $baddr was always 0xffffffff. It should have been 0x1xxxxyyyy. That is, it was using 32-bit integers for the math, and the numbers weren't wrapping around. Maybe it's just my machine, or Perl version (v5.8.0), but I fixed it by using BigInt objects.

I'm attaching my mods, in case anybody else has the same problem.

Xybyre
06-30-2004, 10:45 AM
I've written a tool to analyze the disassembly, much like IDA does. It took me a while to decide to release it, as I spent a lot of work on it, and I don't have the time to support it. However, as it has been so useful to me, I'm sure that some of you will find it invaluable for tracing TiVo programs.

Instead of reiterating the documentation, I'll just point you to it: http://www.xybyre.net/tivo. This is a Windows program.

Here's a screenshot:
http://www.xybyre.net/tivo/images/disasm.png

Enjoy,
Xybyre

NutKase
06-30-2004, 12:38 PM
I've written a tool to analyze the disassembly, much like IDA does.

Great, thanks. I'll have to set aside some playtime tonight.

NutKase

alldeadhomiez
06-30-2004, 01:52 PM
It took me a while to decide to release it, as I spent a lot of work on it, and I don't have the time to support it.

In that case, it would be a good idea to post the source code so that others can continue to fix bugs and maintain it instead of bothering you.

NutKase
08-11-2004, 08:43 PM
OK, I've gotten the mips.dasm.pl to work and have disassembled tivoapp.

When I load it into viewmips I see it processing then "Analysis Complete"

For some reason, I see no information in the window, or on any of the tabs.

...just a blank window.

NutKase

EDIT: I've gotten it working now using drnull's 04.2 update.

Now let's see what all this is telling me. ;)

NutKase
08-11-2004, 09:00 PM
In that case, it would be a good idea to post the source code so that others can continue to fix bugs and maintain it instead of bothering you.

I have to agree with ADH. It would be helpful to have the source. I'd like to add a File select option/menu so I can get away from typing long paths in Run commands.

Great tool by the looks of it. Nice work.

NutKase

Xybyre
08-16-2004, 12:31 PM
The reason I have not released the source code is because I wrote this on my work laptop, which uses IBM VisualAge C++ 3.5. The code won't compile with anything else because I use its class library. I doubt anybody else uses this compiler. However, I do still maintain it, and continue to add features and bug fixes (v1.1 is now available).

I have no problems with adding useful features, if requested. Bug reports are also welcome. I just didn't want to get flooded with requests.

It turns out there aren't as many TiVo reversers as I thought. ...in this forum, anyway ;)

bma
09-07-2004, 08:45 PM
I'm trying to run mips.dasm.pl against tivoapp from 4.0.1, but it keeps dying while in load_sections, saying 'Out of Memory!' I've got 512m physical ram and upped my disk swap to 5g, and upped shmmax, but it still keeps dying.
Any thoughts on how to get this to work? I'm using drnull's 0.4.2 version.

bma
09-07-2004, 08:48 PM
I'm trying to run mips.dasm.pl against tivoapp from 4.0.1, but it keeps dying while in load_sections, saying 'Out of Memory!' I've got 512m physical ram and upped my disk swap to 5g, and upped shmmax, but it still keeps dying.
Any thoughts on how to get this to work? I'm using drnull's 0.4.2 version.
I must be blind, just saw the suggestion to s/SYMBOL/Disassembly/. It's working now :)

Vegas
03-08-2005, 12:20 AM
used this a few times on tivoapp from 3.1.1c and 4.0.1b no problems.
Trying it now on tivoapp from 6.2 and it fails while writting the annotated dump. yet still works fine on the 4.0.1b tivoapp.

I've done some debugging and it looks like its calculating an offset beyond the end of the tivoapp file which causes the file read to fail. Not a perl programer so haven't been able to figure out why. The .S output looks fine up to the failure point.
Attached is the output file. I can put the tivoapp file up on ftp if it would help.

Below is the console output:

Dev101:/home/jowens/tivo/dasm # ./4.2mips.dasm.pl 6tivoapp
tmesis MIPS linux disassembler 2004-05-30 v0.4.2

Disassembling ...
Reading constants ... 767664 bytes
Reading sections ... Loaded 24 data sections
Reading GOT ... 54163 entries
Reading Resources ... 0 entries
Processing symbol table ... 304 entries
Loading syscall numbers ... 1163 entries
Loading procedure definitions ... 0 entries
Finding jump targets ...

Writting annotated dump ...
......Reading file 6tivoapp failed

Vegas

NutKase
03-08-2005, 02:12 AM
I've built a .proc file for 7.1. My question for now regards my tivoapp71a.S disassembly.



Dynamic Section:
NEEDED libhpkoss.so
NEEDED libtvstructures.so
NEEDED libhpkhl.so
NEEDED libtmk.so
NEEDED libtvutil.so
NEEDED libhpkll.so
NEEDED libutil.so.1
NEEDED libdl.so.2
NEEDED libpthread.so.0
NEEDED libm.so.6
NEEDED libc.so.6
INIT 0x410abc
FINI 0x174b174

<small snip>

Version definitions:
1 0x01 0x0b0d5800 tivoapp
2 0x00 0x0d696910 GLIBC_2.0

Version References:
required from libutil.so.1:
0x0d696910 0x00 14 GLIBC_2.0
required from libhpkhl.so:
0x0b0e3b73 0x00 15 HPK_1.0_WITH_EXTRA_SYMS
0x0d512450 0x00 13 HPK_1.0
required from libdl.so.2:
0x0d696910 0x00 10 GLIBC_2.0
0x0d696912 0x00 09 GLIBC_2.2


This is what I've gotten dumped at the beginning and I'm not sure if it's telling me that I 'still' need these files in my .proc files list or if 'these files needed' something during disassembly?

Hope that makes sense.


NutKase

7.1
03-08-2005, 12:25 PM
used this a few times on tivoapp from 3.1.1c and 4.0.1b no problems.
Trying it now on tivoapp from 6.2 and it fails while writting the annotated dump. yet still works fine on the 4.0.1b tivoapp.

I've done some debugging and it looks like its calculating an offset beyond the end of the tivoapp file which causes the file read to fail. Not a perl programer so haven't been able to figure out why. The .S output looks fine up to the failure point.The issue, as you observed, is that the perl script is trying to read beyond the end of the tivoapp file. I've seen this with 7.1 and some 5.X tivoapps too. I believe this is in code DrNull added. Possible solutions: Turn the die into a warn and return zero but continue. Check the file offset against the file size and don't try to read beyond the end of the file. Observe that not every VMA maps to a file offset. For example, the uninitialized data section (bss) is not stored in the elf file since it is "uninitialized" (really, initialized to zero during the startup code). Check the vma address against the section map, and check the flags for the corresponding section to see whether there is data stored in the file for that section.The attached patch (against 0.4.2) implements approach 3.

Vegas
03-08-2005, 03:11 PM
The issue, as you observed, is that the perl script is trying to read beyond the end of the tivoapp file. I've seen this with 7.1 and some 5.X tivoapps too. I believe this is in code DrNull added.

Thanks 7.1, I tracked it down to the added code and I think the subs handle_laj_mode and handle_lalj_mode. This is my first venture into disassembly so it would have taken me forever to figure out how to fix it.

To get around it I had done #1, replace die with printf and a warning along with $addr and $offset. This let the disassembly finish, but I received 6121 warning messages, so didn't think this was a very good approach.

I will try your patch tonight. Looking forward to digging into tivoapp. This should be fun.

Thanks

Vegas

AhoyMatey
03-08-2005, 08:29 PM
The attached patch (against 0.4.2) implements approach 3.That's pretty cool. I especially like the jump target indicators.

Vegas
03-09-2005, 11:14 AM
The attached patch (against 0.4.2) implements approach 3.

The patch fixed up the disassembly for 6.2 tivoapp. I now have dumps of 3.1.1c, 4.0.1b and 6.2 and am off to begin my learning curve for patching binaries... diving into the deep end.
If anyone else would like to dive in, there is some great info in this thread (http://www.dealdatabase.com/forum/showthread.php?t=38242&referrerid=5812). Learned more last night in 3 hours than I did from the $40 assembly book I bought.

Just a heads up to 7.1. When I ran your patched version against 3.1.1c-tivoapp it went into the infinet loop during reading sections. Didn't look into it, just went back and used the unpatched ver.

Vegas

7.1
03-09-2005, 11:40 AM
Just a heads up to 7.1. When I ran your patched version against 3.1.1c-tivoapp it went into the infinet loop during reading sections. Didn't look into it, just went back and used the unpatched ver.Hum. I suspect it is this change:
- while (!/^SYMBOL/o) {
+ while (!/^(Disassembly|DYNAMIC SYMBOL TABLE|SYMBOL TABLE)/o) {
that is at fault. This is supposed to correct the "seems to run forever and consume large amounts of memory" problem with 4.x and above tivoapps discussed earlier in the thread. I guess it's not properly finding the end of the section table. It might be that "SYMBOL TABLE" should be changed to just "SYMBOL". Or it could do more intelligent parsing of the section table lines and bail from the loop when it hits something it can't parse.

If you could send me the first few hundred lines of the .dump file, I could probably figure out what it should be looking for to terminate the section table scan.

alldeadhomiez
03-09-2005, 12:39 PM
Hum. I suspect it is this change:
- while (!/^SYMBOL/o) {
+ while (!/^(Disassembly|DYNAMIC SYMBOL TABLE|SYMBOL TABLE)/o) {


Might want to change that to something like:


while(defined($_) && !/(Disassembly|DYNAMIC SYMBOL TABLE|SYMBOL TABLE)/o) {

so that if objdump aborts for some reason or those markers are not present, it craps out (somewhat) gracefully instead of getting stuck in an infinite loop.

Vegas
03-09-2005, 06:35 PM
If you could send me the first few hundred lines of the .dump file, I could probably figure out what it should be looking for to terminate the section table scan.

Here is the dump of 3.1.1c sniped after the start of disassembly.

Edit I tried the suggestion from ADH. It did jump out of reading sections after about 15 seconds, but writing annotated dump looks like it's going to take hours to complete.

Vegas

AhoyMatey
04-14-2005, 10:47 PM
The one thing that I always would have liked for mips.dasm.pl is the ability to see the raw machine code hex value alongside the address in the anotated dumpfile. The attached patch for mips.dasm.pl against against 0.4.2a accomplishes this. 0.4.2b also has alldeadhomiez' suggested fix for eliminating possible infinite loops.

Usage is now "mips.dasm.pl <executable> [raw]". Without the raw option, the <executable>.dump and <executable>.S files are produced as before. With the raw option these two files are created instead: <executable>.dump_w_raw and <executable>_w_raw.S. Unfortunately, the annotated file in this case may not be loaded into Xbyre's MIPS viewer tool. But both types of .S file may be around simultaneously - just run mips.dasm once with the raw option and once without.

I've also included a patch for xrefs.pl which allows it to work with a <executable>_w_raw.S file. It picks up whether the annotated *.S file is a "raw" file automatically (by searching for "raw" in the filename). I called it version 0.1a, but it's actually created off of drnull's patch of xrefs.pl

(If anyone has problems with the patches, I'll post the entire perl files.)

santa8claws
04-17-2005, 11:35 PM
Just wanted to add a quick note that when I used the precompiled gnu toolchain from sourceforge that objdump ends the last section with:
"Disassembly of section .fini:"

So it would be useful to update the dasm script to include both:

For this routine:
sub find_jump_targets()
{
open(IN, "<$dump") || die "Cannot open file $dump\n";
while (<IN>) {
if (/Disassembly of section/o) { last; }
}

Change "last" to "fini" or maybe to (last|fini)?

BTW, Great tool!
-S8C

jkozee
08-04-2005, 10:43 AM
I am trying to run this on tivoapp (7.1b) and receive this error in the load_got routine:
"Reading GOT ...Reading file tivoapp failed"

Here's a little more info:
OS = RH9 and RH8
Perl = v5.8.0 and ???
Script Version = 0.4 and 0.4.2a
objdump = self compiled and sourceforge
tivoapp file size = 22051388
tivoapp.dump .got line = "21 .got 0003ef24 10088620 10088620 014c8620 2**4"

Variable values at failure:
$i=57303
$got_size=64457
$offset=21792288

Can anyone offer any suggestions as to what I am doing wrong. Let me know if you need any more info. Any help would be appreciated.

7.1
08-04-2005, 11:32 AM
I am trying to run this on tivoapp (7.1b) and receive this error in the load_got routine:
"Reading GOT ...Reading file tivoapp failed"Seems to be a read failure on the exe file. You might want to get the errno so you can see why the read failed. All your data matches mine, and it works for me, so I suspect you have a problem with your tivoapp file. My script version is v.0.4.2b running on FC4 with perl 5.8.6.

jkozee
08-04-2005, 02:40 PM
Thanks for the reply 7.1. So, it looks like it's just me that is the lucky one:)

I think I have solved the problem, but I'm new to perl, so let me know if this makes sense.

I didn't check the errno, but I suspected that the error was from trying to read past the end of the file. I verified that to be the case by repositioning the file with seek(EXE, $offset + $i * 4, 0) during each iteration and didn't get the same error. Then I stumbled onto binmode(). I have now put a binmode() after every open statement and I think this has solved my problems.

So, I guess my questions now are:
1) Does this make sense as the solution?
2) Has something changed between perl 5.8.0 and 5.8.2 in regards to how it opens files by default?
3) Is there a global setting for perl to tell it to open every file in binary mode instead of text mode that should be set?

TIA

7.1
08-04-2005, 03:12 PM
Looks like this is an interaction between perl 5.8.0 (which added a bunch of UTF-8 character set stuff) and RH 8 and 9 that default the locale to UTF-8. Perl 5.8.1 backtracked on the "automatic UTF-8-ification" of file handles, which resolved the problem. See the perl 5.8.1 release notes: here (http://groups.google.com/group/comp.lang.perl.announce/browse_frm/thread/d7d454c512b2d892/50a3020ef6b64420?lnk=st&rnum=26#50a3020ef6b64420).

It's probably a good idea to use binmode on the files anyway. Might even be necessary to run on windows with ActiveState perl.

jkozee
08-04-2005, 03:52 PM
7.1,

Thanks for the insight. I'll upgrade perl tonight to get current, but I'll leave the binmode statements in for completeness. I don't think a posted diff is needed, but I can if anyone needs it.

One other thing. Is Xbyre's MIPS viewer tool available any longer. I'd like to take a look, but it looks like the site is down.

7.1
09-10-2005, 04:31 PM
Here's a 0.4.3 variant of the mips.dasm.pl script that adds support for some of the addressing modes used in the 7.2 tivoapps. For example, procedures are now mostly called with jal instead of jalr, and strings are referenced directly rather than through the global offset table. As far as I know it should still work with older tivoapps, but I didn't test it on anything but the 7.2.0 versions.

Indentation had drifted with the previous patches, and it wasn't "use strict" clean, so I'm posting a full version with those things fixed.

pentium101
07-06-2006, 12:38 PM
OK, I've gotten the mips.dasm.pl to work and have disassembled tivoapp.

When I load it into viewmips I see it processing then "Analysis Complete"

For some reason, I see no information in the window, or on any of the tabs.

...just a blank window.

NutKase

EDIT: I've gotten it working now using drnull's 04.2 update.

Now let's see what all this is telling me. ;)I'm also having trouble with the viewmips program.

I cannot get anything displayed under the disassembly tab. I've made my 6.2 tivoapp.S file using the 0.4.3 mips-dasm file and using the unistd.h file from the version 4 zip file.

Can anyone shed some light on why this may not be working for me?

shutterfriend
07-10-2006, 10:03 AM
Can anyone tell me what I am doing wrong. I have built the toolchain, ran the mips disassembler version 0.4.3 and it creates a dump and S file with no problems. I then try and open the .S file using viewmips program and it keeps crashing (exception 0xc0000005).

What am I doing wrong? I am currently reading the file by opening in textpad and I think I have been able to port the bufferhack but I don't think I am getting all the information to 100% guarantee I am getting the right locations for the patches.

Thanks.

Jamie
07-10-2006, 10:38 AM
Can anyone tell me what I am doing wrong. I have built the toolchain, ran the mips disassembler version 0.4.3 and it creates a dump and S file with no problems. I then try and open the .S file using viewmips program and it keeps crashing (exception 0xc0000005).

What am I doing wrong? I am currently reading the file by opening in textpad and I think I have been able to port the bufferhack but I don't think I am getting all the information to 100% guarantee I am getting the right locations for the patches.viewmips (http://www.dealdatabase.com/forum/showpost.php?p=171965&postcount=41) was released as an unsupported tool without source code. The author hasn't posted here in over a year. You could try to email him directly (there's an email link on the viewmips web page.) My suggestion is you don't use it.

shutterfriend
07-10-2006, 10:42 AM
Jamie,

Thanks for the quick reply. So after I disassemble the tivoapp using mips-dasm.0.4.3.pl and create the .S file what should I use to read the file? Currently I am using just textpad but I am not 100% sure I am doing it right. I am going to see if I can correctly identify the superpatches from 722 to 722b and see if I get the correct locations but is this the best way to do this?

Thanks.

Jamie
07-10-2006, 10:46 AM
Thanks for the quick reply. So after I disassemble the tivoapp using mips-dasm.0.4.3.pl and create the .S file what should I use to read the file? Currently I am using just textpad but I am not 100% sure I am doing it right. I am going to see if I can correctly identify the superpatches from 722 to 722b and see if I get the correct locations but is this the best way to do this?Yes, any text editor that can read the large files should be fine. You can use the perl scripts in this thread, or write your own, to generate call cross references and other analysis.

shutterfriend
07-10-2006, 10:50 AM
Jamie,

Thanks for the help. I will keep plugging away at this. Are you referring to the xrefs.pl and calltree.pl? For the xrefs.pl it requires a start address. What do I put as the start address?

Thanks again for your guidance. I hope to get this figured out this week to hopefully be able to provide patch porting for others to use and be able to contribute to this forum as it has done for me.

shutterfriend
07-10-2006, 10:55 AM
Jamie,

Would the start address be the start address indicated in the .S file near the top of the file?

Thanks.

Jamie
07-10-2006, 10:56 AM
Thanks for the help. I will keep plugging away at this. Are you referring to the xrefs.pl and calltree.pl? For the xrefs.pl it requires a start address. What do I put as the start address?That's not a script I use. If I had to guess, I'd guess the "start address" is the start address of a function and xrefs.pl may go off and find all the callers of that function.

These tools are unsupported too, but at least there is source so you can read them to understand what they do and adjust them as necessary.

shutterfriend
07-10-2006, 11:03 AM
Jamie,

What scripts do you use?

Also, I think I hit a dead end already. For the bufferhack for 7.2.2 and 7.2.2b the AddrA was always between li v0,300 and sw v0,252(s5) but 7.3 doesn't have this at all in the file.

I guess I am way off track.

mrpenguin
08-21-2006, 11:15 PM
How does the .proc file get created? I saw it asked in here but never answered. Thanks

mrpenguin
08-29-2006, 11:00 PM
Figured it out, for future readers, I created my .proc file by just going through the disassembly and finding areas that seemed like would do something specific and just gave it a name. I am sure that part could be much improved. Finding the GetResource routine is huge in making disassembly and patching easier I thought. the resource inclusion is really helpful. to find where the GetResource routine is took some time. In the end, it was not that tough. Look for the code that creates a mail message. in there you will find many JARL's. I basically put them all into the proc file as GetResource entries. one hit as accurate, because the processed file had tons of resources labeled. Then I knew which one it was.

mrpenguin
09-17-2006, 09:05 PM
Attached is a patch I made to get the SBS mode to properly display target memory addresses. I am pretty sure it works, finds/fills in about 850 of them.

nova1
09-20-2006, 10:34 PM
I've been disassembling tivoapp from 6.3 and since jal is used a lot to reference routines in the shared libraries, I've modified mips.dasm.pl to read in the dynamic symbol tables list from all the shared libraries it uses.

I first ran mips-TiVo-linux-nm -D libc.so >libc.sym and then added this code


open(IN, "<$libc_sym") || die "Cannot open file $libc_sym\n";

while (<IN>) {
my @fields = split();
$addr = hex($fields[0]);
if ($addr != 0) {
if ($fields[$#fields] =~ /^#/o) {
$symbols{$addr} = "libc_" . $fields[$#fields];
} else {
$symbols{$addr} = $fields[$#fields];
}
}
}
close(IN);

and then I added this code to the MAIN: loop section


# handle jal <absolute addr>
if (!defined($oper1) && !$comments) {
my $name = lookup_procedure(hex($target));
if (defined($name)) {
$comments = sprintf("$name\n");
}
}

I also modified start_lalj_mode to be this:


if ($instr eq "lui" && $args =~ /^([afgkrstv][0-9pt]),(0x[\da-f]*)/o) {

and at the top of handle_lalj_mode


if ($lalj_stage == 1 && $instr eq "lw") {
# handle lui/lw
$lalj_stage = 2;

# Debug.
#printf("Addr 0x%08X, LUL OFF=%08x\n", $addr, $lalj_off);
#sleep(1);
}

I also modified track_registers to handle the lui/jal/ori sequence with the delay slot


my $flush_reg = 0;
sub track_registers($$$) {
my ($addr, $instr, $args) = @_;
chomp($args);
my ($target, $oper1, $oper2) = split(',', $args);

if ($jump_target[$addr]) {
##flush register values
%regs = {zero => 0};
} elsif ($instr =~ /^j/o) {
# to handle delay slot (lui/jal/ori)
# delay flush of unsaved register values until
# insn after delay slot
$flush_reg = 1;
}
if ($flush_reg == 3) {
## jalr/jal: flush unsaved register values
map {undef $regs{$_}} @unsaved;
$flush_reg = 0;
} elsif ($flush_reg > 0) {
$flush_reg += 1;
}

SpoonsJTD
10-16-2006, 04:23 PM
I've been trying to get up to speed with this. Got the cross-compiler set up and am able to run the latest non-patched tmesis script. Am I correct in that resource strings got moved from the mfs database proper into brf files between versions 3.x and 6.x of the dtivo code? The resource.tcl script provided much earlier just returns a list of numbers, and it appears that tuikhelper.tcl has replaced this functionality somewhat. Has anybody modified the tmesis script to load strings from brf files instead of from the res file, assuming I understand the resource change history correctly?

jeboo
10-17-2006, 09:58 AM
I've been trying to get up to speed with this. Got the cross-compiler set up and am able to run the latest non-patched tmesis script. Am I correct in that resource strings got moved from the mfs database proper into brf files between versions 3.x and 6.x of the dtivo code? The resource.tcl script provided much earlier just returns a list of numbers, and it appears that tuikhelper.tcl has replaced this functionality somewhat. Has anybody modified the tmesis script to load strings from brf files instead of from the res file, assuming I understand the resource change history correctly?

You are correct with regards to strings being moved to brfs in 6.x. As for an updated script, that is not a trivial task to link brf references. MFS string refs were somewhat easy to discern, you only needed to figure out the value passed to one function.

With brfs, you need to identify the file, the ID, and then properly parse the brf. I know we had a pretty good brf parsing thread going a while back. It would be a bit of work, but its doable. I suppose the driving force will be how necessary brf refs are for updating hacks (superpatch, bufferhack, patches, etc). Lastly, brf patching is a bit more complicated. An improperly patched brf can lead to boot failures or crashes.

From what I've seen, most (SP pending??) of the previous hacks ported fine to 6.3.

jeboo

MuscleNerd
10-17-2006, 12:55 PM
I suppose the driving force will be how necessary brf refs are for updating hacks (superpatch, bufferhack, patches, etc).

Well they're very useful for developing new hacks too, of course. :)

SpoonsJTD
10-17-2006, 01:12 PM
I had a minor breakthrough with my own efforts after realizing that running objdump manually and then letting the mips.pl script pick the dump up didn't do the same thing as letting the script do it all by getting objdump in my path. I wasn't getting any of the 'possible reference to string' output messages. I thought I called objdump the same way the script was, but apparently I wasn't.

Now that I've looked at the result, I now understand the ealier reference to finding the sub that does the brf loading by finding the address jal'd right after loading the brf file name. I also now see how brf string extraction would be quite a bit different from the MFS lookup. What you described sounds a lot like the beginnings of a simulator. :)

SpoonsJTD
10-17-2006, 04:14 PM
Question: what c or c++ code when compiled results in the operations used to create the string address put in $a0 that tivoapp's disassembly shows? I've been trying to write some code that does the lui\ori to create the string address, but everything I write ends up generating lw\addiu. Is there some special compiler flag that changes string table handling that I am missing?

I've been playing around with the idea of creating a simple decompiler by looking at the mips dissassembler output from my own code and trying to match patterns in tivoapp, but I can't even write code to match this first pattern. I've tried c and c++, char pointers, literals, static char arrays from a class, and std::string's just to name a few. All generate the string reference using lw\addiu.

MuscleNerd
10-17-2006, 06:15 PM
I've been disassembling tivoapp from 6.3 and since jal is used a lot to reference routines in the shared libraries, I've modified mips.dasm.pl to read in the dynamic symbol tables list from all the shared libraries it uses.[/code]

The fact that tivoapp now heavily depends on dynamic libraries changes things somewhat, compared to when it was completely or mostly statically linked. For instance, the whole section of tivoapp.proc called "C runtime library, by code inspection" is no longer necessary, since the libc procedure addresses are now discovered using a technique like nova1's above.

Now, it's true that libc is the only one of the .so's that have meaningful symbol names...the rest of them are just the cryptic #0abcde variety. But analyzing the .so's and manually translating some of those #0abcde refs to useful symbols can be very useful.

For example, the 6.3a version of tivoapp contains about 152,000 references to the libtmk.so library. 34% of those are to TmkCore::DownRef(), and 14% of them are to TmkString::~TmkString(). Those 72,000 references are a whole lot better off being denoted as such, rather than their #0abcde equivalents.

So all I'm saying is it might be productive to focus on coming up with meaningful symbol names for the .so libs, especially the heavily used ones like libtmk.so.

nova1
10-18-2006, 12:02 PM
Thanks for the pointer.


$ grep tmk_#0E031 tivoapp.S | wc -l
51618
$ grep tmk_#0EC4E tivoapp.S | wc -l
20552

I hope that's right. I've added to my tivoapp.proc


0x02a245d0 TmkCore::DownRef TmkCore::DownRef() tmk_#0E031
0x02a2db70 TmkString::~TmkString TmkString::~TmkString() tmk_#0EC4E

SpoonsJTD
10-18-2006, 01:14 PM
Shouldn't another 14% of them be references to TmkString::TmkString()? :) Or split across several constructors if it has them.

MuscleNerd
10-18-2006, 01:35 PM
Yep nova1, those are right.

As for the TmkString constructors:


2a30534 TmkString::TmkString()
2a30b30 TmkString::TmkString()
2a2d47c TmkString::TmkString(char const*)
2a339b0 TmkString::TmkString(TmkMempool*, char const*)>
2a305fc TmkString::operator=(TmkString const&)

I'm not quite sure of the difference between the first two.

If you examine the hard-coded strings in libtmk.so, you'll find assertion strings and other hints for other functions in it.

You can also do the manual comparisons of assembly of that lib versus when those functions were all in tivoapp statically, although the change in toolchains and compiler options over the years can make that more challenging, comparing the 6.3a disassembly to the 3.1 disassemblyl, for example.

SpoonsJTD
10-18-2006, 01:42 PM
Yeah, my comment was a pseudo-joke about 14% of them being destructors for tmkstring. Where I thought this was headed was using nm -C on libtmk, for example, and using the symbols to augment the .proc or to add to the .proc instead of doing them by hand.

Jamie
10-18-2006, 01:49 PM
Yeah, my comment was a pseudo-joke about 14% of them being destructors for tmkstring. Where I thought this was headed was using nm -C on libtmk, for example, and using the symbols to augment the .proc or to add to the .proc instead of doing them by hand.The symbols in the proprietary libraries are all obfuscated, so nm isn't going to cut it. You still need to map the obfuscated names to meaningful names.

SpoonsJTD
10-18-2006, 01:54 PM
Running nm -C on libtmk.so gives me (the tail end):



00095c40 r TmkCore::~TmkCore()::__FUNCTION__
00095c4c r TmkCore::operator delete(void*)::__FUNCTION__
000962b4 r TmkInit::CommonInit(int*, char**, unsigned, TmkInitInfoNode*)::__FUNCTION__
000964f0 r TmkPipe::InternalInitPipe()::__FUNCTION__
00096880 r TmkSeed::TmkSeed(TmkMempoolUser)::__FUNCTION__
0009688c r TmkSeed::operator delete(void*)::__FUNCTION__
00098b88 r TmkFdSet::Select(TmkFdSet*, TmkFdSet*, TmkFdSet*, TmkTime*, bool (*)())::N_FRAMES
00098440 r UnixFile::Compare(TmkFile*)::__FUNCTION__
00098a70 r TmkServer::~TmkServer()::__FUNCTION__
00098b60 r TmkSocket::IsDataPresent()::__FUNCTION__
00096ddc r TmkThread::Dead()::__FUNCTION__
00096de4 r TmkThread::Join()::__FUNCTION__
00096dd0 r TmkThread::Start(long)::__FUNCTION__
00096dec r TmkThread::unblock()::__FUNCTION__
00097744 r TmkMempool::AssertValidPointer(void const*)::__FUNCTION__ const
00096aac r TmkStringCore::Utf8()::__FUNCTION__ const
00096aa4 r TmkStringCore::String()::__FUNCTION__ const
000de678 b TmkTimerManager::GetNextTimeout(TmkTime&)::zero const


I assumed that's where MuscleNerd was getting the symbols for the TmkString destructor and the DownRef method.

Jamie
10-18-2006, 01:58 PM
Running nm -C on libtmk.so gives me (the tail end):



00095c40 r TmkCore::~TmkCore()::__FUNCTION__
00095c4c r TmkCore::operator delete(void*)::__FUNCTION__
000962b4 r TmkInit::CommonInit(int*, char**, unsigned, TmkInitInfoNode*)::__FUNCTION__
000964f0 r TmkPipe::InternalInitPipe()::__FUNCTION__
00096880 r TmkSeed::TmkSeed(TmkMempoolUser)::__FUNCTION__
0009688c r TmkSeed::operator delete(void*)::__FUNCTION__
00098b88 r TmkFdSet::Select(TmkFdSet*, TmkFdSet*, TmkFdSet*, TmkTime*, bool (*)())::N_FRAMES
00098440 r UnixFile::Compare(TmkFile*)::__FUNCTION__
00098a70 r TmkServer::~TmkServer()::__FUNCTION__
00098b60 r TmkSocket::IsDataPresent()::__FUNCTION__
00096ddc r TmkThread::Dead()::__FUNCTION__
00096de4 r TmkThread::Join()::__FUNCTION__
00096dd0 r TmkThread::Start(long)::__FUNCTION__
00096dec r TmkThread::unblock()::__FUNCTION__
00097744 r TmkMempool::AssertValidPointer(void const*)::__FUNCTION__ const
00096aac r TmkStringCore::Utf8()::__FUNCTION__ const
00096aa4 r TmkStringCore::String()::__FUNCTION__ const
000de678 b TmkTimerManager::GetNextTimeout(TmkTime&)::zero const


I assumed that's where MuscleNerd was getting the symbols for the TmkString destructor and the DownRef method.Ah, ok. I'm looking at a different software version (7.x) and all the libraries there are stripped and have obfuscated names in the dynamic symbol table. If you've found .so's that aren't stripped, that is a find.

SpoonsJTD
10-18-2006, 02:01 PM
Yeah, nm isn't working on the other lib's that look interesting. The libtmk.so I'm using is from 6.3a.

Edit: Correction, later figured out this was a 6.2 file.

MuscleNerd
10-18-2006, 02:02 PM
Ah, ok. I'm looking at a different software version (7.x) and all the libraries there are stripped and have obfuscated names in the dynamic symbol table. If you've found .so's that aren't stripped, that is a find.

Yeah, those unstripped .so's are few and far between!

I know that 6.1.AA4-01-2 has them.

For some reason, when I use mips-TiVo-linux-nm on my 6.3a libs, I don't get the output that SpoonsJTD gets.

% mips-TiVo-linux-nm -C libtmk.so
mips-TiVo-linux-nm: libtmk.so: no symbols

% mips-TiVo-linux-nm -D libtmk.so | grep " #" | wc -l
1961

SpoonsJTD
10-18-2006, 02:06 PM
I think we simulposted, the libtmk.so I'm using is from 6.3a (and you fixed it heh).

Let me double check.

Edit: Correction -- the libtmk.so I'm getting symbols from is from 6.2.

MuscleNerd
10-18-2006, 02:08 PM
I think we simulposted, the libtmk.so I'm using is from 6.3a (and you fixed it heh).

LOL yeah you gotta love the confusion that editing can cause.

SpoonsJTD
10-18-2006, 02:11 PM
My bad, I've got a mix of 6.3 and 6.2 files I am playing with. The libtmk.so with symbols is from 6.2.

SpoonsJTD
10-18-2006, 02:26 PM
Now that I realized I was getting the tmk lib from 6.2, I checked some others and they have symbols too. Bittersweet -- symbols, but from the wrong version.

MuscleNerd
10-18-2006, 02:28 PM
Now that I realized I was getting the tmk lib from 6.2, I checked some others and they have symbols too. Bittersweet -- symbols, but from the wrong version.

About (at least) 180 of the 6.3a libtmk.so routines are unique enough that you can manually match them to 6.2 libtmk.so symbols.

nova1
10-18-2006, 02:42 PM
Here are the main entry points for tivoapp 6.3a


0x00622cec SystemServices_main SystemServices_main(argc, char *argv[],
envp)
0x0062f13c goto_main goto_main(argc, char *argv[], envp)
0x0063a8e0 KnownHost_main KnownHost_main(argc, char *argv[], envp)
0x0063cd80 SiHost_main SiHost_main(argc, char *argv[], envp)
0x006436dc shmemd_main shmemd_main(argc, char *argv[], envp)
0x00694e18 dbgc-mcp_main dbgc-mcp_main(argc, char *argv[], envp)
0x0069c6b4 HmeHost_main HmeHost_main(argc, char *argv[], envp)
0x0069d550 convert-db_main convert-db_main(argc, char *argv[], envp)
0x006a62e0 httpd_main httpd_main(argc, char *argv[], envp)
0x006a7eb0 checkForSwInstall_main checkForSwInstall_main(argc, char *argv[
], envp)
0x006c3334 mcp_main mcp_main(argc, char *argv[], envp)
0x006d7648 tcphonehome_main tcphonehome_main(argc, char *argv[], env
p)
0x006e0264 main main(argc, char *argv[], envp)
0x006e3b10 mfsadd_main mfsadd_main(argc, char *argv[], envp)
0x006f7658 TvLauncher_main TvLauncher_main(argc, char *argv[], envp)
0x006f820c RendezvousDaemon_main RendezvousDaemon_main(argc, char *argv[]
, envp)
0x0070e748 progressiveBootConfirm_main progressiveBootConfirm_main(argc
, char *argv[], envp)
0x0072b3a4 huxley_main huxley_main(argc, char *argv[], envp)
0x007356f0 mfsd_main mfsd_main(argc, char *argv[], envp)
0x0073df6c myworld_main myworld_main(argc, char *argv[], envp)
0x0074baec tivoapp_main tivoapp_main(argc, char *argv[], envp)
0x0074be48 SiHost_Dynamic_main SiHost_Dynamic_main(argc, char *argv[],
envp)
0x0090d694 expect_main expect_main(argc, char *argv[], envp)
0x00951180 ApgManager_main ApgManager_main(argc, char *argv[], envp)
0x00ce2810 knquery_main knquery_main(argc, char *argv[], envp)
0x00cecca4 tivosh_main tivosh_main(argc, char *argv[], envp)
0x00f2a784 mfsassert_main mfsassert_main(argc, char *argv[], envp)
0x01633db4 fsfix_main fsfix_main(argc, char *argv[], envp)
0x01657db8 mfscheck_main mfscheck_main(argc, char *argv[], envp)

MuscleNerd
10-18-2006, 02:59 PM
Here are the main entry points for tivoapp 6.3a

Awesome...well to continue the spirit of collaboration, here's some symbol information for 6.3a's libtmk.so as best as I can determine. (Note that it's not in the format of a normal .proc file...I use a different set of scripts, but you can convert it into a .proc file easily enough).

SpoonsJTD
10-18-2006, 03:37 PM
MuscleNerd, I asked this above, any thoughts on why the string assignment in tivoapp has a different assembly signature than all the various string assignment mechanisms I use to produce comparison assembly code using gcc and g++? This is bugging me to no end that I can't even forward engineer the same assembly output for something as simple as using a string as a pameter in a method call.

MuscleNerd
10-18-2006, 03:50 PM
I don't know, but it might just be something like you're using a different gcc than TiVo uses. TiVo hasn't posted the toolchain for the 6.3 build yet.

Jamie
10-18-2006, 04:17 PM
MuscleNerd, I asked this above, any thoughts on why the string assignment in tivoapp has a different assembly signature than all the various string assignment mechanisms I use to produce comparison assembly code using gcc and g++? This is bugging me to no end that I can't even forward engineer the same assembly output for something as simple as using a string as a pameter in a method call.It's possible they made code generation modifications, but more likely it's just a matter of getting the gcc options right. I may not have picked all the right options in the gcc config in my toolchain build script. I'd look for options that allow you to turn off PIC (position independent code).

MuscleNerd
10-18-2006, 05:39 PM
Here are a handful of subroutine names for the 6.3a tivoapp itself, but I've barely started looking into it. These are just some routines I found interesting enough to look for first. The names are mostly my own, except for the last four.

The "really_need_to_encrypt_clips" routine just returns True. If you make it return false, then you now have an alternative to the nocso patch, except that it's inferior to the nocso patch because it doesn't allow your tivo to play clips that were encrypted before you applied the patch.

The BF_* routines are used in encrypting/decrypting the HMO traffic.

SpoonsJTD
10-22-2006, 10:26 PM
It's possible they made code generation modifications, but more likely it's just a matter of getting the gcc options right. I may not have picked all the right options in the gcc config in my toolchain build script. I'd look for options that allow you to turn off PIC (position independent code).

Ok, I found the fpic and fPIC which turns ON PIC when compiling, but nothing for turning off PIC. At first, I was looking for options to turn off PIC during a build, but re-reading this, are you saying I should be looking in the config for when GCC itself is being built?

'-fno-pic' turns off PIC. Should I be applying this to the build config somewhere for building gcc, or should using this flag during gcc compiles work?'

Update: Using -fno-pic during a compile didn't change the lw/addiu sequence during string assignments.

SpoonsJTD
10-22-2006, 10:47 PM
Interesting, I just found this post:
http://dealdatabase.com/forum/showpost.php?p=166414&postcount=26

DrNull added to the original script to catch the string loading assembly pattern I am getting from my builds which makes me wonder if the other pattern is created from a different code construct.

Update: Ok, doing some research on the web has led me to understand that the lw\addiu vs. lui\ori doesn't have anything to do with string loading per se, they are just different ways to reference a memory location. You use lui\ori for larger addresses that can't be loaded 'immediately' with lw\addiu.

I don't know much about the gcc compiler, but I assume that it puts all the string literals in a string table somewhere? Possibly towards the end of the executable? This might explain why my tiny helloworld app refers to strings using a direct load, but tivoapp, which is much much larger has to load strings using high word, low word memory addressing.

MuscleNerd
11-15-2006, 08:19 PM
For anyone looking for a huge boost to the number of strings referenced by the 6.3a disassembly...

The routine ResourceManager::LookupResource() is a very large "if" statement that maps values from 0x4c4b40 through 0x4fd032 into strings from 24 different BRFs. Those BRFs are simple in that they each consist of only a single list of strings. Combined, they will provide you with over 2100 strings. The strings are referenced about 1900 times in the code, which should help orient you within the code considerably.

The routine also maps values from 0x4fd033 through 0x517b7f into PNG images from 11 different BRFs.

Other versions of tivoapp also have LookupResource() (easily found by looking for that string), but of course the numbers above are all different.

Hope that helps some...

P.S. The LookupResource() routine itself isn't called that many times directly... it's called by FetchResource (at 0x499a40 in v6.3a) that *is* called about 1400 times. For instance:



9e95e4: 3c05004f lui a1,0x4f
9e95e8: 02002021 move a0,s0
9e95ec: 0c126690 jal 0x499a40 ;: <FetchResource>
9e95f0: 34a5cfc7 ori a1,a1,0xcfc7 ;: a1 = 004fcfc7 (BRF_STR "Sorry, recording of music and audio-only programs is not supported at this time.")

jacko101
09-22-2007, 06:08 AM
sorry im noob how do you use the mipsdisassemble

Jamie
09-22-2007, 08:08 AM
sorry im noob how do you use the mipsdisassembleThe instructions are listed in the very first post in this thread. If they don't make sense to you, then you are in way over your head and it is unlikely the tool will be of any use to you.

woracan
09-01-2009, 02:53 PM
Please post tivoapp.proc files for any versions you may have. It would be helpful if you could help me find the version of tivoapp to match the tivoapp.proc file you post. Thanks.

tivo4mevo
09-04-2009, 04:50 PM
There were never very many folks producing *.proc files to begin with, and the versions that you're asking for are circa 2005 (and 6.1 is for a combo box). So I'm not sure how much of a response you'll get.

I see your post (http://www.dealdatabase.com/forum/showthread.php?p=304214#post304214) of the NoCSO patch for 7.2.0a-oth, were you looking for any other patches in particular, or just wanting to explore a bit more?

woracan
09-06-2009, 11:48 PM
There were never very many folks producing *.proc files to begin with, and the versions that you're asking for are circa 2005 (and 6.1 is for a combo box). So I'm not sure how much of a response you'll get.

I see your post (http://www.dealdatabase.com/forum/showthread.php?p=304214#post304214) of the NoCSO patch for 7.2.0a-oth, were you looking for any other patches in particular, or just wanting to explore a bit more?

I'd like to explore the tivoapp more.

tivo4mevo
09-07-2009, 12:52 PM
You're correct that hacking related to guide data isn't discussed here.

If no one has a *.proc file for your specific tivoapp, you may be able to obtain a tivoapp for which the proc file has already been posted, explore that, and then extrapolate that to your specific versions.

jt1134
09-21-2009, 01:39 AM
here's some symbol information from 11.0d tivoapp and libtmk.so, mostly ported from the stuff posted here previously

the tmesis script missed quite a few strings in the 'tivoapp_main()' function, but after manually matching up the missing references, it was a snap to identify most of tivoapp's subprograms

off to bed...

[edit]see post below for updated tivoapp.proc

7.1
09-24-2009, 04:17 PM
Here's some more stuff.

There's an updated mips.dasm (0.4.5). This fixes some bugs and adds some new features. Some of the updates are mine, and others came from another contributor. Unfortunately, I don't have a good change log. It includes most of the updates from this thread. One notable thing still missing is the resource string reference lookup suggested (http://www.dealdatabase.com/forum/showthread.php?p=269413#post269413) by MuscleNerd.

mips.dasm now dumps a .refcount file that lists procedure reference counts.

It also dumps a .proc.new file with candidate .proc entries based on a simple heuristic that a reference to a string that looks like a function prototype might be a good candidate for the containing function prototype.

There's a perl script, libSymbols.pl, that scans the name list from a set of .so files to produce a .proc. This is basically the code that nova1 suggested here (http://www.dealdatabase.com/forum/showthread.php?p=264549#post264549), but taken out of mips.dasm so it can be applied to multiple libraries to produce a .proc file.

There's a perl script, fsmSymbols.pl that scans the data sections of tivoapp looking for string references next to code references. The idea is that the string might be a good identifier for the function. This seems to work well for functions called from the BRF Finite State Machines.

Finally, there's a 11.0d tivoapp.proc. This starts with jt1134's as a base, adds some of the know patches, and the functions identified with the scripts above. There's a lot of cruft in there, but the cruft doesn't seem to cause problems, and some of the additions found by the heuristics are useful.

jt1134
09-27-2009, 07:47 PM
thanks for the update 7.1 (and the unnamed contributor)! i've been trying to get up to speed with perl so that i could try to implement most of this stuff

here's an updated 11.0d tivoapp.proc including everything from above with ~100 more libtmk.so routines and a handful more from tivoapp

7.1
10-09-2009, 09:15 AM
Here's another little addition: brf strings based on MuscleNerds post (http://www.dealdatabase.com/forum/showthread.php?p=269413#post269413).

brfstrings.tcl is a variation on this script (http://www.dealdatabase.com/forum/showthread.php?p=231957#post231957).

brf.sh uses brfstrings.tcl and produces a tivoapp.brfstrings file used by mips.dasm. It is hand written based on the code in ResourceManager::LookupResource. It could probably be automated, but it isn't that hard to do it by hand. I've done 11.0d and 7.2.2b.

mips.dasm-0.4.6.pl now looks up string references in registers from tivoapp.brfstrings.

This all works nicely in 7.2.2b, finding 1800 BRF strings. Unfortunately, there are a lot less in 11.0d: only 206.

woracan
10-11-2009, 12:31 AM
Thanks for the excellent contribution 7.1!

Shouldn't the locations for sp2 and sp3 in the 7.2.2b tivoapp.proc file be unique?


0x00c593d0 sp2
0x00c593d0 sp3

woracan
10-11-2009, 02:26 PM
In the 7.2.2b tivoapp.proc file,

0x00e3834c sp12
should be

0x00e3934c sp12

RandC
11-20-2009, 02:24 AM
brf.sh uses brfstrings.tcl and produces a tivoapp.brfstrings file used by mips.dasm. It is hand written based on the code in ResourceManager::LookupResource. It could probably be automated, but it isn't that hard to do it by hand. I've done 11.0d and 7.2.2b.How do you go about creating or modifying the brf.sh file for 6.4a software. I'm playing around with MIPS disassembler and am interested in what this adds to the output.

7.1
11-20-2009, 12:50 PM
How do you go about creating or modifying the brf.sh file for 6.4a software. I'm playing around with MIPS disassembler and am interested in what this adds to the output.First run tivoapp through mips.dasm-0.4.6.pl to generate a proc file. Then run it through again using that proc file:

mips.dasm-0.4.6.pl tivoapp raw
cat tivapp.proc.new >>tivoapp.proc
mips.dasm-0.4.6.pl tivoapp raw
Now look through tivoapp_w_raw.S in a text editor and find the LocateResource function. You should be able to find it by searching for "LocateResource".

Within that function, you'll see blocks of code that look like this:

0x0047e440 0x0c123e29 jal 0x0048f8a4 # ; </uilib/lib/resources/Thumbs.brf>
0x0047e444 0x00000000 nop
0x0047e448 0x3c05ffae lui a1,0xffae # a1 = 0xffae0000
0x0047e44c 0x34a584a0 ori a1,a1,0x84a0 # a1 = 0xffae84a0Note the name of the brf file (Thumbs.brf in this instance), and the final a1 value (0xffae84a0, in this case). Add a line to brf.sh with this data:
brfstrings.tcl Thumbs.brf 0xffae84a0Repeat for all the similar code blocks in the LookupResource function.

You'll need to fetch the brf files (e.g. with TuikHelper.tcl) and drop them in the same directory with the brf.sh script when you run it. I don't have the 6.4a brf files, so I haven't done it for that software version.

RandC
11-20-2009, 03:55 PM
You'll need to fetch the brf files (e.g. with TuikHelper.tcl) and drop them in the same directory with the brf.sh script when you run it. I don't have the 6.4a brf files, so I haven't done it for that software version.Already Tuik'ed the brf files onto PC. I will run as you posted and see how things turn out.

RandC
11-20-2009, 08:24 PM
Within that function, you'll see blocks of code that look like this:

0x0047e440 0x0c123e29 jal 0x0048f8a4 # ; </uilib/lib/resources/Thumbs.brf>
0x0047e444 0x00000000 nop
0x0047e448 0x3c05ffae lui a1,0xffae # a1 = 0xffae0000
0x0047e44c 0x34a584a0 ori a1,a1,0x84a0 # a1 = 0xffae84a0Note the name of the brf file (Thumbs.brf in this instance), and the final a1 value (0xffae84a0, in this case). Add a line to brf.sh with this data:
brfstrings.tcl Thumbs.brf 0xffae84a0Repeat for all the similar code blocks in the LookupResource function.
Would this qualify as three entries?

0x0059f7ac jal 0x005e11d8 # ; </soma_uicommon/lib/somascheduler/TvSchedulerResultFormat.brf>
0x0059f7b0 addiu a0,sp,24
0x0059f7b4 jal 0x0043040c
0x0059f7b8 move a0,s0
0x0059f7bc move v1,v0
0x0059f7c0 lui v0,0x2 # v0 = 0x00020000
0x0059f7c4 ori v0,v0,0x981c # v0 = 0x0002981c
0x0059f7c8 beq v1,v0,0x0059f978
0x0059f7cc sltu v0,v0,v1
0x0059f7d0 bnez v0,0x0059f8cc
0x0059f7d4 lui v0,0x2 # v0 = 0x00020000
0x0059f7d8 ori v0,v0,0x9817 # v0 = 0x00029817
0x0059f7dc beq v1,v0,0x0059f8ac
0x0059f7e0 sltu v0,v0,v1
0x0059f7e4 bnez v0,0x0059f874
0x0059f7e8 lui v0,0x2 # v0 = 0x00020000
0x0059f7ec ori v0,v0,0x9816 # v0 = 0x00029816

brfstrings.tcl TvSchedulerResultFormat.brf 0x0002981c
brfstrings.tcl TvSchedulerResultFormat.brf 0x00029817
brfstrings.tcl TvSchedulerResultFormat.brf 0x00029816

7.1
11-21-2009, 12:57 PM
Would this qualify as three entries?
...Nope. You aren't in the right section of code. You only want to do this in the LookupResource function. In 6.4a, it starts at 0x0047e160 and ends at 0x0047ea14.

I've attached what I came up with. If you have the .brf files and process them to produce a tivoapp.brfstrings file, please post it.

jt1134
12-09-2009, 09:25 PM
here are .proc and .brfstrings files I've been using with 6.4a

voskrese
04-15-2017, 10:07 AM
Hi, where can I download scripts in topic