Pages

Saturday, May 23, 2009

Overriding a system call in AIX

At some point, I used to write code for AIX kernel. (I have written modules for the Solaris and HPUX kernels too, but these day's I am purely a user-space developer). Still, I sometimes get questions/comments related to AIX on email. Here is a recent one

I went through your blogs..quite nice..
I need some help with the aix kernel.I m currently implement a system call.
What I need to do is basically intercept a system call on aix 5.3.
How can I do this.?.
On linux I have the system call table , which i can replace the current system calls with my own using the NR offset into the sys_call_table.
How do i do the same in aix kernel?


AIX is a different beast altogether, unlike a lot of other Unixes, because its BSD heritage is not as strong and a bit under the hood. So a lot of things work very differently in AIX, and over-riding a system call is one of them.


For the impatient, here is a short answer.
Write a function with the same signature as the system call and load it as a kernel extension. Any application started after this new module has been loaded will see your symbol as a system call, instead of the original one, due to the way AIX loader works.

For most of the time - you are interested in intercepting a system call, rather than overriding it. Let's say you want to intercept getpid(). Now you could write your own version of getpid with an exactly same signature, do whatever you wanted to do, and then transfer control to the original getpid(). IBM has an example here on how could you add a new syscall to AIX.

Follow it, but name your syscall getpid() (or whatever you want to override).

Herein lies the catch. There is no way you can call the original getpid(), because just having overridden that symbol, you can no longer access the original. Any attempts to call getpid() from with getpid() will result in ...

Well, what else, but recursion.

So to work around this problem, over-riding a system call means writing two kernel modules. The first one would merely re-export the original system call with a different name. The second, will actually override the syscall by redefining it, and then call the original one as exported by the first module. To make it cleaner, you can automate the loading of the first module from within the second module, so to the user, it is only one module that they load.

Usually you would do this fairly early on in the boot process, because your intercept will not apply to applications started before the interceptor module was loaded.

I will update this post sometime in the future with the long answer.

12 comments:

chetank said...

hi Qasim,

Thanx for the reply.


like u said..."The first one would merely re-export the original system call with a different name. The second, will actually override the syscall by redefining it, and then call the original one as exported by the first module."

so tell me if i understood this correctly..
lets use getpid().

first module will re-export getpid() with a different name..
so lets call it my_getpid()

my getpid()
{
getpid(); /* original getpid()*/
}

the second module will re-export
getpid() <-- this is my getpid

getpid()
{
/* do some stuff here */
/* load first module & call
my_getpid() */
my_getpid(); /* this will actually call the original getpid() */
}

is my understanding correct.

however there is another way of doing it..

Now i know the address of all the syscalls..this can be found from the debugger.
lets say getpid() is stored at mem location 0x0024

so i write one kern extension..
call it getpid();


getpid()
{
/*use a function pointer to call the original getpid using the address */
void (*fp)();
/* make fp point to original getpid() */
fp = 0x0024;

/* Call original getpid() :) */
(*fp);
}

will this work?

but let me know.

Thanks

Qasim Zaidi said...

Your understanding is correct.

Regarding the function pointers, if you know for sure that the function would always be available at a certain address, you can use function pointers and make do with a single module. The address would, in all likelihood, vary across kernel's and machines, so too fragile to be used in any production environment.

chetank said...

Hi Qasim,

I tried what u said about overriding syscalls and guess what
it WORKED!!!! wooot

But i have issues when i execute other commands
for e.g. i replaced getpid syscall succesfully.
But whn i try to say complile w xlc
the binder process seg faults
dunno why
may return values are not correct or i m doing something wrong in my syscall

i also replaced access syscalls
the system doesnt hang after i load the extension
but when i do something like say
truss cd .
the system hangs
so obvioulsy i m missing something in my syscall.

any idea what mistake i cud be doing?
i code is almost similar the way i explained it..i m not doing anything..new..like mem access or allocation..just simple statements

Qasim Zaidi said...

when you intercept the syscall, are you doing your own processing before executing the syscall, or afterwards?

Also, make sure you are compiling the extension with the right flags, as specified on http://www.ibm.com/developerworks/aix/library/au-kernelext.html#listing3.

Finally, is the issue only with running truss and hooking access(). That is, would getpid() hook work fine for you? Or is it that the access hook having problem only when you run truss cd . (and works fine with cd .)

chetank said...

when you intercept the syscall, are you doing your own processing before executing the syscall, or afterwards?

processing before executing orignal syscall.

Also, make sure you are compiling the extension with the right flags, as specified on http://www.ibm.com/developerworks/aix/library/au-kernelext.html#listing3.

No changes ..followed exactly as given in the article.

Finally, is the issue only with running truss and hooking access(). That is, would getpid() hook work fine for you? Or is it that the access hook having problem only when you run truss cd . (and works fine with cd .)

the system crashes whenever i run
ld or truss so far..
not system crashes..if i use the syscalls or if i simply run cd .
or any other simple commands.

Patel said...

hi Qasim,
Is it possible to trap system calls(e.g: open, execve...) using ptrace on AIX?
help would be greatly appreciated.

~Shaan.

Anonymous said...
This comment has been removed by a blog administrator.
Anonymous said...
This comment has been removed by a blog administrator.
Anonymous said...

Nice post and this post helped me alot in my college assignement. Thank you seeking your information.

Anonymous said...

I just found the website who reviews about
many
home based business reviews

If you want to know more here it is
home based business reviews
www.home-businessreviews.com

Anonymous said...

Predilection casinos? digging this unaccomplished [url=http://www.realcazinoz.com]casino[/url] advisor and press up online casino games like slots, blackjack, roulette, baccarat and more at www.realcazinoz.com .
you can also hamper our additional [url=http://freecasinogames2010.webs.com]casino[/url] lope at http://freecasinogames2010.webs.com and convoy accurate to living tangled currency !
another solo confinement [url=http://www.ttittancasino.com]casino spiele[/url] put out in across is www.ttittancasino.com , preferably than of german gamblers, be repaid magnanimous online casino bonus.

Anonymous said...

Online-Casinos bieten in der Regel Quoten und Payback-Prozentsätze, die ein bisschen höher als Land-basierte Casinos sind. Einige Online-Casinos behaupten höhere Payback-Prozentsätze für Spielautomaten-Spiele spielautomaten online, und einige veröffentlichen Auszahlung Prozentsatz Audits auf ihren Websites.