Overriding a system call in AIX

At some point, I used to write code for AIX kernel. (I have written modules for the Solaris and HPUX kernels too, but these day's I am purely a user-space developer). Still, I sometimes get questions/comments related to AIX on email. Here is a recent one

I went through your blogs..quite nice..
I need some help with the aix kernel.I m currently implement a system call.
What I need to do is basically intercept a system call on aix 5.3.
How can I do this.?.
On linux I have the system call table , which i can replace the current system calls with my own using the NR offset into the sys_call_table.
How do i do the same in aix kernel?

AIX is a different beast altogether, unlike a lot of other Unixes, because its BSD heritage is not as strong and a bit under the hood. So a lot of things work very differently in AIX, and over-riding a system call is one of them.


For the impatient, here is a short answer.
Write a function with the same signature as the system call and load it as a kernel extension. Any application started after this new module has been loaded will see your symbol as a system call, instead of the original one, due to the way AIX loader works.

For most of the time - you are interested in intercepting a system call, rather than overriding it. Let's say you want to intercept getpid(). Now you could write your own version of getpid with an exactly same signature, do whatever you wanted to do, and then transfer control to the original getpid(). IBM has an example here on how could you add a new syscall to AIX.

Follow it, but name your syscall getpid() (or whatever you want to override).

Herein lies the catch. There is no way you can call the original getpid(), because just having overridden that symbol, you can no longer access the original. Any attempts to call getpid() from with getpid() will result in ...

Well, what else, but recursion.

So to work around this problem, over-riding a system call means writing two kernel modules. The first one would merely re-export the original system call with a different name. The second, will actually override the syscall by redefining it, and then call the original one as exported by the first module. To make it cleaner, you can automate the loading of the first module from within the second module, so to the user, it is only one module that they load.

Usually you would do this fairly early on in the boot process, because your intercept will not apply to applications started before the interceptor module was loaded.

I will update this post sometime in the future with the long answer.