Is it possible to force gcc use int instruction for all the system calls, but not sysenter? This question may sound strange but I have to compile some projects like Python and Firefox this way.
Summary
Thanks to jbcreix, I’ve downloaded glibc 2.9 source code, and modified the lines in sysdeps/unix/sysv/linux/i386/sysdep.h, to disable use of sysenter by #undef I386_USE_SYSENTER, and it works.
Recompile your C library after replacing
sysenterbyint 0x80insyscall.sand link again.This is not compiler generated code which means you are lucky.
The ultimate origin of the actual syscall is here, as the OP says:
http://cvs.savannah.gnu.org/viewvc/libc/sysdeps/unix/sysv/linux/i386/sysdep.h?root=libc&view=markup
And as I suspected there really was a syscall.S it’s just that the glibc sources are a labyrinth.
http://cvs.savannah.gnu.org/viewvc/libc/sysdeps/unix/sysv/linux/i386/syscall.S?root=libc&view=markup
So I think he got it right, asveikau.