I’ve been enjoying a hefty process creation penalty on my Windows XP Home SP3 for about two months. The problem is most manifest and annoying with tasks that do create lots of processes, such as shell scripts (incidentally, bash scripts on Cygwin), Makefiles, or unpacking an IzPack package such as the SpringSource Tool Suite installer (lots of separate unpack200.exe JAR extractor processes). I’m sure it’s process creation from observing bash script diagnostic output, or watching processes appear in task manager. There is no noticeable delay once a process is up and running.
I’ve reported that problem on the Cygwin mailing list as I initially and erroneously thought only Cygwin was affected, suspecting a bug in the Cygwin DLL, or whatnot.
Slowdown after update on Win32 (XP Home) – (link to my post to the Cygwin list)
I’m wondering whether something has installed some crap into a process creation hook which I presume might exist on Windows. (As with the security manager in Java.) A virus, or security software? I haven’t consciously installed either. I also suspected a Microsoft update glitch, but I think they would have fixed that by now.
As far as I know, processes on Win32 are created by CreateProcess.
How can I find out why process creation takes so long and what exactly is going on here? Is there something like strace for Linux, or possibly even something better?
Many thanks to Luke for pointing me in the right direction. Procmon is a fantastic tool. Literally, a new world inside the system opened up for me. And the culprit was quickly found by setting a filter to include
parent process ID = WINPID of a Cygwin bash.exeand then just monitoring a singlelscommand. It was malware hooked into a registry key namedAppCertDlls, of which I was of course totally unaware before.Once I moved the malware
clipmote.dllout ofsystem32, process creation speed was immediately back to normal.I guess I contracted this virus by installing poisoned freeware.
I spent some time investigating this issue and found it both scary and interesting, so here’s what I’ve found, and of course I’ll be thankful for all additional information or any comments you might have.
The malware DLL was loaded in every single user process, even Explorer, taskmgr, and procmon itself. Only processes under
SYSTEM(as listed in inprocexp.exe) appear to have remained uninfected.The issue here is that
HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\AppCertDllsis checked (at least on my system, but possibly on many systems, and probably even by default) to load DLLs which may have a say over whether or not some binary is allowed to run or not by returning a value from a function that is called on them:In fact, I had an entry under that key called
sethdown, but the name could be anything.From reports found on the net I conclude that the
AppCertDllshook is a regular part of the Windows operating system, and not some rogue extension. If that is so, it’s a virus entry point, as it allows to dynamically configure malware into a process.Looks like it is actually – and ironically – meant to be a security feature, not dissimilar to the security manger in the JVM (see my original question). A Microsoft utility called
appsec.exeis reported to use this key. The function that is then called on the “security” DLL that is loaded isCreateProcessNotify, as quoted above. From what I’ve understood, it is assumed to just say yes or no – but it can do whatever it wants.In my case and the single instance of running
ls.exethat I analysed, it created five threads. They load or try to load other DLLs and interact with the registry. And who knows what else.If anyone knows more about this dangerous mechanism, I’m all ears.
Here’s what I found on
AppCertDlls:early mention in 01/2007 (Re5: Блокировка запуска приложений)
AppCertDlls reported on Sysinternals forum (Autoruns), 10/2007
About AppCertDlls – virus mentioned 01/2008
more details on the AppCertDlls key and CreateProcessNotify, 01/2008
06/2010 infection report providing some details
The malware tried to steal bank account information (and money, obviously), but it could possibly be configured to do other things, too. There were additional registry entries under
HKEY_CURRENT_USER\Software\AppDataLow\{some GUID here}. It must have done some screen scraping while I was doing online banking; it knew used TANs. I remember I once I had the impression my bank was hacked when shortly before or (probably) after login I saw a screen asking for about twenty TANs at once. WTF, I thought, the browser got the wrong URL, and I closed the window. I should have be more concerned. I wouldn’t have thought the whole issue to be so dangerous. Fortunately, no damage. Just lucky, I guess.Here’s a list of the significant-looking strings I found inside the virus:
client.dllEAPSFILT.dllKERNEL32.dllI think the first two might be other virus libraries it tries to load.
_aullshrCompareStringACreateEventACreateFileACreateFileMappingACreateProcessNotifyCtfImmIsCiceroStartedInThreadCtfImmTIMActivateDllCleanupServerDllStartupServerExitProcessGetThreadContextImmDisableImeImmDisableIMEImmGetConversionListAImmGetVirtualKeyImmRegisterWordAImmSetCandidateWindowInterlockedExchangeAddiswlowerlstrcmpAMapViewOfFilememsetOpenThreadReadFileRealDriveTypeRegenerateUserEnvironmentRestartDialogSetFilePointerSleepstrchrTerminateThreadUnmapViewOfFilevDbgPrintExWithPrefixVirtualQueryExWaitForMultipleObjectsThe
Cicerofunction looked rogue to me, but it is part ofC:\WINDOWS\system32\imm32.dll, the “Windows IMM32 API Client Library” – whatever that is.I ran
sfc.exe, Microsoft’s “System File Checker”. Will do more checking on my program files. And get a virus scanner. Recommendations welcome.