I have very weird problem with web-services. Setup is following: 6 web-services deployed on a x64 single core, 1gb ram machine under IIS7 (win2008), each running in a dedicated application pool. Web-services are connecting via LB to a mysql cluster. We are using mysql odbc connector 5.1.6 with connection pooling off, .net v3.5. Build target platform x64.
Roughly, every 3hrs, one web-service starts consuming 100% of CPU (or better to say 100% of CPU core, but since it is a single core machine it will be 100% of CPU). At the same time mysql servers are idle. Killing “hanging” w3wp from task manager doesn’t help, IIS restart helps. I’m looking at process explorer and see one strange thread inside w3wp with following stack trace, which i can’t understand:
ntoskrnl.exe!IoAcquireRemoveLockEx+0xe7
ntoskrnl.exe!memset+0x22a
ntoskrnl.exe!KeWaitForSingleObject+0x2cb
ntoskrnl.exe!KeDetachProcess+0x120d
ntoskrnl.exe!PsReturnProcessNonPagedPoolQuota+0x3a3
ntoskrnl.exe!CcSetDirtyPinnedData+0x433
myodbc5.dll!SQLTablePrivilegesW+0x22dac
myodbc5.dll!SQLTablePrivilegesW+0x2bffd
myodbc5.dll!SQLTablePrivilegesW+0x107ea
odbc32.dll!SQLAllocHandle+0xba5
odbc32.dll!SQLAllocHandle+0x9c8
mscorwks.dll!IEE+0xa913
System.Data.ni.dll+0x56f8c3
System.Data.ni.dll+0x5c1efe
mscorwks.dll!IEE+0xb0ee
mscorwks.dll!CompareAssemblyIdentity+0x2bb8f
mscorwks.dll!CompareAssemblyIdentity+0x39deb
mscorwks.dll!CompareAssemblyIdentity+0xe63d
mscorwks.dll!CertCreateAuthenticodeLicense+0x21b12f
mscorwks.dll!StrongNameTokenFromPublicKey+0x49f7
mscorwks.dll!PreBindAssembly+0x88a46
mscorlib.ni.dll+0x3988d0
mscorlib.ni.dll+0x39877a
mscorwks.dll!IEE+0xb042
mscorwks.dll!CompareAssemblyIdentity+0x2610d
mscorwks.dll!CompareAssemblyIdentity+0x26358
mscorwks.dll!CompareAssemblyIdentity+0x2ae50
mscorwks.dll!CompareAssemblyIdentity+0x97ad7
mscorwks.dll!CompareAssemblyIdentity+0x97877
mscorwks.dll!CopyPDBs+0x170f3
mscorwks.dll!InitializeFusion+0x5994
mscorwks.dll!GetMetaDataInternalInterfaceFromPublic+0x34ad9
mscorwks.dll!InitializeFusion+0x1282d
mscorwks.dll!CorExitProcess+0x802d
mscorwks.dll!InitializeFusion+0xb5db
mscorwks.dll!CertCreateAuthenticodeLicense+0x240021
mscorwks.dll!CompareAssemblyIdentity+0x97bcc
mscorwks.dll!CompareAssemblyIdentity+0x97877
mscorwks.dll!CreateAssemblyNameObject+0x2c29d
mscorwks.dll!InitializeFusion+0x5994
mscorwks.dll!GetMetaDataInternalInterfaceFromPublic+0x34ad9
mscorwks.dll!InitializeFusion+0x1282d
mscorwks.dll!StrongNameErrorInfo+0x130fe
mscorwks.dll!InitializeFusion+0xbf48
mscorwks.dll!StrongNameErrorInfo+0x76b0
kernel32.dll!BaseThreadInitThunk+0xd
ntdll.dll!RtlUserThreadStart+0x21
I was also able to reproduce the same problem on the other machine which has VS2010 deployed and was able to attach to “hanging” w3wp and saw that GC finalizer thread is sitting with a stacktrace similar to the one below:
[Native to Managed Transition]
System.Data.dll!System.Data.Odbc.OdbcHandle.ReleaseHandle()
[Native to Managed Transition]
[Managed to Native Transition]
mscorlib.dll!System.Runtime.InteropServices.SafeHandle.Dispose(bool disposing)
mscorlib.dll!System.Runtime.InteropServices.SafeHandle.Finalize()
[Appdomain Transition]
And one more stacktrace exists, which I believe is a reflection of the first one at the end.
my_SQLFreeStmtExtended strmov Highest
myodbc5.dll!strmov(char * dst, const char * src) Line 38 + 0xd bytes
myodbc5.dll!cli_safe_read(st_mysql * mysql) Line 731
myodbc5.dll!cli_read_query_result(st_mysql * mysql) Line 2739 + 0x8 bytes
myodbc5.dll!mysql_next_result(st_mysql * mysql) Line 5221 + 0xd bytes
myodbc5.dll!my_SQLFreeStmtExtended(void * hstmt, unsigned short fOption, unsigned int clearAllResults) Line 427 + 0xc bytes
odbc32.dll!000007fef1d43ee9()
[Frames below may be incorrect and/or missing, no symbols loaded for odbc32.dll]
odbc32.dll!000007fef1d43d3e()
mscorwks.dll!000007fef102cd27()
System.Data.ni.dll!000007feebe73ad3()
This problem obviously is also reproducible on production 16 cores machines.
We have performed code review and fixed/re-wrote code in a places where we potentially could have left connection not closed and not disposed.
I’ll appreciate any help with this issue and accept any advice at what else i can look at.
We’ve decided to start using mysql connector .net instead of mysql connector odbc. All we did is changed from OdbcConnection to MySqlConnection, from OdbcCommand to MySqlCommand, from OdbcParameter to MySqlParameter. We ran couple of ramp up tests against it and it looks like we gained not only stability but also improved performance.
So, as a conclusion: use MySql Connector .NET instead of MySql Connector ODBC.