I am experiencing some stange problem in python extension in c++. I appreciate any help or suggestions.
To set context.
I am using embedded python in C++ to execute python scripts. I am also using python extension in c++ to give ability to python script to call the C++ function.
What is problem?
-
When I comment method entry from modules “method table” like I have shown below. Application never crashes or I am not experiencing segfault at all.
static PyMethodDef sa_methods[] = { //{"GetBlue",(PyCFunction)sa_GetBlue,METH_VARARGS,PyDoc_STR("fetches Blue color")}, //{"GetRed",(PyCFunction)sa_GetRed,METH_VARARGS,PyDoc_STR("fetches Red color")}, {"GetYellow",(PyCFunction)sa_GetYellow,METH_VARARGS,PyDoc_STR("fetches yellow color")}, {"GetPink",(PyCFunction)sa_GetPink,METH_VARARGS,PyDoc_STR("fetches pink color")}, {NULL, NULL, 0, NULL} }; -
When I remove method comments from the method entry from modules “method table” like I have shown below. Application always crashes in between processing 20000 to 50000 requests randomly.
static PyMethodDef sa_methods[] = { {"GetBlue",(PyCFunction)sa_GetBlue,METH_VARARGS,PyDoc_STR("fetches Blue color")}, {"GetRed",(PyCFunction)sa_GetRed,METH_VARARGS,PyDoc_STR("fetches Red color")}, {"GetYellow",(PyCFunction)sa_GetYellow,METH_VARARGS,PyDoc_STR("fetches yellow color")}, {"GetPink",(PyCFunction)sa_GetPink,METH_VARARGS,PyDoc_STR("fetches pink color")}, {NULL, NULL, 0, NULL} };
I have kept above two methods empty and they are merely returning Py_False.
Please find below the stack trace that I am getting every time.
#0 0x00002b700146c4d7 in PyNumber_CoerceEx (pv=0x41801798, pw=0x41801790) at Objects/object.c:1599
#1 0x00002b700142d6e3 in binary_op1 (v=0x2aaaac079600, w=0x2b7001750550, op_slot=16) at Objects/abstract.c:929
#2 0x00002b7001431e08 in PyNumber_Multiply (v=0x41801798, w=0x41801790) at Objects/abstract.c:1188
#3 0x00002b70014c4326 in PyEval_EvalFrameEx (f=0x8127780, throwflag=<value optimized out>) at Python/ceval.c:1118
#4 0x00002b70014c8493 in call_function (f=0x996e660, throwflag=<value optimized out>) at Python/ceval.c:3792
#5 PyEval_EvalFrameEx (f=0x996e660, throwflag=<value optimized out>) at Python/ceval.c:2389
#6 0x00002b70014c8493 in call_function (f=0x2aaac30aeb50, throwflag=<value optimized out>) at Python/ceval.c:3792
#7 PyEval_EvalFrameEx (f=0x2aaac30aeb50, throwflag=<value optimized out>) at Python/ceval.c:2389
#8 0x00002b70014c8d9f in PyEval_EvalCodeEx (co=0x2aaaad7dc558, globals=<value optimized out>, locals=<value optimized out>, args=0x2aaac6ab59a0, argcount=2,
kws=0x2aaac6ab59b0, kwcount=0, defs=0x7819128, defcount=1, closure=0x0) at Python/ceval.c:2968
#9 0x00002b70014c6df3 in call_function (f=0x2aaac6ab57d0, throwflag=<value optimized out>) at Python/ceval.c:3802
#10 PyEval_EvalFrameEx (f=0x2aaac6ab57d0, throwflag=<value optimized out>) at Python/ceval.c:2389
#11 0x00002b70014c8d9f in PyEval_EvalCodeEx (co=0x2aaabdf8e828, globals=<value optimized out>, locals=<value optimized out>, args=0x2aaac65bcbf0, argcount=2,
kws=0x2aaac65bcc00, kwcount=0, defs=0x7819c28, defcount=1, closure=0x0) at Python/ceval.c:2968
#12 0x00002b70014c6df3 in call_function (f=0x2aaac65bca40, throwflag=<value optimized out>) at Python/ceval.c:3802
#13 PyEval_EvalFrameEx (f=0x2aaac65bca40, throwflag=<value optimized out>) at Python/ceval.c:2389
#14 0x00002b70014c8493 in call_function (f=0x2aaac6301630, throwflag=<value optimized out>) at Python/ceval.c:3792
#15 PyEval_EvalFrameEx (f=0x2aaac6301630, throwflag=<value optimized out>) at Python/ceval.c:2389
#16 0x00002b70014c8d9f in PyEval_EvalCodeEx (co=0x6c7f558, globals=<value optimized out>, locals=<value optimized out>, args=0x2aaac55963e8, argcount=2,
kws=0x2aaac55963f8, kwcount=0, defs=0xd83fce8, defcount=1, closure=0x0) at Python/ceval.c:2968
#17 0x00002b70014c6df3 in call_function (f=0x2aaac5596240, throwflag=<value optimized out>) at Python/ceval.c:3802
#18 PyEval_EvalFrameEx (f=0x2aaac5596240, throwflag=<value optimized out>) at Python/ceval.c:2389
#19 0x00002b70014c8d9f in PyEval_EvalCodeEx (co=0x6c7fcd8, globals=<value optimized out>, locals=<value optimized out>, args=0x2aaac6fbc328, argcount=2,
kws=0x2aaac6fbc338, kwcount=0, defs=0xd83fc68, defcount=1, closure=0x0) at Python/ceval.c:2968
#20 0x00002b70014c6df3 in call_function (f=0x2aaac6fbc1a0, throwflag=<value optimized out>) at Python/ceval.c:3802
#21 PyEval_EvalFrameEx (f=0x2aaac6fbc1a0, throwflag=<value optimized out>) at Python/ceval.c:2389
#22 0x00002b70014c8d9f in PyEval_EvalCodeEx (co=0x6c7f648, globals=<value optimized out>, locals=<value optimized out>, args=0x1, argcount=0, kws=0x2aaac3e32558,
kwcount=0, defs=0xd83ffa8, defcount=1, closure=0x0) at Python/ceval.c:2968
#23 0x00002b70014c6df3 in call_function (f=0x2aaac3e323d0, throwflag=<value optimized out>) at Python/ceval.c:3802
#24 PyEval_EvalFrameEx (f=0x2aaac3e323d0, throwflag=<value optimized out>) at Python/ceval.c:2389
#25 0x00002b70014c8d9f in PyEval_EvalCodeEx (co=0x2aaaad7f70a8, globals=<value optimized out>, locals=<value optimized out>, args=0x0, argcount=0, kws=0x1325c6a8,
kwcount=0, defs=0x0, defcount=0, closure=0x2aaaad18e990) at Python/ceval.c:2968
#26 0x00002b70014c6df3 in call_function (f=0x1325c480, throwflag=<value optimized out>) at Python/ceval.c:3802
#27 PyEval_EvalFrameEx (f=0x1325c480, throwflag=<value optimized out>) at Python/ceval.c:2389
#28 0x00002b70014c8d9f in PyEval_EvalCodeEx (co=0x2aaaad7f77b0, globals=<value optimized out>, locals=<value optimized out>, args=0x1, argcount=1, kws=0x2aaac5ec1d08,
kwcount=0, defs=0x2aaaacfdf5e8, defcount=1, closure=0x0) at Python/ceval.c:2968
#29 0x00002b70014c6df3 in call_function (f=0x2aaac5ec1b00, throwflag=<value optimized out>) at Python/ceval.c:3802
#30 PyEval_EvalFrameEx (f=0x2aaac5ec1b00, throwflag=<value optimized out>) at Python/ceval.c:2389
#31 0x00002b70014c8493 in call_function (f=0xa926080, throwflag=<value optimized out>) at Python/ceval.c:3792
#32 PyEval_EvalFrameEx (f=0xa926080, throwflag=<value optimized out>) at Python/ceval.c:2389
#33 0x00002b70014c8493 in call_function (f=0x8e4bd40, throwflag=<value optimized out>) at Python/ceval.c:3792
#34 PyEval_EvalFrameEx (f=0x8e4bd40, throwflag=<value optimized out>) at Python/ceval.c:2389
#35 0x00002b70014c8d9f in PyEval_EvalCodeEx (co=0x2aaabc9f7648, globals=<value optimized out>, locals=<value optimized out>, args=0x9ff5f58, argcount=1,
---Type <return> to continue, or q <return> to quit---
kws=0x9ff5f60, kwcount=0, defs=0x0, defcount=0, closure=0x2aaaacfdfc50) at Python/ceval.c:2968
#36 0x00002b70014c6df3 in call_function (f=0x9ff5dc0, throwflag=<value optimized out>) at Python/ceval.c:3802
#37 PyEval_EvalFrameEx (f=0x9ff5dc0, throwflag=<value optimized out>) at Python/ceval.c:2389
#38 0x00002b70014c8493 in call_function (f=0xdb993e0, throwflag=<value optimized out>) at Python/ceval.c:3792
#39 PyEval_EvalFrameEx (f=0xdb993e0, throwflag=<value optimized out>) at Python/ceval.c:2389
#40 0x00002b70014c8d9f in PyEval_EvalCodeEx (co=0x2aaabd820990, globals=<value optimized out>, locals=<value optimized out>, args=0x2aaaacfdf8a8, argcount=1, kws=0x0,
kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2968
#41 0x00002b700145815d in function_call (func=0x7d898c0, arg=0x2aaaacfdf890, kw=0x0) at Objects/funcobject.c:524
#42 0x00002b700142d318 in PyObject_Call (func=0x7d898c0, arg=0x2aaaacfdf890, kw=0x0) at Objects/abstract.c:2492
#43 0x00002b700143cd7f in instancemethod_call (func=<value optimized out>, arg=0x2aaaacfdf890, kw=0x0) at Objects/classobject.c:2579
#44 0x00002b700142d318 in PyObject_Call (func=0x7554690, arg=0x2aaaac04d050, kw=0x0) at Objects/abstract.c:2492
#45 0x00002b70014c0e16 in PyEval_CallObjectWithKeywords (func=0x7554690, arg=0x2aaaac04d050, kw=0x0) at Python/ceval.c:3575
#46 0x00002b70014f9dcd in t_bootstrap (boot_raw=0x2aaac76e67c0) at ./Modules/threadmodule.c:425
#47 0x00000033c720677d in start_thread () from /lib64/libpthread.so.0
#48 0x00000033c66d33ed in clone () from /lib64/libc.so.6
I got solution for this problem which has ultimately gave me as well as my client huge relief.
And here is the reason and solution.
The functions “sa_GetBlue” and “sa_GetRed” was returning “Py_False”. Before returning “Py_False” or “Py_True” you need to increase the reference count of the variable holding these values otherwise It may corrupt memory state of the interpreter; it would not crash immediately but may crash at any point of time. In my case, It was crashing after processing 20000 to 50000 requests.
Please check below link to get more idea about what I am trying to explain.
why-does-python-keep-a-reference-count-on-false-and-true
@doomster and @Omnifarious; Thank you for your comment and direction that you have provided.