Question

I originally asked this question on the Python capi-sig list: How to pass arguments to tp_new and tp_init from subtypes?

I’m reading the Python PEP-253 on subtyping and there are plenty of good recommendations on how to structure the types, call tp_new and tp_init slots, etc.

But, it lacks an important note on passing arguments from sub to super type.
It seems the PEP-253 is unfinished as per the note:

(XXX There should be a paragraph or two about argument passing here.)

So, I’m trying to extrapolate some strategies well known from the Python classes subtyping, especially techniques that each level strips-off arguments, etc.

I’m looking for techniques to achieve similar effect to this, but using plain Python C API (3.x):

class Shape:
    def __init__(self, shapename, **kwds):
        self.shapename = shapename
        super().__init__(**kwds)

class ColoredShape(Shape):
    def __init__(self, color, **kwds):
        self.color = color
        super().__init__(**kwds)

What would be the equivalent in Python C API?

How to deal with similar situation but with arguments specific to derived class expected in different order?
It is arguments given at the end of the args tuple (or kwds dict, I assume principle would be same).

Here is some (pseudo-)code that illustrates the situation:

class Base:
   def __init__(self, x, y, z):
      self.x = x
      self.y = y
      self.z = z

class Derived(Base):
   def __init__(self, x, y, a):
      self.a = a
      super().__init__(x, y, None):

Note, if the a was expected first:

Derived.__init__(self, a, x, y)

it would be similar situation to the Shape and ColoredShape above.
It would also be easier to deal with, I assume.

Could anyone help to figure out the missing XXX comment mentioned above and correct technique for passing arguments from subtype up to super type(s) on construction?

UPDATE 2012-07-17:

Inspired by ecatmur’s answer below I looked through Python 3 sources and I found defdict_init constructor of collections.defaultdict type object interesting. The type is derived from PyDictObject and its constructor takes additional argument of default_factory. The constructor signature in Python class is this:

class collections.defaultdict([default_factory[, ...]])

Now, here is how the default_factory is stripped from original args tuple, so the rest of arguments is forwarded to the tp_init of base type, it is PyDictObject:

int result;
PyObject *newargs;
Py_ssize_t n = PyTuple_GET_SIZE(args);
...
newargs = PySequence_GetSlice(args, 1, n);
...
result = PyDict_Type.tp_init(self, newargs, kwds);

Note, this snipped presence only the relevant part of the defdict_init function.

Answer 1

The problem is that PyArgs_ParseTupleAndKeywords doesn’t provide a way to extract extra *args and **kwargs from the input args and keywords; indeed, any extra arguments result in a TypeError; “Function takes %s %d positional arguments (%d given)”, or “‘%U’ is an invalid keyword argument for this function”.

This means that you’re going to have to parse args and keywords yourself; you’re guaranteed that args is a tuple and keywords is a dict, so you can use the standard methods (PyTuple_GET_ITEM and PyDict_GetItemString) to extract the arguments you’re interested in, and identify and construct a tuple and dict to pass on from the remainder. You obviously can’t modify args, because tuples are immutable; and while popping items from keywords should be OK it does seem a little risky (example crash).

A more ambitious but definitely feasible route would be to copy vgetargskeywords from getargs.c (http://hg.python.org/cpython/file/tip/Python/getargs.c) and extend it to take optional out-parameters for remainder *args and **kwargs. This should be fairly straightforward as you just need to modify the parts where it detects and throws TypeError on extra arguments (extra args; extra keywords). Good luck if you choose this route.