A couple of months ago I was working with Pybind11, a library to create Python bindings for C++ code, and I found it interesting the way they use variadic templates to declare the bindings. Here is a snippet:
1#include <pybind11/pybind11>
2
3using namespace py = pybind11;
4
5/// Function that adds two values. The flag controls whether the second value
6/// should be inverted before adding.
7int add(int lhs, int rhs, bool invert_rhs) {
8 return (invert_rhs) ? lhs - rhs : lhs + rhs;
9}
10
11PYBIND11_MODULE(mylib, m) {
12 m.def("add", add,
13 py::arg("lhs"), py::arg("rhs"), py::kw_only(), py::arg("sub"));
14}
First we define a module using the macro PYBIND11_MODULE that is called mylib and can be manipulated via the variable m. With the module variable, we can define bindings for functions and classes. In the latter case, we call the def method passing information about the function: its name; the function pointer to be called; the name of the arguments; and the special marker py::kw_only that separates positional vs keyword arguments in Python.
What I found most interesting about this method is that you can simply throw parameters at it of different types, in any order and it can handle them naturally. Surely this have to do with variadic templates, but that alone does not explain everything. Let's find out.
First of all, let's check the definition of the method module_::def. As expected it is a template method that builds a cpp_function and adds it to the module namespace. We are looking for how the extra parameters are handled inside cpp_function.
1class module_ : public object {
2 /// Create Python binding for a new function within the module scope.
3 template <typename Func, typename... Extra>
4 module_ &def(const char *name_, Func &&f, const Extra &... extra) {
5 cpp_function func(std::forward<Func>(f), name(name_), /*...*/, extra...);
6 add_object(name_, func, /*...*/);
7 return *this;
8 }
9};
Looking into the constructor of cpp_function, we find a call to the true internal constructor called initialize. Skimming through this function I found a suspicious process_attributes template instantiated with the Extra types.
1class cpp_function : public function {
2 template <typename Func, typename Return, typename... Args, typename... Extra>
3 void initialize(Func &&f, Return(*)(Args...), const Extra &... extra) {
4 // ...
5 process_attributes<Extra...>::init(extra..., rec);
6 // ...
7 }
8};
At this point it is important to pause and think about what do we expect from this call. Since the def call can have an arbitrary number of extra arguments, I expect to see some kind of iteration of each argument and a dispatch for the corresponding type. For example, we would like to handle the py::kw_only() differently than, say, py::arg("lhs"). This is exactly what process_attributes::init does, albeit in a more convoluted manner.
The init function itself takes the extra arguments as input and also an additional function_record which serves as a context for the cpp_function being constructed. For each argument type we would like to interact differently with this function record.
1template <typename... Args>
2struct process_attributes {
3 static void init(const Args &... args, function_record *r) {
4 using expander = int[];
5 (void) expander {
6 0,
7 ((void) process_attribute<typename std::decay<Arg>::type>::init(args, r), 0)...
8 };
9 }
10};
In the body of this method, it first creates a type alias called expander to be an array of integers (int[]). This is optional, but makes the code a tiny bit more readable. Then, it creates a literal array of integers but does not assign it to anything. This is intentional because we just want to use some shenanigans to expand all of our template parameters. The first element of the array is 0 but the second element uses the comma-operator.
The syntax looks something like this:
1((void) func(), 0)
The left operand of the parenthesized expression calls a function and casts the return to void, effectively making the compiler discard the value such that the whole expression evaluates to simply 0. But that is not all. Notice the ellipsis ... after the second array element? This "expansion" will be done for each template argument, essentially creating an array full of zeroes. But we are not interested in the resulting array. We are just abusing the C++ syntax to call a function for each variadic argument.
Instead of a plain function though, the method init of the templated class process_attribute is called. ATTRIBUTE. Not attributes. Mind the difference. This template will basically handle the dispatch for us. For each type we would like to accept in our module_::def method, we should specialize the process_attribute template, like so:
1template <typename T, typename SFINAE = void>
2struct process_attribute;
3
4template<>
5struct process_attribute<arg> {
6 static void init(const arg value, function_record *r) {
7 // Handle the argument of type `py::arg` here...
8 }
9};
10
11template<>
12struct process_attribute<kw_only> {
13 static void init(const kw_only &, function_record *r) {
14 // Handle the argument of type `py::kw_only` here...
15 }
16};
Finally, as the last piece of the puzzle is the std::decay<Arg> trait. You see, as far as template parameters go, an argument of type A and another of type const A are different. But in this case, it does not matter for us, we would like to process them the same. Therefore before tapping into the process_attribute template, we decay the type to remove all qualifiers in front of it and process the unadorned result.
Conclusions #
Variadic templates are a scary feature of C++ for the uninitiated but it allows to do very powerful things at compile time. Yes, everything I mentioned in this post happens at compile time! Variadic templates, the expander snippet together with the full specialization of the process_attribute allows compile time dispatch, creating a very flexible user-facing API. This is so dope!
Acknowledgements #
Thanks to Rodrigo Ceccato de Freitas for reading an early draft of this post and suggesting some changes.