| Guido van Rossum | 1fb071c | 1997-08-25 21:36:44 +0000 | [diff] [blame^] | 1 | <HTML> |
| 2 | |
| 3 | <HEAD> |
| 4 | <TITLE>Metaprogramming in Python 1.5</TITLE> |
| 5 | </HEAD> |
| 6 | |
| 7 | <BODY BGCOLOR="FFFFFF"> |
| 8 | |
| 9 | <H1>Metaprogramming in Python 1.5</H1> |
| 10 | |
| 11 | <H4>XXX Don't link to this page! It is very much a work in progress.</H4> |
| 12 | |
| 13 | <P>While Python 1.5 is only out as a <A |
| 14 | HREF="http://grail.cnri.reston.va.us/python/1.5a3/">restricted alpha |
| 15 | release</A>, its metaprogramming feature is worth mentioning. |
| 16 | |
| 17 | <P>In previous Python releases (and still in 1.5), there is something |
| 18 | called the ``Don Beaudry hook'', after its inventor and champion. |
| 19 | This allows C extensions to provide alternate class behavior, thereby |
| 20 | allowing the Python class syntax to be used to define other class-like |
| 21 | entities. Don Beaudry has used this in his infamous <A |
| 22 | HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> package; Jim |
| 23 | Fulton has used it in his <A |
| 24 | HREF="http://www.digicool.com/papers/ExtensionClass.html">Extension |
| 25 | Classes</A> package. (It has also been referred to as the ``Don |
| 26 | Beaudry <i>hack</i>, but that's a misnomer. There's nothing hackish |
| 27 | about it -- in fact, it is rather elegant and deep, even though |
| 28 | there's something dark to it.) |
| 29 | |
| 30 | <P>Documentation of the Don Beaudry hook has purposefully been kept |
| 31 | minimal, since it is a feature of incredible power, and is easily |
| 32 | abused. Basically, it checks whether the <b>type of the base |
| 33 | class</b> is callable, and if so, it is called to create the new |
| 34 | class. |
| 35 | |
| 36 | <P>Note the two indirection levels. Take a simple example: |
| 37 | |
| 38 | <PRE> |
| 39 | class B: |
| 40 | pass |
| 41 | |
| 42 | class C(B): |
| 43 | pass |
| 44 | </PRE> |
| 45 | |
| 46 | Take a look at the second class definition, and try to fathom ``the |
| 47 | type of the base class is callable.'' |
| 48 | |
| 49 | <P>(Types are not classes, by the way. See questions 4.2, 4.19 and in |
| 50 | particular 6.22 in the <A |
| 51 | HREF="http://grail.cnri.reston.va.us/cgi-bin/faqw.py" >Python FAQ</A> |
| 52 | for more on this topic.) |
| 53 | |
| 54 | <P> |
| 55 | |
| 56 | <UL> |
| 57 | |
| 58 | <LI>The <b>base class</b> is B; this one's easy.<P> |
| 59 | |
| 60 | <LI>Since B is a class, its type is ``class''; so the <b>type of the |
| 61 | base class</b> is the type ``class''. This is also known as |
| 62 | types.ClassType, assuming the standard module <code>types</code> has |
| 63 | been imported.<P> |
| 64 | |
| 65 | <LI>Now is the type ``class'' <b>callable</b>? No, because types (in |
| 66 | core Python) are never callable. Classes are callable (calling a |
| 67 | class creates a new instance) but types aren't.<P> |
| 68 | |
| 69 | </UL> |
| 70 | |
| 71 | <P>So our conclusion is that in our example, the type of the base |
| 72 | class (of C) is not callable. So the Don Beaudry hook does not apply, |
| 73 | and the default class creation mechanism is used (which is also used |
| 74 | when there is no base class). In fact, the Don Beaudry hook never |
| 75 | applies when using only core Python, since the type of a core object |
| 76 | is never callable. |
| 77 | |
| 78 | <P>So what do Don and Jim do in order to use Don's hook? Write an |
| 79 | extension that defines at least two new Python object types. The |
| 80 | first would be the type for ``class-like'' objects usable as a base |
| 81 | class, to trigger Don's hook. This type must be made callable. |
| 82 | That's why we need a second type. Whether an object is callable |
| 83 | depends on its type. So whether a type object is callable depends on |
| 84 | <i>its</i> type, which is a <i>meta-type</i>. (In core Python there |
| 85 | is only one meta-type, the type ``type'' (types.TypeType), which is |
| 86 | the type of all type objects, even itself.) A new meta-type must |
| 87 | be defined that makes the type of the class-like objects callable. |
| 88 | (Normally, a third type would also be needed, the new ``instance'' |
| 89 | type, but this is not an absolute requirement -- the new class type |
| 90 | could return an object of some existing type when invoked to create an |
| 91 | instance.) |
| 92 | |
| 93 | <P>Still confused? Here's a simple device due to Don himself to |
| 94 | explain metaclasses. Take a simple class definition; assume B is a |
| 95 | special class that triggers Don's hook: |
| 96 | |
| 97 | <PRE> |
| 98 | class C(B): |
| 99 | a = 1 |
| 100 | b = 2 |
| 101 | </PRE> |
| 102 | |
| 103 | This can be though of as equivalent to: |
| 104 | |
| 105 | <PRE> |
| 106 | C = type(B)('C', (B,), {'a': 1, 'b': 2}) |
| 107 | </PRE> |
| 108 | |
| 109 | If that's too dense for you, here's the same thing written out using |
| 110 | temporary variables: |
| 111 | |
| 112 | <PRE> |
| 113 | creator = type(B) # The type of the base class |
| 114 | name = 'C' # The name of the new class |
| 115 | bases = (B,) # A tuple containing the base class(es) |
| 116 | namespace = {'a': 1, 'b': 2} # The namespace of the class statement |
| 117 | C = creator(name, bases, namespace) |
| 118 | </PRE> |
| 119 | |
| 120 | This is analogous to what happens without the Don Beaudry hook, except |
| 121 | that in that case the creator function is set to the default class |
| 122 | creator. |
| 123 | |
| 124 | <P>In either case, the creator is called with three arguments. The |
| 125 | first one, <i>name</i>, is the name of the new class (as given at the |
| 126 | top of the class statement). The <i>bases</i> argument is a tuple of |
| 127 | base classes (a singleton tuple if there's only one base class, like |
| 128 | the example). Finally, <i>namespace</i> is a dictionary containing |
| 129 | the local variables collected during execution of the class statement. |
| 130 | |
| 131 | <P>Note that the contents of the namespace dictionary is simply |
| 132 | whatever names were defined in the class statement. A little-known |
| 133 | fact is that when Python executes a class statement, it enters a new |
| 134 | local namespace, and all assignments and function definitions take |
| 135 | place in this namespace. Thus, after executing the following class |
| 136 | statement: |
| 137 | |
| 138 | <PRE> |
| 139 | class C: |
| 140 | a = 1 |
| 141 | def f(s): pass |
| 142 | </PRE> |
| 143 | |
| 144 | the class namespace's contents would be {'a': 1, 'f': <function f |
| 145 | ...>}. |
| 146 | |
| 147 | <P>But enough already about Python metaprogramming in C; read the |
| 148 | documentation of <A |
| 149 | HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> or <A |
| 150 | HREF="http://www.digicool.com/papers/ExtensionClass.html" >Extension |
| 151 | Classes</A> for more information. |
| 152 | |
| 153 | <H2>Writing Metaclasses in Python</H2> |
| 154 | |
| 155 | <P>In Python 1.5, the requirement to write a C extension in order to |
| 156 | engage in metaprogramming has been dropped (though you can still do |
| 157 | it, of course). In addition to the check ``is the type of the base |
| 158 | class callable,'' there's a check ``does the base class have a |
| 159 | __class__ attribute.'' If so, it is assumed that the __class__ |
| 160 | attribute refers to a class. |
| 161 | |
| 162 | <P>Let's repeat our simple example from above: |
| 163 | |
| 164 | <PRE> |
| 165 | class C(B): |
| 166 | a = 1 |
| 167 | b = 2 |
| 168 | </PRE> |
| 169 | |
| 170 | Assuming B has a __class__ attribute, this translates into: |
| 171 | |
| 172 | <PRE> |
| 173 | C = B.__class__('C', (B,), {'a': 1, 'b': 2}) |
| 174 | </PRE> |
| 175 | |
| 176 | This is exactly the same as before except that instead of type(B), |
| 177 | B.__class__ is invoked. If you have read <A HREF= |
| 178 | "http://grail.cnri.reston.va.us/cgi-bin/faqw.py?req=show&file=faq06.022.htp" |
| 179 | >FAQ question 6.22</A> you will understand that while there is a big |
| 180 | technical difference between type(B) and B.__class__, they play the |
| 181 | same role at different abstraction levels. And perhaps at some point |
| 182 | in the future they will really be the same thing (at which point you |
| 183 | would be able to derive subclasses from built-in types). |
| 184 | |
| 185 | <P>Going back to the example, the class B.__class__ is instantiated, |
| 186 | passing its constructor the same three arguments that are passed to |
| 187 | the default class constructor or to an extension's metaprogramming |
| 188 | code: <i>name</i>, <i>bases</i>, and <i>namespace</i>. |
| 189 | |
| 190 | <P>It is easy to be confused by what exactly happens when using a |
| 191 | metaclass, because we lose the absolute distinction between classes |
| 192 | and instances: a class is an instance of a metaclass (a |
| 193 | ``metainstance''), but technically (i.e. in the eyes of the python |
| 194 | runtime system), the metaclass is just a class, and the metainstance |
| 195 | is just an instance. At the end of the class statement, the metaclass |
| 196 | whose metainstance is used as a base class is instantiated, yielding a |
| 197 | second metainstance (of the same metaclass). This metainstance is |
| 198 | then used as a (normal, non-meta) class; instantiation of the class |
| 199 | means calling the metainstance, and this will return a real instance. |
| 200 | And what class is that an instance of? Conceptually, it is of course |
| 201 | an instance of our metainstance; but in most cases the Python runtime |
| 202 | system will see it as an instance of a a helper class used by the |
| 203 | metaclass to implement its (non-meta) instances... |
| 204 | |
| 205 | <P>Hopefully an example will make things clearer. Let's presume we |
| 206 | have a metaclass MetaClass1. It's helper class (for non-meta |
| 207 | instances) is callled HelperClass1. We now (manually) instantiate |
| 208 | MetaClass1 once to get an empty special base class: |
| 209 | |
| 210 | <PRE> |
| 211 | BaseClass1 = MetaClass1("BaseClass1", (), {}) |
| 212 | </PRE> |
| 213 | |
| 214 | We can now use BaseClass1 as a base class in a class statement: |
| 215 | |
| 216 | <PRE> |
| 217 | class MySpecialClass(BaseClass1): |
| 218 | i = 1 |
| 219 | def f(s): pass |
| 220 | </PRE> |
| 221 | |
| 222 | At this point, MySpecialClass is defined; it is a metainstance of |
| 223 | MetaClass1 just like BaseClass1, and in fact the expression |
| 224 | ``BaseClass1.__class__ == MySpecialClass.__class__ == MetaClass1'' |
| 225 | yields true. |
| 226 | |
| 227 | <P>We are now ready to create instances of MySpecialClass. Let's |
| 228 | assume that no constructor arguments are required: |
| 229 | |
| 230 | <PRE> |
| 231 | x = MySpecialClass() |
| 232 | y = Myspecialclass() |
| 233 | print x.__class__, y.__class__ |
| 234 | </PRE> |
| 235 | |
| 236 | The print statement shows that x and y are instances of HelperClass1. |
| 237 | How did this happen? MySpecialClass is an instance of MetaClass1 |
| 238 | (``meta'' is irrelevant here); when an instance is called, its |
| 239 | __call__ method is invoked, and presumably the __call__ method defined |
| 240 | by MetaClass1 returns an instance of HelperClass1. |
| 241 | |
| 242 | <P>Now let's see how we could use metaprogramming -- what can we do |
| 243 | with metaclasses that we can't easily do without them? Here's one |
| 244 | idea: a metaclass could automatically insert trace calls for all |
| 245 | method calls. Let's first develop a simplified example, without |
| 246 | support for inheritance or other ``advanced'' Python features (we'll |
| 247 | add those later). |
| 248 | |
| 249 | <PRE> |
| 250 | import types |
| 251 | |
| 252 | class Tracing: |
| 253 | def __init__(self, name, bases, namespace): |
| 254 | """Create a new class.""" |
| 255 | self.__name__ = name |
| 256 | self.__bases__ = bases |
| 257 | self.__namespace__ = namespace |
| 258 | def __call__(self): |
| 259 | """Create a new instance.""" |
| 260 | return Instance(self) |
| 261 | |
| 262 | class Instance: |
| 263 | def __init__(self, klass): |
| 264 | self.__klass__ = klass |
| 265 | def __getattr__(self, name): |
| 266 | try: |
| 267 | value = self.__klass__.__namespace__[name] |
| 268 | except KeyError: |
| 269 | raise AttributeError, name |
| 270 | if type(value) is not types.FuncType: |
| 271 | return value |
| 272 | return BoundMethod(value, self) |
| 273 | |
| 274 | class BoundMethod: |
| 275 | def __init__(self, function, instance): |
| 276 | self.function = function |
| 277 | self.instance = instance |
| 278 | def __call__(self, *args): |
| 279 | print "calling", self.function, "for", instance, "with", args |
| 280 | return apply(self.function, (self.instance,) + args) |
| 281 | <HR> |
| 282 | |
| 283 | Confused already? |
| 284 | |
| 285 | |
| 286 | <P>XXX More text is needed here. For now, have a look at some very |
| 287 | preliminary examples that I coded up to teach myself how to use this |
| 288 | feature: |
| 289 | |
| 290 | |
| 291 | |
| 292 | <H2>Real-life Examples</H2> |
| 293 | |
| 294 | <DL> |
| 295 | |
| 296 | <DT><A HREF="Enum.py">Enum.py</A> |
| 297 | |
| 298 | <DD>This (ab)uses the class syntax as an elegant way to define |
| 299 | enumerated types. The resulting classes are never instantiated -- |
| 300 | rather, their class attributes are the enumerated values. For |
| 301 | example: |
| 302 | |
| 303 | <PRE> |
| 304 | class Color(Enum): |
| 305 | red = 1 |
| 306 | green = 2 |
| 307 | blue = 3 |
| 308 | print Color.red |
| 309 | </PRE> |
| 310 | |
| 311 | will print the string ``Color.red'', while ``Color.red==1'' is true, |
| 312 | and ``Color.red + 1'' raise a TypeError exception. |
| 313 | |
| 314 | <P> |
| 315 | |
| 316 | <DT><A HREF="Trace.py">Trace.py</A> |
| 317 | |
| 318 | <DD>The resulting classes work much like standard classes, but by |
| 319 | setting a special class or instance attribute __trace_output__ to |
| 320 | point to a file, all calls to the class's methods are traced. It was |
| 321 | a bit of a struggle to get this right. This should probably redone |
| 322 | using the generic metaclass below. |
| 323 | |
| 324 | <P> |
| 325 | |
| 326 | <DT><A HREF="Meta.py">Meta.py</A> |
| 327 | |
| 328 | <DD>A generic metaclass. This is an attempt at finding out how much |
| 329 | standard class behavior can be mimicked by a metaclass. The |
| 330 | preliminary answer appears to be that everything's fine as long as the |
| 331 | class (or its clients) don't look at the instance's __class__ |
| 332 | attribute, nor at the class's __dict__ attribute. The use of |
| 333 | __getattr__ internally makes the classic implementation of __getattr__ |
| 334 | hooks tough; we provide a similar hook _getattr_ instead. |
| 335 | (__setattr__ and __delattr__ are not affected.) |
| 336 | (XXX Hm. Could detect presence of __getattr__ and rename it.) |
| 337 | |
| 338 | <P> |
| 339 | |
| 340 | <DT><A HREF="Eiffel.py">Eiffel.py</A> |
| 341 | |
| 342 | <DD>Uses the above generic metaclass to implement Eiffel style |
| 343 | pre-conditions and post-conditions. |
| 344 | |
| 345 | <P> |
| 346 | </DL> |
| 347 | |
| 348 | </BODY> |
| 349 | |
| 350 | </HTML> |