Gitiles
Code Review Sign In
gerrit-public.fairphone.software / platform / external / python / cpython3 / 7b877a9278b5e8a3dcac5d3a485ed657ac8815fe / . / Demo / metaclasses / index.html
blob: ee91f8f1a22d16f376197f5a9545374e955182de [file] [log] [blame]
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]1<HTML>
2
3<HEAD>
4<TITLE>Metaprogramming in Python 1.5</TITLE>
5</HEAD>
6
7<BODY BGCOLOR="FFFFFF">
8
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]9<H1>Metaprogramming in Python 1.5 (DRAFT)</H1>
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]10
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]11<H4>XXX This is very much a work in progress.</H4>
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]12
13<P>While Python 1.5 is only out as a <A
14HREF="http://grail.cnri.reston.va.us/python/1.5a3/">restricted alpha
15release</A>, its metaprogramming feature is worth mentioning.
16
17<P>In previous Python releases (and still in 1.5), there is something
18called the ``Don Beaudry hook'', after its inventor and champion.
19This allows C extensions to provide alternate class behavior, thereby
20allowing the Python class syntax to be used to define other class-like
21entities. Don Beaudry has used this in his infamous <A
22HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> package; Jim
23Fulton has used it in his <A
24HREF="http://www.digicool.com/papers/ExtensionClass.html">Extension
25Classes</A> package. (It has also been referred to as the ``Don
Guido van Rossum7b877a91997-09-08 02:20:57 +0000[diff] [blame^]26Beaudry <i>hack</i>,'' but that's a misnomer. There's nothing hackish
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]27about it -- in fact, it is rather elegant and deep, even though
28there's something dark to it.)
29
30<P>Documentation of the Don Beaudry hook has purposefully been kept
31minimal, since it is a feature of incredible power, and is easily
32abused. Basically, it checks whether the <b>type of the base
33class</b> is callable, and if so, it is called to create the new
34class.
35
36<P>Note the two indirection levels. Take a simple example:
37
38<PRE>
39class B:
40 pass
41
42class C(B):
43 pass
44</PRE>
45
46Take a look at the second class definition, and try to fathom ``the
47type of the base class is callable.''
48
49<P>(Types are not classes, by the way. See questions 4.2, 4.19 and in
50particular 6.22 in the <A
51HREF="http://grail.cnri.reston.va.us/cgi-bin/faqw.py" >Python FAQ</A>
52for more on this topic.)
53
54<P>
55
56<UL>
57
58<LI>The <b>base class</b> is B; this one's easy.<P>
59
60<LI>Since B is a class, its type is ``class''; so the <b>type of the
61base class</b> is the type ``class''. This is also known as
62types.ClassType, assuming the standard module <code>types</code> has
63been imported.<P>
64
65<LI>Now is the type ``class'' <b>callable</b>? No, because types (in
66core Python) are never callable. Classes are callable (calling a
67class creates a new instance) but types aren't.<P>
68
69</UL>
70
71<P>So our conclusion is that in our example, the type of the base
72class (of C) is not callable. So the Don Beaudry hook does not apply,
73and the default class creation mechanism is used (which is also used
74when there is no base class). In fact, the Don Beaudry hook never
75applies when using only core Python, since the type of a core object
76is never callable.
77
78<P>So what do Don and Jim do in order to use Don's hook? Write an
79extension that defines at least two new Python object types. The
80first would be the type for ``class-like'' objects usable as a base
81class, to trigger Don's hook. This type must be made callable.
82That's why we need a second type. Whether an object is callable
83depends on its type. So whether a type object is callable depends on
84<i>its</i> type, which is a <i>meta-type</i>. (In core Python there
85is only one meta-type, the type ``type'' (types.TypeType), which is
86the type of all type objects, even itself.) A new meta-type must
87be defined that makes the type of the class-like objects callable.
88(Normally, a third type would also be needed, the new ``instance''
89type, but this is not an absolute requirement -- the new class type
90could return an object of some existing type when invoked to create an
91instance.)
92
93<P>Still confused? Here's a simple device due to Don himself to
94explain metaclasses. Take a simple class definition; assume B is a
95special class that triggers Don's hook:
96
97<PRE>
98class C(B):
99 a = 1
100 b = 2
101</PRE>
102
103This can be though of as equivalent to:
104
105<PRE>
106C = type(B)('C', (B,), {'a': 1, 'b': 2})
107</PRE>
108
109If that's too dense for you, here's the same thing written out using
110temporary variables:
111
112<PRE>
113creator = type(B) # The type of the base class
114name = 'C' # The name of the new class
115bases = (B,) # A tuple containing the base class(es)
116namespace = {'a': 1, 'b': 2} # The namespace of the class statement
117C = creator(name, bases, namespace)
118</PRE>
119
120This is analogous to what happens without the Don Beaudry hook, except
121that in that case the creator function is set to the default class
122creator.
123
124<P>In either case, the creator is called with three arguments. The
125first one, <i>name</i>, is the name of the new class (as given at the
126top of the class statement). The <i>bases</i> argument is a tuple of
127base classes (a singleton tuple if there's only one base class, like
128the example). Finally, <i>namespace</i> is a dictionary containing
129the local variables collected during execution of the class statement.
130
131<P>Note that the contents of the namespace dictionary is simply
132whatever names were defined in the class statement. A little-known
133fact is that when Python executes a class statement, it enters a new
134local namespace, and all assignments and function definitions take
135place in this namespace. Thus, after executing the following class
136statement:
137
138<PRE>
139class C:
140 a = 1
141 def f(s): pass
142</PRE>
143
144the class namespace's contents would be {'a': 1, 'f': &lt;function f
145...&gt;}.
146
147<P>But enough already about Python metaprogramming in C; read the
148documentation of <A
149HREF="http://maigret.cog.brown.edu/pyutil/">MESS</A> or <A
150HREF="http://www.digicool.com/papers/ExtensionClass.html" >Extension
151Classes</A> for more information.
152
153<H2>Writing Metaclasses in Python</H2>
154
155<P>In Python 1.5, the requirement to write a C extension in order to
156engage in metaprogramming has been dropped (though you can still do
157it, of course). In addition to the check ``is the type of the base
158class callable,'' there's a check ``does the base class have a
159__class__ attribute.'' If so, it is assumed that the __class__
160attribute refers to a class.
161
162<P>Let's repeat our simple example from above:
163
164<PRE>
165class C(B):
166 a = 1
167 b = 2
168</PRE>
169
170Assuming B has a __class__ attribute, this translates into:
171
172<PRE>
173C = B.__class__('C', (B,), {'a': 1, 'b': 2})
174</PRE>
175
176This is exactly the same as before except that instead of type(B),
177B.__class__ is invoked. If you have read <A HREF=
178"http://grail.cnri.reston.va.us/cgi-bin/faqw.py?req=show&file=faq06.022.htp"
179>FAQ question 6.22</A> you will understand that while there is a big
180technical difference between type(B) and B.__class__, they play the
181same role at different abstraction levels. And perhaps at some point
182in the future they will really be the same thing (at which point you
183would be able to derive subclasses from built-in types).
184
Guido van Rossum7b877a91997-09-08 02:20:57 +0000[diff] [blame^]185<P>At this point it may be worth mentioning that C.__class__ is the
186same object as B.__class__, i.e., C's metaclass is the same as B's
187metaclass. In other words, subclassing an existing class creates a
188new (meta)inststance of the base class's metaclass.
189
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]190<P>Going back to the example, the class B.__class__ is instantiated,
191passing its constructor the same three arguments that are passed to
192the default class constructor or to an extension's metaprogramming
193code: <i>name</i>, <i>bases</i>, and <i>namespace</i>.
194
195<P>It is easy to be confused by what exactly happens when using a
196metaclass, because we lose the absolute distinction between classes
197and instances: a class is an instance of a metaclass (a
198``metainstance''), but technically (i.e. in the eyes of the python
199runtime system), the metaclass is just a class, and the metainstance
200is just an instance. At the end of the class statement, the metaclass
201whose metainstance is used as a base class is instantiated, yielding a
202second metainstance (of the same metaclass). This metainstance is
203then used as a (normal, non-meta) class; instantiation of the class
204means calling the metainstance, and this will return a real instance.
205And what class is that an instance of? Conceptually, it is of course
206an instance of our metainstance; but in most cases the Python runtime
207system will see it as an instance of a a helper class used by the
208metaclass to implement its (non-meta) instances...
209
210<P>Hopefully an example will make things clearer. Let's presume we
211have a metaclass MetaClass1. It's helper class (for non-meta
212instances) is callled HelperClass1. We now (manually) instantiate
213MetaClass1 once to get an empty special base class:
214
215<PRE>
216BaseClass1 = MetaClass1("BaseClass1", (), {})
217</PRE>
218
219We can now use BaseClass1 as a base class in a class statement:
220
221<PRE>
222class MySpecialClass(BaseClass1):
223 i = 1
224 def f(s): pass
225</PRE>
226
227At this point, MySpecialClass is defined; it is a metainstance of
228MetaClass1 just like BaseClass1, and in fact the expression
229``BaseClass1.__class__ == MySpecialClass.__class__ == MetaClass1''
230yields true.
231
232<P>We are now ready to create instances of MySpecialClass. Let's
233assume that no constructor arguments are required:
234
235<PRE>
236x = MySpecialClass()
Guido van Rossum7b877a91997-09-08 02:20:57 +0000[diff] [blame^]237y = MySpecialClass()
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]238print x.__class__, y.__class__
239</PRE>
240
241The print statement shows that x and y are instances of HelperClass1.
242How did this happen? MySpecialClass is an instance of MetaClass1
243(``meta'' is irrelevant here); when an instance is called, its
244__call__ method is invoked, and presumably the __call__ method defined
245by MetaClass1 returns an instance of HelperClass1.
246
247<P>Now let's see how we could use metaprogramming -- what can we do
248with metaclasses that we can't easily do without them? Here's one
249idea: a metaclass could automatically insert trace calls for all
250method calls. Let's first develop a simplified example, without
251support for inheritance or other ``advanced'' Python features (we'll
252add those later).
253
254<PRE>
255import types
256
257class Tracing:
258 def __init__(self, name, bases, namespace):
259 """Create a new class."""
260 self.__name__ = name
261 self.__bases__ = bases
262 self.__namespace__ = namespace
263 def __call__(self):
264 """Create a new instance."""
265 return Instance(self)
266
267class Instance:
268 def __init__(self, klass):
269 self.__klass__ = klass
270 def __getattr__(self, name):
271 try:
272 value = self.__klass__.__namespace__[name]
273 except KeyError:
274 raise AttributeError, name
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]275 if type(value) is not types.FunctionType:
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]276 return value
277 return BoundMethod(value, self)
278
279class BoundMethod:
280 def __init__(self, function, instance):
281 self.function = function
282 self.instance = instance
283 def __call__(self, *args):
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]284 print "calling", self.function, "for", self.instance, "with", args
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]285 return apply(self.function, (self.instance,) + args)
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]286
287Trace = Tracing('Trace', (), {})
288
289class MyTracedClass(Trace):
290 def method1(self, a):
291 self.a = a
292 def method2(self):
293 return self.a
294
295aninstance = MyTracedClass()
296
297aninstance.method1(10)
298
299print "the answer is %d" % aninstance.method2()
300</PRE>
301
302Confused already? The intention is to read this from top down. The
303Tracing class is the metaclass we're defining. Its structure is
304really simple.
305
306<P>
307
308<UL>
309
310<LI>The __init__ method is invoked when a new Tracing instance is
311created, e.g. the definition of class MyTracedClass later in the
312example. It simply saves the class name, base classes and namespace
313as instance variables.<P>
314
315<LI>The __call__ method is invoked when a Tracing instance is called,
316e.g. the creation of aninstance later in the example. It returns an
317instance of the class Instance, which is defined next.<P>
318
319</UL>
320
321<P>The class Instance is the class used for all instances of classes
322built using the Tracing metaclass, e.g. aninstance. It has two
323methods:
324
325<P>
326
327<UL>
328
329<LI>The __init__ method is invoked from the Tracing.__call__ method
330above to initialize a new instance. It saves the class reference as
331an instance variable. It uses a funny name because the user's
332instance variables (e.g. self.a later in the example) live in the same
333namespace.<P>
334
335<LI>The __getattr__ method is invoked whenever the user code
336references an attribute of the instance that is not an instance
337variable (nor a class variable; but except for __init__ and
338__getattr__ there are no class variables). It will be called, for
339example, when aninstance.method1 is referenced in the example, with
340self set to aninstance and name set to the string "method1".<P>
341
342</UL>
343
344<P>The __getattr__ method looks the name up in the __namespace__
345dictionary. If it isn't found, it raises an AttributeError exception.
346(In a more realistic example, it would first have to look through the
347base classes as well.) If it is found, there are two possibilities:
348it's either a function or it isn't. If it's not a function, it is
349assumed to be a class variable, and its value is returned. If it's a
350function, we have to ``wrap'' it in instance of yet another helper
351class, BoundMethod.
352
353<P>The BoundMethod class is needed to implement a familiar feature:
354when a method is defined, it has an initial argument, self, which is
355automatically bound to the relevant instance when it is called. For
356example, aninstance.method1(10) is equivalent to method1(aninstance,
35710). In the example if this call, first a temporary BoundMethod
358instance is created with the following constructor call: temp =
359BoundMethod(method1, aninstance); then this instance is called as
360temp(10). After the call, the temporary instance is discarded.
361
362<P>
363
364<UL>
365
366<LI>The __init__ method is invoked for the constructor call
367BoundMethod(method1, aninstance). It simply saves away its
368arguments.<P>
369
370<LI>The __call__ method is invoked when the bound method instance is
371called, as in temp(10). It needs to call method1(aninstance, 10).
372However, even though self.function is now method1 and self.instance is
373aninstance, it can't call self.function(self.instance, args) directly,
374because it should work regardless of the number of arguments passed.
375(For simplicity, support for keyword arguments has been omitted.)<P>
376
377</UL>
378
379<P>In order to be able to support arbitrary argument lists, the
380__call__ method first constructs a new argument tuple. Conveniently,
381because of the notation *args in __call__'s own argument list, the
382arguments to __call__ (except for self) are placed in the tuple args.
383To construct the desired argument list, we concatenate a singleton
384tuple containing the instance with the args tuple: (self.instance,) +
385args. (Note the trailing comma used to construct the singleton
386tuple.) In our example, the resulting argument tuple is (aninstance,
38710).
388
389<P>The intrinsic function apply() takes a function and an argument
390tuple and calls the function for it. In our example, we are calling
391apply(method1, (aninstance, 10)) which is equivalent to calling
392method(aninstance, 10).
393
394<P>From here on, things should come together quite easily. The output
395of the example code is something like this:
396
397<PRE>
398calling <function method1 at ae8d8> for <Instance instance at 95ab0> with (10,)
399calling <function method2 at ae900> for <Instance instance at 95ab0> with ()
400the answer is 10
401</PRE>
402
403<P>That was about the shortest meaningful example that I could come up
404with. A real tracing metaclass (for example, <A
405HREF="#Trace">Trace.py</A> discussed below) needs to be more
406complicated in two dimensions.
407
408<P>First, it needs to support more advanced Python features such as
409class variables, inheritance, __init__ methods, and keyword arguments.
410
411<P>Second, it needs to provide a more flexible way to handle the
412actual tracing information; perhaps it should be possible to write
413your own tracing function that gets called, perhaps it should be
414possible to enable and disable tracing on a per-class or per-instance
415basis, and perhaps a filter so that only interesting calls are traced;
416it should also be able to trace the return value of the call (or the
417exception it raised if an error occurs). Even the Trace.py example
418doesn't support all these features yet.
419
420<P>
421
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]422<HR>
423
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]424<H1>Real-life Examples</H1>
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]425
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]426<P>Have a look at some very preliminary examples that I coded up to
427teach myself how to use metaprogramming:
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]428
429<DL>
430
431<DT><A HREF="Enum.py">Enum.py</A>
432
433<DD>This (ab)uses the class syntax as an elegant way to define
434enumerated types. The resulting classes are never instantiated --
435rather, their class attributes are the enumerated values. For
436example:
437
438<PRE>
439class Color(Enum):
440 red = 1
441 green = 2
442 blue = 3
443print Color.red
444</PRE>
445
446will print the string ``Color.red'', while ``Color.red==1'' is true,
447and ``Color.red + 1'' raise a TypeError exception.
448
449<P>
450
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]451<DT><A NAME=Trace></A><A HREF="Trace.py">Trace.py</A>
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]452
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]453<DD>The resulting classes work much like standard
454classes, but by setting a special class or instance attribute
455__trace_output__ to point to a file, all calls to the class's methods
456are traced. It was a bit of a struggle to get this right. This
457should probably redone using the generic metaclass below.
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]458
459<P>
460
461<DT><A HREF="Meta.py">Meta.py</A>
462
463<DD>A generic metaclass. This is an attempt at finding out how much
464standard class behavior can be mimicked by a metaclass. The
465preliminary answer appears to be that everything's fine as long as the
466class (or its clients) don't look at the instance's __class__
467attribute, nor at the class's __dict__ attribute. The use of
468__getattr__ internally makes the classic implementation of __getattr__
469hooks tough; we provide a similar hook _getattr_ instead.
470(__setattr__ and __delattr__ are not affected.)
471(XXX Hm. Could detect presence of __getattr__ and rename it.)
472
473<P>
474
475<DT><A HREF="Eiffel.py">Eiffel.py</A>
Guido van Rossum7b877a91997-09-08 02:20:57 +0000[diff] [blame^]476
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]477<DD>Uses the above generic metaclass to implement Eiffel style
478pre-conditions and post-conditions.
479
480<P>
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]481
482<DT><A HREF="Synch.py">Synch.py</A>
483
484<DD>Uses the above generic metaclass to implement synchronized
485methods.
486
487<P>
488
Guido van Rossum7b877a91997-09-08 02:20:57 +0000[diff] [blame^]489<DT><A HREF="Simple.py">Simple.py</A>
490
491<DD>The example module used above.
492
493<P>
494
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]495</DL>
496
Guido van Rossum0cdb8871997-08-26 00:08:51 +0000[diff] [blame]497<P>A pattern seems to be emerging: almost all these uses of
498metaclasses (except for Enum, which is probably more cute than useful)
499mostly work by placing wrappers around method calls. An obvious
500problem with that is that it's not easy to combine the features of
501different metaclasses, while this would actually be quite useful: for
502example, I wouldn't mind getting a trace from the test run of the
503Synch module, and it would be interesting to add preconditions to it
504as well. This needs more research. Perhaps a metaclass could be
505provided that allows stackable wrappers...
506
Guido van Rossum7b877a91997-09-08 02:20:57 +0000[diff] [blame^]507<HR>
508
509<H2>Things You Could Do With Metaclasses</H2>
510
511<P>There are lots of things you could do with metaclasses. Most of
512these can also be done with creative use of __getattr__, but
513metaclasses make it easier to modify the attribute lookup behavior of
514classes. Here's a partial list.
515
516<P>
517
518<UL>
519
520<LI>Enforce different inheritance semantics, e.g. automatically call
521base class methods when a derived class overrides<P>
522
523<LI>Implement class methods (e.g. if the first argument is not named
524'self')<P>
525
526<LI>Implement that each instance is initialized with <b>copies</b> of
527all class variables<P>
528
529<LI>Implement a different way to store instance variables (e.g. in a
530list kept outside the the instance but indexed by the instance's id())<P>
531
532<LI>Automatically wrap or trap all or certain methods
533
534<UL>
535
536<LI>for tracing
537
538<LI>for precondition and postcondition checking
539
540<LI>for synchronized methods
541
542<LI>for automatic value caching
543
544</UL>
545<P>
546
547<LI>When an attribute is a parameterless function, call it on
548reference (to mimic it being an instance variable); same on assignment<P>
549
550<LI>Instrumentation: see how many times various attributes are used<P>
551
552<LI>Different semantics for __setattr__ and __getattr__ (e.g. disable
553them when they are being used recursively)<P>
554
555<LI>Abuse class syntax for other things<P>
556
557<LI>Experiment with automatic type checking<P>
558
559<LI>Delegation (or acquisition)<P>
560
561<LI>Dynamic inheritance patterns<P>
562
563<LI>Automatic caching of methods<P>
564
565</UL>
566
567<P>
568
569<HR>
570
571<H4>Credits</H4>
572
573<P>Many thanks to David Ascher and Donald Beaudry for their comments
574on earlier draft of this paper. Also thanks to Matt Conway and Tommy
575Burnette for putting a seed for the idea of metaclasses in my
576mind, nearly three years ago, even though at the time my response was
577``you can do that with __getattr__ hooks...'' :-)
578
579<P>
580
581<HR>
582
Guido van Rossum1fb071c1997-08-25 21:36:44 +0000[diff] [blame]583</BODY>
584
585</HTML>