Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 1 | ************************* |
| 2 | Python Advocacy HOWTO |
| 3 | ************************* |
| 4 | |
| 5 | :Author: A.M. Kuchling |
| 6 | :Release: 0.03 |
| 7 | |
| 8 | |
| 9 | .. topic:: Abstract |
| 10 | |
| 11 | It's usually difficult to get your management to accept open source software, |
| 12 | and Python is no exception to this rule. This document discusses reasons to use |
| 13 | Python, strategies for winning acceptance, facts and arguments you can use, and |
| 14 | cases where you *shouldn't* try to use Python. |
| 15 | |
| 16 | |
| 17 | Reasons to Use Python |
| 18 | ===================== |
| 19 | |
| 20 | There are several reasons to incorporate a scripting language into your |
| 21 | development process, and this section will discuss them, and why Python has some |
| 22 | properties that make it a particularly good choice. |
| 23 | |
| 24 | |
| 25 | Programmability |
| 26 | --------------- |
| 27 | |
| 28 | Programs are often organized in a modular fashion. Lower-level operations are |
| 29 | grouped together, and called by higher-level functions, which may in turn be |
| 30 | used as basic operations by still further upper levels. |
| 31 | |
| 32 | For example, the lowest level might define a very low-level set of functions for |
| 33 | accessing a hash table. The next level might use hash tables to store the |
| 34 | headers of a mail message, mapping a header name like ``Date`` to a value such |
| 35 | as ``Tue, 13 May 1997 20:00:54 -0400``. A yet higher level may operate on |
| 36 | message objects, without knowing or caring that message headers are stored in a |
| 37 | hash table, and so forth. |
| 38 | |
| 39 | Often, the lowest levels do very simple things; they implement a data structure |
| 40 | such as a binary tree or hash table, or they perform some simple computation, |
| 41 | such as converting a date string to a number. The higher levels then contain |
| 42 | logic connecting these primitive operations. Using the approach, the primitives |
| 43 | can be seen as basic building blocks which are then glued together to produce |
| 44 | the complete product. |
| 45 | |
| 46 | Why is this design approach relevant to Python? Because Python is well suited |
| 47 | to functioning as such a glue language. A common approach is to write a Python |
| 48 | module that implements the lower level operations; for the sake of speed, the |
| 49 | implementation might be in C, Java, or even Fortran. Once the primitives are |
| 50 | available to Python programs, the logic underlying higher level operations is |
| 51 | written in the form of Python code. The high-level logic is then more |
| 52 | understandable, and easier to modify. |
| 53 | |
| 54 | John Ousterhout wrote a paper that explains this idea at greater length, |
| 55 | entitled "Scripting: Higher Level Programming for the 21st Century". I |
| 56 | recommend that you read this paper; see the references for the URL. Ousterhout |
| 57 | is the inventor of the Tcl language, and therefore argues that Tcl should be |
| 58 | used for this purpose; he only briefly refers to other languages such as Python, |
| 59 | Perl, and Lisp/Scheme, but in reality, Ousterhout's argument applies to |
| 60 | scripting languages in general, since you could equally write extensions for any |
| 61 | of the languages mentioned above. |
| 62 | |
| 63 | |
| 64 | Prototyping |
| 65 | ----------- |
| 66 | |
| 67 | In *The Mythical Man-Month*, Fredrick Brooks suggests the following rule when |
| 68 | planning software projects: "Plan to throw one away; you will anyway." Brooks |
| 69 | is saying that the first attempt at a software design often turns out to be |
| 70 | wrong; unless the problem is very simple or you're an extremely good designer, |
| 71 | you'll find that new requirements and features become apparent once development |
| 72 | has actually started. If these new requirements can't be cleanly incorporated |
| 73 | into the program's structure, you're presented with two unpleasant choices: |
| 74 | hammer the new features into the program somehow, or scrap everything and write |
| 75 | a new version of the program, taking the new features into account from the |
| 76 | beginning. |
| 77 | |
| 78 | Python provides you with a good environment for quickly developing an initial |
| 79 | prototype. That lets you get the overall program structure and logic right, and |
| 80 | you can fine-tune small details in the fast development cycle that Python |
| 81 | provides. Once you're satisfied with the GUI interface or program output, you |
| 82 | can translate the Python code into C++, Fortran, Java, or some other compiled |
| 83 | language. |
| 84 | |
| 85 | Prototyping means you have to be careful not to use too many Python features |
| 86 | that are hard to implement in your other language. Using ``eval()``, or regular |
| 87 | expressions, or the :mod:`pickle` module, means that you're going to need C or |
| 88 | Java libraries for formula evaluation, regular expressions, and serialization, |
| 89 | for example. But it's not hard to avoid such tricky code, and in the end the |
| 90 | translation usually isn't very difficult. The resulting code can be rapidly |
| 91 | debugged, because any serious logical errors will have been removed from the |
| 92 | prototype, leaving only more minor slip-ups in the translation to track down. |
| 93 | |
| 94 | This strategy builds on the earlier discussion of programmability. Using Python |
| 95 | as glue to connect lower-level components has obvious relevance for constructing |
| 96 | prototype systems. In this way Python can help you with development, even if |
| 97 | end users never come in contact with Python code at all. If the performance of |
| 98 | the Python version is adequate and corporate politics allow it, you may not need |
| 99 | to do a translation into C or Java, but it can still be faster to develop a |
| 100 | prototype and then translate it, instead of attempting to produce the final |
| 101 | version immediately. |
| 102 | |
| 103 | One example of this development strategy is Microsoft Merchant Server. Version |
| 104 | 1.0 was written in pure Python, by a company that subsequently was purchased by |
| 105 | Microsoft. Version 2.0 began to translate the code into C++, shipping with some |
| 106 | C++code and some Python code. Version 3.0 didn't contain any Python at all; all |
| 107 | the code had been translated into C++. Even though the product doesn't contain |
| 108 | a Python interpreter, the Python language has still served a useful purpose by |
| 109 | speeding up development. |
| 110 | |
| 111 | This is a very common use for Python. Past conference papers have also |
| 112 | described this approach for developing high-level numerical algorithms; see |
| 113 | David M. Beazley and Peter S. Lomdahl's paper "Feeding a Large-scale Physics |
| 114 | Application to Python" in the references for a good example. If an algorithm's |
| 115 | basic operations are things like "Take the inverse of this 4000x4000 matrix", |
| 116 | and are implemented in some lower-level language, then Python has almost no |
| 117 | additional performance cost; the extra time required for Python to evaluate an |
| 118 | expression like ``m.invert()`` is dwarfed by the cost of the actual computation. |
| 119 | It's particularly good for applications where seemingly endless tweaking is |
| 120 | required to get things right. GUI interfaces and Web sites are prime examples. |
| 121 | |
| 122 | The Python code is also shorter and faster to write (once you're familiar with |
| 123 | Python), so it's easier to throw it away if you decide your approach was wrong; |
| 124 | if you'd spent two weeks working on it instead of just two hours, you might |
| 125 | waste time trying to patch up what you've got out of a natural reluctance to |
| 126 | admit that those two weeks were wasted. Truthfully, those two weeks haven't |
| 127 | been wasted, since you've learnt something about the problem and the technology |
| 128 | you're using to solve it, but it's human nature to view this as a failure of |
| 129 | some sort. |
| 130 | |
| 131 | |
| 132 | Simplicity and Ease of Understanding |
| 133 | ------------------------------------ |
| 134 | |
| 135 | Python is definitely *not* a toy language that's only usable for small tasks. |
| 136 | The language features are general and powerful enough to enable it to be used |
| 137 | for many different purposes. It's useful at the small end, for 10- or 20-line |
| 138 | scripts, but it also scales up to larger systems that contain thousands of lines |
| 139 | of code. |
| 140 | |
| 141 | However, this expressiveness doesn't come at the cost of an obscure or tricky |
| 142 | syntax. While Python has some dark corners that can lead to obscure code, there |
| 143 | are relatively few such corners, and proper design can isolate their use to only |
| 144 | a few classes or modules. It's certainly possible to write confusing code by |
| 145 | using too many features with too little concern for clarity, but most Python |
| 146 | code can look a lot like a slightly-formalized version of human-understandable |
| 147 | pseudocode. |
| 148 | |
| 149 | In *The New Hacker's Dictionary*, Eric S. Raymond gives the following definition |
| 150 | for "compact": |
| 151 | |
| 152 | .. epigraph:: |
| 153 | |
| 154 | Compact *adj.* Of a design, describes the valuable property that it can all be |
| 155 | apprehended at once in one's head. This generally means the thing created from |
| 156 | the design can be used with greater facility and fewer errors than an equivalent |
| 157 | tool that is not compact. Compactness does not imply triviality or lack of |
| 158 | power; for example, C is compact and FORTRAN is not, but C is more powerful than |
| 159 | FORTRAN. Designs become non-compact through accreting features and cruft that |
| 160 | don't merge cleanly into the overall design scheme (thus, some fans of Classic C |
| 161 | maintain that ANSI C is no longer compact). |
| 162 | |
Christian Heimes | 18c6689 | 2008-02-17 13:31:39 +0000 | [diff] [blame] | 163 | (From http://www.catb.org/~esr/jargon/html/C/compact.html) |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 164 | |
| 165 | In this sense of the word, Python is quite compact, because the language has |
| 166 | just a few ideas, which are used in lots of places. Take namespaces, for |
| 167 | example. Import a module with ``import math``, and you create a new namespace |
| 168 | called ``math``. Classes are also namespaces that share many of the properties |
| 169 | of modules, and have a few of their own; for example, you can create instances |
| 170 | of a class. Instances? They're yet another namespace. Namespaces are currently |
| 171 | implemented as Python dictionaries, so they have the same methods as the |
| 172 | standard dictionary data type: .keys() returns all the keys, and so forth. |
| 173 | |
| 174 | This simplicity arises from Python's development history. The language syntax |
| 175 | derives from different sources; ABC, a relatively obscure teaching language, is |
| 176 | one primary influence, and Modula-3 is another. (For more information about ABC |
Christian Heimes | 18c6689 | 2008-02-17 13:31:39 +0000 | [diff] [blame] | 177 | and Modula-3, consult their respective Web sites at http://www.cwi.nl/~steven/abc/ |
| 178 | and http://www.m3.org.) Other features have come from C, Icon, |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 179 | Algol-68, and even Perl. Python hasn't really innovated very much, but instead |
| 180 | has tried to keep the language small and easy to learn, building on ideas that |
| 181 | have been tried in other languages and found useful. |
| 182 | |
| 183 | Simplicity is a virtue that should not be underestimated. It lets you learn the |
Christian Heimes | 18c6689 | 2008-02-17 13:31:39 +0000 | [diff] [blame] | 184 | language more quickly, and then rapidly write code -- code that often works the |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 185 | first time you run it. |
| 186 | |
| 187 | |
| 188 | Java Integration |
| 189 | ---------------- |
| 190 | |
| 191 | If you're working with Java, Jython (http://www.jython.org/) is definitely worth |
| 192 | your attention. Jython is a re-implementation of Python in Java that compiles |
| 193 | Python code into Java bytecodes. The resulting environment has very tight, |
| 194 | almost seamless, integration with Java. It's trivial to access Java classes |
| 195 | from Python, and you can write Python classes that subclass Java classes. |
| 196 | Jython can be used for prototyping Java applications in much the same way |
| 197 | CPython is used, and it can also be used for test suites for Java code, or |
| 198 | embedded in a Java application to add scripting capabilities. |
| 199 | |
| 200 | |
| 201 | Arguments and Rebuttals |
| 202 | ======================= |
| 203 | |
| 204 | Let's say that you've decided upon Python as the best choice for your |
| 205 | application. How can you convince your management, or your fellow developers, |
| 206 | to use Python? This section lists some common arguments against using Python, |
| 207 | and provides some possible rebuttals. |
| 208 | |
| 209 | **Python is freely available software that doesn't cost anything. How good can |
| 210 | it be?** |
| 211 | |
| 212 | Very good, indeed. These days Linux and Apache, two other pieces of open source |
| 213 | software, are becoming more respected as alternatives to commercial software, |
| 214 | but Python hasn't had all the publicity. |
| 215 | |
| 216 | Python has been around for several years, with many users and developers. |
| 217 | Accordingly, the interpreter has been used by many people, and has gotten most |
| 218 | of the bugs shaken out of it. While bugs are still discovered at intervals, |
| 219 | they're usually either quite obscure (they'd have to be, for no one to have run |
| 220 | into them before) or they involve interfaces to external libraries. The |
| 221 | internals of the language itself are quite stable. |
| 222 | |
| 223 | Having the source code should be viewed as making the software available for |
| 224 | peer review; people can examine the code, suggest (and implement) improvements, |
| 225 | and track down bugs. To find out more about the idea of open source code, along |
| 226 | with arguments and case studies supporting it, go to http://www.opensource.org. |
| 227 | |
| 228 | **Who's going to support it?** |
| 229 | |
| 230 | Python has a sizable community of developers, and the number is still growing. |
| 231 | The Internet community surrounding the language is an active one, and is worth |
| 232 | being considered another one of Python's advantages. Most questions posted to |
| 233 | the comp.lang.python newsgroup are quickly answered by someone. |
| 234 | |
| 235 | Should you need to dig into the source code, you'll find it's clear and |
| 236 | well-organized, so it's not very difficult to write extensions and track down |
| 237 | bugs yourself. If you'd prefer to pay for support, there are companies and |
| 238 | individuals who offer commercial support for Python. |
| 239 | |
| 240 | **Who uses Python for serious work?** |
| 241 | |
| 242 | Lots of people; one interesting thing about Python is the surprising diversity |
| 243 | of applications that it's been used for. People are using Python to: |
| 244 | |
| 245 | * Run Web sites |
| 246 | |
| 247 | * Write GUI interfaces |
| 248 | |
| 249 | * Control number-crunching code on supercomputers |
| 250 | |
| 251 | * Make a commercial application scriptable by embedding the Python interpreter |
| 252 | inside it |
| 253 | |
| 254 | * Process large XML data sets |
| 255 | |
| 256 | * Build test suites for C or Java code |
| 257 | |
| 258 | Whatever your application domain is, there's probably someone who's used Python |
| 259 | for something similar. Yet, despite being useable for such high-end |
| 260 | applications, Python's still simple enough to use for little jobs. |
| 261 | |
| 262 | See http://wiki.python.org/moin/OrganizationsUsingPython for a list of some of |
| 263 | the organizations that use Python. |
| 264 | |
| 265 | **What are the restrictions on Python's use?** |
| 266 | |
| 267 | They're practically nonexistent. Consult the :file:`Misc/COPYRIGHT` file in the |
Christian Heimes | 2202f87 | 2008-02-06 14:31:34 +0000 | [diff] [blame] | 268 | source distribution, or the section :ref:`history-and-license` for the full |
Christian Heimes | 18c6689 | 2008-02-17 13:31:39 +0000 | [diff] [blame] | 269 | language, but it boils down to three conditions: |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 270 | |
| 271 | * You have to leave the copyright notice on the software; if you don't include |
| 272 | the source code in a product, you have to put the copyright notice in the |
| 273 | supporting documentation. |
| 274 | |
| 275 | * Don't claim that the institutions that have developed Python endorse your |
| 276 | product in any way. |
| 277 | |
| 278 | * If something goes wrong, you can't sue for damages. Practically all software |
Christian Heimes | c3f30c4 | 2008-02-22 16:37:40 +0000 | [diff] [blame] | 279 | licenses contain this condition. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 280 | |
| 281 | Notice that you don't have to provide source code for anything that contains |
| 282 | Python or is built with it. Also, the Python interpreter and accompanying |
| 283 | documentation can be modified and redistributed in any way you like, and you |
| 284 | don't have to pay anyone any licensing fees at all. |
| 285 | |
| 286 | **Why should we use an obscure language like Python instead of well-known |
| 287 | language X?** |
| 288 | |
| 289 | I hope this HOWTO, and the documents listed in the final section, will help |
| 290 | convince you that Python isn't obscure, and has a healthily growing user base. |
| 291 | One word of advice: always present Python's positive advantages, instead of |
| 292 | concentrating on language X's failings. People want to know why a solution is |
| 293 | good, rather than why all the other solutions are bad. So instead of attacking |
| 294 | a competing solution on various grounds, simply show how Python's virtues can |
| 295 | help. |
| 296 | |
| 297 | |
| 298 | Useful Resources |
| 299 | ================ |
| 300 | |
| 301 | http://www.pythonology.com/success |
| 302 | The Python Success Stories are a collection of stories from successful users of |
| 303 | Python, with the emphasis on business and corporate users. |
| 304 | |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 305 | .. http://www.fsbassociates.com/books/pythonchpt1.htm |
| 306 | The first chapter of \emph{Internet Programming with Python} also |
| 307 | examines some of the reasons for using Python. The book is well worth |
| 308 | buying, but the publishers have made the first chapter available on |
| 309 | the Web. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 310 | |
| 311 | http://home.pacbell.net/ouster/scripting.html |
| 312 | John Ousterhout's white paper on scripting is a good argument for the utility of |
| 313 | scripting languages, though naturally enough, he emphasizes Tcl, the language he |
| 314 | developed. Most of the arguments would apply to any scripting language. |
| 315 | |
| 316 | http://www.python.org/workshops/1997-10/proceedings/beazley.html |
| 317 | The authors, David M. Beazley and Peter S. Lomdahl, describe their use of |
| 318 | Python at Los Alamos National Laboratory. It's another good example of how |
| 319 | Python can help get real work done. This quotation from the paper has been |
| 320 | echoed by many people: |
| 321 | |
| 322 | .. epigraph:: |
| 323 | |
| 324 | Originally developed as a large monolithic application for massively parallel |
| 325 | processing systems, we have used Python to transform our application into a |
| 326 | flexible, highly modular, and extremely powerful system for performing |
| 327 | simulation, data analysis, and visualization. In addition, we describe how |
| 328 | Python has solved a number of important problems related to the development, |
| 329 | debugging, deployment, and maintenance of scientific software. |
| 330 | |
| 331 | http://pythonjournal.cognizor.com/pyj1/Everitt-Feit_interview98-V1.html |
| 332 | This interview with Andy Feit, discussing Infoseek's use of Python, can be used |
| 333 | to show that choosing Python didn't introduce any difficulties into a company's |
| 334 | development process, and provided some substantial benefits. |
| 335 | |
Christian Heimes | 5b5e81c | 2007-12-31 16:14:33 +0000 | [diff] [blame] | 336 | .. http://www.python.org/psa/Commercial.html |
| 337 | Robin Friedrich wrote this document on how to support Python's use in |
| 338 | commercial projects. |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 339 | |
| 340 | http://www.python.org/workshops/1997-10/proceedings/stein.ps |
| 341 | For the 6th Python conference, Greg Stein presented a paper that traced Python's |
| 342 | adoption and usage at a startup called eShop, and later at Microsoft. |
| 343 | |
| 344 | http://www.opensource.org |
| 345 | Management may be doubtful of the reliability and usefulness of software that |
| 346 | wasn't written commercially. This site presents arguments that show how open |
| 347 | source software can have considerable advantages over closed-source software. |
| 348 | |
Christian Heimes | dd15f6c | 2008-03-16 00:07:10 +0000 | [diff] [blame] | 349 | http://www.faqs.org/docs/Linux-mini/Advocacy.html |
Georg Brandl | 116aa62 | 2007-08-15 14:28:22 +0000 | [diff] [blame] | 350 | The Linux Advocacy mini-HOWTO was the inspiration for this document, and is also |
| 351 | well worth reading for general suggestions on winning acceptance for a new |
| 352 | technology, such as Linux or Python. In general, you won't make much progress |
| 353 | by simply attacking existing systems and complaining about their inadequacies; |
| 354 | this often ends up looking like unfocused whining. It's much better to point |
| 355 | out some of the many areas where Python is an improvement over other systems. |
| 356 | |