Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 1 | .. _pyporting-howto: |
| 2 | |
| 3 | ********************************* |
| 4 | Porting Python 2 Code to Python 3 |
| 5 | ********************************* |
| 6 | |
| 7 | :author: Brett Cannon |
| 8 | |
| 9 | .. topic:: Abstract |
| 10 | |
Brett Cannon | 4b0c24a | 2011-02-03 22:14:58 +0000 | [diff] [blame] | 11 | With Python 3 being the future of Python while Python 2 is still in active |
| 12 | use, it is good to have your project available for both major releases of |
Brett Cannon | 9ca21b3 | 2014-01-07 11:52:04 -0500 | [diff] [blame] | 13 | Python. This guide is meant to help you figure out how best to support both |
| 14 | Python 2 & 3 simultaneously. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 15 | |
Brett Cannon | 4b0c24a | 2011-02-03 22:14:58 +0000 | [diff] [blame] | 16 | If you are looking to port an extension module instead of pure Python code, |
Éric Araujo | 5405a0b | 2011-02-05 16:03:12 +0000 | [diff] [blame] | 17 | please see :ref:`cporting-howto`. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 18 | |
Brett Cannon | 9ca21b3 | 2014-01-07 11:52:04 -0500 | [diff] [blame] | 19 | If you would like to read one core Python developer's take on why Python 3 |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 20 | came into existence, you can read Nick Coghlan's `Python 3 Q & A`_ or |
| 21 | Brett Cannon's `Why Python 3 exists`_. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 22 | |
Brett Cannon | 9ca21b3 | 2014-01-07 11:52:04 -0500 | [diff] [blame] | 23 | For help with porting, you can email the python-porting_ mailing list with |
| 24 | questions. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 25 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 26 | The Short Explanation |
| 27 | ===================== |
Larry Hastings | 3732ed2 | 2014-03-15 21:13:56 -0700 | [diff] [blame] | 28 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 29 | To make your project be single-source Python 2/3 compatible, the basic steps |
| 30 | are: |
Larry Hastings | 3732ed2 | 2014-03-15 21:13:56 -0700 | [diff] [blame] | 31 | |
Brett Cannon | 2645bad | 2015-03-13 12:49:44 -0400 | [diff] [blame] | 32 | #. Only worry about supporting Python 2.7 |
Brett Cannon | 90783eb | 2014-12-12 15:13:43 -0500 | [diff] [blame] | 33 | #. Make sure you have good test coverage (coverage.py_ can help; |
| 34 | ``pip install coverage``) |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 35 | #. Learn the differences between Python 2 & 3 |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 36 | #. Use Futurize_ (or Modernize_) to update your code (e.g. ``pip install future``) |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 37 | #. Use Pylint_ to help make sure you don't regress on your Python 3 support |
Brett Cannon | 2645bad | 2015-03-13 12:49:44 -0400 | [diff] [blame] | 38 | (``pip install pylint``) |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 39 | #. Use caniusepython3_ to find out which of your dependencies are blocking your |
Brett Cannon | 90783eb | 2014-12-12 15:13:43 -0500 | [diff] [blame] | 40 | use of Python 3 (``pip install caniusepython3``) |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 41 | #. Once your dependencies are no longer blocking you, use continuous integration |
| 42 | to make sure you stay compatible with Python 2 & 3 (tox_ can help test |
Brett Cannon | 90783eb | 2014-12-12 15:13:43 -0500 | [diff] [blame] | 43 | against multiple versions of Python; ``pip install tox``) |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 44 | #. Consider using optional static type checking to make sure your type usage |
| 45 | works in both Python 2 & 3 (e.g. use mypy_ to check your typing under both |
| 46 | Python 2 & Python 3). |
Georg Brandl | 728e4de | 2014-10-29 09:00:30 +0100 | [diff] [blame] | 47 | |
Larry Hastings | 3732ed2 | 2014-03-15 21:13:56 -0700 | [diff] [blame] | 48 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 49 | Details |
| 50 | ======= |
Larry Hastings | 3732ed2 | 2014-03-15 21:13:56 -0700 | [diff] [blame] | 51 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 52 | A key point about supporting Python 2 & 3 simultaneously is that you can start |
| 53 | **today**! Even if your dependencies are not supporting Python 3 yet that does |
| 54 | not mean you can't modernize your code **now** to support Python 3. Most changes |
| 55 | required to support Python 3 lead to cleaner code using newer practices even in |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 56 | Python 2 code. |
Larry Hastings | 3732ed2 | 2014-03-15 21:13:56 -0700 | [diff] [blame] | 57 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 58 | Another key point is that modernizing your Python 2 code to also support |
| 59 | Python 3 is largely automated for you. While you might have to make some API |
| 60 | decisions thanks to Python 3 clarifying text data versus binary data, the |
| 61 | lower-level work is now mostly done for you and thus can at least benefit from |
| 62 | the automated changes immediately. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 63 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 64 | Keep those key points in mind while you read on about the details of porting |
| 65 | your code to support Python 2 & 3 simultaneously. |
Georg Brandl | 2cb2fa9 | 2011-02-07 15:30:45 +0000 | [diff] [blame] | 66 | |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 67 | |
Brett Cannon | 2645bad | 2015-03-13 12:49:44 -0400 | [diff] [blame] | 68 | Drop support for Python 2.6 and older |
| 69 | ------------------------------------- |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 70 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 71 | While you can make Python 2.5 work with Python 3, it is **much** easier if you |
Brett Cannon | 2645bad | 2015-03-13 12:49:44 -0400 | [diff] [blame] | 72 | only have to work with Python 2.7. If dropping Python 2.5 is not an |
| 73 | option then the six_ project can help you support Python 2.5 & 3 simultaneously |
Brett Cannon | 90783eb | 2014-12-12 15:13:43 -0500 | [diff] [blame] | 74 | (``pip install six``). Do realize, though, that nearly all the projects listed |
| 75 | in this HOWTO will not be available to you. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 76 | |
Brett Cannon | 2645bad | 2015-03-13 12:49:44 -0400 | [diff] [blame] | 77 | If you are able to skip Python 2.5 and older, then the required changes |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 78 | to your code should continue to look and feel like idiomatic Python code. At |
| 79 | worst you will have to use a function instead of a method in some instances or |
| 80 | have to import a function instead of using a built-in one, but otherwise the |
| 81 | overall transformation should not feel foreign to you. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 82 | |
Brett Cannon | 2645bad | 2015-03-13 12:49:44 -0400 | [diff] [blame] | 83 | But you should aim for only supporting Python 2.7. Python 2.6 is no longer |
Brett Cannon | 7e93c84 | 2016-12-22 10:02:01 -0800 | [diff] [blame] | 84 | freely supported and thus is not receiving bugfixes. This means **you** will have |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 85 | to work around any issues you come across with Python 2.6. There are also some |
Brett Cannon | 2645bad | 2015-03-13 12:49:44 -0400 | [diff] [blame] | 86 | tools mentioned in this HOWTO which do not support Python 2.6 (e.g., Pylint_), |
| 87 | and this will become more commonplace as time goes on. It will simply be easier |
| 88 | for you if you only support the versions of Python that you have to support. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 89 | |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 90 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 91 | Make sure you specify the proper version support in your ``setup.py`` file |
| 92 | -------------------------------------------------------------------------- |
Brett Cannon | b7e6b89 | 2013-03-09 14:22:35 -0500 | [diff] [blame] | 93 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 94 | In your ``setup.py`` file you should have the proper `trove classifier`_ |
| 95 | specifying what versions of Python you support. As your project does not support |
| 96 | Python 3 yet you should at least have |
| 97 | ``Programming Language :: Python :: 2 :: Only`` specified. Ideally you should |
| 98 | also specify each major/minor version of Python that you do support, e.g. |
| 99 | ``Programming Language :: Python :: 2.7``. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 100 | |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 101 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 102 | Have good test coverage |
| 103 | ----------------------- |
Brett Cannon | 9ca21b3 | 2014-01-07 11:52:04 -0500 | [diff] [blame] | 104 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 105 | Once you have your code supporting the oldest version of Python 2 you want it |
| 106 | to, you will want to make sure your test suite has good coverage. A good rule of |
| 107 | thumb is that if you want to be confident enough in your test suite that any |
| 108 | failures that appear after having tools rewrite your code are actual bugs in the |
| 109 | tools and not in your code. If you want a number to aim for, try to get over 80% |
Brett Cannon | 7e93c84 | 2016-12-22 10:02:01 -0800 | [diff] [blame] | 110 | coverage (and don't feel bad if you find it hard to get better than 90% |
| 111 | coverage). If you don't already have a tool to measure test coverage then |
| 112 | coverage.py_ is recommended. |
Brett Cannon | 9ca21b3 | 2014-01-07 11:52:04 -0500 | [diff] [blame] | 113 | |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 114 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 115 | Learn the differences between Python 2 & 3 |
| 116 | ------------------------------------------- |
Brett Cannon | 9ca21b3 | 2014-01-07 11:52:04 -0500 | [diff] [blame] | 117 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 118 | Once you have your code well-tested you are ready to begin porting your code to |
| 119 | Python 3! But to fully understand how your code is going to change and what |
| 120 | you want to look out for while you code, you will want to learn what changes |
| 121 | Python 3 makes in terms of Python 2. Typically the two best ways of doing that |
| 122 | is reading the `"What's New"`_ doc for each release of Python 3 and the |
Brett Cannon | 90783eb | 2014-12-12 15:13:43 -0500 | [diff] [blame] | 123 | `Porting to Python 3`_ book (which is free online). There is also a handy |
| 124 | `cheat sheet`_ from the Python-Future project. |
Brett Cannon | 9ca21b3 | 2014-01-07 11:52:04 -0500 | [diff] [blame] | 125 | |
Brett Cannon | 9ca21b3 | 2014-01-07 11:52:04 -0500 | [diff] [blame] | 126 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 127 | Update your code |
| 128 | ---------------- |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 129 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 130 | Once you feel like you know what is different in Python 3 compared to Python 2, |
| 131 | it's time to update your code! You have a choice between two tools in porting |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 132 | your code automatically: Futurize_ and Modernize_. Which tool you choose will |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 133 | depend on how much like Python 3 you want your code to be. Futurize_ does its |
| 134 | best to make Python 3 idioms and practices exist in Python 2, e.g. backporting |
| 135 | the ``bytes`` type from Python 3 so that you have semantic parity between the |
| 136 | major versions of Python. Modernize_, |
| 137 | on the other hand, is more conservative and targets a Python 2/3 subset of |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 138 | Python, directly relying on six_ to help provide compatibility. As Python 3 is |
| 139 | the future, it might be best to consider Futurize to begin adjusting to any new |
| 140 | practices that Python 3 introduces which you are not accustomed to yet. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 141 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 142 | Regardless of which tool you choose, they will update your code to run under |
| 143 | Python 3 while staying compatible with the version of Python 2 you started with. |
| 144 | Depending on how conservative you want to be, you may want to run the tool over |
| 145 | your test suite first and visually inspect the diff to make sure the |
| 146 | transformation is accurate. After you have transformed your test suite and |
| 147 | verified that all the tests still pass as expected, then you can transform your |
| 148 | application code knowing that any tests which fail is a translation failure. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 149 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 150 | Unfortunately the tools can't automate everything to make your code work under |
| 151 | Python 3 and so there are a handful of things you will need to update manually |
| 152 | to get full Python 3 support (which of these steps are necessary vary between |
| 153 | the tools). Read the documentation for the tool you choose to use to see what it |
| 154 | fixes by default and what it can do optionally to know what will (not) be fixed |
| 155 | for you and what you may have to fix on your own (e.g. using ``io.open()`` over |
| 156 | the built-in ``open()`` function is off by default in Modernize). Luckily, |
| 157 | though, there are only a couple of things to watch out for which can be |
| 158 | considered large issues that may be hard to debug if not watched for. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 159 | |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 160 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 161 | Division |
| 162 | ++++++++ |
Georg Brandl | 2cb2fa9 | 2011-02-07 15:30:45 +0000 | [diff] [blame] | 163 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 164 | In Python 3, ``5 / 2 == 2.5`` and not ``2``; all division between ``int`` values |
| 165 | result in a ``float``. This change has actually been planned since Python 2.2 |
| 166 | which was released in 2002. Since then users have been encouraged to add |
| 167 | ``from __future__ import division`` to any and all files which use the ``/`` and |
| 168 | ``//`` operators or to be running the interpreter with the ``-Q`` flag. If you |
| 169 | have not been doing this then you will need to go through your code and do two |
| 170 | things: |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 171 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 172 | #. Add ``from __future__ import division`` to your files |
| 173 | #. Update any division operator as necessary to either use ``//`` to use floor |
| 174 | division or continue using ``/`` and expect a float |
Georg Brandl | 2cb2fa9 | 2011-02-07 15:30:45 +0000 | [diff] [blame] | 175 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 176 | The reason that ``/`` isn't simply translated to ``//`` automatically is that if |
Brett Cannon | fdde79d | 2015-02-27 15:10:03 -0500 | [diff] [blame] | 177 | an object defines a ``__truediv__`` method but not ``__floordiv__`` then your |
| 178 | code would begin to fail (e.g. a user-defined class that uses ``/`` to |
| 179 | signify some operation but not ``//`` for the same thing or at all). |
Georg Brandl | 2cb2fa9 | 2011-02-07 15:30:45 +0000 | [diff] [blame] | 180 | |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 181 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 182 | Text versus binary data |
| 183 | +++++++++++++++++++++++ |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 184 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 185 | In Python 2 you could use the ``str`` type for both text and binary data. |
| 186 | Unfortunately this confluence of two different concepts could lead to brittle |
| 187 | code which sometimes worked for either kind of data, sometimes not. It also |
| 188 | could lead to confusing APIs if people didn't explicitly state that something |
| 189 | that accepted ``str`` accepted either text or binary data instead of one |
| 190 | specific type. This complicated the situation especially for anyone supporting |
| 191 | multiple languages as APIs wouldn't bother explicitly supporting ``unicode`` |
| 192 | when they claimed text data support. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 193 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 194 | To make the distinction between text and binary data clearer and more |
| 195 | pronounced, Python 3 did what most languages created in the age of the internet |
| 196 | have done and made text and binary data distinct types that cannot blindly be |
| 197 | mixed together (Python predates widespread access to the internet). For any code |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 198 | that deals only with text or only binary data, this separation doesn't pose an |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 199 | issue. But for code that has to deal with both, it does mean you might have to |
| 200 | now care about when you are using text compared to binary data, which is why |
| 201 | this cannot be entirely automated. |
Georg Brandl | 2cb2fa9 | 2011-02-07 15:30:45 +0000 | [diff] [blame] | 202 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 203 | To start, you will need to decide which APIs take text and which take binary |
| 204 | (it is **highly** recommended you don't design APIs that can take both due to |
| 205 | the difficulty of keeping the code working; as stated earlier it is difficult to |
| 206 | do well). In Python 2 this means making sure the APIs that take text can work |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 207 | with ``unicode`` and those that work with binary data work with the |
| 208 | ``bytes`` type from Python 3 (which is a subset of ``str`` in Python 2 and acts |
| 209 | as an alias for ``bytes`` type in Python 2). Usually the biggest issue is |
| 210 | realizing which methods exist on which types in Python 2 & 3 simultaneously |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 211 | (for text that's ``unicode`` in Python 2 and ``str`` in Python 3, for binary |
| 212 | that's ``str``/``bytes`` in Python 2 and ``bytes`` in Python 3). The following |
| 213 | table lists the **unique** methods of each data type across Python 2 & 3 |
| 214 | (e.g., the ``decode()`` method is usable on the equivalent binary data type in |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 215 | either Python 2 or 3, but it can't be used by the textual data type consistently |
Brett Cannon | 8396b8e | 2015-04-13 16:32:16 -0400 | [diff] [blame] | 216 | between Python 2 and 3 because ``str`` in Python 3 doesn't have the method). Do |
Brett Cannon | fd53f98 | 2015-04-13 16:21:07 -0400 | [diff] [blame] | 217 | note that as of Python 3.5 the ``__mod__`` method was added to the bytes type. |
Brett Cannon | ce71ab2 | 2011-02-05 22:05:05 +0000 | [diff] [blame] | 218 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 219 | ======================== ===================== |
| 220 | **Text data** **Binary data** |
| 221 | ------------------------ --------------------- |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 222 | \ decode |
| 223 | ------------------------ --------------------- |
| 224 | encode |
| 225 | ------------------------ --------------------- |
| 226 | format |
| 227 | ------------------------ --------------------- |
| 228 | isdecimal |
| 229 | ------------------------ --------------------- |
| 230 | isnumeric |
| 231 | ======================== ===================== |
Brett Cannon | ce71ab2 | 2011-02-05 22:05:05 +0000 | [diff] [blame] | 232 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 233 | Making the distinction easier to handle can be accomplished by encoding and |
| 234 | decoding between binary data and text at the edge of your code. This means that |
| 235 | when you receive text in binary data, you should immediately decode it. And if |
| 236 | your code needs to send text as binary data then encode it as late as possible. |
| 237 | This allows your code to work with only text internally and thus eliminates |
| 238 | having to keep track of what type of data you are working with. |
Brett Cannon | ce71ab2 | 2011-02-05 22:05:05 +0000 | [diff] [blame] | 239 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 240 | The next issue is making sure you know whether the string literals in your code |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 241 | represent text or binary data. You should add a ``b`` prefix to any |
| 242 | literal that presents binary data. For text you should add a ``u`` prefix to |
| 243 | the text literal. (there is a :mod:`__future__` import to force all unspecified |
| 244 | literals to be Unicode, but usage has shown it isn't as effective as adding a |
| 245 | ``b`` or ``u`` prefix to all literals explicitly) |
Antoine Pitrou | 5c28cfdc | 2011-02-05 11:53:39 +0000 | [diff] [blame] | 246 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 247 | As part of this dichotomy you also need to be careful about opening files. |
Antoine Pitrou | 5c28cfdc | 2011-02-05 11:53:39 +0000 | [diff] [blame] | 248 | Unless you have been working on Windows, there is a chance you have not always |
| 249 | bothered to add the ``b`` mode when opening a binary file (e.g., ``rb`` for |
| 250 | binary reading). Under Python 3, binary files and text files are clearly |
| 251 | distinct and mutually incompatible; see the :mod:`io` module for details. |
| 252 | Therefore, you **must** make a decision of whether a file will be used for |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 253 | binary access (allowing binary data to be read and/or written) or textual access |
Martin Panter | c04fb56 | 2016-02-10 05:44:01 +0000 | [diff] [blame] | 254 | (allowing text data to be read and/or written). You should also use :func:`io.open` |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 255 | for opening files instead of the built-in :func:`open` function as the :mod:`io` |
| 256 | module is consistent from Python 2 to 3 while the built-in :func:`open` function |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 257 | is not (in Python 3 it's actually :func:`io.open`). Do not bother with the |
| 258 | outdated practice of using :func:`codecs.open` as that's only necessary for |
| 259 | keeping compatibility with Python 2.5. |
Antoine Pitrou | 5c28cfdc | 2011-02-05 11:53:39 +0000 | [diff] [blame] | 260 | |
Brett Cannon | 90783eb | 2014-12-12 15:13:43 -0500 | [diff] [blame] | 261 | The constructors of both ``str`` and ``bytes`` have different semantics for the |
| 262 | same arguments between Python 2 & 3. Passing an integer to ``bytes`` in Python 2 |
| 263 | will give you the string representation of the integer: ``bytes(3) == '3'``. |
| 264 | But in Python 3, an integer argument to ``bytes`` will give you a bytes object |
| 265 | as long as the integer specified, filled with null bytes: |
| 266 | ``bytes(3) == b'\x00\x00\x00'``. A similar worry is necessary when passing a |
| 267 | bytes object to ``str``. In Python 2 you just get the bytes object back: |
| 268 | ``str(b'3') == b'3'``. But in Python 3 you get the string representation of the |
| 269 | bytes object: ``str(b'3') == "b'3'"``. |
| 270 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 271 | Finally, the indexing of binary data requires careful handling (slicing does |
| 272 | **not** require any special handling). In Python 2, |
| 273 | ``b'123'[1] == b'2'`` while in Python 3 ``b'123'[1] == 50``. Because binary data |
| 274 | is simply a collection of binary numbers, Python 3 returns the integer value for |
| 275 | the byte you index on. But in Python 2 because ``bytes == str``, indexing |
| 276 | returns a one-item slice of bytes. The six_ project has a function |
| 277 | named ``six.indexbytes()`` which will return an integer like in Python 3: |
| 278 | ``six.indexbytes(b'123', 1)``. |
Antoine Pitrou | 5c28cfdc | 2011-02-05 11:53:39 +0000 | [diff] [blame] | 279 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 280 | To summarize: |
Antoine Pitrou | 5c28cfdc | 2011-02-05 11:53:39 +0000 | [diff] [blame] | 281 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 282 | #. Decide which of your APIs take text and which take binary data |
| 283 | #. Make sure that your code that works with text also works with ``unicode`` and |
| 284 | code for binary data works with ``bytes`` in Python 2 (see the table above |
| 285 | for what methods you cannot use for each type) |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 286 | #. Mark all binary literals with a ``b`` prefix, textual literals with a ``u`` |
| 287 | prefix |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 288 | #. Decode binary data to text as soon as possible, encode text as binary data as |
| 289 | late as possible |
| 290 | #. Open files using :func:`io.open` and make sure to specify the ``b`` mode when |
| 291 | appropriate |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 292 | #. Be careful when indexing into binary data |
Antoine Pitrou | 5c28cfdc | 2011-02-05 11:53:39 +0000 | [diff] [blame] | 293 | |
Brett Cannon | adcb654 | 2016-03-18 13:23:58 -0700 | [diff] [blame] | 294 | |
| 295 | Use feature detection instead of version detection |
| 296 | ++++++++++++++++++++++++++++++++++++++++++++++++++ |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 297 | |
Brett Cannon | adcb654 | 2016-03-18 13:23:58 -0700 | [diff] [blame] | 298 | Inevitably you will have code that has to choose what to do based on what |
| 299 | version of Python is running. The best way to do this is with feature detection |
| 300 | of whether the version of Python you're running under supports what you need. |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 301 | If for some reason that doesn't work then you should make the version check be |
Brett Cannon | adcb654 | 2016-03-18 13:23:58 -0700 | [diff] [blame] | 302 | against Python 2 and not Python 3. To help explain this, let's look at an |
| 303 | example. |
| 304 | |
| 305 | Let's pretend that you need access to a feature of importlib_ that |
| 306 | is available in Python's standard library since Python 3.3 and available for |
| 307 | Python 2 through importlib2_ on PyPI. You might be tempted to write code to |
| 308 | access e.g. the ``importlib.abc`` module by doing the following:: |
| 309 | |
| 310 | import sys |
| 311 | |
Zachary Ware | 54005af | 2016-08-17 09:51:20 -0500 | [diff] [blame] | 312 | if sys.version_info[0] == 3: |
Brett Cannon | adcb654 | 2016-03-18 13:23:58 -0700 | [diff] [blame] | 313 | from importlib import abc |
| 314 | else: |
| 315 | from importlib2 import abc |
| 316 | |
| 317 | The problem with this code is what happens when Python 4 comes out? It would |
| 318 | be better to treat Python 2 as the exceptional case instead of Python 3 and |
| 319 | assume that future Python versions will be more compatible with Python 3 than |
| 320 | Python 2:: |
| 321 | |
| 322 | import sys |
| 323 | |
Zachary Ware | 54005af | 2016-08-17 09:51:20 -0500 | [diff] [blame] | 324 | if sys.version_info[0] > 2: |
Brett Cannon | adcb654 | 2016-03-18 13:23:58 -0700 | [diff] [blame] | 325 | from importlib import abc |
| 326 | else: |
| 327 | from importlib2 import abc |
| 328 | |
| 329 | The best solution, though, is to do no version detection at all and instead rely |
| 330 | on feature detection. That avoids any potential issues of getting the version |
| 331 | detection wrong and helps keep you future-compatible:: |
| 332 | |
| 333 | try: |
| 334 | from importlib import abc |
| 335 | except ImportError: |
| 336 | from importlib2 import abc |
| 337 | |
| 338 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 339 | Prevent compatibility regressions |
| 340 | --------------------------------- |
Antoine Pitrou | 5c28cfdc | 2011-02-05 11:53:39 +0000 | [diff] [blame] | 341 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 342 | Once you have fully translated your code to be compatible with Python 3, you |
| 343 | will want to make sure your code doesn't regress and stop working under |
| 344 | Python 3. This is especially true if you have a dependency which is blocking you |
| 345 | from actually running under Python 3 at the moment. |
Antoine Pitrou | 5c28cfdc | 2011-02-05 11:53:39 +0000 | [diff] [blame] | 346 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 347 | To help with staying compatible, any new modules you create should have |
| 348 | at least the following block of code at the top of it:: |
Georg Brandl | 2cb2fa9 | 2011-02-07 15:30:45 +0000 | [diff] [blame] | 349 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 350 | from __future__ import absolute_import |
| 351 | from __future__ import division |
Berker Peksag | bd62f0a | 2014-12-13 15:48:22 +0200 | [diff] [blame] | 352 | from __future__ import print_function |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 353 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 354 | You can also run Python 2 with the ``-3`` flag to be warned about various |
| 355 | compatibility issues your code triggers during execution. If you turn warnings |
| 356 | into errors with ``-Werror`` then you can make sure that you don't accidentally |
| 357 | miss a warning. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 358 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 359 | You can also use the Pylint_ project and its ``--py3k`` flag to lint your code |
| 360 | to receive warnings when your code begins to deviate from Python 3 |
| 361 | compatibility. This also prevents you from having to run Modernize_ or Futurize_ |
| 362 | over your code regularly to catch compatibility regressions. This does require |
| 363 | you only support Python 2.7 and Python 3.4 or newer as that is Pylint's |
| 364 | minimum Python version support. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 365 | |
| 366 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 367 | Check which dependencies block your transition |
| 368 | ---------------------------------------------- |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 369 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 370 | **After** you have made your code compatible with Python 3 you should begin to |
| 371 | care about whether your dependencies have also been ported. The caniusepython3_ |
| 372 | project was created to help you determine which projects |
| 373 | -- directly or indirectly -- are blocking you from supporting Python 3. There |
| 374 | is both a command-line tool as well as a web interface at |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 375 | https://caniusepython3.com. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 376 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 377 | The project also provides code which you can integrate into your test suite so |
| 378 | that you will have a failing test when you no longer have dependencies blocking |
| 379 | you from using Python 3. This allows you to avoid having to manually check your |
| 380 | dependencies and to be notified quickly when you can start running on Python 3. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 381 | |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 382 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 383 | Update your ``setup.py`` file to denote Python 3 compatibility |
| 384 | -------------------------------------------------------------- |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 385 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 386 | Once your code works under Python 3, you should update the classifiers in |
| 387 | your ``setup.py`` to contain ``Programming Language :: Python :: 3`` and to not |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 388 | specify sole Python 2 support. This will tell anyone using your code that you |
| 389 | support Python 2 **and** 3. Ideally you will also want to add classifiers for |
| 390 | each major/minor version of Python you now support. |
| 391 | |
Georg Brandl | 2cb2fa9 | 2011-02-07 15:30:45 +0000 | [diff] [blame] | 392 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 393 | Use continuous integration to stay compatible |
| 394 | --------------------------------------------- |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 395 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 396 | Once you are able to fully run under Python 3 you will want to make sure your |
| 397 | code always works under both Python 2 & 3. Probably the best tool for running |
| 398 | your tests under multiple Python interpreters is tox_. You can then integrate |
| 399 | tox with your continuous integration system so that you never accidentally break |
| 400 | Python 2 or 3 support. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 401 | |
Martin Panter | a90a4a9 | 2016-05-30 04:04:50 +0000 | [diff] [blame] | 402 | You may also want to use the ``-bb`` flag with the Python 3 interpreter to |
Brett Cannon | 4269d6d | 2015-04-13 14:37:50 -0400 | [diff] [blame] | 403 | trigger an exception when you are comparing bytes to strings or bytes to an int |
| 404 | (the latter is available starting in Python 3.5). By default type-differing |
Antoine Pitrou | 3764fc2 | 2015-04-13 21:07:57 +0200 | [diff] [blame] | 405 | comparisons simply return ``False``, but if you made a mistake in your |
Brett Cannon | 4269d6d | 2015-04-13 14:37:50 -0400 | [diff] [blame] | 406 | separation of text/binary data handling or indexing on bytes you wouldn't easily |
| 407 | find the mistake. This flag will raise an exception when these kinds of |
| 408 | comparisons occur, making the mistake much easier to track down. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 409 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 410 | And that's mostly it! At this point your code base is compatible with both |
| 411 | Python 2 and 3 simultaneously. Your testing will also be set up so that you |
| 412 | don't accidentally break Python 2 or 3 compatibility regardless of which version |
| 413 | you typically run your tests under while developing. |
Brett Cannon | 8045d97 | 2011-02-03 22:01:54 +0000 | [diff] [blame] | 414 | |
Antoine Pitrou | 8d8f7c5 | 2011-02-05 11:40:05 +0000 | [diff] [blame] | 415 | |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 416 | Consider using optional static type checking |
| 417 | -------------------------------------------- |
Antoine Pitrou | 8d8f7c5 | 2011-02-05 11:40:05 +0000 | [diff] [blame] | 418 | |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 419 | Another way to help port your code is to use a static type checker like |
| 420 | mypy_ or pytype_ on your code. These tools can be used to analyze your code as |
| 421 | if it's being run under Python 2, then you can run the tool a second time as if |
| 422 | your code is running under Python 3. By running a static type checker twice like |
| 423 | this you can discover if you're e.g. misusing binary data type in one version |
| 424 | of Python compared to another. If you add optional type hints to your code you |
| 425 | can also explicitly state whether your APIs use textual or binary data, helping |
| 426 | to make sure everything functions as expected in both versions of Python. |
Antoine Pitrou | 8d8f7c5 | 2011-02-05 11:40:05 +0000 | [diff] [blame] | 427 | |
Antoine Pitrou | 8d8f7c5 | 2011-02-05 11:40:05 +0000 | [diff] [blame] | 428 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 429 | .. _2to3: https://docs.python.org/3/library/2to3.html |
Brett Cannon | 17be09c | 2014-12-05 18:11:05 -0500 | [diff] [blame] | 430 | .. _caniusepython3: https://pypi.python.org/pypi/caniusepython3 |
Brett Cannon | 90783eb | 2014-12-12 15:13:43 -0500 | [diff] [blame] | 431 | .. _cheat sheet: http://python-future.org/compatible_idioms.html |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 432 | .. _coverage.py: https://pypi.python.org/pypi/coverage |
| 433 | .. _Futurize: http://python-future.org/automatic_conversion.html |
Brett Cannon | adcb654 | 2016-03-18 13:23:58 -0700 | [diff] [blame] | 434 | .. _importlib: https://docs.python.org/3/library/importlib.html#module-importlib |
| 435 | .. _importlib2: https://pypi.python.org/pypi/importlib2 |
Sanyam Khurana | 338cd83 | 2018-01-20 05:55:37 +0530 | [diff] [blame^] | 436 | .. _Modernize: https://python-modernize.readthedocs.io/ |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 437 | .. _mypy: http://mypy-lang.org/ |
Brett Cannon | 9ca21b3 | 2014-01-07 11:52:04 -0500 | [diff] [blame] | 438 | .. _Porting to Python 3: http://python3porting.com/ |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 439 | .. _Pylint: https://pypi.python.org/pypi/pylint |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 440 | |
Sanyam Khurana | 338cd83 | 2018-01-20 05:55:37 +0530 | [diff] [blame^] | 441 | .. _Python 3 Q & A: https://ncoghlan-devs-python-notes.readthedocs.io/en/latest/python3/questions_and_answers.html |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 442 | |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 443 | .. _pytype: https://github.com/google/pytype |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 444 | .. _python-future: http://python-future.org/ |
Georg Brandl | e73778c | 2014-10-29 08:36:35 +0100 | [diff] [blame] | 445 | .. _python-porting: https://mail.python.org/mailman/listinfo/python-porting |
Brett Cannon | 9ca21b3 | 2014-01-07 11:52:04 -0500 | [diff] [blame] | 446 | .. _six: https://pypi.python.org/pypi/six |
| 447 | .. _tox: https://pypi.python.org/pypi/tox |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 448 | .. _trove classifier: https://pypi.python.org/pypi?%3Aaction=list_classifiers |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 449 | |
Brett Cannon | 6b33519 | 2014-12-05 10:56:12 -0500 | [diff] [blame] | 450 | .. _"What's New": https://docs.python.org/3/whatsnew/index.html |
Brett Cannon | 5866719 | 2016-12-17 12:37:20 -0800 | [diff] [blame] | 451 | |
Sanyam Khurana | 1b4587a | 2017-12-06 22:09:33 +0530 | [diff] [blame] | 452 | .. _Why Python 3 exists: https://snarky.ca/why-python-3-exists |