Forward port macOS installer updates from 3.7/3.8 (GH-21132) (GH-21135)
- fix installer builds when using latest versions of Python 3
- fix installer builds on newer macOS releases with SIP
- Python Launcher app factory defaults now use python3
diff --git a/Mac/BuildScript/README.rst b/Mac/BuildScript/README.rst
index f8b25fa..94a6bb2 100644
--- a/Mac/BuildScript/README.rst
+++ b/Mac/BuildScript/README.rst
@@ -7,216 +7,84 @@
$DESTROOT, massages that installation to remove .pyc files and such, creates
an Installer package from the installation plus other files in ``resources``
and ``scripts`` and placed that on a ``.dmg`` disk image.
+The installer package built on the dmg is a macOS bundle format installer
+package. This format is deprecated and is no longer supported by modern
+macOS systems; it is usable on macOS 10.6 and earlier systems.
+To be usable on newer versions of macOS, the bits in the bundle package
+must be assembled in a macOS flat installer package, using current
+versions of the pkgbuild and productbuild utilities. To pass macoS
+Gatekeeper download quarantine, the final package must be signed
+with a valid Apple Developer ID certificate using productsign.
+Starting with macOS 10.15 Catalina, Gatekeeper now also requires
+that installer packages are submitted to and pass Apple's automated
+notarization service using the altool command. To pass notarization,
+the binaries included in the package must be built with at least
+the macOS 10.9 SDK, mout now be signed with the codesign utility
+and executables must opt in to the hardened run time option with
+any necessary entitlements. Details of these processes are
+available in the on-line Apple Developer Documentation and man pages.
-For Python 3.4.0, PSF practice is to build two installer variants
-for each release.
+As of 3.8.0 and 3.7.7, PSF practice is to build one installer variants
+for each release. Note that as of this writing, no Pythons support
+building on a newer version of macOS that will run on older versions
+by setting MACOSX_DEPLOYMENT_TARGET. This is because the various
+Python C modules do not yet support runtime testing of macOS
+feature availability (for example, by using macOS AvailabilityMacros.h
+and weak-linking). To build a Python that is to be used on a
+range of macOS releases, always build on the oldest release to be
+supported; the necessary shared libraries for that release will
+normally also be available on later systems, with the occasional
+exception such as the removal of 32-bit libraries in macOS 10.15.
-1. 32-bit-only, i386 and PPC universal, capable on running on all machines
- supported by Mac OS X 10.5 through (at least) 10.9::
+build-installer requires Apple Developer tools, either from the
+Command Line Tools package or from a full Xcode installation.
+You should use the most recent version of either for the operating
+system version in use. (One notable exception: on macOS 10.6,
+Snow Leopard, use Xcode 3, not Xcode 4 which was released later
+in the 10.6 support cycle.)
+
+1. 64-bit, x86_64, for OS X 10.9 (and later)::
/path/to/bootstrap/python2.7 build-installer.py \
- --sdk-path=/Developer/SDKs/MacOSX10.5.sdk \
- --universal-archs=32-bit \
- --dep-target=10.5
+ --universal-archs=intel-64 \
+ --dep-target=10.9
- builds the following third-party libraries
- * NCurses 5.9 (http://bugs.python.org/issue15037)
- * SQLite 3.8.11
- * XZ 5.0.5
-
- - uses system-supplied versions of third-party libraries
-
- * readline module links with Apple BSD editline (libedit)
-
- - requires ActiveState ``Tcl/Tk 8.4`` (currently 8.4.20) to be installed for building
-
- - recommended build environment:
-
- * Mac OS X 10.5.8 Intel or PPC
- * Xcode 3.1.4
- * ``MacOSX10.5`` SDK
- * ``MACOSX_DEPLOYMENT_TARGET=10.5``
- * Apple ``gcc-4.2``
- * bootstrap non-framework Python 2.7 for documentation build with
- Sphinx (as of 3.4.1)
-
- - alternate build environments:
-
- * Mac OS X 10.6.8 with Xcode 3.2.6
- - need to change ``/System/Library/Frameworks/{Tcl,Tk}.framework/Version/Current`` to ``8.4``
- * Note Xcode 4.* does not support building for PPC so cannot be used for this build
-
-2. 64-bit / 32-bit, x86_64 and i386 universal, for OS X 10.6 (and later)::
-
- /path/to/bootstrap/python2.7 build-installer.py \
- --sdk-path=/Developer/SDKs/MacOSX10.6.sdk \
- --universal-archs=intel \
- --dep-target=10.6
-
- - builds the following third-party libraries
-
- * NCurses 5.9 (http://bugs.python.org/issue15037)
- * SQLite 3.8.11
- * XZ 5.0.5
-
- - uses system-supplied versions of third-party libraries
-
- * readline module links with Apple BSD editline (libedit)
-
- - requires ActiveState Tcl/Tk 8.5.15.1 (or later) to be installed for building
-
- - recommended build environment:
-
- * Mac OS X 10.6.8 (or later)
- * Xcode 3.2.6
- * ``MacOSX10.6`` SDK
- * ``MACOSX_DEPLOYMENT_TARGET=10.6``
- * Apple ``gcc-4.2``
- * bootstrap non-framework Python 2.7 for documentation build with
- Sphinx (as of 3.4.1)
-
- - alternate build environments:
-
- * none. Xcode 4.x currently supplies two C compilers.
- ``llvm-gcc-4.2.1`` has been found to miscompile Python 3.3.x and
- produce a non-functional Python executable. As it appears to be
- considered a migration aid by Apple and is not likely to be fixed,
- its use should be avoided. The other compiler, ``clang``, has been
- undergoing rapid development. While it appears to have become
- production-ready in the most recent Xcode 5 releases, the versions
- available on the deprecated Xcode 4.x for 10.6 were early releases
- and did not receive the level of exposure in production environments
- that the Xcode 3 gcc-4.2 compiler has had.
-
-
-* For Python 2.7.x and 3.2.x, the 32-bit-only installer was configured to
- support Mac OS X 10.3.9 through (at least) 10.6. Because it is
- believed that there are few systems still running OS X 10.3 or 10.4
- and because it has become increasingly difficult to test and
- support the differences in these earlier systems, as of Python 3.3.0 the PSF
- 32-bit installer no longer supports them. For reference in building such
- an installer yourself, the details are::
-
- /usr/bin/python build-installer.py \
- --sdk-path=/Developer/SDKs/MacOSX10.4u.sdk \
- --universal-archs=32-bit \
- --dep-target=10.3
-
- - builds the following third-party libraries
-
- * Bzip2
+ * OpenSSL 1.1.1
+ * Tcl/Tk 8.6
* NCurses
- * GNU Readline (GPL)
- * SQLite 3
+ * SQLite
* XZ
- * Zlib 1.2.3
- * Oracle Sleepycat DB 4.8 (Python 2.x only)
+ * libffi
- - requires ActiveState ``Tcl/Tk 8.4`` (currently 8.4.20) to be installed for building
+ - uses system-supplied versions of third-party libraries
+
+ * readline module links with Apple BSD editline (libedit)
+ * zlib
+ * bz2
- recommended build environment:
- * Mac OS X 10.5.8 PPC or Intel
- * Xcode 3.1.4 (or later)
- * ``MacOSX10.4u`` SDK (later SDKs do not support PPC G3 processors)
- * ``MACOSX_DEPLOYMENT_TARGET=10.3``
- * Apple ``gcc-4.0``
- * system Python 2.5 for documentation build with Sphinx
-
- - alternate build environments:
-
- * Mac OS X 10.6.8 with Xcode 3.2.6
- - need to change ``/System/Library/Frameworks/{Tcl,Tk}.framework/Version/Current`` to ``8.4``
-
+ * Mac OS X 10.9.5
+ * Xcode Command Line Tools 6.2
+ * ``MacOSX10.9`` SDK
+ * ``MACOSX_DEPLOYMENT_TARGET=10.9``
+ * Apple ``clang``
General Prerequisites
---------------------
-* No Fink (in ``/sw``) or MacPorts (in ``/opt/local``) or other local
- libraries or utilities (in ``/usr/local``) as they could
+* No Fink (in ``/sw``) or MacPorts (in ``/opt/local``) or Homebrew or
+ other local libraries or utilities (in ``/usr/local``) as they could
interfere with the build.
-* The documentation for the release is built using Sphinx
- because it is included in the installer. For 2.7.x and 3.x.x up to and
- including 3.4.0, the ``Doc/Makefile`` uses ``svn`` to download repos of
- ``Sphinx`` and its dependencies. Beginning with 3.4.1, the ``Doc/Makefile``
- assumes there is an externally-provided ``sphinx-build`` and requires at
- least Python 2.6 to run. Because of this, it is no longer possible to
- build a 3.4.1 or later installer on OS X 10.5 using the Apple-supplied
- Python 2.5.
-
* It is safest to start each variant build with an empty source directory
- populated with a fresh copy of the untarred source.
+ populated with a fresh copy of the untarred source or a source repo.
* It is recommended that you remove any existing installed version of the
Python being built::
sudo rm -rf /Library/Frameworks/Python.framework/Versions/n.n
-
-The Recipe
-----------
-
-Here are the steps you need to follow to build a Python installer:
-
-* Run ``build-installer.py``. Optionally you can pass a number of arguments
- to specify locations of various files. Please see the top of
- ``build-installer.py`` for its usage.
-
- Running this script takes some time, it will not only build Python itself
- but also some 3th-party libraries that are needed for extensions.
-
-* When done the script will tell you where the DMG image is (by default
- somewhere in ``/tmp/_py``).
-
-Building other universal installers
-...................................
-
-It is also possible to build a 4-way universal installer that runs on
-OS X 10.5 Leopard or later::
-
- /usr/bin/python /build-installer.py \
- --dep-target=10.5
- --universal-archs=all
- --sdk-path=/Developer/SDKs/MacOSX10.5.sdk
-
-This requires that the deployment target is 10.5, and hence
-also that you are building on at least OS X 10.5. 4-way includes
-``i386``, ``x86_64``, ``ppc``, and ``ppc64`` (G5). ``ppc64`` executable
-variants can only be run on G5 machines running 10.5. Note that,
-while OS X 10.6 is only supported on Intel-based machines, it is possible
-to run ``ppc`` (32-bit) executables unmodified thanks to the Rosetta ppc
-emulation in OS X 10.5 and 10.6. The 4-way installer variant must be
-built with Xcode 3. It is not regularly built or tested.
-
-Other ``--universal-archs`` options are ``64-bit`` (``x86_64``, ``ppc64``),
-and ``3-way`` (``ppc``, ``i386``, ``x86_64``). None of these options
-are regularly exercised; use at your own risk.
-
-
-Testing
--------
-
-Ideally, the resulting binaries should be installed and the test suite run
-on all supported OS X releases and architectures. As a practical matter,
-that is generally not possible. At a minimum, variant 1 should be run on
-a PPC G4 system with OS X 10.5 and at least one Intel system running OS X
-10.9, 10.8, 10.7, 10.6, or 10.5. Variant 2 should be run on 10.9, 10.8,
-10.7, and 10.6 systems in both 32-bit and 64-bit modes.::
-
- /usr/local/bin/pythonn.n -m test -w -u all,-largefile
- /usr/local/bin/pythonn.n-32 -m test -w -u all
-
-Certain tests will be skipped and some cause the interpreter to fail
-which will likely generate ``Python quit unexpectedly`` alert messages
-to be generated at several points during a test run. These are normal
-during testing and can be ignored.
-
-It is also recommend to launch IDLE and verify that it is at least
-functional. Double-click on the IDLE app icon in ``/Applications/Python n.n``.
-It should also be tested from the command line::
-
- /usr/local/bin/idlen.n
-
diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py
index a2cba32..4fab488 100755
--- a/Mac/BuildScript/build-installer.py
+++ b/Mac/BuildScript/build-installer.py
@@ -2,6 +2,20 @@
"""
This script is used to build "official" universal installers on macOS.
+NEW for 3.9.0 and backports:
+- 2.7 end-of-life issues:
+ - Python 3 installs now update the Current version link
+ in /Library/Frameworks/Python.framework/Versions
+- fully support running under Python 3 as well as 2.7
+- support building on newer macOS systems with SIP
+- fully support building on macOS 10.9+
+- support 10.6+ on best effort
+- support bypassing docs build by supplying a prebuilt
+ docs html tarball in the third-party source library,
+ in the format and filename conventional of those
+ downloadable from python.org:
+ python-3.x.y-docs-html.tar.bz2
+
NEW for 3.7.0:
- support Intel 64-bit-only () and 32-bit-only installer builds
- build and use internal Tcl/Tk 8.6 for 10.6+ builds
@@ -14,28 +28,7 @@
- use generic "gcc" as compiler (CC env var) rather than "gcc-4.2"
TODO:
-- support SDKROOT and DEVELOPER_DIR xcrun env variables
-- test with 10.5 and 10.4 and determine support status
-
-Please ensure that this script keeps working with Python 2.5, to avoid
-bootstrap issues (/usr/bin/python is Python 2.5 on OSX 10.5). Doc builds
-use current versions of Sphinx and require a reasonably current python3.
-Sphinx and dependencies are installed into a venv using the python3's pip
-so will fetch them from PyPI if necessary. Since python3 is now used for
-Sphinx, build-installer.py should also be converted to use python3!
-
-For 3.7.0, when building for a 10.6 or higher deployment target,
-build-installer builds and links with its own copy of Tcl/Tk 8.6.
-Otherwise, it requires an installed third-party version of
-Tcl/Tk 8.4 (for OS X 10.4 and 10.5 deployment targets), Tcl/TK 8.5
-(for 10.6 or later), or Tcl/TK 8.6 (for 10.9 or later)
-installed in /Library/Frameworks. When installed,
-the Python built by this script will attempt to dynamically link first to
-Tcl and Tk frameworks in /Library/Frameworks if available otherwise fall
-back to the ones in /System/Library/Framework. For the build, we recommend
-installing the most recent ActiveTcl 8.6. 8.5, or 8.4 version, depending
-on the deployment target. The actual version linked to depends on the
-path of /Library/Frameworks/{Tcl,Tk}.framework/Versions/Current.
+- test building with SDKROOT and DEVELOPER_DIR xcrun env variables
Usage: see USAGE variable in the script.
"""
@@ -56,14 +49,15 @@
INCLUDE_TIMESTAMP = 1
VERBOSE = 1
-from plistlib import Plist
+RUNNING_ON_PYTHON2 = sys.version_info.major == 2
-try:
+if RUNNING_ON_PYTHON2:
from plistlib import writePlist
-except ImportError:
- # We're run using python2.3
- def writePlist(plist, path):
- plist.write(path)
+else:
+ from plistlib import dump
+ def writePlist(path, plist):
+ with open(plist, 'wb') as fp:
+ dump(path, fp)
def shellQuote(value):
"""
@@ -1096,7 +1090,7 @@
if not os.path.exists(htmlDir):
# Create virtual environment for docs builds with blurb and sphinx
runCommand('make venv')
- runCommand('venv/bin/python3 -m pip install -U Sphinx==2.2.0')
+ runCommand('venv/bin/python3 -m pip install -U Sphinx==2.3.1')
runCommand('make html PYTHON=venv/bin/python')
os.rename(htmlDir, docdir)
os.chdir(curDir)
@@ -1125,8 +1119,7 @@
# Since the extra libs are not in their installed framework location
# during the build, augment the library path so that the interpreter
# will find them during its extension import sanity checks.
- os.environ['DYLD_LIBRARY_PATH'] = os.path.join(WORKDIR,
- 'libraries', 'usr', 'local', 'lib')
+
print("Running configure...")
runCommand("%s -C --enable-framework --enable-universalsdk=/ "
"--with-universal-archs=%s "
@@ -1134,12 +1127,15 @@
"%s "
"%s "
"%s "
+ "%s "
"LDFLAGS='-g -L%s/libraries/usr/local/lib' "
"CFLAGS='-g -I%s/libraries/usr/local/include' 2>&1"%(
shellQuote(os.path.join(SRCDIR, 'configure')),
UNIVERSALARCHS,
(' ', '--with-computed-gotos ')[PYTHON_3],
(' ', '--without-ensurepip ')[PYTHON_3],
+ (' ', "--with-openssl='%s/libraries/usr/local'"%(
+ shellQuote(WORKDIR)[1:-1],))[PYTHON_3],
(' ', "--with-tcltk-includes='-I%s/libraries/usr/local/include'"%(
shellQuote(WORKDIR)[1:-1],))[internalTk()],
(' ', "--with-tcltk-libs='-L%s/libraries/usr/local/lib -ltcl8.6 -ltk8.6'"%(
@@ -1147,6 +1143,24 @@
shellQuote(WORKDIR)[1:-1],
shellQuote(WORKDIR)[1:-1]))
+ # As of macOS 10.11 with SYSTEM INTEGRITY PROTECTION, DYLD_*
+ # environment variables are no longer automatically inherited
+ # by child processes from their parents. We used to just set
+ # DYLD_LIBRARY_PATH, pointing to the third-party libs,
+ # in build-installer.py's process environment and it was
+ # passed through the make utility into the environment of
+ # setup.py. Instead, we now append DYLD_LIBRARY_PATH to
+ # the existing RUNSHARED configuration value when we call
+ # make for extension module builds.
+
+ runshared_for_make = "".join([
+ " RUNSHARED=",
+ "'",
+ grepValue("Makefile", "RUNSHARED"),
+ ' DYLD_LIBRARY_PATH=',
+ os.path.join(WORKDIR, 'libraries', 'usr', 'local', 'lib'),
+ "'" ])
+
# Look for environment value BUILDINSTALLER_BUILDPYTHON_MAKE_EXTRAS
# and, if defined, append its value to the make command. This allows
# us to pass in version control tags, like GITTAG, to a build from a
@@ -1161,21 +1175,24 @@
make_extras = os.getenv("BUILDINSTALLER_BUILDPYTHON_MAKE_EXTRAS")
if make_extras:
- make_cmd = "make " + make_extras
+ make_cmd = "make " + make_extras + runshared_for_make
else:
- make_cmd = "make"
+ make_cmd = "make" + runshared_for_make
print("Running " + make_cmd)
runCommand(make_cmd)
- print("Running make install")
- runCommand("make install DESTDIR=%s"%(
- shellQuote(rootDir)))
+ make_cmd = "make install DESTDIR=%s %s"%(
+ shellQuote(rootDir),
+ runshared_for_make)
+ print("Running " + make_cmd)
+ runCommand(make_cmd)
- print("Running make frameworkinstallextras")
- runCommand("make frameworkinstallextras DESTDIR=%s"%(
- shellQuote(rootDir)))
+ make_cmd = "make frameworkinstallextras DESTDIR=%s %s"%(
+ shellQuote(rootDir),
+ runshared_for_make)
+ print("Running " + make_cmd)
+ runCommand(make_cmd)
- del os.environ['DYLD_LIBRARY_PATH']
print("Copying required shared libraries")
if os.path.exists(os.path.join(WORKDIR, 'libraries', 'Library')):
build_lib_dir = os.path.join(
@@ -1304,7 +1321,13 @@
data = fp.read()
fp.close()
# create build_time_vars dict
- exec(data)
+ if RUNNING_ON_PYTHON2:
+ exec(data)
+ else:
+ g_dict = {}
+ l_dict = {}
+ exec(data, g_dict, l_dict)
+ build_time_vars = l_dict['build_time_vars']
vars = {}
for k, v in build_time_vars.items():
if type(v) == type(''):
@@ -1421,7 +1444,7 @@
vers = getFullVersion()
major, minor = getVersionMajorMinor()
- pl = Plist(
+ pl = dict(
CFBundleGetInfoString="Python.%s %s"%(pkgname, vers,),
CFBundleIdentifier='org.python.Python.%s'%(pkgname,),
CFBundleName='Python.%s'%(pkgname,),
@@ -1443,7 +1466,7 @@
)
writePlist(pl, os.path.join(packageContents, 'Info.plist'))
- pl = Plist(
+ pl = dict(
IFPkgDescriptionDescription=readme,
IFPkgDescriptionTitle=recipe.get('long_name', "Python.%s"%(pkgname,)),
IFPkgDescriptionVersion=vers,
@@ -1459,7 +1482,7 @@
vers = getFullVersion()
major, minor = getVersionMajorMinor()
- pl = Plist(
+ pl = dict(
CFBundleGetInfoString="Python %s"%(vers,),
CFBundleIdentifier='org.python.Python',
CFBundleName='Python',
@@ -1512,7 +1535,7 @@
os.mkdir(rsrcDir)
makeMpkgPlist(os.path.join(pkgroot, 'Info.plist'))
- pl = Plist(
+ pl = dict(
IFPkgDescriptionTitle="Python",
IFPkgDescriptionVersion=getVersion(),
)