Guido van Rossum | e7e578f | 1995-08-04 04:00:20 +0000 | [diff] [blame] | 1 | """New import scheme with package support. |
| 2 | |
Guido van Rossum | 507f15b | 1996-05-28 15:52:00 +0000 | [diff] [blame] | 3 | Quick Reference |
| 4 | --------------- |
| 5 | |
| 6 | - To enable package support, execute "import ni" before importing any |
| 7 | packages. Importing this module automatically installs the relevant |
| 8 | import hooks. |
| 9 | |
| 10 | - To create a package named spam containing sub-modules ham, bacon and |
| 11 | eggs, create a directory spam somewhere on Python's module search |
| 12 | path (i.e. spam's parent directory must be one of the directories in |
| 13 | sys.path or $PYTHONPATH); then create files ham.py, bacon.py and |
| 14 | eggs.py inside spam. |
| 15 | |
| 16 | - To import module ham from package spam and use function hamneggs() |
| 17 | from that module, you can either do |
| 18 | |
| 19 | import spam.ham # *not* "import spam" !!! |
| 20 | spam.ham.hamneggs() |
| 21 | |
| 22 | or |
| 23 | |
| 24 | from spam import ham |
| 25 | ham.hamneggs() |
| 26 | |
| 27 | or |
| 28 | |
| 29 | from spam.ham import hamneggs |
| 30 | hamneggs() |
| 31 | |
| 32 | - Importing just "spam" does not do what you expect: it creates an |
| 33 | empty package named spam if one does not already exist, but it does |
| 34 | not import spam's submodules. The only submodule that is guaranteed |
| 35 | to be imported is spam.__init__, if it exists. Note that |
| 36 | spam.__init__ is a submodule of package spam. It can reference to |
| 37 | spam's namespace via the '__.' prefix, for instance |
| 38 | |
| 39 | __.spam_inited = 1 # Set a package-level variable |
| 40 | |
| 41 | |
| 42 | |
| 43 | Theory of Operation |
| 44 | ------------------- |
| 45 | |
Guido van Rossum | e7e578f | 1995-08-04 04:00:20 +0000 | [diff] [blame] | 46 | A Package is a module that can contain other modules. Packages can be |
| 47 | nested. Package introduce dotted names for modules, like P.Q.M, which |
| 48 | could correspond to a file P/Q/M.py found somewhere on sys.path. It |
| 49 | is possible to import a package itself, though this makes little sense |
| 50 | unless the package contains a module called __init__. |
| 51 | |
| 52 | A package has two variables that control the namespace used for |
| 53 | packages and modules, both initialized to sensible defaults the first |
| 54 | time the package is referenced. |
| 55 | |
| 56 | (1) A package's *module search path*, contained in the per-package |
| 57 | variable __path__, defines a list of *directories* where submodules or |
| 58 | subpackages of the package are searched. It is initialized to the |
| 59 | directory containing the package. Setting this variable to None makes |
| 60 | the module search path default to sys.path (this is not quite the same |
| 61 | as setting it to sys.path, since the latter won't track later |
| 62 | assignments to sys.path). |
| 63 | |
| 64 | (2) A package's *import domain*, contained in the per-package variable |
| 65 | __domain__, defines a list of *packages* that are searched (using |
| 66 | their respective module search paths) to satisfy imports. It is |
| 67 | initialized to the list cosisting of the package itself, its parent |
| 68 | package, its parent's parent, and so on, ending with the root package |
| 69 | (the nameless package containing all top-level packages and modules, |
| 70 | whose module search path is None, implying sys.path). |
| 71 | |
| 72 | The default domain implements a search algorithm called "expanding |
| 73 | search". An alternative search algorithm called "explicit search" |
| 74 | fixes the import search path to contain only the root package, |
| 75 | requiring the modules in the package to name all imported modules by |
| 76 | their full name. The convention of using '__' to refer to the current |
| 77 | package (both as a per-module variable and in module names) can be |
| 78 | used by packages using explicit search to refer to modules in the same |
| 79 | package; this combination is known as "explicit-relative search". |
| 80 | |
| 81 | The PackageImporter and PackageLoader classes together implement the |
| 82 | following policies: |
| 83 | |
| 84 | - There is a root package, whose name is ''. It cannot be imported |
| 85 | directly but may be referenced, e.g. by using '__' from a top-level |
| 86 | module. |
| 87 | |
| 88 | - In each module or package, the variable '__' contains a reference to |
| 89 | the parent package; in the root package, '__' points to itself. |
| 90 | |
| 91 | - In the name for imported modules (e.g. M in "import M" or "from M |
| 92 | import ..."), a leading '__' refers to the current package (i.e. |
| 93 | the package containing the current module); leading '__.__' and so |
| 94 | on refer to the current package's parent, and so on. The use of |
| 95 | '__' elsewhere in the module name is not supported. |
| 96 | |
| 97 | - Modules are searched using the "expanding search" algorithm by |
| 98 | virtue of the default value for __domain__. |
| 99 | |
| 100 | - If A.B.C is imported, A is searched using __domain__; then |
| 101 | subpackage B is searched in A using its __path__, and so on. |
| 102 | |
| 103 | - Built-in modules have priority: even if a file sys.py exists in a |
| 104 | package, "import sys" imports the built-in sys module. |
| 105 | |
| 106 | - The same holds for frozen modules, for better or for worse. |
| 107 | |
| 108 | - Submodules and subpackages are not automatically loaded when their |
| 109 | parent packages is loaded. |
| 110 | |
| 111 | - The construct "from package import *" is illegal. (It can still be |
| 112 | used to import names from a module.) |
| 113 | |
| 114 | - When "from package import module1, module2, ..." is used, those |
| 115 | modules are explicitly loaded. |
| 116 | |
| 117 | - When a package is loaded, if it has a submodule __init__, that |
| 118 | module is loaded. This is the place where required submodules can |
| 119 | be loaded, the __path__ variable extended, etc. The __init__ module |
| 120 | is loaded even if the package was loaded only in order to create a |
| 121 | stub for a sub-package: if "import P.Q.R" is the first reference to |
| 122 | P, and P has a submodule __init__, P.__init__ is loaded before P.Q |
| 123 | is even searched. |
| 124 | |
| 125 | Caveats: |
| 126 | |
| 127 | - It is possible to import a package that has no __init__ submodule; |
| 128 | this is not particularly useful but there may be useful applications |
| 129 | for it (e.g. to manipulate its search paths from the outside!). |
| 130 | |
| 131 | - There are no special provisions for os.chdir(). If you plan to use |
| 132 | os.chdir() before you have imported all your modules, it is better |
| 133 | not to have relative pathnames in sys.path. (This could actually be |
| 134 | fixed by changing the implementation of path_join() in the hook to |
| 135 | absolutize paths.) |
| 136 | |
| 137 | - Packages and modules are introduced in sys.modules as soon as their |
| 138 | loading is started. When the loading is terminated by an exception, |
| 139 | the sys.modules entries remain around. |
| 140 | |
| 141 | - There are no special measures to support mutually recursive modules, |
| 142 | but it will work under the same conditions where it works in the |
| 143 | flat module space system. |
| 144 | |
| 145 | - Sometimes dummy entries (whose value is None) are entered in |
| 146 | sys.modules, to indicate that a particular module does not exist -- |
| 147 | this is done to speed up the expanding search algorithm when a |
| 148 | module residing at a higher level is repeatedly imported (Python |
| 149 | promises that importing a previously imported module is cheap!) |
| 150 | |
| 151 | - Although dynamically loaded extensions are allowed inside packages, |
| 152 | the current implementation (hardcoded in the interpreter) of their |
| 153 | initialization may cause problems if an extension invokes the |
| 154 | interpreter during its initialization. |
| 155 | |
| 156 | - reload() may find another version of the module only if it occurs on |
| 157 | the package search path. Thus, it keeps the connection to the |
| 158 | package to which the module belongs, but may find a different file. |
| 159 | |
| 160 | XXX Need to have an explicit name for '', e.g. '__root__'. |
| 161 | |
| 162 | """ |
| 163 | |
| 164 | |
| 165 | import imp |
| 166 | import string |
| 167 | import sys |
| 168 | import __builtin__ |
| 169 | |
| 170 | import ihooks |
| 171 | from ihooks import ModuleLoader, ModuleImporter |
| 172 | |
| 173 | |
| 174 | class PackageLoader(ModuleLoader): |
| 175 | |
| 176 | """A subclass of ModuleLoader with package support. |
| 177 | |
| 178 | find_module_in_dir() will succeed if there's a subdirectory with |
| 179 | the given name; load_module() will create a stub for a package and |
| 180 | load its __init__ module if it exists. |
| 181 | |
| 182 | """ |
| 183 | |
| 184 | def find_module_in_dir(self, name, dir): |
| 185 | if dir is not None: |
| 186 | dirname = self.hooks.path_join(dir, name) |
| 187 | if self.hooks.path_isdir(dirname): |
| 188 | return None, dirname, ('', '', 'PACKAGE') |
| 189 | return ModuleLoader.find_module_in_dir(self, name, dir) |
| 190 | |
| 191 | def load_module(self, name, stuff): |
| 192 | file, filename, info = stuff |
| 193 | suff, mode, type = info |
| 194 | if type == 'PACKAGE': |
| 195 | return self.load_package(name, stuff) |
| 196 | if sys.modules.has_key(name): |
| 197 | m = sys.modules[name] |
| 198 | else: |
| 199 | sys.modules[name] = m = imp.new_module(name) |
| 200 | self.set_parent(m) |
| 201 | if type == imp.C_EXTENSION and '.' in name: |
| 202 | return self.load_dynamic(name, stuff) |
| 203 | else: |
| 204 | return ModuleLoader.load_module(self, name, stuff) |
| 205 | |
| 206 | def load_dynamic(self, name, stuff): |
| 207 | file, filename, (suff, mode, type) = stuff |
| 208 | # Hack around restriction in imp.load_dynamic() |
| 209 | i = string.rfind(name, '.') |
| 210 | tail = name[i+1:] |
| 211 | if sys.modules.has_key(tail): |
| 212 | save = sys.modules[tail] |
| 213 | else: |
| 214 | save = None |
| 215 | sys.modules[tail] = imp.new_module(name) |
| 216 | try: |
| 217 | m = imp.load_dynamic(tail, filename, file) |
| 218 | finally: |
| 219 | if save: |
| 220 | sys.modules[tail] = save |
| 221 | else: |
| 222 | del sys.modules[tail] |
| 223 | sys.modules[name] = m |
| 224 | return m |
| 225 | |
| 226 | def load_package(self, name, stuff): |
| 227 | file, filename, info = stuff |
| 228 | if sys.modules.has_key(name): |
| 229 | package = sys.modules[name] |
| 230 | else: |
| 231 | sys.modules[name] = package = imp.new_module(name) |
| 232 | package.__path__ = [filename] |
| 233 | self.init_package(package) |
| 234 | return package |
| 235 | |
| 236 | def init_package(self, package): |
| 237 | self.set_parent(package) |
| 238 | self.set_domain(package) |
| 239 | self.call_init_module(package) |
| 240 | |
| 241 | def set_parent(self, m): |
| 242 | name = m.__name__ |
| 243 | if '.' in name: |
| 244 | name = name[:string.rfind(name, '.')] |
| 245 | else: |
| 246 | name = '' |
| 247 | m.__ = sys.modules[name] |
| 248 | |
| 249 | def set_domain(self, package): |
| 250 | name = package.__name__ |
| 251 | package.__domain__ = domain = [name] |
| 252 | while '.' in name: |
| 253 | name = name[:string.rfind(name, '.')] |
| 254 | domain.append(name) |
| 255 | if name: |
| 256 | domain.append('') |
| 257 | |
| 258 | def call_init_module(self, package): |
| 259 | stuff = self.find_module('__init__', package.__path__) |
| 260 | if stuff: |
| 261 | m = self.load_module(package.__name__ + '.__init__', stuff) |
| 262 | package.__init__ = m |
| 263 | |
| 264 | |
| 265 | class PackageImporter(ModuleImporter): |
| 266 | |
| 267 | """Importer that understands packages and '__'.""" |
| 268 | |
| 269 | def __init__(self, loader = None, verbose = 0): |
| 270 | ModuleImporter.__init__(self, |
| 271 | loader or PackageLoader(None, verbose), verbose) |
| 272 | |
| 273 | def import_module(self, name, globals={}, locals={}, fromlist=[]): |
| 274 | if globals.has_key('__'): |
| 275 | package = globals['__'] |
| 276 | else: |
| 277 | # No calling context, assume in root package |
| 278 | package = sys.modules[''] |
| 279 | if name[:3] in ('__.', '__'): |
| 280 | p = package |
| 281 | name = name[3:] |
| 282 | while name[:3] in ('__.', '__'): |
Guido van Rossum | 974e46c | 1996-09-10 17:38:20 +0000 | [diff] [blame] | 283 | p = p.__ |
Guido van Rossum | e7e578f | 1995-08-04 04:00:20 +0000 | [diff] [blame] | 284 | name = name[3:] |
| 285 | if not name: |
| 286 | return self.finish(package, p, '', fromlist) |
| 287 | if '.' in name: |
| 288 | i = string.find(name, '.') |
| 289 | name, tail = name[:i], name[i:] |
| 290 | else: |
| 291 | tail = '' |
| 292 | mname = p.__name__ and p.__name__+'.'+name or name |
| 293 | m = self.get1(mname) |
| 294 | return self.finish(package, m, tail, fromlist) |
| 295 | if '.' in name: |
| 296 | i = string.find(name, '.') |
| 297 | name, tail = name[:i], name[i:] |
| 298 | else: |
| 299 | tail = '' |
| 300 | for pname in package.__domain__: |
| 301 | mname = pname and pname+'.'+name or name |
| 302 | m = self.get0(mname) |
| 303 | if m: break |
| 304 | else: |
| 305 | raise ImportError, "No such module %s" % name |
| 306 | return self.finish(m, m, tail, fromlist) |
| 307 | |
| 308 | def finish(self, module, m, tail, fromlist): |
| 309 | # Got ....A; now get ....A.B.C.D |
| 310 | yname = m.__name__ |
| 311 | if tail and sys.modules.has_key(yname + tail): # Fast path |
| 312 | yname, tail = yname + tail, '' |
| 313 | m = self.get1(yname) |
| 314 | while tail: |
| 315 | i = string.find(tail, '.', 1) |
| 316 | if i > 0: |
| 317 | head, tail = tail[:i], tail[i:] |
| 318 | else: |
| 319 | head, tail = tail, '' |
| 320 | yname = yname + head |
| 321 | m = self.get1(yname) |
| 322 | |
| 323 | # Got ....A.B.C.D; now finalize things depending on fromlist |
| 324 | if not fromlist: |
| 325 | return module |
| 326 | if '__' in fromlist: |
| 327 | raise ImportError, "Can't import __ from anywhere" |
| 328 | if not hasattr(m, '__path__'): return m |
| 329 | if '*' in fromlist: |
| 330 | raise ImportError, "Can't import * from a package" |
| 331 | for f in fromlist: |
| 332 | if hasattr(m, f): continue |
| 333 | fname = yname + '.' + f |
| 334 | self.get1(fname) |
| 335 | return m |
| 336 | |
| 337 | def get1(self, name): |
| 338 | m = self.get(name) |
| 339 | if not m: |
| 340 | raise ImportError, "No module named %s" % name |
| 341 | return m |
| 342 | |
| 343 | def get0(self, name): |
| 344 | m = self.get(name) |
| 345 | if not m: |
| 346 | sys.modules[name] = None |
| 347 | return m |
| 348 | |
| 349 | def get(self, name): |
| 350 | # Internal routine to get or load a module when its parent exists |
| 351 | if sys.modules.has_key(name): |
| 352 | return sys.modules[name] |
| 353 | if '.' in name: |
| 354 | i = string.rfind(name, '.') |
| 355 | head, tail = name[:i], name[i+1:] |
| 356 | else: |
| 357 | head, tail = '', name |
| 358 | path = sys.modules[head].__path__ |
| 359 | stuff = self.loader.find_module(tail, path) |
| 360 | if not stuff: |
| 361 | return None |
| 362 | sys.modules[name] = m = self.loader.load_module(name, stuff) |
| 363 | if head: |
| 364 | setattr(sys.modules[head], tail, m) |
| 365 | return m |
| 366 | |
| 367 | def reload(self, module): |
| 368 | name = module.__name__ |
| 369 | if '.' in name: |
| 370 | i = string.rfind(name, '.') |
| 371 | head, tail = name[:i], name[i+1:] |
| 372 | path = sys.modules[head].__path__ |
| 373 | else: |
| 374 | tail = name |
| 375 | path = sys.modules[''].__path__ |
| 376 | stuff = self.loader.find_module(tail, path) |
| 377 | if not stuff: |
| 378 | raise ImportError, "No module named %s" % name |
| 379 | return self.loader.load_module(name, stuff) |
| 380 | |
| 381 | def unload(self, module): |
| 382 | if hasattr(module, '__path__'): |
| 383 | raise ImportError, "don't know how to unload packages yet" |
| 384 | PackageImporter.unload(self, module) |
| 385 | |
| 386 | def install(self): |
| 387 | if not sys.modules.has_key(''): |
| 388 | sys.modules[''] = package = imp.new_module('') |
| 389 | package.__path__ = None |
| 390 | self.loader.init_package(package) |
| 391 | for m in sys.modules.values(): |
| 392 | if not m: continue |
| 393 | if not hasattr(m, '__'): |
| 394 | self.loader.set_parent(m) |
| 395 | ModuleImporter.install(self) |
| 396 | |
| 397 | |
| 398 | def install(v = 0): |
| 399 | ihooks.install(PackageImporter(None, v)) |
| 400 | |
| 401 | def uninstall(): |
| 402 | ihooks.uninstall() |
| 403 | |
| 404 | def ni(v = 0): |
| 405 | install(v) |
| 406 | |
| 407 | def no(): |
| 408 | uninstall() |
| 409 | |
| 410 | def test(): |
| 411 | import pdb |
| 412 | try: |
| 413 | testproper() |
| 414 | except: |
Guido van Rossum | f15d159 | 1997-09-29 23:22:12 +0000 | [diff] [blame^] | 415 | sys.last_type, sys.last_value, sys.last_traceback = sys.exc_info() |
Guido van Rossum | e7e578f | 1995-08-04 04:00:20 +0000 | [diff] [blame] | 416 | print |
| 417 | print sys.last_type, ':', sys.last_value |
| 418 | print |
| 419 | pdb.pm() |
| 420 | |
| 421 | def testproper(): |
| 422 | install(1) |
| 423 | try: |
| 424 | import mactest |
| 425 | print dir(mactest) |
| 426 | raw_input('OK?') |
| 427 | finally: |
| 428 | uninstall() |
| 429 | |
| 430 | |
| 431 | if __name__ == '__main__': |
| 432 | test() |
Guido van Rossum | 507f15b | 1996-05-28 15:52:00 +0000 | [diff] [blame] | 433 | else: |
| 434 | install() |