Guido van Rossum | e7e578f | 1995-08-04 04:00:20 +0000 | [diff] [blame^] | 1 | """New import scheme with package support. |
| 2 | |
| 3 | A Package is a module that can contain other modules. Packages can be |
| 4 | nested. Package introduce dotted names for modules, like P.Q.M, which |
| 5 | could correspond to a file P/Q/M.py found somewhere on sys.path. It |
| 6 | is possible to import a package itself, though this makes little sense |
| 7 | unless the package contains a module called __init__. |
| 8 | |
| 9 | A package has two variables that control the namespace used for |
| 10 | packages and modules, both initialized to sensible defaults the first |
| 11 | time the package is referenced. |
| 12 | |
| 13 | (1) A package's *module search path*, contained in the per-package |
| 14 | variable __path__, defines a list of *directories* where submodules or |
| 15 | subpackages of the package are searched. It is initialized to the |
| 16 | directory containing the package. Setting this variable to None makes |
| 17 | the module search path default to sys.path (this is not quite the same |
| 18 | as setting it to sys.path, since the latter won't track later |
| 19 | assignments to sys.path). |
| 20 | |
| 21 | (2) A package's *import domain*, contained in the per-package variable |
| 22 | __domain__, defines a list of *packages* that are searched (using |
| 23 | their respective module search paths) to satisfy imports. It is |
| 24 | initialized to the list cosisting of the package itself, its parent |
| 25 | package, its parent's parent, and so on, ending with the root package |
| 26 | (the nameless package containing all top-level packages and modules, |
| 27 | whose module search path is None, implying sys.path). |
| 28 | |
| 29 | The default domain implements a search algorithm called "expanding |
| 30 | search". An alternative search algorithm called "explicit search" |
| 31 | fixes the import search path to contain only the root package, |
| 32 | requiring the modules in the package to name all imported modules by |
| 33 | their full name. The convention of using '__' to refer to the current |
| 34 | package (both as a per-module variable and in module names) can be |
| 35 | used by packages using explicit search to refer to modules in the same |
| 36 | package; this combination is known as "explicit-relative search". |
| 37 | |
| 38 | The PackageImporter and PackageLoader classes together implement the |
| 39 | following policies: |
| 40 | |
| 41 | - There is a root package, whose name is ''. It cannot be imported |
| 42 | directly but may be referenced, e.g. by using '__' from a top-level |
| 43 | module. |
| 44 | |
| 45 | - In each module or package, the variable '__' contains a reference to |
| 46 | the parent package; in the root package, '__' points to itself. |
| 47 | |
| 48 | - In the name for imported modules (e.g. M in "import M" or "from M |
| 49 | import ..."), a leading '__' refers to the current package (i.e. |
| 50 | the package containing the current module); leading '__.__' and so |
| 51 | on refer to the current package's parent, and so on. The use of |
| 52 | '__' elsewhere in the module name is not supported. |
| 53 | |
| 54 | - Modules are searched using the "expanding search" algorithm by |
| 55 | virtue of the default value for __domain__. |
| 56 | |
| 57 | - If A.B.C is imported, A is searched using __domain__; then |
| 58 | subpackage B is searched in A using its __path__, and so on. |
| 59 | |
| 60 | - Built-in modules have priority: even if a file sys.py exists in a |
| 61 | package, "import sys" imports the built-in sys module. |
| 62 | |
| 63 | - The same holds for frozen modules, for better or for worse. |
| 64 | |
| 65 | - Submodules and subpackages are not automatically loaded when their |
| 66 | parent packages is loaded. |
| 67 | |
| 68 | - The construct "from package import *" is illegal. (It can still be |
| 69 | used to import names from a module.) |
| 70 | |
| 71 | - When "from package import module1, module2, ..." is used, those |
| 72 | modules are explicitly loaded. |
| 73 | |
| 74 | - When a package is loaded, if it has a submodule __init__, that |
| 75 | module is loaded. This is the place where required submodules can |
| 76 | be loaded, the __path__ variable extended, etc. The __init__ module |
| 77 | is loaded even if the package was loaded only in order to create a |
| 78 | stub for a sub-package: if "import P.Q.R" is the first reference to |
| 79 | P, and P has a submodule __init__, P.__init__ is loaded before P.Q |
| 80 | is even searched. |
| 81 | |
| 82 | Caveats: |
| 83 | |
| 84 | - It is possible to import a package that has no __init__ submodule; |
| 85 | this is not particularly useful but there may be useful applications |
| 86 | for it (e.g. to manipulate its search paths from the outside!). |
| 87 | |
| 88 | - There are no special provisions for os.chdir(). If you plan to use |
| 89 | os.chdir() before you have imported all your modules, it is better |
| 90 | not to have relative pathnames in sys.path. (This could actually be |
| 91 | fixed by changing the implementation of path_join() in the hook to |
| 92 | absolutize paths.) |
| 93 | |
| 94 | - Packages and modules are introduced in sys.modules as soon as their |
| 95 | loading is started. When the loading is terminated by an exception, |
| 96 | the sys.modules entries remain around. |
| 97 | |
| 98 | - There are no special measures to support mutually recursive modules, |
| 99 | but it will work under the same conditions where it works in the |
| 100 | flat module space system. |
| 101 | |
| 102 | - Sometimes dummy entries (whose value is None) are entered in |
| 103 | sys.modules, to indicate that a particular module does not exist -- |
| 104 | this is done to speed up the expanding search algorithm when a |
| 105 | module residing at a higher level is repeatedly imported (Python |
| 106 | promises that importing a previously imported module is cheap!) |
| 107 | |
| 108 | - Although dynamically loaded extensions are allowed inside packages, |
| 109 | the current implementation (hardcoded in the interpreter) of their |
| 110 | initialization may cause problems if an extension invokes the |
| 111 | interpreter during its initialization. |
| 112 | |
| 113 | - reload() may find another version of the module only if it occurs on |
| 114 | the package search path. Thus, it keeps the connection to the |
| 115 | package to which the module belongs, but may find a different file. |
| 116 | |
| 117 | XXX Need to have an explicit name for '', e.g. '__root__'. |
| 118 | |
| 119 | """ |
| 120 | |
| 121 | |
| 122 | import imp |
| 123 | import string |
| 124 | import sys |
| 125 | import __builtin__ |
| 126 | |
| 127 | import ihooks |
| 128 | from ihooks import ModuleLoader, ModuleImporter |
| 129 | |
| 130 | |
| 131 | class PackageLoader(ModuleLoader): |
| 132 | |
| 133 | """A subclass of ModuleLoader with package support. |
| 134 | |
| 135 | find_module_in_dir() will succeed if there's a subdirectory with |
| 136 | the given name; load_module() will create a stub for a package and |
| 137 | load its __init__ module if it exists. |
| 138 | |
| 139 | """ |
| 140 | |
| 141 | def find_module_in_dir(self, name, dir): |
| 142 | if dir is not None: |
| 143 | dirname = self.hooks.path_join(dir, name) |
| 144 | if self.hooks.path_isdir(dirname): |
| 145 | return None, dirname, ('', '', 'PACKAGE') |
| 146 | return ModuleLoader.find_module_in_dir(self, name, dir) |
| 147 | |
| 148 | def load_module(self, name, stuff): |
| 149 | file, filename, info = stuff |
| 150 | suff, mode, type = info |
| 151 | if type == 'PACKAGE': |
| 152 | return self.load_package(name, stuff) |
| 153 | if sys.modules.has_key(name): |
| 154 | m = sys.modules[name] |
| 155 | else: |
| 156 | sys.modules[name] = m = imp.new_module(name) |
| 157 | self.set_parent(m) |
| 158 | if type == imp.C_EXTENSION and '.' in name: |
| 159 | return self.load_dynamic(name, stuff) |
| 160 | else: |
| 161 | return ModuleLoader.load_module(self, name, stuff) |
| 162 | |
| 163 | def load_dynamic(self, name, stuff): |
| 164 | file, filename, (suff, mode, type) = stuff |
| 165 | # Hack around restriction in imp.load_dynamic() |
| 166 | i = string.rfind(name, '.') |
| 167 | tail = name[i+1:] |
| 168 | if sys.modules.has_key(tail): |
| 169 | save = sys.modules[tail] |
| 170 | else: |
| 171 | save = None |
| 172 | sys.modules[tail] = imp.new_module(name) |
| 173 | try: |
| 174 | m = imp.load_dynamic(tail, filename, file) |
| 175 | finally: |
| 176 | if save: |
| 177 | sys.modules[tail] = save |
| 178 | else: |
| 179 | del sys.modules[tail] |
| 180 | sys.modules[name] = m |
| 181 | return m |
| 182 | |
| 183 | def load_package(self, name, stuff): |
| 184 | file, filename, info = stuff |
| 185 | if sys.modules.has_key(name): |
| 186 | package = sys.modules[name] |
| 187 | else: |
| 188 | sys.modules[name] = package = imp.new_module(name) |
| 189 | package.__path__ = [filename] |
| 190 | self.init_package(package) |
| 191 | return package |
| 192 | |
| 193 | def init_package(self, package): |
| 194 | self.set_parent(package) |
| 195 | self.set_domain(package) |
| 196 | self.call_init_module(package) |
| 197 | |
| 198 | def set_parent(self, m): |
| 199 | name = m.__name__ |
| 200 | if '.' in name: |
| 201 | name = name[:string.rfind(name, '.')] |
| 202 | else: |
| 203 | name = '' |
| 204 | m.__ = sys.modules[name] |
| 205 | |
| 206 | def set_domain(self, package): |
| 207 | name = package.__name__ |
| 208 | package.__domain__ = domain = [name] |
| 209 | while '.' in name: |
| 210 | name = name[:string.rfind(name, '.')] |
| 211 | domain.append(name) |
| 212 | if name: |
| 213 | domain.append('') |
| 214 | |
| 215 | def call_init_module(self, package): |
| 216 | stuff = self.find_module('__init__', package.__path__) |
| 217 | if stuff: |
| 218 | m = self.load_module(package.__name__ + '.__init__', stuff) |
| 219 | package.__init__ = m |
| 220 | |
| 221 | |
| 222 | class PackageImporter(ModuleImporter): |
| 223 | |
| 224 | """Importer that understands packages and '__'.""" |
| 225 | |
| 226 | def __init__(self, loader = None, verbose = 0): |
| 227 | ModuleImporter.__init__(self, |
| 228 | loader or PackageLoader(None, verbose), verbose) |
| 229 | |
| 230 | def import_module(self, name, globals={}, locals={}, fromlist=[]): |
| 231 | if globals.has_key('__'): |
| 232 | package = globals['__'] |
| 233 | else: |
| 234 | # No calling context, assume in root package |
| 235 | package = sys.modules[''] |
| 236 | if name[:3] in ('__.', '__'): |
| 237 | p = package |
| 238 | name = name[3:] |
| 239 | while name[:3] in ('__.', '__'): |
| 240 | p = package.__ |
| 241 | name = name[3:] |
| 242 | if not name: |
| 243 | return self.finish(package, p, '', fromlist) |
| 244 | if '.' in name: |
| 245 | i = string.find(name, '.') |
| 246 | name, tail = name[:i], name[i:] |
| 247 | else: |
| 248 | tail = '' |
| 249 | mname = p.__name__ and p.__name__+'.'+name or name |
| 250 | m = self.get1(mname) |
| 251 | return self.finish(package, m, tail, fromlist) |
| 252 | if '.' in name: |
| 253 | i = string.find(name, '.') |
| 254 | name, tail = name[:i], name[i:] |
| 255 | else: |
| 256 | tail = '' |
| 257 | for pname in package.__domain__: |
| 258 | mname = pname and pname+'.'+name or name |
| 259 | m = self.get0(mname) |
| 260 | if m: break |
| 261 | else: |
| 262 | raise ImportError, "No such module %s" % name |
| 263 | return self.finish(m, m, tail, fromlist) |
| 264 | |
| 265 | def finish(self, module, m, tail, fromlist): |
| 266 | # Got ....A; now get ....A.B.C.D |
| 267 | yname = m.__name__ |
| 268 | if tail and sys.modules.has_key(yname + tail): # Fast path |
| 269 | yname, tail = yname + tail, '' |
| 270 | m = self.get1(yname) |
| 271 | while tail: |
| 272 | i = string.find(tail, '.', 1) |
| 273 | if i > 0: |
| 274 | head, tail = tail[:i], tail[i:] |
| 275 | else: |
| 276 | head, tail = tail, '' |
| 277 | yname = yname + head |
| 278 | m = self.get1(yname) |
| 279 | |
| 280 | # Got ....A.B.C.D; now finalize things depending on fromlist |
| 281 | if not fromlist: |
| 282 | return module |
| 283 | if '__' in fromlist: |
| 284 | raise ImportError, "Can't import __ from anywhere" |
| 285 | if not hasattr(m, '__path__'): return m |
| 286 | if '*' in fromlist: |
| 287 | raise ImportError, "Can't import * from a package" |
| 288 | for f in fromlist: |
| 289 | if hasattr(m, f): continue |
| 290 | fname = yname + '.' + f |
| 291 | self.get1(fname) |
| 292 | return m |
| 293 | |
| 294 | def get1(self, name): |
| 295 | m = self.get(name) |
| 296 | if not m: |
| 297 | raise ImportError, "No module named %s" % name |
| 298 | return m |
| 299 | |
| 300 | def get0(self, name): |
| 301 | m = self.get(name) |
| 302 | if not m: |
| 303 | sys.modules[name] = None |
| 304 | return m |
| 305 | |
| 306 | def get(self, name): |
| 307 | # Internal routine to get or load a module when its parent exists |
| 308 | if sys.modules.has_key(name): |
| 309 | return sys.modules[name] |
| 310 | if '.' in name: |
| 311 | i = string.rfind(name, '.') |
| 312 | head, tail = name[:i], name[i+1:] |
| 313 | else: |
| 314 | head, tail = '', name |
| 315 | path = sys.modules[head].__path__ |
| 316 | stuff = self.loader.find_module(tail, path) |
| 317 | if not stuff: |
| 318 | return None |
| 319 | sys.modules[name] = m = self.loader.load_module(name, stuff) |
| 320 | if head: |
| 321 | setattr(sys.modules[head], tail, m) |
| 322 | return m |
| 323 | |
| 324 | def reload(self, module): |
| 325 | name = module.__name__ |
| 326 | if '.' in name: |
| 327 | i = string.rfind(name, '.') |
| 328 | head, tail = name[:i], name[i+1:] |
| 329 | path = sys.modules[head].__path__ |
| 330 | else: |
| 331 | tail = name |
| 332 | path = sys.modules[''].__path__ |
| 333 | stuff = self.loader.find_module(tail, path) |
| 334 | if not stuff: |
| 335 | raise ImportError, "No module named %s" % name |
| 336 | return self.loader.load_module(name, stuff) |
| 337 | |
| 338 | def unload(self, module): |
| 339 | if hasattr(module, '__path__'): |
| 340 | raise ImportError, "don't know how to unload packages yet" |
| 341 | PackageImporter.unload(self, module) |
| 342 | |
| 343 | def install(self): |
| 344 | if not sys.modules.has_key(''): |
| 345 | sys.modules[''] = package = imp.new_module('') |
| 346 | package.__path__ = None |
| 347 | self.loader.init_package(package) |
| 348 | for m in sys.modules.values(): |
| 349 | if not m: continue |
| 350 | if not hasattr(m, '__'): |
| 351 | self.loader.set_parent(m) |
| 352 | ModuleImporter.install(self) |
| 353 | |
| 354 | |
| 355 | def install(v = 0): |
| 356 | ihooks.install(PackageImporter(None, v)) |
| 357 | |
| 358 | def uninstall(): |
| 359 | ihooks.uninstall() |
| 360 | |
| 361 | def ni(v = 0): |
| 362 | install(v) |
| 363 | |
| 364 | def no(): |
| 365 | uninstall() |
| 366 | |
| 367 | def test(): |
| 368 | import pdb |
| 369 | try: |
| 370 | testproper() |
| 371 | except: |
| 372 | sys.last_type, sys.last_value, sys.last_traceback = ( |
| 373 | sys.exc_type, sys.exc_value, sys.exc_traceback) |
| 374 | print |
| 375 | print sys.last_type, ':', sys.last_value |
| 376 | print |
| 377 | pdb.pm() |
| 378 | |
| 379 | def testproper(): |
| 380 | install(1) |
| 381 | try: |
| 382 | import mactest |
| 383 | print dir(mactest) |
| 384 | raw_input('OK?') |
| 385 | finally: |
| 386 | uninstall() |
| 387 | |
| 388 | |
| 389 | if __name__ == '__main__': |
| 390 | test() |