Analysis

Analysis(scripts, pathex=None, hookspath=None, excludes=None)

scripts

a list of scripts specified as file names.

pathex

an optional list of paths to be searched before sys.path.

hookspath

an optional list of paths used to extend the hooks package.

excludes

an optional list of module or package names (their Python names, not path names) that will be ignored (as though they were not found).

An Analysis has three outputs, all TOCs accessed as attributes of the Analysis.

scripts

The scripts you gave Analysis as input, with any runtime hook scripts prepended.

pure

The pure Python modules.

binaries

The extension modules and their dependencies. The secondary dependencies are filtered. On Windows, a long list of MS dlls are excluded. On Linux/Unix, any shared lib in /lib or /usr/lib is excluded.

Back to Top

PYZ

PYZ(toc, name=None, level=9)

toc

a TOC, normally an Analysis.pure.

name

A filename for the .pyz. Normally not needed, as the generated name will do fine.

level

The Zlib compression level to use. If 0, the zlib module is not required.

Back to Top

PKG

Generally, you will not need to create your own PKGs, as the EXE will do it for you. This is one way to include read-only data in a single-file deployment, however. A single-file deployment including TK support will use this technique.

PKG(toc, name=None, cdict=None, exclude_binaries=0)

toc

a TOC.

name

a filename for the PKG (optional).

cdict

a dictionary that specifies compression by typecode. For example, PYZ is left uncompressed so that it can be accessed inside the PKG. The default uses sensible values. If zlib is not available, no compression is used.

exclude_binaries

If 1, EXTENSIONs and BINARYs will be left out of the PKG, and forwarded to its container (usually a COLLECT).

Back to Top

EXE

EXE(*args, **kws)

args

One or more arguments which are either TOCs or Targets.

kws

Possible keyword arguments:

console

Always 1 on Linux/unix. On Windows, governs whether to use the console executable, or the Windows subsystem executable.

debug

Setting to 1 gives you progress messages from the executable (for a console=0, these will be annoying MessageBoxes).

name

The filename for the executable.

exclude_binaries

Forwarded to the PKG the EXE builds.

icon

Windows NT family only. icon='myicon.ico' to use an icon file, or icon='notepad.exe,0' to grab an icon resource.

version

Windows NT family only. version='myversion.txt'. Use GrabVersion.py to steal a version resource from an executable, and then edit the ouput to create your own. (The syntax of version resources is so arcane that I wouldn't attempt to write one from scratch.)

There are actually two EXE classes - one for ELF platforms (where the bootloader, that is the run executable, and the PKG are concatenated), and one for non-ELF platforms (where the run executable is simply renamed, and expects a exename.pkg in the same directory). Which class becomes available as EXE is determined by a flag in config.dat. This flag is set to non-ELF when using Make.py -n.

Back to Top

DLL

On Windows, this provides support for doing in-process COM servers. It is not generalized. However, embedders can follow the same model to build a special purpose DLL so the Python support in their app is hidden. You will need to write your own dll, but thanks to Allan Green for refactoring the C code and making that a managable task.

Back to Top

COLLECT

COLLECT(*args, **kws)

args

One or more arguments which are either TOCs or Targets.

kws

Possible keyword arguments:

name

The name of the directory to be built.

Back to Top

Tree

Tree(root, prefix=None, excludes=None)

root

The root of the tree (on the build system).

prefix

Optional prefix to the names on the target system.

excludes

A list of names to exclude. Two forms are allowed:

name

files with this basename will be excluded (do not include the path).

*.ext

any file with the given extension will be excluded.

Since a Tree is a TOC, you can also use the exclude technique described above in the section on TOCs.

Back to Top

Buildtime Warnings

When an Analysis step runs, it produces a warnings file (named warnproject.txt) in the spec file's directory. Generally, most of these warnings are harmless. For example, os.py (which is cross-platform) works by figuring out what platform it is on, then importing (and rebinding names from) the appropriate platform-specific module. So analyzing os.py will produce a set of warnings like:

W: no module named dos (conditional import by os)
W: no module named ce (conditional import by os)
W: no module named os2 (conditional import by os)

Note that the analysis has detected that the import is within a conditional block (an if statement). The analysis also detects if an import within a function or class, (delayed) or at the top level. A top-level, non-conditional import failure is really a hard error. There's at least a reasonable chance that conditional and / or delayed import will be handled gracefully at runtime.

Ignorable warnings may also be produced when a class or function is declared in a package (an __init__.py module), and the import specifies package.name. In this case, the analysis can't tell if name is supposed to refer to a submodule of package.

Warnings are also produced when an __import__, exec or eval statement is encountered. The __import__ warnings should almost certainly be investigated. Both exec and eval can be used to implement import hacks, but usually their use is more benign.

Any problem detected here can be handled by hooking the analysis of the module. See Listing Hidden Imports below for how to do it.

Back to Top

Getting Debug Messages

Setting debug=1 on an EXE will cause the executable to put out progress messages (for console apps, these go to stdout; for Windows apps, these show as MessageBoxes). This can be useful if you are doing complex packaging, or your app doesn't seem to be starting, or just to learn how the runtime works.

Back to Top

Getting Python's Verbose Imports

You can also pass a -v (verbose imports) flag to the embedded Python. This can be extremely useful. I usually try it even on apparently working apps, just to make sure that I'm always getting my copies of the modules and no import has leaked out to the installed Python.

You set this (like the other runtime options) by feeding a phone TOC entry to the EXE. The easiest way to do this is to change the EXE from:

EXE(..., anal.scripts, ....)

to:

EXE(..., anal.scripts + [('v', '', 'OPTION')], ...)

These messages will always go to stdout, so you won't see them on Windows if console=0.

Back to Top

Extending the Path

When the analysis phase cannot find needed modules, it may be that the code is manipulating sys.path. The easiest thing to do in this case is tell Analysis about the new directory through the second arg to the constructor:

anal = Analysis(['somedir/myscript.py'],
                ['path/to/thisdir', 'path/to/thatdir'])

In this case, the Analysis will have a search path:

['somedir', 'path/to/thisdir', 'path/to/thatdir'] + sys.path

You can do the same when running Makespec.py:

Makespec.py --paths=path/to/thisdir;path/to/thatdir ...

(on *nix, use : as the path separator).

Back to Top

Listing Hidden Imports

Hidden imports are fairly common. These can occur when the code is using __import__ (or, perhaps exec or eval), in which case you will see a warning in the warnproject.txt file. They can also occur when an extension module uses the Python/C API to do an import, in which case Analysis can't detect anything. You can verify that hidden import is the problem by using Python's verbose imports flag. If the import messages say "module not found", but the warnproject.txt file has no "no module named..." message for the same module, then the problem is a hidden import.

Hidden imports are handled by hooking the module (the one doing the hidden imports) at Analysis time. Do this by creating a file named hook-module.py (where module is the fully-qualified Python name, eg, hook-xml.dom.py), and placing it in the hooks package under PyInstaller's root directory, (alternatively, you can save it elsewhere, and then use the hookspath arg to Analysis so your private hooks directory will be searched). Normally, it will have only one line:

hiddenimports = ['module1', 'module2']

When the Analysis finds this file, it will proceed exactly as though the module explicitly imported module1 and module2. (Full details on the analysis-time hook mechanism is in the Hooks section).

If you successfully hook a publicly distributed module in this way, please send us the hook so we can make it available to others.

Back to Top

Extending a Package's __path__

Python allows a package to extend the search path used to find modules and sub-packages through the __path__ mechanism. Normally, a package's __path__ has only one entry - the directory in which the __init__.py was found. But __init__.py is free to extend its __path__ to include other directories. For example, the win32com.shell.shell module actually resolves to win32com/win32comext/shell/shell.pyd. This is because win32com/__init__.py appends ../win32comext to its __path__.

Because the __init__.py is not actually run during an analysis, we use the same hook mechanism we use for hidden imports. A static list of names won't do, however, because the new entry on __path__ may well require computation. So hook-module.py should define a method hook(mod). The mod argument is an instance of mf.Module which has (more or less) the same attributes as a real module object. The hook function should return a mf.Module instance - perhaps a brand new one, but more likely the same one used as an arg, but mutated. See mf.py: A Modulefinder Replacement for details, and hooks/hook-win32com.py for an example.

Note that manipulations of __path__ hooked in this way apply to the analysis, and only the analysis. That is, at runtime win32com.shell is resolved the same way as win32com.anythingelse, and win32com.__path__ knows nothing of ../win32comext.

Once in awhile, that's not enough.

Back to Top

Changing Runtime Behavior

More bizarre situations can be accomodated with runtime hooks. These are small scripts that manipulate the environment before your main script runs, effectively providing additional top-level code to your script.

At the tail end of an analysis, the module list is examined for matches in rthooks.dat, which is the string representation of a Python dictionary. The key is the module name, and the value is a list of hook-script pathnames.

So putting an entry:

'somemodule': ['path/to/somescript.py'],

into rthooks.dat is almost the same thing as doing this:

anal = Analysis(['path/to/somescript.py', 'main.py'], ...

except that in using the hook, path/to/somescript.py will not be analyzed, (that's not a feature - we just haven't found a sane way fit the recursion into my persistence scheme).

Hooks done in this way, while they need to be careful of what they import, are free to do almost anything. One provided hook sets things up so that win32com can generate modules at runtime (to disk), and the generated modules can be found in the win32com package.

Back to Top

Adapting to being "frozen"

In most sophisticated apps, it becomes necessary to figure out (at runtime) whether you're running "live" or "frozen". For example, you might have a configuration file that (running "live") you locate based on a module's __file__ attribute. That won't work once the code is packaged up. You'll probably want to look for it based on sys.executable instead.

The bootloaders set sys.frozen=1 (and, for in-process COM servers, the embedding DLL sets sys.frozen='dll').

For really advanced users, you can access the iu.ImportManager as sys.importManager. See iu.py for how you might make use of this fact.

Back to Top

Accessing Data Files

In a --onedir distribution, this is easy: pass a list of your data files (in TOC format) to the COLLECT, and they will show up in the distribution directory tree. The name in the (name, path, 'DATA') tuple can be a relative path name. Then, at runtime, you can use code like this to find the file:

os.path.join(os.path.dirname(sys.executable), relativename))

In a --onefile, it's a bit trickier. You can cheat, and add the files to the EXE as BINARY. They will then be extracted at runtime into the work directory by the C code (which does not create directories, so the name must be a plain name), and cleaned up on exit. The work directory is best found by os.environ['_MEIPASS2']. Be awawre, though, that if you use --strip or --upx, strange things may happen to your data - BINARY is really for shared libs / dlls.

If you add them as 'DATA' to the EXE, then it's up to you to extract them. Use code like this:

import sys, carchive
this = carchive.CArchive(sys.executable)
data = this.extract('mystuff')[1]

to get the contents as a binary string. See support/unpackTK.py for an advanced example (the TCL and TK lib files are in a PKG which is opened in place, and then extracted to the filesystem).

Back to Top

One Pass Execution

In a single directory deployment (--onedir, which is the default), all of the binaries are already in the file system. In that case, the embedding app:

• opens the archive
• starts Python (on Windows, this is done with dynamic loading so one embedding app binary can be used with any Python version)
• imports all the modules which are at the top level of the archive (basically, bootstraps the import hooks)
• mounts the ZlibArchive(s) in the outer archive
• runs all the scripts which are at the top level of the archive
• finalizes Python

Back to Top

Two Pass Execution

There are a couple situations which require two passes:

• a --onefile deployment (on Windows, the files can't be cleaned up afterwards because Python does not call FreeLibrary; on other platforms, Python won't find them if they're extracted in the same process that uses them)
• LD_LIBRARY_PATH needs to be set to find the binaries (not extension modules, but modules the extensions are linked to).

The first pass:

• opens the archive
• extracts all the binaries in the archive (in PyInstaller v1.0, this is always to a temporary directory).
• sets a magic environment variable
• sets LD_LIBRARY_PATH (non-Windows)
• executes itself as a child process (letting the child use his stdin, stdout and stderr)
• waits for the child to exit (on *nix, the child actually replaces the parent)
• cleans up the extracted binaries (so on *nix, this is done by the child)

The child process executes as in One Pass Execution above (the magic environment variable is what tells it that this is pass two).

figure 3 - Self Extracting Executable

There are, of course, quite a few differences between the Windows and Unix/Linux versions. The major one is that because all of Python on Windows is in pythonXX.dll, and dynamic loading is so simple-minded, that one binary can be use with any version of Python. There's much in common, though, and that C code can be found in source/common/launch.c.

The Unix/Linux build process (which you need to run just once for any version of Python) makes use of the config information in your install (if you installed from RPM, you need the Python-development RPM). It also overrides getpath.c since we don't want it hunting around the filesystem to build sys.path.

In both cases, while one PyInstaller download can be used with any Python version, you need to have separate installations for each Python version.

Back to Top

ImportTracker

ImportTracker can be called in two ways: analyze_one(name, importername=None) or analyze_r(name, importername=None). The second method does what modulefinder does - it recursively finds all the module names that importing name would cause to appear in sys.modules. The first method is non-recursive. This is useful, because it is the only way of answering the question "Who imports name?" But since it is somewhat unrealistic (very few real imports do not involve recursion), it deserves some explanation.

Back to Top

analyze_one()

When a name is imported, there are structural and dynamic effects. The dynamic effects are due to the execution of the top-level code in the module (or modules) that get imported. The structural effects have to do with whether the import is relative or absolute, and whether the name is a dotted name (if there are N dots in the name, then N+1 modules will be imported even without any code running).

The analyze_one method determines the structural effects, and defers the dynamic effects. For example, analyze_one("B.C", "A") could return ["B", "B.C"] or ["A.B", "A.B.C"] depending on whether the import turns out to be relative or absolute. In addition, ImportTracker's modules dict will have Module instances for them.

Back to Top

Module Classes

There are Module subclasses for builtins, extensions, packages and (normal) modules. Besides the normal module object attributes, they have an attribute imports. For packages and normal modules, imports is a list populated by scanning the code object (and therefor, the names in this list may be relative or absolute names - we don't know until they have been analyzed).

The highly astute will notice that there is a hole in analyze_one() here. The first thing that happens when B.C is being imported is that B is imported and it's top-level code executed. That top-level code can do various things so that when the import of B.C finally occurs, something completely different happens (from what a structural analysis would predict). But mf can handle this through it's hooks mechanism.

Back to Top

code scanning

Like modulefinder, mf scans the byte code of a module, looking for imports. In addition, mf will pick out a module's __all__ attribute, if it is built as a list of constant names. This means that if a package declares an __all__ list as a list of names, ImportTracker will track those names if asked to analyze package.*. The code scan also notes the occurance of __import__, exec and eval, and can issue warnings when they're found.

The code scanning also keeps track (as well as it can) of the context of an import. It recognizes when imports are found at the top-level, and when they are found inside definitions (deferred imports). Within that, it also tracks whether the import is inside a condition (conditional imports).

Back to Top

Hooks

In modulefinder, scanning the code takes the place of executing the code object. mf goes further and allows a module to be hooked (after it has been scanned, but before analyze_one is done with it). A hook is a module named hook-fullyqualifiedname in the hooks package. These modules should have one or more of the following three global names defined:

hiddenimports

a list of modules names (relative or absolute) that the module imports in some untrackable way.

attrs

a list of (name, value) pairs (where value is normally meaningless).

hook(mod)

a function taking a Module instance and returning a Module instance (so it can modify or replace).

The first hook (hiddenimports) extends the list created by scanning the code. ExtensionModules, of course, don't get scanned, so this is the only way of recording any imports they do.

The second hook (attrs) exists mainly so that ImportTracker won't issue spurious warnings when the rightmost node in a dotted name turns out to be an attribute in a package module, instead of a missing submodule.

The callable hook exists for things like dynamic modification of a package's __path__ or perverse situations, like xml.__init__ replacing itself in sys.modules with _xmlplus.__init__. (It takes nine hook modules to properly trace through PyXML-using code, and I can't believe that it's any easier for the poor programmer using that package). The hook(mod) (if it exists) is called before looking at the others - that way it can, for example, test sys.version and adjust what's in hiddenimports.

Back to Top

Warnings

ImportTracker has a getwarnings() method that returns all the warnings accumulated by the instance, and by the Module instances in its modules dict. Generally, it is ImportTracker who will accumulate the warnings generated during the structural phase, and Modules that will get the warnings generated during the code scan.

Note that by using a hook module, you can silence some particularly tiresome warnings, but not all of them.

Back to Top

Cross Reference

Once a full analysis (that is, an analyze_r call) has been done, you can get a cross reference by using getxref(). This returns a list of tuples. Each tuple is (modulename, importers), where importers is a list of the (fully qualified) names of the modules importing modulename. Both the returned list and the importers list are sorted.

Back to Top

Usage

A simple example follows:

>>> import mf
>>> a = mf.ImportTracker()
>>> a.analyze_r("os")
['os', 'sys', 'posixpath', 'nt', 'stat', 'string', 'strop',
're', 'pcre', 'ntpath', 'dospath', 'macpath', 'win32api',
'UserDict', 'copy', 'types', 'repr', 'tempfile']
>>> a.analyze_one("os")
['os']
>>> a.modules['string'].imports
[('strop', 0, 0), ('strop.*', 0, 0), ('re', 1, 1)]
>>>

The tuples in the imports list are (name, delayed, conditional).

>>> for w in a.modules['string'].warnings: print w
...
W: delayed  eval hack detected at line 359
W: delayed  eval hack detected at line 389
W: delayed  eval hack detected at line 418
>>> for w in a.getwarnings(): print w
...
W: no module named pwd (delayed, conditional import by posixpath)
W: no module named dos (conditional import by os)
W: no module named os2 (conditional import by os)
W: no module named posix (conditional import by os)
W: no module named mac (conditional import by os)
W: no module named MACFS (delayed, conditional import by tempfile)
W: no module named macfs (delayed, conditional import by tempfile)
W: top-level conditional exec statment detected at line 47
   - os (C:\Program Files\Python\Lib\os.py)
W: delayed  eval hack detected at line 359
   - string (C:\Program Files\Python\Lib\string.py)
W: delayed  eval hack detected at line 389
   - string (C:\Program Files\Python\Lib\string.py)
W: delayed  eval hack detected at line 418
   - string (C:\Program Files\Python\Lib\string.py)
>>>

Back to Top

ImportManager

ImportManager formalizes the concept of a metapath. This concept implicitly exists in native Python in that builtins and frozen modules are searched before sys.path, (on Windows there's also a search of the registry while on Mac, resources may be searched). This metapath is a list populated with ImportDirector instances. There are ImportDirector subclasses for builtins, frozen modules, (on Windows) modules found through the registry and a PathImportDirector for handling sys.path. For a top-level import (that is, not an import of a module in a package), ImportManager tries each director on it's metapath until one succeeds.

ImportManager hides the semantic complexity of an import from the directors. It's up to the ImportManager to decide if an import is relative or absolute; to see if the module has already been imported; to keep sys.modules up to date; to handle the fromlist and return the correct module object.

Back to Top

ImportDirector

An ImportDirector just needs to respond to getmod(name) by returning a module object or None. As you will see, an ImportDirector can consider name to be atomic - it has no need to examine name to see if it is dotted.

To see how this works, we need to examine the PathImportDirector.

Back to Top

PathImportDirector

The PathImportDirector subclass manages a list of names - most notably, sys.path. To do so, it maintains a shadowpath - a dictionary mapping the names on its pathlist (eg, sys.path) to their associated Owners. (It could do this directly, but the assumption that sys.path is occupied solely by strings seems ineradicable.) Owners of the appropriate kind are created as needed (if all your imports are satisfied by the first two elements of sys.path, the PathImportDirector's shadowpath will only have two entries).

Back to Top

Owner

An Owner is much like an ImportDirector but manages a much more concrete piece of turf. For example, a DirOwner manages one directory. Since there are no other officially recognized filesystem-like namespaces for importing, that's all that's included in iu, but it's easy to imagine Owners for zip files (and I have one for my own .pyz archive format) or even URLs.

As with ImportDirectors, an Owner just needs to respond to getmod(name) by returning a module object or None, and it can consider name to be atomic.

So structurally, we have a tree, rooted at the ImportManager. At the next level, we have a set of ImportDirectors. At least one of those directors, the PathImportDirector in charge of sys.path, has another level beneath it, consisting of Owners. This much of the tree covers the entire top-level import namespace.

The rest of the import namespace is covered by treelets, each rooted in a package module (an __init__.py).

Back to Top

Packages

To make this work, Owners need to recognize when a module is a package. For a DirOwner, this means that name is a subdirectory which contains an __init__.py. The __init__ module is loaded and its __path__ is initialized with the subdirectory. Then, a PathImportDirector is created to manage this __path__. Finally the new PathImportDirector's getmod is assigned to the package's __importsub__ function.

When a module within the package is imported, the request is routed (by the ImportManager) diretly to the package's __importsub__. In a hierarchical namespace (like a filesystem), this means that __importsub__ (which is really the bound getmod method of a PathImportDirector instance) needs only the module name, not the package name or the fully qualified name. And that's exactly what it gets. (In a flat namespace - like most archives - it is perfectly easy to route the request back up the package tree to the archive Owner, qualifying the name at each step.)

Back to Top

Possibilities

Let's say we want to import from zip files. So, we subclass Owner. The __init__ method should take a filename, and raise a ValueError if the file is not an acceptable .zip file, (when a new name is encountered on sys.path or a package's __path__, registered Owners are tried until one accepts the name). The getmod method would check the zip file's contents and return None if the name is not found. Otherwise, it would extract the marshalled code object from the zip, create a new module object and perform a bit of initialization (12 lines of code all told for my own archive format, including initializing a pack age with it's __subimporter__).

Once the new Owner class is registered with iu, you can put a zip file on sys.path. A package could even put a zip file on its __path__.

Back to Top

Compatibility

This code has been tested with the PyXML, mxBase and Win32 packages, covering over a dozen import hacks from manipulations of __path__ to replacing a module in sys.modules with a different one. Emulation of Python's native import is nearly exact, including the names recorded in sys.modules and module attributes (packages imported through iu have an extra attribute - __importsub__).

Back to Top

Performance

In most cases, iu is slower than builtin import (by 15 to 20%) but faster than imputil (by 15 to 20%). By inserting archives at the front of sys.path containing the standard lib and the package being tested, this can be reduced to 5 to 10% slower (or, on my 1.52 box, 10% faster!) than builtin import. A bit more can be shaved off by manipulating the ImportManager's metapath.

Back to Top

Limitations

This module makes no attempt to facilitate policy import hacks. It is easy to implement certain kinds of policies within a particular domain, but fundamentally iu works by dividing up the import namespace into independent domains.

Quite simply, I think cross-domain import hacks are a very bad idea. As author of the original package on which PyInstaller is based, McMillan worked with import hacks for many years. Many of them are highly fragile; they often rely on undocumented (maybe even accidental) features of implementation. A cross-domain import hack is not likely to work with PyXML, for example.

That rant aside, you can modify ImportManger to implement different policies. For example, a version that implements three import primitives: absolute import, relative import and recursive-relative import. No idea what the Python syntax for those should be, but __aimport__, __rimport__ and __rrimport__ were easy to implement.

Usage

Here's a simple example of using iu as a builtin import replacement.

>>> import iu
>>> iu.ImportManager().install()
>>>
>>> import DateTime
>>> DateTime.__importsub__
<method PathImportDirector.getmod
  of PathImportDirector instance at 825900>
>>>

Back to Top