Other Ways to Start Programs






Other Ways to Start Programs

Suppose, just for a moment, that you've been asked to write a big Python book and you want to provide a way for readers to easily start the book's examples on just about any platform that Python runs on. Books are nice, but it's awfully fun to be able to click on demos right away. That is, you want to write a general and portable launcher program in Python for starting other Python programs. What to do?

In this chapter, we've seen how to portably spawn threads, but these are simply parallel functions, not external programs. We've also learned how to go about starting new, independently running programs, with both the fork/exec combination and with tools for launching shell commands such as os.popen and os.system. Along the way, though, I've also been careful to point out numerous times that the os.fork call doesn't work on Windows today. This constraint may be improved by the time you read this book, but it still is a limitation as I write these words. Moreover, for reasons we'll explore later, the os.popen call is prone to blocking (pausing) its caller in some scenarios and requires a potentially platform-specific command-line string.

Luckily, there are other ways to start programs in the Python standard library, some of which are more platform neutral than others:

  • The os.spawnv and os.spawnve calls were originally introduced to launch programs on Windows, much like a fork/exec call combination on Unix-like platforms. Today, these calls work on both Windows and Unix-like systems, and additional variants have been added to parrot os.exec.

  • The os.system call can be used on Windows to launch a DOS start command, which opens (i.e., runs) a file independently based on its Windows filename associations, as though it were clicked. os.startfile makes this even simpler in recent Python releases.

  • Tools in the Python PyWin32 extensions package provide other, less standardized ways to start programs (e.g., the WinExec call).

  • Other tools such as the commands and subprocess modules provide additional options in this domain.

We won't talk about the PyWin32 extensions package in this chapter, but the other tools available in the standard library merit a quick look here.

The os.spawn Calls

The os.spawn family of calls execute a program named by a command line in a new process, on both Windows and Unix-like systems. In basic operation, they are similar to the fork/exec call combination on Unix and can be used as alternatives to the system and popen calls we've already learned. In the following interaction, for instance, we start a Python program with a command line in two traditional ways (the second also reads its output):

>>> print open('makewords.py').read( )
print 'spam'
print 'eggs'
print 'ham'

>>> os.system('python makewords.py')
spam
eggs
ham
0

>>> result = os.popen('python makewords.py').read( )
>>> print result
spam
eggs
ham

The equivalent os.spawn calls achieve the same effect, with a slightly more complex call signature that provides more control over the way the program is launched:

>>> os.spawnv(os.P_WAIT, r'C:\Python24\python', ('python', 'makewords.py'))
spam
eggs
ham
0
>>> os.spawnl(os.P_NOWAIT, r'C:\Python24\python', 'python', 'makewords.py')
1820
>>> spam
eggs
ham

The spawn calls are also much like forking programs in Unix. They don't actually copy the calling process (so shared descriptor operations won't work), but they can be used to start a program running completely independent of the calling program, even on Windows. The script in Figure makes the similarity to Unix programming patterns more obvious. It launches a program with a fork/exec combination in Linux, or an os.spawnv call on Windows.

PP3E\System\Processes\spawnv.py

############################################################
# start up 10 copies of child.py running in parallel;
# use spawnv to launch a program on Windows (like fork+exec)
# P_OVERLAY replaces, P_DETACH makes child stdout go nowhere
############################################################

import os, sys

for i in range(10):
    if sys.platform[:3] == 'win':
        pypath = sys.executable
        os.spawnv(os.P_NOWAIT, pypath, ('python', 'child.py', str(i)))
    else:
        pid = os.fork( )
        if pid != 0:
            print 'Process %d spawned' % pid
        else:
            os.execlp('python', 'python', 'child.py', str(i))
print 'Main process exiting.'

To make sense of these examples, you have to understand the arguments being passed to the spawn calls. In this script, we call os.spawnv with a process mode flag, the full directory path to the Python interpreter, and a tuple of strings representing the DOS command line with which to start a new program. The path to the Python interpreter executable program running a script is available as sys.executable in recent Python releases. In general, the process mode flag is taken from these predefined values:


os.P_NOWAIT and os.P_NOWAITO

The spawn functions will return as soon as the new process has been created, with the process ID as the return value. Available on Unix and Windows.


os.P_WAIT

The spawn functions will not return until the new process has run to completion and will return the exit code of the process if the run is successful or "-signal" if a signal kills the process. Available on Unix and Windows.


os.P_DETACH and os.P_OVERLAY

P_DETACH is similar to P_NOWAIT, but the new process is detached from the console of the calling process. If P_OVERLAY is used, the current program will be replaced (much like os.exec). Available on Windows.

In fact, there are eight different calls in the spawn family, which all start a program but vary slightly in their call signatures. In their names, an "l" means you list arguments individually, "p" means the executable file is looked up on the system path, and "e" means a dictionary is passed in to provide the shelled environment of the spawned program: the os.spawnve call, for example, works the same way as os.spawnv but accepts an extra fourth dictionary argument to specify a different shell environment for the spawned program (which, by default, inherits all of the parent's settings):

os.spawnl(mode, path, ...)
os.spawnle(mode, path, ..., env)
os.spawnlp(mode, file, ...)                 # Unix only
os.spawnlpe(mode, file, ..., env)           # Unix only
os.spawnv(mode, path, args)
os.spawnve(mode, path, args, env)
os.spawnvp(mode, file, args)                # Unix only
os.spawnvpe(mode, file, args, env)          # Unix only

Because these calls mimic the names and call signatures of the os.exec variants, see the section "The fork/exec Combination," earlier in this chapter, for more details on the differences between these call forms. Unlike the os.exec calls, only half of the os.spawn formsthose without system path checking (and hence without a "p" in their names)are currently implemented on Windows. All the process mode flags are supported on Windows, but detach and overlay modes are not available on Unix. To see which are present, read the library manual or ask Python:

>>> import sys, os
>>> sys.platform
'win32'

>>> [x for x in dir(os) if x.startswith('spawn')]
['spawnl', 'spawnle', 'spawnv', 'spawnve']

>>> [x for x in dir(os) if x.startswith('P_')]
['P_DETACH', 'P_NOWAIT', 'P_NOWAITO', 'P_OVERLAY', 'P_WAIT']

Run a few tests or see the Python library manual for more details; things such as standard stream connection policies vary between the P_DETACH and P_NOWAIT modes in subtle ways. Here is the script in Figure at work on Windows, spawning 10 independent copies of the child.py Python program we met earlier in this chapter:

C:\...\PP3E\System\Processes>type child.py
import os, sys
print 'Hello from child', os.getpid( ), sys.argv[1]

C:\...\PP3E\System\Processes>python spawnv.py
Hello from child -583587 0
Hello from child -558199 2
Hello from child -586755 1
Hello from child -562171 3
Main process exiting.
Hello from child -581867 6
Hello from child -588651 5
Hello from child -568247 4
Hello from child -563527 7
Hello from child -543163 9
Hello from child -587083 8

Notice that the copies print their output in random order, and the parent program exits before all children do; all of these programs are really running in parallel on Windows. Also observe that the child program's output shows up in the console box where spawnv.py was run; when using P_NOWAIT, standard output comes to the parent's console, but it seems to go nowhere when using P_DETACH (which is most likely a feature when spawning GUI programs).

Launching Programs on Windows

The os.system, os.popen, and os.spawn calls can be used to start command lines on Windows just as on Unix-like platforms (but with the handful of caveats mentioned earlier). On Windows, though, the DOS start command combined with os.system provides an easy way for scripts to launch any file on the system, using Windows filename associations. Starting a program file this way makes it run as independently as its starter. Figure demonstrates these launch techniques.

PP3E\System\Processes\dosstart.py

####################################################################
# start up 5 copies of child.py running in parallel;
# - on Windows, os.system always blocks its caller,
# - using DOS start command pops up a DOS box (which goes
#   away immediately when the child.py program exits)
# - running child-wait.py with DOS start, 5 independent
#   DOS console windows pop up and stay up (1 per program)
# DOS start command uses registry filename associations to know
# to run Python on the file, as though double-clicked in a Windows
# file explorer GUI (any filename can be started this way);
####################################################################

import os, sys

for i in range(5):
    #print os.popen('python child.py ' + str(i)).read( )[:-1]
    #os.system('python child.py ' + str(i))
    #os.system('start child.py ' + str(i))
     os.system('start child-wait.py ' + str(i))
print 'Main process exiting.'

Uncomment one of the lines in this script's for loop to experiment with these schemes on your computer. On mine, when run with either of the first two calls in the loop uncommented, I get the following sort of outputthe text printed by five spawned Python programs:

C:\...\PP3E\System\Processes>python dosstart.py
Hello from child -582331 0
Hello from child -547703 1
Hello from child -547703 2
Hello from child -547651 3
Hello from child -547651 4
Main process exiting.

The os.system call usually blocks its caller until the spawned program exits; reading the output of an os.popen call has the same blocking effect (the reader waits for the spawned program's output to be complete). But with either of the last two statements in the loop uncommented, I get output that simply looks like this:

C:\...\PP3E\System\Processes>python dosstart.py
Main process exiting.

In both cases, I also see five new and completely independent DOS console windows appear on my display; when the third line in the loop is uncommented, all of the DOS boxes go away right after they appear; when the last line in the loop is active, they remain on the screen after the dosstart program exits because the child-wait script pauses for input before exiting.

Using the DOS start command

To understand why, first you need to know how the DOS start command works in general. Roughly, a DOS command line of the form start command works as if command were typed in the Windows Run dialog box available in the Start button menu. If command is a filename, it is opened exactly as if its name had been double-clicked in the Windows Explorer file selector GUI.

For instance, the following three DOS commands automatically start Internet Explorer on a file named index.html, my registered image viewer program on a file named uk-1.jpg, and my sound media player program on a file named sousa.au. Windows simply opens the file with whatever program is associated to handle filenames of that form. Moreover, all three of these programs run independently of the DOS console box where the command is typed:

C:\temp>start c:\stuff\website\public_html\index.html
C:\temp>start c:\stuff\website\public_html\uk-1.jpg
C:\...\PP3E\System\Processes>start ..\..\Internet\Ftp\sousa.au

Now, because the start command can run any file and command line, there is no reason it cannot also be used to start an independently running Python program:

C:\...\PP3E\System\Processes>start child.py 1

Because Python is registered to open names ending in .py when it is installed, this really does work. The script child.py is launched independently of the DOS console window even though we didn't provide the name or path of the Python interpreter program. Because child.py simply prints a message and exits, though, the result isn't exactly satisfying: a new DOS window pops up to serve as the script's standard output, and it immediately goes away when the child exits (it's that Windows "flash feature" described earlier!). To do better, add a raw_input call at the bottom of the program file to wait for a key press before exiting:

C:\...\PP3E\System\Processes>type child-wait.py
import os, sys
print 'Hello from child', os.getpid( ), sys.argv[1]
raw_input("Press <Enter>")   # don't flash on Windows

C:\...\PP3E\System\Processes>start child-wait.py 2

Now the child's DOS window pops up and stays up after the start command has returned. Pressing the Enter key in the pop-up DOS window makes it go away.

Using start in Python scripts

Since we know that Python's os.system and os.popen can be called by a script to run any command line that can be typed at a DOS shell prompt, we can also start independently running programs from a Python script by simply running a DOS start command line. For instance:

C:\...\PP3E>python
>>> import os
>>>
>>> cmd = r'start c:\stuff\website\public_html\index.html'  # start IE browser
>>> os.system(cmd)                                          # runs independent
0
>>> file = r'gui\gifs\pythonPowered.gif'                # start image viewer
>>> os.system('start ' + file)                          # IE opens .gif for me
0
>>> os.system('start ' + 'Gui/gifs/PythonPowered.gif')  # fwd slashes work too
0
>>> os.system(r'start Internet\Ftp\sousa.au')           # start media player
0

The four Python os.system calls here start whatever web-page browser, image viewer, and sound player are registered on your machine to open .html, .gif, and .au files (unless these programs are already running). The launched programs run completely independent of the Python sessionwhen running a DOS start command, os.system does not wait for the spawned program to exit. For instance, Figure shows the .gif file handler in action on my machine, generated by both the second and the third os.system calls in the preceding code.

Started image viewer (Internet Explorer)


Now, since we also know that a Python program can be started from a command line, this yields two ways to launch Python programs:

C:\...\PP3E>python
>>> os.system(r'python Gui\TextEditor\textEditor.pyw')   # start and wait
0
>>> os.system(r'start  Gui\TextEditor\textEditor.pyw')   # start, go on
0

When running a python command, the os.system call waits (blocks) for the command to finish. When running a start command, it does not; the launched Python program (here, PyEdit, a text editor GUI we'll meet in Chapter 12) runs independent of the os.system caller. And finally, that's why the following call in dosstart.py generates a new, independent instance of child-wait.py:

C:\...\PP3E\System\Processes>python
>>> os.system('start child-wait.py 1')
0

When run, this call pops up a new, independent DOS console window to serve as the standard input and output streams of the child-wait program. It truly is independentin fact, it keeps running if we exit both this Python interpreter session and the DOS console box where the command was typed.[*] An os.popen call can launch a start command too; but since it normally starts commands independently anyhow, the only obvious advantages of start here are the pop-up DOS box and the fact that Python need not be in the system search path setting:

[*] And remember, if you want to start a Python GUI program this way and not see the new DOS standard stream console box at all, simply name the script child-wait.pyw; the "w" on the end tells the Windows Python port to avoid the DOS box. For DOS jockeys: the start command also allows a few interesting options: /m (run minimized), /max (run maximized), /r (run restoredthe default), and /w (don't return until the other program exitsthis adds caller blocking if you need it). Type start /? for help. And for any Unix developers peeking over the fence: you can also launch independent programs with os.systemappend the & background operator to the command line. The standard library webbrowser module also provides a portable way to pop up an HTML file in a web browser, without requiring Windows registry associations.

>>> file = os.popen('start child-wait.py 1')    # versus: python child-wait...
>>> file.read( )
'Hello from child -413849 1\012Press <Enter>'

Which scheme to use, then? Using os.system or os.popen to run a python command works fine, but only if your users have added the python.exe directory to their system search path setting. Running a DOS start command is often a simpler alternative to both running python commands and calling the os.spawnv function, since filename associations are automatically installed along with Python and os.spawnv requires a full directory path to the Python interpreter program (python.exe). On the other hand, running start commands with os.system calls can fail on Windows for very long command-line strings:

>>> os.system('start child-wait.py ' + 'Z'*425)   # OK- 425 Zs in dos pop up
0
>>> os.system('start child-wait.py ' + 'Z'*450)   # fails- msg, not exception
Access is denied.
0
>>> os.popen('python child-wait.py ' + 'Z'*500).read( )    # works if PATH set
>>> os.system('python child-wait.py ' + 'Z'*500)           # works if PATH set

>>> pypath = r'C:\program files\python\python.exe'        # this works too
>>> os.spawnv(os.P_NOWAIT, pypath, ('python', 'child-wait.py', 'Z'*500))

As a rule of thumb, use one of the os.spawn variants if your commands are (or may be) long. For instance, we'll meet a script in Chapter 6 that launches web browsers to view HTML files; even though a start command applied to an HTML file will automatically start a browser program, this script instead must use os.spawnv to accommodate potentially long directory paths in HTML filenames.

The os.startfile call

One more Windows twist: as mentioned previously in this book, recent Python releases also include an os.startfile call, which is essentially the same as spawning a DOS start command with os.system and works as though the named file were double-clicked. The following calls, for instance, have a similar effect:

>>> os.startfile('README.txt')
>>> os.system('start README.txt')

Both pop up the README.txt file in Notepad on my Windows XP computer. Unlike the second of these calls, though, os.startfile provides no option to wait for the application to close (the DOS start command's /WAIT option does) and no way to retrieve the application's exit status (returned from os.system).

On recent versions of Windows, the following has a similar effect too, because the registry is used at the command line (though this form pauses until the file's viewer is closedlike using start /WAIT):

>>> os.system('README.txt')       # 'start' is optional today

A subtle thing: under the IDLE GUI in Python 2.4, the os.startfile call also does not pop up an intermediate DOS console window, whereas running a start command with os.system does, both from IDLE's interactive window and from within a script launched in IDLE. This seems to be just an artifact of the IDLE system, though; neither call pops up a DOS console outside of IDLE, whether typed interactively or run by a program that is launched from a system command line or icon click.

If you start a Python script with any of these call forms, what happens also depends upon the name of the file: a .py file pops up a console window for os.startfile, but a .pyw file will not (exactly as when a file icon is clicked). The os.system call may map the script's output to the interactive session window unless start is used in the command, in which case we get a console pop up again. This is complex and seems like the sort of small nonlanguage detail that may vary over time, so try the variations that you care about on your own to see the DOS pop-up differences among these three call forms

The os.startfile call works only on Windows, because it uses the Windows registry to know how to open the file. If you want to be more platform neutral, consider using os.popen or os.spawnv.

For more information on other Windows-specific program launcher tools, see O'Reilly's Python Programming on Win32, by Mark Hammond and Andy Robinson. Other schemes are less standard than those shown here but are given excellent coverage in that text.

Other Program Launch Options

For a slightly higher-level interface for launching shell commands, see also the standard library commands module, a Unix-only module which is roughly just a wrapper around the os.popen call.

In Python 2.4 and later, the subprocess module also allows you to spawn new processes, connect to their input, output, and error streams, and obtain their return codes. This module can be used to replace several older modules and functions, including os.system, os.spawn*, os.popen*, and commands.*. It provides lower-level control over spawned programs and is generally portable, but it can be more complex to code in some cases. Some advanced roles are made simpler by this module, however. For instance, to emulate shell-level command chaining with pipes, the following Python code:

from subprocess import Popen
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout)
output = p2.communicate( )[0]

is equivalent to this Unix shell language command:

output='dmesg | grep hda'



 Python   SQL   Java   php   Perl 
 game development   web development   internet   *nix   graphics   hardware 
 telecommunications   C++ 
 Flash   Active Directory   Windows