Friday, June 24, 2011

Starting a Python script intelligently

Under Linux (or any Unix-style operating system), it is easy to make a Python script executable. You simply add a "shebang" line at the top of the script:
#! /usr/bin/python
and make the script executable:
chmod +x myscript
But real life is sometimes more complicated than this.
  1. The Python interpreter might be somewhere else in the user's $PATH, for example /usr/local/bin/python or $HOME/bin/python.
  2. There might be multiple Python interpreters and you might want to select one intelligently (e.g., use /usr/bin/python2.6 if it is installed, otherwise fall back to /usr/bin/python2.5, but if neither is available then emit a warning and abort).
  3. Other arbitrary work might have to be done before starting the Python interpreter.
So I was looking for a way to use a small blurb of shell code to intelligently start a Python script. Of course one could write a shell script that invokes a separate Python script, but for convenience, I wanted everything in a single file. And I wanted the bulk of the script to look like plain old Python code, not like a giant string that requires extra quoting. Google failed me, so I put together the following solution:
#! /bin/sh
# -*- mode: python; coding: utf-8 -*-

# This file is used as both a shell script and as a Python script.

# This part is run by the shell.  It looks for an appropriate Python
# interpreter then uses it to re-exec this script.

if test -x /usr/bin/python2.6
elif test -x /usr/bin/python2.5
  echo 1>&2 "No usable Python interpreter was found!"
  exit 1

exec $PYTHON "$0" "$@"
" """

# The rest of the file is run by the Python interpreter.
__doc__ = """This string is treated as the module docstring."""

print "Hello world!"
When this script is run:
  • The first line is a shebang that causes the script to be interpreted using /bin/sh; i.e., as a standard shell script.
  • The shell interpreter ignores the second line as a comment.
  • The line """:" is interpreted as the shell command : (yes, the colon character is the name of a shell command) with some funny quoting around it. The colon command does nothing, so interpretation continues on the next line.
  • The following lines are interpreted, one by one, as shell code. When interpretation reaches the line exec $PYTHON "$0" "$@", it causes the shell process to end and the selected Python interpreter to be executed to interpret the same script with the original command-line arguments. Because the shell interprets lines as they are read, it does not matter that the rest of the script is not valid shell syntax.
  • The Python interpreter starts executing the script. It ignores the shebang line, which just looks like a comment.
  • It interprets the second line as per PEP 0263 as an encoding declaration. It wouldn't be wise to pick a funky encoding, but utf-8 should be safe. This line also tells emacs to edit the file in Python mode even though it has a shell shebang line.
  • The lines between """:" and " """ are seen as a Python multiline string and ignored. (Actually, they are used as the module docstring, but we will overwrite that in a moment.) The only constraint is that the string cannot contain three consecutive quotation marks.
  • The line __doc__ = """...""" overwrites the module docstring. This is where you should document the module.
  • The rest of the file can be arbitrary Python code. This is where you put your main Python script.
This is the tidiest way that I could come up with for making a single file that is both valid shell code and valid Python code. If you know of a better way, please leave a comment.


Robin said...

Do you know about env? You can use a shebang like this:

#!/usr/bin/env python

Env will search $PATH for python and then exec the first one.

Nice hack with the file being both a shell script and python script by the way :).

Michael Haggerty said...

Yes, the /usr/bin/env trick is well-known (e.g., the trick and its pitfalls are described on the wikipedia page that I referenced). It mostly solves problem 1, but doesn't help with problems 2 or 3.

Marcin Kulik said...

This exactly what I was looking for, thanks!