As an example, we’ll be looking at my own project, passacre. There is a python package called passacre immediately in the root of the project (i.e. not in a lib or src directory). Here’s what’s inside it:

1
2
3
4
5
6
7
8
9
10
11
12
13

passacre/__init__.py
passacre/__main__.py
passacre/application.py
passacre/config.py
passacre/generator.py
passacre/multibase.py
passacre/test/__init__.py
passacre/test/test_application.py
passacre/test/test_config.py
passacre/test/test_generator.py
passacre/test/test_multibase.py
passacre/test/util.py
passacre/util.py

Intra-package imports

There is exactly one way code in your package should be importing other code inside your package, and that’s with absolute imports.

Basic imports

So, in this code, passacre.config needs to import some things from passacre.multibase. The import looks like this:

1

from passacre.multibase import MultiBase

Note that the import uses the fully-qualified name of the module, passacre.multibase. This is important for two reasons:

1. It makes your code very explicit in what you’re importing.
2. Your code will continue to work in the future. There’s a very, very old method of doing imports (“implicit relative imports”) wherein one would write from multibase import Multibase instead, but this is gone in python 3. It should’ve been removed from python 2.7, but unfortunately, they forgot to do it.

There is another style of imports also covered in PEP 328, wherein the code would look instead look like this:

1

from .multibase import MultiBase

I’ll only mention explicit relative imports once more in this post; I very much prefer the look of absolute imports, but this is primarily an issue of style.

Test module imports

passacre also comes with a test suite in the passacre.test submodule. These tests necessarily have to import the passacre code to test it. Here’s what the passacre.test.test_multibase module does to import passacre.multibase:

1

from passacre.multibase import MultiBase

“But wait! That’s the same as before!” you cry. Since the fully-qualified module name is always the same, the imports are also always the same. This is another advantage of absolute imports—you always know exactly what you’re getting.

Sibling subpackage imports

For demonstration purposes, we’ll pretend that there’s another subpackage passacre.test2 that contains even more tests. Here’s the imaginary addition to the package structure:

1
2

passacre/test2/__init__.py
passacre/test2/test_multibase.py

This test_multibase module needs to use some utility function from passacre.test.util, so it imports the function:

1

from passacre.test.util import excinfo_arg_0

Using absolute imports means that it doesn’t matter where in the project a module is when you try to import another module. Python will always look up the module to import from the root of your package.

Shadowed standard library module imports

Something I’m sure everyone has done at some point in their python career is name a module the same as a module in the standard library. If I wanted a passacre.socket module that used the standard library socket module, import socket should give me the standard library module instead of passacre.socket. PEP 328 covers this too by adding a __future__ feature absolute_import, which is on by default in python 3.0 and higher.

Activating absolute_import has exactly one effect: disabling implicit relative imports. Since __future__ features only affect the module enabling them, I can simply add from __future__ import absolute_import to passacre/socket.py and any import socket calls in it will import the standard library socket module.

Distribution

Once your package has some code in it that can run, you’ll probably want to let other people use your package as well.

setup.py

setup.py is your package’s entry point into distutils or setuptools. Every package should have one, as that’s how your code gets put into the right place for your users. I won’t go too much into detail in how to use them; the links above are for tutorials which will explain how to get started.

My own rule of thumb is to use distutils unless I absolutely require a feature from setuptools. Some people will say to always use setuptools, and that’s okay too.

Executable scripts

Jean-Paul’s post recommends putting python scripts in the bin directory of your project root. This is fine advice, but there are newer methods which mean you don’t need a bin directory at all.

First, there’s __main__.py. This is not quite the same as an executable script, but it allows a package to be executed via python -m. For example, passacre/__main__.py means that one can execute python -m passacre and the __main__.py file will be executed. PEP 338 has more details on exactly how this works.

passacre/__main__.py is fairly small:

1
2

from passacre.application import main
main()

Since __main__.py is executed directly, the actual main function should live elsewhere if you actually want to be able to test it. This way, test code can also from passacre.application import main and be able to call that function.

python -m is not limited to executing packages. For times when you want to quickly run some module that has an if __name__ == '__main__': block in it, you can use python -m passacre.thatmodule. Importantly, this will ensure that python recognizes your whole package, which means that your imports won’t fail.

Sometimes a project wants explicit binaries instead of requiring their users to use python -m. For this case, setuptools has automatic script creation. Here’s an excerpt from passacre’s setup.py file:

1
2
3
4
5
6
7
8
9
10

from setuptools import setup

# [...]

setup(
    # [...]
    entry_points={
        'console_scripts': ['passacre = passacre.application:main'],
    },
)

(The actual setup.py uses extras, but that’s not important for this blog post.)

This console_scripts definition makes a passacre executable that does the same thing as the __main__.py script did: imports the main function from passacre.application and calls it.

Using distutils or setuptools to generate and/or install your python scripts is important—as a part of the install process, the shebang line (e.g. #!/usr/bin/env python) will be rewritten to use the python that ran setup.py. So, if someone does /usr/local/bin/python2.6 setup.py install, passacre will start with #!/usr/local/bin/python2.6.

Additionally, python -m and automatic script creation can be used beautifully together—a project can have a scripts subpackage containing one module for every executable script. For example, passacre_generate could be passacre_generate = passacre.scripts.generate:main and also contain:

1
2

if __name__ == '__main__':
    main()

For projects with a lot of scripts, this is a good organization method to keep the scripts separate from the more library-ish code.

Documentation

Please have some, especially if you’re distributing your package. Docstrings are a good start, but a tutorial written with sphinx and/or a README with usage examples goes a long way.

Common pitfalls

Don’t directly run modules inside packages

Seriously. Never ever ever do this. What I mean specifically is doing python passacre/__main__.py, or python passacre/test/test_application.py, or anything else that starts with python passacre/. This will prevent python from knowing where the root of your package is, and so absolute imports and explicit relative imports will fail, or not import the code you think it should be importing.

If you think you have to do this, you’re wrong. Instead, you probably want to use python -m, or generate an executable script to run, or use a test runner. If you don’t think any of those cover your use case, leave a comment and I’ll show you a better method.

Don’t set PYTHONPATH to try to make it go

If you think you have to set PYTHONPATH, you’ve probably fallen victim to the first pitfall and are trying to execute a module directly. With proper package layout and proper imports, you won’t need to set PYTHONPATH to run your code.

Don’t modify sys.path from code in your package

Like modifying PYTHONPATH, but worse because it’s easier to affect other people using your code. Never do this, as it will break and make people trying to use your code very annoyed.

Don’t make your project root a package

Your project root should contain a python package, not be a package itself. If you do this, your setup.py will be very confusing (or not work at all) and it will be very difficult to run your code.

Conclusion

Really, writing a python package isn’t hard. The rules above are very simple and will lead to simple, easy-to-read-and-understand python code.

The other crucial part of my local emacs setup has been emacsclient. Setting my local $EDITOR to emacsclient means that I could, for example, run git commit in a terminal and edit the commit message without having to spawn a new emacs instance. Instead, emacsclient tells the existing emacs what files to edit, and waits until emacs says it’s done with them.

Until just yesterday, there was a major flaw in my setup: I couldn’t activate my local emacs instance through emacsclient on a remote host. This meant that $EDITOR had to be set to something that spawned an editor on the remote host, which was becoming increasingly frustrating.

In the past I’d tried doing things like using an ssh tunnel to forward the emacsclient connections from the remote host to my local machine, but emacsclient doesn’t anticipate this case. The emacsclient protocol is very simple: ASCII commands delimited with line feeds. So, for example:

1
2
3
4

$ emacsclient /foo/bar
# and this sends (approximately):
-auth some-gunk-here
-dir /foo -file /foo/bar

The problem is that this is being sent to your local emacs, which then tries to load the local /foo/bar. A TRAMP path is in the form of (approximately) /ssh:example.com:/foo/bar, so if only this could be prepended, emacs could deal with it!

In my most recent investigation of the problem, I found Ryan Barrett’s implementation, which was most of what I wanted. Except,

1. It wasn’t automatic. It’s really convenient to be able to have TRAMP dump the appropriate authentication file onto the server as soon as a connection is made without having to connect using a special function.

2. It was relying on emacsclient sending a -tty argument in a specific format, and then using the TRAMP prefix of the current buffer. In practice this means you have to have your frontmost buffer a file open on the remote server, and only then can you use emacsclient on that remote server.

I started to hack on it, but I was still affected by the same original problem: emacsclient doesn’t know about TRAMP. The default implementation doesn’t even have a way of specifying additional data to be sent over the wire. With the simplicity of the emacsclient protocol, it started looking like writing my own would be the simplest choice.

So, I put together some elisp to write out a special client authentication file that would also include the TRAMP prefix and a shell script that would parse it and write commands using nc. My elisp implementation borrows code from Ryan Barrett’s (and thank you for getting me started with this!) but goes a bit farther. The full workflow is like this:

1. Update your .emacs to load and configure the elisp.

2. Update your .profile to set $EDITOR to the emacsclient.sh script and tell it where the authentication file is.

3. Open a TRAMP connection from your local emacs to write out the authentication file.

4. Go hog wild.

5. Repeat only steps 3-5.

Here are the versions of emacsclient.sh and remote-emacsclient.el as of 2013-06-25:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

#!/bin/sh
nowait=0
if [ "$1" = "-n" ]; then
    nowait=1
    shift
fi

if [ "$#" -eq 0 ]; then
    echo "no files specified"
    exit 0
fi

[ -e "$remote_emacs_auth" ] && client_host=$(sed -n 1p "$remote_emacs_auth")
if [ -z "$client_host" ] || ! nc -z $client_host; then
    echo "no emacs server"
    sleep 1
    exec emacs "$@"
fi

quote () {
    sed 's:&:\&\&:g;s:-:\&-:g;s: :\&_:g'
}

quoteline () {
    printf "%s\n" "$1" | quote
}

unquote () {
    sed 's:&&:\&:g;s:&-:-:g;s:&_: :g;s:&n:\
:g'
}

client_auth=$(sed -n 2p "${remote_emacs_auth}")
tramp_prefix=$(sed -n 3p "${remote_emacs_auth}")
args=$(printf "%s\n%s\n" "${client_auth}" "-dir ${tramp_prefix}${quoted_pwd}")
[ "${nowait}" != 0 ] && args="${args} -nowait"
quoted_pwd=$(quoteline "$(pwd)")

for file; do
    quoted_file=$(quoteline "${file}")
    case "${file}" in
        +*) argument="-position ${quoted_file}";;
        /*) argument="-file ${tramp_prefix}${quoted_file}";;
        *)  argument="-file ${tramp_prefix}${quoted_pwd}/${quoted_file}";;
    esac
    args="${args} ${argument}"
done

printf "%s\n" "${args}" | nc $client_host | unquote

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123

(require 'tramp)
(require 'tramp-sh)

(setq server-use-tcp t)
(server-start)

(defun put-alist (key value alist)
  "Set cdr of an element (KEY . ...) in ALIST to VALUE and return ALIST.
If there is no such element, create a new pair (KEY . VALUE) and
return a new alist whose car is the new pair and cdr is ALIST."
  (let ((elm (assoc key alist)))
    (if elm
        (progn
          (setcdr elm value)
          alist)
      (cons (cons key value) alist))))

(defun update-tramp-emacs-server-port-forward (method-name)
  "Update the specified TRAMP's method to forward the Emacs
 server port to the local host. This lets emacsclient on the
 remote host open files in the local Emacs server."
  (let* ((method (assoc method-name tramp-methods))
         (ssh-args (cadr (assoc 'tramp-login-args method))))
    (put-alist 'tramp-login-args
      (list (put-alist "-R" (let ((port
                                   (process-contact server-process :service)))
        ;; put-alist makes a dotted pair for the key/value, but tramp-methods
        ;; needs a normal list, so put the value inside a list so that the
        ;; second part of the dotted pair (ie the cdr) is a list, which
        ;; converts it from a dotted pair into a normal list.
                              (list (format "%s:127.0.0.1:%s" port port)))
                       ssh-args))
      method)))

(defun tramp-make-tramp-file-name-from-vec (vec file)
  "Convenience function for making a TRAMP path, since this
apparently didn't already exist."
  (tramp-make-tramp-file-name
    (tramp-file-name-method vec)
    (tramp-file-name-user vec)
    (tramp-file-name-host vec)
    file))

(defcustom tramp-default-remote-emacsclient-auth-file
  "~/.emacs.d/remote-server"
  "Default remote path at which to save the remote emacsclient
authentication file. This can be a string or nil to disable
saving an authentication file.

The authentication file is similar to the one written out by the
emacsclient TCP server, except it includes the prefix used for
the TRAMP connection to the remote server."
  :group 'tramp
  :type '(choice (const nil) string))

(defcustom tramp-remote-emacsclient-auth-file-alist nil
  "The remote emacsclient authentication file path to use for
specific host/user pairs. This is an alist of items (HOST USER
PATH). The first matching item specifies the path to use for a
connection which does not specify a method. HOST and USER are
regular expressions or nil, which is interpreted as a regular
expression which always matches. If no entry matches, the
variable `tramp-default-remote-emacsclient-auth-file' takes
effect.

If the connection does not specify the user, lookup is done using
the empty string for the user name.

See `tramp-default-remote-emacsclient-auth-file' for an
explanation of the auth file path."
  :group 'tramp
  :type '(repeat (list (choice :tag "Host regexp" regexp (const nil))
             (choice :tag "User regexp" regexp (const nil))
             (choice :tag "emacsclient auth path" string (const nil)))))

(defun tramp-get-remote-emacsclient-auth-file (vec)
  "Determine the full TRAMP path for the remote emacsclient
authentication file, given a connection vector."
  (let
      ((auth-file
        (let ((choices tramp-remote-emacsclient-auth-file-alist)
              (host (or (tramp-file-name-host vec) ""))
              (user (or (tramp-file-name-user vec) ""))
              lfile item matched)
          (while choices
            (setq item (pop choices))
            (when (and (string-match (or (nth 0 item) "") host)
                       (string-match (or (nth 1 item) "") user))
              (setq lfile (nth 2 item)
                    choices nil
                    matched t)))
          (if matched lfile tramp-default-remote-emacsclient-auth-file))))
    (if auth-file
        (tramp-make-tramp-file-name-from-vec vec auth-file))))

(defun tramp-save-remote-emacsclient-auth-file (&optional vec)
  "Write the remote emacsclient authentication file for a given
connection buffer, or, if used interactively, for the TRAMP
connection of the current buffer."
  (interactive)
  (let ((vec (or vec (tramp-dissect-file-name default-directory))))
    (condition-case err
        (let ((auth-file (tramp-get-remote-emacsclient-auth-file vec))
              (server (process-contact server-process :local)))
          (if auth-file
              (with-temp-file auth-file
                (insert
                 (format "127.0.0.1 %d\n" (elt server (- (length server) 1)))
                 (format "-auth %s\n" (process-get server-process :auth-key))
                 (server-quote-arg (tramp-make-tramp-file-name-from-vec vec ""))
                 "\n"))
            (when (called-interactively-p 'any)
              (message "No remote emacsclient auth file for %s"
                       default-directory))))
      (file-error (message "error saving remote emacsclient auth: %s" err)))))

(defadvice tramp-open-connection-setup-interactive-shell
  (after copy-server-file-by-tramp (proc vec) activate)
  "Automatically write out a remote emacsclient auth file after a
successful connection."
  (tramp-save-remote-emacsclient-auth-file vec))

(provide 'remote-emacsclient)

The most up-to-date versions of each will be in my dotfiles repository. For convenience, here are links to the files within the repository: emacsclient.sh and remote-emacsclient.el.

Configuration is pretty straightforward. In my local .emacs file, (server-start) got changed to:

1
2

(require 'remote-emacsclient)
(update-tramp-emacs-server-port-forward tramp-default-method)

The update-tramp-emacs-server-port-forward function takes a TRAMP method and updates the ssh arguments to that method to include a -R flag forwarding remote TCP connections to the local server. It’s probably not necessary to call it for any TRAMP method other than your default.

By default every new server connection will leave an authentication file. This can be tuned with the tramp-default-remote-emacsclient-auth-file and tramp-remote-emacsclient-auth-file-alist variables. The former can be set to nil to disable auth file creation by default. The latter is in the same form as the tramp-default-method-alist variable for specifying the location of the auth file by host and/or by user.

And in my .zshrc:

1
2
3

: ${remote_emacs_auth:="$HOME/.emacs.d/remote-server"}; export remote_emacs_auth
export EDITOR="$HOME/.dotfiles/emacsclient.sh"
alias emacs="$HOME/.dotfiles/emacsclient.sh -n"

$remote_emacs_auth’s default mirrors remote-emacsclient.el’s default of writing the auth file to ~/.emacs.d/remote-server. Since emacsclient.sh is (for me, by default) checked out in ~/.dotfiles and set executable, this is all that’s required to use it as an editor. As an added bonus, the emacs alias lets me type emacs somefile over ssh to open somefile in my local emacs.

There’s only two caveats (that I’m aware of (I’ve only tried the scp, ssh, and scpc tramp methods with this so far)):

1. Since the authentication key changes every time emacs is started, you must open a new tramp connection to a server before emacsclient will be able to speak to your local emacs. I haven’t found a good solution to this other than forcing the authentication key to always be the same (bad) or making emacs reconnect to everything whenever you start it (worse).

2. There’s no detection of when the server is restarted. Restarting the server in a running emacs also changes the authentication key. Making a new tramp connection isn’t required in this case; there’s a tramp-save-remote-emacsclient-auth-file function that can be run interactively. Running it will write a new auth file for the connection corresponding to the current buffer.

Always initiate from the client

Until a bug in Twisted is resolved, only the client will be able to successfully initiate the StartTLS. Here’s a full runnable example.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

from twisted.internet import protocol, reactor
from twisted.protocols import amp
from twisted.python import log

import sys

class SupCommand(amp.Command):
    pass

class ServerProtocol(amp.AMP):
    @SupCommand.responder
    def sup(self):
        log.msg('got sup')
        return {}

class ServerFactory(protocol.ServerFactory):
    protocol = ServerProtocol

class ClientProtocol(amp.AMP):
    def connectionMade(self):
        d = self.callRemote(amp.StartTLS)
        d.addCallback(lambda ign: self.callRemote(SupCommand))

class ClientFactory(protocol.ClientFactory):
    protocol = ClientProtocol

if sys.argv[1] == 'client':
    reactor.connectTCP('localhost', 9991, ClientFactory())
elif sys.argv[1] == 'server':
    reactor.listenTCP(9991, ServerFactory())

log.startLogging(sys.stdout)
reactor.run()

Using client certificates isn’t hard

Specifying which certificate to use for both peers requires only minor modifications. Here’s another runnable example, assuming you have an ssl.pem file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

from twisted.internet import protocol, reactor, ssl
from twisted.protocols import amp
from twisted.python import log

import sys

cert = ssl.PrivateCertificate.loadPEM(open('ssl.pem').read())
certParams = {'tls_localCertificate': cert}

class SupCommand(amp.Command):
    pass

class ServerProtocol(amp.AMP):
    @SupCommand.responder
    def sup(self):
        log.msg('got sup')
        return {}

    @amp.StartTLS.responder
    def startTLS(self):
        return certParams

class ServerFactory(protocol.ServerFactory):
    protocol = ServerProtocol

class ClientProtocol(amp.AMP):
    def connectionMade(self):
        d = self.callRemote(amp.StartTLS, **certParams)
        d.addCallback(lambda ign: self.callRemote(SupCommand))

class ClientFactory(protocol.ClientFactory):
    protocol = ClientProtocol

if sys.argv[1] == 'client':
    reactor.connectTCP('localhost', 9991, ClientFactory())
elif sys.argv[1] == 'server':
    reactor.listenTCP(9991, ServerFactory())

log.startLogging(sys.stdout)
reactor.run()

ssl.pem must include both the certificate and the key. Though the example doesn’t demonstrate it, certificate authorities to use can also be specified using the tls_verifyAuthorities parameter (see the API docs for StartTLS).