Alex Hunt

Snapshot Testing in Python for Sublime Text Plugins

I recently authored a small Sublime Text plugin called Sublime tmux that provides commands to interact with tmux sessions directly from the editor. Even though its features are concise, I later ended up adding several small enhancements which unveiled a non-trivial number of edge cases which I’d wished were covered by tests.

At Digital Detox we currently write our JavaScript tests using Jest, which allows us to rapidly iterate thanks the use of test snapshots. This involves writing select data fragments to a file as a snapshot and diffing against these in subsequent runs, putting any changes to an API surface up for review. This approach would be a perfect fit here, and as such I was determined try to make this — as far as I can tell — the first use of snapshot testing in a Sublime package.

Writing unit tests for Sublime

Install SublimeText/UnitTesting

This is the main testing framework for Sublime Text 3. It hosts tests written for Python’s built-in unittest framework and allows tests to interact with Sublime API features in the current app instance. It is distributed as a Sublime package installable from Package Control.

notion image

Define a test case

With direction from the provided test examples, I created a single test_open_tmux.py test module in my project. I could then run UnitTesting: Test Current Package from the Command Palette, which executes all tests found and prints the results to an output pane.

test_run_current_file (test_open_tmux.TestOpenTmuxCommand) ... ok

This first test sets a mock return value for the current directory path and invokes the tmux_open plugin command. Depending on what your particular plugin does, the availability of the Sublime API library here is likely to provide necessary hooks for calling whatever logic needs to be asserted.

import sublime
from unittest import TestCase, mock

class TestOpenTmuxCommand(TestCase):
    @mock.patch('os.path.dirname')
    def test_run_current_file(self, os_dirname_mock):
        os_dirname_mock.return_value = '/home/user/example-project'
        sublime.active_window().run_command('open_tmux')

        self.assertTrue(self.subproc_popen_mock.called)

I found mocking APIs with Python’s unittest.mock library convenient to use. Here however, details for self.subproc_popen_mock have been omitted for conciseness. TestCase also provides a complete set of assertion methods.

Pretty cool — but this is not the end. Next, we’re going to assert what was called using snapshots.

Snapshot testing for Python is a thing!

If you’re following what the subject plugin does, it ultimately makes command calls to the running tmux server:

💡
tmux new-window -c /home/alex/Development/dotfiles

During execution, this takes the form of a subprocess.Popen instance with a given set of arguments that contain tmux commands. So writing a snapshot assertion storing the call_args sent is all we need to do to cover this commands sent for each test.

The SnapshotTest library

This all depended on the existence of a Jest-like snapshot test library for Python. Thankfully, the creator of graphene had exactly this in mind with SnapshotTest, a relatively new testing module. We can install this using pip.

pip install snapshottest

Updating our tests

snapshottest.TestCase is based on unittest.TestCase, which means we can substitute one for the other in our class definition, leaving all standard assertion methods available.

import sublime
from unittest import mock
import snapshottest

class TestOpenTmuxCommand(snapshottest.TestCase):
    ...

Our test code now becomes, in fact, less interesting. A single self.assertMatchSnapshot assertion call is all that’s required. This example also shows additional use and mocking of Sublime's window API.

def test_run_unsaved_file_outside_project(self):
    view = sublime.active_window().new_file()

    with mock.patch('sublime.Window.folders') as window_folders_mock:
        window_folders_mock.return_value = []

        with mock.patch('os.path.expanduser') as home_dir_mock:
            home_dir_mock.return_value = '/home/user'
            sublime.active_window().run_command('open_tmux')

    view.window().run_command('close_file')
    self.assertMatchSnapshot(self.subproc_popen_mock.call_args[0])

When this runs (still from the UnitTesting plugin) an entry is written to the newly created file tests/snapshots/snap_test_open_tmux.py with the serialised snapshot data.

snapshots['TestOpenTmuxCommand::test_run_unsaved_file_outside_project 1'] = (
    [
        'tmux',
        'new-window',
        '-c',
        '/home/user'
    ]
)

Looks like the right command arguments (sent as a list)! And this saved output stands to assert it stays that way in future. Happy days.

In review

It worked! With this workflow validated, I finished writing a full set of snapshot tests covering the majority of Sublime tmux’s command scenarios which are now merged, and am now in a much better position to make future changes.

Storing snapshots for each test saved authoring time over defining custom assertions, but this was still a fairly lengthy exercise overall. Specific to this plugin, I definitely spent too much time working out how to correctly shape mock Popen calls for precheck commands, which led me to skip mocking a few more complicated test conditions for now.

For the most part, the SnapshotTest library worked well, and will most likely be a part of my next general Python project. More work is needed to get tests working in CI however, as the Sublime UnitTesting package runs slightly differently as configured and can’t find the snapshottest dependency. This is something I hope to resolve soon though.

All the tests written can be viewed here. If you’re a Sublime package maintainer and are inspired to try any of this out for yourself, please feel free comment below with any questions or feedback, or message me on Twitter.

 
Back to Home