Snapshot Testing in Python for Sublime Text Plugins
I recently authored a small Sublime Text plugin called Sublime tmux that provides commands to interact with tmux sessions directly from the editor. Even though its features are concise, I later ended up adding several small enhancements which unveiled a non-trivial number of edge cases which I’d wished were covered by tests.
Writing unit tests for Sublime
This is the main testing framework for Sublime Text 3. It hosts tests written for Python’s built-in unittest framework and allows tests to interact with Sublime API features in the current app instance. It is distributed as a Sublime package installable from Package Control.
Define a test case
With direction from the provided test examples, I created a single test_open_tmux.py test module in my project. I could then run UnitTesting: Test Current Package from the Command Palette, which executes all tests found and prints the results to an output pane.
test_run_current_file (test_open_tmux.TestOpenTmuxCommand) ... ok
This first test sets a mock return value for the current directory path and invokes the
tmux_open plugin command. Depending on what your particular plugin does, the availability of the Sublime API library here is likely to provide necessary hooks for calling whatever logic needs to be asserted.
import sublime from unittest import TestCase, mock class TestOpenTmuxCommand(TestCase): @mock.patch('os.path.dirname') def test_run_current_file(self, os_dirname_mock): os_dirname_mock.return_value = '/home/user/example-project' sublime.active_window().run_command('open_tmux') self.assertTrue(self.subproc_popen_mock.called)
I found mocking APIs with Python’s unittest.mock library convenient to use. Here however, details for
self.subproc_popen_mock have been omitted for conciseness.
TestCase also provides a complete set of assertion methods.
Pretty cool — but this is not the end. Next, we’re going to assert what was called using snapshots.
Snapshot testing for Python is a thing!
If you’re following what the subject plugin does, it ultimately makes command calls to the running tmux server:
During execution, this takes the form of a
subprocess.Popen instance with a given set of arguments that contain tmux commands. So writing a snapshot assertion storing the call_args sent is all we need to do to cover this commands sent for each test.
The SnapshotTest library
This all depended on the existence of a Jest-like snapshot test library for Python. Thankfully, the creator of graphene had exactly this in mind with SnapshotTest, a relatively new testing module. We can install this using pip.
pip install snapshottest
Updating our tests
snapshottest.TestCase is based on
unittest.TestCase, which means we can substitute one for the other in our class definition, leaving all standard assertion methods available.
import sublime from unittest import mock import snapshottest class TestOpenTmuxCommand(snapshottest.TestCase): ...
Our test code now becomes, in fact, less interesting. A single
self.assertMatchSnapshot assertion call is all that’s required. This example also shows additional use and mocking of Sublime's window API.
def test_run_unsaved_file_outside_project(self): view = sublime.active_window().new_file() with mock.patch('sublime.Window.folders') as window_folders_mock: window_folders_mock.return_value =  with mock.patch('os.path.expanduser') as home_dir_mock: home_dir_mock.return_value = '/home/user' sublime.active_window().run_command('open_tmux') view.window().run_command('close_file') self.assertMatchSnapshot(self.subproc_popen_mock.call_args)
When this runs (still from the UnitTesting plugin) an entry is written to the newly created file tests/snapshots/snap_test_open_tmux.py with the serialised snapshot data.
snapshots['TestOpenTmuxCommand::test_run_unsaved_file_outside_project 1'] = ( [ 'tmux', 'new-window', '-c', '/home/user' ] )
Looks like the right command arguments (sent as a list)! And this saved output stands to assert it stays that way in future. Happy days.
It worked! With this workflow validated, I finished writing a full set of snapshot tests covering the majority of Sublime tmux’s command scenarios which are now merged, and am now in a much better position to make future changes.
Storing snapshots for each test saved authoring time over defining custom assertions, but this was still a fairly lengthy exercise overall. Specific to this plugin, I definitely spent too much time working out how to correctly shape mock
Popen calls for precheck commands, which led me to skip mocking a few more complicated test conditions for now.
For the most part, the SnapshotTest library worked well, and will most likely be a part of my next general Python project. More work is needed to get tests working in CI however, as the Sublime UnitTesting package runs slightly differently as configured and can’t find the snapshottest dependency. This is something I hope to resolve soon though.
All the tests written can be viewed here. If you’re a Sublime package maintainer and are inspired to try any of this out for yourself, please feel free comment below with any questions or feedback, or message me on Twitter.