Testing overview

Zulip takes pride in its extensive, carefully designed test suites. For example, test-backend runs a complete test suite (~98% test coverage; 100% on core code) for the Zulip server in under a minute on a fast laptop; very few web apps of similar scope can say something similar.

This page focused on the mechanics of running automated tests in a development environment; you may also want to read about our testing philosophy and continuous integration setup.

Manual testing with a web browser is primarily discussed in the docs on using the development environment.

Running tests

Zulip tests must be run inside a Zulip development environment; if you're using Vagrant, you may need to enter it with vagrant ssh.

You can run all of the test suites (similar to our continuous integration) as follows:

./tools/test-all

However, you will rarely want to do this while actively developing, because it takes a long time. Instead, your edit/refresh cycle will typically involve running subsets of the tests with commands like these:

./tools/lint zerver/lib/actions.py # Lint the file you just changed
./tools/test-backend zerver.tests.test_markdown.MarkdownTest.test_inline_youtube
./tools/test-backend MarkdownTest # Run `test-backend --help` for more options
./tools/test-js-with-puppeteer navigation.ts
./tools/test-js-with-node utils.js

The commands above will all run in just a few seconds. Many more useful options are discussed in each tool's documentation (e.g. ./tools/test-backend --help).

Major test suites

Zulip has a handful of major tests suite that every developer will eventually work with, each with its own page detailing how it works:

Other test suites

Additionally, Zulip also has about a dozen smaller tests suites:

Each of these has a reason (usually, performance or a need to do messy things to the environment) why they are not part of the handful of major test suites like test-backend, but they all contribute something valuable to helping keep Zulip bug-free.

Internet access inside test suites

As a policy matter, the Zulip test suites should never make outgoing HTTP or other network requests. This is important for 2 major reasons:

As a result, Zulip's major test suites should never access the Internet directly. Since code in Zulip does need to access the Internet (e.g. to access various third-party APIs), this means that the Zulip tests use mocking to basically hardcode (for the purposes of the test) what responses should be used for any outgoing Internet requests that Zulip would make in the code path being tested.

This is easy to do using test fixtures (a fancy word for fixed data used in tests) and the mock.patch function to specify what HTTP response should be used by the tests for every outgoing HTTP (or other network) request. Consult our guide on mocking to learn how to mock network requests easily; there are also a number of examples throughout the codebase.

We partially enforce this policy in the main Django/backend test suite by overriding certain library functions that are used in outgoing HTTP code paths (httplib2.Http().request, requests.request, etc.) to throw an exception in the backend tests. While this is enforcement is not complete (there a lot of other ways to use the Internet from Python), it is easy to do and catches most common cases of new code depending on Internet access.

This enforcement code results in the following exception:

pytb File "tools/test-backend", line 120, in internet_guard raise Exception("Outgoing network requests are not allowed in the Zulip tests." Exception: Outgoing network requests are not allowed in the Zulip tests. ...

Documentation tests

The one exception to this policy is our documentation tests, which will attempt to verify that the links included in our documentation aren't broken. Those tests end up failing nondeterministically fairly often, which is unfortunate, but there's simply no other correct way to verify links other than attempting to access them. The compromise we've implemented is that in CI, these tests only verify links to websites controlled by the Zulip project (zulip.com, our GitHub, our ReadTheDocs), and not links to third-party websites.