Manual testing

As a general rule, we like to have automated tests for everything that can be practically tested. However, there are certain types of bugs that are best caught with old fashioned manual testing (also called manual QA). Manual testing not only catches bugs, but it also helps developers learn more about the system and think about the existing semantics of a feature they're working on.

This doc assumes you know how to set up a local development server and open the Zulip app in the browser. It also assumes a basic knowledge of how to use Zulip.

Basic stuff

When testing Zulip manually, here are things to focus on:

You generally want to test with Cordelia as the primary user, and use Hamlet as her primary conversation partner. Use Iago when you need to test administrative functions. Send messages to Othello or Prospero if you want to verify things such as Cordelia not being able to receive messages not intended for her.

The rest of this document groups tasks into basic areas of functionality of the system. If you have multiple people testing at once, you can divvy up QA tasks by these sections in the doc.

Message view

We mostly test the message view as part of testing everything else, but there are few things to specially test here.

Try using all the navigation hotkeys: - Up/k - Down/j - PgUp/K - PgDn/J/Spacebar - End (or fn-right-arrow on OSX) - also try scrolling aggressively with the mouse

Try narrowing from the message view: - Hotkeys - use a to go to All messages - use s to narrow to a stream (select message first and verify in sidebar) - use S to narrow to the topic (and verify in sidebar) - use v to navigate to private messages - Click on the recipient bar - narrow to a stream - narrow to a topic - narrow to PMs with one user - narrow to a group PM - Click on the Zulip logo - narrow to a topic - click on the Zulip logo (and verify you're in the Recent topics view)

Messagebox

With messagebox we want to test mainly the functioning of all all the keyboard shortcuts and click handlers.

Apart from that there are three views of a message box, we want to test their appearance too: - Message that includes sender - Message without the sender - "/me" message

Here's how we're going to test the message appearances: - narrow to a new topic and send a message (this message will include sender) - edit the message ("(EDITED)" label should appear beside sender name) - send another message (will not include sender) - edit the message ("(EDITED)" label should appear in the left column, where the avatar is) - send a "/me" message (/me test message) - message should appear alongside sender name - edit the message ("(EDITED)" label should appear beside the message)

For all the three cases, we need to test the click handlers and the hotkeys too: - Sender popover: - click on the avatar and sender name for messages which include sender - press 'u' to open the sender popover for all messages - Message reply: - click on message to reply - use 'r' or Return hotkey to reply - use '>' to quote and reply - use '@' to mention and reply - Reactions: - click on the reactions button to open menu - use ':' to open the reactions menu - react to a message - Chevron - click on chevron to open menu - use 'i' to open chevron menu - Message edit - click on message edit/view source button - use 'i' + Return to edit/view source message - click on the 'copy and close' option in view source and verify positioning of 'Copied!' label. - Star a message: - click on the star button in the right column - use 'Ctrl + S' to star a message - Message length - send a long message and see if '[More]' appears - click on the 'more' or 'collapse' link - use i to collapse/expand a message irrespective of message length - use 'v' to show all images in the thread - use 'M' to mute the thread

Play with the screen size to check if the messagebox appears fine in different screens too.

Message editing

With message editing we mainly want to exercise topic changes.

Here are some tasks:

Narrowing

Zulip uses the term "narrowing" to refer to opening different views of your messages, whether by clicking on sidebar options, recipient bars, or by using search. The main focus of these tasks should be watching unread counts. Of course, you also want to see messages show up in the message pane. And, finally, you should make sure that no messages outside the narrow show up in Cordelia's view.

:::{important} Make sure that Cordelia is subscribed to Verona but not subscribed to Denmark; if not, you should use different streams for your testing. :::

When testing narrows, you want to have Hamlet send the same message several times in a row, while cycling Cordelia through various narrows.

Here are the main tasks for Hamlet (and each message gets sent several times):

For each of the above types of messages, you will want to cycle through the following views for Cordelia (and have Hamlet send new messages after each narrow):

There are 56 things to test here. If you can get into a rhythm where you can test each case in about 30 seconds, then the whole exercise is about 30 minutes, assuming no bugs.

Composing messages

We have pretty good automated tests for our Markdown processor, so manual testing is targeted more to other interactions. For composing a message, pay attention to details like what is automatically populated and where the focus is placed.

Popover menus

For this task you just want to go through all of our popover menus and exercise them. The main nuance here is that you occasionally want to click somewhere on the UI outside of an existing popover to see if the popover menu is "too sticky." Also, occasionally actions will be somewhat jarring; for example, if you mute a message in the current view, then the message will disappear from the view.

Here are the things to test:

This is a fairly quick task where we test the search filters on the left sidebar and the buddy list. If Cordelia is not subscribed to Denmark, subscribe her to that stream.

Stream permissions

This is an important category to test, because we obviously do not want to have bugs where people can read messages on streams they should not have access to.

The general flow here is for Hamlet to create the streams and verify that Cordelia has the correct visibility to them.

First, we start off with "positive" tests.

For negative tests, we want to dig a little deeper to find back doors for Cordelia to access the stream. Here are some techniques to try:

For public streams, it's ok for Cordelia to know the stream exists, and she can subsequently subscribe. For private streams, she should not even know they exist (until she's invited, of course).

The main task for testing search is to play around with search suggestions (autocomplete). Once you select an option, verify the message view is consistent with the search and that the left sidebar reflects the current narrow. If a search comes up legitimately empty, have Hamlet send a message that matches the search.

Here are searches you should be able to do with autocomplete: - stream:design - stream:Verona topic:Verona1 - stream:Verona keyword - sent by me - @-mentions - starred messages - messages sent by Hamlet - PMs with Hamlet - PMs with Hamlet matching keyword "foo"

There are some things you can try that don't come up in autocomplete: - -stream:Verona (exclude Verona) - stream:Verona stream:devel (should return no results)

Miscellaneous: - Use the "/" hotkey to start a search. - Use the "x" icon to clear a search. - Use the "Esc" hotkey to clear a search.

Stream settings

Test various UI entry points into stream settings: - Use small gear menu in left sidebar, then filter to "devel". - Use popover menu in left sidebar next to "devel". - Use gear menu above buddy list and filter to "devel". - Use gear menu and click on "devel." - Use gear menu and then click on chevron menu next to "devel." (I'm not sure why we still have the chevron at this writing.)

Create new public stream "public1" and add Hamlet: - Type "public1" in the text box and then click "Create new stream." - Select "People must be invited" and then verify you can't select "Announce stream". - Select "Anyone can join" again to make it be public. - Check the checkbox for Hamlet. - Hit the "Create" button.

Test subscribe/unsubscribe: - Log in as Hamlet and go to his stream settings. - As Cordelia, unsubscribe from "public1" using the checkmark in the streams settings page. - Verify that Hamlet sees that Cordelia has unsubscribed (and the subscriber count should decrement). - As Cordelia, resubscribe to "public1." - Verify Hamlet sees that change.

As Cordelia, exercise different options in Create Stream dialog by creating streams s1, s2, s3, etc.: - s1: anyone can join, announce it, and add Hamlet using filter feature - s2: people must be invited - s3: anyone can join, don't announce - s4: check all, then uncheck all, then invite only Hamlet - s5: invite everybody but Hamlet - s6: - create the stream as public, but don't subscribe anybody initially - then click on stream options to add Hamlet using "Add" button

Test per-stream options: - Use "devel" stream and send a message to it - Do mute and unmute, have Hamlet send messages - Test notifications on/off, have Hamlet send messages - Test pin and unpin, view left sidebar - Change stream color, and then view the left sidebar and the All messages view - Verify stream subscriber counts in the main stream view

User settings

You can modify per-user settings by choosing "Settings" in the gear menu. Do these tasks as Cordelia.

Keyboard shortcuts

We mostly test keyboard shortcuts as part of other tasks.

Here are the tasks for this section: - Use the "?" hotkey to open the keyboard help - Proofread the dialog for typos. - Close the dialog. - Re-open the keyboard help using the gear menu. - Find a hotkey that you don't frequently use and experiment with its usage.

Miscellaneous menu options

Make sure that these options launch appropriate help screens: - Proofread and try a couple random options: - Message formatting - Search operators - Make sure help launches in a separate browser tab: - Desktop and mobile apps - Integrations - API documentation

Inviting users/tutorial

Here are the tasks: - Invite ignore@zulip.com using the link beneath the buddy list but then don't take further action. - Fully invite foo@zulip.com using the gear menu. - Go to the development console to get the login link for foo@zulip.com. - Go through the signup flow. - Follow the tutorial. - Use the gear menu to log out. - Log back in as Cordelia (admittedly, this step doesn't really QA much of our production code, since the login flow is customized for the development environment).

To be continued...

This document does not cover settings/admin options yet. The main things to do when testing the settings system are: - Verify that changes are synced to other users. - Verify error messages appear if you do something wrong and look right. - For organization settings, verify that they look right in read-only mode (i.e. when not logged into an administrator account).