Custom Apps

Definition

Zulip defines a "custom app" to be a piece of code that runs in the Zulip ecosystem, but which is not part of the core Zulip codebase. Custom apps are mostly synonymous with "bots" and "integrations" in the Zulip ecosystem. We currently do not support any kind of browser plugin model.

Problem statement

Zulip wants to enable people in the world to author custom apps with the following goals in mind:

This document describes Zulip's current infrastructure, as well as laying out a roadmap for some future features.

A quick note on bots/integrations

As noted earlier, a custom app is just a generic term for what we often call bots or integrations. We recognize that bots and integrations can have different connotations. A bot typically spends most of its time responding to Zulip messages. An integration usually represents an app that interacts with some large third party system like an issue tracker. We will use both terms in this document in an informal sense, but from an architecture standpoint, we treat bots and integrations as essentially two shades of the same color. Many integrations are implemented as "bots." Likewise, any bot that does stuff outside of Zulip acts as an "integration."

Since the line between what a "bot" is and what an "integration" is can get very blurry, we try to be informal about "bots/integrations" and more formal about how "custom apps" actually function within the system.

Categories of custom apps

Stimulus/response and read/write

At the end of the day, most useful apps respond to some stimulus and produce a response. In the Zulip universe, Zulip can be the source of the stimulus, or the target of the response, or both. Along those lines, we divide custom apps into these three types:

The above three classifications represent kind of a Zulip-centric view of the universe, but we should put ourselves in the shoes of somebody "out in the world."

Some things are a little outside of the scope of this document. We could plausibly extend Zulip some day to host World Reader/Writer apps that don't even write Zulip messages but simply use Zulip as a kind of middleware platform.

More in the short term, we will have custom apps that may read/write from multiple sources. For example, a meeting bot may take input from both a cron job and a Zulip stream, and it may write to both a Zulip stream and a third party calendar tool. For the scope of this document, we won't spend a lot of time talking about how to build these types of apps, but we are aware that any solution needs to accommodate multiple sources and targets.

World Reader/Zulip Reader

Finally, we set the stage for how we talk about custom apps in terms of these two broad categories:

Again, we recognize that there can be overlap between those two categories for complex custom apps, but we mostly leave it as an exercise for the reader how to implement those apps.

Other classifications

We discussed one dimension for classifying custom apps, which is whether they are world-readers or Zulip-readers. Here we cover a few other classification schemes briefly:

A lot of the classification schemes are interrelated. Here are some examples:

World Reader

A World Reader custom app is an app that responds to stimuli from the world outside of Zulip. It typically functions as a Zulip Writer and posts some kind of message to a Zulip stream or user to alert people of world events. Here are some example stimuli:

Setting aside issues of how a custom app is constructed or deployed, you basically have to solve these problems: - Detect events. - Translate events into Zulip messages. - Post the messages to Zulip.

Zulip integrations

Zulip actually supports a bunch of integrations out-of-the-box that perform as World Readers.

The three different integration models basically differ in where they perform the main functions of a World Reader.

Webhook integrations

In a webhook integration, the deployment model is usually this::

3rd party hardware: - detect event - send data to Zulip webhook

Zulip: - support webhook endpoint - translate event to messages - internally post messages

One current limitation of our system is that we don't have a great way to deploy prototypes of webhook-based custom apps before Zulip has vetted the translation and added an official endpoint. Maybe we could set up some kind of webserver that can run translation code outside of Zulip and externally post the messages, and we could think about how to structure the code so that it is easy to eventually turn it into a Zulip-hosted integration.

Python scripts

In script integrations, the deployment model is usually this:

Custom app author's hardware: - detect event by polling a third party system - translate event in the script - externally post messages

These type of integrations are typically easy to prototype, but they can be harder to deploy in production settings, since we rely on the authors to run their own scripts.

In some cases authors might want to at least move the translation/posting code to live on Zulip, by contributing that code to Zulip as a server-side integration. Then, there would still be the challenge of detecting events in the third party system, where maybe the user submits a patch to the third party as well.

Plugin integrations

In plugin integrations, the deployment model is usually this:

Third party system (driver): - detect event

Third party system (plugin): - further detect/triage event - translate event - externally post to Zulip

For third parties that have a plugin model, there are often other issues at play, like the plugins may need to be written in a non-Python language like Ruby. There are probably still some scenarios, however, where a lot of the logic for translation could be moved to a Zulip-side integration, and then we supply very thin client code for the plugin.

Zulip Reader

A Zulip Reader custom app gets stimuli from Zerver itself. Most Zulip Reader apps are packaged/advertised more as what people commonly call "bots" than as "integrations." (But sometimes what is currently a "bot" should really be deployed more like an "integration" in an ideal Zulip universe.)

Example custom Zulip Reader apps can be serious or whimsical.

Serious

Whimsical

Setting aside whether a custom app is performing a serious or whimsical function, there are a few different types of Zulip Readers:

Deployment issues

Zulip currently provides only minimal deployment support for Zulip Reader custom apps:

Local deployment

If you download the API client and write a bot that reads from Zulip, you face the following challenges if you deploy your code on your own devices:

We want to make it easier to deploy Zulip Readers on Zulip hardware. The following document talks about how we want to enable this from a code structuring standpoint:

Writing contrib bots

This document, on the other hand, is more about designing the Zulip backend system to support eventual deployment of reader apps on the Zulip server.

Before we talk about server-side apps, we should consider an intermediate solution.

Non-Zulip dedicated hardware

There are some scenarios, mostly with general-purpose "serious" custom apps, where an app author might use the following development process:

To give a concrete example, let's say that I work for a company that is building an issue tracker, and we want to offer Zulip support. I would start by writing a Zulip Reader that scans for the alert word @ticket on certain public Zulip streams, and part of that app would have logic to post to my company's issue-tracking API.

Once I'm confident in my prototype, I will probably run it on dedicated company hardware that might already have tight physical security, 24/7 IT monitoring, etc.

But what if I don't have this kind of infrastructure available to me? Typically what I will do instead is rent time on some kind of hosting service. Some hosting platforms are basically just remote Unix systems, but others are more oriented toward hosting web apps.

Zulip's current roadmap assumes that authors will likely gravitate toward web-based solutions (even if it's just running a web server on their own Unix host in the cloud).

Zulip intends to offer support for "outgoing webhooks." The term "outgoing webhook" can be confusing, depending on your perspective, but it simply means that an HTTP request is outgoing from Zulip, so that it will hit a web endpoint that runs a third-party custom app.

Zulip will allow the custom app author, probably with the help of a Zulip admin, to configure Zulip to send a subset of Zulip messages to the author's web endpoint, and then the protocol for the custom app will to read the HTTP request and send some kind of HTTP response that optionally results in a message being written to Zulip. Meanwhile, the custom app can mutate the "world" as it sees fit.

Zulip-side support for reader apps

Even for app authors that have access to dedicated hardware, there would be several advantages to running Zulip Readers under the same umbrella as the core Zulip system.

The only problem with the above bullets is that we haven't built out any of that infrastructure yet.

We do have pending PR #1393, which addresses some of the issues that might come up.

In order to run apps inside the Zulip server, we basically need to solve the problems below. (One assumption is that we don't run apps truly in-process.)