How do you separate the signal from the noise of product changes?

What are the heuristics for separating “important” product changes from minor changes that are essentially background noise? If this question sparks your interest, keep reading.

Everyone who works at a software company must perpetually ask the question: “How did our product change?”

The folks who sell, market, or support the product must understand changes to do their jobs. Since they’re removed from the actual building of the product, this poses a challenge.

Even the people who are hands-on in building the product lose grip of product changes as they unfold. For example, a product manager might know that a change was “ready to go live” in a staging environment, but was it actually deployed yet? They hate having to constantly ask an engineer “Is it live?”, but they have little other choice.

Today, there are two general solutions to solving the problem of product change awareness:

• Machine-driven solutions. Automated notifications of product changes based on code versioning or project management tools.
• Human-driven solutions. Hand-crafted changelogs or communications to stakeholders.

Both types of solutions have downsides.

Automated product change notifications are usually too noisy to be useful. People end up muting the Slack channels with the firehose of GitHub merges or Jira status changes.

Human-driven product updates, conversely, are labor-intensive and fallible. Given the time it takes to write communications, many changes go uncommunicated.

The DoubleLoop Solution

At DoubleLoop, we’ve built a “best of both worlds” solution that hybridizes the machine- and human-driven solutions for communicating product changes. We enable teams to efficiently separate the signal from the noise of automated notifications.

DoubeLoop integrates with code versioning tools (GitHub) and project management tools (Jira or Clubhouse) to generate a source-of-truth of product development activity.

The power is in how we classify events from these systems. In many cases, the classifications come directly from the source tools. For example, we classify a GitHub pull request by repository, branch, and creator. We classify Jira issues by project, epic, type, or requester, etc..

Classifications enable teams to configure filtered communication streams tied to email or Slack. You can configure a “view” in DoubleLoop that sends a message to Slack every time a pull request is merged to master or when a feature is completed for a certain project.

However, DoubleLoop goes beyond importing classifications directly from source tools. To help teams separate the signal from the noise, we apply heuristics to automatically classify the importance of events.

Let’s look at an example of a DoubleLoop heuristic. Events in DoubleLoop can be set to have minor, medium, or major importance. By default, pull requests and code commits are set to have “medium” importance. However, we added a heuristic that automatically downgrades the importance of code merges created by bots (e.g., Dependabot). These commits are things like library upgrades that are minimally interesting to someone who wants to know how the product changed. Teams can filter “minor” events out of their communications streams to reduce noise.

DoubleLoop also has the ability to parse information from commit messages based on naming conventions. For example, if a commit contains the string “feat,” we could automatically classify the change as a feature. If it contains “fix,” we could classify it as a bug fix. These classifications can be used in defining filtered views. For example, someone might want to see only a stream of feature launches without the noise of bug fixes and engineering chores.

Seeking design partners

What other heuristics or parsing rules should we add to DoubleLoop to identify the most relevant product changes?

We’re looking for teams to collaborate with in evolving this part of the DoubleLoop system. The ideal design partner

• ships code at high-velocity,
• feels a strong need to separate the signal from the noise
• is able to install our GitHub, Jira, or Clubhouse integration, and
• is willing to provide us feedback on a regular basis, ideally through a cross-company Slack channel.

Interested in participating? Sign up → here.