The 10 Practices of Healthy Engineering Teams – Part 3

Andrew Hao ·

Credit: #WOCInTech Chat

In Part 1 and Part 2 of this series, we introduced the engineering team at SuperStartupCorp. They had built a reputation for delivering quality software, fostering a culture of trust, and developing healthy learning practices. Read up on the first two parts, then join us to discover the final secrets to their success.

7. They play well with the product team

Effective teams have healthy dialogue, mutual trust, and co-ownership of the product development process with product owners and designers.

Having a problem-solving attitude with product owners

Ryan, a former product manager with Carbon Five, appreciates it when developers respond to product owner questions with helpful alternatives instead of stonewalling. He says:

Some developers can come across as curmudgeons to [their non-technical counterparts], seeming to say “no” when what they may actually mean is “what you’re asking is super hard and seems like it isn’t worth it.”

Instead, he says, developers can propose alternatives and inform their product owners about the relative size of the tasks:

Developers can be great partners to product owners by being able to use their expertise to inform the product owner how easy, hard, or risky something might be. Then they can provide alternative approaches that might help the PO achieve their goals in a better way.

Engaging in the design process

Elias, a Carbon Five product designer, appreciates developers who also engage in the design process:

It’s super amazing when a developer is willing to take a chance and think outside the box, [saying] “there’s another way to do this”. It’s also great when a developer pokes a hole in your design or product idea, but doesn’t just leave it at that, [also] offering another tried-and-true solution from their experience.

Oftentimes, a product team’s view of the engineering team can be one where the engineering team seems to always be shooting down their ideas, or wave away their designs with a scoff. Elias enjoys working with engineers who probe into designs with an attitude of empathy and curiosity:

From a design perspective, it has helped immensely to know that when a developer is questioning design questions it’s not to be deflator, but only so that they understand the story/feature better. Framing these thoughts in a constructive matter is very helpful and helps make the product better.

Effective engineers are able to be problem solvers to their product colleagues, providing alternatives and technical insights where needed.

8. Develop just enough architecture process

As SuperStartupCorp’s engineering team grew, they began to see some architectural issues arise from the rapid growth:

  1. Teams working independently of each other found themselves duplicating domain code often because each team had a different view of the problem space. This led to different names for classes, methods and variables, or fragmented, scattered domain logic.
  2. Multiple teams, by default, kept building features into the company’s monolithic codebase, without pausing to evaluate whether it was time to segregate features into separate services. This caused the app to grow larger, be harder to test and deploy, and scale inefficiently.
  3. The scaling demands of the business began outpacing the technical capabilities of the architecture, unable to meet the needs of growing customers.

Engineers first decided that they would help everyone have a consistent view of the system by printing out a giant diagram of the business domain. This was developed as a UML diagram and kept in source control, but could have been as simple as a whiteboard sketch. They printed it out and pasted it in plain view on the wall of the engineering floor.

Secondly, they decided to keep a glossary (also known as a Ubiquitous Language) of their business domain terms codified in an internal company wiki. This glossary contained a list of domain terms and definitions that they would always use when discussing these concepts – in their verbal discussions, in emails, chats, and in the codebase.

Third, they developed a system of architecture reviews. Any time a team started down the road of implementing a large feature, they floated a small one-page design document across to their peers on other teams. It was meant to be small, with a lightweight description of the change in a paragraph, along with a sketch or two if visuals were helpful. Their peers could then have adequate context and time to give feedback – or simply have more insight to upcoming changes to the system.

9. Keep performance a first-class citizen

As years of development passed, the team noticed that their application performance slowly deteriorated. Occasional performance fixes would help, but keeping the application performant always seemed like a distant ideal. However, the engineering team decided to implement a few practices that would ensure that app performance would no longer be ignored.

Engineers made sure that performance issues were monitored and actively addressed at all points in the development cycle. Developers were educated on performance topics like N+1 queries, HTTP caching, and application best practices. Potential issues like these were often discussed in internal code reviews.

Additionally, the team had developed load tests for their web systems, which could be periodically run against their internal integration environments, or even on production. These tests were implemented and tested with third party services like flood.io, or with frameworks and tools like JMeter and Minigun.

Teams have codified a app performance threshold that their systems behave at. For example, the team agreed to keep the 95th percentile server response time at 100 ms for the Authentication API. If the system began exhibiting performance over that threshold, an investigation task would be inserted into the backlog to bring performance back within acceptable levels.

10. Manage the noise of their production systems

The team had a problem: every day, they were inundated with a flood of app notifications from their system logs and app error reporters that made it difficult to separate the truly critical issues from the merely dysfunctional. Developers had even begun to write email inbox filters to completely ignore these messages!

Upon realizing this dysfunction, they made it a priority to triage bugs immediately as they were reported. Through reports from services like New Relic, Bugsnag, and Airbrake, immediate performance regressions or app errors are investigated, triaged and prioritized with fixes as soon as possible. (Unsurprisingly, the best way to remove error messages from production systems is to quickly fix the underlying bug.)

However, engineers still found themselves flooded with notifications on email and on Slack – everything from performance metric reports, to minor warnings about system statuses, to ordinary messages about deployments, commits, and pull requests.

They devoted a sprint to silencing noisy or faulty error reporters. If a logger in the code was writing too much to the logs, its log level was increased. Certain classes of errors were added to a whitelist of ignored errors for each error reporting service. Certain metrics that were “pushed” to developer inboxes were moved to in-house dashboards and screens. App services were reconfigured to report only the most important information over email or Slack channels.

In a radical step, logging and error reporting were turned off for non-critical parts of the codebase, and left on for the truly critical paths. Now, only truly important events make their way through to developers, who can clearly understand how to triage these events.

Conclusion

Great engineering teams take time to build: introducing solid coding practices, keeping due diligence to architecture, and developing curious, empathetic people takes time. The good news is that if your team is like SuperStartupCorp’s, it may begin far from perfect, but it merely needs some direction and time to mature.

With these 10 tips, we hope that we’ve helped you guide your teams to a healthy state. Don’t forget to read Part 1 and Part 2. And of course, you can hire us to help you build some excellent teams and products.

Do you have any things to add? What’s worked well for your teams? Let us know in the comments!

Photo Credit: #WOCInTech Chat

Andrew Hao
Andrew Hao

Andrew is a design-minded developer who loves making applications that matter.