Building a DataOps Team
Data companies are coming to a consensus on what we mean by DataOps. In short, we seek to bring the benefits of a DevOps culture to organizations with a core competency in data analytics. I’ll clarify this in a moment, but I want to draw your attention to the teams that engage in this work.
If you have a modern software engineering team, you are familiar with the concept of continuous build, continuous integration, and continuous deployment. The latter two give us the acronym of CI/CD, but they require an automated build with automated testing so that we know whether a build succeeds or fails.
Break the Build?
In the context of an automated build & test, you can understand the phrase “oops, we broke the build”. This is how engineers refer to the moment when a rapid feedback loop tells them that today’s change failed the tests in the automated build. There’s a lot of significance baked into that phrase, but I want you to notice this: The immediate awareness of the impact of change is essential to the culture of DevOps.
Break the Team?
Here’s the new thought.
Can we create a culture where our teams are aware of when their actions break the team itself? Can we create a rapid feedback loop that tells you whether a change in human interactions was successfully adopted by the team? Can we identify when a change in behavior breaks the team? That’s what I mean by “building the team” — a continuous check to identify when the team is healthy and when it is not.
And my conclusion is that yes, we can. We can measure the health of a data engineering team as effectively as we measure the health of data applications, data pipelines or data content. We can know when we help or harm the team.
Monitoring and measuring a team’s health draws on different tools. You may need soft skills training or a new language of success, but the tools already exist. I will describe a few of them.
DevOps and DataOps
First, let me provide an oversimplification of DevOps and DataOps. You’ll need this as context. Yes, this is oversimplified. Yes, it’s harder than it looks. But it helps to see the bigger picture of what we mean by these “culture-oriented” terms.
DevOps is the culture of collaboration across application development and infrastructure operations, with shared end-to-end responsibility, a bias to automate, and rapid iterations that produce continuous improvement. It is the application of lean manufacturing principles to application development, deployment, delivery and operations.
DataOps is the application of the DevOps culture to an organization whose deliverables are data and analytics rather than software. It is a culture that focuses on velocity, quality, predictability and scalability as they apply to the data and the analytics. These are achieved through the behaviors of communication, collaboration, orchestration, measurement while ensuring security, access and ease of use.
The first thing that I want you to notice about my definitions, above, is that these describe an organization’s culture and the behavioral results. They do not dictate the technology or the tools. They focus on people and process. If you’re familiar with technology management principles that emphasize PPT (People, Process, Technology), you’ll note the emphasis on the first two.
Healthy Individuals
To measure a team’s health, let’s first define it. A healthy team is a well-functioning (small) society of healthy individuals.
I suggest that healthy individuals embody autonomy, a sense that they belong where they are, and competence that grows over time. These traits come from the work of Richard Ryan and Edward Deci and many others who work on Self-Determination Theory. I won’t go any deeper than this for now, but let me repeat the principles. These may be easier to remember if you notice the A,B,C of Autonomy, Belonging, and Competence:
- Individuals must have autonomy.
- Individuals must feel that they belong.
- Individuals must have confidence that they are growing in their professional competence.
Healthy Society
A society of humans, even if it is a small one such as a data engineering team, is complex. Note my carefully chosen term. To be complex is to have patterns of behavior, often surprising and emergent behavior, that are unknown and impossible to decompose into simpler components or behaviors.
Fortunately, there is a large body of literature on healthy societies. A recent book that I found helpful is Blueprint: The Evolutionary Origins of a Good Society, by Nicholas Christakis. The author summarizes research to show universal human societal principles that include, to name a few: cooperation, cohesion, punishment, justice, a tendency to form connections, maintenance of diversity, and the preservation of culture. It is from his book that I borrow the following definition of culture:
Culture is knowledge that is transmitted to individuals and across time, that can be taught and learned, and that is distinctive to groups.
— Nicholas Christakis
I must oversimplify again. Learning to recognize healthy teams and the behaviors that work for or against them can take a lifetime of effort. We will focus on the simplest and most fundamental.
- Define the group.
- Collect the knowledge that is distinct to this group.
- Transmit that knowledge between individuals and across time.
To sum those up in a single sentence: Create and sustain a team culture.
Feedback and Measurement
And now, to bring it all together, only two concepts remain. Let’s return to the original thought of how to recognize when behavioral changes will “break the team” (similar to breaking the build for an automated application build). The two remaining concepts are feedback and measurement.
If you have been exposed to self-governing teams, it will not be hard to guess where I am going to go next. For a team to be healthy, it must be able to self-assess its own health. This blog post is not an article about management — it is an article about healthy teams. All that is needed is for the team to own its own health. I suggest the following measures, to start with. The team will take ownership of them and can improve them as they determine together what works and what does not.
In each iteration’s retrospective, take a few moments to discuss team health. This is how you measure — it’s all self-assessment.
- Do we still feel that we are a team? Or has something happened that is pulling us apart? Have we split into more than one team?[Group identity]
- Do each of us feel that we belong on this team? Or have we failed to state the obvious when one member is not making it? Let’s put it out in the open and talk about it. [Belonging]
- Does our team know why we’re here? Has our charter changed so quickly that we are no longer confident of this? [Distinctive identity]
- As individuals, do we each feel that we were able to choose how to contribute? If one of us is still too green, how can we help them? [Autonomy]
- Do we feel that everyone contributed effectively? If not, what were the blockers? Do we need better tools or additional training? [Competence]
- Do we each feel that we are growing professionally in this role? Yes, you’re allowed to talk about this with your peers, not only with your boss. [Growth in our competence]
- What knowledge did we gain during this iteration? How can we preserve that knowledge? Did we pair effectively so that it wasn’t only one person who learned it? [Knowledge transmission and preservation]
Self-Correcting Team Health
The immediate awareness of the impact of change is essential to a healthy team culture. If we include a self-assessment of team health in each iteration’s retrospective, we provide the immediate feedback that is necessary to notice when changes in behavior are helping or harming the team.
If you break the automated build, every engineer notices. If you adopt the mindset described above, everyone will also notice when the team breaks. And you’ll self-assess quickly enough to self-correct.
