Dive into DevSecOps

DevSecOps is the combination of development, security, and operations practices to reduce the time needed to deliver changes in the system—ensuring high quality with integrated security practices.

Goals:

  • Improve deployment frequency;
  • Faster time to market;
  • Lower failure rate of new releases;
  • Shorter time to fix something;
  • Faster time to recovery (if a new release crashes the current system).

At Mercedes-Benz.io we want to have this in our DNA but we can not only continuously improve on what we do but also need to evolve as everything around this concept also evolves. Mindset, tools, and strategies to improve the teams structure are changing and we need to keep up, take advantages of new findings in the technology community, and, whenever possible, also lead the way.

Organizational challenges for effective DevSecOps 

When we think of DevSecOps, we imagine the idyllic scenario with developers using a stable and secure platform for fast delivery of new product features. But what is actually involved in achieving this ideal? How close are we to reaching this goal? 

From my experience over the years, I have seen some awesome work being developed in this field and experienced what is happening in our and other companies. Thus, I came to the conclusion that DevSecOps success involves 3 main factors: the right mindset, technology and an appropriate organizational structure. 

These factors form a base triangle where several other aspects on DevSecOps should be applied and/or followed, so that we can continuously improve our products and the way that we deliver them to their users.  
 

The vision of having a triangle made with Technology, Mindset and Organization Structure on the DevSecOps picture 

With this triangle in mind, we should guarantee the context where we build our products and chase the “delivery flow” – the place where we are in peace with the delivery process and happy with what we deliver. 

When technology is what we use to build technology 

This point mainly reflects the need of technology to provide solutions for faster and recurring delivery. Many say that DevOps is not about the tools but while I agree it is also true that we need tools—provided by someone else or made by us— to help us on the delivery flow and optimize the delivery process.  

In terms of process, I would add that we should seek to use the tools that can help us optimize the process as early as possible in the team’s delivery process. 

We can cluster the technology in two major clusters: 

  1. Tools that we use or build to improve the delivery flow 
  1. Flexible platform Infrastructure 

Tools to improve delivery flow  

The delivery process has several steps, and we can introduce tools to improve the product and the way that it is built.  

Here some examples of technology usage to improve the product and its delivery: 

  • Tools that help on the Coding/Development phase, for example, Source Code Management Tools for managing the code and the team(s) collaboration model, or tools for Code Quality evaluation over best practices as early as possible.
  • Tests for different product functionalities and business rules should be applied with the help of tools that help on its automation. 
  • We also have tools available to help on the Code Review and Pair Programming, to improve the team collaboration and the result of that work. 
  • Automated pipelines for Continuous Integration and Continuous Delivery/Deployment with automatic evaluation of possible vulnerabilities in terms of Security and Compliance
  • LoggingMonitoring and Alerting are supported by several tools that can connect all information allowing peaceful sleeping and faster troubleshooting. 
  • For Performance we can use tools to improve the overall experience of our users and also to evaluate what can be decreasing performance and be improved. 

The above list has some of the areas and tools where automation is the key for delivery improvement. Every challenge we have can be an opportunity for improvement and to add more items to this list. 

There’s a lot of tools out there that we can use and to choose the best one for what you need is also a challenge by itself.  It’s important to understand why you need it, if it’s better to use a tool already available or if you need to build one that fits best to what we need.   

Additionally, to the value the tool can give, we should also evaluate the effort for integration and maintenance, keeping in mind that we should control the amount of cognitive load that we add to our product teams. 

Flexible infrastructure 

Also related with the technology for DevSecOps, we have the infrastructure solutions to provide a flexible and—we hope—stable base for our products. This infrastructure can be made available as a Service (IaaS), extended to a Containers as a Service (CaaS or KaaS if the container solution is based on Kubernetes) or a Platform as a Service (PaaS) like the one we have with PCF. Each one adds a layer on the previous providing easier to use features but, most of the times, removing some flexibility in the usage of that infrastructure. 

Over this infrastructure base, we can (and should) provide software services—for example, based on tools—and present them as a SaaS (Software as a Service) to be used by the product teams, if possible, in a self-service manner. 

The evaluation and choice of possible tools should be done with everyone involved in the product development, from Architecture to product and platform teams. This is one of the things we are actively doing in several Proof of Value/Proof of Concept initiatives at Mercedes-Benz.io. 

A tool is good if we know how to use it… and if it fulfills a purpose 

When we started to talk about DevOps, around a decade ago, I remember this image of development connected with operations with no barriers. An infinite loop with people working together but also a handout phase pictured with 2 colours: one for Development and the other for Operations. 

Dev and Ops connected in an infinite loop 

Around this infinite loop we can have all the tools to help on each stage and the idea there was to connect Devs and Ops in the most streamlined possible manner.  

For me, that picture is outdated, and the following image is what I can picture nowadays. 

DevOps infinite loop 

On the above picture we have a team the owns the product end2end, in all phases, supported by a platform, enabling the team to focus on the business and on how to improve the product. 

I see this as an updated mindset over DevOps and a key factor for companies to improve the way their products are delivered to their users. No handovers, no predefined/discussed responsibilities, everyone working together to ensure the best products. A platform made by platform teams focused on providing the best platform services, developing a platform as a product. Products with product teams focused on delivering value to their users. 

This mindset is totally aligned with the “you build it, you run it” principle and we can partly extrapolate the “eat your own dog food” practice, as each team should guarantee what is delivered to the product users and get direct feedback when something needs improvement.

From this DevOps picture to DevSecOps, we “just” need to add the Security we need to have around what we deliver, with the intrinsic mindset and some tools to help on that. 

We need a structure to enable the mindset and good usage of technology 

In terms of structure, I have been following the important work that Matthew Skelton and Manuel Pais are doing, around team topologies, in the past few years. In my point of view, they are setting the standards for DevOps team structures, bringing some definitions, a common language and some practices that, when correctly applied, can help organizations improve the way their products are built and delivered.  

Next, I’ll point out what I think should be addressed when you want a good structure for your DevOps teams.  

Design the organization following the architecture you want for your product​ 

If we know that the produced software mimics the team’s communication structure, why don’t we just use the wanted architecture to structure our teams? 

The idea here is to apply the so-called Reverse Conway ​Maneuver​​, starting by the definition of the wanted software architecture for the product(s) and then, defining the teams to produce the software according to that architecture. 

Easier said than done, I know. Not starting from scratch means that a usually non-pacific transformation is needed. But expecting that your product will follow an Architecture guideline that your team structure doesn’t follow—we already know that will not happen.   

Manage the complexity of each team’s domains to decrease cognitive load! ​ 

We should also look at the complexity that each team needs to handle and control.  
As we empirically know, our capacities are always limited and we should maximise the individual capacity to process new information, resolve problems, and learn. 

In a simplistic way, we can look at our working memory when we need to do a task. The working memory is loaded with 3 types of information:

  1. the intrinsic load with all the information we know and need to resolve the task,
  2. the extraneous load with all the information that exists in the context and is unnecessary or unproductive to do the task but is there – like some existent processes and bureaucracy – and,
  3. the germane load that we use to understand and find a solution to solve the task.  
Textfeld

If our context is too complex, our available working memory is already filled with information leaving no space for solving the task in an ideal way. If the team has a lot of processes and bureaucracy to deal with, that information is fulfilling the available working memory, and again, removing space for solving the task in the best way. 

That is why we should simplify the complexity around one team and reduce the unproductive information each member has to deal with, in order to leave space for the germane load, to better solve each issue and deliver innovative solutions. 

Size matters! 

Here I’m talking about the ideal group size to maximize teamwork and results.

Dunbar’s number show us the limits of the number of people we can have, socially, together and have closer connections. We should extrapolate that to the groups in our company structure—underlining the type of relations we have at each organizational groups. 

Groups size and the kinds of relations 

So, yes, size matters and we should use this knowledge to set the best team size on our structure. 

Identify the topology of your teams and their modes of interaction​ 

Team Topologies give us the fundamental topologies that we can use to identify our teams and the interaction modes to define how they interact with each other’s. 

Here are the team topologies we should use to define our teams: 

  • Stream-aligned teams are the full-stack, cross-functional, product teams. 
  • Enabling teams are the ones that help the stream-aligned teams to adopt or modify software, as part of a transition or learning period. 
  • Complicated subsystem teams, previously defined as component teams, should be used only when needed and by a period of time, to manage a complex subsystem that the stream-aligned teams can’t handle by themselves. 
  • Platform teams are the ones supporting stream-aligned teams on their product delivery.  

If used well, these are the only topologies we should need to define the type of teams we have on our DevOps structure. 

Besides the topology of each team, we should also define the interaction modes each team have with the other teams. Here are the core interaction modes we should use: 

  • Collaboration for teams working together for a period of time to discover new things like APIs, practices, technologies, etc. 
  • X-as-a-Service when one team provides, and one team consumes something “as a Service”. 
  • Facilitation when one team helps and mentors another team. 

These definitions should help us identify the teams we have and the teams we need, improving, at the same time, the communication by having a clear and simple definition to connect the organization. 

A practical aspect of these definitions passes by creating small artifacts like the “Thinnest Viable Platform” for platform teams or “Team API” for all teams, to share this throughout the organization, improving the visibility and understanding of its structure. Follow the links to see the relevant information that we should have on those artifacts. 

Use a platform to reduce the cognitive load from product teams​ 

We should define and use a platform to build the common pieces that enable the product delivery, removing some of the complexity, and the inherent cognitive load from the stream aligned teams. 

Common platform used by stream-aligned teams 

This is the so-called platform model and, as stated on the “2020 State of DevOps Report”, when “(…) done right, it simply works, resulting in faster, more efficient delivery of high-quality software that meets an organisation’s business needs — and at scale. One platform to help product teams”. 

This platform model relies on having a platform built as an internal product that has the stream-aligned teams as customers. On the platform we need cross-functional teams to build it and follow the organizational structure concerns mentioned above.  

As we are aiming for the best way to help the delivery process, the platform should, as much as possible, provide self-service solutions to the stream-aligned teams to optimize integration. We can aim for a NoOps concept, providing the delivery environment fully automated and abstracted from the underlying infrastructure. 

What do we need to improve our delivery? 

The triangle as a base for all best practices 

Wrapping it up, we have the triangle formed by TechnologyMindset and Organization Structure, as the base to apply good practices, patterns, frameworks, principles, values, architecture, processes… Everything that we can do to improve our products and the way we deliver them, but requires a good base to be able to work as best as possible. 

As the old sailor saying goes: “there’s no good wind for those who don’t know where they’re going”. Nobody can do it alone and it’s nothing one can command, it’s something for which we are all accountable for. We should use this triangle to guide and direct our efforts, create a common language, and aim for the same place.  

We are not alone: Insights into our Company Exchange Concept

To be Agile is to always question what we have to improve. We can inspect what we practice within our company and try to find ways to improve. However, if we look to what’s happening externally, we can learn new ways of doing things and, most importantly, learn from mistakes of others.

“Learn from the mistakes of others. You can’t live long enough to make them all yourself.”

― Eleanor Roosevelt

This should be a good starting point to make us want to have a way to interact with other companies that share the Agile way of working, to know what they do and understand how they do it.

The Why

Creating a #network of people and interacting with them over a specific topic can be very interesting. We can make people have some cool time to think and discuss about a lot of topics, but why is it so important to do these exchanges with other companies? What can we gain with that? This is the main question we should answer in order to make the efforts and time spent to be worthwhile. 

The other day I heard that what we evolved in the last 2 years to be comparable to what another well-known tech company achieved over the period of 10 years. Growing this fast can be really overwhelming and may cause real damage to us as an organisation, depending in how we deal with our growing pains. 

If we want to be more effective in what we do, striving to build the best digital products on the automotive market, to get in touch with other companies to #share and #learn from them is extremely important. Unless we think that we know it all and can prevent everything. But I personally don’t think so.

Let’s get some valuable knowledge

To get more #knowledge is the immediate thing that comes to my mind when I think on the reasons to participate in exchange initiatives. But what can we learn from other companies?

In terms of #Agility, there are a lot of topics we can gather some insights on from others, understanding their (best) #practices and deriving ones we could apply in our environment. 

This knowledge transfer can be quite different depending on the companies with which we interact. To better understand that, we can look at the following clusters that are relevant in our case:

  1. First and maybe the most important cluster is the companies from the Daimler/Mercedes-Benz universe. Collaboration within this universe will provide us more insights about what is happening in the Daimler world. At the same time, this can make us more relevant by sharing what we do and inspiring other departments and subsidiaries to implement or enhance the Agile way of working. In terms of #compliance, this relationship between companies of the same group is much simpler in what we can #exchange, making this one of our focus.
  2. Another cluster we want to collaborate with includes the large companies that are in the process of Agile transformation. We are especially interested in experiences on the topic of #scaling with Agility. We have grown a lot over a short period of time and are facing the challenges of that process. Sharing with others and learning about their pains and mitigation strategies can help us prevent some problems by acting with conscience on those possible outcomes.
  3. We can also expect that we can learn a lot from the companies of our size assuming they experience similar issues living Agile. The learnings we can get from each other’s experiences may be of an enormous importance because of these similarities – similar problems could be solved by similar solutions. I know, it depends, but it might work or be just #valuable input to work with.
  4. In this last group I put the companies that we must be really careful to share our experience with, our #competitors. To understand with whom we can interact and what kind of information is allowed to be shared from the compliance perspective is one of our main concerns.

Beyond the knowledge

Networking is also an important aspect of this kind of initiative. By working #together in work sessions we can dig into new and old questions and generate new knowledge that comes out of the collective intelligence. This exchange events are not only about knowledge transfer but also a way to generate insights and solutions, encouraging the out-of-the-box thinking.

To get inspiration from new ideas, new perspectives and learning new trends can and must be in the list of objectives. Fresh ideas are always welcome, we really want to use this exchange activities to link the outer world, to get new and valuable information and also, I believe, to enforce our #ideas and ways of working.

As a side effect of all points mentioned above, when organising and participating in knowledge exchange events, we want to raise #attention to what we do and how we want to do it, especially that of our direct stakeholders. Being prominent on the market by being a live actor on organisational topics can also help us consolidate our position in the purpose of making Mercedes-Benz the Digital Champion of the Automotive Industry. On one hand, we are doing it by building products in the way that customers love, and we are proud of. On the other hand, by being pioneers on the organisational level and reinforcing the Agile way of working across the Mercedes-Benz digital universe.

Yes, we are trying to show off. But at the same time, we must be humble and know that we cannot know it all and can always learn from other experiences.

Exchange is a two-way communication path

Obviously, with the knowledge exchange initiative, we seek to get valuable information, but we also want to #give something back. Give-and-Receive concept ensures that we can get interested participants in such activities and brings value to each one of us.

For example, we already received a huge interest in #Holacracy, the organisational framework which is used in Mercedes-Benz.io, and how it connects to an Agile way of doing things. We should not try to impose our ways or make others follow the same path, but we can share our experience of being organised as we are, the challenges and how we are dealing with them. 

To share what we do can be very valuable for others but is also a good #opportunity to show how we organise ourselves, our values and how we foster them as well as how we want to be as an Agile company. While providing some food for thought to others, we are positioning ourselves as a valuable player on the digital companies’ market.

The How after the Why

We work in a holacratic organisation, we have skill-based circles that contains the roles that are needed within the organisation. Once the need to exchange the knowledge with others was identified, we created the Company Exchange Coordinator role in the Agility circle and settled its purpose and accountabilities.

The Agility circle includes two important Scrum roles: Scrum Master and Product Owner and is divided respectively in two sub-circles for “The What” (for POs) and the “The How” (SMs). These are the home base for all the Product Owners and Scrum Masters in the company. 

If you want to know more about Holacracy, check this blogpost from Susanne Kopp.

The Company Exchange Coordinator role for Agility is the role that I fill, along with Karina Ivanenko, and it has the #purpose of hardening and establishing collaboration between Mercedes-Benz.io and other companies for exchange of knowledge and ways of working. We work together in this coordination role, we combine the needs and interests of our two roles, me as a Scrum Master and Karina as a Product Owner, in two different locations, Portugal and Germany, so we can expand the spectrum of possibilities across locations.

What we must do is defined in our role #accountabilities – another important part of a holacratic organisation. In our case, the main point, in my opinion, is to develop a way to exchange information and experiences with other companies in an organised and compliant way.

Since this is a coordination role, the idea is not to do everything but to be a point of contact and guidance for actions related to exchanges with other companies regarding Agility. 

It is important to mention that we are not the only people in the company who are driving the topic of knowledge exchange. Apart from the Company Exchange Coordinator role, there is one dedicated to the collaboration with Universities – University Exchange Coordinator. These both roles exist not only in the Agility circle but also in the circle for Concept, Usability & Design (CUD).

The four dimensions of our company exchange: Mercedes-Benz.io’s Agility and CUD circle, collaborating with companies and universities

We are working together in these 4 dimensions to foster exchanges with 2 different kinds of entities (companies and universities), for now in 2 different areas (Agility and CUD) but in the #future it can be done in other subjects/circles.

Current status

In this group of 4 roles of exchange coordination, we have been working on the definition of the goals and measures to reach them. We have identified some goals that are relevant for all dimensions as well as specific goals that must be handled and fostered in each individual role.

Company Exchange was kicked off by organising our first meet-up in our offices in Stuttgart, in a joint venture with Detecon International GmbH, where we were able to #evaluate the goals that we want to achieve with this initiative. We were happy to see that it was valuable for most of the participants and that such format makes sense in regards to the set goals.

“It is beyond a doubt that all our knowledge begins with experience.”

—  Immanuel Kant

We received companies like Deutsche Telekom, Deutsche Bahn, Bosch or Haufe-Umantis and everyone got a chance to speak out their experiences regarding Agility and the challenges of scaling product development to multiple teams (agile@scale). Different realities, structures, states, sizes, methods, frameworks… a lot of information was shared in presentations with some space to get answers to the raised questions. 

But it was not only a time for company presentations, we gathered the knowledge and experience in the room and set a challenge for a small workshop. It was a great moment for everyone to interact, discuss ideas and reach #common ideas and views over agility topics from people that share a #mindset but have different realities.

This experience gave us a clearer view of what we can achieve with these exchange initiatives, raised a lot of ideas and some doubts for us to work with.

Next steps

We are still working on this exchange concept; we want to make all the goals #transparent to everyone and to define ways to measure the result of the time and effort involved in it.

We also want to establish contact with further companies that may be interested in this kind of exchange. If you are interested in this topic, know some cool companies that may be open to this exchange and may also want to be part of the related initiatives, please contact us and let us know!

And if there’s a thing that I can recommend is, while going fast, look out-of-the-circle if you want to go far.