Devopsdays Ghent 2019: day one

This year I was lucky enough to attend devopsdays for a second time. This time the conference was held in the beautiful Ghent, Belgium.

I’ve been to devopsdays Amsterdam a couple of times, but I never went to one in a different city. Because ten years ago the first devopsdays was organized in Ghent, I thought this was a great moment to visit the “birth place” of devopsdays.

These are the notes I made during/after the talks.

Enterprise DevOps 10 years on: What went wrong? and what went right? — Nigel Kersten

Some enterprises are cargo culting DevOps. They have shiny, polished processes and tools which are not in place out of experience, but out of an idea that that is how you are supposed to do DevOps.

Processes often are like scar tissue—adhesions formed after trauma. That is: something went wrong in the past and a process was created to prevent that thing from going wrong again. Unfortunately these processes are not always reviewed to see if they still match a world that has changed since then.

Most companies are not changing enough. Teams, and perhaps departments, can be successful in adopting DevOps, but often it does not scale up to the whole company.

So what went wrong? When you look at the CALMS framework (which stands for culture, automation, lean, measurement and sharing), culture and sharing are hard. This leaves you with automation, lean and measurement. But:

DevOps should be more than optimizing and automating builds!

The things we build need to be more accessible and more easily shareable.

Why we’re bad at sharing DevOps — Joshua Zimmerman

What does DevOps mean? Often the picture of several blindfolded people examining (parts of) an elephant is shown to demonstrate that nobody is right.

But this approach misses the point. Language changes all the time. Words might loose some of its original meaning and start meaning something else. “Awful” originally was used to describe something worthy of respect or fear. Nowadays it is used to make clear something is extremely bad.

Arguing about the definition of DevOps does not help us reach consensus and also does not learn others what we want to convey about DevOps.

Teaching

Education can be used as a tool of oppression—forcing your ideas onto others. One way of doing this is by being selective in what we teach and what we omit. But how we teach can also make a difference.

Traditional approach in tech uses one-way broadcasts. The teacher fills up the student with knowledge. But people are not empty vessels, we already have experience and knowledge.

Instead of teaching being one-way communication, you can also explore solutions together with the student (problem-posing education). You can work together on a problem and find truth. Students can contribute to the knowledge.

Ways to improve sharing concepts like DevOps:

• Focus on concepts over language. Set a context and move on.
• Pose more problems and offer fewer solutions. Show the problems more clearly instead of focussing on the solutions.
• Create more spaces for collaborative learning.

Reading tip: Pedagogy of the Oppressed by Paulo Freire.

Everything I need to know about DevOps I learned in The Marines — Ken Mugrage

Disclaimer: there are things in Marine boot camp that are counter productive when applied at DevOps, like instructors yelling at recruits.

Words matter a lot. In the marines they have very specific words for things. For example: a bed is a rack, a hat is a cover, a toilet is the head. In organizations it’s similar: a “unit test” means something specific. By having a shared, well defined vocabulary communication is more effective.

Boot camp is full of creating habits. For example: recruits have to sit cross legged on a concrete floor in a “classroom.” Later on in the course it becomes clear that this position is a stable position to fire a gun. By the time the recruits get to actually work with a gun, they are already used to the position.

Having muscle memory, ingraining habits, means the quality goes up.

A different level of risk means a different level of testing. This does not have to imply that deployments take longer. While the developers are running their functional tests, you can also already run additional tests that are required to promote that code to the next stage. So by the time the developers know that the functional tests pass, we also know whether there are security or performance issues that need to be addressed.

The military is not all about command and control. High level objectives are set by the highest in command and these objectives get more detailed in lower levels. In business: people share “corporate goals,” but the product team decides what the best implementation is.

Takeaways:

• Words have meaning
• Train like you play, play like you train
• Diversity and inclusion makes us stronger

Supporting devops with technology procurement — Ramon van Alteren

Procurement is about obtaining stuff. In a modern context this means answering question like “which cloud vendor?” and “why that cloud vendor?”

Spotify was using their own data centers with compute, network and storage. Because provisioning new hardware takes time (e.g. weeks to order a new server), you start managing buffers and flow to be able to fulfil a demand.

Moving to the cloud puts you into a transactional mode, you pay directly for what you use. Moving to the cloud in a way where every team can spin up their own infrastructure also means there is no longer a single place of costs. However, cost is a factor in how you design and run systems. By decentralizing this place of costs, each team has to consider them.

And don’t forget that rules like writing off infrastructure does not work. In the cloud, compute is not “free” anymore after a number of years. When a project has a budget for infrastructure, you have to take into account that in a cloud setting you have to keep paying once the work on the project is finished and it goes to maintenance mode (and the costs shifts from engineering to an operations department).

Lessons:

1. Create a safety net. Predicting costs is not easy. Allow people to experiment, but make it safe for them. (E.g. alert on spikes in costs.)
2. Look at global cost optimization before local. E.g. look at using reserved instances instead of having a team of developers optimize a small thing while they also could be adding value to the company.
3. Haven an explicit frame of reference. Make it clear if cost are high or not.
4. Most importantly: cost reduction should not be a goal! Otherwise you end up with a cheap, very specific service; you’d rather focus on value for users.

You can’t buy DevOps — Julie Gunderson

You cannot buy it, but people will try to sell it to you.

You cannot change culture with a credit card.

Cultural changes rarely work top-down. Instead start small and prove its usefulness.

DevOps is not things like:

• Moving to the cloud
• Forming a “DevOps team”
• Getting rid of the Ops team
• A prescriptive checklist

DevOps is about culture, people over processes. Research by Google about what makes a successful team, shows that psychological safety fosters a culture of empowerment. Having psychological safety means you can embrace failure and learn from your mistakes.

We should stop talking about “a root cause,” but instead discuss “contributing factors.” With our complex systems, there often no longer is a single root cause. There are always contributing factors that allowed e.g. a person to push the button that took down the whole company.

Some practices you can adopt:

• Configuration management
• Automation
• Use the right tools (the actual users of the tools should be able to decide which tools to use)
• Working in small branches instead of massive integration (this makes it easier to rollback if needed)
• Testing

You can start small and prove success. This can affect the entire organization.

What MMA taught me about working in tech — Daniel Maher

Mixed martial arts (MMA) is about mixing techniques.

Fundamentals win fights: there are some basic things (tools) you have to learn. Learn about the individual tools and how to put them together. But don’t be dogmatic about fundamentals; they are just tools. What matters is how you use the tools.

There are more ways than one to throw a jab. It’s the diversity that makes you better. You have to embrace diversity. MMA is all about the mixing, right? How does this relate to tech? You come up with new ideas by surrounding yourself with people who are different and have different experience, knowledge, etc. Fresh eyes and fresh ideas.

Tech needs diversity to succeed

By having a diverse team, you’ll be prepared for things you would not have thought about by yourself.

Admitting that hard problems are hard — Sasha Rosenbaum

Have you ever been at a meeting where your boss explains this horribly complex architecture he wants for a project? We reward complexity. (Resume driven development: use tech because it looks good on your resume, not because you actually needed it.) There is a cost to admitting ignorance and pushing back on this idea.

We should stop pretending that:

• distributed systems are easy (running Kubernetes is not easy, it is error prone and hard)
• people can be available 24/7
• unicorn companies don’t have technical debt
• that new product will solve all of your problems

There are costs to complexity:

• time to learn
• time to build
• maintenance
• increased chances of failure

This costs us money and productivity.

We use heuristics to navigate the world. And we need to because otherwise we would not last very long. Problem is that our current assumptions are not very good (e.g. “women and persons of color are not very smart”). These stereotypes are persistent and actually hurt people. Same resume: 6–14% more interviews for males vs female candidate, 50% more callbacks if the name of the candidate sounded white vs black-sounding names.

A consequence is also that the cost for admitting ignorance and errors are much higher for women, people of color and junior engineers.

Building effective teams (according to the same Google study that was mentioned by Julie Gunderson) is not related to background, personality types or skills, but psychological safety.

But psychological safety does not just happen. It has to be built by leaders.

• Make it safe to express opinions
• Make it safe to admit mistakes (e.g. admit mistakes yourself as a leader)
• Realise that you do not know everything

This isn’t easy. This is a hard problem. But it is going to be worth solving it.

Reading tip: Mindset: The New Psychology of Success by Carol S. Dweck.

Ignites

Ignites are short talks. Each speaker provides twenty slides which automatically advance every fifteen seconds.

The danger of DevOps Certifications — Ken Mugrage

You cannot certify empathy. Most certifications only certify attendance, not actual knowledge or experience.

DevOps is about learning from failure, but companies that hire you to “implement DevOps” because you have a “DevOps certificate” are not going to give you the time to learn. If you fail, the CEO gets to say “see, DevOps doesn’t work (for us).” This can even be a danger for your career.

Training, courses, etc are useful. And if you want to get a certificate, that is also fine. But it’s not the thing that will make your career.

Certification on tools is ok. You cannot certify culture.

Some companies will want certification, fine, but understand that it’s not the key to your success.

4 ways to help people in their careers — Emanuil Tolev

• Coaching: shorter term relationship to learn a specific skill
• Mentoring: longer, exploratory relationship
• Sponsorship: improve how others will think of you (potential, skills)
• Career counselling: less formal version of the previous ways

The stories we tell about ourselves are powerful. Melancholy and contemplation are good things. You should prevent self-reinforcing feelings of doubt though.

For the coaches, mentors, etc: you cannot take the journey of others, but you can help them with their journey.

Serverless changes nothing — Ant Stanley

Serverless does not bring new ideas. Thinks like the 12 factor app, the Unix philosophy or microservices already existed. Operation practices are also the same, often even the same tools are used.

If you want to succeed in serverless, you need to adopt a DevOps way of working first; do the cultural transformation first. You cannot do serverless without DevOps.