Streamlining Signed Artifacts in Container Ecosystems — Tonis Tiigi

It’s possible to sign Docker images, but at the moment most are actually not signed. Also, users should understand what the signature is protecting and what it’s not protecting. We should not want signing just to tick a box on the security checlist, but because of the security it adds. And we need something simple: integrated with existing tools, should not slow down tools.

Buildkit powers “docker build” but is not limited to Dockerfiles. It’s high performance, can build complex builds and has caching.

A modern build is a graph of images, Git repositories, local files, etc. The results are images, binaries, archives.

We need Supply-chain Levels for Software Artifacts (SLSA) provenance: what has actually happened in the build? What was the build config? Et cetera. It’s useful to figure out how an artifact was built.

Buildkit does not sign images by default. GitHub has an example in the documentation to run a build with Buildkit and generate an artifact. It claims to generate an unforgeable statement. But if your GitHub credentials are leaked and the attacker can get your hands on the temporary signing key, they can use it to sign their own artifacts.

Docker created the github-builder repository. It contains reusable GitHub Actions to securely build images. If you use this, your images are signed to prove that they were built from a certain repository, using the configured build steps. Where Buildkit (among other things) provides isolation, github-builder provides signing context. It also protects against build dependency leaks.

So that takes care of the signatures, but how do you verify them?

• The command “docker inspect” now shows verified signatures
• You can manually verify it with cosign
• You can also use sigstore/policy-controller for Kubernetes

Buildx also includes experimental Rego (Open Policy Agent) policy support. This means you can write a matching policy for Dockerfile, e.g. Dockerfile.rego, which is then automatically loaded. All build sources now need to pass policy for the build to continue (images, Git repositories, URLs, etc).

You can do very complex stuff in the policies. As simple example Tonis showed:

package docker

allow if {
  input.image.repo = "org/app"
  docker_github_builder_tag(input.image, "org/app", input.image.tag)
}

This policy should make sure that the image can only be built from this repository and that the image tag should match the Git tag.

Summary:

• No reason not to sign
• Not all signatures are equal
• Software pulling packages should verify pulled content

Link to the conference page

Sequoia git: Making Signed Commits Matter — Neal H. Walfield

Version control systems (also known as VCSs) track the following:

• Changes to the code
• Authorship
• Other metadata
• Commit message

But the author can be faked: the metadata is set by the author, including the author’s name. After a quick “git config” command you can commit as anyone you want, for example Linus Torvalds. Sure, GitHub could see that the committer (the one pushing the commit) and author are different. However, this is not necessarily bad because we might simply want to give proper attribution to the author of the commit.

And in theory the forge might also be compromised, or someone may have gotten permission to push to the project.

To prevent impersonations, we can cryptographically prove who the author is by signing the commits. But now the problem shifts to the certificates. Because anyone can create a key with any name (again, for example Linus) attached to it. So what does a signed commit mean now?

How can we be sure that the author is who they say they are? There are ways:

• You could talk to developer the verify
• You could go to key signing parties
• You can use a central authority that you trust (e.g. keys.openpgp.org, the Linux developer keyring, the distributions-gpg-keys package, or, if you trust Github, use github.com/<username>.gpg)

You can use the following command to show the Git log and the signatures on them:

git log --show-signature

But now you need to actually check that the signatures are indeed made by the certificates you trust.

It’s up to the maintainers of the software to curate a list of contributors and track when contributors join and leave (yes, there is a temporal element as well). This is hard. Maintainer needs tooling. And you would want to detect unauthorized commits (impersonation, a malicious forge, a machine in the middle or for instance when project is given to a new maintainer by a forge/registry).

What does the solution look like?

• Clear semantics
• The project itself maintains signing policy
• Third party uses maintainers’ policy to authenticate project
• Verification, not attestation: do not rely on any external authority

(Note that the maintainers can still be socially engineered to include the key of an attacker in their policy. So they still have to be careful about who is added to the policy.)

Sequoia git provides:

• Specification
• Config
• Tooling

With Sequoia git (which part of the Sequoia PGP project) you can have a signing policy in an openpgp-policy.toml file in the project’s Git repository. It specifies users, their keys and their capabilities. You can use sg-git to help maintain this file.

For instance to add user Alice and then describe the current policy, you can use the following commands:

sq-git policy authorize alice --committer <cert>
sq-git policy describe

A commit is “authenticated” if at least one parent commit says the commit is acceptable (via the policy). To verify that there is an authenticated path from the current state back to a certain commit we trust, use this command:

sq-git log --trust-root <sha of trusted commit>

Projects may have contributions from others that are not included in the policy. To maintain an authenticated path when accepting the contribution, a trusted author needs to merge the contribution via a merge commit that is authenticated. (You may need to use the “--no-ff” on the merge to make sure there is a merge commit though.)

Link to the conference page

An Endpoint Telemetry Blueprint for Security Teams — Victor Lyuboslavsky

With open source we can inspect something that is broken, we can change the defaults. With security we are used to the opposite; it’s a black box. We are not used to owning the data. The data exists on the endpoints, but ownership is transferred to a different team. How can we add more security in a way engineers understand and can use?

Victor presents a blueprint with the following layers:

• Endpoint agents
• Control layer
• Ingestion, streaming & storage
• Detection
• Correlation, intelligence and response

The value is not in the layers themselves, but the boundaries. For example, the ingestion should move the data reliably but should not care which tool collected it. This makes them loosely coupled.

For endpoint agents Victor suggests osquery which allows basic questions about endpoints. Data is structured and consistent. It aligns with open source values. (Alternatives: scripts & cron, log shippers like filebeat or tools like auditd or Event Tracing for Windows.)

Controlling the data (the next layer) means that you want to have:

• Central config
• Live queries
• Consistent schemas

Fleet (disclaimer: Victor works here) is built to manage osquery at scale and a good candidate for this layer.

The control layer needs to work hand-in-hand with ingestion layer. The ingestion layer moves data to downstream system. E.g. Vector or Logstash can be used here.

Ingestion isn’t where you get clever. It’s where you get reliable.

Streaming decouples users from consumers and e.g. allows replay. Note that this is an optional step and it would come after ingestion, not in place of it. For instance Apache Kafka can be used in this layer. Ingestion absorbs the mess. Streaming preserves flexibility.

The storage layer is where telemetry becomes durable. It’s about being able to ask hard questions later. Examples of useful tools: ClickHouse, Elasticsearch (which is better at text search) and Iceberg (which is slower for active investigation).

For the detection layer you might want to use Sigma. It provided portability. Rules are translated to native SQL running on ClickHouse. Intent (Sigma signatures) becomes execution (SQL query to get the data).

Finally the correlation layer: Grafana can be used for correlation and visualisation. Grafana can query ClickHouse. Grafana also has alerting.

Note that response isn’t just about automation. It’s also to pause and ask better questions. The correlation layer should focus on enabling humans to act.

Open endpoint telemetry is not an “EDR killer”. It does not replace it. It adds diversity and complements other tools. It provides a second set of eyes.

Link to the conference page

The Bakery: How PEP810 sped up my bread operations business — Jacob Coffee

Python loads imports eagerly by default. This leads to memory bloat and cold start issues. Explicit lazy imports (see PEP 810) only import a module when it’s first accessed not when the import statement is executed.

Lazy import is scheduled to be included in Python 3.15 and looks like this:

lazy import foo from bar

The design principles applied are that lazy imports are:

• Explicit
• Local
• Granular

When parsing the Python code a proxy module is created. Only when the module is actually used, the proxy is transparently replaced by the real package. You will not always see improvements, so do not blindly replace all imports with lazy imports.

PEP 810 also eliminates the need for TYPE_CHECKING guards. (See the typing docs, in short: importing a module that is expensive and only contains types used for type checking in an “if TYPE_CHECKING:” block.) It also helps for faster test discovery and collection, less memory usage, decrease cold start slowness in e.g. AWS Lambda functions, CLI applications, etc.

Meta (with Cinder) saw a 70% startup time reduction and 40% memory savings. PySide has a 35% startup improvement.

About CLI tools: when using lazy imports you might notice the difference when using --help. There’s no need to load all dependencies to just output the help text of a tool.

Some notes:

• Import time side effects (e.g. logging configuration, DB connections) are also delayed!
• Type checkers need to be updated
• Import errors move to first use (so in runtime, not at launch). Keep that in mind when debugging
• It’s not always faster, so profile your application before migrating and see where you can potentially benefit
• Document your lazy imports!
• You cannot do lazy imports in functions

Circular imports are probably still a problem, but they just show up later.

Link to the repo for this talk

Link to the conference page

Modern Python monorepo with uv, workspaces, prek and shared libraries — Jarek Potiuk

Jarek is, besides his other roles, the number 1 Apache Airflow contributor. The Apache Airflow repo is the monorepo he talks about today. There is also a series of blog posts about this topic: see part 1, which links to the other parts.

Airflow drove early requirements for uv workspaces. They now manage 120+ distributions seamlessly with it. It allows them to combine distributions to work together in a workspace. Also used to import from one distribution in another one.

The project shares a single virtual environment used by uv in root of project. If you run “uv sync” from the top level you get everything. If you run it in a subdirectory (e.g. airflow-core) you only get what is needed for that distribution.

Benefits of the uv workspaces:

• Isolated
• Explicit
• Flexible

Hatch has (or will have, at the time of writing) largely compatible workspaces.

However pre-commit became a bottleneck. They needed to run 170+ pre commit hooks on every commit. Prek is drop-in replacement for pre-commit and works fantastic. It is optimized for speed and monorepos.

Airflow uses symlinked shares libraries (where a shared lib is also a distribution). The Hatchling build backend needs to replace links with physical copies during packaging. They use Prek to maintain consistency.

uv sync detects conflicts between merged requirements files and Prek hooks enforce relative imports in shared code to prevent cross coupling issues (IIRC)

Link to the conference page

PyInfra: Because Your Infrastructure Deserves Real Code in Python, Not YAML Soup — Loïc “wowi42” Tosser

Loïc is a Frenchmen (which, as he himself states, means he must have opinions) and not a YAML fan to put it mildly. That is: YAML as a programming language, e.g. how it is used in Ansible.

PyInfra is an infrastructure as code library to write Python code which is then translated to shell scripts to run on the target hosts. So, in contrast to Ansible, you do not need Python on the target. The target machine only needs SSH and a POSIX shell. You can also configure Docker containers with PyInfra.

If it has SSH, PyInfra can talk to it.

PyInfra has idempotent operations and built-in diff checking. Declarative infrastructure with actual code and not YAML. You can use inventory from Terraform, Coolify or any API.

You can leverage the entire Python packaging ecosystem. Slack integration? Just use the right Python package.

PyInfra is not only a CLI tool, you can also use it as a library.

PyInfra is 10 times faster than Ansible, uses 70% less code, has proper code reuse via import and proper loops instead of with_items. It can have actual unit tests and can scale to thousands of servers. Also you no longer have error messages stating that the error appears to be in … but may be elsewhere in the file … (looking at you Ansible). PyInfra has clear error messages without having to specify -vvvv and wading through hundreds of lines of output.

The suggested migration path:

• Start small, one playbook at a time
• Use your IDE for autocomplete and refactoring
• Leverage Python’s standard library and the ecosystem with all its packages
• Sleep better because you don’t have to debug at 3 AM.

Is PyInfra production ready? Yes! It has a stable API, is already in use in production, it’s actively maintained and is MIT licensed (so no commercial entity behind it to steer its direction).

You can get started today with a simple “pip install pyinfra”.

Link to the conference page

(Note from me, Mark, I found Loïc a great speaker: he has lots of energy, is funny and can transfer his enthusiasm to the room. If the topic interests you and the video becomes available, I would recommend watching this talk as a great sales pitch to get started with PyInfra.)

Ducks to the rescue - ETL using Python and DuckDB — Marc-André Lemburg

ETL stands for Extract, Transform, Load. Nowadays we usually do Extract, Load, Transform because databases are efficient in processing.

DuckDB is open source, in-process analytics data storage (OLAP). It is similar to SQLite, but for OLAP workloads. It has great Python support and uses SQL as standard query language. It’s pip installable, column based (Apache Arrow). It’s single writer but allows for multiple readers, so it’s not a distributed database.

Polars’ streaming can help with processing your data as a line-by-line stream so you don’t have to load the whole file in memory at once.

Example to load a CSV file into DuckDB extremely fast:

SELECT * FROM read_csv(...)

You can load the data into staging tables first to prepare everything and not mess up e.g. existing data. You can then transform data in DuckDB, e.g. filter out unneeded and duplicate data, validate data, fill in missing data, convert data types, etc. You can do the transforms in SQL. You can even use native integrations to write to PostgreSQL, MySQL, etc. Or worst case stream to Python.

Guidelines:

• Know your queries, that is: know how your data is going to be used
• Use the Pareto principle (80/20 rule): optimize for queries that are used often
• Keep a healthy balance between performance and space requirements (which are often trade-offs)

Huge datasets: use the DuckLake extension.

To get started: “uv add duckdb”. Do some experiments and see how it works for you.

Link to the conference page

My takeaways

• Yes, FOSDEM is crowded and you may not be able to get into every talk you want to see in person, but it’s still nice to be there. It’s well organised and there’s a friendly atmosphere. Lots of interesting projects to see and people to talk to. And it’s convenient if you want to sponsor your favorite projects by buying some merchandise.
• It’s worth investigating signing Docker images (in the right way) further.
• Lazy imports look useful! Once Python 3.15 lands it’s worth doing profiling on the projects I work on to see if we can use those to speed things up on startup and save some memory.
• At work we recently decided to go for a monorepo for a project. I want to see if/how uv workspaces and prek can help us.
• I’ve written a bunch of Ansible roles to configure my humble homelab and laptop. Perhaps it’s time to switch to PyInfra? It sounds promising and might be worth the investment of migrating to.

About the trip

This year I had more time, so I booked a room at Falko Hotel for two nights. It’s about a 20–30 minute drive (depending on traffic) to the parking garage I used. And from there about 20 minutes with pubic transport to the Université libre de Bruxelles.

Staying another night meant I had more time for sightseeing, had the time to write this post from my notes and could drive home well rested the next day.

As for tech: besides a phone and laptop, I also brought along two items that made the trip more comfortable:

• A MOJOGEAR Mini Evo powerbank to give my phone extra juice to make it through the day. With 10.000 mAh and up to 22.5W of power it’s more than sufficient for a day at a conference. With its small size and less than 175 grams in weight, it’s also easy to carry around.
• A GL.iNet Opal (GL-SFT1200) travel router. I plug it in, hook it up to the hotel internet, start a VPN connection and all my other devices automatically connect to it and can use the internet without the hotel snooping on my traffic. (Not that I have an indication that my hotel would do that, but theoretically they could if I would not use a VPN.)

Keynote: CAMS Then and Now and the Path Forward — Thomas “VikingOps” Krag

When people think about DevOps, they think of CAMS, which stands for:

• Culture
• Automation
• Measurement
• Sharing

The term CAMS was formalized by John Willis and he wrote down his idea in the article What Devops Means to Me. This talk will focus on the most important aspect: culture, and specifically organizational learning.

Culture

Culture as described by Simon Sinek:

The 5 disciplines from organizational learning:

Shared vision

Have people play the game not just because of the rules, but because they feel responsible for the game.

Mental Models

Breaking your own assumptions and how they can influence your actions. You need to self-reflect on your own beliefs.

Personal Mastery

This is about owning yourself, learning and achieving your personal goals. And to do the latter you first need to define what is important for yourself.

Team learning

Effective teamwork leads to results that persons cannot achieve on their own. And individuals working as a team can learn faster than they would by themselves.

Systems Thinking

Looking into patterns that emerge inside your organization from a holistic viewpoint instead of just looking at your own team.

Automation

Automation is about automating the right things. You don’t want to do it just to automate things, but it has to fit in the system. So you have to start with culture before you think about automation. The ultimate goal is GitOps where everything happens via a pull request.

Measurement

Measurement started out with a focus on the tooling. But measuring how you work is as important as measuring your infrastructure.

Important key metrics, from DORA:

• deployment frequency (how often do you release to production)
• lead time for changes (the amount of time for a change to get into production)
• time to restore service (how long does it take to recover from a failure in production)
• change failure rate (the percentage of deployments causing a failure in production)

Sharing

Share how you are doing (in) DevOps. This is how we ended up here now. Share how you are improving your own organization. But also share information within your organization: documentation, videos, presentations, open spaces, lean coffee sessions

Three ways

CAMS originated in 2010. The three ways are principles that came out of the book The Phoenix Project. They are:

• 1st way: create flow (systems thinking)
• 2nd way: feedback loops
• 3nd way: experimentation, risks, learning

If we map these to CAMS:

• Culture: 3rd way
• Automation: 1st way
• Measurement: 2nd way
• Sharing: 3rd way

Working on your culture is as important as doing your actual work.

So what’s next? CAMS is still as applicable as 10 years ago. It has always been important, but it was only put into words in 2010. We need to continue sharing to get the full value out of it.

Gamification of Chaos Testing — Bram Vogelaar

We think of Usain Bolt as the record breaking athlete, but he’s also the person that worked really hard to get there. It takes a lot of time and effort to become good at something. Pilots and firemen spend most of their time training and not doing what you expect them to do; just to make sure they perform well under pressure. Also note that pilots use a lot of checklists to prevent mistakes.

We should do the same: train for when our platform is in an error state. We should not just be able to detect it, but also solve the problem.

Chaos engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production. This practice started at Netflix with Chaos Monkey.

We need to become comfortable with experimenting. Have game day exercises and analyze what happened, to improve your training. Do not just focus on the result of the exercise itself, but also ask questions like “was it the right experiment?”

Now that we use containers, add sidecar containers with tools to get metrics or detect errors. Or to do chaos engineering e.g. with Toxiproxy.

Since checklists are boring, we can use gamification to spice things up. Celebrate failure, and learn from it!

Living in the year 3000: breaking production on purpose on Saturdays and have the system remedy the problem itself.

Convince management that failure is normal and expected behaviour. Promising 100% uptime is not realistic. Large, complex systems will always be in a (somewhat) degraded state.

Let engineers be scientists to deal with this complex environment. Give them training, allow them to do tests (experiments), which results in having valid monitoring that lead to actionable alerts. Get the engineers in a state where they are comfortable with failures.

(Slides)

Watch Your Wallet! Cost Optimizations in AWS — Renato Losio

Each AWS service has it’s own price components. It’s a complex subject. Even a simple service like a load balancer has multiple components. It looks simple with the “$0.008 per LCU-hour” price tag, but now you have to figure out what an LCU-hour is. Then you learn it has four dimensions that are measured: number of new connections per second, active connections, processed bytes and rule evaluations. Good luck predicting the costs.

This presentation only sticks to the basics since cost optimization it such a big topic.

To start to manage/reduce your costs, you need to enable billing and costs for your DevOps team. If you cannot measure your costs, you cannot manage it. Note that you do not need an excessive amount of tags for cost management. First you need to figure out what you are going to change and how it’s going to affect the bill for your company.

What patterns can we avoid? In most organizations, the most expensive parts of your bill will be:

• Compute
• Storage
• Data transfer

So we will dive into these subjects.

Compute

Tips to reduce costs:

• Avoid fixed IP addresses where possible. This is not so much about the costs of the IP address itself, but mostly because architectures that require a fixed IP tend to be complex and more expensive.
• Use Graviton (ARM) instances. They have a better price to performance ratio. You can also migrate managed services (RDS, Lambda functions) to Graviton.
• Check out Lightsail. It’s less flexible than using EC2 instances, but cheaper.

Data transfer

We usually forget to take data transfer costs into account upfront. It’s also a complex subject and there are a lot of considerations to make.

Tips:

• Multi AZ is always good, but are 3 zones always better than 2 zones?
• Multi region is easy to do nowadays, but expensive with regard to data transfer. Ask yourself if you really need it.
• If you want to use multiple cloud providers always think about the data transfer costs. Perhaps the storage costs are lower at provider X, but if you have to transfer data, you’ll probably pay an egress cost.
• Using a CDN (CloudFront) might be cheaper than paying for the egress data transfer otherwise.

Storage

Initially it was simple: there were only two storage classes. Currently there are 6 different classes with their own prices and characteristics.

The best option is to use lifecycle rules to move data between different classes. Note that you in a lifecycle policy you cannot filter the objects based on an extension (e.g. *.jpg) but instead you need to think “from left to right.” So you need to think upfront about the prefixes you are going to want to use in your bucket.

Managed services can offer you automatic and manual backups, which can be great. But what is the cost of that? Check how much retention you need for example.

With regard to EBS: for most use cases gp3 is better and cheaper than gp2 (except for very large volumes). Note that you can change the EBS volume type without stopping the machine. In most cases you can have a 20% cost saving without affecting your performance.

AWS changes quickly

After each AWS re:Invent, your deployment is probably outdated with regard to cost optimizations. Examples:

• Use gp3 instead of gp2 for your EBS volumes.
• The instance type m6 might be more interesting than the m5 or m4 you may currently be using.
• Dublin was the cheapest region in Europe, but for most things Stockholm is cheaper than Dublin at the moment.

So keep up to date with the offerings.

Keynote: Call for Code with The Linux Foundation: Contributing to Tech-for-Good Even if You’re Not Techincal — Daniel Krook and Demi Ajayi

Call for Code is a multi year program launched in 2018 to address humanitarian issues and help bridge potential solutions. Last year the global challenge was around climate change and a track was added for the social and business impact of the COVID-19 pandemic.

It’s not just about generating ideas to take on the issues. It should eventually also lead to an adopted open source solution that is sustainable.

The 14 projects discussed today can be found on Call for Code page on the Linux Foundation website. You can also read about them via the IBM developer site. GitHub is central for how they iterate on the features. The related organizations are :

• Call for Code with the Linux Foundation
• Call for Code for Racial Justice

The key takeaway for this session is to make us help improve how the projects do DevOps. The goal is to ensure that everyone can contribute to the projects, can do so with confidence and the projects can be deployed with speed.

The Call for Code for Racial Justice open source projects are categorised in three pillars:

• Police reform and judicial accountability:
  • Fair Change
  • Open Sentencing
  • Incident Accuracy Reporting System (IARS)
• Diverse representation:
  • Take Two
• Policy and legislative reform:
  • Legit-Info
  • Five Fifth Voters
  • Truth Loop

Demi talked about each of these projects in the program and their tech stacks. You can read more about them on the Call for Code for Radical Justice section on the IBM developer site.

Daniel in turn talked about other Call for Code projects:

• ClusterDuck
• Pyrrha
• ISAC-SIMO
• OpenEEW
• Liquid Prep
• DroneAid
• Rend-o-matic

Even if you cannot code, there are numerous ways you can contribute, e.g. conducting user research, write/review documentation, do design work, advocacy like speaking at conferences.

There are multiple ways to get involved:

• Via the virtual community (join slack, join events)
• Directly via the projects on GitHub
• Conduct outreach (recruit friends, host events, etc)

DevSecOps Culture: Laughing Through the Failures — Chris Romeo

Chris shared 10 DevSecOps failures and talked about how to change the culture and turn these failures into successes.

#1 Name and brand

This quote nicely sums it up:

To change the culture:

• Embrace security and make it part of DevOps
• Teach this new definition to everyone involved with your project (developers, testers, product managers, executives, etc)
• Create a “DevSecOps” swear jar ;-)

#2 The infinity graph

The problem is that nothing ever gets done with this infinity graph. It’s not an accurate representation.

The solution is to talk about pipelines instead and integrating security into them. Code review should include security, vulnerability scanning should be part of the pipeline, etc. Ban the infinity graph.

#3 Security as a special team

Creating a specific security team is the opposite of what DevOps is about. It’s about working together. Security isn’t a specialty, it is the responsibility of everybody. This requires knowledge and expertise.

The other way around is also true: teach security people to code. They don’t have to become great coders, but it would be nice if they can review code and make suggestions to make things more secure.

#4 Vendor defined DevOps

Sometimes we let vendors define what DevOps and security are for us, via the products that they offer. It would be better to find the best of breed outside of the offering of cloud provider. Take a vendor independent approach and determine what DevOps means to you.

#5 Big company envy

Looking at the big companies can be discouraging. You most likely have not invested the same time in it as e.g. Netflix, Etsy, etc. have. So while you won’t be at the same level, don’t see this as an excuse to give up. Do the DevOps that you do. Don’t fixate on the top of the class. Get on that path and make incremental progress.

Use the OWASP DevSecOps Maturity Model to create a roadmap.

#6 Overcomplicated pipelines and doing everything now

This can be a complicated subject (see for example the DevSecOps Reference Architecture from Sonatype).

But keep it simple! Start with a small subset of security tools. Everybody related to your project should be able to explain the build pipeline. Don’t try to solve all problems immediately. Take a phased approach.

#7 Security as gatekeeper

Security might want to slow down the pipeline and act as a gatekeeper. Don’t say “no,” but “yes, if…” For example: “yes, that would be a great feature if you enable multifactor authentication.”

Practice empathy. Both security people for developers, but also the other way around.

#8 Noisy security tools

You buy a tool and enable every option to “get your money’s worth.” The result is 10,000 JIRA tickets of things that need to be fixed. This does not help.

It would be better to tune the tools and don’t waste time with security findings that do not matter.

Start with a minimal policy focussing on the largest issue. Developers will then start to trust the tool and then you can slowly increase the policy.

#9 Lack of threat modelling

You scanning tools cannot find business logic flaws; there’s no pattern to it.

Perform threat modelling outside of the pipeline. It should be done when new feature assignments go out.

#10 Vulnerable code in the wild

There are lots of vulnerabilities in open source software; this is a supply chain problem.

To improve this: embed software composition analysis (SCA) in all your pipelines. Set the SCA policy to fail when a vulnerability is detected. If you filter out a vulnerability (e.g. because there’s no fix yet), make sure the filter will not be active forever.

Successes

The 10 DevOps successes we can distil from the failures above:

1. Just call it DevOps
2. Pipelines
3. Security for everyone
4. Embrace your DevOps
5. Be content with your DevOps
6. Simple and staged pipeline
7. Security as trusted partner
8. Tuned and valuable security tools
9. Threat modelling for everyone
10. Breaking the build for vulnerabilities

Key takeaways

• Hopefully the real impact of DevOps (when looking back 50 years from now) is going to be security culture related.
• Some of the failures were outside of your control, but you can change them.
• Everybody has to code, choose your DevOps, keep it simple, lower the noise, add the best practices we discussed, do some threat modelling, break the build for vulnerabilities.
• Embrace and laugh at the failures.

Keynote: Managing Risk with Service Level Objectives and Chaos Engineering — Liz Fong-Jones

Besides being a principal developer advocate at Honeycomb, Liz is also a member of the platform on-call rotation. Honeycomb deploys with confidence up to 14 times a day, every day of the week—so also on Fridays. How do they manage to (mostly) meet their Service Level Objectives (SLOs) while also scaling out their user traffic?

Their confidence recipe:

• Quantify the amount of reliability.
• Be able to identify risk areas that might prevent them from fulfilling their targets.
• Test to verify that the assumptions about the systems are correct.
• Respond to that feedback to address the problems found via those experiments or though natural outages.

How to measure reliability?

You need to know how broken is “too broken.” You don’t have to alert on all problems when working at scale. You need to measure success of the service and define SLOs. These are a way to measure and quantify your reliability.

Honeycomb’s jobs is to reliably ingest telemetry, index it, store it safely and let people query it in near-real-time. Honeycomb’s SLOs measure the things that their customers care about.

For example, they have set an SLO that the homepage needs to load quickly (within a few hundred milliseconds) in 99.9% of the times. User queries need to be run successful “only” 99% of the time and are allowed to take up to 10 seconds. On the other hand: ingestion needs to succeed in 99.99% of the time since they only have one shot at it.

Services are not just 100% down or 100% up (most of the time).

These metrics help Honeycomb make decisions about reliability and product velocity. If the service is down too much, they need to invest in reliability (since having features that cannot be used does not add value). On the other hand: if they exceed the SLO, they can move faster.

Recipe for shipping reliably and quickly

Practices used by Honeycomb:

• Code is instrumented. Think beforehand what a success or failure in production would look like.
• Functional and visual testing using libraries that create snapshots.
• The design for feature flag deployment (only making a feature available to a small percentage of users initially).
• Practice automated integration plus human review. (There’s an SLO for how long the tests are allowed to take. Code reviews are high priority tasks since you are blocking someone else by not reviewing their code.)
• The main branch is safe to release any time.
• Automatically roll out the changes.
• After the deployment the engineers have to observe what is happening with their code in production. Only after they confirm that everything is okay, they can go home. So you are free to deploy on Friday at 19:30 as long as you are willing to stick around until your code is deployed and you have confirmed it is not causing issues.

For infrastructure Honeycomb also use infrastructure as code practices:

• They can use CI and feature flags for their infrastructure.
• They can automatically provision fleets if needed.
• They can automatically quarantine certain paths to keep the main fleet from crashing or do performance profiling.

Chaos Engineering

Left-over error budget is used for chaos engineering experiments. This is something where you go test a hypothesis. You need to control the percentage of users affected by it and be able to revert the impact you are causing.

Chaos engineering is engineering. It’s not pure chaos.

This works well for stateless things, but how does it work for stateful things? In the case of the Honeycomb infrastructure, they make sure to only restart one server or service at a time. They do not introduce too much chaos to reduce the likelihood that something goes catastrophically wrong.

Two reasons why you will want to do these experiments at 3 PM and not at 3 AM:

• You want to test at peak traffic instead of low traffic, since the latter could give you a false sense of security in situations when everything may still look normal even though this would have been a problem in a high traffic situation.
• When doing things in the afternoon, there are more people available to deal with something than there would be in the middle of the night.

With the experiments they measure if they had an impact on the customer experience. If they cause a change, does the telemetry reflect this? (Is the node indeed reported as being offline, for example?) When you fix things, you need to repeat the experiment and make sure the change indeed fixed the issue.

When you burn the error budget, the SRE book states that you should freeze deploys. Liz disagrees. If you freeze deploys, but continue with feature development, the risk of the next deployment only increases. Instead, Liz advocates for using the team’s time to work on reliability (i.e. change the nature of the work instead of stopping work).

Fast and reliable: pick both!

You don’t have to pick between fast and reliable. In a lot of ways fast is reliable. If you exercise your delivery pipelines every hour of every day, stopping becomes the anomaly instead of deploying.

Takeaways

• By designing the delivery pipeline for reliability, Honeycomb can meet their SLOs.
• Feature flag can reduce blast radius, and keep you within your SLO.
• And when they cannot: there are other ways to mitigate the risks.
• By discovering risks at 3 PM and not 3 AM, you improve the customer experience since the system is more resilient.
• If something does go catastrophically wrong, remember that the SLO is a guideline not a rule. SLOs are for managing predictable-ish unknown-unknowns and not things that are completely outside of your control.
• We are all part of sociotechnical systems. Customers, engineers and stakeholders alike.
• Outages or failed experiments are learning opportunities, not reasons to fire someone.
• SLOs are an opportunity to have discussions about trade-offs between stability and speed.
• DevOps is about talking to each other and talking to our customers.

Common Pitfalls of Infrastructure as Code (And how to avoid them!) — Tim Davis

We start with the basics: what is Infrastructure as Code (IaC)? With the advent of cloud providers, you no longer use hardware, but a UI to stand up infrastructure. This led to shadow IT since developers ran off with a credit card to provision what they needed themselves, instead of using slow, internal IT systems.

Developers however rather write code and use developer methodologies than click through a UI. This is where IaC started. With it you can create and manage infrastructure by writing code.

What are he pitfalls? The bad news: you get all the pitfalls of infrastructure and all the pitfalls of code. But you’ve probably already got a lot of experience with those issues and teams to handle them. You just use a different methodology.

The first pitfall is not fostering the communication between the groups that have experience and tools.

Infrastructure pitfalls

Which framework/tool do you pick? There are basically two categories: multi-cloud or cloud agnostic tools on the one hand (like Terraform and Pulumi) and cloud specific tools on the other (like CloudFormation). Note that for example with Terraform and Pulumi you still have to rewrite code when switching from one cloud provider to another, but at least the tool is familiar.

Security is a huge thing. You still need to know how to design your VPC, IAM policies, security policies, etc. You still need to communicate with all the teams that have the experience. It’s not just Dev and Ops. With tools like Terrascan and Checkov you can shift-left the security aspect instead of trying to bolt it on afterwards.

Code pitfalls

The biggest thing issue is with default values. If you use the UI, there are a lot of boxes that may be blank or have stuff in them. Some of the boxes can be left blank, for some you need to specify what you want. The UI is going to yell at you; if you use IaC things may be less in your face.

You don’t want to deploy something with an open policy. Open Policy Agent can really help you to make sure you stay within your allowed parameters. For instance you can write a policy to make sure you are only use a specific region, don’t deploy an open S3 bucket or that you only use certain sizes of EC2 instances.

If you hard code certain values in Terraform or other IaC tools, you might need to copy/paste a lot of code if you want to create e.g. a test, acceptance and production environment. To mitigate these DRY (don’t repeat yourself) issues you can for instance use Terraform modules or Terragrunt.

State size can become a problem. If you want to, you can put all of your infrastructure in the same state file (which is where your tool stores the state of the infrastructure). However it means the tool will have to check all resources in the state file to detect if it needs to do something. To mitigate this, you can again use Terraform modules. It helps with performance and makes the codebase more manageable.

Wrap-up

The conference is still ongoing while I publish this post. However, this is it for me for All Day DevOps for this year. I learned new things and got inspired.

Thanks to the organizers, moderators and speakers for hosting another great event.

(On the first day I followed the sessions of the “Building your application for the cloud” learning path.)

Modernizing your infrastructure: moving to Infrastructure as Code (IaC) — Emily Freeman

Operations is changing—modernizing. DevOps reduces or eliminates the friction between the development and operations teams.

One way of defining DevOps is with the CALMS acronym:

• Culture
• Automation
• Lean
• Measurement
• Sharing

SRE has evolved beyond its original form. SRE can be seen as a prescriptive implementation of DevOps. It has a focus on deploying and maintaining applications.

About failure:

In the cloud, sometimes it is rainy.

If you are using the cloud, you are using APIs continuously. This allows us to automate a lot of things. And do configuration via (YAML) files which can be version controlled. This gives us repeatability and reliability.

Toil is defined as manual, repetitive work. Automation removes the toil. By creating code to do the work, we can reuse it.

Azure Resource Manager templates: you can manually deploy infrastructure and then view the template from within the Azure portal. This template is a configuration file which you can store and use over and over again.

Deploying small batches of code is key. The risk of small changes is less than having big (and many) changes. Small batches also make it simpler to identify bugs. Releasing often makes sure feedback on changes is received faster.

Microsoft works hard to make sure that Azure Pipelines work with the tools you already use.

Additional resources:

• Code
• The Small Batches Principle

Monitoring your infrastructure and applications in production — Jason Hand

Why, what and how do we monitor?

Let’s first have a look at a definition of SRE:

Site Reliability Engineering is an engineering discipline devoted to helping an organization sustainably achieve the appropriate level of reliability in their systems, services, and products.

Note the terms “appropriate” and “reliability.” Hundred percent reliability is expensive but not always needed. For planes and pacemakers you want absolute reliability. For applications, this usually is not needed. Ask yourself what the parts of reliability are that you want to be concerned with.

Why monitor?

• Are apps doing what we expect from them?
• Are apps doing what others (customers) expect?
• Are we meeting our goals or at least moving in the right direction?
• Our systems are constantly in change. What is happening with these changes?

Monitoring is the most important part of reliability.

The difference between reacting and responding to incidents is being prepared or not. You’ll want to be prepared to be able to respond, not just react.

Some organizations perform root cause analyses after failures. Complex systems usually don’t have a single root cause though—more often than not it is a combination of causes. Since having more than one root cause is a contradiction in terms, Jason much rather talks about “post-incident reviews.”

Back to the term reliability. This can be measured in a lot of ways. To name a few:

• Availability: will the system respond when I ask it something?
• Latency: “slow is the new down” (we don’t have the time for slow applications).
• Throughput: can you get the volume of data from one point to another in a way you expect?
• Coverage: can a process plough through all the available data?
• Correctness: did it do what it was supposed to do?
• Fidelity: See for example Netflix: perhaps the search is temporarily unavailable but you can still scroll through the catalog and watch a movie.
• Freshness: is the data you get the most recent stuff?
• Durability: if you write data, can you retrieve it again?

It is not just about what we see, but also about what our customer experiences. We should put the customer first and look at things from their perspective.

Service Level Indicators (SLIs) are a ratio/proportion. For instance: the number of successful HTTP calls divided by the total number of HTTP calls.

But where do the numbers come from? A few options:

• Reported by the server
• Reported by clients
• Reported by the application
• Reported by the front-end (load balancer)
• Reported by our monitoring / testing infrastructure

You need to include the way you measure in the SLI.

There is no “right way” to measure things, but there are trade-offs. And you need to be aware of them. Ideally you should monitor as close to the user as possible.

The SLIs are the measurements. The Service Level Objectives (SLOs) determine what is acceptable. If the SLI (indicator) dips below the SLO (objective), we need to get people involved.

How to build a good SLO? Take the thing to monitor (e.g. HTTP requests), look at the SLI proportions and add a time statement. An example: “90% of the HTTP requests as reported by the load balancer succeeded in the last 30 day window.” You can make your SLOs more complex if needed, e.g. by having a lower and upper bound.

Alert on actionable things. Alerts are not logs, notifications, heartbeats or normal situations. Alerts are about things a person needs to go do. And not just any person, but the right person. It should not be something that can be automated. Ideally alerts are pretty simple and include:

• Where is this coming from?
• Which SLO is breached?
• What is the impact for the customer?
• What systems are impacted?
• What are the first couple of steps to take?

Possible future direction of monitoring:

• There’s no one silver bullet. We’ll probably want to integrate multiple monitoring solutions.
• Modular monitoring
• Aggregate monitoring (measure the system as a whole)
• Correlation and anomaly detection (AI / ML)
• Monitoring the monitoring

Additional resources:

• Code
• Azure Monitor overview

Troubleshooting failure in the cloud — Jason Hand

Our systems have become more complex than they have been in the past. We are now using microservices, containers, functions, etc. Our focus for this session: going from the detection phase (previous session) to the response phase.

Before talking about how to detect something is wrong, you need to define that you mean with “wrong.”

Azure has a nice toolbox to help you:

• Azure monitor
• Service health alerts
• Azure Application Insights
• Diagnostic logs (application and system)
• Live debugging via log streams
• Log Analytics

The first question is often: is it just me? Azure (and the other cloud providers) can also have problems. The Azure Resource Health page in the Service Health section shows you if your components have problems. It even has historical data. But because you don’t want to stare at a dashboard all day long, you’ll want to also have health alerts for when something goes wrong.

Guidance for creating service health alerts:

• Don’t create too many, but also not too few. Find your sweet spot.
• Make sure they don’t overlap (to avoid confusion).
• Don’t alert on non-production environments.
• Keep the responders in mind (what would they want).
• Separate alerts for issues, planned maintenance and advisories.

Application Map shows components and how they interact with each other. Where is slowness? Where are problems?

You will need to enable diagnostic logging to get a bunch of logs. And you don’t have to go to the portal, you can use the CLI tool to download them. You can then use a separate tool to analyse the logs.

az webapp log download --resource-group <name> --name <webapp name>

Azure Cloud Shell allows you to have a terminal session in your browser. You don’t need your local CLI.

Use Log Analytics for more complex troubleshooting. It collects a lot of information from a log of sources.

You’ll need to use the Kusto Query Language to get to the information. Queries start with a table, then commands to filter, sort, specify time range, etc. Once you have queries that answer your questions, you can keep them handy by saving them right in the Azure portal.

Azure Monitor Workbooks and Troubleshooting Guides (a.k.a. runbooks) combine text, queries, metrics and parameters into reports. This helps with onboarding and will also help your first responders.

Workbooks are editable by others that have access to the same resources. Troubleshooting Guides are like workbooks, but for dealing with problems.

Future for troubleshooting:

• Observability (OpenCensus and other distributed tracing tools)
• Anomaly detection

Additional resources:

• Code
• Get started with Azure Monitor log queries

Scaling for growth and resiliency — Jeramiah Dooley

How do we take an application and how do we scale it? Bigger picture: asking a lot of “why?” questions.

Do we need to scale out our application and the underlying infrastructure? Maybe.

• Perhaps we should first check if the application is buggy (memory leaks) or if we can optimize it for performance (e.g. database queries).
• Is there a temporary (or artificial) reason the site has a spike?
• Is there unused capacity we can move around?
• Scaling costs money, but if we don’t do it we give the customers a bad experience. This is a risk vs expense issue.
• When is the application no longer meeting our expectations (SLOs)?

We need to factor for both the anticipated and unanticipated spikes.

One can scale up (vertical) or out (horizontal). Scaling up is adding more resources (CPU, memory) to an instance. Scaling out is adding more machines. When scaling out, the application needs to be aware of the change (additional or less instances).

You might not want to scale just one portion of an app since it can cause problems in another part of the app.

Autoscaling can trigger on quite specific conditions: only today if CPU > 80% add more instances.

But should you use autoscaling?

• Do you know your app? When do you need to add more resources? How will your app respond? How quickly? What happens when you scale down?
• Do you know your traffic?
• Autoscaling also has its limits. (E.g. there is going to be a delay.)
• Do you understand the metrics that matter?

It’s not magic, it’s just automation.

An autoscaling profile has:

• Capacity:
  • Minimum
  • Maximum
  • Default
• Triggers (a metric we want to trigger on and a value for that metric)
• Recurrence (when do we want this to happen? Always? First week of the month?)

Warnings with regards to autoscaling:

• Scaling out will happen when any of the triggers are met.
• Scaling in will trigger when all triggers are met.
• Autoscaling always wins over manual scaling settings.

Okay, we know how to handle load. But do we know what to do when we have a localized failure (e.g. a data center that goes down)? Use:

• Multiple regions/geographies
• Region pairs
• Availability zones

Azure Front Door:

• Global HTTP load balancer with instance failover
• Bonuses:
  • SSL offloading
  • Firewall and DDoS Protections
  • Central control plane for traffic orchestration

Future possible directions for scaling:

• Content delivery networks (CDNs)
• Failovers
• Traffic shaping
• Draining data centers

Additional resources:

• Code (Tip: the code apparently has a very nice deployment script.)
• Scale up an app in Azure
• Get started with Autoscale in Azure
• Business continuity and disaster recovery (BCDR): Azure Paired Regions
• What is Azure Front Door Service?

Responding to and learning from failure — Emily Freeman

Failure is inevitable.

Teach people how to solve problems. It will prevent you from being woken up for everything at night.

Having to react to failure leads to:

• Unplanned work
• Context switching
• Confusion
• Increased time to repair

We tolerate a certain level of failure every day. Things that are not fine however:

• Alerts to inbox
• Alerts to a common channel where memes are combined with alerts since you’ll miss the important stuff
• Shoulder tapping
• Delays in communication
• Tiered and off-shore support
• Network Operations Center

We can borrow concepts from other disciplines. Like creating a space to communicate and collaborate. Ideally there is a single space for all communication for an incident; a space devoted to that single incident. The benefits of having such a space: you’ll have a time stamped record (which helps the post-incident review), and it focusses communication.

Train your first responders and provide them with context and clear escalation procedures.

Update your stakeholders throughout the incident.

Important when alerting is adding a severity of the incident:

• Sev1 (critical): complete outage
• Sev2 (critical): major functionality broken, revenue affected
• Sev3 (warning): minor problem
• Sev4 (warning): redundant component failure
• Sev5 (info): false alarm or unactionable alert

The life cycle of an incident:

• Detection: discovering that there is an issue. You’ll have to figure out what is wrong ASAP.
• Response: about 1/3 of your time is spent in this phase. The incident commander (IC) is the person in charge of the incident response. A communication chief (or scribe, or whatever you call this person) documents the situation for a historical record. Don’t use this to place blame, but to analyze what happened.
• Remediation: you know what went wrong and how to fix it. Always under-promise and over-deliver in your timeline. ChatOps tools might help to remediate a situation.
• Analysis: host a post-incident review to learn from what happened. The goal is not to produce artifacts, instead the goal is to maximize organizational learning. Prevent a similar incident in the future. Record counter measures and assign a date and responsible person.
• Readiness: prepare for the future.

Incidents are not linear and neither is the response.

The last two phases are often forgotten while they are the most important. Since these will prevent the same failure in the future from happening.

Additional resources:

• Code
• Rasmussen and practical drift
• Azure Logic Apps Documentation (concerning building automated workflows)
• Overview of alerts in Microsoft Azure
• Build a chat bot with the Azure Bot Service