Building pi in a World of Slop — Mario Zechner

Pi is the engine inside OpenClaw.

Mario has several issues with Claude Code. To name a few: the system prompt changes on every release, zero observability, zero model choice, almost zero extensibility. He gravitated towards OpenCode as a replacement. But Mario found some design choices he did not like.

Started developing pi. It contains four packages:

• pi-ai: a unified LLM API
• pi-tui: a terminal UI framework
• pi-agent-core: an agent loop for tool execution, validation, event streaming and message queuing
• pi-coding-agent: the CLI

Pi has four tools:

• Read
• Write
• Edit
• Bash

That’s it.

It is in YOLO mode by default (in other words: it can execute commands without asking for permission). You need to implement your own guardrails that fit for your security needs. You can ask pi to implement those for you if you want.

Pi is extensible:

• skills
• prompt templates
• themes
• extensions: tools, command, shortcuts, compaction, etc.

All of these hot-reload.

How do you build an extension? You don’t. You specify what you need, and have Pi build it.

But none of this is about pi. It’s about taking control of your own tools.

Clankers are ruining OSS. Mario auto-closes new PRs that look AI generated with the message that the user should try again. Since Clankers don’t read this, he gets rid of the bots this way. Users that indeed try again are not blocked the second time.

Agents are merchants of learned complexity

Models are trained on bad architecture decisions and cargo cult best practices. So it’s trained on garbage. The model fills in blanks in your specs with garbage from the internet.

Humans work differently since we feel pain and fix things to resolve/prevent the pain. Agents don’t learn the way we learn.

How we should work with agents:

• Scoped, so that the agent doesn’t need to load tons of code
• With a closed loop so the agent can evaluate its own work
• Nothing mission critical (instead: dashboards, debugging tools, etc)
• Work on the boring stuff or things you haven’t had time for yourself
• Reproduce cases from user issues

Most importantly:

Slow the fuck down

Think about what you are building and why. Also learn to say “no”. It’s easy to add features, but are they the right features to add? Limit generated code to what you can review. And do review every line if it’s critical code.

The Friction Is Your Judgment — Armin Ronacher and Cristina Poncela Cubeiro

Agents initially felt like an unlocked secret: you get more done. Now you have to use them or you fall behind. It is not sustainable to do reviews and have time to think. It’s addictive. We are tricked into thinking we are doing more work. But we have less time to reflect on whether we are even doing the right thing. It is hard to know when to stop.

Before agents there was a balance between creating and reviewing code. But the creation part is now amplified. The parts the engineer still has to do are not amplified. As a result reviews are skipped or rubber-stamped.

The moments where you want to skip thinking are exactly the moments where it matters most.

Agents are optimized to write code that runs. They are not as good at making an overall good design. Agents introduce more code paths and local failures. And a degrading codebase reinforces itself: the agents will create worse code.

Libraries have clearly defined problems they are solving. And they likely have a simple core. Products, on the other hand, have more interacting components: flags, permissions, billing, etc. The components are more intertwined. One of the problems with that is that the context window cannot hold the full picture.

Your codebase has become infrastructure. So you have to design it in a way the agent can read it. To have an agent-legible codebase you need to have:

• Modularization with clear boundaries so agents can work in a single area at a time
• Known patterns and conventions the agent can use for pattern-matching
• A simple core (push the complexity to layers above)
• No hidden magic. If the agent cannot see it, it cannot take it into account

Examples of mechanical enforcements:

• No bare catch-all. This forces the agent to think about error handling
• No raw SQL outside the abstraction layer to preserve the query interface
• Use components for the UI
• No dynamic imports
• Enforce unique function names so it’s easier to grep for a name
• Use erasableSyntaxOnly TypeScript mode

In pull request reviews you should separate the input going back to the agent from what needs to go to a human to make a judgement call.

We still have to go slow. It becomes harder to understand what is going on in the codebase. This makes cleanup also harder. It’s harder to judge the state of your codebase.

While we like to remove friction, some friction is useful. It makes it possible to steer the project. The friction isn’t your enemy, it’s your judgement.

Context Is the New Code — Patrick Debois

We are prompting to turn context into code. But we are also transforming code back into context in the form of skills.

Patrick is talking about the context development lifecycle today.

Generate: create & curate context

Prompting is using humans as a context engine. If you want to get advanced: create rules/instructions (AGENT.md or CLAUDE.md) to have reusable pieces of context. You can also bring in context in the form of library documentation or pull context from other places (context connectors, like MCP). But spec driven development is also a form of context.

Evaluate: test & measure context quality

We’re not yet writing evals for our code context. We could validate our skills (more or less linting). Use LLM as a judge if the generated code matches the criteria. Compare the outcome with code generated without the context in your AGENT.md to see if the context has an effect.

Once a judge gets agents and can do stuff, you basically get an end-to-end test. The tests give feedback what is working and what is missing. We can generate actions from there to improve context.

Can we run this in CI/CD? That’s hard, because it’s not deterministic. Better to run e.g. 5 or more times to see if it works most of the time.

Distribute

What if you want to reuse context in multiple projects/teams? You want to package the context in one way or another. Then the question becomes: how do you discover the skills? Via a skills marketplace. But most skills on there are crap. (It’s still useful to learn from others though.) A skill contains context, code, etc.

We’re also going to have dependencies and thus dependency hell.

We’ll need security and scan the context. (Snyk has options for this.) Who built this skill, with what model? We’ll need a skill SBOM.

Observe: monitor and improve in production

If you maintain a skill as something someone else can use, how do you get feedback how it’s working for others? Look at agent logs. Agent traces. Any feedback on a PR that it’s not correct is also feedback.

What about running code that’s running in production and was created from a context? You could e.g. use Hud

Agent sandboxing. Is code running in production doing strange things? Having a context filter is like a WAF and can prevent prompt injections. Use harness engineering.

Context is the fuel. Coding agents are the engine.

Related articles:

• CI/CD for Context in Agentic Coding: Same Pipeline, Different Rules
• The Context Development Lifecycle: Optimizing Context for AI Coding Agents
• Harness engineering: leveraging Codex in an agent-first world

(Full disclosure: Patrick works for Tessl.)

Most Enterprise Agentic Projects Are Doomed — Here’s Why — Jess Grogan-Avignon and Jack Wang

The presenters work in the world of large enterprises. A bad deployment can take down critical infrastructure. Control, process, repeatability and governance structures for human speed, not machine speed.

Only 12% of the companies are “AI Achievers”. The other 88% of them remain stuck and are falling behind.

Things that have made companies successful are now holding them back in the time of AI.

Speed

Enterprise speed is running at human speed (with practices such as security reviews, deployment process, etc). Corporations have not invested like tech companies have. An example: an application took 2 weeks to build, but 12 months to get it into production.

Approval infra hasn’t kept up with the speed at which the supply of code is growing. To go faster, every human step in the process should become executable, adaptable code.

Value

Needing a business case is not wrong per se. However, they assume scope, solution, expected value and cost to deliver are known beforehand. But when the cost to prototype drops massively, you can more easily test things that were economically impossible before.

Enterprises need to think as a VC: take a bit of risk, not everything will turn into profit. Try out things.

Delivery

Treating a scientific process like software feature delivery. Utopian design upfront with guaranteed performance and status updates. Instead of building the thing.

IT is not the problem. The team needs to upskill product managers, architects, etc around building confidence.

Trust

Completed features are not the most valuable thing you ship. There is a large trust gap. The trust you build when using an AI is the most valuable. Trust in content quality, accuracy, security, reliability.

Agent autonomy is gated by evidence in outcomes, you need to earn autonomy. Use what the user is saying to iterate. Shadow mode -> advisory mode -> controlled autonomy -> expanded autonomy. Engineer for trust, not completion.

Moat

What is unique for you? When your customer touches our product. Deployment is the starting line, not the finish line. How fast can you iterate? Continuous, compounding feedback loop.

Prescription to succeed:

• Start now: deliver differently and measure in confidence
• Make finance a transformation partner, not a gatekeeper
• Make governance speed an engineering problem
• Redefine your moat as what you compound from today

The Domain-Native AI Organization: How to Leverage Domain Expertise — Chris Lovejoy

We often do not have deep understanding about the processes we are automating. Use domain experts as oracle, evaluator or architect.

• Oracle: directly adds expertise
• Evaluator: define and measure quality
• Architect: build self-improving systems

Most common mistakes:

• Not hiring domain experts (or too late)
• Wrong kind of domain expert
• Not fitting them in the organization properly

Do you need domain experts? Yes! Appraising AI quality requires judgement. And judgement requires domain expertise.

CI/CD Is Dead, Agents Need Continuous Compute and Computers — Hugo Santos and Madison Faulkner

Agentic software is breaking traditional CI/CD. Why is CI/CD dead? At agent scale you have more PRs and more repos. But it still takes the same time to review and verify. Merging all different versions together becomes impossible.

Machine latency in the CI/CD pipeline was hidden behind the ‘slow’ humans. With agents the pain points become clear.

Today we have the human in the loop: we validate the PR. So each time a PR is rejected the agentic software can update the PR fairly quickly only to be blocked again by the human. But since this means there are only so many changes going on, we have a relatively big window to get your stuff merged.

The PR as a unit of work was designed for humans. CI matters because it validates your work. It also facilitates coordination.

Hugo explains a loop we have today: intent + plan (what are you doing) goes into an agent harness loop. The agent will check out your code. Then goes to internal validation to make sure the change is correct. It then reports back to the human for external validation (does it look good?). When done, go to merge queue. This is fast, but not fast enough because there is a human in the loop.

Tomorrow the human will be out of the loop. We’ll have all kinds of LLMs (e.g. one with a security focus, another one with a different focus, etc) to do the external validation. The human only needs to approve a change once it’s in the pre merge queue. So the human is still gatekeeping, but later in the process.

Software Engineering Is Becoming Plan and Review — Louis Knight-Webb

What are we even going to do in the new AI era?

Work humans do:

• Plan
• Write code
• Review my code
• Review other people’s code

If we look at 2021, most time was spent writing code and reviewing other people’s code. If we look at 2025, there is little actual coding; most time is spent reviewing other people’s code.

So work got displaced. The time we no longer spend on coding (because the AI is doing that now) mostly changed into planning and reviewing code.

There are basically two modes to work in:

• Spend a lot of time planning (create a comprehensive plan doc, interrogation, etc). You spend more time planning, but the coding agent can run longer and there’s less time needed for review.
• Spend a lot of time reviewing. Loosely define prompts, at the cost of more manual QA work. If you spend less time planning, the coding agent yields results faster, but there’s more back and forth with agent delivering half baked work.

The mode you’ll want to use depends on the type of work. Plan heavy mode works well for refactoring/migration. For new features heavy planning works for backend tasks, but for the frontend the review heavy mode is a better match.

Spending 5 mins of planning saves you 30 minutes of time reviewing code.

The time an agent can run before human intervention is needed has increased over time. This is good: you want to minimize the time you are spending with the AI. Most of the time the back and forth is done by the agent itself.

When the agent takes longer than 5 minutes, waiting on the AI (and slacking off) is not realistic. So when agents run longer and longer, you need to change your way of working. One option is parallelism (have multiple agents working at the same time) Vibe Kanban is a tool to help you with that approach.

What should the future look like to help the human?

• Focusmaxxing: embrace the fact that you cannot context switch every 30 seconds, instead build to get the most out of the humans
• Write tasks
• QA (websites, APIs)
• Code review
• Shepherd the change until it’s deployed

Then the talk took another turn.

Recently Louis decided to shut down the company behind Vibe Kanban. On stage he instructed his agent to write the Goodbye bloop blog post (with a prompt he had prepared beforehand). Don’t worry though, the project will continue as open source and be maintained by the community.

How Building with AI Can Double the Throughput of Your Engineering Team — Brian Scanlan

Brian works for Intercom. They are the company behind Fin.

Change is hard. You need clear and executive guidance. How to enable change:

• Update job descriptions and expectations
• Constantly talk about the urgency of AI adoption
• Reward great work (financially, socially and publicly)
• Give people room to learn, enable them and give them access to tools
• Be very specific about what you want to see and how it is to be done

Standardizing on a single, skill driven AI platform helps. Prove that it works, optimize its usage. Connect it to everything. Anything the human can do on their laptop, the agent should also be able to do. (You’ll need to be in control of your environment to make sure it doesn’t do anything bad though!) Start using the platform for all technical work. It will make mistakes initially, but it will become a flywheel where it will become more powerful over time.

Engineering is changing. The engineers focus their time on writing specs, validation and improving the agents. The agents write, test and review code.

Internal tools at his company are deprecated in favour of first-class vendor replacements (like Anthropic/Claude Code).

Give agents problems, not tasks

Agents should figure out the necessary tasks on their own. They focus on durable, high quality, sharable skills.

Current bottleneck: code review.

Intercom has extensive feedback loops via lots of hooks to Honeycomb. They measure things like skill invocations, failures, etc.

A side-effect of using AI more was that their defect rate is going down. This wasn’t a goal, but a natural consequence.

Relevant link:

• Brian’s website

Agents Don’t Do Standups: Building the Post-Engineer Engineering Org — Mike Spitz

From “how do we help engineers output more?” to “how do we make agents faster?”

Scrum did not survive

Rituals designed for humans don’t work for agents. Ceremonies became huddle sessions.

Specs are turned into LDDs (lightweight design documents) which become tickets and PRs.

How do you start?

1. Pick the engineers with development and broad system knowledge
2. Scale slowly!
3. Experiment in non-critical systems

Also keep in mind that:

Not everyone can drive a sports car

It’s going to be hard for a few engineers. The curious engineer will smash this.

Some guardrails:

1. Verifiable deterministic tasks (unit tests, e2e tests, linters, PR prerequisites)
2. Agentic code review (human steering, agentic review; opinionated comments are easy to offload)
3. Tiered human in the loop (heavy human review at system design, light review at code (except security), heavy review at end for product feel)

Prerequisites for autonomous loop:

1. Composable skills (all parts of development are abstracted into composable skills)
2. Agent-involved stages (agent involved at every stage: spec, LDD, ticket/branch/PR creation, self-testing, self-QA)
3. Self-healing agents
4. Human multipliers (allow humans to parallelize)

What do the humans do?

1. Security (ensure no shortcuts were taken by the AI)
2. Product feel
3. Scale & engineering complexity for task (Are we spending tokens on work we don’t need to do? Is the agent over-engineering?)

The playbook to get started:

1. Start with boring, repetitive tasks
2. Remove as much redundancy from the process as possible
3. Make sure the good patterns are turned into skills
4. Build guardrails before autonomy
5. Build this with your best engineers
6. Do not onboard everyone all at once
7. Do not try to create a “one size fits all” approach
8. Do not be conservative, otherwise you’ll get behind (compounding effect)
9. Do not try to do too much at once, you want a phased approach

The New Application Layer — Malte Ubl

AI is changing how we build and what we build. Is there a place for the software engineer in the future?

Agents are a new kind of software. They are both the builders and users of software. They allow us to automate things that were economically unviable in the past. But with AI, this changes. Lots of software was too expensive to write (compared to the benefits it would bring). Also, companies are making different decisions with regard to buying from SaaS companies versus making it themselves. The cheaper the software is to make, the more software there will be. This leads to more work for software engineers.

As AI Engineers, our job is to build the next application layer: agents.

Archetypes of practical non-coding agents:

• Always running (24/7)
• Compresses the research (in the chain of business event -> research -> human decision, we can have the AI do the research)
• Surfaces the hidden information (lots of info, but you cannot practically use it)
• The boring things (more time for the human for the interesting tasks)

We also need to realise that software is for agents now. For example, 60% of the visitors of the vercel.com page are agents.

Not writing the code yourself also makes us less opinionated about the infra. It “just has to run”.

We should also have agentic security infrastructure. We need to have an open mindset for how to change things.

The new application layer can thrive independently of models. Now model A is the “best” today, but tomorrow it’s model B. We as engineers should provide a stable interface.

Europe is the leader in AI Engineering. Not models though. But the model companies are commoditising. The application layer is where the real innovation happens.

Harness Engineering: How to Build Software When Humans Steer and Agents Execute — Ryan Lopopolo

Use the models to do the full job. Lean into the idea that the models are software engineers.

Code is free

Hiring the “hands on the keyboard” is constrained by token budgets nowadays. We need to constructively use this capacity. The human skill sets that are now needed are delegation and systems design.

The models are good enough. Code is free (to produce, maintain and refactor). Your role is to unblock your team and your team is infinitely large. Your job is to make use of that team.

Human time and attention is scarce. How do we effectively use it? When we are no longer blocked by this, we can work on the low(er) priority issues. We have infinite coding resources, right? Humans don’t need to concern themselves with implementation, but specs and guardrails. Our job is to build systems and structures to make our teams effective.

What does it mean to do a good job? Used to be years of experience in the field. Lots of little decisions in everyday work. But agents have seen more code than we have seen. We need to write down the non-functional requirements so that the agents can use this. Figure out what the agents are struggling with. Put guardrails in place to guide the agents. Move on to higher level tasks.

Having a single QA expert in the team who can write a good QA plan can benefit the whole team. Same goes for other experts: one engineer has more impact on the whole team than before.

How can we help the agents to make the correct decisions?

• Continuously run review agents
• Check if the code has a secure interface
• “Lint” for non-functional requirements
• Figure out why we are spending time on correcting agents and fix the problem so you can move on
• Adapt your codebase to match the world today, e.g. limit file size so they fit in the context window
• Have good error messages with resolution steps. This helps the agent resolve issues itself

Everything is a prompt: rules, skills, error messages, PR comments, et cetera.

Just build things. Do not hesitate to have the agents do the full job.

Why building eval platforms is hard — Phil Hetzel

Evals and observability are related. Evals is what you do before hitting production, observability is what you do when in production.

Why are evals important?

• LLMs have extreme variability (we love them for it though)
• Agents are becoming the norm, people have come to expect them when interacting with your company
• You need to be confident with the agent’s performance

Evals are not a hard problem: gather a bunch of example inputs, loop through them with the agent and publish the result. Right? Actually it’s way more complex. Think about tracing, alerting, online scoring, topic modeling, etc.

Different stages of doing evals:

1. A spreadsheet plus a for loop. It’s a great place to start, no barrier to entry. However, it’s more about documenting and not really experimenting, and it is hard to compare experiments. It’s also a cumbersome process.
2. Vibe coded UI. Nice next step, probably has proper persistence (database). It’s a bit more bespoke to bring others into the fold. But it’s still more of a reporting tool.
3. Encouraging experimentation. Give a user access to an agent configuration plus a sandbox. Allow the users to tweak prompts and parameters. (He demonstrated with Braintrust). Still no access to production traces.
4. “The Flywheel”. This is where observability and evals are connecting. Understanding actual user behaviour. This unlocks the feedback loop. Downside: you now have to manage the eval platform, at the pace the industry is moving. More importantly: agent traces are nasty (and not like normal application traces), very large and numerous.

This is a new problem because of the specifics of traces: larger spans, highly unstructured, difference in read patterns. And while these aspects individually are not unique problems, the combination of them is new.

Building the right system: specific for agent traces with multipersona workflows. This allows you to measure agent quality at scale, delivering near real time feedback.

Looking forward:

• Surface unknown-unknowns via topic modelling techniques
• Build the platform for users and agents
• Perform observability via an AI proxy or gateway

What Breaks When You Build AI Under Sovereignty Constraints — Bilge Yücel

Sovereign AI is “the ability of an organisation to design, deploy and operate AI systems on its own terms.” In a practical sense: this means having explicit control over data flow, model choices, infrastructure, observability and operations.

The pillars of sovereignty:

Data sovereignty

Data should be stored in “trusted jurisdictions”. If you send your data to a model running in the US, you may already not be compliant anymore.

Infrastructure sovereignty

Maximal control: airgapped (EU AI act safe). Maximal convenience: SaaS (CLOUD Act risk)

Model sovereignty

You do not want to tightly couple with a specific model provider. You want to be able to swap without architectural changes.

Operational sovereignty

Monitor how AI systems behave, have the human in the loop, manage versioning and updates to models and applications.

Sovereignty is a spectrum. Some sectors need more sovereignty (finance, healthcare); some need less (e.g. startups).

What are the engineering challenges? What do you do and what do you break?

• Replace the frontier API with a self hosted model. Consequence: translate API logic
• Private data to required jurisdiction. Consequence: managing multiple databases and instances.
• Replace managed infra with on-prem. Consequence: you discover vendor lock-in and are limited by your hardware.
• Incorporate observability and tracing. Consequence: you have to do version control of the whole system.

Haystack solves some of these problems. It has:

• A consistent interface
• Explicit data flow (know what data was where)
• Serializable to YAML so easy to version
• No black box or hidden assumptions because it is open source

(Full disclosure: Haystack is a product from Deepset, the company Bilge works for.)

Architecture:

• Guardrails (check input)
• Agent (with LLM)
• Guardrails (prevent leaking info)

Sovereignty checklist:

• Can you swap your models without changing the application logic?
• Do you have reproducible run logs stored in a compliant way?
• Can your team respond to an incident without needing the help of the hyperscalers?

Related blog post: AI Guardrails: Content Moderation and Safety with Open Language Models

Software Engineering + AI = ? — Gergely Orosz and swyx

Gergely is the man behind The Pragmatic Engineer.

What does Gergely think about tokenmaxxing? It’s happening at multiple large companies. Token output is measured and e.g. shown on a leaderboard. This results in uncertainty: people start to think performance is also measured by token usage. Token count can be weaponized. The result: people start burning tokens by, for example, asking the AI instead of reading the docs just to get the token count up, even if the AI doesn’t do a good job of answering the question. Some companies even have a minimum token count. It’s a weird time to live in.

Is AI still making us faster? Experienced engineers were holding off, especially the older models on existing codebases. Targeting and measuring token usage came from the C-suite. They had the idea that if their company was not using AI tools they are probably not doing well. There’s a push to use AI.

Why are engineers putting up with that? For the same reason as we have weird leetcode interviews: they select for smart people willing to put up with a bullshit process to get the job. Big tech is a bit weird.

Individually, AI makes us go faster. For teams? Not always. It is hard to retrofit in some situations. An engineer is perhaps not much more productive. But enabling non-coding colleagues with AI unlocks the ability for them to not have to wait for dev but get things done themselves.

These times are hard to understand for us engineers:

• Using AI takes a long time to get good at it
• Knowing the theory behind it and how it works will not make you more productive

How is the role of a software engineer changing? Before AI the role was already changing. AI just sped it up. Startups were already moving fast with smaller teams. More roles collapse into the role of a software engineer (think of DevOps, QA). Now also product management. Software engineers need to know about the business, and take more responsibility.

“Everyone is an engineering manager now” (no longer a software engineer). This is absolute nonsense. Traditional managers take care of people problems, their growth path, etc. But with AI you don’t have to worry about a person. It’s more a tech lead role. You can do more and faster, but are still in control. It’s only orchestration, not management. With agents you have a faster feedback loop than with humans.

E.g. Uber is not building more features into their core product. Instead they are working on infra and people are building more. It’s a low risk way to be hands on with AI and learn. But also the codebase is too big for the context window, building separate things is easier. Plus: anything related to AI gets funded faster.

Taking AI infra seriously is not well understood. We’re learning from each other.

Keynote — Kitze

This talk was fast paced and very enjoyable. I can recommend watching it. (See the AIE Europe Day 1: Keynotes … starting at 8:08:46)

It Ain’t Broke: Why Software Fundamentals Matter More Than Ever — Matt Pocock

With the new paradigm (AI), we should probably chuck out the old rules to make room for new ones, right? There’s a “specs to code” movement where you write the spec and have the AI do the coding. You don’t look at the code. If there is a problem with the code, you go back to the spec and try again. This is driven by the idea that code is cheap.

But ignoring the code doesn’t work. A bad codebase is one that is hard to change. The Pragmatic Programmer has a chapter on software entropy. It basically means that if you do not pay enough attention to the design of the whole system, the codebase will become worse and worse. And this was exactly what he was seeing when he was running the compiler again and again when using specs to code.

Bad code is the most expensive it has ever been. Good codebases matter more than ever.

Software fundamentals matter more than ever

Common failure modes and how to avoid them by going back to old software practices:

The AI didn’t do what I wanted

No-one knows exactly what they want. There is a communication barrier between you and the AI. The Design Of Design speaks about “the design concept”: the idea of what you are building. You and the AI don’t share a design concept. Hence the /grill-me skill. Works towards a shared understanding. Use his other skills to generate a Product Requirements Document or issues. (See Matt’s skills repository for the skills mentioned in this talk.)

The AI is way too verbose

You also see that in the interaction between a developer and a domain expert. You need to establish shared language. Domain Driven Design also needs a ubiquitous language. In our case this is essentially a Markdown file with concepts and what they mean. Use the /ubiquitous-language skill. This generates a file (UBIQUITOUS_LANGUAGE.md) with tables with terminology.

Code that doesn’t work

Use feedback loops: static types, browser access to look around, automated tests.

Doing way too much

The Pragmatic Programmer calls this “outrunning your headlights”. The rate of feedback is your speed limit. Related skill /tdd: write test, make test pass. However, testing is hard (how big of a unit do you want to test, what to mock, what behaviour to test, etc). Good codebases are easy to test. (Because of better feedback loops and more clear boundaries.) Use deep modules: few modules with lots of functionality but simple interfaces.

AI does not understand my code

Again, use deep modules. Easier for the AI to understand the design. How to go from shallow to deep modules? Use the /improve-codebase-architecture skill.

My brain hurts

The human cannot keep up. Again: deep modules make it simpler for you to understand the codebase. You can treat these modules as grey boxes. The AI can handle what’s in the blob. Design the interface but delegate the implementation.

Code is not cheap, it is important. If we think of AI as a programmer, you need someone above that, thinking on the strategic level. And that’s you! And that’s why fundamental software development skills are still important.

How to Build Agents That Run for Hours (Without Losing the Plot) — Ash Prabaker and Andrew Wilson

Why are agents losing the plot?

• Context: the agent can’t carry state (and has “context anxiety” if the context is getting filled up)
• Planning: general models are not great at planning (e.g. running out of context)
• Verification: models are bad at evaluating their own output (it thinks it’s done)

There are two ways to fix this:

• Train the model
• Wrap the model in a harness

Anthropic’s new models were also combined with harness improvements last year. Every release of Claude could run longer unattended.

Harness design for long-running agents. Building the generator/evaluation loop, and then deleting half of it when the model caught up. Splitting up the generator (which builds the thing) from the evaluator (which grades the thing). Most people now use the same instance to build and evaluate.

Tuning a standalone evaluator to be skeptical is tractable. Making a generator self-critical is not.

Added one more role: plan (a 1-line prompt to full spec). This is the input for the generator. If you squint a bit, you’ll notice that this mimics the real world where we have a product manager (the plan role), an individual contributor (the generator) and a QA person (the evaluator).

Before any code gets written, the generator and evaluator negotiate what “done” looks like for this chunk. They iterate via files until they agree (one agent writes, the other reads and responds). The agents agree on a contract. This bridges user stories to testable behaviour.

Ash presented an example where a solo agent built a game vs building a game with a full harness. The solo agent was done in 20 minutes, and the full harness took 6 hours. But the result was significantly better and more thought through.

Out of the box, Claude is a poor QA agent. It would find a bug and then decide itself that it wasn’t a big deal and approve the work anyway.

With Opus 4.6 half of what was described about harnesses became obsolete. The new model needs less scaffolding and the harness can be less complex.

Takeaways:

1. Use an adversarial evaluator, self-evaluation is a trap
2. Structured handoffs are better than compaction
3. Make subjective quality gradable with rubrics the model can apply
4. Read the traces as they are your primary debugging loop
5. Delete scaffolding where the model catches up

The models you are using (e.g. Opus for planning and Sonnet for building) influence the harness. The harness patterns that work tend to get absorbed back into the tools. Most of the loop described is buildable in Claude Code right now with primitives that are already available.

Further reading on the Anthropic Engineering blog:

• Effective harnesses for long-running agents
• Harness design for long-running application development

Building Your Own Software Factory — Eric Zakariasson

Eric spoke about his experience using Cursor to build a software factory. Some parts are running autonomously. It’s hard work. These are his observations.

There are several levels of using AI. At the lowest level you have autocomplete. Then you progress to a coding intern, a junior dev, a developer (where the majority of code is written by the AI tool) to a senior developer and finally a software factory where the AI is a black box and operates like a dark factory.

Why would you want a factory:

• Throughput (24/7, machines do not need sleep)
• Consistent output (like an actual factory, but with AI there is a risk of losing determinism)
• Leverage your taste better

What do you need to build a factory?

• Primitives & patterns (co-located code and usage patterns)
• Guardrails (you want to let the agents free, but not too free, think about rules, hooks and tests)
• Enablers (what can you allow the agents to do to be more free, think about skills, MCPs)

Rules should be created dynamically: if you see an agent doing something you don’t like, write a rule. Note that agents will behave better over time with newer models, so rules may become obsolete.

Other aspects you’ll need:

• Runnable: can the agent start the env?
• Accessible: can the agent use the tools it needs, e.g. Datadog?
• Verifiable: can the agent perform check itself?

For each of the different stages of the software development lifecycle you will need an agent. For example to review changes and automated testing.

You need to shift your way of working:

• You’ll write less code yourself (if any) but are managing your agents
• You are also going from sync to async
• You need to think more about scope and parallelising work (e.g. running some tasks in parallel will guarantee merge conflicts while other tasks do not interfere with each other).
• You still need to know how data flows and what users want
• You’ll want to identify the human in the loop. Is there e.g. a copy/paste action the user is doing now? Automate that away.

Since agents will run for longer periods, you need to trust them more. You get to know the agents: their weaknesses and their strengths and how to prompt them.

When having agents work in parallel, Eric is using separate environments (even in different VMs) to have reproducible and isolated environments without side effects from other branches and ongoing work. This will take more effort to setup, but once you are there, it is easier to scale up the number of agents working on your code.

You need to keep an eye out for where the agents go off the rails. Use this information to improve the factory.

Now how do you go from 5 agents to 100? Same as before: observe the outcome. And:

• Make sure the agents can verify their own work
• Setup automations. Examples:
  • Eric demonstrated asking the agent what actions he does frequently so he can work on automating those
  • Review the comments made on PR reviews so the agents can learn from them
• Move up abstractions

The takeaways:

• Be clear about your intent: what problem are you solving?
• Stay in the loop for important decisions (e.g. which payment system to use, etc)
• Build systems and tools: codify them and give your agents access to them
• Store context for later and keep it up-to-date (since it will evolve)
• Let the agents be free (one team had even given the agents a place to complain and that proved to be very useful)

Build Your Own Deep Research Agent + Technical Writer — Louis-François Bouchard, Paul Iusztin and Samridhi Vaid

The team built a multi-agent pipeline to replace the research and technical writing process. They give a topic and it will write a technical article (without slop or hallucinations). It targets short content, like LinkedIn posts.

The GitHub repo of their project: iusztinpaul/designing-real-world-ai-agents-workshop

Constraints:

• Costs per task
• Latency
• Quality
• Compliance & data privacy

There’s a scale from simple prompts (where you have more control and less costs) via workflows to single agent to multiple agents (where you have more autonomy, and thus less control, and higher costs). It’s best to always use the most simple solution. For example: if the context is known at or before query time and has less than 200K tokens, a simple prompt can suffice, using Context Augmented Generation (CAG).

In a situation where the context is not known beforehand (e.g. because it is private or too recent), you might benefit from including a workflow. (A workflow is a sequence of fixed steps, with the same steps in the same order each time). Think about using Retrieval-Augmented Generation (RAG).

The next step is when you need the system to take autonomous actions or you need dynamic behaviour. Then you get to agents, which can react to what is happening. These agents can use also tools, which can have their own:

• System prompt
• Validation logic
• LLMs

Tools are specialists, but with one shared decision maker which has a global context. Delegation to tools helps with context management. The tools can have their own context windows.

AI products are never just agents, simple workflows or LLM calls and tools. They combine all of them. AI engineers need to understand how to build these complex systems. And deep research systems are a perfect project on how to learn these complex, multi-agent systems.

The MCP server they built (see their GitHub repo):

• Tools: actions the agent can do
• Prompts: instructions the agent can follow
• Resource: data the agent can read

Why both skills and MCP? They are moving to using skills more, but those cannot replace the MCP server completely because some tools are too complex to be turned into skills.

LinkedIn post generation:

• Guidelines: what to write about (topic, angle, etc)
• Profiles: how to write (structure, terminology, character of the post)
• Research

Debugging workflows/agents purely through logs is hard. You want traces (LLM/tool calls with full I/O + metadata), latency and cost tracking. They used Opik for observability.

You also want to automate evals. Generating one post allows for manual review, but when scaling to 100 posts, it’s quite impossible to manually review each one. One small change could break something completely, so you need to review.

They used a three layer architecture:

• Optimization
• Regression testing (evals in CI/CD)
• Production monitoring (using Opik)

They encourage us to run their project ourselves and reading the code to understand the details of what’s going on.

AI Coding For Real Engineers — Matt Pocock

To follow the workshop along, see Matt’s workshop at aihero.dev/s/ai-2026.

Before I begin with my notes: this session is definitely worth watching (again) once it’s available on YouTube! Matt is an excellent teacher, has great energy and great content as well.

Once there are about 100K tokens in the context, the AI starts to get dumber and making increasingly dumb decisions. We don’t want the AI to bite off more than it can chew, so keep your tasks small.

Even with the 1M context window of Claude the “smart zone” is still around 100K. Claude basically just expanded the dumb zone. Good for retrieval, less good for coding.

So how do you tackle big tasks? Multi-phase plans are a common solution. It’s basically a loop. This is where the Ralph Wiggum loop comes from. Matt likes something smarter though.

Every session starts with a system prompt. If you have 200K tokens in here already, you are in the “dumb zone” from the start. To stay in the “smart zone” you can clear the context. This does mean that you lose everything that happened after the system prompt. Alternatively, you can compact.

If you show the number of tokens in the status line of Claude Code (or whatever tool you are using), you know how close to the “dumb zone” you are. When using /compact it squeezes all information. The downside of using /compact over /clear: with the latter you have a deterministic state.

The /grill-me skill (source) is a really nice way of taking inputs from the world. It can interview the user relentlessly about a plan or design until reaching shared understanding (according to the skill itself). Ideally you use the /grill-me skill with both the developer and the domain expert in the room.

After the /grill-me skill, you want to summarize all those valuable tokens into a Product Requirements Document (PRD). This is the definition of done for your agent. You can use the /write-a-prd skill (source) to write this document. Note that there are testing decisions in there too. These are important!

Matt explains that he does not actually read the PRDs that are generated. In the grilling sessions he makes sure he and the AI are on the same wavelength so there’s no need to review the resulting document. Why would he? Doing so would basically only test the LLM’s ability to summarize.

Should he optimize the plan? Matt doesn’t think optimizing the plan to death adds a lot of value. Things will change afterwards anyway.

To see examples of a PRD and issues generated from it, check his course-video-manager repository, in particular the closed issues.

Now that we have our destination, how do we split it? Matt likes creating a Kanban board out of it. He created a skill to do this: /prd-to-issues (source).

As you’ll see in that skill, Matt instructs the AI to use vertical slices. LLMs love to code horizontally, so per layer (database, business logic, frontend). This means you don’t get feedback on your work until all layers are done. If you slice vertical layers, you can test the entire flow sooner. (Also known as “tracer bullets” if you’ve read The Pragmatic Programmer.)

With the Kanban board setup, it’s easier to parallelize working on tasks. And once we have the issues that can be worked on, the human can step out of the loop.

For implementation use the /tdd skill (source). It does red/green refactors: write failing test first and then make it succeed. This not only adds (good) tests to the codebase, but starting with the tests it’s harder to ‘cheat’ with writing the tests (you cannot write the tests to match the implementation because the latter does not exist yet).

How do you conform with existing architecture, coding standards, API design, constraints, etc?

• Push instructions to the LLM (e.g. in CLAUDE.md)
• Pull: give the agent an opportunity to collect info, e.g. via skills.

For the implementer you should use the pull strategy so it can pull what it needs. For the reviewer use push (these are our standards, make sure they are adhered to).

You absolutely need to have feedback loops for the AI. The quality of your feedback loops directly affect the quality of the output.

As you may have noticed, there are two types of work when building something:

• Human in the loop (HITL) tasks (like planning) which need the human
• Away from keyboard (AFK) tasks (like implementation) where the AI can work autonomously

To recap the process thus far: you have an idea, you have the grilling session with the AI and this results in a PRD, which gets turned into issues on a Kanban board. These are HITL steps. Now the AI can take over and handle implementation where one or more agents work (the night shift so to speak). Once the AI is done, the human steps back in the loop for the QA/review step.

And yes, we do need code review. There is no way to avoid this. If we delegate coding to the agent in small PRs, we have to review more code. Matt doesn’t feel good saying that, but it’s his honest answer. The QA step is also where you can impose your opinions on the agent. Note that in the QA phase we also create more issues on the Kanban board.

You can and should have an automated review step though. Only QA manually afterwards. But be careful that the automated review isn’t done in the “dumb zone”, you want to review in the smart zone.

Frontend in particular is tricky. It needs human eyes. AI is not very good at that yet. But you can ask it to create a couple of prototypes to trigger a feedback loop with the agent.

So how does this work in a team? You involve the team in all HITL steps. And while the idea, research, prototype steps look linear, in the messy world you’ll bounce back and forth between those phases.

What if you have a bad, complicated codebase that even humans do a bad job in? How do you improve that? If your files are “shallow modules” (small files with little functionality), it’s hard for AI to navigate. It has to manually track through the repo. Also hard to draw test boundaries. Hard to test interaction between modules. Should the tests mock other modules?

Building a codebase that is easy to test is essential, because the feedback loop is better. “Deep modules” are better; these are modules with more functionality. Dependencies are more clear. So how do you go from a bad codebase to a good one? How to group modules? The /improve-codebase-architecture skill (source) will find places to deepen the modules.

(The concept of shallow/deep modules come from the book A Philosophy of Software Design discusses dependencies.)

If you take only one thing away from today: use the /improve-codebase-architecture skill

You need to have (enough) control over the thing to be able to fix it. The PRD contains which modules are updated, it also helps you keep in control of the beast. Because we delegate more, we lose sense of our codebase. By building deep modules (big shapes), it’s easier to have their mental models in your mind. You don’t need to code review all details in a module, you only need to make sure the shape does what it needs to do.

Code is important, so understanding the tools deeply make you a better developer and you’ll get more out of AI.

When using plan mode, you can tell in CLAUDE.md to be terse (“when talking to me, sacrifice grammar for the sake of concision”). This helps when reading the plans. But Matt dropped this in favour of the grilling session where he and the LLM came to the same shared understanding and he no longer needed to read the plans.

Does he keep the markdown plans for future reference? No clear answer. Matt is wary of outdated documentation (names and requirements have changed). He tends to get rid of the plans and marking the issues as closed.

What does he think of Beads from Steve Yegge? It’s another way to manage Kanban boards.

Sidenote: Sandcastle (also created by Matt) is an orchestrator. It takes Ralph loop from sequential to parallel.

Matt is not selling a way of working. He does recommend buying old programming books (pre AI) since they contain a lot of wisdom that is still applicable.

PyGrunn is a Python focussed, one day conference held in Groningen¹. Or as the organizers more eloquently phrase this:

PyGrunn is the “Python and friends” developer conference with a local footprint and global mindset. Firmly rooted in the open source culture, it aims to provide the leaders in advanced internet technologies a platform to inform, inspire and impress their peers.

Before I start with my notes I want to give a shout-out to Reinout van Rees. His (PyGrunn) summaries are excellent. I’m always impressed by the quality of them and how little time he needs to write them. Where I’m only able to make notes and need to write them out afterwards, Reinout has the summary (as a coherent story) ready before the speaker has unhooked their laptop. So if you are interested in one of the talks he has attended, head over there first.

Platform Engineering: Python Perspective — Andrii Mishkovskyi

Why do platform engineering teams exist? We’ve seen a “shift left” of responsibilities towards the developer e.g. QA, operations (DevOps). But we (software developers) are trained to write code, not to e.g. monitor it.

So where does a humble developer start? There is an abundance of choices on the tools to use. What do you pick for package management? Or continuous integration? Or deployment, code quality, observability, etc. This freedom of choice comes with a cost. We start with discussing which tools to use instead of the problem the customer is facing. And depending on which choice we make, the result may make reasoning about the software more complex.

So what should you do?

Andrii broke it down into three parts:

• You observe (i.e. you read a lot to get the lay of the land)
• You execute (this is the actual software development part)
• And then you collect feedback (you reach out to teams, you observe how their work has changed)

Some of the platform engineering team deliverables:

• Documentation
• Self service portal (tip: look into Backstage)
• Boilerplates
• APIs

The “consumers” are developers, compliance teams and other platform teams. The goal is to:

• Have reasonable defaults
• Remove redundancy
• Keep things consistent

To get a feel for the scale of things: at Andrii’s company there are over 160 services, developed on by 300+ developers in 500+ repositories. They total up to 3 million lines of code and 6 million lines of YAML.

Templates (boilerplates) provide a paved path. The goal is to have teams spend as little time as possible when starting a project. The templates use a certain set tools that are supported. Teams are free to use different tools though if they want to.

Andrii uses cookiecutter templates at work. It’s not his choice perse, but it’s what was already in place when he joined. There are currently three templates in use. They have evolved over time. For example in the last nine years, over 800 changes have been made (that is more than one change per week on average).

The evolution has left its marks: the templates have a lot of code duplication. There is also code specific to a tiny minority of projects (only about 8 out of the 500+). This means that most projects start with deleting code after using the boilerplate.

And that also relates the downside of using cookiecutter the way they do. Instead of just using it to get started with a project, they also use it incrementally. But cookiecutter does not have versioning built-in. So if you remove a file and then reapply cookiecutter again, the file is happily created again.

While Andrii is aware of the issues (and thinks copier might be a better alternative), it is hard to replace practices that are already in use. And cookiecutter is great for getting started with a project.

With regard to the standardization, they use the following in the templates:

• pyproject.toml for all projects
• Poetry (instead of setuptools + pip-tools)
• sprinkle Renovate on top for automatically updating dependencies

As it currently stands, there are 99 project that migrated to pyproject.toml and Poetry in the last 2 years. It makes sense because it takes time for projects to transition. Plus they are not required to migrate; again: the template are there to help, not to limit the users. Renovate has been adopted more quickly.

Migrating from e.g. pkg_resources to pkgutil or PEP-420 for namespaces packages is hard. Templates can help with that. However, cookiecutter does not actually manage files. So if a file has been removed from a template, rerunning cookiecutter does not remove the file from the project. So that requires some care.

When they migrated from a monolithic application to a microservices architecture, authentication/authorization became an issue. There was no visibility for teams, no transparency and no accountability. To combat this, they created an API where applications can declare the required access and scopes in a YAML file. Maintainers can approve this access. And this also allows for CI/CD to check access. A CLI tool can verify the validity of the YAML and check if access is actually approved.

Securing your team, solution and company to embrace chaos — Edzo Botjes

Edzo started by sharing a link to his slides and warned us that it usually takes two weeks for people to digest the contents of his talk. He would overload us with information. And that’s where I decided to solely concentrate on the talk and not on note taking.

Nobody knows what they are doing. Embrace this.

Even a simple, deterministic system like a double pendulum has a nondeterministic outcome. In other (my) words: the whole world is in chaos and unpredictable.

When presented with information everyone processes it differently and understands something else (also see viral phenomenon of the dress).

What worked for Edzo was to embrace chaos. To do this he let go of his desire to control and trying to create a predictable outcome.

Descriptors: Decoding the Magic — Alex Dijkstra

Many people have used descriptors without them even being aware of it.

From the documentation:

You can view descriptors as reusable @properties. A descriptor implements the __get__ and __set__ methods (and when needed __delete__).

Alex showed a bunch of examples. This is the template he showed to introduce descriptors:

class MyDescriptor:

    def __get__(self, obj, owner):
        # owner is the class to which the instance belongs
        return obj.__dict__.get(self.private_name)

    def __set__(self, obj, val):
        # self is descriptor instance.
        obj.__dict__[self.private_name] = val

    def __set_name__(self, owner, name):
        self.public_name = name
        self.private_name = f'_{name}'

His example of how to use this:

class MyClass:
    value = MyDescriptor()


myinstance = MyClass()
myinstance.value = 5  # MyDescriptor.__set__
myinstance.value      # MyDescriptor.__get__

Using descriptors you can do things when getting or setting the value. E.g. in a class you can enforce that an attribute has a certain type.

The __set_name__ method was introduced in Python 3.6. It is not needed to use this in all of your descriptors, but also doesn’t hurt.

Do you want to use descriptors all over the place? No. They can be useful: you can create a clean APIs with them and this helps if the API is used frequently. However, it does create some overhead and the code is a bit more complex.

Resources:

• https://docs.python.org/3/howto/descriptor.html
• Luciano Ramalho’s book Fluent Python (Luciano also did a few talks about descriptors)

Release the KrakenD — Erik-Jan Blanksma

KrakenD is an API gateway product. Erik-Jan likes it so much he wanted to share his experience with it.

Projects can start out simple, with a monolith that is accessed via a web client. Before you know it, there are several services and multiple types of clients. The solution is to introduce an API Gateway in the middle. It can then handle the incoming requests.

API Gateway in short:

• It sits between clients and backend (as a sort of portal).
• It hides internal complexity of backend for the clients.
• The gateway is a great place to introduce things like authorization/authentication, logging, load balancing, caching, etc.

KrakenD is one of the available API Gateways. It is open source, but there’s also an enterprise version with extra features. KrakenD is implemented in Go and offers a bunch of features out of the box (monitoring, throttling, request and response manipulation). KrakenD offers integrations with e.g. tools (Jaeger, Grafana, the Elastic stack), authorization/authentication services and queues (RabbitMQ).

KrakenD is a stateless process, so no database is needed. It takes JSON (or YAML) config. It can combine the results of multiple backends API calls and return it as a single response.

Tips:

• Use the KrakenDesigner (makes it easy to explore what’s possible). Note that you do not want to use this in production.
• You’ll want to split up the configuration when it grows. By using flexible configuration you can combine so called partials, settings and templates.
• KrakenD can check and even audit you configuration.
• Since the configuration is in JSON, you can generate OpenAPI.json from the KrakenD config. You can use this for Swagger.

KrakenD is a great tool to manage your APIs. It is lightweight, fast, easy to configure and has lots of functions out of the box. It is versatile and extensible. By using it you can make your architecture more agile.

However, it also means that you will have to manage the API Gateway configuration. And a change in the configuration means you will have to restart the process.

General tips

Some general notes:

• Have a look at:
  • Pydantic
  • Chalice
  • Xata
• It can be helpful use functional programming (e.g. using closures) instead of by default using classes and methods.

How cognitive biases and ranking can foster an ineffective DevOps culture — Kenny Baas & Evelyn van Kelle

In practice, not all team members are treated equally although we might think they are.

How to make sure everyone said what has to be said?

In each team ranking takes place. It determines who takes the lead, who expresses an opinion, etc.

We need to make sure ranking is made explicit. Your job title or position in the org chart is explicit ranking. But things like your gender, skin colour and level of charisma are part of the implicit ranking.

The result of this ranking can be a situation like this: a woman on the team just explained something on a certain topic, but questions about it are addressed to the white man in the team.

We all have a list of traits we associate with higher rank. So if someone ticks more of those boxes, they are seen as higher in rank. You can also rank yourself lower compared to others. As a result you might not express your opinion because someone you regard as having a higher rank said something that conflicts with what you wanted to say.

The person with the higher rank should be aware of their rank and “share it.” Ask yourself the question “what don’t I know?” in a discussion. Ask the question the other (with a lower rank) is afraid to ask.

Own, play and share your rank.

How can we create and include new insights?

We use cognitive bias to make decisions. Usually this helps us, but sometimes it works against us. What we know hinders us from taking on a new perspective and it limits us in our thinking. We get stuck in what we know (functional fixedness).

Be aware of your biases and try to break free from them.

Who makes decisions and how to get everyone onboard with the decision?

Have less “corporate” meetings and more “campfires” where we share ideas and talk about the why.

There are 4 ways to go into a decision:

• Idea
• Suggestion
• Proposal
• Command

If you have a proposal or command, be open about it: “this is what we are going to do, what does it take to go along with it?” This is better than pretending it is completely open for discussion.

Getting Started: Embrace Your Inner Child — Rain Leander

Learning to ride a bike when you are older is harder than when you are a child. This is not so much a physical thing, but we do not want to be embarrassed when we are older. This goes for everything new we do: we want to be the best—instantly. However you will fall down and fail. You will make mistakes. Embrace that, be brave and learn.

Bravery is something you have to develop. It is hard! And it is not a lack of fear, it is moving forward despite of fear.

The steps to become more brave that worked for Rain:

• Embrace vulnerability
• Admit you have fear
• Do it anyway
• When you fall down, get back up

Play!

Playing is instrumental to life. No one can tell you how to play. Do what is fun for you. And when it stops being fun, just stop doing it. Book time in your agenda to play, each day! (And napping counts.)

Explore! Some of the things Rains uses to stay curious and explore:

• Listen
• Ask questions
• Avoid assumptions
• Listen
• Embrace learning as fun
• Develop a beginner’s mind
• Listen

You are a unicorn. Your experience, your background, your education, etc. makes you unique.

Be willing to fall down, and get back up. Be brave. Play! Conquer the world.

(The original blog post for this talk: Getting Started: Embrace Your Inner Child)

Prevent Heroism: How to Work Today to Reduce Work Tomorrow — Quintessence Anx

We might be familiar with “the angry sysadmin.” It is the person who is highly skilled, has lots of knowledge, gets asked all the questions and puts in overwork to get their own work done. This is basically the description of the character Brent from the book The Phoenix Project.

Being a Brent-like person is stressful and may result in anger, resentment and strained relationships. Brent-like persons typically have issues with things like time management, maintaining focus and allostatic load (where you’re too tired to be awake and too stressed to sleep). But there are business implications as well: diminishing returns, high turnover, reputational damage and financial performance goes down.

Instead of having—or being—a Brent-like person, we rather want no silos and no vacuums (the latter is where you, for instance, get no response to questions).

Steps to get out of this situation and do things different:

• Brainstorm the work (every task done during the day, planned or unplanned)
• Determine which “stream” each task is in: reactive or proactive
• Look for patterns
• Switch to “upstream mode” to prevent work in the future (short term/long term vs permanent/temporary matrix)
• Plan and Prioritize work: upstream work is more important
• Do the work
• Iterate (what works, what does not, what can be improved)

Upstream work is the proactive and preventative work that reduces the need for reactive, or downstream, work

But how does one measure things? Things to keep in mind:

• What are your needs?
• Learn how to measure them
• Establish baselines
• Know your capacity
• Track the rate of change
• Set goals
• Iterate

Slides and additional resources can be found at https://noti.st/quintessence

Real-world Continuous Delivery: Learn, Adapt, Improve — Michiel Rook

This talk is a real life case study, focussing on the deployment of the platform of a customer of Michiel.

There was a pretty long release check list with manual steps. Especially when under pressure steps were forgotten or done incorrectly. Ideally they released every two weeks (after each sprint). It took a team 2–3 days of manual work. Time not spent on features.

To improve this situation, the following goals were set:

• Reduce costs by letting people do more valuable work
• Faster feedback and ability to do more experiments
• Reduce toil

For continuous delivery to work, you have to make things small. This is a key difference between high and low performing teams (see Accelerate).

Although the platform was a monolith, they deferred splitting it up. If they would have broken down the monolith into smaller pieces, they would have more manual steps for each release and made the situation worse. So the strategy was to automate first and then split.

Phase 1

The teams increased the release cadence. They switched to a weekly release cycle instead of a release every two weeks (on good weeks). Releasing more often makes problems visible faster.

The release process was also automated. They created a pipeline to build and deploy the platform. In this phase there was still a manual step between the deployment to acceptance and production. They didn’t trust the system enough yet.

They also made changes in their way of working. The big one: pair programming. This is superior to code reviews, for example because you also discuss code that does not get written and you have architecture discussions.

Other changes:

• A “do not leave broken windows” mentality. This is from the book The Pragmatic Programmer) which states: neglect accelerates the rot faster than any other factor.
• Measure how they were doing in terms of their continuous delivery journey. The book Measuring Continuous Delivery has useful information about this.

Phase 2

In the next phase they had a robot press the button to deploy to production. This required zero downtime deploys. They solved this by doing rolling updates (a load balancer in front of a couple of backend servers and then upgrade one server at a time until they are all up-to-date).

Pipeline failures come in all forms, e.g.:

• Flaky tests
• Timeouts
• Network stability issues
• External dependencies in tests

If you cannot trust a test, remove it. Otherwise you’ll probably work around it, which is more dangerous.

This level of automation also meant they treated pipeline failures as priority 1 issues (which warrants extreme feedback to notify people of a broken build: lights, sounds).

Phases 3

In the third phase they made it faster:

• Scale vertically and horizontally
• Parallelize tests

The Adjacent Possible: Evolution, Innovation & Catastrophe — Jason Yee

The four cornerstones of resilience:

• Monitoring: knowing what to look for
• Responding: knowing what to do
• Learning: knowing what has happened
• Anticipating: knowing what to expect

The term “adjacent possible” comes from biology in the context of evolution. Evolution requires a chain of changes to happen.

Complex systems usually do not have a single root cause, only contributing factors. Perhaps there’s an “adjacent possible” of failure. We cannot anticipate all failures, but perhaps we can explore them.

How do you explore adjacent possible failures?

To move from known knowns to known unknowns, we can use chaos engineering (thoughtful, planned experiments to improve our understanding of how systems work (and fail), so that we can improve them). This way we can explore the known unknowns and these then become the known knowns. From that standpoint, the original unknown unknowns become the known unknowns.

Scientific process:

• Observe (what is normal)
• Hypothesize (how do we think it will react to failure)
• Experiment (inject failure)
• Analyze (what did actually happen)
• Share knowledge

Explore the adjacent possible!

Minimal viable presentations — Mark Smalley

There are four important points to take into consideration when creating a presentation:

Accept that it is not about you

It is the audience’s presentation. What you want to tell might not be what they want to hear. Kill your darlings.

Anticipate silent questions

These questions are:

• Huh? What is he talking about? (Solution: be clear about the self evident stuff)
• Really? (Solution: be clear about the evidence)
• So? How is this relevant for me? What’s in it for me?
• What’s next?

Think of learning objectives

Think about what the audience should know, believe, be able to do and feel after the presentation.

Dare to choose your ideal audience

Accept that the presentation will not be for the majority of the people listening.

Engineering Metrics That Matter — Dan Lines

Before we dive in, let’s start with some KPIs you want to avoid:

• Lines of code
• Number of commits
• Individual stack ranking
• Anything with story points

So what do you want to measure? What will improve your process?

Couple pull request size with cycle time (how much time from coding to deployment)

What you’ll see is that bigger PR size takes longer to review and thus increase cycle time.

Deployment frequency

How often can we get new value into the hands of customers.

Idle time

Waiting for e.g. a release or review. Waiting means context switches which means less productivity.

Code churn

How much code are we reworking in a short period of time? High code churn indicates delays and quality issues.

Review depth

The amount of feedback per PR.

Mean time to restore

How long it takes to fix an incident in production.

Investment profile

New functions for our customers vs bug fixing, backend or non-functional work.

DevOps is no Walk in the Park — Sabine Wojcieszak

In a park you are safe, you do not need a map, can wander where you want and you need no preparation. DevOps is not like that.

The term CALMS has been coined by John Willis to explain DevOps. It stands for:

• Culture
• Automation
• Lean
• Measurement
• Sharing

Most people talk a lot about the automation and some about the measurements part, but very few about sharing and even less about culture.

There is a similarity with Agile where the term began as a niche but became hyped and then mainstream, where people talk about Agile without knowing what it is. We should prevent this from happening with DevOps.

It’s no longer business and IT, but IT is part of business. One could even say IT is business.

DevOps is not cherry picking what we like (automation) or only picking the low hanging fruit and calling it “DevOps.” It needs an holistic approach. But this needs a lot of ingredients. In the right mix.

You should always ask yourself why you are doing DevOps. Do you think it is cheaper because of the level of automation? Because everyone is doing it? To deliver better software?

You need to understand the whole approach.

DevOps means to Sabine: deliver valuable products with better quality sooner to customers/users. This means we need to know what is valuable for our customers, what our customers think of as quality, etc.

The term CI/CD in context of DevOps traditionally stands for continuous integration / continuous delivery. But it could also stand for continuous improvement / continuous development, if we think of products and people. But we can also talk about continuous improvement and continuous discovery of opportunities and outcomes.

You cannot buy DevOps—neither with tools, nor with certifications. But it’s not for free either. You will fail for example.

Some anecdotes of where it went wrong:

• “We are now the DevOps department.” Which means there is still a silo with a lack of communication.
• “We have a DevOps team that is setting up the pipelines.” Again no communication, no transparency.
• “We have a linter but it gave to many warnings so we turned it off.” No understanding about why it is done, no trust that it helps to grow, sign of fear to make mistakes.
• “We are measured by the number of developed features, but not allowed to talk to sales.” It’s a feature factory, measuring the wrong things. Again also not enough communication and silos.
• “IT has not reported any problems.” Be open to the obvious: if customers complain, there is a problem. Ask yourself what is actually monitored? Not business relevant metrics obviously otherwise the problem would have been detected by the company instead of the customers.

DevOps is more than automation, it is CALMS.