Streamlining Signed Artifacts in Container Ecosystems — Tonis Tiigi

It’s possible to sign Docker images, but at the moment most are actually not signed. Also, users should understand what the signature is protecting and what it’s not protecting. We should not want signing just to tick a box on the security checlist, but because of the security it adds. And we need something simple: integrated with existing tools, should not slow down tools.

Buildkit powers “docker build” but is not limited to Dockerfiles. It’s high performance, can build complex builds and has caching.

A modern build is a graph of images, Git repositories, local files, etc. The results are images, binaries, archives.

We need Supply-chain Levels for Software Artifacts (SLSA) provenance: what has actually happened in the build? What was the build config? Et cetera. It’s useful to figure out how an artifact was built.

Buildkit does not sign images by default. GitHub has an example in the documentation to run a build with Buildkit and generate an artifact. It claims to generate an unforgeable statement. But if your GitHub credentials are leaked and the attacker can get your hands on the temporary signing key, they can use it to sign their own artifacts.

Docker created the github-builder repository. It contains reusable GitHub Actions to securely build images. If you use this, your images are signed to prove that they were built from a certain repository, using the configured build steps. Where Buildkit (among other things) provides isolation, github-builder provides signing context. It also protects against build dependency leaks.

So that takes care of the signatures, but how do you verify them?

• The command “docker inspect” now shows verified signatures
• You can manually verify it with cosign
• You can also use sigstore/policy-controller for Kubernetes

Buildx also includes experimental Rego (Open Policy Agent) policy support. This means you can write a matching policy for Dockerfile, e.g. Dockerfile.rego, which is then automatically loaded. All build sources now need to pass policy for the build to continue (images, Git repositories, URLs, etc).

You can do very complex stuff in the policies. As simple example Tonis showed:

package docker

allow if {
  input.image.repo = "org/app"
  docker_github_builder_tag(input.image, "org/app", input.image.tag)
}

This policy should make sure that the image can only be built from this repository and that the image tag should match the Git tag.

Summary:

• No reason not to sign
• Not all signatures are equal
• Software pulling packages should verify pulled content

Link to the conference page

Sequoia git: Making Signed Commits Matter — Neal H. Walfield

Version control systems (also known as VCSs) track the following:

• Changes to the code
• Authorship
• Other metadata
• Commit message

But the author can be faked: the metadata is set by the author, including the author’s name. After a quick “git config” command you can commit as anyone you want, for example Linus Torvalds. Sure, GitHub could see that the committer (the one pushing the commit) and author are different. However, this is not necessarily bad because we might simply want to give proper attribution to the author of the commit.

And in theory the forge might also be compromised, or someone may have gotten permission to push to the project.

To prevent impersonations, we can cryptographically prove who the author is by signing the commits. But now the problem shifts to the certificates. Because anyone can create a key with any name (again, for example Linus) attached to it. So what does a signed commit mean now?

How can we be sure that the author is who they say they are? There are ways:

• You could talk to developer the verify
• You could go to key signing parties
• You can use a central authority that you trust (e.g. keys.openpgp.org, the Linux developer keyring, the distributions-gpg-keys package, or, if you trust Github, use github.com/<username>.gpg)

You can use the following command to show the Git log and the signatures on them:

git log --show-signature

But now you need to actually check that the signatures are indeed made by the certificates you trust.

It’s up to the maintainers of the software to curate a list of contributors and track when contributors join and leave (yes, there is a temporal element as well). This is hard. Maintainer needs tooling. And you would want to detect unauthorized commits (impersonation, a malicious forge, a machine in the middle or for instance when project is given to a new maintainer by a forge/registry).

What does the solution look like?

• Clear semantics
• The project itself maintains signing policy
• Third party uses maintainers’ policy to authenticate project
• Verification, not attestation: do not rely on any external authority

(Note that the maintainers can still be socially engineered to include the key of an attacker in their policy. So they still have to be careful about who is added to the policy.)

Sequoia git provides:

• Specification
• Config
• Tooling

With Sequoia git (which part of the Sequoia PGP project) you can have a signing policy in an openpgp-policy.toml file in the project’s Git repository. It specifies users, their keys and their capabilities. You can use sg-git to help maintain this file.

For instance to add user Alice and then describe the current policy, you can use the following commands:

sq-git policy authorize alice --committer <cert>
sq-git policy describe

A commit is “authenticated” if at least one parent commit says the commit is acceptable (via the policy). To verify that there is an authenticated path from the current state back to a certain commit we trust, use this command:

sq-git log --trust-root <sha of trusted commit>

Projects may have contributions from others that are not included in the policy. To maintain an authenticated path when accepting the contribution, a trusted author needs to merge the contribution via a merge commit that is authenticated. (You may need to use the “--no-ff” on the merge to make sure there is a merge commit though.)

Link to the conference page

An Endpoint Telemetry Blueprint for Security Teams — Victor Lyuboslavsky

With open source we can inspect something that is broken, we can change the defaults. With security we are used to the opposite; it’s a black box. We are not used to owning the data. The data exists on the endpoints, but ownership is transferred to a different team. How can we add more security in a way engineers understand and can use?

Victor presents a blueprint with the following layers:

• Endpoint agents
• Control layer
• Ingestion, streaming & storage
• Detection
• Correlation, intelligence and response

The value is not in the layers themselves, but the boundaries. For example, the ingestion should move the data reliably but should not care which tool collected it. This makes them loosely coupled.

For endpoint agents Victor suggests osquery which allows basic questions about endpoints. Data is structured and consistent. It aligns with open source values. (Alternatives: scripts & cron, log shippers like filebeat or tools like auditd or Event Tracing for Windows.)

Controlling the data (the next layer) means that you want to have:

• Central config
• Live queries
• Consistent schemas

Fleet (disclaimer: Victor works here) is built to manage osquery at scale and a good candidate for this layer.

The control layer needs to work hand-in-hand with ingestion layer. The ingestion layer moves data to downstream system. E.g. Vector or Logstash can be used here.

Ingestion isn’t where you get clever. It’s where you get reliable.

Streaming decouples users from consumers and e.g. allows replay. Note that this is an optional step and it would come after ingestion, not in place of it. For instance Apache Kafka can be used in this layer. Ingestion absorbs the mess. Streaming preserves flexibility.

The storage layer is where telemetry becomes durable. It’s about being able to ask hard questions later. Examples of useful tools: ClickHouse, Elasticsearch (which is better at text search) and Iceberg (which is slower for active investigation).

For the detection layer you might want to use Sigma. It provided portability. Rules are translated to native SQL running on ClickHouse. Intent (Sigma signatures) becomes execution (SQL query to get the data).

Finally the correlation layer: Grafana can be used for correlation and visualisation. Grafana can query ClickHouse. Grafana also has alerting.

Note that response isn’t just about automation. It’s also to pause and ask better questions. The correlation layer should focus on enabling humans to act.

Open endpoint telemetry is not an “EDR killer”. It does not replace it. It adds diversity and complements other tools. It provides a second set of eyes.

Link to the conference page

The Bakery: How PEP810 sped up my bread operations business — Jacob Coffee

Python loads imports eagerly by default. This leads to memory bloat and cold start issues. Explicit lazy imports (see PEP 810) only import a module when it’s first accessed not when the import statement is executed.

Lazy import is scheduled to be included in Python 3.15 and looks like this:

lazy import foo from bar

The design principles applied are that lazy imports are:

• Explicit
• Local
• Granular

When parsing the Python code a proxy module is created. Only when the module is actually used, the proxy is transparently replaced by the real package. You will not always see improvements, so do not blindly replace all imports with lazy imports.

PEP 810 also eliminates the need for TYPE_CHECKING guards. (See the typing docs, in short: importing a module that is expensive and only contains types used for type checking in an “if TYPE_CHECKING:” block.) It also helps for faster test discovery and collection, less memory usage, decrease cold start slowness in e.g. AWS Lambda functions, CLI applications, etc.

Meta (with Cinder) saw a 70% startup time reduction and 40% memory savings. PySide has a 35% startup improvement.

About CLI tools: when using lazy imports you might notice the difference when using --help. There’s no need to load all dependencies to just output the help text of a tool.

Some notes:

• Import time side effects (e.g. logging configuration, DB connections) are also delayed!
• Type checkers need to be updated
• Import errors move to first use (so in runtime, not at launch). Keep that in mind when debugging
• It’s not always faster, so profile your application before migrating and see where you can potentially benefit
• Document your lazy imports!
• You cannot do lazy imports in functions

Circular imports are probably still a problem, but they just show up later.

Link to the repo for this talk

Link to the conference page

Modern Python monorepo with uv, workspaces, prek and shared libraries — Jarek Potiuk

Jarek is, besides his other roles, the number 1 Apache Airflow contributor. The Apache Airflow repo is the monorepo he talks about today. There is also a series of blog posts about this topic: see part 1, which links to the other parts.

Airflow drove early requirements for uv workspaces. They now manage 120+ distributions seamlessly with it. It allows them to combine distributions to work together in a workspace. Also used to import from one distribution in another one.

The project shares a single virtual environment used by uv in root of project. If you run “uv sync” from the top level you get everything. If you run it in a subdirectory (e.g. airflow-core) you only get what is needed for that distribution.

Benefits of the uv workspaces:

• Isolated
• Explicit
• Flexible

Hatch has (or will have, at the time of writing) largely compatible workspaces.

However pre-commit became a bottleneck. They needed to run 170+ pre commit hooks on every commit. Prek is drop-in replacement for pre-commit and works fantastic. It is optimized for speed and monorepos.

Airflow uses symlinked shares libraries (where a shared lib is also a distribution). The Hatchling build backend needs to replace links with physical copies during packaging. They use Prek to maintain consistency.

uv sync detects conflicts between merged requirements files and Prek hooks enforce relative imports in shared code to prevent cross coupling issues (IIRC)

Link to the conference page

PyInfra: Because Your Infrastructure Deserves Real Code in Python, Not YAML Soup — Loïc “wowi42” Tosser

Loïc is a Frenchmen (which, as he himself states, means he must have opinions) and not a YAML fan to put it mildly. That is: YAML as a programming language, e.g. how it is used in Ansible.

PyInfra is an infrastructure as code library to write Python code which is then translated to shell scripts to run on the target hosts. So, in contrast to Ansible, you do not need Python on the target. The target machine only needs SSH and a POSIX shell. You can also configure Docker containers with PyInfra.

If it has SSH, PyInfra can talk to it.

PyInfra has idempotent operations and built-in diff checking. Declarative infrastructure with actual code and not YAML. You can use inventory from Terraform, Coolify or any API.

You can leverage the entire Python packaging ecosystem. Slack integration? Just use the right Python package.

PyInfra is not only a CLI tool, you can also use it as a library.

PyInfra is 10 times faster than Ansible, uses 70% less code, has proper code reuse via import and proper loops instead of with_items. It can have actual unit tests and can scale to thousands of servers. Also you no longer have error messages stating that the error appears to be in … but may be elsewhere in the file … (looking at you Ansible). PyInfra has clear error messages without having to specify -vvvv and wading through hundreds of lines of output.

The suggested migration path:

• Start small, one playbook at a time
• Use your IDE for autocomplete and refactoring
• Leverage Python’s standard library and the ecosystem with all its packages
• Sleep better because you don’t have to debug at 3 AM.

Is PyInfra production ready? Yes! It has a stable API, is already in use in production, it’s actively maintained and is MIT licensed (so no commercial entity behind it to steer its direction).

You can get started today with a simple “pip install pyinfra”.

Link to the conference page

(Note from me, Mark, I found Loïc a great speaker: he has lots of energy, is funny and can transfer his enthusiasm to the room. If the topic interests you and the video becomes available, I would recommend watching this talk as a great sales pitch to get started with PyInfra.)

Ducks to the rescue - ETL using Python and DuckDB — Marc-André Lemburg

ETL stands for Extract, Transform, Load. Nowadays we usually do Extract, Load, Transform because databases are efficient in processing.

DuckDB is open source, in-process analytics data storage (OLAP). It is similar to SQLite, but for OLAP workloads. It has great Python support and uses SQL as standard query language. It’s pip installable, column based (Apache Arrow). It’s single writer but allows for multiple readers, so it’s not a distributed database.

Polars’ streaming can help with processing your data as a line-by-line stream so you don’t have to load the whole file in memory at once.

Example to load a CSV file into DuckDB extremely fast:

SELECT * FROM read_csv(...)

You can load the data into staging tables first to prepare everything and not mess up e.g. existing data. You can then transform data in DuckDB, e.g. filter out unneeded and duplicate data, validate data, fill in missing data, convert data types, etc. You can do the transforms in SQL. You can even use native integrations to write to PostgreSQL, MySQL, etc. Or worst case stream to Python.

Guidelines:

• Know your queries, that is: know how your data is going to be used
• Use the Pareto principle (80/20 rule): optimize for queries that are used often
• Keep a healthy balance between performance and space requirements (which are often trade-offs)

Huge datasets: use the DuckLake extension.

To get started: “uv add duckdb”. Do some experiments and see how it works for you.

Link to the conference page

My takeaways

• Yes, FOSDEM is crowded and you may not be able to get into every talk you want to see in person, but it’s still nice to be there. It’s well organised and there’s a friendly atmosphere. Lots of interesting projects to see and people to talk to. And it’s convenient if you want to sponsor your favorite projects by buying some merchandise.
• It’s worth investigating signing Docker images (in the right way) further.
• Lazy imports look useful! Once Python 3.15 lands it’s worth doing profiling on the projects I work on to see if we can use those to speed things up on startup and save some memory.
• At work we recently decided to go for a monorepo for a project. I want to see if/how uv workspaces and prek can help us.
• I’ve written a bunch of Ansible roles to configure my humble homelab and laptop. Perhaps it’s time to switch to PyInfra? It sounds promising and might be worth the investment of migrating to.

About the trip

This year I had more time, so I booked a room at Falko Hotel for two nights. It’s about a 20–30 minute drive (depending on traffic) to the parking garage I used. And from there about 20 minutes with pubic transport to the Université libre de Bruxelles.

Staying another night meant I had more time for sightseeing, had the time to write this post from my notes and could drive home well rested the next day.

As for tech: besides a phone and laptop, I also brought along two items that made the trip more comfortable:

• A MOJOGEAR Mini Evo powerbank to give my phone extra juice to make it through the day. With 10.000 mAh and up to 22.5W of power it’s more than sufficient for a day at a conference. With its small size and less than 175 grams in weight, it’s also easy to carry around.
• A GL.iNet Opal (GL-SFT1200) travel router. I plug it in, hook it up to the hotel internet, start a VPN connection and all my other devices automatically connect to it and can use the internet without the hotel snooping on my traffic. (Not that I have an indication that my hotel would do that, but theoretically they could if I would not use a VPN.)

The conference

This has been the third time I went to devopsdays Amsterdam. And I love this conference!

Some of the reasons:

• The organizers manage to get great speakers with interesting talks on stage each year.
• Pakhuis de Zwijger is a great location.
• Excellent Wi-Fi.
• Great atmosphere.
• Good food.

The workshops

Go

I had heard about Go, some of my co-workers have some experience with it, but I never wrote anything in the language. I was curious about it though.

The workshop from Michael Hausenblas was a nice intro. Based on what he told and showed us I cannot say that I expect that Go will replace Bash and Python for me. However, I will make some time to actually write some code myself to get a better feel for it.

Monitoring with Elastic

We are already using the Elastic Stack in some places at work, but I have not used it for monitoring purposes. (I gravitate towards Prometheus combined with Alertmanager for alerting and Grafana for dashboards with graphs.) However, Philipp Krenn showed us that you can also do very interesting things with Kibana in the monitoring and debugging realm. Especially since you can correlate metrics with logs in the same tool.

Kubernetes

I could say that Bridget Kromhout’s Kubernetes workshop was a nice refresher of what I had learned in the Kubernetes workshop last year but, to be honest, that would be a lie. I am glad I took this workshop.

It was a good workshop with lots of hands-on tasks. But it went a bit too fast to make it stick. I would have to spend more time on a Kubernetes cluster to really understand everything and get fluent with it. Luckily there is lots of information on container.training (including the sheets of this workshop) and there are plenty of cloud providers where you can get a Kubernetes cluster without having to create or maintain it yourself.

The talks

The talk that resonated most with me this year was the one from Waldo Grunenwald about product teams. Perhaps because (in my opinion) this is something that could be better in my job. Product management, development and operations are three different teams with different managers. Then again, I currently try to be the “ops guy” in our development team so that’s also DevOps, right? :)

The other most memorable talks for me were:

• Bridget Kromhout’s keynote: Cloud, containers, k8s
• Armon Dadgar on service meshes
• Jason Yee relating Dutch peculiarities to DevOps
• Lee Atchison about monitoring in a dynamic (cloud) environment

Miscellaneous

I have been using Emacs for quite a while. I was a Vim user in the past, but switched somewhere between 2007 and 2009. (The first time I wrote about Emacs here was in 2009.)

I have tried PyCharm a couple of times and it is a really nice editor with very useful features. It just never stuck with me and I always went back to Emacs after a while.

During the conference I used Visual Studio Code to write my notes. And I have to say I quite liked it. I intend to also give it a go at work. Who knows, I might even switch…

Go for Ops — Michael Hausenblas (Red Hat)

Michael walked us through the features of the Go language by giving numerous examples. This is a workshop that usually takes a full day so we were in for a nice ride.

One thing he mentioned that he liked about the language is that there is (almost) no magic involved.

Some things that stood out to me, Mark, (as someone who writes Python most of the time and does not know much about Go):

• There are no objects in Go; they are “structs” and methods (functions bound to a struct) (Note from Mark: Steve Francia wrote “Is Go an Object Oriented language?” which seems like a useful article).
• You need to create a file first before you can write to it. (In Python you open a file for writing and it is created if needed).
• To format dates you have to use a special date in the formatter: Jan 2, 15:04:05, 2006 (which is basically 1, 2, 3, 4, 5, 6).
• The standard library is very comprehensive. (This is actually something Go has in common with Python.)
• Code formatting is enforced via gofmt.

Common pattern to handle errors:

  mail, err := mailof(uid, aproject)
  if err != nil {
      ...
      os.Exit(1)
  }

Slide 27: if you feed the printf function a different type, e.g. a string, it will not even compile. (Mark: this is something I’m not used to, coming from Python.)

To expose things like functions (make them available to other packages): start the name with an uppercase letter. Functions starting with a lowercase letter are internal/private to the package. If you try to access an internal function, you get a nice error message (again: at compile time).

Slide 31: “log.Fatalf()” triggers the os.Exit(1) you can see when you run this example.

You can add a call to the defer function at the end of a scope (e.g. “defer f.Close()” in slide 33). Since the Go runtime will execute this always (even if there was an error), you can use this e.g. as a cleanup of an open file. You can have as many defers as you like; they will be executed in reverse order.

Starting with writing tests is quite simple: create file with <module name>_test.go. The function name of the test is irrelevant as long as it starts with “Test”. Run the tests with “go test” (plus options, if you like). Go offers test coverage information. Tip: use a nice editor/IDE and integrate running the tests and code coverage there.

As you can see on slide 39 it is possible to add encodings to your struct to be able to, for instance, encode and decode JSON. There are other encodings, see e.g. https://pkg.go.dev/encoding#section-directories

Google, where Go was created, uses a monorepo. As a result they did not need dependency management in Go. Use e.g. dep to help you out here. It looks like vgo will be part of the language in the future.¹

You can either trust upstream (and Github to be available) and not put your dependencies in your repo, or chose not to and version control the code you depend on yourself.

About running a Go application in a container: you can either pick an image with debug tools (like centos:7), or pick a minimal image like alpine or scratch as the basis of your image. You have to decide whether you want the smallest image possible or want (some) tools included.

For Michael, Go replaced a lot of Bash and Python. However, Michael is not convinced that Go is a good fit to write a complete web application in, for instance. But decide for yourself. On slide 56 there are a couple of links to some pages with criticism.

As already stated, Go has an extensive standard library. Michael advises to use it. If it does not have or do what you want, your second best option is to use a drop-in replacement. Only if that is not possible, search for a package with a different API.

Useful resources:

• The sheets for this workshop can be found via https://go-talks.appspot.com/github.com/mhausenblas/go4ops/main.slide#1.
• Go by example is a nice resource to learn about the concepts in Go.
• https://goreportcard.com/
• https://golang.org/doc/

Monitor Your Microservices — Logs, Metrics, Pings, And Traces — Philipp Krenn (Elastic)

Distributed services make debugging … interesting.

The code for this workshop, a highly monitored “hello world” app can be found on Github.

The server provided for the workshop is an Amazon Lightsail instance created with Terraform and provisioned with Ansible. (The code for this deployment is also included in the aforementioned repo.)

Notable changes in Kibana 6.3:

• It has tools to manage the Elasticsearch indices.
• In visualizations the aggregation previously called “calculation” has been renamed to “math.”

Packetbeat is using libpcap, just like Wireshark. Philipp thinks the future of Packetbeat is in tracking down DNS + TLS errors since you should encrypt the data between your services (which means that Packetbeat can no longer extract much information from the packets).

Previously you used Logstash to get the Nginx access logs into Elasticsearch. Filebeat modules can help you there. Filebeat is just forwarding the data; the parsing is done by Elasticsearch. Filebeat has processors to enrich events with e.g. cloud and host metadata (quite cheaply actually since this information is collected on startup of Filebeat and cached).

Auditbeat has the same type configuration as auditd.

Journalbeat (from a third party) can be used for journald support. Philipp doesn’t guarantee anything, but this is on the list of the Elastic team and he hopes there will be official support for journald.

You can have a rule to collect multiline messages, like stack traces, together in one document by telling Filebeat that if a line start with e.g. a timestamp, it is the start of a new line and if it starts with e.g. a space it is part of a stack trace. You could also use structured logs (which is recommended if you can).

As of version 6 you can tell beats to enable (and update) the related dashboards in Kibana.

For alerting with the Elastic stack you need a commercial license.

The machine learning (also only available in the commercial X-Pack license) takes three iterations to detect a pattern. For example the pattern of how much traffic your application receives on a workday can be learned in three days. For a weekday/weekend pattern, it would need three weeks.

Kibana also has support for APM (Application Performance Monitoring). There are agents for e.g. Python and Node and a bunch of others (some in beta or alpha stage, see the docs).

Elastic is working on Index Lifecycle Management (ILM) which will run as part of the cluster. Philipp is not sure when it will be available though. For now use Curator.

Elasticsearch already supports metrics aggregation (called “rollups”) via the API. In a future version there will also be an graphical interface to configure this.

Philipp compared his workshop to Lego. He showed us some configuration, visualizations, etcetera but “some assembly is required.”

Kubernetes 101 — Bridget Kromhout (Microsoft)

This was a fast paced, highly interactive workshop about Kubernetes so I only took a few notes. However, the slides have so much information on them, you can follow the workshop perfectly fine without comments from me.

Resources:

• Slides: https://devopsdaysams2018.container.training/
• Git repo: https://github.com/jpetazzo/container.training

Warning: we have done stuff you should not do in production. :)

Kubernetes is highly unopinionated.

By default Kubernetes uses one big, flat network. However, you can configure Kubernetes so that customers cannot access each other.

In real life you would not host your own Docker registry in the production environment. We do it in the workshop because it is easier than messing with credentials to other registries.

Kubernetes has extensive role based access control support.

Before we got started

While I was on my way to Amsterdam, I was reading up on my RSS feeds and ran across the most recent comic on turnoff.us. It was so appropriate that I decided to copy it here:

Setup your own Ansible/Docker Workshop/Raising an Ansible Army — Arnab Sinha (TATA Consultancy Services)

Arnab wanted to be able to easily create lab environments for trainings. This workshop not only discusses how the lab is setup but also uses such a lab environment (in this case to provide an Ansible training environment).

The nature of the setup of the lab he used today: each participant got a control node and two managed nodes. Each node was in fact a Docker container which was managed by Ansible.

The first part of the workshop was basically an introduction to Ansible with topics like the history of Ansible and basic command line usage. Arnab demonstrated how to use a custom inventory file, limiting plays to a group or certain tasks (or skipping tasks) and how to syntax check your playbook.

A few examples:

$ ansible all -i "localhost," -c local -m shell -a whoami
$ ansible -i demo.ini all -m shell -a whoami -v
$ ansible-playbook playbook.yml --syntax-check

Some best practices:

• Use the .ini extension for your inventory file.
• Use a separate inventory file for each environment (develop, test, production, etc).
• Use tags so you can specify which tasks you want to run. (Use “ansible-playbook --list-tags playbook.yml” to show all available tags.)

In the category “today I learned”:

• Ansible has a pull mode (ansible-pull). Who knew? :-)
• Ansible comes with documentation: ansible-doc.
• Looping over sequences with with_sequence (see the docs).
• You can make a playbook executable by adding

  #!/usr/bin/env ansible-playbook
  

  at the top (and using chmod).

If you want to run your own lab, you can use Arnab’s GitHub repo: arnabsinha4u/ansible-traininglab. Note that this assumes a CentOS host.

In order to be able to log in to the “master” node (via ssh ansiblelabuser1@localhost) I had to enable PasswordAuthentication in /etc/ssh/sshd_config. But since I had run the Ansible playbook already, I was not allowed to change that file. I first had to run this command:

$ chattr -i /etc/ssh/sshd_config

Other GitHub repos from Arnab that you can use:

• arnabsinha4u/docker-traininglab
• arnabsinha4u/launchpad

Introduction To Kubernetes — Andy Repton (Schuberg Philis)

Kubernetes is a container orchestration platform. It has a huge open source backing and new features are being built quickly. It does one thing (in an elegant way).

Kubernetes has three main components:

• Masters: the brains of the cluster. Consists of: Apiserver, controller manager, scheduler.
• Nodes: the brains of individual nodes. Consists of: kubelet, kube proxy.
• etcd: replicated key/value store; the state store and clustering manager of kubernetes.

When you look at it from a ‘physical’ perspective, you have a Kubernetes node and this node runs Docker, which in turn runs the containers. Pods are a logical wrapper around containers; we don’t care about nodes.

Pods are mortal. What this means is that processes are expected to die. But we do not care because Kubernetes ensures availability by making sure that there are enough of them running.

During the workshop we used the following GitHub repo: Seth-Karlo/intro-to-kubernetes-workshop.

The pod you can create with the pod/pod.yml file can be used for a toolbox to examine other pods.

More terminology: a replica set is basically a way of saying “make sure there are N copies of a pod.” If you look at the specification of a replica set, you can see that it contains a Pod spec.

Using the readinessProbe directive you can make sure that a container does not receive traffic until it is actually ready. Note that this is different from Docker’s health check which is meant to determine if a container is still working or should be killed.

With the replica set example in aforementioned repo, Kubernetes will automatically start a pod again if it is killed. Even if you kill a pod yourself—Kubernetes doesn’t care why it has gone down.

If you edit a replica set (e.g. to update to a newer version of an image), it has no immediate impact because the pod spec is nested. Deployments can enforce changes are being executed though.

To get the whole configuration of a pod, including the default and not just the stuff we specified, run:

$ kubectl get pod <podname> -o yaml

Note that volumeMounts appear by default on every pod you create.

Secrets, although the name implies something different, are not encrypted; all pods in the same namespace can access the secret and decode it (base64). It is an easy way to put information in a pod, it is not secure!

Services don’t “exist” like containers do. A service is a purely logical idea. A service exposes pods to other pods.

A service automatically gets a DNS entry: <service name>.<namespace name>. This means that from inside your containers, you can use DNS to access other containers.

About scheduling:

• You can label nodes and then make sure that pods are scheduled on nodes with a certain label.
• Kubernetes will distribute pods across nodes as ’evenly’ as possible.
• Kubernetes will not auto reschedule pods when you add a new node.

For this workshop we used kops because it was easier. At Schuberg Philis they actually use Terraform to manage their cluster(s). Note that you can use a flag and then kops will spit out Terraform code.

If you are worried about your pods going down gracefully, you are doing your pods wrong.

If your application depends on long running processes: don’t use Kubernetes. Use the right tool for the right application.

Combine containers inside pods if latency matters, if they need to share configuration files or if they need to connect via loopback device.

Miscellaneous:

• Kompose: a tool to convert Docker Compose files to Kubernetes YAML files.
• With horizontal pod autoscaling you can automatically scale up/down the number of pods to handle load.
• You can set limits on your pods so Kubernetes will kill it off when it goes over the limits, e.g when it uses too much memory.

Resources:

• Slides
• Command cheat sheet

Hands-On OpenShift Developer Workshop (In Azure) — Alessandro Vozza (Microsoft) & Samuel Terburg (Red Hat)

Why OpenShift: because developers need a platform to be able to deploy their applications. OpenShift is a platform to run your containers at scale. Meant for enterprise: not necessarily the latest features, but focus on stability.

OpenShift was originally written in Ruby, but it has been rewritten in Go and it is built upon Kubernetes. Openshift is always one release (circa three months) behind on Kubernetes.

Everything you can deploy in Kubernetes, you can deploy on OpenShift.

OpenShift Origin is community supported. If you want a commercially supported version, you have to run on Red Hat Enterprise Linux (RHEL). Red Hat OpenShift uses RHEL images, where OpenShift Origin uses CentOS.

OpenShift Online runs on AWS, but you can for instance also run it on bare metal if you want. But public clouds are a more natural fit for cloud-native applications.

Pods are the orchestrated units in OpenShift. Containers in a pod can talk to each other via localhost and local sockets. The security boundary is extended from the container to the pod. Containers can see each others processes and files. You only want to run one process in a container though.

A service can be seen as a sort of load balancer to redirect traffic to the right pods. Internally it is using iptables.

OpenShift provides its own Docker registry which you can use if you want to.

OpenShift has solved the persistent storage problem before Kubernetes did. You can use the native storage for your solution (e.g. EBS for AWS). Note that block storage solutions require mounting/unmounting and thus takes a little longer.

As with Kubernetes, there is no built-in autoscale for OpenShift. Red Hat CloudForms can monitor your cluster and do the scaling for you.

The routing layer is your entrypoint into the cluster. It’s based on HAProxy. Comparable with Kubernetes’ Ingress.

RHEL Atomic is a minimalistic OS designed to run Docker containers. (It is similar to CoreOS, but Red Hat wanted to have its own OS.) Everything you want to run has to run in a container. You can install OpenShift on RHEL Atomic.

Fun fact: you can create resources in Azure with Ansible.

Unfortunately there were some problems with the Red Hat OpenShift Azure Test Drive. As an alternative I used minishift to run OpenShift on my laptop. With it, I could work on the workshop.

Further reading:

• Kube by Example
• Azure Red Hat OpenShift
• ansible-azure repository
• OpenShift Documentation
• Minishift repository