Since it took me a little while to figure out how to get it working in the first place, and get a comfortable workflow in the second place, I decided to write it down here for future reference.

Context

First a bit of context.

One of the AWS best practices is to enable multi-factor authentication (MFA) for privileged users. You can even create a policy to require MFA. This means that users affected by the policy will have to enter their MFA code to log in via the web console, but also if they want to access the AWS APIs from the command line (e.g. in a Python script or using the AWS CLI).

Without MFA, you can configure your credentials in ~/.aws/credentials and that file might look like this:

[default]
aws_access_key_id = AKIAIFFJM737PF5LH4VA
aws_secret_access_key = +gGgW9ElnjG73//uJxziR8dvgQzs++SFbR4at1Q9

(Don’t worry, the credentials have already been revoked.)

Assuming you have installed the AWS CLI, you can now interact with the AWS API like this:

$ aws s3 ls example.user.bucket
2019-02-19 20:23:44 3023 Vagrantfile
2019-02-19 20:23:42 372 config.yml
2019-02-19 20:23:44 206115 screenshot-1.png

But when the use of MFA is enforced, you’ll see this:

$ aws s3 ls example.user.bucket

An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

Assign an MFA device

Before we can continue, you’ll have to configure an MFA device. See the official documentation on enabling a Virtual Multi-factor Authentication (MFA) Device for instructions.

Make note of the assigned device, we’ll need it in a moment.

Manual authentication

To be able to use the CLI, you’ll have to request a so called “session token”:

$ aws sts get-session-token \
    --serial-number arn:aws:iam::xxxxxxxxxxxx:mfa/example.user \
    --token-code 287632
{
    "Credentials": {
        "AccessKeyId": "ASIARYSXPEDHOE73FL3V",
        "SecretAccessKey": "S1rS...F8KL",
        "SessionToken": "FQoG...seMF",
        "Expiration": "2019-02-20T07:36:27Z"
    }
}

Note the 6 digit code at the end of the command, this is the code generate by your MFA device (probably Google Authenticator or a similar app).

You can use these credentials in several ways:

• Expose them via environment variables:

  $ export AWS_ACCESS_KEY_ID=ASIARYSXPEDHOE73FL3V
  $ export AWS_SECRET_ACCESS_KEY=S1rS...F8KL
  $ export AWS_SESSION_TOKEN=FQoG...1seMF
  

• Put them in the credentials file (~/.aws/credentials) as a new profile:

  [default]
  aws_access_key_id = AKIAIFFJM737PF5LH4VA
  aws_secret_access_key = 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
  
  [session]
  aws_access_key_id=ASIARYSXPEDHOE73FL3V
  aws_secret_access_key=S1rS...F8KL
  aws_session_token=FQoG...1seMF
  

  Note that with this solution, you’ll have to specify the new profile when using the CLI, e.g. “aws s3 ls example.user.bucket --profile session”

Since the token is only valid for 12 hours in my case, the manual workflow means you would have to do the copy/paste routine at least once per (work)day. Since I don’t like repetitive tasks and this is an excellent task to (at least partially) automate, let’s make the computer do the repetitive work for us.

Assisted management of session tokens

Obviously, I’m not the first one to want this, so I use a tool that already exists: aws-mfa (PyPI, GitHub).

There are alternatives that do, more or less, the same thing. What I liked about this tool:

• MIT license
• They use the boto3 library instead of starting a subprocess, running AWS CLI and parsing the JSON output.

As described in the documentation of the tool, you’ll have to modify your credentials file (again: ~/.aws/credentials) a little bit:

[default-long-term]
aws_access_key_id = AKIAIFFJM737PF5LH4VA
aws_secret_access_key = +gGgW9ElnjG73//uJxziR8dvgQzs++SFbR4at1Q9
aws_mfa_device = arn:aws:iam::xxxxxxxxxxxx:mfa/example.user

Note that the profile name has changed and that the “aws_mfs_device” line has been added.

With this configuration in place, you can request an session token:

$ aws-mfa
INFO - Validating credentials for profile: default
INFO - Short term credentials section default is missing, obtaining new credentials.
Enter AWS MFA code for device [arn:aws:iam::xxxxxxxxxxxx:mfa/example.user] (renewing for 43200 seconds): 466139
INFO - Fetching Credentials - Profile: default, Duration: 43200
INFO - Success! Your credentials will expire in 43200 seconds at: 2019-02-20 08:03:25+00:00

If you check your credentials file again, you’ll see that a “default” profile has been added.

[default]
assumed_role = False
aws_access_key_id = ASIARYSXPEDHNH5EOH3I
aws_secret_access_key = fcD3...+FL3A
aws_session_token = FQoG...seMF
aws_security_token = FQo...seMF
expiration = 2019-02-20 08:03:25

[default-long-term]
aws_access_key_id = AKIAIFFJM737PF5LH4VA
aws_secret_access_key = +gGgW9ElnjG73//uJxziR8dvgQzs++SFbR4at1Q9
aws_mfa_device = arn:aws:iam::xxxxxxxxxxxx:mfa/example.user

Your original credentials in the “default-long-term” section are unchanged, but the new “default” profile has all the short lived credentials.

As you can see, the new profile has an “aws_session_token” and an “aws_security_token” entry with the same token. Apparently this is to support both boto and boto3

Once again the AWS CLI is working again:

$ aws s3 ls example.user.bucket
2019-02-19 20:23:44 3023 Vagrantfile
2019-02-19 20:23:42 372 config.yml
2019-02-19 20:23:44 206115 screenshot-1.png

The biggest difference with the manual workflow is that this takes a whole lot less typing and is much less error prone.

These are just notes. They are not proper summaries of the talks.

Three years of Lessons from Running Potentially Malicious Code Inside Containers — Ben Hall (Katacoda)

Katacoda is an interactive learning environment where people can get shell access on machines provided by Katacoda via their browser. People are prone to experiment. Katacoda needs to make sure that one person cannot have an impact on other customers.

Docker has a number of security related features, like:

• Running as a non root user in the container.
• File isolation via copy on write.
• Limits on what resources users can use (memory, CPU), via cgroups.
• Constrict a container via namespaces.
• Restrict privileges so you cannot use e.g. setuid/setgid.
• Seccomp to limit system calls.
• Running profiles via AppArmor.

You need visibility into your containers. If the CPU spikes, is there cryptocurrency mining going on, or is it a legitimate machine learning job?

How do we solve a problem like mining? Cgroups are not enough. You would like to have traffic shaping to e.g. limit the amount of data a container can send. You can also:

• Kill long running processes (easier to do when using orchestrators like Docker Swarm or Kubernetes)
• Block or restrict outbound network traffic (block by default and only allow legit usage)

You could use iptables for the latter. However, this does not work with Docker since it uses a separate Docker chain. When running in Kubernetes it would be easier via a service mesh (like Consul or Istio).

But if you cannot block everything, it fundamentally is a cat and mouse game.

When the host is attacked via a container, having privileged containers is the worst since you can easily break out of them (inserting SSH keys, finding Docker credentials, etc, etc).

You can limit the number of processes to prevent e.g. fork bombs (--pids-limit=2048).

Visibility into your system is key.

Tips:

• Log iptables drops
• Sysdig Falco
• Have a security pipeline (e.g. using Aqua)

Are containers secure? Fundamentally yes. But you also have to think about the security measures that are in place.

Quick wins:

• Don’t allow root
• Read only file systems
• Block egress (why do you need run apt-get in production?)
• Network policies / service mesh / Istio!
• Create visibility

Cloud Object Storage: The Right Way — Orit Wasserman (Lightbits Labs)

Objects are organized in a bucket (like a folder, but they cannot be nested) and are immutable. Cloud object storage means you use a restful API to access the objects. Common clouds: AWS S3, Swift (OpenStack), Google cloud storage and Ceph (which can be used as private cloud object storage solution).

You’ll want to use cloud object storage:

• if you need to scale
• have lots of large objects that are rarely updated
• have small objects that are infrequently updated and are not performance sensitive
• slow hard drives

When not to use cloud object storage:

• When the application does a lot of in place writes in big files.
• Legacy applications that cannot use the object storage API.

Special features:

• Multipart upload
  • Upload one object as a set of parts (initiate object, upload the bits and complete the object).
  • Benefits are improved throughput, quick recovery from network issues, pause/resume upload.
  • Pitfalls: not recommended for small objects, limit for regular uploads is 5GB, unless you are careful you can leave orphans after incomplete uploads.
• Versioning
  • Not enables by default because it takes up space (and you pay for what you use).
• Lifecycle
  • Expire old objects.
  • Move objects to colder storage.
• Virtual hierarchy
  • By adding a prefix you can ‘fake’ hierarchy.
• Static website
  • Cloud storage already has a webserver. You can serve your website directly from cloud object storage.

To increase security of your cloud object storage usage, you should use signatures if your provider and SDK support it.

You should also encrypt the traffic (there is a performance penalty though). Server side encryption is not enough because this means you will have to trust your cloud provider. You should also encrypt client side (SSE-C or SSE-KMS).

Pay attention to access control. It might not be simple or convenient, but please use it. Give as little permission as needed. Be very careful with public buckets!

Even though your storage is now safe, you can still have a disaster you need to recover from. You need to have a disaster recovery plan and test that plan regularly. Ceph provides geo replication.

Replication in Ceph makes a distinction between metadata and data operations. Metadata is usually small and updated infrequently. It is replicated synchronous.

Data on the other hand is usually about large amounts of data and there are frequent operations. These operations are asynchronous.

(Slides)

Cloud-init: The cross-cloud Magic Sauce — Scott Moser & Chad Smith (Canonical)

Cloud-init helps with initial initialization, which is convenient for you. It also allows cloud platform providers to have a few generic images and only on instantiation make them specific (instead of having thousands of specific images to begin with). Cloud-init supports a bunch of distributions, not just Ubuntu. It also supports many cloud providers.

The goals of cloud-init:

• Remove the need to make a custom image or reboot. There are a couple of moment where you can execute code: ASAP, once the network is up or at the end.
• Option to create an image from a snapshot. You can choose to execute code on boot once, per instance or per boot.
• Make it easier to migrate to different clouds.
• OSes should only have to ship one single image.

Instance input:

• Disk image
• Meta data (hostname, authorization info like SSH keys, etc)
• Network configuration (multiple NICs and IPs, bridges, vlans, etc)
• User data (e.g. commands to execute)
• Vendor data (like vendor data but for the platform)

Cloud-init wanted to provide a common interface since every cloud provider does things (slightly) different.

Cloud-init provides specific configuration to (amongst others):

• Grow the root partition
• Add/upgrade packages
• Set the timezone/locale
• Run arbitrary code
• Start Puppet or Chef

Notable changes in cloud-init versions 18.4 and 18.5:

• Standardized instance metadata in /run/cloud-init/instance-data.json
• Extended command line tooling (query, analyze and status).
• Automatic network configuration based on metadata/udev events (to come in 18.5)

You can use Jinja templates for your cloud-init config. The example from the slides:

## template: jinja
#cloud-config
{% set HN='oss-' ~ v1.platform ~ '-' ~ v1.region ~ '-' ~ range(9) | random %}
puppet:
  conf:
    agent:
    server: puppetserver.blackboxsw.com
    certname: {{ HN }}
hostname: {{ HN }}
...
{% if v1.region == 'us-east-2' and v1.cloud_name == 'aws' -%}
echo 'Installing custom proxies ...’
{%- endif %}

(Slides)

Note: these are my notes. They are not necessarily representative or complete summaries of the talks.

Container security monitoring using open source — Madhu Akula (AppSecCo)

Why is container security monitoring (CSM) important? An attacker could gain access to one of your containers. Or you may be using an image from the Docker Hub with malicious code. CSM can help you to take actions based on the monitoring alerts.

For CSM it is important to collect your logs on a central location. Then we can analyse them to find suspicious patterns. Based on the patterns we identified, we can set thresholds and create pattern to trigger alerts.

How to look at logs? E.g. using commands like “docker events” or “docker logs”. Using “docker diff <container>” you can check which files have changed in your container. Although not security specific, for system level monitoring you can use cAdvisor.

For security specific monitoring you can have a look at AppArmor, Auditd, Elastic Beats (e.g. the auditd module) and Sysdig Falco.

Sysdig Falco is not an enforcement tool, but an auditing and monitoring tool. With Falco you can load rules that trigger on events, e.g. when a shell was spawned or a sensitive file has been opened. Falco comes with rules, but you can also customize rules yourself.

You can even create an automated defence to e.g. deploy new versions when an alert triggers (as was shown in a demo).

7 Container design patterns you need to know — Christian Melendez (Equinix)

Most of your problems are not unique, both in software development and container usage. Having patterns makes it easier to communicate: they provide a shared language to known problems. Having patterns also allows you to reuse shared components (solutions).

Patterns for a single container

Single-container pattern

A base for other patterns.

• Establish resource boundaries
• Isolate resources from others
• Separate concerns — the container only has one responsibility

Sidecar pattern

Extend an application container with extra tools.

• Co-schedule the sidecar in the same node (same pod in the case of Kubernetes)
• Shared resources (volumes)

Ambassador pattern

The ambassador is a broker or proxy between an application and consumers.

• Modular
• Reusable in other applications

Adapter pattern

An application container to modify the interface of the application to e.g. provide a consistent interface.

• Helps to set a communication standard
• Can translate between protocols
• Reusable modules, perhaps environment variables to configure behaviour
• Examples: collect metrics, parse and store logs, custom health checks

Multi-node patterns

Used to scale at the node level for loosely coupled applications. Communication is via network calls. They are the base for microservices.

Scatter/Gather pattern

Split a big task to into smaller tasks that can run simultaneous and combine the results afterwards.

• Partial processing
• Multiple threads for requests and process
• Response time is the sum of all requests

Leader election pattern

How to select a new leader when the old leader fails.

• Reuse, don’t invent your own wheel
• Container implementations like etcd, Zookeeper and Consul

Work Queue

One manager container with multiple worker containers. The manger concurrently distributes the tasks to the workers.

How To Build An Effective Container-Based Local Development Environment — Martino Fornasa (Kiratech)

Having a local development workflow is important:

• Easier onboarding
• Consistent software dependency management
• Environment similar to production
• Local testing (faster than waiting for CI tool)
• You build knowledge

Creating a local environment is hard. Lots of variables to take into account like languages, preferences of developers, you need to keep allow a certain flexibility.

A first attempt led to issues:

• Speed
• Sync issues
• Cleanup
• Resources (CPU/memory)

And people hated it.

Goals for a proper solution:

• Tune for performance and reusability
• Allow people to use their preferred tools
• Make it easy to do the right thing.

When your target is Kubernetes, your development workflow should also take this into account. Benefits of a Kubernetes workflow:

• Easier to setup complex environments
• Same/similar tools for development and deployment
• Each developer is able to run the full system (perhaps using external systems)

Three approaches:

• Auto rebuild the image on each code change, push to registry and deploy in the cluster. Related tools: draft and skaffold. Skaffold is probably more convenient for interpreted languages.
• Run on cluster + sync the filesystem between the local system and the cluster. Higher risk of drift between local image and image built by your CI system. Tools: skaffold and ksync.
• Run locally + develop against remote resources. Tool: telepresence.

DevSecOps Toolchain: Blocking Dangerous Containers — Siamak Sadeghianfar (Red Hat) & Georgios Kryparos (Tink)

DevSecOps is a shift in both mindset (everyone is responsible for security) and security work (security as code).

We’ve gone from running applications on hardware to running them in a VM using a hypervisor. Then we started running applications in containers (which may or may not run in a VM). Now the world is even more complex with container orchestrators. In summary: many layers to take into account. And then we haven’t even touched on the registry for images…

Containers have brought good things (consistency from dev to production, faster deployments), but they have also brought security concerns:

• isolation breakouts
• outdated apps and dependencies
• secret management is an additional headache
• auditing and monitoring are still issues we need to deal with

You need vulnerability scanners, configuration auditors, secret scanners and traffic monitors on all levels: host, container engines, orchestrator and registry.

Tools you can use to help you:

• SonarQube: code analysis
• Quay Security Scanner (using Clair)
• Aqua Container Security Platform

Takeaways:

• Automation! Shift security to the left.
• Shared visibility
• Train first, enforce later
• Collaboration between dev, sec and ops teams, but also e.g. quality assurance

Zero Gain vs. Zero Loss: Practical Tales of Enterprise Decision-Making — Amélie Koran (HHS)

Leadership would like to see as little loss as possible. Selling the value of what you do on a day-to-day basis to others in the organization is important.

A “play” is an action you can take to affect a certain outcome. Examples: skills development and applicability of what you do to the mission of the business.

Play one: metaphors

Metaphors allow you to connect on a common touchstone. E.g. use physical security measures as a metaphor for “cyber” security measures; executives can conceptualize the former, but perhaps not the latter.

Don’t talk tech, talk about something they are familiar with. Examples: risk, saving money, spend a little to prevent a greater loss.

Play two: participation

Having a seat at the table is valuable. Think about design sprints, strategy sessions, budget planning sessions. This leads to deeper connections, reduces communication barriers, creates more empathy and enables awareness.

Play three: conceptualization

Defining stuff is important. Having a shared lexicon helps communication. The terms and definitions enable knowledge sharing.

Clear communication has all kinds of benefits like reducing questions, stress and anxiety.

Story time

Why do people want dashboards? They would love to have a single number to give them insight in how things are going. But what is the question that is to be answered? What decision should the dashboard support? Is what is being presented actionable?

Tips:

• Aim to reduce user errors (via good UX/UI)
• Automate tests
• Don’t deploy to production, invest in development and test environments
• Perform root cause analysis for major issues
• Include the proper team members (for instance: include security people)

Monitoring the easy way — Daniel Barker (National Association of Insurance Commissioners)

Some definitions:

Observability

Based on the output the system state should be deterministic.

Controllability

Based on the input the system state should be deterministic.

Observability tools (in implementation order):

• Metrics aggregation
• Log aggregation
• Alerting and visualizations
• Distributed tracing (harder than the others to implement, not all systems need it)

Let’s have a look at each of those tools.

Metrics

Metrics aggregation tools:

• Prometheus, used a lot with Kubernetes
• Graphite
• InfluxDB
• OpenTSDB

The two last ones are great for long term storage for Prometheus.

OpenMetrics is a metrics exposition format. Implement your metrics once and different products can understand them. Datadog is currently the only commercial product that support this standard; it is still early days though.

Logs

Log aggregation tools:

• Splunk (commercial)
• ELK stack
• Graylog
• FluentD (not log aggregation, but you can use it to replace Logstash in your ELK stack because it is a bit more efficient)

Tips:

• Assume your logs arrive in a random order in your log aggregation service so include a timestamp
• Use UTC for your timestamps
• Format logs in JSON
• Log all application errors, don’t eat them without logging them, otherwise you don’t find out about them until your users start complaining
• Turn logging on
• Make the log messages human readable because they are meant to be read by humans
• Do not log informational/debug data in production
• Do not log anything a human cannot read or react on
• Only log significant events
• Do not print personally identifiable information (PII)

Alerting and visualization

Only send alerts when action can be taken. Information is never an alert.

You would not go running around the town square yelling the weather.

Visualization: you should be able to look at it and understand the system better.

Tools:

• Prometheus AlertManager
• Bosun
• Cabot
• StatsAgg (does a lot more than just alerting)
• Grafana
• Vizceral
• Flame graphs

Distributed tracing

Track requests throughout the system

Tools:

• Zipkin
• Jaeger
• AppDash

More information in the article The open source guide to DevOps monitoring tools.

Getting started with chaos engineering — Ana Medina (Gremlin)

Definition of chaos engineering: thoughtful, planned experiments designed to reveal the weakness in our systems.

Chaos engineering is not about unexpected or unmonitored experiments. It is also not about creating outages.

Like a vaccine, we inject harm to build immunity.

Use cases: outage reproduction, on-call training, strengthen products and battle test infrastructure.

Prerequisites for chaos engineering:

• Monitoring and observability in place
• On-call and incident management (a good SRE practice in general)
• Know the cost of downtime per hour

You can inject chaos on all layers. Nice places to do it are e.g. Elasticsearch, Cassandra, etc.

To get started with chaos engineering, pick one of the top 5 most critical systems. Whiteboard the system and determine what experiment you want to run (e.g. maximize CPU, kill processes, packet loss) and determine the blast radius (for example only impact 1% of your traffic).

Chaos day is a dedicated day where the whole company focuses on resilience. There’s a tutorial on the Gremlin site if you want help to plan your own chaos day.

Examples of experiments:

• Reproduce actual outage
• Data center failure
• Large traffic spikes
• Maxing out CPU and disk

Tools:

• Chaos Monkey
• Simian Army
• Kube Monkey
• Litmus
• Powerfulseal
• Gremlin: failure as a service

What is SRE and why every DevOps professional should care — Dave Rensin (Google)

How do we choose between DevOps and SRE?

DevOps principles:

• Reduce organizational silos
• Accept failure as normal
• Implement gradual changes
• Leverage tooling and automation
• Measure everything (otherwise you run your business by luck)

SRE principles:

• Reliability is the most important feature of a system (every system, not just software).
• The only measure that actually matters is the reliability experienced by your users.
• 100% is always the wrong goal all the time. (Your users also don’t need 100% reliability, they won’t even notice it.)

The space between 100% and your availability target (e.g. 99.99%) is called the “error budget.” If you achieve higher reliability than you targeted for, you are wasting budget.

Terms:

SLI

A service level indicator tells us how our user is experiencing our system.

SLO

A service level objective is an SLI plus a goal.

SLA

A service level agreement, in contrast to SLI and SLO, points outward. SLA = SLO + margin + consequences. Purely a business discussion; there is no engineering involved.

SRE practices:

• Metrics & monitoring (are we measuring and alerting on the right thing?)
• Capacity planning
• Change management (consistency is important because you want to automate this)
• Emergency response
• Culture (toil management, engineering alignment, blamelessness)

Why choose between SRE and Devops? It is not one or the other. SRE is a concrete, highly opinionated implementation of DevOps! (Also see the video class SRE implements DevOps.) It provides a consistent and optimized way of implementing DevOps principles.

Some nice quotes:

The closer to zero you can drive the blast radius of a mistake, the closer to infinity you can drive your risk.

(And SRE manages risk.)

SRE is what happens when you ask a software engineer to design operations.

Take the SLO and calculate your error budget. Then apply risk analysis and calculate how many bad minutes/year something cost (or could cost). Then you start working on the items that are > 25% of your error budget. Tool to play with this: CRE Risk Analysis Template

How to start with SRE? The most important thing is willingness to do SRE. You cannot adopt SRE unless you have executive buy-in. (Note that you ‘only’ need buy-in, it does not work if executives enforce implementation.)

The next important thing is to pick one application (defined as a “discrete failure domain”) first. This will reduce the number of people you need buy-in from. But it has to be a meaningful application, otherwise the lessons you learn won’t be internalized. Once the reliability and velocity are increasing and the frustration between dev and ops is decreasing, others will want to do it too!

When you have the willingness and know which application to work on, you start with defining your SLIs, SLOs and error budgets. Answer the question “what do my customers care about?” Pick the smallest number of things that answer that question. So not “CPU utilization” but e.g. “latency of the application.” Then determine the first set of objectives. Also determine a policy, for instance: feature freezes until you go below the error budget.

Note that the SLOs don’t have to be perfect. You can re-evaluate them periodically (monthly or quarterly for instance).

Now, before you do any engineering, get your alerting and monitoring in order. Make sure you monitor the things that are relevant for your SLOs. Usually you can turn off half of your alerts without impacting the experience of your user. More alerting is almost never better. More logging and measurements are (probably) better. Simple test: only alert on something your customer would want to be alerted on.

An important prerequisite for doing SRE is having a blameless culture! The highest return on investment of a mistake is if you learn as much as possible from it. You can only do that in a blameless culture.

Take one step at a time:

1. Define SLIs, SLOs and error budgets
2. Audit and adjust monitoring and alerting
3. Model and reward blameless postmortems

If you do this, you will improve things!

More information: sre.google

DevSecOps - securing a great developer experience — Stefan Streichsbier (GuardRails)

Some stats: 51% of the world’s population has internet access (June 2017) and there are 31 million developers on GitHub (October 2018).

Having a website used to be simple: you uploaded the needed files to your hosting provider via FTP. There was no to little collaboration (just one “web master”), no build steps (PHP, ASP, etc) and you tested by checking if there were parse errors.

Now there is a whole bunch of additional (and different) steps required. There are code repositories, build pipelines, several environments (test, production). Deployment on e.g. a Kubernetes cluster is much more complex than putting some files somewhere. And then there’s the complexity of all the development tools and all the products that the cloud providers offer.

So how does security fit in?

Michael Wittig wrote an AWS security primer in which he has a mind map of all security related tasks and activities there are in AWS. On Hacker News someone posted a comment on it:

I’ve worked extensively with AWS over the last 4 years, and I can barely wrap my head around the scope of managing security in AWS. We have an entire department dedicated to security in our company, and none of them are remotely close to being experts in AWS security either.

I’m starting to get curious if there even is an expert who could set up and maintain a bulletproof AWS Account.

Steps in the evolution of security:

• Penetration testing
• Secure software development life cycle
• DevSecOps

Going to DevSecOps is not easy for typical security teams. They have to get into the mindset of software development. Security tools are typically also not made with developers in mind.

Tools are not everything. We need to have a proper workflow around it. We should not “automate the nagging” by building in security tools in the development pipeline.

Only a fraction of the 31 million developers on GitHub have access to (enterprise grade) security tools and penetration testing.

Quality had to face some of the challenges security is currently facing. Quality is now more and more a first class citizen.

Some definitions:

• Usability: can the user accomplish his/her goal very effectively when they interface with a product?
• User experience: everything around using a certain product
• Developer experience: user experience where your audience is exclusively developers.

To increase the developer experience with regards to security we have to keep some things in mind:

• Signal vs noise
  • Security tools should not add more noise.
  • Instead they should focus on high impact issues.
  • Make sure that issues are reported with a high level of accuracy.
• Lost in translation
  • Security has to speak the same language as developers. Otherwise developers have to become security experts as well to understand what the security team is talking about.
  • Use the right communication channel. So not a PDF report or Excel sheet per email, but notifications in the workflow (e.g. in a pull request, issue, chat application).
  • Describe the issues clear so that developer know how to solve them.
• Make it easy
  • Make it simple to get started and try out (not: “schedule an appointment with our sales rep”)
  • Tight integration with the workflow
  • Provide all functionality in one tool.

How to make security a part of DevOps?

• Acknowledge that developers are key. Tools should bring them joy using it and provide value.
• Respect that security is a complex topic but provide clear focus and make it easy to consume for developers.
• Security has to become a commodity, not just for big enterprises.