Title Text

The State of Kubernetes Development Tooling

October 2019

Kubernetes the hard way falsely implies that there's an easy way.

@pczarkowski

@ellenkorbes

@ellenkorbes

  • Run on Kubernetes.
  • Are not Kubernetes itself. E.g. CRDs & operators.

Who is this talk for?

@ellenkorbes

Developers working on apps that...

  • Their pre-commit workflow.
  • The part of their day where code gets written.

Who is this talk for?

  • Their CI/CD pipelines.
  • Ops stuff not directly related to writing code.

But not necessarily...

Developers looking to improve...

@ellenkorbes

You put the whole app in one container?!

@7hunderbird

 

@ellenkorbes

  • Provisioning
  • Configuration files
  • Bootstrap
  • Write code
  • Build
  • Deploy to dev
  • Test
  • Debug
  • Git & CI
  • Ship to prod. & CD
  • Observability

Development

Cycle:

@ellenkorbes

Pre-Commit Stage:

  • Provisioning
  • Configuration files
  • Bootstrap
  • Write code
  • Build
  • Deploy to dev
  • Test
  • Debug
  • Git & CI
  • Ship to prod. & CD
  • Observability

@ellenkorbes

  • Configuration files
  • Bootstrap
  • Write code
  • Build
  • Deploy to dev
  • Test
  • Debug

Setup

Dev

Feedback

@ellenkorbes

  • Configuration files
  • Bootstrap
  • Write code
  • Build
  • Deploy to dev
  • Test
  • Debug

Wishlist

  • Manageable size
  • Create them for me?

@ellenkorbes

  • Setup tasks
    (like migrations)

@ellenkorbes

  • Configuration files
  • Bootstrap
  • Write code
  • Build
  • Deploy to dev
  • Test
  • Debug

Wishlist

  • Automate docker/kubectl cycle
  • Short feedback loop
  • Unobtrusive
  • Hot reload
  • Easy to on-board
  • Dependency-aware

@ellenkorbes

  • Configuration files
  • Bootstrap
  • Write code
  • Build
  • Deploy to dev
  • Test
  • Debug

Wishlist

  • Do testing
  • Highlight failures
  • End-to-end tests

@ellenkorbes

  • Configuration files
  • Bootstrap
  • Write code
  • Build
  • Deploy to dev
  • Test
  • Debug

Wishlist

  • Log streaming
  • Highlight issues
  • Allow breakpoints
  • Not tracing

@ellenkorbes

  • Configuration files
  • Bootstrap
  • Write code
  • Build
  • Deploy to dev
  • Test
  • Debug

Wishlist

Maybe containers were a mistake.

@jessfraz

@ellenkorbes

@ellenkorbes

  • Debug
  • Connect
  • Build & Deploy Cycle

Categories

  • Draft
  • Garden
  • Skaffold
  • Tilt
  • Jenkins X
  • Forge
  • Helm

@ellenkorbes

Tools

  • ksync
  • kubefwd
  • Okteto
  • Telepresence
  • inlets
  • Squash
  • Stern
  • Debug
  • Connect
  • Build & Deploy Cycle

Categories

  • kubeval
  • yq
  • kubectx
  • kubens
  • ytt
  • kpxg
  • kbld
  • kapp
  • kwt
  • ...

Others...

@ellenkorbes

@ellenkorbes

Trivia:

Which of these

isn't a real tool?

  • kubeval
  • yq
  • kubectx
  • kubens
  • ytt
  • kpxg
  • kbld
  • kapp
  • kwt
  • ...

Others...

Kubernetes: complicating simple things since 2014.

@texmandie

@ellenkorbes

Debug

  • kubectl
  • Squash
  • Stern
  • Easily shows you the status of your pods, deployments, services.
  • Learn it.
  • kubectl logs --follow -n [namespace] [pod]

Notes

@ellenkorbes

@ellenkorbes

Debug

  • kubectl
  • Squash
  • Stern
  • Steps and breakpoints on deployed containers!
  • No more Printf debugging 🙌
  • IDE or CLI
  • Delve, gdb, Java, Node, Python
  • Fun!

@ellenkorbes

Notes:

Debug

  • kubectl
  • Squash
  • Stern

@ellenkorbes

Debug

  • kubectl
  • Squash
  • Stern
  • Streams logs from multiple pods
  • Color-coded
  • Easy to filter
  • It's the best
  • I love it 💜
  • Use it!

@ellenkorbes

Notes:

Debug

  • kubectl
  • Squash
  • Stern

@ellenkorbes

Debug

  • kubectl
  • Squash
  • Stern
  • "[...] think of it as a very fancy shell script that does a docker build, docker push, kubectl apply [...]" — rdldw
  • Builds and deploys! Doesn't watch.
  • Requires YAML, Dockerfile, Forge config.
  • Allows profiles e.g. dev/canary/production.
  • Development seems stopped 😢

@ellenkorbes

Notes:

Connect

  • Forge
  • Helm
  • inlets
  • ksync
  • kubefwd
  • Okteto
  • Telepresence
  • Package manager for Kubernetes application structure.
  • Install and uninstall off-the-shelf apps. Think apt-get, brew, npm, pip, gem.
  • App lifecycle management. Deploy a service's new version, roll back, delete.
  • Omnipresent: Haven't used it? You will.
  • Pro & con: Go templating on configs.

@ellenkorbes

Notes:

  • Forge
  • Helm
  • inlets
  • ksync
  • kubefwd
  • Okteto
  • Telepresence

Connect

  • Reverse proxy & websocket tunnel to expose internal endpoints to the internet.
  • Similar to ngrok or Argo Tunnel, but free and open source.
  • Has a Kubernetes operator so you can seamlessly expose a local Kubernetes cluster.
  • Use it to share works in progress, set up webhooks in development, and so on.
  • ACHTUNG! You're exposing internals to the internet. Be mindful of security.

@ellenkorbes

Notes:

  • Forge
  • Helm
  • inlets
  • ksync
  • kubefwd
  • Okteto
  • Telepresence

Connect

@ellenkorbes

  • inlets
  • Watches your file system
  • Syncs local files with files in your cluster
  • Straight forward and easy to use
  • Built on top of Syncthing
  • You do need a watcher inside the container to act on the updated files!

@ellenkorbes

Notes:

  • Forge
  • Helm
  • inlets
  • ksync
  • kubefwd
  • Okteto
  • Telepresence

Connect

  • Forward all possible ports—as opposed to kubectl which forwards individual ports.
  • Access any service locally like a pod in the cluster
  • Easy! Fun! Buy it now! (Just kidding.)

@ellenkorbes

Notes:

  • Forge
  • Helm
  • inlets
  • ksync
  • kubefwd
  • Okteto
  • Telepresence

Connect

  • Forge
  • Helm
  • inlets
  • ksync
  • kubefwd
  • Okteto
  • Telepresence

Connect

  • Swaps a deployment for a development environment.
  • Syncs & watches files, gives you a remote terminal, does port forwarding.
  • Ideal to build admission controllers, operators, things deeply integrated with k8s.
  • One service at a time.

@ellenkorbes

Notes:

  • Forge
  • Helm
  • inlets
  • ksync
  • kubefwd
  • Okteto
  • Telepresence

Connect

  • Hot swaps a running pod and creates a local shell where the network is proxied into the cluster.
  • Run app locally; it acts as if in the cluster.
  • Use local IDE, debugger, etc.
  • Vs. Squash: Single instance. Might cause conflicts like env vars and docker version.
  • Vs. Okteto: Local machine pretends to be a node vs. moving dev into the cluster.

@ellenkorbes

Notes:

  • Forge
  • Helm
  • inlets
  • ksync
  • kubefwd
  • Okteto
  • Telepresence

Connect

Build & Deploy

  • Draft
  • Garden
  • Jenkins X
  • Skaffold
  • Tilt

@ellenkorbes

  • Generates config files: Great if you're new to containers or Kubernetes! Helm based.
  • Supports bootstrapping tasks.
  • Works per-service: You must "draft up" for every service.
  • "The team originally working on Draft have to put it on the backburner while focusing on efforts like Brigade, Helm 3, and CNAB."
  • No hot reload!

Notes:

@ellenkorbes

Build & Deploy

  • Draft
  • Garden
  • Jenkins X
  • Skaffold
  • Tilt

@ellenkorbes

  • No Kubernetes manifests required. (Unless you want to!)
  • Allows in-cluster builds. Local Kubernetes, Docker, minikube not required.
  • Project awareness: It tracks build and deploy dependencies.
  • Tests are first-class citizens. Locally & CI.
  • Pluggable: Custom module types for e.g. OpenFaaS, Java, Terraform. (Soon also Go!)

Notes:

Build & Deploy

  • Draft
  • Garden
  • Jenkins X
  • Skaffold
  • Tilt

@ellenkorbes

Build & Deploy

  • Draft
  • Garden
  • Jenkins X
  • Skaffold
  • Tilt

@ellenkorbes

Build & Deploy

  • Draft
  • Garden
  • Jenkins X
  • Skaffold
  • Tilt

@ellenkorbes

Build & Deploy

  • Draft
  • Garden
  • Jenkins X
  • Skaffold
  • Tilt

@ellenkorbes

Build & Deploy

  • Draft
  • Garden
  • Jenkins X
  • Skaffold
  • Tilt

@ellenkorbes

  • Something about provisioning.
  • Something about CI.
  • Uses Skaffold and ksync in the background.
  • Has build packs that generate config files similarly to Draft! 😊

Notes:

Build & Deploy

  • Draft
  • Garden
  • Jenkins X
  • Skaffold
  • Tilt

@ellenkorbes

  • Automatically builds & deploys!
  • For those already familiar with k8s, it doesn't require learning new concepts.
  • Requires k8s manifests! Not for beginners.
  • Easy set up if you already have them.
  • No support for dependencies, tests, or bootstrapping.

Notes:

Build & Deploy

  • Draft
  • Garden
  • Jenkins X
  • Skaffold
  • Tilt

Build & Deploy

  • Draft
  • Garden
  • Jenkins X
  • Skaffold
  • Tilt

Build & Deploy

  • Draft
  • Garden
  • Jenkins X
  • Skaffold
  • Tilt

@ellenkorbes

  • Like Skaffold: Requires k8s manifests; easy set up if you already have them.
  • Gorgeous UI, dashboard, and logging.
  • Vs. Skaffold? Skaffold's concepts map almost directly into Tilt. 
  • Starlark configs!

Notes:

Build & Deploy

  • Draft
  • Garden
  • Jenkins X
  • Skaffold
  • Tilt

@ellenkorbes

Build & Deploy

  • Draft
  • Garden
  • Jenkins X
  • Skaffold
  • Tilt

@ellenkorbes

Build & Deploy

  • Draft
  • Garden
  • Jenkins X
  • Skaffold
  • Tilt

@ellenkorbes

Build & Deploy

  • Draft
  • Garden
  • Jenkins X
  • Skaffold
  • Tilt

Toolchains:

  • Helm
  • kubectl
  • Draft ☠️
  • Garden
  • Jenkins X
  • Skaffold
  • Tilt
  • Forge ☠️
  • inlets
  • ksync
  • kubefwd
  • Okteto
  • Telepresence
  • kubefwd
  • Squash
  • Stern

Debug:

Connect:

Build&Deploy Cycle:

@ellenkorbes

Omnipresent:

It works on my cluster...

@maeddes

 

@ellenkorbes

Still missing:

  • Better docs!
  • Consistent naming for things across different tools.
  • Better support for tests!
  •  
  • Beginner friendliness.
  • Clarity. A cohesive story.

?

@ellenkorbes

(ls *.go | entr -r ./mytest.sh)

Links:

@ellenkorbes

@krisnova

You can't have a clusterf!$& without a cluster.

@ellenkorbes

  • developer relations
  • ellen@garden.io
  • @ellenkorbes
  • #garden@slack.k8s.io
  • they/them

ellen körbes

ellen@garden.io | @ellenkorbes

The State of Kubernetes Development Tooling v2 GOTOBLN

By ellenkorbes

Private

The State of Kubernetes Development Tooling v2 GOTOBLN

More from ellenkorbes