
Title Text

The State of Kubernetes Development Tooling
October 2019

Kubernetes the hard way falsely implies that there's an easy way.


@pczarkowski
@ellenkorbes

@ellenkorbes
- Run on Kubernetes.
- Are not Kubernetes itself. E.g. CRDs & operators.
Who is this talk for?

@ellenkorbes
Developers working on apps that...
- Their pre-commit workflow.
- The part of their day where code gets written.
Who is this talk for?

- Their CI/CD pipelines.
- Ops stuff not directly related to writing code.
But not necessarily...
Developers looking to improve...
@ellenkorbes

You put the whole app in one container?!


@7hunderbird
@ellenkorbes

- Provisioning
- Configuration files
- Bootstrap
- Write code
- Build
- Deploy to dev
- Test
- Debug
- Git & CI
- Ship to prod. & CD
- Observability
Development
Cycle:
@ellenkorbes
Pre-Commit Stage:

Provisioning- Configuration files
- Bootstrap
- Write code
- Build
- Deploy to dev
- Test
- Debug
Git & CIShip to prod. & CDObservability
@ellenkorbes

- Configuration files
- Bootstrap
- Write code
- Build
- Deploy to dev
- Test
- Debug
Setup
Dev
Feedback
@ellenkorbes

- Configuration files
- Bootstrap
- Write code
- Build
- Deploy to dev
- Test
- Debug
Wishlist
- Manageable size
- Create them for me?
@ellenkorbes

- Setup tasks
(like migrations)
@ellenkorbes
- Configuration files
- Bootstrap
- Write code
- Build
- Deploy to dev
- Test
- Debug
Wishlist

- Automate docker/kubectl cycle
- Short feedback loop
- Unobtrusive
- Hot reload
- Easy to on-board
- Dependency-aware
@ellenkorbes
- Configuration files
- Bootstrap
- Write code
- Build
- Deploy to dev
- Test
- Debug
Wishlist

- Do testing
- Highlight failures
- End-to-end tests
@ellenkorbes
- Configuration files
- Bootstrap
- Write code
- Build
- Deploy to dev
- Test
- Debug
Wishlist

- Log streaming
- Highlight issues
- Allow breakpoints
- Not tracing
@ellenkorbes
- Configuration files
- Bootstrap
- Write code
- Build
- Deploy to dev
- Test
- Debug
Wishlist

Maybe containers were a mistake.


@jessfraz
@ellenkorbes

@ellenkorbes
- Debug
- Connect
- Build & Deploy Cycle
Categories

- Draft
- Garden
- Skaffold
- Tilt
- Jenkins X
- Forge
- Helm
@ellenkorbes
Tools
- ksync
- kubefwd
- Okteto
- Telepresence
- inlets
- Squash
- Stern
- Debug
- Connect
- Build & Deploy Cycle
Categories

- kubeval
- yq
- kubectx
- kubens
- ytt
- kpxg
- kbld
- kapp
- kwt
- ...
Others...
@ellenkorbes

@ellenkorbes
Trivia:
Which of these
isn't a real tool?
- kubeval
- yq
- kubectx
- kubens
- ytt
- kpxg
- kbld
- kapp
- kwt
- ...
Others...

Kubernetes: complicating simple things since 2014.


@texmandie
@ellenkorbes

Debug
- kubectl
- Squash
- Stern

- Easily shows you the status of your pods, deployments, services.
- Learn it.
- kubectl logs --follow -n [namespace] [pod]
Notes
@ellenkorbes

@ellenkorbes

Debug
- kubectl
- Squash
- Stern

- Steps and breakpoints on deployed containers!
- No more Printf debugging 🙌
- IDE or CLI
- Delve, gdb, Java, Node, Python
- Fun!

@ellenkorbes
Notes:
Debug
- kubectl
- Squash
- Stern





@ellenkorbes
Debug
- kubectl
- Squash
- Stern

- Streams logs from multiple pods
- Color-coded
- Easy to filter
- It's the best
- I love it 💜
- Use it!
@ellenkorbes
Notes:
Debug
- kubectl
- Squash
- Stern


@ellenkorbes
Debug
- kubectl
- Squash
- Stern


- "[...] think of it as a very fancy shell script that does a docker build, docker push, kubectl apply [...]" — rdldw
- Builds and deploys! Doesn't watch.
- Requires YAML, Dockerfile, Forge config.
- Allows profiles e.g. dev/canary/production.
- Development seems stopped 😢
@ellenkorbes
Notes:
Connect
- Forge
- Helm
- inlets
- ksync
- kubefwd
- Okteto
- Telepresence

- Package manager for Kubernetes application structure.
- Install and uninstall off-the-shelf apps. Think apt-get, brew, npm, pip, gem.
- App lifecycle management. Deploy a service's new version, roll back, delete.
- Omnipresent: Haven't used it? You will.
- Pro & con: Go templating on configs.

@ellenkorbes
Notes:
- Forge
- Helm
- inlets
- ksync
- kubefwd
- Okteto
- Telepresence
Connect

- Reverse proxy & websocket tunnel to expose internal endpoints to the internet.
- Similar to ngrok or Argo Tunnel, but free and open source.
- Has a Kubernetes operator so you can seamlessly expose a local Kubernetes cluster.
- Use it to share works in progress, set up webhooks in development, and so on.
- ACHTUNG! You're exposing internals to the internet. Be mindful of security.
@ellenkorbes
Notes:
- Forge
- Helm
- inlets
- ksync
- kubefwd
- Okteto
- Telepresence
Connect

@ellenkorbes
- inlets


- Watches your file system
- Syncs local files with files in your cluster
- Straight forward and easy to use
- Built on top of Syncthing
- You do need a watcher inside the container to act on the updated files!

@ellenkorbes
Notes:
- Forge
- Helm
- inlets
- ksync
- kubefwd
- Okteto
- Telepresence
Connect

- Forward all possible ports—as opposed to kubectl which forwards individual ports.
- Access any service locally like a pod in the cluster
- Easy! Fun!
Buy it now! (Just kidding.)

@ellenkorbes
Notes:
- Forge
- Helm
- inlets
- ksync
- kubefwd
- Okteto
- Telepresence
Connect


- Forge
- Helm
- inlets
- ksync
- kubefwd
- Okteto
- Telepresence
Connect

- Swaps a deployment for a development environment.
- Syncs & watches files, gives you a remote terminal, does port forwarding.
- Ideal to build admission controllers, operators, things deeply integrated with k8s.
- One service at a time.

@ellenkorbes
Notes:
- Forge
- Helm
- inlets
- ksync
- kubefwd
- Okteto
- Telepresence
Connect
- Hot swaps a running pod and creates a local shell where the network is proxied into the cluster.
- Run app locally; it acts as if in the cluster.
- Use local IDE, debugger, etc.
- Vs. Squash: Single instance. Might cause conflicts like env vars and docker version.
- Vs. Okteto: Local machine pretends to be a node vs. moving dev into the cluster.
@ellenkorbes


Notes:
- Forge
- Helm
- inlets
- ksync
- kubefwd
- Okteto
- Telepresence
Connect

Build & Deploy
- Draft
- Garden
- Jenkins X
- Skaffold
- Tilt

@ellenkorbes
- Generates config files: Great if you're new to containers or Kubernetes! Helm based.
- Supports bootstrapping tasks.
- Works per-service: You must "draft up" for every service.
- "The team originally working on Draft have to put it on the backburner while focusing on efforts like Brigade, Helm 3, and CNAB."
- No hot reload!
Notes:

@ellenkorbes




Build & Deploy
- Draft
- Garden
- Jenkins X
- Skaffold
- Tilt

@ellenkorbes
- No Kubernetes manifests required. (Unless you want to!)
- Allows in-cluster builds. Local Kubernetes, Docker, minikube not required.
- Project awareness: It tracks build and deploy dependencies.
- Tests are first-class citizens. Locally & CI.
- Pluggable: Custom module types for e.g. OpenFaaS, Java, Terraform. (Soon also Go!)
Notes:
Build & Deploy
- Draft
- Garden
- Jenkins X
- Skaffold
- Tilt

@ellenkorbes

Build & Deploy
- Draft
- Garden
- Jenkins X
- Skaffold
- Tilt

@ellenkorbes

Build & Deploy
- Draft
- Garden
- Jenkins X
- Skaffold
- Tilt

@ellenkorbes

Build & Deploy
- Draft
- Garden
- Jenkins X
- Skaffold
- Tilt

@ellenkorbes

Build & Deploy
- Draft
- Garden
- Jenkins X
- Skaffold
- Tilt

@ellenkorbes

- Something about provisioning.
- Something about CI.
- Uses Skaffold and ksync in the background.
- Has build packs that generate config files similarly to Draft! 😊
Notes:
Build & Deploy
- Draft
- Garden
- Jenkins X
- Skaffold
- Tilt


@ellenkorbes
- Automatically builds & deploys!
- For those already familiar with k8s, it doesn't require learning new concepts.
- Requires k8s manifests! Not for beginners.
- Easy set up if you already have them.
- No support for dependencies, tests, or bootstrapping.
Notes:
Build & Deploy
- Draft
- Garden
- Jenkins X
- Skaffold
- Tilt


Build & Deploy
- Draft
- Garden
- Jenkins X
- Skaffold
- Tilt


Build & Deploy
- Draft
- Garden
- Jenkins X
- Skaffold
- Tilt


@ellenkorbes
- Like Skaffold: Requires k8s manifests; easy set up if you already have them.
- Gorgeous UI, dashboard, and logging.
- Vs. Skaffold? Skaffold's concepts map almost directly into Tilt.
- Starlark configs!
Notes:
Build & Deploy
- Draft
- Garden
- Jenkins X
- Skaffold
- Tilt

@ellenkorbes



Build & Deploy
- Draft
- Garden
- Jenkins X
- Skaffold
- Tilt





@ellenkorbes

Build & Deploy
- Draft
- Garden
- Jenkins X
- Skaffold
- Tilt

@ellenkorbes

Build & Deploy
- Draft
- Garden
- Jenkins X
- Skaffold
- Tilt

Toolchains:
- Helm
- kubectl
- Draft ☠️
- Garden
- Jenkins X
- Skaffold
- Tilt
- Forge ☠️
- inlets
- ksync
- kubefwd
- Okteto
- Telepresence
- kubefwd
- Squash
- Stern
Debug:
Connect:
Build&Deploy Cycle:
@ellenkorbes
Omnipresent:

It works on my cluster...


@maeddes
@ellenkorbes

Still missing:
- Better docs!
- Consistent naming for things across different tools.
- Better support for tests!
- Beginner friendliness.
- Clarity. A cohesive story.
?
@ellenkorbes
(ls *.go | entr -r ./mytest.sh)

- Draft → draft.sh
- entr → eradman.com/entrproject/
- Forge → forge.sh
- Garden → garden.io
- Helm → helm.sh
- inlets → github.com/inlets/inlets & inlets-operator
- Jenkins X → jenkins-x.io
- ksync → github.com/vapor-ware/ksync
- kubectl 'Debug Services' → kubernetes.io/docs/tasks/debug-application-cluster/debug-service/
- kubefwd → kubefwd.com
- Okteto → okteto.com
- Skaffold → skaffold.dev
- Squash → github.com/solo-io/squash
- Stern → github.com/wercker/stern
- Telepresence → www.telepresence.io
- Tilt → tilt.dev
Links:
@ellenkorbes



@krisnova
You can't have a clusterf!$& without a cluster.
@ellenkorbes

- developer relations
- ellen@garden.io
- @ellenkorbes
- #garden@slack.k8s.io
- they/them
ellen körbes

ellen@garden.io | @ellenkorbes

The State of Kubernetes Development Tooling v2 GOTOBLN
By ellenkorbes