Implement fast multi-node via an in-cluster proxy

.dockerignore

@@ -0,0 +1 @@
+bin/

.github/workflows/pull_request.yaml

@@ -24,7 +24,7 @@ jobs:
     - name: Unit Tests
       run: make test
     - name: Codecov
-      uses: codecov/codecov-action@v1
+      uses: codecov/codecov-action@v2
       with:
         file: cover-unit.out
         flags: unit-tests
@@ -41,6 +41,10 @@ jobs:
       name: Check out code into the Go module directory
       with:
         fetch-depth: 0
+    - name: Build proxy image and load in containerd
+      run: |
+        make BUILD_CMD="docker build" image save-image
+        sudo ctr --namespace k8s.io image import ./bin/buildkit_proxy_image.tar
     - name: Setup containerd cluster
       run: |
         set -x
@@ -62,9 +66,9 @@ jobs:
         kubectl wait --for=condition=ready --timeout=30s node --all
         kubectl get nodes -o wide
     - name: Run integration tests
-      run: make integration EXTRA_GO_TEST_FLAGS=-v
+      run: make integration TEST_FLAGS=-v
     - name: Gather integration coverage results
-      uses: codecov/codecov-action@v1
+      uses: codecov/codecov-action@v2
       with:
         file: cover-int.out
         flags: integration-tests
@@ -81,6 +85,8 @@ jobs:
       name: Check out code into the Go module directory
       with:
         fetch-depth: 0
+    - name: Build proxy image
+      run: make BUILD_CMD="docker build" image
     - name: Setup kubeadm cluster with default docker runtime
       run: |
         set -x
@@ -104,9 +110,9 @@ jobs:
         kubectl get nodes -o wide
 
     - name: Run integration tests
-      run: make integration EXTRA_GO_TEST_FLAGS=-v
+      run: make integration TEST_FLAGS=-v
     - name: Gather integration coverage results
-      uses: codecov/codecov-action@v1
+      uses: codecov/codecov-action@v2
       with:
         file: cover-int.out
         flags: integration-tests
@@ -136,7 +142,7 @@ jobs:
     - uses: actions/checkout@v2
       name: Check out code into the Go module directory
     - name: Build
-      run: make dist
+      run: make BUILD_CMD="docker build" dist
     - uses: actions/upload-artifact@v2
       with:
         name: darwin.tgz
@@ -151,4 +157,4 @@ jobs:
       with:
         name: windows.tgz
         path: bin/windows.tgz
-        retention-days: 1
+        retention-days: 1

CONTRIBUTING.md

@@ -25,6 +25,17 @@ make build
 sudo make install
 ```
 
+The proxy image version is baked into the CLI, and derived from your local git repo state.  When developing locally, this can lead to the builder failing to start due to a missing proxy image.  The simplest option is to override the proxy image to use an official published image, then you can build your local image with the builder.  After you build the image, then your local CLI will create functional builders as the image is present in the local runtime.
+
+```
+# Force the builder to use an existing published proxy image
+kubectl buildkit create --proxy-image ghcr.io/vmware-tanzu/buildkit-proxy:v0.2.0
+
+# Use the builder you just created to build the local image
+make image
+```
+
+
 To run the **unit tests**, run
 ```
 make test
@@ -37,7 +48,7 @@ make integration
 
 If you want to run a single suite of tests while working on a specific area of the tests or main code, use something like this:
 ```
-make integration EXTRA_GO_TEST_FLAGS="-run TestConfigMapSuite -v"
+make integration TEST_FLAGS="-run TestConfigMapSuite -v"
 ```
 Hint: find the current test suites with `grep "func Test" integration/suites/*.go`
 
@@ -46,6 +57,18 @@ To check your code for **lint/style consistency**, run
 make lint
 ```
 
+If you have a custom buildkit image you want to test with, (e.g. `docker.io/moby/buildkit:local`)
+you can pass this to the integration tests (excluding "default" buidler tests) with:
+```
+make integration TEST_ALT_BUILDKIT_IMAGE=docker.io/moby/buildkit:local
+```
+
+### Docker Hub Rate Limiting
+The Makefile for this project has an optional prefix variable set up to allow use of a Docker Hub Proxy to mitigate rate limiting.  For example, if you use Harbor, this guide can help you set up a proxy for your organization: https://tanzu.vmware.com/developer/guides/harbor-as-docker-proxy/ 
+```
+export DOCKER_HUB_LIBRARY_PROXY_CACHE=registry.acme.com/dockerhub-proxy-cache/library/
+```
+
 ## Reporting Issues
 
 A great way to contribute to the project is to send a detailed report when you run into problems.

Makefile

@@ -1,11 +1,25 @@
 # Copyright (C) 2020 VMware, Inc.
 # SPDX-License-Identifier: Apache-2.0
 
-VERSION?=$(shell git describe --tags --first-parent --abbrev=7 --long --dirty --always)
-TEST_KUBECONFIG?=$(HOME)/.kube/config
+VERSION?=dev
 TEST_TIMEOUT=600s
 TEST_PARALLELISM=3
 
+# Set this to your favorite proxy cache if you experience rate limiting problems with Docker Hub.
+# Remember to include the prefix for library images
+DOCKER_HUB_LIBRARY_PROXY_CACHE?=
+export DOCKER_HUB_LIBRARY_PROXY_CACHE
+# TODO wire this up so it's passed into the build of the binaries instead of replicating the hardcoded string
+BUILDKIT_PROXY_IMAGE=ghcr.io/vmware-tanzu/buildkit-proxy
+TEST_IMAGE_BASE=$(DOCKER_HUB_LIBRARY_PROXY_CACHE)busybox
+BUILDER_BASE?=$(DOCKER_HUB_LIBRARY_PROXY_CACHE)golang:1.17-bullseye
+RUNTIME_BASE?=$(DOCKER_HUB_LIBRARY_PROXY_CACHE)debian:bullseye-slim
+
+
+export TEST_IMAGE_BASE 
+BUILD_CMD=kubectl buildkit build
+PUSH_CMD=docker push
+
 # Verify Go in PATH
 ifeq (, $(shell which go))
 $(error You must install Go to build - https://golang.org/dl/ )
@@ -26,10 +40,12 @@ CI_BUILD_TARGETS=$(foreach os,$(CI_OSES),\
 CI_ARCHIVES=$(foreach os,$(CI_OSES),$(BIN_DIR)/$(os).tgz)
 
 GO_MOD_NAME=github.com/vmware-tanzu/buildkit-cli-for-kubectl
-GO_DEPS=$(foreach dir,$(shell go list -deps -f '{{.Dir}}' ./cmd/kubectl-buildkit ./cmd/kubectl-build),$(wildcard $(dir)/*.go)) Makefile
+GO_DEPS=$(foreach dir,$(shell go list -deps -f '{{.Dir}}' ./cmd/kubectl-buildkit ./cmd/kubectl-build ./cmd/buildkit-proxy),$(wildcard $(dir)/*.go)) Makefile
 REVISION=$(shell git describe --match 'v[0-9]*' --always --dirty --tags)
-GO_FLAGS=-ldflags "-X $(GO_MOD_NAME)/version.Version=${VERSION}" -mod=vendor
+GO_FLAGS=-ldflags "-X $(GO_MOD_NAME)/version.Version=${VERSION} -X $(GO_MOD_NAME)/version.DefaultHelperImage=$(BUILDKIT_PROXY_IMAGE)" -mod=vendor
 GO_COVER_FLAGS=-cover -coverpkg=./... -covermode=count
+UNIT_TEST_PACKAGES=$(shell go list ./... | grep -v "/integration/")
+COVERAGE_FILTERS=grep -v "\.pb\.go" | grep -v "/integration/"
 
 .PHONY: help
 help:
@@ -41,14 +57,36 @@ clean:
 	-rm -rf $(BIN_DIR) cover*.out cover*.html
 
 .PHONY: build
-build: $(BIN_DIR)/$(NATIVE_ARCH)/kubectl-buildkit $(BIN_DIR)/$(NATIVE_ARCH)/kubectl-build
+build: $(BIN_DIR)/$(NATIVE_ARCH)/kubectl-buildkit $(BIN_DIR)/$(NATIVE_ARCH)/kubectl-build $(BIN_DIR)/$(NATIVE_ARCH)/buildkit-proxy
 
 $(BIN_DIR)/%/kubectl-buildkit $(BIN_DIR)/%/kubectl-buildkit.exe: $(GO_DEPS)
 	GOOS=$* go build $(GO_FLAGS) -o $@ ./cmd/kubectl-buildkit
 
 $(BIN_DIR)/%/kubectl-build $(BIN_DIR)/%/kubectl-build.exe: $(GO_DEPS)
 	GOOS=$* go build $(GO_FLAGS) -o $@  ./cmd/kubectl-build
 
+$(BIN_DIR)/%/buildkit-proxy $(BIN_DIR)/%/buildkit-proxy.exe: $(GO_DEPS)
+	GOOS=$* go build $(GO_FLAGS) -o $@  ./cmd/buildkit-proxy
+
+.PHONY: image
+image:
+	$(BUILD_CMD) -t $(BUILDKIT_PROXY_IMAGE):$(VERSION) \
+		--build-arg BUILDKIT_PROXY_IMAGE=$(BUILDKIT_PROXY_IMAGE) \
+		--build-arg VERSION=$(VERSION) \
+		--build-arg BUILDER_BASE=$(BUILDER_BASE) \
+		--build-arg RUNTIME_BASE=$(RUNTIME_BASE) \
+		-f ./builder/Dockerfile .
+
+# TODO refine so this can support native kubectl build/save
+.PHONY: save-image
+save-image:
+	@mkdir -p $(BIN_DIR)
+	docker save $(BUILDKIT_PROXY_IMAGE):$(VERSION) > $(BIN_DIR)/buildkit_proxy_image.tar
+
+.PHONY: push
+push:
+	$(PUSH_CMD) $(BUILDKIT_PROXY_IMAGE):$(VERSION)
+
 install: $(BIN_DIR)/$(NATIVE_ARCH)/kubectl-buildkit $(BIN_DIR)/$(NATIVE_ARCH)/kubectl-build
 	cp $(BIN_DIR)/$(NATIVE_ARCH)/kubectl-buildkit $(BIN_DIR)/$(NATIVE_ARCH)/kubectl-build $(INSTALL_DIR)
 
@@ -59,33 +97,46 @@ print-%:
 build-ci: $(CI_BUILD_TARGETS)
 
 .PHONY: dist
-dist: $(CI_BUILD_TARGETS) $(CI_ARCHIVES)
+dist: $(CI_BUILD_TARGETS) $(CI_ARCHIVES) image save-image
 
 $(BIN_DIR)/%.tgz: $(BIN_DIR)/%/*
 	cd $(BIN_DIR)/$* && tar -czvf ../$*.tgz kubectl-*
 
+.PHONY: generate
+generate:
+	go generate ./...
+
+
 .PHONY: test
 test:
-	go test $(GO_FLAGS) $(GO_COVER_FLAGS) -coverprofile=./cover-unit.out ./...
+	go test $(GO_FLAGS) \
+		-parallel $(TEST_PARALLELISM) \
+		$(TEST_FLAGS)  \
+		$(GO_COVER_FLAGS) -coverprofile=./cover-unit-full.out \
+		$(UNIT_TEST_PACKAGES)
+	cat ./cover-unit-full.out | $(COVERAGE_FILTERS) > ./cover-unit.out
+	rm -f ./cover-unit-full.out
+
 
 .PHONY: integration
 integration:
-	@echo "Running integration tests with $(TEST_KUBECONFIG)"
+	@echo "Running integration tests"
 	@kubectl config get-contexts
-	TEST_KUBECONFIG=$(TEST_KUBECONFIG) go test -timeout $(TEST_TIMEOUT) $(GO_FLAGS) \
+	go test -timeout $(TEST_TIMEOUT) $(GO_FLAGS) \
 		-parallel $(TEST_PARALLELISM) \
-		$(EXTRA_GO_TEST_FLAGS)  \
-		$(GO_COVER_FLAGS) -coverprofile=./cover-int.out \
+		$(TEST_FLAGS)  \
+		$(GO_COVER_FLAGS) -coverprofile=./cover-int-full.out \
 		./integration/...
-
+	cat ./cover-int-full.out | $(COVERAGE_FILTERS) > ./cover-int.out
+	rm -f ./cover-int-full.out
 	go tool cover -html=./cover-int.out -o ./cover-int.html
 
 .PHONY: coverage
 coverage: cover.html
 
 cover.html: cover-int.out cover-unit.out
-	cp cover-int.out cover.out
-	tail +2 cover-unit.out >> cover.out
+	cat cover-int.out > cover.out
+	tail +2 cover-unit.out | $(COVERAGE_FILTERS) >> cover.out
 	go tool cover -html=./cover.out -o ./cover.html
 	go tool cover -func cover.out | grep total:
 	open ./cover.html

builder/Dockerfile

@@ -0,0 +1,17 @@
+# TODO - refactor this as a cross-compilation process
+
+ARG BUILDER_BASE=golang:1.17-bullseye
+ARG RUNTIME_BASE=debian:bullseye-slim
+
+FROM ${BUILDER_BASE} as build
+COPY . /go/src/github.com/vmware-tanzu/buildkit-cli-for-kubectl
+WORKDIR /go/src/github.com/vmware-tanzu/buildkit-cli-for-kubectl
+ARG BUILDKIT_PROXY_IMAGE
+ARG VERSION
+RUN go install \
+    -ldflags "-extldflags -static -X github.com/vmware-tanzu/buildkit-cli-for-kubectl/version.Version=${VERSION} -X github.com/vmware-tanzu/buildkit-cli-for-kubectl/version.DefaultHelperImage=${BUILDKIT_PROXY_IMAGE}" \
+    -tags "osusergo netgo static_build" ./cmd/buildkit-proxy
+
+FROM ${RUNTIME_BASE} as final
+COPY --from=build /go/bin/buildkit-proxy /bin/
+ENTRYPOINT ["/bin/buildkit-proxy"]

cmd/buildkit-proxy/main.go

@@ -0,0 +1,66 @@
+package main
+
+import (
+	"context"
+	"fmt"
+	"os"
+
+	"github.com/sirupsen/logrus"
+	"github.com/spf13/cobra"
+	"github.com/spf13/pflag"
+	"github.com/vmware-tanzu/buildkit-cli-for-kubectl/pkg/proxy"
+	"github.com/vmware-tanzu/buildkit-cli-for-kubectl/version"
+)
+
+func main() {
+	if err := doMain(context.Background()); err != nil {
+		os.Exit(1)
+	}
+}
+
+// TODO - explore wiring up a unit test wrapper so we can gather proxy code coverage in integration tests
+
+func doMain(ctx context.Context) error {
+	pflag.CommandLine = pflag.NewFlagSet("buildkit-proxy", pflag.ExitOnError)
+	cfg := &proxy.ServerConfig{}
+	var debug bool
+
+	root := &cobra.Command{
+		Use:   "buildkit-proxy CMD [OPTIONS]",
+		Short: "Run the BuildKit proxy gRPC service",
+		//SilenceUsage: true,
+	}
+
+	cmd := &cobra.Command{
+		Use:   "serve",
+		Short: "run the gRPC server",
+		RunE: func(cmd *cobra.Command, args []string) error {
+			if debug {
+				logrus.SetLevel(logrus.DebugLevel)
+			}
+			srv, err := proxy.NewProxy(ctx, *cfg)
+			if err != nil {
+				return err
+			}
+			return srv.Serve(ctx)
+		},
+		SilenceUsage: true,
+	}
+	flags := cmd.Flags()
+	flags.StringVar(&cfg.BuildkitdSocketPath, "buildkitd", "/run/buildkit/buildkitd.sock", "Specify the buildkitd socket path")
+	flags.StringVar(&cfg.ContainerdSocketPath, "containerd", "", "Connect to local containerd with the specified socket path")
+	flags.StringVar(&cfg.DockerdSocketPath, "dockerd", "", "Connect to local dockerd with the specified socket path")
+	flags.StringVar(&cfg.HelperSocketPath, "listen", "/run/buildkit/buildkit-proxy.sock", "Socket path for this proxy to listen on")
+	flags.BoolVar(&debug, "debug", false, "enable debug level logging")
+
+	root.AddCommand(cmd,
+		&cobra.Command{
+			Use:   "version",
+			Short: "Show version information ",
+			RunE: func(cmd *cobra.Command, args []string) error {
+				fmt.Printf("%s\n", version.GetProxyImage())
+				return nil
+			},
+		})
+	return root.Execute()
+}

cmd/buildkit-proxy/main_test.go

@@ -0,0 +1,14 @@
+package main
+
+import (
+	"context"
+	"testing"
+
+	"github.com/stretchr/testify/require"
+)
+
+func TestDoMain(t *testing.T) {
+	ctx := context.Background()
+	err := doMain(ctx)
+	require.NoError(t, err)
+}

codecov.yml

@@ -0,0 +1,4 @@
+ignore:
+- "vendor/"
+- "integration/"
+- "**/*.pb.go"

docs/architecture.md

@@ -8,13 +8,13 @@ When you run `kubectl build` or `kubectl buildkit` the kubectl CLI detects this
 
 The CLI plugin then operates just like kubectl does.  It loads up your kubeconfig, uses the currently active context, and interacts with the kubernetes API on your cluster.
 
-In order to build a container image, `kubectl build` needs one or more BuildKit builders. These builders can be created explicitly with the `kubectl buildkit create` command, or one will be created automatically the first time you run `kubectl build`. The builders are modeled as a kubernetes Deployment, but we'll probably add DaemonSet support too.
+In order to build a container image, `kubectl build` needs one or more BuildKit builders. These builders can be created explicitly with the `kubectl buildkit create` command, or one will be created automatically the first time you run `kubectl build`. The builders are modeled as a kubernetes Deployment, but we'll probably add DaemonSet support too.  The Deployment uses anti-affinity to distribute pods across the nodes in the cluster.
 
 The pod spec by default tries to mount the socket for the container runtime on the host so it can communicate directly to the runtime. This works for both `containerd` and `dockerd` runtimes. This allows the builder to build and load the images you build directly into the container runtime so kubernetes can run other pods with those images. You can opt-out of this model if you prefer a de-privileged approach, but then it can't load the images directly. For those cases you will have to push any built images to a registry and then rely on the kubelet to pull them.
 
-When you run a command like `build` the CLI finds one running/healthy pod, and does the equivalent of a `kubectl exec` to connect to the pod, then runs a small ephemeral proxy inside the container to be able to route gRPC API calls over the stdin/stdout pipe from the exec to the `buildkitd` running inside the pod.  That proxy is implemented inside the `buildkitd` CLI itself.
+When you run a command like `build` the CLI finds one running/healthy pod, and does the equivalent of a `kubectl exec` to connect to the pod.  Within the pod we use 2 containers.  One container has the BuildKit daemon, and the other contains a proxy component used by this CLI to communicate with `buildkitd`. The CLI uses this proxy to be able to route gRPC API calls over the stdin/stdout pipe from the exec to the `buildkitd`.  This proxy helps offload the work of loading the final image from `buildkitd` into the local runtime, unless you've opted out or have requested to push directly to a registry. 
 
-At present, `buildkitd` can't talk to each of the other builders directly. To make your freshly built image available in a multi-node cluster, the CLI detects multiple builder pods are running, and performs an *export* of the image from the pod that built the image. It then loads the image to the container runtimes on the other nodes using the same `kubectl exec` approach described above. It does this by talking directly to the container runtime socket inside the pod to perform loading the image.  This is typically transparent and seamless to the user, but can be noticable when building large images when using a limited bandwidth link between the CLI and the kubernetes cluster.  If you specify `--push` during the build, it skips this export/load step during the build.
+To make your freshly built image available in a multi-node cluster, you must scale the Deployment definition for the builder up, and the CLI detects multiple builder pods are running.  When detected, the proxy container inside the pods facilitate transferring the image between the nodes across the internal kubernetes cluster network.  This avoids transferring the image back-and-forth between the builder and CLI, which may be running remotely.  If you specify `--push` during the build, it skips this transfer step during the build.
 
 
 ## Code/Repo Layout
@@ -28,4 +28,5 @@ The following lists the major components that make up the repo
 * [pkg/driver/kubernetes/execconn/](../pkg/driver/kubernetes/execconn/) The exec tunnel to send gRPC commands to the builder pod
 * [pkg/driver/kubernetes/manifest/](../pkg/driver/kubernetes/manifest/) Handles creation of the kubernetes resource definitions for the builder
 * [pkg/driver/kubernetes/podchooser/](../pkg/driver/kubernetes/podchooser/) Selects which pod to send build requests to
+* [pkg/proxy/](../pkg/proxy/) Runs within the builder pod to offload CLI operations related to image loading and transfer between nodes
 * [integration/](../integration/) Integration tests which exercise the CLI via go test running on a live kubernetes cluster