From 226c2712fe067d2f26ee7e48a354d5d3bb0f17ec Mon Sep 17 00:00:00 2001 From: SurbhiAgarwal Date: Sun, 18 Oct 2026 14:19:44 +0530 Subject: [PATCH] docs: Add a 10,000 m view of kpt and Porch Add a comprehensive architectural guide providing a high-level (10,000 m) overview of kpt and Porch within the Configuration as Data (CaD) paradigm. - Add guides/10000-m-view.md covering the core CaD principles, client-side kpt CLI capabilities, server-side Porch control plane architecture, Mermaid interaction and lifecycle state diagrams, comparison table, and end-to-end multi-cluster workflows. - Update guides/_index.md to link to the new guide. - Update book/02-concepts/_index.md to introduce Porch under toolchain components and reference the 10,000 m view guide. Closes #4806 Part of #4711 Signed-off-by: SurbhiAgarwal --- .../content/en/book/02-concepts/_index.md | 8 + .../content/en/guides/10000-m-view.md | 202 ++++++++++++++++++ documentation/content/en/guides/_index.md | 1 + 3 files changed, 211 insertions(+) create mode 100644 documentation/content/en/guides/10000-m-view.md diff --git a/documentation/content/en/book/02-concepts/_index.md b/documentation/content/en/book/02-concepts/_index.md index cc2f371ccb..014fb9c858 100644 --- a/documentation/content/en/book/02-concepts/_index.md +++ b/documentation/content/en/book/02-concepts/_index.md @@ -67,6 +67,14 @@ The kpt toolchain includes the following components: configuration easy to create and transform, via reusable functions. Because they are expected to be used for in-place transformation, the functions need to be idempotent. +- [**Porch (Package Orchestration)**](https://docs.porch.nephio.org/): A Kubernetes-native control plane ("kpt-as-a-service") that exposes package operations as declarative Kubernetes resources (`PackageRevision`). Porch automates package cloning, server-side function evaluation, and multi-stage lifecycle governance (`Draft` ➔ `Proposed` ➔ `Published`) across enterprise fleets. Separate GitOps reconcilers such as Config Sync, Flux, or Argo CD synchronize published configuration to workload clusters. + +### The 10,000 m View: kpt and Porch + +While the **kpt CLI** provides an agile client-side developer experience on local filesystems and Git repositories, **Porch** provides a centralized, server-side package orchestration engine running inside Kubernetes. Together, they form an end-to-end Configuration as Data platform. + +For a detailed architectural comparison, end-to-end workflows, and guidance on when to use each, see the [10,000 m View of kpt and Porch]({{% relref "/guides/10000-m-view" %}}) guide. + ## Packages kpt manages KRM resources in bundles called **packages**. diff --git a/documentation/content/en/guides/10000-m-view.md b/documentation/content/en/guides/10000-m-view.md new file mode 100644 index 0000000000..faf1ee0068 --- /dev/null +++ b/documentation/content/en/guides/10000-m-view.md @@ -0,0 +1,202 @@ +--- +title: "A 10,000 m View of kpt and Porch" +linkTitle: "10,000 m View: kpt & Porch" +description: | + High-level architectural overview of kpt, Porch, Configuration as Data, + and how they orchestrate Kubernetes configuration at scale. +toc_hide: false +menu: + main: + parent: "Guides" + weight: 5 +--- + +Managing Kubernetes configuration across multiple environments, dozens of clusters, and distributed teams is one of the most demanding challenges in cloud-native platforms. Traditional approaches—such as imperative scripts, monolithic Helm charts with hundreds of parameters, or complex domain-specific languages (DSLs)—frequently lead to configuration drift, excessive templating complexity, and brittle delivery pipelines. + +The **kpt** project addresses this challenge through the **Configuration as Data (CaD)** paradigm, delivering a comprehensive ecosystem built on two complementary pillars: + +1. **kpt**: A client-side, package-centric CLI and toolchain for authoring, transforming, and deploying configuration. +2. **Porch (Package Orchestration)**: A Kubernetes-native server-side control plane that turns package management into a declarative, multi-repository service. + +This guide provides a high-level, 10,000-meter architectural view of how kpt and Porch fit together, their shared foundation, and how they empower teams to manage configuration at enterprise scale. + +--- + +## The Foundation: Configuration as Data (CaD) + +At the heart of both kpt and Porch is the principle of **Configuration as Data**: + +* **Declarative and Uniform**: Configuration is represented purely as data conforming to the [Kubernetes Resource Model (KRM)](https://github.com/kubernetes/design-proposals-archive/blob/main/architecture/resource-management.md), stored directly as serializable YAML/JSON. +* **Separation of Code and Data**: Instead of embedding programming logic or template directives inside configuration files (such as `{{ if .Values.enabled }}`), configuration files remain clean KRM data. +* **In-Place Transformation via KRM Functions**: Specialized, containerized programs (KRM functions) read KRM resources from an input stream, mutate or validate them, and output the updated resources in-place. +* **Versioned Source of Truth**: Configuration is stored separately from live state in versioned repositories. Git is a common choice, while Porch also supports OCI repositories. + +By keeping configuration as plain data, machines, controllers, and humans can read, modify, and validate the same configuration without losing comments, structure, or schema fidelity. + +--- + +## The Two Pillars + +While kpt and Porch share the same package format and philosophy, they operate at different layers of the infrastructure stack: + +```mermaid +flowchart TD + subgraph ClientSide ["Client-Side Tooling: kpt CLI"] + direction TB + Dev["Developer / CI Pipeline"] -->|kpt pkg get/update| LocalPkg["Local Package on Filesystem"] + LocalPkg -->|kpt fn render| FnRun["KRM Functions Engine"] + FnRun -->|In-place mutation/validation| LocalPkg + LocalPkg -->|kpt live apply| ClusterDirect["Direct Kubernetes Cluster"] + end + + subgraph ServerSide ["Server-Side Control Plane: Porch"] + direction TB + UserUI["Platform Engineer / Web UI / Orchestrator"] -->|Kubernetes API / kubectl| PorchSvc["Porch Extension API Server"] + PorchSvc -->|Manage PackageRevisions| Repos["Upstream & Downstream Git Repos"] + PorchSvc -->|Server-side fn evaluation| PorchRunner["Function Evaluator"] + Repos -->|GitOps Sync: Config Sync / Flux / Argo CD| ClusterFleet["Cluster Fleet"] + end + + LocalPkg -.->|Publish Blueprint| Repos +``` + +### 1. kpt (The Client-Side Engine) + +**kpt** is a client-side command-line tool optimized for developers, platform engineers, and CI/CD pipelines working locally or within workflow runners. + +* **Package Management (`kpt pkg`)**: Fetches, inspects, and versions packages from remote Git repositories. It uses **3-way merge** (`kpt pkg update`) to intelligently integrate upstream updates while preserving local modifications. +* **Function Pipelines (`kpt fn`)**: Executes pipelines of containerized or binary KRM functions declared in the `Kptfile` to mutate (e.g., set namespace, add labels, inject sidecars) or validate (e.g., check resource quotas, verify schema constraints) configuration files in-place. +* **Actuation and Inventory (`kpt live`)**: Directly applies rendered packages to a target Kubernetes cluster with resource pruning, dependency ordering, and real-time status monitoring via the ResourceGroup inventory model. + +### 2. Porch (The Server-Side Control Plane) + +**Porch** (Package Orchestration) lifts the capabilities of kpt into a Kubernetes-native, multi-tenant service—often described as **"kpt-as-a-service"**. + +Instead of requiring individual engineers to manually clone repositories and run CLI commands, Porch runs as an extension API server inside a management Kubernetes cluster, exposing packages as first-class Kubernetes custom resources: + +* **`Repository`**: Represents an external Git or OCI repository registered with Porch (for catalogs, blueprints, or deployment targets). +* **`PackageRevision`**: Represents a specific version or branch of a configuration package stored in a repository. +* **`PackageRevisionResources`**: Exposes the actual KRM resource YAML files within a package revision directly through the Kubernetes API. + +#### Porch Package Lifecycle & Governance + +Porch introduces a formal lifecycle state machine for configuration packages: + +```mermaid +stateDiagram-v2 + [*] --> Draft: Create / Clone Package + Draft --> Draft: Edit resources, run KRM functions + Draft --> Proposed: Propose for review + Proposed --> Draft: Reject / Request changes + Proposed --> Published: Approve & Publish + Published --> DeletionProposed: Propose deletion + DeletionProposed --> [*]: Confirm deletion +``` + +* **Draft**: The package is in an editable state. Resources can be added, removed, or mutated via KRM functions. Changes are tracked as commits on a workspace branch in Git. +* **Proposed**: The package is frozen and submitted for review/approval by platform administrators or automated policy gates. +* **Published**: The package revision is finalized, tagged in Git (e.g., `package-name/v1`), and made available for deployment or downstream consumption. + +--- + +## Detailed Comparison: kpt vs Porch + +| Dimension | kpt CLI | Porch (Package Orchestration) | +| :--- | :--- | :--- | +| **Primary Role** | Client-side authoring, rendering, and deployment tool | Centralized server-side package lifecycle control plane | +| **Execution Environment** | Local developer workstation, CI/CD runner | Management Kubernetes cluster (as an API service) | +| **Interface** | CLI commands (`kpt pkg`, `kpt fn`, `kpt live`) | Kubernetes API (`kubectl`, CRDs, controllers, Web UIs) | +| **Storage Model** | Local filesystem, working with Git repos | Direct integration with registered Git / OCI repositories | +| **Function Execution** | Local Docker / Podman or binary function runner | Server-side containerized function runners in Kubernetes | +| **Lifecycle Workflow** | Ad-hoc (developer commits and pushes to Git) | Formal approval workflow (`Draft` ➔ `Proposed` ➔ `Published`) | +| **Deployment Model** | Client-driven direct apply (`kpt live`) or Git push | GitOps integration (writes to Git; GitOps agents sync to clusters) | +| **Primary Personas** | Package authors, developers, automation scripts | Platform teams, enterprise SREs, multi-cluster orchestrators | + +--- + +## End-to-End Workflow: How kpt and Porch Work Together + +In a mature enterprise or multi-cluster architecture, kpt and Porch are not competing alternatives—they work together seamlessly across the package lifecycle: + +```mermaid +sequenceDiagram + autonumber + actor Author as Package Author + participant kpt as kpt CLI + participant Catalog as Blueprint Catalog (Git) + actor Platform as Platform Engineer / Orchestrator + participant Porch as Porch Control Plane + participant DeployRepo as Deployment Repo (Git) + participant GitOps as GitOps Engine (Config Sync) + participant Cluster as Workload Cluster + + Note over Author,Catalog: Phase 1: Blueprint Authoring + Author->>kpt: kpt pkg init & author KRM resources + Author->>kpt: kpt fn render (validate locally) + Author->>Catalog: Push vetted package blueprint (e.g. database/v1.0.0) + + Note over Platform,DeployRepo: Phase 2: Fleet Specialization & Orchestration + Platform->>Porch: Clone blueprint to target cluster repo (Draft) + Porch->>Porch: Evaluate KRM functions (set cluster ID, IP, domain) + Platform->>Porch: Propose & Approve (Draft -> Proposed -> Published) + Porch->>DeployRepo: Commit rendered package to deployment branch + + Note over DeployRepo,Cluster: Phase 3: Actuation & Deployment + DeployRepo->>GitOps: Pull published package + GitOps->>Cluster: Apply resources to cluster + Cluster-->>Platform: Status & health reported back +``` + +### 1. Blueprint Authoring (kpt CLI) +A platform or application expert uses `kpt` on their workstation to create a reusable package blueprint (e.g., an enterprise PostgreSQL setup with monitoring and backup policies). They define the package pipeline in `Kptfile`, test it using `kpt fn render`, and commit it to an upstream catalog repository. + +### 2. Fleet Orchestration (Porch) +When a development team needs a database instance in their development, staging, or production cluster: +1. Porch clones the blueprint from the catalog repository into the team's deployment repository as a `Draft` revision. +2. Porch runs the package's configured mutator functions to inject site-specific configuration (such as namespace, storage classes, network policies, or references to externally managed secrets). +3. The package transitions from `Draft` to `Proposed`. Automated policy engines (like Gatekeeper or Kyverno) or team leads review and approve the revision. +4. Once marked `Published`, Porch writes the finalized manifests to the target branch in the deployment repository. + +### 3. Continuous Deployment (GitOps) +A GitOps agent running on the target cluster (such as **Config Sync**, **Argo CD**, or **Flux**) continuously syncs the published package from the deployment repository and actuates it onto the cluster. + +### 4. Lifecycle Upgrades (3-Way Merge in Both) +When the upstream blueprint is updated (e.g., from `v1.0.0` to `v1.1.0` with a security patch): +* Both kpt and Porch leverage the **3-way merge algorithm** to merge upstream changes into downstream copies without overwriting site-specific customizations. +* Porch can execute this upgrade across hundreds of downstream package instances simultaneously, opening new draft revisions for review. + +--- + +## When to Use What + +{{% alert title="Choosing Between kpt CLI and Porch" color="primary" %}} +* **Use kpt CLI when:** + * You are authoring or customizing individual packages on your local machine. + * You are developing and testing new KRM functions. + * You are implementing a lightweight CI/CD pipeline that validates or renders packages before committing. + * You want direct client-side deployment to a cluster using `kpt live` without running a server-side control plane. + +* **Use Porch when:** + * You are managing configuration packages across dozens or hundreds of clusters. + * You need a declarative Kubernetes API to create, clone, update, and delete packages programmatically. + * You need a centralized package catalog and structured approval workflows (`Draft` / `Proposed` / `Published`). + * You are building higher-level orchestration systems, self-service developer portals, or automated controllers (such as [Project Nephio](https://nephio.org)). +{{% /alert %}} + +--- + +## The Broader Ecosystem + +* **[KRM Functions Catalog](https://catalog.kpt.dev)**: Reusable, off-the-shelf functions maintained by the kpt community for common mutations and validations. +* **[KRM Function SDKs](https://github.com/kptdev/krm-functions-sdk)**: Libraries in Go and TypeScript to write custom functions with minimal boilerplate. +* **[Nephio](https://nephio.org)**: A Linux Foundation project that uses kpt and Porch as its core engine for orchestrating cloud infrastructure and complex 5G network functions across massive edge deployments. +* **GitOps Synchronizers**: Porch produces standard GitOps-ready repositories compatible with Google Cloud Config Sync, Flux, and Argo CD. + +--- + +## Where to Go Next + +* Dive deeper into the core concepts: [The kpt Book: Concepts]({{% relref "/book/02-concepts" %}}) +* Learn about package management: [Working with Packages]({{% relref "/book/03-packages" %}}) +* Understand how updates work: [Understanding 3-Way Merge in kpt]({{% relref "/guides/3-way-merge" %}}) +* Explore the Porch project: [Porch Documentation](https://docs.porch.nephio.org/) diff --git a/documentation/content/en/guides/_index.md b/documentation/content/en/guides/_index.md index 6f8560792d..c9e6dbcddf 100644 --- a/documentation/content/en/guides/_index.md +++ b/documentation/content/en/guides/_index.md @@ -9,6 +9,7 @@ menu: # Reference +- [A 10,000 m View of kpt and Porch]({{% relref "/guides/10000-m-view" %}}) - [The Rationale Behind kpt]({{% relref "/guides/rationale" %}}) - [Namespace Provisioning CLI]({{% relref "/guides/namespace-provisioning-cli" %}}) - [Variant Constructor Pattern]({{% relref "/guides/variant-constructor-pattern" %}})