English | 简体中文
A CLI tool for creating and deploying offline bundles for CSGHub in air-gapped environments. Run create on a machine with internet access to build the bundle, then deploy on the offline machine. This tool runs as a command-line utility and does not require any runtime daemon or background service.
- Create offline bundles containing K3s, Helm charts, and container images
- Deploy CSGHub from offline bundles without internet access
- Clean up K3s and all services when needed
- Multi-architecture support (amd64, arm64)
- Selective components with a unified
--skipflag (charts, images, runtime images, packages, tar, K3s) - Interactive runtime image selection (TUI) during
create - List runtime images with category and compute type filters
- No daemon - Pure CLI tool that runs on demand
- User Guide — command reference, bundle layout, and common workflows
- User Guide (简体中文)
- README (简体中文)
- Go 1.25.7 or higher (for building)
- Make (optional, for using Makefile)
- Root/sudo access (for deployment and cleanup operations)
make buildmake build-linuxOr build individually:
make build-linux-amd64
make build-linux-arm64make build-darwinOr build individually:
make build-darwin-amd64
make build-darwin-arm64make build-allmake cleanNotes: For default settings regarding charts, please refer to internal/manifest/charts
Make the binary executable:
sudo chmod +x ./csghub-airgap-linux-amd64| Command | Description |
|---|---|
create |
Create an offline bundle on an internet-connected machine |
deploy |
Deploy CSGHub from an offline bundle on an offline machine |
cleanup |
Remove K3s and all deployed services |
list-images |
List available runtime images before creating a bundle |
Create an offline bundle containing all necessary components:
sudo csghub-airgap create [options]| Flag | Alias | Description | Default |
|---|---|---|---|
--output |
-o |
Output directory for the bundle | ./csghub-airgap-<date> |
--arch |
Target architecture (amd64, arm64) | Current system | |
--debug |
-d |
Enable debug logging | false |
--proxy |
Use ghfast.top proxy prefix for GitHub downloads | false |
|
--force-pull |
Skip the local image cache (containerd cache + file cache) and always re-pull from the registry | false |
|
--no-pkg-cache |
Skip the cached NVIDIA package directory and always re-download apt/dnf/zypper packages | false |
|
--k3s-env |
Extra environment variables for K3s (KEY=VALUE) | [] |
|
--k3s-arg |
Extra CLI arguments for K3s (e.g., --k3s-arg=--default-runtime=nvidia) |
[] |
|
--skip |
Skip components: comma-separated chart names (csghub, longhorn, nvdp) and/or kinds (k3s, charts, image, runtime, package, tar) | ||
--quiet |
-q |
Skip interactive prompts; runtime image selection is skipped (no images selected) | false |
--dry-run |
Print the operations that would be performed (downloads, image exports, packing) without executing them | false |
|
--set |
-s |
Set Helm values (e.g., csghub.key=value, supports multiple) | [] |
- Architecture check - verify the current platform can run (creating on macOS is supported, see notes below).
- NVIDIA package download - only when the target arch matches the host, the nvdp chart is enabled, and
--skip packageis not set. - K3s download - k3s binary, airgap images, install script, helm binary (unless
--skip k3s). - Per-chart processing (each enabled chart):
- Pull the Helm chart (unless
--skip charts; if--skip chartsbut the chart file is missing locally, it is still pulled). - Export the chart's main images (unless
--skip image). - For the
csghubchart: show the runtime image selection UI, then export the selected runtime images (unless--skip runtimeor--quiet).
- Pull the Helm chart (unless
- Generate
metadata.yamlrecording the bundle configuration. - Pack - pack the whole output directory into
<dirname>.tar(unless--skip tar).
Runtime image selection timing: runtime images are selected and exported after the csghub main image export, not at program start.
After the csghub main image export, the program reads csghub-server's runtime image config (default from GitHub main branch) and shows a TUI for you to select:
Select runtime images (space=toggle, a=all, A=none, enter=confirm, empty=skip)
spacetoggle the current itemaselect allAselect noneenterconfirm (selecting nothing = skip, i.e. no runtime images bundled)q/Esccancel (also skips runtime images)
# Basic bundle creation (interactive runtime image selection)
sudo csghub-airgap create
# Specify output directory and use proxy
sudo csghub-airgap create \
--output /opt/csghub-bundle \
--proxy
# Create an arm64 bundle
sudo csghub-airgap create --arch arm64
# No runtime image selection (non-interactive)
sudo csghub-airgap create --quiet
# Skip runtime images entirely
sudo csghub-airgap create --skip runtime
# Export only runtime images (skip main chart images too)
sudo csghub-airgap create --skip k3s,charts,package,tar,image
# Image-only mode: skip k3s / charts / packages (shows the main image selection UI)
sudo csghub-airgap create --skip k3s,charts,package
# Keep the plain directory (no .tar), useful with deploy --skip tar
sudo csghub-airgap create --skip tar
# Skip specific charts
sudo csghub-airgap create --skip longhorn
sudo csghub-airgap create --skip longhorn,nvdp
# Force re-pull all images (ignore local cache)
sudo csghub-airgap create --force-pull
# Custom Helm values and K3s args
sudo csghub-airgap create \
--set csghub.global.gateway.external.domain=csghub.mycompany.com \
--set longhorn.persistence.defaultClassReplicaCount=3 \
--k3s-arg "--disable=servicelb"
# Extra K3s environment variables
sudo csghub-airgap create \
--k3s-env "K3S_NODE_NAME=server01" \
--k3s-env "K3S_TOKEN=mysecrettoken"After create finishes, besides the directory below, a csghub-airgap-<date>.tar archive of the whole directory is generated next to it for easy transfer. During deploy you can point --input at the archive (auto-unpacked) or at an already-unpacked directory (with --skip tar).
csghub-airgap-<date>/
├── metadata.yaml # bundle config (auto-applied at deploy)
├── k3s/
│ ├── bin/k3s
│ ├── images/k3s-airgap-images-<arch>.tar.zst
│ ├── scripts/k3s-install.sh
│ ├── scripts/k3s-patch.sh
│ └── helm/helm-v3.20.1-linux-<arch>.tar.gz
├── packages/ # NVIDIA container toolkit packages (Linux only)
├── csghub/
│ ├── charts/csghub-<version>.tgz
│ └── images/csghub-<version>.tar
├── csghub-runtime/ # only when runtime images were selected
│ └── images/csghub-runtime-<version>.tar
├── longhorn/
│ ├── charts/longhorn-<version>.tgz
│ └── images/longhorn-<version>.tar
└── nvidia-device-plugin/
├── charts/nvidia-device-plugin-<version>.tgz
└── images/nvidia-device-plugin-<version>.tar
- Cross-architecture: NVIDIA packages are not downloaded when the target arch differs from the host; fetch them manually on a Linux machine (e.g.
apt/dnf/zypper install nvidia-container-toolkit=...). - Creating on macOS: supported, but NVIDIA packages cannot be downloaded on non-Linux hosts; handle them manually.
--flannel-iface: detected from the build machine (e.g.en0on macOS); if not overridden with--k3s-argat deploy time, it is replaced with the target machine's default NIC.
Deploy CSGHub from an offline bundle (requires root):
sudo csghub-airgap deploy [options]| Flag | Alias | Description | Default |
|---|---|---|---|
--input |
-i |
Input bundle directory (or the .tar archive) |
./csghub-airgap-<date> |
--debug |
-d |
Enable debug logging | false |
--proxy |
Use ghfast.top proxy prefix for GitHub downloads | false |
|
--k3s-env |
Extra environment variables for K3s (KEY=VALUE) | [] |
|
--k3s-arg |
Extra CLI arguments for K3s | [] |
|
--data |
-D |
Specify data directory for K3s and Longhorn | (system default) |
--domain |
Specify CSGHub domain | csghub.example.com |
|
--quiet |
-q |
Skip interactive prompts; multipathd is auto-disabled (no prompt) | false |
--dry-run |
Print the operations that would be performed (incl. k3s install command, chart install) without executing them | false |
|
--skip |
Same as create: k3s, charts, image, runtime, package, tar and chart names |
||
--set |
-s |
Set Helm values (e.g., csghub.key=value, supports multiple) | [] |
- System check - must be Linux; the bundle architecture must match the target machine.
- NVIDIA GPU detection - auto-disable the nvdp chart (and package install) when the host has no NVIDIA GPU.
- NVIDIA package install - when nvdp is enabled and
--skip packageis not set. - Kernel parameters - write required sysctl parameters (skipped with
--skip k3s, i.e. an own cluster). - K3s install - install k3s (unless
--skip k3s). - multipathd preflight - when multipathd is running, prompt to stop and disable it (auto under
--quiet). Longhorn conflicts with multipath and may hang PVC mounting. - Per-chart deploy (each enabled chart):
- Import main images into containerd (unless
--skip image). - For the
csghubchart: import runtime images (unless--skip runtime; only when runtime images were selected at create time). - Install or upgrade the chart (unless
--skip charts).
- Import main images into containerd (unless
- K3s patch - apply the domain and other config (writes
/etc/hostsand patches the cluster CoreDNS). When k3s was not installed by this tool (--skip k3s, own cluster), prompts for confirmation first;--quietapplies it by default.
# Basic deployment
sudo csghub-airgap deploy --input /opt/csghub-bundle
# Specify domain and data directory
sudo csghub-airgap deploy \
--input /opt/csghub-bundle \
--domain csghub.mycompany.com \
--data /mnt/data
# Non-interactive deploy (auto-disables multipathd)
sudo csghub-airgap deploy --input /opt/csghub-bundle --quiet
# Skip K3s install (only import images and install charts)
sudo csghub-airgap deploy --input /opt/csghub-bundle --skip k3s
# Skip all image imports (images already loaded another way)
sudo csghub-airgap deploy --input /opt/csghub-bundle --skip image
# Skip runtime image import
sudo csghub-airgap deploy --input /opt/csghub-bundle --skip runtime
# Skip specific charts
sudo csghub-airgap deploy --input /opt/csghub-bundle --skip longhorn
sudo csghub-airgap deploy --input /opt/csghub-bundle --skip nvdp
# Override Helm values at deploy time
sudo csghub-airgap deploy \
--input /opt/csghub-bundle \
--set csghub.global.persistence.size=200Gi
# Extra K3s args and env
sudo csghub-airgap deploy \
--input /opt/csghub-bundle \
--k3s-arg "--default-runtime=nvidia" \
--k3s-env "K3S_NODE_NAME=server01"Remove K3s and all deployed services:
sudo csghub-airgap cleanup [options]| Flag | Alias | Description | Default |
|---|---|---|---|
--yes |
-y |
Skip confirmation and run cleanup directly | false |
# Interactive cleanup (confirmation prompt + 10s countdown, Ctrl+C to abort)
sudo csghub-airgap cleanup
# Force cleanup without confirmation
sudo csghub-airgap cleanup --yes
⚠️ Warning: cleanup removes the entire k3s cluster on this machine and ALL its data (not just CSGHub resources), including/var/lib/rancher/k3s(PVCs, Longhorn volumes),/var/lib/kubelet, the k3s binary and systemd units. A k3s you installed yourself on this host will be deleted too — make sure this is intended.Interactive mode requires typing
yesto confirm, then waits through a 10-second countdown (Ctrl+Cto cancel). This effectively runs/usr/local/bin/k3s-uninstall.sh. If it fails, run that script manually.
List available runtime images from csghub-server configs before creating a bundle (no root needed):
csghub-airgap list-images [options]| Flag | Alias | Description | Default |
|---|---|---|---|
--source |
Where to fetch configs: github, local |
github |
|
--version |
-v |
Git branch or tag for csghub-server (e.g., main, v1.2.0) |
main |
--local-path |
Local directory containing csghub-server configs (used with --source local) |
. |
|
--category |
-c |
Filter by category (can specify multiple): evaluation, finetune, inference, job, notebook, space |
(all) |
--compute |
Filter by compute type (comma-separated): cpu, gpu, dcu |
(all) | |
--proxy |
Use ghfast.top proxy prefix for GitHub downloads | false |
|
--format |
-f |
Output format: table, json |
table |
# List all runtime images (fetches from GitHub)
csghub-airgap list-images
# List only inference images
csghub-airgap list-images --category inference
# List images from a specific csghub-server version
csghub-airgap list-images --version v1.2.0
# List GPU images across all categories
csghub-airgap list-images --compute gpu
# List CPU images for finetune and inference
csghub-airgap list-images --category finetune --category inference --compute cpu
# List images from a local csghub-server directory (air-gapped)
csghub-airgap list-images \
--source local \
--local-path /path/to/csghub-server \
--category inference
# Output as JSON for scripting
csghub-airgap list-images --category notebook --format json--skip accepts comma-separated, repeatable values (case-insensitive). Two kinds of values:
Kinds (coarse-grained)
| Value | create behavior | deploy behavior |
|---|---|---|
k3s |
Skip K3s download | Skip K3s install |
charts |
Skip chart pull (reuse existing files), still export main images | Skip chart install/upgrade, still import images |
image |
Skip main chart image export | Skip main chart image import |
runtime |
Skip runtime image export | Skip runtime image import |
package |
Skip NVIDIA package download | Skip NVIDIA package install |
tar |
Skip packing: no .tar, keep the plain directory |
Skip unpacking: --input must point at an already-unpacked directory |
Chart names
| Value | Description |
|---|---|
csghub |
Skip the csghub chart (pull, its images, runtime images) |
longhorn |
Skip the longhorn chart |
nvdp |
Skip the NVIDIA device plugin chart (and package operations) |
--skip imageand--skip runtimeare independent:imagecontrols main chart images,runtimecontrols runtime images. Use both to skip all images. Chart names not present in the bundle are ignored with a warning.
Image-only mode: --skip k3s,charts,package (without --skip image) enters "export images only" mode and shows a main image selection UI per chart. Adding --quiet skips the UI and exports all main images directly.
# Export only images, skip the main image selection (export all main images)
sudo csghub-airgap create --skip k3s,charts,package --quiet# 1) On the internet-connected machine (produces /opt/csghub-bundle.tar)
sudo csghub-airgap create --output /opt/csghub-bundle
# 2) Copy the archive to the air-gapped machine (USB, SCP, etc.)
scp /opt/csghub-bundle.tar user@airgap-host:/root/
# 3) Deploy on the air-gapped machine (auto-unpacked)
sudo csghub-airgap deploy \
--input /opt/csghub-bundle.tar \
--domain csghub.internal.company.com# Bundle (includes nvdp and NVIDIA packages)
sudo csghub-airgap create --output /opt/csghub-gpu
# Deploy (auto-installs nvdp when a GPU is detected; auto-disables without one)
sudo csghub-airgap deploy \
--input /opt/csghub-gpu \
--domain csghub.gpu.company.com# On the bundle machine
sudo csghub-airgap create --output /opt/csghub-img --skip k3s,charts,package
# On the target machine (K3s already installed, import images only)
sudo csghub-airgap deploy --input /opt/csghub-img --skip k3s,chartssudo csghub-airgap cleanup --yes
sudo csghub-airgap deploy --input /opt/csghub-bundle --domain csghub.company.com# Check CSGHub pods are ready
kubectl get pods -n csghub
# List namespaces
kubectl get nsMost commands require root privileges. Always run with sudo:
sudo csghub-airgap [command]Ensure the bundle architecture matches your system; re-create the bundle with the matching --arch:
uname -m
sudo csghub-airgap create --arch $(uname -m)Enable debug logging for detailed output:
sudo csghub-airgap create --debug
sudo csghub-airgap deploy --debug --input /path/to/bundleIf cleanup fails, you can manually remove K3s:
# Run the K3s uninstall script
/usr/local/bin/k3s-uninstall.sh
# Or remove K3s completely
/usr/local/bin/k3s-killall.sh
/usr/local/bin/k3s-uninstall.sh| Error | Solution |
|---|---|
permission denied |
Run command with sudo |
architecture mismatch |
Create bundle with correct --arch flag |
bundle not found |
Verify --input path is correct |
k3s already running |
Run csghub-airgap cleanup first |
| stale image cache | Re-create with --force-pull to force a fresh pull |
| GitHub access restricted | Add --proxy; or use list-images --source local |
- This tool requires root privileges for deployment and cleanup operations
- Bundle files may contain sensitive configuration data - secure them appropriately
- Always verify bundle integrity before deployment in air-gapped environments
- The tool does not run any background daemons or services
To completely remove csghub-airgap:
# Remove the binary
sudo rm /usr/local/bin/csghub-airgap
# Remove any created bundles (optional)
sudo rm -rf /opt/csghub-* ./csghub-airgap*
# Clean up any deployments
sudo csghub-airgap cleanup --yes 2>/dev/null || true