Skip to content
Published on

Managing CI/CD Pipelines as Code with Dagger: Same Pipeline from Local to CI

🇰🇷KO🇺🇸EN🇯🇵JA
Authors
  • Name
    Twitter

What is Dagger?

Dagger is an open-source tool that lets you write CI/CD pipelines in programming languages such as Go, Python, and TypeScript. Its core value proposition is "Write once, run anywhere" — a pipeline you run on your local laptop works identically on GitHub Actions, GitLab CI, or Jenkins.

Problems with Traditional CI/CD

  • YAML hell: Complex pipelines expressed in YAML suffer from poor readability and reusability
  • No local testing: Pipelines can only run in CI, resulting in long feedback loops
  • Vendor lock-in: GitHub Actions and GitLab CI have completely different syntax
  • Difficult debugging: Reproducing the CI environment locally is hard

How Dagger Solves This

Local development ──→ Dagger Engine ──→ Same result
GitHub Actions ──→ Dagger Engine ──→ Same result
GitLab CI ──→ Dagger Engine ──→ Same result

The Dagger Engine runs all tasks in containers, eliminating environment differences.

Installation and Project Initialization

Installing the Dagger CLI

# macOS
brew install dagger/tap/dagger

# Linux
curl -fsSL https://dl.dagger.io/dagger/install.sh | sh

# Verify version
dagger version

Project Initialization

# Using the Go SDK
mkdir my-pipeline && cd my-pipeline
dagger init --sdk=go --name=my-pipeline

# Generated file structure
# ├── dagger.json
# ├── main.go
# └── go.mod

First Pipeline: Building a Go Project

Writing main.go

package main

import (
    "context"
    "dagger/my-pipeline/internal/dagger"
)

type MyPipeline struct{}

// Build function: Builds the Go project
func (m *MyPipeline) Build(ctx context.Context, source *dagger.Directory) *dagger.Container {
    // Configure Go build environment
    builder := dag.Container().
        From("golang:1.22-alpine").
        WithDirectory("/src", source).
        WithWorkdir("/src").
        WithEnvVariable("CGO_ENABLED", "0").
        WithExec([]string{"go", "mod", "download"}).
        WithExec([]string{"go", "build", "-o", "/app", "./cmd/server"})

    // Lightweight runtime image
    return dag.Container().
        From("alpine:3.19").
        WithFile("/app", builder.File("/app")).
        WithEntrypoint([]string{"/app"})
}

// Test function: Runs tests
func (m *MyPipeline) Test(ctx context.Context, source *dagger.Directory) (string, error) {
    return dag.Container().
        From("golang:1.22-alpine").
        WithDirectory("/src", source).
        WithWorkdir("/src").
        WithExec([]string{"go", "mod", "download"}).
        WithExec([]string{"go", "test", "-v", "-race", "./..."}).
        Stdout(ctx)
}

// Lint function: Runs golangci-lint
func (m *MyPipeline) Lint(ctx context.Context, source *dagger.Directory) (string, error) {
    return dag.Container().
        From("golangci/golangci-lint:v1.57").
        WithDirectory("/src", source).
        WithWorkdir("/src").
        WithExec([]string{"golangci-lint", "run", "--timeout", "5m"}).
        Stdout(ctx)
}

Running Locally

# Build
dagger call build --source=.

# Test
dagger call test --source=.

# Lint
dagger call lint --source=.

# Export build result as a Docker image
dagger call build --source=. export --path=./image.tar

Writing a Pipeline with the Python SDK

# dagger/src/main/__init__.py
import dagger
from dagger import dag, function, object_type

@object_type
class MyPipeline:
    @function
    async def test(self, source: dagger.Directory) -> str:
        """Test a Python project"""
        return await (
            dag.container()
            .from_("python:3.12-slim")
            .with_directory("/src", source)
            .with_workdir("/src")
            .with_exec(["pip", "install", "-r", "requirements.txt"])
            .with_exec(["pip", "install", "pytest", "pytest-cov"])
            .with_exec(["pytest", "-v", "--cov=app", "--cov-report=term-missing"])
            .stdout()
        )

    @function
    async def build(self, source: dagger.Directory) -> dagger.Container:
        """Build a Docker image"""
        # Dependency caching
        pip_cache = dag.cache_volume("pip-cache")

        return (
            dag.container()
            .from_("python:3.12-slim")
            .with_mounted_cache("/root/.cache/pip", pip_cache)
            .with_directory("/app", source)
            .with_workdir("/app")
            .with_exec(["pip", "install", "--no-cache-dir", "-r", "requirements.txt"])
            .with_entrypoint(["python", "main.py"])
        )

    @function
    async def publish(
        self,
        source: dagger.Directory,
        registry: str = "ghcr.io",
        image_name: str = "my-app",
        tag: str = "latest",
        registry_user: dagger.Secret | None = None,
        registry_pass: dagger.Secret | None = None,
    ) -> str:
        """Build image and push to registry"""
        container = await self.build(source)

        if registry_user and registry_pass:
            container = container.with_registry_auth(
                registry,
                await registry_user.plaintext(),
                registry_pass,
            )

        ref = f"{registry}/{image_name}:{tag}"
        digest = await container.publish(ref)
        return digest

Caching Strategy

Dagger's caching dramatically improves build speed:

func (m *MyPipeline) BuildWithCache(ctx context.Context, source *dagger.Directory) *dagger.Container {
    // Go module cache
    goModCache := dag.CacheVolume("go-mod-cache")
    goBuildCache := dag.CacheVolume("go-build-cache")

    return dag.Container().
        From("golang:1.22-alpine").
        WithMountedCache("/go/pkg/mod", goModCache).
        WithMountedCache("/root/.cache/go-build", goBuildCache).
        WithDirectory("/src", source).
        WithWorkdir("/src").
        WithExec([]string{"go", "build", "-o", "/app", "./cmd/server"})
}

Caching Performance Comparison

First build:        2 min 30 sec (including dependency download)
Second build:       15 sec (cache hit)
Source-only change: 20 sec (dependency cache reused)

Secret Management

func (m *MyPipeline) Deploy(
    ctx context.Context,
    source *dagger.Directory,
    kubeconfig *dagger.Secret,
    registryToken *dagger.Secret,
) (string, error) {
    // Secrets are never exposed in logs
    return dag.Container().
        From("bitnami/kubectl:latest").
        WithMountedSecret("/root/.kube/config", kubeconfig).
        WithSecretVariable("REGISTRY_TOKEN", registryToken).
        WithDirectory("/manifests", source.Directory("k8s")).
        WithExec([]string{"kubectl", "apply", "-f", "/manifests/"}).
        Stdout(ctx)
}

# Passing secrets during local execution
dagger call deploy \
  --source=. \
  --kubeconfig=file:$HOME/.kube/config \
  --registry-token=env:REGISTRY_TOKEN

CI Integration

GitHub Actions

# .github/workflows/ci.yml
name: CI
on:
  push:
    branches: [main]
  pull_request:

jobs:
  ci:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dagger/dagger-for-github@v6
        with:
          version: 'latest'
          verb: call
          args: test --source=.
      - uses: dagger/dagger-for-github@v6
        with:
          version: 'latest'
          verb: call
          args: lint --source=.
      - uses: dagger/dagger-for-github@v6
        with:
          version: 'latest'
          verb: call
          args: build --source=.

GitLab CI

# .gitlab-ci.yml
stages:
  - test
  - build

test:
  stage: test
  image: registry.dagger.io/engine:latest
  services:
    - docker:dind
  script:
    - dagger call test --source=.

build:
  stage: build
  image: registry.dagger.io/engine:latest
  services:
    - docker:dind
  script:
    - dagger call build --source=. export --path=image.tar
  artifacts:
    paths:
      - image.tar

Multi-Platform Build

func (m *MyPipeline) BuildMultiPlatform(
    ctx context.Context,
    source *dagger.Directory,
) (string, error) {
    platforms := []dagger.Platform{
        "linux/amd64",
        "linux/arm64",
    }

    platformVariants := make([]*dagger.Container, len(platforms))
    for i, platform := range platforms {
        platformVariants[i] = dag.Container(dagger.ContainerOpts{Platform: platform}).
            From("golang:1.22-alpine").
            WithDirectory("/src", source).
            WithWorkdir("/src").
            WithEnvVariable("CGO_ENABLED", "0").
            WithExec([]string{"go", "build", "-o", "/app", "./cmd/server"})
    }

    // Push multi-platform image
    digest, err := dag.Container().
        Publish(ctx, "ghcr.io/my-org/my-app:latest",
            dagger.ContainerPublishOpts{
                PlatformVariants: platformVariants,
            })

    return digest, err
}

Integrated Pipeline: Full CI/CD

func (m *MyPipeline) CI(ctx context.Context, source *dagger.Directory) error {
    // 1. Lint
    _, err := m.Lint(ctx, source)
    if err != nil {
        return fmt.Errorf("lint failed: %w", err)
    }
    fmt.Println("✅ Lint passed")

    // 2. Test
    _, err = m.Test(ctx, source)
    if err != nil {
        return fmt.Errorf("test failed: %w", err)
    }
    fmt.Println("✅ Tests passed")

    // 3. Build
    container := m.Build(ctx, source)
    _, err = container.Sync(ctx)
    if err != nil {
        return fmt.Errorf("build failed: %w", err)
    }
    fmt.Println("✅ Build succeeded")

    return nil
}

Dagger vs Traditional CI Tools Comparison

FeatureGitHub ActionsGitLab CIDagger
Pipeline definitionYAMLYAMLGo/Python/TS
Local executionact (limited)Not possibleFully supported
DebuggingDifficultDifficultIDE debugger available
Vendor lock-inGitHubGitLabNone
CachingManual setupManual setupAutomatic (content-based)
ReusabilityMarketplaceTemplatesPackages/Modules

Troubleshooting

Common Issues and Solutions

# Check Dagger Engine status
docker ps | grep dagger-engine

# Reset cache
dagger query <<< '{ defaultPlatform }'

# Verbose log output
dagger call --debug test --source=.

# Shell access at a specific step (debugging)
dagger call build --source=. terminal
Review Quiz (5 Questions)

Q1. What is Dagger's biggest advantage over traditional CI/CD tools?

You can run the same pipeline in both local and CI environments. Since pipelines are written in programming languages, you get IDE support, type safety, and debugging capabilities.

Q2. How do you implement caching in Dagger?

Create cache volumes with dag.CacheVolume() and mount them to containers with WithMountedCache(). Caching is automatic and content-based.

Q3. How do you securely pass secrets in Dagger?

Accept dagger.Secret types as function parameters and pass them to containers using WithMountedSecret or WithSecretVariable. They are never exposed in logs.

Q4. What action do you use to run Dagger in GitHub Actions?

The dagger/dagger-for-github@v6 action.

Q5. What options do you use for multi-platform builds in Dagger?

The Platform field in dagger.ContainerOpts and the PlatformVariants option in Publish.

Discuss on Twitter • View on GitHub