CI/CD with Tekton
Tekton is an open source framework for creating CI/CD systems. With the help of Tekton, developers can continuously build, test and deploy their applications. Tekton can be used as a CI/CD system in Kubernetes environments. Tekton is also used in OpenShift as the OpenShift Pipeline Operator.
Tekton resources
Tekton provides various resources, such as Pipelines, Tasks, Triggers and EventListener. The combination of these resources and their configuration is handled via Custom Resource Definitions (CRD) for Kubernetes in the form of YAML files.
A minimal Tekton pipeline usually consists of a task resource and a pipeline resource in which this task is used. A simple task can be structured as follows, for example:
apiVersion: tekton.dev/v1
kind: Task
metadata:
name: say-hello
spec:
params:
- name: username
type: string
steps:
- name: echo
image: ubuntu
script: |
#!/bin/bash
echo "Hello $(params.username)!"
A simple pipeline that executes this task can look like this:
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
name: greeting
spec:
params:
- name: username
type: string
tasks:
- name: say-hello
taskRef:
name: say-hello
params:
- name: username
value: $(params.username)
A detailed introduction to Tekton, its resources and the associated concepts can be found in the official documentation.
Reusability of resources
Certain Tekton resources, such as tasks, are designed to be easily reused, e.g. in several pipeline resources. Nevertheless, in larger projects, numerous YAML files containing the configuration of the various resources are quickly created. If triggers and event listeners are used in addition to tasks and pipelines, this problem is even worse. For each new project or application that is to be added to the CI/CD system based on Tekton, many new YAML files must be created and adapted. This task can be carried out via copy-paste-modify, with the help of merge strategies on the basis of kustomize or via a simple templating mechanism.
YAML templating with ytt
In my experience, the YAML files used for certain Tekton resources within a team or project quickly become very similar in terms of content. For example, pipelines for similar applications usually consist of a similar sequence of tasks. These resources are therefore very suitable for generating on the basis of a template. ytt is a simple command line tool for generating YAML files based on templates. Among other things, it supports the use of placeholders. The installation of ytt is described in the documentation.
Templates for Tekton resources
The use of ytt in conjunction with Tekton will be demonstrated using a pipeline resource as an example. The following pipeline resource contains the task git-clone.
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
name: pipeline-project-a
spec:
params:
- name: branch
type: string
tasks:
- name: git-clone
params:
- name: url
value: ssh://git@scm.acme.com/project-a.git
- name: branch
value: $(params.branch)
taskRef:
kind: Task
name: git-clone
It is easy to imagine that every pipeline resource has such a task for every application that is managed in a Git repository. The only difference between the pipeline resources of different applications is the concrete path to the Git repository. The concrete pipeline resource can easily be converted into a template. To do this, the variable positions are provided with ytt-specific placeholder comments and an import statement is added at the beginning of the file, which causes ytt to replace the placeholders with the help of so-called data values from the @ytt:data module. A template file pipeline-template.yaml would therefore have the following content:
#@ load("@ytt:data", "data")
---
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
name: #@ "pipeline-" + data.values.project.name
spec:
params:
- name: branch
type: string
tasks:
- name: git-clone
params:
- name: url
value: #@ "ssh://git@scm.acme.com/" + data.values.project.name + ".git"
- name: branch
value: $(params.branch)
taskRef:
kind: Task
name: git-clone
The available data values are defined using a schema. The schema file schema.yaml for this example has the following content:
#@data/values-schema
---
project:
name: "project-name"
The placeholder data.values.project.name has the default value project-name.
Generating Tekton resources
New YAML files can now be generated using the template and the schema. The specific values for the placeholders can be defined in a separate file, such as values.yaml:
---
project:
name: "project-xyz"
ytt is then called from the command line:
ytt --file schema.yaml --file pipeline-template.yaml --data-values-file values.yaml --output-files ./generated --file-mark "pipeline-template.yaml:path=pipeline/pipeline-xyz.yaml"
The result of this call is a concrete pipeline resource for project-xyz in the directory ./generated/pipeline with the file name pipeline-xyz.yaml.
Combination with kustomize
If kustomize is used in conjunction with Kubernetes, the generated files must be added to the corresponding kustomization.yaml files. This can be automated with the help of yq. yq is also a command line tool and can be used to manipulate YAML files. A newly created file pipeline-xyz.yaml can be added to the resources list of a kustomization.yaml with the following command.
yq --inplace '.resources += "pipeline-xyz.yaml"' ./kustomization.yaml
Both ytt and yq can easily be combined into a bash script, making it easy to create scripts that can automate these types of repetitive tasks and help avoid errors that can quickly occur with the copy-paste-modify approach, for example.
Summary
With the help of Tekton, CI/CD systems can be created in Kubernetes or OpenShift environments. The available CRDs enable the implementation of flexible pipelines, among other things. Configuration via YAML files is familiar in the Kubernetes universe, but with many applications and many pipelines it quickly leads to numerous files. The initial manual effort required to create a pipeline for a new application, for example, can be reduced using the templating mechanism shown with yyt. If additional Tekton components such as triggers or event listeners are used, the number of YAML files increases and the use of templates becomes all the more worthwhile.