How to build cloud-ready images with Cloud Native Buildpacks.

19.05.2021•Stefan Welsch
Cloud Cloud native Kubernetes DevOps Framework How-to

Buildpacks transform application source code into images that can be run directly in the cloud. The code is examined to determine which dependencies are needed.

Buildpacks was first designed by Heroku in 2011. Since then the project has been adapted by Cloud Foundry and other Platform as a Service (PaaS) providers (Google App Engine, Gitlab, …).

The Cloud Native Buildpacks project was initiated by Pyvotal and Heroku in January 2018. Only 9 months later, in October 2018, the project joined the CNCF. The aim of the project is to standardize the buildpack ecosystem. There is also the Platform-To-Build contract, which has been precisely defined and contains findings from years of experience by Pyvotal and Heroku.

Cloud Native Buildpacks support modern container standards such as OCI (Open Container Initiative) and always use the latest capabilities.

Components

Let’s look at the most important components of build packs.

Builder

A builder is an image that contains all the components needed to execute a build. A builder image consists of a build image, lifecycle, buildpacks and other files that are necessary for configuration.

build pack

A buildpack is a unit that looks at the application code and uses it to formulate a plan for how the application will be executed. The correct build pack is determined based on the code and then the build is carried out with all necessary installations.

Lifecycle

The lifecycle orchestrates buildpack executions and then merges the artifacts into the app image

Platform

The platform is, for example, the Pack CLI or in the CICD process a plug-in which generates the OCI image from the lifecycle, build pack and application code.

Let’s build our first app

So that we can get started, we first need a platform with which we can generate our OCI image. We can simply use the Pack CLI locally for this purpose. The installation of the Pack CLI is possible in various forms and is described very well on the Pack CLI page. There is, for example, a binary for Windows, Mac and the various Linux distributions, which can be easily installed using the relevant package manager.

Since we want to remain independent of the operating system, I want to use the official Docker image here.

In the first step, let’s take a look at which builders are available to us. We can achieve this with the following command.

1
2
3
4
5
6
7
8

docker run -ti buildpacksio/pack builder suggest                                                                                                                                                                                                                                                              09:54:32
Suggested builders:
	Google:                gcr.io/buildpacks/builder:v1      Ubuntu 18 base image with buildpacks for .NET, Go, Java, Node.js, and Python
	Heroku:                heroku/buildpacks:18              Base builder for Heroku-18 stack, based on ubuntu:18.04 base image
	Heroku:                heroku/buildpacks:20              Base builder for Heroku-20 stack, based on ubuntu:20.04 base image
	Paketo Buildpacks:     paketobuildpacks/builder:base     Ubuntu bionic base image with buildpacks for Java, .NET Core, NodeJS, Go, Ruby, NGINX and Procfile
	Paketo Buildpacks:     paketobuildpacks/builder:full     Ubuntu bionic base image with buildpacks for Java, .NET Core, NodeJS, Go, PHP, Ruby, Apache HTTPD, NGINX and Procfile
	Paketo Buildpacks:     paketobuildpacks/builder:tiny     Tiny base image (bionic build image, distroless-like run image) with buildpacks for Java Native Image and Go

We can now use this builder to build our OCI image directly from the source code of our application. So first I create a go file with the following content:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

package main

import (
	"fmt"
	"log"
	"net/http"
)

func main() {
	port := "8080"

	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		_, err := fmt.Fprint(w, "hello from b-nova")
		if err != nil {
			log.Fatalf("error printing message to response writer %s", err)
			return
		}
	})

	log.Printf("Now listening on port %s.", port)
	log.Fatal(http.ListenAndServe(fmt.Sprintf(":%s", port), nil))
}

Who is not familiar with Go: We are starting a simple HTTP server on port 8080, which makes a static output as soon as we call the URL.

Now we can already create our first image from the source code. So in the console we enter the following:

1
2
3
4
5

docker run \
    -v /var/run/docker.sock.raw:/var/run/docker.sock \
    -u501 \
    -v $PWD:/workspace -w /workspace \
    buildpacksio/pack build go-sample --builder=gcr.io/buildpacks/builder:v1

We are now receiving a long output that we want to take a closer look at

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

Unable to find image 'buildpacksio/pack:latest' locally
latest: Pulling from buildpacksio/pack
...
Status: Downloaded newer image for buildpacksio/pack:latest
v1: Pulling from buildpacks/builder
...
Status: Downloaded newer image for gcr.io/buildpacks/builder:v1
v1: Pulling from buildpacks/gcp/run
...
Status: Downloaded newer image for gcr.io/buildpacks/gcp/run:v1

We see here that different images are being downloaded.

buildpacksio/pack → This is our platform with which we can create the OCI image together with the lifecycle, the build pack and the application code.

buildpacks/builder → Here is our build image. This image is used to create the build environment. The lifecycle and build packs are then executed in the build environment.

buildpacks/gcp/run → The run image is the base image for our application image.

Build image and run image are also called stack. You always need them together to create an image.

Next comes the lifecycle. We see the following output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

===> DETECTING
4 of 6 buildpacks participating
google.go.runtime  0.9.1
google.go.gopath   0.9.0
google.go.build    0.9.0
google.utils.label 0.0.1
===> ANALYZING
Previous image with name "go-sample" not found
Restoring metadata for "google.go.runtime:go" from cache
===> RESTORING
Restoring data for "google.go.runtime:go" from cache
===> BUILDING
=== Go - Runtime (google.go.runtime@0.9.1) ===
...
Using latest runtime version: 1.16.3
=== Go - Gopath (google.go.gopath@0.9.0) ===
--------------------------------------------------------------------------------
Running "go get -d (GOPATH=/layers/google.go.gopath/gopath GO111MODULE=off)"
Done "go get -d (GOPATH=/layers/google.go.gopath/gopath GO111MODUL..." (118.624084ms)
=== Go - Build (google.go.build@0.9.0) ===
--------------------------------------------------------------------------------
Running "go list -f {{if eq .Name \"main\"}}{{.Dir}}{{end}} ./..."
/workspace
Done "go list -f {{if eq .Name \"main\"}}{{.Dir}}{{end}} ./..." (61.671871ms)
--------------------------------------------------------------------------------
Running "go build -o /layers/google.go.build/bin/main ./. (GOCACHE=/layers/google.go.build/gocache)"
Done "go build -o /layers/google.go.build/bin/main ./. (GOCACHE=/l..." (529.000116ms)
=== Utils - Label Image (google.utils.label@0.0.1) ===
===> EXPORTING
Adding layer 'google.go.build:bin'
Adding 1/1 app layer(s)
Adding layer 'launcher'
Adding layer 'config'
Adding layer 'process-types'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
Setting default process type 'web'
Saving go-sample...
*** Images (b5c6cd0a9637):
      go-sample
Reusing cache layer 'google.go.runtime:go'
Successfully built image 'go-sample'

As we can see, the lifecycle consists of:

DETECT (1): Suitable build packs that are used during the build phase are found here

ANALYZE (7): Here files are restored which can optimize the build and export phase

RESTORE (10): Layers are restored from the cache here

BUILD (12): Here the application code is transformed into an executable artifact which can be packed in a container.

EXPORT (29): The OCI image is created here

After our image has been built we can simply start this with the docker run command

1
2

docker run -p8080:8080 go-sample
2021/05/04 05:46:20 Now listening on port 8080.

Our own buildpack

Now we have seen how you can transform your application into an image with an already existing buildpack. Now let’s see how you can write your own buildpack. This is useful, for example, if you want to modify the build process. To do this, we first install the Pack CLI Binary.

1

brew install buildpacks/tap/pack

Then we create the necessary files for our own build pack. The project then has the following structure:

We now add the following to the files:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# buildpack.toml
 
# Buildpack API version
api = "0.5"

# Buildpack ID and metadata
[buildpack]
id = "com.bnova/go-sample"
version = "0.0.1"
name = "Go Sample"

# Stacks that the buildpack will work with
[[stacks]]
id = "io.buildpacks.samples.stacks.bionic"

1
2
3
4
5
6

# detect

#!/usr/bin/env bash
set -eo pipefail

exit 1

1
2
3
4
5
6
7

# build

#!/usr/bin/env bash
set -eo pipefail

echo "---> Go Buildpack"
exit 1

Then we have to make the two files in the bin folder executable.

1

chmod +x go-buildpack/bin/detect go-buildpack/bin/build

To test our buildpack, we have to run the buildpack against our go application. To do this, we do the following in the CLI.

1
2

# Set the default builder
pack config default-builder cnbs/sample-builder:bionic

Now we build our application with our buildpack.

1
2
3
4
5
6
7
8

pack build go-sample --path . --buildpack ./go-buildpack

Output: 
===> DETECTING
[detector] err:  com.bnova/go-sample@0.0.1 (1)
[detector] ERROR: No buildpack groups passed detection.
[detector] ERROR: failed to detect: buildpack(s) failed with err
ERROR: failed to build: executing lifecycle. This may be the result of using an untrusted builder: failed with status code: 101

As we can see, the build fails because we only return an error in our detect script. If we want to adapt this so that we no longer receive an error with a go file.

1
2
3
4
5
6
7
8

# detect

#!/usr/bin/env bash
set -eo pipefail

if [[ -f *.go ]]; then
   exit 100
fi

If we build our application again, we get the following output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

pack build go-sample --path . --buildpack ./go-buildpack

Output: 
===> DETECTING
[detector] com.bnova/go-sample 0.0.1
===> ANALYZING
===> RESTORING
===> BUILDING
[builder] ---> Go Buildpack
[builder] ERROR: failed to build: exit status 1
ERROR: failed to build: executing lifecycle. This may be the result of using an untrusted builder: failed with status code: 145

Now we have to write our build script to build the application. The finished script looks like this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

#!/usr/bin/env bash
set -eo pipefail

echo "---> Go Buildpack"

# 1. GET ARGS
layersdir=$1

# 2. CREATE THE LAYER DIRECTORY
golayer="$layersdir"/go
mkdir -p "$golayer"

# 3. DOWNLOAD GO
echo "---> Downloading and extracting Go"
go_url=https://golang.org/dl/go1.16.3.linux-amd64.tar.gz
wget -q -O - "$go_url" | tar -xzf - -C "$golayer"

# 4. MAKE GO AVAILABLE DURING LAUNCH
echo -e 'launch = true' > "$layersdir/go.toml"

# 5. MAKE GO AVAILABLE TO THIS SCRIPT
export PATH="$golayer"/go/bin:$PATH

# 6. BUILD THE APP
go build

# ========== ADDED ===========
# 7. SET DEFAULT START COMMAND
cat > "$layersdir/launch.toml" <<EOL
[[processes]]
type = "web"
command = "./go-sample"
EOL

Let’s take a closer look at the script.

Steps 1-5:

We create the layer for the go installation and make it available for the build.

Step 6:

We build our application and generate the finished binary

Step 7:

We have to give our application a default start command. We can specify several processes here if we have different entry points (for example to run an async task). The “web” process is currently the default process.

Now we can run our build again and shouldn’t get any more errors.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

pack build go-sample --path . --buildpack ./go-buildpack

Output:
bionic: Pulling from cnbs/sample-builder
Digest: sha256:a674cd6b556924e0b36000c00f0cda8ee42c20aa9be45e4ddfc65ea43c5423e7
Status: Image is up to date for cnbs/sample-builder:bionic
bionic: Pulling from cnbs/sample-stack-run
Digest: sha256:0e6d2966062c26f0a0660c89c5bd1dba7e1fa019e6d68ef5c3694eafde1ab805
Status: Image is up to date for cnbs/sample-stack-run:bionic
0.10.2: Pulling from buildpacksio/lifecycle
Digest: sha256:c3a070ed0eaf8776b66f9f7c285469edccf5299b3283c453dd45699d58d78003
Status: Image is up to date for buildpacksio/lifecycle:0.10.2
===> DETECTING
[detector] b-nova.com/go-sample 0.0.1
===> ANALYZING
===> RESTORING
===> BUILDING
[builder] ---> Go Buildpack
[builder] ---> Downloading and extracting Go
===> EXPORTING
[exporter] Adding layer 'b-nova.com/go-sample:go'
[exporter] Adding 1/1 app layer(s)
[exporter] Reusing layer 'launcher'
[exporter] Adding layer 'config'
[exporter] Reusing layer 'process-types'
[exporter] Adding label 'io.buildpacks.lifecycle.metadata'
[exporter] Adding label 'io.buildpacks.build.metadata'
[exporter] Adding label 'io.buildpacks.project.metadata'
[exporter] Setting default process type 'web'
[exporter] *** Images (10b297b97276):
[exporter]       go-sample
Successfully built image go-sample

environment variables

Now that we have our own build pack, we can, for example, pass environment variables to our build process. Let’s take a look at a small example. Let us assume that we should be able to configure the port of the application in the build process.

So we change the initialization of the port in our application as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

alt: 
func main() {
  port := "8080"

neu:
var Port string
func main() {
  port := Port
  if port == "" {
    port = "8080"
  }

So we read the port from the PORT environment variable, which is transferred to our Go application.

Then we have to modify our build file so that we have access to the environment variables. We add the following under step 1 “GET ARGS”:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# 1. GET ARGS
layersdir=$1

# ENV VARS
platform_dir=$2
env_dir=${platform_dir}/env
echo "     env_dir: ${env_dir}"
echo "     env vars:"
if compgen -G "${env_dir}/*" > /dev/null; then
  for var in ${env_dir}/*; do
    declare "$(basename ${var})=$(<${var})"
  done
fi
export | sed 's/^/       /'

and change the build command as follows

1
2

# 6. BUILD THE APP
go build -ldflags "-X main.Port=$PORT"

Now we can give the port as an environment variable when building the image. We carry out the build as follows:

1

pack build go-sample --path . --env="PORT:8081" --buildpack ./go-buildpack

The Http server should now be running on port 8081 instead of port 8080.

The entire source code can be found in our GitHub repository: https://github.com/b-nova/buildpacks-go-sample

Outlook

Cloud Native Buildpacks are a very powerful and simple means of quickly transforming your source code into an image. Today we saw a little preview of how you can create your own buildpack to customize the build. I will be looking at Tekton in connection with Cloud Native Buildpacks in the next few days. But it is not yet certain whether there will be another blog post about it.

What is certain, however, is the fact that we at b-nova will continue to deal with interesting topics relating to the topics of Cloud, GitOps and DevOps. Stay tuned.

This text was automatically translated with our golang markdown translator.

Stefan Welsch

Stefan Welsch – Pionier, Stuntman, Mentor. Als Gründer von b-nova ist Stefan immer auf der Suche nach neuen und vielversprechenden Entwicklungsfeldern. Er ist durch und durch Pragmatiker und schreibt daher auch am liebsten Beiträge die sich möglichst nahe an 'real-world' Szenarien anlehnen.