We’re always looking for ways to enhance your experience with Docker, whether you’re using an integration, extension, or directly in product. Docker Desktop 4.18 focuses on improvements in the command line and in Docker Desktop.

Read on to learn about new CLI features in Docker Scout, and find out about Docker init, an exciting CLI Beta feature to help you quickly add Docker to any project. We also review new features to help you get up and running with Docker faster: Container File Explorer, adminless macOS install, and a new experimental feature in Docker Compose.

Docker Scout CLI

In Docker Desktop 4.17, we introduced Docker Scout, a tool that provides visibility into image vulnerabilities and recommendations for quick remediation. We are delighted to announce the release of several new features into the Docker Scout command line, which ships with Docker Desktop 4.18. These updates come after receiving an overwhelming amount of community feedback.

The 4.18 release of Docker Scout includes a vulnerability quickview, image recommendations directly on the command line, improved remediation guidance with BuildKit SBOM utilization, and a preview feature comparing images (imagine using diff, but for container images).

Quickview

Suppose that you have created a new container image and would like to assess its security posture. You can now run docker scout quickview for an instant, high-level security insight into your image. If any issues are found, Docker Scout will guide you on what to do next.

`docker scout quickview` output showing image vulnerability information

Command-line recommendations

If you’ve previously used docker scout cves to understand which CVEs exist in your images, you may have wondered what course of action to take next. With the new docker scout recommendations command, you receive a list of recommendations that directly suggest available updates for the base image.

The docker scout recommendations command analyzes the image and displays recommendations to refresh or update the base image, along with a list of benefits, including opportunities to reduce vulnerabilities or how to achieve smaller image sizes.

‘docker scout recommendations’ output showing available image updates for vulnerable images

BuildKit provenance and SBOM attestations

In January 2023, BuildKit was extended to support building images with attestations. These images can now use the docker scout command line to process this information and determine relevant next steps. We can support this as the docker scout command-line tool knows exactly what base image you built with and can provide more accurate recommendations.

If an image was built and pushed with an attached SBOM attestation, docker scout reads the package information from the SBOM attestation instead of creating a new local SBOM.

To learn how to build images with attestations using BuildKit, read “Generating SBOMs for Your Image with BuildKit.”

Compare images

Note: This is an experimental Docker Scout feature and may change and evolve over time.

Retrospectively documenting the changes made to address a security issue after completing a vulnerability remediation is considered a good practice. Docker Desktop 4.18 introduces an early preview of image comparison.

Comparison of vulnerability differences between two images

This feature highlights the vulnerability differences between two images and how packages compare. These details include the package version, environment variables in each image, and more. Additionally, the command-line output can be set up in a markdown format. This information can then be used to generate diff previews in pull requests through GitHub Actions.

We’d love to know what scenarios you could imagine using this diff feature in. You can do this by opening up Docker Desktop, navigating to the Images tab, and selecting Give feedback.

Read the documentation to learn more about these features.

Container File Explorer

Another feature we’re happy to announce is the GA release of Container File Explorer. When you need to check or quickly replace files within a container, Container File Explorer will help you do this — and much more — straight from Docker Desktop’s UI.

You won’t need to remember long CLI commands, fiddle with long path parameters on the docker cp command, or get frustrated that your container has no shell at all to check the files. Container File Explorer provides a simple UI that allows you to:

• Check a container file system
• Copy files and folders between host and containers
• Easily drag and drop files to a container
• Quickly edit files with syntax highlighting
• Delete files

With Container File Explorer, you can view your containers’ files at any state (stopped/running/paused/…) and for any container type, including slim-containers/slim-images (containers without a shell). Open the dashboard, go to the Containers tab, open the container action menu, and check your files:

Container File Explorer UI in Docker Desktop

Adminless install on macOS

We’ve adjusted our macOS install flow to make it super easy for developers to install Docker Desktop without granting them admin privileges. Some developers work in environments with elevated security requirements where local admin access may be prohibited on their machines. We wanted to make sure that users in these environments are able to opt out of Docker Desktop functionality that requires admin privileges.

The default install flow on macOS will still ask for admin privileges, as we believe this allows us to provide an optimized experience for the vast majority of developer use cases. Upon granting admin privileges, Docker Desktop automatically installs the Docker CLI tools, enabling third-party libraries to seamlessly integrate with Docker (by enabling the default Docker socket) and allowing users to bind to privileged ports between 1 and 1024.

If you want to change the settings you configured at install, you can do so easily within the Advanced tab of Docker Desktop’s Settings.

Docker init (Beta)

Another exciting feature we’re releasing in Beta is docker init. This is a new CLI command that lets you quickly add Docker to your project by automatically creating the required assets: Dockerfiles, Compose files, and .dockerignore. Using this feature, you can easily update existing projects to run using containers or set up new projects even if you’re not familiar with Docker.

You can try docker init by updating to the latest version of Docker Desktop (4.18.0) and typing docker init in the command line while inside a target project folder. docker init will create all the required files to run your project in Docker.

Refer to the docker init documentation to learn more.

The Beta version of docker init ships with Go support, and the Docker team is already working on adding more languages and frameworks, including Node.js, Python, Java, Rust, and .Net, plus other features in the coming months. If there is a specific language or framework you would like us to support, let us know. Submit other feedback and suggestions in our public roadmap.

Note: Please be aware that docker init should not be confused with the internally-used docker-init executable, which is invoked by Docker when utilizing the –init flag with the docker run command. Refer to the docs to learn more.

`docker init` command-line output on how to get started

Docker Compose File Watch (Experimental)

Docker Compose has a new trick! Docker Compose File Watch is available now as an Experimental feature to automatically keep all your service containers up-to-date while you work.

With the 4.18 release, you can optionally add a new x-develop section to your services in compose.yaml:

services: web: build: . # !!! x-develop is experimental !!! x-develop: watch: - action: sync path: ./web target: /app/web - action: rebuild path: .package.json

Once configured, the new docker compose alpha watch command will start monitoring for file edits within your project:

• On a change to ./web/App.jsx, for example, Compose will automatically synchronize it to /src/web/App.jsx inside the container.
• Meanwhile, if you modify package.json (such as by installing a new npm package), Compose will rebuild the image and replace the existing service with an updated version.

Compose File Watch mode is just the start. With nearly 100 commits since the last Docker Compose release, we’ve squashed bugs and made a lot of quality-of-life improvements. (A special shout-out to all our recent first-time contributors!)

We’re excited about Docker Compose File Watch and are actively working on the underlying mechanics and configuration format.

Conclusion

That’s a wrap for our Docker Desktop 4.18 update. This release includes many cool, new features, including some that you can help shape! We also updated the Docker Engine to address some CVEs. As always, we love hearing your feedback. Please leave any feedback on our public GitHub roadmap and let us know what else you’d like to see.

Check out the release notes for a full breakdown of what’s new in Docker Desktop 4.18.

Performance testing is a critical component of software testing and performance evaluation. It involves simulating a large number of users accessing a system simultaneously to determine the system’s behavior under high user loads. This process helps organizations understand how their systems will perform in real-world scenarios and identify potential performance bottlenecks. Testing the performance of your application under different load conditions also helps identify bottlenecks and improve your application’s performance.

In this article, we provide an introduction to the Ddosify Docker Extension and show how to get started using it for performance testing.

The importance of performance testing

Performance testing should be regularly performed to ensure that your application is performing well under different load conditions so that your customers can have a great experience. Kissmetrics found that a 1-second delay in page response time can lead to a seven percent decrease in conversions and that half of the customers expect a website to load in less than 2 seconds. A 1-second delay in page response could result in a potential loss of several million dollars in annual sales for an e-commerce site.

Meet Ddosify

Ddosify is a high-performance, open-core performance testing platform that focuses on load and latency testing. Ddosify offers a suite of three products:

1. Ddosify Engine: An open source, single-node, load-testing tool (6K+ stars) that can be used to test your application from your terminal using a simple JSON file. Ddosify is written in Golang and can be deployed on Linux, macOS, and Windows. Developers and small companies are using Ddosify Engine to test their applications. The tool is available on GitHub.

2. Ddosify Cloud: An open core SaaS platform that allows you to test your application without any programming expertise. Ddosify Cloud uses Ddosify Engine in a distributed manner and provides a web interface to generate load test scenarios without code. Users can test their applications from different locations around the world and can generate advanced reports. We are using different technologies including Docker, Kubernetes, InfluxDB, RabbitMQ, React.js, Golang, AWS, and PostgreSQL within this platform and all working together transparently for the user. This tool is available on the Ddosify website.

3. Ddosify Docker Extension: This tool has similarities to Ddosify Engine, but has an easy-to-use user interface thanks to the extension capability of Docker Desktop. This feature allows you to test your application within Docker Desktop. The Ddosify Docker Extension is available free of charge from the Extension marketplace. The Ddosify Docker Extension repository is open source and available on GitHub. The tool is also available from the Docker Extensions Marketplace.

In this article, we will focus on the Ddosify Docker Extension.

The architecture of Ddosify

Ddosify Docker Extension uses the Ddosify Engine as a base image under the hood. We collect settings, including request count, duration, and headers, from the extension UI and send them to the Ddosify Engine.

The Ddosify Engine performs the load testing and returns the results to the extension. The extension then displays the results to the user (Figure 1).

Figure 1: Overview of Ddosify.

Why Ddosify?

Ddosify is easy to use and offers many features, including dynamic variables, CSV data import, various load types, correlation, and assertion. Ddosify also has different options for different use cases. If you are an individual developer, you can use the Ddosify Engine or Ddosify Docker Extension free of charge. If you need code-free load testing, advanced reporting, multi-geolocation, and more requests per second (RPS), you can use the Ddosify Cloud.

With Ddosify, you can:

• Identify performance issues of your application by simulating high user traffic.
• Optimize your infrastructure and ensure that you are only paying for the resources that you need.
• Identify bugs before your customers do. Some bugs are only triggered under high load.
• Measure your system capacity and identify its limitations.

Why run Ddosify as a Docker Extension?

Docker Extensions help you build and integrate software applications into your daily workflows. With Ddosify Docker Extension, you can easily perform load testing on your application from within Docker Desktop. You don’t need to install anything on your machine except Docker Desktop. Features of Ddosify Docker Extension include:

• Strong community with 6K+ GitHub stars and a total of 1M+ downloads on all platforms. Community members contribute by proposing/adding features and fixing bugs.
• Currently supports HTTP and HTTPS protocols. Other protocols are on the way.
• Supports various load types. Test your system’s limits across different load types, including:
  • Linear
  • Incremental
  • Waved
• Dynamic variables (parameterization) support. Just like Postman, Ddosify supports dynamic variables.
• Save load testing results as PDF.

Getting started

As a prerequisite, you need Docker Desktop 4.10.0 or higher installed on your machine. You can download Docker Desktop from our website.

Step 1: Install Ddosify Docker Extension

Because Ddosify is an extension partner of Docker, you can easily install Ddosify Docker Extension from the Docker Extensions Marketplace (Figure 2). Start Docker Desktop and select Add Extensions. Next, filter by Testing Tools and select Ddosify. Click on the Install button to install the Ddosify Docker Extension. After a few seconds, Ddosify Docker Extension will be installed on your machine.

Figure 2: Installing Ddosify.

Step 2: Start load testing

You can start load testing your application from the Docker Desktop (Figure 3). Start Docker Desktop and click on the Ddosify icon in the Extensions section. The UI of the Ddosify Docker Extension will be opened.

Figure 3: Starting load testing.

You can start load testing by entering the target URL of your application. You can choose HTTP Methods (GET, POST, PUT, DELETE, etc.), protocol (HTTP, HTTPS), request count, duration, load type (linear, incremental, waved), timeout, body, headers, basic auth, and proxy settings. We chose the following values:

URL:

https://testserver.ddosify.com/account/register/

Method:

POST

Protocol:

HTTPS

Request Count:

100

Duration:

5

Load Type:

Linear

Timeout:

10

Body:

{“username”: “{{_randomUserName}}”, “email”: “{{_randomEmail}}”, “password”: “{{_randomPassword}}”}

Headers:

User-Agent: DdosifyDockerExtension/0.1.2Content-Type: application/json

In this configuration, we are sending 100 requests to the target URL for 5 seconds (Figure 4). The RPS is 20. The target URL is a test server that is used to register new users with body parameters. We are using dynamic variables (random) for username, email, and password in the body. You can learn more about dynamic variables from the Ddosify documentation.

Figure 4: Parameters for sample load test.

Then click on the Start Load Test button to begin load testing. The results will be displayed in the UI (Figure 5).

Figure 5: Ddosify test results.

The test results include the following information:

• 48 requests successfully created users. Response Code: 201
• 20 requests failed to create users because of the duplicate username and emails with the server. Response Code: 400
• 32 requests failed to create users because of the timeout. The server could not respond within 10 seconds, so we should increase the timeout value or optimize the server

You can also save the load test results. Click on the Report button to save the results as a PDF file (Figure 6).

Figure 6: Save results as PDF.

Conclusion

In this article, we showed how to install Ddosify Docker Extension and quickly start load testing your application from Docker Desktop. We created random users on a test server with 100 requests for 5 seconds, and we saw that the server could not handle all the requests because of the timeout.

If you need help with Ddosify, you can create an issue on our GitHub repository or join our Discord server.

Resources

• Ddosify
• Ddosify Engine
• Ddosify Docker Extension
• Ddosify Docker Extension source codes

This article is a result of open source collaboration. During Hacktoberfest 2022, the project was announced in the Black Forest Docker meetup group and received contributions from members of the meetup group and other Hacktoberfest contributors. Almost all of the code in the GitHub repo was written by Stefan Ruf, Himanshu Kandpal, and Sreekesh Iyer.

The MEAN stack is a fast-growing, open source JavaScript stack used to develop web applications. MEAN is a diverse collection of robust technologies — MongoDB, Express.js, Angular, and Node.js — for developing scalable web applications.

The stack is a popular choice for web developers as it allows them to work with a single language throughout the development process and it also provides a lot of flexibility and scalability. Node, Express, and Angular even claimed top spots as popular frameworks or technologies in Stack Overflow’s 2022 Developer Survey.

In this article, we’ll describe how the MEAN stack works using an Event Posting app as an example.

How does the MEAN stack work?

MEAN consists of the following four components:

• MongoDB — A NoSQL database
• ExpressJS —  A backend web-application framework for NodeJS
• Angular — A JavaScript-based front-end web development framework for building dynamic, single-page web applications
• NodeJS — A JavaScript runtime environment that enables running JavaScript code outside the browser, among other things

Here’s a brief overview of how the different components might work together:

• A user interacts with the frontend, via the web browser, which is built with Angular components.
• The backend server delivers frontend content, via ExpressJS running atop NodeJS.
• Data is fetched from the MongoDB database before it returns to the frontend. Here, your application displays it for the user.
• Any interaction that causes a data-change request is sent to the Node-based Express server.

Why is the MEAN stack so popular?

The MEAN stack is often used to build full-stack, JavaScript web applications, where the same language is used for both the client-side and server-side of the application. This approach can make development more efficient and consistent and make it easier for developers to work on both the frontend and backend of the application.

The MEAN stack is popular for a few reasons, including the following:

• Easy learning curve — If you’re familiar with JavaScript and JSON, then it’s easy to get started. MEAN’s structure lets you easily build a three-tier architecture (frontend, backend, database) with just JavaScript and JSON.
• Model View Architecture — MEAN supports the Model-view-controller architecture, supporting a smooth and seamless development process.
• Reduces context switching — Because MEAN uses JavaScript for both frontend and backend development, developers don’t need to worry about switching languages. This capability boosts development efficiency.
• Open source and active community support — The MEAN stack is purely open source. All developers can build robust web applications. Its frameworks improve the coding efficiency and promote faster app development.

Running the Event Posting app

Here are the key components of the Event Posting app:

• MongoDB
• Express.js
• Angular
• Node.js
• Docker Desktop

Deploying the Event Posting app is a fast process. To start, you’ll clone the repository, set up the client and backend, then bring up the application.

Then, complete the following steps:

git clone https://github.com/dockersamples/events cd events/backend npm install npm run dev

General flow of the Event Posting app

The flow of information through the Event Posting app is illustrated in Figure 1 and described in the following steps.

Figure 1: General flow of the Event Posting app.

1. A user visits the event posting app’s website on their browser.
2. AngularJS, the frontend framework, retrieves the necessary HTML, CSS, and JavaScript files from the server and renders the initial view of the website.
3. When the user wants to view a list of events or create a new event, AngularJS sends an HTTP request to the backend server.
4. Express.js, the backend web framework, receives the request and processes it. This step includes interacting with the MongoDB database to retrieve or store data and providing an API for the frontend to access the data.
5. The back-end server sends a response to the frontend, which AngularJS receives and uses to update the view.
6. When a user creates a new event, AngularJS sends a POST request to the backend server, which Express.js receives and processes. Express.js stores the new event in the MongoDB database.
7. The backend server sends a confirmation response to the front-end, which AngularJS receives and uses to update the view and display the new event.
8. Node.js, the JavaScript runtime, handles the server-side logic for the application and allows for real-time updates. This includes running the Express.js server, handling real-time updates using WebSockets, and handling any other server-side tasks.

You can then access Event Posting at http://localhost:80 in your browser (Figure 2):

![Screenshot of blue "Add new event" button.](https://www.docker.com/wp-content/uploads/2023/03/Add-new-event-2-1110x392.png "Screenshot of blue "Add new event" button. - Add new event 2")

Figure 2: Add a new event.

Select Add New Event to add the details (Figure 3).

Figure 3: Add event details.

Save the event details to see the final results (Figure 4).

Figure 4: Display upcoming events.

Why containerize the MEAN stack?

Containerizing the MEAN stack allows for a consistent, portable, and easily scalable environment for the application, as well as improved security and ease of deployment. Containerizing the MEAN stack has several benefits, such as:

• Consistency: Containerization ensures that the environment for the application is consistent across different development, testing, and production environments. This approach eliminates issues that can arise from differences in the environment, such as different versions of dependencies or configurations.
• Portability: Containers are designed to be portable, which means that they can be easily moved between different environments. This capability makes it easy to deploy the MEAN stack application to different environments, such as on-premises or in the cloud.
• Isolation: Containers provide a level of isolation between the application and the host environment. Thus, the application has access only to the resources it needs and does not interfere with other applications running on the same host.
• Scalability: Containers can be easily scaled up or down depending on the needs of the application, resulting in more efficient use of resources and better performance.

Containerizing your Event Posting app

Docker helps you containerize your MEAN Stack — letting you bundle your complete Event Posting application, runtime, configuration, and operating system-level dependencies. The container then includes everything needed to ship a cross-platform, multi-architecture web application.

We’ll explore how to run this app within a Docker container using Docker Official Images. To begin, you’ll need to download Docker Desktop and complete the installation process. This step includes the Docker CLI, Docker Compose, and a user-friendly management UI, which will each be useful later on.

Docker uses a Dockerfile to create each image’s layers. Each layer stores important changes stemming from your base image’s standard configuration. Next, we’ll create an empty Dockerfile in the root of our project repository.

Containerizing your Angular frontend

We’ll build a multi-stage Dockerfile to containerize our Angular frontend.

A Dockerfile is a plain-text file that contains instructions for assembling a Docker container image. When Docker builds our image via the docker build command, it reads these instructions, executes them, and creates a final image.

With multi-stage builds, a Docker build can use one base image for compilation, packaging, and unit testing. A separate image holds the application’s runtime. This setup makes the final image more secure and shrinks its footprint (because it doesn’t contain development or debugging tools).

Let’s walk through the process of creating a Dockerfile for our application. First, create the following empty file with the name Dockerfile in the root of your frontend app.

touch Dockerfile

Then you’ll need to define your base image in the Dockerfile file. Here we’ve chosen the stable LTS version of the Node Docker Official Image. This image comes with every tool and package needed to run a Node.js application:

FROM node:lts-alpine AS build

Next, let’s create a directory to house our image’s application code. This acts as the working directory for your application:

WORKDIR /usr/src/app

The following COPY instruction copies the package.json and src file from the host machine to the container image.

The COPY command takes two parameters. The first tells Docker which file(s) you’d like to copy into the image. The second tells Docker where you want those files to be copied. We’ll copy everything into our working directory called /usr/src/app.

COPY package.json .

COPY package-lock.json . RUN npm ci

Next, we need to add our source code into the image. We’ll use the COPY command just like we previously did with our package.json file.

Note: It’s common practice to copy the package.json file separately from the application code when building a Docker image. This step allows Docker to cache the node_modules layer separately from the application code layer, which can significantly speed up the Docker build process and improve the development workflow.

COPY . .

Then, use npm run build to run the build script from package.json:

RUN npm run build

In the next step, we need to specify the second stage of the build that uses an Nginx image as its base and copies the nginx.conf file to the /etc/nginx directory. It also copies the compiled TypeScript code from the build stage to the /usr/share/nginx/html directory.

FROM nginx:stable-alpine

COPY nginx.conf /etc/nginx/nginx.conf COPY --from=build /usr/src/app/dist/events /usr/share/nginx/html

Finally, the EXPOSE instruction tells Docker which port the container listens on at runtime. You can specify whether the port listens on TCP or UDP. The default is TCP if the protocol isn’t specified.

EXPOSE 80

Here is our complete Dockerfile:

# Builder container to compile typescript FROM node:lts-alpine AS build WORKDIR /usr/src/app

Install dependencies

COPY package.json . COPY package-lock.json . RUN npm ci

Copy the application source

COPY . .

Build typescript

RUN npm run build

FROM nginx:stable-alpine

COPY nginx.conf /etc/nginx/nginx.conf COPY --from=build /usr/src/app/dist/events /usr/share/nginx/html

EXPOSE 80

Now, let’s build our image. We’ll run the docker build command as above, but with the -f Dockerfile flag. The -f flag specifies your Dockerfile name. The “.” command will use the current directory as the build context and read a Dockerfile from stdin. The -t tags the resulting image.

docker build . -f Dockerfile -t events-fe:1

Containerizing your Node.js backend

Let’s walk through the process of creating a Dockerfile for our backend as the next step. First, create the following empty Dockerfile in the root of your backend Node app:

# Builder container to compile typescript FROM node:lts-alpine AS build WORKDIR /usr/src/app

Install dependencies

COPY package.json . COPY package-lock.json . RUN npm ci

Copy the application source

COPY . .

Build typescript

RUN npm run build

FROM node:lts-alpine WORKDIR /app COPY package.json . COPY package-lock.json . COPY .env.production .env

RUN npm ci --production

COPY --from=build /usr/src/app/dist /app

EXPOSE 8000 CMD [ "node", "src/index.js"]

This Dockerfile is useful for building and running TypeScript applications in a containerized environment, allowing developers to package and distribute their applications more easily.

The first stage of the build process, named build, is based on the official Node.js LTS Alpine Docker image. It sets the working directory to /usr/src/app and copies the package.json and package-lock.json files to install dependencies with the npm ci command. It then copies the entire application source code and builds TypeScript with the npm run build command.

The second stage of the build process, named production, also uses the official Node.js LTS Alpine Docker image. It sets the working directory to /app and copies the package.json, package-lock.json, and .env.production files. It then installs only production dependencies with npm ci --production command, and copies the output of the previous stage, the compiled TypeScript code, from /usr/src/app/dist to /app.

Finally, it exposes port 8000 and runs the command node src/index.js when the container is started.

Defining services using a Compose file

Here’s how our services appear within a Docker Compose file:

services: frontend: build: context: "./frontend/events" dockerfile: "./Dockerfile" networks: - events_net backend: build: context: "./backend" dockerfile: "./Dockerfile" networks: - events_net db: image: mongo:latest ports: - 27017:27017 networks: - events_net proxy: image: nginx:stable-alpine environment: - NGINX_ENVSUBST_TEMPLATE_SUFFIX=.conf - NGINX_ENVSUBST_OUTPUT_DIR=/etc/nginx volumes: - ${PWD}/nginx.conf:/etc/nginx/templates/nginx.conf.conf ports: - 80:80 networks: - events_net

networks: events_net:

Your example application has the following parts:

• Four services backed by Docker images: Your Angular frontend, Node.js backend, MongoDB database, and Nginx as a proxy server
• The frontend and backend services are built from Dockerfiles located in ./frontend/events and ./backend directories, respectively. Both services are attached to a network called events_net.
• The db service is based on the latest version of the MongoDB Docker image and exposes port 27017. It is attached to the same events_net network as the frontend and backend services.
• The proxy service is based on the stable-alpine version of the Nginx Docker image. It has two environment variables defined, NGINX_ENVSUBST_TEMPLATE_SUFFIX and NGINX_ENVSUBST_OUTPUT_DIR, that enable environment variable substitution in Nginx configuration files.
• The proxy service also has a volume defined that maps the local nginx.conf file to /etc/nginx/templates/nginx.conf.conf in the container. Finally, it exposes port 80 and is attached to the events_net network.
• The events_net network is defined at the end of the file, and all services are attached to it. This setup enables communication between the containers using their service names as hostnames.

You can clone the repository or download the docker-compose.yml file directly from Dockersamples on GitHub.

Bringing up the container services

You can start the MEAN application stack by running the following command:

docker compose up -d

Next, use the docker compose ps command to confirm that your stack is running properly. Your terminal will produce the following output:

$ docker compose ps
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS events-backend-1 events-backend "docker-entrypoint.s…" backend 29 minutes ago Up 29 minutes 8000/tcp events-db-1 mongo:latest "docker-entrypoint.s…" db 5 seconds ago Up 4 seconds 0.0.0.0:27017->27017/tcp events-frontend-1 events-frontend "/docker-entrypoint.…" frontend 29 minutes ago Up 29 minutes 80/tcp events-proxy-1 nginx:stable-alpine "/docker-entrypoint.…" proxy 29 minutes ago Up 29 minutes 0.0.0.0:80->80/tcp

Viewing the containers via Docker Dashboard

You can also leverage the Docker Dashboard to view your container’s ID and easily access or manage your application (Figure 5):

Figure 5: Viewing running containers in Docker Dashboard.

Conclusion

Congratulations! You’ve successfully learned how to containerize a MEAN-backed Event Posting application with Docker. With a single YAML file, we’ve demonstrated how Docker Compose helps you easily build and deploy your MEAN stack in seconds. With just a few extra steps, you can apply this tutorial while building applications with even greater complexity. Happy developing!

After listening to feedback and consulting our community, it’s clear that we made the wrong decision in sunsetting our Free Team plan. Last week we felt our communications were terrible but our policy was sound. It’s now clear that both the communications and the policy were wrong, so we’re reversing course and no longer sunsetting the Free Team plan:

• If you’re currently on the Free Team plan, you no longer have to migrate to another plan by April 14.
• Customers who upgraded from a Free Team subscription to a paid subscription between the sunsetting announcement on March 14 and today’s announcement will automatically receive a full refund for the transaction in the next 30 days, allowing them to use their new paid subscription for free for the duration of the term they purchased.
• Customers who requested a migration to a Personal or Pro plan will be kept on their current Free Team plan. (Or they can choose to open a new Personal or Pro account via our website.)
• In the past 10 days we received & accepted more applications for our Docker-Sponsored Open Source program (DSOS) than we did in the previous year. We encourage eligible open source projects to continue to apply and are currently processing applications within a couple of business days.

For more details, you can visit our FAQ. We apologize for both the communications and the policy, and vow to be an ever-more trustworthy community member in the future.

If you have any questions, you’re welcome to contact me directly on Twitter @scottcjohnston or by emailing [email protected].

The Hugging Face Hub is a platform that enables collaborative open source machine learning (ML). The hub works as a central place where users can explore, experiment, collaborate, and build technology with machine learning. On the hub, you can find more than 140,000 models, 50,000 ML apps (called Spaces), and 20,000 datasets shared by the community.

Using Spaces makes it easy to create and deploy ML-powered applications and demos in minutes. Recently, the Hugging Face team added support for Docker Spaces, enabling users to create any custom app they want by simply writing a Dockerfile.

Another great thing about Spaces is that once you have your app running, you can easily share it with anyone around the world. 🌍

This guide will step through the basics of creating a Docker Space, configuring it, and deploying code to it. We’ll show how to build a basic FastAPI app for text generation that will be used to demo the google/flan-t5-small model, which can generate text given input text. Models like this are used to power text completion in all sorts of apps. (You can check out a completed version of the app at Hugging Face.)

Prerequisites

To follow along with the steps presented in this article, you’ll need to be signed in to the Hugging Face Hub — you can sign up for free if you don’t have an account already.

Create a new Docker Space 🐳

To get started, create a new Space as shown in Figure 1.

![Screenshot of Hugging Face Spaces, showing "Create new Space" button in upper right.](https://www.docker.com/wp-content/uploads/2023/03/create-new-hugging-face-space-1110x612.png "Screenshot of Hugging Face Spaces, showing "Create new Space" button in upper right. - create new hugging face space")

Figure 1: Create a new Space.

Next, you can choose any name you prefer for your project, select a license, and use Docker as the software development kit (SDK) as shown in Figure 2.

Spaces provides pre-built Docker templates like Argilla and Livebook that let you quickly start your ML projects using open source tools. If you choose the “Blank” option, that means you want to create your Dockerfile manually. Don’t worry, though; we’ll provide a Dockerfile to copy and paste later. 😅

Figure 2: Adding details for the new Space.

When you finish filling out the form and click on the Create Space button, a new repository will be created in your Spaces account. This repository will be associated with the new space that you have created.

Note: If you’re new to the Hugging Face Hub 🤗, check out Getting Started with Repositories for a nice primer on repositories on the hub.

Writing the app

Ok, now that you have an empty space repository, it’s time to write some code. 😎

The sample app will consist of the following three files:

• requirements.txt — Lists the dependencies of a Python project or application
• app.py — A Python script where we will write our FastAPI app
• Dockerfile — Sets up our environment, installs requirements.txt, then launches app.py

To follow along, create each file shown below via the web interface. To do that, navigate to your Space’s Files and versions tab, then choose Add file → Create a new file (Figure 3). Note that, if you prefer, you can also utilize Git.

![Screenshot showing selection of "Create a new file" under "Add file" dropdown menu.](https://www.docker.com/wp-content/uploads/2023/03/hugging-face-space-files-versions-1110x400.png "Screenshot showing selection of "Create a new file" under "Add file" dropdown menu. - hugging face space files versions")

Figure 3: Creating new files.

Make sure that you name each file exactly as we have done here. Then, copy the contents of each file from here and paste them into the corresponding file in the editor. After you have created and populated all the necessary files, commit each new file to your repository by clicking on the Commit new file to main button.

Listing the Python dependencies

It’s time to list all the Python packages and their specific versions that are required for the project to function properly. The contents of the requirements.txt file typically include the name of the package and its version number, which can be specified in a variety of formats such as exact version numbers, version ranges, or compatible versions. The file lists FastAPI, requests, and uvicorn for the API along with sentencepiece, torch, and transformers for the text-generation model.

fastapi==0.74.* requests==2.27.* uvicorn[standard]==0.17.* sentencepiece==0.1.* torch==1.11.* transformers==4.*

Defining the FastAPI web application

The following code defines a FastAPI web application that uses the transformers library to generate text based on user input. The app itself is a simple single-endpoint API. The /generate endpoint takes in text and uses a transformers pipeline to generate a completion, which it then returns as a response.

To give folks something to see, we reroute FastAPI’s interactive Swagger docs from the default /docs endpoint to the root of the app. This way, when someone visits your Space, they can play with it without having to write any code.

from fastapi import FastAPI from transformers import pipeline

Create a new FastAPI app instance

app = FastAPI()

Initialize the text generation pipeline

This function will be able to generate text

given an input.

pipe = pipeline("text2text-generation", model="google/flan-t5-small")

Define a function to handle the GET request at `/generate`

The generate() function is defined as a FastAPI route that takes a

string parameter called text. The function generates text based on the # input using the pipeline() object, and returns a JSON response

containing the generated text under the key "output"

@app.get("/generate") def generate(text: str): """ Using the text2text-generation pipeline from `transformers`, generate text from the given input text. The model used is `google/flan-t5-small`, which can be found [here](https://huggingface.co/google/flan-t5-small). """ # Use the pipeline to generate text from the given input text output = pipe(text)

# Return the generated text in a JSON response
return {"output": output\[0\]\["generated\_text"\]}

Writing the Dockerfile

In this section, we will write a Dockerfile that sets up a Python 3.9 environment, installs the packages listed in requirements.txt, and starts a FastAPI app on port 7860.

Let’s go through this process step by step:

FROM python:3.9

The preceding line specifies that we’re going to use the official Python 3.9 Docker image as the base image for our container. This image is provided by Docker Hub, and it contains all the necessary files to run Python 3.9.

WORKDIR /code

This line sets the working directory inside the container to /code. This is where we’ll copy our application code and dependencies later on.

COPY ./requirements.txt /code/requirements.txt

The preceding line copies the requirements.txt file from our local directory to the /code directory inside the container. This file lists the Python packages that our application depends on

RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt

This line uses pip to install the packages listed in requirements.txt. The --no-cache-dir flag tells pip to not use any cached packages, the --upgrade flag tells pip to upgrade any already-installed packages if newer versions are available, and the -r flag specifies the requirements file to use.

RUN useradd -m -u 1000 user USER user ENV HOME=/home/user \\ PATH=/home/user/.local/bin:$PATH

These lines create a new user named user with a user ID of 1000, switch to that user, and then set the home directory to /home/user. The ENV command sets the HOME and PATH environment variables. PATH is modified to include the .local/bin directory in the user’s home directory so that any binaries installed by pip will be available on the command line. Refer the documentation to learn more about the user permission.

WORKDIR $HOME/app

This line sets the working directory inside the container to $HOME/app, which is /home/user/app.

COPY --chown=user . $HOME/app

The preceding line copies the contents of our local directory into the /home/user/app directory inside the container, setting the owner of the files to the user that we created earlier.

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "7860"]

This line specifies the command to run when the container starts. It starts the FastAPI app using uvicorn and listens on port 7860. The --host flag specifies that the app should listen on all available network interfaces, and the app:app argument tells uvicorn to look for the app object in the app module in our code.

Here’s the complete Dockerfile:

# Use the official Python 3.9 image FROM python:3.9

Set the working directory to /code

WORKDIR /code

Copy the current directory contents into the container at /code

COPY ./requirements.txt /code/requirements.txt

Install requirements.txt

RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt

Set up a new user named "user" with user ID 1000

RUN useradd -m -u 1000 user

Switch to the "user" user

USER user

Set home to the user's home directory

ENV HOME=/home/user \\ PATH=/home/user/.local/bin:$PATH

Set the working directory to the user's home directory

WORKDIR $HOME/app

Copy the current directory contents into the container at $HOME/app setting the owner to the user

COPY --chown=user . $HOME/app

Start the FastAPI app on port 7860, the default port expected by Spaces

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "7860"]

Once you commit this file, your space will switch to Building, and you should see the container’s build logs pop up so you can monitor its status. 👀

If you want to double-check the files, you can find all the files at our app Space.

Note: For a more basic introduction on using Docker with FastAPI, you can refer to the official guide from the FastAPI docs.

Using the app 🚀

If all goes well, your space should switch to Running once it’s done building, and the Swagger docs generated by FastAPI should appear in the App tab. Because these docs are interactive, you can try out the endpoint by expanding the details of the /generate endpoint and clicking Try it out! (Figure 4).

![Screenshot of FastAPI showing "Try it out!" option on the right-hand side.](https://www.docker.com/wp-content/uploads/2023/03/fastapi-swagger-docs-1110x646.png "Screenshot of FastAPI showing "Try it out!" option on the right-hand side. - fastapi swagger docs")

Figure 4: Trying out the app.

Conclusion

This article covered the basics of creating a Docker Space, building and configuring a basic FastAPI app for text generation that uses the google/flan-t5-small model. You can use this guide as a starting point to build more complex and exciting applications that leverage the power of machine learning.

If you’re interested in learning more about Docker templates and seeing curated examples, check out the Docker Examples page. There you’ll find a variety of templates to use as a starting point for your own projects, as well as tips and tricks for getting the most out of Docker templates. Happy coding!

Today, Hugging Face and Docker are announcing a new partnership to democratize AI and make it accessible to all software engineers. Hugging Face is the most used open platform for AI, where the machine learning (ML) community has shared more than 150,000 models; 25,000 datasets; and 30,000 ML apps, including Stable Diffusion, Bloom, GPT-J, and open source ChatGPT alternatives. These apps enable the community to explore models, replicate results, and lower the barrier of entry for ML — anyone with a browser can interact with the models.

Docker is the leading toolset for easy software deployment, from infrastructure to applications. Docker is also the leading platform for software teams’ collaboration.

Docker and Hugging Face partner so you can launch and deploy complex ML apps in minutes. With the recent support for Docker on Hugging Face Spaces, folks can create any custom app they want by simply writing a Dockerfile. What’s great about Spaces is that once you’ve got your app running, you can easily share it with anyone worldwide! 🌍 Spaces provides an unparalleled level of flexibility and enables users to build ML demos with their preferred tools — from MLOps tools and FastAPI to Go endpoints and Phoenix apps.

Spaces also come with pre-defined templates of popular open source projects for members that want to get their end-to-end project in production in a matter of seconds with just a few clicks.

Spaces enable easy deployment of ML apps in all environments, not just on Hugging Face. With “Run with Docker,” millions of software engineers can access more than 30,000 machine learning apps and run them locally or in their preferred environment.

“At Hugging Face, we’ve worked on making AI more accessible and more reproducible for the past six years,” says Clem Delangue, CEO of Hugging Face. “Step 1 was to let people share models and datasets, which are the basic building blocks of AI. Step 2 was to let people build online demos for new ML techniques. Through our partnership with Docker Inc., we make great progress towards Step 3, which is to let anyone run those state-of-the-art AI models locally in a matter of minutes.”

You can also discover popular Spaces in the Docker Hub and run them locally with just a couple of commands.

To get started, read Effortlessly Build Machine Learning Apps with Hugging Face’s Docker Spaces. Or try Hugging Face Spaces now.

I’ve been a long-time user and avid fan of both Docker and Kubernetes, and have many happy memories of attending the Docker Meetups in London in the early 2010s. I closely watched as Docker revolutionized the developers’ container-building toolchain and Kubernetes became the natural target to deploy and run these containers at scale.

Today we’re happy to announce Telepresence for Docker, simplifying how teams develop and test on Kubernetes for faster app delivery. Docker and Ambassador Labs both help cloud-native developers to be super-productive, and we’re excited about this partnership to accelerate the developer experience on Kubernetes.

What exactly does this mean?

• When building with Kubernetes, you can now use Telepresence alongside the Docker toolchain you know and love.
• You can buy Telepresence directly from Docker, and log in to Ambassador Cloud using your Docker ID and credentials.
• You can get installation and product support from your current Docker support and services team.

Kubernetes development: Flexibility, scale, complexity

Kubernetes revolutionized the platform world, providing operational flexibility and scale for most organizations that have adopted it. But Kubernetes also introduces complexity when configuring local development environments.

We know you like building applications using your own local tools, where the feedback is instant, you can iterate quickly, and the environment you’re working in mirrors production. This combination increases velocity and reduces the time to successful deployment. But, you can face slow and painful development and troubleshooting obstacles when trying to integrate and test code into a real-world application running on Kubernetes. You end up having to replicate all of the services locally or remotely to test changes, which requires you to know about Kubernetes and the services built by others. The result, which we’ve seen at many organizations, is siloed teams, deferred deploying changes, and delayed organizational time to value.

Bridging remote environments with local development toolchains

Telepresence for Docker seamlessly bridges local dev machines to remote dev and staging Kubernetes clusters, so you don’t have to manage the complexity of Kubernetes, be a Kubernetes expert, or worry about consuming laptop resources when deploying large services locally.

The remote-to-local approach helps your teams to quickly collaborate and iterate on code locally while testing the effects of those code changes interactively within the full context of your distributed application. This way, you can work locally on services using the tools you know and love while also being connected to a remote Kubernetes cluster.

How does Telepresence for Docker work?

Telepresence for Docker works by running a traffic manager pod in Kubernetes and Telepresence client daemons on developer workstations. As shown in the following diagram, the traffic manager acts as a two-way network proxy that can intercept connections and route traffic between the cluster and containers running on developer machines.

Once you have connected your development machine to a remote Kubernetes cluster, you have several options for how the local containers can integrate with the cluster. These options are based on the concepts of intercepts, where Telepresence for Docker can re-route — or intercept — traffic destined to and from a remote service to your local machine. Intercepts enable you to interact with an application in a remote cluster and see the results from the local changes you made on an intercepted service.

Here’s how you can use intercepts:

• No intercepts: The most basic integration involves no intercepts at all, simply establishing a connection between the container and the cluster. This enables the container to access cluster resources, such as APIs and databases.
• Global intercepts: You can set up global intercepts for a service. This means all traffic for a service will be re-routed from Kubernetes to your local container.
• Personal intercepts: The more advanced alternative to global intercepts is personal intercepts. Personal intercepts let you define conditions for when a request should be routed to your local container. The conditions could be anything from only routing requests that include a specific HTTP header, to requests targeting a specific route of an API.

Benefits for platform teams: Reduce maintenance and cloud costs

On top of increasing the velocity of individual developers and development teams, Telepresence for Docker also enables platform engineers to maintain a separation of concerns (and provide appropriate guardrails). Platform engineers can define, configure, and manage shared remote clusters that multiple Telepresence for Docker users can interact within during their day-to-day development and testing workflows. Developers can easily intercept or selectively reroute remote traffic to the service on their local machine, and test (and share with stakeholders) how their current changes look and interact with remote dependencies.

Compared to static staging environments, this offers a simple way to connect local code into a shared dev environment and fuels easy, secure collaboration with your team or other stakeholders. Instead of provisioning cloud virtual machines for every developer, this approach offers a more cost-effective way to have a shared cloud development environment.

Get started with Telepresence for Docker today

We’re excited that the Docker and Ambassador Labs partnership brings Telepresence for Docker to the 12-million-strong (and growing) community of registered Docker developers. Telepresence for Docker is available now. Keep using the local tools and development workflow you know and love, but with faster feedback, easier collaboration, and reduced cloud environment costs.

You can quickly get started with your Docker ID, or contact us to learn more.

We recently announced the first Technical Preview of Docker+Wasm, a special build that makes it possible to run Wasm containers with Docker using the WasmEdge runtime. Starting from version 4.15, everyone can try out the features by activating the containerd image store experimental feature.

We didn’t want to stop there, however. Since October, we’ve been working with our partners to make running Wasm workloads with Docker easier and to support more runtimes.

Now we are excited to announce a new Technical Preview of Docker+Wasm with the following three new runtimes:

• spin from Fermyon
• slight from Deislabs
• wasmtime from Bytecode Alliance

All of these runtimes, including WasmEdge, use the runwasi library.

What is runwasi?

Runwasi is a multi-company effort to make a library in Rust that makes it easier to write containerd shims for Wasm workloads. Last December, the runwasi project was donated and moved to the Cloud Native Computing Foundation’s containerd organization in GitHub.

With a lot of work from people at Microsoft, Second State, Docker, and others, we now have enough features in runwasi to run Wasm containers with Docker or in a Kubernetes cluster. We still have a lot of work to do, but there are enough features for people to start testing.

If you would like to chat with us or other runwasi maintainers, join us on the CNCF’s #runwasi channel.

Get the update

Ready to dive in and try it for yourself? Great! Before you do, understand that this is a technical preview build of Docker Desktop, so things might not work as expected. Be sure to back up your containers and images before proceeding.

Download and install the appropriate version for your system, then activate the containerd image store (Settings > Features in development > Use containerd for pulling and storing images), and you’ll be ready to go.

![Features in development screen with "Use containerd" option selected.](https://www.docker.com/wp-content/uploads/2023/03/docker-desktop-containerd-enabled-1110x669.png "Features in development screen with "Use containerd" option selected. - docker desktop containerd enabled")

Figure 1: Docker Desktop beta features in development.

• Mac (Intel)
• Mac (Arm)
• Linux (deb, Intel)
• Linux (deb, Arm)
• Linux (rpm, Intel)
• Linux (Arch)
• Windows

Let’s take Wasm for a spin

The WasmEdge runtime is still present in Docker Desktop, so you can run:

$ docker run --rm --runtime=io.containerd.wasmedge.v1 --platform=wasi/wasm secondstate/rust-example-hello:latest Hello WasmEdge!

You can even run the same image with the wasmtime runtime:

$ docker run --rm --runtime=io.containerd.wasmtime.v1 --platform=wasi/wasm secondstate/rust-example-hello:latest Hello WasmEdge!

In the next example, we will deploy a Wasm workload to Docker Desktop’s Kubernetes cluster using the slight runtime. To begin, make sure to activate Kubernetes in Docker Desktop’s settings, then create an example.yaml file:

cat > example.yaml <<EOT apiVersion: apps/v1 kind: Deployment metadata: name: wasm-slight spec: replicas: 1 selector: matchLabels: app: wasm-slight template: metadata: labels: app: wasm-slight spec: runtimeClassName: wasmtime-slight-v1 containers: - name: hello-slight image: dockersamples/slight-rust-hello:latest command: ["/"] resources: requests: cpu: 10m memory: 10Mi limits: cpu: 500m memory: 128Mi

apiVersion: v1 kind: Service metadata: name: wasm-slight spec: type: LoadBalancer ports: - protocol: TCP port: 80 targetPort: 80 selector: app: wasm-slight EOT

Note the runtimeClassName, kubernetes will use this to select the right runtime for your application.

You can now run:

$ kubectl apply -f example.yaml

Once Kubernetes has downloaded the image and started the container, you should be able to curl it:

$ curl localhost/hello hello

You now have a Wasm container running locally in Kubernetes. How exciting! 🎉

Note: You can take this same yaml file and run it in AKS.

Now let’s see how we can use this to run Bartholomew. Bartholomew is a micro-CMS made by Fermyon that works with the spin runtime. You’ll need to clone this repository; it’s a slightly modified Bartholomew template.

The repository already contains a Dockerfile that you can use to build the Wasm container:

FROM scratch COPY . . ENTRYPOINT [ "/modules/bartholomew.wasm" ]

The Dockerfile copies all the contents of the repository to the image and defines the build bartholomew Wasm module as the entry point of the image.

$ cd docker-wasm-bartholomew $ docker build -t my-cms . [+] Building 0.0s (5/5) FINISHED => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 147B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load build context 0.0s => => transferring context: 2.84kB 0.0s => CACHED [1/1] COPY . . 0.0s => exporting to image 0.0s => => exporting layers 0.0s => => exporting manifest sha256:cf85929e5a30bea9d436d447e6f2f2e 0.0s => => exporting config sha256:0ce059f2fe907a91a671f37641f4c5d73 0.0s => => naming to docker.io/library/my-cms:latest 0.0s => => unpacking to docker.io/library/my-cms:latest 0.0s

You are now ready to run your first WebAssembly micro-CMS:

$ docker run --runtime=io.containerd.spin.v1 -p 3000:80 my-cms

If you go to http://localhost:3000, you should be able to see the Bartholomew landing page (Figure 2).

Figure 2: Bartholomew landing page.

We’d love your feedback

All of this work is fresh from the oven and relies on the containerd image store in Docker, which is an experimental feature we’ve been working on for almost a year now. The good news is that we already see how this hard work can benefit everyone by adding more features to Docker. We’re still working on it, so let us know what you need.

If you want to help us shape the future of WebAssembly with Docker, try it out, let us know what you think, and leave feedback on our public roadmap.

UPDATE: As of March 24, 2023, Docker has reversed its decision to sunset the “Docker Free Team” plan. Read our new blog post for details.

We apologize for how we communicated and executed sunsetting Docker “Free Team” subscriptions, which alarmed the open source community.

For those of you catching up, we recently emailed accounts that are members of Free Team organizations, to let them know that they will lose features unless they move to one of our supported free or paid offerings. This impacted less than 2% of our users. Note that this change does not affect Docker Personal, Docker Pro, Docker Team, or Docker Business accounts, Docker-Sponsored Open Source members, Docker Verified Publishers, or Docker Official Images.

The Docker Free Team subscription was deprecated in part because it was poorly targeted. In particular, it didn’t serve the open source audience as well as our recently updated Docker-Sponsored Open Source program, the latter offering benefits that exceed those of the deprecated Free Team plan.

We’d also like to clarify that public images will only be removed from Docker Hub if their maintainer decides to delete them. We’re sorry that our initial communications failed to make this clear.

We apologize again for the poor communication and execution surrounding this deprecation and promise to continue to listen to community feedback.

UPDATED March 24, 2023: Please visit our new FAQ. The old FAQ is accessible on the Wayback Machine.

Graph databases have become a popular solution for storing and querying complex relationships between data. As the amount of graph data grows and the need for high concurrency increases, a distributed graph database is essential to handle the scale.

Finding a distributed graph database that automatically shards the data, while allowing businesses to scale from small to trillion-edge-level without changing the underlying storage, architecture of the service, or application code, however, can be a challenge.

In this article, we’ll look at NebulaGraph, a modern, open source database to help organizations meet these challenges.

Meet NebulaGraph

NebulaGraph is a modern, open source, cloud-native graph database, designed to address the limitations of traditional graph databases, such as poor scalability, high latency, and low throughput. NebulaGraph is also highly scalable and flexible, with the ability to handle large-scale graph data ranging from small to trillion-edge-level.

NebulaGraph has built a thriving community of more than 1000 enterprise users since 2018, along with a rich ecosystem of tools and support. These benefits make it a cost-effective solution for organizations looking to build graph-based applications, as well as a great learning resource for developers and data scientists.

The NebulaGraph cloud-native database also offers Kubernetes Operators for easy deployment and management in cloud environments. This feature makes it a great choice for organizations looking to take advantage of the scalability and flexibility of cloud infrastructure.

Architecture of the NebulaGraph database

NebulaGraph consists of three services: the Graph Service, the Storage Service, and the Meta Service (Figure 1). The Graph Service, which consists of stateless processes (nebula-graphd), is responsible for graph queries. The Storage Service (nebula-storaged) is a distributed (Raft) storage layer that persistently stores the graph data. The Meta Service is responsible for managing user accounts, schema information, and Job management. With this design, NebulaGraph offers great scalability, high availability, cost-effectiveness, and extensibility.

Figure 1: Overview of NebulaGraph services.

Why NebulaGraph?

NebulaGraph is ideal for graph database needs because of its architecture and design, which allow for high performance, scalability, and cost-effectiveness. The architecture follows a separation of storage and computing architecture, which provides the following benefits:

• Automatic sharding: NebulaGraph automatically shards graph data, allowing businesses to scale from small to trillion-edge-level data volumes without having to change the underlying storage, architecture, or application code.
• High performance: With its optimized architecture and design, NebulaGraph provides high performance for complex graph queries and traversal operations.
• High availability: If part of the Graph Service fails, the data stored by the Storage Service remains intact.
• Flexibility: NebulaGraph supports property graphs and provides a powerful query language, called Nebula Graph Query Language (nGQL), which supports complex graph queries and traversal operations.
• Support for APIs: It provides a range of APIs and connectors that allow it to integrate with other tools and services in a distributed system.

Why run NebulaGraph as a Docker Extension?

In production environments, NebulaGraph can be deployed on Kubernetes or in the cloud, hiding the complexity of cluster management and maintenance from the user. However, for development, testing, and learning purposes, setting up a NebulaGraph cluster on a desktop or local environment can still be a challenging and costly process, especially for users who are not familiar with containers or command-line tools.

This is where the NebulaGraph Docker Extension comes in. It provides an elegant and easy-to-use solution for setting up a fully functional NebulaGraph cluster in just a few clicks, making it the perfect choice for developers, data scientists, and anyone looking to learn and experiment with NebulaGraph.

Getting started with NebulaGraph in Docker Desktop

Setting up

Prerequisites: Docker Desktop 4.10 or later.

Step 1: Enable Docker Extensions

You’ll need to enable Docker Extensions under the Settings tab in Docker Desktop. Within Docker Desktop, confirm that the Docker Extensions feature is enabled (Figure 2). Go to Settings > Extensions and select Enable Docker Extensions.

Figure 2: Enabling Docker Extensions within the Docker Desktop.

All Docker Extension resources are hidden by default, so, to ensure its visibility, go to Settings > Extensions and check the Show Docker Extensions system containers.

Step 2: Install the NebulaGraph Docker Extension

The NebulaGraph extension is available from the Extensions Marketplace in Docker Desktop and on Docker Hub. To get started, search for NebulaGraph in the Extensions Marketplace, then select Install (Figure 3).

Figure 3: Installing NebulaGraph from the Extensions Marketplace.

This step will download and install the latest version of the NebulaGraph Docker Extension from Docker Hub. You can see the installation process by clicking Details (Figure 4).

Figure 4: Installation progress.

Step 3: Waiting for the cluster to be up and running

After the extension is installed, for the first run, it normally takes fewer than 5 minutes for the cluster to be fully functional. While waiting, we can quickly go through the Home tab and Get Started tab to see details of NebulaGraph and NebulaGraph Studio, the WebGUI Utils.

We can also confirm whether it’s ready by observing the containers’ status from the Resources tab of the Extension as shown in Figure 5.

Figure 5: Checking the status of containers.

Step 4: Get started with NebulaGraph

After the cluster is healthy, we can follow the Get Started steps to log in to the NebulaGraph Studio, then load the initial dataset, and query the graph (Figure 6).

Figure 6: Logging in to NebulaGraph Studio.

Step 5: Learn more from the starter datasets

In a graph database, the focus is on the relationships between the data. With the starter datasets available in NebulaGraph Studio, you can get a better understanding of these relationships. All you need to do is click the Download button on each dataset card on the welcome page (Figure 7).

Figure 7: Starter datasets.

For example, in the demo_sns (social network) dataset, you can use the following query to find new friend recommendations by identifying second-degree friends with the most mutual friends:

Einstein

Figure 8: Query results shown in the Nebula console.

Instead of just displaying the query results, you can also return the entire pattern and easily gain insights. For example, in Figure 9, we can see LeBron James is on two mutual friend paths with Tim:

Figure 9: Graphing the query results.

Another example can be found in the demo_fraud_detection (graph of loan) dataset, where you can perform a 10-degree check for risky applicants, as shown in the following query:

MATCH p_=(p:`applicant`)-[*1..10]-(p2:`applicant`) WHERE id(p)=="p_200" AND p2.`applicant`.is_risky == "True" RETURN p_ LIMIT 100

The results shown in Figure 10 indicate that this applicant is suspected to be risky because of their connection to p_190.

Figure 10: Results of query showing fraud detection risk.

By exploring the relationships between data points, we can gain deeper insights into our data and make more informed decisions. Whether you are interested in finding new friends, detecting fraudulent activity, or any other use case, the starter datasets provide a valuable starting point.

We encourage you to download the datasets, experiment with different queries, and see what new insights you can uncover, then share with us in the NebulaGraph community.

Try NebulaGraph for yourself

To learn more about NebulaGraph, visit our website, documentation site, star our GitHub repo, or join our community chat.

Docker has introduced several new features and updates to its platform. One of the most notable updates is the release of Docker Scout 4.18, which adds new features, including vulnerability queries, process build attestations for images using BuildKit, and a "docker scout recommendations" command that suggests updates. Additionally, Docker Desktop 4.18 introduces image comparison, which allows users to document vulnerabilities and make improvements. Docker Container File Explorer has also been released, which allows users to view, edit and delete files within Docker containers.

In the latest update, Docker Compose has added a watch mode to monitor file edits and automatically sync them. Additionally, Docker is seeking feedback on updates for File Watch, which will offer improved functionality for Docker Compose. Docker has also introduced a CLI command, docker init, that quickly adds Docker to your environment. Finally, users can submit feedback, requests for framework/language support, and provide suggestions by navigating to the Images tab in Docker Desktop and selecting "Give Feedback."

The MEAN stack is a JavaScript-based three-tier web development platform that comprises MongoDB, ExpressJS, Angular, and NodeJS. The stack provides a high-performance environment for web application development. It runs on a wide range of platforms, including Linux, Mac, and Windows.

One of the advantages of the MEAN stack is that it can be containerized with Docker to scale efficiently. Containerizing the MEAN stack allows for scalability by dividing the application into small, independent, and self-sufficient components that can be scaled up or down according to demand.

To containerize a MEAN stack application, a Dockerfile needs to be created for the Angular frontend and Node.js backend, and they can be attached using Docker-compose to a specified network, such as events_net. The Docker build process uses Docker images, which store the changes made to the application.

The Docker Dashboard provides a user-friendly interface to view Docker containers and information about them. The Dashboard enables developers to monitor the status of the containers, and troubleshoot any issues that arise.

Overall, MEAN stack applications offer developers an efficient, easy-to-use web development platform which can be containerized to support scalability using Docker.

Docker is constantly improving with new features and capabilities that make it more accessible and powerful. The recent integration of WebAssembly, or Wasm, with Docker has opened up new possibilities for containerization and microservice management.

Developers can build Wasm modules using familiar languages like C++ or Rust and then package them into Docker images for easy deployment in Kubernetes clusters. The runwasi project, which enables running Wasm images within a Docker container, has even moved to the Cloud Native Computing Foundation.

One of the benefits of running a Docker container is the ability to create a micro-CMS for a website, simplifying the process of content management. Using Docker and a content management system like WordPress or Drupal, for example, a developer can package everything into a single container, making it easy to deploy and manage on any environment.

In addition to Wasm support, new features like the deployment YAML file make it easy to configure and manage Docker containers in a Kubernetes cluster. The YAML file defines the desired state of the deployment and can be used to automate many tasks like scaling, rolling updates, and health checking.

Another promising development in the Docker ecosystem is the integration with Telepresence and Ambassador Labs. This partnership provides a seamless way to bridge the gap between local and remote Kubernetes environments, allowing for shared development clusters and cost-effective collaboration. Telepresence also simplifies Kubernetes development by intercepting local traffic and routing it to the remote cluster for interaction, making it easier to test and deploy applications.

Overall, Docker is making AI and microservices more accessible to developers by democratizing the AI process and allowing for more cost-efficient development. The recent advancements in the Docker ecosystem, such as the integration of Wasm and the Telepresence partnership, offer exciting new possibilities for containerization and microservice management.

Docker had previously announced its decision to sunset the "Free Team" plan, which caused confusion and concern among the open-source community. However, Docker recently announced a reversal of this decision due to the poor targeting and lack of benefits for the open-source audience. As things stand, the Free Team Plan has been deprecated, which means that public images will not be removed from Docker Hub unless the maintainer decides to delete them. Docker has apologized for poor communication and promised to provide updates on the matter.

It is worth noting that the "Free Team" plan was designed to provide individual developers and small teams with free access to Docker's team features, including private repositories and unlimited collaborators. The Plan was intended to empower developers to build and deploy containerized applications without incurring exorbitant costs.

Docker Hub is a cloud-based registry that allows developers to store and share container images. The platform provides a convenient way for developers to host, distribute, and manage Docker images. The deprecation of the Free Team Plan means that the old FAQ may still be accessible, but individual accounts or paid organizations were not affected.

In conclusion, Docker's decision to sunset the Free Team Plan and its subsequent reversal have caused significant confusion among users, but Docker has promised to provide updates on the matter. Until then, the public images will only be removed from Docker Hub if the maintainer decides to delete them.

We often hear how indispensable Docker Compose is as a development tool. Running docker compose up offers a streamlined experience and scales from quickly standing up a PostgreSQL database to building 50+ services across multiple platforms.

And, although “building a Docker image” was previously considered a last step in release pipelines, it’s safe to say that containers have since become an essential part of our daily workflow. Still, concerns around slow builds and developer experience have often been a barrier towards the adoption of containers for local development.

We’ve come a long way, though. For starters, Docker Compose v2 now has deep integration with BuildKit, so you can use features like RUN cache mounts, SSH Agent forwarding, and efficient COPY with –link to speed up and simplify your builds. We’re also constantly making quality-of-life tweaks like enhanced progress reporting and improving consistency across the Docker CLI ecosystem.

As a result, more developers are running docker compose build && docker compose up to keep their running development environment up-to-date as they make code changes. In some cases, you can even use bind mounts combined with a framework that supports hot reload to avoid the need for an image rebuild, but this approach often comes with its own set of caveats and limitations.

An early look at the watch command

Starting with Compose v2.17, we’re excited to share an early look at the new development-specific configuration in Compose YAML as well as an experimental file watch command (Figure 1) that will automatically update your running Compose services as you edit and save your code.

This preview is brought to you in no small part by Compose developer Nicolas De Loof (in addition to more than 10 bugfixes in this release alone).

![Screenshot of Docker compose command line showing the new "watch" command.](https://www.docker.com/wp-content/uploads/2023/04/Docker-watch-command-F1.gif "Screenshot of Docker compose command line showing the new "watch" command. - Docker watch command F1")

Figure 1: Preview of the new watch command.

An optional new section, x-develop, can be added to a Compose service to configure options specific to your project’s daily flow. In this release, the only available option is watch, which allows you to define file or directory paths to monitor on your computer and a corresponding action to take inside the service container.

Currently, there are two possible actions:

• sync — Copy changed files matching the pattern into the running service container(s).
• rebuild — Trigger an image build and recreate the running service container(s).

services: web: build: . x-develop: watch: - action: sync path: ./web target: /src/web - action: rebuild path: package.json

In the preceding example, whenever a source file in the web/ directory is changed, Compose will copy the file to the corresponding location under /src/web inside the container. Because Webpack supports Hot Module Reload, the changes are automatically detected and applied.

Unlike source code files, adding a new dependency cannot be done on the fly, so whenever package.json is changed, Compose will rebuild the image and recreate the web service container.

Behind the scenes, the file watching code shares its core with Tilt. The intricacies and surprises of file watching have always been near and dear to the Tilt team’s hearts, and, as Dockhands, the geeking out has continued.

We are going to continue to build out the experience while gated behind the new docker compose alpha command and x-develop Compose YAML section. This approach will allow us to respond to community feedback early in the development process while still providing a clear path to stabilization as part of the Compose Spec.

Docker Compose powers countless workflows today, and its lightweight approach to containerized development is not going anywhere — it’s just learning a few new tricks.

Try it out

Follow the instructions at dockersamples/avatars to quickly run a small demo app, as follows:

git clone https://github.com/dockersamples/avatars.git cd avatars docker compose up -d docker compose alpha watch

open http://localhost:5735 in your browser

If you try it out on your own project, you can comment on the proposed specification on GitHub issue #253 in the compose-spec repository.

In the world of software development, using Docker for local development purposes has become essential nowadays. It provides an environment where developers can work in a controlled and isolated space while allowing them to build and test their applications with ease. One command that has made the development process much easier is docker compose up. This command streamlines the process from small tasks to building multiple services, making the build process simpler and quicker.

The latest release of Compose (v2.17) has introduced a new feature aimed at improving the development process - the watch command. This feature allows developers to monitor file and directory changes and perform actions in corresponding service containers. For instance, Compose automatically copies changed source files found in the web/ directory to the container's /src/web location, with Hot Module Reload support. Moreover, changes to package.json triggers Compose to rebuild the image with the latest modules.

As a response to community feedback, Docker also introduced a new docker compose alpha command and x-develop YAML section. This new development allows early feedback from the community while stabilizing the Compose specification.

{{!!automatisation-de-flux-rss-avec-un-llm/Automatisation des flux RSS avec un LLM 88e22c51c8114b2eab4d0f51ef22d9d6.html!!}}

Docker has revolutionized the way developers build, package, and deploy their applications. Docker containers provide a lightweight, portable, and consistent runtime environment that can run on any infrastructure. And now, the Docker team has developed docker init, a new command-line interface (CLI) command introduced as a beta feature that simplifies the process of adding Docker to a project (Figure 1).

Note: Docker Init should not be confused with the internally -used docker-init executable, which is invoked by Docker when utilizing the –init flag with the docker run command.

Figure 1: With one command, all required Docker files are created and added to your project.

Create assets automatically

The new  docker init command automates the creation of necessary Docker assets, such as Dockerfiles, Compose files, and .dockerignore files, based on the characteristics of the project. By executing the docker init command, developers can quickly containerize their projects. Docker init is a valuable tool for developers who want to experiment with Docker, learn about containerization, or integrate Docker into their existing projects.

To use docker init, developers need to upgrade to the version 4.19.0 or later of Docker Desktop and execute the command in the target project folder. Docker init will detect the project definitions, and it will automatically generate the necessary files to run the project in Docker.

The current Beta release of docker init supports Go, Node, and Python, and our development team is actively working to extend support for additional languages and frameworks, including Java, Rust, and .NET. If there is a language or stack that you would like to see added or if you have other feedback about docker init, let us know through our Google form.

In conclusion, docker init is a valuable tool for developers who want to simplify the process of adding Docker support to their projects. It automates the creation of necessary Docker assets  and can help standardize the creation of Docker assets across different projects. By enabling developers to focus on developing their applications and reducing the risk of errors and inconsistencies, Docker init can help accelerate the adoption of Docker and containerization.

See Docker Init in action

To see docker init in action, check out the following overview video by Francesco Ciulla, which demonstrates building the required Docker assets to your project.

Check out the documentation to learn more.

If you’re using a Docker-based development approach, you’re already well on your way toward creating cloud-native software. Containerizing your software ensures that you have all the system-level dependencies, language-specific requirements, and application configurations managed in a containerized way, bringing you closer to the environment in which your code will eventually run.

In complex systems, however, you may need to connect your code with several auxiliary services, such as databases, storage volumes, APIs, caching layers, message brokers, and others. In modern Kubernetes-based architectures, you also have to deal with service meshes and cloud-native deployment patterns, such as probes, configuration, and structural and behavioral patterns.

Kubernetes offers a uniform interface for orchestrating scalable, resilient, and services-based applications. However, its complexity can be overwhelming, especially for developers without extensive experience setting up Kubernetes clusters. That’s where Gefyra comes in, making it easier for developers to work with Kubernetes and improve the process of creating secure, reliable, and scalable software.

What is Gefyra?

Gefyra, named after the Greek word for “bridge,” is a comprehensive toolkit that facilitates Docker-based development with Kubernetes. If you plan to use Kubernetes as your production platform, it’s essential to work with the same environment during development. This approach ensures that you have the highest possible “dev/prod-parity,” minimizing friction when transitioning from development to production.

Gefyra is an open source project that provides docker run on steroids. It allows you to connect your local Docker with any Kubernetes cluster and run a container locally that behaves as if it would run in the cluster. You can write code locally in your favorite code editor using the tools you love.

Additionally, Gefyra does not require you to build a container image from your code changes, push the image to a registry, or trigger a restart in the cluster. Instead, it saves you from this tedious cycle by connecting your local code right into the cluster without any changes to your existing Dockerfile. This approach is useful not only for new code but also when introspecting existing code with a debugger that you can attach to a running container. That makes Gefyra a productivity superstar for any Kubernetes-based development work.

How does Gefyra work?

Gefyra installs several cluster-side components that enable it to control the local development machine and the development cluster. These components include a tunnel between the local development machine and the Kubernetes cluster, a local DNS resolver that behaves like the cluster DNS, and sophisticated IP routing mechanisms. Gefyra uses popular open source technologies, such as Docker, WireGuard, CoreDNS, Nginx, and Rsync, to build on top of these components.

The local development setup involves running a container instance of the application on the developer machine, with a sidecar container called Cargo that acts as a network gateway and provides a CoreDNS server that forwards all requests to the cluster (Figure 1). Cargo encrypts all the passing traffic with WireGuard using ad hoc connection secrets. Developers can use their existing tooling, including their favorite code editor and debuggers, to develop their applications.

Figure 1: Local development setup.

Gefyra manages two ends of a WireGuard connection and automatically establishes a VPN tunnel between the developer and the cluster, making the connection robust and fast without stressing the Kubernetes API server (Figure 2). Additionally, the client side of Gefyra manages a local Docker network with a VPN endpoint, allowing the container to join the VPN that directs all traffic into the cluster.

Figure 2: Connecting developer machine and cluster.

Gefyra also allows bridging existing traffic from the cluster to the local container, enabling developers to test their code with real-world requests from the cluster and collaborate on changes in a team. The local container instance remains connected to auxiliary services and resources in the cluster while receiving requests from other Pods, Services, or the Ingress. This setup eliminates the need for building container images in a continuous integration pipeline and rolling out a cluster update for simple changes.

Why run Gefyra as a Docker Extension?

Gefyra’s core functionality is contained in a Python library available in its repository. The CLI that comes with the project has a long list of arguments that may be overwhelming for some users. To make it more accessible, Gefyra developed the Docker Desktop extension, which is easy for developers to use without having to delve into the intricacies of Gefyra.

The Gefyra extension for Docker Desktop enables developers to work with a variety of Kubernetes clusters, including the built-in Kubernetes cluster, local providers such as Minikube, K3d, or Kind, Getdeck Beiboot, or any remote clusters. Let’s get started.

Installing the Gefyra Docker Desktop

Prerequisites: Docker Desktop 4.8 or later.

Step 1: Initial setup

In Docker Desktop, confirm that the Docker Extensions feature is enabled. (Docker Extensions should be enabled by default.) In Settings | Extensions select the Enable Docker Extensions box (Figure 3).

![ Screenshot showing Docker Desktop interface with "Enable Docker Extensions" selected.](https://www.docker.com/wp-content/uploads/2023/06/F3-Gefyra-Enable-Docker-Extensions-1110x628.png "Screenshot showing Docker Desktop interface with "Enable Docker Extensions" selected. - F3 Gefyra Enable Docker Extensions")

Figure 3: Enable Docker Extensions.

You must also enable Kubernetes under Settings (Figure 4).

![Screenshot of Docker Desktop with "Enable Kubernetes" and "Show system containers (advanced)" selected.](https://www.docker.com/wp-content/uploads/2023/06/F4-Gefyra-Enable-Kubernetes-1110x594.png "Screenshot of Docker Desktop with "Enable Kubernetes" and "Show system containers (advanced)" selected. - F4 Gefyra Enable Kubernetes")

Figure 4: Enable Kubernetes.

Gefyra is in the Docker Extensions Marketplace. In the following instructions, we’ll install Gefyra in Docker Desktop.

Step 2: Add the Gefyra extension

Open Docker Desktop and select Add Extensions to find the Gefyra extension in the Extensions Marketplace (Figure 5).

![Screenshot showing search for "Gefyra" in Docker Extensions Marketplace.](https://www.docker.com/wp-content/uploads/2023/06/F5-Gefyra-Extensions-Marketplace-1110x635.png "Screenshot showing search for "Gefyra" in Docker Extensions Marketplace. - F5 Gefyra Extensions Marketplace")

Figure 5: Locate Gefyra in the Docker Extensions Marketplace.

Once Gefyra is installed, you can open the extension and find the start screen of Gefyra that lists all containers that are connected to a Kubernetes cluster. Of course, this section is empty on a fresh install.
To launch a local container with Gefyra, just like with Docker, you need to click on the Run Container button at the top right (Figure 6).

Figure 6: Gefyra start screen.

The next steps will vary based on whether you’re working with a local or remote Kubernetes cluster. If you’re using a local cluster, simply select the matching kubeconfig file and optionally set the context (Figure 7).

For remote clusters, you may need to manually specify additional parameters. Don’t worry if you’re unsure how to do this, as the next section will provide a detailed example for you to follow along with.

![Screenshot of Gefyra interface showing blue "Choose Kubeconfig" button.](https://www.docker.com/wp-content/uploads/2023/06/F7-Select-Kubeconfig-1110x628.png "Screenshot of Gefyra interface showing blue "Choose Kubeconfig" button. - F7 Select Kubeconfig")

Figure 7: Selecting Kubeconfig.

The Kubernetes demo workloads

The following example showcases how Gefyra leverages the Kubernetes functionality included in Docker Desktop to create a development environment for a simple application that consists of two services — a backend and a frontend (Figure 8).

Both services are implemented as Python processes, and the frontend service uses a color property obtained from the backend to generate an HTML document. Communication between the two services is established via HTTP, with the backend address being passed to the frontend as an environment variable.

Figure 8: Frontend and backend services.

The Gefyra team has created a repository for the Kubernetes demo workloads, which can be found on GitHub.

If you prefer to watch a video explaining what’s covered in this tutorial, check out this video on YouTube.

Prerequisite

Ensure that the current Kubernetes context is switched to Docker Desktop. This step allows the user to interact with the Kubernetes cluster and deploy applications to it using kubectl.

kubectl config current-context docker-desktop

Clone the repository

The next step is to clone the repository:

git clone https://github.com/gefyrahq/gefyra-demos

Applying the workload

The following YAML file sets up a simple two-tier app consisting of a backend service and a frontend service with communication between the two services established via the SVC_URL environment variable passed to the frontend container.

It defines two pods, named backend and frontend, and two services, named backend and frontend, respectively. The backend pod is defined with a container that runs the quay.io/gefyra/gefyra-demo-backend image on port 5002. The frontend pod is defined with a container that runs the quay.io/gefyra/gefyra-demo-frontend image on port 5003. The frontend container also includes an environment variable named SVC_URL, which is set to the value backend.default.svc.cluster.local:5002.

The backend service is defined to select the backend pod using the app: backend label, and expose port 5002. The frontend service is defined to select the frontend pod using the app: frontend label, and expose port 80 as a load balancer, which routes traffic to port 5003 of the frontend container.

/gefyra-demos/kcd-munich> kubectl apply -f manifests/demo.yaml pod/backend created pod/frontend created service/backend created service/frontend created

Let’s watch the workload getting ready:

kubectl get pods NAME READY STATUS RESTARTS AGE backend 1/1 Running 0 2m6s frontend 1/1 Running 0 2m6s

After ensuring that the backend and frontend pods have finished initializing (check for the READY column in the output), you can access the application by navigating to http://localhost in your web browser. This URL is served from the Kubernetes environment of Docker Desktop.

Upon loading the page, you will see the application’s output displayed in your browser. Although the output may not be visually stunning, it is functional and should provide the necessary functionality for your needs.

![Blue bar displaying "Hello World" in black text.](https://www.docker.com/wp-content/uploads/2023/06/hello-world-e1682692682514.png "Blue bar displaying "Hello World" in black text. - hello world e1682692682514")

Now, let’s explore how we can correct or adjust the color of the output generated by the frontend component.

Using Gefyra “Run Container” with the frontend process

In the first part of this section, you will see how to execute a frontend process on your local machine that is associated with a resource based on the Kubernetes cluster: the backend API. This can be anything ranging from a database to a message broker or any other service utilized in the architecture.

Kick off a local container with Run Container from the Gefyra start screen (Figure 9).

![Screenshot of Gefyra interface showing blue "Run container" button.](https://www.docker.com/wp-content/uploads/2023/06/F9-Run-local-container-1110x626.png "Screenshot of Gefyra interface showing blue "Run container" button. - F9 Run local container")

Figure 9: Run a local container.

Once you’ve entered the first step of this process, you will find the kubeconfig` and context to be set automatically. That’s a lifesaver if you don’t know where to find the default kubeconfig on your host.

Just hit the Next button and proceed with the container settings (Figure 10).

![Screenshot of Gefyra interface showing the "Set Kubernetes Settings" step.](https://www.docker.com/wp-content/uploads/2023/06/F10-Gefyra-containers-1110x628.png "Screenshot of Gefyra interface showing the "Set Kubernetes Settings" step. - F10 Gefyra containers")

Figure 10: Container settings.

In the Container Settings step, you can configure the Kubernetes-related parameters for your local container. In this example, everything happens in the default Kubernetes namespace. Select it in the first drop-down input (Figure 11).

In the drop-down input below Image, you can specify the image to run locally. Note that it lists all images that are being used in the selected namespace (from the Namespace selector). Isn’t that convenient? You don’t need to worry about the images being used in the cluster or find them yourself. Instead, you get a suggestion to work with the image at hand, as we want to do in this example (Figure 12). You could still specify any arbitrary images if you like, for example, a completely new image you just built on your machine.

![Screenshot of Gefyra interface showing "Select a Workload" drop-down menu under Container Settings.](https://www.docker.com/wp-content/uploads/2023/06/F11-Gefyra-namespace-1110x634.png "Screenshot of Gefyra interface showing "Select a Workload" drop-down menu under Container Settings. - F11 Gefyra namespace")

Figure 11: Select namespace and workload.

Figure 12: Select image to run.

To copy the environment of the frontend container running in the cluster, you will need to select pod/frontend from the Copy Environment From selector (Figure 13). This step is important because you need the backend service address, which is passed to the pod in the cluster using an environment variable.

Finally, for the upper part of the container settings, you need to overwrite the following run command of the container image to enable code reloading:

poetry run flask --app app debug run --port 5002 --host 0.0.0.0

![Screenshot of Gefyra interface showing selection of “pod/frontend” under “Copy Environment From."](https://www.docker.com/wp-content/uploads/2023/06/F13-Copy-frontend-1110x628.png "Screenshot of Gefyra interface showing selection of “pod/frontend” under “Copy Environment From." - F13 Copy frontend")

Figure 13: Copy environment of frontend container.

Let’s start the container process on port 5002 and expose this port on the local machine. In addition, let’s mount the code directory (/gefyra-demos/kcd-munich/frontend) to make code changes immediately visible. That’s it for now. A click on the Run button starts the process.

Figure 14: Installing Gefyra components.

It takes a few seconds to install Gefyra’s cluster-side components, prepare the local networking part, and pull the container image to start locally (Figure 14). Once this is ready, you will get redirected to the native container view of Docker Desktop from this container (Figure 15).

Figure 15: Log view.

You can look around in the container using the Terminal tab (Figure 16). Type in the env command in the shell, and you will see all the environment variables coming with Kubernetes.

Figure 16: Terminal view.

We’re particularly interested in the SVC_URL variable that points the frontend to the backend process, which is, of course, still running in the cluster. Now, when browsing to the URL http://localhost:5002, you will get a slightly different output:

![Blue bar displaying "Hello KCD" in black text](https://www.docker.com/wp-content/uploads/2023/06/Hello-KCD.png "Blue bar displaying "Hello KCD" in black text - Hello KCD")

Why is that? Let’s look at the code that we already mounted into the local container, specifically the app.py that runs a Flask server (Figure 17).

Figure 17: App.py code.

The last line of the code in the Gefyra example displays the text Hello KCD!, and any changes made to this code are immediately updated in the local container. This feature is noteworthy because developers can freely modify the code and see the changes reflected in real-time without having to rebuild or redeploy the container.

Line 12 of the code in the Gefyra example sends a request to a service URL, which is stored in the variable SVC. The value of SVC is read from an environment variable named SVC_URL, which is copied from the pod in the Kubernetes cluster. The URL, backend.default.svc.cluster.local:5002, is a fully qualified domain name (FQDN) that points to a Kubernetes service object and a port.

These URLs are commonly used by applications in Kubernetes to communicate with each other. The local container process is capable of sending requests to services running in Kubernetes using the native connection parameters, without the need for developers to make any changes, which may seem like magic at times.

In most development scenarios, the capabilities of Gefyra we just discussed are sufficient. In other words, you can use Gefyra to run a local container that can communicate with resources in the Kubernetes cluster, and you can access the app on a local port. However, what if you need to modify the backend while the frontend is still running in Kubernetes? This is where the “bridge” feature of Gefyra comes in, which we will explore next.

Gefyra “bridge” with the backend process

We could choose to run the frontend process locally and connect it to the backend process running in Kubernetes through a bridge. However, this approach may not always be necessary or desirable, especially for backend developers who may not be interested in the frontend. In this case, it may be more convenient to leave the frontend running in the cluster and stop the local instance by selecting the stop button in Docker Desktop’s container view.

First of all, we have to run a local instance of the backend service. It’s the same as with the frontend, but this time with the backend container image (Figure 18).

![Screenshot of Gefyra interface showing "pod/backend" setup.](https://www.docker.com/wp-content/uploads/2023/06/F18-Gefyra-Run-backend-1110x628.png "Screenshot of Gefyra interface showing "pod/backend" setup. - F18 Gefyra Run backend")

Figure 18: Running a backend container image.

Compared to the frontend example from above, you can run the backend container image (quay.io/gefyra/gefyra-demo-backend:latest), which is suggested by the drop-down selector. This time we need to copy the environment from the backend pod running in Kubernetes. Note that the volume mount is now set to the code of the backend service to make it work.

After starting the container, you can check http://localhost:5002/color, which serves the backend API response. Looking at the app.py of the backend service shows the source of this response. In line 8, this app returns a JSON response with the color property set to green (Figure 19).

![Screenshot showing app.py code with "color" set to "green".](https://www.docker.com/wp-content/uploads/2023/06/F19-Gefyra-Check-color.png "Screenshot showing app.py code with "color" set to "green". - F19 Gefyra Check color")

Figure 19: Checking the color.

At this point, keep in mind that we’re only running a local instance of the backend service. This time, a connection to a Kubernetes-based resource is not needed as this container runs without any external dependency.

The idea is to make the frontend process that serves from the Kubernetes cluster on http://localhost (still blue) pick up our backend information to render its output. That’s done using Gefyra’s bridge feature. In the next step, we will overlay the backend process running in the cluster with our local container instance so that the local code becomes effective in the cluster.

Getting back to the Gefyra container list on the start screen, you can find the Bridge column on each locally running container (Figure 20). Once you click this button, you can create a bridge of your local container into the cluster.

![Screenshot of Gefyra interface showing "Bridge" column on far right.](https://www.docker.com/wp-content/uploads/2023/06/F20-Gefyra-Bridge-1110x628.png "Screenshot of Gefyra interface showing "Bridge" column on far right. - F20 Gefyra Bridge")

Figure 20: The Bridge column is visible on the far right.

In the next dialog, we need to enter the bridge configuration (Figure 21).

Figure 21: Enter the bridge configuration.

Let’s set the “Target” for the bridge to the backend pod, which is currently serving the frontend process in the cluster, and set a timeout for the bridge to 60 seconds. We also need to map the port of the proxy running in the cluster with the local instance.

If your local container is configured to listen on a different port from the cluster, you can specify the mapping here (Figure 22). In this example, the service is running on port 5003 in both the cluster and on the local machine, so we need to map that port. After clicking the Bridge button, it takes a few seconds to return to the container list on Gefyra’s start view.

Figure 22: Specify port mapping.

Observe the change in the icon of the Bridge button, which now depicts a stop symbol (Figure 23). This means the bridge function is now operational and can be terminated by simply clicking this button again.

Figure 23: The Bridge column showing a stop symbol.

At this point, the local code is able to handle requests from the frontend process in the cluster by using the URL stored in the SVC_URL variable, without making any changes to the frontend process itself. To confirm this, you can open http://localhost in your browser (which is served from the Kubernetes of Docker Desktop) and check that the output is now green. This is because the local code is returning the value green for the color property. You can change this value to any valid one in your IDE, and it will be immediately reflected in the cluster. This is the amazing power of this tool.

Remember to release the bridge of your container once you are finished making changes to your backend. This will reset the cluster to its original state, and the frontend will display the original “beautiful” blue H1 again. This approach allows us to intercept containers running in Kubernetes with our local code without modifying the Kubernetes cluster itself. That’s because we did not make any changes to the Kubernetes cluster itself. Instead, we kind of intercepted containers running in Kubernetes with our local code and released that intercept afterwards.

Conclusion

Gefyra is an easy-to-use Docker Desktop extension that connects with Kubernetes to improve development workflows and team collaboration. It lets you run containers as usual while being connected with Kubernetes, thereby saving time and ensuring high dev/prod parity.

The Blueshoe development team would appreciate a star on GitHub and welcomes you to join their Discord community for more information.

About the Author

Michael Schilonka is a strong believer that Kubernetes can be a software development platform, too. He is the co-founder and managing director of the Munich-based agency Blueshoe and the technical lead of Gefyra and Getdeck. He talks about Kubernetes in general and how they are using Kubernetes for development. Follow him on LinkedIn to stay connected.

Docker Desktop release 4.19 is now available. In this post, we highlight features added to Docker Desktop in the past month, including performance enhancements, new language support, and a Moby update.

5x faster container-to-host networking on macOS

In Docker Desktop 4.19, we’ve made container-to-host networking performance 5x faster on macOS by replacing vpnkit with the TCP/IP stack from the gVisor project.

Many users work on projects that have containers communicating with a server outside their local Docker network. One example of this would be workloads that download packages from the internet via npm install or apt-get. This performance improvement should help a lot in these cases.

Over the next month, we’ll keep track of the stability of this new networking stack. If you notice any issues, you can revert to using the legacy vpnkit networking stack by setting "networkType":"vpnkit" in Docker Desktop’s settings.json config file.

Docker Init (Beta): Support for Node and Python

In our 4.18 release, we introduced docker init, a CLI command in Beta that helps you easily add Docker to any of your projects by creating the required assets for you. In the 4.19 release, we’re happy to add to this and share that the feature now includes support for Python and Node.js.

You can try docker init with Python and Node.js by updating to the latest version of Docker Desktop (4.19) and typing docker init in the command line while inside a target project folder.

The Docker team is working on adding more languages and frameworks for this command, including Java, Rust, and .Net. Let us know if you would like us to support a specific language or framework. We welcome any feedback you may have as we continue to develop and improve Docker Init (Beta).

Docker Scout (Early Access)

With Docker Desktop release 4.19, we’ve made it easier to view Docker Scout data for all of your images directly in Docker Desktop. Whether you’re using an image stored locally in Docker Desktop or a remote image from Docker Hub, you can see all that data without leaving Docker Desktop.

A nudge toward Compose v2

Compose v1 has reached end-of-life and will no longer be bundled with Docker Desktop after June 2023.

In preparation, a new warning will be shown in the terminal when running Compose v1 commands. Set the COMPOSE_V1_EOL_SILENT=1 environment variable to suppress this message.

You can upgrade by enabling Use Compose v2 in the Docker Desktop settings. When active, Docker Desktop aliases docker-compose to Compose v2 and supports the recommended docker compose syntax.

Moby 23

We updated the Docker Engine and the CLI to Moby 23.0,  where we are upstreaming open source internal developments such as the containerd integration and Wasm support, which will ship with Moby 24.0. Moby 23.0 includes additional enhancements such as the --format=json shorthand variant of --format=“{{ json . }}” and support of relative source paths to the run command in the -v/--volume and -m/--mount flags. You can read more about Moby 23.0 in the release notes.

Conclusion

We love hearing your feedback. Please leave any feedback on our public GitHub roadmap and let us know what else you’d like to see. Check out the Docker Desktop 4.19 release notes for a full breakdown of what’s new in the latest release.

Coming up with a title that explains the full story here was difficult, so I'm going to try to explain quickly.

Yesterday, Docker sent an email to any Docker Hub user who had created an "organisation", telling them their account will be deleted including all images, if they do not upgrade to a paid team plan. The email contained a link to a tersely written PDF (since, silently edited) which was missing many important details which caused significant anxiety and additional work for open source maintainers.

As far as we know, this only affects organisation accounts that are often used by open source communities. There was no change to personal accounts. Free personal accounts have a a 6 month retention period.

Why is this a problem?

1. Paid team plans cost 420 USD per year (paid monthly)
2. Many open source projects including ones I maintain have published images to the Docker Hub for years
3. Docker's Open Source program is hostile and out of touch

Why should you listen to me?

I was one of the biggest advocates around for Docker, speaking at their events, contributing to their projects and being a loyal member of their voluntary influencer program "Docker Captains". I have written dozens if not hundreds of articles and code samples on Docker as a technology.

I'm not one of those people who think that all software and services should be free. I pay for a personal account, not because I publish images there anymore, but because I need to pull images like the base image for Go, or Node.js as part of my daily open source work.

When one of our OpenFaaS customers grumbled about paying for Docker Desktop, and wanted to spend several weeks trying to get Podman or Rancher Desktop working, I had to bite my tongue. If you're using a Mac or a Windows machine, it's worth paying for in my opinion. But that is a different matter.

Having known Docker's new CTO personally for a very long time, I was surprised how out of touch the communication was.

I'm not the only one, you can read the reactions on Twitter (including many quote tweets) and on Hacker News.

Let's go over each point, then explore options for moving forward with alternatives and resolutions.

The issues

1. The cost of an organisation that hosts public images has risen from 0 USD / year to 420 USD / year (paid monthly). Many open source projects receive little to no funding. I would understand if Docker wanted to clamp down on private repos, because what open source repository needs them? I would understand if they applied this to new organisations.

2. Many open source projects have published images to the Docker Hub in this way for years, openfaas as far back as 2016. Anyone could cybersquat the image and publish malicious content. The OpenFaaS project now publishes its free Community Edition images to GitHub's Container Registry, but we still see thousands of pulls of old images from the Docker Hub. Docker is holding us hostage here, if we don't pay up, systems will break for many free users.

3. Docker has a hostile and out of touch definition of what is allowable for their Open Source program. It rules out anything other than spare-time projects, or projects that have been wholly donated to an open-source foundation.

"Not have a pathway to commercialization. Your organization must not seek to make a profit through services or by charging for higher tiers. Accepting donations to sustain your efforts is permissible."

This language has been softened since the initial email, I assume in an attempt to reduce the backlash.

Open Source has a funding problem, and Docker was born in Open Source. We the community were their king makers, and now that they're turning over significant revenue, they are only too ready to forget their roots.

The workarounds

Docker's CTO commented informally on Twitter that they will shut down accounts that do not pay up, and not allow anyone else to take over the name. I'd like to see that published in writing, as a written commitment.

In an ideal world, these accounts would continue to be attached to the user account, so that if for some reason we wanted to pay for them, we'd have access to restore them.

Squatting and the effects of malware and poison images is my primary concern here. For many projects I maintain, we already switched to publishing open source packages to GitHub's Container Registry. Why? Because Docker enforced unrealistic rate limits that means any and every user who downloads content from their Docker Hub requires a paid subscription - whether personal or corporate. I pay for one so that I can download images like Prometheus, NATS, Go, Python and Node.

Maybe you qualify for the "open source" program?

If the project you maintain is owned by a foundation like the CNCF or Apache Foundation, you may simply be able to apply to Docker's program. However if you are independent, and have any source of funding or any way to financial sustainability, I'll paraphrase Docker's leadership: "sucks to be you."

Let's take an example? The curl project maintained by Daniel Stenberg - something that is installed on every Mac and Linux computer and certainly used by Docker. Daniel has a consulting company and does custom development. Such a core piece of Internet infrastructure seems to be disqualified.

There is an open-source exemption, but it's very strict (absolutely no "pathway to commercialization" - no services, no sponsors, no paid addons, and no pathway to ever do so later) and they're apparently taking >1 year to process applications anyway.

— Tim Perry (@pimterry) March 14, 2023

Cybersquat before a bad actor can

If you are able to completely delete your organisation, then you could re-create it as a free personal account. That should be enough to reserve the name to prevent hostile take-over. Has Docker forgotten Remember leftpad?

This is unlikely that large projects can simply delete their organisation and all its images.

If that's the case, and you can tolerate some downtime, you could try the following:

• Create a new personal user account
• Mirror all images and tags required to the new user account
• Delete the organisation
• Rename the personal user account to the name of the organisation

Start publishing images to GitHub

GitHub's Container Registry offers free storage for public images. It doesn't require service accounts or long-lived tokens to be stored as secrets in CI, because it can mint a short-lived token to access ghcr.io already.

Want to see a full example of this?

We covered it on the actuated blog: The efficient way to publish multi-arch containers from GitHub Actions

If you already have an image on GitHub and want to start publishing new tags there using GitHub's built-in GITHUB_TOKEN, you'll need to go to the Package and edit its write permissions. Add the repository with "Write" access.

Make sure you do not miss the "permissions" section of the workflow file.

How to set up write access for an existing repository with GITHUB_TOKEN

Migrate your existing images

The crane tool by Google's open source office is able to mirror images in a much more efficient way than running docker pull, tag and push. The pull, tag and push approach also doesn't work with multi-arch images.

Here's an example command to list tags for an image:

crane ls ghcr.io/openfaas/gateway | tail -n 5

0.26.1
c26ec5221e453071216f5e15c3409168446fd563
0.26.2
a128df471f406690b1021a32317340b29689c315
0.26.3

The crane cp command doesn't require a local docker daemon and copies directly from one registry to another:

crane cp docker.io/openfaas/gateway:0.26.3 ghcr.io/openfaas/gateway:0.26.3

On Twitter, a full-time employee on the CNCF's Harbor project also explained that it has a "mirroring" capability.

Wrapping up

Many open source projects moved away from the Docker Hub already when they started rate-limiting pulls of public open-source images like Go, Prometheus and NATS. I myself still pay Docker for an account, the only reason I have it is to be able to pull those images.

I am not against Docker making money, I already pay them money and have encouraged customers to do the same. My issue is with the poor messaging, the deliberate anxiety that they've created for many of their most loyal and supportive community users and their hypocritical view of Open Source sustainability.

If you're using GitHub Actions, then it's easy to publish images to GHCR.io - you can use the example for the inlets-operator I shared.

But what about GitHub's own reliability?

I was talking to a customer for actuated only yesterday. They were happy with our product and service, but in their first week of a PoC saw downtime due to GitHub's increasing number of outages and incidents.

We can only hope that whatever has caused issues almost every day since the start of the year is going to be addressed by leadership.

Is GitHub perfect?

I would have never predicted the way that Docker changed since its rebirth - from the darling of the open source community, on every developer's laptop, to where we are today. So with the recent developments on GitHub like Actions and GHCR only getting better, with them being acquired by Microsoft - it's tempting to believe that they're infallible and wouldn't make a decision that could hurt maintainers. All businesses need to work on a profit and loss basis. A prime example of how GitHub also hurt open source developers was when it cancelled all Sponsorships to maintainers that were paid over PayPal. This was done at very short notice, and it hit my own open source work very hard - made even worse by the global downturn.

Are there other registries that are free for open source projects?

I didn't want to state the obvious in this article, but so many people contacted me that I'm going to do it. Yes - we all know that GitLab and Quay also offer free hosting. Yes we know that you can host your own registry. There may be good intentions behind these messages, but they miss point of the article.

What if GitHub "does a Docker on us"?

What if GitHub starts charging for open source Actions minutes? Or for storage of Open Source and public repositories? That is a risk that we need to be prepared for and more of a question of "when" than "if". It was only a few years ago that Travis CI was where Open Source projects built their software and collaborated. I don't think I've heard them mentioned since then.

Let's not underestimate the lengths that Open Source maintainers will go to - so that they can continue to serve their communities. They already work day and night without pay or funding, so whilst it's not convenient for anyone, we will find a way forward. Just like we did when Travis CI turned us away, and now Docker is shunning its Open Source roots.

See what people are saying on Twitter:

Is Docker saying that the OSS openfaas organisation on Docker Hub will get deleted if we don't sign up for a paid plan?

What about Prometheus, and all the other numerous OSS orgs on the Docker Hub?

cc @justincormack pic.twitter.com/FUCZPxHz1x

— Alex Ellis (@alexellisuk) March 14, 2023

Updates

Update: 17 March

There have been hundreds of comments on Hacker News, and endless tweets since I published my article. The community's response has been clear - abject disappointment and confusion.

Docker has since published an apology, I'll let you decide whether the resulting situation has been improved for your open source projects and for maintainers - or not.

The requirements for the "Docker-Sponsored Open Source (DSOS)" program have not changed, and remain out of touch with how Open Source is made sustainable.

Update: 24 March

Over 105k people read my article and hundreds of people voiced their concerns on both Hacker News and Twitter, following this pressure, Docker Inc reconsidered their decision.

10 days later, they emailed the same group of people - "We’re No Longer Sunsetting the Free Team Plan"

You can use actuated's new CLI to calculate the total number of build minutes you're using across an organisation with GitHub Actions.

I'm also going to show you:

• How to build tools rapidly, without worrying
• The best way to connect to the GitHub API using Go
• How to check your remaining rate limit for an access token
• A better way to integrate than using Access Tokens
• Further ways you could develop or contribute to this idea

Why do we need this?

If you log into the GitHub UI, you can request a CSV to be sent to your registered email address. This is a manual process and can take a few minutes to arrive.

It covers any paid minutes that your account has used, but what if you want to know the total amount of build minutes used by your organisation?

We wanted to help potential customers for actuated understand how many minutes they're actually using in total, including free-minutes, self-hosted minutes and paid minutes.

I looked for a way to do this in the REST API and the GraphQL API, but neither of them could give this data easily. It was going to involve writing a lot of boilerplate code, handling pagination, summing in the values and etc.

So I did it for you.

The actions-usage CLI

The new CLI is called actions-usage and it's available on the self-actuated GitHub organisation: self-actuated/actions-usage.

As I mentioned, a number of different APIs were required to build up the picture of true usage:

• Get a list of repositories in an organisation
• Get a list of workflow runs within the organisation for a given date range
• Get a list of jobs for each of those workflow runs
• Add up the minutes and summarise the data

The CLI is written in Go, and there's a binary release available too.

I used the standard Go flags package, because I can have working code quicker than you can say "but I like Cobra!"

flag.StringVar(&orgName, "org", "", "Organization name")
flag.StringVar(&token, "token", "", "GitHub token")
flag.IntVar(&since, "since", 30, "Since when to fetch the data (in days)")

flag.Parse()

In the past, I used to make API calls directly to GitHub using Go's standard library. Eventually I stumbled upon Google's "github-go" library and use it everywhere from within actuated itself, to our Derek bot and other integrations.

It couldn't be any easier to integrate with GitHub using the library:

auth := oauth2.NewClient(context.Background(), oauth2.StaticTokenSource(
  &oauth2.Token{AccessToken: token},
))
page := 0
    opts := &github.RepositoryListByOrgOptions{ListOptions: github.ListOptions{Page: page, PerPage: 100}, Type: "all"}

If you'd like to learn more about the library, I wrote A prototype for turning GitHub Actions into a batch job runner.

The input is a Personal Access Token, but the code could also be rewritten into a small UI portal and use an OAuth flow or GitHub App to authenticate instead.

• How to integrate with GitHub without PATs
• Build and deploy a React app with OpenFaaS

How to get your usage

The tool is designed to work at the organisation level, but if you look at my example for turning GitHub Actions into a batch job runner, you'll see what you need to change to make it work for a single repository, or to list all repositories within a personal account instead.

Or create a Classic Token with: repo and admin:org and save it to ~/pat.txt. Create a short lived duration for good measure.

Download a binary from the releases page

./actions-usage --org openfaas --token $(cat ~/pat.txt)

Fetching last 30 days of data (created>=2023-01-29)

Total repos: 45
Total private repos: 0
Total public repos: 45

Total workflow runs: 95
Total workflow jobs: 113
Total usage: 6h16m16s (376 mins)

The openfaas organisation has public, Open Source repos, so there's no other way to get a count of build minutes than to use the APIs like we have done above.

What about rate-limits?

If you remember above, I said we first call list repositories, then list workflow runs, then list jobs. We do manage to cut back on rate limit usage by using a date range of the last 30 days.

You can check the remaining rate-limit for an API token as follows:

curl -H "Authorization: token $(cat ~/pat.txt)" \
  -X GET https://api.github.com/rate_limit

{
  "rate": {
    "limit": 5000,
    "used": 300,
    "remaining": 4700,
    "reset": 1677584468
  }

I ran the tool twice and only used 150 API calls each time. In an ideal world, GitHub would add this to their REST API since they have the data already. I'll mention an alternative in the conclusion, which gives you the data, and insights in an easier way.

But if your team has hundreds of repositories, or thousands of builds per month, then the tool may exit early due to exceeding the API rate-limit. In this case, we suggest you run with -days=10 and multiply the value by 3 to get a rough picture of 30-day usage.

Further work

The tool is designed to be used by teams and open source projects, so they can get a grasp of total minutes consumed.

Why should we factor in the free minutes?

Free minutes are for GitHub's slowest runners. They're brilliant a lot of the time, but when your build takes more than a couple of minutes, become a bottleneck and slow down your team.

Ask me how I know.

So we give you one figure for total usage, and you can then figure out whether you'd like to try faster runners with flat rate billing, with each build running in an immutable Firecracker VM or stay as you are.

What else could you do with this tool?

You could build a React app, so users don't need to generate a Personal Access Token and to run a CLI.

• How to integrate with GitHub without PATs
• Build and deploy a React app with OpenFaaS

You could extend it to work for personal accounts as well as organisations. Someone has already suggested that idea here: How can I run this for a user account? #2

The code is open source and available on GitHub:

• self-actuated/actions-usage

This tool needed to be useful, not perfect, so I developed in my "Rapid Prototyping" style.

My new style for rapid prototyping in @golang:

* All code goes in main.go, in main(), no extra methods, no packages, no extra files
* Use Go's flags and log packages
* Maybe create a few separate methods/files, still in the main package

For as long as possible.. pic.twitter.com/9TEpN6XSCA

— Alex Ellis (@alexellisuk) October 8, 2022

If you'd like to gain more insights on your usage, to adopt Arm builds or speed up your team, Actuated users don't currently need to run tools like this to track their usage, we do it automatically for them and bubble it up through reports:

Actuated can also show jobs running across your whole organisation, for better insights for Team Leads and Engineering Managers:

]

Find out more about what we're doing to make self-hosted runners quicker, more secure and easier to observe at actuated.dev

Around 6-8 months ago I started exploring MicroVMs out of curiosity. Around the same time, I saw an opportunity to fix self-hosted runners for GitHub Actions. Actuated is now in pilot and aims to solve most if not all of the friction.

There's three parts to this post:

1. A quick debrief on Firecracker and MicroVMs vs legacy solutions
2. Exploring friction with GitHub Actions from a hosted and self-hosted perspective
3. Blazing fast CI with Actuated, and additional materials for learning more about Firecracker

We're looking for customers who want to solve the problems explored in this post. Register for the pilot

1. A quick debrief on Firecracker 🔥

Firecracker is an open source virtualization technology that is purpose-built for creating and managing secure, multi-tenant container and function-based services.

I learned about Firecracker mostly by experimentation, building bigger and more useful prototypes. This helped me see what the experience was going to be like for users and the engineers working on a solution. I met others in the community and shared notes with them. Several people asked "Are microVMs the next thing that will replace containers?" I don't think they are, but they are an important tool where hard isolation is necessary.

Over time, one thing became obvious:

MicroVMs fill a need that legacy VMs and containers can't.

If you'd like to know more about how Firecracker works and how it compares to traditional VMs and Docker, you can replay my deep dive session with Richard Case, Principal Engineer (previously Weaveworks, now at SUSE).

Join Alex and Richard Case for a cracking time. The pair share what's got them so excited about Firecracker, the kinds of use-cases they see for microVMs, fundamentals of Linux Operating Systems and plenty of demos.

2. So what's wrong with GitHub Actions?

First let me say that I think GitHub Actions is a far better experience than Travis ever was, and we have moved all our CI for OpenFaaS, inlets and actuated to Actions for public and private repos. We've built up a good working knowledge in the community and the company.

I'll split this part into two halves.

What's wrong with hosted runners?

Hosted runners are constrained

Hosted runners are incredibly convenient, and for most of us, that's all we'll ever need, especially for public repositories with fast CI builds.

Friction starts when the 7GB of RAM and 2 cores allocated causes issues for us - like when we're launching a KinD cluster, or trying to run E2E tests and need more power. Running out of disk space is also a common problem when using Docker images.

GitHub recently launched new paid plans to get faster runners, however the costs add up, the more you use them.

What if you could pay a flat fee, or bring your own hardware?

They cannot be used with public repos

From GitHub.com:

We recommend that you only use self-hosted runners with private repositories. This is because forks of your public repository can potentially run dangerous code on your self-hosted runner machine by creating a pull request that executes the code in a workflow.

This is not an issue with GitHub-hosted runners because each GitHub-hosted runner is always a clean isolated virtual machine, and it is destroyed at the end of the job execution.

Untrusted workflows running on your self-hosted runner pose significant security risks for your machine and network environment, especially if your machine persists its environment between jobs.

Read more about the risks: Self-hosted runner security

Despite a stern warning from GitHub, at least one notable CNCF project runs self-hosted ARM64 runners on public repositories.

On one hand, I don't blame that team, they have no other option if they want to do open source, it means a public repo, which means risking everything knowingly.

Is there another way we can help them?

I spoke to the GitHub Actions engineering team, who told me that using an ephemeral VM and an immutable OS image would solve the concerns.

There's no access to ARM runners

Building with QEMU is incredibly slow as Frederic Branczyk, Co-founder, Polar Signals found out when his Parca project was taking 33m5s to build.

I forked it and changed a line: runs-on: actuated-aarch64 and reduced the total build time to 1m26s.

This morning @fredbrancz said that his ARM64 build was taking 33 minutes using QEMU in a GitHub Action and a hosted runner.

I ran it on @selfactuated using an ARM64 machine and a microVM.

That took the time down to 1m 26s!! About a 22x speed increase. https://t.co/zwF3j08vEV pic.twitter.com/ps21An7B9B

— Alex Ellis (@alexellisuk) October 20, 2022

They limit maximum concurrency

On the free plan, you can only launch 20 hosted runners at once, this increases as you pay GitHub more money.

Builds on private repos are billed per minute

I think this is a fair arrangement. GitHub donates Azure VMs to open source users or any public repo for that matter, and if you want to build closed-source software, you can do so by renting VMs per hour.

There's a free allowance for free users, then Pro users like myself get a few more build minutes included. However, These are on the standard, 2 Core 7GB RAM machines.

What if you didn't have to pay per minute of build time?

What's wrong with self-hosted runners?

It's challenging to get all the packages right as per a hosted runner

I spent several days running and re-running builds to get all the software required on a self-hosted runner for the private repos for OpenFaaS Pro. Guess what?

I didn't want to touch that machine again afterwards, and even if I built up a list of apt packages, it'd be wrong in a few weeks. I then had a long period of tweaking the odd missing package and generating random container image names to prevent Docker and KinD from conflicting and causing side-effects.

What if we could get an image that had everything we needed and was always up to date, and we didn't have to maintain that?

Self-hosted runners cause weird bugs due to caching

If your job installs software like apt packages, the first run will be different from the second. The system is mutable, rather than immutable and the first problem I faced was things clashing like container names or KinD cluster names.

You get limited to one job per machine at a time

The default setup is for a self-hosted Actions Runner to only run one job at a time to avoid the issues I mentioned above.

What if you could schedule as many builds as made sense for the amount of RAM and core the host has?

Docker isn't isolated at all

If you install Docker, then the runner can take over that machine since Docker runs at root on the host. If you try user-namespaces, many things break in weird and frustrating aways like Kubernetes.

Container images and caches can cause conflicts between builds.

Kubernetes isn't a safe alternative

Adding a single large machine isn't a good option because of the dirty cache, weird stateful errors you can run into, and side-effects left over on the host.

So what do teams do?

They turn to a controller called Actions Runtime Controller (ARC).

ARC is non trivial to set up and requires you to create a GitHub App or PAT (please don't do that), then to provision, monitor, maintain and upgrade a bunch of infrastructure.

This controller starts a number of re-usable (not one-shot) Pods and has them register as a runner for your jobs. Unfortunately, they still need to use Docker or need to run Kubernetes which leads us to two awful options:

1. Sharing a Docker Socket (easy to become root on the host)
2. Running Docker In Docker (requires a privileged container, root on the host)

There is a third option which is to use a non-root container, but that means you can't use sudo in your builds. You've now crippled your CI.

What if you don't need to use Docker build/run, Kaniko or Kubernetes in CI at all? Well ARC may be a good solution for you, until the day you do need to ship a container image.

3. Can we fix it? Yes we can.

Actuated ("cause (a machine or device) to operate.") is a semi-managed solution that we're building at OpenFaaS Ltd.

A semi-managed solution, where you provide hosts and we do the rest

A semi-managed solution, where you provide hosts and we do the rest.

You provide your own hosts to run jobs, we schedule to them and maintain a VM image with everything you need.

You install our GitHub App, then change runs-on: ubuntu-latest to runs-on: actuated or runs-on: actuated-aarch64 for ARM.

Then, provision one or more VMs with nested virtualisation enabled on GCP, DigitalOcean or Azure, or a bare-metal host, and install our agent. That's it.

If you need ARM support for your project, the a1.metal from AWS is ideal with 16 cores and 32GB RAM, or an Ampere Altra machine like the c3.large.arm64 from Equinix Metal with 80 Cores and 256GB RAM if you really need to push things. The 2020 M1 Mac Mini also works well with Asahi Linux, and can be maxed out at 16GB RAM / 8 Cores. I even tried Frederic's Parca job on my Raspberry Pi and it was 26m30s quicker than a hosted runner!

Whenever a build is triggered by a repo in your organisation, the control plane will schedule a microVM on one of your own servers, then GitHub takes over from there. When the GitHub runner exits, we forcibly delete the VM.

You get:

• A fresh, isolated VM for every build, no re-use at all
• A fast boot time of ~ <1-2s
• An immutable image, which is updated regularly and built with automation
• Docker preinstalled and running at boot-up
• Efficient scheduling and packing of builds to your fleet of hosts

It's capable of running Docker and Kubernetes (KinD, kubeadm, K3s) with full isolation. You'll find some examples in the docs, but anything that works on a hosted runner we expect to work with actuated also.

Here's what it looks like:

Want the deeply technical information and comparisons? Check out the FAQ

You may also be interested in a debug experience that we're building for GitHub Actions. It can be used to launch a shell session over SSH with hosted and self-hosted runners: Debug GitHub Actions with SSH and launch a cloud shell

Wrapping up

We're piloting actuated with customers today. If you're interested in faster, more isolated CI without compromising on security, we would like to hear from you.

Register for the pilot

We're looking for customers to participate in our pilot.

Register for the pilot 📝

Actuated is live in pilot and we've already run thousands of VMs for our customers, but we're only just getting started here.

Pictured: VM launch events over the past several days

Other links:

• Read the FAQ
• Watch a short video demo
• Follow actuated on Twitter

What about GitLab?

We're focusing on GitHub Actions users for the pilot, but have a prototype for GitLab. If you'd like to know more, reach out using the Apply for the pilot form.

Just want to play with Firecracker or learn more about microVMs vs legacy VMs and containers?

• Watch A cracking time: Exploring Firecracker & MicroVMs
• Try my firecracker lab on GitHub - alexellis/firecracker-init-lab

What are people saying about actuated?

"We've been piloting Actuated recently. It only took 30s create 5x isolated VMs, run the jobs and tear them down again inside our on-prem environment (no Docker socket mounting shenanigans)! Pretty impressive stuff."

Addison van den Hoeven - DevOps Lead, Riskfuel

"Actuated looks super cool, interested to see where you take it!"

Guillermo Rauch, CEO Vercel

"This is great, perfect for jobs that take forever on normal GitHub runners. I love what Alex is doing here."

Richard Case, Principal Engineer, SUSE

"Thank you. I think actuated is amazing."

Alan Sill, NSF Cloud and Autonomic Computing (CAC) Industry-University Cooperative Research Center

"Nice work, security aspects alone with shared/stale envs on self-hosted runners."

Matt Johnson, Palo Alto Networks

"Is there a way to pay github for runners that suck less?"

Darren Shepherd, Acorn Labs

"Excited to try out actuated! We use custom actions runners and I think there's something here 🔥"

Nick Gerace, System Initiative

It is awesome to see the work of Alex Ellis with Firecracker VMs. They are provisioning and running Github Actions in isolated VMs in seconds (vs minutes)."

Rinat Abdullin, ML & Innovation at Trustbit

"This is awesome!" (After reducing Parca build time from 33.5 minutes to 1 minute 26s)

Frederic Branczyk, Co-founder, Polar Signals

When I heard about the Microsoft Dev Kit 2023, I was surprised by how generous the specifications were for the price? Naturally, I wanted to know if I could run Linux on it. You may also wonder. The answer is: kinda.

I'm not sure why you are interested in ARM computers, but for me I got involved with them when porting software to the Raspberry Pi and helping support other Open Source projects to do the same. I maintain OpenFaaS, and it's always had support to run on ARM devices as well as larger on-premises or cloud servers. The team at Equinix Metal gave me complimentary access to large enterprise-grade ARM servers, Ampere even shipped me a server once, which was unfortunately far too loud to leave on in the house.

Then came the M1 chip from Apple, this is probably where interest in ARM turned mainstream, and binaries for common software started appearing on GitHub releases pages, Docker announced a partnership with ARM and AWS released two generations of server CPUs with an ARM architecture, including support for Lambda.

See also: raspberrypi.org: Five years of Raspberry Pi clusters

The Dev Kit offer

The Dev Kit comes in a small black plastic case with a matte finish, it's got 3x USB-A ports on the back, 2x USB-C on the side and a mini Display Port connector. Inside there's a 8-core Snapdragon 8c 64-bit ARM processor, 32GB of RAM and a 512GB NVMe.

"OpenBSD Dev Kit 2023" by Patrick Wildt

Why does that look like such good value? Well there really aren't any other computers in this price range that have a 64-bit ARM processor, and the Raspberry Pi 4 is miles slower in comparison.

Some comparisons on options for ARM64 compute. https://t.co/9mmL84m4tr pic.twitter.com/T4LIioxEo0

— Alex Ellis (@alexellisuk) October 25, 2022

Comparing the Raspberry Pi 4 8GB against some more expensive machines.

What about the Mac Mini M1 2020? Despite being over 2 years old, it still costs 899 GBP and comes with half the RAM and half the storage. That makes the Dev Kit great value at 570 GBP.

One thing we do know about the Mac Mini is that it's worse value, but a good performer, and Asahi Linux seems relatively stable. It even runs Firecracker.

The Mac Mini M1 with Ashai Linux installed really is quite fast for the money

Here it is compared to a Raspberry Pi 4 building a custom Linux Kernel.

The Ampere Altra is much faster, even with only 30/80 of the cores allocated to the build. pic.twitter.com/fKqDhlfZcX

— Alex Ellis (@alexellisuk) October 26, 2022

What I've tried

The first thing I figured out was how to disable secure boot.

Disable secure boot by holding the power button and small circular button down and powering up.

That was painless, then I flashed my trusty USB pen drive with Ubuntu 22.04 and held the two larger buttons to "Boot from USB"

No Linux for you

Grub showed its boot menu, and I thought I'd got it sorted. I couldn't have been more wrong.

So I did what any sane person would do next, try a newer Ubuntu version - 22.10. It broke in exactly the same way.

Now support for the Snapdragon 8c was merged into Linux v6.0, so I thought perhaps I needed to build a new Kernel?

Not booting a v6.0 Kernel

Well that didn't work either.

So I thought maybe I had built the Kernel wrong, and would use a nightly snapshot of Debian sid, which had a v6.0 Kernel built in. That also didn't work.

I spent a day building and rebuilding USB drives and running debootstrap, hoping that one variation or changed Kernel configuration setting would get me booted at least into the Kernel's start-up screen.

Patrick Wildt, an OpenBSD maintainer replied to one of my tweets and told me he had OpenBSD 7.2 up and running on his, so I thought I would at least try that out.

OpenBSD running!

I was able to install Golang (Go) and port some of my own software over ([inlets/mixctl](https://github.com/inlets/mixctl/ - a TCP load-balancer written in Go):

But I need to be able to run Linux, with KVM for this to be useful for my work on actuated - managed and isolated self-hosted CI runners using Firecracker.

I even tried a USB flash drive known to boot with the Lenovo x13s, but it didn't get past Grub.

Not quite giving up

I've used Linux on the desktop since 2018 and it suits my current work very well to be able to run containers directly on the host I'm using, to have a lightweight and responsive UI and the ease of configuration.

Whilst all the various OSS projects I release all have Windows binaries, and it's a decent Operating System, I don't want to have to contend with it on a daily basis.

So I thought, if I can't run Linux on the machine directly, what if I could use WSL2?

Windows Dev Kit 2023 - Linux (WSL) vs Mac Mini with Asahi Linux

The difference isn't as marked as you would have thought.

I tested with Geekbench 5, hdparm and dd.

But.. pic.twitter.com/3E6jnFbNwk

— Alex Ellis (@alexellisuk) October 31, 2022

WSL started really quite quickly and so I ran Geekbench 5, hdparm and dd to test the CPU/memory, along with the read and write speed of the disk.

The single-core speed was better, whilst the multi-core speed had dropped off a little compared to running Geekbench directly in Windows 11.

Now because I needed KVM to run Firecracker, I typed in "cpu-checker" and to my dismay, I saw the module hadn't been made available in this Kernel.

At OpenFaaS Ltd, we've been building Kernels to run in guest VMs using Firecracker for actuated CI runners, for ARM and Intel processors, so it's something I'm very familiar with.

Hayden Barnes has written about how to build and install a custom Kernel for WSL, so I took his instructions and updated them for ARM.

Building my Kernel with KVM enabled

Then just like all my other experiments with trying to get Linux to work how I needed, it felt flat on its face:

For whatever reason, KVM isn't built into the Kernel either as a built-in module or as a loadable module, and a custom Kernel isn't yet supported for ARM64.

Why we need ARM for CI and how my Raspberry Pi beat GitHub's hosted runner

Let's take a quick look at why access to real ARM hardware is important vs. emulation with QEMU.

A Raspberry Pi 8GB kitted out with an NVMe, bought before the global shortage

A start-up founder in Berlin tweeted complaining that his tests were taking 33m30s to execute against an ARM64 CPU profile.

Does anyone have a @github actions self-hosted runner manifest for me to throw at a @kubernetesio cluster? I'm tired of waiting for emulated arm64 CI runs taking ages.

— Frederic 🧊 Branczyk (@fredbrancz) October 19, 2022

I took a look at his builds and quickly found out that the reason for the slowness was user-space emulation that his team had set up with the "qemu" tool.

qemu is what allows us to create multi-arch builds from a PC that work on ARM computers. But it can also run programs and operating systems.

I cloned his Parca project, then moved it to an organisation where I'd set up actuated, and changed ubuntu-latest to actuated-aarch64

The first build went onto an ARM64 host at Equinix Metal costing 2.5 USD / hour. The build usually took over 33 minutes on a Hosted Runner without emulation, it took just 1min26s on the Equinix Metal host.

And that was with only 4/80 cores allocated, if it had say 32 allocated, it would have probably completed even quicker.

QEMU vs bare-metal ARM64

Next, I set up the actuated agent on a Raspberry Pi 4. The initial run was 9m30s, 3x faster than emulation, on a device that has a one-time cost of 30-80 GBP total. I then noticed that his build was spending a long time resolving and downloading Go modules. I ran go mod vendor and started another build.

That took 7min20s. Saving 2 minutes.

Vendoring to the rescue

So then I looked into what the cheapest ARM64 host would be on the cloud, and it turns out AWS has an a1.metal with 16 cores and 32GB of RAM for 0.48 USD / hour, or 350 USD / mo. So for 350 USD / mo, you can have an ARM64 build-queue 3-4 deep and builds that complete in 1 minute instead of 34.

If some day a Dev Kit 2023 actually works with Linux and KVM, then you could pay for vs an AWS a1.metal instance in just two months.

If you want ARM64 builds or end to end tests for your project, feel free to reach out to me.

Conclusion

The Dev Kit 2023 from Microsoft is a snappy Windows 11 machine with excellent support for Microsoft's WSL2 "Linux experience". WSL2 in this configuration does not support virtualisation or custom Kernel builds. Systemd is turned off by default, which means common software may not work out the box. I didn't test the support, but I was told that you can enable it using these instructions.

When is Linux coming then?

Booting Linux isn't an option at the moment, and may require Microsoft to release a Device Tree Blob (DTB), or a third-party to reverse engineer this. My understanding is that this wasn't required for OpenBSD because it can boot in APCI mode, and the Thinkpad X13s boots because its vendor provided custom DTBs.

Whilst I enjoy this kind of tinkering, it was disappointing that a "Dev Kit", built for developers can neither boot Linux, nor enable a custom Linux Kernel for WSL2.

What's that I hear Hacker News and Reddit cry? "It's a Dev Kit for Windows, you moron!" That may be the case, but Microsoft "Loves Linux", and clearly has worked hard to make WSL2 available out of the box on these devices.

If you're looking for a step above the Raspberry Pi B 8GB for running headless Linux, the 2020 Mac Mini, configured with 16GB of RAM and 256GB of disk space runs to ~ 899 GBP. I wish it were cheaper, but with Ashai Linux it runs Firecracker, KVM, Docker and just about anything else I've thrown at it.

If you get further than I did, please feel free to reach out. For my use-case of building Kubernetes clusters, and supporting Open Source projects and companies to build on ARM64, headless use is absolutely fine.

See also:

• The Past, Present, and Future of Kubernetes on Raspberry Pi - Alex Ellis - KubeCon
• State of netbooting Raspberry Pi in 2021
• Walk-through — install Kubernetes to your Raspberry Pi in 15 minutes

Thanks to:

• Patrick Wildt for encouraging me to try out OpenBSD
• Lucas Lombard for giving me access to his 2020 Mac Mini M1 for testing out actuated

Check out the new thing I'm doing with ARM

With actuated, we're trying to make CI faster, more secure, and isolated whilst taking away a lot of the management and common issues.

Our actuated solution is primarily for Intel/AMD users, but also supports ARM runners.

Feel free to check out the docs, or watch a quick demo of it in action launching Firecracker VMs for each CI job:

I want to show you how to use arkade in your GitHub Actions to get the typical kinds of tools you'll need to build code and run end-to-end tests. By the end of the post, I hope you'll see why developers are using arkade on their workstations and in CI instead of the more clunky alternatives.

arkade has 3.1k GitHub Stars, 157 releases, over 110k downloads and over 72 contributors.

Let's start right at the beginning.

When I started the arkade project in 2019, it was all about making it easier to install Helm charts for Kubernetes clusters. In one instance, I was writing a tutorial for Civo, and needed 5000 words to install the boilerplate software to get a TLS certificate.

Arkade's job was to reduce these 5000 words to 5 lines of text, and it worked. That's where arkade install came about. It could convert lengthy, repetitive technical content into concise steps:

arkade install ingress-nginx
arkade install cert-manager
arkade install openfaas
arkade install openfaas-ingress \
    --email [email protected] 
    --domain faas.openfaas.com

How's that different from Helm? I've got a surprise for you. It is Helm, but wrapped with defaults so you can focus on your $dayjob instead of flicking through values.yaml files. Every "app" for Kubernetes has flags that we've hand-picked, see arkade install openfaas --help for instance.

But please don't let me distract you, this post is about arkade get. A much faster alternative to brew, and a much more up to date alternative to apt-get.

Then, it quickly became necessary to have CLIs at hand on Windows, MacOS, Linux and even ARM hosts like a Raspberry Pi, arkade get was born.

Works on MacOS, Darwin M1 and Intel, Linux and Linux ARM:

arkade get kubectl
arkade get [email protected]
arkade get kubectl faas-cli helm inletsctl

Try it out for yourself, you can download arkade to your workstation here: arkade

Extending GitHub Actions

There are three types of GitHub Action, we'll look at two of them: composite and JavaScript. For all intents and purposes, a composite action is what you use when you want to write bash.

Here's an example I wrote to setup SSHD with the actor's SSH keys pre-installed: alexellis/setup-sshd-actor

Using the action looks the same as any other:

  - name: Setup SSH server for Actor
    uses: alexellis/setup-sshd-actor@master

JavaScript actions are a better fit when you need some application logic, but don't want to ship and host a separate binary to be run in a composite action.

My earliest was [alexellis/upload-assets] which is a GitHub Action to upload multiple assets to a release using a wildcard. I made it because GitHub's own didn't support wildcards, and I needed that everywhere and it was incredibly useful in the transition from Travis CI.

Here's how you use it:

      - name: Upload release binaries
        uses: alexellis/[email protected]
        env:
          GITHUB_TOKEN: ${{ github.token }}
        with:
          asset_paths: '["./bin/release-it*"]'

Square peg, round CI system

Installing arkade inside a GitHub Action is as simple as:

name: Install arkade
run: curl -sLS https://get.arkade.dev | sudo sh

But ardent fans of GitHub Actions wanted more.

So I sat down and wrote a composite action to install the binary in a similar way, here it is:

# Copyright (c) 2022 OpenFaaS Ltd
name: 'Install arkade'
description: 'Install arkade'
branding:
  icon: 'arrow-right-circle'
  color: 'gray-dark'
runs:
    using: 'composite'
    steps:
      - name: Install Install
        shell: bash
        id: download
        run: |
          if ! [ -x "$(command -v curl)" ]; then
            sudo apt update && sudo apt install -qy curl
          fi
          curl -sLS https://get.arkade.dev | sudo sh
          echo "PATH=$HOME/.arkade/bin:$PATH" >> $GITHUB_ENV

Now you can write this instead:

uses: alexellis/setup-arkade@v2

What's that I hear? You're not impressed?

I thought we could do better too.

So I thought how I could build an action for the get command. I'd seen a friend (Stefan Prodan) had a hard-coded list of tools (that were also available in arkade).

He'd used inputs to control which tools to install:

      - name: Run Kubernetes tools
        uses: stefanprodan/kube-tools@v1
        with:
          kubectl: 1.23.0
          kustomize: 4.4.1
          helm: 2.17.0
          helmv3: 3.7.2
          kubeseal: 0.16.0
          kubeval: v0.16.1

Unfortunately, that also meant hard-coding the list in his actions.yml file, and having to write new code for every tool he added.

inputs:
  command:
    description: 'command to run'
    required: true
  kubectl:
    description: 'kubectl version'
  kustomize:
    description: 'kustomize version'

So could I do better?

I took inspiration from his design and came up with a tool that executed arkade get -o list and then created a new YAML file with all the inputs printed out, a bit like Stefan's file.

Rather than his dozen tools, you get access to 106 tools, with more added every week:

arkade get -o list |wc -l
106

Check out the program I wrote in Go: to-inputs/main.go

This program gets run every night to repopulate the actions.yml file using a good ole trick called envsubst.

Here how envsubst works:

cat<<EOF > template.txt
# Inputs from arkade:

$INPUTS

EOF

INPUTS=$(arkade get -o list)
cat template.txt | envsubst > result.txt

head -n 5 result.txt

# Inputs from arkade:

argocd
argocd-autopilot
arkade
...

You can see the full version of the nightly GitHub Action here which regenerates the list of tools and commits back to the repository: https://github.com/alexellis/arkade-get/blob/master/.github/workflows/update-tools.yml

Here's what it looks like now:

    - uses: alexellis/setup-arkade@v1
    - uses: alexellis/arkade-get@master
      with:
        kubectl: v1.25.0
        helm: latest
        argocd: latest
        inletsctl: latest
        faas-cli: 0.14.10
    - name: check for faas-cli
      run: |
        faas-cli version

Wrapping up

I came here to show you some of what I learned creating both composite and JavaScript actions, and how to work around a limitation for custom actions, where you can only specify a static list of inputs.

Need more convincing?

Two year update: Building an Open Source Marketplace for Kubernetes

What if you love brew?

That's OK, you're allowed to try arkade from time to time, especially if you don't like waiting to download CLIs.

What if you're an arduent apt-get convert and can't be convinced otherwise?

That's OK too, if you like old really packages or waiting for PPAs to update.

What if there's already a GitHub action to install this one tool?

Then feel free to use it, but for the ones that are missing their own action, or where that becomes too much to manage, Arkade has 106 tools at the moment, with more being added every week and month.

What else does arkade do?

Glad you asked, check out the README, it's full of practical examples: arkade.dev

How do I contribute a CLI to arkade?

Check out a recent PR where we added flyctl: add flyctl tool #769

Want a full example?

Here's what it looks like to install K3s using K3sup and then go on to deploy OpenFaaS. It's an example from the actuated docs.

name: k3sup-tester

on: push
jobs:
  k3sup-tester:
    runs-on: actuated
    steps:
    - uses: alexellis/setup-arkade@v1
    - uses: alexellis/arkade-get@master
      with:
        kubectl: v1.25.0
        k3sup: latest
        faas-cli: latest

      - name: Install K3s with k3sup
        run: |
          mkdir -p $HOME/.kube/
          k3sup install --local --local-path $HOME/.kube/config
      - name: Wait until nodes ready
        run: |
          k3sup ready --quiet --kubeconfig $HOME/.kube/config --context default
      - name: Install OpenFaaS
        run: arkade install openfaas

Whilst I've got you here

At OpenFaaS Ltd, we've been trying to make the experience for self-hosted runners faster, completely isolated and more efficient.

See if this resonates?

GitHub Actions Managed/Hosted runners are great most of the time and super convenient

but also:

- limited on CPU/RAM
- slower than your own hardware or cloud provider
- limited to Intel runners
- limited on concurrent jobs
- and those e2e tests are making you miserable

So..

— Alex Ellis (@alexellisuk) September 24, 2022

Or watch the video:

Live demo and walk-through of actuated for secure self-hosted GitHub Actions

I wanted to run a container for a customer only once, but the UX just wasn't simple enough. So I created a new OSS utility with Go and the Kubernetes API.

A Deployment will usually restart, so will a Pod. You can set the "restartPolicy" to "Never", but it's still not a good fit for something that's meant to start - run and complete.

Why would you want to run a one-time job?

Here are a few ideas:

• Running kubectl within the cluster under a specific Service Account
• Checking curl works from within the cluster
• Checking inter-pod DNS is working
• Cleaning up a DB index
• A network scan with nmap
• Tasks that you'd usually run on cron
• Dynamic DNS updates
• Generating a weekly feed / report / update / sync
• Running a container build with Kaniko

Another use-case may be where you're running bash, but need to execute something within the cluster and get results back before going further.

I'll show you what working with a Job looks like today and how we can make it better with a little Go code.

What a Job looks like

So I looked again at the Kubernetes Job, which is an API that I rarely see used. They look a bit like this:

apiVersion: batch/v1
kind: Job
metadata:
  name: pi
spec:
  suspend: true
  parallelism: 1
  completions: 5
  template:
    spec:
      containers:
      - name: pi
        image: perl:5.34.0
        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: Always
  backoffLimit: 4

Example from the Kubernetes docs

So the overall spec is similar to a Pod, but it's unfamiliar enough to cause syntax errors when composing these by hand.

What's more: a Job isn't something that you run, then get its logs and continue with your day.

You have to describe the job, to find a Pod that it created with a semi-random name, and then get the logs from that.

kubectl describe jobs/myjob

Name:           myjob
...
Events:
  Type    Reason            Age   From            Message
  ----    ------            ----  ----            -------
  Normal  SuccessfulCreate  12m   job-controller  Created pod: myjob-hlrpl
  Normal  SuccessfulDelete  11m   job-controller  Deleted pod: myjob-hlrpl
  Normal  Suspended         11m   job-controller  Job suspended
  Normal  SuccessfulCreate  3s    job-controller  Created pod: myjob-jvb44
  Normal  Resumed           3s    job-controller  Job resumed

So you end up with something like this:

apiVersion: batch/v1
kind: Job
metadata:
  name: checker
  namespace: openfaas
spec:
  completions: 1
  parallelism: 1
  template:
    metadata:
      name: checker
    spec:
      serviceAccount: openfaas-checker
      containers:
      - name: checker
        image: ghcr.io/openfaas/config-checker:latest
        imagePullPolicy: Always
      restartPolicy: Never

And in my instance, I also wanted RBAC.

Then you'd:

• Apply the RBAC file

• Apply the Job

• Describe the Job

• Find the Pod name

• Get the Pod logs

• Delete the Job/Pod

  #!/bin/bash

  JOBUUID=$(kubectl get job -n openfaas $JOBNAME -o "jsonpath={.metadata.labels.controller-uid}") PODNAME=$(kubectl get po -n openfaas -l controller-uid=$JOBUUID -o name)

  kubectl logs -n openfaas $PODNAME > $(date '+%Y-%m-%d_%H_%M_%S').txt

  kubectl delete -n openfaas $PODNAME kubectl delete -f ./artifacts/job.yaml

Can we do better?

I think we can. And in a past life, all the way back in 2017, before Kubernetes won me over, I was using Docker Swarm.

I wrote a little tool called "jaas" and it turned out to be quite popular with people using it for one-time tasks that would run on Docker Swarm.

View the code: alexellis/jaas

I also remembered that Stefan Prodan who was a friend of mine and a past contributor to OpenFaaS had once had this itch and created something called kjob.

Stefan's not touched the code for three years, but his approach was to take a CronJob as a template, then to add in a few alterations. It's flexible because it means you can do just about anything you want with the spec, then you override one or two fields.

But I wanted to move away from lesser-known Kubernetes APIs and get a user-experience that could be as simple as:

kubectl apply -f ./artifacts/rbac.yaml
run-job ./job.yaml -o report.txt

Enter run-job

So that's how run-job was born.

My previous example became:

name: checker
image: ghcr.io/openfaas/config-checker:latest
namespace: openfaas
service_account: openfaas-checker

And this was run with run-job -f job.yaml.

Then I thought, let's make this more fun. The cows function in OpenFaaS is one of the most popular for demos and was also part of the codebase for jaas.

Here's its Dockerfile:

FROM --platform=${TARGETPLATFORM:-linux/amd64} alpine:3.16 as builder

ARG TARGETPLATFORM
ARG BUILDPLATFORM
ARG TARGETOS
ARG TARGETARCH

RUN mkdir -p /home/app

RUN apk add --no-cache nodejs npm

# Add non root user
RUN addgroup -S app && adduser app -S -G app
RUN chown app /home/app

WORKDIR /home/app
USER app

COPY package.json    .
RUN npm install --omit=dev
COPY index.js        .

CMD ["/usr/bin/node", "./index.js"]

It's multi-arch and you can either use buildx or faas-cli to compile it for various architectures.

$ cat <<EOF > cows.yaml
# Multi-arch image for arm64, amd64 and armv7l
image: alexellis2/cows:2022-09-05-1955
name: cows
EOF

Here's what it looks like:

$ run-job -f cows.yaml

        ()  ()
         ()()
         (oo)
  /-------UU
 / |     ||
*  ||w---||
   ^^    ^^
Eh, What's up Doc?

With kubectl get events -w we can see what's actually happening behind the scenes:

0s          Normal   SuccessfulCreate   job/cows         Created pod: cows-5qld5
0s          Normal   Scheduled          pod/cows-5qld5   Successfully assigned default/cows-5qld5 to k3s-eu-west-agent-1
0s          Normal   Pulling            pod/cows-5qld5   Pulling image "alexellis2/cows:2022-09-05-1955"
0s          Normal   Pulled             pod/cows-5qld5   Successfully pulled image "alexellis2/cows:2022-09-05-1955" in 877.707423ms
1s          Normal   Created            pod/cows-5qld5   Created container cows
0s          Normal   Started            pod/cows-5qld5   Started container cows
0s          Normal   Completed          job/cows         Job completed

An example to use at work

But we can also do for-work kinds of things with one-shot tasks.

Imagine that you wanted to get metadata on all your Kubernetes nodes.

You'd need an RBAC file for that:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubectl-run-job
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: run-job
  name: kubectl-run-job
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: run-job
  name: kubectl-run-job
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubectl-run-job
subjects:  
  - kind: ServiceAccount
    name: kubectl-run-job
    namespace: default
---

Then you'd need a container image with kubectl pre-installed, so let's make that:

FROM --platform=${TARGETPLATFORM:-linux/amd64} alpine:3.16 as builder

ARG TARGETPLATFORM
ARG BUILDPLATFORM
ARG TARGETOS
ARG TARGETARCH

RUN mkdir -p /home/app

# Add non root user
RUN addgroup -S app && adduser app -S -G app
RUN chown app /home/app

RUN apk add --no-cache curl && curl -SLs https://get.arkade.dev | sh

WORKDIR /home/app
USER app

RUN arkade get [email protected] --quiet

CMD ["/home/app/.arkade/bin/kubectl", "get", "nodes", "-o", "wide"]

We'd also create a YAML file for the job:

name: get-nodes
image: alexellis2/kubectl:2022-09-05-2243
namespace: default
service_account: kubectl-run-job

Finally, we'd run the job:

$ kubectl apply ./examples/kubectl/rbac.yaml
$ run-job -f ./examples/kubectl/kubectl_get_nodes_job.yaml

Created job get-nodes.default (4097ed06-9422-41c2-86ac-6d4a447d10ab)
....
Job get-nodes.default (4097ed06-9422-41c2-86ac-6d4a447d10ab) succeeded 
Deleted job get-nodes

Recorded: 2022-09-05 21:43:57.875629 +0000 UTC

NAME           STATUS   ROLES                       AGE   VERSION        INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
k3s-server-1   Ready    control-plane,etcd,master   25h   v1.24.4+k3s1   192.168.2.1   <none>        Raspbian GNU/Linux 10 (buster)   5.10.103-v7l+      containerd://1.6.6-k3s1
k3s-server-2   Ready    control-plane,etcd,master   25h   v1.24.4+k3s1   192.168.2.2   <none>        Raspbian GNU/Linux 10 (buster)   5.10.103-v7l+      containerd://1.6.6-k3s1
k3s-server-3   Ready    control-plane,etcd,master   25h   v1.24.4+k3s1   192.168.2.3   <none>        Raspbian GNU/Linux 10 (buster)   5.10.103-v7l+    containerd://1.6.6-k3s1

Now, you can also specify the command and arguments for the command in the job file, so you could get the same output in JSON and pipe it through to jq to automate things in bash:

name: get-nodes
image: alexellis2/kubectl:2022-09-05-2243
namespace: default
service_account: kubectl-run-job
command:
 - kubectl
args:
 - get nodes
 - -o
 - json

Summing up

The point of run-job really is to make running a one-time container on Kubernetes simpler than the experience of crafting YAML and typing in half a dozen kubectl commands.

If you think it may be useful for you, then run-job is available via arkade get run-job along with over 100 other CLIs that download much quicker than brew.

We've also had the first contribution to the repo, which was to add the "command" and "args" options to the YAML file. The idea isn't to replicate every field in the Pod and Job specification, but just enough to make "run-job -f job-file.yaml` useful and convenient for supporting customers and running ad-hoc tasks.

Star on GitHub: alexellis/run-job

At OpenFaaS Ltd, we've used run-job with a customer to check their OpenFaaS Deployments and the configuration of their functions. The container is a Go program and you may like to take a look at it, to adapt it to check your own systems?

For those of us who are interested in manipulating and querying a Kubernetes cluster from code, the Go client provides a really good experience with hundreds of OSS examples available.

Read the code: openfaas/config-checker

If you're really into checking remote Kubernetes clusters for your customers, then Replicated have a similar, but more generic tool to openfaas/config-checker called "troubleshoot" which can be used to create bundles.

The bundles collect data from the customer's environment for you and you get to write in a DSL.

This is my guide to troubleshooting applications deployed on a Kubernetes cluster. It's based upon my experience with the technology since I switched over from Docker in early 2017.

There are many third-party tools and techniques available, but I want to focus in on things that you'll find on almost every computer, or CLIs that you can download quickly for MacOS, Windows or Linux.

What if you're a Kubernetes expert?

If you're already supporting clusters for customers, or developing software, you may know some of these. So perhaps you may find this content useful to send over to your own users, and colleagues.

In any case, stay with me and see whether there's a new flag or technique you may pick up. You may also like some of the more advanced supplementary material that I'm linking to throughout the post.

Got a comment, question or suggestion? Let's talk on Twitter

Is it there?

When your application isn't working, you may want to check that all its resources have been created.

The first command you learn is probably going to be "kubectl get pods"

But remember, that Kubernetes supports various namespaces in order to segregate and organise workloads. What you've installed may not be in the default namespace.

Here's how can change the namespace to look into the "kube-system" or "openfaas-fn" namespace for instance:

kubectl get pods --namespace kube-system
kubectl get pods -n openfaas

You can query all of the namespaces available with:

kubectl get pods --all-namespaces
kubectl get pods -A

The -A flag was added to kubectl in the 2-3 years, and means you can save on typing.

Now of course, Pods are just one of the things we care about. The above commands can also take other objects like Service, Deployment, Ingress and more.

Why isn't it working?

When you ran "kubectl get", you may have seen your resource showing as 0/1, or even as 1/1 but in a crashing or errored state.

How do we find out what's going wrong?

You may be tempted to reach for "kubectl logs", but this only shows logs from applications that have started, if your pod didn't start, then you need to find out what's preventing that.

You're probably running into one of the following:

• The image can't be pulled
• There's a missing volume or secret
• No space in the cluster for the workload
• Taints or affinity rules preventing the pod from being scheduled

Now kubectl get events on its own isn't very useful, because all the rows come out in what appears to be a random order. The fix is something you'll have to get tattooed somewhere prominent, because there's no shorthand for this yet.

kubectl get events \
  --sort-by=.metadata.creationTimestamp

This will print out events in the default namespace, but it's very likely that you're working in a specific namespace, so make sure to include the --namespace or --all-namespaces flag:

kubectl get events \
  --sort-by=.metadata.creationTimestamp \
  -A

Events are not just useful for finding out why something's not working, they also show you how pods are pulled, scheduled and started on a cluster.

Add "--watch" or "-w" to the command to watch an OpenFaaS function being created for instance:

kubectl get events \
  --sort-by=.metadata.creationTimestamp \
  -n openfaas-fn

And with the watch added on:

kubectl get events \
  --sort-by=.metadata.creationTimestamp \
  -n openfaas-fn \
  --watch

LAST SEEN   TYPE      REASON              OBJECT                           MESSAGE
0s          Normal    Synced              function/figlet                  Function synced successfully
0s          Normal    ScalingReplicaSet   deployment/figlet                Scaled up replica set figlet-5485896b55 to 1
1s          Normal    SuccessfulCreate    replicaset/figlet-5485896b55     Created pod: figlet-5485896b55-j9mbd
0s          Normal    Scheduled           pod/figlet-5485896b55-j9mbd      Successfully assigned openfaas-fn/figlet-5485896b55-j9mbd to k3s-pi
0s          Normal    Pulling             pod/figlet-5485896b55-j9mbd      Pulling image "ghcr.io/openfaas/figlet:latest"
0s          Normal    Pulled              pod/figlet-5485896b55-j9mbd      Successfully pulled image "ghcr.io/openfaas/figlet:latest" in 632.059147ms
0s          Normal    Created             pod/figlet-5485896b55-j9mbd      Created container figlet
0s          Normal    Started             pod/figlet-5485896b55-j9mbd      Started container figlet

There's actually quite a lot that's going on in the above events. You can then run something like kubectl scale -n openfaas-fn deploy/figlet --replicas=0 to scale it down and watch even more events getting generated as the pod is removed.

Now, there is a new command called kubectl events which you may also want to look into, however my kubectl version was too old, and it's only an alpha feature at present.

You can upgrade kubectl using arkade whether you're on Windows, MacOS or Linux:

arkade get [email protected]
$HOME/.arkade/bin/kubectl alpha events -n openfaas-fn

Now as I understand it, this new command does order the events, to keep an eye on how it progresses to see if the Kubernetes community promote it to generally available (GA) status or not.

It starts, but doesn't work

So the application starts up with 1/1 pods, or starts up then keeps crashing, so you're seeing a lot of restarts when you type in:

kubectl get pod
NAME                                        READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-54d8b558d4-59lj2   1/1     Running   5          114d

This is probably where the age old favourite "kubectl logs" comes in.

Most people do not work with Pods directly, but create a Deployment, which in turn creates a number of Pods depending on the replicas field. That's true of the way I've installed ingress-nginx, which you can see has restarted 5 times. It's now been running for 114 days or nearly 4 months.

kubectl logs ingress-nginx-controller-54d8b558d4-59lj2|wc -l

Wow. I just saw 328 lines pass by, and that was too much information.

Let's filter down to just the past 10 lines, and who needs to be typing in pod names? That's no longer necessary in Kubernetes.

Find the deployment name and use that instead:

kubectl get deploy

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
ingress-nginx-controller   1/1     1            1           114d


kubectl logs deploy/ingress-nginx-controller \
  --tail 10

That's better, just the past 10 lines show up. But what if I want to monitor the logs as the application runs?

Well kubectl logs doesn't have a --watch flag, but it does have a --follow (-f) flag that we can use instead:

kubectl logs deploy/ingress-nginx-controller \
  --tail 10 \
  --follow

Pods can have more than one container, and when they do, it means more typing, because we have to pick one of them.

Did you notice that I added an extra flag here? --container

kubectl logs -n openfaas deploy/gateway

error: a container name must be specified for pod gateway-7c96d7cbc4-d47wh, choose one of: [gateway operator]

We can also filter down to logs that were emitted during a specific time-frame:

kubectl logs -n openfaas deploy/gateway \
  --container gateway \
  --since 30s

You can also enter a specific date in RFC3339 format by using the --since-time flag.

There are too many pods!

When your deployment has more than one replica, then the kubectl logs command will select only one of them, and you'll miss out on potentially important information.

kubectl logs -n openfaas deploy/queue-worker
Found 2 pods, using pod/queue-worker-755598f7fb-h2cfx
2022/05/23 15:24:32 Loading basic authentication credentials
2022/05/23 15:24:32 Starting Pro queue-worker. Version: 0.1.5	Git Commit: 8dd99d2dc1749cfcf1e828b13fe5fda9c1c921b6	Ack wait: 60s	Max inflight: 25
2022/05/23 15:24:32 Initial retry delay: 100ms	Max retry delay: 10s	Max retries: 100

We can see that there are two pods:

kubectl get pods -n openfaas|grep queue
queue-worker-755598f7fb-h2cfx           1/1     Running   1          28d
queue-worker-755598f7fb-ggkn9           1/1     Running   0          161m

So kubectl only attached us to the oldest pod, not the newer one created just over 2 hours ago.

The fix here is to either use a label selector, which matches common labels across both pods, or to use a third-party tool.

kubectl logs -n openfaas --l app=queue-worker

Using a label selector means that we will not get new pods created since we stat the command, so a third party tool is going to be better for anything that can auto-scale or crash and restart.

Recently, I would have recommended kail, but whilst working with an OpenFaaS customer, we discovered that the maintainer doesn't cater to Window users.

Instead, we switched to a very similar tool called stern.

You can install this tool with arkade onto Windows, MacOS and Linux:

arkade get stern

But can you curl it?

Whilst Kubernetes can run batch jobs and background tasks, most of the time, you will see teams deploying websites, microservices, APIs and other applications with HTTP or TCP endpoints.

So a good question to ask is "can I curl it?"

Accessing services requires its own article, so I wrote that up a few weeks ago: A Primer: Accessing services in Kubernetes

Why have we run out of RAM already?

The metrics-server project is an add-on for Kubernetes that can quickly show you how much RAM and CPU is being used by Pods across your cluster. It'll also show you how well balanced the workloads are across nodes.

Here's a complete installation of OpenFaaS on a 2x node Raspberry Pi cluster. It also includes a bunch of extras like OpenFaaS Pro, Grafana, an inlets tunnel for connectivity, the new UI dashboard and the metrics-server itself. I'm also running multiple replicas of some services like the queue-worker which has two separate pods.

kubectl top pod -A
NAMESPACE     NAME                                        CPU(cores)   MEMORY(bytes)   
default       ingress-nginx-controller-54d8b558d4-59lj2   4m           96Mi            
grafana       grafana-5bcd5dbf74-rcx2d                    1m           22Mi            
kube-system   coredns-6c46d74d64-d8k2z                    5m           10Mi            
kube-system   local-path-provisioner-84bb864455-wn6v5     1m           6Mi             
kube-system   metrics-server-ff9dbcb6c-8jqp6              36m          13Mi            
openfaas      alertmanager-5df966b478-rvjxc               2m           6Mi             
openfaas      autoscaler-6548c6b58-9qtbw                  2m           4Mi             
openfaas      basic-auth-plugin-78bb844486-gjwl6          4m           3Mi             
openfaas      dashboard-56789dd8d-dlp67                   0m           3Mi             
openfaas      gateway-7c96d7cbc4-d47wh                    12m          24Mi            
openfaas      inlets-portal-client-5d64668c8d-8f85d       1m           5Mi             
openfaas      nats-675f8bcb59-cndw8                       2m           12Mi            
openfaas      pro-builder-6ff7bd4985-fxswm                1m           117Mi           
openfaas      prometheus-56b84ccf6c-x4vr2                 10m          28Mi            
openfaas      queue-worker-6d4756d8d9-km8g2               1m           2Mi             
openfaas      queue-worker-6d4756d8d9-xgl8w               1m           2Mi             
openfaas-fn   bcrypt-7d69d458b7-7zr94                     12m          16Mi            
openfaas-fn   chaos-fn-c7b647c99-f9wz7                    2m           5Mi             
openfaas-fn   cows-594d9df8bc-zl5rr                       2m           12Mi            
openfaas-fn   shasum-5c6cc9c56c-x5v2c                     1m           3Mi

Now we can see how well the pods are balanced across machines:

kubectl top node
NAME          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k3s-agent-1   236m         5%     908Mi           23%       
k3s-pi        528m         13%    1120Mi          14%  

So we actually have plenty of headroom to deploy some more workloads.

Now you cannot add any kind of "--watch" or "--follow" flag to this command, so if you want to watch it whilst you scale some functions or deploy a bunch of new pods, you need to use a bash utility like "watch".

Try this for example:

# Terminal 1
watch "kubectl top pod -A"

# Terminal 2
watch "kubectl top node"

The metrics-server is an optional add-on, which you can install with arkade or helm:

arkade install metrics-server

Have you turned it off and on again?

It's tragic, but true, that turnings things off and on again can fix many errors that we run into on a daily basis.

Restarting a deployment in Kubernetes may fix issues due to forcing your code to reconnect to services, pull down an updated image, or just release memory and database connections.

The command you're looking for is kubectl rollout restart.

Let's restart the 2x OpenFaaS queue-workers, whilst watching the logs with stern, and the events from the namespace in another window.

# Terminal 1
kubectl get events \
  -n openfaas
  --sort-by=.metadata.creationTimestamp \
  --watch

# Terminal 2
stern -n openfaas queue-worker.* --since 5s

# Terminal 3
kubectl rollout restart \
  -n openfaas deploy/queue-worker

Note the syntax for stern is a regular expression, so it'l match on anything that starts with "queue-worker" as a prefix. The --since 5s is similar to what we used with kubectl logs, to keep what we're looking at recent.

So we see our two new pods show up in stern, something that kubectl logs would not be able to do for us:

queue-worker-6d4756d8d9-km8g2 queue-worker 2022/06/01 10:47:27 Connect: nats://nats.openfaas.svc.cluster.local:4222
queue-worker-6d4756d8d9-xgl8w queue-worker 2022/06/01 10:47:30 Connect: nats://nats.openfaas.svc.cluster.local:4222

And the events for the queue-worker show the new pods being created and the older ones being removed from the cluster.

LAST SEEN   TYPE     REASON              OBJECT                               MESSAGE
69s         Normal   Pulling             pod/queue-worker-6d4756d8d9-xgl8w    Pulling image "ghcr.io/openfaasltd/queue-worker:0.1.5"
68s         Normal   Pulled              pod/queue-worker-6d4756d8d9-xgl8w    Successfully pulled image "ghcr.io/openfaasltd/queue-worker:0.1.5" in 597.180284ms
68s         Normal   Created             pod/queue-worker-6d4756d8d9-xgl8w    Created container queue-worker
68s         Normal   Started             pod/queue-worker-6d4756d8d9-xgl8w    Started container queue-worker
67s         Normal   ScalingReplicaSet   deployment/queue-worker              Scaled down replica set queue-worker-755598f7fb to 0
67s         Normal   SuccessfulDelete    replicaset/queue-worker-755598f7fb   Deleted pod: queue-worker-755598f7fb-h2cfx
67s         Normal   Killing             pod/queue-worker-755598f7fb-h2cfx    Stopping container queue-worker

What we didn't talk about

I need to check on applications for two reasons. The first is that I actually write software for Kubernetes such as openfaas and inlets-operator. During development, I need most of the commands above to check when things go wrong, or to see output from changes I've made. The second reason is that my company supports OpenFaaS and inlets users in production. Supporting users remotely can be challenging, especially if they are not used to troubleshooting applications on Kubernetes.

There are so many things that need to be checked in a distributed system, so for OpenFaaS users, we wrote them all down in a Troubleshooting guide and you'll see some of what we talked about today covered there.

In my experience, application-level metrics are essential to being able to evaluate how well a service is performing. With very little work, you can record Rate, Error and Duration (RED) for your service, so that when the errors are more subtle, you can start to understand what may be going on.

Metrics are beyond the scope of this article, however if you'd like to get some experience, in my eBook Everyday Go I show you how to add metrics to a Go HTTP server and start monitoring it with Prometheus.

Adding Prometheus metrics to a HTTP server in Everyday Go

You can learn more about how we use Prometheus in OpenFaaS in this talk from KubeCon: How and Why We Rebuilt Auto-scaling in OpenFaaS with Prometheus

I see Kubernetes as a journey and not a destination. I started my walk in 2017, and it was very difficult for me at the time. If, like me, you now have several years under your belt, try to have some empathy for those who are only starting to use the technology now.

You may also like these two free courses that I wrote for the Linux Foundation/CNCF:

• Kubernetes at the Edge with K3s
• Introduction to Serverless on Kubernetes

Kubernetes, K3s, OpenFaaS, Docker, Prometheus, Grafana and many other Cloud Native tools are written in Go, learn what you need to know with my eBook Everyday Golang

Got a comment, question or suggestion? Let's talk on Twitter

Do you need to know Spanish to get by in Spain? Probably not, with many younger people learning English in school and Google Translate on our phones, we can probably survive.

However, knowing a few words can help you get more out of your stay and make connections with people that you may meet.

I'm breaking down a few key words into categories, hope you find them helpful.

It's never too late to learn at least a few words, and you'll never know how helpful they could be during your stay.

How are you?

¿Cómo estás? How are you - informal
¿Cómo está? / ¿Cómo está usted? How are you - formal

Spanish has a concept of a formal and informal way to address others (tu vs usted), which is similar to French (tu vs vous), or German (Du vs Sie).

¡Hasta luego! See you later
¡Adios! Bye
¡Buenas noches! Good night

¡No te preocupes! - "don't worry" - something I heard said quite a bit (preocupar is the verb and it sounds like "preoccupied")

Paying for things

¿Cuánto cuesta? ("kwanto" "kwesta") - how much is it?

Now, you may not know your numbers, but at least you can express your intent. Perhaps say: "escribir lo, por favor" (write it down please)

La cuenta, por favor - the bill, when sitting down to a meal or having drinks.

Beware, you may sound odd asking for "la cuenta" in a "tienda" (shop) or "supermercado" (supermarket). There, you tend to need to ask for "un recibo" (sounds like receipt, doesn't it?)

Take a photo of your receipts for Concur when you get home!

Drinks

Un cafe - a coffee
Un té - a tea

una taza de té - a cup of tea, una taza de cafe - a cup of coffee

"z" is said like "th", like the way Amercians say "ma_th_"

"e" at the end of a word is said like "ey", like "h_ey_"

Un cafe con leche - a coffee with milk, similar in size to a latte
Un cortado - a short coffee finished with milk, you should try these
Un cafe solo - an espresso, a short black coffee
Un cafe sin leche - a black coffe, without milk
Un té con azucar - a tea with sugar

Leche de soja - Soya milk
Leche de almendras - Almond milk

Quisiera un cafe con leche de soja - I'd like a coffee with soy milk

See some other phrases involving coffee

agua - water
agua sin gas - still water
agua con gas - fizzy water
agua potable - drinkable water

A note from a local, Ivan Pedrazas:

Not all water in Spain is considered drinkable, even if you ask: "¿Qué es el agua potable?" (said: "pot-ab-ley"), the levels of chlorine may give you an upset stomach. Best stick to bottled water, buy a large bottle and take it to your hotel room.

I learned this the hard way when walking across Spain in my 20s for El Camino De Santiago. Me duele el estomago!

Do you like juice?

zumo de naranja - orange juice
zumo de manzana - apple juice

Once you've had a coffee or two, you may need the bathroom.

¿Dónde está el baño?

¿Dónde está el baño, por favor? - if you're not in a hurry, you can add a "por favor" to the end of the sentence, which means "please". I like to think of this like saying "for a favour".

Food

Desayuno - Breakfast
Almuerzo - Lunch
Cena - Dinner, the evening meal

Huevos fritos - fried eggs
Hamburguesa - Hamburger
Patatas fritos - fries (potatoes)

When I walked on my camino, it was common to eat a "menú del día" for lunch which may include fried egg and chips along with a yoghurt or dessert.

Jamón - serano ham, cured pork leg

Un bocadillo de jamón (said like boka-di-i-yo) - a ham sandwich.

Pan integral - wholewheat bread

You can generally ask for food by saying its name, then adding "por favor" to the end.

Un bocadillo de jamón, por favor.

Don't be surprised when your sandwich comes out without butter or mayo. It may come across a bit dry to us in the UK or USA, however the bread is usually softer to make up for this.

Vegetarian? Playing it safe?

Un de queso - a cheese sandwich

Days of the week

You may be getting to Spain on Sabado, having a rest on Domingo and a walk around on Lunes.

Spanish days correspond to the names of the planets.

• Lunes - Moon - Monday
• Martes - Mars - Tuesday
• Miércoles - Mercury - Wednesday
• Jueves - Jupiter - Thursday
• Viernes - Venus - Friday
• Sabado - Saturn - Saturday
• Domingo - dominicus - of the Lord (from Latin)

When did you arrive on this trip?

¿Cuándo llegaste? Llegué el Martes.

(el) noche - night
(el) día - day (bear in mind, despite ending in "a" this is a masculine word"
(la) mañana - the morning, or "mañana" - tomorrow
Hoy - today
Ayer - yesterday

Directions

Dónde está? Dónde está su hotel? Where's your hotel?

Dónde está el mercado? Where's the market?

Todo recto - keep going straight on
A la izquierda - on the left
A la derecha - on the right
Baja la calle - walk down the road

If you're looking for something, but aren't sure if there is one, you can say:

¿Dónde hay una farmacia? (Is there a pharmacy?) Or simply: (¿Hay una farmacia por aqui?): Is there a pharmacy around here?

If you've got some pain, you can say:

Me duele aquí (it hurts me here, pointing to where)

Tengo dolor a la cabeza - I have a headache (I have a pain of the head)

Social

De dónde eres? Where are you from?

Soy de Inglaterra - I'm from England
Soy de los Estados Unidos - I'm from the US
Soy de Alémania - I'm from Germany

Me gusta el ciclismo - I like cycling
Me gusta correr - I like running
Me gusta nadar - I like swimming

In Spanish there is a formal way to address people using "usted" and a less formal way using "tu". It's likely that you'll be forgiven for getting this wrong, and should try to focus on getting the message across.

Te gusta nadar? Do you like to swim? (asked to a peer)
Le gusta nadar? Do you like to swim? (formally)

Being comfortable

Tengo hambre (silent "h") - I'm hungry
Estoy cansado (for a male) - I'm tired
Tengo calor - I'm feeling warm
Tengo frío - I'm feeling cold
Tengo que ir al baño - I need to go to the toilet

Hace calor - it's warm
Hace frío - it's cold
Hace sol - it's sunny
Llueve - it's raining

Taking it all in

I've given you a few of the words that I've found useful in the past, and I hope you'll recognise some of these when you're in Spain for KubeCon.

You may be able to pick up a Spanish phrase book before you leave, in a shop near your hotel, or even order on Amazon to your hotel. These almost always have a "menu reader" at the back of the book to help with food and navigating things like intolerances and preferences.

I don't drink dairy, so I may say: "contiene lactosa?" Does it contain lactose? Gluten free would be: "sin gluten". Perhaps, "lleve trigo?" Does it contain wheat? This is literally like saying "does it wear wheat"? See also llevé

If you have a serious allergy, rather than an intolerance do be very careful.

One trick you can use if you forget a word is to simply ask:

"¿Qué cosa es eso? ¿Qué es esto?"

As a rule, I found that Spanish people enjoy it when we tourists make an effort, even if it's a small one, so have courage and try out some of the words we've covered today.

And if you head to Kubecon North America later in the year, you'll also be able to use Spanish in more places than you may realise. Last time I was over in San Diego I heard Spanish spoken in shops, cafes, the conference center and it can be more precise to ask someone a question in Spanish than in English at times. Especially if they're a native speaker.

I will almost never remember a new word until I write it down, so perhaps write down some of the words we've covered, or if you see a new one write it down and practice it. For pronounciation, you'll find help on Google Translate, and many videos on YouTube too.

Enjoy your trip and come and feel free to reach out to me on Twitter to meet up for a cafe con leche!

Memorias de Kubecon 2019 con @ewilde, @TheAxeR, @mccabejohn, @AngelBarrera92 y @paurosello pic.twitter.com/4ywxscur4F

— Alex Ellis (@alexellisuk) May 12, 2022

Of course, I am not Spanish, nor a native. If you've found a problem or want to suggest a more idomatic expression, send me a quick email to [email protected]

Thank you to Ivan Pedrazas for keeping me honest and checking over what I've written up.

What if Kubernetes is more like your kind of language?

I've also got a special offer for you if you'd like to do some reading and learning between talks or flights.

The first 50 people will get 20% off with code estudiante:

• Everyday Go - learn Go, the language used to write Kubernetes and Docker
• Serverless For Everyone Else - Learn how to make the most of functions with Node.js

Feel free to check out the reviews and scan the table of contents to make sure the material is a good fit for you.

Many of us own at least one Raspberry Pi, and if it's not doing duty as a media player, retro gaming console, blocking ads, or reporting the weather, then it may well be gathering dust.

I'm writing this article as an excuse for you to blow the dust off those microchips, and to put your pocket-sized silicon to work as the world's smallest API-driven cloud.

By following the instructions in this article, you'll be able to deploy scheduled tasks, webhook receivers, web pages and functions to your pocket-sized cloud from anywhere using a REST API.

My own pocket-sized cloud, running 7x applications 24/7 on a Raspberry Pi 3 in my lounge.

Outline:

• What's a full-size cloud look like?
• A pocket-sized definition
• Dust off your Raspberry Pi
• Set up faasd for API-driven deployments
• Try out a store function
• Explore the asynchronous invocation system
• Deploy with GitHub Actions
• Explore monitoring and metrics
• Conclusion and further resources

Join me for a live stream on YouTube - Friday 25th March 12.00-13.30 GMT - Your pocket-sized cloud with OpenFaaS and GitHub Actions

Follow me, or share & discuss this article on Twitter

What's a full-size cloud look like?

According to NIST's final defintion of cloud:

"cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction."

When I read this definition, it seems like the authors are describing a managed cloud provider like AWS, GCP or some installable software like Kubernetes or one of the major hypervisors.

Now, we cannot get AWS from our Raspberry Pis, but we can get quite close to large parts of the definition by installing Kubernetes to a cluster of Raspberry Pis, configuring a network storage layer and using netbooting for easy provisioning and reconfiguration of hosts. I talk about how to do this in my workshop: Netbooting workshop for Raspberry Pi with K3s

But we're not going to do that for two reasons. The first is that there's a shortage of Raspberry Pis, so this is really about using one of your existing devices, and putting it to work. The second is that there's a non-trivial cost to Kubernetes, which isn't entirely necessary for a pocket-sized cloud.

A pocket-sized definition

Our definition of cloud is going to be scaled down, to fit in your pocket, in less than 1GB of RAM. What we're building today is a place to run our code, which can be configured through an API - using containers for packaging, so that it's easy to automate and maintain. We will have a single point of failure in the architecture, but we'll counter that by using an NVMe SSD instead of an SD card, which extends our time to failure of storage. The software is easy to deploy, so we can reduce our Mean Time To Recovery (MTTR) to a few minutes.

Getting to API-driven deployments

I spend a bit too much of my time on Twitter, and have quite often seen tweets that go a bit like this:

"I've written a HTTP server in Go/Python/Node and deployed it to a VM somewhere, but now I don't know how to re-deploy it remotely"

In the good old days, we would use FTP, or SFTP to transfer code or binaries to production, and then an SSH connection or control-panel UI to restart the service. That is probably one of the simplest solutions available today, but it's hardly "API-driven".

What if you could package your code in a container and then deploy the new version with a curl request?

You may have heard of OpenFaaS - a project I started in 2016 to find a way to run functions on any cloud, on servers that I managed, with containers being the primitive, instead of zip files. I wanted to pick a timeout that suited my needs, instead of whatever made sense for the vendor's SLA. I wanted to use containers so that I could deploy and test my work locally.

Perhaps you've tried OpenFaaS, or maybe you just wrote it off because you didn't see a use-case, or felt that you "had to be ready for serverless"? Quite frankly, that's my own fault for doing a poor job of messaging. Let's try to fix that.

OpenFaaS is a platform for running HTTP servers, packaged in containers, with a management API. So we can use it for running our code, and the version of OpenFaaS that we'll explore today can also add stateful services like a Postgresql, MongoDB, Redis, or a Grafana dashboard.

The idea of running a HTTP server for someone was so compelling that Google Cloud released "Cloud Run" in 2018, and if you squint, they look very similar. It's a way to run a HTTP server from a container image, and not much else.

The OpenFaaS stack comes with a few core components - a gateway with a REST management API and built-in Prometheus monitoring and a queue-system built around NATS for running code in the background.

Conceptual overview of OpenFaaS

The templates system takes a lot of the drudgery away from building HTTP services with containers - the Dockerfile gets abstracted way along with the HTTP framework. You get a handler where you fill out your code, and a simple command to package it up.

The function store also provides a quick way to browse pre-made images, like machine learning models and network utilities like nmap, curl, hey or nslookup.

You can access either via:

$ faas-cli template store list

NAME                     SOURCE             DESCRIPTION
csharp                   openfaas           Classic C# template
dockerfile               openfaas           Classic Dockerfile template
go                       openfaas           Classic Golang template
java8                    openfaas           Java 8 template
java11                   openfaas           Java 11 template
java11-vert-x            openfaas           Java 11 Vert.x template
node14                   openfaas           HTTP-based Node 14 template
node12                   openfaas           HTTP-based Node 12 template
node                     openfaas           Classic NodeJS 8 template
php7                     openfaas           Classic PHP 7 template
python                   openfaas           Classic Python 2.7 template
python3                  openfaas           Classic Python 3.6 template
python3-dlrs             intel              Deep Learning Reference Stack v0.4 for ML workloads
ruby                     openfaas           Classic Ruby 2.5 template
ruby-http                openfaas           Ruby 2.4 HTTP template
python27-flask           openfaas           Python 2.7 Flask template
....

$ faas-cli template store pull ruby-http

For functions:

faas-cli store list

FUNCTION                                 DESCRIPTION
NodeInfo                                 Get info about the machine that you'r...
alpine                                   An Alpine Linux shell, set the "fproc...
env                                      Print the environment variables prese...
sleep                                    Simulate a 2s duration or pass an X-S...
shasum                                   Generate a shasum for the given input
Figlet                                   Generate ASCII logos with the figlet CLI
curl                                     Use curl for network diagnostics, pas...
SentimentAnalysis                        Python function provides a rating on ...
hey                                      HTTP load generator, ApacheBench (ab)...
nslookup                                 Query the nameserver for the IP addre...
SSL/TLS cert info                        Returns SSL/TLS certificate informati...
Colorization                             Turn black and white photos to color ...
Inception                                This is a forked version of the work ...
Have I Been Pwned                        The Have I Been Pwned function lets y...
Face Detection with Pigo                 Detect faces in images using the Pigo...
Tesseract OCR                            This function brings OCR - Optical Ch...
...

$ faas-cli store deploy hey

Now, because Kubernetes is too big for our pocket, we'll run a different version of OpenFaaS called "faasd". Where full-size OpenFaaS targets Kubernetes, which eventually, through many layers of indirection, runs your HTTP server in a container, faasd runs your container directly.

OpenFaaS on Kubernetes

The result of tearing away all those layers is something that remains API-driven, is immensely quick and the learning curve goes down dramatically. It'll no longer be highly-available, clustered, or multi-node, so think of it more like an appliance. Do you have 3 toasters or ovens in your kitchen, just in case one of them fails?

The distance faasd creates from Kubernetes also makes it very stable - there's not much to go wrong, and very few breaking changes or "deprecations" to worry about. Upgrading a faasd appliance is a case of replacing the "faasd" binary and perhaps the containerd binary too.

OpenFaaS scheduling containers with faasd and containerd

There are end-users who deploy faasd instead of OpenFaaS on Kubernetes for production. faasd is a great way to deploy code for an internal intranet, client project, or an edge device. Package it as a Virtual Machine, load in your functions and you can pretty much forget about it.

I spoke at Equinix Metal's Proximity event last year on "Getting closer to the metal" (going lower down in the stack, away from Kubernetes). The talk was based around faasd and things we could do that weren't possible with K8s. For instance, we were able to get the cold start to sub-second and schedule 100x more containers on a single node, making it more cost effective.

Dust off that silicon

You need at least a Raspberry Pi 3 which has 1GB of RAM for functions or the Raspberry Pi 4, which usually has at least 2GB of RAM available. The Raspberry Pi Zero W 2 will also work, but only has 512MB RAM.

See also: First Impressions with the Raspberry Pi Zero 2 W

I recommend either the "classic" Raspberry Pi 32-bit Operating System Lite edition (Buster) or Ubuntu Server 20.04. Bullseye does work, however there are a few changes in the OS, that you can work around by using Buster.

Flash the image to an SD card using either dd from a Linux host, or a graphical UI like etcher.io. I have a Linux host set aside that I use for flashing SD cards because I do it so often.

Now we are almost ready to boot up and get headless access, without a keyboard or monitor. For the Raspberry Pi OS, you will have to create a file named ssh in the boot partition. For Ubuntu, SSH is already enabled.

Connect to your Raspberry Pi via ssh ssh [email protected] for Raspberry Pi OS, and run nmap -p 22 192.168.0.0/24 if you used Ubuntu, this will show you any new hosts on your network that you can connect to over SSH.

Change the default password and the hostname, for Raspberry Pi OS, use raspi-config

Install faasd

Installing faasd from that SSH session is relatively simple:

git clone https://github.com/openfaas/faasd --depth=1
cd faasd

./hack/install.sh

At the end of the installation, you'll get a command for how to find the password.

Now because our little cloud is API-driven, we should not run any more commands on it directly, but use it as a server from our laptop.

Go over to your workstation and install the OpenFaaS CLI:

# Omit sudo if you wish, then move faas-cli to /usr/local/bin/
curl -SLs https://cli.openfaas.com | sudo sh

Log into your cloud:

export OPENFAAS_URL=http://IP:8080
faas-cli login --password-stdin

Type in the password from the previous session and hit enter.

Try out a function from the store

Now you can deploy a sample function from the function store. This is a blocking command and it may take a few seconds to complete

faas-cli store deploy nodeinfo

Then list your functions, describe it and invoke it:

$ faas-cli list -v
Function                        Image                                           Invocations     Replicas
nodeinfo                        ghcr.io/openfaas/nodeinfo:latest                0               1  

$ faas-cli describe nodeinfo

Name:                nodeinfo
Status:              Ready
Replicas:            1
Available replicas:  1
Invocations:         0
Image:               
Function process:    node index.js
URL:                 http://127.0.0.1:8080/function/nodeinfo
Async URL:           http://127.0.0.1:8080/async-function/nodeinfo

$ curl http://127.0.0.1:8080/function/nodeinfo
Hostname: localhost

Arch: arm
CPUs: 4
Total mem: 476MB
Platform: linux
Uptime: 5319.62

Any function can be invoked asynchronously, which is important for long running functions - scheduled tasks and webhook receivers.

$ curl -d "" -i http://127.0.0.1:8080/async-function/nodeinfo
HTTP/1.1 202 Accepted
X-Call-Id: 9c766581-965a-4a11-9b6d-9c668cfcc388

You'll receive an X-Call-Id that can be used to correlate requests and responses. An X-Callback-Url is optional, and can be used to post the response to a HTTP bin, or some other function that you have, to create a chain.

For instance:

# Netcat running on 192.168.0.68
$ nc -l 8888

$ curl -d "" -H "X-Callback-Url: http://192.168.0.86:8888/" \
    -i http://127.0.0.1:8080/async-function/nodeinfo

HTTP/1.1 202 Accepted
X-Call-Id: 926c4181-6b8e-4a43-ac9e-ec39807b0695

And in netcat, we see:

POST / HTTP/1.1
Host: 192.168.0.86:8888
User-Agent: Go-http-client/1.1
Content-Length: 88
Content-Type: text/plain; charset=utf-8
Date: Wed, 23 Mar 2022 10:44:13 GMT
Etag: W/"58-zeIRnHjAybZGzdgnZVWGeAGymAI"
Keep-Alive: timeout=5
X-Call-Id: 926c4181-6b8e-4a43-ac9e-ec39807b0695
X-Duration-Seconds: 0.086076
X-Function-Name: nodeinfo
X-Function-Status: 200
X-Start-Time: 1648032253651730445
Accept-Encoding: gzip
Connection: close

Hostname: localhost

Arch: arm
CPUs: 4
Total mem: 476MB
Platform: linux
Uptime: 5577.28

Build your own function

You can deploy a predefined HTTP server that listens to requests on port 8080, use a custom Dockerfile, or an openfaas template from the store.

Building and Deploying Functions, credit: Ivan Velichko

Create a function called "mystars" - we'll use it to receive webhooks from GitHub when someone stars one of our repos:

export OPENFAAS_PREFIX="ghcr.io/alexellis"
faas-cli new --lang node16 mystars

This creates:

mystars.yml
mystars/handler.js
mystars/handler.json

The mystars.yml defines how to build and deploy the function.

The mystars/handler.js defines what to do in response to a HTTP request.

'use strict'

module.exports = async (event, context) => {
  const result = {
    'body': JSON.stringify(event.body),
    'content-type': event.headers["content-type"]
  }

  return context
    .status(200)
    .succeed(result)
}

This is very similar to AWS Lambda, and you can find the full documentation in the eBook Serverles For Everyone Else, and a more limited overview in the docs at: Node.js templates (of-watchdog template)

Then you can install any NPM modules you may need like octokit by running cd mystars and run npm install --save

To try out the function on your Raspberry Pi, build it on your own host, using Docker, publish the image to your GHCR account and then deploy it.

faas-cli publish -f mystars.yml \
  --platforms linux/amd64,linux/arm64,linux/arm/7

You can remove the other platforms if you want quicker builds, but the above cross-compiles your function to run on a 32-bit ARM OS, 64-bit OS like Ubuntu and a PC like your workstation/laptop or a regular cloud server.

Once published, deploy the function with:

faas-cli deploy -f mystars.yml

Finally, you can invoke your function synchronously, or asynchronously just like we did earlier with the nodeinfo function.

Want more code examples?

• For JavaScript and Node.js see: Serverless For Everyone Else
• For Golang (Go), see: Everyday Go, the final chapter focuses on functions written in Go

There's also plenty of blog posts on the OpenFaaS site on using Python, Java, C#, existing Dockerfiles etc.

CI/CD with GitHub Actions

For a pocket-sized cloud, we want to do as much automation as possible, which includes building our images, and deploying them. My favourite solution is GitHub Actions, but you can use similar techniques with whatever you're most familiar with.

GitHub Actions, plus GitHub's Container Registry ghcr.io make a perfect combination for our functions. Any public repos and images are free to build, store and deploy later on.

In a simple pipeline, we can install the faas-cli, pull down the templates we need and publish our container images.

I've separated out the build from the deployment, but you could combine them too.

Rather than using mystars.yml, rename the file to stack.yml, so that we have one less thing to configure in our build.

name: build

on:
  push:
    branches:
      - '*'
  pull_request:
    branches:
      - '*'

permissions:
  actions: read
  checks: write
  contents: read
  packages: write

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@master
        with:
          fetch-depth: 1
      - name: Get faas-cli
        run: curl -sLSf https://cli.openfaas.com | sudo sh
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
      - name: Get TAG
        id: get_tag
        run: echo ::set-output name=TAG::latest-dev
      - name: Get Repo Owner
        id: get_repo_owner
        run: >
          echo ::set-output name=repo_owner::$(echo ${{ github.repository_owner }} |
          tr '[:upper:]' '[:lower:]')
      - name: Login to Container Registry
        uses: docker/login-action@v1
        with:
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GITHUB_TOKEN }}
          registry: ghcr.io
      - name: Publish functions
        run: >
          OWNER="${{ steps.get_repo_owner.outputs.repo_owner }}" 
          TAG="latest"
          faas-cli publish
          --extra-tag ${{ github.sha }}
          --platforms linux/arm/v7

In order to replace the default tag of "latest" for your container image in stack.yml, and change the field: image: ghcr.io/alexellis/mystars:latest to image: ghcr.io/alexellis/mystars:${TAG:-latest}.

This next part deploys the function. It runs whenever I create a release in the GitHub repo, and will deploy the latest image using the SHA pushed to GHCR.

name: publish

on:
  push:
    tags:
      - '*'

permissions:
  actions: read
  checks: write
  contents: read
  packages: read

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@master
        with:
          fetch-depth: 1
      - name: Get faas-cli
        run: curl -sLSf https://cli.openfaas.com | sudo sh
      - name: Pull template definitions
        run: faas-cli template pull
      - name: Get TAG
        id: get_tag
        run: echo ::set-output name=TAG::latest-dev
      - name: Get Repo Owner
        id: get_repo_owner
        run: >
          echo ::set-output name=repo_owner::$(echo ${{ github.repository_owner }} |
          tr '[:upper:]' '[:lower:]')
      - name: Login to Container Registry
        uses: docker/login-action@v1
        with:
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GITHUB_TOKEN }}
          registry: ghcr.io
      - name: Login
        run: >
          echo ${{secrets.OPENFAAS_PASSWORD}} | 
          faas-cli login --gateway ${{secrets.OPENFAAS_URL}} --password-stdin
      - name: Deploy
        run: >
          OWNER="${{ steps.get_repo_owner.outputs.repo_owner }}"
          TAG="${{ github.sha }}"
          faas-cli deploy --gateway ${{secrets.OPENFAAS_URL}}

The final part - the deployment can be made over an inlets HTTPS tunnel.

Your inlets URL can be configured via a secret called OPENFAAS_URL along with your password for the OpenFaaS API in OPENFAAS_PASSWORD.

For more on inlets see:

• When Your ISP Won't Give You A Static IP
• Automated HTTP tunnel server

The above examples are from Serverless For Everyone Else, you can see the example repo here: alexellis/faasd-example

Triggering your functions

If you set up an inlets tunnel, then you'll have a HTTPS URL for the OpenFaaS API, and potentially a number of custom domains for your functions on top of that.

For instance, with my Treasure Trove function the insiders.alexellis.io domain maps to http://127.0.0.1:8080/function/trove in OpenFaaS.

So you can provide a URL to an external system for incoming webhooks - from Stripe, PayPal, GitHub, Strava, etc or you can share the URL of your custom domain, like I did with the Treasure Trove. In the example I mentioned above, I provided a URL to IFTTT, to send tweets in JSON format, which are then filtered before being sent off to a Discord channel.

It's up to you whether you want to use the synchronous or asynchronous URL. If your function is slow, then the originating system may retry the message, in that instance, the asynchronous URL would be better.

The UI is served on port 8080, but can be put behind a TLS-terminating reverse proxy like Caddy or Nginx.

Trying out the OpenFaaS UI

After HTTP, the second most popular way to invoke a function is via a cron schedule.

Imagine that you had a function that ran every 5 minutes, to send out password reset emails, after having queried a database for pending rows.

It might look a bit like this:

version: 1.0
provider:
  name: openfaas
  gateway: http://127.0.0.1:8080
functions:
  mystars:
    lang: node16
    handler: ./mystars
    image: ghcr.io/alexellis/send-password-resets:latest
    topic: cron-function
    schedule: "*/5 * * * *"
    secrets:
    - aws-sqs-secret-key
    - aws-sqs-secret-token
    - postgresql-username
    - postgresql-password
    environment:
      db_name: resets

In addition to the code, you can supply a number of secrets and environment variables to configure the function.

You can add stateful containers to faasd by editing its docker-compose.yaml file. Don't be confused by that name, Docker and Compose are not used in faasd, but we do use the same familiar spec to define what to run.

Here's how we add NATS, with a bind-mount so that its data is kept between reboots or restarts:

  nats:
    image: docker.io/library/nats-streaming:0.22.0
    user: "65534"
    command:
      - "/nats-streaming-server"
      - "-m"
      - "8222"
      - "--store=file"
      - "--dir=/nats"
      - "--cluster_id=faas-cluster"
    volumes:
      - type: bind
        source: ./nats
        target: /nats

You can learn how to configure cron schedules, stateful services, secrets and environment variables in Serverless For Everyone Else.

Monitoring

OpenFaaS emits metrics in Prometheus format for its own API and any functions that you invoke through the OpenFaaS gateway.
Browse the metrics in the docs: OpenFaaS Metrics

I built a simple dashboard is monitoring several key functions:

• Derek - installed on various GitHub organisations, derek helps reduce maintainer fatigue by reducing repetitive work
• Treasure Trove - my portal for GitHub sponsors - access all my technical writing back to 2019 and discounts on my eBooks
• Filter-tweets - triggered by If This Then That (IFTTT) - this function reads a JSON body, filters out spam and tweets by the openfaas account or by myself, then forwards the message onto a discord channel.
• Stars - installed as a webhook on the inlets, openfaas GitHub organisations and on some of my personal repos. It receives JSON events from GitHub and translates these to well-formatted Discord messages

Built-in monitoring

The errors, or non-200/300 messages are from Derek, when the code receives an event from GitHub that it cannot process.

Wrapping up

I wrote this article to give you a taste of what kinds of things you can do with a pocket-sized cloud. My goals were to make it small, nimble, API-driven and most importantly, attainable. Kubernetes is an extremely versatile platform, but requires dozens more moving parts to run it in production.

Follow me, or share & discuss this article on Twitter

This isn't a Highly-Available (HA) or Fault Tolerant setup, but it's not meant to be either. One thing you can do to make things better, is to switch to a an NVMe SSD instead of using an SD card: Upgrade your Raspberry Pi 4 with a NVMe boot drive

If you think this concept is interesting and useful for your side projects, or a client project, you can also set up all the software on a cloud like DigitalOcean, Linode or AWS using an EC2 instance.

One of my functions running on faasd

I've been running the Treasure Trove on a Raspberry Pi 3 since 2019, I don't remember any downtime, but if the host went down for some reason, I'd simply run 3-5 commands and redeploy from a GitHub Action. If I was present at home, this could probably be completed within 30 minutes.

How many AWS or Kubernetes-related outages have you recovered from in 30 minutes?

For a deep dive on OpenFaaS, check out Ivan Velichko's write up: OpenFaaS - Run Containerized Functions On Your Own Terms

What about Kubernetes?

Kubernetes, with K3s is more suitable if you want to run more than device, or really fancy setting up your own cluster and getting closer to the NIST definition of "cloud".

You'll need 3-5 Raspberry Pis, so that you can tolerate at least one device failure, which will be a bigger investment, and eat up more of your compute power. But you may find the skils you built with your RPi cluster are useful at work, or your next gig. Here's my guide to Self-hosting Kubernetes on your Raspberry Pi

If you're interested in learning more about netbooting, I wrote up a workshop, a video tutorial and a set of automation scripts: Netbooting workshop for Raspberry Pi with K3s

Join us for a live event

On Friday 25th March at 12.00-13.30 GMT, I'll be joined by Martin Woodward - GitHub's Director of DevRel to build a pocket-sized cloud using a single Raspberry Pi. Together, we'll show you how to combine faasd and GitHub Actions for remote deployments of scheduled tasks, APIs and web portals.

Hit subscribe & remind, so you can watch live, or get a notification when the recording goes up:

Friday 25th March 12.00-13.30 GMT - Your pocket-sized cloud with OpenFaaS and GitHub Actions

Whenever I work on a local or remote Kubernetes cluster, I tend to want to connect to my application to send it a HTTP request or something similar.

There are a number of options for doing this and if (like me), you work with Kubernetes on a daily basis, it's important to know the most efficient option and the best fit for our own needs.

Let's say that we're trying to access a Grafana dashboard? We could use any application here that receives HTTP traffic. TCP traffic is similar, but one option I'll show you (Ingress) won't work for TCP traffic.

I will use arkade to fetch the CLIs that we are going to need for the examples, but you are welcome to do things the "long way" if you prefer that. (Searching for a README, following lots of links, looking for the latest stable version, downloading it, unzipping it etc)

arkade is an open-source Kubernetes marketplace for helm charts, apps and DevOps CLIs

arkade get kind
arkade get [email protected]

kind create cluster

# Install Grafana via helm with sane defaults
arkade install grafana

This creates a Service of type ClusterIP, which we cannot access from outside of the cluster:

kubectl get svc -n grafana
NAME      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
grafana   ClusterIP   10.96.19.104   <none>        80/TCP    9d

Let's take a look at LoadBalancers, NodePorts, port-forwarding and Ingress. As a spoiler: port-forwarding tends to be the lowest common denominator which will work on any cloud or local distribution mostly seamlessly. For production, you will almost always deploy an Ingress Controller and expose port 80 and 443 via a managed LoadBalancer.

I'm not going to cover any specific feature of a local distribution such as KinD's extra port mappings or Minikube add-ons.

Let's keep this focused on Kubernetes API objects and approaches that work everywhere, without implying that one local or remote distribution is better than another.

On a side-note, did you know that minikube can now run in Docker instead of using VirtualBox? For me, that puts it on par with KinD and K3d. Both minikube and KinD both work really well on Apple Silicon with Docker Desktop. Another bonus for me.

LoadBalancer

A TCP load-balancer is offered by most managed clouds, you can allocate a port such as 8080, 443, etc and have a piece of infrastructure created to allow access to your Service.

For Grafana, the port of the internal service is 80, so we'd create a LoadBalancer either through YAML or through kubectl expose.

kubectl expose -n grafana deploy/grafana \
  --target-port=80 \
  --port=8080 \
  --name grafana-external \
  --type LoadBalancer

service/grafana-external exposed

This will create an LB for port 8080 and we'll have to pay AWS or GKE ~ 15-20 USD / mo for this.

Here's the equivalent YAML via kubectl get -n grafana service/grafana-external -o yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: grafana
  name: grafana-external
  namespace: grafana
spec:
  ports:
  - nodePort: 30180
    port: 8080
    protocol: TCP
    targetPort: 80
  selector:
    app.kubernetes.io/instance: grafana
    app.kubernetes.io/name: grafana
  type: LoadBalancer
status:
  loadBalancer: {}

The status field will be populated with a public IP address when using a managed K8s engine.

However, if we're running locally, we will never get an IP address populated and our LoadBalancer will be useless.

There's two solutions here:

1. Install the inlets-operator, which will create a virtual machine on a public cloud and connect us using an inlets Pro tunnel. The advantage here is that the LB will be just like a cloud LB, and accessible publicly. The inlets tunnel can work on any kind of network since it uses an outgoing websocket, even on a captive WiFi portals, VPNs, and behind HTTP proxies. See also: arkade install inlets-operator
2. Option two is to use something like the MetalLB project (see arkade install metallb). This project can allocate a local IP address to the LB and advertise it on our local network range with ARP. At that point we can access it locally on our own LAN at home, but not anywhere else. If we're on the road and plug our laptop into another network with a different network range, or connect to the VPN or go to the office, then we will probably run into issues.

Unfortunately, exposing a HTTP service over a LoadBalancer is less than ideal because it adds no form of encryption such as TLS. Our traffic is in plaintext. In the final section I will link you to a tutorial to combine Ingress with a LoadBalancer for secure HTTPS traffic on port 443.

NodePorts

A NodePort is closely related to a LoadBalancer, in fact if you look at the YAML from our previous example, you'll note that LoadBalancers require a node-port to be allocated to operate.

A NodePort is a port that is allocated in a high port range such as 30080. Any machine in your cluster that receives traffic on port 30080 will forward it to the corresponding service.

Pros: works on most public clusters, no need to pay for an LB.
Cons: doesn't always work on Docker Desktop due to the way networking is configured. Doesn't work well with KinD. Port number looks suspicious.

kubectl expose -n grafana deploy/grafana \
  --target-port=80 \
  --port=8080 \
  --name grafana-external \
  --type NodePort

service/grafana-external exposed

The NodePort will be allocated randomly. My service got: 32181.

Here's the equivalent YAML via kubectl get -n grafana service/grafana-external -o yaml

If you write your own YAML file, you can specify a port for the NodePort, but make sure it doesn't clash with others that you've created.

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: grafana
  name: grafana-external
  namespace: grafana
spec:
  ports:
  - nodePort: 30080
    port: 8080
    protocol: TCP
    targetPort: 80
  selector:
    app.kubernetes.io/instance: grafana
    app.kubernetes.io/name: grafana
  type: NodePort
status:
  loadBalancer: {}

Note that we are still working with plaintext HTTP connections, with no encryption.

Port-forwarding

Port-forwarding is my preferred option for accessing any HTTP or TCP traffic within a local or remote cluster. Why?

Pros: works with remote and local clusters, is encrypted by default using TLS. No special steps required
Cons: clunky user-experience, often disconnects, always needs to be restarted when a pod or service is restarted. Will not load-balance between replicas of a deployment, binds only to one pod.

So there are some significant cons with this approach, namely that if you are redeploying or restarting a service which you forwarded, then you have to kill the port-forward command and start it over again. It's a very manual process.

kubectl port-forward \
 -n grafana \
 svc/grafana 8080:80

You must also pay attention to ports, and avoid clashes:

Unable to listen on port 8080: Listeners failed to create with the following errors: [unable to create listener: Error listen tcp4 127.0.0.1:8080: bind: address already in use unable to create listener: Error listen tcp6 [::1]:8080: bind: address already in use]
error: unable to listen on any of the requested ports: [{8080 3000}]

So let's change the port to 3001 instead:

kubectl port-forward \
 -n grafana \
 svc/grafana 3001:80
 
Forwarding from 127.0.0.1:3001 -> 3000
Forwarding from [::1]:3001 -> 3000

Now access the service via http://127.0.0.1:3001

But what if you want to access this port-forwarded service from another machine on your network? By default it's only bound to localhost for security reasons.

curl -i http://192.168.0.33:3001
curl: (7) Failed to connect to 192.168.0.33 port 3001: Connection refused


kubectl port-forward \
 -n grafana \
 svc/grafana 3001:80 \
 --address 0.0.0.0
 
Forwarding from 0.0.0.0:3001 -> 3000

Now try again:

curl -i http://192.168.0.33:3001/login

HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Type: text/html; charset=UTF-8
Expires: -1
Pragma: no-cache
X-Frame-Options: deny
Date: Fri, 04 Feb 2022 11:20:37 GMT
Transfer-Encoding: chunked

<!DOCTYPE html>
<html lang="en">

Exercise caution with --address - it will allow any machine on your network to access your port-forwarded service.

But there's still an issue here. Whenever we restart grafana because of a config change, look what happens:

kubectl rollout restart -n grafana deploy/grafana
deployment.apps/grafana restarted

kubectl port-forward -n grafana svc/grafana 3001:80 --address 0.0.0.0
Handling connection for 3001

E0204 11:21:49.883621 2977577 portforward.go:400] an error occurred forwarding 3001 -> 3000: error forwarding port 3000 to pod 5ffadde834d4c96f1d0f634e23b87a6c3816faade8645a5f78715f27390929df, uid : network namespace for sandbox "5ffadde834d4c96f1d0f634e23b87a6c3816faade8645a5f78715f27390929df" is closed

Now the only solution is find the tab running kubectl, kill the process with Control + C and then start over with the command again.

When you're iterating on a service over and over, as I often do, this grows tedious very quickly. And when you're port-forwarding 3 different services like OpenFaaS, Prometheus and Grafana it's that pain x3.

what if we had three replicas of the OpenFaaS gateway, and then port-forwarded that with kubectl?

arkade install openfaas
kubectl scale -n openfaas deploy/gateway --replicas=3
kubectl port-forward -n openfaas svc/gateway 8080:8080

Unfortunately, if you have three replicas of a Pod or Deployment, kubectl port-forward will not load-balance between them either. So is there a smarter option?

Smarter port-forwarding

With inlets, you can set up your local machine (laptop) as a tunnel server, then deploy a Pod to the cluster running a client. This is the opposite way that inlets is usually used.

inlets was originally created in 2019 to bring tunnels to containers and Kubernetes, so you could expose local services on the Internet from any network.

The K8s cluster will run the inlets client which is a reverse proxy. The server portion on my local machine will send requests into the cluster and then they will be send onto the right pod.

This port-forwarding with inlets only uses up one port on your host, and will allow you to access a number of services at once. If you restart pods, that's fine, there's nothing for you to do unlike with kubectl port-forward.

Inlets will also load-balance between any pods or services within your cluster, where as kubectl port-forward only binds to a single port, which is why it has to be restarted if that pod gets terminated.

inlets.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inlets-client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: inlets-client
  template:
    metadata:
      labels:
        app: inlets-client
    spec:
      containers:
      - name: inlets-client
        image: ghcr.io/inlets/inlets-pro:0.9.3
        imagePullPolicy: IfNotPresent
        command: ["inlets-pro"]
        args:
        - "http"
        - "client"
        - "--url=wss://192.168.0.33:8123"
        - "--upstream=prometheus.svc.local=http://prometheus.openfaas:9090"
        - "--upstream=gateway.svc.local=http://gateway.openfaas:8080"
        - "--upstream=grafana.svc.local=http://grafana.grafana:80"
        - "--token=TOKEN_HERE"
        - "--license=LICENSE_HERE"
---

Set the --token and --license then run: kubectl apply -f inlets.yaml. Change the IP from 192.168.0.33 to your local machine's IP.

Check the logs, it's detected our upstream URLs and is ready to go. You can see where each hostname will go to in the cluster including the namespace.

kubectl logs deploy/inlets-client

2022/02/04 10:53:52 Licensed to: alex <[email protected]>, expires: 37 day(s)
2022/02/04 10:53:52 Upstream: prometheus.svc.local => http://prometheus.openfaas:9090
2022/02/04 10:53:52 Upstream: gateway.svc.local => http://gateway.openfaas:8080
2022/02/04 10:53:52 Upstream: grafana.svc.local => http://grafana.grafana:80
time="2022/02/04 10:53:52" level=info msg="Connecting to proxy" url="wss://192.168.0.33:8123/connect"
time="2022/02/04 10:53:52" level=info msg="Connection established" client_id=65e97620ff9b4d2d8fba29e16ee91468

Now we run the server on our own host. It's set up to use encryption with TLS, so I've specified the IP of my machine 192.168.0.33.

inlets-pro http server \
  --auto-tls \
  --auto-tls-san 192.168.0.33 \
  --token TOKEN_HERE \
  --port 8000

The --port 8000 value sets port 8000 as where we will connect to access any of the forwarded services.

Now, edit /etc/hosts and add in the three virtual hosts we specified in the inlets-client YAML file and save it.

127.0.0.1	prometheus.svc.local
127.0.0.1	gateway.svc.local
127.0.0.1	grafana.svc.local

I can now access all three of the services from my machine:

curl http://prometheus.svc.local:8000
curl http://gateway.svc.local:8000
curl http://grafana.svc.local:8000

If you have more services to add, just edit the above steps and redeploy the inlets client. The tunnel server doesn't need to be restarted.

Note that none of this traffic is going over the Internet, and it is ideal for a local development environment with WSL, Docker Desktop and VMs.

Ingress

Kubernetes Ingress is the last option we should cover.

Ingress isn't actually a way to access services directly, it's more of a way to multiplex connections to different back-end services.

Think of upstreams in Nginx, or VirtualHosts in Apache HTTP Server.

To make use of Ingress, you need to install an IngressController, of which there are dozens.

The most popular (in my opinion) are:

• ingress-nginx
• Traefik

There are many many others.

Once an Ingress Controller is deployed, you then have a chicken and egg problem again. It has a TCP port 80 and 443, and you have to expose that somehow.

Your options are the same as above:

To access it publicly: a managed cloud LoadBalancer or inlets-operator

To access it locally: MetalLB, NodePorts or port-forwarding.

Remember that NodePorts don't give out good ports like 80 and 443, do you really wan to go to https://example.com:30443? I didn't think so.

With Kubernetes, so many things are possible, but not advisable. You can actually change security settings so that you can bind NodePorts to port 80 and 443. Do not do this.

You can install ingress-nginx with arkade:

arkade install ingress-nginx

You'll then see its service in the default namespace:

kubectl get svc ingress-nginx-controller
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.96.213.226   <pending>     80:30208/TCP,443:31703/TCP   3d20h

This is type of LoadBalancer, because it's expected.

I'm going to install the inlets-operator and have it provision IPs for me on DigitalOcean VMs.

arkade install inlets-operator \
  --provider digitalocean \
  --region lon1 \
  --token-file ~/do-token.txt

I got the API token do-token.txt from DigitalOcean's web portal and created a token with Read/Write scope.

Now, almost as soon as that command was typed in, I got a public IP:

kubectl get svc -n default -w
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP                   PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.96.213.226   178.128.36.66   80:30208/TCP,443:31703/TCP   3d20h


curl -s http://178.128.36.66

Not found

Let's create a basic Ingress record that will always direct us to Grafana.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana
  namespace: grafana
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - backend:
          service:
            name: grafana
            port:
              number: 80
        path: /
        pathType: Prefix
status: {}

This ingress record matches any traffic and then sends to to grafana.grafana:80 within our cluster.

Typically, when we're using Ingress it's to get some benefit of a reverse proxy like being able:

• apply rate limits
• authenticate and gate requests
• load-balance between services
• multiplex a number of domains under one IP or LoadBalancer
• encrypt traffic with a TLS certificate

I've just given you a brief example of how to use Ingress, but you can follow this tutorial to see how to use multiple domains and how to get TLS records:

• Tutorial: Expose a local IngressController with the inlets-operator

What about a "Service Mesh"? Linkerd advocates for using ingress-nginx or similar to get incoming network access. Istio ships its own Gateway called an IstioGateway which can replace Ingress on Kubernetes. For an example of Istio with OpenFaaS see: Learn how Istio can provide a service mesh for your functions (adapt as necessary for your workloads).

Wrapping up

We looked at LoadBalancers, NodePorts, kubectl and inlets port-forwarding and Ingress. This is not an extensive guide, but I hope you have a better handle on what's available to you and will go off to learn more and experiment.

If, like me you do a lot of application development, you may find inlets useful and more versatile vs kubectl port-forward. We showed how to multiplex 3 different HTTP services from our cluster, but inlets also supports TCP forwarding and a developer at UK Gov wrote to me to explain how he used this to debug NATS from a staging environment.

You may also like: Fixing the Developer Experience of Kubernetes Port Forwarding

So what is my preferred setup?

For a production cluster, serving websites and APIs over HTTP - the "de facto" setup is a managed cloud LoadBalancer to expose TCP port 80 and 443 from your Ingress Controller.

As I explained in the introduction, our focus is on local development techniques that work for every kind of local Kubernetes cluster. We are not trying to tie our destinies to KinD or Minikube or Microk8s etc.

I will often set up the inlets-operator, cert-manager and ingress-nginx on my local environments so that I can get full TLS certs and custom domains during local development. This mirrors the recommended configuration for a production setup and also works with Istio.

I also use port-forwarding (kubectl port-forward) a lot, as I mentioned in the introduction - it's the lowest common denominator and, if you have access to kubectl, it will work everywhere.

I use either a Linux desktop with Docker CE installed and KinD for Kubernetes, or my Apple M1 MacBook Air with Docker Desktop and KinD. Arkade is smart enough to download the right version of KinD, kubectl, inlets-pro and etc by looking at the output of uname -a.

More recently, I've found myself needing access to Grafana, OpenFaaS and Prometheus all at once, and my example with an inlets client running as a Pod gave me a huge productivity boost over kubectl. Your mileage may vary. You can try inlets on a monthly subscription with no long-term commitment. There are other tools available that specialise in port-forwarding for local development, but I am biased since I created inlets to solve problems that I was facing which these do not.

Getting more in-depth experience

If you want to gain experience with Kubernetes objects and APIs, why not try the course I wrote for the LinuxFoundation? It covers all the basics whilst also introducing you to K3s:

Introduction to Kubernetes on Edge with K3s

I also have another course commissioned by the Cloud Native Computing Foundation (CNCF) that covers Serverless on Kubernetes with OpenFaaS being used for many of the more detailed examples:

Introduction to Serverless on Kubernetes

Premium materials

I also have my own premium learning materials available in eBook, code sample and video format. Feel free to browse the table of contents and see what others are saying.

Learn use-cases for functions and how to build our own with Node.js: Serverless for Everyone Else

Start contributing to Cloud Native projects like Kubernetes and OpenFaaS, or write your own code in Go with my book - Everyday Go

Join me for a walk through seven of my highlights from 2021 - of independent business, community-building, hobbies and book recommendations.

In this article I'll share my highlights of 2021, but if you don't know where I'm coming from, you can get a quick recap with 2020's edition: Still in the game: My 2020 Year in Review.

"Still in the game" was a reference to Simon Sinek's concept of finite and infinite games. Many corporations and large brands talk about "winning" and "loosing" - but Simon challenges this as flawed rhetoric. You can only win if there is an end to the game, and a set of rules - governed and enforced by some well defined entity. So my article was about my second year of independent business, still learning, earning and still doing business. If there is no way to win, and no game then Simon hints that following a mission and continuing to participate are what we must focus on instead.

In 2019 when I started my own company, I was hard on myself. I vastly underestimated how long it would take to start building a client base and set of customer stories. There's so much still to learn, and several unresolved problems that I'm wrestling with - like funding for my open source work and projects.

I've become more comfortable with the ebb and flow of a business that is primarily based upon non-recurring revenue. You eat what you kill. In one month I may receive next to nothing financially, yet 30 days after that, I may see as much as a quarter of that year's total revenue deposited into the company account.

This year I've also been able to slowly convert some income from high-value marketing consulting to licensing and support. Why? Because my primary goal is to serve the needs of OpenFaaS users.

Making plans is overrated. I insulated this space to be my Cloud Native shed, and now it's my woodworking workshop. Things change, being adaptable is important to staying in the game.

So on to this year's highlights.

• Getting paid for more of the work I'm doing
• Seeing inlets become a product people pay for
• Starting and seeing the impact of Growlab
• Rediscovering woodworking and making things with my hands
• Becoming an eBook author
• Learning the fundamentals of positioning, sales and the business mindset
• Doubling down on my strategy for staying in the business of making OpenFaaS

Build it and they will come

As entrepreneurs, our mission is to build something of value for a group of people, so useful that they would pay money for that saving in time, that utility, that increase in status or productivity.

On the other hand: many open source projects start out of scratching our own itch. Even Apple started from Steve and Wozniak wanting to own computers and to be able to democratise them for their friends.

A discipline that's paid off for me this year: reading, note-taking and experimentation.

So on one hand, business books will teach that one must first deeply understand a market, their problems, their dreams, their daily routines - as much as possible, before starting to build a solution to help them. And on the other - open source and some business success stories talk about a need or pain point that we ourselves know deeply.

Whatever way you go about this, from personal experience, or observing as an outsider - I saw a common theme in what I read. Experiment, prototype, test, iterate and defer.

And the most important change for this year was to get paid for the work I was doing, the value I was creating, the service I was giving.

inlets - network tunnels reimagined for Cloud Native

I pivoted inlets this year from open source, with a separate commercial version, to a commercial version with open-source automation and tooling. Why? The Open Source model wasn't serving the needs of my business, all the value was given away for free in the open source version and only a few people saw value in paying for my time, skill and expertise.

Inlets is a network tunnel that runs in userspace, is self-contained in a single binary and integrates deeply with containers and Kubernetes.

So I flipped this around. If you only used inlets because it was free, find an alternative (there are a bunch of them, with different tradeoffs) or pay for it. Teams were happy to pay the full price for inlets, and for 12 months up front, but the other segment of users were personal users like you and me. They were less happy to pay for 12 months up front, even if the target market was developers earning well over $100k a year.

I didn't want to build a complex billing, subscription and tax collection system, but I kept hearing that monthly billing was important. Carlos, the creator of Goreleaser mentioned that he'd just started selling his software using Gumroad's license server. So after taking a look at the API, I added support that day.

Read about it here: Introducing the inlets monthly subscription

In a short period of time, inlets gained its first two dozen subscribers.

There was a little bit of churn here and there, and I also noticed that some people started to abuse the subscription - by paying for the month, then cancelling it immediately. Then a few weeks later paying for a month and cancelling it again. Eventually I reached out to one of these developers and he told me that he was "gaming the system" and only wanted to use the product some months of the year.

This could be a sign that the price for inlets is too low and attracting people who don't value it enough.

In the past, these people used to abuse the free trial - requesting a trial whenever they needed to use the software. One of the other things I did was to remove the free trial - so that users either pay for one month and invest something of themselves, or use an alternative that they are happy also meets their needs.

I want to be in business for a long time, that means that a proportion of people who only use my things because they are free, will leave. I'm OK with that.

Growlab

I had previously incubated seeds indoors and seen better germination rates than outdoors. I'd even recorded the process with a Raspberry Pi in the past and created videos from it. This year was a very cold and windy start. So much so that many of the seeds I'd bought and planted just didn't convert into seedlings.

I wanted an enclosure to hold soil whilst the seeds had a chance to grow in the warmth, but I also wanted to record the progress with a camera, mounted over the top. I went out into the garden and realised that I didn't have a clue what I was doing. I spoke to a few friends and eventually decided that some copper pipe glued into a wooden base might do.

The first growlab, one of many.

Given that I'd invested some time and resources into the project, I decided to build a community around it. First, I started sharing my own updates and using the #growlab hashtag - then I noticed people asking me questions about it. The Raspberry Pi and technological aspect was interesting to other developers and with the pandemic, many people were interested in growing their own food also.

Having learned a lot about sales and value that year, I thought I'd give the people a "What's in it for me" - primarily a contest, where they could win prizes by completing a set of achievements. The gamification worked quite well, and by the end of the year, there were 30 participants.

On the growlab website, I wrote up a few benefits that I'd heard people had received from the project:

• Learning new skills
• Connecting with others
• Being mindful in the present moment
• Giving to others

Each participant was called a technician - a citizen scientist who had built his or her own incubator - not only growing seeds, but sharing updates and encouraging others to get involved.

Checkout the Growlab website

Woodworking renaissance

The word "renaissance" means "rebirth" and to some extent that's what happened to me with woodworking. Growlab was the catalyst for a renewed interest in working with wood and creating things with my hands. Many British children learn how to work with wood at an early age, but without the tools or motivation, this was something that I wasn't able to continue after leaving school.

For most of the year, I'd been focused on management and marketing, rather than on making. So when I realised that creating things with my hands gave me a similar sense of satisfaction to coding, it seemed like a natural hobby to invest in.

But it's not a hobby for the sake of it. A friend of mine had made a desk using tongue and groove flooring from a local big box store and I was so impressed, that I wanted to try my own projects too.

Here's a few of my projects from this year:

One of my favourite projects - an outdoor seating bench with tusked mortise and tenon joinery.

The plans:

The cut list and plans for the bench

I wouldn't have even known where to start with this last year.

I made my first wrist-rest for my mechanical keyboard using very primitive tools and found that other people wanted these too.

One of my earliest wrist rests

You can buy one here.

The highlight of my woodworking journey has to be the three Butcher Blocks that I was able to gift to family members as Christmas presents.

My wife holding her bonus Christmas gift of a Butcher Block made by yours truly.

Becoming an eBook author

I wrote and launched my first eBook in January 2021 - Serverless For Everyone Else. It's a handbook for building functions in JavaScript with faasd - the version of OpenFaaS that can run on a single VM or Raspberry Pi. Reception was better than expected and I matched the total sponsorships via GitHub for the OpenFaaS account in one month of sales.

I then went back to my business plan and ideas that I'd written down in 2019 when setting up the business. I wanted to write a book about my Go blog posts that had done so well throughout the years. Refreshing, updating and extending the most popular content on crafting CLIs, unit-testing like your live depended on it and integrating metrics and monitoring. It was called Everyday Go and you can read a bit more about it here: I wrote a book about Everyday Go

Read more about my self-publishing journey: I wish I'd self-published sooner

All the books you can read

Around May time, I set up an Amazon Wishlist called "Books I'd like to read" and something unexpected happened. People started to send me books from the list and I got used to seeing an Amazon Prime driver come by the house every few days.

For each book, sent to me I write the name and date on the front cover. That means I'll think of you whenever I pick it up.

Some of my favourite books from this year:

• The 1-Page Marketing Plan: Get New Customers, Make More Money, And Stand Out From The Crowd by Alan Dibb
• 7 Strategies for Wealth & Happiness: Power Ideas from America's Foremost Business Philosopher - Jim Rohn
• Zig Ziglar's Secrets of Closing the Sale
• Million Dollar Consulting by Alan Weiss

Alan Dibb helped me understand pricing, premium branding, unique selling points and positioning.

Jim Rohn helped me understand that seasons can apply to business in the same way that they did to growlab. He suggests going with the flow, and having a mindset of belief and action. There are many recordings on YouTube that you can listen to such as: How to Take Charge of Your Life

Zig Ziglar helped me understand the difference between price and cost, between suspects and prospects - people ready to waste my time & resources and people who might be a good fit for my business.

Alan Weiss helped hammer home the difference between selling time and selling value. This mindset is ultimately what gave me the confidence to build a premium consulting brand and to know when to walk away from a deal.

I also found fresh inspiration for my entrepreneurial journey through Tony Robbins' podcast. He interviews successful founders who are not afraid to share their struggle and where they've come from.

Now, someone asked me on Twitter whether I read the books people send me. For one - the books are those that I've chosen ahead of time and prioritised on my wishlist. Many of them have a core message that you receive after reading 3-5 chapters and then go on to solidify throughout the remaining pages.

If you'd like to know my thoughts on each, you can find my reviews and insights in my Insiders' Updates Archive going back to 2019: Insiders' Portal

Changes for OpenFaaS

A couple of weeks ago, I wrote about Operation Getting Paid - a broad strategy to move from giving away my time, expertise and knowledge for free, to being paid instead.

This has involved more investment into OpenFaaS Pro - a commercial distribution of OpenFaaS for production use. Some consulting to understand more deeply the challenges faced by OpenFaaS users and to inform the roadmap, and to automate/offload some of the support demand from the project through training materials like Serverless For Everyone Else.

faasd has seen its API filled out, now there's very little that it cannot do. faasd is also being used to run my social media, Gumroad and Derek integrations.

OpenFaaS Pro now has a function builder that can scale well for large users - where they don't want to maintain 2000-5000 GitLab jobs, but instead want an API that exchanges code for a container image. The OpenFaaS Kafka integration has been hardened and tuned with customers in production and is happily churning away at millions of records per hour for them.

We celebrated the 5th Birthday for OpenFaaS this month. Not only have we served the needs of many users in production, but we've also helped community contributors to grow and in some instances triple their salary, way above even my own earnings at the time.

Learn how OpenFaaS impacted the serverless community through the lens of its contributors.

Watch now:

GopherCon UK 2021: Alex Ellis - Zero to OpenFaas in 60 months

Wrapping up

I'd like to thank my clients for taking a risk on an independent entrepreneur and for coming back for repeat business. Thanks to my GitHub Sponsors for generating just enough recurring revenue so that I can continue to dedicate a portion of my time to open source. So if you'd like to see that continue, you can support it here.

If you're building your own independent business or open source project and want to chat with me, I can offer you a 1:1 via my Calendly page. Insiders save around 50% on the cost and a 20% discount code for all my eBooks on Gumroad.

Today the Raspberry Pi Foundation released the Raspberry Pi Zero 2 W. I'm going to share my thoughts and experiences with it. Can you run containers? Can you run OpenFaaS? What about K3s? Is it worth buying or will it gather dust?

I've got my faasd t-shirt on and you'll find out why below.

The Raspberry Pi Foundation sent me one of these boards for testing earlier in the year. The original Zero / Zero W had a CPU that was only compatible with armv6 architectures and became quickly underpowered for my Node.js and Go applications.

Note: this blog post is best viewed on a PC, some images will not show if you're on a mobile device

What's new?

It's been several years since I first reviewed the original Raspberry Pi Zero W, so what's changed and what's it good for?

Twinning - on the left, the original and on the right the new Zero 2.

The new version has the same processor as the original Raspberry Pi 3. From the official press release:

Priced at $15, Raspberry Pi Zero 2 W uses the same Broadcom BCM2710A1 SoC die as the launch version of Raspberry Pi 3, with Arm cores slightly down-clocked to 1GHz, bundled into a single space-saving package alongside 512MB of LPDDR2 SDRAM. The exact performance uplift over Zero varies across workloads, but for multi-threaded sysbench it is almost exactly five times faster.

Unfortunately the team were unable to upgrade the RAM to 1GB, which would have made this a smaller Raspberry Pi 3. Yes there are a lot of variations of the Raspberry Pi, so where does this fit?

As I mentioned, running Node.js used to take several seconds to launch, and compiling Go programs could take minutes, where as on a Raspberry Pi 4 these tasks would complete much quicker. This meant that the Raspberry Pi was often last picked for new projects, especially when I wanted to use infrastructure projects.

Over time, many open source projects (including OpenFaaS that I maintain) dropped support for its ARMv6 processor due to its slow speed and issues in the ecosystem.

That lead me to ask the question of what to do with my original Raspberry Pi Zeros. I'm going to give you a quick refresher on what they do well, before getting hands on with the newer version and answering the questions that I know you have.

What do we do with all these slow RPi Zeros?

A stack of the original Raspberry Pi Zeros

I remember meeting up with the founders of BitScope and we were on our way to see Eben Upton, wondering what we could do with all the Raspberry Pi Zeros that we had in our collections.

We couldn't use them for clustering because they were too slow, and support was deprecated in many infrastructure projects, sometimes unintentionally.

My ZumoPi robot commissioned by Pimoroni and Linux User and Developer Magazine

Where they had really shined was in IoT and robotics projects where space and power were on a premium.

My growlab build was the first, but it inspired 20 others to build their own and join me

This year I launched the growlab project to help people all around the world build a lab to grow and monitor seeds and plants. The original Raspberry Pi Zero was a perfect fit here due to its size, cost and plethera of interfacing options: from 40 GPIO pins, to add-ons and a camera port. Its built-in WiFi adapter meant I could place a growlab anywhere with power.

The fruits of my labour, but no Raspberries this year!

The Raspberry Pi Zeros had a temperature sensor connected to them and a wide-angle camera, so you could capture images like this and upload them to a static webpage to share your progress with the community.

Community images from the growlab project

Next, I wanted to aggregate the sensor data in a time-series database. My tool of choice for microservice metrics is Prometheus, but for sensor data InfluxDB is better suited with long term storage and retention policies. It's also fairly light weight. Even so, it could not really fit on my Raspberry Pi along with Grafana, a tool that is often used to visualise data.

So I set up a Raspberry Pi 4 with 2GB of RAM and used the faasd project to host Grafana, InfluxDB and a couple of functions for ingesting data.

The following app runs on the Raspberry Pi Zero: sender/main.py

And then you run this function on the RPi 4 with OpenFaaS: submit-sample/handler.py

The result is that you can then get a beautiful set of graphs showing the environmental data:

I had insulated my shed and installed a Raspberry Pi Zero with a sensor running the sender/main.py code. It was very very hot in there.

You can see how the functions and faasd setup works here: Data Logger with faasd

All the code for the growlab projects are available on GitHub

You can find out more about growlab here: https://growlab.dev and in my video presentation from Equinix Metal's GIFEE event

Enter the RPi Zero 2 W

We can actually run containers directly on the Raspberry Pi Zero 2 W, because it supports the ARMv7 architecture, which has much better support across the board.

Self-hosted GitHub Actions runners

Even GitHub provide a self-hosted actions runner for ARMv7, and I tried it out this morning:

See the steps are running on my new RPi Zero 2 in my house

Why would this be useful? Well I guess that the GitHub Actions team saw their product as a CI tool for building binaries, but it has become much more and we don't have time to go into that here. Needless to say, it's become more of a general purpose task runner and a scaleable solution for running batch jobs and extending the GitHub platform.

At the beginning of the post I showed you a picture of me wearing my faasd t-shirt and here's why.

Serverless Raspberries

Earlier in the year, I tried really hard to rebuild OpenFaaS, containerd, CNI and all the various components needed as armv6 containers and in the end I had to give up and admit defeat. It was just too slow and too much work, for no real reward.

You can read about how far I got here: One last trip down memory lane with the Raspberry Pi Zero

faasd always worked on the Raspberry Pi 3, and now that the two shared the same CPU, it meant I could probably run faasd on this tiny device again.

A screenshot showing containerd and OpenFaaS running

You can also install fully-fledged Docker on these devices using the well-known bash script: curl -sLS https://get.docker.com | sh or by using a package manager.

You'll probably want to adjust your /boot/config.txt file to squeeze out a few more MB of usable RAM. Just add gpu_mem=16 to ring out a few megabytes from the GPU, it won't be needing them when running headless.

So whilst the GitHub Actions Runner dials home, and then receives control messages from GitHub's control-plane, what if you wanted to host some functions or webpages and access them from your Raspberry Pi Zero 2?

Fortunately, inlets has you covered there. Unlike tools like Ngrok, it's actually built for use in production and integrates extremely well with containers.

I ran the inlets-pro client on my computer using a tunnel server that I created on DigitalOcean.

Feel free to try out my sample functions over the tunnel:

$ curl -sL https://zero2.exit.o6s.io/function/nodeinfo

Hostname: localhost

Arch: arm
CPUs: 4
Total mem: 477MB
Platform: linux
Uptime: 350

$ curl -sL https://zero2.exit.o6s.io/function/figlet -d `whoami`
       _ 
 _ __ (_)
| '_ \| |
| |_) | |
| .__/|_|
|_|      

$ curl -sL https://zero2.exit.o6s.io/function/cows

         (__)       (----------)
         (--) . . . ( *>YAWN<* )
   /------\/        (----------)
  /|     ||
 * ||----||
Cow after pulling an all-nighter

Why not try out the cows function in your browser?

There's templates for building your own functions in Node.js, Go, JavaScript, Bash, Python and other languages too. See an example of each on the homepage.

We've also published a set of multi-arch functions to the store which you can see via faas-cli store list. Here's the store's GitHub repo to see how they work with common CLIs like curl, ffmpeg, nslookup, hey, nmap and so forth containerised and ready to run on your RPis: openfaas/store-functions

faas-cli list shows the amount of invocations so far on the various functions.

Of course OpenFaaS isn't just about having fun. It's got plenty of real-world use-cases, that you can learn about: Exploring Serverless Use-cases from Companies and the Community

I wrote a blog post about setting up faasd on a Raspberry Pi, and toured some of the features like asynchronous invocations and multi-arch container images. It's now much much easier to install, but you may like seeing what else it can do: faasd - lightweight Serverless for your Raspberry Pi

You can set up your own tunnel with custom domains using the guides below:

• Serve traffic through a private self-hosted tunnel

Inlets can also turn your RPi Zero into a handy file sharing server:

• The simple way to share files directly from your computer

To learn how to setup faasd on your Raspberry Pi and build functions with practical examples, try my eBook: Serverless For Everyone Else

What about K3s?

I would not recommend running K3s on a Raspberry Pi 3 due to its well-known limits on I/O. These cause timeouts and break K3s when running with etcd, the tool used to maintain state across the cluster.

If you're only wanting to run a single node Kubernetes cluster, then you may have a better experience with faasd which is free and open source and foregoes any bogging down with clustering.

That said, I thought you'd like to see if it would install.

Running a Kubernetes cluster over WiFi is a terrible idea. I added a USB network adapter for this experiment.

You need to append cgroup_memory=1 cgroup_enable=memory to the /boot/cmdline.txt file and reboot before attempting this.

k3sup can be used to install K3s via SSH and merge a new context into your KUBECONFIG for use with kubectl. All without logging into the RPi itself.

k3sup install \
  --ip 192.168.0.87 \
  --user pi \
  --k3s-channel v1.22 \
  --merge \
  --context rpizero2 \
  --local-path $HOME/.kube/config
  

Merging with existing kubeconfig at /home/alex/.kube/config
Saving file to: /home/alex/.kube/config

# Test your cluster with:
export KUBECONFIG=/home/alex/.kube/config

To my surprised, it actually worked, but seemed to get really bogged down starting up, taking multiple seconds to return kubectl get nodes

Shortly after it choked.

In the logs for the k3s service, I saw very high latency from etcd:

Oct 28 11:17:56 zero-pi k3s[484]: I1028 11:17:56.769385     484 trace.go:205] Trace[1920479996]: "GuaranteedUpdate etcd3" type:*v1.Endpoints (28-Oct-2021 11:17:54.460) (total time: 2308ms):
Oct 28 11:17:56 zero-pi k3s[484]: Trace[1920479996]: ---"initial value restored" 196ms (11:17:54.657)
Oct 28 11:17:56 zero-pi k3s[484]: Trace[1920479996]: ---"Transaction prepared" 1606ms (11:17:56.263)
Oct 28 11:17:56 zero-pi k3s[484]: Trace[1920479996]: ---"Transaction committed" 504ms (11:17:56.768)

Followed by kubectl get nodes eventually timing out.

$ kubectl get node
Unable to connect to the server: net/http: TLS handshake timeout

Then I saw hundreds of lines of errors.

It's probably safe to say that whilst this device had no issues running faasd, it's just not suitable for K3s, even with a single node. The limited RAM and I/O is just not a good combination for a container orchestor built to run on much more powerful hardware. Use RPi4s instead.

See also: Self-hosting Kubernetes on your Raspberry Pi

Building a little Go program

This uses Go 1.17 to build arkade which has very few dependencies, and can be used to download CLIs in a similar way to homebrew, but for binaries. It also has app installers for around 40 Kubernetes projects.

sudo rm -rf /usr/local/go

sudo apt install -qy git

curl -SL -o go.tgz https://golang.org/dl/go1.17.2.linux-armv6l.tar.gz

sudo mkdir -p /usr/local/go
sudo tar -xvf ./go.tgz -C /usr/local/go --strip-components=1
export PATH=$PATH:/usr/local/go/bin

Clone arkade:

git clone https://github.com/alexellis/arkade --depth=1
cd arkade

# Resolve and download dependencies according to go.mod
time go mod download

# Build a binary
time go build

Here's the original RPi Zero

pi@zero-pi-1:~/arkade $ time go mod download

real	5m54.399s
user	3m51.653s
sys	1m23.257s

pi@zero-pi-1:~/arkade $ time go build

real	5m29.314s
user	4m37.119s
sys	0m40.533s

And the new model:

pi@zero-pi-2:~/arkade$ time go mod download

real	0m44.392s
user	0m40.020s
sys	0m5.177s

pi@zero-pi-2:~/arkade$ time go build

real	0m50.969s
user	1m48.109s
sys	0m10.250s
pi@zero-pi-2:~/arkade$ 

And an RPi 4:

pi@k4s-1:~/arkade $ time go mod download

real	0m33.653s
user	0m41.719s
sys	0m10.816s

pi@k4s-1:~/arkade $ time go build

real	0m20.983s
user	0m45.630s
sys	0m8.521s

So we have a difference of 5m54.399s vs 0m50.969s for just downloading and resolving a few dependencies from the go.mod file. That is a marked improvement, but still not ideal. Even the RPi 4 took 33 seconds for this operation.

For the build, things were even worse, with the original RPi Zero taking real 5m29.314s which is so long that I almost gave up waiting. The RPi Zero 2 completed in 0m50.969s

Just for fun, I also built arkade on my Apple MBA M1:

bash-3.2$ time go mod download

real	0m6.409s
user	0m2.748s
sys	0m1.223s

bash-3.2$ time go build

real	0m0.174s
user	0m0.260s
sys	0m0.286s

Even the RPi 4 is relatively sluggish. Who wants to wait 20s between changing code and running it? For me, cross-compiling Go on a faster computer is a better alternative.

Here's a snippet from arkade's Makefile, which is how we cross-compile inside a GitHub Action:

	CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -ldflags $(LDFLAGS) -a -installsuffix cgo -o bin/arkade-armhf

That binary can then be copied to the Raspberry Pi using scp or downloaded via HTTP endpoint.

Building hello-world instead

Let's compare these results to a trival app I wrote for my eBook Everyday Go. It simply accepts a couple of version flags during a build, then outputs them back to the user at runtime.

package main

import "fmt"

var (
	Version   string
	GitCommit string
)

func main() {
	fmt.Printf("release-it: %s, commit: %s\n", Version, GitCommit)
}


git clone https://github.com/alexellis/release-it --depth=1
cd release-it

time go build -mod=vendor

RPi Zero 1: real 0m5.638s
RPi Zero 2: real 0m1.622s
RPi 4: real 0m0.766s

So building Go programs is much much faster on the RPi Zero 2, but it's still not responsive enough for me and I would recommend building your code on a regular Intel computer and copying the binary across for testing.

In fact, that's what we do for arkade in the release process which you can see here: arkade/.github/workflows/publish.yml

Before I leave you, let's see how quickly Node.js will launch on the new processor?

curl -sLS -o node.tar.xz https://nodejs.org/dist/v16.13.0/node-v16.13.0-linux-armv7l.tar.xz
sudo tar -xvf node.tar.xz -C /usr/local/ --strip-components=1

cat <<EOF > index.js

console.log("Hi " + process.env.USER)

EOF

time node index.js

To further put my point home about armv6, the Node.js project has stopped building Node for the original RPi Zero, so I had to use an unofficial build here: https://unofficial-builds.nodejs.org/download/release/v14.10.0/node-v14.10.0-linux-armv6l.tar.xz

RPi Zero 1: real 0m1.551s
RPi Zero 2: real 0m0.557s
RPi 4: real 0m0.355s

1.5s to load Node.js, with a program with zero dependencies is far too long. The RPi Zero 2 brings that down to half a second, which I think most people could proably tolerate. This number will be higher if there is more that needs to be loaded in at start-up.

Why is loading Node.js important? I mentioned the GitHub Actions runner earlier, that uses a bundled copy of Node.js to execute some of your workflows.

Wrapping up

Should you buy a Raspberry Pi Zero 2 W?

Building and running microservices and functions in Go and Node.js will be faster. You can run more tools than before like the GitHub Actions self-hosted runner, and various open source projects which only build for ARMv7 CPUS. In short, it will cope better with pretty much anything you throw at it.

If you'd like to run K3s or experiment with containers and clustering, you need a Raspberry Pi 4, ideally with 2GB of RAM or more. I have write up on clustering here: State of netbooting Raspberry Pi in 2021

What about 64-bit OSes like Ubuntu? According to my sources, a 32-bit OS suits this board better, so who are we to argue? I was told that the 64-bit OS is slower for most things compared to a 32-bit OS. Booting the graphical desktop taking an extra second, building the kernel taking ~ 2s more, and launching a webpage in Chrome taking ~ 5s more. Do you really need a 64-bit ARM OS, even if it's demonstrably slower? Well, if you're absolutely sure, then an RPi4 may be a better choice.

For faasd and serverless functions, you can try it out on your RPi Zero 2, but the minimum I would recommend is an RPi 3 or 4. You will thank me later, when you can pack more functions in and get a more responsive experience.

For sensors and low-powered operations that may need to run from a battery, the original Raspberry Pi Zero still may be the best option from a power consumption point of view. According to my Kill-O-Watt, my Zero 2 is drawing 4.8W at idle, and around 8.1W during the build of arkade. The original RPi Zero draws 2.3W at idle, around half of that.

If you're reading this, then the chances are that (just like me) you've already ordered one. There are more things you can run on it than with the older generation, I'm sure you'll have fun playing with it.

What are you going to do with yours? Follow me on Twitter @alexellisuk if you'd like to join the discussion and find out more.

You may also like:

• A Tour of Inlets - A Tunnel Built for the Cloud
• State of netbooting Raspberry Pi in 2021
• Will it cluster? k3s on your Raspberry Pi

There's been some talk on Twitter recently about a new feature emerging on GitHub Actions. It allows an action to mint an OpenID Connect (OIDC) token, which can then be used to deploy artifacts into other systems and clouds.

I'll give you a bit of context, then show you the AWS and GCP story, followed by how I integrated this with OpenFaaS so that a set list of users on GitHub could deploy to my OpenFaaS cluster without having to give them any credentials.

What's the point of federation?

I rememeber seeing a keynote in 2018 at KubeCon where Kelsey Hightower demoed an integration between two services with one of them running on AWS and the other running on GCP. Normally that would mean producing a number of secrets and configuring them on either side. His demo was different because there was a level of trust between the two sides, fedration if you will.

The Oxford dictionary describes "federation" as:

the action of forming states or organizations into a single group with centralized control.

Centralised control within a single group sounds useful and just like Kelsey showed us GCP and AWS working in harmony, later on I'll show you GitHub Actions deploying to OpenFaaS without any shared credentials.

This video is definitely worth a watch, even if you were there live. He explains how serverless platforms play well with Kubernetes, integrating with managed cloud services.

Kelsey was probably using GCP's Workload Identity Federation technology.

AWS has a similar technology with its IAM (Identity Access Management), IdP (Identity Provider) and Secure Token Service (STS).

Why could credential sharing be an anti-pattern?

For some systems, credentials are tied to a human identity. We've all probably created a GitHub Personal Access Token PAT) once or twice in our time. These tokens are equivalent to us taking actions, and most of the time have very coarse-level permissions, i.e. "read/write to all repos".

If I were to use a PAT in an integration, then I left the company, my access token would still be in use, and my identity would be tied to that. On the other hand, if I left and deactived my account, the integration would stop working.

Even when API tokens and service accounts decouple identity from access tokens, there's still the need to share, store and rotate these secrets which presents a risk. The less of this we do, the lower the risk of something going wrong.

What does it look like?

Here's what the author of the GCP integration said:

Before federation:

1. Create a Google Cloud service account and grant IAM permissions
2. Export the long-lived JSON service account key
3. Upload the JSON service account key to a GitHub secret

After:

1. Create a Google Cloud service account and grant IAM permissions
2. Create and configure a Workload Identity Provider for GitHub
3. Exchange the GitHub Actions OIDC token for a short-lived Google Cloud access token

In short, the token and identity that GitHub Actions provides is enough to deploy to GCP or AWS when configured in this way. That means using the SDK, CLIs, Terraform and other similar tooling. It could probably also be made to work with Kubernetes authentication and authorization.

The GCP example

The GCP example is where I learned the most about how this works, since GitHub haven't documented the API yet.

This GitHub Action exchanges a GitHub Actions OIDC token into a Google Cloud access token using Workload Identity Federation. This obviates the need to export a long-lived Google Cloud service account key and establishes a trust delegation relationship between a particular GitHub Actions workflow invocation and permissions on Google Cloud.

From reading the source code:

The Action is provided with two variables: ACTIONS_ID_TOKEN_REQUEST_URL and ACTIONS_ID_TOKEN_REQUEST_TOKEN

By posting ACTIONS_ID_TOKEN_REQUEST_TOKEN to ACTIONS_ID_TOKEN_REQUEST_URL, you receive a JSON response containing an undocumented count variable and a value property

Next, the resulting JWT token is used with Google's STS service to create a short-lived token for Google Cloud.

Then from what I read, that token can be used with the gcloud CLI and so forth.

Some prior configuration is required with a service account on the Google side.

See Seth Vargo's example here: @google-github-actions/auth

The AWS example

Aidan Steele appears to have done the heavy lifting here.

Aidan writes:

GitHub Actions has new functionality that can vend OpenID Connect credentials to jobs running on the platform. This is very exciting for AWS account administrators as it means that CI/CD jobs no longer need any long-term secrets to be stored in GitHub. But enough of that, here’s how it works:

He also mentions:

At the time of writing, this functionality exists but has yet to be announced or documented. It works, though!

That's important to note, because right when I was in the middle of demoing an integration with OpenFaaS, the team changed a URL for where the tokens are issued from. It's probably not a good idea to put this straight into production, until its announcement, if it should make it to GA. I hope it does!

Aidan shares an example of the JWT he received from the OIDC endpoint if you'd like to poke around.

If you're an AWS customer, try Aidan's example here: AWS federation comes to GitHub Actions

After writing this article, I reached out to Aiden who told me this was a feature he requested in early 2020, you can see the GitHub Support request here.

Making it work with OpenFaaS

Now both of the examples we've seen give you the ability to access the APIs of Google Cloud or AWS - that's very powerful, but also very broadly scoped.

I wondered if I could follow the same principles and make an authentication plugin for OpenFaaS Pro that would act in the same way.

First of all, I wrote a tiny HTTP server in Go, and deployed it on my machine to print out webhooks.

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"os"
)

func handler(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "OK")
	fmt.Fprintf(os.Stdout, "Method: %s Path: %s \nHeaders:\n", r.Method, r.URL.Path)

	for k, v := range r.Header {
		fmt.Fprintf(os.Stdout, "%s=%s", k, v)
	}

	if r.Body != nil && r.ContentLength > 0 {
		defer r.Body.Close()
		fmt.Fprintf(os.Stdout, "\nBody (%d bytes):\n", r.ContentLength)

		io.Copy(os.Stdin, r.Body)
	}
	fmt.Fprintf(os.Stdout, "\n")

}

func main() {
	http.HandleFunc("/", handler)
	log.Println("Starting server on 8080")
	log.Fatal(http.ListenAndServe(":8080", nil))
}

Want to learn patterns and practices for writing idiomatic Go including microservices, CLIs and building Go via GitHub Actions? See my new book: Everyday Go

Then I ran an inlets tunnel which gives me a secure, private and self-hosted way to receive webhooks. I didn't want this token going through the Ngrok or Cloudflare tunnel shared servers.

inlets-pro http client --token=$TOKEN \
  --url=wss://minty-tunnel.exit.o6s.io \
  --upstream http://127.0.0.1:8080 \
  --license-file $HOME/.inlets/LICENSE \
  --auto-tls=false

Getting the first OIDC token

Next I wrote an Action to dump out environment variables. I was wondering if the token would already be minted and available.

Note that the id-token: write permission must be given in order to obtain an OIDC token from the built-in endpoint.

name: federate

on:
  push:
    branches:
    - '*'

jobs:
  auth:

    # Add "id-token" with the intended permissions.
    permissions:
      contents: 'read'
      id-token: 'write'

    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@master
        with:
          fetch-depth: 1
      - name: Dump env
        run: env

After running it, I noticed that it wasn't, but two things caught my eye:

• ACTIONS_ID_TOKEN_REQUEST_URL
• and ACTIONS_ID_TOKEN_REQUEST_TOKEN

So I decided to post that token to the given URL. Then with some searching around, I saw the Google Aaction did the same.

Another bonus to hitting "." is that you get the search from VSCode which is easier to use and faster IMHO than using the GitHub UI

This runs on your local machine, so no need to pay for or wait for a Codespace to launch. pic.twitter.com/Lbq8UnSs9E

— Alex Ellis (@alexellisuk) October 6, 2021

Rather than searching in the UI, I hit . which is a new GitHub UI shortcut. It opens VSCode in a client-side only experience that has a much better search capability.

So then I ran the action:

name: federate

on:
  push:
    branches:
    - '*'

jobs:
  auth:

    # Add "id-token" with the intended permissions.
    permissions:
      contents: 'read'
      id-token: 'write'

    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@master
        with:
          fetch-depth: 1
      - name: Dump env
        run: env
      - name: Post the token
        run: |
          OIDC_TOKEN=$(curl -sLS "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=minty.exit.o6s.io" -H "User-Agent: actions/oidc-client" -H "Authorization: Bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN")

          curl -i -s --data-binary "$OIDC_TOKEN" \
            https://minty.exit.o6s.io/github-oidc

Bear in mind that the aud field should always be set to the URL that will consume the token, in this case: minty.exit.o6s.io.

Within the microservice, after validating the JWT against the issuer's public key, you should validate then that the aud field is set as expected.

See also: JWT RFC 7519

Checking out the fields

To my surprise, it worked.

I saw a response like this:

{
  "count: 3118,
  "value": "VALID_JWT_TOKEN",
}

Next, I extracted the .value property and threw away the count. I still don't know what that integer means, I hadn't run that many token requests for instance.

Pasting the token into JWT.io gave me an output much like Aidan's.

{
  "actor": "aidansteele",
  "aud": "https://github.com/aidansteele/aws-federation-github-actions",
  "base_ref": "",
  "event_name": "push",
  "exp": 1631672856,
  "head_ref": "",
  "iat": 1631672556,
  "iss": "https://token.actions.githubusercontent.com",
  "job_workflow_ref": "aidansteele/aws-federation-github-actions/.github/workflows/test.yml@refs/heads/main",
  "jti": "8ea8373e-0f9d-489d-a480-ac37deexample",
  "nbf": 1631671956,
  "ref": "refs/heads/main",
  "ref_type": "branch",
  "repository": "aidansteele/aws-federation-github-actions",
  "repository_owner": "aidansteele",
  "run_attempt": "1",
  "run_id": "1235992580",
  "run_number": "5",
  "sha": "bf96275471e83ff04ce5c8eb515c04a75d43f854",
  "sub": "repo:aidansteele/aws-federation-github-actions:ref:refs/heads/main",
  "workflow": "CI"
}

I don't think these tokens can be abused once they have expired, but I decided not to share one of mine with you verbatim.

I found these fields interesting:

• actor - who triggered this action? Do we want them to access OpenFaaS?
• iss - who issued this token? We can use this URL to get their public key and then verify the JWT
• repository_owner - who owned the repo? Is it part of our company organisation?

Token exchange

Now I started to look into how Google did its token exchange and came across a lesser-known OAuth2 grant_type called token exchange.

It turns out that various IdPs like Okta, Keycloak and Auth0 consider it experimental, and it may need additional configuration to enable it.

You can read about the OAuth 2.0 Token Exchange in its draft Internet Engineering Task Force (IETF) paper: rfc8693

The grant_type field should be populated as urn:ietf:params:oauth:grant-type:token-exchange and you should be aware of the many other fields prefixed with urn:ietf:params:oauth. They are not all required in the spec, but in some of the IdPs I looked into, many of the optional fields were listed as required.

Don't you love standardisation?

What I did instead

Any IdP can be used with OpenFaaS, even those which do not support Token Exchange, so I decided to try validating the OIDC token from GitHub directly.

Here's the way it works:

• The iss field maps to https://token.actions.githubusercontent.com
• By adding the OIDC configuration endpoint to the path /.well-known/openid-configuration, we get a JSON bundle back with various URLs that we can use to then download the public key of the server
• The public key can be used to verify the JWT token
• If the token is good, and hasn't expired, then we know it came from GitHub Actions

Here's the result from the OIDC Discovery URL URL as specified in the OIDC Spec

curl -s https://token.actions.githubusercontent.com/.well-known/openid-configuration


{
  "issuer": "https://token.actions.githubusercontent.com",
  "jwks_uri": "https://token.actions.githubusercontent.com/.well-known/jwks",
  "subject_types_supported": [
    "public",
    "pairwise"
  ],
  "response_types_supported": [
    "id_token"
  ],
  "claims_supported": [
    "sub",
    "aud",
    "exp",
    "iat",
    "iss",
    "jti",
    "nbf",
    "ref",
    "repository",
    "repository_owner",
    "run_id",
    "run_number",
    "run_attempt",
    "actor",
    "workflow",
    "head_ref",
    "base_ref",
    "event_name",
    "ref_type",
    "environment",
    "job_workflow_ref"
  ],
  "id_token_signing_alg_values_supported": [
    "RS256"
  ],
  "scopes_supported": [
    "openid"
  ]
}

The field we care about is jwks which we can follow to download the public key used to sign JWT tokens.

{
  "keys": [
    {
      "n": "zW2j18tSka65aoPgmyk7aUkYE7MmO8z9tM_HoKVJ-w_alYIknkf7pgBeWWfqRgkRfmDuJa8hATL20-bD9cQZ8uVAG1reQfIMxqxwt3DA6q37Co41NdgZ0MUTTQpfC0JyDbDwM_ZIzis1cQ1teJcrPBTQJ3TjvyBHeqDmEs2ZCmGLuHZloep8Y_4hmMBfMOFkz_7mWH7NPuhOLWnPTIKxnMuHl4EVdNL6CvIYEnzF24m_pf3IEM84vszL2s6-X7AbFheZVig8WqhEwiVjbUVxXcY4PtbK0z3jhgxcpjc6WTH0JlRedpq2ABowWZg-pxOoWZUAETfj6qBlbIn_F9kpyQ",
      "kty": "RSA",
      "kid": "DA6DD449E0E809599CECDFB3BDB6A2D7D0C2503A",
      "alg": "RS256",
      "e": "AQAB",
      "use": "sig",
      "x5c": [ "MIIDrDCCApSgAwIBAgIQBJyUm+htTmG6lfzlIyswTjANBgkqhkiG9w0BAQsFADA2MTQwMgYDVQQDEyt2c3RzLXZzdHNnaHJ0LWdoLXZzby1vYXV0aC52aXN1YWxzdHVkaW8uY29tMB4XDTIxMDkwODE4MTEyN1oXDTIzMDkwODE4MjEyN1owNjE0MDIGA1UEAxMrdnN0cy12c3RzZ2hydC1naC12c28tb2F1dGgudmlzdWFsc3R1ZGlvLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM1to9fLUpGuuWqD4JspO2lJGBOzJjvM/bTPx6ClSfsP2pWCJJ5H+6YAXlln6kYJEX5g7iWvIQEy9tPmw/XEGfLlQBta3kHyDMascLdwwOqt+wqONTXYGdDFE00KXwtCcg2w8DP2SM4rNXENbXiXKzwU0Cd0478gR3qg5hLNmQphi7h2ZaHqfGP+IZjAXzDhZM/+5lh+zT7oTi1pz0yCsZzLh5eBFXTS+gryGBJ8xduJv6X9yBDPOL7My9rOvl+wGxYXmVYoPFqoRMIlY21FcV3GOD7WytM944YMXKY3Olkx9CZUXnaatgAaMFmYPqcTqFmVABE34+qgZWyJ/xfZKckCAwEAAaOBtTCBsjAOBgNVHQ8BAf8EBAMCBaAwCQYDVR0TBAIwADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwNgYDVR0RBC8wLYIrdnN0cy12c3RzZ2hydC1naC12c28tb2F1dGgudmlzdWFsc3R1ZGlvLmNvbTAfBgNVHSMEGDAWgBSWfvMfh/F4A7d7WzhYMmM2jZhYNDAdBgNVHQ4EFgQUln7zH4fxeAO3e1s4WDJjNo2YWDQwDQYJKoZIhvcNAQELBQADggEBAJxwcMczvuXRVZUAF+jYaWKLdaa7HeeU3vOVrgeuPehLh9BquEu+asKVswMdEDvLMsrVrMRhJjXYaOW+B1UnlKHiKZzIx030e3GypAci/KNBXSvFB3KCZ4yk1Yvs3+hWV+5DWGRjDf5x3pp+zNWcHG12I+1F1KdC4vvPZ0G624imeucDzZurRD66SrLE/PqlMaFos8YqRr3QaY7hGhEtnwuu5P2POD6iRGIU60EpIkmFauuTv7eXRKN1u/RaQf6Qc4LGNysHT46gqEp9AGts/0AeAYFpEnvAdBXcHPrXhzPD72eAEdVzIFcwtzbB++sf2lBEqQxYPIfjFmiwB24T+bM="
      ],
      "x5t": "2m3USeDoCVmc7N-zvbai19DCUDo"
    },
  }
}

Now the kid present on this URL is also sent over in the OIDC token from GitHub Actions, but in the header rather than the body that Aidan shared on his blog.

After matching the kid from the token to the JWKS payload, you can find the correct public key and verify the OIDC token sent to you. That's what the OpenFaaS plugin does.

But at this point all we've done is allow anyone using a GitHub Action to deploy to our cluster. So we are not quite done yet, because I want to restrict that to just my friends and colleagues.

I updated the inlets tunnel so it pointed at my OpenFaaS instance running on KinD:

kubectl port-forward -n openfaas deploy/gateway 8080:8080

inlets client .. \
  --upstream http://127.0.0.1:8080

I could have also deployed the tunnel into KinD using the helm chart, but didn't need this to be permanent.

Next up, I changed GitHub Action to install the OpenFaaS CLI and to run faas-cli list using the token which I'd extacted into an environment variable.

name: federate

on:
  workflow_dispatch:
  push:
    branches:
    - '*'

jobs:
  auth:

    # Add "id-token" with the intended permissions.
    permissions:
      contents: 'read'
      id-token: 'write'

    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@master
        with:
          fetch-depth: 1
      - name: Install faas-cli
        run: curl -sLS https://cli.openfaas.com | sudo sh
        
      - name: Get token and use the CLI
        run: |
          OIDC_TOKEN=$(curl -sLS "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=minty.exit.o6s.io" -H "User-Agent: actions/oidc-client" -H "Authorization: Bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN")
          JWT=$(echo $OIDC_TOKEN | jq -j '.value')
          export OPENFAAS_URL=https://minty.exit.o6s.io/

          faas-cli list --token "$JWT"

Normally we'd run faas-cli login followed by list, deploy and so forth, but here we're using --token. OpenFaaS Pro supports various OAuth2 flows, you can read more here: OpenFaaS Single Sign-On

To my surprise, it worked!

There was just a little bit more work to do, I needed to write an Access Control List (ACL) and use one of GitHub Actions' fields to authorize just my friends. I used the actors field and enabled anyone in that field to be an admin for the OpenFaaS REST API.

Then I needed some help from a tester. Martin Woodward, Director of DevRel at GitHub was only to happy to help.

He first got "unauthorized" then I added him to the ACL and it worked.

Later on, Lucas Roesler a core contributor to OpenFaaS deployed a function of his own and it worked as expected. Lucas is very knowledgeable about OAuth and OIDC, both of which have a lot of moving parts, so he is a good person to run things by when I'm doing something new.

The OIDC Specification is also very readable and many client libraries exist for various languages. You should probably use these whenever you can instead of writing your own.

🎉🎉 https://t.co/6csQKTbtd1

No copying secrets or anything!!! https://t.co/3Ljji9QZ4t pic.twitter.com/RRUBaljKSM

— Lucas Roesler 🇺🇸🇩🇪 (@TheAxeR) October 6, 2021

Why do I like this so much?

In the past I built a complex PaaS called OpenFaaS Cloud which integrated tightly with GitHub and GitLab, but those were much earlier times and predated GitHub Actions. OpenFaaS Cloud took you from commiting code into a linked repository, to having a live endpoint within a few seconds.

However, it was quite a lot of code to maintain and required many additional components. After all, they do say that you can have a secure system or a simple one. There are some parts I think we did well like the multi-user UI, secret support and user experience. But users wanted to customise everything, or build with their existing CI system.

If you'd like to know more about what we built back then, the culimation is probably this conference talk from 2019: KubeCon: OpenFaaS Cloud + Linkerd: A Secure, Multi-Tenant Serverless Platform - Charles Pretzer & Alex Ellis

With the new OIDC configuration, GitHub Actions and multiple namespace support in OpenFaaS, you can get very close to a multi-tenant, highly-integrated and portable serverless platform.

But that's not all.

Taking it further

The authorization rules could also be enhanced further so that Lucas could only deploy to a set of mapped namespaces and we wouldn't bump into each other that way. Perhaps we'd set up a central shared OpenFaaS server and we could just deploy whatever we needed there.

This has all been done on Kubernetes so far, but faasd has become a promising alternative way to run OpenFaaS. It uses containerd, but has no clustering support meaning it can run very well on a 5 USD VM, a work hypervisor or even on an edge compute device like a Raspberry Pi. All the OpenFaaS Pro components will work on faasd since they are not specific to Kubernetes, but I've not had a customer ask for that yet.

OpenFaaS functions are container images, and GitHub Actions has really impressive support for building them with Docker including caching, multi-arch support with buildx. It can be tricky to put all this together for the first time, so in my eBook Serverless For Everyone Else, I give a reference example.

How I would expect this plug-in to be used would be to: to get a commit event, build a set of multi-archtecture images, push them to GitHub Container Registry, and then trigger a deployment using the OIDC federation. If the cluster was public, just the OpenFaaS gateway URL would need to be shared, which is non-confidential data. If the function is going to an edge device or a private cluster, then an inlets tunnel would be needed to access the gateway.

GitLab also offers OIDC tokens during CI jobs when used in combination with Hashicorp Vault. As long as the proper issuer is configured in the new openfaas plugin, then any valid issuer would work in the same way as GitHub Actions. One thing I really enjoy is writing code once and then reusing it again.

I'd like to offer the OpenFaaS GitHub Actions federation to the community to try out and give feedback. Ping me on OpenFaaS Slack to try it out.

If you can't wait for that, OpenFaaS Pro already has SSO support with any OIDC-compatible IdP like Azure LDAP, Auth0, Okta and Keycloak. Customers also get access to scale to zero for better efficiency, Kafka event integration, retries with exponential backoff and a chance to influence the future roadmap.

My Go application and the GitHub Actions workflow I shared are both available on my GitHub account.

As a software developer, Docker is a valuable tool for containerizing projects. It simplifies the process of adding Docker assets with the "docker init" command that automates the creation of necessary Docker files.

When traveling to Spanish-speaking countries like Spain, it's essential to be familiar with basic Spanish words and phrases. This includes vocabulary for socializing, expressing emotions, and communicating needs like asking for the location of bathrooms.

Travelers should also be mindful of food concerns, drink options, and allergies. Spanish coffee and drink options are widely available, with soy milk being a popular alternative to regular milk. However, it's also crucial to beware of tap water and ask "¿Qué es esto?" when unsure about the ingredients of a dish.

For those attending KubeCon NA, it's important to note that Spanish is widely used in many places. It would be helpful to learn the language's basic words and phrases for directions and location.

Moreover, as reading and learning are useful for personal growth and career advancement, one can take advantage of the offer to get 20% off on the first 50 spots using the code "estudiante" to sign up for online courses during free time.