How to Make Your Own Salted Kubernetes Raspberry Pi Cluster

By August 15, 2018Blog

At a few recent conferences, I displayed a custom Raspberry Pi Kubernetes cluster that was built with the purpose of showing off various features of the Salt project. Salt is a powerful remote execution and configuration management platform, but because it’s built on a messaging bus, it can do so much more. Here’s how to build your own Raspberry Pi cluster running Kubernetes:

Pi Cluster Details

The Pi cluster is made up of twelve Raspberry Pi 3s: currently, four are the original Raspberry Pi 3s and the remaining eight are the Raspberry Pi 3Bs. The Raspberry Pis are powered using two Anker USB power bricks and they utilize a Netgear 10/100 switch. The rack is made up of a 3D printed case specifically designed for the Raspberry Pi and a custom wooden enclosure. One Pi acts as the Kubernetes control node and another acts as the Salt Master; the remaining ten are Salt Minions and Kubernetes nodes.

Kubernetes Setup

There are many great how-to guides for setting up Kubernetes, particularly on Raspberry Pi clusters, so we’re not going to go into too much detail on setting up Kubernetes. There are also a handful of really nice Salt formulas that make the entire process very straight forward; unfortunately, given that this cluster is using Raspberry Pis, the setup is a bit outside the scope that the formulas handle, so we need to do a few things manually. For Kubernetes versions, I had the best luck using version 1.9.7 and below; later versions have some issues during the setup phase. I also had the best luck using the Flannel network fabric.

Salt + Kubernetes

In order to show off some of the features of Salt, the software should work with Kubernetes by controlling it, orchestrating it, and reacting to its actions (more on that later).  In order for Salt to interact with Kubernetes, the Python Kubernetes library will be need to be installed on the Kubernetes control node. Make sure to install version 3.0.0 of the python-kubernetes library, as later versions are not compatible with the current stable version of Salt.

Once the Kubernetes python library is installed, make sure that the Salt Minion running on the Kubernetes control node is properly configured to talk to Kubernetes. Looking at the admin.conf file under the /etc/kubernetes directory, copy a few options over to the Salt minion configuration.  

Include the api_url so that Salt knows the right address to send Kubernetes commands. Also include the certificate-authority-data, client-certificate-data, and client-key-data items to prove that Salt is authorized to send the commands to Kubernetes. All of these configuration items should be prefaced with kubernetes in the minion configuration. Include these in a file called kubernetes.conf under the /etc/salt/minion.d directory so that it’s easier to maintain in the future.

Once that change is made and you’ve restarted the Salt Minion on the Kubernetes control node, verify that the connection between Salt and Kubernetes is functioning properly. Do this by running one of the functions from the Kubernetes Salt module from the Salt Master targeted against the Kubernetes control node:

“salt k8s-master-1 kubernetes.ping”

If everything is working as expected, this should return back a simple True message.

Now that Salt is able to control Kubernetes, it’s time to do something a little more exciting: The following is a simple Salt state that will tell Kubernetes to deploy an nginx container into the cluster with a random number between 2–25 of replicas or copies across the cluster.

my-nginx:
  kubernetes.deployment_present:
    - namespace: default
      spec:
        replicas: {{ range(2, 25) | random }}
        template:
          metadata:
            labels:
              run: my-nginx
          spec:
            containers:
            - name: my-nginx
              image: garethgreenaway/nginx:latest
              ports:
              - containerPort: 80

Additionally, you can schedule a job in the Salt scheduler by running the following Salt State on the Kubernetes control node:

recycle-kubernetes-deployments:
  schedule.present:
    - function: state.sls 
    - job_args:
        - states.kubernetes
    - seconds: '30'
    - maxrunning: 1

This state will tell Salt to run the init.sls state in /srv/salt/states/kubernetes/ on the Kubernetes control node every 30 seconds. You’ll see the significance of this in the demonstration later.

Raspberry Pi Lights

The purpose of this demonstration is to provide a visual representation of Salt and its event bus, particularly when it is orchestrating Kubernetes. For this project, I wanted to use something involving lights that could be used on each individual Raspberry Pi.  Fortunately, Pimoroni sells the Blinkt, a nice little add-on board for the Raspberry Pi that has 8 individual LEDs, each one capable of displaying a range of 256 colors as well as other effects such as blinking and pulsing.  As an added bonus, there is an open source Python library that can be used to interact with the board, appropriately also named blinkt.

Blinkt Module

Now that the Raspberry Pis have their lights, Salt and Kubernetes needs a way to control them. Using the blinkt python library, build a custom Salt module that can interact with the Blinkt board from within Salt:

# -*- coding: utf-8 -*-
'''
Module for controlling the Blinkt LED matrix

.. versionadded:: Fluorine

:maintainer:    Gareth J. Greenaway <gareth@saltstack.com>
:maturity:      new
:depends:       blinkt Python module

'''

from __future__ import absolute_import, unicode_literals, print_function

import colorsys
import logging
import time
import random

__virtualname__ = 'blinkt'

try:
    import blinkt
    HAS_BLINKT = True
except (ImportError, NameError):
    HAS_BLINKT = False
log = logging.getLogger(__name__)


def __virtual__():
    '''
    Only load the module if Blinkt is available
    '''
    if HAS_BLINKT:
        return __virtualname__
    else:
        return False, "The blinkt excecution module can not be loaded."


def random_colors():
    '''
    Set the LEDs to random colors

    CLI Example:

    .. code-block:: bash

        salt '*' blinkt.random_colors

    '''
    blinkt.set_clear_on_exit()
    blinkt.set_brightness(0.1)

    for i in range(blinkt.NUM_PIXELS):
        blinkt.set_pixel(i, random.randint(0, 255),
                         random.randint(0, 255),
                         random.randint(0, 255))
    blinkt.show()
    return {'result': True, 'comment': 'Set LEDs to random colors'}


def rainbow():
    '''
    Set the LEDs to the colors of the rainbow

    CLI Example:

    .. code-block:: bash

        salt '*' blinkt.rainbow

    '''
    spacing = 360.0 / 16.0
    hue = 0

    blinkt.set_clear_on_exit()
    blinkt.set_brightness(0.1)

    hue = int(time.time() * 100) % 360
    for x in range(blinkt.NUM_PIXELS):
        offset = x * spacing
        h = ((hue + offset) % 360) / 360.0
        r, g, b = [int(c*255) for c in colorsys.hsv_to_rgb(h, 1.0, 1.0)]
        blinkt.set_pixel(x, r, g, b)
    blinkt.show()
    return {'result': True, 'comment': 'Set LEDs to rainbow'}


def one_rgb(pixel=0, red=0, green=0, blue=0):
    '''
    Set the one LED to a color

    CLI Example:

    .. code-block:: bash

        salt '*' blinkt.one_rgb 5 255 120 10

    '''
    if pixel > blinkt.NUM_PIXELS:
        return False, 'Invalid pixel'

    blinkt.set_pixel(pixel, red, green, blue)
    blinkt.show()
    return {'result': True,
            'comment': 'Set pixel {0} to rgb ({1},{2},{3})'.format(pixel, red, green, blue)}


def range_rgb(start=0, end=1, red=255, green=255, blue=255):
    '''
    Set a range of LEDs to a color

    CLI Example:

    .. code-block:: bash

        salt '*' blinkt.range_rgb start=2 end=5 red=0 green=255 blue=0

    '''
    if start > blinkt.NUM_PIXELS:
        return False, 'Invalid start pixel'

    if end > blinkt.NUM_PIXELS:
        return False, 'Invalid end pixel'

    for pixel in range(start, end + 1):
        blinkt.set_pixel(pixel, red, green, blue)
        blinkt.show()
    return {'result': True,
            'comment': 'Set range {0}-{1} to rgb ({2},{3},{4})'.format(start, end, red, green, blue)}


def all_rgb(red=255, green=255, blue=255):
    '''
    Set all the LEDs to a color

    CLI Example:

    .. code-block:: bash

        salt '*' blinkt.all_rgb red=0 green=255 blue=0

    '''
    blinkt.set_all(red, green, blue)
    blinkt.show()
    return {'result': True,
            'comment': 'Set all pixels to rgb ({0},{1},{2})'.format(red, green, blue)}


def clear(**kwargs):
    '''
    Clear  one pixel, a range of pixels or all pixels

    CLI Example:

    .. code-block:: bash

        salt '*' blinkt.clear 5

        salt '*' blinkt.clear 3 5

        salt '*' blinkt.clear

    '''
    if 'pixel' in kwargs:
        ret = clear_one(kwargs['pixel'])
    elif 'start' in kwargs and 'end' in kwargs:
        ret = clear_range(kwargs['start'], kwargs['end'])
    else:
        blinkt.set_all(0, 0, 0)
        blinkt.show()
        comment = 'Clear all pixels'
        ret = {'result': True,
               'comment': comment}

    return ret


def clear_one(pixel):
    '''
    Clear one pixel

    CLI Example:

    .. code-block:: bash

        salt '*' blinkt.clear_one 5

    '''
    blinkt.set_pixel(pixel, 0, 0, 0)
    blinkt.show()
    comment = 'Clear pixel {0}'.format(pixel)
    return {'result': True,
            'comment': comment}


def clear_range(start_pixel, end_pixel):
    '''
    Clear one pixel

    CLI Example:

    .. code-block:: bash

        salt '*' blinkt.clear_range 2 6

    '''
    for pixel in range(start_pixel, end_pixel + 1):
        blinkt.set_pixel(pixel, 0, 0, 0)
    blinkt.show()
    comment = 'Clear pixel range {0}-{1}'.format(start_pixel, end_pixel)
    return {'result': True,
            'comment': comment}

Name the custom module blinkt_mod.py since we’re going to be controlling the blinkt device.  Include the _mod to avoid any conflicts with the blinkt python library, which is named blinkt. At the top of our module, include the file encoding: in this case, UTF-8 and a doc string so that everyone will know what our module is designed to do. Then import some items from the __future__ library to ensure that our module is both Python 2 and Python 3 compliant.

After that, import the needed Python libraries and define the __virtualname__ variable.  This variable allows us to include a name that can be called from Salt that might be different than the name of the file. The file can be named blinkt_mod.py but the module name will simply be blinkt. Next, try to load the blinkt Python library in a try and except block; if that’s successful, set a variable called HAS_BLINKT to True, otherwise the variable is set to False.  

Then, use that variable later in the __virtual__ function.  This function is an option function that, if it exists, if checked and run for each Salt module. It is typically used to determine if a module should be made available. Following the __virtual__ function, you’ll have several functions that will perform various actions on the blinkt device. They each include their own docstring that indicates which function they are performing. Each module then calls the appropriate function from the blinkt library, calls the show function, and finally returns a True return code and a message of what was performed.

Now that you’ve got a custom module, ensure that it’s available across all of the Raspberry Pis.  Place the module under /srv/salt/_modules on the Salt Master, then run the command “salt \* saltutil.sync_modules,” which will cause any of the custom modules under this directory to be copied to the appropriate location on all of the Salt Minions.

Verify that the new module is in place by turning some lights on.

Now that the Raspberry Pis have their lights, Salt and Kubernetes need a way to control them.  That’s coming up in Part 2, so stay tuned!

Join the discussion One Comment

Leave a Reply