How to Make Your Own Salted Kubernetes Pi Cluster—Part 2

By August 21, 2018Blog

In part 1 of our Salted Kubernetes Pi Cluster tutorial, we showed you how to set up Kubernetes and turn the lights on to your Raspberry Pis. Now you’ll learn how to control those lights and become the ruler of your very own Pi Cluster (try not to go mad with power).

Blinkt Engine

Most of the functions of the Blinkt Salt module will work correctly, but there are a few that will probably not function as expected.  If you want to set all the LEDs to be a single color or have each LED be a random color, then that action will function perfectly. However, if you want to do something more complex, such as have the LEDs continuously change colors or blink, then the module in its current state is not going to allow that. Because of the way the module is currently set up, once the Salt run has finished, the execution module also finishes. In order to accomplish the more complex scenarios, you’ll need something running in a continuous manner to constantly update the Blinkt device.

You could create a specialized application that runs in the background, commonly called a daemon, to accomplish this.  But you don’t have to: Salt already provides a mechanism that does the same thing called a Salt Engine.  

Salt Engines are long-running, external system processes that leverage Salt.  They have access to Salt configuration, execution modules, and Runners. And they’re executed in a separate process that is monitored by Salt. If a Salt Engine stops, it is restarted automatically.  Salt Engines are available to run on the Salt Master and on Salt Minions.

In a similar fashion to how we created the custom Salt module to control the Blinkt device, we’ll also create a custom engine:

# -*- coding: utf-8 -*-
'''
Engine for controlling the Blinkt LED matrix

.. versionadded:: Fluorine

:maintainer:    Gareth J. Greenaway <gareth@saltstack.com>
:maturity:      new
:depends:       blinkt Python module

'''

# Import python libs
from __future__ import absolute_import, print_function, unicode_literals
import datetime
import colorsys
import logging
import random
import time

try:
    import blinkt
    HAS_BLINKT = True
except (ImportError, NameError):
    HAS_BLINKT = False

# Import salt libs
import salt.utils.event

log = logging.getLogger(__name__)

# Define the module's virtual name
__virtualname__ = 'blinkt'

def __virtual__():
    '''
    Only load the module if Blinkt is available
    '''
    if HAS_BLINKT:
        return True
    return False

class BlinktEngine(object):
    '''
    Blinkt engine
    '''
    def __init__(self):
        self.stop_time = None

    def run(self):
        '''
        Main loop function
        '''
        if __opts__['__role'] == 'master':
            event_bus = salt.utils.event.get_master_event(
                __opts__,
                __opts__['sock_dir'],
                listen=True)
            tag = '/salt/master/blinkt'
        else:
            event_bus = salt.utils.event.get_event(
                'minion',
                transport=__opts__['transport'],
                opts=__opts__,
                sock_dir=__opts__['sock_dir'],
                listen=True)
            tag = '/salt/minion/blinkt'

        _kwargs = {}
        mode = None
        while True:
            now = datetime.datetime.now()
            event = event_bus.get_event(full=True)
            if (event and 'tag' in event 
                    and event['tag'].startswith(tag)):
                mode = event['data'].get('mode', None)
                _kwargs = event['data'].get('kwargs', {})
                if _kwargs.get('timeout', None):
                    self.stop_time = now + datetime.timedelta(seconds=_kwargs.get('timeout'))

            if mode:
                func = getattr(self, mode, None)
                if func:
                    func(**_kwargs)
                    if self.stop_time and self.stop_time >= now:
                        mode = None
                        self.clear(**_kwargs)

    def random_blink_colors(self, **kwargs):
        '''
        Set the LEDs to random colors
        '''
        blinkt.set_clear_on_exit()
        blinkt.set_brightness(0.1)

        for i in range(blinkt.NUM_PIXELS):
            blinkt.set_pixel(i, random.randint(0, 255),
                             random.randint(0, 255), random.randint(0, 255))
        blinkt.show()

    def rainbow(self, **kwargs):
        '''
        Set the LEDs to the colors of the rainbow
        '''

        spacing = 360.0 / 16.0
        hue = 0

        blinkt.set_clear_on_exit()
        blinkt.set_brightness(0.1)

        hue = int(time.time() * 100) % 360
        for x in range(blinkt.NUM_PIXELS):
            offset = x * spacing
            h = ((hue + offset) % 360) / 360.0
            red, green, blue = [int(c*255) for c in colorsys.hsv_to_rgb(h, 1.0, 1.0)]
            blinkt.set_pixel(x, red, green, blue)
        blinkt.show()

    def one_rgb(self, **kwargs):
        '''
        Set the one LED to a color
        '''
        pixel = kwargs.get('pixel')
        red = kwargs.get('red')
        green = kwargs.get('green')
        blue = kwargs.get('blue')
        blinkt.set_pixel(pixel, red, green, blue)
        blinkt.show()

    def range_rgb(self, **kwargs):
        '''
        Set a range of LEDs to a color
        '''
        start_pixel = kwargs.get('start_pixel')
        end_pixel = kwargs.get('end_pixel')
        red = kwargs.get('red')
        green = kwargs.get('green')
        blue = kwargs.get('blue')
        for pixel in range(start_pixel, end_pixel + 1):
            blinkt.set_pixel(pixel, red, green, blue)
            blinkt.show()

    def all_rgb(self, **kwargs):
        '''
        Set all the LEDs to a color
        '''
        red = kwargs.get('red')
        green = kwargs.get('green')
        blue = kwargs.get('blue')
        blinkt.set_all(red, green, blue)
        blinkt.show()

    def clear(self, **kwargs):
        '''
        Clear one pixel, a range of pixels or all pixels
        '''
        if 'pixel' in kwargs:
            self.clear_one(kwargs['pixel'])
        elif 'start' in kwargs and 'end' in kwargs:
            self.clear_range(kwargs['start'], kwargs['end'])
        else:
            blinkt.set_all(0, 0, 0)
            blinkt.show()

    def clear_one(self, pixel):
        '''
        Clear one pixel
        '''
        blinkt.set_pixel(pixel, 0, 0, 0)
        blinkt.show()

    def clear_range(self, start_pixel, end_pixel):
        '''
        Clear range of pixels
        '''
        for pixel in range(start_pixel, end_pixel + 1):
            blinkt.set_pixel(pixel, 0, 0, 0)
        blinkt.show()

def start(interval=1):
    '''
    Listen to events and write them to a log file
    '''
    client = BlinktEngine()
    client.run()

Follow a similar naming convention as the one used for the Blinkt module:  in this case, name the engine blinkt_engine.py with a virtualname of Blinkt to be descriptive, but to also avoid any potential conflicts with the Blinkt Python library. The structure of the engine will look very similar to the module that you wrote, with many similar function names. The major difference is that all the functions won’t reside in a BlinktEngine class.  

Jumping down to the end of the engine, you’ll see there is a function called “start.” This function is what Salt looks for when attempting to run an engine; each Salt Engine needs a start function.  Examining this particular start function, you’ll see that it is creating a client object using the BlinktEngine class, then running the “run” function on that object.

Taking a look at the “run” function, you’ll see that it’s taking advantage of the Salt Engine’s ability to interact with the Salt event bus. Depending on whether or not this engine is running on a Salt Minion or a Salt Master, it calls the appropriate function from the salt.utils.event to create an event listener. Grab an event off the event bus; if that event is not empty and includes the tag, then set the mode and kwargs variables to be the respective values from the event.  

If the event contains a value for timeout, set the stop_time variable to be the current time plus the value of the timeout. This will be useful later when you want certain actions to only last a certain amount of time. Check to see if the mode is valid and attempt to set func to be a function within the class with the name of “mode,” defaulting to None if the function doesn’t exist.  If func is valid, attempt to execute it and pass in the kwargs variable. Following that, check to see if stop_time is valid and if it is greater than or equal to the current time. If so, set the mode back to None and run the clear function to reset the Blinkt device.

Make sure the Salt Module is available across all the Raspberry Pis. Place the module under /srv/salt/_engines on the Salt Master, then run the command “salt \* saltutil.sync_all,” which will cause any of the customizations, including our new engine, to be copied to the appropriate location on all of Salt Minions.

Blinkt Module + Blinkt Engine

Now that you have the engine in place eagerly waiting for events, you’ll need to update the original Blinkt module to send commands to the engine rather than sending them along to the Blinkt device directly.

# -*- coding: utf-8 -*-
'''
Module for controlling the Blinkt LED matrix

.. versionadded:: Fluorine

:maintainer:    Gareth J. Greenaway <gareth@saltstack.com>
:maturity:      new
:depends:       blinkt Python module

'''

from __future__ import absolute_import, unicode_literals, print_function

import logging

__virtualname__ = 'blinkt'

try:
    import blinkt
    HAS_BLINKT = True
except (ImportError, NameError):
    HAS_BLINKT = False
log = logging.getLogger(__name__)


def __virtual__():
    '''
    Only load the module if Blinkt is available
    '''
    if HAS_BLINKT:
        return __virtualname__
    return False, "The blinkt excecution module can not be loaded."


def random_colors(timeout=None):
    '''
    Set the LEDs to random colors

    CLI Example:

    .. code-block:: bash

        salt '*' blinkt.random_colors

    '''
    try:
        res = __salt__['event.fire']({'mode': 'random_blink_colors',
                                      'kwargs': {'timeout': timeout}},
                                     '/salt/minion/blinkt')
        ret = {'result': True, 'comment': 'Set LEDs to random colors'}
    except KeyError:
        # Effectively a no-op, since we can't really return without an event system
        ret = {}
        ret['comment'] = 'Event module not available.'
        ret['result'] = True
    return ret


def rainbow(timeout=None):
    '''
    Set the LEDs to the colors of the rainbow

    CLI Example:

    .. code-block:: bash

        salt '*' blinkt.rainbow

    '''
    try:
        res = __salt__['event.fire']({'mode': 'rainbow',
                                      'kwargs': {'timeout': timeout}},
                                     '/salt/minion/blinkt')
        ret = {'result': True, 'comment': 'Set LEDs to rainbow'}
    except KeyError:
        # Effectively a no-op, since we can't really return without an event system
        ret = {}
        ret['comment'] = 'Event module not available.'
        ret['result'] = True
    return ret


def one_rgb(pixel=0, red=0, green=0, blue=0, timeout=None):
    '''
    Set the one LED to a color

    CLI Example:

    .. code-block:: bash

        salt '*' blinkt.one_rgb 5 255 120 10

    '''
    if pixel > blinkt.NUM_PIXELS:
        return False, 'Invalid pixel'

    try:
        res = __salt__['event.fire']({'mode': 'one_rgb',
                                      'kwargs': {'pixel': pixel,
                                                 'red': red,
                                                 'green': green,
                                                 'blue': blue,
                                                 'timeout': timeout}},
                                     '/salt/minion/blinkt')
        ret = {'result': True,
               'comment': 'Set pixel {0} to rgb ({1},{2},{3})'.format(pixel, red, green, blue)}
    except KeyError:
        # Effectively a no-op, since we can't really return without an event system
        ret = {}
        ret['comment'] = 'Event module not available.'
        ret['result'] = True
    return ret


def range_rgb(start=0, end=1, red=255, green=255, blue=255, timeout=None):
    '''
    Set a range of LEDs to a color

    CLI Example:

    .. code-block:: bash

        salt '*' blinkt.range_rgb start=2 end=5 red=0 green=255 blue=0

    '''
    if start > blinkt.NUM_PIXELS:
        return False, 'Invalid start pixel'

    if end > blinkt.NUM_PIXELS:
        return False, 'Invalid end pixel'

    try:
        res = __salt__['event.fire']({'mode': 'range_rgb',
                                      'kwargs': {'start_pixel': start,
                                                 'end_pixel': end,
                                                 'red': red,
                                                 'green': green,
                                                 'blue': blue,
                                                 'timeout': timeout}},
                                     '/salt/minion/blinkt')
        ret = {'result': True,
               'comment': 'Set range {0}-{1} to rgb ({2},{3},{4})'.format(start, end, red, green, blue)}
    except KeyError:
        # Effectively a no-op, since we can't really return without an event system
        ret = {}
        ret['comment'] = 'Event module not available.'
        ret['result'] = True
    return ret


def all_rgb(red=255, green=255, blue=255, timeout=None):
    '''
    Set all the LEDs to a color

    CLI Example:

    .. code-block:: bash

        salt '*' blinkt.all_rgb red=0 green=255 blue=0

    '''
    try:
        res = __salt__['event.fire']({'mode': 'all_rgb',
                                      'kwargs': {'red': red,
                                                 'green': green,
                                                 'blue': blue,
                                                 'timeout': timeout}},
                                     '/salt/minion/blinkt')
        ret = {'result': True,
               'comment': 'Set all pixels to rgb ({0},{1},{2})'.format(red, green, blue)}
    except KeyError:
        # Effectively a no-op, since we can't really return without an event system
        ret['comment'] = 'Event module not available.'
        ret['result'] = True
    return ret


def clear(**kwargs):
    '''
    Clear  one pixel, a range of pixels or all pixels

    CLI Example:

    .. code-block:: bash

        salt '*' blinkt.clear 5

        salt '*' blinkt.clear 3 5

        salt '*' blinkt.clear

    '''
    try:
        res = __salt__['event.fire']({'mode': 'clear', 'kwargs': kwargs}, '/salt/minion/blinkt')

        if 'pixel' in kwargs:
            comment = 'Clear pixel {0}'.format(kwargs['pixel'])
        elif 'start' in kwargs and 'end' in kwargs:
            comment = 'Clear pixel range {0}-{1}'.format(kwargs['start_pixel'], kwargs['end_pixel'])
        else:
            comment = 'Clear all pixels'

        ret = {'result': True,
               'comment': comment}

    except KeyError:
        # Effectively a no-op, since we can't really return without an event system
        ret = {}
        ret['comment'] = 'Event module not available.'
        ret['result'] = True

    return ret

The updated module should look very similar to the original; the major difference is that now all the functions have been updated.  Rather than calling the Blinkt Python library directly, each function is now using the dunder Salt dictionary to call the “event.fire” Salt module.

Pass in the appropriate function name in the “mode” variable along with the kwargs dictionary, which contains all the additional parameters passed to this function when calling the Salt module.  

Wrap the call to the event.fire function inside a try & except block, just in case the event system is unavailable. After the event.fire function has been called, set the “ret” dictionary with a result of True and a comment of what has happened.  This dictionary is then returned to the user and Salt’s default outputer presents it in a friendly way.

Another change to this version of the Blinkt module is that all of our module functions now include an additional “timeout” parameter.  By default, the value is set to None, but if you include a value when calling one of the mode functions, this will signal the engine that it is only to have this mode active for the value of the timeout.  You’ll see how this will be useful later when you start tying this into the Kubernetes cluster.

Salt + Docker Events

By creating the custom Blinkt module and engine, you’ve seen how easy it can be to put events onto the Salt event bus and then pull those events off to control an external system or device, but what about putting events onto the event bus from an external system? One of the engines that is included with Salt does just that: the Docker engine. This particular engine will listen to a Docker URL, usually the local unix socket URL, using the Docker API, and then place events that occur from Docker onto the Salt Event Bus.

These events are generally related to the containers that Docker is handling and include events such as create, start, terminate, and destroy. Once these events from Docker are published to the Salt Event Bus, Salt can react to it just like any other event. Typically, this reaction comes in the form of running a particular pre-defined state file on the Salt Master.  

There are a few different options available for reactions. There is the local reaction, which allows you to run a remote execution module on a specific Minion or collection of Minions. There is the runner reaction, which allows us to run a Salt Runner function. Finally, there are the wheel and caller reactions, which allow you to run a wheel function on the Master and a remote execution function on a Minion that is running without a Master.

As when writing Salt states, you have a few options for the formats you can use.  Most reactor state files will be written using YAML format, but when the situation calls for something a bit more powerful, you can turn to writing out state files in pure Python. That is the format you’ll use for the reactors that will tie together the Kubernetes cluster with the work you’ve done on the Blinkt module and engine.

Blinkt + Kubernetes

Because the Kubernetes setup uses Docker for its container, if you were to set up the Docker Salt Engine on each of the Kubernetes nodes, every time something happens with one of the containers on that node, an event would appear on the Salt Event Bus.  An event which we could then react to. We’ll use a state file to push out a configuration file to all of our Kubernetes nodes to ensure that both the Docker engine and our new Blinkt engine are running.

/etc/salt/minion.d/engines.conf:
  file.managed:
    - source: salt://states/salt/engines.conf
    - user: root
    - group: root
    - mode: 644

The above state file is telling Salt that you want to manage the file /etc/salt/minion.d/engines.conf on each of the Kubernetes nodes, and that the file should be owned by root and have a group with read and write permissions for the owner (and just read access for anyone else).  Have the source for that file be a file that resides on the Salt Master at the provided path. Here are the contents:

engines:
  - docker_events:
      docker_url: unix://var/run/docker.sock
  - blinkt

This tells the Salt Minion that you want to have both the docker_events and Blinkt engines running when the Salt Minion starts up, and you want the docker_events engine to listen for events using the local domain socket URL. Once that file is in place,  restart the Salt Minion on all of the Kubernetes nodes.

Once the Minions have all been restarted, you should start seeing some Docker events on the Event Bus. You can verify that by using the “event” function in the “state” runner on our Salt Master with the command:

salt-run state.event pretty=True

Typically, there are many events being generated when Salt is running,  but you’re currently only interested in seeing if the Docker events are available, so update that command with a filter to ensure you only see the events that the docker_events engine is using:

salt-run state.event 'salt/engines/docker_events/*' pretty=True

Assuming everything is working correctly, you’ll see some Docker events streaming past as the Salt schedule adjusts the Kubernetes deployment. Once you’ve verified that the Salt Master is receiving the Docker events, you can set up the reactors. Configure them on the Salt Master by creating a file under /etc/salt/master.d named reactors.conf with the following contents:

reactor:
    - 'salt/engines/docker_events/destroy':
      - /srv/reactors/docker/destroy.sls
    - 'salt/engines/docker_events/stop':
      - /srv/reactors/docker/stop.sls
    - 'salt/engines/docker_events/create':
      - /srv/reactors/docker/create.sls
    - 'salt/engines/docker_events/start':
      - /srv/reactors/docker/start.sls

This configuration adds Reactors to the Salt Master to look for the following events and then tells the Salt Master to run the corresponding reactor state file. For example, if the Salt Master sees an event on the event bus with the tag of “salt/engines/docker_events/destroy,” it will execute the reactor state file “/srv/reactors/docker/destroy.sls”.

Create the destroy.sls along with the other reactor files under the path /srv/reactors/docker on the Salt Master.

/srv/reactors/docker/destroy.sls:
#!py

def run():

    work = {}

    kwargs = {'start': 6,
              'end': 7,
              'red': 255,
              'blue': 255,
              'green': 0,
              'timeout': 5}

    work['docker_destroy'] = {
        'local.blinkt.range_rgb': [
            {'tgt': data['id']},
            {'kwarg': kwargs},
        ]
    }

    return work

/srv/reactors/docker/stop.sls:
#!py

def run():

    work = {}

    kwargs = {'start': 4,
              'end': 5,
              'red': 255,
              'blue': 0,
              'green': 0,
              'timeout': 20}

    work['docker_stop'] = {
        'local.blinkt.range_rgb': [
            {'tgt': data['id']},
            {'kwarg': kwargs},
            ]
    }

    return work

/srv/reactors/docker/start.sls:
#!py

def run():

    work = {}

    kwargs = {'start': 2,
              'end': 3,
              'red': 0,
              'blue': 0,
              'green': 255,
              'timeout': 20}

    work['docker_start'] = {
        'local.blinkt.range_rgb': [
            {'tgt': data['id']},
            {'kwarg': kwargs},
        ]
    }

    return work

/srv/reactors/docker/create.sls:
#!py

def run():

    work = {}

    kwargs = {'start': 0,
              'end': 1,
              'red': 0,
              'blue': 255,
              'green': 0,
              'timeout': 30}

    work['docker_destroy'] = {
        'local.blinkt.range_rgb': [
            {'tgt': data['id']},
            {'kwarg': kwargs},
        ]
    }

    return work

All of these reactors are fairly similar. For a future improvement to this project, you could consolidate the reactors into one reactor file with some logic to determine what kind of Docker event the reactor has received, but for now just kept them separate.

Using the create reactor as an example, the first line indicates that you’re using Salt’s built-in Python render for this state file, and the file includes a “run” function. Salt will look for this function when attempting to run a reactor of this type.

The rest of the state file is some simple Python code. We declare a dictionary named “work” and another named “kwargs,” which hold the values for start and end values as well as some color values and a timeout value. After that comes the interesting part: this particular reactor is a local reactor, so it will run the execution module function on a particular Minion. In this case it will run the “range_rgb” function from our Blinkt module.  

The “range_rgb” function is expecting to receive start and end parameters so that it knows the range of lights on the Blinkt device that it should turn on. It is also expecting red, green, and blue color values so that it knows which color it should make the range of lights. Be sure to include a timeout value so that the range of lights only remains lighted for the specified amount of time.

Also, include the “tgt” parameter using the value of “id” from a dictionary called “data” that is passed to every reactor from the reactor system.  The value passing to “tgt” corresponds to the Salt Minion where the Docker event originated, so this is the target where we want to run the blinkt.range_rgb function to indicate that a Docker container was just created on this Minion. In this particular case, you’ll be turning on the first and second lights in the color of blue for 30 seconds. Then return the results of the function.

The remaining three functions work in a similar fashion with a call to the same “range_rgb” function within the Blinkt module; the difference is the color of LEDs on the Blinkt device that will be turned on. The destroy, stop and start events will cause the Blinkt lights be set to purple, red, and green, respectively. Assuming that everything is working correctly, as Docker containers are created and destroyed, you should see the various lights on the Blinkt devices turn off and on as well.

The Saltiest Results

Running a Kubernetes on a collection of Raspberry Pis, even when controlled via SaltStack, does not have many real world applications. Neither does having LED lights reflecting the current state of the cluster and its Docker containers. But hopefully the above examples have given some insight into how powerful the SaltStack system is and it can do so much more beyond just remote execution and configuration management.

Leave a Reply