Create a workflow that generates events

In an earlier tutorial we showed how Vamp workflows can be used to automate all kinds of useful behaviour, for example to automate canary releasing with rollbacks, or autoscaling. In this tutorial, we will explain the Vamp workflow system in more detail and create a workflow that accesses the Vamp events system. We will first create and retrieve some custom events manually using the Vamp REST API, then create a simple workflow to automate these actions. Let’s get started!

Requirements

The Vamp events system

Vamp is a distributed system tied together by a central events stream. Every action in Vamp creates events, which in turn can be used as triggers for new actions. For example, gateway updates are triggered by deployments (synchronisation events), while canary releases and autoscaling actions are based on calculated health and metrics events. Vamp events consist of an event type, one or more event tags and an optional event value. They are described in the format:

['tag', 'tag1:tag2'], value, event_type

Event types

Vamp works with a number of default event types and custom event types can also be created using the /events API endpoint (more on that later). If no event type is specified when creating an event, the generic event type event will be used. It is advisable to always specify an event type to allow for easy filtering.

Default event types Description
archive Store/update of static Vamp artifacts (breeds, blueprints etc.)
synchronisation Successful matching of a desired state with an observed state (for example, a successful update from 3 to 5 running instances)
event Generic event type, will be used if no event type is specified when creating an event
health Generated by the health workflow and used by the Vamp UI
metrics Generated by the metrics workflow and used by the Vamp UI
allocation Generated by the allocation workflow and used to calculate resource usage
workflow Generated by workflows used in Vamp Runner

Event tags

Each event created by Vamp is given one or more tags. Tags provide meta-information for stored events and are used for filtering searches and listening. Tags can be either a single tag or a combination of two tags separated by a :. The Vamp convention is to use the first tag as a generic label (for example a group) and the second tag as a specific label (i.e. a specific item in a group). generic_tag:specific_tag

Event value

The event value is optional and could be anything you choose to store along with an event. Values are not analysed and can’t be used for search. If no value is specified, it will be blank

Storing events

Vamp events are stored by default in Elasticsearch using the integrated Vamp Pulse module. Elasticsearch indexing is based on the event type and is updated for each event received by the API. Each event type will be indexed individually, including custom event types .
Read more about events

Create events with the REST API

The /events endpoint of the Vamp API can be used to create and retrieve events. We are going to use this endpoint to create an event with the custom type hello_api. Note that custom event type names may only include alphanumerics, ‘_’ and ‘-’.

Use Postman or curl to POST the below JSON to the /api/v1/events endpoint.

{
  "tags": [
    "tag1:tag2"
  ],
  "value": "optional_value",
  "type": "hello_api"
}

If all runs to plan, you will receive a response from the API with the accepted JSON in the body.

{
  "tags": [
    "tag1",
    "tag1:tag2"
  ],
  "value": "optional_value",
  "timestamp": "2017-01-09T10:55:08.026Z",
  "type": "hello_api"
}

You will notice that a timestamp has been added and the the event tag has been expanded. This means we will be able to search for this event using both the generic tag (tag1) and the combined generic:specific tag (tag1:tag2). If you check in the Vamp UI, you will see the created event show up in the EVENTS stream at the bottom of the screen (of course you need to open the events stream panel first :).

Create events with a simple workflow

That’s great - we stored our first event! Now let’s take this a step further and create a workflow to automate the process.

A bit about workflows

Vamp workflows are a convenient way of creating Node JS based scripts that run as containers and access the Vamp API. JavaScript workflows run in Vamp workflow agent containers (github.com/magneticio - Vamp workflow agent) and are managed just like any other container inside your cluster, making them robust, scalable and dynamic. You can schedule workflows to run as a daemon, be triggered by Vamp events or to run at specified times. You could also create your own workflows using your language of preference - create an application or script that accesses the Vamp API and build it into a docker container that can be deployed by Vamp.

We are going to create a workflow that runs as a daemon (i.e. all the time). We’ll start by creating a breed with the required workflow-script and then deploy this as a Vamp workflow. Our workflow will simply store an event using the /events API endpoint, but you could also use Vamp workflows to aggregate and calculate metrics and send these out as events again, like the default “metrics” and “health” workflows and events that are used by the Vamp UI to create graphs.

Create a breed

The breed will hold the JavaScript to be run for our workflow. It is always advisable to create a Vamp breed artifact containing the workflow script and then reference the breed from a workflow definition. Note that we have included the Vamp Node.js Client library, this will allow the Node.js application to easily interact with the Vamp API, see the gitHub project for more details (github.com/magneticio - Vamp Node.js Client).

  1. Go the Breeds page in the Vamp UI
  2. Click Add
  3. Paste in the below breed YAML and click Save
name: hello_workflow
deployable:
  type: application/javascript
  definition: |
    'use strict';

    var vamp = require('vamp-node-client'); // required to interact with the Vamp API

    var api = new vamp.Api(); // a new Vamp API object

    // create a Vamp event - tag, value, event_type
    api.event('tag1:tag2', 'optional_value', 'hello_workflow');   
  

Deploy the hello_workflow breed as a workflow

After successfully saving the hello_workflow breed, we can create a workflow that references it. Once the workflow is created, the script from the breed will be deployed inside a Vamp workflow agent container. We will schedule our workflow to run as a daemon, so it will immediately start running and use environment variables to set the run interval (execution period) to 5 seconds.

  1. Go to the Workflows page in the Vamp UI
  2. Click Add (top right)
  3. Paste in the below workflow YAML and click Save
name: hello_workflow
breed: hello_workflow
environment_variables:
  VAMP_WORKFLOW_EXECUTION_PERIOD: '5' 
  VAMP_WORKFLOW_EXECUTION_TIMEOUT: '4'  # should be less than VAMP_WORKFLOW_EXECUTION_PERIOD
schedule: daemon

The workflow will be deployed and you will see the created events appearing in the Vamp UI EVENTS stream.

Update the running workflow

Updates made to a breed or workflow will not be directly carried over to the running workflow. To apply updates to a running workflow, it must be restarted. Let’s update our running workflow so it creates an event every second. We will need to suspend it first as It is not possible to update workflows while they are running.

  1. Go the Workflows page in the Vamp UI
  2. Open the action menu on the hello_workflow workflow and click Suspend
  3. Open the hello_workflow workflow and click Edit
  4. Set the VAMP_WORKFLOW_EXECUTION_PERIOD environment variable to 1 (this will cause the events to be generated every second)
  5. Click Save
  6. Go back to the Workflows page
  7. Open the action menu on the hello_workflow workflow and click Start
    • The workflow will be restarted, applying the new changes

Check the events stream again. hello_workflow will now be generating events every second - that’s a lot of events! Feel free to click Suspend to temporarily stop the workflow running or delete it completely.

Retrieve and filter events using the REST API

Now we have a lot of events stored we can use the same REST API /events endpoint to retrieve them. We can search for specific events and filter for specific types and tags. You can try this out in Postman or curl by sending a GET request to the /api/v1/events endpoint, you can even do this in your browser. Search through stored events using ?type= or ?tags= (following the tag format generic or generic:specifc). Note that it is not possible to filter for specific values.
See using the Vamp API for details on pagination.

The API can also be used to retrieve an events stream /api/v1/events/stream

Summing up

You should now know a bit more about the Vamp events system and how it is used by the distributed components of Vamp, such as workflows. Now you understand how to build, deploy and update your own workflows there should be no stopping you - what will you automate?

Looking for more of a challenge?

Just for fun, you could try these:

What next?