Escalations

An escalation is a workflow triggered by an escalation event. Vamp checks for these escalation events using a continuous background process with a configurable interval time. If the events match the escalation handlers defined in the DSL, the action is executed.

Again: escalation events can be generated by third party systems and they will be handled in the same manner as events created by Vamp SLA workflows.

Escalation handlers

Any escalation that is triggered should be handled by an escalation handler

Vamp ships with the following set of escalation handlers - scale_instances, scale_cpu and scale_memory. These handlers can be composed into intricate escalation systems.

scale_instances

Scales up the number of running instances. It is applied only to the first service in the cluster (old or “A” version). You can set upper limits to how far you want to scale out or in, effectively guaranteeing a minimum set of running instances. This is very much like AWS auto-scaling.

Example - scale_instances

---
type: scale_instances
target: monarch  # Target cluster for the scale up/down.
                 # If it's not specified, by default it's the 
                 # current cluster where SLA escalations are 
                 # specified.
minimum: 1       # Minimum number of instances.
maximum: 3       # Maximum number of instances.
scale_by: 1      # Increment/decrement to use on current 
                 # number of running instances.

scale_cpu

Scales up the number of CPUs per instances. It is applied only on the first service in the cluster (old or “A” version).

Example - scale_cpu

---
type: scale_cpu
target: monarch  
minimum: 1
maximum: 3 
scale_by: 0.5

scale_memory

Scales up the memory per instance. It is applied only on the first service in the cluster (old or “A” version).

Example - scale_memory

---
type: scale_memory
target: monarch  
minimum: 512     # In MB.
maximum: 4096    # In MB.
scale_by: 512    # In MB.

Composing escalation handlers

Vamp has a set of predefined escalation handler types that deal with escalations. You can compose these handlers in the DSL to get the desired outcome of an escalation event. The escalation handlers to_all and to_one are supported:

to_all

This is a “group” escalation handler that contains a list of escalations. On each escalation event it will propagate the escalation to all escalation handlers from its list. No order or hierarchy.

Example - to_all

---
to_all:
  escalations:
    # Scale up/down.
    - scale_instances
    # And notify for each event.
    - notify

to_one

This is a group escalation handler that contains a list of escalations. On each escalation event it will propagate the escalation to each escalation handler (from its list) until one can handle it. On an Escalate event, it will start at the head of the list. During a DeEscalate event is will start at the rear.
When is this useful? Well, Vamp could try to scale up the service and if that;s not possible anymore (e.g. reached the upper limit of allowed scale) then an email can be sent.

Example - simple use of to_one

---
to_one:
  escalations:
    # First try to escalate.
    - scale_instances
    # If it's not possible, proceed with notifying.
    - notify

Example - complex use of to_one

---
name: monarch

gateways:
  80: monarch1/port

environment_variables:
  monarch2.password: secret

clusters:

  monarch1:
    breed:
      name: monarch1
      deployable: vamp/monarch1
      ports:
        port: 80/http
      environment_variables:
        STORAGE_HOST: $storage.host
        DB_PORT: $storage.ports.port
        STORAGE_PASS: $storage.environment_variables.password
      
      dependencies:
        storage: monarch2

    scale:
      cpu: 1
      memory: 1024MB
      instances: 1
    
    sla:
      type: response_time_sliding_window
      threshold:
        upper: 1000
        lower: 100
      window:
        interval: 600
        cooldown: 600
      escalations:
        - 
          to_one:
            escalations:
              -
                type: scale_instances
                # First try to scale up storage service.
                target: monarch2
                minimum: 1
                maximum: 3
                scale_by: 1
              -
                type: scale_instances
                # If we cannot scale up storage anymore, scale up this.
                target: monarch1
                minimum: 1
                maximum: 3
                scale_by: 1
              
  monarch2:
    breed:
      name: monarch2
      deployable: vamp/monarch2
      ports:
        port: 3306
      environment_variables:
        STORAGE_PASS: password
    scale:
      cpu: 1
      memory: 1024MB
      instances: 1

What next?