An escalation is a workflow triggered by an escalation event. Vamp checks for these escalation events using a continuous background process with a configurable interval time. If the events match the escalation handlers defined in the DSL, the action is executed.
Again: escalation events can be generated by third party systems and they will be handled in the same manner as events created by Vamp SLA workflows.
Any escalation that is triggered should be handled by an escalation handler
Vamp ships with the following set of escalation handlers -
scale_memory. These handlers can be composed into intricate escalation systems.
Scales up the number of running instances. It is applied only to the first service in the cluster (old or “A” version). You can set upper limits to how far you want to scale out or in, effectively guaranteeing a minimum set of running instances. This is very much like AWS auto-scaling.
Example - scale_instances
--- type: scale_instances target: monarch # Target cluster for the scale up/down. # If it's not specified, by default it's the # current cluster where SLA escalations are # specified. minimum: 1 # Minimum number of instances. maximum: 3 # Maximum number of instances. scale_by: 1 # Increment/decrement to use on current # number of running instances.
Scales up the number of CPUs per instances. It is applied only on the first service in the cluster (old or “A” version).
Example - scale_cpu
--- type: scale_cpu target: monarch minimum: 1 maximum: 3 scale_by: 0.5
Scales up the memory per instance. It is applied only on the first service in the cluster (old or “A” version).
Example - scale_memory
--- type: scale_memory target: monarch minimum: 512 # In MB. maximum: 4096 # In MB. scale_by: 512 # In MB.
Composing escalation handlers
Vamp has a set of predefined escalation handler types that deal with escalations. You can compose these handlers in the DSL to get the desired outcome of an escalation event. The escalation handlers
to_one are supported:
This is a “group” escalation handler that contains a list of escalations. On each escalation event it will propagate the escalation to all escalation handlers from its list. No order or hierarchy.
Example - to_all
--- to_all: escalations: # Scale up/down. - scale_instances # And notify for each event. - notify
This is a group escalation handler that contains a list of escalations. On each escalation event it will propagate the escalation to each escalation handler (from its list) until one can handle it. On an
Escalate event, it will start at the head of the list. During a
DeEscalate event is will start at the rear.
When is this useful? Well, Vamp could try to scale up the service and if that;s not possible anymore (e.g. reached the upper limit of allowed scale) then an email can be sent.
Example - simple use of to_one
--- to_one: escalations: # First try to escalate. - scale_instances # If it's not possible, proceed with notifying. - notify
Example - complex use of to_one
--- name: monarch gateways: 80: monarch1/port environment_variables: monarch2.password: secret clusters: monarch1: breed: name: monarch1 deployable: vamp/monarch1 ports: port: 80/http environment_variables: STORAGE_HOST: $storage.host DB_PORT: $storage.ports.port STORAGE_PASS: $storage.environment_variables.password dependencies: storage: monarch2 scale: cpu: 1 memory: 1024MB instances: 1 sla: type: response_time_sliding_window threshold: upper: 1000 lower: 100 window: interval: 600 cooldown: 600 escalations: - to_one: escalations: - type: scale_instances # First try to scale up storage service. target: monarch2 minimum: 1 maximum: 3 scale_by: 1 - type: scale_instances # If we cannot scale up storage anymore, scale up this. target: monarch1 minimum: 1 maximum: 3 scale_by: 1 monarch2: breed: name: monarch2 deployable: vamp/monarch2 ports: port: 3306 environment_variables: STORAGE_PASS: password scale: cpu: 1 memory: 1024MB instances: 1