Events and metrics

The information on this page is written for Vamp v0.9.2

To provide an effective feedback loop, HTTP/TCP logs should be collected, stored and analyzed. Collection and storing is done with a combination of HAProxy, VGA and Logstash setup. Logs are stored in Elasticsearch and can be later visualised by Kibana.

How traffic is logged

Vamp uses Logstash to format and store logs from running applications HAProxy (via Vamp Gateway Agent) in Elasticsearch indices. Logstash listens on UDP port and formats incoming log raw data in json format. Vamp API actions (including those generated by Vamp workflows and the Vamp UI) and running service events are stored by the Vamp API to specific Elasticsearch indices. Data and events are read by Vamp components either directly from Elasticsearch or via the Vamp API:

  • The Vamp UI reads data and events via the Vamp API. Health and Metrics events (generated by Vamp workflows) are required by the Vamp UI.
  • Vamp workflows read formatted log data directly from Elasticsearch and events are via the Vamp API. In theory, workflows could also read events directly from Elasticsearch.

Formatted raw data (logs)

HAProxy

HAProxy generates logs and makes them accessible via open socket - check the HAProxy configuration of log (github.com/magneticio - haproxy.cfg). VGA listens on log socket and any new messages are forwarded to the Logstash instance. The HAProxy log format is configurable in Vamp configuration vamp.gateway-driver.haproxy (github.com/magneticio - reference.conf). In general, for each HTTP/TCP request to HAProxy, several log messages are created (e.g. for gateway, service and instance level).

Logstash

A simple Logstash configuration should be sufficient for dozens of requests per second - or even more, depending on whether Elastic Stack (ELK) is also used for custom application/service logs etc. This example Logstash configuration (github.com/magneticio - logstash.conf), together with the default vamp.gateway-driver.haproxy log format, will transform logs to plain JSON, which can be parsed easily later on (e.g. for Kibana visualisation).
For alternative Logstash/Elasticsearch setups you can check these examples (elastic.co - Deploying and Scaling Logstash) and the Logstash command line parameter (github.com/magneticio - Logstash section).

Note!

  • Logstash listens on UDP port, but in principle any other listener can receive logs forwarded by VGA.
  • Different VGAs can use different Logstash instances.

Events

Vamp collects events on all running services. Interaction with the API also creates events, like updating blueprints or deleting a deployment. Furthermore, Vamp allows third party applications to create events and trigger Vamp actions. All events are stored and retrieved using the Event API, part of the Vamp API.

Kibana

Vamp can be configured to create Kibana searches, visualisations and dashboards automatically with the vamp.gateway-driver.kibana.enabled configuration parameter. Vamp will do this by inserting ES documents to the Kibana index, so only the URL to access ES is needed (by default reusing the same as for persistence). Read more about Vamp configuration

What next?