Events and metrics

The information on this page is written for Vamp v0.9.4

To provide an effective feedback loop, HTTP/TCP logs should be collected, stored and analyzed.

How traffic is logged

Vamp works with Filebeat to store logs from running applications and HAProxy (via Vamp Gateway Agent) in Elasticsearch indices. Logs are stored with a unique ID and can be later visualised by Kibana. Vamp API actions (including those generated by Vamp workflows and the Vamp UI) and running service events are stored in Elasticsearch indices according to event type. Data and events are read by Vamp components either directly from Elasticsearch or via the Vamp API:

  • The Vamp UI reads data and events via the Vamp API. Health and Metrics events (generated by Vamp workflows) are required by the Vamp UI.
  • Vamp workflows read formatted log data directly from Elasticsearch and events are via the Vamp API. In theory, workflows could also read events directly from Elasticsearch.

Formatted raw data (logs)

HAProxy

HAProxy generates logs and makes them accessible via open socket - check the HAProxy configuration of log (github.com/magneticio - haproxy.cfg). The HAProxy log format is configurable in Vamp configuration vamp.gateway-driver.haproxy (github.com/magneticio - reference.conf). In general, for each HTTP/TCP request to HAProxy, several log messages are created (e.g. for gateway, service and instance level).

Vamp uses Filebeat to ship HAproxy logs from the Vamp Gateway Agents to Elasticsearch. Vamp components read data either directly from Elasticsearch or via the Vamp API.

Events

Vamp collects events on all running services. Vamp API actions (including those generated by Vamp workflows and the Vamp UI) and running service events are stored by the Vamp API to specific Elasticsearch indices. Third party applications can also create events and trigger Vamp actions. All events are stored and retrieved using the Event API, part of the Vamp API.

Kibana

Vamp can be configured to create Kibana searches, visualisations and dashboards automatically with the vamp.gateway-driver.kibana.enabled configuration parameter. Vamp will do this by inserting ES documents to the Kibana index, so only the URL to access ES is needed (by default reusing the same as for persistence). Read more about Vamp configuration

What next?