Troubleshoot - full Vamp installation

The information on this page is written for Vamp v0.9.3

Try this first

The steps below will help you debug problems encountered when following our full install instructions.

Check the Vamp components

  • Check the step by step instructions for your container scheduler DC/OS, Mesos/Marathon, Kubernetes, Rancher or Docker.
  • Check the information panel in the Vamp UI (click the i-icon in the top right corner, or check Extended info under Admin). If some fields are empty, you may need to adjust your configuration. See the example Vamp configurations.
    You can view and update the configuration in the Vamp UI at Admin > Backend configuration > Applied
  • Check you are running supported versions of all components.
  • Check the Vamp release notes for known issues and breaking changes.
  • Check that the vamp and vamp-gateway-agent docker containers are running using docker ps.
  • Check the vamp and vamp-gateway-agent logs docker logs {container ID}
    Any errors here should be clear, in case they aren’t report the issue.
  • Check the Vamp configuration (using the API at /api/v1/config or in the Admin section of the Vamp UI):
    • You should use http:// in front of TCP/HTTP addresses, but not in front of UDP connections.
    • Make sure there are no double ports (ie 9200:9200)

Check Elasticsearch

Vamp uses Elasticsearch to store HAProxy logs and calculated metrics from its event system. The Health and Metrics indices are generated by Vamp workflows. To do this, they must have access to a valid Elasticsearch VGA index and the Vamp API.

  • Check the Elasticsearch indices required by Vamp have been created:
    GET <elasticsearch url>/_cat/indices?v
    HAProxy logs (VGA index): vamp-vga-YYYY-MM-DD
    Health workflow events: vamp-pulse-health-YYYY-MM-DD
    Metrics workflow events: vamp-pulse-metrics-YYYY-MM-DD
    vamp-vwa-YYYY-MM-DD
    vamp-pulse-vamp-archive-YYYY-MM-DD
Problem Action
no indices at all Check Vamp Gateway Agent (or your custom HAProxy) is storing logs in the expected format (HAproxy 1.7.1 log format).
no VGA index Check the VGA container logs: docker logs
Check if Filebeat is running in the VGA container: docker exec ps
Check that VGA can connect to Elasticsearch: docker exec curl -sSL
no Health or Metrics indices SSH into a Vamp Workflow Agent container and check you can telnet to the Elasticsearch url and the Vamp API set in your Vamp configuration (available from the Admin section of the Vamp UI, /api/v1/info or /api/v1/config or open a workflow and check the environment variables).

Check the key value store

  • Check the key value store reports as connected in Vamp info GET <vamp url>/api/v1/info
    If not, correct the configuration in application.conf.
    You can view and update the applied Vamp configuration in the Vamp UI under Admin > Backend configuration
  • Check you can communicate with the configured key value store from where Vamp is running.
    Zookeeper:
    Consul:
    etcd:

Check the Vamp workflows

The default Vamp workflows should be running continuously and not restarting.

  • Check that the three default workflow containers are there using docker ps
  • Check the logs for each vamp-workflow-agent container using docker logs {container ID}
    If a failure is reported, the workflow may not be able to talk to Elasticsearch
  • Check the Elasticsearch configuration in application.conf.
    Any errors here should be clear, in case they aren’t report the issue.

If you’re still hitting problems, please report the issue.

Report an issue

If you’ve tried the above steps and are still stuck, then let us know. We accept bug reports and pull requests on the GitHub repo for each project (github.com - magneticio).

When reporting issues, please include the following details:

  • Description of issue.
  • Vamp info, from the API: GET <vamp url>/api/v1/info
  • Vamp config, from the API; GET <vamp url>/api/v1/config
  • Reproduction steps.
  • Which orchestrator are you using, DC/OS, Kubernetes, Rancher, etc.
  • Log output from Vamp and it’s various components. You can get these from the docker logs command on the relevant containers.
  • Any other information that you might consider important.

Did this help?

If you find some of the instructions not clear enough or lacking information, please raise an issue on GitHub.