add docs and docker config for local and remote app telemetry collection and display

This commit is contained in:
Levi Schoen 2023-12-13 13:49:37 -08:00
parent ffd306ef52
commit 1aa1348944
4 changed files with 1650 additions and 0 deletions

86
docs/Telemetry.md Normal file
View File

@ -0,0 +1,86 @@
# Telemetry
[example metrics emitted by Kava application](./example-prometheus-metrics.txt)
## Enabling Kava Metric Telemetry
To enable the kava app to emit telemetry during operation, update the relevant config values to enable metrics:
`config.toml`
```toml
[instrumentation]
# When true, Prometheus metrics are served under /metrics on
# PrometheusListenAddr.
# Check out the documentation for the list of available metrics.
prometheus = true
# Address to listen for Prometheus collector(s) connections
prometheus_listen_addr = ":8888"
```
`app.toml`
```toml
[telemetry]
# Prefixed with keys to separate services.
service-name = ""
# Enabled enables the application telemetry functionality. When enabled,
# an in-memory sink is also enabled by default. Operators may also enabled
# other sinks such as Prometheus.
enabled = true
```
Then restart the service with the updated settings
## Running local prometheus collector and grafana services
To collect app metrics and visualize them locally, you can run the prometheus collector and grafana services with docker compose from the repo root directory
```bash
docker compose -f prometheus.docker-compose.yml
```
Navigate to localhost:3000 to view the grafana unix
Login with `admin` as the username and `admin` as the password
Hook up grafana to the local prometheus collector by navigating to `http://localhost:3000/connections/datasources/new`, selecting prometheus, entering `http://prometheus:9090` for the url, and clicking `Save & test` at the bottom of the screen
See [grafana docs](https://grafana.com/docs/grafana/latest/dashboards/) for information on how to construct queries and build dashboards
### Collecting from local host
Update [prometheus config](../prometheus.yml) to collect metrics from your local source
```yaml
metrics_path: /
static_configs:
- targets:
- localhost:8888
```
### Collecting from remote host
Update the kava config on the host and restart using the instructions from `Enabling Kava Metric Emission`
Install [ngrok](https://ngrok.com/download) on the remote host
Run ngrok on the remote host to forward the prometheus metric port
```bash
ngrok http 8888
```
```yaml
scrape_configs:
- job_name: proxy
scheme: https
metrics_path: /
static_configs:
- targets:
- 4efb-18-207-102-158.ngrok-free.app
```

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,36 @@
version: '3.8'
networks:
monitoring:
driver: bridge
volumes:
prometheus_data: {}
services:
grafana:
image: grafana/grafana-enterprise
container_name: grafana
restart: unless-stopped
ports:
- "3000:3000"
networks:
- monitoring
prometheus:
image: prom/prometheus:latest
container_name: prometheus
restart: unless-stopped
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--web.enable-lifecycle'
expose:
- 9090
networks:
- monitoring

10
prometheus.yml Normal file
View File

@ -0,0 +1,10 @@
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: proxy
scheme: https
metrics_path: /
static_configs:
- targets:
- 4efb-18-207-102-158.ngrok-free.app