> ## Documentation Index
> Fetch the complete documentation index at: https://tero-0926be64-video-walkthrough.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Prometheus

> Deploy Edge as a sidecar proxy for Prometheus metrics scraping on Kubernetes

Deploy Tero Edge as a sidecar to your application to filter Prometheus metrics
before Prometheus scrapes them. Edge sits between Prometheus and your application,
applying policies as each scrape passes through.

## How it works

Edge runs as a sidecar container in the same pod as your application. Prometheus
scrapes Edge instead of your application. Edge proxies the request to
your app's `/metrics` endpoint, applies policies to filter metrics, and returns
the filtered response.

```mermaid theme={null}
flowchart LR
    Prometheus --> Edge[Tero Edge]

    subgraph Pod
        Edge --> App[Your App]
    end

    style Prometheus fill:#e6522c,stroke:#e6522c,color:#fff
    style Edge fill:#10b981,stroke:#10b981,color:#fff
    style App fill:#262626,stroke:#262626,color:#fafafa
```

## Key features

* **Streaming processing**: Edge filters metrics line-by-line as they stream
  through, keeping memory usage bounded regardless of response size
* **Dual byte limits**: Configure `max_input_bytes_per_scrape` to bound memory
  usage and `max_output_bytes_per_scrape` to cap filtered response size
* **Zero-copy forwarding**: Edge forwards metrics that pass policy checks
  without extra allocations
* **Fail-open behavior**: If policy evaluation fails, metrics pass through
  unchanged

## Prerequisites

* Application exposing Prometheus metrics on Kubernetes
* Prometheus configured to scrape your pods
* `kubectl` access to your cluster
* Tero account

## Connect

<Steps>
  <Snippet file="create-edge-api-key.mdx" />

  <Step title="Create the secret">
    Store your API key as a Kubernetes secret:

    ```bash theme={null}
    kubectl create secret generic tero-edge \
      --from-literal=api-key=YOUR_API_KEY
    ```
  </Step>

  <Step title="Create the Edge ConfigMap">
    Create a ConfigMap with your Edge configuration:

    ```yaml tero-edge-config.yaml theme={null}
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: tero-edge-config
    data:
      config.json: |
        {
          "listen_address": "0.0.0.0",
          "listen_port": 9090,
          "upstream_url": "http://localhost:8080",
          "log_level": "info",
          "prometheus": {
            "max_input_bytes_per_scrape": 10485760,
            "max_output_bytes_per_scrape": 10485760
          },
          "policy_providers": [
            {
              "id": "tero",
              "type": "http",
              "url": "https://sync.usetero.com/v1/policy/sync",
              "headers": [
                { "name": "Authorization", "value": "Bearer ${TERO_API_KEY}" }
              ],
              "poll_interval_secs": 60
            }
          ]
        }
    ```

    ```bash theme={null}
    kubectl apply -f tero-edge-config.yaml
    ```

    <Note>
      Set `upstream_url` to your application's metrics endpoint. If your app
      exposes metrics on port 8080 at `/metrics`, use `http://localhost:8080`.
    </Note>
  </Step>

  <Step title="Add Edge as a sidecar">
    Add the Edge container to your application deployment:

    ```yaml theme={null}
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: my-app
    spec:
      template:
        spec:
          containers:
            # Your application container
            - name: app
              image: my-app:latest
              ports:
                - name: http
                  containerPort: 8080
                - name: metrics-internal
                  containerPort: 8080  # Your app's metrics port

            # Tero Edge sidecar
            - name: tero-edge
              image: ghcr.io/usetero/edge-prometheus:latest
              args:
                - /etc/tero/config.json
              ports:
                - name: metrics
                  containerPort: 9090  # Prometheus scrapes this port
              env:
                - name: TERO_API_KEY
                  valueFrom:
                    secretKeyRef:
                      name: tero-edge
                      key: api-key
              resources:
                requests:
                  cpu: 50m
                  memory: 32Mi
                limits:
                  cpu: 200m
                  memory: 128Mi
              volumeMounts:
                - name: tero-edge-config
                  mountPath: /etc/tero
                  readOnly: true
              livenessProbe:
                httpGet:
                  path: /_health
                  port: 9090
                initialDelaySeconds: 5
                periodSeconds: 10
              readinessProbe:
                httpGet:
                  path: /_health
                  port: 9090
                initialDelaySeconds: 2
                periodSeconds: 5

          volumes:
            - name: tero-edge-config
              configMap:
                name: tero-edge-config
    ```
  </Step>

  <Step title="Update Prometheus scrape config">
    Update your Prometheus configuration to scrape the Edge sidecar port instead
    of your application's metrics port:

    ```yaml theme={null}
    scrape_configs:
      - job_name: "my-app"
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          # Scrape the tero-edge metrics port (9090) instead of app port
          - source_labels: [__meta_kubernetes_pod_container_name]
            action: keep
            regex: tero-edge
          - source_labels:
              [__address__, __meta_kubernetes_pod_container_port_number]
            action: replace
            regex: ([^:]+):.*
            replacement: $1:9090
            target_label: __address__
    ```

    Or if using ServiceMonitor (Prometheus Operator):

    ```yaml theme={null}
    apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: my-app
    spec:
      selector:
        matchLabels:
          app: my-app
      endpoints:
        - port: metrics  # Points to Edge's port 9090
          interval: 30s
    ```
  </Step>

  <Step title="Verify">
    Check that both containers are running:

    ```bash theme={null}
    kubectl get pods -l app=my-app
    ```

    Test the metrics endpoint through Edge:

    ```bash theme={null}
    kubectl exec -it <pod-name> -c tero-edge -- wget -qO- http://localhost:9090/metrics | head -20
    ```

    Check Edge logs for filtering activity:

    ```bash theme={null}
    kubectl logs <pod-name> -c tero-edge --tail=50
    ```
  </Step>
</Steps>

## Configuration

### Prometheus settings

Configure Prometheus-specific settings in the `prometheus` section:

```json theme={null}
{
  "prometheus": {
    "max_input_bytes_per_scrape": 104857600,
    "max_output_bytes_per_scrape": 10485760
  }
}
```

| Setting                       | Default | Description                                                                                    |
| ----------------------------- | ------- | ---------------------------------------------------------------------------------------------- |
| `max_input_bytes_per_scrape`  | 10MB    | Maximum bytes to read from upstream per scrape. Limits memory for buffering input.             |
| `max_output_bytes_per_scrape` | 10MB    | Maximum bytes to forward to client per scrape. Set lower than input if filtering reduces data. |

**Example: High-cardinality filtering**

If your application exposes 1GB of metrics but policies filter it down to 1MB,
configure a high input limit with a lower output limit:

```json theme={null}
{
  "prometheus": {
    "max_input_bytes_per_scrape": 1073741824,
    "max_output_bytes_per_scrape": 10485760
  }
}
```

This allows Edge to process the full 1GB response while capping the filtered
output at 10MB.

### Policy providers

Edge supports multiple policy sources. Configure them in the `policy_providers`
array.

#### File provider

Load policies from a local file. Pick this for static policies bundled in the
ConfigMap.

```json theme={null}
{
  "id": "local",
  "type": "file",
  "path": "/etc/tero/policies.json"
}
```

#### HTTP provider

Fetch policies from a remote endpoint. Use this for dynamic policies managed
via the Tero API.

```json theme={null}
{
  "id": "tero",
  "type": "http",
  "url": "https://sync.usetero.com/v1/policy/sync",
  "headers": [{ "name": "Authorization", "value": "Bearer ${TERO_API_KEY}" }],
  "poll_interval_secs": 60
}
```

The `${TERO_API_KEY}` variable is injected from the Kubernetes secret via the
container environment configuration.

## Memory tuning

Edge's streaming architecture keeps memory usage predictable. Key factors:

1. **Input limit**: `max_input_bytes_per_scrape` caps how much data Edge reads
   from upstream. This bounds memory for buffering input data.

2. **Output limit**: `max_output_bytes_per_scrape` caps how much data Edge
   forwards to clients. Set this high if you have aggressive filtering.

3. **Line buffer**: Edge processes each metric line with a 4KB buffer. Edge
   passes lines over the 4KB limit through unfiltered.

4. **Concurrent scrapes**: Memory scales with concurrent scrapes. Each active
   scrape can use up to `max_input_bytes_per_scrape`.

For high-cardinality workloads with aggressive filtering:

```json theme={null}
{
  "prometheus": {
    "max_input_bytes_per_scrape": 1073741824,
    "max_output_bytes_per_scrape": 52428800
  }
}
```

This allows processing up to 1GB of metrics while capping output at 50MB.

## Troubleshooting

**Prometheus can't scrape metrics**

Verify Edge is running and healthy:

```bash theme={null}
kubectl describe pod <pod-name>
kubectl logs <pod-name> -c tero-edge
```

Ensure Prometheus is configured to scrape port 9090 (Edge) not your app's
metrics port directly.

**Metrics not being filtered**

Check that policies loaded successfully:

```bash theme={null}
kubectl logs <pod-name> -c tero-edge | grep -i policy
```

Verify your policy targets `metric` telemetry type with `METRIC_FILTER` stage.

**Scrapes timing out**

If your app has high-cardinality metrics, increase resource limits:

```yaml theme={null}
resources:
  limits:
    cpu: 500m
    memory: 256Mi
```

Also check `max_input_bytes_per_scrape` isn't truncating large responses.

**Some metrics missing**

Check if scrapes are being truncated due to input or output limits:

```bash theme={null}
kubectl logs <pod-name> -c tero-edge | grep -i truncat
```

Increase the limit if needed, or add policies to drop unwanted metrics earlier
in the stream.