Observability & Log Shipping¶

OpenFlight can ship session logs to Grafana Cloud for long-term storage, querying, and dashboarding. This uses Grafana Alloy to tail JSONL session files and push them to Loki.

Overview¶

Session Logs (JSONL)  →  Grafana Alloy  →  Grafana Cloud Loki  →  Grafana Dashboards
~/openflight_sessions/     (local agent)       (cloud storage)       (query & visualize)

• What gets shipped: Every JSONL entry — session start/end, shot data (ball speed, club speed, spin, launch angle, carry), trigger events, I/Q readings
• Labels: app=openflight, host=<hostname>, log_type (shot_detected, session_start, etc.), mode, club
• Extracted fields: ball_speed, club_speed, carry, spin_rpm, launch_v, launch_h, angle_source, club_aoa, club_path, shot_number
• Buffering: Local WAL (write-ahead log) buffers during network outages

Setup¶

Prerequisites¶

• Raspberry Pi running OpenFlight
• A Grafana Cloud account (free tier works)

1. Get Grafana Cloud Credentials¶

1. Log in to Grafana Cloud
2. Go to your stack → Connections → Loki
3. Click Details to find:
4. URL: Loki push endpoint (e.g., https://logs-prod-us-central1.grafana.net/loki/api/v1/push)
5. User: Numeric instance ID
6. Generate an API key with logs:write scope

2. Install Alloy¶

sudo ./scripts/setup/setup_alloy.sh

This script: - Adds the Grafana APT repository - Installs Grafana Alloy - Writes the Alloy config to /etc/alloy/config.alloy (from config/alloy.alloy) - Creates a credentials template at /etc/alloy/credentials.env - Configures the systemd service to run as your user (so it can read session logs) - Enables the Alloy service

3. Configure Credentials¶

sudo vim /etc/alloy/credentials.env

Fill in your Grafana Cloud values:

LOKI_URL=https://logs-prod-us-central1.grafana.net/loki/api/v1/push
LOKI_USER=123456
LOKI_API_KEY=glc_your_api_key_here

4. Start Alloy¶

sudo systemctl start alloy

Or just run scripts/start-kiosk.sh — it starts Alloy automatically if credentials are configured.

5. Verify¶

Check Alloy is running:

sudo systemctl status alloy

Check for errors:

sudo journalctl -u alloy -f

Then query in Grafana Cloud → Explore → Loki:

{app="openflight"}

Querying Session Data¶

Basic Queries (LogQL)¶

# All shot data
{app="openflight", log_type="shot_detected"}

# Shots with a specific club
{app="openflight", log_type="shot_detected", club="driver"}

# Session starts
{app="openflight", log_type="session_start"}

# Filter by ball speed (using JSON parsing)
{app="openflight", log_type="shot_detected"} | json | ball_speed_mph > 150

Metric Queries¶

# Average ball speed over time
avg_over_time({app="openflight", log_type="shot_detected"} | json | unwrap ball_speed_mph [1h])

# Shot count per session
count_over_time({app="openflight", log_type="shot_detected"} [24h])

# Spin detection rate (shots with spin > 0)
count_over_time({app="openflight", log_type="shot_detected"} | json | spin_rpm > 0 [24h])

# Processing failures during a session
{app="openflight", log_type="error"}

How It Works¶

Session Log Format¶

OpenFlight writes JSONL files to ~/openflight_sessions/session_<timestamp>_<mode>.jsonl. Each line is a JSON object with a type field:

Alloy Pipeline¶

The config at config/alloy.alloy defines a three-stage pipeline:

1. File Discovery (local.file_match): Watches for session_*.jsonl files
2. JSON Processing (loki.process): Extracts fields as labels and structured metadata, uses the ts field as the log timestamp
3. Loki Push (loki.write): Ships to Grafana Cloud with basic auth and retry on rate-limiting

Auto-Start¶

When scripts/start-kiosk.sh runs, it checks: 1. Is Alloy installed? 2. Is /etc/alloy/credentials.env configured (not empty)? 3. Is the Alloy service already running?

If credentials are configured but Alloy isn't running, it starts the service automatically.

Troubleshooting¶

"Alloy installed but no credentials file found"¶

Run the setup script:

sudo ./scripts/setup/setup_alloy.sh

Then fill in /etc/alloy/credentials.env.

"Alloy installed but credentials not configured"¶

The credentials file exists but has empty values. Edit it:

sudo vim /etc/alloy/credentials.env

401 Unauthorized¶

Your API key is invalid or missing the logs:write scope. Generate a new key in Grafana Cloud → Loki → Details.

Permission Denied on Config¶

sudo chmod 755 /etc/alloy
sudo chmod 644 /etc/alloy/config.alloy
sudo chmod 600 /etc/alloy/credentials.env

Alloy Won't Start (CHDIR Error)¶

The data directory may not exist:

sudo mkdir -p /var/lib/alloy/data
sudo chown -R $(whoami):$(whoami) /var/lib/alloy

No Logs Appearing in Grafana¶

1. Verify Alloy is running: sudo systemctl status alloy
2. Check Alloy logs: sudo journalctl -u alloy -f
3. Verify session logs exist: ls ~/openflight_sessions/
4. Make sure session logging is enabled (it's on by default, including mock mode)

Configuration Reference¶

Files¶

Environment Variables¶