docs
Sidecar Monitoring

GSwarm Sidecar Monitoring System

Easily monitor your Gensyn AI nodes—no coding required!

This tool helps you track the health and performance of your Gensyn AI nodes. Just update a simple config file, then launch the monitor with a single command. No need to change your existing node setup or write any code.

What is the GSwarm Sidecar?

The GSwarm Sidecar is a monitoring tool for Gensyn AI nodes. It:

  • Watches your Gensyn AI node log files in real time
  • Sends important events and metrics to a central dashboard (API)
  • Helps you keep track of node health, training progress, and more
  • Requires no changes to your Gensyn AI node code

Quick Start for Beginners

1. Get Your JWT Token

To use the monitor, you need a JWT token from the GSwarm dashboard:

  1. Go to https://gswarm.dev (opens in a new tab)
  2. Connect your Metamask wallet (you'll see a prompt on the site)
  3. After logging in, go to your dashboard/settings and copy your JWT token

2. Edit the Configuration File

Before running the monitor, tell it where to find your log files and where to send the data.

  • Open the file: configs/config.yaml
  • Paste your JWT token into the jwt_token field:
jwt_token: "YOUR_JWT_TOKEN_HERE"
  • Update these lines to match your setup:
log_monitoring:
  api_endpoint: "https://central-api.example.com/gswarm/metrics"  # Where to send metrics
  log_files:
    - "/path/to/rl-swarm/logs/swarm.log"   # Main log file
    - "/path/to/rl-swarm/logs/yarn.log"    # Login server log
#   - "/path/to/rl-swarm/logs/wandb/debug.log"  # (Optional) Advanced metrics

3. Start the Monitor

If you have Docker installed (recommended for easiest setup):

docker compose up

Or, if you prefer to run it directly (requires Go):

make run

4. How to Know It's Working

  • The monitor will print logs to your terminal window.
  • If you see messages about reading your log files and sending data, it's working!
  • You can also check your dashboard at https://gswarm.dev (opens in a new tab) to see if your node is reporting.

Need Help?

  • Double-check your log file paths in configs/config.yaml
  • Make sure Docker is installed (for easiest setup)
  • For advanced help, see the DEVELOPMENT.md or ask your Gensyn support contact

What Does This Do?

  • Watches your Gensyn AI node log files in real time
  • Sends important events and metrics to a central dashboard (API)
  • Helps you keep track of node health, training progress, and more
  • No changes needed to your Gensyn AI node code

Advanced: Development & Customization

If you want to build, test, or develop the monitor yourself, see below for more details.

Project Structure

gswarm-sidecar/
├── cmd/monitor/main.go              # Main entry point
├── internal/config/config.go        # Configuration management
├── internal/monitor/monitor.go      # Main monitor coordinator
├── internal/logs/monitor.go         # Log file monitoring
├── internal/dht/monitor.go          # DHT network monitoring
├── internal/blockchain/monitor.go   # Blockchain event monitoring
├── internal/system/monitor.go       # System resource monitoring
├── configs/config.yaml              # Configuration file
├── Dockerfile                       # Container definition
├── docker-compose.yml               # Deployment configuration
├── go.mod                           # Go module dependencies
├── Makefile                         # Development tasks
├── .golangci.yml                    # Linting configuration
├── .pre-commit-config.yaml          # Pre-commit hooks
├── .editorconfig                    # Editor configuration
├── DEVELOPMENT.md                   # Development guide
└── README.md                        # This file

Features

  • Gensyn AI Node Monitoring: Tracks training progress, model updates, and node participation
  • Log File Monitoring: Tracks swarm.log, yarn.log, and wandb logs from Gensyn nodes
  • DHT Network Monitoring: Monitors Hivemind DHT peer connections and model synchronization
  • Blockchain Monitoring: Tracks smart contract events on Gensyn testnet (training submissions, rewards)
  • System Resource Monitoring: Hardware metrics and Docker container monitoring for AI nodes
  • Health Check Endpoints: REST API for Gensyn node health and metrics
  • Data Processing: Aggregates and normalizes metrics data from distributed training
  • Alerting: Generates alerts for critical events in the Gensyn AI network

Running Without Docker (Command Line Go)

You can run the GSwarm Sidecar directly from the command line using Go, without Docker. This is useful for development, debugging, or running on bare metal.

Prerequisites

Setup

Clone the repository and install dependencies:

git clone <repository-url>
cd gswarm-sidecar
go mod download

Edit your configuration: Copy and edit configs/config.yaml to match your environment (log file paths, blockchain settings, API tokens, etc).

Running the Sidecar

You can run the monitor directly with Go:

go run cmd/monitor/main.go

Or build a binary and run it:

go build -o gswarm-sidecar cmd/monitor/main.go
./gswarm-sidecar

Stopping: Press Ctrl+C to gracefully stop the sidecar.

Notes:

  • The sidecar will read configs/config.yaml by default. To use a different config, set the CONFIG_PATH environment variable: 
    CONFIG_PATH=/path/to/your_config.yaml go run cmd/monitor/main.go
  • All logs will be printed to stdout by default.
  • Make sure your Go version matches the required version in go.mod for best compatibility.

Configuration Reference

Edit configs/config.yaml to customize:

  • Log file paths for Gensyn AI node logs
  • DHT bootstrap peers for the Gensyn network
  • Blockchain contract details (Gensyn testnet)
  • System monitoring intervals
  • Storage locations for node metrics

Security

  • If auth_token is set, an Authorization: Bearer <token> header is added to each request.
  • Use HTTPS for secure transmission.

Example Event JSON Structure

{
  "node_id": "node-123",
  "timestamp": "2024-06-07T12:34:56Z",
  "event_type": "training_progress",
  "details": {
    "epoch": 5,
    "accuracy": 0.92
  }
}
IntroductionTelegram Monitoring