Observability

Netsy exposes Prometheus-compatible metrics and structured logs for monitoring cluster health, debugging failures, and alerting on degradation.

The observability design is intended to explain:

• the Node’s current role and state
• if the Primary is using sync or quorum writes
• whether Replicas are healthy enough for quorum
• if startup, catchup, draining, or election has stalled
• whether object storage, replication, or compaction is the bottleneck

Metrics

Naming and instrumentation rules:

• All metrics use the netsy_ prefix.
• Use base units in metric names e.g. durations use _seconds, sizes use _bytes.
• Use counters for cumulative events, gauges for current state, histograms for latency/size distributions.
• Keep labels bounded and low-cardinality. Avoid labels containing keys, revisions, object names, addresses, or error strings.
• Role-specific metrics use netsy_primary_, netsy_elector_, or netsy_replica_ prefixes and are exposed via a custom prometheus.Collector that only emits them while the Node is currently in that role.
• When a Node leaves a role, role-specific metrics disappear from scrape output. They are not set to 0.
• Do not rely on gauges for short-lived workflows that may complete between scrapes. Prefer counters, histograms, and structured logs for loading stages, preflight work, elections, and flushes.
• Standard gRPC server interceptor metrics (e.g. grpc_server_handled_total, grpc_server_handling_seconds_bucket) should be registered via go-grpc-middleware or equivalent for per-RPC observability across both Client and Peer gRPC servers. These complement the Netsy-specific metrics and are not documented individually here.

Node State

Each state metric is a gauge vector with a state label. Exactly one label value is 1 at any time and all others are 0.

Revisions

Startup, Catchup, and Draining

These metrics make long-running phases visible beyond the high-level state gauges.

Client API (Read and Write Proxy)

These metrics cover the Client API surface, including range (read) requests served directly by any Node and write requests proxied by Replicas to the Primary.

Write Path

The Primary chooses between synchronous object storage writes and quorum writes based on Replica health and quorum configuration.

Replica Health and Replication

These metrics describe the Primary’s view of Replica health and quorum eligibility for write decisions.

Elector Cluster View and Elections

These metrics describe the Elector’s view of registered and healthy Nodes, and the results of Primary election attempts.

Chunk Buffer

Object Storage

Write metrics carry kind (chunk or snapshot) and mode (sync or async) labels so operators can distinguish client-facing sync writes from background buffer flushes. Read metrics carry only result — reads are off the hot path (bootstrap, preflight, discovery). Only explicitly instrumented write paths record metrics; internal metadata writes (discovery, registration) are excluded.

Snapshots

Snapshot creation is a Primary-only maintenance operation that compacts Chunk files into a single Snapshot file. These metrics are separate from the general object storage write metrics to give visibility into snapshot scheduling and lifecycle.

Retries

Every retry path has a corresponding counter so operators can see degradation before it becomes a failure.

Service Discovery and Registration

Watches and Compaction

Note: netsy_watch_min_revision is emitted per Node. Detecting compaction-blocking skew requires comparing this metric across Prometheus instance labels (e.g. min(netsy_watch_min_revision) by (instance)).

Structured Logging

Netsy uses structured logs for all operationally significant events. Each log entry includes at minimum:

• msg
• cluster_id
• node_id
• timestamp

Logs should use stable event names and stable key names. Put detailed error text in error. Put short, bounded failure categories in reason so logs can be aggregated and alerted on safely.

State and Lifecycle Events

Logged whenever a Node changes state or enters or exits a major lifecycle phase.

When local database state is newly created, discarded, or rebuilt, Netsy should emit dedicated lifecycle logs such as local_db_initialized, local_db_rebuild_started, and local_db_rebuild_completed with a bounded reason field.

Election Events

Logged when an election starts, advances, completes, or fails.

Write Path and Transaction Events

Logged when the Primary switches write mode or when a write fails in an operationally interesting way.

The write_transaction message is logged at debug level for every completed write transaction on the Primary. It includes path, revision, healthy_replicas, required_receipts, received_receipts (quorum only), and duration_ms. This is intentionally debug-level to avoid noise during normal operation, but is invaluable for diagnosing why individual transactions chose a particular write path.

Registration Events

Logged by the Elector when Node registration changes.

Object Storage and Compaction Events

Debugging

Key Relationships

When diagnosing issues, the following metric relationships are useful:

• Quorum eligibility: compare netsy_primary_healthy_replicas and netsy_primary_required_receipts. If healthy Replicas fall below the required threshold, netsy_primary_write_path{path="sync"} becomes 1.
• Replica-specific degradation: compare netsy_primary_healthy_replicas, netsy_primary_required_receipts, netsy_primary_replication_streams, and each Replica’s own netsy_replica_receipt_age_seconds. Use structured logs to identify which specific Replica is timing out, disconnected, or no longer counted toward quorum.
• Replication lag: compare netsy_latest_revision and netsy_committed_revision across Nodes to identify Replicas that are behind the current committed point.
• Object storage lag: on the Primary, compare netsy_latest_revision and netsy_primary_object_storage_revision to measure how far async object storage writes are behind quorum-committed data.
• Buffer pressure: rising netsy_primary_chunk_buffer_bytes together with rising netsy_primary_chunk_buffer_age_seconds suggests async flushes are not keeping up.
• Retry pressure: rising netsy_retries_total for operation="object_storage_write", operation="heartbeat_send", or operation="receipt_send" indicates degradation in progress. A sustained increase in object storage write retries may precede the Primary entering Draining.
• Loading stalls: if netsy_state_health{state="loading"} remains 1, inspect recent loading_stage_started and loading_stage_completed logs together with netsy_loading_stage_duration_seconds to see which startup step is slow or failing.
• Local DB rebuild churn: increases in netsy_local_db_rebuilds_total indicate repeated local-database resets or rebuilds. Correlate with local_db_rebuild_started and local_db_rebuild_completed logs to see the trigger.
• Election stalls: inspect netsy_elector_primary_election_duration_seconds, netsy_elector_primary_election_failures_total, netsy_elector_primary_election_contacts_total, and election logs to determine whether the cluster is blocked on prior-Primary contact, node contactability, or candidate validation.
• Compaction stalls: rising netsy_watch_min_revision skew across Nodes (compare by Prometheus instance label), long netsy_compaction_duration_seconds, or repeated increments in netsy_primary_compaction_confirmation_failures_total indicate watch-admission, confirmation, or local compaction-work issues.
• Snapshot staleness: a rising netsy_primary_snapshot_age_seconds or repeated netsy_primary_snapshot_creations_total{result="error"} increments indicate snapshots are not being created, which increases loading/recovery time for new Nodes.
• Registration issues: rising netsy_retries_total{operation="node_registration"} or long netsy_node_registration_duration_seconds during loading indicate Service Discovery or Elector connectivity problems. Correlate with node_registered / node_deregistered logs.
• Client API health: rising netsy_client_requests_total{result="error"} or elevated netsy_client_request_duration_seconds indicates client-facing degradation. Compare netsy_replica_proxy_requests_total error rates on Replicas to determine whether failures originate from the Replica’s proxy path or the Primary itself.

Alerting Notes

• Alerts should be written with Prometheus staleness semantics in mind. Role-specific metrics disappear when a Node is no longer the Primary or Elector.
• Prefer alerting on sustained conditions using for: rather than single scrape failures.
• Prefer counters for rate-based alerts and gauges for current-role or current-health dashboards.