Learning

Seven concepts that separate operators from maintainers.

Not a tutorial series. Not a certification prep track. These are the ideas we come back to on every engagement — the ones that decide whether a platform pays for itself or quietly bleeds budget.

The three pillars — and why they aren’t enough

Logs, metrics, and traces are the classical pillars of observability. The pillars tell you what happened; they don’t tell you why. Mature platforms add profiles, events, and continuous change data — and correlate all of them through shared identifiers (trace_id, service.name, resource attributes) so an engineer can pivot from a metric spike to the exact log line without rebuilding context in their head.

logs metrics traces profiles correlation

OpenTelemetry is the contract, not the vendor

OpenTelemetry (OTel) is a vendor-neutral specification for how telemetry is produced, propagated, and shipped. Instrument once in OTel, and you can send the same data to Elastic, Datadog, Grafana, Honeycomb, or a home-grown ClickHouse warehouse without touching application code. Teams that skip OTel end up locked into whichever agent they installed first — and pay for it at renewal.

OpenTelemetry OTLP collector vendor-neutral instrumentation

Cardinality is the hidden cost of observability

Every unique combination of metric label values (user_id, request_id, k8s pod name) creates a new time series. Metric cardinality grows multiplicatively and quietly eats budget — a single unbounded label can 10× your bill in a week. Design your label taxonomy on day one, cap high-cardinality dimensions at the collector, and put the ones you actually need on the query in logs or traces, not metrics.

cardinality labels TSDB cost engineering

Relevance is a design decision, not a default

Search engines like Elasticsearch and OpenSearch ship with BM25 — a strong statistical baseline for ranking. It is almost never the right answer for production. Real relevance is engineered: analyzer choices per language, boosts per field, learning-to-rank models, hybrid dense+sparse retrieval, and rigorous offline evaluation with judged queries. If your search team can’t show you an NDCG chart, they’re guessing.

BM25 relevance tuning NDCG hybrid search learning to rank

Vector search is a different data structure, not a magic upgrade

Embeddings turn text (or images, or code) into fixed-length vectors, and ANN indexes (HNSW, IVF) let you retrieve the nearest ones in milliseconds. That solves semantic recall — questions where keywords don’t match. It does not solve exactness (product SKUs, error codes), governance (freshness, deletions), or ranking beyond similarity. The right architecture is usually hybrid: BM25 for lexical precision, vectors for semantic recall, reranking on top.

embeddings HNSW ANN hybrid retrieval RAG

SIEM value comes from detections, not from data volume

Onboarding every log source into your SIEM feels like progress and reads like a bill. The lever that matters is detection engineering: MITRE ATT&CK-aligned rules, tuned to your environment, versioned in git, with unit tests and false-positive budgets. Pipeline tools like Cribl or Elastic ingest pipelines let you route, reduce, and enrich data before it hits the license tier — cutting cost while keeping the detections you actually run.

SIEM detection engineering MITRE ATT&CK Cribl data reduction

Retention & tiering is where architecture meets the invoice

Hot storage (SSD, fast query) is expensive; warm and cold tiers (object storage, decompressed on demand) are 10–50× cheaper. The engineering decision is which data belongs where and for how long — driven by legal retention, incident-response reachback (typically 90 days hot, 12 months warm), and compliance frames (PCI, HIPAA, SOX). Platforms like Elastic frozen tier, Splunk SmartStore, and ClickHouse tiered storage exist for this exact reason; use them.

hot/warm/cold frozen tier SmartStore retention compliance

Keep going

Want the terms defined?

Our Glossary covers 100 of the most-searched terms across observability, search, and security data — plain-English definitions from engineers, not marketing.

Open the Glossary Talk to an engineer