Grafana has announced the release of Loki 3, with improvements including query acceleration with Bloom filters and native OpenTelemetry support.

Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. It consists of a set of components that can be composed into a fully featured logging stack.

Compared to other log aggregation systems, Loki does not do full text indexing on logs. Instead, it stores compressed, unstructured logs and only indexes metadata, making it simpler to operate and cheaper to run. It also indexes and groups log streams using the same labels used by Prometheus, so users can switch between metrics and logs using the same labels.

The main headline improvement to this version of Loki is query acceleration with Bloom filters, an experimental feature designed to help users find specific log data faster.

The developers say that previous releases of Loki were good at finding log data very quickly given information about roughly where to look such as label selectors to filter on. The addition of Bloom filters to this release are designed for searches where it isn't possible to use filters. They say that until now, if a user wanted to find one particular message in a log, Loki had to look every log line within all matching streams. Bloom filters add the ability to search for strings, such as order ID or user ID, skipping large chunks of data that don't contain the information you're seeking. They say that early internal tests suggest that with Bloom filters, Loki can skip a significant percentage of log data while running queries:

"Our dev environment tests show that we can now filter out between 70% and 90% of the chunks we previously needed to process a query"

A bloom filter is a probabilistic data structure that is based on hashing, named after the inventor, Burton Howard Bloom. 

The second main change is support for OpenTelemetry, which the Loki team say 54% of the respondents in Grafana's 2024 Observability Survey said they were using more than they did last year. OpenTelemetry aims to provide a vendor-agnostic way to easily instrument applications or systems, regardless of their language, infrastructure, or runtime environment.

Until now, if you wanted to view OpenTelemetry format logs in Loki, they had to be ingested using Loki Exporter which serialized the data in JSON, meaning it then had to be deserialized at query time to interact with OTel attributes and metadata in log records.

The new version has native OpenTelemetry ingestion support, avoiding the need for these stages.

Loki 3 is available now.

More Information

Grafana Website

Loki On GitHub

Related Articles

Grafana 11 Improves Metrics

What's the Best Way to Effectively Monitor a Kubernetes Cluster?

Grafana Adds New Tools

Grafana Ships Grafana 10

Grafana Updates Cloud Platform

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info