The Observability industry is built on a fundamental promise: Send all of your telemetry data to your vendor of choice, and they will apply their magic sauce to turn that telemetry into “insights.”
In the beginning this promise was predicated on fundamental truths inherent in all SaaS models of the era: the data being sent was not too large, and not too valuable. We were content to ship our data off to become part of our vendors’ statistical data models.
But that balance has shifted. Data volumes have grown exponentially – observability data doubly so. With this increased volume has come an even larger increase in Observability costs that can no-longer be ignored. Every vendor now charges directly or based on some approximation of bytes-ingested.
And now we all dream of using our data to train our own models; wouldn’t that be liberating? This is only possible if we retain our data, or if our vendors were to somehow open up APIs allowing us to access it within their walled gardens.
OpenTelemetry is just the beginning of this sea-change. First it solved the problem of massive duplicated effort and lack of interoperability in the instrumentation layer. Now organizations are beginning to adopt the OpenTelemetry Collector as a way to change telemetry destinations in an instant.
But Observability can never be truly vendor-neutral – and open – until the storage layer is open, standardized, and hostable anywhere.
Josh is a seasoned software developer with over a decade of experience, specializing in a broad range of topics including operations, observability, agile methodologies, and accessibility. His passion for technology is matched by his enthusiasm for sharing knowledge through public speaking. Currently, Josh serves as a Developer Advocate for Altinity, where he creates educational content on ClickHouse and OpenTelemetry, and he is a contributor to the OpenTelemetry project.