Observability is an exfiltration surface: what our analytics collected by default

We added analytics to answer a performance question. A page was slow and lab tools disagreed with each other, so we instrumented real-user metrics and waited a week for data.

When the data arrived we ran a query to see what properties were available. That query returned one raw event, and the raw event was the interesting part.

It contained a full IP address, latitude and longitude to four decimal places, city, postal code, country, timezone, the complete user agent string, and the full URL including its query string.

We hadn't decided any of that. It arrived because the defaults are permissive and we hadn't read them.

The thing you add to be careful is the thing nobody audits

Telemetry gets installed for good reasons. You want to know what's slow, what breaks, where people give up. It's the responsible move, and it's usually made by whoever cares most about doing the job properly.

Which is exactly why it doesn't get reviewed. A feature that touches user data gets scrutiny. A monitoring tool that touches all of it gets a config block and a deploy.

We treat a database as something to protect. We treat a third-party analytics account as somewhere the numbers live. It holds a copy of your users' behavior, keyed to their location and device, retained on someone else's schedule, readable by anyone with the login.

So we stopped reading performance data and audited what we were collecting instead.

Session recording was on, and we hadn't turned it on

The audit found session replay enabled at the project level, sampling every session, with no minimum duration and no URL blocklist.

Nobody had chosen that. It was on because it was on.

The recordings included our admin panel. An authenticated back office where we edit posts, read partner inquiries, and log in, captured as video, at full sampling, for thirty days.

That's the part worth sitting with. Not that a setting was wrong, but that a system was recording our most sensitive surface for weeks and the only reason we found out is that we happened to be looking at something else.

Input masking doesn't mask text

The configuration had maskAllInputs set to true, which is the reassuring one. Type a password, the replay shows dots.

It masks input field values. It does not mask text rendered on the page. That's a separate setting, and it was at its default of off.

So the login form was protected. So was the public contact form, which we'd deliberately excluded from capture and had verified.

The page that was not protected is the one inside the admin panel where a submitted inquiry is displayed for us to read. A name, an email address and a message, rendered as ordinary text on a page nobody had thought of as a place where that data appears.

The form was the surface we'd secured. The inbox was the one we hadn't.

The name reads as complete coverage. It isn't wrong, exactly. It does what it says. But "all inputs" sounds like "everything a person typed," and the two are different once the thing a person typed is being displayed back somewhere else.

We've written about this shape before. A check that reports success for the thing it measures, while a closely related property nobody measured goes the other way. The setting was doing its job. Our reading of it was doing a different job.

The documentation and the artifact disagreed twice

There's a configuration option for IP handling. We went to set it.

Reading the installed bundle rather than the documentation, it logs that the option has no effect at all and has been deprecated. IP collection and geolocation happen server-side, after the request arrives, and no client configuration touches them.

The fix exists, but it's a dashboard toggle, not a line of code. If we'd set the option and moved on, we'd have shipped a config change that read as a fix, passed review, and did nothing.

Worth generalizing: for anything you can't verify from the outside, the installed artifact is a better source than the documentation. Docs describe intent at the time of writing. The bundle is what's running.

There's a sharper version of that, which we found when checking what the replacement setting actually does. The current documentation says one thing. An older issue thread, written against a previous plugin architecture, says the exact opposite in plain terms. Both were accurate about their own moment.

A search that surfaced the older one first would have produced a confident, wrong paragraph in this post. In this post, specifically, arguing that you should read the artifact rather than the documentation.

Two mechanisms were collecting the same thing

Our own metrics event was firing correctly and skipping the admin panel, which we'd verified.

The SDK's native performance capture was also on, enabled server-side, and firing on admin. Same four metrics, twice, from two systems, one of which nobody had enabled.

It hadn't corrupted anything, because every query we'd run named our event explicitly. But it easily could have. A percentile over a blended set from two mechanisms with different sampling would have been wrong in a way that looks fine.

The audit turned up a related surprise while counting those events. The analytics project isn't scoped to this site. A second property reports into it, which we knew, and had never thought about as a scope question. Any project-level setting we changed would have changed for both, and any event volume we reasoned about was two properties' worth.

That one is small and it's the same thing again: a default whose reach nobody had checked.

The policy asserted something nobody had checked

Our privacy policy said we don't use session recording.

Not an omission. A positive claim about a property of a system, written in good faith, never verified against the system.

That's the same failure as a passing test that isn't testing anything, applied to a document instead of a suite. Someone wrote down what they believed the configuration to be. Nobody went and looked.

The uncomfortable version of this is that a privacy policy is a public assertion about behavior, and most of them are written from the same place: what the team intends, rather than what the stack does. Ours was accurate about the parts we'd built and wrong about the parts we'd installed.

What we changed

Session recording is off. Two-factor is on for the account with access. IP addresses are discarded before storage, and location enrichment still runs because it happens before the discard, so we keep a country and a city and lose the address. The duplicate performance capture is disabled. Analytics no longer initializes at all on admin routes, rather than initializing and being told to stay quiet.

One detail about that location, since the draft of this post got it wrong and the correction is more interesting than the claim. The original event carried latitude and longitude to four decimal places, which reads as eleven-metre precision. It isn't. It's a city centroid printed at a precision it doesn't have. The number looks like where someone is standing and means roughly which city they're in.

The admin change had a detail worth keeping. The SDK installs global listeners when it initializes, and it offers no way to un-initialize. Once it's running in a tab, no later check can stop it capturing clicks. Suppressing pageviews would have looked like a fix while leaving the larger category untouched. The only version that works is never starting it.

The recordings are gone now. We deleted some of them. The rest expired on their own thirty-day clock while we were still deciding what to do about them, which produces the same end state and is not the same thing.

The through-line

None of this was a breach. Nothing left a system it shouldn't have. The blast radius was small because the site is small.

What it was is a copy of behavioral data, in a third-party system, on someone else's retention schedule, in a shape nobody had chosen, protected by one login, described by a policy that said it wasn't happening. Every part of that arrived through a default.

The instinct that installs observability is the same instinct that ships tests and writes checks. It's the instinct to know rather than assume. It just stops one step short, at the point where the tool itself becomes the thing you're assuming about.

Read what you installed, not what you meant to install.

Written by Synthetixis, an AI-native product studio. More on what most AI software gets wrong.