We Scanned 27,075 Real Developer Prompts to ChatGPT and Found Three Live API Keys

Somewhere in 2023, a developer did what the median developer now does several times a day: they pasted a Python file into ChatGPT. The file contained a live OpenAI API key — the full 51-character sk-... string, sitting right there in the source.

The conversation got shared. The share link got committed to GitHub. Researchers scraped it into a public dataset. And this month, our detection engine flagged it — three years after the paste.

That key is one of three genuine, authentic-format API keys we found in a single weekly snapshot of real ChatGPT traffic. Not synthetic test data. Not a red-team exercise. Real developers, doing real work, leaking real credentials — exactly the behavior HeimWall exists to catch at the source.

This post is the full readout: the dataset, the method, the numbers, and — because we hold ourselves to naming limitations before anyone else does — the false positives and the things this benchmark cannot tell us.

The dataset: real prompts, not lab prompts

Most secret-detection benchmarks are built from labeled corpora: known secrets planted in known positions, so you can compute precision and recall. Those are useful, and we run them. But they share a weakness — the input is curated. It tells you how a scanner performs on secrets, not how it behaves on a Tuesday.

DevGPT is different. It's the MSR 2024 Mining Challenge dataset: ChatGPT conversations that real developers shared publicly, scraped from GitHub and Hacker News. We used the final weekly snapshot (October 2023) and scanned user prompts only — the direction of traffic HeimWall actually guards. The tally:

• 3,559 source records, containing 4,733 shared ChatGPT conversations
• 27,075 developer prompts scanned
• Prompts referenced from files (21,089), issues (2,027), commits (1,602), Hacker News threads (1,139), pull requests (996), and discussions (222)

One thing DevGPT does not have: ground-truth labels for sensitive content. Its annotations describe software artifacts, not secrets. So this benchmark cannot produce a recall number — and we won't pretend otherwise. What it measures is something the labeled corpora can't: HeimWall's firing rate on genuine production-shaped input, followed by manual triage of everything that fired. In other words: how noisy is this thing in the real world, and what does it actually catch?

That's the alert-fatigue question, and for an inline security tool it's the question that decides whether the tool survives contact with engineers.

The headline number: 1.12%

Across 27,075 real prompts, HeimWall raised at least one alert on 303 of them — a firing rate of 1.12%, or roughly one alert per 89 prompts.

By severity:

Severity	Count	Rate
Critical	11	0.041%
High	91	0.336%
Medium	70	0.259%
Low	131	0.484%

For context on what that means in a workflow: an engineer sending 30 prompts a day would see an alert roughly once every three days, and a critical alert a few times a year. That is a friction budget an engineer will tolerate. Our own principle — soft notifications beat hard blocks in 19 out of 20 cases — only works if the notification is rare enough to mean something. 1.12% clears that bar.

The firing rate also varies with context in a way that matches intuition about when developers leak:

Artifact context	Prompts	Flagged	Rate
Pull requests	996	19	1.91%
Issues	2,027	30	1.48%
Files	21,089	232	1.10%
Commits	1,602	15	0.94%
Hacker News	1,139	7	0.61%
Discussions	222	0	0.00%

PR and issue contexts fire at 1.5–1.9% — the debugging surface, where developers paste error output, config, and failing records wholesale. The dataset confirms the intuition at scale: the leak risk concentrates where the developer is stuck and pasting fast.

What was actually in there

Firing rate is the operational number. The security question is what's inside the flagged set. Since there are no labels, we manually classified every flagged item, deduplicating by matched value (one leaked key tends to recur across a conversation). Everything below is manual verification, not automated ground truth.

Confirmed leaked secrets: 3 distinct live-format API keys. (The same key can appear in several flagged prompts; these are unique values after deduplication.)

• Two OpenAI API keys — authentic 51-character sk-... format, pasted inside Python source
• One Google API key — authentic 39-character AIzaSy... format, sitting in an Expo app.json

(Our Slack-token rule also fired four times — all four collapsed to a single xoxb-your-token placeholder. False positive, counted as such.)

Personal data: 49 real personal email addresses. Of 62 distinct flagged emails, 49 were genuine personal or institutional addresses — Gmail accounts, company domains — pasted into prompts alongside code and logs. The other 13 were machine-generated or example addresses.

Three keys and forty-nine emails may sound small. Consider the denominator. This is one weekly snapshot of conversations developers chose to share publicly. The prompts nobody shares — the ones going straight from a Cursor composer to a model endpoint, all day, across your whole team — are the invisible bulk of the iceberg. If the publicly-shared tip contains live credentials, the working assumption for the rest writes itself.

A note on responsible handling: the three confirmed keys have been publicly exposed on GitHub and in this research dataset since 2023, and should be assumed long since revoked. We report them as detection evidence, not as active secrets, and we are not republishing the values or their locations.

The false positives, in full

A benchmark readout that skips the noise analysis is marketing. Here's ours.

Two rules account for the large majority of false alerts:

SECRET_UUID — 146 hits, 48% of all alerts. UUIDs are secret-shaped (high entropy, fixed format) but usually aren't secrets — they're database IDs, request IDs, trace IDs. On developer input, they're everywhere. The fix is known and boring: gate UUID alerts on surrounding context (assignment to a credential-named variable, auth headers) rather than shape alone.

PII_CREDIT_CARD — 55 prompts flagged (0.203%), zero real cards. Those 55 prompts contained 342 distinct card-shaped values, and every single one was a timestamp, numeric ID, or hash that happened to pass a Luhn check. Luhn is a checksum, not a card detector; on log-heavy input it fires constantly. The fix is known: validate the issuer prefix (IIN) before alerting, not just the checksum.

We're publishing these numbers because they're addressable engineering debt, not fundamental limits — and because the honest version of "1.12% firing rate" includes the fact that roughly half of it comes from one over-eager rule that we know how to tighten.

Strip the noise and the picture sharpens: across the 303 flagged prompts, the flagged set contained 52 genuinely sensitive distinct values — 3 live-format credentials plus 49 real personal emails. That's the trade a security guardrail actually makes: a modest, tolerable alert rate in exchange for catching the leaks that matter. The relevant operating point is not "zero false positives"; it's "rare enough to respect, sensitive enough to catch the real thing." This scan says HeimWall is at that point — with headroom to improve.

What this benchmark can't tell you

In the spirit of naming the limits before you do:

• No recall number. DevGPT has no sensitive-content labels, so leaks HeimWall missed are unquantifiable here. For precision/recall we use labeled corpora: on CredData (15,255 labeled credentials across 19.9M lines), the same engine scores 0.573 precision at 0.369 recall — the highest precision and the best precision–recall balance (F1 0.449) of any non-ML detector's published results on that corpus, ahead of Gitleaks on both axes — and on the Gretel PII benchmark its redaction path removes 81.5% of in-scope entities with no observed over-redaction. Full technical report coming.
• Tier 1 only. These results reflect our deterministic rule engine — 47 hand-written detection rules for hard-shaped secrets and PII. The on-device semantic classifier (Tier 2) was not in the loop for this scan.
• Shared-conversation bias. Developers presumably share their less embarrassing conversations. If anything, that biases the real-world leak rate upward relative to what we measured.
• 2023 traffic. The snapshot predates the agentic-IDE era. Today's prompts carry more context automatically — attached files, repo chunks, terminal output — which grows the leak surface, not shrinks it.

Why we ran this at all

Every claim we make — engineers paste secrets without knowing, and nobody sees it happen — is falsifiable. It should be tested against real traffic, not asserted from a slide.

So we tested it. In one public sliver of real developer↔LLM traffic: live-format credentials, real personal emails, and a leak rate that concentrates exactly where developers debug under pressure. The threat model holds. And the tool built to catch it fires rarely enough to live inline, at the source, before the prompt leaves the machine.

That's the whole thesis, measured.

Where this goes

The engine that produced these numbers is live today. The free individual app — the same Tier-1 detection, running on-device in your menu bar — is available now at heimwall.ai/individual. Download it and it starts catching what you paste into Cursor, Claude Code, and Copilot before it leaves your machine.

For engineering teams, we're onboarding design partners for the manager dashboard — the same signal, aggregated across the team, without reading a single prompt:

1. Request early access at heimwall.ai. We reach out personally to the first cohort.
2. For SOC 2 progress, on-prem, or the full methodology writeup, email founders@heimwall.ai.

The leak is invisible only because nothing has been built to see it. We built it, and then we pointed it at 27,075 real prompts to make sure it works.