A missing settings row deleted my ANTHROPIC_API_KEY at boot

On a headless ARM box, I put LLM_PROVIDER=anthropic and ANTHROPIC_API_KEY in .env and started the gateway. It came up with no provider. You could call this a config-precedence bug, but “the database wins” undersells it. The settings store had no value for the key, and it still deleted the value the environment had.

llm_provider is null, so delete process.env.ANTHROPIC_API_KEY runs

Our gateway’s LLM loader reads the provider from a SQLite settings table. The setup wizard is what fills that table, and because the box was headless, nobody ran the wizard.

const dbProvider = getSetting('llm_provider');
if (dbProvider === 'anthropic') {
  const dbAnthropicKey = getSetting('anthropic_api_key');
  if (dbAnthropicKey) process.env.ANTHROPIC_API_KEY = dbAnthropicKey;
} else {
  delete process.env.ANTHROPIC_API_KEY;
}

null is not 'anthropic', so the key was deleted. I had assumed LLM_PROVIDER picked the provider, but that branch never reads it. The log said No provider configured — open Settings to select one. That was true of the table and false of the machine, and it sent me to a settings UI that a headless box doesn’t have.

The delete is deliberate. A Claude CLI child process that inherits the key bills the API instead of the subscription. The intent was right. The trigger was wrong. The minimal fix is to read what the operator set first, and to delete only when someone explicitly chose a different provider:

const provider = process.env.LLM_PROVIDER ?? getSetting('llm_provider');

if (provider === 'anthropic') {
  if (!process.env.ANTHROPIC_API_KEY) {
    const dbAnthropicKey = getSetting('anthropic_api_key');
    if (dbAnthropicKey) process.env.ANTHROPIC_API_KEY = dbAnthropicKey;
  }
} else if (provider) {
  // An operator explicitly picked a non-Anthropic provider.
  delete process.env.ANTHROPIC_API_KEY;
}
// provider unset everywhere: touch nothing, log it, let the operator decide.

That patch stops the boot-time delete. It still writes process.env from the loader, though, and that’s the part I got wrong in the first place. The better shape comes later in the post.

One common fix is env-over-DB, and here it’s only half the answer

One common fix is to let env vars win. Podly’s issue #190 does exactly that: env vars take precedence over DB and UI values and never get persisted. LiteLLM’s PR #32024 fixes a close relative, where a stale DB false beat an explicit STORE_MODEL_IN_DB=True.

Here is what env-first would and wouldn’t have fixed for me. If the loader had read LLM_PROVIDER from the environment before the table, my box resolves to anthropic, the else never runs, and the key survives. So for this incident, precedence alone would have been enough. It would not have removed the delete. Suppose an operator sets only ANTHROPIC_API_KEY and leaves LLM_PROVIDER unset. Env-first reads undefined, falls through to the empty table, gets null, and deletes the key again. Precedence decides which value wins. Mutation means the losing side can destroy a value the operator set. My bug needed both.

Invariant: never mutate the parent’s env; scrub the child’s at spawn

The rule I should have started with:

With the settings store empty, resolved config equals the environment. After boot and after the first request, process.env still matches what the operator started the process with. A secret a child must not see gets removed from that child’s env when you spawn it, never from the parent’s.

That fits the reason for the delete instead of fighting it. The billing concern belongs to the Claude CLI child, so the fix belongs at the spawn:

import { spawn } from 'node:child_process';

function spawnClaudeCli(args: string[], useSubscription: boolean) {
  const env = { ...process.env };           // a copy
  if (useSubscription) delete env.ANTHROPIC_API_KEY;
  return spawn('claude', args, { env });
}

The parent keeps the key for its own API calls. The child never sees it. No boot ordering or missing row can erase a value the operator set.

Check your own stack: diff the env after the loader has run

Point your service at a fresh, empty settings database, however it normally chooses one, and skip every seed or wizard step. Then snapshot the environment before import, and snapshot it again only after the config loader has actually run. A snapshot taken right after import will miss a loader that runs lazily on the first request or on a settings reload. It will pass, and it will be wrong. So force the loader to run: make one real request, or call the loader function directly.

// env-diff.mjs — run from your project root with your normal .env loaded
const before = { ...process.env };

await import('./server.js');              // your entry point

// Make the loader run before the second snapshot. Pick one:
await fetch('http://127.0.0.1:3000/');    // a real request to your port
// or: (await import('./config.js')).loadConfig();

const after = { ...process.env };
const keys = new Set([...Object.keys(before), ...Object.keys(after)]);
const removed = [...keys].filter(k => k in before && !(k in after));
const added   = [...keys].filter(k => !(k in before) && k in after);
const changed = [...keys].filter(k => k in before && k in after && before[k] !== after[k]);

console.log({ removed, added, changed });
process.exit(removed.length || added.length || changed.length ? 1 : 0);

It passes when all three lists are empty. If the service has a settings-reload path, trigger that too, then diff again.

For a Python service, the same idea as a sketch. create_app and load_config stand in for whatever your framework calls them:

# sketch — replace create_app / load_config with your own entry points
import os
before = dict(os.environ)
app = create_app()        # pointed at an empty settings store
load_config(app)          # or make one real request through a test client
after = dict(os.environ)
print("removed:", sorted(before.keys() - after.keys()))
print("added:",   sorted(after.keys() - before.keys()))
print("changed:", sorted(k for k in before.keys() & after.keys() if before[k] != after[k]))

A diff only catches the paths you exercised, so also grep for every place the code writes the environment:

grep -rnE \
  -e 'delete process\.env' \
  -e 'process\.env(\.[A-Za-z_][A-Za-z0-9_]*|\[[^]]*\])[[:space:]]*(\?\?|\|\||&&)?=[^=]' \
  -e 'Object\.assign\([[:space:]]*process\.env' \
  -e 'os\.environ(\[[^]]*\][[:space:]]*=[^=]|\.pop|\.update|\.clear|\.setdefault|\.popitem)' \
  -e 'del os\.environ' \
  -e 'os\.(putenv|unsetenv)' \
  -e '(^|[^A-Za-z_.])(setenv|unsetenv|putenv|clearenv)\(' \
  -e 'env::(set_var|remove_var)' \
  -e 'os\.(Setenv|Unsetenv|Clearenv)' \
  --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=dist \
  --exclude-dir=build --exclude-dir=target --exclude-dir=vendor \
  .

This matches process.env.X = undefined and = '', ??= and ||=, Object.assign(process.env, …), os.environ.update and .clear(), putenv and setenv from C or libc wrappers, Rust’s std::env::set_var and remove_var, and Go’s os.Setenv and os.Unsetenv. It scans the whole tree, because the loader is rarely where you’d expect. Zero hits is the pass condition. Every hit is a place where config gets written instead of read. Move the write into a copy you pass to the child (spawn(..., { env }), subprocess.run(..., env=...), Rust’s Command::env_remove, Go’s cmd.Env), or into a config object that your own code reads. My minimal patch above would still show up here. That’s the point: the grep isn’t satisfied until the parent’s environment is read-only.