Incident 2026-11: production database and cache reachable from the internet

Status: contained and remediated, 2026-11-06. Open questions are listed below. Published: 2026-11-06. This note will be updated if we learn more.

Summary

From the first production deploy on 2026-07-05 until 2026-11-06, the production Postgres database (port 5432) and Redis cache (port 6379) of the hosted service at agentwatch.in were reachable from the internet. Redis had no password. Postgres had a password, but a weak default one. The host firewall did not protect them: Docker's own network rules bypass it.

We found this on 2026-11-04 while checking a deploy. On 2026-11-06 we closed the ports, changed every credential the database held, and checked the servers. We cannot rule out that someone read data during the exposure, so we treat everything stored before 2026-11-06 as possibly read.

On-premises and self-hosted installs were not affected by our hosting. If you run AgentWatch yourself from docker-compose.yml from before 2026-11-06, check that ports 5432 and 6379 are not published on a public interface.

Timeline (UTC; times on 2026-11-06 are approximate)

Date	Event
2026-07-05	Production data stores started from the development compose file, which published both ports on all interfaces
2026-07 to 2026-10	Redis log records repeated commands from outside addresses that changed its replication settings
2026-09-06 onwards	Postgres log records about 21,450 failed login attempts, almost all for role names that do not exist on our server
2026-10-17 to 2026-11-04	An unknown party set a password on Redis. The API could no longer reach Redis, and /health reported degraded
2026-11-04	Exposure found during a post-deploy check
2026-11-06 11:06	Outside access to both ports blocked at the firewall; confirmed closed from outside
2026-11-06 11:10	Both data stores re-bound to the loopback interface only; new Postgres and Redis passwords
2026-11-06 11:12	All user API keys and named API keys rotated; internal service secret and session secret rotated
2026-11-06 11:15	Old credentials confirmed rejected (401), new ones accepted; /health back to ok

What was exposed

• Redis (no password until an unknown party set one). It holds short-lived data: the background job queue (jobs can carry the step content that async detectors analyse), rate-limit counters, and live-update messages for the console. Anyone who connected could read what it held at that moment.
• Postgres (weak password). It holds runs, steps, findings, accounts and configuration. User API keys and named API keys were stored in plaintext in this database.

What was not exposed

• Upstream provider credentials and the evidence-signing private key are stored encrypted, with a key that lives outside the database.
• Gateway keys (awgk_…) are stored only as hashes, and none were active.
• MCP OAuth tokens are stored only as hashes, and none existed.
• Environment files with service secrets live on the host, not in either data store.

What the access review showed

• Redis: the log shows outside hosts repeatedly trying to make our Redis copy data from servers they control, a known technique for loading code into Redis. We found no extra module files, no unexpected processes in the container, and no unexpected outbound connections from the host. The old container is kept, unchanged, as evidence.
• Postgres: the failed logins were for names such as postgres, admin and root, which do not exist on our server. Successful logins were not logged at the time, so we cannot say whether anyone logged in with the real role.
• Host: no unexpected processes, outbound connections or scheduled jobs. SSH accepts keys only, and the authorised keys were unchanged since 2026-10-25.
• Audit ledger: the hash chain of the main tenant verifies. This shows the audit rows are consistent. It does not prove nobody changed them: a party with database access could have recomputed the chain.

What we changed

• Both data stores listen on the loopback interface only. A firewall rule drops outside traffic to their ports as a second layer.
• New strong passwords for Postgres and Redis, kept outside the repository.
• Every user API key and named API key was replaced. The internal dashboard-to-API secret and the session secret were replaced, so everyone has to sign in again.
• docker-compose.yml now publishes both ports on 127.0.0.1 only. A test (backend/tests/test_datastore_ports.py) fails the build if any compose file or deploy script publishes a data-store port on a public interface. It was checked against the old file and fails there.

Later the same day:

• Postgres now logs every connection and disconnection with the client address (from 14:10 UTC). Had this been on before, we could say whether the database was read.
• API keys are stored only as SHA-256 hashes. A key is shown once, when it is created or rotated, and can't be revealed afterwards. A copy of the users table no longer lets anyone act as a tenant. Dashboard sign-ins use their own short-lived credentials, also stored as hashes.
• Email/password accounts were reset. Their passwords were held in the dashboard server's configuration, not in the database, so this incident did not expose them; we reset them anyway and now store scrypt hashes. Each holder sets a new password from a one-time link.

What remains unknown

• Whether anyone logged in to Postgres with the real role, and if so, what they read or changed.
• What Redis held at each moment it was accessed.
• Who set the Redis password, and why.

What you should do

• Sign in again at app.agentwatch.in.
• Replace any API key you use in the SDK, Cursor hooks, scripts or integrations: Account → Danger zone → Rotate primary API key. The new key is shown once; copy it then. Old keys now return 401.
• If you sign in with email and password, ask your workspace admin for your reset link. Google and GitHub sign-in are unaffected.
• Assume that data you sent to the hosted service before 2026-11-06 may have been read. That includes prompts, tool inputs and outputs captured as steps. If any of it contained credentials, rotate them.

Questions: agentwatchsupport@gmail.com.