memind/docs/agent-run-worker-rollout-runbook.md

# Agent Run Worker Rollout Runbook

Date: 2026-07-02

This runbook controls the long-running Tool Gateway / external `agent-run-worker` rollout.

Production safety rules:

- Do not delete PG data.
- Do not delete or rewrite MindSpace user data. On current 103 production, MindSpace Service root is `/Users/john/MindSpace`; `/Users/john/Project/Memind/MindSpace` is legacy compatibility/storage context. Confirm against [103 runtime topology](./103-runtime-topology.md) before any file operation.
- Keep H5 public traffic on `https://mm.tkmind.cn`.
- Treat `m.tkmind.cn` / 105 tunnel as legacy rollback transport, not the primary H5 path.
- Keep default production state closed unless a rollout window is explicitly active.

## Default Closed State

Expected production state:

```bash
cd /Users/john/Project/Memind
curl -fsS https://mm.tkmind.cn/api/runtime/status
node scripts/check-agent-run-worker.mjs
node scripts/runtime-slo-report.mjs
```

Expected values:

```text
toolRuntime.codeRunPolicy.enabled=false
toolRuntime.codeRunsEnabled=false
toolRuntime.queue.autoDispatch=true
toolRuntime.queue.toolGateway.enabled=false
check-agent-run-worker.ok=true
check-agent-run-worker.expected=disabled
check-agent-run-worker.running=false
runtime-slo-report.ok=true
runtime-slo-report.failures=[]
```

## One-user Canary Profile

Use only for a controlled john-user canary.

Portal `.env` additions:

```bash
MEMIND_AGENT_RUN_AUTODISPATCH=0
MEMIND_AGENT_CODE_RUNS_ENABLED=1
MEMIND_AGENT_CODE_RUNS_USER_IDS=a6fb1e97-2b0f-447b-b138-4561d8e5c53e
MEMIND_AGENT_CODE_RUN_TASK_TYPES=small_patch
MEMIND_AGENT_CODE_RUNS_REQUIRE_VALIDATION=1
MEMIND_TOOL_GATEWAY_ENABLED=0
MEMIND_TOOL_GATEWAY_DRY_RUN=0
```

Worker installer environment:

```bash
MEMIND_AGENT_RUN_WORKER_START=1
MEMIND_AGENT_RUN_WORKER_POLL_MS=3000
MEMIND_AGENT_RUN_WORKER_BATCH_SIZE=1
MEMIND_AGENT_RUN_QUEUE_CONCURRENCY=1
MEMIND_TOOL_GATEWAY_ENABLED=1
MEMIND_TOOL_GATEWAY_DRY_RUN=0
MEMIND_AGENT_RUN_WORKDIR_OVERRIDE=/Users/john/Project/memind_canary/<rollout-id>
MEMIND_AGENT_RUN_WORKDIR_USER_ID=a6fb1e97-2b0f-447b-b138-4561d8e5c53e
```

All canary code writes must use `/Users/john/Project/memind_canary/...`.

## All-user Gray Profile

Current production rollout can use all H5 users by leaving the user and task-type allowlists empty. H5 default code-run now auto-attaches receipt validation, so required validation stays enabled.

Portal `.env` additions:

```bash
MEMIND_AGENT_RUN_AUTODISPATCH=0
MEMIND_AGENT_CODE_RUNS_ENABLED=1
MEMIND_AGENT_CODE_RUNS_USER_IDS=
MEMIND_AGENT_CODE_RUN_TASK_TYPES=
MEMIND_AGENT_CODE_RUNS_REQUIRE_VALIDATION=1
MEMIND_AGENT_RUN_QUEUE_CONCURRENCY=2
MEMIND_AGENT_RUN_WORKER_BATCH_SIZE=2
MEMIND_AGENT_RUN_HEARTBEAT_MS=30000
MEMIND_AGENT_RUN_WORKER_EXPECT_RUNNING=1
MEMIND_TOOL_GATEWAY_ENABLED=0
MEMIND_TOOL_GATEWAY_DRY_RUN=0
VITE_AGENT_CODE_RUNS_ENABLED=1
VITE_AGENT_CODE_RUNS_AUTODETECT=1
VITE_AGENT_CODE_RUNS_USER_IDS=
```

Worker installer environment:

```bash
MEMIND_AGENT_RUN_WORKER_START=1
MEMIND_AGENT_RUN_WORKER_POLL_MS=3000
MEMIND_AGENT_RUN_WORKER_BATCH_SIZE=2
MEMIND_AGENT_RUN_QUEUE_CONCURRENCY=2
MEMIND_AGENT_RUN_HEARTBEAT_MS=30000
MEMIND_TOOL_GATEWAY_ENABLED=1
MEMIND_TOOL_GATEWAY_DRY_RUN=0
```

Expected values:

```text
toolRuntime.codeRunPolicy.enabled=true
toolRuntime.codeRunPolicy.userAllowlist=[]
toolRuntime.codeRunPolicy.taskTypeAllowlist=[]
toolRuntime.codeRunPolicy.requireValidation=true
toolRuntime.codeRunsEnabled=true
toolRuntime.queue.autoDispatch=false
toolRuntime.queue.maxConcurrentRuns=2
toolRuntime.queue.toolGateway.enabled=false
check-agent-run-worker.expected=running
check-agent-run-worker.running=true
check-agent-run-worker.processes includes --limit 2
```

## Enable Procedure

1. Backup `.env`.

```bash
backup_dir="/Users/john/Project/memind_backups/$(date +%Y%m%d-%H%M)-agent-run-worker-rollout"
mkdir -p "$backup_dir"
cp /Users/john/Project/Memind/.env "$backup_dir/.env.before"
```

2. Append the Portal `.env` additions from the canary profile.

3. Restart Portal.

```bash
launchctl kickstart -k "gui/$(id -u)/cn.tkmind.memind-portal"
```

4. Verify Portal policy.

```bash
curl -fsS https://mm.tkmind.cn/api/runtime/status | node -e '
let s="";process.stdin.on("data",d=>s+=d);process.stdin.on("end",()=>{
  const j=JSON.parse(s);
  console.log(JSON.stringify({
    ok:j.ok,
    codeRunPolicy:j.toolRuntime?.codeRunPolicy,
    autoDispatch:j.toolRuntime?.queue?.autoDispatch,
    toolGateway:j.toolRuntime?.queue?.toolGateway?.enabled
  }, null, 2));
})'
```

5. Create canary repo.

```bash
canary_dir="/Users/john/Project/memind_canary/agent-run-worker-rollout-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$canary_dir"
git -C "$canary_dir" init
printf '# rollout canary\n' > "$canary_dir/README.md"
git -C "$canary_dir" add README.md
git -C "$canary_dir" commit -m 'init rollout canary'
```

6. Install and start worker with the worker installer environment.

```bash
MEMIND_AGENT_RUN_WORKER_START=1 \
MEMIND_AGENT_RUN_WORKDIR_OVERRIDE="$canary_dir" \
MEMIND_AGENT_RUN_WORKDIR_USER_ID="a6fb1e97-2b0f-447b-b138-4561d8e5c53e" \
MEMIND_TOOL_GATEWAY_ENABLED=1 \
MEMIND_TOOL_GATEWAY_DRY_RUN=0 \
MEMIND_AGENT_RUN_QUEUE_CONCURRENCY=1 \
bash /Users/john/Project/Memind/scripts/install-agent-run-worker-agent.sh
```

7. Verify worker running.

```bash
MEMIND_AGENT_RUN_WORKER_EXPECT_RUNNING=1 node /Users/john/Project/Memind/scripts/check-agent-run-worker.mjs
```

The worker LaunchAgent must have `KeepAlive=true` during all-user gray so a clean worker exit does not leave code-run queue processing stopped.

Check stale running recovery before and after worker changes:

```bash
node /Users/john/Project/Memind/scripts/agent-run-worker.mjs --recover-stale --stale-ms 900000 --limit 5
```

Check worker heartbeat status during all-user gray:

```bash
MEMIND_AGENT_RUN_WORKER_EXPECT_RUNNING=1 node /Users/john/Project/Memind/scripts/check-agent-run-worker.mjs
```

Expected queue fields:

```text
heartbeatMs=30000
oldestRunningHeartbeatAgeMs=0 when no running rows
runningWithoutHeartbeatCount=0 during normal operation
```

Runtime SLO report should also expose the same heartbeat summary:

```bash
node /Users/john/Project/Memind/scripts/runtime-slo-report.mjs
```

Expected SLO fields:

```text
runtime.toolQueueSlo.heartbeatMs=30000
runtime.toolQueueSlo.oldestRunningHeartbeatAgeMs=0 when no running rows
runtime.toolQueueSlo.runningWithoutHeartbeatCount=0 during normal operation
failures=[] unless heartbeat age exceeds MEMIND_AGENT_RUN_HEARTBEAT_STALE_MS
```

For the 24h all-user gray soak, keep the daily SLO report unchanged and install the separate hourly snapshot agent:

```bash
MEMIND_RUNTIME_SLO_SOAK_INTERVAL_SECONDS=3600 \
MEMIND_RUNTIME_SLO_SOAK_RETENTION_DAYS=7 \
bash /Users/john/Project/Memind/scripts/install-runtime-slo-soak-agent.sh
```

Expected soak state:

```text
LaunchAgent label=cn.tkmind.memind-runtime-slo-soak
StartInterval=3600
reportDir=/Users/john/Project/Memind/reports/runtime-slo-soak
latest JSON/Markdown snapshot includes runtime.toolQueueSlo.heartbeatMs
```

Apply only when the listed running rows are known stale:

```bash
node /Users/john/Project/Memind/scripts/agent-run-worker.mjs --recover-stale --apply-recovery --stale-ms 900000 --limit 5
```

8. Submit only code runs with validation metadata.

Required user message metadata:

```json
{
  "metadata": {
    "memindRun": {
      "validation": {
        "expectedFile": {
          "path": "RESULT.md",
          "contains": "EXPECTED_TOKEN"
        }
      }
    }
  }
}
```

## Pause Procedure

Pause worker only:

```bash
launchctl bootout "gui/$(id -u)/cn.tkmind.memind-agent-run-worker" 2>/dev/null || true
launchctl disable "gui/$(id -u)/cn.tkmind.memind-agent-run-worker" 2>/dev/null || true
node /Users/john/Project/Memind/scripts/check-agent-run-worker.mjs
```

Portal remains in canary mode until `.env` is restored.

## Disable Procedure

1. Stop and disable worker.

```bash
launchctl bootout "gui/$(id -u)/cn.tkmind.memind-agent-run-worker" 2>/dev/null || true
launchctl disable "gui/$(id -u)/cn.tkmind.memind-agent-run-worker" 2>/dev/null || true
```

2. Restore `.env`.

```bash
cp "$backup_dir/.env.before" /Users/john/Project/Memind/.env
launchctl kickstart -k "gui/$(id -u)/cn.tkmind.memind-portal"
```

3. Reinstall disabled worker plist.

```bash
MEMIND_AGENT_RUN_WORKER_START=0 bash /Users/john/Project/Memind/scripts/install-agent-run-worker-agent.sh
```

4. Verify closed state.

```bash
curl -fsS https://mm.tkmind.cn/api/runtime/status
node /Users/john/Project/Memind/scripts/check-agent-run-worker.mjs
node /Users/john/Project/Memind/scripts/runtime-slo-report.mjs
```

## Incident Checks

If code tasks appear stuck:

```bash
node /Users/john/Project/Memind/scripts/check-agent-run-worker.mjs
tail -200 /Users/john/Library/Logs/memind-agent-run-worker.log
curl -fsS https://mm.tkmind.cn/api/runtime/status
```

If `oldestPendingAgeMs` grows while worker is running:

- Check executor keys and model mapping.
- Check `MEMIND_TOOL_GATEWAY_ENABLED=1` in worker plist.
- Check `MEMIND_AGENT_RUN_WORKDIR_OVERRIDE` points to canary or intended workspace.
- Do not delete queued/failed rows; inspect events first.

## Auto-pause Guard

Install the guard after enabling the external worker:

```bash
bash /Users/john/Project/Memind/scripts/install-agent-run-guard-agent.sh
```

Default guard thresholds:

```text
recent failures >= 3 in 10 minutes
oldest queued/retryable age >= 5 minutes
queued + retryable count >= 10
oldest running age >= 15 minutes
```

When a threshold trips, `agent-run-guard.mjs --apply`:

```text
backs up .env
sets MEMIND_AGENT_CODE_RUNS_ENABLED=0
sets MEMIND_AGENT_RUN_AUTODISPATCH=1
stops and disables cn.tkmind.memind-agent-run-worker
kickstarts cn.tkmind.memind-portal
```

Read-only check:

```bash
node /Users/john/Project/Memind/scripts/agent-run-guard.mjs --dry-run
```

If validation fails:

- Treat it as a product correctness failure, not transport failure.
- Validation failed is non-retryable by design.
- Inspect `tool_gateway_result.stdoutTail`, `stderrTail`, and `tool_gateway_validation_failed`.

## Restore Proof

Final closed proof should include:

```text
codeRunPolicy.enabled=false
codeRunsEnabled=false
queue.autoDispatch=true
queue.toolGateway.enabled=false
check-agent-run-worker.ok=true
check-agent-run-worker.running=false
runtime-slo-report.failures=[]
```