The evaluation category is control-boundary integrity, not request-routing feature count. Gate must prove that AI activity was authorized, policy-evaluated, budget-valid, and auditable before execution.
Unknown control state is denial, not permission. This reduces unauthorized execution risk and increases availability impact during degraded conditions.
Control plane integrity
| Model | Control plane role | Failure behavior | Control risk |
|---|---|---|---|
| App-local controls | Each application embeds policy and logging. | Behavior varies by repo, team, and runtime. | Inconsistent enforcement and difficult incident reconstruction. |
| API gateway control plane | Configures routing, auth, quotas, and traffic policy. | Data plane may continue with cached config. | Stale policy can permit execution that current policy would deny. |
| AI request console | Manages provider routing, keys, logs, semantic checks, and budgets. | Behavior depends on product design and must be validated. | Visibility may continue while enforcement state is incomplete. |
| Syndicate Gate control plane | Supplies versioned policy, budget state, provider authorization, and audit requirements. | Requests fail closed unless covered by signed offline policy. | Lower bypass tolerance with higher availability impact during degraded control-state conditions. |
Auditability depth model
Syndicate Gate should target Levels 3 to 5 by default and Level 6 where independent verification is required. A deployment reaches Level 6 only when key management, storage durability, retention, access controls, and verification procedures support independent review.
| Level | Name | Description | Incident response value |
|---|---|---|---|
| 0 | No reliable audit | Events may exist in application logs but are incomplete or inconsistent. | Cannot reconstruct execution with confidence. |
| 1 | Basic logging | Request and response metadata are recorded. | Useful for debugging, weak for enforcement proof. |
| 2 | Structured audit | Events include actor, application, provider, model, policy outcome, and reason codes. | Supports incident timeline and accountability. |
| 3 | Decision-linked audit | Events link to policy version, budget decision, data classification, and enforcement outcome. | Shows why execution was allowed or denied. |
| 4 | Tamper-evident audit | Events are hashed, signed, sequenced, or protected against undetected modification. | Supports evidence integrity review. |
| 5 | Replayable audit | Inputs, policy versions, model identifiers, context references, and outputs are preserved or content-addressed. | Enables controlled reconstruction of the decision path. |
| 6 | Independently verifiable audit | Evidence can be verified outside the runtime with retained keys, hashes, manifests, and storage proofs. | Strongest posture for audits, investigations, and regulator review. |
Cost governance enforcement
Gate evaluates actor, tenant, team, application, workflow, provider, model, estimated tokens, maximum request spend, remaining allowance, and fallback cost envelope before model invocation. If budget state is unavailable and the request requires budget validation, the request is denied.
| Capability | Reporting system | Enforcing system |
|---|---|---|
| Token count | Reports usage after provider response. | Estimates before execution and reconciles after response. |
| Budget report | Shows spend by team, model, or provider. | Blocks, downgrades, or routes requests before budget breach. |
| Alerting | Notifies after threshold crossing. | Prevents execution when threshold would be exceeded. |
| Provider fallback | Routes around outage or latency. | Routes only when fallback satisfies data, jurisdiction, policy, and budget constraints. |
Multi-provider risk handling
| Decision factor | Required evaluation |
|---|---|
| Data classification | Is this provider allowed to receive this data class? |
| Jurisdiction | Does the route satisfy residency and transfer requirements? |
| Retention | Does provider retention behavior match policy? |
| Model approval | Is the model approved for this workflow and actor? |
| Safety behavior | Are provider-side safety controls known and acceptable for the use case? |
| Tool semantics | Are tool-call semantics compatible with Claw and Code controls? |
| Cost envelope | Is estimated spend within the allowed budget? |
| Evidence sufficiency | Can request and response metadata be captured sufficiently for review? |
A timeout from Provider A must not silently route sensitive data to Provider B unless Provider B satisfies the same data, jurisdiction, retention, safety, cost, and evidence obligations.
Security model
| Layer | Examples | Controls | Does not fully control |
|---|---|---|---|
| Network security | mTLS, IP allowlists, private connectivity, firewalls. | Who can reach the gateway or provider path. | Whether a prompt is authorized or a tool call is safe. |
| Identity and access | SSO, service accounts, RBAC, workload identity. | Who or what is acting. | Whether this request is allowed in this workflow state. |
| API policy | Rate limits, quotas, auth headers, route policy. | Traffic shape and API access. | Semantic intent, prompt injection, and model output risk. |
| Semantic policy | Prompt classification, data classification, provider eligibility, tool-call constraints. | Whether content and intent satisfy policy. | Model truthfulness or complete prompt-injection prevention. |
| Execution authorization | Claw workflow approval, Code action control, change gates. | Whether AI output may cause downstream action. | Activity outside the governed boundary. |
| Audit integrity | Signed events, hash chains, retention controls. | Whether evidence can be trusted. | Events never produced because traffic bypassed Gate. |
Prompt injection detection, PII detection, and output validation reduce risk but are not perfect. High-impact actions require deterministic authorization gates outside the model.
Benchmark methodology
Published numbers should separate Gate overhead from provider latency. Results must disclose topology, request shape, policy complexity, audit mode, cryptographic settings, budget mode, and failure conditions.
| Dimension | Required disclosure |
|---|---|
| Deployment topology | Same-region or cross-region, self-hosted or managed, network path, TLS configuration. |
| Provider behavior | Mock provider, real provider, streaming mode, model latency distribution. |
| Request shape | Prompt size, context size, tool schema size, output size, concurrency. |
| Policy complexity | Number of policy checks, local versus remote dependencies, cache hit rate. |
| Audit mode | Synchronous write, buffered write, signed artifact, hash chain, storage backend. |
| Crypto mode | Hashing only, signing, external KMS or HSM, key rotation state. |
| Budget mode | Cached budget, strongly consistent budget, external ledger call. |
| Failure conditions | Normal operation, provider timeout, audit delay, policy dependency degradation. |
| Metrics | p50, p95, p99, error rate, denial rate, queue depth, audit write latency, and provider latency separated from Gate overhead. |
Statement format: Under [topology], with [request shape], [policy mode], [audit mode], and [provider simulation], Syndicate Gate added [p50/p95/p99 overhead] before provider invocation and [p50/p95/p99 overhead] after provider response. These numbers do not include provider model latency unless explicitly stated.
Expanded failure scenarios
| Scenario | Behavior | Blocked | Allowed | Evidence | Cannot guarantee |
|---|---|---|---|---|---|
| Control plane outage | Gate denies requests requiring live policy, budget, provider, or tenant state. | New high-risk execution, provider changes, tool authorization, code actions. | Health checks, diagnostics, explicitly pre-authorized low-risk signed offline policy scope. | Denial reason, dependency status, last known policy version, caller identity, request hash if capture succeeds. | Whether unavailable current policy would allow the request. |
| Audit storage failure | Gate refuses execution requiring durable audit and attempts alternate sinks if configured. | Provider invocation, downstream action, workflow progress. | Retry audit write, failover audit sink, diagnostics. | Audit failure event in available sink; local volatile record if no durable sink is available. | Complete durable record if every audit path fails before persistence. |
| Provider timeout | Provider result is incomplete. Fallback only if approved for the same policy obligations. | Tool execution from missing response, workflow completion, success claims. | Controlled retry, approved fallback, caller-visible failure. | Provider attempt ID, timeout class, retry or fallback decision, budget reconciliation state. | Whether provider eventually processed the request without provider evidence. |
| Malformed provider response | Gate rejects response for downstream action. | Tool-call execution, structured workflow transition, code mutation. | Return validation error or retry if policy permits. | Raw response hash, validation failure code, rejected action metadata. | Provider intent or correctness beyond observable response. |
| Network partition | Gate blocks requests dependent on unreachable control, audit, provider, or budget services. | Execution requiring unreachable dependencies. | Local status, partition diagnostics, signed offline policy scope if configured. | Partition event, dependency graph, denial events for affected requests. | Global event ordering across partitions until reconciliation. |
| Partial policy failure | Any unknown required check results in denial. | Request execution, provider routing, tool authorization. | Non-executing diagnostics. | Policy trace showing failed evaluator and unknown result. | Whether the missing evaluator would have allowed the request. |
| Budget ledger unavailable | Budget-controlled requests are denied unless explicitly covered by reserved quota. | Spend-incurring model calls outside reserved quota. | Requests covered by pre-reserved budget and diagnostics. | Budget dependency failure and reserved quota decision if used. | Current global budget if the ledger is unreachable. |
| Claw unavailable | Workflow transitions are blocked. Gate may allow non-executing output only if policy permits. | Workflow advancement, approval bypass, action-bearing tool use. | Read-only model response if classified safe and non-actionable. | Claw dependency failure and blocked transition event. | Whether Claw would have approved the transition. |
| Code unavailable | Code-affecting actions are blocked even if Gate and Claw evaluate the request. | File changes, repository operations, CI actions, deployment actions. | Non-mutating explanation or plan generation if policy permits. | Blocked execution event with requested action and actor identity. | Outcome of code action because it did not execute. |
Regulatory mapping
Use evidence language: supports review for, produces artifacts for, or may support control operation. Do not claim that the product alone ensures compliance with SOC 2, ISO 27001, GDPR, or incident response obligations.
| Framework | Relevant concern | Syndicate evidence that may support review | Not covered by product alone |
|---|---|---|---|
| SOC 2 Security | Access control, change control, operational telemetry, incident response. | Actor identity, policy version, allow or deny decisions, privileged action records, tamper-evident audit, admin changes. | Organizational control design, access reviews, vendor management, and full incident process. |
| SOC 2 Availability | Resilience, failure handling, operational telemetry. | Fail-closed events, dependency health, denied execution during outages, recovery timeline. | SLA commitments, capacity planning, and disaster recovery testing outside Syndicate. |
| ISO 27001 | Risk treatment, access control, logging, operational security. | Policy-as-code records, enforcement evidence chains, evidence retention, role separation, denied actions. | Full ISMS, risk register ownership, and internal audit program. |
| GDPR | Lawful basis, data minimization, processor controls, data subject rights. | Data classification, provider routing restrictions, retention policy references, prompt and response redaction records. | Lawful basis determination, DPA negotiation, transfer assessment, and DSAR operations. |
| Incident response | Reconstruction and containment. | Timeline, actor, action, provider, model evidence, failed policy checks, blocked execution records. | Human decision quality, external provider logs, and systems outside the governed path. |
Attack surface analysis
| Threat | Example | Gate control | Residual risk |
|---|---|---|---|
| Prompt injection | Retrieved content instructs the model to ignore policy or leak data. | Data classification, prompt and tool policy, response validation, Claw and Code authorization before action. | Detection may miss novel injection and the model may still produce misleading text. |
| Replay attack | A previously valid request is resent under changed conditions. | Nonce and idempotency controls, policy version check, timestamp window, budget re-evaluation. | Distributed replay detection depends on shared state availability. |
| Policy bypass | An application calls the provider directly. | Credential isolation, network egress controls, provider key custody, audit reconciliation. | Gate cannot govern paths that do not pass through it. |
| Tool-call forgery | The model emits an unauthorized tool invocation. | Tool schema validation, Claw authorization, Code execution control. | Unsafe custom tools outside Syndicate may still execute if independently exposed. |
| Provider fallback abuse | A request routes to a less restrictive provider after timeout. | Provider eligibility policy, data residency checks, fallback allowlist. | Misclassified data may route incorrectly. |
| Audit tampering | An operator modifies logs after an incident. | Signed or hash-linked artifacts, restricted write paths, external verification. | Key compromise or misconfigured storage can weaken integrity. |
| Budget exhaustion | An actor burns shared model budget. | Per-actor, team, and workflow budgets with pre-execution estimates. | Token estimates can differ from provider billing. |
| Malformed response exploitation | Provider returns invalid structured output that tricks a downstream parser. | Schema validation, strict parsing, deny on malformed output. | Application code outside Syndicate may parse differently. |
| Agent privilege escalation | An agent acts outside delegated scope. | Actor-agent binding, scoped credentials, workflow state checks. | Incorrect identity mapping can over-authorize. |
| Data exfiltration through output | Model includes sensitive source data in a response. | Output classification, redaction, response policy. | Classifiers are imperfect; encoded leakage may evade detection. |
Data governance and provenance
Caller and agent identity, app, tenant, workflow, prompt, tool schemas, data labels, policy decision, budget decision, provider, model, response artifact or hash, Claw and Code authorization, signatures, hash references, and retention metadata.
Which actors may call which models, which data classes may reach which providers, which workflows may use AI output, which tool calls require approval, and which requests are blocked when evidence cannot be produced.
Gate cannot prove facts not captured at request time, provider internals, identical future model output, or external tool behavior outside Claw and Code.
Replay depends on preserved inputs, policy versions, provider and model identifiers, context references, output artifacts, and decision traces. External state and stochastic models may not reproduce exactly.
Syndicate ecosystem
Caller -> Gate authentication -> policy, budget, provider, audit decision -> approved provider -> response validation -> Claw workflow authorization -> Code execution control -> final evidence
Competitor comparisons
Portkey is a logging and routing platform. Gate is an enforcement boundary. Compare fail-closed behavior, pre-execution policy evaluation, and tamper-evident audit evidence.
LangSmith is a tracing and evaluation platform. Gate is an enforcement boundary. Compare what gets recorded after execution vs what gets evaluated before.