Design Records

• 1: Bucket Configuration Replication: Source Times, Deletions, and Deterministic Convergence
• 2: An Unsigned Header Is Not Part of the Request
• 3: Replication Reliability: Delete Completion, MRF Visibility, and Resync Cancellation
• 4: Replicated Tag Ordering: Revision Timestamps, Tombstones, and Resurrection
• 5: SILO Server 20260903 Pre-release Review
• 6: Replica Metadata Normalization: What a Trusted Copy May Not Re-Inject
• 7: Request-Header Deadlines: Absolute Header Limits and Rolling Body Idle Timeouts
• 8: Conditional DELETE: Why the Condition Must Be Evaluated Once
• 9: DSN-Only Database Notifications: A Compatibility Boundary for #53
• 10: Preview Text, Never Execute It: SILO Console Text Preview PRD
• 11: Go 1.27 TLS Defaults and OIDC Discovery Failure Modes
• 12: No I/O Before Auth, No Privilege From Headers
• 13: One Endpoint, Two Privileges: Separating User and Group Status
• 14: Config Environment Files Are Not Shell Scripts
• 15: Two SSE-C Keys, One CopyObject Response
• 16: Why CompleteMultipartUpload Must Return ChecksumType: Review of PR #57
• 17: When the Total Is Unknown: Folder Download Progress
• 18: A ListObjects Shortcut Must Not Turn a Missing Bucket into an Empty One
• 19: Read-Only Checksum Audit and Reliable CLI Output
• 20: Optional Checksums, Mandatory Failure: Repairing UploadPart and UploadPartCopy Compatibility
• 21: BadDigest, InvalidRequest, and the CompleteMultipartUpload Checksum Contract
• 22: Should SILO Fix ListMultipartUploads? Design Review of Issue #79

1 - Bucket Configuration Replication: Source Times, Deletions, and Deterministic Convergence

#77 is a reproduced site-replication correctness defect. A receiver replaces source time with arrival time and may then reject a genuinely newer deletion. Some configuration types stop exporting their timestamp after deletion, preventing heal from recovering a delete missed during an outage. Adding a DELETE branch alone cannot solve both problems.

Merge status (2026-09-12): the repair and research archive were merged into main through PR #180, commit 48ec10312. #77 is closed. All nine pre-merge CI checks passed.
Review boundary: the plan went through four Claude Code Opus 5 Max review rounds, passing the final two. Full implementation review, remediation review, and focused final acceptance returned GO_WITH_NONBLOCKING_NOTES in all three rounds. Final blockers are zero; the requested full cmd and final lint checks have now passed.
Applicability: this page describes the repair now included in main. Check the specific version of downloads and running installations; full deletion recovery still requires every node to be upgraded and deletion export to be enabled consistently.

Existing work and scope

The earlier release notes, security hardening record, and Server compatibility page already document the #77 deletion limitation. They do not provide a complete record of its state model, alternatives, or verification boundaries. This page supplies that reasoning.

The source repository archive retains the original reproduction, plan versions, final reports and invocation identities for all seven review rounds, finding dispositions, executed test logs, source/binary hashes, and a rerunnable two-site driver. Raw model reasoning streams, binaries, and temporary lab volumes are excluded. Original artifacts and copies with normalized workstation paths and document links have separate hashes.

This repair covers Policy, Tags, SSE, Quota, Versioning, and Object Lock. Lifecycle/expiry has its own merged-payload time semantics; CORS keeps its separate mechanism. Notification, IAM, object replication, MRF, resync, and public counters are not rewritten. Object replication reliability has a separate record.

The supported release target is the maintained silo, silo-console, mc, and silo-pkg stack. Compatibility with unmodified upstream MinIO/MC remains best effort and does not require downgrading maintained components or recreating dependency forks.

What the reproduction established

The original regression reproduced on both ErasureSD and 16-disk Erasure ObjectLayers. A peer PUT originated at a past time T, but the receiving disk stored current arrival time. A subsequent DELETE carrying T+1 minute was rejected as older. RPC success alone therefore cannot establish correct final state.

Three further entry-point defects matter: an old bulk event can overwrite a newer field; a request that checks state before queuing for the lock can act on an obsolete decision; and remote Tag heal omits UpdatedAt. Each requires a repair at the actual entry point, beyond changing source selection in heal.

The facts a field needs

Reuse the existing payload, field UpdatedAt, and bucket Created. No disk format, SDK field, or persisted deployment ID is added.

Empty Versioning and Object Lock events remain no-ops. Sharing a helper must not give these types deletion semantics. Zero field times use Created as a comparison baseline; this does not turn historical emptiness into a new delete.

Ordering first gives real states priority over baselines, then compares source time between real states. At the same time, a deletion wins over a live value. Conflicting live values at the same rank use a stable content key, with the lexicographically greater key winning. An identical effective state causes neither a save nor a notification. Live baselines also use content-key ordering; a later creation default cannot outrank a real change.

This is deterministic conflict resolution. It does not mean a lexicographically greater configuration better expresses business intent. Operators must still choose and resubmit the intended value after conflicting concurrent changes.

Comparison must match persistence

Policy sets are map-backed, so ordinary JSON encoding can depend on iteration order. The Server sorts the complete JSON tree of an already validated policy, including Statement, Action/NotAction, Resource/NotResource, Principal, and Condition. Sid and numeric precision are retained. This adds no syntax unsupported by the existing parser.

That parser accepts NotAction and NotResource, but the original structure’s required Action/Resource encoding can fail on the corresponding empty sets. Explicit field encoding addresses this, and Policy GET/admin export use it too. The purpose is to keep an accepted policy readable, beyond making its comparison key deterministic.

Quota keys use the existing parsed representation encoded as JSON. {}, JSON null, and valid zero-quota documents remain live documents, not implicit deletion events. An empty Policy instead follows the established peer deletion interpretation: a valid empty-policy PUT succeeds, and a subsequent GET returns the existing NotFound response.

XML configurations use the bytes of a valid document. There is no general XML canonicalization layer. Versioning needs one exception: apply the existing Object Lock constraint before comparing the effective document that will actually be persisted. Otherwise comparison can accept a value that Save rewrites, causing heal to send it again next time.

Lock the decision as well as the save

The six fields share one .metadata.bin record. Loading raw state, validation, comparison, modification, and saving must all happen under the existing metadata.lock. Comparing outside the lock still permits stale decisions; separate field locks would allow whole-record read/modify/write operations to overwrite one another.

A local write allocates its time inside the lock:

max(current UTC time, Created + 1ns, current field time + 1ns)

This lets a local correction advance beyond an already stored future field time. Timestamped peer events retain their original time. Identical or older states return without a write.

Bulk processing handles only explicitly supplied fields, validates them, and saves at most once. Omission preserves a field; explicit null follows its type’s semantics. An invalid field cannot leave half the bulk update persisted. Import assigns a common time for the selected six-type fields under the final per-bucket commit lock. Disk state and outbound events use that commit’s final snapshot, not a later read combined with an earlier time. A normalized empty Policy uses the existing dedicated deletion event so bulk omitempty cannot lose it.

The save helper returns its normalized snapshot. Public Update/Delete signatures stay unchanged, while other metadata retains its existing processing and notification behavior.

Adoption must not manufacture a deletion

Adoption can change Created. Moving it earlier while retaining an empty field’s old default timestamp makes an empty baseline appear to be a real deletion. Moving it later can invalidate historical initial values.

The narrow repair, under the existing adoption lock, rebases only these six fields whose previous time was zero or equal to old Created. Actual update and deletion times remain unchanged. An empty payload alone is not proof of a default, and existing-bucket configuration protection is not redesigned. Genuinely different bucket generations still require operator intervention.

One rule across the real entry points

Initial synchronization retains its existing five-type send path. Versioning is still initialized through MakeBucketHook and aligned through heal. No extra initialization path is added merely to make the table symmetric.

Heal filters valid candidates before selecting the maximum, instead of seeding from the first map entry and only then filtering defaults. Public mismatch counters cannot be the sole gate: equal content with different source times still needs synchronization. Conversely, equal effective states need no further write or RPC.

Why rollout needs a default-off switch

The startup setting MINIO_SITE_REPLICATION_METADATA_TOMBSTONES defaults to off. It controls visibility of newly exposed deletion information and does not detect remote capability.

Old implementations cannot safely consume all newly exposed deletion information. For example, old Quota heal can clear the payload while retaining an already parsed cache value. An instruction to upgrade does not itself isolate this rolling-upgrade window, so the default stays off.

Upgrade every node at every participating site to a build containing the repair, ensure consistent settings within each site, and drain old requests. Then set on consistently and restart. Before a downgrade, first set off and restart every fixed node, then roll back the software. The old software’s original defects return with it.

While off, hidden Tags/SSE/Quota tombstones can cause repeated stale heal RPCs that a fixed receiver rejects. A subsequent heal with zero RPCs is expected only when complete state is visible and stable.

Retained and rejected alternatives

The initial implementation adds 729 and removes 692 production Go lines, a net increase of 37, chiefly replacing duplicated apply and heal branches. Line count does not establish minimality. Necessity must connect each mechanism to a concrete failure; sufficiency must cover every real entry point; minimality asks which failure returns if a mechanism is removed.

Validation and its limits

The environment is local go1.27.1 darwin/arm64. Four groups of pre-implementation audit tests failed on the unfixed baseline. The resulting regression suite passes on both ObjectLayers; its main entry points are in cmd/site-replication-metadata{,-heal,-gate}_test.go. The baseline audit log retains the original failures alongside the passing existing tests.

Local evidence contains baseline failures, test logs, a rerunnable two-site driver, metadata snapshots, and source/binary SHA-256 identities. The first two-site binary reports c8f264f79 + dirty; its actual Go build-info and SHA-256 have now been recorded. After review repairs and cleanup, runs were repeated on binaries built from clean 62cf066ff and fcbb93e89. Actual --version, Go build metadata, and SHA-256 identities are retained with the baseline binary built from 5c5765816. Final 461e9a721 differs from fcbb93e89 only in test formatting and equivalent conditional syntax; that diff is recorded separately.

The table above records local tests and isolated-process observations. Remote integration evidence is separate: all nine checks on PR #180 passed, comprising six Go CI jobs, DCO, vulnerability analysis, and release-pipeline validation. The merged main tree was verified to match the PR merge tree that passed these checks. This is not proof of real Linux multi-node cluster behavior, official release artifacts, or production deployment.

Adversarial review record

The plan used actual Claude Code claude-opus-5 --effort max for four rounds. The first two drove corrections to state comparison, committed snapshots, historical baselines, and import boundaries. The last two returned GO_WITH_NONBLOCKING_NOTES with zero pre-implementation blockers. Plan approval is not proof that implementation is correct.

The separate implementation review pinned 4089113e3 and used the same model and effort to inspect the complete diff, production call chains, formal tests, and runtime evidence independently, challenging sufficiency, minimality, rollout safety, and evidence identity. Its verdict was GO_WITH_NONBLOCKING_NOTES: no unconditional blocker, one conditional blocker, and nine further findings. It confirmed the core convergence mechanism and found no counterexample to ordering, deletion, or duplicate suppression within the declared contract. Three findings were real defects the change had introduced and were repaired in 62cf066ff. The subsequent fcbb93e89 closes the diagnostic gap when no valid source exists and adds an initial-sync regression.

A stub is worth calling out separately. The original recovery test injected an object layer whose creation probe returned the expected time, so it passed against code that could never behave that way in production. The replacement stamps the bucket directory on every local drive and drives the real object layer, and it fails on the unrepaired code.

The second review pinned 62cf066ff, again using actual claude-opus-5 --effort max. It returned GO_WITH_NONBLOCKING_NOTES with zero conditional or unconditional blockers. It retraced production paths, checked the author’s F1/F2 reproduction results from formal tests with old production code overlaid, and revised the first review’s assessment of the Policy encoder. The reviewer did not execute the tests.

Diagnostic regressions now use the existing logger target to check warning level, distinct reasons, repeated calls, and zero RPCs. The initial-sync test drives a real source ObjectLayer and the complete outbound sequence; its peer only acknowledges RPCs. That proves outbound content. The real two-process experiment supplies a separate level of evidence, and the two must not be conflated.

The third focused acceptance pinned final production code fcbb93e89 and again returned GO_WITH_NONBLOCKING_NOTES, with zero blockers. It also inspected the test-only style diff in 461e9a721 and confirmed equivalent semantics. Full cmd and lint were still running when the reviewer read their logs; both subsequently exited with code 0. Test formatting caused lint to fail at fcbb93e89 itself; corrected 461e9a721 is the delivery baseline that satisfies the required checks.

Three nonblocking improvements remain outside this repair: heal diagnostics can show a zero source time after decode/parse failure; the initial-sync unit test does not execute the local peer branch and therefore does not prove that recovered Created is persisted locally (that production path was reviewed); and strict system-log capture could need isolation if concurrent background logging is introduced. No additional production accessor or test hook was added for these points. A counterfactual failure proves the assertion actually reached, such as empty-baseline noise. Warning-level and per-reason deduplication assertions pass after repair; that is not a claim that each was separately demonstrated failing on old code.

Operational boundaries that remain

1. Historical timestamp pollution is not reconstructable. Arrival-time replacement and old untimestamped events have lost source facts. Installing the repair cannot recover their true historical order. Inspect every site and resubmit the intended configuration or deletion at the authoritative site.
2. Zero-time typed events remain compatible. They use monotonic local time and emit legacy-zero; the existing zero-time constraint for bulk remains. These events are outside the timestamped-source convergence guarantee.
3. Bucket identity conflicts are not merged automatically. Resolve generation differences first. Events before target Created are not applied; unknown creation time is recovered only from a real physical bucket. Unknown or missing buckets are not written. The recovered value is a physical approximation from bucket-directory modification time. It can change with top-level entries, differ across drives, and be later than actual creation. Older source events are still skipped; one event is insufficient to lower the bucket identity. A successful configuration write or initial site sync records the recovered value. Until then, status reports the unknown time and periodic healing skips the bucket in both directions.
4. A physical clock is not a causal clock. Local correction can advance beyond an already known future field time, but cannot infer the business intent of all concurrent writes.
5. Diagnostics are bounded; success does not mean applied. legacy-zero, before-created, indeterminate, unreachable, and peer-error reuse LogOnceIf with stable keys and error text, details in attributes, and existing hourly cleanup. Each reason keeps its own key, so one condition cannot deduplicate another away. Empty baselines and peers that do not yet have the bucket stay quiet. Invalid existing states emit indeterminate even when no valid source can be selected, without causing an RPC. Normal duplicates and older events remain quiet. This is a per-bucket/field/reason bound: both unreachable-peer warnings and indeterminate-state warnings can grow with bucket and populated-field count, not a fixed site-wide limit.
6. Code, integration, and release require separate acceptance. Issue state, Server version, images, packages, published documentation, and production settings each need their own evidence. The main-branch merge recorded here does not establish that release artifacts or production installations have been upgraded.

2 - An Unsigned Header Is Not Part of the Request

This record describes the unsigned-header coverage repair committed to SILO as 123325430 and merged through PR #173, tracked as SN-2026-011. It was reported by Oren Yomtov against a released build and reproduced locally on both signature paths.

Status on 2026-09-11: the original repair is pushed and merged through PR #173. The follow-up signing and payload-verification fixes described below are also merged through PR #177, with all eight PR checks passing. Source validation and published releases are separate: the currently published September 3 Server release does not contain these fixes.
Scope: SigV4 header coverage, consistent policy inputs and body-checksum verification. S3 field names, object and bucket metadata formats, replication protocols, encryption formats and client commands are unchanged.
Security property: unsigned client-supplied x-amz-* operation headers cannot change an authorized request; policy evaluation and body verification use the effective signed inputs.

Too Long; Didn’t Read (TL;DR)

A presigned PUT URL signs exactly one header, host. SILO confirmed that each header named in the signed-headers list had arrived, but it never walked the headers that actually arrived, so an x-amz-* header outside that list was accepted and used. cmd/api-router.go routes any PUT carrying x-amz-copy-source to CopyObjectHandler on that header alone. Together these turned a write grant for one object into a server-side copy that reads any object the signing key can reach, executed as the signer — a confused deputy. The Authorization-header path behaved the same way when the header was left out of SignedHeaders.

The repair states one invariant:

As far as x-amz-* semantics go, the signed-headers list IS the request.
Any x-amz-* header not covered by it is refused before the handler runs.

This matches AWS S3, which refuses the same request with AccessDenied (“There were headers present in the request which were not signed”). The signing code was inherited byte-for-byte from upstream minio/minio, so every earlier SILO release, and upstream itself, carry the gap.

Failure: the coverage gap

extractSignedHeaders in cmd/signature-v4-utils.go iterates the signed-headers list and, for each name, pulls the value from the request (or the query string). It proves that every promised header is present. It never asks the opposite question — is every x-amz-* header that arrived actually in the list?

The one place that walked the arriving headers, checkMetaHeaders, matched only the X-Amz-Meta- prefix and was called only from the presigned path (doesPresignedSignatureMatch). The Authorization-header verifier (doesSignatureMatch) called nothing equivalent. So an unsigned x-amz-copy-source — or any other operation-shaping x-amz-* header — sailed through on both paths:

presigned PUT (SignedHeaders=host)  ->  add unsigned  x-amz-copy-source: /src/secret
  -> router sees x-amz-copy-source  -> CopyObjectHandler
  -> copy runs as the signer, reading a bucket the URL never named

Reproduced locally on RELEASE-style builds: the control PUT returns 200 with an empty body; the same URL plus the one unsigned header returns 200 with a CopyObjectResult whose ETag is the md5 of the victim object, and the destination reads back the victim’s bytes. Where the destination bucket already allows anonymous GetObject, the copied private bytes are then readable with no credentials at all.

Provenance

The gap is inherited from upstream MinIO; SILO did not introduce it. The SigV4 verifier in cmd/signature-v4-utils.go and the Authorization-header path doesSignatureMatch in cmd/signature-v4.go are original MinIO code dating to 2016, and the header-driven CopyObject dispatch in cmd/api-router.go traces to 2019. The only routine that ever walked the arriving headers, checkMetaHeaders, was added upstream on 2023-07-27 in minio/minio#17737 (535f97ba6). Upstream therefore recognized the class — an unsigned header must match the signed set — but scoped the check to the X-Amz-Meta- prefix and to the presigned path, leaving x-amz-copy-source and the whole Authorization-header path uncovered. The window has been open for the life of MinIO’s S3 layer.

Before the unsigned-header repair, SILO’s change to cmd/signature-v4-utils.go was the one-line dependency-path migration in 9b11dc946, moving the policy import to pgsty/silo-pkg/v3. The vulnerable verification behavior came from upstream. The original repair (123325430) and the follow-ups in PR #177 change that boundary. The vulnerable code predates SILO’s fork baseline, the upstream 2025-12-03 maintenance-mode commit from which the first SILO release was cut.

Upstream minio/minio is archived as of that handoff, so there is no upstream maintainer to take the patch. SILO inherited the code unchanged and is the only place it is fixed, as the security ledger already records for other inherited findings.

The repair

checkMetaHeaders becomes checkUnsignedHeaders, is broadened from the X-Amz-Meta- prefix to all of X-Amz-, and is called on both the presigned and Authorization-header paths. A header that is not covered by the signed set is refused with ErrUnsignedHeaders before any handler logic runs.

Four decisions shaped the exact boundary. Each had a plausible alternative that was rejected for a concrete reason.

Membership, not value equality

The inherited check compared signedHeadersMap.Get(k) == val[0]. For a header absent from the signed map, Get returns the empty string, so a header whose first value is empty compared equal and passed. A multi-value header such as X-Amz-Copy-Source: ["", "/src/secret"] could therefore smuggle an unsigned copy-source past a value-equality check. The repair tests membership in the signed set instead. A signed header’s value is already bound by the signature, so value equality was never the property that mattered; presence in the list is.

Exempt X-Amz-Content-Sha256

X-Amz-Content-Sha256 can be omitted from SignedHeaders because the effective payload hash is bound separately in the canonical request. For presigned requests, the query value takes precedence, with a header fallback when the query value is absent. An explicit UNSIGNED-PAYLOAD remains valid. PR #177 aligns the policy condition with this effective value while preserving header-presence semantics, and checks header-only presigned body hashes in the generic authentication path as well as upload paths. The exception does not permit policy evaluation or body verification to use a different value.

Derive signature age from the signed date

The original repair exempted an internal x-amz-signature-age scratch header written after verification. That was too late for PUT and UploadPart authorization, which runs before signature verification. PR #177 instead derives s3:signatureAge directly from the signed X-Amz-Date, and removes the scratch header, its constant and its exemption. A forged date fails signature verification; an unsigned client header under the old name is rejected. Verification remains idempotent without mutating request headers.

Inject X-Amz-Tagging after authentication

PutObjectTaggingHandler derives an X-Amz-Tagging header from the request body so that policy conditions can read it, and it previously did so before authenticateRequest. With the broadened check, that server-synthesized header — which the client never signs — would be refused as unsigned. The injection now happens after signature verification and before authorization, which still has it for policy conditions. Rejected alternative: blanket-exempt X-Amz-Tagging the way content-sha256 is exempt. That would let a client set object tags through an unsigned header on any signed or presigned write, reopening a smaller version of the same class of bug.

Scope across signature modes

• Authorization-header (signed) and presigned SigV4: both now enforced. These are the reachable paths.
• Streaming SigV4: not reachable for a copy. authenticateRequest returns ErrSignatureVersionNotSupported for streaming auth types, so CopyObjectHandler’s checkRequestAuthType rejects a streaming-signed copy before any copy occurs. The check is not added to the streaming verifier because the dispatch cannot reach it; a regression test pins that rejection.
• SigV2: unaffected. V2 canonicalization folds the x-amz-* headers into the string-to-sign by construction, so an added x-amz-* header changes the computed signature and is rejected as a signature mismatch.

Status code: 400 versus 403

AWS returns 403 Forbidden for an unsigned header; SILO returns 400 AccessDenied (ErrUnsignedHeaders), inherited from upstream. The attack is refused either way, and the error Code string is identical; only the HTTP status differs. Raising it to 403 is a one-line change to cmd/api-errors.go that also shifts the pre-existing meta-header rejection. It is left as a deliberate, reversible election rather than folded silently into a security fix, because it is a behavior change for the existing unsigned-meta-header path and is not required to close the vulnerability.

Tests

Several existing tests built a signed request and then set x-amz-copy-source, x-amz-copy-source-range, or x-amz-metadata-directive after signing — that is, they depended on the very behavior this fix removes. They now re-sign with signRequestV4 after setting those headers, which is what every real S3 client does. signRequestV4 excludes the Authorization header from its own signed set, so re-signing is safe. Current coverage includes checkUnsignedHeaders unit cases for empty first values, the payload-hash exception and rejection of the obsolete unsigned signature-age header and TestPresignedVerifyIdempotent, which verifies the same presigned request twice.

Evidence

• A built server reproduced the confused deputy on both the presigned and Authorization-header paths, then refused both after the fix while the control PUT, a real minio-go CopyObject, PutObject with user metadata and tags, and body-based PutObjectTagging all continued to work.
• go test ./cmd/ passes on the fix tree; gofmt, gofumpt, and vet are clean.
• Adversarial review (round one) independently surfaced three defects in the first draft — non-idempotent verification via the scratch header, the empty-first-value bypass, and over-rejection of an unsigned x-amz-content-sha256 — each of which is addressed above and confirmed by re-running the reviewer’s own adversarial test suite against the final tree.
• Adversarial review (round two, against the committed fix) found no regression and confirmed the re-signed tests keep their original intent: an invalid access key still returns InvalidAccessKeyId, and a wrong SSE-C key still returns 403 after the signature validates. It surfaced three adjacent, pre-existing gaps that also fail on the parent commit and are out of this change’s scope; they are recorded under follow-ups below.

Compatibility and operations

• Ordinary clients: no request change. Every AWS SDK, minio-go, and mc already signs the x-amz-* headers it sends.
• Unsigned x-amz-* headers: now refused with AccessDenied, as on AWS. A client that added such a header without signing it was already outside the SigV4 contract.
• Rolling upgrade: wire and storage formats are unchanged. Upgraded nodes enforce the boundary; nodes still running an older build remain exposed until upgraded, so behavior can differ by node during the rolling window.
• Rollback: data written by the fixed version stays readable by the previous version, but rollback reopens the confused deputy.

Residual risks and follow-ups

• Release delivery: a source fix and a public engineering record do not establish that a published binary or image contains the fix. Verify the selected release and artifact separately.
• CVE: the reporter requested one; the finding carries the stable fork-local SN-2026-011 identifier until a CVE is assigned.
• Status code election: the 400-versus-403 choice above is open.
• Adjacent signing fixes: PR #177 addresses repeated copy-source ambiguity, signature-age authorization ordering, and the effective payload-hash policy value. It also closes the separately reproduced header-only presigned body-checksum gap. The regression set covers signed and presigned requests, policy enforcement before upload verification, and real HTTP bucket-policy tampering. These follow-ups are distinct from the original SN-2026-011 finding; their merge and release status is recorded above.
• The general question: this repair covers x-amz-* request headers. Any future control that lets request syntax select an operation must answer the same question this one did — is this value covered by the signature before it is allowed to mean anything? The repeated-header gap above is the same question in a different guise: the value the signature binds and the value the handler consumes must be the one and the same.

Conclusion

The signature is the request. Everything an x-amz-* header claims is a claim until the signature covers it:

Confirming that the promised headers arrived is not the same as confirming that the arrived headers were promised. Refuse any unsigned x-amz-* header before the handler runs, on every signature path a handler can be reached through.

3 - Replication Reliability: Delete Completion, MRF Visibility, and Resync Cancellation

This page records the analysis, design choices, review, and implementation of #153, #152, and #137. They belong to the same replication reliability series, but affect operation classification, recovery visibility, and task lifecycle respectively. One general retry patch cannot repair all three.

As of 2026-09-09: PR #162 is merged as d1105bbb, and all three issues are closed. All eight checks on the tested PR head, followed by main Go CI and VulnCheck, passed.
Second round, 2026-09-16: PR #196 (fix 0c61128d2, verification record aea3882c9) repaired the replication worker’s own delete exits and the persisted MRF marker-recovery path — a different surface from the first round. It is on main and not in Server 20260903; see the second-round section.
Review: the plan was discussed with the installed Claude Code Fable 5.1 Max, followed by a review of the implementation. The final verdict was GO.
Delivery boundary: this work completed code, tests, and main integration. It did not create a Server tag or formal release, and does not establish that existing packages, images, or production deployments contain the fixes.

Overall decision and the surrounding series

The selection rule was to repair a reproduced invariant at the smallest boundary that owns it, retain existing recovery mechanisms, and use deterministic tests to prove that work can finish. Additional complexity needs a concrete counterexample.

The surrounding series had already separated three concepts that are easy to confuse:

• #136 / PR #138 repaired counter completeness: receive and apply the final result before persisting terminal state, without waiting for the one-minute periodic flush.
• #139 repaired outcome accuracy: an existing destination object does not prove that this update succeeded. Success and failure must come from the target’s actual replication result.
• #137 repairs cancellation and resource lifecycle: the task must stop, its walker, workers, and result consumer must exit, and an old run must not pollute a new run’s state. This change preserves the first two contracts rather than introducing another accounting mechanism.

Authorization to use internal replication semantics belongs to the earlier CORS and replication trust record. Authoritative Object Lock state across pools remains tracked separately in #133, still open at this record’s date. Closing these three issues does not mean every replication concern is resolved.

The release-gating target is the maintained pgsty/silo stack with Console, mcli, and silo-pkg. Compatibility with upstream MinIO/MC remains best effort. Neither a mechanism proposed for upstream nor an external experiment can automatically be treated as an observation on current SILO.

#153: distinguish marker replication from permanent version deletion

The report and the reproduction differ

The original issue described a sustained HTTP 405 storm involving ILM and replication, and proposed treating every delete-marker 405 probe as completion. Current SILO source and measurements do not justify adopting that explanation and patch directly.

The baseline was 450dcb848. The experiment used two locally built SILO servers, separate disposable data directories, and maintained mcli / minio-go clients. A critical observation was that mcli uses bulk DELETE even for a single key; that route was already correct. A direct single-object S3 DELETE was therefore necessary to exercise the faulty entry point.

Checking only that the remote version disappeared would falsely declare success. Source metadata must be inspected too. The reproduction establishes incorrect initial purge classification and completion state, plus an unnecessary probe. It did not reproduce the external report’s sustained 405 storm or request-volume figures. Existing scanner/heal and resync producers already select the proper purge path; they cannot be described as necessarily repeating that erroneous probe every cycle.

Why one classification condition is sufficient

When deleting an existing marker, the object layer can return both DeleteMarker=true and a nonempty VersionPurgeStatus. The former describes the version being operated on; the latter describes the operation now required. They are compatible facts.

The old single-delete handler checked only DeleteMarker and populated DeleteMarkerVersionID. Completion then updated ordinary ReplicationStatus instead of completing the purge. The final condition is:

if objInfo.DeleteMarker && objInfo.VersionPurgeStatus.Empty() {
    dmVersionID = objInfo.VersionID
} else {
    versionID = objInfo.VersionID
}

This matches the classification already used by other producers. The fix belongs in the DeleteObject handler. It requires no storage-format change, relaxation of replica deletion protection, or new recovery task. Legacy PENDING entries remain recoverable through the existing scanner/heal purge path.

Why a 405 is not unconditional success

A 405 from a versioned HEAD can establish that the marker exists at the target. For replicating a delete marker, that can mean idempotent completion. For permanently deleting that version, it means there is still work to do.

Writing VersionPurgeComplete merely because HEAD returned 405 could let the source remove its metadata while leaving the unwanted target version behind. The implementation retains existing 405 semantics. Real remote failures such as 403, 405, and 503 must not be disguised as successful permanent deletion.

Source history traces marker-based classification to an upstream 2020-11-19 commit. The 2023-07-10 optimization changed scheduling from dsc.ReplicateAny() to the returned object’s replication/PENDING purge state while retaining classification based only on DeleteMarker. At this location, the 2025-04-02 commit merely moved Pending to replication.VersionPurgePending; it did not first introduce that scheduling condition. This identifies lineage, not a bisect across every historical release, and does not establish when the entire externally reported storm was introduced.

#152: expose drops from an already bounded queue

MRF, or Most Recent Failures, records recent failed replication work for background processing. It already had limits: mrfSaveCh has capacity 100000, and mrfRetryLimit is 3. The code drops a queue entry when RetryCount > mrfRetryLimit or the save channel is full.

The source object and its pending replication state remain. Dropping a queue entry does not mean losing source data. However, subsequent repair depends on the scanner; a prompt retry can turn into a wait for a scan, so this mechanism does not establish a fixed recovery deadline.

The actual defect was that TotalDroppedCount / TotalDroppedBytes increased internally but were omitted from both public statistics snapshots and from Prometheus v2/v3. A deterministic capacity-one fixture demonstrates the failure: one entry is admitted, a 20-byte overflow entry and a 30-byte retry-exhausted entry are dropped, and internal totals become 2 / 50 while administration reports 0 / 0.

The selected minimum change

1. Atomically read the existing counters into both administration snapshots.
2. Register and load cumulative counters in metrics v2 and v3, and document their meaning.
3. Warn at the two actual drop sites: retry exhaustion and a full MRF channel. Fixed messages and deduplication keys prevent changing counter values from defeating deduplication. Handing ordinary worker overflow to MRF is not itself a drop, so it does not gain a warning here.
4. Route ordinary objects, healing, and deletion consistently by (bucket, objectName) to restore worker affinity for the same object.

These counters accumulate since Server startup and reset on restart. Operations count entries, not unique objects; one object can contribute repeatedly. Bytes cover known sizes, with deletion entries contributing zero bytes. They measure neither data loss nor the complete backlog. Operators should examine increases alongside replication backlog and target health.

This work does not enlarge the queue, increase retry limits, add a persistent retry scheduler, or introduce another backoff framework. Existing limits continue to bound memory use, and the scanner remains the eventual repair mechanism. The repair makes an invisible condition observable; it does not promise a recovery deadline under arbitrary failures.

#137: make cancellation part of a run’s lifecycle

One token cannot cancel a group

Previously, cancellation placed one unkeyed token into a shared channel. One site resync can cover several buckets, with up to 10 concurrent bucket runs, while dispatchers and workers compete to consume the token. Only one bucket might stop; an unrelated task might consume it; or a leftover token might affect a later task.

Two blocking phases had independent defects: a bare receive from Walk output did not observe cancellation, and sending to a full worker channel did not observe it either. Once a worker exited, the dispatcher could block forever on a channel with no receiver. Walk also inherited the parent context, so an early function return could not stop its own walker.

State handling had related faults: site updateState modified a local value without writing it back to the map; bucket Canceled handling was incomplete; and an old finalizer could overwrite canceled state with Completed.

Registration, cancellation, and status share a write boundary

Each resyncBucket creates an owned context.WithCancelCause and registers before waiting for a concurrency slot. Registration and cancelResyncID share the resyncer’s state lock:

• Registration validates that the target still exists and the resync ID still matches, and reads its current cancellation state.
• Cancellation marks matching Pending / Started states Canceled before canceling all matching registered contexts.
• A registered run still waiting for a slot receives cancellation; a run registered after cancellation sees the canceled state.
• Walk, dispatch, workers, and result sends observe that run’s context. Unrelated IDs are unaffected, and there is no buffered token for a later task to consume.

A dedicated operation mutex serializes site start/cancel configuration setup. Canceling running work still executes if part of the target-configuration loop fails. Bucket finalizers and counter updates check the current target and resync ID, ignoring late results from removed, replaced, or canceled runs. Site state is actually written back, and a late Completed result cannot overwrite Canceled.

Success and failure require different finalization order

The result-consumer contract from #136 must remain intact: persist terminal state only after the consumer finishes.

Success: close worker inputs → join workers → drain results
         → persist terminal state → return slot → cancel owned context and unregister

Failure/cancellation: cancel context first to unblock work → join workers and consumer
                      → persist the appropriate terminal state → return slot → unregister

Canceling before joining workers on a normal completion path could discard pending work and turn a successful run into Failed. The final code calls finish exactly once from one defer, using defer order to complete cleanup. It therefore needs no additional sync.Once.

WithCancelCause distinguishes an explicit user cancellation from parent-context interruption. User cancellation becomes Canceled; parent interruption observed during finalization must not leave an interrupted run marked Completed. Shutdown while a run is still waiting for a slot preserves the existing Pending state for restart recovery.

Protect terminal state and resumed work

Cancellation can still arrive between the context check and the terminal save. Under its lock, markStatus must persist an existing Canceled state even if the finalizer previously computed Completed. An old resync ID must also be unable to overwrite a new run.

This is not a transaction across metadata files. If a bucket completed and persisted before cancellation, a subsequent site cancellation can leave different records reading “bucket Completed, site Canceled.” Completion happened first. The guarantee is that late completion cannot turn an already canceled run back into Completed, not that cancellation erases work completed before it.

The recovery loader, loadResync, previously launched goroutines and then immediately executed defer cancel(). Inspection of SILO’s actual shared-lock implementation confirmed that this cancel ends the merged leader context; it is not a no-op. A WaitGroup now retains that context until resumed runs exit. Losing leadership still cancels the existing context; replacing it with a global context would bypass the leadership constraint. Loading disk state must also preserve newer in-memory start/cancel state.

Fable review and the complexity decisions

The review used the installed Claude Code with model argument claude-fable-5-1[1m] and --effort max. Baseline reproductions and the minimal proposal produced agreement on all three repairs. The final patch and validation were then reviewed again, yielding GO. The conclusion rests on code and evidence, not model agreement alone.

The final review accepted two further implementation boundaries. The infrequent resync start operation holds the status lock across configuration reads and writes, consistent with existing terminal saves. The active registry uses resyncOpts, including resyncBefore, as its key; current callers reuse the same in-memory values, and no identity mismatch was reproduced. Future changes that reconstruct time values or restore task identity should revisit equivalence, rather than adding another registry without evidence now.

Validation and reproducible evidence

Regressions were added against unchanged production code first, and failed before the fixes. New tests exercise the production handler, real erasure storage, actual metric registration, and resyncBucket, rather than only testing helper logic that repeats the implementation.

Tests pinned to the merged revision: delete markers, MRF visibility, and cancellation lifecycle. With that revision and Go 1.27.1, the relevant checks can be repeated with:

go test ./cmd -run '^(TestReplication|TestReplicateDeleteMarker|TestResync|TestSiteResync)' -count=1
go test -race ./cmd -run '^(TestReplication|TestReplicateDeleteMarker|TestResync|TestSiteResync)' -count=1
go test ./cmd -run '^TestResyncCancel' -count=100
go test ./... -count=1 -timeout=30m

After complete local validation, only new-test formatting and fixtures changed: an existing ARN was reused, and the collector supplied the metric prefix, preventing the compatibility scanner from treating test strings as new protocol identifiers. The guard was not weakened. Relevant tests, lint, and compatibility checks were rerun after those adjustments, and remote CI checked the final commit.

Maintenance and release decisions

Future changes must continue to establish operation identity, visible failure, truthful per-object outcomes, complete terminal counters, and cancellation that releases its own resources. Neither an API returning Completed nor an object existing at the destination can replace those checks.

The code verdict is GO, and the delivery facts are main integration and passing CI. A formal release still requires selecting a tag, verifying packages and images, and establishing that deployments contain the repair. The scanner-dependent MRF recovery delay, the open cross-pool issue, and the externally reported 405 storm not reproduced on current SILO remain part of this decision record.

Second round (2026-09-16): worker-side purge classification and persisted MRF recovery

The first round classified deletes at the handler entry (#153), exposed MRF queue drops (#152), and completed resync cancellation (#137). The second round — PR #196, fix 0c61128d2, integration verification aea3882c9 — repairs a different surface: the replication worker’s own exits, the outer aggregation, and the persisted MRF recovery path for delete markers. The two rounds are complementary; neither subsumes the other.

Status: on verified main 40220bd836cb, not in Server 20260903. All evidence is synthetic (real single-drive and 16-drive storage, signed DELETEs, controlled HTTP targets, real persisted-MRF disk files replayed through a fresh worker pool). The externally reported 405 storm is not reproduced and not attributed to these paths.

What was still broken

• Legacy-shape tasks were skipped entirely. A task with an empty VersionID, a non-empty DeleteMarkerVersionID, a COMPLETED creation target, and a PENDING purge target never issued its DELETE — the per-target creation early-return suppressed it.
• Fixing only the target function made aggregation lie. The outer status selection keys on VersionID and creation state, so a failed legacy-shape purge aggregated as COMPLETED, emitted ObjectReplicationComplete, and skipped queueing to persisted MRF — worse than the baseline.
• Persisted MRF dropped every marker 405. Replaying a disk MRF entry for a delete-marker version fetched real marker metadata plus MethodNotAllowed, and the error path discarded it — the marker MRF recovery route was a dead end.
• Failed purges overwrote successful resync markers, completed purges were re-sent, and a not-yet-ready HEAD unconditionally overwrote creation state.
• A multi-target empty-state regex misparse could corrupt creation state once. A disk marker carrying only creation metadata, replayed against a task with purge state, let two empty target states parse as a bogus Pending; the deleted-flag guard then rewrote the whole creation block as empty entries with fresh timestamps — a one-shot corruption that requires disk/task divergence to reach.

Root causes: the worker had no task-level purge classification (the per-target predicate in the community’s #184 was itself wrong — a composite purge status can never reach COMPLETE through it; its investigation and proposed fix nevertheless shaped this follow-up, with thanks to Julien Laurenceau); MRF recovery had no valid-405 identity gate; and the delete task carried no retry count, so the existing mrfRetryLimit drop was unreachable on the delete path.

The repair

• One classification, every exit. isVersionPurge() (non-empty VersionID, or a non-empty DeleteMarkerVersionID with a composite purge status) drives both the inner target function and the outer aggregation. Purge exits write only VersionPurgeStatus and leave ReplicationStatus empty — the storage layer’s “do not update” signal.
• A three-field clear guard empties the composite status on the purge path, making the multi-target misparse unreachable at disk writes.
• Purges send the canonical permanent-delete request — explicit versionId, ReplicationDeleteMarker=false, no HEAD/readiness probe (authorization is the DELETE’s own). This also prevents a lost-response retry from re-creating a marker at an unversioned fallback.
• A valid-405 gate for MRF recovery. A MethodNotAllowed schedules recovery only when the returned object is a delete marker, the bucket/object/version identity matches, and the modification time is non-zero.
• A bounded retry budget. Delete tasks carry a retry counter, incremented at all three persisted-MRF entry points (aggregation failure, lock failure, queue-full fallback), respecting the existing limit; after exhaustion, the scanner can still re-raise healing.
• Audit and event status map COMPLETE to COMPLETED at the statistics/event boundary only, reusing the existing legacy constant.

What 405 means, precisely

For creation (replicating a delete marker), a HEAD 405 on the target marker version means already created — idempotent completion. For purge (permanently deleting a version), a 405 from MRF identity probing with full marker identity means work remains; the purge’s own success is decided by the DELETE alone, and a DELETE 403/405/503 is always a real failure. Empty or null version markers return ObjectNotFound, not 405, and sit outside the gate.

Boundaries that remain

The legacy in-memory task shape does not serialize across restart and has no current producer — its handling is robustness, not an active repair. Targets without a configured client still only log. Target-level resync replacing a purge subset is a pre-existing defect this round neither caused nor fixed. Persistence was driven directly in tests; timer-based flush and process-crash durability are not claimed. Multi-process site-replication meshes and cross-region acceptance are out of scope. For the tag-ordering and replica-metadata repairs in the same reliability series, see Replicated Tag Ordering and Replica Metadata Normalization.

4 - Replicated Tag Ordering: Revision Timestamps, Tombstones, and Resurrection

This page records the analysis and repair of two defects in how object tags keep their ordering across replication, merged into Server main as PR #193 (fix 03027727d) and PR #196 (fix 680eac66e).

As of 2026-09-16: both fixes are on verified main 40220bd836cb. They are not in the published Server 20260903; use the linked PRs to identify a build containing them.
Delivery boundary: source acceptance (regression tests plus the R4–R8 integration run whose PR #196 checks passed) only. No tag, package, image, or production rollout is established by this record.
Evidence class: every mechanism below was demonstrated with synthetic experiments against real single-drive and 16-drive erasure backends. No customer incident is attributed to these paths.

The shared model: a tag value and its revision are one state

Tags replicate with an internal revision timestamp (x-minio-internal-tagging-timestamp). The ordering rule on the receiving side is simple: a replicated tagging state wins only when its revision is newer than the stored one. Both defects in this family break that rule by making the revision — not the value — the part that gets lost:

• R4 dropped the timestamp on the way into an SSE-KMS destination, so newer tag updates lost to stale stored state inside the storage-layer reconciliation.
• R5 meant an empty value (a deletion) carried no revision at all, so the protocol could not express “deleted at time T” — and a delayed event could resurrect what a client had already deleted.

R4: SSE-KMS destination copies dropped tag revision timestamps

Failure form. A trusted replication COPY to an SSE-KMS encrypted destination returned 200, the object was correctly encrypted, plaintext GETs worked — and the destination’s tags and their timestamps stayed at the old values. Because the HTTP request succeeded, nothing surfaced the loss. The storage-layer reconciliation (reconcileStoredObjectTags) compares revisions under the object write lock; without the incoming timestamp, the old stored state won.

Trigger surface. Not only explicit SSE-KMS headers. Bucket-default KMS and global automatic encryption hit the same code path, so the defect could fire with no KMS header in the request at all.

Root cause. The option builder for PUT-like requests parsed the trusted source-tagging timestamp into ObjectOptions, then the SSE-KMS branch constructed and returned a different ObjectOptions carrying mtime, ETag, replication trust, and two Object Lock timestamps — but not ReplicationSourceTaggingTimestamp. The omission dated back to upstream c4373ef290 (2021); a 2026 Object Lock repair added two more timestamps to that literal and still missed this one. The field’s only consumer is the COPY tag-ordering comparison, which is why PUT and multipart were not affected — that split is also what keeps R4 separate from R5.

Fix. One field added to the existing SSE-KMS ObjectOptions literal (03027727d), nothing else. Regression tests cover every destination encryption (none, SSE-S3, SSE-KMS with and without key context, SSE-C) crossed with trusted/untrusted source, missing/valid/malformed timestamps, and 50 signed ordered COPY+GET cycles across two single-pool backends (single-drive and 16-drive erasure) with 1–3 ns event spacing. The KMS cases use a test stub.

No backfill. A lost source-tagging timestamp cannot be reconstructed at the destination. After upgrading, new tag events replicate in order; replay of old events still follows the timestamp comparison: an incoming event must be strictly newer; the stored value wins ties and rejects older events.

Deferred observation. The same SSE-KMS literal also omits the proxy and speedtest option fields; the speedtest flag is read on the storage path, so global auto-encryption would drop it on a speedtest PUT. Registered as a separate follow-up; deliberately not bundled into this repair.

R5: empty tag values had no revision, so deletions could resurrect

Failure form. Nine baseline regressions, all against real storage:

• A successful DeleteObjectTagging never minted a new revision, so a later-arriving trusted metadata COPY with an older view of the tags re-instated them.
• A newer COPY carrying an empty tagging state was ignored — empty meant “nothing to say” instead of “deleted”.
• The first replica PUT parsed the source tag timestamp and never persisted it.
• Equal visible values with equal timestamps collapsed to “no replication needed”, so a delete → re-add sequence never re-established ordering.
• A queued replication event’s completion callback wrote the old tags from its snapshot back over an already-committed deletion — and without a timestamp, the resurrected set inherited the deletion’s newer revision, which is worse than the originally-reported symptom.

Root cause. A tag value and its timestamp (including the empty value’s timestamp) constitute one state. The old protocol could only represent non-empty states: DELETE-tagging never minted a revision, the sender only attached timestamps in the non-empty branch, and the receiver only made decisions in the non-empty branch.

Fix (680eac66e):

1. Every tagging mutation mints one revision. PUT/DELETE tagging handlers unconditionally stamp a single UTC RFC3339Nano revision, whether or not replication selects the object. The storage layer enforces monotonicity under the write lock: a local revision that is not strictly newer is advanced to stored+1 ns, and multi-pool backends compute one value that strictly exceeds every pool’s copy.
2. The sender transmits tombstones. Empty values carry their recorded revision; empty-without-revision is not fabricated; a malformed recorded timestamp fails closed rather than being silently repaired.
3. The receiver accepts tombstones. Replication COPY captures the stored tag pair before rebuilding, so an empty value with a timestamp enters the existing reconciliation as a state that can win. Duplicate suppression is relaxed only for a strictly newer source revision. Multipart completion orders tags from the persisted upload metadata. The delete-acknowledgement path no longer writes snapshot tags back.

Operator-visible changes:

• Equal timestamps now resolve to stored-wins on unqualified and explicit null COPY requests. The old handler let the incoming state win ties on the unqualified path; the change is a compatibility-visible tightening in the correct direction.
• An ordinary COPY with tagging REPLACE and no tags now genuinely clears destination tags. The old default-metadata path carried source tags over — an S3 consistency improvement, but a behavior change.
• Every ordinary COPY (including key rotations that change nothing) records one new local revision.
• Both ends of a replication pair must upgrade together. An old peer still drops empty-value revisions, so a new sender’s tombstones are invisible to it.
• Objects with recorded revisions incur one extra metadata COPY per object during explicit resync/heal. When the destination has bucket-default or automatic KMS encryption, that “metadata” COPY rewrites the object data — budget accordingly for bulk resync.

Limits, stated as limits:

• No migration: history with missing or wrong deletion timestamps cannot be reconstructed, and no historical tombstones are fabricated.
• Replication rules with tag filters evaluate target eligibility against the post-deletion (empty) tagging state, so a rule filtered on the deleted tag never sees the deletion. This is a pre-existing scoping decision, unchanged here.
• Arbitrary clock skew is not a total order. The repair establishes per-hop ordering, not multi-site causality.
• A malformed recorded timestamp makes the sender’s construction fail permanently and the event retry through MRF until an explicit, correct tag change replaces it — deliberately fail-closed.

Rejected alternatives

• Synthesize a tombstone from ModTime for empty-without-revision objects. Every object that never carried tags would gain a revision; combined with forced metadata replication, every object would take the metadata-COPY path on every hop.
• Transmit only when the value is empty. Withdrawn by its own proposer during review: the equal-value re-add sequence (set X at T1, delete at T2, re-add X at T3) loses the re-add’s revision under that condition. A regression test (TestTaggingRepeatedValueNeedsRevisionDelivery) pins the counterexample.
• A new HEAD revision protocol. The pinned minio-go metadata extractor discards internal response headers, so this needs a new wire contract for marginal benefit; the worst case is still a forced metadata COPY.
• A distributed causal clock, or regenerating timestamps at commit. The wall-clock model plus the in-lock monotonic guard is the minimal correct fix.
• Per-pool monotonic guards only. Ordinary reads return single-pool object info, so a sender could emit a stale primary-pool revision; the unified multi-pool value is required.

Verification and what it does not prove

R4 regressions cover the encryption × trust × timestamp matrix and the ordered sequence on two single-pool backends (single-drive and 16-drive erasure), with a stub KMS. R5 separately has a multi-pool tagging-deletion regression. The R4–R8 integration run repeated targeted tests on the merged tree, including the isolated R5 multi-pool test; all eleven PR #196 checks passed. These establish the tested ordering cases. They do not establish multi-site scheduling under real clock skew, cross-region failover, or behavior of deployments that upgrade one end of a pair only.

The upgrade summary and release boundary live in the component matrix; the sibling repair that stops normalized replica metadata from being re-injected is recorded separately in Replica Metadata Normalization.

5 - SILO Server 20260903 Pre-release Review

This is the durable pre-release engineering record behind SILO 20260903. It explains why an earlier “all issues are solved” assessment was not accepted at face value, what the independent review found, how the fixes were narrowed, and which gates still remained at the review point. The linked release note records the later publication result.

Decision: the source candidate at 6e112d1856d4f3655f30fc81ee47e9f43d50d8f3 is a code-level GO for remote review. Production release remains a conditional GO until remote CI, Test Release, tag and artifact verification, signing, container publication, and public pull checks complete.
Baseline: RELEASE.2026-08-06T00-00-00Z at 3be10fcc1a44f6620ded0bd303461f9d688cca23.
Scope: SILO Server behavior and its embedded/pinned runtime components. Documentation, the standalone Console, mcli, package repositories, images, and the deployed site are separate deliverables.
Publication closure: the later final tree 9b11dc9469e650815b775cb47b039610644f5da4 was published as RELEASE.2026-09-03T13-18-01Z on 2026-09-04 after the remote, package, provenance, container, and public-download gates below completed. The conditional decision in this page remains the historical review criterion, not the current release state.

2026-09-09 follow-up: the repairs and main validation for #153, #152, and #137 are recorded separately in Replication Reliability. That record preserves #136’s accounting contract, explains cancellation lifecycle decisions, and keeps #133 separately tracked. It does not change the historical assessment of the 0903 release candidate below.

Why the second review was necessary

The first implementation pass had strong test results and resolved most reported defects. Its conclusion was nevertheless too broad: it treated green tests and a clean worktree as proof that every security invariant had been closed.

An adversarial review asked different questions:

• Can the same invariant be bypassed by a different valid wire representation?
• Does a pre-authentication fast path still perform I/O or acquire state?
• What happens when metadata exists but cannot be loaded?
• Do two individually correct read-modify-write paths share the same serialization boundary?
• Does request sanitization preserve all SigV4 streaming state?
• Does a validation claim describe the final tree or an earlier one?
• Is a complex mechanism protecting a reproduced failure, or only a hypothetical future?

That pass found real defects after the initial “ready” claim. The correct response was not to distrust all prior work, but to narrow every assertion to an invariant and an observed tree.

Review result by area

The invariants that now define the candidate

Trust is derived once, after authentication

An internal-looking header is still client input. The request must first pass the existing authentication path in its original signed form. Only then can the handler combine:

1. an exact, single replication marker;
2. a non-anonymous authenticated identity;
3. s3:ReplicateObject or s3:ReplicateDelete on the addressed resource;
4. replica status where the narrower replica-only semantics require it.

The result lives in private request context. Header stripping is defense in depth for legacy consumers, not the source of authority.

This ordering matters because SigV4 may sign the headers. Sanitizing first would reject legitimate replication with SignatureDoesNotMatch. The sanitized clone also has to share the request trailer: trailers arrive after the initial header parse and carry streaming checksums.

The complete receiver-wide model is in No I/O Before Auth, No Privilege From Headers.

A shared record has one write boundary

Policy, lifecycle, SSE, tags, quota, replication, Object Lock, versioning, and CORS are logical fields but physical members of one bucket record. A per-field mutex cannot protect a whole-record read-modify-write.

The selected repair is deliberately smaller than a new database or transaction layer:

acquire metadata.lock
  load or reuse current record
  mutate the requested field
  parse/normalize the complete record
  persist atomically
  publish the in-memory record
release metadata.lock

The lock does not cover object data I/O and is bounded to a bucket-metadata operation. Migration and healing must participate because they also replace the whole record. Replication receivers merge only changed fields so an older remote snapshot cannot erase unrelated local state.

Failure is a state, not the same thing as absence

The CORS hot path must distinguish four states:

The second row is why a failed-bucket set survives the simplification pass. A pre-signed URL is already authorized by its signature and may access a private object without bucket-policy evaluation. In that case the bucket CORS document is the browser-origin boundary. Losing the failure bit and using a permissive global fallback would weaken that boundary.

The set remains bounded by real bucket load attempts and is maintained through two helpers. Successful load, removal, stale-bucket cleanup, refresh, reset, and concurrent load all have tests.

Object Lock is semantic, not textual

Any valid enabled Object Lock configuration implies versioning. XML whitespace, element order, and the presence of a Default Retention rule do not change that meaning. Therefore normalization follows parsing, not a byte comparison against one canonical document.

The resulting versioning record is plain Enabled. A suspended state and an exclude-prefix extension are incompatible with the lock invariant and are removed on update, read-back, and reload.

Complexity audit

The pre-release pass explicitly looked for over-design, duplication, defensive programming without a threat model, and stale compatibility machinery.

Complexity retained because it protects a reproduced failure

• One metadata lock: retained because a deterministic cross-type lost-update test reproduced data loss.
• CORS tombstones: retained because site replication cannot distinguish deletion from “never observed” without them.
• CORS load-failure state: retained because a pre-signed URL provides an authenticated, policy-independent counterexample.
• Two replication trust levels: retained because ordinary replication and replica-ciphertext/SSE semantics do not use identical wire shapes.
• Post-authentication sanitization: retained because sanitizing before SigV4 verification breaks legitimate signed requests.
• Adversarial multi-pool/null-version tests: retained because single-pool happy paths do not exercise the state-selection failures they caught.

Complexity removed or narrowed

• CORS failure-set mutations were centralized in noteLoadFailure and clearLoadFailure.
• The replication import path now applies changed fields rather than copying an entire possibly stale record.
• Obsolete encryption helpers, dead event-target functions, and abandoned handler branches were deleted.
• The compatibility guard stopped inventorying every exported source symbol and now protects the actual served routes and frozen wire/configuration surfaces.
• The old wait_pipe lint exemption was removed; gomodguard_v2 replaced deprecated configuration.
• Dynamic timeout tests no longer call global rand.Seed from a parallel package.
• The server returned from the temporary silo-go fork to the reviewed upstream-compatible minio-go revision.

Changes deliberately not introduced

• no generic metadata transaction framework;
• no second CORS cache or unbounded negative cache;
• no new public “trusted replication” request header;
• no cross-repository release gate that makes the server depend on a later Console or documentation release;
• no partial conditional-delete contract in the release candidate;
• no broad rewrite of inherited site-replication registers without dedicated convergence tests.

Deferrals and why they do not all have the same severity

“Inherited” does not mean harmless. It means the defect was not introduced by this change set and should be evaluated against the documented release contract. A deployment that depends on one of the affected paths inherits a deployment-specific stop condition even when the general release remains conditional GO.

Evidence

Full acceptance tree

The full local acceptance corresponds to ebac0ca73bbf251b070bb6df4d8005015841f901:

• full cmd and internal suites;
• complete cmd race suite: 365.448 seconds, pass;
• lint: 0 issues;
• rebrand/compatibility and generated-file guards;
• govulncheck with no reachable vulnerability;
• six make verify deployment shapes: 174 PASS / 0 FAIL.

The first two make verify attempts encountered environment/setup failures while obtaining mcli, not test failures. The successful run used the locally checksum-pinned mcli, retained the outbound proxy for GitHub downloads, bypassed it for localhost, and placed GNU userland tools first in PATH. That distinction is part of the evidence rather than something to hide.

Post-acceptance candidate

The only code change after that full run is 84e1580a4, which clears one CORS failure-state bit after a successful on-demand metadata reload. The candidate merge adds no code; 6e112d185 changes only Helm release metadata and documentation. On the final candidate, the following pass:

• git diff --check;
• targeted CORS and Object Lock go test -race;
• rebrand guard;
• generated-file check;
• lint with 0 issues.
• Helm lint, default and optional renders, chart packaging, and the seven-resource legacy-upgrade identity guard.

This evidence is proportional to a one-line state-transition fix, but the remote CI and release workflows must still run against the pushed tree.

Go, no-go, and ownership of the remaining gates

Code decision: GO

No confirmed code defect from the two review rounds remains unresolved in the candidate. The fixes are covered at the layer where their invariants live, and the retained complexity corresponds to reproduced counterexamples.

Production decision: conditional GO

The server must not be described as released until all of these are facts:

1. candidate commits are pushed and reviewed;
2. remote CI and Test Release pass on the pushed head;
3. the intended tag points at the reviewed chart 7.0.2/server 0903/client 0903 release tree;
4. Draft artifacts, checksums, SBOMs, attestations, and signed RPMs verify;
5. finalize and Docker release publish both classic and distroless variants;
6. anonymous download and pull tests pass;
7. release notes are updated from the tagged facts and the documentation site is deployed.

Any failure in steps 1–6 is a release blocker. A local green suite cannot substitute for them.

Deployment-specific stop conditions

Operators should delay even a successfully published release when they cannot yet:

• update every node in a distributed cluster within one coordinated maintenance operation;
• update every member of a site-replication group before using bucket CORS;
• revise IAM policies for s3:DeleteObjectVersion and status-action separation;
• avoid or explicitly accept the known #10, #77, #79, #99, or #100 path their workload depends on.

The final conclusion at the review point was intentionally narrower than “everything is fixed”: the reviewed candidate was ready to enter the release machinery, the remaining limitations were explicit, and production publication was gated by verifiable artifacts rather than confidence. Those gates later completed for the release linked above; the deployment-specific conditions remain applicable.

Restart and readback verification (2026-09-11, #116)

A later acceptance (#116, run on 2026-09-11) established the restart-persistence half of readiness. The durable part is the method, which any operator can reuse when validating a restart or upgrade window:

• Acknowledgement ledger. Every acknowledged PUT writes to a unique versioned key, and the acknowledgement records the VersionId, byte count, and SHA-256 immediately. Readback fetches by exact VersionId and asserts all three — an early confirmation cannot be silently replaced by a later write.
• Readback timing. Periodic re-reads happen at 15/30/60 s after the data canary succeeds, through every peer, and the final check includes writes made during a single-node outage and after its rejoin.
• No retry masking. Each canary carries a hard 60 s deadline covering setup, request, response-body read, and sleep, and SDK retries are disabled so a recovery window cannot be papered over by client-side retries.
• Driver topology. Four Linux/arm64 containers on one host with independent network identities, one drive each (EC 2+2), and tmpfs volumes kept mounted by a holder container through the full shutdown; nodes stop in parallel with a 10 s grace period, then start in parallel.

The operational finding worth remembering: admin-endpoint readiness is not data readiness. In the recorded run the candidate’s admin endpoint was online at roughly 2.5 s while the first full data canary completed only near 14 s. A readiness probe against admin/health says nothing about the data plane during that window, and no fixed sleep substitutes for an actual data-plane check.

Boundaries, stated as boundaries: this acceptance covers process/container restart and TCP peer reconnection on a single Linux host. It does not prove persistence across independent hosts, host reboots, or physical media failure, and the timings are individual observations, not latency guarantees. The run artifacts are retained outside the documentation tree; the method above is the part that generalizes.

6 - Replica Metadata Normalization: What a Trusted Copy May Not Re-Inject

This page records the repair of how a trusted replication receiver restores metadata for a replica, merged into Server main as PR #194 (fix 4fcdf37ce, merged as 9f3037e941).

As of 2026-09-16: the fix is on verified main 40220bd836cb. It is not in the published Server 20260903.
Provenance: the production logic follows PR #187 by Mikhail Khadarenka; the merged change keeps that authorship and narrows it to a review-validated boundary.
Evidence class: an HTTP-level baseline of 64 leaf cases (44 controls passing, 20 defect failures before the fix) against real single-drive and 16-drive erasure backends, plus counterfactual replay of the same suite against the baseline helper. No customer incident is attributed.

What went wrong

The ordinary PUT path normalizes metadata: it strips the transport-only aws-chunked token from Content-Encoding, and removes the X-Amz-Meta-X-Amz-Unencrypted-Content-Length/-Md5 user-metadata keys that a GHSA mitigation deliberately deletes. The trusted replication receiver, however, restored replica metadata by re-running the same permissive extractor with replication allowed — replaying every supported header and all user metadata from the original request. Concretely, on a trusted replica write the server could store and later return via GET/HEAD:

• Content-Encoding: aws-chunked (a pure transport encoding that must never be stored, per the AWS SigV4 streaming rules), or the unsplit aws-chunked,gzip string instead of gzip;
• the two GHSA-redacted user-metadata keys — a partial rollback of that mitigation, limited to trusted replica writes;
• for Snowball entries without their own PAX header: the outer archive’s content-type, cache-control, and user metadata.

The object bytes themselves were not necessarily damaged; the stored metadata was wrong. The regression was introduced by 56fa63bfd (2026-04-15, the replication header trust boundary hardening, CVE-2026-34204) — whose trust protection is correct and stays.

The fix

One file (cmd/handler-utils.go). The boolean dual-mode helper is deleted:

• the ordinary extractor unconditionally skips replication-only keys;
• a new replica extractor walks only the replication-to-internal header map and restores only the six replication-scoped fields: the SSE-C sealed key material, sealed algorithm, IV, and encrypted-multipart marker (the empty marker is honored by key presence), the actual object size, and the SSE-C checksum identity mapping;
• it never re-reads ordinary supported headers or user metadata.

Expected stored encodings after the fix:

aws-chunked, gzip (note the space) still stores gzip with a leading space, and gzip, aws-chunked stores the whole string. These are documented status quo, asserted by tests as such — not claims of repair.

Operator-visible changes

• Trusted replicas of Snowball entries without a PAX header no longer inherit the outer archive’s ordinary metadata. The six replication-scoped fields still apply to authorized entries. This matches ordinary (non-trust) Snowball behavior, and no producer in the repository ships the auto-extract marker, so nothing in-tree depends on the old inheritance.
• The GHSA-redacted keys are no longer written back on replica restore — matching what every ordinary PUT already did.
• Authentication, permission gating, and replication trust semantics are unchanged; the ordinary extraction path is byte-for-byte equivalent.
• Rolling back the code reopens the injection path but does not repair already-stored metadata.

Upgrade does not fix stored objects

An upgrade stops new pollution; it does not scan or rewrite existing objects. Two consequences matter:

• Objects whose authoritative source is still polluted will be judged inconsistent after the upgrade and re-selected for metadata replication on heal/resync — repeatedly. Fix the authoritative source first, then let the copies converge.
• Ordinary S3 self-COPY is not a general remediation API: it can create new versions or shift timestamps rather than rewriting one version’s metadata in place.

The stored-metadata remediation proposal — status

A design for a future operation exists: build an inventory (including non-current versions, not just the latest), verify by comparing against the trusted source version or an independent checksum — never by guessing from the wrong response header, and never by re-decompressing gzip just because a label says so — process the authoritative source’s exact versions first, then converge copies; keep immutable inventories and metadata backups; use small validation batches with a rehearsed rollback. Where no supported path exists for an object, stop and leave it — editing xl.meta directly is not a supported operation.

Any selected procedure must preserve the required version identity and current version relationship, Object Lock retention/legal hold, tags, replication state, and encryption context. Check for concurrent changes before writing; blocked or unverifiable versions stay untouched. A local clone must prove the chosen operation and rollback before this proposal becomes an executable runbook.

This is a design proposal awaiting separate approval, not an executed procedure. No production inventory scan, object write, version change, or deployment has been performed as part of it. Treat it as the shape of a future runbook, not as a validated one.

Known limits

• The POST-form upload path (bucket-handlers.go) calls the low-level extractor directly and never normalized encodings; that behavior is unchanged and flagged for a separate issue.
• Local verification ran with a test-only capacity accommodation (a full host disk); the merged-tree rerun in the R4–R8 integration record covers the unchanged-tree case.
• No two-site scheduler, restart, or network-failure acceptance is claimed.

Verification

Regression tests (TestExtractReplicationMetadata*, TestAPIReplicaContentEncoding, TestAPISnowballReplicaContentEncoding, plus race-included trust/SSE-C round-trips) cover the mapping table, the six restored fields, and the ordinary path’s equivalence; the counterfactual run (unchanged tests against the baseline helper) reproduces the 20 defect failures, demonstrating the tests bite. The upgrade summary lives in the component matrix; the sibling tag-ordering repair is recorded in Replicated Tag Ordering.

7 - Request-Header Deadlines: Absolute Header Limits and Rolling Body Idle Timeouts

This page records the repair of Server’s HTTP read deadlines, merged into main as part of PR #196 (fix 055030ea5).

As of 2026-09-16: the fix is on verified main 40220bd836cb. It is not in the published Server 20260903.
Evidence class: synthetic — a direct TCP comparison (header limited to 100 ms, header arriving over 400 ms, standard Go refuses where old SILO returned 204), plus a real single-drive process probe where flag and environment settings both rejected a 400 ms slow header while healthy requests continued. No production incident is attributed; the issue that motivated the investigation is a slow-HTTP DoS scanner report that has not been reproduced against a deployed cluster.

Two timeout classes, one connection

• Request-header absolute deadline. ReadHeaderTimeout bounds the total time from the start of header reading to its completion. Trickle-feeding bytes cannot extend it. On HTTP/1 keep-alive connections, Go first uses IdleTimeout while waiting for the next request’s initial bytes, then starts a fresh header deadline. ReadHeaderTimeout also participates in the separate TLS-handshake timeout calculation.
• Body rolling idle timeout. Once headers parse and the connection enters the active phase, the existing rolling semantics return: every successful read extends the deadline, and only a stall between bytes (the configured idle timeout) kills the connection. A long upload or download that keeps making progress is not capped in total duration by this HTTP/1 repair; other protocol, proxy, and application timeouts still apply.

Before this repair, the first class did not exist in practice: the connection-layer wrapper replaced the socket deadline with now + idle + 250 ms before every partial read, overwriting whatever absolute deadline net/http had set — so a slow reader could hold a connection open indefinitely by sending one byte per idle window.

Two independent defects

1. The connection layer neutralized the absolute deadline. The DeadlineConn wrapper’s read path reset the socket deadline on every partial read, defeating the read-header deadline Go’s server sets. The direct-TCP baseline proved it in isolation: with a 100 ms header limit and a 2 s idle window, a header that finishes at 400 ms was accepted.
2. The configuration never reached the server. The CLI accepted --read-header-timeout and MINIO_READ_HEADER_TIMEOUT, parsed defaults and all — and the server-context builder copied IdleTimeout while dropping ReadHeaderTimeout entirely, so the running server always saw zero. Fixing defect 1 alone left the real process accepting slow headers; the second fix is one line next to the idle-timeout binding.

The reason this stayed invisible for so long: the flag’s default (30 s) equals the idle timeout’s default, and with the flag unwired the server fell back to exactly that same 30 s — so every observable default behaved as if configured.

Configuration

• Flag: --read-header-timeout
• Environment: MINIO_READ_HEADER_TIMEOUT
• Default: 30 s (equal to the idle timeout default)
• There is no YAML configuration field for either timeout; the value binds once at startup from flag > environment > default.

What each protocol gets

• HTTP/1: headers and keep-alive waits are absolute; the body keeps the rolling idle timeout. The connection-state hook composes with (rather than replaces) any caller hook.
• TLS: handshake reads take the minimum of the positive header deadline and the existing read/write timeouts; after the handshake, a fresh header limit begins. The handshake’s write side remains rolling — this repair is not a complete TLS-handshake resource limit.
• HTTP/2: untouched. When h2 is negotiated, the phase switching is skipped entirely; h2 keeps its own native per-stream read timeout, which is absolute, and ReadHeaderTimeout never enters the h2 configuration.
• Internal callers: Linux internode dialing uses its own rolling semantics; grid-hijacked connections unwrap to the raw TCP connection before any of this applies.

Rejected alternatives

• Clamp all future deadlines globally. Go 1.27 sets a whole-request deadline in some paths; with the read timeout equal to the idle timeout, this would hard-cap entire HTTP/1 requests — header plus body — and kill every large upload.
• Drop the read timeout and reinterpret zero as rolling idle. Zero is net/http’s “never time out” for background reads and hijacked connections; reinterpreting it would break long handlers, and h2 would lose its per-stream timeout.
• Wrap the body reader / response controller. Full chunked/drain/EOF accounting with h2 special cases is a far larger change than the header defect requires. (A later, unmerged branch explores a body-side response controller for the same DoS family; as of this record it is not part of main and not part of this repair’s claims.)
• Reconstruct the standard library’s deadline arithmetic in the hook. Duplicates stdlib internals that drift between Go versions; remembering the value stdlib actually asked for is the robust form.
• Auto-derive strictness from value comparisons. With all three defaults equal at 30 s, “shorter than the idle window” is indistinguishable in production defaults; such logic only works in test configurations.

Verification and limits

Tests pin the connection wrapper across three consecutive update periods (no extrapolation of the absolute cap), the phase transitions over HTTP/1 keep-alive, TLS, and HTTP/2-only negotiation, and the flag/env binding on the real CLI context; a process probe exercised a live server with a 100 ms header limit rejecting a header that takes 400 ms. Known limits: the TLS handshake write side stays rolling; handler CPU/storage waits have no deadline; the absolute header cap carries no slack while the rolling idle keeps its ~250 ms update slack; and multi-node, cross-region long-transfer acceptance is future work — the integration record explicitly does not count a scripted S3 long transfer as passed for this repair.

The upgrade note (a shorter header timeout also narrows the TLS handshake window; it is not a total-duration limit for uploads or downloads) is in the component matrix.

8 - Conditional DELETE: Why the Condition Must Be Evaluated Once

Follow-up, 2026-09-13: single-object If-Match DELETE later landed as 40bee4b7b, followed by main-branch multi-pool serialization and cleanup fixes. Latest published Server 20260903 does not contain these later changes. The August 26 design status below is historical; see the component matrix and Server changelog.

This document records the analysis, design discussion, and repair decision for SILO PR #12.

Status on 2026-08-26: PR #12 remains open at head 5b71a75e, 118 commits behind the latest main. Its commit has no DCO sign-off and GitHub reports no check runs. The improved design described here has been implemented, tested, and reviewed twice in an isolated local worktree, and committed on the local branch codex/pr12-conditional-delete; it has not been pushed, merged, or released.
Scope: correctly support If-Match for the single-object DeleteObject API, and fail closed instead of silently deleting when an unsupported per-object DeleteObjects ETag is received. Full batch-condition execution and bucket-policy enforcement remain separate deliverables.
Release boundary: local implementation, tests, review, commit, push, remote CI, merge, tag, image publication, and production deployment are independent gates.

Too Long; Didn’t Read (TL;DR)

The underlying problem is real. SILO currently ignores If-Match on DELETE, so a client can believe it is performing compare-and-delete while the server performs an unconditional deletion. PR #12 targets the right problem and correctly recognizes that the condition must be evaluated against fresh object state while holding a lock.

The original implementation puts the same HTTP callback into every erasure pool. Different pools can retain copies from different points in time, so each pool evaluates and mutates against its own ETag. A two-pool test reproduced both failures:

• the request ultimately returns 412 after the older matching copy has already been deleted;
• the request succeeds after deleting the current copy, while an older non-matching copy remains and becomes visible again.

The selected repair introduces no new condition framework. It follows the established multi-pool GET pattern: select the current object under the outer namespace lock, evaluate the condition exactly once, then clear the callback before calling lower pools. A false condition mutates no pool. A true condition allows the existing cleanup to run without reinterpreting the client condition per copy.

Why this is a real problem

Silently dropping the condition is unsafe

AWS conditional-delete documentation now defines the behavior for general-purpose buckets and both DeleteObject and DeleteObjects:

Ignoring the header is therefore not a harmless unsupported extension. It removes the concurrency guard the caller used to avoid deleting another writer’s newer object.

Not every S3 client sends conditional deletes, so prevalence is unknown. Severity for each relying caller is high: one silent downgrade can remove newly committed data.

A delete marker is not merely an ETag comparison case

PR #12 reuses the generic isETagEqual, which returns true whenever the right-hand value is *. Consequently, isETagEqual("", "*") is also true.

The more fundamental bypass occurs one layer above. erasureServerPools.DeleteObject returns success immediately when the current object is already a delete marker. That happens before the callback added by the PR. A diagnostic test observed zero callback calls and a successful result.

The impact needs precise wording:

• the delete-marker fast path did not remove a historical version or create another marker in the reproduced case; it bypassed the condition and falsely reported success;
• the multi-pool counterexamples do mutate storage on a failed request or leave a stale copy after success.

Changing only isETagEqual("", "*") cannot cross the outer fast path and risks altering a comparator shared by GET, PUT, and COPY.

What the original PR got right

Its high-level algorithm is sound:

1. detect If-Match in the handler;
2. read fresh ObjectInfo after acquiring the storage lock;
3. return before mutation when the condition is false;
4. encode the result as an S3 response.

This avoids the obvious TOCTOU window of a separate HEAD followed by DELETE. The PR also adds handler, helper, and erasure-layer tests. Its ordinary single-pool path correctly returns 412 and preserves the object for a wrong specific ETag.

The defect is not the decision to evaluate under a lock. It is choosing the wrong layer and therefore the wrong object state.

Where the atomicity boundary lives

The deletion path has two layers:

DeleteObjectHandler
    -> erasureServerPools.DeleteObject
         holds the namespace write lock
         selects the latest current pinfo across pools
         handles the delete-marker fast path
         selects a single-pool or all-pool deletion
             -> erasureSets / erasureObjects.DeleteObject

Only erasureServerPools.DeleteObject knows:

• which copy represents the current object;
• which pools still contain older copies or inconsistent metadata;
• whether the delete-marker fast path applies;
• whether multiple pools will be mutated concurrently.

The client condition therefore belongs at this layer. A single pool knows only its local copy and cannot reinterpret a condition on the logical current object.

Two-pool counterexamples

The test places an older object in pool 0 and a newer object with a different ETag in pool 1. Reads select pool 1 as current, while an unversioned delete cleans both pools.

Condition matches the old copy

The original PR lets pool 0 pass and delete its copy while pool 1 fails. The aggregate result follows the current pool and returns 412, even though storage changed.

Condition matches the current copy

Pool 1 passes and deletes the current copy. Pool 0 fails and retains the old copy. The request returns success, after which the old object becomes visible again.

The callback also captures a single http.ResponseWriter. Calling it concurrently from multiple pools can make multiple goroutines write the same HTTP response. Storage replicas should not concurrently decide wire-level output.

A pre-existing degraded-pool limitation

There is one related but inherited limitation outside this patch. If the selected current pool is readable and writable but an older, non-current pool is degraded, the existing all-pool delete path can return the selected pool’s success while an error from the older pool is not surfaced. That copy can remain and reappear after recovery.

The new condition does not create this behavior: it evaluates the readable current object correctly and then enters the same unversioned multi-pool cleanup used by an unconditional delete. Repairing error aggregation and recovery for partially degraded old pools should be tracked separately because it changes the guarantees of every unversioned multi-pool delete, not only conditional requests.

The selected minimal repair

1. Evaluate exactly once at the outer layer

After erasureServerPools.DeleteObject acquires the namespace write lock:

1. save opts.CheckPrecondFn;
2. remove it from options passed to lower layers;
3. inspect all pools and select the current pinfo;
4. if the current object cannot be read reliably, return a quorum error without calling the callback;
5. call the saved callback exactly once with pinfo.ObjInfo;
6. on success, continue through the existing deletion path with no lower-layer reinterpretation.

This pattern already exists in multi-pool GetObjectNInfo: save the callback, clear it below, select the latest object, and evaluate once. Reusing it limits the DELETE change to the real atomicity boundary.

2. Treat * as current-representation existence

The DELETE-specific check separates wildcard and ETag semantics:

specific ETag -> compare the client-visible ETag of the current object
*             -> require a non-empty object name and no delete marker

A missing key already returns Not Found during object selection. A current delete marker reaches the callback and returns 412. The generic isETagEqual remains unchanged.

3. Require read permission for a specific ETag

The handler first checks s3:DeleteObject. When the normalized condition is not a bare *, it additionally checks s3:GetObject:

• delete-only policy plus *: allowed;
• delete-only policy plus a specific ETag: 403 and no mutation;
• Get plus Delete and a matching ETag: allowed.

Authorization completes before any storage mutation.

4. Do not require SSE-C content decryption for DELETE

The original PR invokes the GET/PUT-oriented DecryptObjectInfo, which rejects an SSE-C object when SSE-C read headers are absent. Conditional DELETE needs the client-visible ETag, not plaintext content or decrypted size.

The selected implementation uses the established getDecryptedETag projection only for a specific ETag. Wildcard requests do not read the ETag. This reuses existing ETag behavior without imposing content-decryption requirements on DELETE.

5. Evaluate the current version

AWS specifies that conditional-delete evaluation applies to the current version. SILO’s outer pool selection already reads the current object, while preserving an explicit versionId for the eventual version deletion.

A regression test requests deletion of a historical version while matching that historical ETag rather than the current ETag. It must return 412 and preserve both versions.

6. Reject silent downgrades at unsupported edges

Two small guards keep the single-object feature from being bypassed:

• an empty or whitespace-only If-Match is rejected instead of becoming an unconditional delete;
• If-Match cannot be combined with the internal recursive x-minio-force-delete extension, whose prefix semantics cannot represent one object’s ETag condition; the HTTP handler rejects it and the storage layer also refuses any internal prefix-delete plus callback combination.

The batch XML decoder now also recognizes per-object <ETag> values. Until atomic per-item execution is implemented, any non-empty batch ETag rejects the entire request with NotImplemented before deletion begins. This is not batch conditional-delete support; it is a narrow data-safety guard against silently discarding a condition.

Rejected alternatives

Change only isETagEqual

It does not address the outer delete-marker fast path and risks changing several APIs that share the comparator.

Keep per-pool callbacks and aggregate the result

An aggregate error cannot roll back a copy already deleted by another pool. The condition applies to the logical current object, not independently to every physical copy.

Introduce a new condition object or transaction coordinator

The current feature has one If-Match condition, and CheckPrecondFn already expresses it. GET demonstrates the correct one-shot consumption pattern. A new DSL, state machine, or cross-pool transaction abstraction is unnecessary.

Complete every conditional-delete feature in one PR

DeleteObjects and policy conditions cross different API and repository boundaries. Combining XML parsing, per-item responses, IAM, quiet mode, and dependency publication with the core deletion repair would make the change harder to validate.

Test and acceptance contract

The minimally sufficient matrix is:

The original PR’s quorum test merely took 8 of 16 disks offline and asserted that some error occurred. Delete write quorum was already unavailable, so the same test passed on main without conditional DELETE. The replacement asserts the specific quorum result, zero callback calls, and object survival after restoring the disks.

Independent adversarial review

Two read-only local Claude Code reviews used the Fable model at xhigh effort against the exact server diff and both design records. Both verdicts were GO WITH NON-BLOCKING NOTES, with no P0, P1, or P2 findings after the first round’s changes were applied.

The first review found the conditional force-delete bypass, whitespace-only downgrade, silent batch-ETag discard, missing versioned-success coverage, and the inherited degraded-old-pool limitation. Those findings produced the guards, tests, and limitation text above. The second review confirmed the outer atomicity boundary, error handling, auth split, batch-field blast radius, response-writer behavior, bilingual parity, and minimality. Its remaining actionable P3 was a hypothetical internal caller combining prefix deletion with a callback; the storage layer now rejects that combination too.

One reviewer sentence suggested that SSE-C without customer-key headers would necessarily fail the condition. Direct inspection showed the opposite established behavior: getDecryptedETag projects the stored client-visible suffix without asking to decrypt object contents. A focused regression test now pins that behavior. Remaining non-blocking notes are multiple-header normalization and the deliberate 501-before-auth error-ordering nuance. A live-AWS differential check for specific ETag versus a current delete marker and versionId plus If-Match would still be useful before claiming byte-for-byte behavioral parity beyond the published contract.

Deliberate follow-up scope

Per-object conditions in DeleteObjects

The AWS DeleteObjects API accepts an <ETag> per <Object> and returns each outcome under <Deleted> or <Error> in the same 200 response.

The safety patch adds an ETag field to ObjectToDelete only so the handler can detect the condition and reject the entire request before mutation. This closes the previous silent unconditional-delete behavior, but it does not implement AWS’s required per-object evaluation or mixed <Deleted> / <Error> response.

Full compatibility remains a separate high-priority change: evaluate every item against the logical current object under the correct lock, apply the exact-ETag permission rule per item, preserve quiet-mode behavior, and report each failed condition without blocking unrelated items.

The s3:if-match policy condition key

AWS policies can enforce conditional deletes. SILO’s silo-pkg does not yet define s3:if-match. Full support requires:

1. the condition key and action map in silo-pkg;
2. a new silo-pkg release;
3. correct condition values for a single-delete header and batch per-item ETags;
4. a server dependency update and policy compatibility tests.

That is a separate cross-repository deliverable, not a prerequisite for making single-object execution correct.

Complexity, benefit, and cost

Production code remains small: one DELETE-specific condition helper, one extra authorization check, roughly a dozen lines that consume the callback once at the outer layer, and narrow fail-closed guards for malformed/recursive and as-yet unsupported batch conditions. Most complexity belongs in tests because deletion spans pools, versions, markers, quorum, and permissions.

The benefit exceeds the cost. It removes a dangerous silent unconditional delete and places the condition at an existing global consistency boundary. Reusing the current outer-lock/latest-object pattern is the minimal, sufficient, and necessary design.

Merge and release gates

The single-object repair becomes mergeable only after:

1. targeted condition, permission, versioning, quorum, and two-pool tests pass;
2. go test ./cmd, go vet ./cmd, formatting, and diff checks pass;
3. an independent adversarial review has no unresolved blocker;
4. the contribution is organized on current main with a valid author DCO sign-off;
5. DCO, Go CI, VulnCheck, and other required remote workflows are green;
6. the PR description distinguishes complete DeleteObject support from the batch fail-closed guard and links the full batch/policy follow-ups.

A merge is still not a release. Users can rely on the behavior only after a corresponding SILO release, package, docker.io/pgsty/silo image, deployment, and real-client verification have independently completed.

Conclusion

Conditional DELETE is worth implementing. PR #12 has the right goal and the useful insight that fresh state must be checked under a lock. The required correction is the boundary: a client condition belongs to the logical current object and cannot be interpreted independently by every physical copy.

The selected design moves one callback to the erasureServerPools layer that already selects the current object, preserves the generic comparator, handles wildcard/delete-marker semantics explicitly, and adds the specific-ETag read permission. It changes no storage format, dependency, or public condition framework. The batch change is deliberately limited to refusing an unsupported condition before mutation; full batch execution and policy support remain separate work.

That is the minimum complexity needed to make the feature sufficient and safe.

9 - DSN-Only Database Notifications: A Compatibility Boundary for #53

This document is the product requirements and final design record for SILO issue #53. It records the accepted compatibility boundary, implementation, and verification for PostgreSQL and MySQL bucket-notification targets.

Decision

SILO will retain PostgreSQL and MySQL notification targets, but support exactly one current configuration form for each:

• PostgreSQL requires a complete connection_string.
• MySQL requires a complete dsn_string.

The old five-field form — host, port, username, password, and database — remains unsupported by the current KV configuration system. SILO will not re-register those keys and will not synthesize a DSN from them during legacy migration.

The legacy migration contract is deliberately narrow:

This is a configuration-boundary decision, not removal of the database-notification feature.

Status: implemented in server commit f1ba68358; release pending.
Owner: SILO server repository.
Tracking: pgsty/silo#53.
Target: the next SILO patch release after implementation and verification.

Context

SILO inherited two generations of database-notification configuration from MinIO.

The pre-KV JSON configuration could describe a database connection either as a complete string or as five fields:

host
port
username
password
database

The current KV configuration exposes only the driver-native form:

notify_postgres  -> connection_string
notify_mysql     -> dsn_string

This direction is not new. MinIO deprecated the five discrete fields in RELEASE.2020-04-10T03-34-42Z and instructed operators to move to connection_string or dsn_string. SILO’s current help tables, environment-variable documentation, and examples already present the complete string as the supported interface.

SILO is a new community fork with an explicit migration step. Its compatibility contract prioritizes the S3 and Admin APIs, current MINIO_* settings, on-disk data, and current KV configuration. It does not need to perpetuate every pre-2020 configuration spelling when a supported canonical form has existed for years.

The defect

Before the fix, the legacy migration helpers, SetNotifyPostgres and SetNotifyMySQL, wrote both forms into the new KV configuration. Even when the old target already had a complete connection string, the helpers also emitted all five discrete keys, usually with empty values.

The new parser rejects those keys because neither DefaultPostgresKVS nor DefaultMySQLKVS registers them. Key validation checks key presence, not whether the corresponding value is empty. Both legacy source forms therefore fail:

old complete string -> canonical string + five empty unknown keys -> rejected
old discrete fields -> empty canonical string + five populated unknown keys -> rejected

The failure is amplified by notification initialization. FetchEnabledTargets is fail-fast across notification subsystems: the first invalid subsystem returns an error and a nil target list. The caller logs the error and continues starting the object server, leaving healthy Webhook, Kafka, NATS, and other targets unavailable as well.

Merely returning an error from the two migration helpers does not fix that behavior. The error propagates through readConfigWithoutMigrate and initConfig, but initConfigSubsystem currently logs non-retriable configuration errors as “some features may be missing” and returns success. The server then starts without assigning globalServerConfig; notification failure is only one consequence, because region, storage class, compression, identity, and other stored settings may also be absent. The implementation must therefore carry a typed database-migration error to the startup boundary and make that error fatal. Classifying it as retriable is also wrong because the server would retry forever without any state change that could repair the configuration.

The resulting behavior is especially dangerous because object I/O still works. Operators can see a healthy S3 service while every configured event pipeline has stopped. Targets are never constructed, so delivery or later replay of events produced during the outage must not be assumed.

There is also a diagnostic-exposure issue. The unregistered password key has no sensitivity metadata and may be copied verbatim into health or diagnostic material. The registered connection_string and dsn_string keys are already treated as sensitive values.

Why the first fix was reverted

The first repair registered the five discrete keys and taught the parser to read them. That made migrated targets pass CheckValidKeys, and it appeared attractive because the target argument structures and constructors still contain code for the old fields.

It also broke the documented connection-string path.

The shared mc admin config set tokenizer discovers field boundaries by looking for registered key names. It is not fully quote-aware. Once port became a registered key, this valid input contained what looked like a second top-level field:

connection_string="host=db port=5432 dbname=events user=app"

The tokenizer split at the port= inside the quoted value, truncated connection_string, and handed the remainder to the port parser. The command then failed with invalid port.

Under the current tokenizer, registering common words such as host, port, and password creates a direct conflict between the connection-string grammar and the top-level KV grammar. The attempted registration fix was therefore reverted. Re-registering those keys is not an acceptable solution.

Product judgment

Database notification targets are a specialized but useful capability. They provide a direct database-backed namespace view or access journal without requiring an external event bus. That remains valuable for small deployments and for users already operating PostgreSQL or MySQL.

The legacy spelling of their connection parameters has much less value. A five-field model cannot represent the useful range of driver options: TLS modes and certificates, connection timeouts, application names, Unix sockets, multi-host PostgreSQL settings, MySQL driver parameters, and future driver capabilities. Supporting both forms also creates precedence, merging, redaction, and testing questions that do not exist with one canonical value.

The complete string is the better abstraction boundary: SILO owns notification semantics, while the database driver owns connection syntax.

The product decision is therefore to keep the capability and remove the compatibility illusion. An unsupported legacy target must be rejected clearly; it must not be accepted and transformed into a configuration that later disables unrelated targets.

Goals

1. Establish connection_string and dsn_string as the only supported live configuration interfaces for database notifications.
2. Allow a legacy JSON target that already contains the canonical string to cross the migration boundary without modification to its connection semantics.
3. Reject enabled discrete-only legacy targets before a partial or invalid KV configuration is activated.
4. Replace the current silent runtime failure mode of #53 — healthy targets disabled while the server appears healthy — with an explicit startup-time failure that operators must resolve before the server runs.
5. Ensure no migration error, log line, health report, or diagnostic bundle exposes a database password.
6. Remove the ten Postgres/MySQL exceptions from the source-level unregistered-write audit.
7. Make the compatibility boundary and operator remediation explicit in release and migration documentation.

Non-goals

• Supporting both DSN and discrete database fields in the current KV interface.
• Automatically synthesizing a DSN from old discrete fields.
• Rewriting the shared KV tokenizer.
• Changing FetchEnabledTargets fail-fast semantics in this patch.
• Silently skipping an enabled database target and continuing with partial notification coverage.
• Removing PostgreSQL or MySQL notification targets.
• Deleting the legacy struct fields needed to decode and identify unsupported input. They remain on shared target argument structs that are also used by live constructors, whose discrete-field connection-string synthesis is unreachable from current KV configuration; those fields must not become supported configuration keys.
• Correcting ignored errors from the other eight legacy notification setters. Their pre-existing silent-skip behavior remains unchanged in this narrowly scoped database-migration patch and requires a separate audit and design decision.

Functional requirements

Current configuration

1. notify_postgres accepts connection_string; notify_mysql accepts dsn_string.
2. The five discrete keys remain unregistered and rejected by current configuration commands.
3. Existing full strings must continue to support the database driver’s syntax, including parameters whose names contain host, port, user, password, or database.
4. No new public environment variables or KV keys are introduced.
5. The declared legacy variables MINIO_NOTIFY_POSTGRES_HOST/PORT/USERNAME/PASSWORD/DATABASE and their MySQL equivalents are not wired into current parsing and remain unsupported. They must not be documented as working alternatives to the complete-string variables.

Legacy migration

1. SetNotifyPostgres must return without emitting a target when the legacy target is disabled.
2. For an enabled target, SetNotifyPostgres must require a non-empty ConnectionString and write only registered Postgres keys. If both a canonical string and discrete fields are present, the canonical string wins and every discrete value is discarded.
3. SetNotifyMySQL must apply the equivalent rule to DSN.
4. Neither helper may emit host, port, username, password, or database.
5. A missing canonical string must return a typed or wrapped migration error identifying the subsystem and target name.
6. cmd/config-migrate.go must check and propagate both helper errors. Ignoring them is forbidden.
7. No partially migrated configuration may be activated or persisted after either helper fails.
8. Error text may name the required key and remediation, but must not include any connection-field value.
9. The propagated typed migration error must abort server startup. It must not be downgraded to the non-fatal “some features may be missing” path in initConfigSubsystem, and it must not enter the retriable-error loop.
10. Validation errors for a supplied canonical string follow the same startup-fatal and secrecy rules; wrapping must add target context without repeating the DSN or its components.

Recommended error shape:

notify_postgres:archive uses unsupported legacy discrete connection fields;
set connection_string before migrating to SILO

Operator remediation

An operator encountering the error must choose an explicit remediation path. This applies both before an initial switch to SILO and when upgrading a deployment that is already running SILO: legacy migration output is not persisted, so the same old JSON source can re-enter migration on every start. A deployment that currently starts with notifications silently broken can therefore fail to start after this repair until the source configuration is corrected.

1. On a compatible intermediate MinIO release, replace the old fields with connection_string or dsn_string, verify the target, and then migrate to SILO.
2. Disable or remove the legacy database target, migrate the server, and recreate the target with the canonical string afterward.
3. For a fresh SILO installation, create the target directly with the canonical string; no legacy migration is involved.
4. For an existing SILO deployment that still reads a legacy JSON file, stop on the previous working release, back up the source configuration, then convert, disable, or remove the database target before starting the fixed release. Do not delete or rewrite unrelated configuration.

Documentation must not suggest that a discrete-only target will be converted automatically.

Availability trade-off

This decision intentionally turns one unsupported configuration from a degraded startup into a hard startup failure. The immediate availability cost is real: a server that previously served objects while all notifications were silently dead may refuse to start after the repair.

That cost is accepted because an object server that appears healthy while configured event sinks are absent creates silent, potentially unrecoverable downstream data loss. SILO is a new fork with an explicit migration boundary, and the discrete form has been deprecated since 2020. A fatal, actionable precondition is preferable to an upgrade that reports success with reduced notification coverage. The release note must make this startup behavior prominent; it must not be buried as an internal migration cleanup.

Security requirements

1. The unsupported-input error must never format the legacy argument structure or its values.
2. Tests must use a sentinel password and assert that it is absent from returned errors and captured logs.
3. Migrated output must contain the registered sensitive connection-string key and no standalone password key.
4. If a diagnostic bundle was exported from an affected deployment before this repair, operators should treat the database password as potentially disclosed and rotate it.

Alternatives considered

Register and parse the discrete fields

Benefit: preserves the old source form and uses already existing argument fields.
Rejected because: registration makes common field names visible to the shared tokenizer and corrupts quoted connection strings. It also expands the supported public configuration surface after the fields were deprecated in 2020.

Synthesize a canonical string during migration

Benefit: preserves discrete-only legacy installations.
Rejected because: it creates permanent code and test ownership for an obsolete input form, including PostgreSQL quoting, MySQL DSN formatting, socket and IPv6 behavior, defaults, and future driver drift. For a new fork with an explicit migration boundary, the benefit does not justify the continuing surface.

Skip only the unsupported target

Benefit: keeps the object server and other notification targets running.
Rejected because: silently discarding a configured event sink can cause unobservable and unrecoverable event loss. A clear migration failure is safer than an apparently successful upgrade with reduced notification coverage.

Change global notification fail-fast behavior

Benefit: limits the blast radius of future invalid targets.
Rejected for this change because: it neither repairs the database target nor closes the credential-exposure path, and it changes system-wide error semantics. It may be evaluated independently with its own operational contract.

Remove database notification targets

Benefit: removes the complete database-specific maintenance surface.
Rejected because: the targets remain useful and self-contained. The defect belongs to an obsolete configuration form, not to the notification capability itself.

Implementation scope

The server change should remain narrow:

1. Update internal/config/notify/legacy.go so the two database setters emit only canonical registered keys and reject enabled targets without a canonical string.
2. Update cmd/config-migrate.go to propagate the two database-helper errors with subsystem and target context.
3. Define a typed database-migration error and update cmd/server-main.go so initConfigSubsystem returns it as fatal instead of logging and ignoring it. It must remain non-retriable.
4. Leave ignored errors from the other eight legacy notification setters unchanged in this patch; record them for a separate audit rather than expanding #53 implicitly.
5. Remove all ten Postgres/MySQL entries from knownUnregisteredWrites; the ratchet should become empty unless another independently justified legacy exception exists.
6. Add focused migration, startup, validation, secrecy, and coexistence tests.
7. Update database-notification and migration documentation in silo.pgsty.com.

The patch must not register the old keys, change the generic tokenizer, or refactor unrelated notification targets.

Acceptance criteria

The implementation is complete only when all of the following are demonstrated:

1. A legacy PostgreSQL target with a complete connection string migrates, passes CheckValidKeys, and is returned by GetNotifyPostgres unchanged.

2. A legacy MySQL target with a complete DSN does the equivalent.

3. Discrete-only enabled targets for both databases fail before target initialization with an actionable error containing the subsystem and target name, and server startup aborts.

4. Missing-string and malformed-string errors contain none of the sentinel host, username, password, database, or DSN values.

5. Disabled discrete legacy targets do not create configuration entries and do not block migration.

6. Migrated KVS output contains none of the ten discrete keys, including empty ones.

7. When a legacy target contains both a canonical string and conflicting discrete values, only the canonical string is migrated and no discrete sentinel appears in any output KVS value.

8. A SetKVS regression test using the real DefaultPostgresKVS and DefaultMySQLKVS key sets accepts a quoted connection string containing port=, host=, or password=.

9. A configuration containing healthy Webhook, Kafka, or NATS targets cannot reach FetchEnabledTargets with an invalid migrated database target because readConfigWithoutMigrate fails without yielding, persisting, or activating a partial configuration, and startup aborts on that typed error.

10. initConfigSubsystem returns the typed migration error; it neither logs-and-continues nor enters the retriable loop.

11. knownUnregisteredWrites no longer contains Postgres or MySQL exceptions.

12. The following verification passes:

    go test ./internal/config/notify ./internal/config ./internal/event/target -count=1
    go test -v ./cmd -run 'Test(ReadConfigWithoutMigrate|InitConfigSubsystem)' -count=1
    git diff --check

    The verbose cmd output must show that tests with both prefixes actually ran; a zero-match warning is a failed acceptance check. The normal server CI suite must also pass. In the documentation checkout, run make check.

Implementation result

Server commit f1ba68358 implements the accepted design without expanding the public configuration surface:

• the two legacy database setters emit only connection_string or dsn_string plus registered target settings;
• disabled targets remain ignored, while enabled targets without a canonical string return a value-free LegacyDatabaseTargetError;
• only the two database migration errors are newly propagated;
• the typed error is non-retriable, escapes initConfigSubsystem, and is classified as fatal by serverMain before logger.FatalIf exits the process;
• the ten Postgres/MySQL exceptions were removed from knownUnregisteredWrites;
• focused tests cover complete-string round trips, canonical precedence, discarded discrete values, secrecy, failed-migration atomicity, startup classification, and the real tokenizer key sets.

The final local Claude Code review used Claude Fable 5 at max effort and returned GO with high confidence and no blocking findings. Verification included the focused package set, race tests, go vet ./cmd, and the complete go test ./cmd -count=1 suite. The review authorized only the six-file server commit; publication remains a separate gate.

Cross-repository review found no implementation changes are required in pgsty/mc, pgsty/silo-pkg, or pgsty/silo-console: the client forwards configuration text, the package repository owns no notification schema, and Console already serializes its form into the canonical connection_string or dsn_string. The public reference and compatibility documentation is updated with this record.

Release and compatibility statement

The release note must describe this as an enforced compatibility boundary:

SILO database notification targets require connection_string for PostgreSQL and dsn_string for MySQL. The pre-2020 discrete host/port/username/password/database form is not migrated. Convert or recreate such targets before switching the deployment to SILO.

Deployments already running SILO with an old-format source configuration are equally affected: after this release the server will not start until each enabled legacy database target is converted, disabled, or removed.

The issue should close only after the repair is present in a published server tag. A merged patch, a local site build, and a published release are separate completion gates.

Review record

Claude Fable 5 reviewed the first draft at xhigh effort on 2026-08-23 and returned approve with required changes. The required calibration was incorporated: startup-fatal propagation now extends through initConfigSubsystem; already-running SILO deployments are covered; the availability trade-off is explicit; canonical-string precedence, dead legacy environment variables, other ignored helper errors, and executable tests are specified.

The same model then completed a final source-backed verification pass. Final verdict: approve, with no blocking findings. It confirmed that the English and Chinese records are aligned, the requirements are implementable against the current server tree, and the acceptance criteria cover the startup, migration, parser-regression, and secrecy boundaries.

After implementation, a separate local Claude Code review using Claude Fable 5 at max effort traced the path through ExitFunc(1), inspected driver error behavior, ran the focused, race, vet, and full cmd suites, and returned GO with high confidence and no blocking findings.

10 - Preview Text, Never Execute It: SILO Console Text Preview PRD

Status: shipped in SILO Console 2.2.0 · Owner: pgsty/silo-console · Tracking: pgsty/silo#17 · Review: consensus of product, security, and frontend architecture reviews

SILO Console can preview images, PDFs, audio, and video, but not the small logs, text files, JSON documents, and XML documents that operators inspect every day. A correctly stored Content-Type does not help: these objects are classified as unsupported before the preview renderer is selected.

Restoring the old browser-native behavior would be easy. It would also be the wrong fix. An object in storage is controlled by the user who uploaded it. Loading that object as a same-origin HTML or XML document would turn a convenience feature into an execution boundary.

The accepted design therefore makes a stronger promise:

SILO previews eligible objects as bounded UTF-8 text. It never asks the browser to interpret their markup, MIME type, or file contents as a document.

This record fixes the product boundary, the resource limit, the security invariants, the implementation shape, and the evidence required before the feature can ship.

Decision

The first release will add a dedicated text preview type and a PreviewText component.

The contract is:

1. Preserve every existing image, PDF, audio, and video classification.
2. Only when the existing classifier returns none, consider a text fallback.
3. Admit the four target extensions or four exact passive text MIME types.
4. Fetch bytes through the ordinary authenticated download path, without preview=true.
5. Enforce a hard application read limit of 1 MiB.
6. Decode only strict UTF-8 and reject binary-looking content.
7. Render one React text node inside a scrollable <pre>.
8. Never use an iframe, HTML parser, XML parser, or HTML injection API.
9. Show the complete object or no object; do not show a truncated JSON or XML document.
10. Keep download available for files that are too large, invalidly encoded, or otherwise unavailable.

No Console API or S3 API change is required. The backend inline MIME allowlist is not expanded.

Current behavior

The defect is present in SILO Console v2.1.1, the version currently pinned by SILO when this design was written.

The frontend preview union contains only:

image | pdf | audio | video | none

Its extension table contains media formats, but not .log, .txt, .json, or .xml. Its MIME classifier likewise ignores text/plain, application/json, application/xml, and text/xml.

Runtime verification produced this split:

The object-detail action also uses the wrong conjunction when deciding whether Preview should be disabled. An authorized user can click Preview for an unsupported object and receive only the unavailable message; in other combinations, the UI can offer an action before the server rejects it.

The preview component still contains a generic same-origin iframe fallback. It is unreachable under the current type union, so the current defect is not an exploitable text-preview XSS. The dead branch is nevertheless hazardous: adding text to the union and letting it fall through would reactivate precisely the document-loading behavior this design rejects.

Root cause

This is contract drift across three independently evolved layers.

Classification drift

The browser code decides eligibility from filename and object metadata, but its closed type union has no text representation. Correct metadata cannot select a renderer that does not exist.

Response-policy drift

The Console server separately decides whether a response may be inline. It still treats plain text and JSON as safe passive MIME types, while XML and HTML remain attachments. That server decision is not reflected in the frontend classifier.

Renderer drift

The old generic iframe remains after the set of reachable preview types became media-only. The code therefore suggests a capability that the type system can no longer invoke.

The repair must realign the three layers without making MIME metadata a security boundary.

Why same-origin iframe preview is rejected

X-Frame-Options: SAMEORIGIN is not a sandbox. It controls who may embed a response; it does not limit what code inside a same-origin frame can do.

If uploader-controlled HTML, XHTML, SVG, or active XML were ever served as an inline same-origin document, it could act with the Console origin. An HttpOnly cookie would prevent direct cookie reads, but it would not prevent authenticated same-origin requests. A permissive or accidentally widened MIME rule would then turn stored content into stored application code.

nosniff, Content Security Policy, and Content-Disposition remain useful defense in depth, but none replaces the core invariant:

untrusted object bytes
        |
        v
strict text decoder
        |
        v
React textContent

never:
iframe / innerHTML / DOMParser / XML parser / executable document

Product contract

The feature is a read-only text viewer, not a web previewer and not an online editor.

The user should be able to:

• open a small eligible object from either the list or object-detail surface;
• read whitespace-preserving source text in the existing preview modal;
• select and copy text using browser-native behavior;
• understand whether a failure is caused by size, encoding, permission, object replacement, or network error;
• download the original bytes at any time.

The user must never be led to believe that:

• formatted JSON is the stored object;
• a partial XML document is complete;
• replacement characters are original bytes;
• an unsupported encoding has been decoded faithfully;
• an active HTML/XML document has been safely “sanitized” and executed.

Goals and non-goals

Goals

1. Preview small logs, text, JSON, and XML without a local download.
2. Keep object content inert regardless of extension, MIME, or payload.
3. Bound retained response bytes and rendered text to 1 MiB.
4. Preserve the stored text rather than silently reformatting it.
5. Keep list and detail actions consistent with permissions and type eligibility.
6. Support current object versions and explicitly selected historical versions.
7. Preserve anonymous-access and subpath-hosting behavior.
8. Ship the feature in Console first, then consume that exact Console revision in SILO.

Non-goals

• HTML or XHTML rendering.
• XML parsing, XSLT, external entities, or schema validation.
• Markdown rendering.
• JSON pretty-printing.
• YAML or CSV-specific behavior.
• Editing or saving.
• Syntax highlighting, line numbers, search, folding, ANSI rendering, or linkification.
• Head, tail, or truncated previews for large objects.
• Lossy decoding or automatic detection of GBK, UTF-16, Latin-1, or other encodings.
• A new backend text-preview endpoint.
• Changes to the existing SVG, media, PDF, download, share, or storage contracts.

An object such as notes.md may still be shown as raw text when its exact MIME type is text/plain. It does not gain Markdown semantics.

Eligibility contract

Eligibility is deliberately two-stage.

Stage 1: preserve the legacy media decision

Run the current image, PDF, audio, and video classifier unchanged. If it returns anything other than none, return that result.

This preserves historical behavior for conflicting filename and MIME combinations.

Stage 2: apply text fallback

Only after the legacy result is none:

1. Reject final extensions .html, .htm, and .xhtml.

2. Match the final filename extension case-insensitively against:

   • .log
   • .txt
   • .json
   • .xml

3. Normalize Content-Type by removing parameters, trimming whitespace, and lowercasing it.

4. Match the normalized MIME exactly against:

   • text/plain
   • application/json
   • application/xml
   • text/xml

An allowed extension or an allowed exact MIME is sufficient. Broad matches such as text/, substring tests, and application/+json are forbidden in this release.

The resulting matrix is normative:

Filename and MIME affect product eligibility only. They never select an executable rendering mode.

Resource contract

The binary limit is:

MAX_TEXT_PREVIEW_BYTES = 1,048,576

Exactly 1 MiB is eligible. 1 MiB plus one byte is not.

Known sizes

• If the selected version has a known size greater than the limit, do not request its body.
• If its known size is zero, show the empty-file state.
• If its known size is within the limit, begin a bounded request.
• An absent size is not the same as zero; it enters the bounded unknown-size path.

The current list-to-modal handoff must therefore preserve undefined rather than converting it to zero with a truthy fallback.

Bounded request

For a small or unknown size, request:

Range: bytes=0-1048576

The extra byte is an over-limit sentinel.

The client must:

1. Inspect Content-Range and Content-Length when present.
2. Read the response as a stream rather than calling response.text() or building a complete Blob.
3. Retain at most the limit plus the sentinel byte.
4. Cancel immediately when the sentinel byte is observed.
5. Enforce the same limit when the server ignores Range and returns 200.
6. Render only after end-of-stream proves that the complete object is within the limit.

An over-limit object opens an explanation state with its known size, the 1 MiB policy, and a Download action. It never shows a prefix fragment.

Request identity and cancellation

A preview request is identified by:

bucket + object name + version ID

The request must use the existing generated API client or an equivalent base-path-safe helper so that it preserves:

• same-origin credentials;
• the current Console subpath;
• version_id;
• anonymous-mode X-Anonymous: 1;
• current error handling and permission boundaries.

Close, object change, version change, bucket change, and component unmount must abort the active request and clear the old content.

Abort alone is insufficient. A generation token or invalidation flag must also prevent a response that already completed reading or decoding from updating a newer preview.

An aborted request is not an error and must not produce an error toast.

Encoding and fidelity

The first release supports strict UTF-8 only:

new TextDecoder("utf-8", { fatal: true })

Requirements:

• handle the UTF-8 BOM without displaying it;
• preserve Unicode text, emoji, tabs, LF, and CRLF;
• reject invalid UTF-8 rather than inserting replacement characters;
• reject decoded NUL characters as binary or unsupported content;
• do not guess another encoding;
• do not log or persist object text;
• always retain Download as the original-byte escape hatch.

The unsupported-encoding state should explain:

This object is not valid UTF-8 text or contains binary data. Download it to inspect the original bytes.

JSON and XML are displayed exactly as decoded source text. The first release must not run JSON.parse followed by JSON.stringify: that can alter unsafe integers, duplicate keys, whitespace, lexical forms, and the text users copy.

Safe renderer

The success state renders one text node:

<pre>{content}</pre>

The implementation must not use:

• iframe, object, or embed;
• dangerouslySetInnerHTML or innerHTML;
• DOMParser or an XML parser;
• Markdown or HTML rendering;
• an HTML data/blob URL;
• per-line or per-token spans;
• automatic links, ANSI escapes, or syntax markup.

One bounded text node keeps the DOM cost predictable and the security property inspectable.

The preformatted region uses a monospace font, preserves whitespace, defaults to no wrapping, owns both scrollbars, is keyboard focusable, and supports native selection and copy. No-wrap is intentional: it preserves aligned logs and avoids expensive layout of a single very long line.

UI states and permissions

The Preview action is enabled only when:

eligible preview type
AND object read permission
AND not a delete marker
AND not a prefix

The object-detail conjunction bug must be fixed, and list and detail surfaces must share the same eligibility function.

An eligible over-limit object still offers Preview. The modal explains why content is not loaded; disabling the button would leave the user unable to distinguish size, permission, and type failures.

The modal distinguishes:

HTTP error bodies must never be decoded and displayed as object content.

All new user-facing strings go through the existing translation layer and ship in English and Chinese together. The content region and controls must remain usable in light and dark themes and at narrow widths.

Functional and security requirements

Functional requirements

• FR1: Existing media and PDF classification remains unchanged.
• FR2: The text fallback follows the normative extension/MIME matrix.
• FR3: Eligible complete objects up to 1 MiB render as strict UTF-8 source.
• FR4: Over-limit objects render no partial content.
• FR5: Empty objects have a distinct successful empty state.
• FR6: Current and selected historical versions use the same version for metadata, size, and body.
• FR7: Anonymous access and subpath hosting retain their current request behavior.
• FR8: List and detail actions apply the same type and permission decision.
• FR9: Download, share, media, PDF, and storage behavior do not change.

Security requirements

• SR1: Object bytes can reach the DOM only through text content.
• SR2: Text Preview contains no document renderer or parser.
• SR3: At most 1 MiB plus one sentinel byte is retained.
• SR4: Closing or changing identity invalidates every previous response.
• SR5: Invalid UTF-8 and NUL content are not shown as faithful text.
• SR6: Errors, Redux, local storage, logs, and telemetry never retain preview text.
• SR7: Server authorization remains authoritative for direct requests.
• SR8: No CSP or backend inline MIME relaxation is introduced.

Implementation scope

Expected Console changes:

1. Refactor preview classification so the current media decision is preserved and text is an explicit fallback.
2. Add text to the preview type union.
3. Add a dedicated PreviewText component with streaming bounds, strict decode, request cancellation, and explicit states.
4. Route text objects explicitly to that component.
5. Remove the unreachable generic iframe fallback.
6. Fix the object-detail Preview disable expression and share eligibility logic with the list surface.
7. Preserve unknown size instead of coercing it to zero.
8. Add English and Chinese strings.
9. Add classification, component, resource, security, permission, version, and browser tests.

Expected unchanged areas:

• Console and S3 API paths;
• the backend safeMimeTypes list;
• Content Security Policy;
• object storage and metadata formats;
• image, PDF, audio, video, download, and share handlers;
• external frontend dependencies.

If a future product requires tailing, server-side transcoding, organization-wide policy, or reliable behavior through proxies that ignore Range, a dedicated server endpoint may be designed separately.

Rejected alternatives

Keep text preview disabled

Benefit: no new code or browser memory use.
Rejected because: logs and configuration objects are a routine object-storage workflow, and download-only inspection is an avoidable Console regression.

Reuse the same-origin iframe

Benefit: minimal code and browser-native presentation.
Rejected because: it turns uploader-controlled content and mutable MIME metadata into a same-origin document boundary. It also leaves resource use unbounded.

Add a backend preview API now

Benefit: central server-side limits and normalized text responses.
Rejected for the first release because: the user already has object-read permission, and the existing download endpoint provides versioning, authorization, and Range. A new API would duplicate contracts without establishing a new data-access boundary.

Show the first 1 MiB of a large object

Benefit: better large-log convenience.
Rejected because: partial JSON/XML is structurally misleading, UTF-8 boundaries need additional handling, and a single “preview” action would no longer mean complete content.

Decode invalid UTF-8 with replacement characters

Benefit: some damaged or legacy logs remain partially readable.
Rejected because: copied text would no longer faithfully represent the stored object. Lossy viewing and other encodings require a separate, explicit product mode.

Auto-format JSON

Benefit: more readable indentation.
Rejected because: parse/stringify can alter numbers, duplicate keys, lexical representation, and copied content. A future opt-in formatted view may sit beside, never replace, the raw default.

Add Monaco or another code editor

Benefit: line numbers, search, highlighting, and folding.
Rejected because: bundle, worker, CSP, and maintenance costs exceed the needs of a bounded read-only preview. A native <pre> is smaller and easier to audit.

Acceptance and test plan

Classification matrix

Automated tests must lock every normative matrix row, extension case handling, MIME parameter stripping, explicit HTML/XHTML denial, and unchanged media conflicts.

Resource tests

Cover:

• 0 bytes;
• 1 byte;
• exactly 1,048,576 bytes;
• 1,048,577 bytes;
• known over-limit size with zero body requests;
• unknown size;
• 206 with a revealing Content-Range;
• server ignores Range and returns 200;
• missing or false Content-Length;
• close and identity changes during streaming.

No case may retain or render more than the complete allowed object.

Encoding and fidelity tests

Cover UTF-8 Chinese, emoji, tabs, LF, CRLF, BOM, invalid byte sequences, NUL bytes, JSON unsafe integers, duplicate keys, original whitespace, XML declarations, DOCTYPE, CDATA, and stylesheet processing instructions.

The raw success view must preserve decoded text. Invalid and binary cases must show their dedicated state.

Security tests

Payloads containing <script>, event attributes, iframe tags, SVG handlers, XML stylesheets, external entities, and suspicious URLs must:

• appear literally in <pre>.textContent;
• create no corresponding DOM elements;
• execute no script or dialog;
• cause no object-content-originated request;
• encounter no iframe, object, embed, HTML parser, or XML parser in Text Preview.

Permission and race tests

Verify:

• no GetObject means no usable action and no retained body;
• historical versions require their corresponding permission;
• metadata and body use the same version ID;
• a late old response cannot replace a new object’s preview;
• 401, 403, 404, 416, and 5xx bodies never become preview content;
• anonymous access and Console subpaths do not regress.

Browser regression

Use a real SILO/Console test instance to inspect both English and Chinese routes, light and dark themes, and narrow and desktop widths. Media, PDF, download, share, and version workflows require smoke coverage alongside the new text states.

Delivery and completion gates

The change belongs to pgsty/silo-console, even though the user report is tracked in the SILO server repository.

Delivery is staged:

1. Merge the focused Console source and test change.
2. Pass TypeScript checking, production build, automated matrices, and real-browser security regression.
3. Update Console release notes and regenerate the actual embedded web assets.
4. Publish a Console version; a minor release is appropriate for the new visible capability.
5. Update SILO’s github.com/minio/console => github.com/pgsty/silo-console replacement to the exact new pseudo-version.
6. Build a SILO candidate from that exact dependency and repeat integration checks.
7. Publish the SILO binary and image, naming the first version that contains the feature.

These are separate states:

Issue #17 should not be described as fixed for users merely because a local preview or Console source PR exists.

Trade-off summary

The accepted design favors:

• explicit scope over a generic browser viewer;
• complete small files over partial large files;
• source fidelity over automatic formatting;
• strict UTF-8 over silent lossy decoding;
• one inert text node over a full editor;
• the existing download API over a new backend contract;
• a verifiable security invariant over convenient same-origin rendering.

The cost is real: large logs and legacy encodings still require download, and the first release has no search, line numbers, wrapping toggle, or highlighting. Those omissions are deliberate. They make the feature small enough to audit and strong enough to trust.

Review record

The design was independently reviewed from three perspectives:

• product scope, delivery, and acceptance;
• security and frontend architecture;
• compatibility and current-source verification.

The reviewers initially differed on MIME-only eligibility and lossy UTF-8 fallback. After cross-review they reached a single contract:

• existing media classification wins;
• text fallback accepts the four target extensions or four exact normalized MIME types;
• HTML/XHTML extensions are explicitly excluded;
• strict UTF-8 and NUL rejection are required;
• lossy viewing is deferred to a separate proposal.

No unresolved design question remains. Implementation may proceed against this record.

11 - Go 1.27 TLS Defaults and OIDC Discovery Failure Modes

After SILO’s toolchain moved to Go 1.27, the Server TLS repair 48e184652 (“fix(tls): honor Go key exchange defaults across transports”) removed its explicit curve overrides. This page records the TLS changes and the diagnostic method for OIDC discovery failures that came out of issue #154, and the operational facts an administrator needs when identity goes missing at startup.

Release boundary, 2026-09-16: Server 20260903 already uses Go 1.27.1, but does not contain 48e184652. That later TLS repair is on main; upgrading the compiler and adopting this repair are separate changes.

Evidence class, stated up front. Every mechanism below is verified by synthetic experiments: ClientHello captures, fresh-process CA probes, and fixture reproductions. The #154 customer’s discovery URL and ingress configuration were never obtained, so no root cause is claimed for that deployment — two locally verified mechanisms could each produce the reported symptom, and either the ingress rejecting the new handshake, or a proxy rejecting the changed User-Agent, remains plausible. #154 stays open for an affected-environment retest.

What Go 1.27 changed

• Explicit curve preferences now override the ML-KEM compat switches. GODEBUG=tlsmlkem=0 (and tlssecpmlkem=0) only remove post-quantum hybrids from the default curve set. An application that configures CurvePreferences explicitly keeps ML-KEM in whatever list it names — a deliberate Go 1.27 change. SILO had eight TLS configuration points setting an explicit list including X25519MLKEM768; the fix removes all eight assignments and retires the helper, so these Server configuration points follow Go’s defaults and the compat switches work again. The stack review found pkg, mcli, and Console clients already used defaults; Console’s HTTPS listener retains its separate P-256-only policy.
• ClientHello now offers ML-DSA signature algorithms (identifiers 0x0904–0x0906). ML-DSA is a signature scheme and distinct from ML-KEM: disabling hybrid key exchange does not disable the ML-DSA offer, and an ingress that rejects ML-DSA is not fixed by any ML-KEM switch.
• The ClientHello grew. Measured on the same source and dependencies: Go 1.26.5 default 1497 bytes; Go 1.27.1 default 1509 bytes; the old explicit list under tlsmlkem=0 produced a 275-byte hello with no ML-KEM, while Go 1.27.1 with an explicit list still produced 1509 bytes containing ML-KEM. Rebuilding with a different compiler alone changed the handshake.
• macOS root-CA behavior flips with the module’s Go directive. A fresh process honoring SSL_CERT_FILE/SSL_CERT_DIR instead of the Keychain is governed by the x509sslcertoverrideplatform GODEBUG default, which follows the main module’s go directive: go 1.26 modules ignore those variables on macOS (platform store wins), go 1.27 modules honor them — and the consuming application’s directive wins even when a library module is older. Operators on macOS should know that setting either variable replaces Keychain trust wholesale with the file/directory given; a stale or incomplete path then breaks chains the Keychain would have accepted, and unsetting restores the Keychain.
• Not everything changed. TLS versions, cipher suites, certificate and hostname verification, proxy handling, and HTTP/2 selection are unaffected. The standard-library drain cap (256 KiB / 50 ms) and other audited Go 1.27 changes showed no SILO dependency. Go 1.27 binaries require macOS 13 or newer. Downgrading is not a supported path: the module graph requires Go ≥ 1.27.1 across Server, Console, and mc.

Why the OIDC-only patch was withdrawn

The investigation first produced a minimal candidate: clear CurvePreferences on the OIDC discovery transport only. It was deliberately not shipped. The same transport serves identity plugins, notification and lambda reachability checks, audit webhooks, and S3 cloud-backend tiers — fixing two OIDC call sites would have left every other consumer on the defective explicit list. The merged repair removes the explicit curves at all eight Server configuration points so the compat switches apply there, keeps certificate verification strict, and adds no protocol downgrade or automatic fallback. The archived one-transport patch must not be reapplied on top of the merged fix.

Diagnosing a discovery failure by phase

The startup chain is: server start → identity system init → fetch .well-known/openid-configuration (discovery) → fetch the jwks_uri keys → IAM store ready → Console initializes. Console’s own OIDC configuration dialog validates through the same server-side transport. A failure anywhere in the chain leaves IAM offline; a successful discovery does not clear the JWKS fetch, and a 503 on JWKS blocks IAM just as hard.

Discriminate by where the connection dies:

• Reset right after the TLS ClientHello (tls_start then reset): suspect the ingress’s ClientHello handling — proxy CONNECT rules, TLS terminators, or anything keyed on hello size or contents. This is where the Go 1.27 changes land.
• Reset after TLS completes (wrote_request then reset): the TLS layer is fine; look at HTTP-layer policy — WAF rules, User-Agent allowlists (the server’s UA changed from MinIO to Silo with the rebrand), routing. Replacing certificates or key exchange here has no targeted effect.
• x509 errors: compare the chain actually received, the SNI, and the trust store the process resolves (see the macOS section above).
• Always test from the same network position as the failing process — a fresh container does not inherit the failing container’s network namespace, and same-IP/same-proxy controls come first.

The health endpoint that tells the truth

/minio/health/live and /minio/health/ready both stay 200 while IAM is offline — readiness as deployed does not cover the identity system. The endpoint that reports it is /minio/health/cluster, which checks identity initialization and returns 503 with the X-Minio-Server-Status: iam-offline marker. Monitoring that should catch a broken IdP integration should probe the cluster health, plus one authenticated operation.

Recovery is automatic: identity initialization retries at randomized 0–3 s intervals, and a recovered IdP brings IAM back without a restart (observed sub-second to ~1.4 s locally). Retrying cannot fix a persistent incompatibility — a hello the ingress rejects stays rejected.

Transport facts worth knowing

The discovery/JWKS client builds its own transport: HTTP/2 disabled (no ALPN, HTTP/1.1), proxies taken only from HTTPS_PROXY/NO_PROXY (uppercase preferred; ALL_PROXY unused), a 30 s DNS cache, dialing that walks addresses in order without shuffling, and timeouts of 5 s per TCP dial, 10 s for the TLS handshake, and 1 min to response headers. There is no total timeout on the discovery or JWKS fetch itself — a slow IdP can hold startup indefinitely; tightening that is a known, separately-sized follow-up.

Attribution

This record distills the issue #154 investigation and the September Go 1.27 stack review; the reproduction artifacts and the full evidence chain are retained outside the documentation tree. The supported statement is: the merged fix restores Go key-exchange defaults at the eight affected Server configuration points, verified with synthetic negative controls — it does not claim to have diagnosed any specific hidden deployment, and #154 remains open pending a retest in the affected environment.

12 - No I/O Before Auth, No Privilege From Headers

This record describes the CORS hot-path and replication-request trust repair merged into SILO as PR #101 (938603458 through 04b097fd9).

Status on 2026-09-03: PR #101 merged into main on 2026-09-01 with four follow-up commits: per-entry Snowball trust isolation (ff44527a3), request defaults preserved across Snowball workers (ab3ae99ca), and replication validity probes that verify the replication permissions (c9ad74673) under the rule prefix (5db7be4ee). Implementation, focused and race tests, the complete server package suite, object-lock tests, vet, build, two rounds of Fable 5 design review, repeated Opus 5 adversarial acceptance, and a real local TLS two-site replication run are complete. The pre-release cleanup kept the resident-only lookup with its fail-closed startup and load-failure states, dropped only the internal-namespace special case, and made the header-stripped request clone share the original request trailer so streaming-checksum uploads keep working for untrusted requests. Tag, package, image, deployment, and production verification remain separate gates.
Scope: HTTP request interpretation before and inside the S3 handlers. No S3 wire field, object format, bucket metadata format, replication protocol, encryption format, or client command changes.
Security properties: pre-authentication CORS processing performs no object-layer I/O; a header never grants replication semantics by itself; SSE-C ciphertext paths and replica-only metadata require both authentication and the corresponding replication permission.

Too Long; Didn’t Read (TL;DR)

Two bugs looked unrelated:

1. an Origin header made the outermost CORS middleware treat the first URL segment as a bucket and synchronously load its metadata before authentication;
2. X-Minio-Source-Replication-Request made downstream code believe a request was internal replication merely because the header existed.

They shared the same design failure: untrusted request shape was allowed to acquire expensive or privileged internal meaning before an authorization boundary.

The repair establishes two invariants:

before authentication:  parse cheaply; never load bucket metadata
after authentication:   derive one trust decision; downstream code consumes it

For CORS, the outer middleware now reads only metadata already resident in memory. For replication, handlers authenticate the original signed request first, authorize the appropriate replication action, and then attach a private trust decision to the request context. Untrusted internal headers are stripped only after signature verification. The context decision—not header removal—is the authority used by option builders, encryption paths, object lock, event generation, and metadata persistence.

Failure A: pre-authentication CORS amplification

corsHandler wraps the complete server router. Any request carrying Origin reaches it before S3 authentication, request validity checks, and the normal API limiter.

The per-bucket CORS implementation originally called the normal bucket metadata getter:

Origin-bearing request
  -> first URL segment becomes "bucket"
  -> GetCorsConfig
  -> GetConfig cache miss
  -> read .metadata.bin
  -> probe ten legacy config paths
  -> cache a default BucketMetadata record

When .metadata.bin did not exist, the loader intentionally searched legacy configuration files. With none found, it returned a valid empty metadata record rather than NoSuchBucket. The generic getter then inserted that record into metadataMap.

An unauthenticated client could therefore vary otherwise plausible names and obtain two effects per distinct value:

• repeated erasure/object metadata reads before the normal request limiter;
• growth of the in-memory bucket metadata map.

Name validation alone cannot repair this. An attacker can generate an effectively unbounded sequence of syntactically valid, nonexistent bucket names. Distributed deployments eventually prune stale map entries during the 15-minute metadata refresh; single-node deployments do not start that refresh loop, so their synthetic entries persist until restart.

Failure B: a marker header became authority

SILO and its MinIO-compatible clients use internal headers to preserve source state during replication. The most important marker is:

X-Minio-Source-Replication-Request: true

Before this repair, several paths treated header presence—or its raw string value—as proof that the request was a replication request. That affected more than metadata extraction:

• GET of an SSE-C object could set NoDecryption and return ciphertext without the customer key to a caller holding only ordinary read permission;
• source ETag and modification time could replace server-generated values;
• source tagging, retention, and legal-hold timestamps could enter last-writer-wins comparisons;
• a past object-lock retention date could be accepted through a raw marker check;
• delete-marker identity and modification time could be supplied by the caller;
• successful object events could be suppressed;
• multipart actual size and encrypted checksum metadata could be injected at completion;
• X-Amz-Replication-Status could be persisted from ordinary PUT, COPY, or POST-policy metadata extraction.

The earlier CVE-2026-34204 repair correctly stopped ordinary PUT and COPY from importing the replication SSE metadata that could make objects unreadable. It did not yet provide one authority shared by every reader of the marker, source fields, event state, object-lock exceptions, or multipart completion metadata.

Selected design

One exact marker, two trust levels

The marker is accepted only when it appears exactly once and its value is exactly lowercase true. Duplicate values, mixed case, and any other value are untrusted.

The handler then derives two related decisions:

The split is required by the real wire protocol. Not every legitimate replication request repeats X-Amz-Replication-Status: REPLICA.

The receiver follows this matrix:

The explicit 403 for an unauthorized REPLICA request prevents a claimed replica write from being silently downgraded into a new ordinary object that may be replicated again.

Authenticate the original, then sanitize

SigV4 signs request headers. Removing an internal header before authentication would change the canonical request and turn a valid signature into SignatureDoesNotMatch.

The ordering is therefore mandatory:

original request
  -> existing signature/authentication path
  -> ordinary S3 action authorization
  -> replication action authorization
  -> derive trusted / replicaTrusted
  -> bind decision to request context
  -> clone and strip untrusted internal fields
  -> option parsing, encryption, object lock, storage, events

The audit logger retains the original request. The effective request clone retains public S3, SSE, checksum, object-lock, copy-source, proxy, and replication-validity headers. It strips only internal source/replication controls, including source ETag/MTime/delete-marker/timestamps, replication SSE state, actual object size, encrypted checksum transfer, and the request use of X-Amz-Replication-Status.

Header stripping is defense in depth. All privileged consumers use the private context decision or an explicit Boolean; they do not infer trust by looking at the clone.

Replica status is not generic user metadata

X-Amz-Replication-Status is an S3 response header that MinIO-compatible servers also use as an internal request control. It no longer belongs to the generic supported-request-metadata list.

Ordinary PUT, COPY, multipart initiation, Snowball/PAX extraction, and POST policy cannot persist it merely by submitting the field. The receiver sets REPLICA explicitly only in a replicaTrusted branch.

This closes a subtle POST-policy path: a form field could previously store REPLICA, causing the resulting object to evade normal replication scheduling even though the POST principal never held replication permission.

Object lock receives an explicit decision

The object-lock parser used to accept past retention dates when the raw marker header was present. That package now receives allowPastRetainDate explicitly from replicaTrusted state.

The surrounding handler also uses the same decision when deciding whether an existing compliance/legal-hold version may be overwritten by a replica. This removes an internal-header dependency from the reusable object-lock package.

Actual replication wire matrix

The design was checked against the silo-go v7.3.1 emitter selected by the server’s go.mod, not inferred from comments or upstream documentation.

s3:ReplicateDelete is the trust gate, not the receiver’s only permission. For compatibility with deployed target policies, a trusted replication delete also requires s3:DeleteObject; an explicit deny on s3:DeleteObjectVersion still blocks a named-version purge. Ordinary clients do not use this compatibility path: an explicit UUID or versionId=null requires an allow for s3:DeleteObjectVersion.

Requiring REPLICA for every trusted operation would break PutPart, multipart completion, and batch replication. Trusting every marker would recreate the vulnerability. Stored multipart provenance bridges the two requirements for encrypted raw parts.

CORS resident-only state machine

The outer CORS middleware must remain cheaper than the request it is about to route. It now calls a dedicated resident-only getter that takes one read lock and examines only in-memory state.

The lookup consults the resident map and a bounded set of real buckets whose metadata failed to load at startup or during a refresh. That set is filled only from disk-derived bucket lists, never from a client path, never records a bucket that is already resident, and a successful load, Set, bucket removal, stale-bucket reconciliation, and subsystem reset clear it. Both non-resident states fail closed: a presigned URL is authenticated by its own signature, so the bucket’s CORS document is the only origin boundary a browser enforces for it, and answering with the global policy would let a leaked URL be used from any origin. The internal .minio.sys namespace no longer has a special case; like any reserved or invalid name it is not a bucket, gets the global fallback, and is rejected downstream.

Alternatives rejected

Implementation boundary

The selected change is intentionally layered:

1. a small request-trust module defines exact marker parsing, replication authorization, private context state, and the post-authentication effective request;
2. object option builders parse source fields only when their caller provides trusted state;
3. DecryptObjectInfo, event request parameters, multipart completion, delete options, and object lock consume the same decision;
4. handlers calculate trust immediately after their existing authentication path;
5. multipart part handling combines current-request trust with stored MPU replica provenance;
6. generic metadata extraction does not accept replica status;
7. CORS middleware uses a separate resident-only metadata accessor and never calls the load-on-miss getter.

No object-layer API needs to infer HTTP trust. Programmatic internal callers that construct ObjectOptions{ReplicationRequest: true} remain unchanged.

Verification and adversarial review

Regression coverage includes:

• hundreds of distinct valid missing bucket names, both actual and preflight CORS requests, with zero metadata reads and no map growth;
• Console, reserved, invalid, startup, internal namespace, and invalid stored CORS paths;
• least-privilege SSE-C GET, HEAD, and GetObjectAttributes callers with correct, missing, wrong-case, and unauthorized markers;
• marker-only batch-style PUT preserving source ETag/MTime only with s3:ReplicateObject;
• unauthorized REPLICA PUT and DELETE returning 403;
• POST policy unable to forge replica status;
• object-lock past-date parsing with and without replica trust;
• marker-only CopyObject with SSE-C source headers copying plaintext rather than ciphertext;
• fake marker on an ordinary SSE-C MPU failing instead of storing raw bytes;
• a real in-process SSE-C multipart replication chain: encrypted source, raw ciphertext part, trusted replica initiation, marker-only PutPart and Complete, and exact plaintext recovery with the original key.

The final local tree passed focused and race tests, the complete cmd suite, object-lock tests, vet, build, and diff checks.

A separate black-box run started two TLS-enabled SILO instances built from the candidate and enabled real site replication. It verified:

• an SSE-C 4 KiB object;
• an SSE-C 12 MiB, three-part multipart object;
• an SSE-C CopyObject result;
• a replicated delete marker.

Source and target ETag, size, version ID, SSE-C key MD5, decrypted SHA-256, and delete-marker version ID matched; targets reported REPLICA.

Two Fable 5 review rounds first corrected the trust model for marker-only batch and multipart calls, then audited the implementation. A final independent Claude Code Opus 5 review reported GO, with no P0/P1 findings, and independently reran build, vet, race, object-lock, and full cmd tests.

Compatibility and operations

• Ordinary clients: no request change. Untrusted internal headers are ignored instead of acquiring internal semantics.
• Unauthorized claimed replica writes: requests carrying X-Amz-Replication-Status: REPLICA now return 403 where some multipart subpaths previously lacked a uniform check.
• Batch replication: destination credentials must include s3:ReplicateObject, as documented in the batch replication requirements. Without it, the receiver processes marker-only writes as ordinary writes and does not preserve source ETag/MTime.
• SSE-C: ordinary reads still require the customer key. Authorized replica reads may use the raw ciphertext path needed to preserve encrypted bytes.
• Events: only trusted replication suppresses replica creation/access events; a forged marker no longer silences them.
• Object lock: replica exceptions are permission-derived rather than header-derived.
• Performance: CORS removes pre-authentication backend work. Trusted writes add policy checks already required by the replication contract; no additional object pass is introduced.
• Rolling upgrade: wire and storage formats are unchanged. New receivers enforce the trust boundary; old receivers remain vulnerable to the old header semantics until upgraded. Per-bucket CORS behavior can therefore differ by node during the rolling window.
• Rollback: data written by the repaired version remains readable by the previous version, but rollback reopens both trust defects and restores pre-authentication metadata loads.

Residual risks and follow-ups

• 2026-09-09 replication reliability follow-up: Delete completion, MRF visibility, and resync cancellation records the reproductions, minimal fixes, Fable review, and PR #162 validation for #153, #152, and #137. It addresses reliability after trusted requests enter the replication pipeline, preserving this page’s authorization boundary.

• Emit a rate-limited diagnostic when a marker-bearing request lacks replication permission; the safe ordinary fallback is otherwise easy to misdiagnose as an ETag/MTime mismatch.

• Replication validity probes now verify the replication permissions the target credentials need and place the synthetic validation key under the rule prefix (c9ad74673, 5db7be4ee).

• This review covers the named source/replication headers. Other future internal controls must still answer the same question: which authenticated decision allowed this client value to acquire internal meaning?

Conclusion

An internal-looking header is still client input. A bucket-shaped URL segment is still attacker input. The durable repair is to stop either one from becoming authority by accident:

Before authentication, do no backend work. After authentication, derive trust once and pass the decision—not the claim—downstream.

That rule is broader than CORS or replication. It is the boundary future SILO handlers should preserve whenever inexpensive public request syntax meets expensive or privileged internal state.

13 - One Endpoint, Two Privileges: Separating User and Group Status

This document records the discussion, repair, and final authorization design for upstream issue minio/minio#21478 and SILO PR #73.

Status on 2026-08-26: SILO PR #73 was merged as 2e2377d1c, preserving the signed-off repair commit 58735ee38. All eight reported checks passed. Upstream issue #21478 and PR #21482 remain open, but minio/minio is archived and read-only, so no further issue comment or merge can be made there.
Group follow-up on 2026-08-28: final release review found the same fixed-action defect in set-group-status. Signed-off server commit 229fe2b3c now selects admin:EnableGroup or admin:DisableGroup from the requested target state and adds a real four-way IAM authorization test. Local verification and independent review are complete; it was merged into main on 2026-08-29, and tag and delivery remain pending.
Scope: authorize enabling and disabling a user with their respective existing Admin Actions. Do not change the route, status values, account storage, replication record, or client API.
Security property: possessing admin:DisableUser must not grant the ability to enable an account, and possessing admin:EnableUser must not grant the ability to disable one.
Release boundary: merge, tag, release package, container image, deployment, and production verification remain separate gates.

Too Long; Didn’t Read (TL;DR)

SILO exposes both admin:EnableUser and admin:DisableUser, but the shared set-user-status handler historically authorized every request with admin:EnableUser. A policy that granted only admin:DisableUser therefore could not disable an account. The workaround was to grant admin:EnableUser as well, which destroyed the least-privilege boundary that the two action names promised.

The selected repair derives exactly one required action from the requested target state before authorization:

The handler then calls validateAdminReq once. A four-way IAM test proves both positive operations and both denied cross-action operations. This is intentionally stricter than preserving the accidental historical behavior in which an Enable-only policy could also disable users.

The same rule now applies to group status:

Before the follow-up, an EnableGroup-only principal could disable a group, while a DisableGroup-only principal received AccessDenied for that exact operation. The group repair uses the same one-selector, one-authorization design rather than treating the two actions as aliases.

The reported defect

The Admin API uses one route for both state transitions:

PUT /minio/admin/v3/set-user-status
    ?accessKey=<target>
    &status=enabled|disabled

Before the repair, the handler checked one fixed action before reading the requested status:

objectAPI, creds := validateAdminReq(ctx, w, r, policy.EnableUserAdminAction)

The later call to SetUserStatus correctly received either enabled or disabled, but authorization had already treated both as Enable operations. admin:DisableUser existed in the policy vocabulary and documentation while being ineffective for this endpoint on its own.

Issue #21478 supplied the practical counterexample: an operator wanted a policy that could disable accounts during an incident without being able to restore them. A policy containing admin:DisableUser received AccessDenied; adding admin:EnableUser made the request work, but also gave the operator the more powerful recovery transition that the policy intentionally withheld.

This is not a missing convenience permission. It is a mismatch between the policy model and the enforcement point:

policy says:     DisableUser only
request says:    target state = disabled
handler checked: EnableUser
result:          legitimate disable denied
workaround:      grant an unwanted enable capability

Why two actions must mean two capabilities

An account state transition has direction. Disabling is commonly delegated to incident responders, fraud controls, compliance automation, or a break-glass process. Enabling restores access and may require a separate approver.

If either action authorizes both transitions, a policy author cannot express that separation. The server would publish two names while enforcing one combined capability. The design contract is therefore strict:

The built-in consoleAdmin policy grants admin:*, so full administrators retain both operations. The compatibility impact is limited to custom restricted policies that relied on the old accidental behavior.

The public PBAC reference now states the same contract for admin:EnableUser and admin:DisableUser.

Design goals and non-goals

Goals

1. Make both existing Admin Actions enforceable according to their names.
2. Preserve least privilege in both directions.
3. Perform one authorization decision and write at most one authorization error.
4. Preserve the route, request values, response format, self-mutation guard, IAM storage call, and site-replication hook.
5. Encode the contract in tests that fail if the two permissions are broadened or swapped again.

Non-goals

• split the endpoint into separate enable and disable routes;
• add a new combined action or change policy syntax;
• change user status persistence or replication;
• redesign Console permissions;
• infer release, image, deployment, or production delivery from a source merge.

Alternatives considered

Keep checking admin:EnableUser for both states

This preserves behavior but leaves admin:DisableUser unusable and forces over-privileged policies. It is the defect, not a compatibility contract worth retaining.

Require both actions for either transition

This makes the two labels decorative and prevents delegated disable-only operation. It is stricter in quantity but weaker in expressiveness and least privilege.

Try Enable authorization, then retry Disable authorization

Upstream PR #21482 attempted this shape for a disabled request. It first called validateAdminReq with EnableUser, then called it again with DisableUser if the first result was nil.

That helper has an important contract: when it returns a nil object layer, it has already written an error response. A Disable-only request can therefore commit a 403 response before the second authorization succeeds and the handler proceeds to mutate account state. Authorization fallback must never continue after an error response has been committed.

Accept either Enable or Disable for a disabled request

validateAdminReq already accepts multiple actions and succeeds if any one is allowed, so compatibility behavior could be implemented safely with one variadic call. That would let Disable-only policies work while preserving the historical ability of Enable-only policies to disable.

SILO rejected this option because the historical ability was the enforcement bug. It would solve the reporter’s positive case but retain a cross-action privilege that contradicts the two-action model. Operators who want both transitions can grant both actions explicitly.

Validate the status before authenticating

Rejecting unknown status values first would change error precedence: a caller that previously had to pass the Enable authorization gate could now receive a validation result before authorization. The repair does not need that broader behavioral change.

Unknown values therefore retain admin:EnableUser as the authorization default. Valid disabled is the only value that selects admin:DisableUser; the existing IAM layer remains responsible for rejecting invalid status values after authorization.

The selected implementation

The repair adds a pure selector:

func setUserStatusAdminAction(status string) policy.AdminAction {
    if madmin.AccountStatus(status) == madmin.AccountDisabled {
        return policy.DisableUserAdminAction
    }
    return policy.EnableUserAdminAction
}

The handler reads the route variables, selects the action, and authorizes exactly once:

vars := mux.Vars(r)
accessKey := vars["accessKey"]
status := vars["status"]

objectAPI, creds := validateAdminReq(ctx, w, r, setUserStatusAdminAction(status))
if objectAPI == nil {
    return
}

Everything after the gate remains unchanged:

• a caller still cannot enable or disable its own account;
• globalIAMSys.SetUserStatus validates and persists the requested status;
• site replication records the same status and timestamp;
• response and audit behavior use the existing path.

The selector depends only on the requested target state. It does not load the current user, infer a transition from stored state, or make authorization depend on whether the target exists. This keeps authorization deterministic and avoids a read-before-authentication dependency.

Why the repair is safe

The correctness argument consists of five invariants:

1. Every valid status maps to exactly one Admin Action.
2. validateAdminReq is invoked once, so a failed authorization cannot be followed by mutation.
3. The mutation call is reachable only after the selected action succeeds.
4. Invalid status values preserve the old Enable authorization boundary and are still rejected by the existing status-validation path.
5. No storage, replication, wire, or client contract changes; only the permission required to reach the existing mutation changes.

The change is a deliberate authorization tightening for Enable-only custom policies that used the disable operation. That tightening is the mechanism that makes admin:DisableUser a real independent capability.

Test design

Pure action mapping

The unit test fixes three selector cases:

Four-way IAM authorization matrix

The integration test creates separate users and policies, then exercises the real Admin API:

1. a Disable-only client successfully disables a target;
2. the same client receives AccessDenied when enabling it;
3. an Enable-only client successfully enables the target;
4. the same client receives AccessDenied when disabling it.

Positive assertions alone would not prove least privilege: both policies could accidentally authorize both states and still pass. The two negative cross-action assertions are the security regression tests.

The test removes every temporary user and policy after execution. It runs inside the existing IAM server suite, so it covers request signing, policy attachment, handler authorization, persistence, and Admin-client error decoding rather than testing only the helper.

Repair and verification record

The server checkout originally contained unrelated dependency, generated-credit, checksum-test, and security-document changes, while local main was behind the remote. The two user-status files were isolated into a clean worktree based on current origin/main; no unrelated file entered the repair commit.

Local verification passed:

go test ./cmd -run '^TestSetUserStatusAdminAction$' -count=1
go test ./cmd -run '^TestIAMInternalIDPServerSuite$' -count=1
git diff --check

The signed-off commit 58735ee38 was pushed in PR #73. Its eight remote checks all passed:

• DCO sign-off;
• format, build, and vet;
• lint and generated files;
• cmd/ tests;
• internal/ tests;
• race detector and S3 Select;
• cross compilation;
• vulnerability analysis.

The PR was merged with the repository’s normal merge strategy as 2e2377d1c. Local main was then fast-forwarded only after the two original working files were byte-for-byte and patch-ID identical to the merged result. The unrelated local changes remained intact, and the temporary worktree and task branch were removed after the code became recoverable from main and PR #73.

Least-privilege policy examples

Disable-only operator

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "admin:DisableUser",
        "admin:GetUser"
      ]
    }
  ]
}

This principal can inspect and disable another user, but cannot enable it.

Enable-only operator

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "admin:EnableUser",
        "admin:GetUser"
      ]
    }
  ]
}

This principal can inspect and enable another user, but cannot disable it. Grant both actions explicitly to roles responsible for the complete account lifecycle.

Group-status follow-up

The group endpoint has the same shape as the user endpoint:

PUT /minio/admin/v3/set-group-status
    ?group=<target>
    &status=enabled|disabled

It also publishes two existing actions, admin:EnableGroup and admin:DisableGroup. The inherited handler nevertheless authorized every request with EnableGroup before reading status. This was not merely a dead permission: it reversed least privilege in both directions. The wrong principal could disable a group, and the intended disable-only principal could not.

The follow-up adds setGroupStatusAdminAction, deliberately matching setUserStatusAdminAction:

func setGroupStatusAdminAction(status string) policy.AdminAction {
    if madmin.GroupStatus(status) == madmin.GroupDisabled {
        return policy.DisableGroupAdminAction
    }
    return policy.EnableGroupAdminAction
}

The integration test creates separate EnableGroup-only and DisableGroup-only administrators and a real target group. It proves:

1. DisableGroup-only can disable;
2. DisableGroup-only cannot enable;
3. EnableGroup-only can enable;
4. EnableGroup-only cannot disable.

The suite exercises signed Admin requests, policy attachment, handler authorization, IAM mutation, response decoding, and cleanup. Invalid status still selects the legacy Enable action before the existing validation error, so the change does not expose a new pre-authentication oracle. The successful site-replication hook remains after mutation and is not called for denied requests.

This follow-up changes no user behavior and introduces no new policy action. It makes the two already documented group actions enforce the same state-specific contract as their user counterparts.

Compatibility and migration

No client or API migration is required. The endpoint, query parameters, status strings, success response, and Admin-client method are unchanged.

Policy review is required for restricted administrative roles:

• a role that should only disable users needs admin:DisableUser;
• a role that should only enable users needs admin:EnableUser;
• a role that must do both needs both actions;
• consoleAdmin and other admin:* policies are unaffected;
• a legacy custom policy containing only admin:EnableUser can no longer use that permission to disable users and must add admin:DisableUser if both operations are intended.

The equivalent rules now apply to group-management roles:

• a role that should only disable groups needs admin:DisableGroup;
• a role that should only enable groups needs admin:EnableGroup;
• a role that must do both needs both actions;
• a legacy EnableGroup-only role can no longer disable groups.

This is a source-level compatibility change in authorization behavior, not a wire-protocol break.

Upstream disposition

As of this record, upstream issue #21478 and PR #21482 are still displayed as open. The upstream repository is archived and read-only. An attempt to leave the single-authorization analysis on the PR was rejected by GitHub because archived, locked discussions cannot accept comments.

The upstream artifacts remain useful provenance but are no longer an actionable delivery path. SILO owns its implemented semantics, tests, merge, release note, and eventual production verification.

Delivery state

Conclusion

The repairs make the authorization model tell the truth. Enabling and disabling users or groups are opposite state transitions with different operational risk, and SILO already exposes different policy actions for each direction. Each handler must therefore select the action from the requested target state and authorize once before mutation.

The code change is small because the design boundary is clear. The durable result is larger: an explicit permission matrix, rejected compatibility alternatives, an invalid-input rule, a four-way integration test, a clean merge record, migration guidance, and an honest release boundary.

14 - Config Environment Files Are Not Shell Scripts

This record defines the startup contract for MINIO_CONFIG_ENV_FILE and explains the compatibility repair committed in SILO as 2aea7fe9c.

Status on 2026-08-28: implementation, focused tests, the complete cmd and internal suites, tagged tests, race tests, vet, lint, generated-file checks, rebrand guards, build, and an independent local Fable Max review are complete. The commit was merged into main on 2026-08-29 as 2aea7fe9c; tag, package, image, deployment, and production verification remain separate gates.
Scope: environment-file parsing and named-target discovery only. No configuration key, subsystem, value, precedence, storage format, or client API changes.
Compatibility rule: the file is a SILO input format. Supporting an optional export prefix does not make it a POSIX shell program.

Too Long; Didn’t Read (TL;DR)

SILO can load startup variables from a file:

export MINIO_CONFIG_ENV_FILE=/etc/default/silo
silo server /data

The parser accepts assignments such as:

MINIO_ROOT_USER = silo-admin
MINIO_ROOT_PASSWORD = "  significant surrounding spaces  "
MINIO_NOTIFY_WEBHOOK_ENABLE_my-hook = off
MINIO_NOTIFY_WEBHOOK_ENDPOINT_my-hook = https://events.example.com/minio

The last two names are important. Multi-target configuration appends the target name verbatim after an underscore. The configuration subsystem does not restrict a target to a shell identifier; names containing -, ., :, digits, or printable Unicode can be discovered and resolved exactly.

A hardening change accidentally validated every key as [A-Za-z_][A-Za-z0-9_]*. It made my-hook invalid and stopped the server during restart even though the previous loader and the configuration target model accepted it. The repair validates what SILO actually needs instead:

• the name is non-empty, valid UTF-8, and made of visible non-whitespace characters;
• = and NUL are not allowed in a name;
• NUL is not allowed in a value;
• invalid input reports file and line without reporting the value;
• the complete file is parsed before any assignment is applied.

Why the regression was real

The environment-file loader calls os.Setenv after parsing. An operating-system environment is a list of strings, not a shell variable namespace. Shell assignment syntax is narrower because the shell must tokenize and expand variable names in its own language.

Named SILO configuration targets are built differently:

MINIO_<SUBSYSTEM>_<PARAMETER>_<target>

For example:

MINIO_NOTIFY_WEBHOOK_ENABLE_my-hook
MINIO_NOTIFY_WEBHOOK_ENDPOINT_my-hook

Target discovery lists variables by the fixed parameter prefix and treats the remaining suffix as the target name. Target lookup reconstructs the same name without uppercasing or sanitizing that suffix. Rejecting - in the file parser therefore broke a valid discover-to-resolve path; it did not protect a shell evaluation path because no shell evaluates the file.

The failure is operationally sharp. MINIO_CONFIG_ENV_FILE is loaded only at startup. A server can continue running with an old process environment, then fail on its next restart after the file or binary changes. Startup must fail on malformed input, but it must not invent a narrower target grammar than the configuration system.

The file grammar

Lines and comments

• blank lines are ignored;
• a line whose first non-whitespace character is # is ignored;
• an optional standalone export followed by whitespace is removed;
• exportFOO=value remains the key exportFOO; it is not mistaken for the prefix;
• the first = separates key and value, so additional = characters remain part of the value.

The file is not a shell. It does not perform variable expansion, command substitution, backslash processing, or inline-comment interpretation.

Keys

Surrounding whitespace around the key is removed. The remaining key must:

1. be non-empty valid UTF-8;
2. contain only Unicode graphic characters;
3. contain no whitespace, =, NUL, control, or invisible format characters.

This preserves OS-compatible names and multi-target suffixes while rejecting visually empty or structurally ambiguous keys. A key beginning with a digit or punctuation is accepted by the parser; SILO still reads only the exact names used by its configuration and runtime components.

Values and quoting

Unquoted values are trimmed. To retain leading or trailing spaces, quote the complete value with matching single or double quotes:

PLAIN = value
SPACED = "  value with significant spaces  "
TOKEN = scheme://user:password@example.com?a=b
EMPTY =

The parser removes one matching outer quote pair. It does not interpret escapes inside the quoted value. NUL is always rejected because it cannot be represented in an environment entry.

Failure and secrecy contract

Syntax errors stop startup. Diagnostics include the file path, line number, and the invalid key or error class, but never the value. A password on a malformed line must not be copied into logs.

Parsing is all-or-nothing: a syntax error returns no entries, and assignment starts only after the complete file has parsed. If the operating system rejects a validated assignment, SILO also stops startup and identifies the key and file. Since the process exits, it never serves requests with a partially loaded environment.

The file itself remains a privileged secret-bearing input. Operators must protect it with appropriate ownership and mode; parser validation is not a substitute for filesystem permissions.

Regression matrix

The committed tests cover:

• spaces and tabs around =;
• quoted values with significant spaces;
• standalone export, including Unicode whitespace after it;
• keys beginning with _, a digit, or punctuation;
• named targets using -, ., :, and Unicode;
• exact named-target discovery through the configuration subsystem;
• empty keys, whitespace, NUL, and invisible format characters;
• NUL values;
• multiple = characters in URLs and tokens;
• file-and-line diagnostics that redact values;
• all-or-nothing parse results.

The implementation passed the complete local server verification matrix and a read-only adversarial review. Windows-specific os.Setenv behavior has not been exercised on a Windows runner; unsupported platform rejection remains fail-fast rather than silent.

Compatibility and delivery

No configuration migration is required. Existing ordinary environment names behave unchanged. Files using shell-style whitespace become more predictable, and previously accepted named targets work again.

The visible compatibility changes are intentional:

• invalid or invisible names now fail instead of being silently ignored;
• unquoted surrounding value whitespace is trimmed; quote it when significant;
• malformed input stops startup with a redacted location-aware error;
• a valid punctuation-bearing target is no longer rejected merely because a shell could not assign it with NAME=value syntax.

This record describes a source commit, not a delivered release. Until the commit is pushed, tested remotely, merged, tagged, packaged, imaged, and deployed, operators must not assume a public SILO binary contains this parser contract.

Conclusion

Configuration compatibility depends on validating the format SILO actually consumes. MINIO_CONFIG_ENV_FILE borrows a small amount of dotenv-like syntax for operator convenience, but it is not executed by a shell. The repair restores named-target compatibility while retaining strict NUL, invisibility, redaction, and fail-fast guarantees.

15 - Two SSE-C Keys, One CopyObject Response

This record explains the CopyObject SSE-C checksum response repair committed in SILO as e73436c99.

Status on 2026-08-28: implementation, encryption and key-rotation tests, complete server suites, race tests, static checks, build, and independent Fable Max acceptance review are complete. The commit was merged into main on 2026-08-29 as e73436c99; tag, package, image, deployment, and production verification remain separate gates.
Scope: the successful CopyObject XML and HTTP response after the destination object has committed. Stored object bytes, checksum metadata, encryption format, source decryption, federation, replication, and historical objects are unchanged.
Security property: source SSE-C headers may decrypt only source state; destination SSE-C headers may decrypt only committed destination state.

Too Long; Didn’t Read (TL;DR)

An SSE-C copy can use two independent keys:

SILO correctly wrote the destination with its destination key. However, after commit, both the XML generator and the generic PUT-response header helper received the complete CopyObject request. The checksum metadata decrypter intentionally prefers copy-source SSE-C headers when they are present. That priority is correct while reading the source, but wrong when interpreting the committed destination.

With source key A and destination key B:

source body       --decrypt A--> logical bytes
logical bytes     --encrypt B--> committed destination
destination csum  --seal B-----> stored checksum metadata
response decoder  --try A------> key mismatch, checksum omitted

The object and stored checksum were correct; only the successful response was incomplete. The repair constructs a destination response-header view by removing exactly the three copy-source SSE-C customer headers. It decrypts the destination checksum once, then reuses the resulting map for both XML and HTTP response headers.

Observable failure

The failure requires a checksum-bearing destination and distinct source/destination SSE-C contexts. A representative request supplies:

x-amz-copy-source: /bucket/source
x-amz-copy-source-server-side-encryption-customer-algorithm: AES256
x-amz-copy-source-server-side-encryption-customer-key: <key A>
x-amz-copy-source-server-side-encryption-customer-key-md5: <md5 A>
x-amz-server-side-encryption-customer-algorithm: AES256
x-amz-server-side-encryption-customer-key: <key B>
x-amz-server-side-encryption-customer-key-md5: <md5 B>
x-amz-checksum-algorithm: CRC32

Before the repair:

• CopyObject returned HTTP 200;
• reading the destination with key B returned the correct body;
• stored destination checksum metadata decrypted with key B and matched the logical bytes;
• the CopyObject XML and HTTP response omitted CRC32 and ChecksumType.

This is a response-contract defect, not evidence of corrupted object data.

The same ambiguity affects same-object SSE-C key rotation. After metadata has been resealed under key B, the request still carries source key A in the copy-source headers. Response generation must describe the post-rotation object, so it must use B.

Why the global decrypter must not change

The metadata decrypter’s copy-source priority is not itself a bug. Earlier in CopyObject, the server examines source checksum metadata to decide whether to preserve its algorithm, recompute a full-object value, or add the default CRC64NVME checksum. For an SSE-C source, that metadata is protected by the source object key and therefore requires the copy-source headers.

Changing the global priority to prefer destination SSE-C headers would fix the final response while breaking source checksum interpretation. The safe boundary is temporal and object-specific:

before commit: full request headers, source context
after commit:  destination-only SSE-C headers, destination context

The repair applies only at that post-commit boundary.

Selected implementation

Destination response view

The handler clones the request headers and removes exactly:

• X-Amz-Copy-Source-Server-Side-Encryption-Customer-Algorithm;
• X-Amz-Copy-Source-Server-Side-Encryption-Customer-Key;
• X-Amz-Copy-Source-Server-Side-Encryption-Customer-Key-MD5.

Regular destination SSE-C headers remain. SSE-S3 and SSE-KMS destination metadata needs no customer key and continues through the existing path.

Decrypt once, project twice

Before the repair, CopyObject called decryptChecksums once while building XML and again while writing success headers. For SSE-S3 or SSE-KMS this could repeat KMS unseal work.

The repaired flow is:

committed ObjectInfo
  -> decryptChecksums(destination headers) once
  -> CopyObjectResult XML fields
  -> x-amz-checksum-* and x-amz-checksum-type response headers

The generic setPutObjHeaders wrapper remains available to PutObject, CompleteMultipartUpload, and DeleteObject. CopyObject calls a narrow helper that accepts the already decrypted checksum map. ETag, VersionID, delete-marker, lifecycle prediction, and checksum header behavior remain in one shared implementation.

Regression matrix

The tests cover:

• plaintext source to SSE-C destination;
• compressed and uncompressed SSE-C destinations;
• SSE-C source key A to destination key B;
• checksum value and type in both CopyObject XML and HTTP headers;
• stored checksum decrypted with destination key B;
• destination body readable with B;
• same-object key rotation from A to B;
• checksum response after rotation;
• SSE-S3 source and destination combinations;
• all object-layer backends used by the API test harness.

The final combined tree passed focused encryption tests, the complete cmd and internal suites, the project’s tagged test configuration, full go test -race ./..., vet, lint, generated-file checks, rebrand guards, and a local build. A mirror Fable Max review reported no P0–P2 findings and independently confirmed that source decryption still receives the full request while destination response decryption receives the filtered view.

Compatibility and operational impact

• Successful CopyObject responses: checksum fields that were previously missing now appear when the committed destination has a checksum.
• Stored objects: no rewrite, migration, metadata-format, or encryption-format change.
• Existing objects: unaffected; the defect existed only in the one-time successful response.
• Clients: no request change. Clients already providing both source and destination SSE-C keys receive a more complete S3-compatible result.
• Performance: one metadata checksum decryption instead of two; no additional object read or hash pass.
• Rolling upgrade: old nodes may omit the fields while new nodes return them. Stored objects remain mutually readable.
• Rollback: restores response omission but does not damage objects created while the repair was present.
• Security: no key or digest value is added to logs or error responses. The response carries only the checksum already authorized for the successful write.

This repair does not resolve the separately deferred legacy federation CopyObject branch and does not audit or modify historical compressed-object checksums. Those questions have different data and operational boundaries.

Conclusion

CopyObject is one request with two object identities. Reusing the full request after commit erased that distinction: a source key was allowed to shadow the destination key while describing destination metadata. The durable repair is not a new encryption scheme; it is an explicit context boundary, followed by one decryption and two faithful response projections.

16 - Why CompleteMultipartUpload Must Return ChecksumType: Review of PR #57

This is the design, review, and decision record for SILO #47 and PR #57.

Status on 2026-08-26: PR #57 was approved and merged as a96116b1; #47 closed automatically. All nine checks on the tested PR head passed, followed by green Go CI and VulnCheck runs on main. No tagged release, package, container image, deployment, or production endpoint has yet been verified to contain the fix.
Scope: return the already-known checksum type from CompleteMultipartUploadResult; do not add new checksum algorithms.
Owner: pgsty/silo, the SILO server repository.
Release boundary: code review, merge, a green main, a tagged release, packages, container images, deployment, and production verification are separate gates.

Too Long; Didn’t Read (TL;DR)

SILO already computed and persisted the correct checksum type for a completed multipart object. HEAD, ListParts, and GetObjectAttributes could expose it. The completion response could not, because its Go response struct had checksum value fields but no ChecksumType field.

PR #57 adds that field, copies the existing value from the checksum map, registers the new exported symbol in the compatibility baseline, and tests FULL_OBJECT, COMPOSITE, and the no-checksum case. It does not recalculate data, change metadata, migrate objects, or weaken integrity checks.

The repair is correct and intentionally narrow. Maintainers approved the fork workflows, refreshed the stale PR branch onto current main, required every new check to pass, submitted an approving review, and merged while preserving the contributor’s signed-off commit. Repository integration is complete; release delivery remains a separate gate.

Where the defect came from

The defect was found while investigating #31, where a real boto3 client exposed several adjacent multipart-checksum incompatibilities. #31 was the data-path failure: a FULL_OBJECT CRC32 multipart upload could fail at completion. It was fixed independently by 0cff48f6c and 75859690b, then closed on 2026-08-04. That review deliberately split four adjacent findings into #46, #47, #48, and #50 instead of treating them as one checksum bug.

After the object completed successfully, another inconsistency remained:

complete_multipart_upload() -> ChecksumType: None
head_object()               -> ChecksumType: FULL_OBJECT

AWS S3 returned FULL_OBJECT in both places. SILO returned the checksum value in the completion XML, and the committed object retained the correct type, but the completion SDK result exposed a null type.

That observation became #47. It is a presentation defect, not a checksum-calculation or storage defect. It does not explain the earlier InvalidPart failure from #31, and repairing it does not replace the server-side part-checksum work tracked in #46, which later landed independently as 7fea6d5a5.

The S3 response contract

The AWS CompleteMultipartUpload API defines ChecksumType as an element of CompleteMultipartUploadResult. Its valid values are:

When an object has no additional S3 checksum, the element should be absent. A server must not invent a type with no checksum value.

This distinction matters to clients. The same Base64 field name can describe either a direct full-object checksum or a multipart composition. A client that validates the completion result needs the type to interpret the checksum correctly and to compare the response with the mode selected at CreateMultipartUpload.

What SILO did before the PR

The completion handler already passed the committed ObjectInfo to generateCompleteMultipartUploadResponse. That generator already called:

cs, _ := oi.decryptChecksums(0, h)

The checksum decoder returned a map containing both the algorithm value and the normalized object type:

CRC32                 -> "...Base64..."
x-amz-checksum-type    -> "FULL_OBJECT" or "COMPOSITE"

The response struct copied the values for CRC32, CRC32C, CRC64NVME, SHA1, and SHA256. It simply had nowhere to put the type:

committed ObjectInfo.Checksum
        -> decryptChecksums
        -> checksum values + x-amz-checksum-type
        -> CompleteMultipartUploadResponse
        -> checksum values copied, type discarded
        -> XML without <ChecksumType>
        -> SDK returns None / null

Other surfaces used the same state correctly. ListParts and GetObjectAttributes already returned ChecksumType; HEAD also reported the stored type. The loss was isolated to the success XML for CompleteMultipartUpload.

What PR #57 changes

The contributed diff contains one signed-off commit, three files, 60 added lines, and no deletions. Only two production lines change. A maintainer later merged current main into the contributor branch to refresh its CI context; that merge changed history, not the three-file product diff.

Add the response field

ChecksumType string `xml:"ChecksumType,omitempty"`

omitempty is part of the compatibility contract: checksum-free uploads retain the old XML shape.

Copy the existing normalized value

ChecksumType: cs[xhttp.AmzChecksumType],

The generator does not infer the type from an ETag, algorithm name, or part count. It uses the same decoded metadata that already supplies the checksum values.

Test the response surface

The added test covers:

• no checksum: the Go field is empty and <ChecksumType> is absent;
• a full-object checksum: the field is FULL_OBJECT and the tag is present;
• a multipart composite checksum: the field is COMPOSITE and the tag is present.

It checks the response value before XML encoding and separately checks omission/presence after encoding.

Record the exported compatibility symbol

CompleteMultipartUploadResponse.ChecksumType is an exported Go field. SILO’s rebrand guard performs an exact comparison of the exported compatibility surface, so the PR correctly adds the field to buildscripts/rebrand-guard/compat-baseline.json. This is an acknowledgement of an intentional public surface change, not a bypass of the guard.

Why the repair works

The correctness argument is a short chain of existing invariants.

1. ObjectInfo.Checksum is the committed checksum metadata. The completion response is generated only after the object layer returns the committed ObjectInfo.
2. decryptChecksums(0, h) uses the existing metadata-decryption path, including the request headers needed for SSE-C. No second decryption mechanism is added.
3. The checksum decoder writes x-amz-checksum-type only when it has decoded a non-empty checksum value.
4. Existing ChecksumType.ObjType() logic normalizes reachable states to FULL_OBJECT or COMPOSITE.
5. Indexing a nil or missing map entry returns the empty string.
6. XML omitempty removes the element for that empty string.

The resulting behavior is deterministic:

The change is therefore a missing projection from established state to the wire response. It does not create new checksum state and cannot make an incorrect checksum correct. It makes the response describe the state the server has already validated and committed.

Review and verification

The PR was reviewed after the contributor branch was refreshed onto current main. The update produced head c4b9d38d; the resulting tree hash, 39ec44c6b390c441413e490370f70fbacc4e6a91, exactly matched the isolated local no-commit merge. The result was clean and included the intervening checksum work on main.

Local verification on that exact merge result included:

targeted ChecksumType regression test
CGO_ENABLED=0 go test ./cmd/ -count=1 -timeout 30m
go vet ./cmd/
gofmt and git diff --check
rebrand compatibility guard
local DCO rule

The targeted regression completed in 2.174 seconds and the full cmd package test completed in 168.956 seconds. The commit author email matches its Signed-off-by trailer. Cryptographic Git commit signing is independent of DCO and is not required by this repository.

A separate read-only local Claude Code adversarial review inspected the merged diff, checksum serialization, XML path, current main, tests, DCO, and compatibility guard. Its verdict was COMMENT: the production change was correct and safe, but it preferred an additional HTTP-level completion test before merge. The maintainer agreed that such a test would improve fidelity, but disagreed that it was blocking: the handler delegates directly to the tested generator, while existing real MPU tests already cover persisted FULL_OBJECT and COMPOSITE states. The formal GitHub review therefore recorded APPROVED with the HTTP-level test as a follow-up.

Actions, branch refresh, and merge

The first four action_required runs had been created on 2026-08-09 against the PR’s old base. After approval, DCO passed but the old VulnCheck run used Go 1.26.5 and failed on newly published standard-library vulnerabilities fixed in Go 1.26.6. Current main had already moved to Go 1.27.0, and its latest VulnCheck was green. Treating the stale failure as either a product regression or an ignorable red check would both have been wrong.

The decision was to refresh the test context, not rerun or waive the stale result:

1. GitHub’s update-branch API merged current main (8d76a255c) into contributor head d014a12cf, producing c4b9d38d without conflicts.
2. GitHub created four new fork workflow runs for the refreshed head; all four were explicitly approved again.
3. All nine reported checks passed: DCO, VulnCheck, six jobs in Go CI, and the Test Release Pipeline. The release validation job completed in 11 minutes 26 seconds.
4. A formal approving review was submitted against c4b9d38d.
5. Merge used an expected-head guard and the repository’s normal merge strategy, producing a96116b1. This preserved the contributor’s signed-off commit rather than rewriting it through a squash. The PR’s Resolves #47 relationship closed the issue one second later.
6. The post-merge main VulnCheck and all six Go CI jobs also passed; cross-compilation, the slowest job, completed in 9 minutes 54 seconds.

This sequence matters because “the patch passed once” was not the acceptance criterion. The exact tree merged into current main had to be the tree reviewed and tested, and a stale CI environment could not substitute for that proof.

Evaluation of the PR

What is strong

• The scope matches the defect. Two production lines restore one missing response element.
• It reuses authoritative state. There is no duplicate type derivation and no new checksum algorithm branch.
• Backward compatibility is explicit. omitempty preserves checksum-free responses.
• The test covers both valid values and absence. A regression cannot silently restore the null result.
• The compatibility baseline is updated deliberately. CI is not weakened.
• DCO provenance is complete. The sole commit has a matching sign-off.

Non-blocking review notes

The test is correct for the changed generator but its fixtures are not byte-for-byte models of every production multipart metadata flag:

• the FULL_OBJECT fixture reaches the right value through a non-multipart checksum state rather than a completed multipart state carrying ChecksumMultipart, ChecksumIncludesMultipart, and ChecksumFullObject;
• the COMPOSITE fixture carries the multipart flag but omits the persisted per-part checksum block.

Existing API-level tests already exercise genuine FULL_OBJECT and COMPOSITE completion and verify their committed types. PR #57 tests the remaining projection from decoded state to the response field and XML. Adding an assertion to those full API tests would improve test fidelity, but it is not required for this two-line repair.

The PR places ChecksumType before the algorithm-specific fields, while AWS’s example response and SILO’s newer CopyObjectResponse place it after them. Mainstream S3 SDKs parse XML by element name, so this is a parity and style detail rather than a compatibility blocker. Moving the field is optional.

Finally, the contributor commit title says feat: even though the PR correctly marks itself as a bug fix. The final merge preserved that signed-off commit instead of rewriting it. This is a history/style imperfection, not a protocol or release blocker.

Why new algorithms do not belong in this PR

AWS now documents additional fields such as SHA512, MD5, and XXHASH variants. Adding those XML fields alone would create false compatibility.

SILO’s current checksum implementation supports CRC32, CRC32C, CRC64NVME, SHA1, and SHA256. A real new algorithm requires coordinated support across:

• request header parsing and validation;
• streaming checksum calculation;
• multipart FULL_OBJECT or COMPOSITE semantics;
• on-disk checksum encoding and decoding;
• UploadPart, UploadPartCopy, completion, copy, replication, HEAD, GET, ListParts, and GetObjectAttributes;
• SDK/client interoperability and a full encrypted/compressed/versioned test matrix.

PR #57 should not grow response-only placeholders for algorithms the server cannot calculate or persist. Each new algorithm family needs a separate compatibility decision, implementation, and review.

Compatibility and operational impact

• S3 clients: checksum-aware clients receive ChecksumType from future successful multipart completions instead of null.
• Wire format: one additive XML element appears only when an additional checksum exists. Clients that ignore unknown elements remain unaffected.
• Integrity: no checksum is recalculated or accepted differently. Existing validation semantics are unchanged.
• Stored data: no object, part, metadata, or erasure format changes. No migration or backfill.
• Existing objects: object state remains correct. A past completion response cannot be replayed; use HEAD or GetObjectAttributes to inspect an existing object’s type.
• Encryption: the response uses the established checksum metadata-decryption path. No key material or new secret is exposed.
• Performance: one map lookup and one optional XML element; no extra object read, hashing pass, or allocation proportional to object size.
• Rolling upgrade: old nodes omit the element and new nodes return it. Requests and stored objects remain compatible, but client-visible behavior stabilizes only after all serving nodes are upgraded.
• Rollback: rolling back removes the response element from future completions; it does not damage objects created while the fix was present.
• Other repositories: no server dependency, silo-pkg, MCLI, or Console change is required. Public documentation belongs in this site.

This is an additive compatibility repair, not a release feature that requires operators to rewrite data. Its only externally visible effect is a more complete success response.

Merge and release decision

The final decision had six parts:

1. accept the narrow projection fix without recalculating checksums or changing storage;
2. keep SHA512, MD5, and XXHASH families out of #57 until they have end-to-end server support;
3. record an HTTP-level completion test as useful follow-up work, not a blocker for the directly tested generator repair;
4. reject stale CI as merge evidence, update the branch to current main, and approve the newly created workflows;
5. merge only after the refreshed head was formally approved and every check was green, using an expected-head guard and a normal merge that preserved the DCO-signed contribution;
6. let Resolves #47 close the issue, then verify the resulting main workflows independently.

No dependency update, storage migration, or cross-repository implementation was required. That decision is now complete at the repository-integration gate.

A green main still does not prove that a SILO tag, release package, container image, deployment, or production endpoint contains the repair. Those delivery gates remain unverified and must be recorded separately when the next release ships.

Conclusion

PR #57 is a good example of a small compatibility fix whose correctness comes from respecting an existing source of truth. The checksum type was already calculated, validated, persisted, decryptable, and visible through other APIs. The completion response simply failed to project it into XML.

The accepted repair does exactly that projection and nothing more. It makes the wire response honest without touching user data, checksum mathematics, storage layout, or algorithm scope. The fork workflows, refreshed-head review, merge, automatic issue closure, and post-merge main verification are complete. What remains is delivery discipline: distinguish this merged fix from a tagged, packaged, imaged, deployed, and production-verified release.

17 - When the Total Is Unknown: Folder Download Progress

Status: Shipped in SILO Console 2.2.0 (16960f7ab); the server embeds it since its Console pin was updated (4d6e1ea8e) · Priority: P1 · Owner: pgsty/silo-console · Related issue: pgsty/silo#62 · PRD review: Claude Fable 5 (xhigh) — APPROVE · Implementation review: Claude Fable 5 (xhigh), 2026-08-23 — APPROVE, no P0/P1/P2 findings

SILO Console shows NaN% in Downloads / Uploads while downloading a folder. The ZIP normally keeps streaming and the stored objects are intact, but the progress bar has crossed from “unknown” into an invalid determinate state. Users see a full-looking bar, assume the transfer failed or finished, and retry it.

The proposed repair is intentionally narrow:

A download may enter determinate mode only when it has a finite, positive total measured in bytes applicable to that response. Without such a total, it remains indeterminate until completion, failure, or cancellation.

The server keeps streaming ZIPs. Ordinary files keep their percentages. The frontend gains one safe calculation boundary, reuses its existing indeterminate renderer, and closes one missing cancellation transition. This record defines why that is both sufficient and the smallest truthful fix.

The observed failure

The defect was observed in the then-current silo-console v2.1.1, which is embedded by Silo RELEASE.2026-08-06T00-00-00Z.

Reproduction:

1. Put several objects below a prefix such as folder/.
2. Stay in the parent listing, select folder/, and click Download.
3. Open Downloads / Uploads before the transfer finishes.
4. The row displays NaN%; the ZIP request continues.

The runtime check used a prefix containing about 88.7 MiB and throttled Chromium to preserve the observation window. Two independent downloads produced the same NaN% state.

This is a frontend correctness bug. It is not evidence of corrupted objects, an altered disk format, or a failed S3 GET.

What is actually happening

The visible NaN% is the end of a contract mismatch across three layers.

A prefix has no object size

S3 folders are common prefixes, not stored directory objects. In the listing model, a prefix ends in / and carries size=0. The Console already renders that size as -, correctly treating it as not applicable.

The generated API model marks size as omitempty, so logical zeroes are absent from listing JSON. The single-selection thunk nevertheless passes object.size straight into the download helper: a prefix or zero-byte object therefore supplies undefined at runtime (while synthetic prefix records may supply 0). Neither value is a valid denominator.

A streamed ZIP has no known wire length

The server recognizes the trailing /, recursively lists the objects, then connects a zip.Writer to an io.Pipe. Objects are read, deflated, and copied to the HTTP response as the archive is produced.

That behavior is desirable: the server can send the first bytes without holding the complete archive in memory or on disk. Its consequence is equally deliberate: the final compressed byte length does not exist when headers are sent, so the response has Content-Type: application/zip and a filename, but no Content-Length.

The sum of source object sizes is not a substitute. Source sizes are uncompressed bytes; ProgressEvent.loaded counts response bytes after ZIP compression and framing. They are different units.

A progress event does not imply a computable percentage

The client currently computes every event as:

Math.round((event.loaded / fileSize) * 100)

For a prefix, the denominator is zero or absent. Depending on the value and event, JavaScript produces NaN (loaded / undefined or 0 / 0) or Infinity (positive bytes divided by zero).

The progress callback then writes that non-finite value into Redux and sets waitingForFile=false. That second operation is the decisive state error: the task leaves the existing indeterminate branch merely because an event arrived, not because the event contained a usable total. The determinate progress component receives the invalid value and renders an invalid label.

The complete chain is:

common prefix: size = 0
        |
        v
download(..., fileSize = 0)
        |
        v
streamed deflated ZIP, no Content-Length
        |
        v
event.loaded / 0 => NaN or Infinity
        |
        v
invalid percentage enters Redux; waitingForFile becomes false
        |
        v
determinate ProgressBar renders NaN%

Ordinary non-empty files avoid the defect because the server can stat the object, sets Content-Length, and the list size is positive. If the browser emits a progress event for an empty response, a zero-byte file reaches the same arithmetic boundary as a prefix even though it is a real object; it therefore belongs in the regression contract.

Product contract

The UI needs one honest distinction:

• Determinate means both transferred bytes and total bytes are known in the same unit.
• Indeterminate means the request is active but the total is unknown.

This yields four load-bearing invariants:

determinate  => total is finite and total > 0
determinate  => percentage is finite and 0 <= percentage <= 100
unknown total => indeterminate
terminal state => not indeterminate

These invariants are more general than objectPath.endsWith("/"): they cover prefixes, zero-byte files, malformed metadata, and any future unknown-length response without inventing object-type exceptions.

Goals and non-goals

Goals

1. A folder download never displays NaN%, Infinity%, or a fabricated percentage.
2. Unknown-length transfers use the existing indeterminate animation.
3. Known-length ordinary files retain their current percentage behavior.
4. Completion, failure, and cancellation always leave indeterminate mode.
5. A zero-byte file never produces a non-finite percentage and still reaches success.
6. No non-finite or out-of-range download percentage enters Redux.
7. The fix can ship in Console first and then be consumed by Silo as a dependency update.

Non-goals

• Do not pre-generate or buffer a complete ZIP on the server.
• Do not use the sum of uncompressed object sizes as network progress.
• Do not redesign the entire Object Manager state model.
• Do not route folders through the current immediately-completing BrowserDownload path.
• Do not solve the browser memory cost of XMLHttpRequest.responseType="blob" here.
• Do not change whether a cancelled row remains visible until the user clears it.
• Do not redesign mid-stream ZIP error signaling after HTTP headers have been sent.
• Do not modify the S3 API, Console API, object layout, or archive contents.

Those are legitimate follow-ups, but coupling them to this defect would enlarge risk without being necessary to restore truthful progress.

The decision

The minimum production repair has four parts.

D1. Calculate only from a valid total

Add a small pure function, separate from DOM and Redux side effects:

type DownloadProgressEvent = Pick<
  ProgressEvent,
  "loaded" | "lengthComputable" | "total"
>;

export const calculateDownloadPercent = (
  event: DownloadProgressEvent,
  objectSize: number,
): number | null => {
  let total: number | null = null;

  if (Number.isFinite(objectSize) && objectSize > 0) {
    total = objectSize;
  } else if (
    event.lengthComputable &&
    Number.isFinite(event.total) &&
    event.total > 0
  ) {
    total = event.total;
  }

  if (
    total === null ||
    !Number.isFinite(event.loaded) ||
    event.loaded < 0
  ) {
    return null;
  }

  return Math.min(
    100,
    Math.max(0, Math.round((event.loaded / total) * 100)),
  );
};

The source priority preserves compatibility:

1. A finite positive objectSize retains the current ordinary-file calculation.
2. If object size is unavailable but the browser declares the response length computable and supplies a finite positive event.total, use it.
3. Otherwise return null: no truthful percentage exists yet.

The helper’s output contract is complete: either null, or a finite number in [0,100].

D2. Keep unknown totals indeterminate

Change the XHR handler to dispatch only a real percentage:

req.addEventListener("progress", (event) => {
  const percent = calculateDownloadPercent(event, fileSize);

  if (percent !== null) {
    progressCallback(percent);
  }

  // No valid total: preserve waitingForFile=true so the existing UI remains
  // indeterminate instead of manufacturing a determinate value.
});

Download rows already start with waitingForFile=true, and ObjectHandled already renders that state with variant="indeterminate". There is no need to widen Redux to number | null, add another boolean, or change MDS.

When the first valid percentage arrives, the existing updateProgress action stores it and sets waitingForFile=false. When no valid percentage ever arrives, the row remains indeterminate until a terminal action.

D3. Make cancellation terminal

Completion and failure already clear waitingForFile. Cancellation does not. Add the missing transition in cancelObjectInList:

item.waitingForFile = false;

Without that line, the repaired prefix download would remain in the indeterminate rendering branch after abort, masking the Cancelled state. The row continues to follow the current product behavior: it remains as a cancelled record and can be removed manually. Automatic removal is not part of this change.

There is one event-order guard at the XHR boundary as well. abort() first produces readystatechange(DONE, status=0) and only then the abort event; without a status-zero return, the generic DONE branch marks the request failed before onabort can mark it cancelled. DONE/status zero is therefore left to the dedicated onerror or onabort handler, and onabort removes the stored request reference.

D4. Normalize an omitted zero-byte size

The single-selection thunk passes object.size || 0, matching the other download entry point. This restores the API model’s omitted logical zero before the helper checks Blob.size === fileSize, so an HTTP 200 zero-byte object completes at 100% instead of being reported as incomplete.

D5. Keep the server stream unchanged

The folder handler continues to generate a deflated ZIP through io.Pipe and omit Content-Length. No API, archive, storage, or resource-management contract changes.

State machine

The state does not move back from determinate to indeterminate. If a later event lacks a valid total after a valid percentage was observed, the handler simply retains the last valid value.

Failed and Cancelled both set done=true in the existing reducers. ObjectHandled uses done to change its close button from “abort request” to “remove record”; this repair preserves that behavior. The cancelled Redux value remains 0, while the existing ProgressBarWrapper renders a full orange terminal bar with a Cancelled label because ready=true. That established presentation is not part of this repair.

waitingForFile is not the ideal long-term name for “no computable progress.” Renaming it or replacing the booleans with a discriminated union would improve the model, but that is a separate refactor. In this repair, the field already expresses and renders the required state, so reusing it minimizes compatibility risk.

Why this is sufficient

The repair closes the bug by cases.

Ordinary non-empty file

objectSize > 0, so the helper uses the same denominator as today. The result is finite and clamped, updateProgress enters determinate mode, and completion still sets 100%.

Current streamed folder

objectSize is normalized to 0, while lengthComputable=false and event.total=0. The helper returns null; no invalid action is dispatched, so the row remains indeterminate. Completion sets waitingForFile=false, percentage=100, and done=true.

Future response with a real length

If a proxy or later server implementation provides a trustworthy response total, lengthComputable=true and event.total>0. The same code automatically produces a real percentage without another product change.

Zero-byte file

The omitted listing size is normalized to zero, and both totals are then zero, so an intermediate percentage is mathematically undefined. The row stays indeterminate for its usually brief lifetime; the zero-byte Blob now equals the normalized expected size, and the successful response transitions directly to 100%. 0/0 is never evaluated.

Failure and cancellation

Failure already exits indeterminate. The added cancellation transition does the same on abort. No terminal row can continue to look active merely because its total was unknown.

Mathematically, division occurs only when total belongs to (0, +infinity). The result is then clamped to [0,100]. Therefore neither NaN nor Infinity can cross the calculation boundary into Redux or the determinate renderer.

Rejected alternatives

Buffer the ZIP to obtain Content-Length

The server could generate the complete archive in memory or a temporary file, measure it, and then send it. That would provide an exact wire total, but at the cost of memory or disk pressure, delayed first byte, cleanup complexity, and worse concurrent-download behavior. An observability defect does not justify discarding streaming.

Sum the objects under the prefix

That sum is uncompressed logical data. event.loaded measures compressed response bytes plus ZIP framing. The units differ, so the bar could stop below 100%, exceed 100%, or move according to compression ratio rather than transfer completion. Reject.

Convert invalid progress to 0%

This hides the string but lies about the state: determinate 0% means the total is known and no portion has transferred. Users would still interpret the transfer as stalled. Unknown must remain unknown.

Special-case paths ending in /

That fixes the reported prefix but misses a real zero-byte object, invalid metadata, and other unknown-length responses. The correct boundary is denominator capability, not object type.

Send folders through BrowserDownload

The current large-file path creates an anchor and immediately calls the completion callback after clicking it. It cannot report true completion, console-managed cancellation, or a subsequent HTTP failure. It may be the basis of a later streaming-download design, but today it would replace one lie with another.

Sanitize inside ProgressBar

A generic component guard could be useful defense in depth, but it would leave invalid data in Redux and hide the broken state transition from every other consumer. The primary repair belongs where progress becomes application state.

Introduce percentage: number | null now

A discriminated progress state would be cleaner than the current booleans if the Object Manager were being redesigned. Adding null while retaining waitingForFile, done, failed, and cancelled would instead create more contradictory combinations. Removing the old fields is larger than this bug requires. Reuse the already-rendered indeterminate state now; redesign it separately.

Requirements and acceptance

Functional requirements

• FR1: An unknown total keeps the task indeterminate.
• FR2: A finite positive object size preserves ordinary-file percentages.
• FR3: A finite positive event.total is a fallback only when lengthComputable=true.
• FR4: Every dispatched percentage is finite and within [0,100].
• FR5: A zero-byte file never displays non-finite progress and reaches success.
• FR6: Completion, failure, and cancellation leave indeterminate mode.
• FR7: Versioned objects, anonymous downloads, previews, and long-filename entry points retain their existing call contract.

Non-functional requirements

• No new server CPU, memory, disk-buffer, or request cost.
• No new frontend dependency or build step.
• No change to the S3 API, Console API, ZIP content, or stored objects.
• The calculation must be testable without a DOM or live store.
• TypeScript typecheck and the production frontend build must pass.

Acceptance criteria

1. While a folder ZIP without Content-Length is active, its row shows an indeterminate animation and no percentage text.
2. On successful completion, the row reports success/100% and the ZIP can be opened.
3. A normal non-empty file continues to show finite determinate progress and completes at 100%.
4. A zero-byte file never shows NaN% or Infinity% and completes successfully.
5. Cancelling an unknown-total download aborts the request and shows Cancelled, not an active animation.
6. No download path can place a non-finite or out-of-range percentage in Redux.

Test plan

Pure calculation matrix

Use the existing @playwright/test runner for the pure module rather than adding a test framework. This needs one config-only addition in web-app/playwright.config.ts: a dependency-free unit project, for example with testMatch: /.*\.unit\.ts/. The existing chromium project depends on the auth setup against a live Console at localhost:9090; pure calculation and reducer tests must not be gated by that environment. No new dependency is introduced.

State tests

Cover the transition contract directly:

1. A new download starts with waitingForFile=true.
2. No valid progress action means it remains indeterminate.
3. Valid progress produces a finite value and waitingForFile=false.
4. Complete produces done=true, waitingForFile=false, percentage=100.
5. Failure produces failed=true, done=true, waitingForFile=false.
6. Cancel produces cancelled=true, done=true, waitingForFile=false, percentage=0.

Browser regression

Use the real Console test instance and Chromium:

1. Create a temporary bucket with several objects below folder/.
2. Select the prefix from its parent and start the download.
3. Apply CDP download throttling so the intermediate state is observable. Throttled runs must raise the default 30-second test timeout with test.setTimeout.
4. Open Downloads / Uploads and verify that the row exists, has no percentage label, and contains neither NaN% nor Infinity%.
5. Cancel it and verify the Cancelled terminal state.
6. Restore network conditions in finally.
7. Download again without throttling, wait for the browser download, and verify the ZIP.
8. Repeat the relevant assertions for one ordinary non-empty file and one zero-byte file.
9. Remove the bucket, objects, downloads, and temporary files in teardown.

The current Playwright project is Chromium-only, so CDP is an acceptable test mechanism. If Firefox or WebKit projects are later enabled, keep the pure and state tests cross-browser and gate only the throttled observation behind the Chromium project.

Implementation boundary

Expected Console changes:

1. Add downloadProgress.ts containing the pure calculation.
2. Change Objects/utils.ts to dispatch only a non-null percentage, let status-zero terminal events reach their dedicated handlers, and clean up an aborted request.
3. Normalize omitted zero sizes in the single-selection thunk.
4. Change cancelObjectInList to clear waitingForFile.
5. Add calculation, state, and browser regression coverage using existing dependencies, with a dependency-free unit project in playwright.config.ts.

Expected unchanged code and contracts:

• The Go folder-download handler and its streaming ZIP.
• ObjectHandled, ProgressBarWrapper, and MDS.
• IFileItem.percentage: number and the existing thunk callback types.
• S3 and Console API routes.
• Stored object and archive formats.

Delivery and rollback

The fix belongs in pgsty/silo-console, not the Silo server repository where the issue was reported.

Delivery order:

1. Transfer or cross-reference issue #62 to pgsty/silo-console.
2. Implement the bounded Console change.
3. Pass typecheck, production build, pure/state tests, and real browser regression.
4. Publish a new Console release.
5. Update Silo’s pinned Console pseudo-version or release dependency.
6. Build a Silo candidate and repeat folder, ordinary-file, zero-byte, cancel, and ZIP-integrity checks.
7. Publish Silo and record both affected and fixed versions on the issue.

There is no data migration. If the frontend change regresses, Silo can roll back only the Console dependency; server data and API behavior remain compatible.

Definition of done

• The calculation returns only null or a finite [0,100] number.
• Active unknown-total folder downloads render indeterminate.
• Ordinary files retain determinate progress.
• Zero-byte files never render invalid progress.
• Complete, failed, and cancelled rows all leave indeterminate mode.
• The streamed ZIP and server response contract remain unchanged.
• Typecheck, production build, and automated regressions pass locally.
• A Console release is published.
• Silo updates the Console dependency and passes candidate verification.

Follow-up work

Four adjacent improvements deserve separate design records:

1. Stream large folder downloads directly to the browser or filesystem instead of holding the full Blob in memory.
2. Replace the Object Manager’s boolean combination with a discriminated progress/terminal state.
3. Improve end-to-end integrity and error signaling for ZIP failures after headers have been sent.
4. Add a generic non-finite-value guard to shared progress components as defense in depth.
5. Repair the pre-existing Blob JSON error decoder and request-trace cleanup on HTTP failure paths.

None is required to stop the current UI from lying. The next maintenance iteration should first restore the smallest honest contract: known totals get percentages; unknown totals remain unknown.

18 - A ListObjects Shortcut Must Not Turn a Missing Bucket into an Empty One

This document records the problem analysis, design discussion, and repair decision for SILO #32 and PR #37.

Status on 2026-08-26: PR #37 was updated to the DCO-signed head e9c5340be, formally approved, and merged as 49c8aeac4; #32 closed automatically. DCO, VulnCheck, and all six Go CI jobs passed on the exact PR head; the post-merge main VulnCheck and all six Go CI jobs also passed. No tagged release, package, container image, deployment, or production endpoint has yet been verified to contain the repair.
Scope: verify bucket existence only for three listing shortcuts that bypass storage; do not restore the generic checkBucketExist, change the normal listing path, or introduce an existence cache.
Release boundary: local commit, push, remote CI, merge, tag, package, container image, deployment, and production verification are independent gates.

Too Long; Didn’t Read (TL;DR)

The problem is real and worth fixing. A normal ListObjects, ListObjectsV2, or ListObjectVersions request against a missing bucket reaches storage and receives BucketNotFound. Three inputs, however, return early:

• a marker outside the prefix;
• max-keys=0;
• a prefix beginning with /, including the Prefix="/" boto3 reproduction from #32.

Those branches return io.EOF directly. The caller treats EOF as a successful end of listing, so the client receives an empty 200 rather than S3’s 404 NoSuchBucket. The identity of the same missing resource changes from an error to success solely because the selection parameters differ. That breaks S3 compatibility and blocks a real user’s upgrade from the pre-regression release.

The repair must not put an expensive bucket check back in every listing. The selected design replaces only the three bare io.EOF returns with a small helper. The helper calls GetBucketInfo once: it returns the real error if the bucket is absent or cannot be confirmed, and preserves io.EOF when the bucket exists. The normal listing hot path is untouched. Only requests that would otherwise exit before storage pay the extra peer-and-disk fan-out.

That decision has now been executed: the strengthened repair passed local review, the exact PR head passed every remote check, and the expected-head-guarded merge entered a green main.

What is the problem?

One API exposes two bucket-existence semantics

#32 reproduces the defect by calling the following against a missing bucket:

s3.list_objects(Bucket="missing-bucket", Prefix="/")

AWS S3 raises NoSuchBucket; SILO returns a successful empty listing. The difference is not in authentication, routing, or XML serialization. It comes from the object-layer listPath control flow:

regular prefix
  -> enter listMerged
  -> consult storage
  -> missing volume/bucket becomes BucketNotFound
  -> HTTP 404 NoSuchBucket

shortcut input
  -> listPath returns io.EOF early
  -> storage is never consulted
  -> the caller treats EOF as normal completion
  -> HTTP 200 with an empty listing

/ is not the only trigger:

For an existing bucket, returning an empty listing from these branches is a reasonable optimization. For a missing bucket, the same EOF masks the resource error that should take precedence.

The regression has a known origin

The reporter confirmed correct behavior in RELEASE.2024-01-29T03-56-32Z and the regression beginning with RELEASE.2024-01-31T20-20-33Z. The corresponding upstream change is minio/minio#18917 / 80ca12008. It removed GetBucketInfo from generic argument checks and relied on actual Put, List, and Multipart storage operations to expose a missing bucket.

That optimization works on normal paths but leaves a gap: an early-return path never reaches the storage operation that is now responsible for producing the error. #32 does not require a broad rollback of the upstream optimization. It repairs the overlooked control-flow exits.

Why fix it?

The S3 contract explicitly requires NoSuchBucket

Both AWS ListObjects and ListObjectsV2 define NoSuchBucket as HTTP 404 when the specified bucket does not exist. prefix, marker, start-after, and max-keys select listing results; they must not turn a missing bucket identity into a successful request.

ListObjectVersions shares the same object-layer listing engine. Giving V1, V2, and version listings the same existence behavior on the same shortcut inputs prevents the three public APIs from diverging further.

An empty 200 changes client decisions

An empty 200 and a 404 are not interchangeable presentation details:

• 404 tells provisioning or test code to create the bucket, fix configuration, or stop;
• an empty 200 asserts that the bucket exists but has no matching objects;
• SDKs, synchronization tools, and integration tests continue down different branches;
• a test using SILO as an S3 substitute can pass locally and fail against AWS.

#32 also establishes a direct upgrade impact: an application relying on the older correct behavior cannot upgrade past the regression. The repair restores both S3 parity and upgrade compatibility.

The repair surface is narrow and testable

The bug is confined to three adjacent early returns. It does not involve object data, metadata formats, sorting, pagination-token encoding, permissions, or wire schemas. A very small production change can be pinned down with object-layer and HTTP-level contracts, so the benefit clearly exceeds the implementation risk.

Why not restore the global check?

Upstream did not remove generic GetBucketInfo as incidental cleanup. The motivation for #18917 states that checking the bucket before every Put, List, and Multipart operation fans out across servers; even after vectorization, the cost becomes visible beyond 100 nodes.

In current SILO, erasureServerPools.GetBucketInfo calls S3PeerSys.GetBucketInfo. That operation concurrently asks every peer and reduces quorum per pool, while each peer checks its local bucket state. It is not a cheap in-memory map lookup.

Two extremes are therefore unacceptable:

• never check: keep the incorrect empty 200;
• check before every List: restore semantics while undoing a critical large-cluster optimization.

The actual design question is whether the check can be confined to branches that never touch storage and therefore cannot discover the missing bucket naturally. It can.

How is it fixed?

Replace only three bare EOF returns

In cmd/metacache-server-pool.go, each shortcut previously executed:

return entries, io.EOF

It now executes:

return entries, z.listPathShortcutEOF(ctx, o.Bucket)

The helper has only two classes of outcome:

func (z *erasureServerPools) listPathShortcutEOF(ctx context.Context, bucket string) error {
    if _, err := z.GetBucketInfo(ctx, bucket, BucketOptions{}); err != nil {
        return err
    }
    return io.EOF
}

• existing bucket: preserve the previous empty-list behavior;
• missing bucket: pass BucketNotFound into the existing error mapping, producing HTTP 404 NoSuchBucket;
• state cannot be confirmed: propagate quorum, offline, timeout, or context errors instead of fabricating success.

The normal listMerged, metacache scan, sorting, pagination, and response-generation paths do not change.

Why the helper belongs here

The check must sit next to the shortcut for three reasons:

1. only this layer knows that it is about to bypass every storage access;
2. moving it into generic argument validation charges every call;
3. moving it into the scan layer cannot help because these branches never scan.

The name intentionally states the boundary. This is not a new generic checkBucketExist; it restores missing existence semantics immediately before a shortcut returns EOF.

Do not add a cache

A bucket-existence cache could reduce fan-out but immediately creates invalidation questions for create, delete, site replication, recovery, and expiry. Adding a second source of truth for three low-frequency shortcuts costs more complexity and consistency risk than it saves.

The selected implementation uses the existing GetBucketInfo source of truth. If future telemetry shows that large clusters receive frequent max-keys=0, slash-prefix, or disjoint-marker probes, the project can evaluate a dedicated metadata fast path, rate limiting, or a carefully invalidated cache using real data rather than speculative machinery in this compatibility patch.

Test and review evidence

Object-layer contract

The object-layer test runs against single-drive and multi-drive erasure setups and exercises four inputs:

• slash-prefixed prefix;
• zero limit;
• marker outside prefix;
• a regular prefix as a control that still receives the error naturally from storage.

Each case covers ListObjects, ListObjectsV2, and ListObjectVersions, using the typed isErrBucketNotFound predicate rather than brittle English error-string comparison.

HTTP contract

The handler test sends genuine signed requests for all three public APIs:

The HTTP test uses the real slash-prefix reproduction from #32. The other two shortcuts are enumerated at the object layer. This proves final wire behavior without repeating the full matrix in the slower handler fixture.

Local quality gates

The improved local commit passed:

go test ./cmd -count=1
the new object-layer and HTTP regressions (10 subcases)
focused go test -race
related existing listing tests
CGO_ENABLED=0 go build ./...
go vet ./...
CI-scope gofmt and git diff --check
post-commit focused regression rerun

The full local cmd test completed in 116.215 seconds. An independent local Claude Code review used the Fable model at Max effort to inspect the exact tree, call paths, error mapping, tests, performance boundary, and this decision. Its verdict was GO, with no mandatory pre-merge change.

The DCO-signed PR head e9c5340be then passed eight remote checks: DCO, VulnCheck, and six jobs in Go CI. After merge, the resulting main commit 49c8aeac4 independently passed VulnCheck and all six Go CI jobs. The slowest checks were PR cross-compile at 9 minutes 47 seconds and post-merge cross-compile at 9 minutes 30 seconds.

Can it introduce new problems?

Shortcut requests now fan out across the cluster

This is the most important and deliberately accepted cost. A shortcut on an existing bucket used to be little more than a local branch; it now calls GetBucketInfo. Directional local microbenchmarks observed:

These numbers show local relative cost only; they are not a latency prediction for a 100+ node deployment. Real distributed execution adds peer networks, quorum, and slowest-node tail latency, potentially making the gap much larger. That is precisely why the check must not expand into the normal listing path.

The risk concentrates in malformed or probe-style traffic. A misconfigured client polling max-keys=0, a slash prefix, or disjoint markers at high frequency can amplify what was a cheap request into peer-and-disk work. After merge, the actual frequency of these inputs should be observed through S3 traces or metrics; rate limiting or optimization should follow evidence.

A degraded cluster exposes more real errors

Previously, a shortcut could return an empty 200 while peers were offline or bucket quorum was unavailable because it never consulted cluster state. The repair can return quorum, timeout, or service errors in those conditions.

That is more honest behavior, not an availability regression: if the server cannot establish that the bucket exists, it must not assert a valid empty bucket. Clients depending on unconditional empty success will nevertheless observe a behavior change.

Bucket create/delete races are not linearizable

GetBucketInfo and returning the empty result are two actions. The bucket can be deleted immediately after the check, or created immediately after a missing-bucket result is formed. This patch does not and should not add a transaction spanning bucket lifecycle to a listing shortcut.

This is the same concurrency class as other APIs that validate a resource before acting. The repair guarantees that the request no longer succeeds with no existence evidence at all; it does not promise a cross-node, cross-lifecycle linearizable snapshot of an empty listing.

Clients relying on the bug will receive 404

Some clients may have adopted the missing bucket’s empty 200 as fact. They will now enter an error branch. This is a visible compatibility change, but it restores the documented S3 contract and the pre-regression behavior. Preserving the bug merely transfers upgrade cost to clients that correctly rely on 404.

Two adjacent edges remain out of scope

The adversarial review recorded two non-blocking P3 boundaries:

1. When resuming a metacache continuation, the c.fileNotFound branch still returns bare io.EOF. A stale or crafted continuation token used after bucket deletion could theoretically receive an empty 200. Adding GetBucketInfo there would affect normal continuation traffic and needs a separate performance and error-precedence design.
2. Some V1 and version-list marker/prefix combinations return NotImplemented during HTTP handler validation before reaching the object layer; the V2 start-after route can reach it. This patch fixes storage shortcuts masking a missing bucket; it does not redefine precedence between malformed parameters and resource errors.

Neither blocks merge. The first is outside #32’s ordinary initial-list reproduction; the second is inherited handler behavior. Recording them prevents “all three shortcuts are covered” from being overstated as byte-for-byte AWS parity for every possible parameter combination.

Alternatives considered

Keep upstream behavior

This has zero performance change and minimizes fork divergence. It also keeps a documented S3 incompatibility, a regression with a known release boundary, and a misleading result when SILO is used as an integration-test substitute. For a narrow and well-tested compatibility repair, that tradeoff is no longer justified.

Restore generic checkBucketExist

This covers every path at once but reintroduces peer fan-out into every Put, List, and Multipart operation, directly undoing the large-cluster optimization from #18917. The cost is disproportionate and the option is rejected.

Fix only Prefix="/"

That passes the single issue reproduction but leaves the same root defect in max-keys=0 and marker-outside-prefix. The branches are adjacent and share the same semantics, so one helper is simpler and less likely to regress.

Add a bucket-existence cache

This makes shortcuts cheaper but requires semantics for create, delete, replication, recovery, and stale TTL windows. There is no telemetry showing enough shortcut traffic to justify that complexity, so it is not selected.

Complexity and cost-benefit

The overall cost-benefit is favorable. The reason is not that GetBucketInfo is cheap—it is not—but that its cost is strictly limited to three shortcuts that otherwise cannot discover the missing bucket. A narrow performance cost in exchange for explicit protocol correctness is better than either a global rollback or indefinitely preserving the incorrect behavior.

Acceptance decision and remaining gates

The final decision was: accept and merge the strengthened PR #37 revision without expanding the production scope.

The accepted sequence was:

1. replace the old fork head with the current-main, DCO-signed revision while preserving Jason Lin as a co-author;
2. retain typed error predicates, V1/V2/version-list object-layer coverage, and HTTP-level 404 / NoSuchBucket assertions;
3. update the PR description with the shortcut fan-out cost and unchanged normal-path boundary;
4. approve the fork workflows and require all eight reported checks to pass on exact head e9c5340be;
5. submit a formal approving review against that head;
6. merge with an expected-head guard, producing 49c8aeac4, automatically close #32, and require the resulting main Go CI and VulnCheck to pass independently.

No cache, feature flag, additional abstraction, or continuation-token redesign was required. High-frequency shortcut traffic and large-cluster tail latency remain observability follow-ups, not reasons for speculative code expansion.

Repository integration is complete. A tag, package, docker.io/pgsty/silo image, deployment, and real S3-client verification must still complete before the repair can be described as delivered to users.

Conclusion

The issue is not merely “a slash prefix reports the wrong error.” The listing engine uses io.EOF to mean two different things: an empty result from an existing bucket and an early exit that never established whether the bucket exists. Removing generic existence checks for large-cluster performance was a sound upstream optimization, but the shortcuts violate its premise that a real storage operation will naturally surface a missing bucket.

The selected repair restores that premise by calling the existing GetBucketInfo only at three storage-bypassing exits. It makes those requests more expensive and exposes real errors on degraded clusters; both are explicit costs. In return, SILO restores S3’s 404 semantics, upgrade compatibility, and test fidelity while preserving the upstream optimization on the normal listing hot path.

This worthwhile, controlled compatibility fix is now merged and green on main; release delivery remains a separate gate.

19 - Read-Only Checksum Audit and Reliable CLI Output

This is the design and implementation record for MCLI’s read-only checksum verification workflow and pgsty/mc#5, the non-TTY output defect found during release review.

Status: shipped in the final mcli 20260903 release. The command merged to main through pull requests #8 and #13, is exercised against a real SILO server in hosted CI, and pgsty/mc#5 is closed. Bundling the client into the Server image remains a separate gate.
Owner: pgsty/mc.
Tracking: pgsty/mc#5.
Safety boundary: verification is read-only; repair is not part of this command.

Too Long; Didn’t Read (TL;DR)

Historical CopyObject implementations could calculate a stored additional checksum over transformed storage bytes instead of the logical bytes returned by S3. mcli checksum verify inventories objects and independently streams the logical body through the recorded algorithm. Each candidate becomes MATCH, MISMATCH, NO_CHECKSUM, WOULD_VERIFY (dry run), one of ten UNKNOWN_* classifications, or one of three SKIPPED_* results.

The first implementation worked in a terminal but printed nothing when stdout was redirected. MCLI automatically marked non-TTY execution as quiet to disable progress UI, and the new command accidentally treated that internal state as a user request to suppress audit records. The repair separates semantic output from progress suppression without changing global quiet behavior or enabling progress bars in CI.

Command and scope

mcli checksum verify ALIAS/BUCKET/OBJECT
mcli checksum verify --recursive ALIAS/BUCKET[/PREFIX]
mcli checksum verify --manifest candidates.jsonl ALIAS

Version one supports CRC32, CRC32C, CRC64NVME, SHA1, and SHA256 checksums marked as FULL_OBJECT. It can select one object, an exact VersionID, current objects under a prefix, all versions, or exact entries from a JSON Lines manifest. It also supports SSE-C key mappings, time and size filters, dry-run estimation, bounded workers, download limits, JSON output, and an optional JSON Lines report.

It does not verify COMPOSITE checksums, infer type from an ETag, inspect xl.meta, identify the historical writer with certainty, or repair metadata. The endpoint must report the checksum type (x-amz-checksum-type) alongside the checksum; on one that does not, every checksummed object is classified UNKNOWN_CHECKSUM_TYPE rather than guessed at.

Read-only data path

For every selected object, MCLI:

1. sends HEAD with checksum mode enabled and retains every supported checksum plus ChecksumType;
2. rejects unsupported or ambiguous states as UNKNOWN_* instead of guessing;
3. streams GET logical bytes through bounded hashers without writing the body to disk;
4. uses VersionID pinning, or If-Match plus a second HEAD for mutable unversioned/null objects;
5. compares independently calculated values with the stored values.

The S3 boundary allows LIST, HEAD, and GET only. Tests fail if a write method reaches the mock endpoint.

Result and exit contract

Every candidate produces one stable result:

The summary carries objects, a verified count, the count of every result status, and incomplete. verified is MATCH plus MISMATCH: the only results that actually streamed a body through a hasher. A run that enumerated many objects and verified none is visible as such.

--fail-on accepts mismatch, unknown, no-checksum, any, or none. The default any returns exit 1 for mismatches and incomplete verification. no-checksum returns exit 1 when any object carries no checksum or when nothing was verified at all, so an empty prefix or a stale manifest cannot pass as a clean audit. Dry-run does not apply --fail-on. Argument, authentication, enumeration, and report-write failures remain command failures rather than object classifications.

In particular, SKIPPED_TOO_LARGE makes the default any return exit 1 because the size cap leaves the audit incomplete. Time-filter and delete-marker skips do not fail by themselves.

Output and automation contract

Object records and the final summary are semantic output:

• Unless the caller explicitly sets --quiet, -q, or MC_QUIET=true, stdout receives every object record and the final summary in both TTY and non-TTY execution.
• Non-TTY --json emits exactly one compact JSON value per line. TTY JSON keeps MCLI’s existing pretty presentation.
• Global flags work at the app, checksum, and verify levels.
• --report is independent of stdout. It still writes object records and the final summary as JSON Lines when explicit quiet suppresses stdout.
• Output transport does not change --fail-on decisions.

The distinction matters because MCLI’s historical globalQuiet has two inputs: an explicit quiet flag and an automatic non-TTY state used to disable progress UI. Changing that global would risk re-enabling progress output across copy, get, put, mirror, and other commands.

The selected repair is command-local. It walks the full CLI context chain for explicit quiet/JSON flags because the CLI library’s GlobalBool stops at the nearest ancestor flag set. It also restores JSON Lines mode inside the checksum action because nested Before hooks can reset it after an app-level --json. No other command’s progress or output behavior changes.

Report, secrets, and operational cost

Report files are created with mode 0600, must not already exist, and contain metadata/results rather than object bodies or SSE-C keys. The manifest likewise contains only bucket, key, and optional VersionID.

Verification downloads every supported object body. Operators should use --dry-run, --max-size, time filters, --max-workers, and the global download limit to bound cost and load. NO_CHECKSUM and UNKNOWN_* counts must remain visible; neither may be presented as successful verification.

What a mismatch proves

A mismatch proves only that the additional checksum returned at verification time does not describe the logical bytes returned at verification time. It does not prove that a particular historical compression defect created the object, and it is not an external source-of-truth comparison.

Do not overwrite checksum metadata in place. Audit and classify first. For a confirmed, operationally relevant mismatch, prefer a new key or new version, verify the replacement, then switch consumers deliberately. Leave UNKNOWN_* objects out of automatic repair.

Verification record and release boundary

The local acceptance matrix covers TTY human/JSON, non-TTY pipes, regular-file redirects, app/parent/leaf JSON and quiet flags, environment quiet, report under quiet, report-write failure, and MISMATCH/UNKNOWN exit status. It also includes real historical MATCH, MISMATCH, and unsupported-composite objects on a local S3 server.

The command shipped in the final mcli 20260903 release from a signed tag at the tip of main, with the functional suite - including a checksum verification run against a real SILO server - green for that commit, and pgsty/mc#5 is closed. Bundling the client into the Server image and a production audit remain later, separately evidenced gates.

20 - Optional Checksums, Mandatory Failure: Repairing UploadPart and UploadPartCopy Compatibility

This is the complete design and implementation record for SILO #46. The repair was not merely a changed if statement. One apparently optional S3 header reached into multipart completion semantics, copy responses, compression and encryption pipelines, compatibility baselines, and release verification.

Status: merged into main as 7fea6d5a5 on 2026-08-24 (pgsty/silo#46 closed); release and production verification pending.
Owner: pgsty/silo, the SILO server repository.
Tracking: #46.
Independent follow-ups: #63 CopyObject + compression checksum, #64 federated UploadPartCopy checksum.
Adversarial review: local Claude Code, Fable 5, --effort max; final verdict GO, with no blocking findings.

Too Long; Didn’t Read (TL;DR)

A multipart upload splits a large file into smaller parts. A client may attach a checksum to each part so the server can verify the transfer, but AWS defines that checksum as optional. SILO used to treat it as mandatory: an ordinary UploadPart failed without one, and UploadPartCopy could never work because it has no part-body checksum to provide.

After the repair, SILO still validates a checksum when the client sends one. When the client omits it, SILO computes the checksum while reading the original bytes and saves the result. This happens before compression and encryption, requires no second read, and changes no on-disk format. The result is AWS-compatible behavior without weakening data integrity.

Decision

When a multipart upload declares a checksum algorithm in CreateMultipartUpload, SILO applies this contract:

1. If the client supplies a part checksum, the server continues to validate it. A wrong value or algorithm fails and is never hidden by fallback computation.
2. If the client omits the part checksum, the server computes it in one pass with the MPU algorithm over the logical plaintext stream, before compression and encryption, and persists the result.
3. A normal UploadPart echoes a checksum response header only when the client supplied the checksum. A server-computed fallback is not echoed.
4. UploadPartCopy has no client part-body checksum, so the server computes the value and returns it in CopyPartResult.
5. ListParts returns the persisted part checksum.
6. FULL_OBJECT completion continues to linearize the full checksum from stored part checksums. COMPOSITE completion continues to require a checksum for every part; clients can recover those values with ListParts.
7. Computation occurs during the existing read. Completion never re-reads the entire object merely to manufacture missing state.

In one sentence:

The optional input is the client-provided checksum value, not the server’s responsibility to maintain a consistent checksum-enabled MPU.

How we found it

The defect surfaced while investigating a different multipart checksum issue, #31.

#31 concerned CompleteMultipartUpload: for FULL_OBJECT, a client can complete with part numbers, ETags, and an optional full-object checksum without retaining every part checksum in the completion XML. Tracing that path backward exposed a stronger, earlier condition in erasureObjects.PutObjectPart:

if cs := fi.Metadata[hash.MinIOMultipartChecksum]; cs != "" {
    if r.ContentCRCType().String() != cs {
        return InvalidArgument{/* checksum missing */}
    }
}

Once an MPU declared a checksum algorithm, every UploadPart had to carry the matching x-amz-checksum-* value. Omitting it returned:

400 InvalidArgument:
checksum missing, want "CRC32", got ""

API-level probes reproduced the behavior on both the single-drive and erasure backends.

Reviewing CopyObjectPartHandler raised the severity from a client-configuration incompatibility to P0. UploadPartCopy has no request body for the caller to checksum. The handler reads the source object, constructs an internal reader, and eventually enters the same PutObjectPart implementation. There is no client header and no SDK setting that can repair the request. Every checksum-enabled MPU therefore rejected UploadPartCopy by construction.

What AWS requires

This cannot be decided by saying that MinIO has historically behaved a certain way. The S3 protocol is the authority.

The AWS UploadPart API describes each algorithm-specific checksum header as something that “can be used as a data integrity check.” More importantly, its response fields say that the checksum is present only when it was provided in the request.

The AWS UploadPartCopy API is different: when the MPU was created with an algorithm, the copy result contains that part checksum. There is no copy request body, so this is necessarily a server-computed value.

The AWS ListParts API is the standard way to recover checksums for parts in an upload that is still in progress.

The algorithm/type matrix also rules out treating the repair as one Boolean flag:

FULL_OBJECT is limited to CRCs that can be linearized, but SHA1 and SHA256 still need correct per-part digests for COMPOSITE completion.

SDK configuration makes the gap practical. Current AWS SDKs usually calculate request checksums when an operation supports them, but users can choose request_checksum_calculation = when_required, and low-level callers can initiate an algorithm without repeating it on every part. S3 accepts those requests; SILO did not.

Why removing the check is not a fix

The most tempting patch is to delete the comparison and allow a checksum-less part to proceed. That only moves the failure to completion.

SILO does not reconstruct and re-read all object bytes during MPU completion. It reads ObjectPartInfo.Checksums from each part.N.meta:

• a missing entry immediately becomes InvalidPart;
• FULL_OBJECT calls Checksum.AddPart, combining digests with their part lengths;
• COMPOSITE concatenates the raw digest bytes and hashes them into the object checksum.

The actual invariant is therefore:

checksum-enabled MPU
        => every committed part has a checksum for the MPU algorithm

Deleting the upload check without filling the metadata would make UploadPart appear successful, leave ListParts incomplete, omit the UploadPartCopy response value, and fail later during completion. A delayed failure is harder to diagnose than the original immediate one.

Alternatives considered

What the commercial edition taught us

We downloaded and verified the then-current MinIO AIStor RELEASE.2026-08-07T18-34-35Z. Without a commercial license the server enters offline mode and denies S3 operations, so the evidence came from Go pclntab and ARM64 disassembly, not a black-box compatibility run.

The static analysis showed that AIStor already:

• installs a server hasher when the client checksum is absent;
• persists the result in part metadata;
• exposes checksum fields in CopyPartResult.

It nevertheless applies fallback only to CanMerge() algorithms—CRC32, CRC32C, and CRC64NVME. SHA1/SHA256 COMPOSITE still follows the old checksum missing path. More importantly, the hasher is attached in the object layer to the current r.Reader; under compression or encryption that reader may already represent transformed storage bytes.

AIStor validated the general direction—compute and store—but not an implementation that SILO could copy mechanically.

How adversarial review overturned the first design

The first plan tried to centralize every decision inside erasureObjects.PutObjectPart: read the MPU metadata in the object layer and install a server hasher when the incoming reader had no client checksum. It looked attractive because all internal callers would share one rule.

The first Fable 5 Max adversarial review found that this design was wrong for compression.

newS2CompressReader is not a lazy wrapper. Construction immediately launches a goroutine:

go func() {
    _, err := io.Copy(comp, r)
    // ...
}()

The S2 writer also reads several blocks concurrently. After constructing the compressor, the handler still performs option parsing, encryption preparation, and the object-layer call. By the time PutObjectPart installed a hasher, the plaintext reader could already have lost several MiB:

• a large part would get a checksum with a missing prefix;
• a small part could reach EOF before installation and produce no result;
• mutating ServerSideHasher concurrently with Read would be a data race.

That finding changed the responsibility split:

The handler installs the hasher before any eager transform starts; the object layer validates the algorithm, requires a result, and persists it atomically.

This was the decisive turn in the design. Putting logic in the lowest layer may look more uniform, but stream correctness depends equally on when bytes begin moving and which representation of those bytes a layer can see.

Final implementation

A dedicated logical checksum reader

PutObjReader originally distinguished two concepts:

• Reader, the stream sent to storage, possibly compressed or encrypted;
• rawReader, used by older ETag and checksum code.

Under compression, even rawReader may not directly see plaintext; it can merely carry an ETag through an etag.Tagger chain. The repair therefore did not overload it. It added an unexported field:

checksumReader *hash.Reader

This reader always represents the logical S3 part bytes. WithEncryption can replace the storage Reader, but it must preserve checksumReader.

Unexported accessors on PutObjReader then:

• return the effective client or server checksum type;
• prefer the client value whenever it exists;
• otherwise return the server result finalized at EOF.

Keeping the mechanism unexported minimizes public Go API growth and gives #63 a shared internal path without prematurely changing ordinary CopyObject behavior.

Preparing the hasher before transformations

prepareMultipartChecksumReader loads the algorithm and checksum type saved with the MPU:

1. no declared algorithm means no work;
2. an existing client checksum is compared by base algorithm;
3. a wrong algorithm preserves the InvalidArgument rejection;
4. an omitted client checksum installs the corresponding server hasher on the plaintext reader.

For normal UploadPart:

• the compressed path prepares actualReader after request-checksum parsing but before newS2CompressReader;
• the uncompressed path prepares the request hash reader before the encryption reader is constructed.

For UploadPartCopy:

• a checksum-enabled MPU first gets an inner hash reader over the logical source range;
• a range copy hashes only the selected bytes;
• compression and destination encryption start only after that reader is ready.

The object layer remains authoritative

Early handler preparation does not replace the storage invariant. erasureObjects.PutObjectPart still:

• re-parses the expected MPU algorithm;
• requires an effective checksum type that matches;
• obtains the checksum map after erasure encoding finishes;
• reports an internal error instead of committing if an enabled algorithm has no result;
• writes the checksum with the ETag, sizes, and index into part.N.meta, then atomically renames the part.

An internal caller that bypasses the HTTP handler without preparing a valid checksum is therefore rejected just as before. It cannot silently commit a part that violates the MPU invariant.

CopyPart response shape

CopyObjectPartResponse gained the five algorithms supported by this source tree:

ChecksumCRC32
ChecksumCRC32C
ChecksumCRC64NVME
ChecksumSHA1
ChecksumSHA256

All are omitempty, so an MPU without checksums produces the old XML. Normal UploadPart still uses the existing TransferChecksumHeader and echoes only a client request value; fallback computation does not alter that response.

Why it works

After the repair, the data flow is:

logical plaintext part
        |
        +--> client checksum verifier (if supplied)
        |         or
        +--> server-side hasher (if omitted)
        |
        v
compression (optional)
        |
        v
encryption (optional)
        |
        v
erasure encode / storage
        |
        v
persist ETag + size + logical part checksum atomically

This satisfies four requirements that previously appeared to conflict:

1. Protocol compatibility: omitting an optional header succeeds.
2. No integrity downgrade: a supplied client value is still checked end to end and is never hidden by server fallback.
3. Correct object semantics: the checksum covers logical S3 bytes, not compressed data or ciphertext.
4. Controlled cost: hashing shares the existing read and adds CPU, not a second disk or network pass.

EOF has a precise role. hash.Reader finalizes ServerSideChecksumResult only when it reaches EOF. Closing the compression pipe synchronizes the compressor goroutine with the storage read; the object layer reads the result only after encoding returns. Targeted -race tests verified that concurrency boundary.

The compatibility-baseline blocker

The five new CopyObjectPartResponse fields are exported Go API. SILO’s buildscripts/rebrand-guard rescans imports, environment variables, headers, routes, storage markers, and exported symbols, then compares them in both directions with buildscripts/rebrand-guard/compat-baseline.json. An unacknowledged symbol makes CI fail.

After recording the five #46 fields, the guard still reported two additions:

internal/config/notify:notify:type:LegacyDatabaseTargetError
internal/config/notify:notify:method:LegacyDatabaseTargetError.Error

They did not come from #46. They belong to the earlier database-notification repair f1ba68358 on the local main branch. The cmd startup path intentionally needs the exported type for errors.As, but that earlier commit had not updated the compatibility baseline. Every later change based on that HEAD would therefore fail the CI guard.

We chose “option A”: acknowledge the two notification symbols as part of their original repair while retaining the five #46 fields. The final baseline diff is exactly seven additions and zero deletions, and the guard reports:

exported=9021
Silo rebrand compatibility baseline is unchanged

This does not disable the check. Exact set equality means that acknowledging a nonexistent symbol also fails. The change explicitly records two intentional compatibility-surface additions.

golangci-lint has not yet run locally; it remains a remote go.yml gate. Green local go test, go vet, race, and rebrand-guard results do not substitute for green remote CI.

Verification evidence

The new tests execute 76 subtests across:

• CRC32, CRC32C, and CRC64NVME FULL_OBJECT;
• CRC32, SHA1, and SHA256 COMPOSITE;
• correct client checksums, wrong algorithms, and wrong values;
• absence of a server-computed checksum in normal UploadPart responses;
• server values in UploadPartCopy responses and ListParts;
• a real 5 MiB + 1 KiB two-part full-object merge;
• zero-length parts and overwriting the same part number;
• a range copy whose SHA256 covers only the copied interval;
• single-drive and 16-drive erasure backends;
• default, versioned, compressed, encrypted, and compressed-plus-encrypted modes;
• explicit SSE-C and SSE-S3.

Local validation included:

go test -race ./cmd -run '^TestAPIUploadPartServerSideChecksum' -count=1
go test ./cmd -count=1
go test ./... -count=1
go vet ./cmd
git diff --check
go run ./buildscripts/rebrand-guard

All passed. Two subsequent Claude Code Fable 5 Max implementation reviews and the final acceptance review returned GO with no blocking findings.

Cost, risk, and release boundary

When a client omits its value, the server performs one additional hash over the part. CRC cost is small; SHA costs more CPU. Both share the read that already had to occur, without buffering an entire part in memory or adding a completion-time second pass.

During a rolling upgrade, old and new nodes may answer the same checksum-less request differently: a new node accepts it while an old node returns 400. ObjectPartInfo.Checksums did not change format, so stored data remains downgrade-readable, but client-visible behavior stabilizes only after all serving nodes have upgraded. The release note must call that out.

This record describes a local main worktree. The implementation has not been committed, pushed, run through remote CI, or packaged into a release. SILO documentation belongs to silo.pgsty.com; a successful local Hugo build does not mean that the product in the wider pgsty.com ecosystem has shipped.

Why two follow-ups remain separate

Adversarial review found two related but independent issues.

#63: CopyObject + compression

Ordinary CopyObject can also attach a server-side checksum to a transformed stream. It shares the root cause and the new checksumReader mechanism, but it is a different API with a different test matrix and rollback boundary. We chose a separate repair and require that PR to reuse this plaintext-reader contract instead of inventing a second abstraction.

#64: legacy federation

Legacy etcd federation turns UploadPartCopy into an ordinary remote UploadPart. Under the AWS response semantics preserved here, that remote request does not return a server fallback value, so the proxy may still lack the checksum required for CopyPartResult. A follow-up must independently choose between a remote-returned value and an ETag-verified ListParts fallback. It must not make all external UploadPart responses non-compliant merely to simplify an internal proxy.

Separating them does not abandon consistency. Consistency is maintained through one shared rule:

Every server-computed S3 checksum binds to the logical plaintext stream, is installed before any eager transform, and is validated and persisted by the object layer that owns the storage invariant.

Lessons retained

The repair leaves lessons more durable than its individual lines of code:

1. An optional header does not make internal state optional. If the protocol lets the client omit a value, the server must produce the state its own completion path needs.
2. Request acceptance and response disclosure are separate contracts. A normal UploadPart may compute internally and still omit the value; UploadPartCopy must return it.
3. Stream layers are defined by byte semantics. The lowest layer is not automatically correct if it no longer sees logical bytes, and an eager goroutine turns “install later” into a race.
4. A commercial implementation is evidence, not the specification. AIStor showed the direction and the boundary that could not be copied.
5. A compatibility guard is a change-acknowledgment mechanism. compat-baseline.json exists to assign every new compatibility surface, not merely to make CI quiet.
6. Independent defects should ship independently while sharing invariants. #63 and #64 remain separate, but both must cite and obey the checksum-reader contract established here.

The final result is not a broad relaxation. It is a stricter and more accurate boundary: clients may omit optional information; the server may not omit correctness.

21 - BadDigest, InvalidRequest, and the CompleteMultipartUpload Checksum Contract

This is the design, investigation, and verification record for SILO #48, with the decision boundary for the related SILO #50.

Status: pgsty/silo#74 merged as 590aeaa7d, and pgsty/silo.pgsty.com#6 merged as 9805dd7; full local verification, remote CI, and independent Opus 5 Max acceptance review completed on the linked changes. Tag, release, package, image, deployment, and production verification remain separate pending gates.
2026-08-28 follow-up: signed-off server commit 7e079ff05 closes the remaining type-only and invalid-token bypass without changing CRC64NVME canonicalization. Complete local, tagged, race, static, build, and Fable Max verification passed; it was merged into main on 2026-08-29, and tag and delivery remain pending.
2026-09-02 update: the CRC64NVME exception described below no longer holds. main now rejects CRC64NVME combined with COMPOSITE at multipart initiation, in trailers, and at completion with InvalidArgument (d28885d0e, d4c8da162, 32b2aa49f), resolving pgsty/silo#50 by rejection rather than canonicalization. The reasoning below is kept as the record of the earlier decision.
Owner: pgsty/silo, the SILO server repository.
Implementation scope: CompleteMultipartUpload error semantics only; no storage-format, checksum-math, dependency, Console, package, or client change.
Independent decision: #50 remains probe-gated and is not part of this repair.

Too Long; Didn’t Read (TL;DR)

Issue #48 is valid and should be fixed, with two corrections to the original report.

First, the checksum-type comparison is worse than the issue states. SILO used bitmask containment instead of equality. An upload created as FULL_OBJECT and completed as COMPOSITE failed, but the reverse COMPOSITE to FULL_OBJECT direction could pass the type check. The repair must compare the base algorithm and normalized multipart object type independently and symmetrically.

Second, the missing-part-checksum row originally lacked a direct AWS capture. That evidence now exists in the official boto/s3transfer project: issue #241 records a real S3 InvalidRequest response naming sha256 and missing part 1, and PR #242 repaired the client and added tests. This is strong enough to implement the response contract without a new AWS account probe.

The accepted behavior is:

The repair uses completion-specific error types. It deliberately does not change the global mapping of hash.ChecksumMismatch, so PutObject, UploadPart, streaming trailers, and other operations retain their existing XAmzContentChecksumMismatch contract.

Issue #50 is a separate question. AWS documents that CRC64NVME is full-object only, but the available sources do not prove that S3 rejects an explicit CRC64NVME + COMPOSITE initiation instead of canonicalizing it. Upstream MinIO intentionally implemented canonicalization and exposes FULL_OBJECT in the initiation response, so the behavior is not silent. A raw AWS probe is required before changing it.

Scope and decision

This record answers two different questions:

1. Are the #48 error-code deviations real, externally observable compatibility defects with enough evidence to repair?
2. Does the same evidence authorize changing the CRC64NVME canonicalization described by #50?

The decisions are:

• #48: accept with corrections and implement. The error codes are part of the S3 wire contract. Returning a different code makes SDK behavior and operator diagnosis diverge even when the request is rejected in both systems.
• #50: do not implement yet. The capability matrix proves the resulting checksum must be full-object. It does not establish whether an invalid requested type is rejected, ignored, or canonicalized. Those are different wire contracts.

The repair is intentionally narrow. It does not add algorithms, recalculate stored data, reinterpret successful uploads, or change the optionality rules repaired for #31 and #46.

Evidence ledger

Not all evidence has the same authority. The implementation decision uses the following hierarchy.

This distinction matters. The original #48 comment correctly downgraded the third row while it was supported only by secondary reports. The boto transcript and the merged client repair close that evidence gap.

The observable contract

Object checksum mismatch

For a FULL_OBJECT multipart upload, SILO combines stored part checksums and compares the result with the optional object checksum supplied on completion. The old code returned hash.ChecksumMismatch. A global API mapping converted that type to:

400 XAmzContentChecksumMismatch

AWS explicitly documents BadDigest for the corresponding completion integrity failure. Reusing the existing generic ErrBadDigest code without a custom message would still be misleading because its static text says Content-MD5; CRC32, CRC32C, and CRC64NVME are not Content-MD5.

The new response is therefore operation-specific:

400 BadDigest
The CRC32 checksum you specified did not match the calculated checksum.

The response does not disclose the expected or supplied digest.

Checksum type mismatch

The checksum type saved by CreateMultipartUpload is part of the upload’s contract. A completion may not switch between COMPOSITE and FULL_OBJECT.

The old test was:

!provided.Type.Is(expectedType)

ChecksumType.Is is a containment operation over a bitmask, not equality. For CRC32:

created FULL_OBJECT + completed COMPOSITE => rejected
created COMPOSITE   + completed FULL_OBJECT => containment succeeds

The second request could proceed using the persisted composite rules. If the caller supplied the composite checksum value under a FULL_OBJECT declaration, completion could even succeed. This is a protocol validation bypass, not merely the wrong error label.

The repair normalizes both values into multipart checksum types, then compares:

1. base algorithm equality; and
2. object type equality (COMPOSITE versus FULL_OBJECT).

For algorithms whose two object-type forms are both syntactically accepted—currently CRC32 and CRC32C—both mismatch directions now return 400 BadDigest. SHA1 and SHA256 with FULL_OBJECT are rejected earlier as InvalidArgument; CRC64NVME is the canonicalized special case discussed under #50 below. Base-algorithm mismatch remains a separate InvalidArgument path because #48 and the cited AWS type contract do not authorize broadening that behavior.

Type-only assertions and invalid tokens

The first #48 repair remembered whether x-amz-checksum-type was present, but its object-layer comparison was still nested under WantChecksum != nil. WantChecksum is populated only when completion carries a checksum value. A caller could therefore send a type assertion without a whole-object checksum:

CreateMultipartUpload:   CRC32 + COMPOSITE
CompleteMultipartUpload: x-amz-checksum-type: FULL_OBJECT
                         no x-amz-checksum-crc32 value

The server returned success and persisted the initiated composite state. It did not corrupt the object, but it accepted an explicit integrity assertion that contradicted the upload contract.

There was a second parser asymmetry. In the header-without-algorithm path used by completion, an unknown value such as NOT_A_TYPE could be ignored when a checksum header was also present. Relying on ChecksumType.ObjType() after creating an invalid bitmask would not be safe: an invalid non-multipart value can fall through to the full-object default. Raw enum validation must happen first.

The follow-up stores the explicit raw type string in ObjectOptions, accepts only COMPOSITE or FULL_OBJECT, and compares it with the initiated multipart type independently of WantChecksum. The order is deliberate:

1. reject every unknown non-empty token as InvalidArgument;
2. compare the base algorithm when a checksum value is supplied;
3. compare the explicit object type whenever the upload recorded a checksum algorithm;
4. report an explicit type mismatch as BadDigest even when no object checksum value was supplied.

CRC64NVME remains a deliberate exception. A raw COMPOSITE token is normalized to FULL_OBJECT before comparison, preserving the inherited behavior pending the #50 AWS probe. A legal type-only header on an upload that recorded no checksum algorithm remains outside the comparison because there is no initiated checksum type to assert against; its exact AWS error semantics remain unproven and were not expanded into this repair.

Missing composite part checksum

For a composite upload, the completion XML must include the selected checksum for every listed part. SILO previously compared an empty client value with the stored part checksum and returned InvalidPart.

That conflated three different states:

• the part or ETag does not exist;
• a checksum was supplied but has the wrong value or algorithm;
• the required checksum element is absent.

The third state now has a dedicated error. Its wire message follows the AWS response captured by boto/s3transfer:

400 InvalidRequest
The upload was created using a sha256 checksum. The complete request must include
the checksum for each part. It was missing for part 1 in the request.

The error is emitted for the first missing part and includes its actual number. FULL_OBJECT behavior is unchanged: a completion may omit per-part checksum elements, while any supplied part checksum must still be valid.

Root cause in the upstream lineage

The behavior is inherited rather than a SILO-specific redesign.

• MinIO PR #15433 introduced extended checksum handling and the global hash.ChecksumMismatch to XAmzContentChecksumMismatch mapping. That mapping is suitable for streaming upload validation but too broad for completion semantics.
• MinIO PR #20855 added full-object checksums and CRC64NVME. It introduced the checksum-type comparison and intentionally canonicalized CRC64NVME to full-object with the comment that AWS appears to ignore the supplied mode.
• MinIO PR #20953 tightened invalid algorithm/type combinations but retained the CRC64NVME special case. That is evidence of deliberate upstream behavior, not an accidental missing branch.
• MinIO issue #20944 reported an AWS BadDigest versus MinIO InvalidPart difference. The divergence was acknowledged but not repaired.

The upstream repository is now archived. SILO therefore owns the compatibility decision, tests, and maintenance burden rather than waiting for an upstream correction.

Repair design

Operation-scoped errors

Changing the global hash.ChecksumMismatch mapping would alter every operation that uses it. That would be a larger, weakly evidenced compatibility change.

The repair adds three package-private, sentinel-backed error helpers in the server command package. Keeping the helpers and the request-header-presence flag private avoids expanding SILO’s exported Go compatibility surface:

• completeMultipartChecksumMismatch, mapped to BadDigest with a checksum-aware description;
• completeMultipartChecksumTypeMismatch, mapped to BadDigest with provided and initiated types;
• missingPartChecksum, mapped to InvalidRequest with algorithm and part number.

Only CompleteMultipartUpload produces these types. The global mapping remains:

hash.ChecksumMismatch => XAmzContentChecksumMismatch

This preserves PutObject and UploadPart behavior and makes the compatibility boundary visible in code.

Symmetric type validation

Both persisted and supplied types are normalized with the multipart flags before comparison. This is necessary because a bare CRC checksum type describes a non-multipart full-object checksum through ObjType(), while the same base value means composite after multipart context is applied. Object type is compared only when the completion request explicitly contains x-amz-checksum-type; omitting an optional header does not synthesize a COMPOSITE assertion.

The resulting invariant is:

provided base algorithm == initiated base algorithm
AND
provided multipart object type == initiated multipart object type

The second condition applies only to an explicitly supplied type. This comparison is symmetric and remains compatible with the existing CRC64NVME canonicalization. It fixes #48 without silently deciding #50.

Precise missing-value detection

For each part, the server already builds a map of all checksum fields supplied in the completion XML. The repair distinguishes:

COMPOSITE + no checksum field supplied => missingPartChecksum / InvalidRequest
expected field present but wrong value => InvalidPart
only a different algorithm supplied    => InvalidPart
FULL_OBJECT + no checksum field         => allowed
FULL_OBJECT + any checksum field        => validate it

This is intentionally narrower than converting every part checksum failure to InvalidRequest. Only the state demonstrated by AWS evidence changes.

The same edit corrects the internal InvalidPart expected/actual field order. The generic S3 InvalidPart wire response did not expose those digest values, but internal error text and logs should still describe them correctly.

Regression and detection matrix

The API-level tests exercise signed HTTP requests through both the single-drive and erasure object-layer backends.

The committed type-mismatch regression uses CRC32, while an independent acceptance probe covered CRC32C as well. The follow-up additionally covers type-only, unknown, lowercase, matching, and checksum-bearing invalid-token cases. The same matrix confirms that SHA1/SHA256 FULL_OBJECT requests stop earlier at the existing invalid-combination check and that CRC64NVME still canonicalizes an explicit COMPOSITE token. Those distinctions are protocol boundaries, not untested claims that every algorithm reaches the same error mapper.

Focused verification command:

go test ./cmd -run 'TestAPIErrCode$|TestAPICompleteMultipart(FullObjectChecksumMismatch|CompositeStillRequiresPartChecksums|CompositeChecksumMismatch|ChecksumTypeMismatch)$|TestAPIUploadPartServerSideChecksumDoesNotMaskClientErrors$' -count=1

Observed result on 2026-08-27:

ok  github.com/minio/minio/cmd

The complete local package gate was then rerun after the review-driven additions:

go test ./cmd ./internal/hash -count=1

ok  github.com/minio/minio/cmd           121.462s
ok  github.com/minio/minio/internal/hash   0.566s

git diff --check also passed. Independent review of the final diff remains a separate gate. A local pass is not remote CI, a merged commit is not a release, and a release is not production deployment.

Independent adversarial review

The first review of the actual server diff was performed with local Claude Code in read-only safe mode. Its verdict was GO with no blocking findings. It independently confirmed the operation-scoped mapping, symmetric bitmask normalization, per-part missing-value detection, both object-layer backends, and preservation of UploadPart behavior.

The review identified four useful gaps that were incorporated before the second full test run:

• distinguish an omitted optional type header from an explicit COMPOSITE assertion;
• separate value-mismatch and type-mismatch error types;
• exercise the composite checksum-of-checksums mismatch path;
• pin missing part 2, algorithm mismatch, and unchanged CRC64NVME canonicalization.

One first-review concern was rejected by primary evidence: it questioned whether checksum type mismatch should return InvalidRequest. The AWS CompleteMultipartUpload reference and AWS CLI reference explicitly specify BadDigest when the completion type differs from initiation.

The final first-round re-review verdict was FINAL GO, no blockers. It explicitly withdrew the earlier error-code concern, agreed with accepting #48 and deferring #50, verified that the new guards preserve the intended non-changes, and found no English/Chinese drift.

A subsequent independent acceptance run used Claude Code claude-opus-5 with maximum effort. It returned ACCEPT, no blocking findings, reproduced the old composite-as-FULL_OBJECT bypass end to end against the pre-fix code, verified that the new API assertions fail against that code, and probed all five checksum algorithms in both type directions.

The 2026-08-28 follow-up received a separate local Fable Max mirror review over the complete uncommitted release-review diff. It returned GO, with no P0–P2 findings. The primary review independently checked its seven P3 observations: five were non-blocking boundaries, while two proposed causes were disproved by the actual config and key-rotation call paths. The review confirmed that raw invalid types are rejected before normalization, type-only mismatch is enforced, source-side checksum decryption still receives the full request, and CRC64NVME canonicalization remains untouched.

Five pre-existing or deliberately deferred, non-blocking observations remain outside these repairs:

• SHA1/SHA256 FULL_OBJECT combinations are rejected by the existing parser as InvalidArgument before the new type-mismatch mapper; only CRC32/CRC32C reach both mismatch directions;
• CRC64NVME treats any type value as full-object state, so completion with an explicit COMPOSITE token is still accepted through canonicalization pending the #50 AWS probe;
• when initiation recorded no checksum algorithm but completion supplies an object checksum, SILO returns BadDigest; AWS documentation says such a value is accepted and ignored, so this should be triaged as a separate compatibility issue;
• composite part-count and value mismatches both become BadDigest with the same description;
• a full-object checksum carrying a -N suffix has that suffix ignored while its digest is still validated.

None is introduced by these patches, and none changes the #48 decision. They should be triaged separately if strict message or invalid-header parity becomes a maintenance priority.

Why #50 is not included

Issue #50 says CRC64NVME + COMPOSITE should be rejected at initiation. Three facts are confirmed:

1. AWS’s algorithm matrix supports CRC64NVME only as a full-object checksum.
2. SILO and upstream MinIO canonicalize the request to full-object state.
3. The server returns x-amz-checksum-type: FULL_OBJECT from CreateMultipartUpload, so the substitution is externally visible rather than silent.

What is not confirmed is the decisive wire behavior: does AWS reject the explicit invalid combination, or accept it and return/carry full-object state? A capability matrix does not answer that question.

The upstream history also argues against guessing. PR #20855 added the canonicalization intentionally, and PR #20953 preserved it while tightening other invalid combinations. That may be based on an AWS observation, but the comment is not a reproducible transcript.

The same representation also affects completion: FullObjectRequested treats every CRC64NVME checksum as full-object state, so a stored FULL_OBJECT upload completed with the raw header value COMPOSITE is accepted as full-object rather than rejected as a type mismatch. This completion-side residue falls under the same raw-token-versus-canonical-state evidence question. It is explicitly not claimed fixed by #48.

PutObject must not be bundled into this decision. Its API reference does not define x-amz-checksum-type, so accepting, rejecting, or ignoring that header is a separate undocumented-header question.

Required AWS probe

Before changing #50, capture a raw SigV4 request and response against a general-purpose AWS S3 bucket:

1. send CreateMultipartUpload with x-amz-checksum-algorithm: CRC64NVME and x-amz-checksum-type: COMPOSITE;
2. record the HTTP status, error code/message, request ID, and all checksum response headers;
3. if accepted, upload one part and complete it, recording whether S3 requires per-part values and which type HeadObject reports;
4. repeat with FULL_OBJECT as the control;
5. probe PutObject separately, explicitly labeling it as an undocumented-header experiment.