Never resend a CAS blob's object-creating S3 request

[filimonov]

On S3-compatible stores the ETag of a single PUT or CopyObject is derived from the bytes. When the SDK resent a CAS blob publication after the first attempt had already landed, and GC had condemned and deleted that incarnation in between, the resend brought back the old ETag. A delayed exact-ETag DELETE from GC then removed a blob that a newer writer had just reused, leaving a committed reference without data. Blob publication now sends each object-creating request at most once; the CAS publish loop retries with a fresh envelope instead.

Changes

• S3ObjectStorage::copyObject honours the write settings' retry override, as writeObject already does.
• New WriteSettings::object_storage_single_attempt_object_creation: PutObject, CopyObject and CompleteMultipartUpload go out once, with adaptive first-attempt timeouts off (there is no second attempt). CreateMultipartUpload, UploadPart and UploadPartCopy keep their retries, so large multipart blobs still survive throttling. The flag also bounds the NO_SUCH_KEY reissue loops of WriteBufferFromS3 and copyS3File to one request (applyWriteErrorRetryOverride). A copy with the flag must use ObjectStorageCopyMode::NativeOnly; copyObject refuses anything else with BAD_ARGUMENTS.
• Client::CompleteMultipartUpload on a single-attempt client no longer reports NoSuchUpload as success when some object exists at the key, and skips the GCS compose (a second creating request; off by default).
• A failed single-attempt creating request is not logged at Error: WriteBufferFromS3 logs it at Debug, copyS3File logs its multipart failure at Debug, and an abort that finds no upload after a lost completion is not an error on either path. A NO_SUCH_KEY loop no longer announces "will retry" on its last attempt.
• CAS blob publication (ObjectStorageBackend::publish) sets the flag. PartWriteTxn::ensureBlobPresent logs each failed attempt at Warning (rate-limited) and counts it in CASBlobPublishAttemptFailed; CASBlobPublishGaveUp counts publications that ended with an exception after a failed attempt, at the attempt cap or the deadline.

Commits 1–3 and 6 contain no CAS code; commits 4 and 7 wire the flag into CAS; commit 5 reworks the tests; commit 8 updates the CAS docs; commits 9 and 10 address review follow-ups (abort log on the copy path, the give-up counter, test trims, wording).

Relation to the earlier retry work

• d528693 (2026-09-05), on top of the adc0fa7 engine, forbade the CAS engine from reissuing a blob publication and recorded this hazard. The transport below stayed on the retrying client: the SDK retry strategy, Client::doRequestWithRetryNetworkErrors and the NO_SUCH_KEY loops.
• The only single-attempt tool then was the control-plane write settings: the SingleAttempt retry profile plus conditionalWriteSettings, with a 5 s attempt budget and a single PUT on GCS, which 940b168 had just removed for blobs. The profile alone swaps the client for the whole buffer, so one lost UploadPart loses the whole multipart body.
• This PR adds the missing per-verb split: creating requests go out once, parts keep their retries. It completes that work rather than fixing a mistake in it.
• The "fresh envelope" argument of 940b168 holds per engine attempt, not per wire request: an SDK resend carries the same bytes.
• The CaBlobPublishCore model had no resend below the engine until the 2026-10-12 audit added SdkResend.

Risks / notes

• Retries are not removed, they move up one level. Before: the S3 SDK and Client::doRequestWithRetryNetworkErrors resent the same request (same bytes, same ETag). Now a failed or ambiguous creating request fails the attempt, and PartWriteTxn::ensureBlobPresent retries it. Each attempt first re-observes the key (HEAD of the body, then the marker) and, if the blob is still needed, publishes a new envelope with a fresh incarnation_tag, so it can never recreate a condemned ETag. A failed CopyObject of a staged blob is retried as a streaming upload (the staged copy is used once). Definite refusals (e.g. access denied) are not retried, as before.
• Retry budget: a creating request runs under the disk's request_timeout_ms, with its connect timeout capped like the control plane's. The loop makes up to 8 attempts and starts no new one after the 90 s deadline (Retry::standard), then the insert fails with a retryable error. An attempt already started can run past the deadline. A failed CompleteMultipartUpload or CopyObject re-uploads the whole body.
• Mixed versions: the fix applies per writer node. A node without it can still resend a blob PUT it publishes.
• The CompleteMultipartUpload change reaches every single-attempt clone, including the CAS control-plane conditional writes. It is more conservative there: NoSuchUpload becomes an S3 error, the write classifies as unresolved, and the engine settles it with a read. Reporting another writer's object as this upload's success was wrong for them too.
• Residuals: CreateMultipartUpload, UploadPart and AbortMultipartUpload on the failure path keep the disk client's retries (about 500 by default), so one engine iteration can block much longer than 90 s; a follow-up measures and bounds that tail. A CompleteMultipartUpload slower than 30 s on RustFS/MinIO was previously rescued by a retry plus the NoSuchUpload → HEAD check; it now costs a re-upload. Neither is measured.
• Part requests (CreateMultipartUpload, UploadPart, UploadPartCopy) keep their SDK retries unchanged.
• Resends after a definite refusal (credentials refresh, region redirect, HTTP 307) are kept; they cannot duplicate an applied request.
• Generation-token stores (GCS) were not affected by the loss, only by the NoSuchUpload rewrite.

Testing

• New suites, red before their fix: S3ObjectStorageCopyObject, S3SingleAttemptObjectCreation (lost responses and NO_SUCH_KEY per creating verb, no Error log for a lost response or for the abort after a lost completion (write and copy), no "will retry" on the last NO_SUCH_KEY attempt, a broken response body, part/create/part-copy retries kept, EntityTooLarge → one multipart copy, adaptive timeouts off on the creation clone plus one slow PUT, default writers still resend), S3ClientSingleAttemptCompletion.
• CAS: CASBlobPublishTransport.LostCompleteIsOneWireRequest drives the backend directly, and two composition tests run the real S3ObjectStorage under the CAS publication loop against a storing S3 stub: a lost PUT is republished after a new HEAD with a different incarnation_tag; a lost staged CopyObject is republished by streaming. CASBackend.ConditionalMultipartNoSuchUploadIsUnresolved pins the control-plane effect.
• Gates on the final tree: release CAS* 2627/2627, ASan CAS* 2631/2631; S3ObjectStorageCopyObject*:S3SingleAttempt*:S3ClientSingleAttempt* release 33/33, ASan 34/34 (includes the death test).
• TLA+: CaBlobPublishCore reproduces the loss with an SDK resend of an identical envelope on content-ETag stores and is green with single-attempt creation; generation stores are green either way.

Changelog category (leave one):

• Critical Bug Fix (crash, data loss, RBAC)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fixes a rare data loss on CAS (content-addressed storage) disks backed by S3 stores with content-derived ETags (GCS is unaffected): a retried blob upload could bring back a blob that garbage collection had already scheduled for deletion, and the delayed deletion then removed a blob still in use. Blob uploads no longer resend the request that creates the object; CAS retries with a new upload instead. Under a store outage an insert may now fail with a retryable error after 8 attempts or once the 90-second deadline for starting attempts passes.

Documentation entry for user-facing changes

No user-facing setting; the new WriteSettings field is internal. The CAS docs (blob-protocol.md, backend.md, correctness.md, configuration.md, troubleshooting.md) and the Antalya profile events page describe the new behaviour and counters.

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [fc72b78]

[k-morozov]

Model found:
The documentation promises a 90-second wall-clock bound that the code does not enforce.

[k-morozov]

Seems here also need to override settings + objectCreationClient.

[filimonov]

Fixed in fc72b78: the S3-to-S3 shortcut in copyObjectToAnotherObjectStorage now applies the write-error retry override and the single-attempt creation client, and requires NativeOnly like copyObject. A non-S3 destination is rejected when the flag is set (BAD_ARGUMENTS without NativeOnly, NOT_IMPLEMENTED with it). A new test checks that a lost CopyObject or NoSuchKey is one wire request.

[filimonov]

The 90 s is a deadline for starting attempts, not a wall-clock bound: no new attempt starts after it, but an attempt already started can run longer (each creating request waits up to request_timeout_ms, and part uploads keep their own retries). The docs and the PR description are corrected in fc72b78. A follow-up measures and bounds the longer tail.