diff --git a/docs/guides/native-candidate.md b/docs/guides/native-candidate.md index d99e19594..294c42a0a 100644 --- a/docs/guides/native-candidate.md +++ b/docs/guides/native-candidate.md @@ -1343,6 +1343,14 @@ For a fully ready graph whose foreground owner died while its pool remains on th same boot, `graph recover-live-owner --run-id RUN --expect-receipt SHA256` selects only that graph. It requires matching dead foreground and relay-owner identities, unchanged receipt and resource inventories, and exclusive publication locks. The +relay endpoint may also be absent after the managed owner exited: both relay files +must be absent under the original held operation lock, and the exact dead +foreground process and ready receipt must select the control root derived for +this run, runtime owner, and current guest boot. Recovery persists that absence +witness and rechecks its directory/lock identity, process death, endpoint absence, +and receipt bytes at cleanup boundaries. A partial publication, replacement +endpoint or lock, legacy unscoped root, or changed receipt is refused. This path +does not recreate relay metadata or invent a live-owner acknowledgement. The operation stops verified guest dependency listeners, removes owned containers and network resources, retains persistent data, retires stale foreground and relay publications, and releases the graph's dependency reservation before unlocking the @@ -1425,8 +1433,10 @@ boot, journal and exact artifact hashes, with no active or pending journal and a independently confirmed current retention proof. Eviction of its old stopped receipt does not require repeating completed archival. First-time or interrupted archival still requires the exact selected cleanup proof and ownership checks. -The same-boot completion proof does not authorize direct data removal: restore the -graph and use ordinary cleanup for that operation. Native routed recovery has been +An explicitly requested data removal can select a completed same-boot recovery +after publisher retirement. It pins the exact completion, current guest boot and +archived publisher identity; retries keep the selected proof source and refuse +changed evidence. This does not fabricate a relay acknowledgement. Native routed recovery has been qualified with repeated owner crashes, retained data and an unaffected sibling's HTTPS route; this does not establish normal source-mounted worktree parity. diff --git a/packages/runtime-core/README.md b/packages/runtime-core/README.md index be1a74ba5..c1d3b85cf 100644 --- a/packages/runtime-core/README.md +++ b/packages/runtime-core/README.md @@ -204,6 +204,23 @@ history format. Stable v4 uses its separate state and is unaffected. ## Named-volume integrity qualification +The ignored fixture +`foreground::native_test::same_boot_absent_relay::absent_relay_cleanup_retries_preserve_data_and_live_sibling` +uses a caller-owned running development pool and a pinned image containing +`/bin/sh` and `sleep`. Set `HACK_LOCAL_TEST_ROOT` to that private candidate home +and `HACK_LOCAL_TEST_IMAGE` to the loaded `sha256:`. Compile the +all-feature library test with `--no-run`, then invoke its exact test name with +`--ignored --exact --nocapture --test-threads=1` under a 300-second external +watchdog. It starts two unique control-only graphs, lets one managed relay exit +normally, and checks explicit same-boot recovery after interrupted intent, +container removal, completed cleanup, and publisher retirement. Persistent data +and the live sibling must survive; substituted endpoint/lock/directory, receipt +bytes, and completed-retry process evidence must refuse. It removes only its own +graph resources on success, including an interrupted explicit volume removal and +idempotent retry, and never restarts the pool. On failure, inspect its +retained graph evidence before another attempt. This fixture does not qualify +application authentication, dependency listeners, browser routing, or publication. + The ignored foreground-owner fixture `foreground::native_test::stop14::fourteen_services_retain_named_data_across_cleanup_and_restore` qualifies normal retaining cleanup of thirteen running services and one completed @@ -317,7 +334,10 @@ After confirmed foreground cleanup and clean owner retirement, explicit `graph cleanup --run-id --remove-data --json` can remove retained data. The operation holds the foreground publication lock, rechecks retirement and resource ownership, and records volume identities before removal so an interrupted attempt -can resume against the same resources. Active owners still handle their own cleanup; +can resume against the same resources. Completed same-boot recovery is supported +with exact completion and archived publisher evidence; the removal intent pins its +proof source so a retry cannot switch to a different recovery generation. +Active owners still handle their own cleanup; stale owner artifacts, uncertain cleanup and changed identities refuse removal. Ordinary volume identity is captured when removal is admitted; shared caches also retain their existing provenance and shared-resource protections. Persistent data diff --git a/packages/runtime-core/src/provider/graph/foreground/native_test.rs b/packages/runtime-core/src/provider/graph/foreground/native_test.rs index e8a7a56ab..9eabfe23c 100644 --- a/packages/runtime-core/src/provider/graph/foreground/native_test.rs +++ b/packages/runtime-core/src/provider/graph/foreground/native_test.rs @@ -552,5 +552,6 @@ mod host_pin_recovery; #[cfg(feature = "native-http-probe")] mod retired_rebind_history; mod retired_recovery_cleanup; +mod same_boot_absent_relay; mod same_boot_recovery; mod source_device_rebind; diff --git a/packages/runtime-core/src/provider/graph/foreground/native_test/same_boot_absent_relay.rs b/packages/runtime-core/src/provider/graph/foreground/native_test/same_boot_absent_relay.rs new file mode 100644 index 000000000..3c9478cfa --- /dev/null +++ b/packages/runtime-core/src/provider/graph/foreground/native_test/same_boot_absent_relay.rs @@ -0,0 +1,427 @@ +//! Opt-in production cleanup proof. Children construct real ready graphs, then +//! drop only their managed relay runtime before exiting. No receipt is fabricated +//! to manufacture the absent endpoint. Fault controls affect only owned fixtures. +use super::*; +use crate::provider::{identity::ProcessIdentity, relay_owner::publication::dead}; +use std::collections::{BTreeMap, BTreeSet}; + +const OWNER: &str = "provider::graph::foreground::native_test::same_boot_absent_relay::owner_child"; +const RECOVERY: &str = + "provider::graph::foreground::native_test::same_boot_absent_relay::recovery_child"; +const REMOVE_DATA: &str = + "provider::graph::foreground::native_test::same_boot_absent_relay::remove_data_child"; +fn candidate() -> Candidate { + Candidate::discover(Path::new(&std::env::var("HACK_LOCAL_TEST_ROOT").unwrap())).unwrap() +} +fn run() -> String { + std::env::var("HACK_LOCAL_GRAPH_RUN").unwrap() +} +fn sha(bytes: &[u8]) -> String { + format!("{:x}", Sha256::digest(bytes)) +} +fn options(project: &Path) -> project::PlanOptions<'_> { + project::PlanOptions { + branch: None, + project, + compose_file: Path::new("compose.yaml"), + profiles: &[], + } +} +#[test] +#[ignore = "Only the isolated native parent fixture invokes this owned child"] +fn owner_child() { + let candidate = candidate(); + let run = run(); + let project = PathBuf::from(std::env::var("HACK_ABSENT_RELAY_PROJECT").unwrap()); + let armed = PathBuf::from(std::env::var("HACK_ABSENT_RELAY_ARMED").unwrap()); + let plan = project::plan(&candidate, options(&project)).unwrap(); + let mut runtime = HostRelayRuntime::new_for_run( + &candidate, + Path::new("/tmp/unused-control-only-artifact"), + &"a".repeat(64), + Vec::new(), + &run, + ) + .unwrap(); + let _publication = Publication::bind(&candidate, &run).unwrap(); + let readiness = BTreeMap::from([("app".into(), graph::Condition::Started)]); + let values = BTreeMap::new(); + let receipt = graph::run_with_host_dependencies_until( + &candidate, + graph::RunOptions { + live_source: false, + shared_source: false, + release_initializer_cache: BTreeSet::new(), + routing_enrolled: false, + project: options(&project), + expected_plan: &plan.plan_id, + source_revision: None, + non_secret_values: &values, + readiness: &readiness, + run_id: &run, + timeout: Duration::from_secs(90), + }, + &mut runtime, + &BTreeMap::new(), + Instant::now() + Duration::from_secs(90), + ) + .unwrap(); + assert_eq!(receipt.phase, "ready-observed"); + state::write(&armed, &json!({"run":run})).unwrap(); + let mut input = libc::pollfd { + fd: 0, + events: libc::POLLIN, + revents: 0, + }; + // SAFETY: one valid writable pollfd. Parent EOF also permits owned exit. + assert!( + unsafe { libc::poll(&mut input, 1, 270_000) } > 0, + "owner child deadline" + ); + let mut byte = [0]; + let _ = std::io::stdin().read(&mut byte).unwrap(); + drop(runtime); +} +#[test] +#[ignore = "Only the isolated native parent fixture invokes this owned child"] +fn recovery_child() { + graph::recover_live_owner( + &candidate(), + &run(), + &std::env::var("HACK_ABSENT_RELAY_RECEIPT").unwrap(), + ) + .unwrap(); +} +#[test] +#[ignore = "Only the isolated native parent fixture invokes this owned child"] +fn remove_data_child() { + graph::foreground::cleanup_request(&candidate(), &run(), true).unwrap(); +} +fn child(candidate: &Candidate, run: &str, name: &str, extra: &[(&str, &str)]) -> Process { + let mut command = Command::new(std::env::current_exe().unwrap()); + command + .env_clear() + .env("PATH", "/usr/bin:/bin:/usr/sbin:/sbin"); + if let Some(home) = std::env::var_os("HOME") { + command.env("HOME", home); + } + command + .args(["--ignored", "--exact", name]) + .env("HACK_LOCAL_TEST_ROOT", &candidate.checkout) + .env("HACK_LOCAL_GRAPH_RUN", run) + .envs(extra.iter().copied()) + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()); + let child = command.spawn().unwrap(); + nonblocking(child.stdout.as_ref().unwrap().as_raw_fd()); + nonblocking(child.stderr.as_ref().unwrap().as_raw_fd()); + Process { + child, + out: Vec::new(), + err: Vec::new(), + } +} +fn wait_file(process: &mut Process, path: &Path, deadline: Instant) { + while !path.exists() { + assert!( + process.poll().is_none(), + "owned child exited before selected boundary" + ); + assert!(Instant::now() < deadline, "selected boundary deadline"); + std::thread::sleep(Duration::from_millis(20)); + } +} +fn pause_recovery( + candidate: &Candidate, + run: &str, + expected: &str, + point: &str, + deadline: Instant, +) { + let mut process = child( + candidate, + run, + RECOVERY, + &[ + ("HACK_ABSENT_RELAY_RECEIPT", expected), + ("HACK_LOCAL_GRAPH_FAULT", point), + ], + ); + let marker = graph::directory(candidate, run) + .unwrap() + .join(format!("fault-{point}.json")); + assert!(!marker.exists()); + wait_file( + &mut process, + &marker, + deadline.min(Instant::now() + Duration::from_secs(40)), + ); + assert_eq!(state::read::(&marker).unwrap()["run"], run); + process.child.kill().unwrap(); + process.wait(deadline); +} +fn refused(candidate: &Candidate, run: &str, expected: &str) { + let path = graph::directory(candidate, run).unwrap().join("state.json"); + let before = fs::read(&path).unwrap(); + assert!(graph::recover_live_owner(candidate, run, expected).is_err()); + assert_eq!(fs::read(path).unwrap(), before); +} +fn marker(candidate: &Candidate, run: &str, write: bool) -> Value { + let engine = graph::Engine::connect_cleanup(candidate).unwrap(); + let (receipt, _) = graph::load(candidate, &engine, run).unwrap(); + let resource = &receipt.resources["volume:data"]; + let volume = graph::inspect_resource(&engine, &receipt, resource) + .unwrap() + .unwrap(); + let mount = volume["Mountpoint"].as_str().unwrap(); + assert_eq!( + mount, + format!("/var/lib/docker/volumes/{}/_data", resource.name) + ); + let script = if write { + "set -eu; test -d \"$1\"; test ! -L \"$1\"; test ! -e \"$1/absence-marker\"; printf absent-relay-retained-v1 > \"$1/absence-marker\"; printf verified" + } else { + "set -eu; test -d \"$1\"; test ! -L \"$1\"; test \"$(cat \"$1/absence-marker\")\" = absent-relay-retained-v1; printf verified" + }; + assert_eq!( + engine.guest().execute_cleanup(script, &[mount]).unwrap(), + "verified" + ); + json!({"name":resource.name,"created":volume["CreatedAt"]}) +} +fn running_container(candidate: &Candidate, run: &str) -> Value { + let engine = graph::Engine::connect_cleanup(candidate).unwrap(); + let (receipt, _) = graph::load(candidate, &engine, run).unwrap(); + let resource = &receipt.resources["container:app"]; + let container = graph::inspect_resource(&engine, &receipt, resource) + .unwrap() + .unwrap(); + assert_eq!(container["State"]["Running"], true); + assert!(container["Id"].as_str().is_some_and(|id| !id.is_empty())); + assert!( + container["State"]["StartedAt"] + .as_str() + .is_some_and(|started| !started.is_empty()) + ); + json!({"id":container["Id"],"started_at":container["State"]["StartedAt"]}) +} +#[test] +#[ignore = "Caller-owned running disposable pool, pinned sh/sleep image, and external 300s watchdog required"] +fn absent_relay_cleanup_retries_preserve_data_and_live_sibling() { + let deadline = Instant::now() + Duration::from_secs(270); + let candidate = candidate(); + let image = std::env::var("HACK_LOCAL_TEST_IMAGE").unwrap(); + assert!( + image + .strip_prefix("sha256:") + .is_some_and(|digest| graph::hex(digest, 64)) + ); + let fixtures = [graph::tests::Fixture::new(), graph::tests::Fixture::new()]; + let runs = [ + graph::probes::token().unwrap(), + graph::probes::token().unwrap(), + ]; + let mut owners = Vec::new(); + for (fixture, run) in fixtures.iter().zip(&runs) { + state::write(&fixture.0.join("compose.yaml"), &json!({"services":{"app":{ + "image":image,"read_only":true,"network_mode":"none","init":true,"user":"0:0", + "entrypoint":["/bin/sh","-c","exec sleep 300"],"command":[],"volumes":["data:/data"]}},"volumes":{"data":{}}})).unwrap(); + let armed = fixture.0.join("armed.json"); + let mut owner = child( + &candidate, + run, + OWNER, + &[ + ("HACK_ABSENT_RELAY_PROJECT", fixture.0.to_str().unwrap()), + ("HACK_ABSENT_RELAY_ARMED", armed.to_str().unwrap()), + ], + ); + wait_file(&mut owner, &armed, deadline); + owners.push(owner); + } + let before = graph::inspect(&candidate, &runs[0]).unwrap().receipt; + let sibling = graph::inspect(&candidate, &runs[1]).unwrap().receipt; + let sibling_container = running_container(&candidate, &runs[1]); + let root = graph::directory(&candidate, &runs[0]).unwrap(); + let expected = sha(&fs::read(root.join("state.json")).unwrap()); + let boot = crate::provider::lifecycle::status(&candidate) + .unwrap() + .guest_boot_id; + let markers = [ + marker(&candidate, &runs[0], true), + marker(&candidate, &runs[1], true), + ]; + refused(&candidate, &runs[0], &expected); + drop(owners[0].child.stdin.take()); + assert!(owners[0].wait(deadline).success()); + let control = before + .relay_startup + .as_ref() + .unwrap() + .control_root + .join("relay-control"); + assert!(!control.join("owner.json").exists() && !control.join("control.sock").exists()); + assert!(control.join("operation.lock").is_file()); + // No partial endpoint can turn into an absence selection. + for name in ["owner.json", "control.sock"] { + fs::write(control.join(name), b"owned-negative-control").unwrap(); + refused(&candidate, &runs[0], &expected); + fs::remove_file(control.join(name)).unwrap(); + } + let original_bytes = fs::read(root.join("state.json")).unwrap(); + fs::write( + root.join("state.json"), + [original_bytes.as_slice(), b"\n"].concat(), + ) + .unwrap(); + refused(&candidate, &runs[0], &expected); + fs::write(root.join("state.json"), &original_bytes).unwrap(); + pause_recovery( + &candidate, + &runs[0], + &expected, + "live-owner-after-intent", + deadline, + ); + // Replacing the selected lock or directory cannot resume its committed intent. + fs::rename(control.join("operation.lock"), control.join("saved.lock")).unwrap(); + drop(state::Lock::acquire(&control).unwrap()); + refused(&candidate, &runs[0], &expected); + fs::remove_file(control.join("operation.lock")).unwrap(); + fs::rename(control.join("saved.lock"), control.join("operation.lock")).unwrap(); + let saved_control = control.with_file_name("relay-control-saved"); + fs::rename(&control, &saved_control).unwrap(); + state::private_directory(&control).unwrap(); + drop(state::Lock::acquire(&control).unwrap()); + refused(&candidate, &runs[0], &expected); + fs::remove_file(control.join("operation.lock")).unwrap(); + fs::remove_dir(&control).unwrap(); + fs::rename(saved_control, &control).unwrap(); + pause_recovery( + &candidate, + &runs[0], + &expected, + "cleanup-after-remove", + deadline, + ); + assert_eq!( + state::read::(&root.join("state.json")) + .unwrap() + .phase, + "cleanup-intent" + ); + assert_eq!(marker(&candidate, &runs[0], false), markers[0]); + pause_recovery( + &candidate, + &runs[0], + &expected, + "live-owner-after-completion", + deadline, + ); + let intent_path = root.join("live-owner-cleanup.json"); + let intent_bytes = fs::read(&intent_path).unwrap(); + let mut substituted: Value = serde_json::from_slice(&intent_bytes).unwrap(); + let mut process: ProcessIdentity = + serde_json::from_value(substituted["relay"]["absent"]["process"].clone()).unwrap(); + process.start_micros += 1; + let witness = dead::CleanupWitness::acquire( + &before.relay_startup.as_ref().unwrap().control_root, + graph::host_relay::context(&before.owner, substituted["boot"].as_str().unwrap()).unwrap(), + &process, + ) + .unwrap(); + substituted["relay"]["absent"]["process"] = json!(process); + substituted["relay"]["absent"]["witness_sha256"] = json!(witness.selection_sha256().unwrap()); + drop(witness); + state::write(&intent_path, &substituted).unwrap(); + refused(&candidate, &runs[0], &expected); + fs::write(&intent_path, &intent_bytes).unwrap(); + pause_recovery( + &candidate, + &runs[0], + &expected, + "live-owner-after-socket-retirement", + deadline, + ); + for name in ["owner.json", "control.sock"] { + fs::write(control.join(name), b"owned-negative-control").unwrap(); + refused(&candidate, &runs[0], &expected); + fs::remove_file(control.join(name)).unwrap(); + } + for _ in 0..2 { + assert_eq!( + graph::recover_live_owner(&candidate, &runs[0], &expected).unwrap()["publisher_retired"], + true + ); + } + let stopped = graph::inspect(&candidate, &runs[0]).unwrap().receipt; + assert_eq!(stopped.phase, "stopped-data-retained"); + assert_eq!(marker(&candidate, &runs[0], false), markers[0]); + assert_eq!(marker(&candidate, &runs[1], false), markers[1]); + assert_eq!( + serde_json::to_value(graph::inspect(&candidate, &runs[1]).unwrap().receipt).unwrap(), + serde_json::to_value(&sibling).unwrap() + ); + assert!(owners[1].poll().is_none()); + assert_eq!(running_container(&candidate, &runs[1]), sibling_container); + assert_eq!( + crate::provider::lifecycle::status(&candidate) + .unwrap() + .guest_boot_id, + boot + ); + // Stop only the remaining fixture owner, then prove both retaining cleanups. + drop(owners[1].child.stdin.take()); + assert!(owners[1].wait(deadline).success()); + let sibling_expected = sha(&fs::read( + graph::directory(&candidate, &runs[1]) + .unwrap() + .join("state.json"), + ) + .unwrap()); + graph::recover_live_owner(&candidate, &runs[1], &sibling_expected).unwrap(); + // Interrupt the new destructive request after an actual owned volume removal, + // before completion publication. Retry must use the persisted same-boot proof. + let mut removal = child( + &candidate, + &runs[0], + REMOVE_DATA, + &[( + "HACK_LOCAL_GRAPH_FAULT", + "retained-data-after-volume-remove", + )], + ); + let removal_marker = root.join("fault-retained-data-after-volume-remove.json"); + wait_file( + &mut removal, + &removal_marker, + deadline.min(Instant::now() + Duration::from_secs(40)), + ); + assert_eq!( + state::read::(&removal_marker).unwrap()["run"], + runs[0] + ); + removal.child.kill().unwrap(); + removal.wait(deadline); + assert_eq!( + graph::inspect(&candidate, &runs[0]).unwrap().receipt.phase, + "cleanup-intent" + ); + assert_eq!(marker(&candidate, &runs[1], false), markers[1]); + let removal_path = root.join("retired-data-removal.json"); + let removal_bytes = fs::read(&removal_path).unwrap(); + assert_eq!( + state::read::(&removal_path).unwrap()["recovery_source"], + "same-boot" + ); + for run in &runs { + graph::foreground::cleanup_request(&candidate, run, true).unwrap(); + graph::foreground::cleanup_request(&candidate, run, true).unwrap(); + assert_eq!( + graph::inspect(&candidate, run).unwrap().receipt.phase, + "removed" + ); + } + assert_eq!(fs::read(&removal_path).unwrap(), removal_bytes); +} diff --git a/packages/runtime-core/src/provider/graph/foreground/retained_data.rs b/packages/runtime-core/src/provider/graph/foreground/retained_data.rs index b8e57d4b0..cf3d02ffc 100644 --- a/packages/runtime-core/src/provider/graph/foreground/retained_data.rs +++ b/packages/runtime-core/src/provider/graph/foreground/retained_data.rs @@ -15,6 +15,53 @@ struct Intent { volumes: BTreeMap, #[serde(default, skip_serializing_if = "Option::is_none")] recovery: Option, + // Absent on legacy intents: those always selected previous-boot recovery. + #[serde(default, skip_serializing_if = "Option::is_none")] + recovery_source: Option, +} +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +enum RecoverySource { + PreviousBoot, + SameBoot, +} +impl RecoverySource { + fn selection( + self, + root: &Path, + receipt: &Receipt, + boot: &str, + expected: Option<&str>, + ) -> Result { + match self { + Self::PreviousBoot => { + graph::dead_owner_cleanup::removal_selection(root, receipt, boot, expected) + } + Self::SameBoot => { + graph::live_owner_cleanup::removal_selection(root, receipt, boot, expected) + } + } + } +} +/// Dispatch by the exact current completion only on first admission. Persisted +/// retries never switch proof sources, even when another sidecar is present. +fn recovery_source( + root: &Path, + receipt: &Receipt, + intent: Option<&Intent>, +) -> Result { + if let Some(intent) = intent { + return Ok(intent + .recovery_source + .unwrap_or(RecoverySource::PreviousBoot)); + } + let previous = graph::dead_owner_cleanup::current_completion(root, receipt)?; + let same = graph::live_owner_cleanup::current_completion(root, receipt)?; + match (previous, same) { + (false, true) => Ok(RecoverySource::SameBoot), + (_, false) => Ok(RecoverySource::PreviousBoot), + (true, true) => Err(refused()), + } } fn refused() -> CandidateError { CandidateError::new( @@ -44,7 +91,8 @@ fn allowed( boot: &str, recovery: Option<&str>, ) -> Result<(), CandidateError> { - if receipt.relay_startup.is_none() + if intent.is_some_and(|i| i.recovery_source.is_some() && i.recovery.is_none()) + || receipt.relay_startup.is_none() || (recovery.is_none() && receipt .relay_cleanup @@ -137,9 +185,25 @@ fn remove_guarded(candidate: &Candidate, run: &str, guard: Guard) -> Result None, Err(_) => return Err(refused()), }; + let source = if needs_recovery(&receipt, intent.as_ref()) { + no_pending(&root, "live-owner-cleanup.pending")?; + no_pending(&root, "dead-owner-cleanup.pending")?; + Some(match &guard { + Guard::Retired(_) => recovery_source(&root, &receipt, intent.as_ref())?, + Guard::Recovered(_) => { + if intent.as_ref().and_then(|i| i.recovery_source) == Some(RecoverySource::SameBoot) + { + return Err(refused()); + } + RecoverySource::PreviousBoot + } + }) + } else { + None + }; let retired_proof = match &guard { Guard::Retired(retired) if needs_recovery(&receipt, intent.as_ref()) => { - let proof = graph::dead_owner_cleanup::removal_selection( + let proof = source.ok_or_else(refused)?.selection( &root, &receipt, engine.guest().boot_id(), @@ -153,7 +217,13 @@ fn remove_guarded(candidate: &Candidate, run: &str, guard: Guard) -> Result(()) }; @@ -229,11 +299,20 @@ fn remove_guarded(candidate: &Candidate, run: &str, guard: Guard) -> Result(json!({"version":1,"binding":"b","boot":"boot","volumes":{},"recovery":"p","recovery_source":"unknown"})).is_err()); + intent.recovery = None; + assert!(allowed(&receipt, Some(&intent), "boot", None).is_err()); + std::fs::write(fixture.0.join("live-owner-cleanup.json"), b"malformed").unwrap(); + assert!(recovery_source(&fixture.0, &receipt, None).is_err()); + } + #[test] fn recovered_removal_intent_does_not_fabricate_relay_acknowledgement() { let mut receipt = receipt(); @@ -279,6 +419,7 @@ mod tests { boot: "boot".into(), volumes: BTreeMap::new(), recovery: Some(proof.clone()), + recovery_source: None, }; for phase in ["cleanup-intent", "removed"] { receipt.phase = phase.into(); @@ -297,6 +438,7 @@ mod tests { boot: "boot".into(), volumes: BTreeMap::new(), recovery: None, + recovery_source: None, }; receipt.phase = "cleanup-intent".into(); assert!(allowed(&receipt, None, "boot", None).is_err()); diff --git a/packages/runtime-core/src/provider/graph/foreground/transport.rs b/packages/runtime-core/src/provider/graph/foreground/transport.rs index 677293b13..05f78be33 100644 --- a/packages/runtime-core/src/provider/graph/foreground/transport.rs +++ b/packages/runtime-core/src/provider/graph/foreground/transport.rs @@ -373,6 +373,43 @@ pub(in crate::provider::graph) fn retire_recovered_publisher_locked( ) } +/// Read the exact selected dead process before retirement or after either +/// interrupted rename. The caller holds the publication lock throughout its +/// cleanup fences; a self-consistent external process receipt is insufficient. +pub(in crate::provider::graph) fn selected_retirement_process( + candidate: &Candidate, + run: &str, + expected_owner: &str, + expected_receipt: &str, + lock: &state::Lock, +) -> Result { + let root = root(candidate, run)?; + super::super::acknowledged_publisher::require_no_pending_missing_lock(candidate, run)?; + super::super::host_pin_recovery::exact_lock_path(&root, lock)?; + let path = retirement_path(&root, expected_owner); + if metadata(&path)?.is_some() { + let intent: Retirement = state::read(&path).map_err(|_| retirement_refused())?; + verify_retirement( + candidate, + run, + (expected_owner, expected_receipt), + &root, + lock, + &intent, + None, + )?; + return Ok(intent.owner.process); + } + let pin = Pin::read(candidate, run)?; + if format!("{:x}", Sha256::digest(&pin.bytes)) != expected_owner + || identity::alive(pin.record.process.pid).unwrap_or(true) + { + return Err(retirement_refused()); + } + no_listener(&root.join("control.sock"))?; + Ok(pin.record.process) +} + /// The caller's current cleanup proof remains valid at each publication effect. pub(in crate::provider::graph) fn retire_recovered_publisher_locked_fenced( candidate: &Candidate, @@ -438,6 +475,12 @@ pub(in crate::provider::graph) fn retire_recovered_publisher_locked_fenced( fs::File::open(&root) .and_then(|file| file.sync_all()) .map_err(|_| retirement_refused())?; + #[cfg(test)] + super::super::fault_pause( + &super::super::directory(candidate, run)?, + run, + "live-owner-after-socket-retirement", + )?; } if record_original { verify_cleanup()?; diff --git a/packages/runtime-core/src/provider/graph/foreground/transport_tests.rs b/packages/runtime-core/src/provider/graph/foreground/transport_tests.rs index 8b5dac3bc..d9b362781 100644 --- a/packages/runtime-core/src/provider/graph/foreground/transport_tests.rs +++ b/packages/runtime-core/src/provider/graph/foreground/transport_tests.rs @@ -165,6 +165,15 @@ fn cleanup_fence_refuses_effects_and_resumes_each_retirement_rename() { directory.join("owner.json") }; assert_eq!(fs::read(&owner_path).unwrap(), before); + let expected_process = serde_json::from_slice::(&before).unwrap().process; + assert_eq!( + selected_retirement_process(&candidate, &run, &owner, &receipt, &lock).unwrap(), + expected_process, + ); + assert!( + selected_retirement_process(&candidate, &run, &"e".repeat(64), &receipt, &lock) + .is_err() + ); let journal = retirement_path(&directory, &owner); assert_eq!(journal.exists(), fail_at >= 4); assert_eq!(directory.join("control.sock").exists(), fail_at < 5); @@ -201,6 +210,10 @@ fn cleanup_fence_refuses_effects_and_resumes_each_retirement_rename() { .unwrap(); assert!(!directory.join("control.sock").exists()); assert!(!directory.join("owner.json").exists()); + assert_eq!( + selected_retirement_process(&candidate, &run, &owner, &receipt, &lock).unwrap(), + expected_process, + ); assert_eq!( fs::read(retired_path(&directory, &owner, false)).unwrap(), before diff --git a/packages/runtime-core/src/provider/graph/live_owner_cleanup.rs b/packages/runtime-core/src/provider/graph/live_owner_cleanup.rs index 84fed4e83..01596ee10 100644 --- a/packages/runtime-core/src/provider/graph/live_owner_cleanup.rs +++ b/packages/runtime-core/src/provider/graph/live_owner_cleanup.rs @@ -4,6 +4,7 @@ use super::*; use crate::provider::relay_owner::publication::dead; use sha2::{Digest, Sha256}; use std::path::Path; +mod relay; const FILE: &str = "live-owner-cleanup.json"; #[derive(Serialize, Deserialize)] #[serde(deny_unknown_fields)] @@ -13,7 +14,7 @@ struct Intent { original: Receipt, original_sha256: String, foreground_sha256: String, - relay: dead::Selection, + relay: relay::Selection, environment: Value, bridges: bridges::cleanup::Selection, prior_bridges: Option, @@ -33,6 +34,36 @@ pub(super) fn current_completion( Ok(intent.complete_sha256.as_deref() == Some(complete.as_str())) } +/// Select an independent data-removal request from the exact completed same-boot +/// generation. A retry pins the entire committed proof, not just receipt phases. +pub(super) fn removal_selection( + root: &Path, + receipt: &Receipt, + boot: &str, + expected: Option<&str>, +) -> Result { + if exists(&root.join("live-owner-cleanup.pending"))? { + return Err(refused()); + } + let intent: Intent = state::read(&root.join(FILE))?; + validate(&intent, receipt, &intent.original_sha256, boot)?; + let complete = intent.complete_sha256.as_deref().ok_or_else(refused)?; + let proof = digest(&intent)?; + match expected { + None if receipt.phase == "stopped-data-retained" && digest(receipt)? == complete => {} + Some(value) + if value == proof + && ["stopped-data-retained", "cleanup-intent", "removed"] + .contains(&receipt.phase.as_str()) => {} + _ => return Err(refused()), + } + Ok(dead_owner_cleanup::RemovalProof { + owner: intent.foreground_sha256, + complete: complete.into(), + digest: proof, + }) +} + fn refused() -> CandidateError { error( "graph_live_owner_recovery", @@ -104,7 +135,8 @@ fn validate( boot: &str, ) -> Result<(), CandidateError> { ready(&intent.original)?; - if intent.version != 1 + intent.relay.validate()?; + if intent.version != intent.relay.version() || intent.boot != boot || boot.is_empty() || !hex(expected, 64) @@ -132,7 +164,11 @@ fn save(root: &Path, intent: &Intent) -> Result<(), CandidateError> { )?; state::write(&root.join(FILE), intent) } -fn stop_listeners(engine: &Engine<'_>, receipt: &Receipt) -> Result<(), CandidateError> { +fn stop_listeners( + engine: &Engine<'_>, + receipt: &Receipt, + verify: &dyn Fn() -> Result<(), CandidateError>, +) -> Result<(), CandidateError> { let startup = receipt.relay_startup.as_ref().ok_or_else(refused)?; for (name, service) in &startup.services { if service.phase == startup::Phase::Completed { @@ -149,6 +185,7 @@ fn stop_listeners(engine: &Engine<'_>, receipt: &Receipt) -> Result<(), Candidat let (uid, gid) = launcher::identity(&value["Config"])?; for binding in service.bindings.values() { let process = binding.process.ok_or_else(refused)?; + verify()?; engine.guest().stop_orphan_relay_listener( crate::provider::lifecycle::RelayLaunch { container: resource.id.as_deref().ok_or_else(refused)?, @@ -273,7 +310,12 @@ pub fn recover_live_owner( foreground.verify_retirement_ready()?; let startup = receipt.relay_startup.as_ref().ok_or_else(refused)?; let context = host_relay::context(&receipt.owner, engine.guest().boot_id())?; - let relay = dead::Witness::acquire(&startup.control_root, context, foreground.process())?; + let relay = + dead::CleanupWitness::acquire(&startup.control_root, context, foreground.process())?; + if matches!(relay, dead::CleanupWitness::Absent(_)) { + relay::require_scoped_root(candidate, &receipt, engine.guest().boot_id())?; + relay::verify_receipt(&root, &receipt)?; + } archive_prior(&root, &receipt, expected)?; let mut intent = if exists(&root.join(FILE))? { state::read::(&root.join(FILE))? @@ -286,13 +328,14 @@ pub fn recover_live_owner( for resource in receipt.resources.values() { inspect_resource(&engine, &receipt, resource)?; } + let selected = relay::Selection::capture(&relay, foreground.process())?; Intent { - version: 1, + version: selected.version(), boot: engine.guest().boot_id().into(), original: receipt.clone(), original_sha256: expected.into(), foreground_sha256: foreground.fingerprint(), - relay: relay.selection(), + relay: selected, environment: serde_json::to_value(environment::cleanup_inventory( candidate, &engine, &receipt, &root, )?) @@ -304,9 +347,10 @@ pub fn recover_live_owner( } }; validate(&intent, &receipt, expected, engine.guest().boot_id())?; - if intent.foreground_sha256 != foreground.fingerprint() || intent.relay != relay.selection() { + if intent.foreground_sha256 != foreground.fingerprint() { return Err(refused()); } + intent.relay.verify(&relay, foreground.process())?; startup::require_dependency_rebind_recovery_complete(&root, &intent.original, &intent.boot)?; foreground.verify_retirement_ready()?; relay.verify()?; @@ -317,16 +361,35 @@ pub fn recover_live_owner( } bridges::cleanup::verify_live_remaining(candidate, &engine, &receipt, &intent.bridges)?; save(&root, &intent)?; - if !intent.listeners_retired { - stop_listeners(&engine, &receipt)?; + #[cfg(test)] + fault_pause(&root, run, "live-owner-after-intent")?; + let verify = || { + engine.guest().verify()?; foreground.verify_retirement_ready()?; relay.verify()?; + if matches!(relay, dead::CleanupWitness::Absent(_)) { + relay::verify_receipt(&root, &intent.original)?; + } + Ok(()) + }; + if !intent.listeners_retired { + stop_listeners(&engine, &receipt, &verify)?; + verify()?; intent.listeners_retired = true; save(&root, &intent)?; } bridges::cleanup::verify_recovery_file(&root, &intent.bridges, intent.prior_bridges.as_ref())?; + verify()?; bridges::cleanup::recover_persist(&root, &intent.bridges)?; - let cleaned = cleanup_owned(candidate, &engine, receipt, &root, false)?; + let cleaned = cleanup_owned_fenced( + candidate, + &engine, + receipt, + &root, + false, + matches!(relay, dead::CleanupWitness::Absent(_)), + verify, + )?; host_relay::inspect_cleanup( candidate, &engine, @@ -335,10 +398,11 @@ pub fn recover_live_owner( &environment, &intent.bridges, )?; - foreground.verify_retirement_ready()?; - relay.verify()?; + verify()?; intent.complete_sha256 = Some(digest(&cleaned)?); save(&root, &intent)?; + #[cfg(test)] + fault_pause(&root, run, "live-owner-after-completion")?; // Commit independently verified owned absence before archival. A crash during // a rename resumes through the completed intent, never through host adoption. drop(relay); @@ -384,23 +448,56 @@ fn finish_retirement( } startup::verify_cleanup(engine, receipt)?; probes::verify_cleanup(engine, receipt)?; - archive_completed_refresh(root, &intent.original, receipt, &intent.boot, &|| { - engine.guest().verify() - })?; - let startup = receipt.relay_startup.as_ref().ok_or_else(refused)?; - dead::retire( - &startup.control_root, - host_relay::context(&receipt.owner, &intent.boot)?, - &intent.relay, + let publisher = foreground::transport::root(candidate, &receipt.run)?; + acknowledged_publisher::require_no_pending_missing_lock(candidate, &receipt.run)?; + let publisher_lock = state::Lock::acquire_existing(&publisher)?; + acknowledged_publisher::require_no_pending_missing_lock(candidate, &receipt.run)?; + let process = foreground::transport::selected_retirement_process( + candidate, + &receipt.run, + &intent.foreground_sha256, + &digest(receipt)?, + &publisher_lock, )?; - foreground::retire_publisher_path( + let absent = intent + .relay + .absent_witness(candidate, receipt, &intent.boot, &process)?; + let verify = || { + engine.guest().verify()?; + if let Some(witness) = &absent { + let current = foreground::transport::selected_retirement_process( + candidate, + &receipt.run, + &intent.foreground_sha256, + &digest(receipt)?, + &publisher_lock, + )?; + intent.relay.verify(witness, ¤t)?; + relay::verify_receipt(root, &intent.original)?; + } + Ok(()) + }; + archive_completed_refresh(root, &intent.original, receipt, &intent.boot, &verify)?; + let startup = receipt.relay_startup.as_ref().ok_or_else(refused)?; + if let relay::Selection::Present(selected) = &intent.relay { + dead::retire( + &startup.control_root, + host_relay::context(&receipt.owner, &intent.boot)?, + selected, + )?; + } + foreground::transport::retire_recovered_publisher_locked_fenced( candidate, &receipt.run, &intent.foreground_sha256, &digest(receipt)?, + None, + &publisher_lock, + &verify, )?; + verify()?; dependency_slots::recover_cleaned(candidate, receipt, None)?; - engine.guest().verify() + verify() } /// Completed cleanup grants archival, not refresh replay. The same exact ready @@ -720,6 +817,135 @@ mod tests { } } #[test] + fn same_boot_removal_requires_exact_completion_and_pins_retry_proof() { + let fixture = super::super::tests::Fixture::new(); + let original = generation(1, "ready-observed"); + let mut stopped = generation(1, "stopped-data-retained"); + let mut intent = completed(original); + let path = fixture.0.join(FILE); + state::write(&path, &intent).unwrap(); + let proof = removal_selection(&fixture.0, &stopped, "boot", None).unwrap(); + assert_eq!(proof.owner, intent.foreground_sha256); + assert_eq!(proof.complete, intent.complete_sha256.clone().unwrap()); + let bytes = fs::read(&path).unwrap(); + for phase in ["stopped-data-retained", "cleanup-intent", "removed"] { + stopped.phase = phase.into(); + assert!(removal_selection(&fixture.0, &stopped, "boot", Some(&proof.digest)).is_ok()); + assert!( + removal_selection(&fixture.0, &stopped, "other-boot", Some(&proof.digest)).is_err() + ); + assert!( + removal_selection(&fixture.0, &stopped, "boot", Some(&"0".repeat(64))).is_err() + ); + if phase != "stopped-data-retained" { + assert!(removal_selection(&fixture.0, &stopped, "boot", None).is_err()); + } + } + stopped.phase = "stopped-data-retained".into(); + stopped.resources.get_mut("container:web").unwrap().id = Some("9".repeat(64)); + assert!(removal_selection(&fixture.0, &stopped, "boot", Some(&proof.digest)).is_err()); + stopped = generation(1, "stopped-data-retained"); + for case in 0..5 { + let mut changed = completed(generation(1, "ready-observed")); + match case { + 0 => changed.complete_sha256 = None, + 1 => changed.complete_sha256 = Some("9".repeat(64)), + 2 => changed.listeners_retired = false, + 3 => changed.original_sha256 = "9".repeat(64), + 4 => changed.foreground_sha256 = "9".repeat(64), + _ => unreachable!(), + } + state::write(&path, &changed).unwrap(); + assert!(removal_selection(&fixture.0, &stopped, "boot", Some(&proof.digest)).is_err()); + } + intent.complete_sha256 = Some(digest(&stopped).unwrap()); + state::write(&path, &intent).unwrap(); + fs::write(fixture.0.join("live-owner-cleanup.pending"), b"partial").unwrap(); + assert!(removal_selection(&fixture.0, &stopped, "boot", None).is_err()); + assert_eq!(fs::read(&path).unwrap(), bytes); + } + + #[test] + fn absent_relay_completion_supports_independent_removal_selection() { + let fixture = super::super::tests::Fixture::new(); + let stopped = generation(1, "stopped-data-retained"); + let mut proof = serde_json::to_value(completed(generation(1, "ready-observed"))).unwrap(); + proof["version"] = json!(2); + proof["relay"] = json!({"absent": { + "process": {"pid":999999, "start_micros":1, "uid":unsafe {libc::geteuid()}, "executable":"/bin/sleep"}, + "witness_sha256":"a".repeat(64) + }}); + state::write(&fixture.0.join(FILE), &proof).unwrap(); + let selected = removal_selection(&fixture.0, &stopped, "boot", None).unwrap(); + assert_eq!(selected.complete, digest(&stopped).unwrap()); + removal_selection(&fixture.0, &stopped, "boot", Some(&selected.digest)).unwrap(); + proof["relay"]["absent"]["witness_sha256"] = json!("b".repeat(64)); + state::write(&fixture.0.join(FILE), &proof).unwrap(); + assert!(removal_selection(&fixture.0, &stopped, "boot", Some(&selected.digest)).is_err()); + } + + #[test] + fn recovery_versions_distinguish_absence_from_legacy_publication() { + let original = receipt(); + let legacy = completed(original.clone()); + let value = serde_json::to_value(&legacy).unwrap(); + assert!(value["relay"]["bytes"].is_array()); + assert!(value["relay"].get("absent").is_none()); + let restored: Intent = serde_json::from_value(value.clone()).unwrap(); + validate(&restored, &original, &digest(&original).unwrap(), "boot").unwrap(); + let mut absent = value; + absent["relay"] = json!({"absent": { + "process": {"pid":999999, "start_micros":1, "uid":unsafe {libc::geteuid()}, "executable":"/bin/sleep"}, + "witness_sha256":"a".repeat(64) + }}); + let old_version: Intent = serde_json::from_value(absent.clone()).unwrap(); + assert!(validate(&old_version, &original, &digest(&original).unwrap(), "boot").is_err()); + absent["version"] = json!(2); + let selected: Intent = serde_json::from_value(absent.clone()).unwrap(); + validate(&selected, &original, &digest(&original).unwrap(), "boot").unwrap(); + assert!( + validate( + &selected, + &original, + &digest(&original).unwrap(), + "changed-boot" + ) + .is_err() + ); + absent["relay"]["absent"]["witness_sha256"] = json!("malformed"); + let malformed: Intent = serde_json::from_value(absent.clone()).unwrap(); + assert!(validate(&malformed, &original, &digest(&original).unwrap(), "boot").is_err()); + absent["relay"]["unexpected"] = json!(true); + assert!(serde_json::from_value::(absent).is_err()); + } + #[test] + fn absence_receipt_fence_rejects_changed_bytes_and_resources() { + let fixture = super::super::tests::Fixture::new(); + let original = receipt(); + let path = fixture.0.join("state.json"); + state::write(&path, &original).unwrap(); + relay::verify_receipt(&fixture.0, &original).unwrap(); + let mut bytes = fs::read(&path).unwrap(); + bytes.push(b'\n'); + fs::write(&path, &bytes).unwrap(); + assert!(relay::verify_receipt(&fixture.0, &original).is_err()); + let mut changed = original.clone(); + changed.phase = "cleanup-intent".into(); + state::write(&path, &changed).unwrap(); + relay::verify_receipt(&fixture.0, &original).unwrap(); + changed.owner = "f".repeat(32); + state::write(&path, &changed).unwrap(); + assert!(relay::verify_receipt(&fixture.0, &original).is_err()); + changed = original.clone(); + changed.relay_startup.as_mut().unwrap().control_root = "/private/foreign".into(); + state::write(&path, &changed).unwrap(); + assert!(relay::verify_receipt(&fixture.0, &original).is_err()); + changed = original.clone(); + changed.phase = "removed".into(); + state::write(&path, &changed).unwrap(); + assert!(relay::verify_receipt(&fixture.0, &original).is_err()); + } + #[test] fn historical_recovery_survives_eviction_without_lending_current_authority() { let fixture = super::super::tests::Fixture::new(); let prior = completed(generation(1, "ready-observed")); diff --git a/packages/runtime-core/src/provider/graph/live_owner_cleanup/relay.rs b/packages/runtime-core/src/provider/graph/live_owner_cleanup/relay.rs new file mode 100644 index 000000000..d4dadca12 --- /dev/null +++ b/packages/runtime-core/src/provider/graph/live_owner_cleanup/relay.rs @@ -0,0 +1,263 @@ +//! Persist absence as a selected locked-directory/process witness, never as a +//! substitute relay acknowledgement or a fallback from invalid publication. +use super::*; +use crate::provider::identity::ProcessIdentity; + +/// The untagged present variant preserves version-one recovery records exactly. +#[derive(Serialize, Deserialize)] +#[serde(untagged, deny_unknown_fields)] +pub(super) enum Selection { + Present(dead::Selection), + Absent { absent: Absent }, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Absent { + process: ProcessIdentity, + witness_sha256: String, +} + +impl Selection { + pub(super) fn capture( + witness: &dead::CleanupWitness, + process: &ProcessIdentity, + ) -> Result { + Ok(match witness.present_selection() { + Some(selected) => Self::Present(selected), + None => Self::Absent { + absent: Absent { + process: process.clone(), + witness_sha256: witness.selection_sha256()?, + }, + }, + }) + } + + pub(super) fn version(&self) -> u8 { + match self { + Self::Present(_) => 1, + Self::Absent { .. } => 2, + } + } + + pub(super) fn validate(&self) -> Result<(), CandidateError> { + if let Self::Absent { absent } = self { + // SAFETY: geteuid takes no arguments. + crate::provider::identity::verify( + &absent.process, + &absent.process, + &absent.process.executable, + unsafe { libc::geteuid() }, + )?; + if !absent.process.executable.is_absolute() || !hex(&absent.witness_sha256, 64) { + return Err(refused()); + } + } + Ok(()) + } + + pub(super) fn verify( + &self, + witness: &dead::CleanupWitness, + process: &ProcessIdentity, + ) -> Result<(), CandidateError> { + witness.verify()?; + match (self, witness.present_selection()) { + (Self::Present(expected), Some(current)) if *expected == current => Ok(()), + (Self::Absent { absent }, None) + if absent.process == *process + && absent.witness_sha256 == witness.selection_sha256()? => + { + Ok(()) + } + _ => Err(refused()), + } + } + + /// Reacquire the original absence lock through publication retirement. There + /// is deliberately no unlink operation for an absent selection. + pub(super) fn absent_witness( + &self, + candidate: &Candidate, + receipt: &Receipt, + boot: &str, + process: &ProcessIdentity, + ) -> Result, CandidateError> { + let Self::Absent { absent } = self else { + return Ok(None); + }; + require_scoped_root(candidate, receipt, boot)?; + let startup = receipt.relay_startup.as_ref().ok_or_else(refused)?; + let witness = dead::CleanupWitness::acquire( + &startup.control_root, + host_relay::context(&receipt.owner, boot)?, + &absent.process, + )?; + self.verify(&witness, process)?; + Ok(Some(witness)) + } +} + +/// A ready receipt and its exact foreground process bind the otherwise absent +/// endpoint to this run. Legacy unscoped roots cannot establish that binding. +pub(super) fn require_scoped_root( + candidate: &Candidate, + receipt: &Receipt, + boot: &str, +) -> Result<(), CandidateError> { + let expected = startup::control_root( + &candidate.state_root, + &receipt.owner, + Some(boot), + Some(&receipt.run), + )?; + if receipt + .relay_startup + .as_ref() + .ok_or_else(refused)? + .control_root + != expected + { + return Err(refused()); + } + Ok(()) +} + +/// Retain the raw-byte selection: parsing and reserializing must not hide an +/// external replacement. Only canonical receipt commits from owned cleanup may +/// advance phase; immutable run/resource/startup identities cannot change. +pub(super) fn verify_receipt(root: &Path, original: &Receipt) -> Result<(), CandidateError> { + let bytes = host_pin_recovery::read_raw(&root.join("state.json"), 2 * 1024 * 1024)?; + let current: Receipt = serde_json::from_slice(&bytes).map_err(|_| refused())?; + if bytes != serde_json::to_vec_pretty(¤t).map_err(|_| refused())? + || immutable(¤t)? != immutable(original)? + || !matches!( + current.phase.as_str(), + "ready-observed" | "cleanup-intent" | "stopped-data-retained" + ) + { + return Err(refused()); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::provider::{identity, relay_owner::Context}; + use std::{ + os::unix::fs::DirBuilderExt, + process::{Child, Command, Stdio}, + }; + + struct Process(Child); + impl Drop for Process { + fn drop(&mut self) { + let _ = self.0.kill(); + let _ = self.0.wait(); + } + } + + #[test] + fn selected_absence_refuses_live_process_partial_publication_and_recreated_endpoint() { + let candidate_root = super::super::super::tests::Fixture::new(); + let candidate = Candidate::discover(&candidate_root.0).unwrap(); + let mut receipt: Receipt = serde_json::from_value(json!({ + "version":1,"run":"a".repeat(32),"owner":"b".repeat(32), + "namespace":"c".repeat(64),"plan_id":"d".repeat(64), + "phase":"ready-observed","readiness":{},"resources":{}, + "relay_startup":{"control_only":true,"guest_root":null, + "control_root":"/private/foreign","artifact":"e".repeat(64),"services":{}} + })) + .unwrap(); + let boot = "11111111-1111-1111-1111-111111111111"; + assert!(require_scoped_root(&candidate, &receipt, boot).is_err()); + let root = startup::control_root( + &candidate.state_root, + &receipt.owner, + Some(boot), + Some(&receipt.run), + ) + .unwrap(); + // Exclusive creation and the fixture's unique state root scope this + // disposable control directory; the test never touches a runtime home. + fs::DirBuilder::new().mode(0o700).create(&root).unwrap(); + let root = super::super::super::tests::Fixture(root); + receipt.relay_startup.as_mut().unwrap().control_root = root.0.clone(); + require_scoped_root(&candidate, &receipt, boot).unwrap(); + assert!(require_scoped_root(&candidate, &receipt, "changed-boot").is_err()); + let context = Context { + runtime: [1; 16], + boot: [2; 16], + }; + let control = root.0.join("relay-control"); + fs::DirBuilder::new().mode(0o700).create(&control).unwrap(); + let lock = state::Lock::acquire(&control).unwrap(); + drop(lock); + let mut child = Process( + Command::new("/bin/sleep") + .arg("60") + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .spawn() + .unwrap(), + ); + let process = identity::observe(child.0.id() as i32).unwrap(); + assert!(dead::CleanupWitness::acquire(&root.0, context, &process).is_err()); + child.0.kill().unwrap(); + child.0.wait().unwrap(); + let witness = dead::CleanupWitness::acquire(&root.0, context, &process).unwrap(); + let selected = Selection::capture(&witness, &process).unwrap(); + assert_eq!(selected.version(), 2); + selected.verify(&witness, &process).unwrap(); + let restored: Selection = + serde_json::from_slice(&serde_json::to_vec(&selected).unwrap()).unwrap(); + restored.verify(&witness, &process).unwrap(); + let mut changed_process = process.clone(); + changed_process.start_micros += 1; + assert!(restored.verify(&witness, &changed_process).is_err()); + fs::write(control.join("owner.json"), b"foreign").unwrap(); + assert!(restored.verify(&witness, &process).is_err()); + drop(witness); + assert!(dead::CleanupWitness::acquire(&root.0, context, &process).is_err()); + fs::remove_file(control.join("owner.json")).unwrap(); + let witness = dead::CleanupWitness::acquire(&root.0, context, &process).unwrap(); + restored.verify(&witness, &process).unwrap(); + let endpoint = + std::os::unix::net::UnixListener::bind(control.join("control.sock")).unwrap(); + assert!(restored.verify(&witness, &process).is_err()); + drop(endpoint); + fs::remove_file(control.join("control.sock")).unwrap(); + restored.verify(&witness, &process).unwrap(); + drop(witness); + let resumed = restored + .absent_witness(&candidate, &receipt, boot, &process) + .unwrap() + .unwrap(); + restored.verify(&resumed, &process).unwrap(); + drop(resumed); + let substituted = + dead::CleanupWitness::acquire(&root.0, context, &changed_process).unwrap(); + let self_consistent = Selection::capture(&substituted, &changed_process).unwrap(); + self_consistent + .verify(&substituted, &changed_process) + .unwrap(); + assert!(self_consistent.verify(&substituted, &process).is_err()); + drop(substituted); + assert!( + self_consistent + .absent_witness(&candidate, &receipt, boot, &process) + .is_err() + ); + fs::rename( + control.join("operation.lock"), + control.join("original.lock"), + ) + .unwrap(); + drop(state::Lock::acquire(&control).unwrap()); + let replacement = dead::CleanupWitness::acquire(&root.0, context, &process).unwrap(); + assert!(restored.verify(&replacement, &process).is_err()); + } +} diff --git a/packages/runtime-core/src/provider/graph/mod.rs b/packages/runtime-core/src/provider/graph/mod.rs index dac57f75e..c142ed97f 100644 --- a/packages/runtime-core/src/provider/graph/mod.rs +++ b/packages/runtime-core/src/provider/graph/mod.rs @@ -1741,6 +1741,10 @@ fn cleanup_owned_fenced( // only the final receipt needs another commit, including environment-slot retirement. #[cfg(test)] fault_pause(root, &receipt.run, "cleanup-after-remove")?; + #[cfg(test)] + if kind == Kind::Volume { + fault_pause(root, &receipt.run, "retained-data-after-volume-remove")?; + } } } fence()?; diff --git a/packages/runtime-core/src/provider/graph/startup.rs b/packages/runtime-core/src/provider/graph/startup.rs index 6397a5185..2321ccfe1 100644 --- a/packages/runtime-core/src/provider/graph/startup.rs +++ b/packages/runtime-core/src/provider/graph/startup.rs @@ -370,6 +370,8 @@ pub(super) fn attach( #[cfg(target_os = "macos")] mod runtime; #[cfg(target_os = "macos")] +pub(super) use runtime::control_root; +#[cfg(target_os = "macos")] pub use runtime::{Dependency, HostRelayRuntime, RefreshPolicy}; /// Incomplete native dependency mutation cannot authorize a fresh execution. diff --git a/packages/runtime-core/src/provider/graph/startup/runtime.rs b/packages/runtime-core/src/provider/graph/startup/runtime.rs index 3b9ce78fb..a21a2babf 100644 --- a/packages/runtime-core/src/provider/graph/startup/runtime.rs +++ b/packages/runtime-core/src/provider/graph/startup/runtime.rs @@ -281,7 +281,7 @@ fn relay_admission_gate( } } } -fn control_root( +pub(in crate::provider::graph) fn control_root( state_root: &Path, owner: &str, boot: Option<&str>, diff --git a/packages/runtime-core/src/provider/relay_owner/publication/dead.rs b/packages/runtime-core/src/provider/relay_owner/publication/dead.rs index cccf5266b..963496c6f 100644 --- a/packages/runtime-core/src/provider/relay_owner/publication/dead.rs +++ b/packages/runtime-core/src/provider/relay_owner/publication/dead.rs @@ -66,7 +66,9 @@ pub(crate) struct Witness { } /// A clean managed owner exit unlinks both publication paths while leaving the /// operation lock. Absence is only a current endpoint fact: callers must bind -/// the original owner/process/publication to a selected coordinator record. +/// the original owner/process/publication to a selected coordinator record, or +/// bind the exact dead foreground process to a ready graph receipt and its +/// current-boot run-derived control root. Absence alone grants no cleanup. pub(crate) enum CleanupWitness { Present(Witness), Absent(AbsentWitness), @@ -190,6 +192,7 @@ impl AbsentWitness { } } impl Witness { + #[cfg(test)] pub(crate) fn acquire( root: &Path, context: Context,