Skip to main content

Recover from partial deployment

Use this page when a deployment completed only partially and the stack cannot be trusted again until the incomplete state is bounded and recovered.

When to use this page​

  • A deploy stopped mid-sequence or left services inconsistent.
  • Runtime topology no longer matches intended deployment truth.
  • You need to choose between bounded repair, rerun, or rollback.

Prerequisites​

  • You can identify which deployment stage failed or stopped.
  • You can capture service posture and runtime-materialization evidence.

Architecture / Flow​

Step-by-step procedure​

1. Identify the partial boundary​

Record:

  • which stage completed
  • which stage stopped
  • whether the current state is safe to verify in place

2. Capture current service posture​

Use:

GET /api/v1/platform/services

3. Compare runtime against intended deployment truth​

Use:

python ./scripts/verify_runtime_materialization.py --check

4. Choose the narrowest recovery option​

The owner decision should choose between:

  • bounded repair
  • rerun
  • rollback

If redeploy is the owner-approved safer path:

bash ./scripts/redeploy-host-stack.sh

5. Re-verify the full posture​

After recovery, confirm:

  • the service topology is consistent again
  • the bounded post-deployment verification path is green

Verification​

This runbook is being used correctly when:

  1. the failed deployment boundary is explicit
  2. intended and running truth were compared before recovery
  3. the stack is not considered trusted until service posture is consistent again

Common failure modes​

Continuing deployment after a partial failure​

Problem:

  • later stages hide or worsen the original inconsistency.

Better path:

  • stop at the partial boundary and recover first

Recovering without topology comparison​

Problem:

  • rerun and rollback choices are made blindly.

Better path:

  • compare current services and runtime materialization first

Source Truth​

  • contracts/docs/public_manufacturer_docs_post_v1_runbooks_v1.json
  • JaddaHelpifyr/jhf-deployment:docs/TROUBLESHOOTING.md
  • JaddaHelpifyr/jhf-deployment:docs/OPERATIONS.md
  • JaddaHelpifyr/helpifyr-fabric:docs/operations/STACK_RECOVERY_RUNBOOK.md

Next paths​