Skip to content

Crash Recovery

This playbook describes how to recover safely after a process crash or host restart.

What Happened

  • The process terminated unexpectedly.
  • Orders or positions may have changed while the process was down.

Why It Happens

  • OS restarts, power outages, or unhandled exceptions.
  • Connectivity or auth failures causing process exits.

What To Check

  • logs/fyers_broker_errors.log and logs/trading.log.
  • State DB health (state_db_path) and file permissions.
  • REST connectivity and token validity.

What To Do Next

  1. Restart the process.
  2. Confirm the broker reconciles state on startup.
  3. Monitor for ReconcileFailureError and resolve before retrying.

Safety Notes

  • Do not delete the state DB during live recovery unless you intend to reset order state.
  • If the state DB is corrupted, stop trading and rebuild state only after manual verification.

Scenario Example

Scenario: The process restarts and fails with "state schema mismatch." What happened: - State DB schema version no longer matches the code. What to check: - Recent code changes or upgrades. What to do next: - Stop all processes, back up the state DB, then delete and restart to rebuild.