Tasks

API Confirmations Needed (Hard Gates)

[x] Confirm REST base URL and endpoints or method mapping for: place/modify/cancel order, orderbook, tradebook, positions, holdings, funds, history, quotes.
[x] Confirm order WebSocket payload schema, fill-id availability, sequencing fields, heartbeat/ping, and reconnect/resubscribe rules.
[x] Confirm market data WebSocket payload schema, heartbeat/ping, subscription batching limits, and reconnect behavior.
[x] Confirm instrument master source and symbol to fyToken mapping workflow.
[x] Confirm how daily IST bars should map to Backtrader session dates.

Phase 1: Minimal Store Skeleton

[x] Create src/fyers_store/ package structure with adapters/state/models and tests/ scaffolding.
[x] Define store/broker/data feed class scaffolding based on sample_codes patterns.
[x] Wire logging categories using existing logging utilities.
[x] Define adapter interfaces for REST and WS.
[x] Update README.md with Phase 1 usage notes.

Phase 1.5: Adapter Contracts & Validation

[x] Implement concrete REST/WS adapters that satisfy start()/stop()/get_queue() and normalized bar contracts.
[x] Add pytest coverage for normalize_bar and Store queue-drain behavior.
[x] Confirm FYERS order side/type values and codify constants.
[x] Define and document logging policy (categories, log levels, noisy WS sampling, and setup_logging usage) across Store/DataFeed/Broker/Adapters.

Phase 2: Historical REST Feed

[x] Implement REST history adapter with pagination and partial-candle rule.
[x] Normalize timestamps, resolution mapping, and timezone handling.
[x] Add pluggable historical cache (default SQLite/WAL) with bars table, per-context watermarks, and data-driven gap detection; keep adapters stateless and wire DB-first lookups.
[x] Unit tests for pagination and timestamp boundaries.
[x] Implement logging policy in code (WS sampling, helper utilities, and ensure setup_logging in entrypoints/examples).
[x] Add schema/validation for config/fyers_ws.yaml to prevent misconfiguration.
[x] Update README.md.

Phase 2b: Historical Cache (SQLite) Hardening + Flexibility

Note: Phase 2 delivered the baseline historical cache + gap detection wiring. This section tracks follow-up hardening/flexibility work that builds on the Phase 2 cache. Plan: - [x] Confirm cache requirements and FYERS resolution codes; document the agreed plan and constraints. - [x] Harden SQLite cache for multi-process use (WAL + busy timeout + retry on lock) while preserving DB-first + REST gap fill. - [x] Replace cache config YAML parsing with yaml.safe_load (remove manual _parse_db_path_from_yaml parsing). - [x] Add basic multi-process write contention handling (SQLite busy_timeout + bounded retry/backoff on database is locked). - [x] Make cache gap detection market-aware using NSE session rules (config/nse_calendar.yaml) to ignore weekends/off-session; holidays configurable. - [x] Lazy-load generated holiday sources by requested date range; make year-range config optional. - [x] Add optional periodic background maintenance (WAL checkpoint + PRAGMA optimize/quick_check; VACUUM optional) driven by config/cache.yaml. - [x] Confirm and enforce FYERS-supported history resolutions (no synthetic caching). - [x] Document the exact FYERS history() resolution values we support in docs/fyers_mapping.md (source: FYERS docs, values confirmed 2026-01-17). - [x] Ensure unknown/unsupported values fail fast with a clear error (Backtrader resampling handles synthetic timeframes). - [x] Add config knobs and tests for cache concurrency and resolution handling. - [x] Update docs at phase end: docs/README.md, docs/Architecture.md (including the decision log), and docs/tasks.md.

Questions: - [x] Confirm whether FYERS supports weekly/monthly resolutions for history(); if not, rely on Backtrader resampling only (no synthetic caching in SQLite). Decision: weekly/monthly not used; Backtrader resampling only. - [x] Should “update historical data” mean forced refresh that overwrites existing bars in a range, or only gap fill? Decision: gap fill only. - [x] How many concurrent Python processes will write to the same SQLite cache, and do you expect high write contention? Decision: multi-process sharing one DB; contention expected at startup/bursts; tune via config/cache.yaml: concurrency. - [x] For rate limits, do you want per-process limits (current) or a shared/global limiter? Decision: per-process limits (reduce per-process values when running many processes). - [x] Cache scope: all NSE instruments proactively, or on-demand when strategies request history? Decision: on-demand. - [x] Cache location preference and acceptable disk growth limits? Decision: one shared path via config/cache.yaml: db_path; treat DB as rebuildable cache and cap growth operationally by choosing scope/base resolution.

Phase 3: Live L1 Data + Backfill

[x] Implement data WS client for L1 symbol updates.
[x] Gap detection + REST backfill integration.
[x] Reconnect and resubscribe behavior with idempotent state.
[x] Allow WS adapters to load and validate config/fyers_ws.yaml via config_path.
[x] Enforce 100-symbol batching for data WS subscribe/unsubscribe calls.
[x] Make data WS batch size configurable (data_ws_batch_size, 1-100).
[x] Add data WS subscribe throttling and idle health warnings.
[x] Update README.md.

Phase 4: Broker Sync Core

[x] Implement REST submit/modify/cancel and order snapshot sync.
[x] Implement order WS parser for orders/trades/positions.
[x] Implement partial fill handling + idempotent fill keys.
[x] Backtrader order lifecycle invariants and notifications.
[x] Map order WS messages into Backtrader orders (translate FYERS payloads to bt.Order states and trigger notify_order/notify_store in the broker loop).
[x] Broker cash/value wiring from REST funds/positions snapshots (replace hardcoded getcash/getvalue).
[x] Update README.md.
[x] Backtrader contract alignment: create bt.Order objects on submit, map order.ref ↔ FYERS order_id, auto-register for notifications, and return bt.Order from buy/sell.
[x] Startup sync: seed orders/trades/positions from REST snapshots on broker.start().
[x] Automatic notification loop: ensure dispatch_order_notifications is called from the main loop (e.g., data feed tick) without strategy intervention.
[x] Position sync: update Backtrader Position objects from fills/snapshots (self.position/getposition correctness).
[x] Order ID aliasing and signed fill handling; periodic REST reconciliation configurable via config/broker.yaml.

Checkpoint (after Phase 4)

[x] Human review: read docs/critical_eval_2.md before proceeding to Phase 5.
[x] Consolidate critical eval findings into actionable tasks in docs/tasks.md (and update the decision log in docs/Architecture.md where needed).

Phase 5: Multi-Strategy Isolation + Persistence

[x] Clean up auth logger setup by removing dead/commented fallback code.
[x] Fix failing tests: bt.Order missing ordtype when constructed in broker._submit_and_wrap (tests/test_broker_bt_order_creation.py, tests/test_broker_order_aliases.py).
[x] Prevent store.broker overwrite; add multi-broker registry/routing for notifications.
[x] Disable CO/BO/MARGIN/MTF via broker allow list; strategies manage exits (CNC/INTRADAY only).
[x] Enable Backtrader in test env so skipped tests run (tests/test_broker_order_aliases.py, tests/test_broker_positions_sync.py).
[x] Per-strategy routing and order ownership boundaries.
[x] Crash-safe persistence and restart recovery (persist open orders/positions, on restart pull REST snapshots: orderbook/tradebook/positions/holdings/funds to rebuild state before consuming WS).
[x] Define persisted state schema (versioned) for: account_id, strategy_id, open orders, last seen tradeNumber per account, last processed WS message timestamps.
[x] Decide persistence backend for runtime state: JSON file vs SQLite table vs separate state DB (keep separate from historical bars cache).
[x] Implement src/fyers_store/state/ storage API (save/load/clear) and wire into Store/Broker lifecycle.
[x] Startup sequence: load persisted state → fetch REST snapshots → reconcile/repair state → then start WS adapters and begin consuming WS queues.
[x] Reconnect sequence: after WS reconnect, run a bounded REST reconcile (orderbook/tradebook/positions) to cover WS gaps before processing new messages.
[x] Idempotency rules: dedupe fills by tradeNumber (or synthetic key fallback), persist dedupe watermark so restarts don't double-apply fills.
[x] Failure mode: if persisted state schema mismatch/corruption, fail fast with a clear message and provide a “reset state” option.
[x] Add tests: restart recovery replay (mocked REST/WS), schema mismatch handling, and dedupe across restart.
[x] Broker cash/value wiring from REST funds/positions snapshots (replace hardcoded getcash/getvalue).
[x] Update README.md.

Phase 6: Hardening + Tests

[x] REST rate-limit throttling + backoff for 429.
[x] Add retry/backoff for FYERS auth HTTP calls (OTP/PIN/token).
[x] Add explicit adapter-level WS reconnect backoff + jitter (avoid SDK-only reconnect behavior).
[x] REST order endpoint hardening + unit tests.
What/Why: enforce explicit timeout/backoff rules and user-facing error messages so order flows either complete or surface "unknown state" guidance instead of silent retries; validate with mocked SDK failures to prevent regressions.
[x] Confirm FYERS order APIs do not support client-provided idempotency keys (assume "no" for now).
[x] place_order: do not auto-retry; on timeout/exception, raise an "unknown state" error and recommend reconciliation via Order WS + orderbook().
[x] modify_order/cancel_order/orderbook: add bounded retry/backoff+jitter for transient failures (429/5xx/network) with clear logs.
[x] Add unit tests with mocked SDK exceptions to validate: no retry on place, retries on cancel/modify/orderbook, and error messages are actionable.
[x] Reconcile fail-fast policy + disconnect/reconnect integration tests.
What/Why: define a strict stop-the-world rule on reconcile failure so strategies do not trade on stale state; prove via mocked reconnect/disconnect scenarios that the broker halts and logs actionable instructions when snapshots cannot be trusted.
[x] Define a strict policy: if reconcile fails, raise a clear exception to stop the strategy/broker instead of silently backing off.
[x] Add tests that force sync_order_state() to fail and assert the failure surfaces immediately with actionable messaging.
[x] Mocked integration tests for WS reconnect/disconnect to verify fail-fast behavior and bounded REST reconcile on restart.
[x] Position sync edge-case coverage.
What/Why: broaden normalization tests so schema variants, empty snapshots, closed-position removals, and short/invalid quantities are handled deterministically, preventing silent drift in Backtrader positions.
[x] Cover schema variants (positions vs netPositions), empty snapshots, and removal of closed positions.
[x] Cover shorts (negative qty) and missing/invalid numeric fields handling.
[x] Load tests for 1000 symbols (mocked).
What/Why: exercise subscription batching, throttling, and queue handling at scale using mocked WS/REST to ensure performance ceilings are understood without live API usage.
[x] Update README.md.
- What/Why: document the Phase 6 hardening behaviors (timeouts/retries/fail-fast rules, position handling, and test coverage) so operators know expected runtime responses.

Phase 7: Auth Expiry + Backtest Reliability (To-Do)

Background: In production, FYERS sessions can expire daily and can also be invalidated immediately when the same credentials are used from another device. We observed the following gaps while running the backtest example and diagnosing expired-token behavior.

Gaps observed: - Token expiry/invalidation is not consistently surfaced across endpoints. Some SDK calls return explicit auth errors (example: code=-8 + "token has expired"), while others degrade into s='no_data' with code=200 and missing candles, which can create misleading "no data" flows. - With an invalid session, REST history pagination/chunking can repeatedly return no_data, producing log spam and wasting time instead of failing fast. - WS adapters can reconnect forever even when the token is invalid, causing reconnect storms and "stuck" strategies with no clear operator action. - Backtests can fail with cryptic Backtrader indicator errors (e.g., EMA array bounds) when the feed yields too few/zero bars. - Operators expected token-expiry errors to appear in REST log files; errors may instead only show up in FYERS SDK internal logs or in stdout depending on setup_logging() and log_path wiring. - Cache vs REST behavior can be surprising: if gaps are detected (or the cache DB path differs), Store may hit REST even when the operator believes data was already cached.

Core (Must Implement): - [x] Define a single repo-standard AuthError + operator message. - [x] Add Store startup preflight auth validation. - [x] Add runtime REST auth failure detection; stop retry storms; treat no_data as unknown until auth validated. - [x] Add WS auth failure detection and stop reconnect storms. - [x] Add opt-in, single-attempt token refresh/restart orchestration. - [x] Backtest hardening: min bars + actionable errors (canonical backtest entrypoint). - [x] Add unit tests for REST/WS auth paths.

New Additions: - [x] Define clean process exit behavior on AuthError (entrypoints catch, log, sys.exit(1)). - [x] Add recovery guardrails (one attempt only, logged).

Optional / Later:

Phase 8: Distributable Library Conversion (Spec Alignment)

[x] Add centralized resource resolver with YAML version enforcement and env overrides.
[x] Refactor config loaders to use resolver paths (no CWD dependencies).
[x] Update config templates to include version: 1 and ship packaged defaults.
[x] Normalize logging defaults (NullHandler + packaged logging config fallback).
[x] Add trading environment guards, REST/WS timeout enforcement, and Store health_check.
[x] Add packaging metadata (pyproject), entrypoints, and migration CLI.
[x] Add example workflows and finalize docs updates.

Phase 9: Documentation Wiki (Operator-Focused)

[x] Create entrypoint docs (docs/README.md, docs/QUICKSTART.md) and core architecture/ops/config pages.
[x] Add usage, developer deep dive, FYERS mapping, troubleshooting, testing, migration docs, and refresh visuals.

Phase 10: Namespace Cleanup

[x] Move resolver implementation under src/fyers_store/utils/ and remove the top-level resolver module.

Phase 11: Docs Hosting

[x] Add mkdocs.yml and a GitHub Actions workflow for Cloudflare Pages deployment.

To-Do (Rolling)

[x] Phase 6 hardening: README update.
Removed: docs/gemini_improvements.md (spec rolled into current docs and decision log).
[ ] Full troubleshooting checklist.
[ ] Optional metrics/log counters for auth events.
[ ] Extended history behavior docs.
[ ] Cache-only backtest mode.
[ ] Cache-mode unit tests.
[ ] Watchdog/auto-restart hooks (deferred to a later operational phase).