Skip to content

Overview

This repository provides a Backtrader Store/Broker/DataFeed integration for FYERS. It is designed for reliability and operational clarity in live trading, while remaining testable and deterministic in a local environment. It is also usable outside Backtrader as a set of small, composable adapters plus a thin orchestration layer.

What This Project Delivers

This project focuses on the parts that typically fail in production systems: normalization, resilience, and recovery.

  • It provides a Store that owns adapter lifecycles and exposes poll-friendly helpers for Backtrader.
  • It provides a DataFeed that can serve both historical and live market data without blocking the Backtrader loop.
  • It provides a Broker that maps FYERS order and trade updates into Backtrader order lifecycle events, including partial fills and idempotency.
  • It provides a SQLite historical cache (WAL) with market-aware gap detection so repeated strategy runs do not repeatedly hit FYERS history endpoints.
  • It provides crash-safe runtime state persistence in a separate SQLite database so restarts can recover without double-applying fills.

How To Read The Documentation

This documentation is split by intent so that operators and developers can quickly find what they need.

  • docs/README.md is the entrypoint for setup, configuration, and quickstart.
  • docs/Architecture.md explains the system structure, data flows, and the decision log.
  • docs/visuals/ contains Mermaid diagrams for each major subsystem and supporting utilities/configuration.
  • docs/fyers_mapping.md is the canonical FYERS SDK and WebSocket mapping reference and includes sample payloads.
  • docs/ops.md contains operational guidance, including restart strategies and scheduler examples.
  • docs/tasks.md tracks phase tasks, hard gates, and rolling to-dos.

Important Operational Assumptions (FYERS Reality)

FYERS imposes a few operational constraints that directly shape the design of this repository. These constraints are important because they explain several "fail fast" and "unknown state" behaviors in the Broker and REST adapter.

  • FYERS order APIs do not support client-provided idempotency keys. This means that "retrying a place order call" can create duplicate real orders when the first call actually reached the server but the client timed out or crashed before receiving a response.
  • The FYERS SDK does not consistently enforce a per-request timeout for every API call. This means that an SDK call can hang longer than expected under network or server issues unless the SDK itself exposes a timeout hook for that call.
  • FYERS API sessions automatically expire at the end of each trading day, which requires a fresh login daily for live trading continuity.
  • FYERS sessions can also expire immediately due to multi-device invalidation if you log into the API from a different device using the same credentials.

In this repo, these realities are handled by avoiding unsafe retries on place_order, requiring explicit reconciliation of state via Order WebSocket and REST snapshots, and treating reconciliation failures as a critical failure that should halt trading rather than "carry on with stale assumptions".

Repository Structure (Business-Friendly Map)

The top-level directories separate configuration, application code, documentation, and tests so that deployments can be audited and reproduced.

  • The src/ directory contains all application code, including the FYERS Store/Broker/DataFeed implementation, adapters, persistence layers, and logging/auth helpers.
  • The config/ directory contains YAML configuration files that define runtime behavior such as credentials, caching paths, WebSocket behavior, logging, and rate-limit tuning.
  • The tests/ directory contains pytest tests. Most tests are fully mocked and do not require live FYERS connectivity.
  • The docs/ directory contains the documentation set.
  • The logs/ directory is the default sink for rotating logs and is created and managed by the logging setup.
  • The secrets/ directory stores runtime tokens and should never be committed.
  • The tmp/ directory is a scratch space used by tests and local workflows.

Example Workflows (Quick Start)

The examples under examples/ are meant to be copy-paste starting points. They show how to wire the Store, adapters, and strategy logic for common workflows.

  • examples/portfolio_backtest_two_symbols.py runs a multi-stock Backtrader backtest from FYERS history (REST + cache).
  • examples/backtest_to_live.py runs a backtest first, then switches to live trading with the same strategy.
  • examples/diy_backtest_no_backtrader.py shows how to run a simple SMA backtest without Backtrader.
  • examples/live_signal_loop.py demonstrates a guarded live loop that drains WS data, checks triggers, and handles order WS updates.

Live examples assume you have config/state.yaml configured and FYERS_ENVIRONMENT=LIVE (or allow_live_orders: true) when you intend to place real orders.

Key Configuration (What Operators Typically Change)

This project is intentionally configuration-driven so that operational behavior can be tuned without code changes.

  • The file config/fyers_credentials.yaml contains FYERS authentication inputs and retry/backoff tuning for auth calls. This file is sensitive and should be handled like a secret.
  • The file config/cache.yaml defines the required SQLite historical cache path. This cache is treated as rebuildable, but it is valuable for performance and rate-limit protection.
  • The file config/state.yaml defines the separate SQLite runtime state database used for crash-safe recovery. This path must be different from the historical cache path.
  • The file config/fyers_ws.yaml defines WebSocket settings (data_type, reconnect policy, SDK logging flags, batch size, throttling, health checks). The WebSocket adapters validate this configuration and treat it as the source of truth when provided.
  • The file config/fyers_rest_limits.yaml defines REST rate limits and backoff parameters used by the REST adapter to reduce 429/503 pressure.
  • The file config/logging_config.yaml defines log handlers and category routing so operational errors and high-volume traffic can be separated.
  • The file config/broker.yaml defines broker-specific runtime knobs such as reconcile intervals and allow lists.
  • The file config/data_feed.yaml defines DataFeed drain limits so live processing remains bounded under load.
  • The file config/nse_calendar.yaml defines market session and holiday rules so historical gap detection avoids pointless refetches on non-trading days.
  • The file config/symbol_master_sources.yaml defines symbol master CSV sources and headers as a planned configuration surface. It is currently not consumed by src/ and exists so future symbol master tooling can be wired without changing code.

Key Modules (What Developers Typically Touch)

The code is organized so that business logic is separated from transport, and the Backtrader-facing surface stays small and stable.

  • The module src/fyers_store/store.py owns adapter lifecycles and exposes the polling and DB-first history helpers used by Backtrader integration.
  • The module src/fyers_store/data_feed.py drains the market WebSocket queue and normalizes messages into bars. It is designed to avoid blocking and to handle high message rates.
  • The module src/fyers_store/broker.py builds order intents, maps FYERS order/trade updates to Backtrader order lifecycle events, and enforces fail-fast reconciliation rules.
  • The package src/fyers_store/adapters/ contains thin transport adapters. The REST adapter owns auth/session usage and backoff logic, while WebSocket adapters own reconnect/backoff and subscription tracking.
  • The package src/fyers_store/persistence/ contains the historical cache interface and the SQLite/WAL implementation.
  • The package src/fyers_store/state/ contains the runtime state persistence layer used for restart recovery.
  • The package src/logging_module/ contains logging configuration and category-based routing helpers.
  • The module src/fyers/auth.py contains FYERS auth flow helpers and client initialization.

For visual maps of these modules and their interactions, see docs/visuals/README.md.

Core Runtime Flows

Historical Data (Cache-First, Gap-Fill)

When a strategy requests historical bars, the Store reads from the local SQLite cache first. If gaps are detected in the requested range, only the missing windows are fetched from FYERS history and then upserted back into the cache. Gap detection respects market session rules so weekends and configured holidays do not cause repeated refetch attempts.

Live Market Data (Queue-Based Polling)

The market WebSocket adapter pushes messages into a queue. The DataFeed drains that queue in bounded batches with a time budget to keep the Backtrader loop responsive. Live updates are normalized into bars (including volume delta logic) and invalid or out-of-order messages are dropped deterministically.

Orders and Trades (REST Actions, WS Notifications)

Order create/modify/cancel calls are REST-only. Order and trade updates arrive through the order WebSocket and are translated into Backtrader order events. Partial fills are supported, and fills are deduped using trade identifiers so restarts do not double-apply executions.

Restart Recovery

At process start, the Broker loads persisted state from the runtime state DB and reconciles with REST snapshots before consuming WebSocket messages. If reconciliation fails, the broker fails fast to avoid trading on stale state. Operational restarts should be owned by an external scheduler, as documented in docs/ops.md.

Testing Strategy

Most tests are mocked and focus on deterministic behavior under edge cases such as partial fills, reconnect sequences, cache gap detection, and retry policy rules. A small number of integration-style tests exist, and any tests that require live FYERS connectivity should be clearly marked and safe to skip in CI environments.

Source Code Walkthrough (Module-by-Module)

This section is intentionally detailed. It walks the src/ tree in a "module-by-module" order, and for each module it answers the following questions in plain English.

  1. What the module does, and why it exists in the system.
  2. Which configuration files and environment variables it depends on.
  3. How you use it as part of Backtrader.
  4. How you use it outside Backtrader.
  5. Which limitations and operational caveats you should be aware of.

If you prefer diagrams instead of text, start with docs/visuals/system_overview.md and then open the per-component diagrams referenced throughout this walkthrough.

src/__init__.py (Repository Package Marker)

This file exists only to mark src/ as a Python package root for import purposes. It is intentionally empty and has no runtime behavior.

src/main.py (Sample Entry Point)

This file exists to provide a minimal "smoke test" that verifies credentials, token handling, network access, and logging setup without involving Backtrader.

  • What it does: It configures logging, loads the FYERS auth configuration, creates an SDK client, calls get_profile(), and logs the response.
  • Configuration dependencies: It relies on config/fyers_credentials.yaml via fyers.auth.config_from_yaml(). It writes token material to the token path configured in that YAML (default is secrets/fyers_token.json).
  • Utility dependencies: It relies on logging_module.config.setup_logging() and logging_module.utils.get_logger(...).
  • How to use with Backtrader: You do not use this file with Backtrader. It is only a standalone entry point.
  • How to use outside Backtrader: Run python src/main.py as a credential and connectivity check.
  • Limitations: This file does not wire the Store/Broker/DataFeed. It only validates authentication and basic API access.

src/fyers/ (FYERS Authentication and Client Helpers)

The src/fyers/ package contains the login flow and token persistence logic. It is intentionally separate from the Store and adapters so authentication can be tested and reasoned about in isolation.

src/fyers/__init__.py

This file exists only to mark src/fyers/ as a Python package. It currently does not define a public API surface and is intentionally empty.

src/fyers/auth.py

This module provides the FYERS authentication flow and a helper that returns a ready-to-use FYERS SDK client.

  • What it does: It implements the multi-step FYERS login flow (OTP/TOTP + PIN + auth code + token exchange), persists the access token to disk, and reuses a valid token when possible to avoid repeated logins during the day.
  • Primary public surface:
  • The dataclass FyersAuthConfig defines the configuration schema. It is designed so YAML keys can match field names directly.
  • The function config_from_yaml(...) loads config/fyers_credentials.yaml into FyersAuthConfig.
  • The function get_fyers_client(config) returns a FYERS SDK client instance initialized with a usable access token (either loaded from disk or freshly generated).
  • Configuration dependencies:
  • The default credentials file is config/fyers_credentials.yaml. This file is sensitive and should be handled like a secret.
  • The environment variable FYERS_CREDENTIALS_YAML can override the credentials file path when running this module directly (for example, python -m fyers.auth).
  • Token storage defaults to secrets/fyers_token.json via the token_path field. This file should never be committed.
  • Auth request timeouts and retry/backoff are controlled by the config fields request_timeout_seconds, auth_max_retries, auth_backoff_base_seconds, auth_backoff_max_seconds, and auth_backoff_jitter_seconds.
  • The field token_refresh_time_ist controls the daily cutoff time used to decide whether a cached token should be treated as still valid.
  • Utility dependencies: It uses logging_module.utils.fyers_auth_logger(...) so auth-related logs can be routed separately from trading logs.
  • How to use with Backtrader:
  • In normal operation, you use this module indirectly. You obtain a token or client here and then pass it into the REST/WS adapters that the Store will use.
  • You should treat "auth" and "trading" as separate concerns. Auth creates and refreshes the session token, while Store/Broker/DataFeed use that session to trade and stream data.
  • How to use outside Backtrader:
  • You can use this module directly in scripts to obtain a token and build a FYERS client for one-off operations (profile fetch, order snapshots, history pulls, and so on).
  • Limitations and caveats:
  • Token validity is treated as a daily concept. This aligns with FYERS sessions expiring at the end of the trading day, but you should still expect unexpected invalidation due to multi-device logins.
  • This module makes network calls and should not be invoked during unit tests unless explicitly mocked.

src/logging_module/ (Logging Configuration and Category Routing)

The src/logging_module/ package exists so operational visibility is consistent across the system. It provides a single "logging setup entry point" and a category mechanism that can route different types of messages to different log files.

src/logging_module/__init__.py

This file exists only to mark src/logging_module/ as a Python package. It currently does not define a public API surface and is intentionally empty.

src/logging_module/config.py

This module is responsible for loading a YAML logging configuration and applying it exactly once per process.

  • What it does: It reads config/logging_config.yaml, validates that referenced handlers actually exist, creates any required log directories, and applies the configuration via logging.config.dictConfig(...).
  • Configuration dependencies: It uses config/logging_config.yaml by default. The default path is resolved relative to the repository, which keeps behavior consistent across local runs and deployments.
  • How to use with Backtrader: Call logging_module.config.setup_logging() before wiring the Store/Broker/DataFeed so category-based routing is active from the first log line.
  • How to use outside Backtrader: Call setup_logging() at the start of any script that uses the FYERS modules or adapters.
  • Limitations and caveats:
  • If you force a reload (force=True), root handlers are cleared so you do not get duplicated log lines. This is deliberate and is safer than "silently stacking handlers".

Related diagrams and docs: - docs/visuals/logging.md

src/logging_module/utils.py

This module provides the "category" abstraction used by the rest of the codebase.

  • What it does: It returns logging.LoggerAdapter instances that always carry a category field, which the YAML logging configuration can route to dedicated log files.
  • Why it matters: Category routing lets you separate high-volume telemetry (for example, sampled WebSocket payloads) from critical failures (for example, broker reconciliation errors).
  • How to use with Backtrader: Most modules already use this helper internally. You only need to call it directly when you add new modules or want a category-specific logger in your own strategy code.
  • How to use outside Backtrader: Use it in scripts to keep logs consistent with the rest of the repository.
  • Limitations: Categories are a convention. If you bypass the helper and log directly with the Python logger, your logs may not be routed as expected.

src/logging_module/filters.py

This module defines filters that are referenced by config/logging_config.yaml to route messages.

  • What it does: It provides a generic CategoryFilter and a specialized auth filter that detects auth/token messages so they can be routed to dedicated error logs.
  • How to use with Backtrader: You normally do not call this module directly. It is wired through YAML configuration.
  • How to use outside Backtrader: You typically do not call this module directly, unless you are building your own logging config and want to reuse the same filters.
  • Limitations: Filters rely on the category field being present, so the system expects callers to use logging_module.utils.get_logger(...) for consistent routing.

src/fyers_store/ (Backtrader Integration Layer)

The src/fyers_store/ package contains the Store/Broker/DataFeed and the internal utilities they depend on. It is organized so that transport concerns (REST and WebSocket) are isolated under adapters/, domain normalization lives in models/, and persistence logic is split into "historical cache" versus "runtime state".

If you only remember one thing about this package, it should be this. The Store is an orchestrator, adapters transport, and the broker and feed translate between FYERS and Backtrader expectations.

Related diagrams and docs: - docs/visuals/system_overview.md - docs/visuals/store.md - docs/visuals/broker.md - docs/visuals/data_feed.md - docs/visuals/adapters.md - docs/visuals/historical_cache.md - docs/visuals/state_persistence.md

src/fyers_store/__init__.py (Public API Boundary)

This file defines what external callers should treat as "stable imports" for the fyers_store package.

  • What it does: It re-exports FyersStore, FyersBroker, FyersData, and basic order enums so users can import them from fyers_store directly.
  • How to use with Backtrader: Import the Store/Broker/DataFeed from fyers_store as your stable entry point rather than importing deep internal modules.
  • How to use outside Backtrader: You can still import the Store/Broker/DataFeed from here, but for lower-level usage you may prefer importing adapters and persistence modules directly.
  • Limitations: Re-exported names are intended to remain stable, but internal module paths are allowed to change during refactors.

src/fyers_store/constants.py (Order Constants)

This module holds FYERS order constants that are used across broker and adapter payload construction.

  • What it does: It defines OrderSide and OrderType enums and a small label mapping that keeps intent payloads readable.
  • How to use with Backtrader: Use these enums when building explicit "order intent" dictionaries to pass into FyersBroker.buy(...) or FyersBroker.sell(...).
  • How to use outside Backtrader: Use these enums when building raw order payloads for FYERS REST calls (or when validating payloads in your own integration).
  • Limitations: These constants mirror FYERS payload conventions. If FYERS changes its API contract, these constants may need updates.

src/fyers_store/models/ (Normalized Domain Types)

The models package exists so that "what a bar looks like" is defined once and enforced at adapter boundaries.

src/fyers_store/models/__init__.py

This file re-exports NormalizedBar and bar normalization helpers for convenience. It exists so other modules can import NormalizedBar consistently without coupling to the internal file layout.

src/fyers_store/models/domain.py

This module defines the normalized OHLCV schema used across the system.

  • What it does: It defines NormalizedBar (a typed dictionary schema) and normalize_bar(...) which validates and normalizes bars.
  • Why it matters: Adapters and the Store apply normalize_bar(...) so downstream modules do not need to keep re-validating shape and types on every bar.
  • Configuration dependencies: This module is configuration-free.
  • How to use with Backtrader: The DataFeed writes normalized bars into Backtrader line buffers. If you want to inject bars for tests, you can use FyersData.push_bar(...) and rely on this normalization logic.
  • How to use outside Backtrader: If you ingest bars from another source, you can call normalize_bar(...) as a "schema gate" before writing bars into your own database or queue.
  • Limitations and caveats:
  • The module can require timezone-aware datetimes and can optionally validate an expected timezone offset, but it cannot decide which timezone is correct without context such as resolution and exchange session rules.

src/fyers_store/resolution.py (Resolution Canonicalization)

This module ensures that resolution tokens are handled consistently across the REST adapter, cache, and Store.

  • What it does: It canonicalizes FYERS-supported resolutions (seconds, minutes, daily) and provides helpers such as resolution_to_timedelta(...), is_daily_resolution(...), and fyers_resolution_aliases(...).
  • Why it matters: A single canonical form prevents duplicated cache keys, inconsistent pagination, and silent resolution drift caused by aliases such as "1m" versus "1".
  • Configuration dependencies: This module is configuration-free.
  • How to use with Backtrader: When you request history or wire a data feed, use FYERS-native base resolutions (for example "1", "5", or "D"). If you need synthetic timeframes (for example 15m bars), you should fetch a base resolution and resample using Backtrader.
  • How to use outside Backtrader: Use canonicalize_fyers_resolution(...) as a validation layer at your boundaries so you fail fast instead of sending an invalid token to FYERS.
  • Limitations: This repository intentionally does not create synthetic resolutions because it would blur the boundary between "data transport" and "strategy-level transformation".

src/fyers_store/market_calendar.py (NSE Calendar for Gap Detection)

This module provides a lightweight NSE trading calendar so the historical cache can distinguish "real missing bars" from "expected no-trade time".

  • What it does: It loads config/nse_calendar.yaml and provides an NseCalendar object with helpers to check trading days, session bounds, and whether a time window contains any expected bar timestamps.
  • How it is used: SQLiteHistoricalCache.detect_gaps(...) uses it to filter out gaps that fall entirely in weekends, holidays, and off-session periods, which reduces pointless REST backfills.
  • Configuration dependencies: It reads config/nse_calendar.yaml. The holiday_list_url key is treated as metadata and is not fetched automatically in-process.
  • How to use with Backtrader: You generally do not call this module directly. It exists to make the historical cache's gap detection market-aware.
  • How to use outside Backtrader: If you build your own ingestion process, you can use the calendar to decide whether a missing span is actionable or expected.
  • Limitations and caveats:
  • If the calendar file is missing or invalid, the cache falls back to naive gap detection, which can cause extra REST history calls around weekends and holidays.
  • The calendar is intentionally local-file-driven so the trading process never depends on network access for calendar updates.

src/fyers_store/persistence/ (Historical Cache)

This package provides a pluggable persistence contract for historical bars and a default SQLite/WAL implementation.

src/fyers_store/persistence/__init__.py

This file defines the public exports for the persistence package.

  • What it does: It re-exports the HistoricalCache contract and the default SQLiteHistoricalCache implementation, along with config helpers such as resolve_cache_db_path(...).
  • How to use with Backtrader: You normally do not import this directly. The Store uses these exports to create the default cache when you did not inject your own.
  • How to use outside Backtrader: You can import the cache implementation directly if you want cache-first history without the Store.
src/fyers_store/persistence/cache.py (Cache Contract)

This module defines the abstract contract that all historical caches must implement.

  • What it does: It defines the HistoricalCache interface and the Gap structure used by the Store to decide what needs to be backfilled from REST.
  • Why it matters: The Store can depend on "what a cache does" without coupling to "which database implementation is used".
  • How to use with Backtrader: You normally do not call this contract directly, but it is a useful reference if you plan to plug in another backend.
  • How to use outside Backtrader: If you implement your own historical cache (for example a shared Postgres cache), this interface defines the required behavior.
src/fyers_store/persistence/sqlite_schema.py (Cache Schema Helpers)

This module centralizes the SQLite DDL and WAL enabling logic for the historical cache.

  • What it does: It defines the bars and bar_watermarks tables, creates indexes, and enables WAL mode with sane PRAGMA tuning.
  • How it is used: SQLiteHistoricalCache.initialize() calls into this module to ensure the DB is ready.
  • Limitations and caveats:
  • This module intentionally does not act as a general migration framework. The history cache is treated as rebuildable, so schema changes are handled operationally.
src/fyers_store/persistence/sqlite_cache.py (SQLite/WAL Historical Cache)

This is the default "zero-ops" historical cache backend.

  • What it does:
  • It stores bars in SQLite with a primary key on (symbol, resolution, ts) so upserts are naturally idempotent.
  • It stores per-context watermarks so different ingestion contexts do not block each other.
  • It detects gaps and can filter out "expected gaps" using the NSE calendar when configured.
  • It supports optional background maintenance such as WAL checkpointing and lightweight integrity checks.
  • Configuration dependencies:
  • config/cache.yaml must provide db_path. The Store will fail fast if it is missing.
  • config/cache.yaml can optionally provide a maintenance: section with keys such as enabled, interval_seconds, start_jitter_seconds, and wal_checkpoint_mode.
  • config/cache.yaml can optionally provide a concurrency: section with keys such as busy_timeout_ms and bounded write lock retry/backoff settings.
  • config/nse_calendar.yaml is optionally loaded to provide market-aware gap detection.
  • How to use with Backtrader: You usually do not instantiate this cache yourself. The Store will create it on startup if you did not inject a cache instance.
  • How to use outside Backtrader: You can instantiate SQLiteHistoricalCache directly if you want a cache-backed history process without the Store.
  • Limitations and caveats:
  • SQLite supports many readers and a single concurrent writer. In multi-process deployments (multiple strategies/processes sharing the same cache DB), you should expect occasional database is locked errors, which this module mitigates using WAL mode, busy_timeout, and bounded retry/backoff.
  • The cache is treated as rebuildable. If you change the bar schema or cache layout, you should plan an operational cache reset rather than an in-place migration.

src/fyers_store/state/ (Crash-Safe Runtime State)

This package is separate from the historical cache because "historical bars" are rebuildable, while "runtime broker state" is used to prevent duplicate fill processing after crashes.

src/fyers_store/state/__init__.py

This file defines the public exports for the state package.

  • What it does: It re-exports the SQLite state store types and the config loader so the Store and Broker can import them cleanly.
  • How to use with Backtrader: You typically do not import this module directly. The Broker and Store handle state persistence for you.
  • How to use outside Backtrader: If you build your own broker loop, importing the state store types from here is the cleanest entry point.
src/fyers_store/state/state_store.py (SQLite State Store)

This module provides the SQLite state store used by the broker for crash-safe restart recovery.

  • What it does:
  • It loads a state config from config/state.yaml (with environment overrides).
  • It creates a versioned schema for orders, positions, trade dedupe identifiers, and WebSocket processing watermarks.
  • It exposes load/upsert/delete methods so the broker can persist snapshots deterministically.
  • Configuration dependencies:
  • config/state.yaml must define state_db_path and account_id.
  • Environment overrides are supported via FYERS_STATE_YAML, FYERS_STATE_DB_PATH, and FYERS_STATE_ACCOUNT_ID.
  • How to use with Backtrader: You normally do not interact with this module directly. The Store creates a per-broker state store using the configured account id and a store-managed strategy id.
  • How to use outside Backtrader: You can use this module directly to store orders, positions, and trade dedupe keys if you implement your own event loop.
  • Limitations and caveats:
  • Schema upgrades are intentionally fail-fast. If the schema version changes, you are expected to delete the state DB (or call clear_state()) and restart with a clean store.
  • The Store enforces that the state DB path must be different from the historical cache DB path to prevent accidental mixing of concerns.

src/fyers_store/adapters/ (Transport Adapters)

The adapter layer exists so "transport and retries" are isolated from "business decisions". In practice, this means the following.

  • The Broker constructs order payloads and implements the order lifecycle.
  • The REST adapter only transports payloads and enforces safe retry/backoff policies.
  • The WebSocket adapters only manage connectivity, reconnect, subscriptions, and message queues.
src/fyers_store/adapters/__init__.py

This file re-exports the adapter interfaces and the concrete implementations so importing adapters is ergonomic and stable.

  • What it does: It exports RestAdapter, WsAdapter, MessageQueue, and the concrete FYERS SDK-backed adapter classes.
  • How to use with Backtrader: You typically create adapter instances and inject them into the Store.
  • How to use outside Backtrader: You can import and use adapters directly without involving the Store if you want lower-level control.
src/fyers_store/adapters/interfaces.py (Adapter Contracts)

This module defines the adapter contracts used across the system.

  • What it does: It defines the abstract interfaces RestAdapter and WsAdapter, plus a minimal MessageQueue protocol that supports non-blocking polling.
  • Why it matters: The Store and DataFeed can depend on these interfaces and remain stable even if the adapter implementation changes (for example, switching from FYERS SDK to a direct HTTP client).
  • How to use with Backtrader: You do not usually import this module directly. It is primarily an internal boundary.
  • How to use outside Backtrader: If you want to implement your own adapters (for example, a fully mocked adapter, or a broker simulator), this contract defines the minimal required behavior.
src/fyers_store/adapters/fyers_rest.py (FYERS REST Adapter)

This module provides the concrete REST adapter backed by the FYERS SDK.

  • What it does:
  • It owns FYERS SDK client initialization and therefore owns the REST "session lifecycle".
  • It normalizes history bars and enforces a timezone policy for history boundaries (intraday UTC, daily IST by default).
  • It applies a per-second and per-minute limiter for history calls, plus bounded retry/backoff for retryable failures.
  • It hardens order calls by making place_order single-shot and making modify/cancel/orderbook calls retryable with bounded backoff.
  • Configuration dependencies:
  • It reads rate limit and backoff settings from config/fyers_rest_limits.yaml by default.
  • It depends on a valid FYERS SDK client or an auth config produced by src/fyers/auth.py.
  • How to use with Backtrader: Provide an instance of FyersSdkRestAdapter to a FyersStore, then let the Broker and Store call it.
  • How to use outside Backtrader: You can use FyersSdkRestAdapter directly for history pulls and account/order snapshots.
  • Limitations and caveats:
  • place_order(...) does not retry by design because FYERS does not support client-provided idempotency keys. If it fails, you must reconcile via Order WebSocket and/or orderbook() before deciding whether to place another order.
  • The adapter does not guarantee per-request timeouts for SDK calls because the FYERS SDK may not expose a uniform timeout mechanism per call.
  • The current RateLimiter enforces only per-second and per-minute budgets. The per_day setting exists for operator visibility but is not currently enforced in-process.
src/fyers_store/adapters/fyers_ws.py (FYERS WebSocket Adapters)

This module provides queue-based WebSocket adapters for market data and order updates with explicit reconnect backoff.

  • What it does:
  • It validates and loads config/fyers_ws.yaml so configuration errors are caught early and predictably.
  • It wraps FYERS SDK WebSocket clients in adapters that push parsed messages into an internal queue.
  • It implements a reconnect loop with exponential backoff and jitter, and it deliberately disables SDK auto-reconnect so reconnect behavior is deterministic and testable.
  • It implements subscription batching to respect FYERS guidance (batches of up to 100 symbols) and optional subscription cooldown to reduce subscribe/unsubscribe bursts.
  • It optionally runs a health monitor thread that warns when message flow becomes stale.
  • Configuration dependencies:
  • It reads config/fyers_ws.yaml via load_ws_config(...).
  • The config includes shared keys (logging flags, reconnect settings) and data-socket-specific keys (batch size, queue sizing, health checks).
  • How to use with Backtrader:
  • Inject FyersDataWsAdapter (market data) into the Store and let FyersData drain it.
  • Inject FyersOrderWsAdapter (order updates) into the Store and let FyersBroker consume it via dispatch_order_notifications(...).
  • For tests and demos, InMemoryWsAdapter can be used to simulate WS messages deterministically.
  • How to use outside Backtrader: You can use the adapters in your own loop by periodically draining adapter.get_queue() and processing the returned message mappings.
  • Limitations and caveats:
  • WebSocket messages are queue-based. If your consumer does not drain regularly, backlogs can build and memory can grow if the queue is unbounded.
  • Subscription batching and throttling are important for large symbol sets. The adapter is designed to enforce safe defaults, but your config still determines how aggressive the process is.

src/fyers_store/store.py (Store / Orchestration Layer)

The Store is the central orchestrator for adapters and persistence. It is deliberately kept thin so it is easy to audit and reason about in live trading environments.

  • What it does:
  • It owns adapter start/stop lifecycle and keeps explicit "started" flags to avoid double-start behavior.
  • It provides a DB-first history method (get_history) that uses the historical cache and only calls REST for missing windows.
  • It provides passthrough wrappers for REST calls such as orderbook(), positions(), and funds().
  • It provides poll-friendly queue drain methods for market and order WebSocket messages (drain_market_messages, drain_order_messages).
  • It supports multi-broker routing so multiple brokers (and therefore multiple strategies) can share one Store while keeping order updates isolated.
  • It creates a per-broker state store (SQLiteStateStore) using account_id (from config) and a store-managed strategy_id.
  • Configuration dependencies:
  • config/cache.yaml is required when you want historical caching. The required key is db_path.
  • Optional cache tuning is read from config/cache.yaml under maintenance: and concurrency:.
  • config/state.yaml is required for broker crash recovery state (state_db_path, account_id), and the Store enforces that the state DB and history cache DB are not the same file.
  • The Store does not directly read config/fyers_ws.yaml or config/fyers_rest_limits.yaml. Those configs are consumed by the adapters themselves.
  • Utility dependencies: It uses logging_module.utils.get_logger(...), the cache exports from fyers_store.persistence, and the state exports from fyers_store.state.
  • How to use with Backtrader:
  • You typically instantiate one Store per process and pass it to both your Broker and DataFeed so they share sessions, caches, and routing state.
  • The Store is started indirectly by broker.start() and/or data.start(). You can also call store.start() explicitly in custom wiring.
  • How to use outside Backtrader:
  • You can treat the Store as a "connection manager" and call start(), subscribe_market(...), and drain_market_messages(...) in your own loop.
  • You can call get_history(...) to read from cache and fill gaps from REST without running Backtrader.
  • Limitations and caveats:
  • The classmethods FyersStore.getdata() and FyersStore.getbroker() are scaffolding helpers. They create a fresh Store instance per call. In most live trading setups you should share a Store instance across broker and feed to avoid duplicated sessions and split routing state.
  • Order ownership routing relies on recognizing an order id field in messages. When a message arrives with an unknown order id (for example, when REST returns only a request id), the Store routes it to the primary broker and claims ownership to keep future updates consistent.

src/fyers_store/data_feed.py (Backtrader DataFeed)

This module provides the Backtrader-facing data feed wrapper and the queue draining logic for live market messages.

  • What it does:
  • It supports a historical-only mode that prefetches history from the Store and then returns bars until the queue is empty.
  • It supports live mode by draining market WebSocket messages in bounded batches and normalizing them into NormalizedBar objects.
  • It includes stall detection so operators see when live feed delivery becomes stale.
  • Configuration dependencies:
  • It reads config/data_feed.yaml when present.
  • It uses stall_warning_seconds and the drain tuning keys market_max_messages, market_batch_size, market_drain_budget_ms, and market_backlog_warn_after.
  • Utility dependencies: It uses logging_module.utils.get_logger(...) and the bar normalization functions from fyers_store.models.
  • How to use with Backtrader:
  • Use this class as the bt.feeds data source by wiring a FyersStore instance and setting dataname (symbol) and resolution.
  • The DataFeed is designed to avoid blocking Backtrader's _load() loop, which is important for predictable cycle timing.
  • How to use outside Backtrader:
  • Outside Cerebro, you will usually consume the Store's market WS queue directly rather than using a Backtrader DataBase subclass.
  • This module still provides useful test hooks such as push_bar(...) to inject a known bar series.
  • Limitations and caveats:
  • In live mode, SymbolUpdate messages are converted into "minimal bars" where OHLC are all equal to the last traded price. This is useful for tick-level strategies that only need last price, but it is not a true OHLC bar series.
  • The drain limits are important. If your strategy is slow or you subscribe to many symbols, you should tune market_max_messages and related settings to avoid persistent queue backlogs.

src/fyers_store/broker.py (Backtrader Broker)

This module provides the Backtrader-facing broker wrapper with explicit reconciliation and crash recovery behaviors.

  • What it does:
  • It starts the Store in a safe order. It starts REST first, loads persisted state, reconciles with REST snapshots, and only then starts the order WebSocket.
  • It constructs order payloads ("intents") and delegates transport to the Store and REST adapter.
  • It consumes order WebSocket messages, maps them to Backtrader order lifecycle events, supports partial fills, and deduplicates fills by trade identifiers.
  • It persists state snapshots (open orders, positions, trade numbers, and WS watermarks) so a restart does not double-apply fills.
  • It implements a reconcile policy that is intentionally fail-fast by default. If reconciliation fails, it raises a hard exception and halts to avoid trading on stale state.
  • Configuration dependencies:
  • It reads config/broker.yaml to load reconcile_interval_seconds and allowed_product_types.
  • It uses the Store-provided state store, which depends on config/state.yaml.
  • Utility dependencies: It uses logging_module.utils.get_logger(...) and relies on Store adapter wiring for all external IO.
  • How to use with Backtrader:
  • Set this broker on your bt.Cerebro instance so orders and account state integrate with the Backtrader lifecycle.
  • Ensure that order updates are drained regularly. A common pattern is to call broker.dispatch_order_notifications(max_messages=...) via a timer so order updates are processed even when the market data feed is idle.
  • If you run multiple strategies, use one Store with multiple brokers and rely on the Store's order routing to isolate events to the correct broker.
  • How to use outside Backtrader:
  • You can use FyersBroker as a higher-level order state machine even without Cerebro by setting broker.notify_order to your own handler and calling dispatch_order_notifications(...) to process WS messages.
  • If you prefer lower-level control, you can skip the broker and call the Store's REST adapter directly, but then you own reconciliation and dedupe correctness.
  • Limitations and caveats:
  • "Unknown state" on order placement is a real operational condition. It can happen due to SDK/network timeouts, rate limits, or session expiry. Because there is no client idempotency key, the safe response is to reconcile via Order WebSocket and REST orderbook before retrying.
  • The broker's reconcile failure guidance explicitly calls out daily session expiry and multi-device invalidation, because those are common causes of sudden REST snapshot failures.
  • Reconciliation is a critical control point. If you disable fail-fast, you must accept the risk of operating with stale or incomplete state and you must own the operational playbook for what to do during outages.

Backtrader Usage Patterns (Practical Guidance)

This repository is intentionally conservative about "automagic" behavior. Most actions that can materially affect trading safety (such as order retries and reconciliation) are explicit. The patterns below describe how the pieces are intended to be used together.

For most live trading deployments, you should create exactly one FyersStore per Python process and share it across:

  • A single FyersBroker instance, which owns reconciliation and state persistence for that process.
  • One or more FyersData instances, which drain market data messages and present them to Backtrader.

This pattern keeps adapter lifecycles, cache usage, and order routing deterministic.

Explicit Market Subscription

Market data subscriptions are explicit because symbol sets can be large and strategies often need different symbol universes.

  • The FyersData feed drains messages, but it does not automatically subscribe to a symbol by itself.
  • You should call store.subscribe_market([...]) (or subscribe via your own adapter wiring) before expecting live bars.

Order Updates Need a Heartbeat

Backtrader runs notify_order(...) when it processes its internal broker notification queue. If your market data is quiet (or you are running an order-only strategy), you can end up with order updates waiting in the queue longer than you want.

  • The broker exposes dispatch_order_notifications(max_messages=...) so you can drain order WS updates even when there are no new market bars.
  • A common pattern is to call this method from a Backtrader timer (or another lightweight heartbeat) so order updates are processed regularly.

If you are running outside Cerebro, you can set broker.notify_order to your own handler and call dispatch_order_notifications(...) directly to process updates.

Failure Behavior Is Deliberately Fail-Fast

The broker reconciliation policy is designed to stop trading when the process cannot establish a correct view of account state.

  • If reconciliation fails, the broker raises a ReconcileFailureError by default.
  • Operators should treat this as a critical condition and rely on an external scheduler to restart the process after the underlying issue (session expiry, multi-device invalidation, network outage, or rate limiting) is resolved.

For operational restart strategies, see docs/ops.md.

Configuration Map (Quick Reference)

This section lists the configuration files under config/ and explains which modules use them. It is intentionally redundant with the module-by-module walkthrough so operators can answer "what do I change for X" quickly.

config/fyers_credentials.yaml

This file defines the FYERS authentication inputs used by src/fyers/auth.py. It is sensitive and should be treated like a secret.

  • Used by: src/fyers/auth.py
  • Typical required fields: totp_key, username, pin, client_id, app_id, app_type, secret_key, redirect_uri
  • Typical operational fields:
  • Token storage: token_path, token_refresh_time_ist
  • Network safety: request_timeout_seconds
  • Bounded retries: auth_max_retries, auth_backoff_base_seconds, auth_backoff_max_seconds, auth_backoff_jitter_seconds
  • Environment override: FYERS_CREDENTIALS_YAML (credentials file path when running python -m fyers.auth)

config/fyers_rest_limits.yaml

This file defines REST rate limits and backoff tuning used by the FYERS SDK-backed REST adapter.

  • Used by: src/fyers_store/adapters/fyers_rest.py
  • Key fields:
  • Limiting: per_second, per_minute
  • Backoff: max_backoff_retries, backoff_base_seconds, backoff_cap_seconds, backoff_jitter_seconds
  • Note: per_day may appear in defaults for visibility, but the current in-process limiter enforces only per-second and per-minute budgets.

config/fyers_ws.yaml

This file defines WebSocket settings and is validated strictly by load_ws_config(...).

  • Used by: src/fyers_store/adapters/fyers_ws.py
  • Required fields: log_path, write_to_file, litemode, reconnect, reconnect_retry, data_type
  • Optional queue sizing:
  • data_ws_queue_maxsize (0 means unbounded)
  • order_ws_queue_maxsize (0 means unbounded)
  • Optional data-socket subscription tuning:
  • data_ws_batch_size (must be between 1 and 100)
  • data_ws_subscribe_cooldown_seconds
  • Optional health checks:
  • data_ws_health_check_seconds
  • data_ws_stale_after_seconds
  • Optional adapter-level reconnect backoff:
  • reconnect_backoff_base_seconds
  • reconnect_backoff_max_seconds
  • reconnect_backoff_jitter_seconds
  • reconnect_stable_reset_seconds

config/cache.yaml

This file defines the historical cache location and optional maintenance/concurrency knobs. The Store treats db_path as required and fails fast if it is missing.

  • Used by: src/fyers_store/store.py, src/fyers_store/persistence/sqlite_cache.py
  • Required fields: db_path
  • Optional sections:
  • maintenance: (WAL checkpointing and periodic PRAGMA operations)
  • concurrency: (busy_timeout and bounded write-lock retry/backoff settings)

config/state.yaml

This file defines the crash-safe runtime state database used by the broker.

  • Used by: src/fyers_store/state/state_store.py, src/fyers_store/store.py, src/fyers_store/broker.py
  • Required fields: state_db_path, account_id
  • Environment overrides: FYERS_STATE_YAML, FYERS_STATE_DB_PATH, FYERS_STATE_ACCOUNT_ID
  • Important rule: state_db_path must not be the same file as the history cache DB path.

config/broker.yaml

This file defines broker runtime knobs that affect reconciliation and validation.

  • Used by: src/fyers_store/broker.py
  • Key fields:
  • reconcile_interval_seconds (set to 0 to disable periodic reconcile; this emits a warning)
  • allowed_product_types (defaults to CNC and INTRADAY if not provided)

config/data_feed.yaml

This file defines live feed drain budgets and stall warning behavior.

  • Used by: src/fyers_store/data_feed.py
  • Key fields:
  • stall_warning_seconds
  • market_max_messages, market_batch_size, market_drain_budget_ms, market_backlog_warn_after

config/nse_calendar.yaml

This file defines market session rules and holiday lists for market-aware gap detection.

  • Used by: src/fyers_store/market_calendar.py, src/fyers_store/persistence/sqlite_cache.py
  • Key fields:
  • timezone, session_start, session_end
  • weekend_days (names like mon, sat, etc)
  • holidays (a list of YYYY-MM-DD strings)
  • holiday_list_url (metadata only; not fetched in-process)

config/logging_config.yaml

This file defines Python logging handlers, formatters, filters, and logger routing.

  • Used by: src/logging_module/config.py, src/logging_module/filters.py, and the entire codebase via category logging.
  • Operational note: If logs are missing, misrouted, or duplicated, start by verifying that all referenced handler names exist and that the configured log directories are writable.

config/symbol_master_sources.yaml

This file is currently a planned configuration surface for symbol master ingestion.

  • Used by: Not consumed by src/ yet.
  • Why it exists: It allows symbol master sources and column mappings to be configured without changing code when the symbol master tooling is wired in.

Test Map (By Component)

The tests under tests/ are organized around failure modes and component responsibilities. This section provides a "where do I look" index so you can quickly find coverage for a behavior.

Auth and Token Handling

  • tests/test_fyers_auth_retries.py covers bounded retries and backoff for auth network calls.
  • tests/test_fyers_auth_timezones.py covers IST-based token validity and refresh cutoff behavior.

Logging

  • tests/test_logging_config_setup.py covers logging config validation and setup behavior.

Domain Normalization and Resolution

  • tests/test_domain.py covers NormalizedBar validation and timezone enforcement behavior.
  • tests/test_resolution.py covers resolution canonicalization rules and supported tokens.
  • tests/test_store_history_resolution_canonicalization.py covers Store-level canonicalization behavior for history requests.

Market Calendar and Gap Detection

  • tests/test_market_calendar_gap_detection.py covers trading-calendar-aware expected bar detection used by the cache.

Historical Cache (SQLite/WAL)

  • tests/test_sqlite_cache.py covers cache inserts, reads, gap detection, watermark behavior, and concurrency-related retry behavior.
  • tests/test_store_cache_config_concurrency.py covers Store wiring of cache concurrency config keys.
  • tests/test_store_history_cache.py covers Store DB-first behavior and REST gap fill behavior.

Runtime State Persistence

  • tests/test_state_persistence.py covers restart recovery behavior for orders, positions, and trade dedupe keys.

REST Adapter (Orders and History)

  • tests/test_fyers_rest_history.py covers history normalization, timezone rules, and pagination/backoff behavior.
  • tests/test_fyers_rest_orders.py covers order call hardening, retry policy boundaries, and unknown-state error signaling.

WebSocket Adapters (Market and Order Sockets)

  • tests/test_fyers_ws_config.py covers strict config validation for config/fyers_ws.yaml and related overrides.
  • tests/test_fyers_ws_backoff_policy.py covers reconnect backoff policy behavior.
  • tests/test_fyers_ws_adapter_resubscribe.py covers resubscription behavior on reconnect and batching constraints.
  • tests/test_fyers_ws_load.py covers mocked load behavior at scale (subscription batching, throttling, and queue handling).
  • tests/test_order_ws_parser.py covers order WS message parsing and routing assumptions.

Store/Broker Integration and Order Routing

  • tests/test_store_queue.py covers queue draining semantics.
  • tests/test_store_broker_routing.py and tests/test_store_strategy_ownership.py cover multi-broker routing and order ownership safety.
  • tests/test_store_order_sync.py covers REST snapshot syncing helpers.

Broker Order Lifecycle and Reconciliation

  • tests/test_broker_bt_order_creation.py covers Backtrader order wrapper creation and basic lifecycle wiring.
  • tests/test_broker_order_aliases.py covers request-id to order-id alias behavior used in unknown-order-id scenarios.
  • tests/test_broker_partial_fills.py covers partial fill aggregation and trade dedupe behavior.
  • tests/test_broker_positions_sync.py covers positions snapshot handling and edge cases.
  • tests/test_broker_product_type_allowlist.py covers broker validation of product types against allow lists.
  • tests/test_broker_reconcile_interval.py covers reconcile interval config loading and disable behavior.
  • tests/test_broker_reconcile_fail_fast.py covers fail-fast behavior and error messaging.
  • tests/test_broker_reconnect_integration.py covers broker behavior around WS reconnect and reconcile sequences.

DataFeed Behavior

  • tests/test_data_feed_live.py covers live-mode queue draining, bar normalization, and backlog/stall signaling.

Optional Integration Test

  • tests/test_macd_strategy.py is an integration-style test that may require live FYERS connectivity and valid credentials, and it is expected to be skip-safe in environments without those prerequisites.