Real-time Snapshot World Index¶

Overview¶

Description: Get near real-time of world indicies by snapshot

Specification.

The example screenshot for dataset:

SAD - System architechture design¶

Logical View¶

Below is the logical view for the pipeline sync data from provider Yahoo into the endpoint.

flowchart TB

  %% Component
  task_push[Task Declare]
  madillo[Madillo]
  postgres[Postgres]
  api[API]
  orchestration[Orchestration]

  subgraph pipeline[Extract pipeline]
    source_chart[Chart API]
    source_summary[Summary API]
    subgraph basement[Basement]
      flatten_chart[Flatten Chart]
      flatten_summary[Flatten Summary]
      join[Join by Ticker]
      dump[Dump Data]
    end

    source_chart -- Fetch (1) --> flatten_chart
    source_summary -- Fetch (2) --> flatten_summary
    flatten_chart -- (3) --> join
    flatten_summary -- (4) --> join
    join -- (5) --> dump
  end

  subgraph gcp[Google Cloud Platform]
    subgraph pubsub[PubSub]
      topic[Topic]
      sub[Subcription]
    end

    subgraph task_push[Workflow Task Push]
      payload[Payload] --> http[HTTP build Task]
    end
    subgraph sync_dataset[Workflow Sync Dataset]
      event[Event] -- call --> madillo[Pipeline Madillo] -- transfer --> postgres[Postgres]
    end

    subgraph task[Cloud Task]
      queue[Queue]
    end

    sub --> task_push <---> queue <---> sync_dataset --> api
  end

  %% Flow
  orchestration --> pipeline
  dump -- Publish message (6) --> topic
  topic -- Push subscribe into --> sub

Step 1: For the extract layer, fetch data from the provider by JSON on 2 components (Snapshot + Timeseries Close) then merged with the ticker primary key.

Step 2: Publish the payload to registed topic (PubSub) then the topics will dispatch with the subcription.

Step 3: By the subscription then will send the payload to create task into queue.

Step 4: Queue dispatch event into sync datasets

Step 5: User can query from the API itself.

Physical Plan¶

The IAM: sa-dragon-knight

ERD¶

The ERD design for the target sync table:

Table: asset_security_index_global_snapshot

For the Quote components

Component of @metadata:

Example on JSON of @metadata

"@metadata": {
  "time_granularity": "5m",
  "range": "1d",
  "type": "close",
  "start": 1716384600,
  "end": 1716385200,
  "timezone": "MTY",
  "gmtoffset": 14000
}

Return output¶

{
  "id": "a2cd773c-d12c-4a99-856e-8eaa30e2b0b7",
  "status": "ok",
  "data": [
    {
      "ticker": "^GSPC",
      "short_name": "S&P 500",
      "language": "en-US",
      "region": "US",
      "quote_type": "INDEX",
      "currency": "USD",
      "market": "us_market",
      "exchange": "SNP",
      "exchange_full_name": "SNP",
      "exchange_timezone_name": "America/New_York",
      "open": 5319.28,
      "high": 5286.01,
      "low": 5323.18,
      "close": 5307.01,
      "previous_close": 5321.41,
      "volume": 2079437000,
      "change": -14.400391,
      "percent_change": -0.2706123,
      "intraday_high_low": "5286.01 - 5323.18",
      "fifty_two_week_range": "4103.78 - 5325.49",
      "quote": {
        "@metadata": {
          "time_granularity": "5m",
          "range": "1d",
          "type": "close",
          "start": 1716384600,
          "end": 1716385200,
          "timezone": "MTY",
          "gmtoffset": 14000
        },
        "timestamp": [1716384600, 1716384900, 1716385200],
        "data": [5318.33, 5318.32, 5320.53]
      },
      "last_updated_ts": 1716434187759875
    }
  ],
  "pagination": {
    "total": 1,
    "page": 1,
    "limit": 100,
    "url": {
      "first": "http://<ORIGIN_URL>/security/index/global/snapshot?ticker=%5EGSPC&page=1&limit=100",
      "next": null,
      "previous": null,
      "last": "http://<ORIGIN_URL>/security/index/global/snapshot?ticker=%5EGSPC&page=1&limit=100"
    }
  }
}

Appendix¶

Appendix A: Record of Changes¶

Table: Record of changes

Appendix B: Related Repository¶