SAD¶

Overview¶

Inno Data Operation is an internal portal platform to centralized operations for data system.

Table content:

• SAD
• Overview
• SAD - System architechture design
  • Logical View
  • Physical View
  • Global view on UI service (SERVICE-01)
    • Overall
    • The table component
    • Global Configurations
    • Deployment Environment
    • Detail on component
    • 1 | UI Service - Repository inno-data-operation
    • 2 | UI Service - Repository inno-infra
    • 3 | UI Service - Cloud Run
    • 4 | UI Service - CloudBuild
    • 5 | UI Service - Load Balancing
    • 6 | UI Service - IAP
    • 7 | UI Service - Secret Manager
    • 8 | UI Service - IAM
    • 9 | UI Service - Cloud Artifact
    • 10 | UI Service - Cloud Logging
    • Permission
    • Securities
    • Authentication
    • UI component with site-map
  • Technical stack
• Internal Operation
  • Service Health Check
  • CICD remove resource
• Appendix
  • Appendix A: Record of Changes
• Source Reference

SAD - System architechture design¶

The data operation included the following position

flowchart LR

  %% Component
  oschestration["[1]" Oschestration]
  operation["[2]" Data Operation]
  api["[3]" Internal API endpoints]
  database["[4]" Internal Database]
  lake["[5]" Centralized Lake House]
  client["[6]" Client API]

  %% Flow
  oschestration & operation -- interactive --> api -- storage buffer database --> database -- streaming --> lake -- transform --> client

Logical View¶

The logical view related to relevant services in the system.

flowchart LR

  %% Component
  subgraph user[User]
    direction LR
    stakeholder[Stakeholder]
    data[Data Officer]
    subgraph operator[Operator]
      direction LR
      maker[Maker]
      checker[Checker]
    end
  end

  subgraph cloud[Cloud Environment]

    endpoint[Endpoint Website URL]

    subgraph auth[Auth Service]
      auth_idp[Auth Provider]
    end

    subgraph fe[Frontend Service]
      ui
    end

    subgraph be[Backend Service]
      subgraph database_layer[Database Layer]
        api
      end
    end

    subgraph internal_storage[Internal Storage]
      internal_bucket[Internal Bucket]
    end

  end

  %% FLow
  user -- access --> endpoint -- direct--> auth_idp
  auth_idp -- interactive --> ui[UI console] -- input/modify/apply --> api[Frontbase API] & internal_bucket

The targeted users for this platform:

• Stakeholder to see the activity in the system of data platform.

• Operator (included Marker-Checker) are the pool of users can actions (input/modify/changes/aplly) interactive with data records

• Data officer that handle the system of data through UI console

There are 4 related services:

Physical View¶

Global view on UI service (SERVICE-01)¶

Overall¶

Here is the physical view to establish the UI service (SERVICE-01) into cloud environment

flowchart LR

  %% Service in Cloud Platform
  subgraph Google Cloud Platform

    cbuild[Cloud Build]
    lb[Load Balancing]
    iap[Cloud IAP]
    secret[Cloud Secret Manager]
    logging[Cloud Logging]
    iam[Cloud IAM]
    artifact[Cloud Artifact]

    subgraph ui[UI service]
      run[Cloud Run]
    end

    subgraph related[Related Component]
      be[Backend Endpoint]
      notification_service[Notification Center]
    end
  end

  %% Service for Code Storage
  subgraph gh[GitHub]
    repository[Repository]
  end

  %% Flow
  lb -- navigate --> iap -- handle authentication --> run
  gh <-- sync/trigger --> cbuild -- deploy --> run
  secret -- control secret --> cbuild
  iam -- control access --> cbuild
  cbuild -- store images --> artifact

  run -- invoke --> be
  run -- yield logs --> logging
  run -- notification --> notification_service

The table component¶

Note:

(*) $ENV meaning prefix of environment

Global Configurations¶

Deployment Environment¶

The environment seperated into 3 environments: Local Development, Staging, Production

The strategy for deployment based on changes (push) on GitHub repositories, which related to developers activities

flowchart LR

  %% Component
  developers[Developers]
  gh[GitHub]
  cicd[CICD Platform]
  resource[Targeted Resources]

  %% Flow
  developers <-- trigger --> gh <-- sync --> cicd <-- deploy --> resource

For detail of environment:

Staging:

Production:

Detail on component¶

1 | UI Service - Repository inno-data-operation¶

2 | UI Service - Repository inno-infra¶

3 | UI Service - Cloud Run¶

Based on each development, this has different instruction and resources configuration for Cloud Run

Environment: Production:

Environment: Staging:

4 | UI Service - CloudBuild¶

5 | UI Service - Load Balancing¶

In charge by: Network Admin

Deployment and managed by: Terraform in inno-infra

For the deployment guidance, go with Infra deployment instruction of load balancing

6 | UI Service - IAP¶

In charge by: Network Admin

Deployment and managed by: Terraform in inno-infra

7 | UI Service - Secret Manager¶

In charge by: Maintainer of project

8 | UI Service - IAM¶

In charge by: Infra Admin

9 | UI Service - Cloud Artifact¶

10 | UI Service - Cloud Logging¶

Permission¶

For GitHub

For Google Cloud Platform:

The tables of service account within project

sa-muerta: Project builder:

Service account: sa-muerta@$PROJECT_ID.iam.gserviceaccount.com

Alias: sa-muerta

sa-visage: Runner of the service:

Service account: sa-visage@$PROJECT_ID.iam.gserviceaccount.com

Alias: sa-visage

Securities¶

Authentication¶

At the first stage, all the users comes with IAP and Load balancer

UI component with site-map¶

Design site-map and reference element for internal operation

Table of routes:

Template Design:

[1] The template related to the datasets and pipelines

[2] The template on error block

Site-map design:

For the syntax:

For the overall picture of the site-map:

flowchart LR

  %% Component
  %% - Convention:
  %% (a) Path with have prefix with `p_*`
  %% (b) Dataset has prefix of `*_ds_*`
  %% (c) Pipeline has prefix of `*_pipe_*`
  root[Root page origin URL]
  p_home[`/homepage`]
  p_datasets[`/datasets/home`]
    p_ds_settlement[`datasets/settlement`]
    p_ds_quote[`datasets/quote`]
    p_ds_corporate_action[`datasets/corporate-action`]
  p_pipelines[`/pipelines/home`]
    p_pipe_adjustment[`pipelines/adjustment`]
  p_admin[`/admin/home`]

  %% Page site-map
  root --> p_home
  root --> p_datasets --> p_ds_quote & p_ds_settlement & p_ds_corporate_action
  root --> p_pipelines --> p_pipe_adjustment
  root --> p_admin

This mapping on the view folder for the design component with following matrix table

Technical stack¶

Based on each service, the stack is choosen based on the team handle the component

Prefix of technical stack: using prefix of TS-*

Frontend - Website:

Backend - Auth service: Reference to SAD of the internal-auth service

Backend - API service: Reference to SAD of the submarine service

Internal Operation¶

Service Health Check¶

The Cloud Monitoring will monitoring the through the HTTP check uptime of related service

Each endpoint will exposed path to check health on service with following metadata

Example on the json output should returned

// Response on health check
// Status code: 200
{
  "version": "2.3.1", // Version that project are in, in semantic version
  "revision": "2024-04-01", // Revision of the service, sometime reference to date that deployed the service
  "timestamp": "2024-05-04T17:25:44Z" // Should be in UTC timestamp format
}

The service health will be monitoring by Cloud Monitoring and code be in inno-infra

CICD remove resource¶

CICD to remove resource on holidays and Sat, Sun.

Appendix¶

Appendix A: Record of Changes¶

Table: Record of changes

Source Reference¶

• [1] Enabling CloudRun with IAP