Notification: Basement Error Report¶

Overview¶

Notification is responsibilities for providing statistic about state of job in a given duration (aka report) or online alert after running job.

Table of Content:

• Notification: Basement Error Report
• Overview
• SAD
  • Logical View
  • Physical View
• Resource
  • Cloud Scheduler
  • Service account
  • Processing component
• Component
  • Flow
  • ERD

SAD¶

Logical View¶

---
title: Notification mechanism
---
flowchart LR

  %% Source
  subgraph source[Source]
      scheduler[Scheduler]
      job[Running Job]
  end
  %% Component
  subgraph processing[Processing]
      direction TB
      gateway[Gateway]
      noti_handler[Notification Handler]
      data_repository[Job metadata storage]
  end
  %% Sink
  subgraph sink[Messaging app]
  end

  %% Flow
  scheduler --Trigger--> gateway
  job --Push--> gateway
  gateway --Determine--> noti_handler
  gateway --Determine--> noti_handler
  noti_handler --Fetch job metadata--> data_repository
  data_repository --> sink
  noti_handler --Push--> sink
  noti_handler --Push--> sink

  %% Style
  linkStyle 0,2,4,6 stroke-width:2px,fill:none,stroke:green;
  linkStyle 1,3,7 stroke-width:2px,fill:none,stroke:blue;
  linkStyle 5 display:none;

Physical View¶

---
title: Notification mechanism
---
flowchart LR

  %% Source
  subgraph source[Google Cloud]
      scheduler[Cloud Scheduler]
      subgraph job[Running Job]
          cloudbuild[Cloud Build]
          workflow_pipeline[Workflow Pipeline]
      end
  end
  %% Component
  subgraph processing[Processing]
      direction LR
      subgraph gateway[Gateway - Google Cloud]
          direction LR
          task_queue[Task Queue]
          workflow_gateway[Workflow - Gateway]
      end
      noti_handler[Otter - Cloud Run]
      subgraph data_repository[Job metadata storage]
          database[(Dawnbreaker - MySQL)]
          workflow_store[Workflow Metadata]
      end
  end
  %% Sink
  subgraph sink[Messaging app]
      slack[Slack]
      email[Email]
  end

  %% Flow
  scheduler --Trigger--> gateway
  job --Push--> gateway
  gateway --Determine--> noti_handler
  gateway --Determine--> noti_handler
  data_repository --> sink
  noti_handler --Fetch job metadata--> data_repository
  noti_handler --Push--> sink
  noti_handler --Push--> sink
  task_queue --Rate limit--> workflow_gateway

  %% Style
  linkStyle 0,2,5,6 stroke-width:2px,fill:none,stroke:green;
  linkStyle 1,3,7 stroke-width:2px,fill:none,stroke:blue;
  linkStyle 4 display:none;

Resource¶

Cloud Scheduler¶

Scheduler for report locate at project which need to have report

Service account¶

• Deploy scheduler:

**sa-natures-prophet: Project builder**:

Service account: `sa-natures-prophet@$PROJECT_ID.iam.gserviceaccount.com`

Alias: `sa-natures-prophet`

| Permissions                                | Identifiers                     | Performtion                                     |
| ------------------------------------------ | ------------------------------- | ----------------------------------------------- |
| `roles/cloudbuild.builds.builder`          | Cloud Build                     | Cloudbuild builder                              |
| `roles/iam.serviceAccountUser`             | IAM                             | Impersion service account on a targeted service |
| `roles/iam.serviceAccountTokenCreator`     |                                 | To impersonate a service account                |
| `roles/secretmanager.secretAccessor`       | Secret                          | Access secrets                                  |
| `roles/storage.admin`                      | `GCS::inno-internal-cloudbuild` | Related to write log into bucket                |
| `roles/run.developer`                      | Cloud Run                       | Deploy Run service                              |
| `PROJECT::roles/run.services.setIamPolicy` | Cloud Run                       | Set IAM for run service                         |
| `roles/cloudscheduler.admin`               | Cloud Schedule                  | Create and update Scheduler job                 |
| `roles/monitoring.metricWriter`            | Cloud Monitoring                | Writing monitoring data to a metrics scope      |
| `roles/logging.logWriter`                  | Cloud Logging                   | Write log entries                               |

Processing component¶

About resource for this component, reference Transflow

Component¶

Flow¶

flowchart LR

  %% Component
  crawler[Crawler]
  logging[Logging]
  reporter[Reporter]
  email[Email]

  %% Flow
  crawler --> logging
  logging --read--> reporter
  reporter --sent notification--> email

ERD¶

classDiagram

  class Error {
    int id
    nvarchar name
    nvarchar description
    PK (id)
  }

  class Provider {
    int id
    nvarchar name
    nvarchar description
    PK (id)
  }

  class Runner {
    int id
    int provider_id
    float scheduled_date
    float scheduled_time
    str identifier
    int error_id
    PK (scheduled_date, scheduled_time, identifier)
    FK (provider_id -> Provider.id, error_id -> Error.id)
  }

  Error <|-- Runner : n-n
  Provider <|-- Runner : 1-1

  class Error_Summary {
    int id
    int provider_id
    float scheduled_date
    str identifier
    int operation_status
  }