Skip to content

Notification: Pipeline Invoke

Overview

Data pipelines running in background in data platform to make sure the data team membes and related affected stakeholder (other team relies on the data results) to know about the health status of the data pipelines.

Table of Content:

SAD - System architechture design

Logical View

Cloud Build publishes messages to a registed portal (currently is a PubSub topic) by itself about build's status changes.

Each message contains a base64 JSON string representation of your Build resource in the message.data attribute. The build's unique ID and the build's status can be found in the message.attributes field.

You can use a push or pull model for your Pub/Sub subscriptions.

flowchart LR

  %% Component
  notification[Notification Service]

  subgraph gcp[Google Cloud Platform]
    subgraph cloudbuild[CloudBuild]
      build[Build]
    end
    build -- change status --> build
    build -- """publish message
      on change""" --> portal[Portal]
    notification -. registration .- portal
    portal -- dispatch --> rlm[Rate Limit] -- routing --> notification
    notification -- listen on targeted route --> notification
  end
  subgraph 3pt[Third Party]
    agent[Agent]
  end
  notification --> agent -- notice --> user[User]

The table component

Code Resource Identifer Description
COMPONENT-01 GitHub Repository inno-notificaiton Contain logic of the notification for the state changes
COMPONENT-02 CloudBuild Private in asia-southeast1 CICD platform
COMPONENT-03 PubSub [Topic] Topic in asia-southeast1: cloud-builds The topic that CloudBuild published messages into
COMPONENT-04 PubSub [Sub] Subcription asia-southeast1 Serverless deployment for application service
COMPONENT-05 Workflow [Dispatch] Subcription asia-southeast1 Serverless deployment for application service
COMPONENT-06 Workflow [Sub] Subcription asia-southeast1 Serverless deployment for application service
COMPONENT-07 IAP At asia-southeast1 Serverless deployment for application service
COMPONENT-08 Cloud Run $ENV-inno-notification Serverless of the notification 
#!/bin/bash

# Element
source script/util.sh;

# Component of [Pipeline]
#
#   <Workflow>           <Task>                  <Workflow>             <Cloud Run>
# ++++++++++++++      ++++++++++++++++++       ++++++++++++++         ++++++++++++++
# +  Executor  +  --> +  Task Declare  +  -->  +  Workflow  +  ---->  +    Otter   +
# ++++++++++++++      ++++++++++++++++++       ++++++++++++++         ++++++++++++++

# Variable
declare PIPELINE=otter

# Component
declare TASK_QUEUE_NAME=prod-transflow-transition-otter-hub

# Default SA
declare WORKFLOW_TASK_DECLARE_SERVICE_ACCOUNT_NAME=sa-chen
declare WORKFLOW_EXECUTOR_SERVICE_ACCOUNT_NAME=sa-warlock

# Validate
check_non_empty_variable PIPELINE
check_non_empty_variable WORKFLOW_TASK_DECLARE_SERVICE_ACCOUNT_NAME
check_non_empty_variable WORKFLOW_EXECUTOR_SERVICE_ACCOUNT_NAME
  • sa-chen: Deploy notification task declare
SA Permissions Identifiers Performtion
sa-chen roles/cloudtasks.enqueuer Cloud Tasks Access to create tasks.
roles/iam.serviceAccountUser Service Accounts Run operations as the service account.
roles/workflows.invoker Workflows Execute workflows and manage the executions using the API.
  • Documentation on service account
#!/bin/bash

# Element
source script/util.sh;

# Component of [Pipeline]
#
#   <Workflow>           <Task>                  <Workflow>             <Cloud Run>
# ++++++++++++++      ++++++++++++++++++       ++++++++++++++         ++++++++++++++
# +  Executor  +  --> +  Task Declare  +  -->  +  Workflow  +  ---->  +    Otter   +
# ++++++++++++++      ++++++++++++++++++       ++++++++++++++         ++++++++++++++

# Variable
declare PIPELINE=otter

# Component
declare TASK_QUEUE_NAME=prod-transflow-transition-otter-hub

# Default SA
declare WORKFLOW_TASK_DECLARE_SERVICE_ACCOUNT_NAME=sa-chen
declare WORKFLOW_EXECUTOR_SERVICE_ACCOUNT_NAME=sa-warlock

# Validate
check_non_empty_variable PIPELINE
check_non_empty_variable WORKFLOW_TASK_DECLARE_SERVICE_ACCOUNT_NAME
check_non_empty_variable WORKFLOW_EXECUTOR_SERVICE_ACCOUNT_NAME
  • Docs CLient ID is public
# Client ID : Is used to identify the application. Let's say you are building an App that would like to access google maps APIs, You need to register the app with google & google will give you client id which is an id to identify the client in our case it's your app. Client ID is publicly available. For example, If you use 3 legged oAuth like signIn with Google, you can see client id in URL. So, You cannot use client id as a secret.

# Client Secret : This is the true secret key, which is stored on server side securely & not available to public.

# Remember, Client ID & Client Secret is common for many other grant types apart from "Resource owner credentials grant". Yes, In resource owner password credentials client id is not exposed anywhere to public but it is supposed to be a public key in overall OAuth context. As per oAuth standard you need both Client ID & Client Secret along with user credentials to generate an access token. It's the standard defined by OAuth.

# A straight forward answer is, It's OAuth specification. You need to send them both & store the secret securely in backend server along with client Id.

Ref: https://www.googlecloudcommunity.com/gc/Apigee/Client-Id-vs-Client-Secret/m-p/13501#:~:text=Client%20ID%20is%20publicly%20available,securely%20%26%20not%20available%20to%20public.

Removed: sa_mars, sa_warlock, sa_chen

-------: sa_techies, sa_slardar, sa_terrorblade

Dispatch

Report

flowchart LR

  %% Component
  scheduler[Scheduler]
  workflow[Workflow]
  notification_service[Notification Service]
  slack[Slack]
  email[Email]

  %% Flow

  subgraph gcp[Google Cloud Platform]
    scheduler
    workflow
    notification_service
  end

  subgraph notification[Notification]
    slack
    email
  end

  scheduler --> workflow --> notification_service
  notification_service --> slack & email

Appendix

Appendix A: Record of Changes

Table: Record of changes

Version Date Author Description of Change
0.1.0 05/20/2024 Bao Truong Initation documentation