Notification: Build Status Change¶
Overview¶
Receive and handle the Cloud Build build events and notification based on filtered set of build status.
- Notification: Build Status Change
- Overview
- SAD - System architechture design
- Appendix
- Source Reference
SAD - System architechture design¶
Logical View¶
Cloud Build publishes messages to a registed portal (currently is a PubSub topic) by itself about build's status changes.
Each message contains a base64 JSON string representation of your Build resource in the message.data
attribute. The build's unique ID and the build's status can be found in the message.attributes
field.
Then there are 2 phases can applied to the calling service by:
[1] Method 1: Go to rate limit zone with dispatch and routing services
[2] Method 2: Direct message into the notification service
The flow has been described at below
flowchart LR
%% Component
notification[Notification Service]
subgraph gcp[Google Cloud Platform]
subgraph cloudbuild[CloudBuild]
build[Build]
end
build -- change status --> build
build -- """publish message
on change""" --> portal[Portal]
notification -. registration .- portal
portal -- [1] dispatch --> rlm[Rate Limit] -- [1] routing --> notification
portal -- [2] direct message --> notification
notification -- listen on targeted route --> notification
end
subgraph 3pt[Third Party]
agent[Agent]
end
notification --> agent -- notice --> user[User]
The enums of possible status of a build or build step has been defined below:
State | Description |
---|---|
STATUS_UNKNOWN | Status of the build is unknown. |
PENDING | Build has been created and is pending execution and queuing. It has not been queued. |
QUEUED | Build or step is queued; work has not yet begun. |
WORKING | Build or step is being executed. |
SUCCESS | Build or step finished successfully. |
FAILURE | Build or step failed to complete successfully. |
INTERNAL_ERROR | Build or step failed due to an internal cause. |
TIMEOUT | Build or step took longer than was allowed. |
CANCELLED | Build or step was canceled by a user. |
EXPIRED | Build was enqueued for longer than the value of queueTtl. |
Reference: API Reference Project Build Status
The messages constructed by below
For the handler
Physical View¶
Based on the instruction CloudBuild Notifier from Google.
Based on use case, the following stack has been choosen:
Stack | Selection |
---|---|
Subcription Method | Using direct message method to call to the service |
Authentication Service | Using the authentication from the IAP service with service account. |
Subscriptions model | push model |
Agent | Channel: Slack |
flowchart LR
%% Component
iap[IAP]
%% Block
subgraph gcp[Google Cloud Platform]
subgraph cloudbuild[CloudBuild]
subgraph trigger[Trigger]
end
subgraph build[Build]
end
trigger --> build
end
subgraph pubsub[PubSub]
pub[Topic]
sub[Subcription]
end
subgraph run[Cloud Run]
notification[Notification Service]
end
build -- push --> pub -- sink --> sub -- http push --> iap -- authentication --> notification
notification <-- """
filter build status,
construct message,
annotate metadata""" --> notification
end
subgraph notification_agent[Notification Agent]
agent[Agent]
end
notification --> notification_agent
The table component¶
Code | Resource | Identifer | Description |
---|---|---|---|
COMPONENT-01 | GitHub | Repository inno-notificaiton | Contain logic of the notification for the status changes |
COMPONENT-02 | GitHub | Repository inno-infra | Contain the PubSub element, IAP context |
COMPONENT-03 | CloudBuild | Private in asia-southeast1 | CICD platform |
COMPONENT-04 | PubSub [Topic] | Topic in asia-southeast1 : cloud-builds | The topic that CloudBuild published messages into |
COMPONENT-05 | PubSub [Sub] | Subcription asia-southeast1 | Serverless deployment for application service |
COMPONENT-06 | IAP | At asia-southeast1 | IAP authentication service |
COMPONENT-07 | Cloud Run | $ENV-inno-notification | Serverless of the notification |
COMPONENT-08 | Slack | Slack application | Agent of notification |
Detail on component¶
1 | Repository inno-notification
¶
Property | Value |
---|---|
ID | COMPONENT-01 |
Resource | Repository |
Identifier | Repository: Inno-Notification |
Role | Contain code of logic handler on listen and construct message payload |
2 | Repository inno-infra
¶
Property | Value |
---|---|
ID | COMPONENT-02 |
Resource | Repository |
Identifier | Repository: Inno-Notification |
Role | Contain the PubSub element, IAP context, IAM |
3 | CloudBuild¶
Property | Value |
---|---|
ID | COMPONENT-03 |
Resource | Cloud Build |
Identifier | Data project service |
Role | CICD platform |
4 | Cloud PubSub [Topic]¶
Property | Value |
---|---|
ID | COMPONENT-04 |
Resource | Cloud PubSub |
Identifier | Topic: cloud-builds |
Role | Handle event from build |
Project | $PROJECT_ID |
Message Retention Duration | 6 hour, in 21600s seconds |
For the labels, matching following component
Label | Value |
---|---|
team | data |
environment | production |
managed_by | infra |
managed_repository | inno-infra |
Note:
At the May 2024, Cloud Build not supported publish event into custom PubSub topic subscribe-build-notification. So that, do not change this.
5 | Cloud PubSub [Sub]¶
Property | Value |
---|---|
ID | COMPONENT-05 |
Resource | Cloud PubSub |
Identifier | Subcription: prod-sub-notification-cloudbuild |
Role | Subcription of the build events |
Message Retention Duration | 6 hour, in 21600s seconds |
Expriration period | Never expire |
Acknowledgegement deadline | 60s |
Retain acked messages | True |
Exactly once delivery | False |
Dead lettering | Not enabled |
Push config: Endpoint | https://internal.notification.data.innotech.vn/event/cloudbuild |
Push config: Service Account | sa-spirit-breaker@${PROJECT_ID}.iam.gserviceaccount.com |
Push config: Audience | Audiance for IAP OAuth Client ID |
Retry config | minimum_backoff="30s", maximum_backoff="600s" |
Note:
The audiance
has the syntax of $PROJECT_NUMBER-$RANDOM_TEXT.apps.googleusercontent.com
For the labels, matching following component
Label | Value |
---|---|
team | data |
environment | production |
managed_by | infra |
managed_repository | inno-infra |
6 | IAP - Identity Aware Proxy¶
Property | Value |
---|---|
ID | COMPONENT-06 |
Resource | IAP - Identity Aware Proxy |
Identifier | iap-lb-be-internal-notification-endpoint |
Role | Authentication service apply on the notification service |
Note: This working on the production only.
Context of IAP:
Targeted into: internal.notification.data.innotech.vn
Type: BINDING
method
Type | Identity |
---|---|
Principal service account | sa-spirit-breaker@${PROJECT_ID}.iam.gserviceaccount.com |
Domain | domain:innotech.vn |
Required IAM roles of roles/iap.httpsResourceAccessor
in the binding of IAP context
7 | Notification Service on Cloud Run¶
Property | Value |
---|---|
ID | COMPONENT-07 |
Resource | $ENV-inno-notification |
Identifier | $ENV-inno-notification |
Role | CICD platform |
Handle the listen on CloudBuild at the route: /event/cloudbuild
Seperated with 2 environment:
Staging:
Property | Value |
---|---|
Deployment Strategy | CICD |
Deployment Triggers | On push into master branch |
Targeted resource | Google Cloud Platform |
Endpoint | https://staging-inno-notification-2yrayayjeq-as.a.run.app |
Production:
Property | Value |
---|---|
Deployment Strategy | CICD |
Deployment Triggers | On push into production branch |
Targeted resource | Google Cloud Platform |
Endpoint | https://internal.notification.data.innotech.vn |
8 | Integrate Agent Slack¶
Property | Value |
---|---|
ID | COMPONENT-08 |
Resource | Slack |
Identifier | Slack Application |
Role | The application in charge for publish message in the agent |
Service Account¶
Default service agent of PubSub:
Identity: serviceAccount:service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com
Alias: SA_AGENT_PUBSUB
Service | Permissions | Identifiers | Performtion |
---|---|---|---|
SA_AGENT_PUBSUB | roles/iam.serviceAccountTokenCreator | Subcription | generated OIDC token on a service account. |
Default service agent of IAP service:
Because the notification go through IAP service for authentication purpose
Identity: serviceAccount:service-${PROJECT_NUMBER}@gcp-sa-iap.iam.gserviceaccount.com
Alias: SA_DEFAULT_IAP
Managed by: Terraform in GitHub::inno-infra
Service | Permissions | Identifiers | Performtion |
---|---|---|---|
SA_DEFAULT_IAP | roles/run.invoker | Cloud Run | Invoke Cloud Run service related to COMPONENT-07 |
Service account on Subcription:
Identity: serviceAccount:sa-spirit-breaker@${PROJECT_ID}.iam.gserviceaccount.com
Alias: sa_spirit_breaker
Service | Permissions | Identifiers | Performtion |
---|---|---|---|
sa_spirit_breaker | roles/run.invoker | Cloud Run | Invoke Cloud Run service related to COMPONENT-07 |
roles/iap.httpsResourceAccessor | IAP | On context access to IAP |
Method to obtaining an OIDC token¶
In all other cases, use the IAM credentials API to generate an OIDC token by impersonating a target service account right before accessing an IAP-secured resource.
This process involves the following steps:
Provide the calling service account (the service account associated with the code that is obtaining the ID token) with the Service Account OpenID Connect Identity Token Creator role (roles/iam.serviceAccountOpenIdTokenCreator
).
This gives the calling service account the ability to impersonate the target service account.
Use the credentials provided by the calling service account to call the generateIdToken method on the target service account.
Set the audience
field to your client ID.
Reference: Programmatic authentication
Implemented Screenshot¶
Appendix¶
Appendix A: Record of Changes¶
Table: Record of changes
Version | Date | Author | Description of Change |
---|---|---|---|
0.1.0 | 05/15/2024 | Bao Truong | Initation documentation |
0.2.0 | 05/15/2024 | Bao Truong | Updated logical and physical view of the project |
0.3.0 | 05/16/2024 | Bao Truong | Updated service account |
0.4.0 | 05/16/2024 | Bao Truong | Updated Build component and examples |
0.5.0 | 05/16/2024 | Bao Truong | Updated layout of the SAD |
0.6.0 | 05/16/2024 | Bao Truong | Updated the component of architechure (IAP) |
0.7.0 | 05/16/2024 | Bao Truong | Updated the message |
0.8.0 | 05/16/2024 | Bao Truong | Added permission on the service account |
1.0.0 | 05/16/2024 | Bao Truong | SAD component for all related service |