Changelog¶

2024-06-28¶

Bump version of project into 4.5.207

• Bump version image of flow deployment into 1.30.10

For Workflow Orchestration component

• Updated client name into deployment-motherplane-agent

• Updated namespace of CRON in prefect.yaml

• Added deployment for sync data from HSC into Prefect

• Removed cron in on-premies server related

• Handle the sync of the adjustment action from: (a) Action of Corporate actions (b) Manual

• Removed src/core element

• Added sync for adjustment event with: (a) From VSD source (b) From mannual specs

• Added adjustment flow

• Documentation for advanced Prefect: Task, Access Endpoint, ...

• Updated the configuration for prefect.yaml namespace

• Fixed empty resources of sync from HNX::quote

• Healthcheck for production of API of HSC service within 60 min

• Test removed __init__.py at provider level

• Migration for database:

• Quote: Full refresh to fetch data

  • Quote: Full refresh to fetch data

• Fixed endpoint for SSI flow with:

• Updated the base-url for the client

• Refector script flow

• Updated the documentation on config component

• [CICD] For the element of ruff: Handle the install without quote by multiple regex pattern

• Added components for Prefect:

• [Flow] Removed config of logging on flow

• Added util for batch generator

• Supported yield with session of database, handlers of client for Submarine.

• Added authentication flow for submarine client

• Bump depdendencies of the project

• submarine-client into v0.2.10

• inno-finance-core into v0.9.10

• Add email of SendGrid to expand the mail with reduced itegrated with inno-notification

2024-06-20¶

Bump version of project into 4.5.83:

• [CICD] Enhance layout of Dockerfile that enhance the steps flow.

• Bump version of workers of Prefect at 1.18.5

• [CICD] Added mypy integrate with pre-commit

• [CICD] Enhance build time into 4.2x times of deployment prefect flow.

• Remove old version of Yahoo snapshot and renamed to 1 version only.

• Fix flow of Vietstock indicator flow

• Edit prefect task map on vietstock indicator flow

• Fix argument of period that returned values

• Created task to push at the end of task flow for each ticker and config retry errors.

• Added authentication into profile

• Refactor src of HNX future quote, future profile. Bump version into 2.6.0

• [Enhancement] Updated the coding style for flow folder.

• Added serialized for webdriver for flow

2024-06-19¶

Bump version of the project into 4.5.78

• Added autopep control flow folder

• Small fix element entrypoint in the flow components

• [Docs] Added the local deployment configuration for Prefect

• [Datasets] Added prefect deployment for Vietstock indicator

• [Prefect] Updated timeout for deployment into 1 hour.

• [Dependencies] Updated numpy to supported M1 machine with min version at numpy release 1.21.4

• Fix small in the components

2024-06-17¶

Bump project into version v4.5.70:

Prefect Client:

Bump client into version v1.15.1

• Removed config gcp_sa_credential_secret_name

• Fixed the variables set up with register namespace

• Updated docs for secret

Internal:

• Variables Configuration both for Native and Prefect paradigm

• Fixed the configuration parsers for global orders: File Setting (.ini, .yaml) > Environment Variables

• Updated the namespace of configuration with prefix of BASEMENT_*

• Updated documentation variable config.

• Remove all the pubsub metadata

• Added make target: venv, venv-refresh, auth-adc

2024-06-16¶

Bump version of project into v4.5.58:

• Fixed deployment component

• Documentation after deployment

• Troubleshooting ssh status outside of the company network

• Updated configuration

• Updated PAT of Deer Runner

• Updated badge for project in README documentation

• Fixed deployment date

2024-06-14¶

Bump version of project into v4.5.53

Datasets:

• Added authentication into Quote on (a) Daily, (b) Index and © Refresh component.

Internal:

• 📝 Updated README with badges, endpoints

• 🔮 Updated maintainers of the project

• 📝 Updated internal documentation for:

• Setting variables SAD, permissions

• Updated navigation changes on layout

• Configuration for NTP in remote instance

• [CICD] Disable workflow server-align-base-date.yml

• Updated Dockerfile of prefect-workers

• Updated labels

• Updated version and configuration

• Spend times to VSD entry

• Added deployment checklist

• Updated errors for DeprecationWarning

# D:\PROJECTS\inno-basement\component\provider\vietstock\indicator_snapshot.py:455: DeprecationWarning: `map_dict` is deprecated. It has been renamed to `replace`.
# The default behavior has changed to keep any values not present in the mapping unchanged. Pass `default=None` to keep existing behavior.

• Fix component\entry\vsd\event_entry.py when run this

# D:\PROJECTS\inno-basement\venv\lib\site-packages\google\cloud\sql\connector\refresh_utils.py:214: CryptographyDeprecationWarning:
# Properties that return a naïve datetime object have been deprecated. Please switch to not_valid_after_utc.
#   expiration = x509.not_valid_after

• Fix

# D:\PROJECTS\inno-basement\tests\unit\provider\vsd\test_event_detail_functional.py:3295: DeprecationWarning: `frame_equal` is deprecated. It has been renamed to `equals`.
#   if not pl.DataFrame(data_element, schema=schema, orient="row").frame_equal(_element):

• Bump dependencies:

• Bump polars to min supported 0.19.16

• Updated packages of google cloud platform Restful API

• submarine client has been upgraded 0.1.35

• Added the the authentication related to internal auth

• Added variables: AUTH_ORIGIN_URL, AUTH_SERVICE_NAME

• Makefile

• Makefile with supported target: make info, make install-dev.

• Supported load .env into the make process

• pyarrow package has been supported into pyarrow-16.1.0

# D:\PROJECTS\inno-basement\venv\lib\site-packages\dask\dataframe\_pyarrow_compat.py:23: UserWarning: You are using pyarrow version 8.0.0 which is known to be insecure.
# See https://www.cve.org/CVERecord?id=CVE-2023-47248 for further details. Please upgrade to pyarrow>=14.0.1 or install pyarrow-hotfix to patch your current version.

• Fixed the path on old adjustment element

2024-06-12¶

Bump version of project into v4.4.2

Internal:

• Bump version of nick-fields/retry@v3

• Updated documentation for setup prefect

• Add revision into project.yaml

• Check the --no-cache-dir on the production deployment

• Updated target for Makefile

• Updated cloudbuild component of prefect

2024-05-17¶

• 📑 [Documentation] Updated the basement project component

• Bump version of the project

• inno-python-client-submarine.git into version 0.1.31

• Bump pip from 23.2.1 into 24.0

• Bump version of slackapi/slack-github-action from v1.23.0 into v1.26.0

• Updated the entrypoint system and mapping it to call by the config

• Removed unnecessary of table:

• FIINS related: fiin_*, included fiin_financial_statements

• Event detail: [vsd_event, vsd_events_contents, vsd_dim_events_tracking, vsd_event_failed_breakdown, vsd_events_failed_breakdown, vsd_events_details]

• Securities information: [vsd_ticker_detail_information, vsd_ticker_registration_document, vsd_ticker_registration_history]

• Core: [manual_entry_settlement_closing]

• Align with declarative in the in the inno-lake-prep project

2024-05-10¶

• Bump dependencies

• Updated the submarine-client into version into 0.1.29

• Bump project version

• Replace logsBucket into all file cloudbuild

• Removed source with using the internal package instead (src/util_mysql, src/submarine, src/str_tune)

2024-04-20¶

• Updated the documentation space for project

• Added the .markdownlint.json config

• Removed all retired stuff on Vietstock, FIIN, SSI

• Add the requirement different for Centos7 with supported of polars-lts-cpu old style

2024-04-16¶

• [Feature] [Internal] The declaration config by YAML and try/except for the schedule

• Add new generation of CRON schedule that supported:

• Declare model through yaml file system

• Avoid error of 1 model affected overall model

• Send notification on failure model

• Handle the report for synced model

• Add the validation composite element

2024-04-12¶

• Add local documentation for project

• Use mkdocs with theme material

• Config the config with mkdocs

• Supported requirements for developers

2024-03-25¶

• Bump dependencies for project

• submarine into v0.0.12 that supported: Quote Index, Quote

• inno_finance_core into v0.6.7

• Bump version for polars

  • Add polars config from file in the init component

  • Remove the depreciated function

  • is_not() into not_()

  • keep_name() into .name.keep()

  • strip is deprecated. It has been renamed to strip_chars.

• Upgrade pyarrow for insecure version

• Remove usage of src/util_mysql

2024-03-20¶

• Run supported sciprt related to submarine client

• Internal flow

• Added config for IDE style for overall project by .editorconfig

• Updated the GitHub token in deployment

• Added inno-finance-core into the component

• Fixed requirement component on declare on platform with targeted component.

• Removed shebang in src folder to avoid Illegal instruction

• Added mypy for experiment. Note mypy required python 3.8 version.

2023-11-03¶

• 🐛 [Bugfix] Fixed illegal instruction of polars for Centos7. PR #225 by @thinh.luu, @bao.truong

• Add timeout for fetch list world index when fetching

• Catch response with status code 5xx from response

• Handle the empty result output

2023-10-15¶

• 🐛 [Bugfix] Fixed illegal instruction of polars for Centos7. PR #214 by @bao.truong

• 📑 [Documentation] Add troubleshooting for Illegal instruction error

• 🌾 [Refactor] [HSC] Expand schema the OHLCV model. PR #193 by @dat.phan, @bao.truong

• Added beta version of HSC model with new properties in the ohlcv-eod and ohlcv-eod-index

• Fixed the Vietstock OHLCV when non-data in the result with the schema included.

• Fixed some grammar in the script layout

• Upgrade functional related to depreciated func namespace:

  • apply() to map_elements()

  • str.strip() to str.strip_chars()

  • lengths() into len_bytes()

2023-10-05¶

• [VSD] Enhancement model recognization of VSD:

• [VSD] Model of event-summary has been take more source. Fetching from the News of VSD

  • The bucket category of news supported include: Issuer, Depository Member, Clearing Member, Fund Manager

  • The event source captured the pipe of data. E.g: NEWS::CATEGORY:ISSUER, NEWS::CATEGORY:FUND_MANAGER, RIGHT::TAB:EXERCISE

• There are 5 reports with daily frequency to handle the flow of dataset

Internal:

• Added component to avoid deadlock when update on multiprocessing.

2023-09-22¶

• 🌾 [Yahoo] Snapshot World Indices from Yahoo provider in real-time. PR #189 by @thanh.vo, @thinh.luu, @bao.truong.

• ⚓ Bump version of pip into version 23.2.1

Internal:

• Remove Innotech Slack application and transfer notification "postman" position into Otter.

• Update Slack channel name:

• Rename into 0-code-regulation instead of coding-standard

• Rename into 0-test-application instead of application-testing

• Rename into 1-task-schedule instead of schedule-task-log

• Change config variable of Slack from LOG_CHANNEL_NAME into JOB_CHANNEL_NAME

• Update the secret in the GitHub Action for deployment

2023-09-01¶

• ⚓ Bump version of project basement into 2.8.0

• 🔮 Update the contributor list for the project

• 📑 Write update document for changelog, README (folder layout) and headstart for developer

• 🌾 Refactor extractor content of event for action from VSD. PR #174 by @bao.truong

• ⚓ Bump version of provider VSD at src/provider/vsd into 6.0.0 with:

  • Handle the CorporateAction more powerful

  • The process of detect the event is interactive and ignore the failed case

  • Updated the list of pool of error for easy controlable later

  • Updated the regex pattern for almost cases

  • Has the class::EventDetail to storage per event and easy serialized

• Refactor script of VSD for event parser and element

• ⚓ Bump version selenium into 4.11.2 and test all related script is working

• Remove usage of webdriver-manager because the selenium version >=4.11.2 has been control by merge the package.

  • Remove config of CHROME_DRIVER_VERSION for driver of Chrome in the project

• Updated the Excel constant to /shared component

• Updated tests/ layout with unitest and integration test

  • Split test cases into unit and integration for more controlable

  • The folder structure of tests

  📦tests
  ┣ 📂integration
  ┃ ┣ 📂core
  ┃ ┗ 📂provider
  ┃ ┃ ┣ 📂cafef
  ┃ ┃ ┣ 📂hnx
  ┃ ┃ ┣ 📂ssi
  ┃ ┃ ┗ 📂vsd
  ┣ 📂unit
  ┃ ┗ 📂provider
  ┃ ┃ ┣ 📂ssc
  ┃ ┃ ┣ 📂vsd
  

  • Rename the term of vendor into provider

  • Updated the target channel for test handle

  • Added test cases for event_detail, annotate, patterns, detect_action_from_title

• Fixed all issues with test cases, all passed script

  • Skip test related to provider block by using controller tool (Like SSI, Vietstock) or block by token (Vietstock)

  • Upgrade test case element

• 🎉 VAS: Data for Account Chart. PR #175 by @sieu.huynh

• Added script VAS account to handle multiple version

  • Added 3 component

  • INNO_VAS_DIM_ACCOUNT_LEVEL_1

  • INNO_VAS_DIM_ACCOUNT_LEVEL_2

  • INNO_VAS_DIM_ACCOUNT_LEVEL_3

  • Added data for version 2014 and 2018

• Fixed issued related to import for VSIC

  • INNO_CORE_DIM_CLASSIFICATION_VSIC_CLASS

  • INNO_CORE_DIM_CLASSIFICATION_VSIC_DIVISION

  • INNO_CORE_DIM_CLASSIFICATION_VSIC_GROUP

  • INNO_CORE_DIM_CLASSIFICATION_VSIC_SECTION

  • INNO_CORE_DIM_CLASSIFICATION_VSIC_SUBCLASS

• 🌾 Fix statement on data component at VAS and VSIC. PR #176 by @sieu.huynh

• 🌾 Fix VAS Element. PR #178 by @sieu.huynh

• 🎉 [Feature] Upgrade the listen to exchange for both HSX and HNX. PR #177 by @thanh.vo

• 🍹 [Data] Entry data for VSD event on 2023-08-23 within frame: 2023-08-21 to 2023-09-01. PR #181 by @thanh.vo

• 🍹 [Data] Entry data for VSD event on 2023-08-16 within frame: 2023-08-16 to 2023-08-21. PR #179 by @bao.truong

2023-07-31¶

• Added ruff and pre-commit enforce

• Added required validate Python version must be greater than 3.6.x in deployment mode

• Added data for VSIC for 2007, 2018

• Merge inno-social-listening into basement because related template format and component

• Change color on job for success template

• Added dispose on engine will seperated the session between parent, child process when 'spawn' mode

• Pipeline: Thống kê giao dịch tự doanh (From Exchange).

2023-07-15¶

• Implement local calculated last-trading-date and next-trading-date replace for the fetch from the database. This reduced the overhead cost for script.

• [Core] Layout structure for core component

• Folder format:

  • data is folder included files (Excel, CSV, ...) for dataset, defined by:

  Syntax: [PREFIX]_[REVISION].[EXTENSION], included:

  • Prefix: Prefix of revision dataset.

    E.g: ticker_gics_

  • Revision: Revision date in YYYYMMDD format

  • Extension: Dataset file type, can be one of [csv, xlsx, xls, txt]

  Each component consistent of:

  • <script>.py to load any revision version to target component

  • <script>.yml to push config check for CICD

• Create --process arguments for overall script.

• Updated the date in machine to not in line with the time counter

• Change all core to use sessionmaker with GCP authentication

• Reduced memory sharing between threads by ThreadProcessor.

• Re-write package job for control:

• Folder input/output/external control

• Revamp color related to job status

• Enhance to pydantic control instead of base object

• Convention naming for variable related to job

• Index Current Market Capitalization

• Finish the list of index captured

• Notification for update the metadata of instituents

• Fetch the latest constitutent for HNX

2023-06-16¶

• Pipeline: Index Snapshot Metadata by @phuc.huynh

• [Fix] Change the schedule for VNINDEX

• [Fix] Change the database schema for HSX Index Constituent

• [Government] [Enhancement] Reboot server required for release resouce

• Create GH Actions to schedule outside the server with diffenrent runner at PR#141 by bao.truong@innotech.vn

• Added component to send message for reboot

• Enhancement Otter to send message when reboot at path /server/reboot

2023-06-07¶

• Updated document state of document

2023-02-30¶

Features:

• ⚓ Bump version of GitHub Actions to v4

• [Feature] Get exchange file from vendors. PR #97 by @phuc.huynh

Bug Fix:

• [Fix] Enhancement folder navigator. PR #12

• Removed inno-staging-tuktuk, change into inno-processor

• Fixed the folder navigator to listener only.

• Updated the component arguments for listener on bucket and endpoint

2023-02-15¶

• [Maintaince] Repository layout structure

• [Feature] Add Load Balancer to Internal Docs

2023-01-05¶

• Rename repository to follow project conventions naming

• Change GCS storage into inno-basement to mapping with name.

• Upgrade GitHub Actions component to using Node16, included:

• ⚓ Bump version of slackapi/slack-github-action from @v1.19.0 to @v1.23.0

• Clean repository layout

• Removed healthcheck normal cases.

2022-12-02¶

• Successful deployments for CICD job

Failed Case example

Success Case example

• ⚓ Bump version of metadata, fixed the pull_metadata

• Fixed issues on date elements with date verify, more output log setting

2022-11-22¶

• Added core component: MARKET_SECTOR_SEGMENTAION, VSD_DIM_TRACKING_EVENTS

• Updated job module:

• Enhance the logging for debug

• Provider fixed and revesed code in client platfrom (Slack)

• reference_link has plural form

• Review overall the module for adjustment with next-trading-date

2022-11-01¶

• Added headers to prevent CORS check in OHLCV of ta.vietstock.vn

• Added get data from Exchange from CSH source at API document

• Fixed order code layout on import

• Bump google-bigquery to 3.3.5 and added db-types support for output parsing.

• Fixed batch-write for Excel with captured by group size of 50,000 per sheet in Excel

2022-09-14¶

• Transfer backend architecture to GCP instead of AWS.

• Set up configurations at project level.

• Seperate packages for development and production run.

2022-02-12 - Naruto in the Lunar New Year¶

• [Technical Change] Replace Airflow into Kubernes with Argo with supported of container orcheschation department. And in the time of replacement, using CronTab to scheduled task; The default tutorial you can read here Scheduled Task with python-crontab

• Added VSD-RIGHTS task to get data from VSD.

• Configure Slack connection with updated credential.