Next Generation for the Data-*¶

Overview¶

This is the my guidance for the team to build the next-genaration for my team

Detail classification¶

The knowledge¶

Documents¶

The list of documentations that developers required to read through:

1. The project detail implement ideas.

2. The project architecture.

3. The document from Cloud Provider, currently is Google Cloud Platform

Install https://cloud.google.com/sdk/gcloud

https://cloud.google.com/docs/authentication/provide-credentials-adc

1. The connector of database backend, currently is MySQL.

a) The connector: https://dev.mysql.com/doc/connector-python/en/connector-python-introduction.html

b) The configuriton

c) PEP 0249: https://peps.python.org/pep-0249/

d) Connection ways with Cloud SQL: Connect Overview

e) Read official document of PyMySQL at Github

Notes:

• The currently supported driver for MySQL is pymysql. Ref: https://cloud.google.com/sql/docs/mysql/connect-connectors#python_1

f) Connection String use PyMysql:

Ref: https://pypi.org/project/cloud-sql-python-connector/

mssql+pymssql://<username>:<password>@<freetds_name>/?charset=utf8

Ref: https://docs.sqlalchemy.org/en/14/dialects/mssql.html#mssql-pyodbc-access-tokens

1. The use cases of service account in the project

2. Development CI/CD process

For SSH connection, used: https://zellwk.com/blog/github-actions-deploy/

1. The Python way: https://docs.python.org/3/using/windows.html

2. Python Setup YAML: https://pyyaml.org/wiki/PyYAMLDocumentation

3. Reading Shebang: https://realpython.com/python-shebang/

Know basic of term related to API design¶

• API Design Guidance: https://cloud.google.com/apis/design

[1] Google API Design Guide API design guide | Cloud APIs | Google Cloud

[2] IBM API Connect Defining components for an API

[3] Platforms Research at [*] [API] Platforms Research

• Data mesh: https://finance-basement.atlassian.net/wiki/spaces/BASEMENT/pages/578191361/1i-STP+Data+Strategy+for+1invest

https://github.com/bradmontgomery/django-querycount/blob/main/querycount/middleware.py

• Python headstart: https://devguide.python.org/getting-started/

• DBT

Topics

Requirement

Methodology¶

How to change

Carreer Path

Motivation

Learning Curvce

This checklist required for team member

Learn: https://testdriven.io/blog/fastapi-crud/

• Configurations for dev in docker-compose. Ref: MySQL docker-compose

On senior path:

• Know about https://github.com/simfin/simfin-tutorials/blob/master/04_Signals.ipynb

• Know about: logging, debug process

• Must well-form the structure

• Active the PR killer

• Follow the next-generation guidance

• Learn alembic and implement https://www.jeffastor.com/blog/testing-fastapi-endpoints-with-docker-and-pytest/

• Write Document at https://www.jeffastor.com/blog/testing-fastapi-endpoints-with-docker-and-pytest

The financial is big¶

Know your project¶

For each project

make sure you follow up the README. It is the first start

The genenral metadata you should follow up

Start install the first proggraming langugaes

make sure this in head:

The tool¶

As the core element, we using a lot

Python: must see the Python-interpreet

Concept of Fixture of Pytest

ref: https://docs.pytest.org/en/6.2.x/fixture.html

ORM concept

https://docs.sqlalchemy.org/en/20/intro.html

Install:

pip install ruff

Debug¶

The compliance¶

Create a Minimal, Reproducible Example¶

When asking a debugging question, people will be better able to provide help if you provide code as text in your question that prospective answerers can easily understand and use to reproduce the problem.

This is variously referred to as creating a "minimal, reproducible example" (MRE), a "minimal, complete and verifiable example" (MCVE), a "minimal, workable example" (MWE), or a "reprex". Regardless of how it's communicated to you, it boils down to ensuring you have included code in your question that follows the following guidelines:

Your code examples should be…

Minimal: Use as little code as possible that still produces the same problem Complete: Provide all parts someone else needs to reproduce your problem in the question itself Reproducible: Test the code you're about to provide to make sure it reproduces the problem The rest of this help article provides guidance on these aspects of writing a minimal, reproducible example.

For more information on how to debug your program so that you can create a minimal example, Eric Lippert has written a fantastic blog post on the subject: How to debug small programs.

https://stackoverflow.com/help/minimal-reproducible-example