Migration table within the database (MySQL)¶
Overview¶
To handle the historicala of dataset (which can not erazer the process), we need a process to consitent the table itself
Flow¶
- Detect the records that required which
(a) The schema of SELECT is mapping to the targeted resources
(b) Confirm the unique on Primary Key (Filter or Technical exclude by ROW_NUMBER)
© Created key to filter not required
Build MD5 hash key on the primary key to detect the change
Filter
- Read the limit of the CDC process in the internal database
https://cloud.google.com/datastream/docs/sources-mysql#mysqlknownlimitations
- Using INSERT INTO by SELECT statement
/* Backend: MySQL */
INSERT INTO Customers (CustomerName, ContactName, Address, City, PostalCode, Country)
SELECT SupplierName, ContactName, Address, City, PostalCode, Country FROM Suppliers;
- Validate the process
TODO: Handle the truncatelog context