Conversation
The modular information is present in Transformation Rules - ETL Pipelines | Conversation
As of now 3 tables are being created for conversations data in the database. The respective schema for each table is as follows:
In the revamping of this ETL job, we are using Data Build Tool (dbt) along with EF Data Platform for data transformation. The new strategy with the data would be stored is:
-
We create a bronze level table for the complete conversations data (conversation_bronze). This will have the semi-flattened data for conversation in which the objects would be stores as it is as JSON array in the database table.
-
Then by using dbt models we create individual silver layer tables (conversatons, conversation_data, conversation_participants).
-
Lastly create a gold layer data table (if required) with consolidated data from each table for a conversation.
The proposed updated schema for the revamping of this ETL job is linked here: https://docs.google.com/spreadsheets/d/1lBLSu-zM8cfNybOSMMTfaVGO5miKrEm3p3CApQRcVhI/edit?usp=sharing
Some dependencies are on the core team for data provision of conversation participants. Need to be discussed with the team.