Campaigns Schema Data Pipeline

The Campaigns Schema Data Pipeline is responsible for processing campaigns schema data from CX. It extracts data from source (MongoDB) applies necessary transformations to align with schema requirements, and loads the data into target database. The respective target database schema for this pipeline is given in the following document: Reporting Database Schema | campaigns . Key tasks included for this data pipeline are:

• Validating data fields for accuracy and ensuring required fields are present.

• Mapping data structure fields to the target schema.

• Handling data updates and upserts to avoid data duplication.

Changes required in the campaigns_data_pipeline_config.yaml file:

Query

A query in the yaml file is a dictionary containing query configurations for different pipelines. For campaigns schema data pipeline the query is as follows

• campaigns: Configurations for the campaigns schema pipeline.

  • database: Name of the MongoDB database from where activities data is being extracted. Example: "campaignsDb".

  • collection_name: Name of the MongoDB collection. Example: "campaigns".

  • filter: Query filter applied to fetch data.

  • replication_key: Field used to track updates. Example: "updatedAt".

  • transformation: Transformation function name. Example: "transform_campaigns_data".

  • num_batches: Number of data batches. Example: 50.

  • query_keys: Reserved for gold queries ( for loading data in gold table if needed ).