Configurable Log Masking

This document outlines the changes introduced to make log masking configurable in the Expertflow environment. It also explains how to apply updates or add new masking patterns for both Java and Node components.

Logback Configuration File

• The logback-spring.xml file contains the Logback configuration, including regex-based masking patterns.

• It is stored in a designated directory for each java/spring component:
  ~/cim-solution/kubernetes/pre-deployment/logback/

• The ef-node-logback-configmap.yaml file, which contains masking patterns for Node components, is also in the same directory.

ConfigMap

• A ConfigMap (e.g., ef-logback-cm) is created from the logback-spring.xml file for each Java component.

• This ConfigMap is mounted as a volume in the connector deployment, making the configuration available to the application at runtime.

• Similarly, ef-node-logback-cm is created from ef-node-logback-configmap.yaml for all Node components and mounted as a volume in those deployments.

Environment Variables

• Each deployment sets the LOGGING_CONFIG environment variable to point to the mounted Logback config file (e.g., /logback/logback-spring.xml).

• Node components set the MASK_ATTRIBUTES_PATH environment variable to /sensitive.js, defined in the ef-common-environment.yaml ConfigMap.

Log Masking Pattern

• A custom layout class (e.g., com.ef.connector.utility.MaskingPatternLayout) is defined in the relevant java component's ConfigMap.

• This class applies regex-based patterns to mask sensitive data before it is logged.

• For Node components, ef-node-logback-configmap.yaml defines a list of sensitiveAttributes that can be updated as needed.

2. Process to Modify Log Masking Patterns

a) Java/Spring Components

1. Locate and Edit logback-spring.xml

• Path: ~/cim-solution/kubernetes/pre-deployment/logback/logback-spring.xml

• Modify or add regex patterns for the required data (e.g., emails, tokens).

2. Update the ConfigMap

kubectl delete configmap ef-logback-cm -n expertflow 
kubectl create configmap ef-logback-cm --from-file=<path-to-logback-spring.xml> -n expertflow 

Replace <path-to-logback-spring.xml> with the actual path.

3. Restart the Affected Pods

kubectl rollout restart deployment/<deployment-name> -n expertflow

• Replace <deployment-name> with the actual deployment name for the connector.

b) Node Components

1. Locate and Edit ef-node-logback-configmap.yaml

• Modify the sensitiveAttributes array (e.g., add "lastName" to mask last names).

2. Update the ConfigMap

kubectl -n expertflow delete configmap ef-node-logback-cm 
kubectl -n expertflow create configmap ef-node-logback-cm --from-file=<path-to-ef-node-logback-configmap.yaml> 

Ensure the correct filename is used (ef-node-logback-cm, not ef-logback-cm in the second command).

3. Restart Node Component Pods

Currently, only ef-agent-manager has log masking enabled.

kubectl -n expertflow rollout restart deployment/ef-agent-manager 

3. Adding Log Masking to a New Component

a) For New Java Components

1. Update logback-spring.xml with necessary patterns.

2. Update the connector's ConfigMap with the updated layout and variables.

3. Modify the deployment:

   • Add volume and mount for logback-spring.xml.

   • Set the LOGGING_CONFIG env variable accordingly.

4. Restart pods and verify masked logs after deployment.

b) For New Node Components

1. Update ef-node-logback-configmap.yaml:

   • Add new sensitive attributes in sensitiveAttributes array.

2. Update the ConfigMap as shown above.

3. Update the Deployment:

   • Add volume:

     CODE

     - name: sensitive-config
       configMap:
         name: ef-node-logback-cm

   • Set environment variable:

     CODE

     - name: MASK_ATTRIBUTES_PATH
       valueFrom:
         configMapKeyRef:
           key: MASK_ATTRIBUTES_PATH
           name: variables-common-env-cm

   • Mount volume:

     CODE

     - name: sensitive-config
       mountPath: /sensitive.js
       subPath: sensitive.js

4. Deploy the Changes:

   1. Update the deployment with the new configurations and restart the pods.

   2. Ensure that the new component's logs are correctly masked by testing and verifying the output.

5. Testing:

   • After deployment, thoroughly test the new component's logs to ensure that sensitive data is being properly masked according to the new patterns.

This document applies to all java components and node components within the Expertflow environment. It provides a consistent approach to making log masking configurable, outlines steps to modify existing patterns, and explains how to add new components for masking.