Migrate EF IAM users to different instances/realms

Overview

This guide provides step-by-step instructions for exporting users from a source EF IAM instance and importing them into a target EF IAM (Keycloak) realm. The process involves extracting user data, cleaning it for import compatibility, and performing a partial import to the destination realm.

Prerequisites

Access to the source EF IAM (Keycloak) deployment (Kubernetes environment)
kubectl configured and authenticated to access the source cluster
Target EF IAM instance with administrative access
SSH access to the machine hosting EF IAM
Administrative privileges on the local machine

Step 1: Install and Verify jq (JSON Processor)

The jq command-line JSON processor is required for cleaning the exported user data.

For Ubuntu/Debian Systems:

BASH

sudo apt-get update
sudo apt-get install jq

For CentOS/RHEL Systems:

BASH

# For CentOS/RHEL 7/8
sudo yum install epel-release
sudo yum install jq

# For CentOS/RHEL 9
sudo dnf install epel-release
sudo dnf install jq

Verify jq Installation:

BASH

jq --version

Expected output should show the version number (e.g., jq-1.6).

Step 2: Access the Source EF IAM (Keycloak) Pod

Connect to the Keycloak pod in your source Kubernetes environment:

BASH

kubectl exec -it -n <Namespace> <Keycloak Pod> -- /bin/bash

Example:

BASH

kubectl exec -it -n ef-external keycloak-0 -- /bin/bash

Step 3: Export Users from Source Realm

Inside the EF IAM (Keycloak) pod, execute the export command to extract realm and user data:

BASH

/opt/bitnami/keycloak/bin/kc.sh export --realm <Realm Name> --users different_files --users-per-file <number of users per file> --dir /tmp/

Example:

BASH

/opt/bitnami/keycloak/bin/kc.sh export --realm expertflow --users different_files --users-per-file 10000 --dir /tmp/

Parameters Explanation:

--realm: Specifies the realm name to export
--users different_files: Exports users to separate files
--users-per-file: Number of users per exported file (recommended: 10000)
--dir: Destination directory for exported files

Verify that file named <realm-name>-users-0.json is in tmp directory. Exit the pod after successful export verification:

BASH

ls /tmp/
exit

Step 4: Copy Exported Files to Local Machine

Copy the exported user files from the pod to your local machine:

BASH

kubectl cp -n <Namespace> <Keycloak Pod>:/tmp/<realm-name>-users-0.json /tmp/<realm-name>-users-0.json

Example:

BASH

kubectl cp -n ef-external keycloak-0:/tmp/expertflow-users-0.json /tmp/expertflow-users-0.json

Step 5: Create and Execute Cleanup Script

Create the Cleanup Script

Create a file named ef_iam_users_list_clean.sh:

BASH

vi ef_iam_users_list_clean.sh

Copy the following script content:

BASH

#!/bin/bash
# Script to clean Keycloak users JSON file
# Removes: realm field, user id, createdTimestamp, credentials id, and credentials createdDate

# Check if file is provided
if [ $# -eq 0 ]; then
    echo "Usage: $0 <input_file> [output_file]"
    echo "If no output file is specified, the input file will be modified in place"
    exit 1
fi

INPUT_FILE="$1"
OUTPUT_FILE="${2:-$INPUT_FILE}"

# Check if input file exists
if [ ! -f "$INPUT_FILE" ]; then
    echo "Error: Input file '$INPUT_FILE' not found"
    exit 1
fi

# Check if jq is installed
if ! command -v jq &> /dev/null; then
    echo "Error: jq is required but not installed"
    echo "Please install jq: sudo apt-get install jq (Ubuntu/Debian) or brew install jq (macOS)"
    exit 1
fi

# Create temporary file
TEMP_FILE=$(mktemp)

# Process the JSON file
jq '
  # Remove realm field if it exists
  if has("realm") then del(.realm) else . end |
  
  # Process users array if it exists
  if has("users") then
    .users |= map(
      # Remove user-level fields
      del(.id, .createdTimestamp) |
      
      # Process credentials array if it exists
      if has("credentials") then
        .credentials |= map(
          # Remove credential-level fields
          del(.id, .createdDate)
        )
      else . end
    )
  else . end
' "$INPUT_FILE" > "$TEMP_FILE"

# Check if jq command was successful
if [ $? -eq 0 ]; then
    # Move temp file to output file
    mv "$TEMP_FILE" "$OUTPUT_FILE"
    echo "Successfully cleaned JSON file. Output saved to: $OUTPUT_FILE"
    
    # Show summary of changes
    echo ""
    echo "Removed fields:"
    echo "- realm (if present)"
    echo "- id from each user object"
    echo "- createdTimestamp from each user object"
    echo "- id from each credentials object"
    echo "- createdDate from each credentials object"
else
    echo "Error: Failed to process JSON file. Please check if the file contains valid JSON."
    rm -f "$TEMP_FILE"
    exit 1
fi

Make Script Executable

BASH

sudo chmod +x ef_iam_users_list_clean.sh

Execute the Cleanup Script

Run the script to clean the exported JSON file:

BASH

sudo ./ef_iam_users_list_clean.sh /tmp/<realm-name>-users-0.json

Expected Output:

CODE

Successfully cleaned JSON file. Output saved to: /tmp/expertflow-users-0.json

Removed fields:
- realm (if present)
- id from each user object
- createdTimestamp from each user object
- id from each credentials object
- createdDate from each credentials object

Step 6: Transfer File to Local Machine

Transfer the cleaned file to your local machine:

BASH

sudo scp <user>@<machine_ip>:/tmp/<realm-name>-users-0.json ./<desired_directory>/

Example:

BASH

sudo scp admin@192.168.1.100:/tmp/expertflow-users-0.json ./ef-iam-exports/

Step 7: Set File Permissions

Ensure proper file permissions:

BASH

sudo chmod -R 755 <realm-name>-users-0.json

Step 8: Import Users to Target Keycloak Realm

Access Target EF IAM Admin Console

Navigate to your target EF IAM (Keycloak) Admin Console
Log in with administrative credentials
Select the target realm where users will be imported

Perform Partial Import

In the Admin Console, navigate to Realm Settings
Click on the Action dropdown menu in the top-right corner
Select Partial Import
In the Import dialog:
- Click Select file and choose your cleaned JSON file (<realm-name>-users-0.json)
- Configure import options:
  - If a user already exists: Choose appropriate action (Overwrite)
Click Upload to preview the import
Review the import summary showing:
- Number of users to be imported
Click Import to execute the import process

Verify Import Success

Navigate to Users in the Admin Console
Verify that the imported users appear in the user list
Check a few user profiles to ensure data integrity
Test user login functionality (recommended)