Customize Bulk Deduplication Using Exclusions and Pre-defined Masters

deduplicate customer data

Customize Duplicate Bulk Merging By Excluding Records and Pre-Defining Masters

You have many records with duplicate data, but you need complete control over what records are merged together and which record they are merged into as the master record.  Or, maybe your records don't have anything in common that Insycle can use to match duplicates and merge. In these situations, you can use CSV files to customize your bulk merging, tagging master records and excluding records from deduplication in your CSV, then using Insycle to run the merge operation, with complete control over the outcome. 

Process Summary

  1. Create a preview CSV from the Merge Duplicates module.
  2. Analyze the duplicates in the CSV report.
  3. Add new columns to mark records to be excluded from merging or used as masters.
  4. Import the records into the CRM adding the new columns as custom fields.
  5. Run the Merge Duplicates module and use the new, custom fields in the filter and master selection rules.


Step-by-Step Instructions

Step 1: Create a CSV Report of Duplicates

Use the Preview Mode in the Merge Duplicates module to simulate the deduplication process. Follow the steps in the Bulk Merge Duplicate People and Companies article up until the Preview Changes in CSV Report step.

Once you have the filters and master record set up, use Preview to simulate the deduplication process for analysis purposes without modifying any data in the CRM.

preview mode

This produces a CSV report of the identified duplicates in your database based on your matching settings. The CSV will have a row per record and include the following columns:

  • Record ID
  • Deeplink (to open the record in the CRM)
  • All fields selected in Step 1
  • All fields selected in Step 4
Step 2: Edit CSV to Specify Excluded Records and Set Master Records

Open the CSV report in a spreadsheet editor and add two new columns:

  • Deduplication Exclude
  • Deduplication Master

Now given the context from all the fields available in the report, update the CSV as needed:

  • Enter TRUE in the Deduplication Exclude column for any row you'd like to exclude from the merge.
  • Enter TRUE in the Deduplication Master for the row you'd like to be designated as the master from within a duplicate group.
add fields

When you're done updating the CSV report with exclusion and master selection, remove all columns from the CSV except for the following:

  • Record ID
  • Deduplication Exclude
  • Deduplication Master
add tags

When finished, save a copy of the CSV file.

Step 3: Create Custom Fields in the CRM

In your CRM, create two custom fields:

  • Deduplication Exclude
  • Deduplication Master

For the field type pick "Boolean" or "Checkbox."

After creating the CRM fields, refresh the data in Insycle by logging out and then logging back in, or navigate to Settings > Sync Status, select the account, and click the Refresh Accounts List button.

Step 4: Use Magical Import to Tag Duplicates

Open the Magical Import module, and load the saved CSV file with your new custom fields.

Map the CRM ID column to the CSV Record ID field. The other two fields will map automatically.

map fields

In Step 2 of the Magical Import, pick Only update existing because those records already exist in the CRM. You're just "tagging" those existing records for deduplication.

only update existing

Then click the Import button at the bottom to import the CSV.

Step 5: Deduplicate in Update Mode with Customizations

Now that records have the deduplication tags in the CRM, you can use the custom fields to tailor the cleanup process.

Return to the Merge Duplicates module. To exclude records from deduplication, in Step 1, click the Filter button.

filter button

For the filter settings select "Deduplication Exclude," "Not Any Of," and enter "Yes" in the Value field.

deduplication exclude, not any of, yes

To designate your chosen records as the masters, in Step 4, remove all rules and add just one rule: "Deduplication Master," "is," "Yes."

deduplication master is yes
Preview Changes in CSV Report, Then Update in CRM

Now you can use Preview mode again to confirm that the deduplication works as expected with all the customizations. 

If everything in CSV looks correct, return to Insycle and move forward with applying the changes in Update mode.


Additional Resources

Related Help Articles

Related Blog Articles