Customize Bulk Deduplication Using Exclusions and Pre-defined Masters

deduplicate customer data

Insycle makes it easy to deduplicate records in custom ways, using exclusion lists and pre-defined masters to give you complete control over your deduplication process.

In some cases, you may need more granular customization for picking duplicate records to include or exclude from the deduplication process. Insycle allows you to manually exclude records from a merge, or manually pick master records in a CSV, then merge the duplicates in Insycle.

For more information on the deduplication process, read Bulk Merge Duplicate People, Companies.

How It Works

Typically, Insycle allows you to identify duplicates with a filter. But if the relevant records do not have anything in common to filter by, and there are too many of them to include or exclude by ID, you can manually exclude records and define masters in a CSV file, then use Insycle to merge the record.

Similarly, for picking the master, you'd typically apply a set of rules in Step 4. But if you have a pre-defined list of records that you'd like to pick master records individually, and those master selections cannot be applied using rules because they'd be different for each duplicate group, you can use that same approach below to customize the master selection in bulk.

Quick Summary

First, start by running the Merge Duplicates module in preview mode. Then, analyze the CSV report of the duplicates, and "tag" records for exclusion or as masters. Then you import the "tags" into the CRM as custom fields. Lastly, you run the Merge Duplicates module and use those "tag" fields in the filter and master selection rules.

Step 1: Deduplicate in Preview Mode

You can simulate the deduplication process for analysis purposes without modifying any data in the CRM by running deduplication in Preview Mode.

This also produces a CSV report of the identified duplicates in your database.

preview mode


The CSV report will have a row per record and include the following columns:

  • Record ID
  • Deeplink to open the record in the CRM.
  • All fields selected in Step 1
  • All fields selected in Step 4

Pro Tip

Since every field that is specified in "Step 4: Master Selection" is also included in the CSV report, you can include additional fields to get more context about the records in the CSV report. For example, add "Industry exists" as the last rule to have the "Industry" field included in the CSV report.

Open the CSV report in Google Sheets or Excel and add two new columns:

  • Deduplication Exclude
  • Deduplication Master

Now given the context from all the fields available in the report, update the CSV as needed:

  • Enter TRUE in the "Deduplication Exclude" column for any row that you'd like to exclude from deduplication.
  • Enter TRUE in the "Deduplication Master" for the row that you'd like to be designated as the master from within a deduplication group.
add fields


When you're done updating the CSV report with exclusion and master selection, remove all columns from the CSV except for the following:

  • Record ID
  • Deduplication Exclude
  • Deduplication Master

Then save the report to a CSV file.

add tags

Step 2: Create Custom Fields in the CRM

Create two custom fields in the CRM:

  • Deduplication Exclude
  • Deduplication Master

For the field type pick "Boolean" or "Checkbox".

After creating the new custom fields in the CRM, refresh the data in Insycle by logging out and then logging back in, or under SettingsSync Status.

Step 3: Use Magical Import to Tag Duplicates

Open the Magical Import module, and load the CSV file you saved in Step 1.

Map the ID column to the relevant record ID field. The other two fields will map automatically.

map fields


In Step 2 of Magical Import, pick "Only update existing" because those records already exist in the CRM, you're simply "tagging" those existing records for deduplication.

only update existing


Then, click to import the CSV.

Step 4: Deduplicate in Update Mode with Customizations

Now that records are tagged with deduplication customizations you can use the custom fields to customize the process.

To exclude records from deduplication, in Step 1, click the Filter button.

filter button


Enter: "Deduplication Exclude," "Not Any Of," and "Yes."

deduplication exclude, not any of, yes


To designate record as master, in Step 4, remove all rules and add just one rule: "Deduplication Master" is "Yes."

deduplication master is yes


You can run deduplication in Preview mode again to confirm that it works as expected with all the customizations and then run it in Update mode to merge the duplicates in the CRM.

Preview Changes Before They Go Live

In some Insycle modules, including Merge Duplicates, you can preview the changes that you are making to your data before those changes are pushed to your live database. When you run modules in Insycle, you have the option of choosing between Preview Mode and Update Modeonce you click the button.

Audit Trail and History

The Activity Tracker lets you review all changes made through Insycle. At any time you can download a CSV report of the operation and records affected.

Additional Resources

Related Blog Articles

Related Help Articles

Additional Resources