Customize Bulk Deduplication Using Exclusions and Pre-Defined Masters

deduplicate customer data

How to Customize Merging Duplicates in Bulk Using a CSV

You have many records with duplicate data, but you need granular control over what records are merged together and which ones they are merged into. It could be that your records don't have anything in common, so Insycle can't find matches.

With Insycle, you can use CSV files to customize your bulk merging, designate master records, and exclude records from deduplication. Then you can import the CSV from the Magical Import, and use the Merge Duplicates module for complete control over the final merge operation. 

Process Summary

  1. Create a preview CSV report of duplicates.
  2. Update the CSV to specify masters or records to be excluded from merging.
  3. Create custom fields in your CRM.
  4. Import the edited CSV records into the CRM.
  5. Use the new, custom fields to merge the duplicates in your CRM.

 

Step-by-Step Instructions

1. Create a CSV Report of Duplicates

Navigate to the Merge Duplicates module, select the database, and the record type in the top menu. Then explore the default templates for a pre-built deduplication solution, or build your own custom Merge Duplicates template.

Next, go through Steps 1-5 of the module to create a Preview CSV. Make sure to pause there; you don't want to Update your records yet. Follow the instructions found in the Bulk Merge Duplicate People and Companies article.

cus1.png

Insycle will generate a preview CSV and send it to your email.

2. Edit the CSV to Specify Excluded Records and Set Master Records

Open the CSV file from your email in a spreadsheet application.

The CSV will have a row per record and include the following columns:

  • The Result of the action
  • A Message with details
  • The Duplicate Group ID which indicates which records will be merged together
  • A record Status that identifies which were picked as master and which were identified as duplicates and merged into the master. See below for more details
  • All fields used to identify the duplicates in Step 1
  • The record ID, record name or email, and Deeplink to the CRM record
  • All fields selected for the Record and Fields rules in Step 4. Note that if a field is used on both tabs, it will only appear once in the CSV.

Add two new columns:

  • Deduplication Exclude
  • Deduplication Master

Now given the context from all the fields available in the report, update the CSV as needed:

  • Enter TRUE in the Deduplication Exclude column for any row you'd like to exclude from the merge.
  • Enter TRUE in the Deduplication Master for the row you'd like to designate as the master from within a duplicate group.

cus2.png

When you're done filling in the Deduplication Exclude and Master columns, remove all columns from the CSV except for:

  • Record ID
  • Deduplication Exclude
  • Deduplication Master

cus3.png

When finished, save the CSV file to your computer.

3. Create Custom Fields in the CRM

In your CRM, create two custom fields:

  • Deduplication Exclude
  • Deduplication Master

For the field type, pick "Boolean" or "Checkbox."

The underlying field names for these fields in the CRM are: 

  • deduplication_exclude
  • deduplication_master

After creating the CRM fields, refresh the data in Insycle by logging out and then logging back in. You could also navigate to Settings > Sync Status, then select the account, and click the Refresh Accounts List button.

4. Use Magical Import to Tag Duplicates

Open the Magical Import module, and load the saved CSV file with your new custom fields.

Under Step 1, map the CRM ID column to the CSV Record ID field. The other two fields will map automatically.

cus4.png

Under Step 2, set the ID field for the Identity Field and select the Records Mode, Only update existing, because those records already exist in the CRM. You're just "tagging" those existing records for deduplication.

cus5.png

Then click the Import button at the bottom to import the CSV.

5. Use Merge Duplicates to Run the Operation 

Now that records have the deduplication tags in the CRM, you can use the custom fields to tailor the cleanup process.

Return to the Merge Duplicates module. In Step 1, click the Filter button to exclude records from deduplication.

step-1-simple-tab-w-filter-arrow.png

For the filter settings in the popup, select the Deduplication Exclude field, the "not any of" condition, and enter "Yes" in the value field.

cus7.png

To designate your chosen records as the masters, in Step 4, remove all rules and add just one rule: Deduplication Master, "is," "Yes."

cus8.png

Preview Your Changes in a New CSV Report, Then Apply the Updates to Your CRM

Under Step 5, use Preview mode to confirm that the deduplication works as expected with all the customizations. 

If everything in the CSV looks correct, return to Insycle and move forward with applying the changes in Update mode.

Frequently Asked Questions

How can I find duplicates when one field is empty?

When using two or more fields to identify duplicates, records can still be considered matches even if one of the field values is blank. You just need to specify which field(s) allow a blank value.

Under Step 1, configure your matching rules in the Simple tab, then click the Conditions tab.

step-1-conditions-tab-arrow.png

All the matching fields you included will automatically appear with the Value Required in All Records condition selected. Change the condition to Empty Allowed in Any Record to allow empty values for certain fields. You can also use the At Least One Record with Non-Empty condition to help you determine which is the master record. Make sure at least one field remains required and is a reliable unique identifier to ensure the records are really duplicates.

step-1-conditions-empty-not-empty.png

For example, on the Simple tab, you may have the matching fields: First Name, Last Name, and Phone Number. But on some of your records, the Phone Number field may be empty. Using the conditions "Empty Allowed in Any Record," or "At Least One Record with Non-Empty," all records with the same name, same phone number, and no phone number will be considered duplicates.

step-2-group-w-empty.png

Can I match duplicates using two different fields?

Yes, you can match duplicates using data in two separate fields. For example, you can look at both the Business Phone and Mobile Phone field values as a single pool of values to compare records and identify duplicates.

Using the Related Fields feature, you can use two different fields (that contain similar data) as matching fields to catch more duplicates. You can set up Related Fields in the Advanced tab of Step 1.

merge-duplicates-salesforce-contacts-first-last-name-phone-step-1-advanced-tab.png

Do the fields I use to match need to be exactly the same?

No, your field data does not need to match exactly. The Similar Match Comparison Rule found in Step 1 looks for values that may be close but with a one-character difference (maybe a typo) and broadens the search.

similar-match.png

This search behaves like when Google shows results for a slightly different term, or says “Did you mean...” For example, if an Email of, “huey@coahulldu.co” is found, it could include records with the values “hueyy@coahulldu.co," or "hue.y@coahulldu.co,” as a match.

similar-match-_1.png

When you do use Similar Match, make sure to carefully review the results to ensure the duplicates being identified are what you're expecting.

Review the Understanding Similar Matching best practices for more detail.

Why can I only process 50 duplicate groups at a time?

Insycle shows 50 records on the module screen as a preview, this isn't the entire list of records. View the Preview CSV report to see the results for all records.

Insycle can process thousands of duplicate groups in one operation. Potentially, you could deduplicate your entire database in one operation. 

How many duplicates can I merge into one master record?

You can merge up to 100 duplicates into one single master record. 

If you have duplicate groups that contain more than five records, you may want to change the value in Skip duplicate groups with more than 5 records per group under Step 3 to make sure you can get them all.

merge-duplicates-step-3-bulk.png

This is a precaution to ensure that if you use a duplicate matching filter that is too broad in Step 1, you do not accidentally merge many non-duplicate records together. If you are going to set this number at a high level, it is a good idea to run Preview Mode first to make sure your deduplication template is operating as you intend.

Are there any limits on the number of records that can be identified and merged with my paid subscription?

All plans include unlimited usage, unlimited users, and unlimited operations. During the free trial, there is a cap of 500 records updated, cleansed, or merged. See the pricing page for more details.

Additional Resources

Related Help Articles

Related Blog Posts