Customize Bulk Deduplication Using Exclusions and Pre-Defined Masters

deduplicate customer data

How to Customize Merging Duplicates in Bulk Using a CSV

You have many records with duplicate data, but you need granular control over what records are merged together and which ones they are merged into. it could be that your records don't have anything in common, so Insycle can't find matches.

With Insycle, you can use CSV files to customize your bulk merging, designate master records, and exclude records from deduplication. Then you can import the CSV from the Magical Import, and use the Merge Duplicates module for complete control over the final merge operation. 

Process Summary

  1. Create a preview CSV report of duplicates.
  2. Update the CSV to specify masters or records to be excluded from merging.
  3. Create custom fields in your CRM.
  4. Import the edited CSV records into the CRM.
  5. Use the new, custom fields to merge the duplicates in your CRM.

 

Step-by-Step Instructions

1. Create a CSV Report of Duplicates

Navigate to the Merge Duplicates module, select the database, and the record type in the top menu. Then explore the default templates for a pre-built deduplication solution, or build your own custom Merge Duplicates template.

Next, go through Steps 1-5 of the module to create a Preview CSV. Make sure to pause there, you don't want to Update your records yet. Follow the instructions found in the Bulk Merge Duplicate People and Companies article.

cus1.png

Insycle will generate a preview CSV and send it to your email.

2. Edit the CSV to Specify Excluded Records and Set Master Records

Open the CSV file from your email in a spreadsheet application.

The CSV will have a row per record and include the following columns:

  • Record ID
  • Deeplink (to open the record in the CRM)
  • All fields selected in Step 1
  • All fields selected in Step 4
  • The Result column identifies which were picked as master, and which were identified as duplicates and merged into the master.

Add two new columns:

  • Deduplication Exclude
  • Deduplication Master

Now given the context from all the fields available in the report, update the CSV as needed:

  • Enter TRUE in the Deduplication Exclude column for any row you'd like to exclude from the merge.
  • Enter TRUE in the Deduplication Master for the row you'd like to designate as the master from within a duplicate group.

cus2.png

When you're done filling in the Deduplication Exclude and Master columns, remove all columns from the CSV except for:

  • Record ID
  • Deduplication Exclude
  • Deduplication Master

cus3.png

When finished, save the CSV file to your computer.

3. Create Custom Fields in the CRM

In your CRM, create two custom fields:

  • Deduplication Exclude
  • Deduplication Master

For the field type pick "Boolean" or "Checkbox."

The underlying field names for these fields in the CRM are: 

  • deduplication_exclude
  • deduplication_master

After creating the CRM fields, refresh the data in Insycle by logging out and then logging back in. You could also navigate to Settings > Sync Status, then select the account, and click the Refresh Accounts List button.

4. Use Magical Import to Tag Duplicates

Open the Magical Import module, and load the saved CSV file with your new custom fields.

Under Step 1, map the CRM ID column to the CSV Record ID field. The other two fields will map automatically.

cus4.png

Under Step 2, set the ID field for the Identity Field and select the Records Mode, Only update existing, because those records already exist in the CRM. You're just "tagging" those existing records for deduplication.

cus5.png

Then click the Import button at the bottom to import the CSV.

5. Use Merge Duplicates to Run the Operation 

Now that records have the deduplication tags in the CRM, you can use the custom fields to tailor the cleanup process.

Return to the Merge Duplicates module. To exclude records from deduplication, in Step 1, click the Filter button.

step-1-simple-tab-w-filter-arrow.png

For the filter settings in the popup, select "Deduplication Exclude," "Not Any Of," and enter "Yes" in the Value field.

cus7.png

To designate your chosen records as the masters, in Step 4, remove all rules and add just one rule: "Deduplication Master," "is," "Yes."

cus8.png

Preview Your Changes in a New CSV Report, Then Apply the Updates to Your CRM

Under Step 5 use Preview mode to confirm that the deduplication works as expected with all the customizations. 

If everything in the CSV looks correct, return to Insycle and move forward with applying the changes in Update mode.

Frequently Asked Questions

Why can I only process 50 duplicate groups at a time?

Insycle shows 50 records on the module screen as a preview, this isn't the entire list of records. View the Preview CSV report to see the results for all records.

Insycle can process thousands of duplicate groups in one operation. Potentially, you could deduplicate your entire database in one operation. 

How many duplicates can I merge into one master record?

You can merge up to 100 duplicates into one single master record. 

If you have duplicate groups that contain more than five records, you may want to change the value in Skip duplicate groups with more than 5 records per group under Step 3 to make sure you can get them all.

cus9.png

This is a precaution to ensure that if you use a duplicate matching filter that is too broad in Step 1, you do not accidentally merge many non-duplicate records together. If you are going to set this number at a high level, it is a good idea to run Preview Mode first to make sure your deduplication template is operating as you intend.

Does the paid subscription have any limits on the number of records that can be identified and merged?

All plans include unlimited usage, unlimited users, and unlimited operations. See the pricing page for more details. During the free trial, there is a cap of 500 records updated, cleansed, or merged.

Do my matching fields have to match each other exactly?

No. The Similar Match Comparison Rule found in Step 1 looks for values that may be close but with a one-character difference (maybe a typo) and broadens the search. This search behaves like when Google shows results for a slightly different term, or says “Did you mean...” For example, if a Company Name of, “Acme” is found, it could include records with the Company Name values “Akme, acm, Acma,” etc., as a match.

You should be careful when using Similar Match as the looser criteria can incorrectly identify non-duplicates as duplicates. 

Review the Be Careful With Similar Matching best practices for more detail.

Additional Resources

Related Help Articles

Related Blog Articles