Customize Bulk Deduplication Using Exclusions and Pre-Defined Masters

Navigate to the Merge Duplicates module, select the database, and the record type in the top menu. Then explore the default templates for a pre-built deduplication solution, or build your own custom Merge Duplicates template.

Next, go through Steps 1-5 of the module to create a Preview CSV. Make sure to pause there, you don't want to Update your records yet. Follow the instructions found in the Bulk Merge Duplicate People and Companies article.

Insycle will generate a preview CSV and send it to your email.

Open the CSV file from your email in a spreadsheet application.

The CSV will have a row per record and include the following columns:

• Record ID
• Deeplink (to open the record in the CRM)
• All fields selected in Step 1
• All fields selected in Step 4

• The Result column identifies which were picked as master, and which were identified as duplicates and merged into the master.

Add two new columns:

• Deduplication Exclude
• Deduplication Master

Now given the context from all the fields available in the report, update the CSV as needed:

• Enter TRUE in the Deduplication Exclude column for any row you'd like to exclude from the merge.
• Enter TRUE in the Deduplication Master for the row you'd like to designate as the master from within a duplicate group.

When you're done filling in the Deduplication Exclude and Master columns, remove all columns from the CSV except for:

• Record ID
• Deduplication Exclude
• Deduplication Master

When finished, save the CSV file to your computer.

In your CRM, create two custom fields:

• Deduplication Exclude
• Deduplication Master

For the field type pick "Boolean" or "Checkbox."

The underlying field names for these fields in the CRM are: 

• deduplication_exclude
• deduplication_master

After creating the CRM fields, refresh the data in Insycle by logging out and then logging back in. You could also navigate to Settings > Sync Status, then select the account, and click the Refresh Accounts List button.

Open the Magical Import module, and load the saved CSV file with your new custom fields.

Under Step 1, map the CRM ID column to the CSV Record ID field. The other two fields will map automatically.

Under Step 2, set the ID field for the Identity Field and select the Records Mode, Only update existing, because those records already exist in the CRM. You're just "tagging" those existing records for deduplication.

Then click the Import button at the bottom to import the CSV.

Now that records have the deduplication tags in the CRM, you can use the custom fields to tailor the cleanup process.

Return to the Merge Duplicates module. To exclude records from deduplication, in Step 1, click the Filter button.

For the filter settings in the popup, select "Deduplication Exclude," "Not Any Of," and enter "Yes" in the Value field.

To designate your chosen records as the masters, in Step 4, remove all rules and add just one rule: "Deduplication Master," "is," "Yes."

Preview Your Changes in a New CSV Report, Then Apply the Updates to Your CRM

Under Step 5 use Preview mode to confirm that the deduplication works as expected with all the customizations. 

If everything in the CSV looks correct, return to Insycle and move forward with applying the changes in Update mode.

Insycle shows 50 records on the module screen as a preview, this isn't the entire list of records. View the Preview CSV report to see the results for all records.

Insycle can process thousands of duplicate groups in one operation. Potentially, you could deduplicate your entire database in one operation. 

You can merge up to 100 duplicates into one single master record. 

If you have duplicate groups that contain more than five records, you may want to change the value in Skip duplicate groups with more than 5 records per group under Step 3 to make sure you can get them all.

This is a precaution to ensure that if you use a duplicate matching filter that is too broad in Step 1, you do not accidentally merge many non-duplicate records together. If you are going to set this number at a high level, it is a good idea to run Preview Mode first to make sure your deduplication template is operating as you intend.

All plans include unlimited usage, unlimited users, and unlimited operations. See the pricing page for more details. During the free trial, there is a cap of 500 records updated, cleansed, or merged.

No. The Similar Match Comparison Rule found in Step 1 looks for values that may be close but with a one-character difference (maybe a typo) and broadens the search. This search behaves like when Google shows results for a slightly different term, or says “Did you mean...” For example, if a Company Name of, “Acme” is found, it could include records with the Company Name values “Akme, acm, Acma,” etc., as a match.

You should be careful when using Similar Match as the looser criteria can incorrectly identify non-duplicates as duplicates. 

Review the Be Careful With Similar Matching best practices for more detail.