Customize Bulk Deduplication Using Exclusions and Pre-Defined Masters

Navigate to Data Management > Merge Duplicates, and select the database and record type in the top menu. Then explore the default templates for a pre-built deduplication solution, or build your own custom Merge Duplicates template.

Next, go through Steps 1-5 of the module to create a Preview CSV. Make sure to pause there; you don't want to Update your records yet. Follow the instructions found in the Bulk Merge Duplicate People and Companies article through to Step 5: Preview Merged Changes in CSV Report.

Insycle will generate a preview CSV and send it to your email.

Open the Preview CSV file from your email in a spreadsheet application.

The CSV will have a row per record and include the following columns:

• The Result of the action
• A Message with details
• The Duplicate Group ID which indicates which records will be merged together
• A record Status that identifies which were picked as master and which were identified as duplicates and merged into the master. See below for more details
• All fields used to identify the duplicates in Step 1
• The record ID, record name or email, and Deeplink to the CRM record
• All fields selected for the Record and Fields rules in Step 4. Note that if a field is used on both tabs, it will only appear once in the CSV

Add two new columns to the CSV:

• Deduplication Exclude
  and
• Deduplication Master

Now given the context from all the fields available in the report, update the CSV as needed:

• Enter TRUE in the Deduplication Exclude column for any row you'd like to exclude from the merge.
• Enter TRUE in the Deduplication Master for the row you'd like to designate as the master from within a duplicate group.

When you're done filling in the Deduplication Exclude and Master columns, remove all columns from the CSV except for:

• Record ID
• Deduplication Exclude
• Deduplication Master

When finished, save the CSV as a new file to your computer.

In your CRM, create two custom fields:

• Deduplication Exclude
  and
• Deduplication Master

For the field type, pick "Boolean" or "Checkbox."

The underlying field names for these fields in the CRM are: 

• deduplication_exclude
• deduplication_master

After creating the CRM fields, refresh the data in Insycle by logging out and then logging back in. You could also navigate to Settings > Sync Status, then select the account, and click the Refresh Accounts List button.

Open the Magical Import module, and load the saved CSV file with your new custom fields.

Under Step 1, map the CRM ID column to the CSV Record ID field. The other two fields will map automatically.

Under Step 2, set the ID field for the Identity Field and select the Records Mode, "Only update existing," because those records already exist in the CRM. You're just "tagging" those existing records for deduplication.

Then click the Import button under Step 5 to import the CSV. ⚠️ Note that there is no preview step in the Magical Import module. All changes are immediately applied to your CRM.

Now that records have the deduplication tags in the CRM, you can use the custom fields to tailor the cleanup process.

Return to the Merge Duplicates module. In Step 1, click the Filter button to exclude records from deduplication.

For the filter settings in the popup, select the Deduplication Exclude field, the "not any of" Condition, and enter "Yes" in the value field.

To designate your chosen records as the masters, in Step 4, remove all rules and add just one rule: Deduplication Master, "is," "Yes."

Preview Your Changes in a New CSV Report, Then Apply the Updates to Your CRM

Under Step 5, use Preview mode to confirm that the deduplication works as expected with all the customizations. 

If everything in the CSV looks correct, return to Insycle and move forward with applying the changes to your CRM using Update mode.

When using two or more fields to identify duplicates, records can still be considered matches even if one of the field values is blank. You just need to specify which field(s) allow a blank value.

Under Step 1, configure your matching rules in the Simple tab, then click the Conditions tab.

All the matching fields you included will automatically appear with the Value Required in All Records condition selected. Change the condition to Empty Allowed in Any Record to allow empty values for certain fields. You can also use the At Least One Record with Non-Empty condition to help you determine which is the master record. Make sure at least one field remains required and is a reliable unique identifier to ensure the records are really duplicates.

For example, on the Simple tab, you may have the matching fields: First Name, Last Name, and Phone Number. But on some of your records, the Phone Number field may be empty. Using the conditions "Empty Allowed in Any Record," or "At Least One Record with Non-Empty," all records with the same name, same phone number, and no phone number will be considered duplicates.

Yes, you can match duplicates using data in two separate fields. For example, you can look at both the Business Phone and Mobile Phone field values as a single pool of values to compare records and identify duplicates.

Using the Related Fields feature, you can use two different fields (that contain similar data) as matching fields to catch more duplicates. You can set up Related Fields in the Advanced tab of Step 1.

You can merge specific records using a CSV file containing the records you want to combine. Here's how:

• Prepare a CSV file with columns for the record IDs and a "Merge Master" column. In the "Merge Master" column, mark which record should be kept after merging.
• Create a custom field called "Merge Master" in your CRM.
• Use the Magical Import module to import your CSV file into the CRM, updating the "Merge Master" field for the relevant records.
• Go to the Merge Duplicates module and set up a filter to select records based on the "Merge Master" field.

See the Step-by-Step Instructions above for full details.

No, your field data does not need to match exactly. The Similar Match Comparison Rule found in Step 1 looks for values that may be close but with a one-character difference (maybe a typo) and broadens the search.

This search behaves like when Google shows results for a slightly different term, or says “Did you mean...” For example, if an Email of, “huey@coahulldu.co” is found, it could include records with the values “hueyy@coahulldu.co," or "hue.y@coahulldu.co,” as a match.

When you do use Similar Match, make sure to carefully review the results to ensure the duplicates being identified are what you're expecting.

Review the Understanding Similar Matching best practices for more detail.

Insycle shows 50 records on the module screen as a preview, this isn't the entire list of records. View the Preview CSV report to see the results for all records.

Insycle can process thousands of duplicate groups in one operation. Potentially, you could deduplicate your entire database in one operation. 

You can merge up to 100 duplicates into one single master record. 

If you have duplicate groups that contain more than five records, you may want to change the value in Skip duplicate groups with more than 5 records per group under Step 3 to make sure you can get them all.

This is a precaution to ensure that if you use a duplicate matching filter that is too broad in Step 1, you do not accidentally merge many non-duplicate records together. If you are going to set this number at a high level, it is a good idea to run Preview Mode first to make sure your deduplication template is operating as you intend.

All plans include unlimited usage, unlimited users, and unlimited operations. During the free trial, there is a cap of 500 records updated, cleansed, or merged. See the pricing page for more details.