Deduplication Best Practices

Insycle has to use the existing data in your CRM to identify duplicate records. Matching duplicates requires unique identifiers—data that is unlikely to be shared by any other record unless it is a duplicate.

For instance, many CRMs match first names, last names, and email addresses. If all of those match, or are similar, you can confidently determine that the record is a duplicate.

Commonly used unique identifying fields include:

• First + Last Name
• Company Name
• Email Address
• Email or Website Domain
• Phone Number
• Website URL
• Various ID Numbers
• Mailing Address

Uniquely identifying data is critical for matching. If you don't use unique identifiers, you are likely to identify unrelated records as duplicates and may accidentally merge them.

Most CRMs include very basic duplicate detection systems. Often, they identify duplicates using name and email or company name. This is an effective way to find surface-level duplicates.

But by broadening the way that you match duplicates, you'll be able to identify more of them in your database.

First, you can use both exact and similar matching in Insycle. With similar matching, you can match duplicates even when the data has slight differences between the two duplicate records.

Using these options, Insycle's pre-built deduplication templates can help you match duplicates in a variety of ways.

You can require that field values are an Exact Match to be considered a duplicate:

Or you can use the Similar Match Comparison Rule on one or more of the fields to account for close variations on a value:

If you'd like to look at the data in two different fields (that contain similar data) as if it were one, you can set up Related Fields under the Advanced tab. For example, you might want to look at both the Phone Number and Mobile Phone Number fields for duplicate values:

Building multiple templates for deduplication allows you to catch and merge more duplicates that are gumming up your processes.

Learn more about detecting duplicates in advanced ways:
Hidden Duplicates: 11 Advanced Ways to Identify & Deduplicate Customer Data

The Similar Match Comparison Rule found in Step 1 will detect matches that are two keystroke deviations away from each other, like:

• insertion: bar → barn
• deletion: barn → bar
• substitution: barn → bark

It looks for values that may be close but with a one-character difference (maybe a typo) and broadens the search.

This search behaves like when Google shows results for a slightly different term, or says “Did you mean...” For example, if an Email address of, “hueyy@coahulldu.co” is found, it could include records with the Email values “hue.y@coahulldu.co, huey@coahulldu.co,” etc., as a match.

Similar matching can be a great way to detect more duplicates, but you have to be thoughtful about how it is used, or you risk matching non-duplicate records and merging them together.

For example, if you used similar matching on a phone number, colleagues of a contact might be mistakenly identified as a duplicate because they have a similar number. There may be other fields where similar matching can produce unwanted results.

However, using similar matching on a First Name or Email field would be more likely to catch duplicate records that have typos, so it's a great way to identify more duplicates, as long as you're thoughtful.

You can set up ongoing deduplication for HubSpot companies and Salesforce accounts automation with Insycle on the module level, using Recipes, or integrating with Workflows for HubSpot users. You can view all scheduled automations on the Automations page of your dashboard.

You can schedule your Merge Duplicates templates to run on an automated, set schedule. You do this by clicking the Review button at the bottom of the module page

Then, you go through a three-step process to run the operation. In the third step, you can choose the Automate tab, and schedule your template to run on a set schedule.

When you merge duplicates, you also have to consider how data is merged. The record that all duplicates are merged down into is known as the master record. Insycle gives you complete control over how data is merged into your master record using advanced rules.

You can identify duplicate records and merge them in bulk, using advanced rules to select the master record and control data retention.

Master Selection By Record

When merging in bulk, the master record from each duplicate group is determined using rules via an elimination process. For each duplicate group, Insycle eliminates records that do not meet the rule criteria until it has the master record.

Rules are read in order, from top to bottom. So if a master record meets the first criteria (in the example below, the first criteria is “highest number of marketing emails clicked”) the others on the list will not impact the selected master record.

Configure Rules That Determine Values to Keep

The Merge Duplicates module allows you to control the values saved in the master record after the merge regardless of the default merge behavior. By adding each field you want to control the data retention for and selecting a Condition, you can tell Insycle where the data for the field should be taken from and how to handle it.

Under Step 4, click the Fields tab. For each field you want to control the data retention for, you need to select a Field, and tell it where the data for the field should be taken from. This is merged into the master. Any data that is not in the master or not copied to the master is deleted.

The Criteria dropdown gives you different options for choosing the data to keep:

• From master record – Use the value that exists in the master record.
• From master record (even empty) – If the field on the selected master record is blank, keep it that way. Don’t automatically fill it in with a value from the most recently updated record.
• From record where value – Select data from one of the records in the duplicate group based on the values. These options vary depending on the field type. For example, retain the data in the Email field that is using a professional domain rather than a free one.
• From record based on other field value – Look at the value in a different field to decide which value from the duplicate group should be kept. The example above highlights how a Last Modified Date value can be used to determine which Lifecycle Stage value to use.
• Combine and append all values – You can merge the values from the selected field for all records in the group. For example, if there is some type of Notes field, you could keep the notes from all of the records in the duplicate group.
• Collect all values from other field – Select a destination field to copy and combine values into, then select what field the data should come from for each record in the duplicate group. For example, this could be used to keep the record Owner values of all duplicates in a group and combine them into a custom field.
• Collect non-master values from other field - Aggregate the values of all the duplicates that are not the master, meaning this excludes the value from the master record itself. This can be especially helpful if you want a record of the object IDs that were removed, so you can also remove them from another system and keep the master. Select a destination field to copy and combine values into, then select what field the data should come from for each record in the duplicate group. 

Learn more about master records:
Bulk Merge Duplicate People, Companies

When you click the Review button in Insycle, you'll get to choose from two modes: Preview Mode and Update Mode.

Preview mode does not update your live CRM data, but instead produces a CSV showing you the results that the deduplication process would have produced.

When using two or more fields to identify duplicates, records can still be considered matches even if one of the field values is blank. You just need to specify which field(s) allow a blank value.

Under Step 1, configure your matching rules in the Simple tab, then click the Conditions tab.

All the matching fields you included will automatically appear with the Value Required in All Records condition selected. Change the condition to Empty Allowed in Any Record to allow empty values for certain fields. You can also use the At Least One Record with Non-Empty condition to help you determine which is the master record. Make sure at least one field remains required and is a reliable unique identifier to ensure the records are really duplicates.

For example, on the Simple tab, you may have the matching fields: First Name, Last Name, and Phone Number. But on some of your records, the Phone Number field may be empty. Using the Condition, "Empty Allowed in Any Record," or "At Least One Record with Non-Empty," all records with the same name, same phone number, and no phone number will be considered duplicates.

Yes. This can be done, for example, if you want to look at both the Phone Number field values and Mobile Phone Number field values as a single pool of values to compare between records and identify duplicates.

Using the Related Fields feature, you can use two different fields (that contain similar data) as matching fields to catch more duplicates. You can set up Related Fields in the Advanced tab.

Currently, there are two ways to make sure that the records that you are merging are indeed duplicate records.

First, always run your deduplication templates in Preview Mode before running them in Update Mode. This produces a CSV that shows you how your records would have been merged. Then you can ensure that your Merge Duplicates template is working as expected and not merging non-duplicate records together.

Additionally, you can reduce the risk when merging duplicates by narrowing your duplicate matching settings in Step 1. Try the Exact Match Comparison Rule instead of Similar Match. Then make sure that you are using actual uniquely identifying fields—first name, last name, email, and phone number are popular choices. The more tightly defined your filter is, the less likely you are to merge non-duplicate records.

If the Result column of the CSV report displays this error:

Cannot determine master record because multiple records (#) satisfy the master selection rules. In ‘Master Selection’, change/add/reorder the rules such that only one record satisfies them (if cannot determine master based on field values, use ‘ID is lowest’ as the last rule).

This error means that based on the master rules you set, Insycle could not figure out which would be the master.

Check Step 4 to ensure that you have Priority Match selected and not Absolute Match.

With Priority Match, the rules configured in the Records tab of Step 4 are processed in order and your master record only has to match one rule. Using Absolute Match, your master record would have to meet all of the rule criteria. The majority of the time it is best to select Priority Match.

If Priority Match was used, then none of the records meet any of the criteria on the list more than the others. In this case, you'll need to experiment with Step 4, reordering or adding additional rules for fields likely to have unique values.

When merging HubSpot contact records using the “From master record (even empty)” data retention rule, the property history in HubSpot shows that Insycle set the value to “empty.” This is a nuance of how HubSpot manages the history of empty values. You can verify that the master record value before the merge was indeed empty by reviewing the Activity Tracker report in Insycle.

Yes. You can use a customized list of duplicates and use the Magical Import module to tag duplicates in your CRM, then use the Merge Duplicates module to deduplicate in bulk. Include ID numbers from your CRM in your CSV.

Yes, Insycle allows you to select which field data is retained in the master record using the Fields tab under Step 4. See the Bulk Merge Duplicate People, Companies article for more detail.

Yes. You can exclude records from deduplication by including a "Deduplication Exclude" field in your CSV, as detailed in the Customize Bulk Deduplication Using Exclusions and Pre-Defined Masters article.

Yes, there are several ways to share details and get approval before merging duplicates.

You can manually approve master records and mark them in a CSV, then use Insycle to bulk deduplicate down to those master records. Consult with this Customize Bulk Deduplication Using Exclusions and Pre-Defined Masters article to learn more.

Or, you can run the Merge Duplicates module in Preview Mode, then deliver the preview CSV that Insycle generates. The CSV report that Insycle generates includes your entire merge operation down to individual duplicate groups but does not update your live data. Then your team can approve the merge based on this report, before running Merge Duplicates in Update Mode.

Additionally, team members can review duplicates and manually select the master for each record under Step 4. Review the Manually Merge Duplicates article for more detail.

No. The Similar Match Comparison Rule found in Step 1 looks for values that may be close but with a one-character difference (maybe a typo) and broadens the search. This search behaves like when Google shows results for a slightly different term, or says “Did you mean...” For example, if a Company Name of, “Acme” is found, it could include records with the Company Name values “Akme, acm, Acma,” etc., as a match.

You should be careful when using Similar Match as the looser criteria can incorrectly identify non-duplicates as duplicates. 

Review the Understanding Similar Matching best practices for more detail.

Why can I only process 50 duplicate groups at a time?

Insycle shows 50 records on the module screen as a preview, this isn't the entire list of records. Include All records when you view the Preview CSV report to see everything.

Insycle can process thousands of duplicate groups in one operation. Potentially, you could deduplicate your entire database in one operation. 

You can merge up to 100 duplicate records into a single master record. 

If you have duplicate groups that contain more than five records, you may want to change the value in Skip duplicate groups with more than 5 records per group under Step 3 to make sure you can get them all.

This is a precaution to ensure that if you use a duplicate matching filter that is too broad in Step 1, you do not accidentally merge many non-duplicate records together. If you are going to set this number at a high level, it is a good idea to run Preview Mode first to make sure your deduplication template is operating as you intend.

All plans include unlimited usage, unlimited users, and unlimited operations. See the pricing page for more details. During the free trial, there is a cap of 500 records updated, cleansed, or merged.