Deduplication Best Practices

 

Duplicate records inhibit your entire organization. Your marketing team can't effectively segment and personalize communications. Sales teams step on each other's toes and lack vital context in conversations. Support teams miss important information, and analysis and reporting are skewed.

Insycle helps you merge duplicate contacts, companies, and deals flexibly and powerfully with the Merge Duplicates module.

However, getting the best results from your deduplication efforts involves considering how duplicates are matched and how that data is merged.

Here are some best practices to ensure good results.

Choosing Matching Fields

Insycle has to use the existing data in your CRM to identify duplicate records. Matching duplicates requires unique identifiers—data that is unlikely to be shared by any other record unless it is a duplicate.

For instance, many CRMs match first names, last names, and email addresses. If all of those match, or are similar, you can confidently determine that the record is a duplicate.

Matching rules

 

Other unique identifying fields that are commonly used in deduplication include:

  • phone number
  • mailing address
  • ID numbers

Uniquely identifying data is critical for matching. If you don't use unique identifiers, you are likely to identify unrelated records as duplicates and may accidentally merge them.

Format & Standardize Matching Fields First, Then Merge

With some matching fields, it is important that you format and standardize the data before deduplicating so that duplicates can be effectively matched.

For instance, let's say you had two duplicate records, and were matching them using first name, last name, and customer ID. If you are using Exact Match in Insycle, it is important that the data in those records is standardized so that matches can be identified. If one record has the first name "jane S." and the other "Jane," these would not be identified under Exact Match settings.

The same goes for the mailing address. If one record has the state listed as "NY" and the other "New York," they would not be matched. Addresses should be standardized before being included in deduplication processes.

When choosing matching fields, consider whether standardization would be required.

Learn More About Formatting and Standardizing Using Insycle:

Smartly Broaden Your Matching to Catch More Duplicates

Most CRMs include very basic duplicate detection systems. Often, they identify duplicates using name and email or company name. This is an effective way to find surface-level duplicates.

But by broadening the way that you match duplicates, you'll be able to identify more of them in your database.

First, you can use both exact and similar matching in Insycle. With similar matching, you can match duplicates even when the data has slight differences between the two duplicate records.

Using these options, Insycle's pre-built deduplication templates can help you match duplicates in a variety of ways. You could match using first name, last name, and email domain:

Matching rules

 

Or name and company name:

 

Or similar first name, email username, and email domain, to account for company.com/company.co or johnb@email.com/john.b@email.com variations:

similar

 

Or name and phone numbers:

deduplicate names and phone numbers

 

Building multiple templates for deduplication allows you to catch and merge more duplicates that are gumming up your processes.

Learn More About Detecting Duplicates in Advanced Ways:
Hidden Duplicates: 11 Advanced Ways to Identify & Deduplicate Customer Data

Be Careful With Similar Matching

Insycle's Merge Duplicates module gives you two options for matching fields—exact match and similar match.

Similar match will detect matches that are two keystroke deviations away from each other, like:

  • insertion: bar → barn
  • deletion: barnbar
  • substitution: barnbark

This can help you to account for differences in formatting and typos. Like adding new fields, similar matching can be a great way to detect more duplicates. But you have to be thoughtful about how it is used, or you risk matching non-duplicate records and merging them together.

For example, if you used similar matching on a phone number, colleagues of a contact might be mistakenly identified as a duplicate because they have a similar number. There may be other fields where similar matching can produce unwanted results.

While similar matching is a great way to catch more duplicates, you need to be careful about which fields you use it for.

Implement Automation

You can set up ongoing deduplication for HubSpot companies and Salesforce accounts automation with Insycle on the module level, using Recipes, or integrating with Workflows for HubSpot users. You can view all scheduled automations on the Automations page of your dashboard.

You can schedule your Merge Duplicates templates to run on an automated, set schedule. You do this by clicking the Review button at the bottom of the module page

Then, you go through a three-step process to run the operation. In the third step, you can choose the Automate tab, and schedule your template to run on a set schedule.

merge duplicates salesforce automation

 

Doing this, you can consistently ensure that HubSpot companies are merged while aligning with Salesforce accounts and keeping the sync active.

Considerations for Choosing Master Records & Fields

When you merge duplicates, you also have to consider how data is merged. The record that all duplicates are merged down into is known as the master record. Insycle gives you complete control over how data is merged into your master record using advanced rules.

You can choose to automatically select the master record using rules on the record level or on the individual field level.

Master Selection By Record

When merging in bulk, the master record from each Duplicate Group is determined using rules via an elimination process. For each Duplicate Group, Insycle eliminates records that do not meet the rule criteria until it has the master record.

Rules are read in order, from top to bottom. So if a master record meets the first criteria (in the example below, the first criteria is “highest marketing emails clicked”) the others on the list will not impact the selected master record.

bulk master selection

 

Master Selection By Field

You can also choose what data is retained in your master record on a field-by-field basis. Then, the master record will be decided based on the rules that you set for individual fields. For instance, you might want to retain Annual Revenue from the master record, but want to retain the Number of Employees from the record that has the highest number. 

Let's take a look:

By-field duplicate feature

 

Learn More About Master Records:
Bulk Merge Duplicate People, Companies

Start in Preview Mode

When you click the Review button in Insycle, you'll get to choose from two modes: Preview Mode and Update Mode.

preview mode

 

Preview mode does not update your live CRM data, but instead produces a CSV showing you the results that the deduplication process would have produced.

The first time you run a new deduplication template, it is always a good idea to run it in Preview Mode first, to ensure that the template is working as you intended. Once you confirm that it is, then you can run it in Update Mode or schedule it for automation.

Frequently Asked Questions

I already have a list of duplicates, can Insycle bulk merge them?

Yes. You can use a customized list of duplicates, as detailed in this help article, and use Insycle to deduplicate the list in bulk. Include ID numbers from your connected CRM in your CSV.

Why can I only process 50 duplicate groups at a time?

Insycle can process thousands of duplicate groups in one operation. Potentially, you could deduplicate your entire database in one operation. Insycle shows 50 records on the module screen as a preview. You do not necessarily need to select the duplicate groups individually, because you will be able to select all relevant records after clicking the Review button to begin the operation.

deduplicate in bulk

How do I ensure that I am not merging non-duplicate records together?

Currently, there are two ways to make sure that the records that you are merging are indeed duplicate records.

First, always run your deduplication templates in Preview Mode before running them in Update Mode. This produces a CSV that shows you how your records would have been merged. Then you can ensure that your Merge Duplicates template is working as expected and not merging non-duplicate records together.

Additionally, you can reduce risk when merging duplicates by narrowing your filter in Step 1. Use Exact Match instead of Similar Match. Make sure that you are using true uniquely identifying fields—first name, last name, email, and phone number are popular choices. The more tightly defined your filter is, the less likely you are to merge non-duplicate records.

How many duplicates can I merge into one master record?

You can merge as many duplicates as you would like into one single master record. This setting is available in Step 3 of the Merge Duplicates module.

maximum merge

This setting ensures that if you set too broad a filter, you will not accidentally merge many non-duplicate records together by accident. This is precautionary, but you can change the setting as needed to match the size of your duplicate groups. If you are going to set this number at a high level, it is a good idea to run Preview Mode first to make sure your deduplication template is operating as you intend.

Can I select which data is retained in my master record in a field-by-field basis?

Yes, Insycle allows you to select which fields are retained in the master record based on predetermined rules. To learn more, please see this help article.

By-field duplicate feature

I need to exclude some records from deduplication. Can I do that?
Yes. You can exclude records from deduplication by including a "Deduplication Exclude" field in your CSV, as detailed in this help article.
My team needs to review and approve the master, can I accommodate that with Insycle?

Yes. You can manually approve master records and mark them in a CSV, then use Insycle to bulk-deduplicate down to those master records. Consult with this help article to learn more.

Alternatively, you can run the Merge Duplicates module in Preview Mode, then deliver the preview CSV that Insycle generates. This CSV shows exactly how the merge would take place, but does not update your live data. Then your team can approve the merge based on this document before running Merge Duplicates in Update Mode.

Additionally, you can choose the master manually choosing the "Manual" tab in Step 4.

Do my matching fields have to match each other exactly?
No, Insycle allows you to use similar matching to identify duplicates, although you should be careful not to catch records that are not actually duplicates using similar matching. Please read more about that in this help article.
How can I find duplicates when one field is empty?

Insycle requires the values to match Step 1. If the value for the matching fields listed in Step 1 are blank, Insycle will not include those records in the deduplication.

You can work around an empty matching value by removing that condition/field from Step 1 and adding it in Step 4. Then, the field will be included in the report generated for you to review and identify when duplicates weren't merged due to a missing field.

For example, in Step 1 you may have the matching fields: first name, last name, phone number. But sometimes, the phone number field may be empty.

Remove the phone number field, and add Phone Number Exists as the last rule in Step 4. Now, Insycle will only use first name and last name to identify the duplicates, but the phone number field will be included in the report for those contacts. Then you can use a customized approach to deduplication to merge records that are missing the phone number.

Additional Resources

Related Blog Articles

Related Help Articles