Deduplication Best Practices

Insycle has to use the existing data in your CRM to identify duplicate records. Matching duplicates requires unique identifiers—data that is unlikely to be shared by any other record unless it is a duplicate.

For instance, many CRMs match first names, last names, and email addresses. If all of those match, or are similar, you can confidently determine that the record is a duplicate.

 

Other unique identifying fields that are commonly used in deduplication include:

• phone number
• mailing address
• ID numbers

Uniquely identifying data is critical for matching. If you don't use unique identifiers, you are likely to identify unrelated records as duplicates and may accidentally merge them.

Most CRMs include very basic duplicate detection systems. Often, they identify duplicates using name and email or company name. This is an effective way to find surface-level duplicates.

But by broadening the way that you match duplicates, you'll be able to identify more of them in your database.

First, you can use both exact and similar matching in Insycle. With similar matching, you can match duplicates even when the data has slight differences between the two duplicate records.

Using these options, Insycle's pre-built deduplication templates can help you match duplicates in a variety of ways. You could match using first name, last name, and email domain:

Or name and company name:

Or similar first name, email username, and email domain, to account for company.com/company.co or johnb@email.com/john.b@email.com variations:

Or name and phone numbers:

Building multiple templates for deduplication allows you to catch and merge more duplicates that are gumming up your processes.

Learn More About Detecting Duplicates in Advanced Ways:
Hidden Duplicates: 11 Advanced Ways to Identify & Deduplicate Customer Data

Insycle's Merge Duplicates module gives you two options for matching fields—exact match and similar match.

Similar match will detect matches that are two keystroke deviations away from each other, like:

• insertion: bar → barn
• deletion: barn → bar
• substitution: barn → bark

This can help you to account for differences in formatting and typos. Like adding new matching fields, similar matching can be a great way to detect more duplicates. But you have to be thoughtful about how it is used, or you risk matching non-duplicate records and merging them together.

For example, if you used similar matching on a phone number, colleagues of a contact might be mistakenly identified as a duplicate because they have a similar number. There may be other fields where similar matching can produce unwanted results.

However, using similar matching on a First Name field would be more likely to catch duplicate records that have typos, 

While similar matching is a great way to catch more duplicates, you need to be careful about which fields you use it for.

You can set up ongoing deduplication for HubSpot companies and Salesforce accounts automation with Insycle on the module level, using Recipes, or integrating with Workflows for HubSpot users. You can view all scheduled automations on the Automations page of your dashboard.

You can schedule your Merge Duplicates templates to run on an automated, set schedule. You do this by clicking the Review button at the bottom of the module page

Then, you go through a three-step process to run the operation. In the third step, you can choose the Automate tab, and schedule your template to run on a set schedule.

When you merge duplicates, you also have to consider how data is merged. The record that all duplicates are merged down into is known as the master record. Insycle gives you complete control over how data is merged into your master record using advanced rules.

You can choose to automatically select the master record using rules on the record level or on the individual field level.

Master Selection By Record

When merging in bulk, the master record from each Duplicate Group is determined using rules via an elimination process. For each Duplicate Group, Insycle eliminates records that do not meet the rule criteria until it has the master record.

Rules are read in order, from top to bottom. So if a master record meets the first criteria (in the example below, the first criteria is “highest marketing emails clicked”) the others on the list will not impact the selected master record.

 

Master Selection By Field

You can also choose what data is retained in your master record on a field-by-field basis. Then, the master record will be decided based on the rules that you set for individual fields. For instance, you might want to retain Annual Revenue from the master record, but want to retain the Number of Employees from the record that has the highest number. 

Let's take a look:

 

Learn More About Master Records:
Bulk Merge Duplicate People, Companies

When you click the Review button in Insycle, you'll get to choose from two modes: Preview Mode and Update Mode.

 

Yes. You can use a customized list of duplicates, as detailed in this help article, and use Insycle to deduplicate the list in bulk. Include ID numbers from your connected CRM in your CSV.

Insycle can process thousands of duplicate groups in one operation. Potentially, you could deduplicate your entire database in one operation. Insycle shows 50 records on the module screen as a preview. You do not necessarily need to select the duplicate groups individually, because you will be able to select all relevant records after clicking the Review button to begin the operation.

Currently, there are two ways to make sure that the records that you are merging are indeed duplicate records.

First, always run your deduplication templates in Preview Mode before running them in Update Mode. This produces a CSV that shows you how your records would have been merged. Then you can ensure that your Merge Duplicates template is working as expected and not merging non-duplicate records together.

Additionally, you can reduce risk when merging duplicates by narrowing your filter in Step 1. Use Exact Match instead of Similar Match. Make sure that you are using true uniquely identifying fields—first name, last name, email, and phone number are popular choices. The more tightly defined your filter is, the less likely you are to merge non-duplicate records.

You can merge as many duplicates as you would like into one single master record. This setting is available in Step 3 of the Merge Duplicates module.

This setting ensures that if you set too broad a filter, you will not accidentally merge many non-duplicate records together by accident. This is precautionary, but you can change the setting as needed to match the size of your duplicate groups. If you are going to set this number at a high level, it is a good idea to run Preview Mode first to make sure your deduplication template is operating as you intend.

Yes, Insycle allows you to select which fields are retained in the master record based on predetermined rules. To learn more, please see this help article.

Yes. You can exclude records from deduplication by including a "Deduplication Exclude" field in your CSV, as detailed in this help article.

Yes. You can manually approve master records and mark them in a CSV, then use Insycle to bulk-deduplicate down to those master records. Consult with this help article to learn more.

Alternatively, you can run the Merge Duplicates module in Preview Mode, then deliver the preview CSV that Insycle generates. This CSV shows exactly how the merge would take place, but does not update your live data. Then your team can approve the merge based on this document before running Merge Duplicates in Update Mode.

Additionally, you can choose the master manually choosing the "Manual" tab in Step 4.

No, Insycle allows you to use similar matching to identify duplicates, although you should be careful not to catch records that are not actually duplicates using similar matching. Please read more about that in this help article.

Insycle requires the values to match Step 1. If the value for the matching fields listed in Step 1 are blank, Insycle will not include those records in the deduplication.

You can work around an empty matching value by removing that condition/field from Step 1 and adding it in Step 4. Then, the field will be included in the report generated for you to review and identify when duplicates weren't merged due to a missing field.

For example, in Step 1 you may have the matching fields: first name, last name, phone number. But sometimes, the phone number field may be empty.

Remove the phone number field, and add Phone Number Exists as the last rule in Step 4. Now, Insycle will only use first name and last name to identify the duplicates, but the phone number field will be included in the report for those contacts. Then you can use a customized approach to deduplication to merge records that are missing the phone number.