How to Deduplicate People, Companies in Bulk and Automatically

You have duplicate records for contacts and companies in your CRM. This causes your sales team to step on each other's toes, hurts customer experiences, balloons your marketing budget, impacts reporting, damages your brand reputation, and causes headaches across your organization as your teams are forced to check multiple records to gain full context. 

Insycle's Merge Duplicates module can identify duplicate records and merge them flexibly, in bulk, using advanced rules and master record selection logic.

Process Summary

  1. Set rules to identify duplicates.
  2. Review and analyze the identified duplicates.
  3. Choose Bulk Mode.
  4. Set master record selection and rules determining what field data is retained.
  5. Deduplicate your records

Step-by-Step Instructions

Step 1: Find Duplicates

Navigate to the Merge Duplicates module. From the top of the screen, choose whether you will be deduplicating contacts or companies.

For a quick start, explore the default templates. There might be a template that already does exactly what you need. You can also use an existing template as a starting point and adjust it to your requirements.

Deduplication Templates

To find duplicates, you need to define how to match records.

For example, to find duplicate people, you might match using First Name, Last Name, and Email Domain fields. In that example, people with the same first name AND last name AND email domain will show as possible duplicates.

Insycle gives you deep control over how to match the values of those fields.

field-button

Choose fields that, in combination, give a high degree of certainty that the matched records are duplicate records.

Commonly used matching fields in deduplication include:

  • First Name + Last Name
  • Company Name
  • Email
  • Email Domain
  • Company Website
  • Phone Number

You can add fields by clicking the "+" button, or remove fields by clicking the X button. When you click Find, Insycle will identify duplicate records based on the matching filter.

Advanced Matching Rules
Exact Match vs. Similar Match
  • Exact Match looks for records that match exactly, with no differences from one record to the next. This is great for identifying surface-level duplicates while remaining highly accurate.
  • Similar Match analyzes data and identifies customer records that are similar, but not exact matches. It works by analyzing the “closeness” of two different data points.

Closeness is determined by measuring the number of changes necessary to make the two data points match.

  • insertion: bar → barn
  • deletion: barnbar
  • substitution: barnbark
Ignored
Ignored allows you to pick parts of the value to ignore. For example, ignore symbols and whitespace when comparing phone numbers, or ignore HTTP, www, subdomain, or top-level domain (.com vs co.uk) when comparing websites or email domains. This is great for catching more advanced duplicates.
Match Parts
Match Parts allows you to compare specific portions of a particular field. For instance, you can pick whether to compare the entire value, any two words, just the first five letters, or a certain number of final characters, etc. This is great for specific use cases.

When finished, click the Find button and Insycle will generate a list of duplicates for you to review.

Step 2: Review Duplicates

Preview the identified duplicates and double-check to make sure that your filter is producing accurate results. You want to avoid merging records that are non-duplicates.

Review duplicates

Click the Find button. The Record Viewer shows the identified duplicate groups. A duplicate group is when two or more records represent the same entity (person, company, or other), based on your matching rules. For example, if you had four records for the same person, it would count as one duplicate group with four records.

In this view, each row represents a duplicate group. When you click a row, it expands and shows the records that belong to that group.

Duplicate group

Dig into your identified duplicate groups. Check for accuracy and decide the best way to determine which master record other records will be merged into in Step 4.

Add more columns to the view using the gear button on the right to get more context and analyze the records.

Step 3: Choose Bulk Mode

You can merge duplicates manually or in bulk using rules.

In Bulk Mode, you set rules for determining the master record automatically.

When merging in bulk, Insycle limits the duplicate group size to five and skips groups that contain more records. You can adjust the default value of five records as needed, up to 100 per duplicate group. This is a precaution to ensure that if you use a duplicate matching filter that is too broad in Step 1, you do not accidentally merge many non-duplicate records together.

mceclip0.png

In addition to Bulk Mode, there is also Manual Mode, which gives you full control of what records are merged on a case-by-case basis.

Step 4: Select the Master Record

The master record is the record that will remain after the duplicates are merged. If three records are identified as duplicates and merged, the other two will not exist anymore, leaving the master behind.  

Here you set rules for determining the master record automatically when merging.

 

Insycle gives you complete control over how data is merged into your master record using advanced rules. You can set rules for determining your master record, then add rules for determining what data is retained on a field-by-field basis. 

Master Selection By Record

The master record from each duplicate group is determined using rules via an elimination process. For each duplicate group, Insycle eliminates records that do not meet the rule criteria.

Rules are read in order, from top to bottom. So if a master record meets the first criteria, the others on the list will not impact the selected master record. In the example below, the first criterion is the highest number of marketing emails clicked.

bulk master selection

Here we have four records that represent the same person—Britany Calvert. That is one duplicate group.

Here is the data that we have for this duplicate group:

Record ID Name Email Contact Owner

1000

Britany Calvert

britany@acme.com

 

1001

Britany Calvert

bcalvert@acme.com

 

1002

Britany Calvert

britany.alvert@acme.co.uk

Jane Adams

1003

Britany Calvert

britany.alvert@gmail.com

 

We haven't sent any emails to Britany yet. That means that when Insycle processes the first three rules—emails clicked, emails opened, and emails bounced—it cannot eliminate any record because they all have no value in the field.

In the next rule about contact owner, records 1000, 1001, and 1003 are eliminated since no contact owner exists for those records. Now only record 1002 remains, making it the master record.

You can add or remove rules using standard and custom fields and reorder the rules as needed using the arrow buttons.

Priority Match vs. Absolute Match

On the right-hand side of Step 4, you will find the option to select Priority and Absolute Match.

  • Priority Match: Go down the master selection rules in order, one by one. As soon as a record meets one of the criteria, it becomes the master record. The vast majority of duplicate templates should use Priority Match.
  • Absolute Match: The master record must meet all of the listed rules in the Record tab in Step 4. If a record does not match every rule listed, no master record will be identified. Absolute match is less commonly used but is ideal for specific use cases.

Choosing Absolute Match can often result in no master record being identified since the record has to match every rule listed. The vast majority of the time, you should select Priority Match.

Master Selection By Field

Insycle also allows you to pick which data to retain in the master record on a field-by-field basis.

For example, you might want the First Name and Last Name fields to be retained from the earliest created record in the duplicate group. But, you may want the email to come from the most recently updated record. Insycle makes this possible. In this situation, the First Name, Last Name, and Email will be pulled from records in the group based on your settings, but the master record will retain its data for all additional fields.

Let's take a look at another example:

By-field duplicate feature

There are multiple options for retaining field data:

  • From master record: The value for this field will always be retained from the master record you set in the Record tab in this step.
  • From record where value: For example, retain the data in the Employee # field that is associated with the record with the most employees.
  • From record based on other field: Retain a field based on criteria for a secondary field. In the example above, we retain the data in the Contact Owner field from the record with the earliest create date.
  • Combine and append all values: Additionally, you can append data into master selection fields as well. This is useful for Notes fields or multi-select picklist fields.

In this example, we have set master selection rules for five different fields; Annual Revenue, Employees, Contact Owner, Lifecycle Stage, and My Custom Field.

In the case of Lifecycle Stage, there are two rules. Insycle will look at the first rule in the list, and if it finds a record that matches the criteria, the second rule will be ignored. In the example above, if Insycle were to find a record in the duplicate group that had the Lifecycle Stage of Customer, then it would ignore the following rule looking for SQL.

Using field-by-field master selection rules gives you unparalleled control over what data is retained in your master record during a bulk merge.

 Tip: Add Fields to Report Adding fields to this section also ensures that these fields show up on your report, whether you run in Preview Mode or Update Mode. For instance, if you want to show the Lifecycle Stage of each contact, you could include the Lifecycle Stage field and instruct Insycle to retain data from the master record. Then, your template will run as normal and the field will be included in your run report.
Customizations

For situations where you need more granular customization for picking duplicate records to include—or exclude—from the deduplication process, you can customize bulk deduplication using exclusions and pre-defined masters. Additionally, you can use this process when there are no common rules you can apply to choose the master record.

Separately, it is also possible to customize the merge logic behavior. For example, you can instruct Insycle to copy values from field A to field B as part of the merge, or combine multiple fields into one. Customized merge logic requires an Enterprise plan. Please reach out via chat to discuss your specific requirements.

Save Deduplication Template

When you are satisfied with how your Merge Duplciates template operations, you can save all of the configurations as a template. Return to the Template menu at the top of the page and click the + to name your template and then click the Save button.

merge3.png

Preview Merging Changes in CSV Report

Now with the filters and master record set up, you can preview the changes you are making to your data. That way, you can check to ensure your deduplication configuration is working as expected before those changes are pushed to your live database. The CSV report that Insycle generates includes your entire merge operation down to individual duplicate groups.

Click the Review button and select Preview mode.

merge duplicates preview and review

Click the Next button to go to the Notify screen, where you can select recipients for the email report. You can also add additional context on this screen.

share report

On the When tab, select which records to apply the change to (in most cases this will be All), then click Run Now.mceclip6.png

Open the CSV file from your email in a spreadsheet application.

excel.png

The report shows the following columns (in order):

  • Each field selected in Step 1 will be included as a column.
  • The Record ID, identifier (this varies depending on record type: Contacts-Email, Company- Company name, Deal- Deal Name), and Deeplink columns are the defaults included in all reports.
  • Each field included in Step 4 will have a column. You could add other fields to Step 4 to provide more context about the records in the report. 
  • The Duplicate Group ID column uses the ID of the master record.
  • The Result column identifies which were picked as master, and which were identified as duplicates and merged into the master. You'll see the values:
    • Duplicate – The record is part of a duplicate group.
    • Master The master record that was chosen for the duplicate group based on your rules.
    • Master (After) – For each duplicate group, the Result column will show the data the final record will contain, based on master selection settings both for by-record and by-field configurations.
    • Error – If Insycle was not able to determine which record would be the master, an error message will appear here.
  • When a field value says "(Default)," it means that the CRM will be using its default processes for dealing with the field. This is typically done for blank fields, system IDs, and other specific situations.

If everything in the After column looks correct, return to Insycle and move forward with applying the changes.

Troubleshooting

If the Result column of the CSV displays an error, read the error text for help figuring out how to resolve the problem.

excel2.png

The most common error is:

Cannot determine master record because multiple records (#) satisfy the master selection rules. In ‘Master Selection’, change/add/reorder the rules such that only one record satisfies them (if cannot determine master based on field values, use ‘ID is lowest’ as the last rule).

This means that based on all the rules, Insycle could not figure out which would be the master. None of the records meet more of the rules than others. In this case, you'll need to experiment with reordering or adding additional fields likely to have unique values under Step 4.

Apply Changes To Your CRM

Click the Review button under Step 5, then in the popup, select the Update option and click Next.

merge1.png

On the Notify tab add any additional recipients, then click Next.

On the When tab you can choose to run and apply the update now, or you can set up an automation to run the template on a set schedule. The first time you apply these changes to the CRM, we suggest you use Run Now.

Click the Run Now tab and select which records to apply the changes to. Then click the Run Now button, then confirm.

merge2.png

Automate Deduplication to Keep Your CRM Duplicate Free

With a working deduplication template, you can then schedule it to run automatically on a set schedule.

On the When tab when launching your operation, select the Automate tab.

Then, you can set a schedule for this template to automatically run. Click Schedule, then confirm the automation in the popup.

mceclip8.png

All automation can be reviews on the Automations page, located on the main lefthand menu. Learn more about setting up an automation.

Troubleshooting

If you're not seeing the results you expect when merging duplicates, consider these issues:

Not all identified duplicates merging into the master?

If you have duplicate records that have been identified by Insycle using your settings in Step 1, but not all of them are merging into the master, check to see how many duplicates are in the affected duplicate groups. You may need to change your maximum merge settings in Step 3, in the Bulk tab.

duplicate-groups.png

This setting protects against accidental merging of non-duplicate records if the filter in Step 1 is too broad.
Insycle unable to determine master record?

If the Result column of the CSV report displays this error:

Cannot determine master record because multiple records (#) satisfy the master selection rules. In ‘Master Selection’, change/add/reorder the rules such that only one record satisfies them (if cannot determine master based on field values, use ‘ID is lowest’ as the last rule).

First, check Step 4 to ensure that you have Priority Match selected and not Absolute Match.

priority vs absolute match

With Priority Match, the rules are processed in order and your master record only has to match one rule. In Absolute Match, your master record would have to meet all of the criteria in the Records tab in Step 4.

This error means that based on the master rules you set, Insycle could not figure out which would be the master. None of the records meets the criteria more than the others. In this case, you'll need to experiment with reordering or adding additional fields likely to have unique values under Step 4.

The vast majority of the time it is best to select Priority Match.

Merging non-duplicate records together?

Currently, there are two ways to make sure that the records that you are merging are indeed duplicate records.

First, always run your deduplication templates in Preview Mode before running them in Update Mode. Then you can ensure that your Merge Duplicates template is working as expected and not merging non-duplicate records together.

Additionally, you can reduce risk when merging duplicates by narrowing your filter in Step 1:

  • Use Exact Match instead of Similar Match
  • Use truly uniquely identifying fields—a combination of first name, last name, email, website, email domain, and phone number are common choices.

The more tightly defined your filter is, the less likely you are to merge non-duplicate records.

Lastly, check your settings in the Bulk tab of Step 3. This is a safety net setting, limiting the number of duplicates in a duplicate group that can be merged together if you accidentally set your filter too broad in Step 1. 

Insycle isn't finding any duplicates?

This is a complex question without further context. Most of the time when Insycle can't find duplicates, it is due to your matching rules in Step 1. To better understand how to setup your template, it is important that analyze the underlaying data. 

A useful exercise can be to setup your match filters to look at exact match First Name and Last Name. 

first-last-name-duplicate-analyze.png

Of course, you never want to merge duplicates using this template. Many people have the same first and last name but are not duplicates.

But if you click the Find button, and this template lets you see a broad overview of what duplicates are potentially in your database, and what fields might be useful to include in your matching fields. 

To get further context, click the gear button on the right side of the Record Veiwer pane. Here, you can add any field in your database to the Record Viewer to better understand the data inside of these records. 

merge4.png

For a complete guide to troubleshooting issues with Insycle, refer to our article on Troubleshooting Issues

Frequently Asked Questions

I already have a list of duplicates, can Insycle bulk merge them?

Yes. You can use a customized list of duplicates and use Insycle's Magical Import module to tag duplicates in your CRM, then use the Merge Duplicates module to deduplciate in bulk. Include ID numbers from your connected CRM in your CSV.

Why can I only process 50 duplicate groups at a time?

Insycle can process thousands of duplicate groups in one operation. Potentially, you could deduplicate your entire database in one operation. Insycle shows 50 records on the module screen as a preview. You do not necessarily need to select the duplicate groups individually, because you will be able to select all relevant records after clicking the Review button to begin the operation.

deduplicate in bulk
How do I ensure that I am not merging non-duplicate records together?

Currently, there are two ways to make sure that the records that you are merging are indeed duplicate records.

First, always run your deduplication templates in Preview Mode before running them in Update Mode. This produces a CSV that shows you how your records would have been merged. Then you can ensure that your Merge Duplicates template is working as expected and not merging non-duplicate records together.

Additionally, you can reduce risk when merging duplicates by narrowing your filter in Step 1. Use Exact Match instead of Similar Match. Make sure that you are using true uniquely identifying fields—first name, last name, email, and phone number are popular choices. The more tightly defined your filter is, the less likely you are to merge non-duplicate records.

How many duplicates can I merge into one master record?

You can merge as many duplicates as you would like into one single master record. This setting is available in Step 3 of the Merge Duplicates module.

maximum merge

This setting ensures that if you set too broad of a filter, you will not accidentally merge many non-duplicate records together by accident. This is precautionary, but you can change the setting as needed to match the size of your duplicate groups. If you are going to set this number at a high level, it is a good idea to run Preview Mode first to make sure your deduplication template is operating as you intend.

Does the amount of identified and merged records have any limits in the paid subscription?

All plans include unlimited usage, unlimited users, and unlimited operations. See the pricing page for more details. During the free trial, there is a cap of 500 records updated, cleansed, or merged.

Insycle is having trouble determining a master record. What could be causing this issue?

If the Result column of the CSV report displays this error:

Cannot determine master record because multiple records (#) satisfy the master selection rules. In ‘Master Selection’, change/add/reorder the rules such that only one record satisfies them (if cannot determine master based on field values, use ‘ID is lowest’ as the last rule).

First, check Step 4 to ensure that you have Priority Match selected and not Absolute Match.

priority vs absolute match

With Priority Match, the rules are processed in order and your master record only has to match one rule. In Absolute Match, your master record would have to meet all of the criteria in the Records tab in Step 4.

This error means that based on the master rules you set, Insycle could not figure out which would be the master. None of the records meets the criteria more than the others. In this case, you'll need to experiment with reordering or adding additional fields likely to have unique values under Step 4.

The vast majority of the time it is best to select Priority Match.

Can I select which data is retained in my master record on a field-by-field basis?

Yes, Insycle allows you to select which fields are retained in the master record based on predetermined rules. 

By-field duplicate feature
I need to exclude some records from deduplication. Can I do that?

Yes. You can exclude records from deduplication by including a Deduplication Exclude field in your CSV, as detailed in this Customize Bulk Deduplication Using Exclusions and Pre-Defined Masters article.

My team needs to review and approve the master, can I accommodate that with Insycle?

Yes. You can manually approve master records and mark them in a CSV, then use Insycle to bulk deduplicate down to those master records. Consult with this Customize Bulk Deduplication Using Exclusions and Pre-Defined Masters article to learn more.

Alternatively, you can run the Merge Duplicates module in preview mode, then deliver the preview CSV that Insycle generates. This CSV shows exactly how the merge would take place, but does not update your live data. Then they can approve the merge based on this document, before running Merge Duplicates in Update Mode.

Additionally, you can choose the master manually choosing the "Manual" tab in Step 4.

master record
Do my matching fields have to match each other exactly?

No, Insycle does allow you to use similar matching to identify duplicates, although you should be careful not to catch records that are not actually duplicates using similar matching. Please read more about that in this help article and more about deduplication best practices.

How can I find duplicates when one field is empty?

Insycle requires the values to match Step 1. If the value for the matching fields listed in Step 1 are blank, Insycle will not identify those records as duplicates. 

You can work around an empty matching value by removing that condition/field from Step 1 and adding it in Step 4. Then, the field will be included in the report generated for you to review and identify when duplicates weren't merged due to a missing field.

For example, in Step 1 you may have the matching fields: First Name, Last Name, Phone Number. But sometimes, the Phone Number field may be empty.

Remove the Phone Number Field, and add Phone Number Exists as the last rule in Step 4. Now, Insycle will only use First Name and Last Name to identify the duplicates, but the Phone Number field will be included in the report for those contacts. Then you can use a customized approach to deduplication to merge records that are missing the phone number.

Additional Resources

Related Help Articles

Related Blog Articles