Weird duplicates cartoon

How to Analyze and Merge Duplicates Manually for Granular Deduplication Control

You have duplicate records in your CRM but you need a controlled, careful process to merge these records, so bulk merging isn't an option.

With the Merge Duplicates module, you can surface duplicates, analyze the records to determine which have the relevant data, and merge them manually, one at a time. 

Process Summary

  1. Identify duplicates.
  2. Review and analyze the duplicates.
  3. Select the records that need merging.
  4. Choose the master record.
  5. Individually select or modify values to be retained.
  6. Merge duplicate records.

 

Step-by-Step Instructions

1. Configure Rules to Identify Duplicates

Navigate to the Merge Duplicates module, pick the record type, and explore the default templates for a pre-built solution.

To find duplicates, you need to define how to match records. Step 1 looks through the records in your database, examining the fields that you specify for matches. Each row is for a field you want to look at for duplicates. 

For example, to find duplicate Contacts you may use the "First Name," "Last Name," and "Email Domain" fields. Contacts with the same first name AND last name AND email domain will show as possible duplicates. 

Choose fields that, in combination, give a high degree of certainty that the matched records are duplicate records. 

step-1-fields.png

See the Advanced How-Tos for more detail on selecting fields to use and narrowing your results with the filter.

When finished, click the Find button and Insycle will generate a list of duplicates for you to review.

Expand Criteria for Matching Duplicates

If you'd like to look at the data in two different fields (that contain similar data) as if it were one, you can set up Related Fields under the Advanced tab. For example, you might want to look at both the Email and Additional Email fields for duplicate values.

step-1-advanced-tab-related-email.png

The Conditions tab provides rules one or more of the records in a duplicate group will need to meet.

  • Value Required in All Records - Each record must contain a value in this field to be considered a duplicate.
  • Empty Allowed in Any Record - A record can still be considered a duplicate if this field is blank. Allowing empty values requires using two or more fields to identify duplicates.
  • At Least One Record With Non-Empty - At least one record in the duplicate group must contain a value.
  • At Least One Record Match - At least one record in the duplicate group must match the specified value, and the other records cannot be blank. If none of the records have the specified value, the duplicate group will not be merged.
  • Only One Record Match - If more than one record in a duplicate group contains the specified field value, the duplicate group is skipped (not merged).

merge-duplicates-salesforce-contacts-step-1-conditions-all-5.png

2. Analyze the Identified Duplicates

Records that have the same values in the fields specified in Step 1 are considered matches. When two or more records represent the same entity (person, company, or other), they are clustered together into duplicate groups. Each duplicate group shows the total number of records that were identified as duplicates. For example, if you had four records for the same person, it would count as one duplicate group with four records.

Check the box in a row to expand and see the records in the group.

dup2.png

Explore the record data in the duplicate groups. Double-check to make sure that the fields you set up in Step 1 are showing what you expected. 

Add more columns to the view using the gear button on the right to help your analysis.

3. Choose Manual Mode and Select the Records to Merge

Select Manual mode to have complete control over which records are merged. You'll work with a single duplicate group through the entire merge process.

man3.png

When you select Manual, an additional set of checkboxes will appear in Step 2 beside the individual records in each duplicate group. When you check the boxes, you are choosing which records will be merged. The data in unchecked records will not be merged. 

Under Step 2, click the checkbox by the duplicate group, then select the individual records you want to merge together.

man4.png

4. Select the Master Record

The master is the record that will remain after the duplicates are merged. If you select three records and merge them, the other two will not exist anymore. By default, data from your chosen master will be retained, and if there are any blank values, this data will automatically be filled in from the other records. If you'd like more control over the data saved in the master record, that is done in Step 5.

Under Step 4, choose the master record that the other records will be merged into.

man5.png

5. Choose or Edit Values to Merge into the Master, then Merge

If you want to control which values are kept, you can choose or edit specific fields under Step 5. This is an optional step—if you don’t pick specific values, the platform's default merging logic will be followed.

Under Step 5, only five fields are initially displayed, so to see all of the fields, change the number of rows shown per page.

merge-duplicates-manually-merge-contacts-step-5-rows.png

There are three tabs showing different field sets. The Conflicts tab shows only the fields that have differences in the values, making it easy to focus only on fields that need attention. The Read Only tab lists only the non-writable fields from your database, and the Full tab shows all of the record fields, even those without data. Use the search to find specific fields. 

On a field-by-field basis, select which values to keep and have merged into the master. For example, you could choose Contact Owner from one and Company Name from another. 

step-5-pick-values.png

To edit the individual field values one-by-one, hover over a value you want to update and click the pencil icon.

merge-duplicates-manually-merge-contacts-step-5-edit-pencil.png

In the Edit popup, update the value and click Save

merge-duplicates-manually-merge-contacts-step-5-edit-save.png

Apply the Merge to Your CRM

When you have all critical fields chosen, click the Merge button at the bottom to merge your selected duplicates. 

Note that there is no preview; this change is applied directly to your CRM.

man8.png

When To Use Manual Merging

Manual merge is great when you have only a handful of duplicates to address, need to merge records carefully, want to employ a manual review process when merging, or just want to explore a few duplicates to understand what you have and how best to merge them.

In most cases, large datasets are a better candidate for bulk deduplication

For situations where you have a large number of records to fix and need granular control for picking records to include - or exclude - from the deduplication process, or for picking the master record, and there are no common rules you can apply for all or some of the records, you can customize bulk deduplication using exclusions and pre-defined masters via a CSV file.

Advanced How-Tos

Step 1: Setting Up the Fields to Find Duplicates

Each row in your matching fields setup is cumulative, so records must meet all of the criteria. For example, looking for records that have the same First Name and Last Name and Phone Number returns only results where all three values are the same.

  The minimum required length for the matching values is four characters or more. Values such as "Joe" or "Ace" will be disregarded.

Field Name Comparison Rule Ignored Match Parts

Pick a field that you think has some duplicate values.

Running a very simple match operation like just First and Last Name is okay for giving you an idea of what you have, but it is too broad to use for reliable analysis and deduplication. There may be legitimate duplicate names–different people with the same first and last name. You need additional, unique criteria to narrow it down.

Choosing Unique Identifiers

Matching duplicates requires unique identifiers—data that is unlikely to be shared by any other record unless it is a duplicate. If you don't use unique identifiers, you are likely to identify unrelated records as duplicates and may accidentally merge them.

Many CRMs match first names, last names, and email addresses. If all of those match, or are similar, you can confidently determine that the record is a duplicate.

Other unique identifying fields that are commonly used in deduplication include:

    • Phone number
    • Domain name
    • Mailing address
    • ID number
Step 1: Matching Using Two Different Fields

Sometimes, you might want to match duplicates using data in two separate fields. For example, you might want to compare your Phone Number field to a Mobile Phone Number field to identify duplicates.

Using the Related Fields feature, you can use two different fields (that contain similar data) as matching fields to catch more duplicates.

You can set up Related Fields in the Advanced tab.

bulk-merge_2.png

Common Examples of Related Field Matching

Matching Field Related Fields
Phone Number Mobile Phone Number, Company Phone
Email Domain Website, Company Domain
Email Additional Email Addresses
Address Company Address
Step 1: Commonly Used Matching Fields in Deduplication

The following unique identifying fields, in combination, give a high degree of certainty that the matched records are truly duplicates that should be merged:

  • First Name + Last Name
  • Company Name
  • Email
  • Email Domain
  • Company Website
  • Phone Number
  • ID Numbers
Step 1: Narrowing Down the Records with a Filter 

Use the filter to work with a segment or smaller pool of records. Then Insycle will only analyze the remaining records for duplicates. To add filters, click the Filter button on Step 1, then choose the field to look at, select the condition, and set the value to look for. The filter is applied before the matching step runs. 

man9.png

You may want to use a filter if:

  • You know you only want to work with a subset of your data. In this case, there’s no need to run the operation on your whole database.
  • There are an overwhelming number of duplicate results. Add a filter to work with a reasonably sized subset while you work to get the configuration right. 
  • You want the operation to run faster. A refined segment can speed things up since there are fewer records to analyze.

Most of the options in the Field dropdown match the fields that are found in your CRM, and for Contact records, there are three additional options related to the Email value: 

  • Email Username: The portion of the email address before the “@.” For example, if the email address were “maria@acmewidgets.com,” the username value would be “maria.” 
  • Free Email Provider Domain: Choose True to filter out records where the email domain is Gmail, Hotmail, Yahoo, and about 10,000 other free email providers. This filter helps ensure these are real clients, or can determine which record is the legitimate one because it’s most likely customer companies aren't using free Gmail accounts (though a contact may have accidentally emailed us from it at some point). 
  • Email Top-Level Domain: The top-level domain (TLD) is everything that follows the final dot of a domain name. For example, in the domain name acmewidgets.com', '.com' is the TLD. Some other popular TLDs include '.org', '.uk', and '.edu'. 
Step 1: Matching Using Two Different Fields 

Sometimes, you might want to match duplicates using data in two separate fields. For example, you might want to compare your Phone Number field to a Mobile Phone Number field to identify duplicates.

Using the Related Fields feature, you can use two different fields (that contain similar data) as matching fields to catch more duplicates.

You can set up Related Fields in the Advanced tab.

bulk-merge_2.png

Common Examples of Related Field Matching

Matching Field Related Fields
Phone Number Mobile Phone Number, Company Phone
Email Domain Website, Company Domain
Address Company Address
Step 1: Allowing Empty Values When Matching

When using two or more fields to identify duplicates, records can still be considered matches even if one of the field values is blank. You just need to specify which field(s) allow a blank value.

Under Step 1, configure your matching rules in the Simple tab, then click the Conditions tab.

step-1-conditions-tab-arrow.png

All the matching fields you included will automatically appear with the Value Required in All Records condition selected. Change the condition to Empty Allowed in Any Record to allow empty values for certain fields. You can also use the At Least One Record with Non-Empty condition to help you determine which is the master record. Make sure at least one field remains required and is a reliable unique identifier to ensure the records are really duplicates.

step-1-conditions-empty-not-empty.png

For example, on the Simple tab, you may have the matching fields: First Name, Last Name, and Phone Number. But on some of your records, the Phone Number field may be empty. Using the Empty Allowed in Any Record or At Least One Record with Non-Empty, all records with the same name, same phone number, and no phone number will be considered duplicates.

step-2-group-w-empty.png

Troubleshooting

Insycle isn't finding any duplicates

Most of the time, when Insycle can't find duplicates, it is due to your matching rules in Step 1. It is important to analyze the underlying data to better understand how to set up your rules. A useful exercise can be to set up your matching filters to look for exact matches of just First Name and Last Name

step-1-fname-Lname-only.png

When you click Find, these rules can show you a broad overview of what duplicates are potentially in your database and what fields might be useful to include in your matching fields. These settings are just for discovery and should not be used for a final merge operation; many people can have the same first and last names and are not duplicates.

To get further context, on Step 2, click the layout gear button on the right side of the title bar. Here, you can add any field in your database as a column to the duplicate group review to better understand the data inside these records. 

man11.png

It's taking a long time for Insycle to find duplicates

It can take a while for Insycle to find and match duplicates if the fields being used to identify them have very long values. The longer the values, the longer it takes Insycle to process the data and generate the results. This might come up when looking for matches based on long ID numbers, LinkedIn bio links, or other URLs with long strings attached (ex, https://www.linkedin.com/in/svadin%C3%ADr-n%C4%9Bmec-1234b31a3/).

You can speed this up by limiting how much of the value Insycle looks at.

If the beginning or ending portion of the values are all unique, you can limit the comparison to the first or last several characters using the Match Parts parameter under Step 1

merge-duplicates-linkedin-bio-step-1-match-parts-last-9-chars.png

merge-duplicates-linkedin-bio-step-2-last-9-chars.png

Or use the Ignore Text (Substrings) parameter, then click the Terms button.

merge-duplicates-linkedin-bio-step-1-ignored-text-terms-button.png

On the Ignored Text tab of the popup, add the common portion of the URL or text string.

merge-duplicates-linkedin-bio-step-1-ignored-text-popup.png

For a complete guide to troubleshooting issues with Insycle, please refer to our article on Troubleshooting Issues.

Frequently Asked Questions

Can I merge specific duplicates within a duplicate group?

Yes. You can select individual records within a duplicate group for manual merging. Under Step 2, select the duplicate group. Then, select the records that you would like to merge. 

man12.png

Can I view more fields to help me analyze my duplicates?

To help you analyze and determine which records are the right ones to merge, you can change the fields that show up in this preview under Step 2 by clicking on the gear button to alter the layout. 

man13.png

Additional Resources

Related Help Articles

Related Blog Posts