Mailchimp Merge Duplicates Overview

Each row in your matching fields setup is cumulative, so records must meet all of the criteria. For example, looking for records that have the same First Name, AND Last Name, AND Phone Number returns only results where all three values are the same.

To match against one field value OR another, you will need to run two different templates. For example, if you want to use fields like Phone Number OR Mobile Phone Number, you’ll run one template for Phone Number, then a second configured the same except with the Mobile Phone Number field.

The searched value must have four or more characters. For example, values of “Joe” will be ignored.

Sometimes, you might want to match duplicates using data in two separate fields. For example, you might want to compare your Phone Number field to a Mobile Phone Number field to identify duplicates.

Using the Related Fields feature, you can use two different fields (that contain similar data) as matching fields to catch more duplicates.

You can set up Related Fields in the Advanced tab.

Common Examples of Related Field Matching

Matching Field	Related Fields
Phone Number	Mobile Phone Number, Company Phone
Email Domain	Website, Company Domain
Address	Company Address

When using two or more fields to identify duplicates, records can still be considered matches even if one of the field values is blank. You just need to specify which field(s) allow a blank value.

Under Step 1, configure your matching rules in the Simple tab, then click the Conditions tab.

All the matching fields you included will automatically appear with the Value Required in All Records condition selected. Change the condition to Empty Allowed in Any Record to allow empty values for certain fields. You can also use the At Least One Record with Non-Empty condition to help you determine which is the master record. Make sure at least one field remains required and is a reliable unique identifier to ensure the records are really duplicates.

For example, on the Simple tab, you may have the matching fields: First Name, Last Name, and Phone Number. But on some of your records, the Phone Number field may be empty. Using the conditions "Empty Allowed in Any Record," or "At Least One Record with Non-Empty," all records with the same name, same phone number, and no phone number will be considered duplicates.

Priority Match: Looks through the master selection rules in order, one by one. As soon as a record meets one of the criteria, Insycle makes the master selection and skips the rest of the rules on the list. The vast majority of duplicate templates should use Priority Match.

Absolute Match: The master record must meet all of the listed rules in the Record tab in Step 4. If a record does not match every rule listed, no master record will be identified. Absolute Match is appropriate for less flexible master selection.

For example, if a company wanted to ensure the chosen master record is in their sales pipeline and already has a sales rep working the record, they can choose Absolute Match and set the Record rules:

• Lifecycle Stage is lead
• Contact Owner exists

Choosing Absolute Match can often result in no master record being identified since the record has to match every rule listed, so in most cases, you should select Priority Match.

Let's say that we have four records that represent the same person—Marta Vaskovitch. The Merge Duplicates module will identify this as one duplicate group that consists of four records.

Here is the data that we have for this duplicate group:

And here are the master selection rules we have set up:

We haven't sent any emails to Marta yet, so when Insycle processes the first three rules—Marketing emails clicked, emails bounced, and emails opened—Insycle cannot eliminate any record because they all have the same value of zero.

In the next rule about contact owner, records 61301, 61201, and 61251 are eliminated since no contact owner exists for those records. Now there is only one record remaining, 61351, and therefore that's the master record.

The Merge Duplicates module allows you to control the values saved in the master record after the merge regardless of the default merge behavior. By adding each field you want to control the data retention for and selecting a Condition, you can tell Insycle where the data for the field should be taken from and how to handle it.

The master record can use values from several different records from the duplicate group, based on the rules that you set in the Fields tab under Step 4. 

By default, any fields not specified here will use the master record values. However, if the master field is blank, the value from the most recently updated duplicate will be used.

In this first example, the Ownership value from the record with the most recent Modified Date will be kept, and all the Account Owner values from the records in the duplicate group will be saved to the Merged Owners custom field.

In this example, the most recent interaction data for several fields will be used in the merged record.

For situations where there are no common rules you can apply for identifying duplicates for all or some of the records, you may need more granular control for picking records to include or exclude from the process. You can customize bulk deduplication using exclusions and pre-defined masters via a CSV file.

You have duplicate records that have been identified by Insycle but not all of them are merging into the master. Check to see how many duplicates are in the affected duplicate groups. If you have duplicate groups that contain more than five records, you may want to change the value in Skip duplicate groups with more than 5 records per group to make sure you can get them all.

This setting is intended to protect against the accidental merging of non-duplicate records if the filter in Step 1 is too broad.

If the Result column of the CSV report displays this error:

Cannot determine master record because multiple records (#) satisfy the master selection rules. In ‘Master Selection’, change/add/reorder the rules such that only one record satisfies them (if cannot determine master based on field values, use ‘ID is lowest’ as the last rule).

This error means that based on the master rules you set, Insycle could not figure out which would be the master.

Check Step 4 to ensure that you have Priority Match selected and not Absolute Match.

With Priority Match, the rules configured in the Records tab of Step 4 are processed in order and your master record only has to match one rule. Using Absolute Match, your master record would have to meet all of the rule criteria. The majority of the time it is best to select Priority Match.

If Priority Match was used, then none of the records meet any of the criteria on the list more than the others. In this case, you'll need to experiment with Step 4, reordering or adding additional rules for fields likely to have unique values.

There are a couple of things to look at that may be misidentifying records as duplicates.

First, you may need a better unique identifier. Under Step 1, if you only use fields that could correctly contain the same values in multiple records, these aren't unique identifiers. In this case, you are likely to identify unrelated records as duplicates and may accidentally merge them.

Unique identifiers are data that is unlikely to be shared by any other record unless it represents the same underlying entity. Fields that are commonly used in deduplication include phone numbers, email, mailing addresses, or ID numbers.

Second, this may indicate the Comparison Rule under Step 1 is too broad. Try using the Exact Match comparison rule instead of Similar Match. Similar Match looks for values that may be close but with a one-character difference (maybe a typo) which broadens the search. 

Remember, always run your deduplication in Preview Mode to confirm things are working as expected before running them in Update Mode and applying the changes to your CRM records.

Most of the time when Insycle can't find duplicates, it is due to your matching rules in Step 1. To better understand how to set up your rules, it is important to analyze the underlying data. A useful exercise can be to set up your matching filters to look for exact matches of just First Name and Last Name. 

When you click the Find button, these rules can show you a broad overview of what duplicates are potentially in your database, and what fields might be useful to include in your matching fields. These settings are just for discovery and should not be used for a final merge operation; many people can have the same first and last names and are not duplicates. 

To get further context, click the gear button on the right side of the Record Viewer pane. Here, you can add any field in your database as a column to the Record Viewer to better understand the data inside of these records. 

If the Result column of the CSV displays an error, read the error text for help figuring out how to resolve the problem.

The most common error is:

Cannot determine master record because multiple records (#) satisfy the master selection rules. In ‘Master Selection’, change/add/reorder the rules such that only one record satisfies them (if cannot determine master based on field values, use ‘ID is lowest’ as the last rule).

This means that based on all the rules, Insycle could not figure out which would be the master. None of the records meet more of the rules than others. In this case, you'll need to experiment with reordering or adding additional fields likely to have unique values under Step 4.

It can take a while for Insycle to find and match duplicates if the fields being used to identify them have very long values. The longer the values, the longer it takes Insycle to process the data and generate the results. This might come up when looking for matches based on long ID numbers, LinkedIn bio links, or other URLs with long strings attached (ex, https://www.linkedin.com/in/svadin%C3%ADr-n%C4%9Bmec-1234b31a3/).

You can speed this up by limiting how much of the value Insycle looks at.

If the beginning or ending portion of the values are all unique, you can limit the comparison to the first or last several characters using the Match Parts parameter under Step 1. 

Or use the Ignore Text (Substrings) parameter, then click the Terms button.

On the Ignored Text tab of the popup, add the common portion of the URL or text string.

Yes, Insycle does cross-audience deduplication in Mailchimp by default. See the Cross-Audience Deduplication in Mailchimp article to learn more.

When using two or more fields to identify duplicates, records can still be considered matches even if one of the field values is blank. You just need to specify which field(s) allow a blank value.

Under Step 1, configure your matching rules in the Simple tab, then click the Conditions tab.

All the matching fields you included will automatically appear with the Value Required in All Records condition selected. Change the condition to Empty Allowed in Any Record to allow empty values for certain fields. You can also use the At Least One Record with Non-Empty condition to help you determine which is the master record. Make sure at least one field remains required and is a reliable unique identifier to ensure the records are really duplicates.

For example, on the Simple tab, you may have the matching fields: First Name, Last Name, and Phone Number. But on some of your records, the Phone Number field may be empty. Using the Conditions "Empty Allowed in Any Record," or "At Least One Record with Non-Empty," all records with the same name, same phone number, and no phone number will be considered duplicates.

Yes. This can be done, for example, if you want to look at both the Phone Number field values and Mobile Phone Number field values as a single pool of values to compare between records and identify duplicates.

Using the Related Fields feature, you can use two different fields (that contain similar data) as matching fields to catch more duplicates. You can set up Related Fields in the Advanced tab.

Currently, there are two ways to make sure that the records that you are merging are indeed duplicate records.

First, always run your deduplication templates in Preview Mode before running them in Update Mode. This produces a CSV that shows you how your records would have been merged. Then you can ensure that your Merge Duplicates template is working as expected and not merging non-duplicate records together.

Additionally, you can reduce the risk when merging duplicates by narrowing your duplicate matching settings in Step 1. Try the Exact Match Comparison Rule instead of Similar Match. Then make sure that you are using actual uniquely identifying fields—first name, last name, email, and phone number are popular choices. The more tightly defined your filter is, the less likely you are to merge non-duplicate records.

If the Result column of the CSV report displays this error:

Cannot determine master record because multiple records (#) satisfy the master selection rules. In ‘Master Selection’, change/add/reorder the rules such that only one record satisfies them (if cannot determine master based on field values, use ‘ID is lowest’ as the last rule).

This error means that based on the master rules you set, Insycle could not figure out which would be the master.

Check Step 4 to ensure that you have Priority Match selected and not Absolute Match.

With Priority Match, the rules configured in the Records tab of Step 4 are processed in order and your master record only has to match one rule. Using Absolute Match, your master record would have to meet all of the rule criteria. The majority of the time it is best to select Priority Match.

If Priority Match was used, then none of the records meet any of the criteria on the list more than the others. In this case, you'll need to experiment with Step 4, reordering or adding additional rules for fields likely to have unique values.

Yes. You can use a customized list of duplicates and use the Magical Import module to tag duplicates in your CRM, then use the Merge Duplicates module to deduplicate in bulk. Include ID numbers from your CRM in your CSV.

Yes, Insycle allows you to select which field data is retained in the master record using the Fields tab under Step 4. See the Bulk Merge Duplicate People, Companies article for more detail.

Yes. You can exclude records from deduplication by including a "Deduplication Exclude" field in your CSV, as detailed in the Customize Bulk Deduplication Using Exclusions and Pre-Defined Masters article.

Yes, there are several ways to share details and get approval before merging duplicates.

You can manually approve master records and mark them in a CSV, then use Insycle to bulk deduplicate down to those master records. Consult with this Customize Bulk Deduplication Using Exclusions and Pre-Defined Masters article to learn more.

Or, you can run the Merge Duplicates module in Preview Mode, then deliver the preview CSV that Insycle generates. The CSV report that Insycle generates includes your entire merge operation down to individual duplicate groups but does not update your live data. Then your team can approve the merge based on this report, before running Merge Duplicates in Update Mode.

Additionally, team members can review duplicates and manually select the master for each record under Step 4. Review the Manually Merge Duplicates article for more detail.

No. The Similar Match Comparison Rule found in Step 1 looks for values that may be close but with a one-character difference (maybe a typo) and broadens the search. This search behaves like when Google shows results for a slightly different term, or says “Did you mean...” For example, if a Company Name of, “Acme” is found, it could include records with the Company Name values “Akme, acm, Acma,” etc., as a match.

You should be careful when using Similar Match as the looser criteria can incorrectly identify non-duplicates as duplicates. 

Review the Similar Matching best practices for more detail.

Insycle shows 50 records on the module screen as a preview, this isn't the entire list of records. Include All records when you view the Preview CSV report to see everything.

Insycle can process thousands of duplicate groups in one operation. Potentially, you could deduplicate your entire database in one operation. 

You can merge up to 100 duplicate records into a single master record. 

If you have duplicate groups that contain more than five records, you may want to change the value in Skip duplicate groups with more than 5 records per group under Step 3 to make sure you can get them all.

This is a precaution to ensure that if you use a duplicate matching filter that is too broad in Step 1, you do not accidentally merge many non-duplicate records together. If you are going to set this number at a high level, it is a good idea to run Preview Mode first to make sure your deduplication template is operating as you intend.

All plans include unlimited usage, unlimited users, and unlimited operations. See the pricing page for more details. During the free trial, there is a cap of 500 records updated, cleansed, or merged.