Mailchimp Merge Duplicates Overview

Insycle supports merging the following Mailchimp record types:

• Contacts

Step 1: Configure Rules to Identify Duplicates

In the Data Management > Merge Duplicates module, pick the record type, and explore the default templates for an existing solution similar to what you need.

Each row in Step 1 is for a field you want to look at for duplicates, along with some parameters on what to look for. You want to choose fields that, in combination, give a high degree of certainty that the matched records are duplicate records. 

If you'd like to look at the data in two different fields (that contain similar data) as if it were one, you can set up Related Fields under the Advanced tab. 

The Conditions tab provides rules that one or more of the records in a duplicate group must meet. These options let you choose fields that are required, can be empty, or specify values that must be included.

Step 2: Analyze the Identified Duplicates

When two or more records represent the same entity (person, company, or other) based on your matching rules, they are clustered together into duplicate groups. Each duplicate group shows the total number of records that were identified as duplicates based on your settings. For example, if you had four records for the same person, it would count as one duplicate group with four duplicate records.

Step 3: Choose Whether to Merge in Bulk or Manually

The most efficient and sustainable way to merge duplicates is in Bulk mode. This allows you to set rules for determining the master record automatically across all records in your database. You'll be able to use saved templates and recipes to repeat the process on a regular basis. 

In Manual mode, you have complete control over which records are merged together by selecting them from the Record Viewer. Manual mode should be reserved only for cases where you need a careful, controlled process. Learn more about merging duplicates in Manual mode.

Step 4: Set Rules for Master Record Selection and Data Retention

After selecting Bulk mode in Step 3, you need to define how all of the matching duplicate groups should be merged at scale in Step 4.

Configure Rules to Automatically Select the Master Record

On the Master tab of Step 4, you define how the duplicate groups should be merged at scale by creating rules that tell Insycle how to select the record from each group to become the master. The master is the record that will remain after the merge.

For example, if you had four records representing the same contact, they would all be merged into one master record. The other three records would not exist anymore.

Here, you also select the processing method—By Priority or Absolute. Most de-duplication operations should use By Priority. Learn more about these options in the Bulk Merge Duplicate People, Companies article.

Configure Rules That Determine Values to Keep

Duplicates may be exact match versions of another record, but often there is only partial data overlap between them. When data is split between two records, both may contain unique and important information you want to keep.

Under Step 4, click the Fields tab. For each field you want to control the data retention for, you need to tell Insycle which record the data should be taken from. This is merged into the master. Any data that is not in the master or not copied to the master is removed.

Learn more about configuring data retention and master record selection.

Preview Merged Changes in CSV Report

Now, with the filters and master record set up, you can preview the changes you are making to your data in a CSV. That way, you can check to ensure your deduplication configuration is working as expected before those changes are pushed to your live database.

Under Step 5, click the Review button and select Preview mode.

On the Notify tab, select recipients and add context to the report email.

On the When tab, click the Run Now tab, and select which records to apply the change to (you could do All, but if you have a large number of records, you may just want to do a chunk for your preview), then click the Run Now button.

Insycle will generate a preview CSV and send it to your email. Open the CSV file from your email in a spreadsheet application.

The Duplicate Group ID indicates which records will be merged together. 

The Status column indicates:

• Duplicate – The record is part of a duplicate group.
• Master – The master record chosen for the duplicate group based on default behavior and your Master rules. Review the selections in this row to determine whether the appropriate records are being chosen.
• Master (After) – For each duplicate group, a special row –Master (After)– will appear in the CSV report. The Master (After) row shows the values the final record will contain based on your Field rules and the default behavior. 
• Error – If Insycle is not able to determine which record would be the master, an error message will appear here. See the Troubleshooting section below for more details.

If everything looks good, return to Insycle and move forward with applying the changes.

Apply Changes to Your Mailchimp Records

When you're satisfied with the results in your preview, you can apply the merge changes to Mailchimp.

Under Step 5, click the Review button again, and this time select Update mode.

On the When tab, you should use Run Now the first time you apply these changes to Mailchimp. If you have a large number of records, you may want to do a smaller batch to review the results in Mailchimp.

After you've seen the results in Mailchimp and you are satisfied with how the operation runs, you can save your configuration as a template and set up automation so this merge operation runs on a set schedule. If you have several templates you'd like to run together automatically, you can create a Recipe.

Return to the Template menu at the top of the page and click Copy to save your configurations as a new version of the template you started with. Then click the pencil to edit your new template name.

Under Step 5, click the Review button, and select Update mode.

On the Notify tab, select the send option appropriate for your automation: Always send, Send when errors, or Do not email.

Add any additional recipients who should receive the CSV (hitting Enter after each address) and provide context in the message subject or body.

On the When tab, select Automate, and configure the frequency you'd like the template to run. When finished, click Schedule.

You can view all your scheduled automations at any time on the Operations > Automations page.

When you have a set of templates that reliably merge your records, you can put them together into a longer, ordered sequence as a Recipe. Then, you can schedule that Recipe to run on a consistent, set schedule. Your templates will run one after another in the order that you set.

With the Activity Tracker, you have a complete audit trail and history of changes made through Insycle, including processes run in Preview mode or data syncs. At any time, you can download a CSV report that shows all the changes made in a given operation run.

Navigate to Operations > Activity Tracker, search by module, app, or template name, then click the Run ID for the operation.

If you've configured the Insycle Run ID property in Mailchimp, you can cross-reference these operations in your CRM records.

When merging duplicates, Insycle goes through the following steps for all Mailchimp record types:

1. Select the master record according to the rules set in Step 4 on the Master tab.
2. Determine field values to retain in the master record based on rules configured in Step 4 on the Fields tab.

3. For fields without specific rules, identify non-empty values using a "fill in the blanks" method:

   • Check whether the master record has any empty fields.

   • If empty, copy values from the most recently updated record in the duplicate group.

4. Update the master record by:

   • Applying the field values chosen in step 2 (from Field rules).

   • Adding the non-empty values identified in step 3 (using "fill in the blanks").

   • Preserving all other existing values in the master record.

5. Merge the duplicate records and reassign all related objects to the master record.
6. Store an audit trail report accessible from the Activity Tracker and send an email notification to the designated recipients.

If you're uncertain about how conflicting field values will merge, you can add those fields to the Preview CSV report by including them in the Layout via the gear icon in Step 2.

This is also useful if there are fields you're worried about backing up; including them in the CSV report allows you to refer back to the values in the CSV sent to your email or downloaded from the Activity Tracker.

Companies generally want to avoid having duplicate contacts in multiple audiences to prevent contacting them repeatedly or missing valuable context in their communications. do have.

Unfortunately, deduplicating across audiences cannot be handled from within Mailchimp. However, Insycle allows you to deduplicate across Mailchimp audiences by default.

To learn more, see Cross-Audience Deduplication in Mailchimp.

Each row in your matching fields setup is cumulative, so records must meet all of the criteria. For example, looking for records that have the same First Name, AND Last Name, AND Phone Number returns only results where all three values are the same.

To match against one field value OR another, you will need to run two different templates. For example, if you want to use fields like Phone Number OR Mobile Phone Number, you’ll run one template for Phone Number, then a second configured the same except with the Mobile Phone Number field.

The searched value must have four or more characters. For example, values of “Joe” will be ignored.

At the bottom of the Master tab under Step 4, select the processing method that will be applied to your master selection rules.

By Priority: Look through the master selection rules in order, one by one. As soon as a record meets one of the criteria, Insycle makes the master selection and skips the rest of the rules on the list. The vast majority of duplicate templates should use By Priority.

Absolute: The master record must meet all of the listed master selection rules. If a record does not match every rule listed, no master record will be identified. Absolute is appropriate for less flexible master selection.

For example, if a company wants to ensure the chosen master record is in their sales pipeline and already has a sales rep working the record, they can choose Absolute and set the Master rules:

• Lifecycle Stage is lead
• Contact Owner active user

Choosing Absolute can often result in no master record being identified since the record has to match every rule listed, so in most cases, you should select By Priority.

Let's say that we have four records that represent the same person—Marta Vaskovitch. The Merge Duplicates module will identify this as one duplicate group that consists of four records.

Here is the data that we have for this duplicate group (viewed here in the Grid Edit module):

Here are the master selection rules we have set up:

We haven't sent any emails to Marta yet, so when Insycle processes the first three rules—Marketing emails clicked, emails bounced, and emails opened—Insycle cannot eliminate any record because they all have the same value of zero.

None of the record email addresses use a work domain, so there isn't a single choice for the master using the Email rule either.

In the next rule about contact owner, records 61301, 61201, and 61251 are eliminated since no contact owner exists for those records. Now there is only one record remaining, 61351, and therefore that's the master record.

The Merge Duplicates module allows you to control the values saved in the master record after the merge regardless of the default merge behavior. By adding each field you want to control the data retention for and selecting a Condition, you can tell Insycle where the data for the field should be taken from and how to handle it.

The master record can use values from several different records from the duplicate group, based on the rules that you set in the Fields tab under Step 4. 

By default, any fields not specified here will use the master record values. However, if the master field is blank, the value from the most recently updated duplicate will be used.

The Criteria dropdown gives you various options for choosing the data to keep, and the Group Fields lets you keep values of multiple fields from the same record:

• From record where value – Select data from one of the records in the duplicate group based on the values. These options vary depending on the field type. For example, retain the data in the Email field that is using a professional domain rather than a free one. 
  HubSpot and Salesforce users can select an Owner field, the “From record where value” Criteria, and the Condition “active user.”
• Most frequent value* – If the same value appears in multiple records, use the one that appears most frequently.
• From record based on other field value – Look at the value in a different field to decide which value from the duplicate group should be kept. The example above highlights how a Last Modified Date value can be used to determine which Lifecycle Stage value to use.
• From master record – Use the value that exists in the master record.
• From master record (even empty) – If the field on the selected master record is blank, keep it that way. Don’t automatically fill it in with a value from the most recently updated record.
• Combine and append all values* – You can merge the values from the selected field for all records in the group. For example, if there is some type of Notes field, you could keep the notes from all of the records in the duplicate group.
• Rollup numbers* – On number fields, you can add the values from all records (sum), calculate the average, or keep only the largest (max) or smallest (min) values.
• Collect non-master values from other field* – Aggregate the values of all the duplicates that are not the master and not the same as the master, meaning all instances of that value are excluded from collection. This can be especially helpful if you want a record of the object IDs that were removed, so you can also remove them from another system. Select a destination field to copy and combine values into, then select what field the data should come from.
• Collect all values from other field* – Select a destination field to copy and combine values into, then select what field the data should come from. For example, this could be used to keep the record Owner values of all duplicates and combine them into a custom field.

* Indicates Criteria options that do not allow you to Group Fields.

For situations where there are no common rules you can apply for identifying duplicates for all or some of the records, you may need more granular control for picking records to include or exclude from the process.

To do this in bulk, you can create a CSV and use the Magical Import module.

In these cases, you can use CSV files to customize your bulk merging, designate master records, and exclude records from deduplication. Then you can import the CSV from the Magical Import, and use the Merge Duplicates module for complete control over the final merge operation. Learn how to customize merging Duplicates in bulk using a CSV.

To do this one record at a time, you can use Manual mode of the Merge Duplicates module.

In Manual mode, you have complete control over which records are merged together by selecting them from the Record Viewer. Manual mode should be reserved only for cases needing a careful, controlled process. Learn more about merging duplicates in Manual mode.

You have duplicate records that have been identified by Insycle but not all of them are merging into the master. Check to see how many duplicates are in the affected duplicate groups. If you have duplicate groups that contain more than five records, you may want to change the value in Skip duplicate groups with more than 5 records per group to make sure you can get them all.

This setting is intended to protect against the accidental merging of non-duplicate records if the filter in Step 1 is too broad.

If the Message column of the CSV report displays this text:

Change rules in Step 4 'Master Selection'. Failed to pick master record because multiple records (2) meet the selection criteria. In 'Master Selection', change, add, or reorder the rules such that only one record matches (if cannot determine master based on field values, use 'Record ID is lowest' as the last rule).

This error means that based on the master rules you set, Insycle could not figure out which would be the master.

There are a couple of things you can try to resolve this:

1. Under Step 4, on the Master tab, experiment with reordering or adding additional fields that are likely to have unique values.
2. Also on the Step 4, Master tab, check to ensure that you have By Priority selected and not Absolute.

   
   With By Priority, your master record only has to match one rule. Using Absolute, your master record would have to meet all of the rule criteria. The majority of the time, it is best to select By Priority.

   If By Priority was used, this indicates none of the records in the duplicate group meet any of the criteria on the list more than the others. In this case, you'll need to experiment on the Master tab, reordering or adding additional rules for fields likely to have unique values.

There are a couple of things to look at that may be misidentifying records as duplicates.

First, you may need a better unique identifier. Under Step 1, if you only use fields that could correctly contain the same values in multiple records, these aren't unique identifiers. In this case, you are likely to identify unrelated records as duplicates and may accidentally merge them.

Unique identifiers are data that is unlikely to be shared by any other record unless it represents the same underlying entity. Fields that are commonly used in deduplication include phone numbers, email, mailing addresses, or ID numbers.

Second, this may indicate the Comparison Rule under Step 1 is too broad. Try using the Exact Match comparison rule instead of Similar Match. Similar Match looks for values that may be close but with a one-character difference (maybe a typo) which broadens the search. 

Remember, always run your deduplication in Preview Mode to confirm things are working as expected before running them in Update Mode and applying the changes to your Mailchimp records.

Most of the time when Insycle can't find duplicates, it is due to your matching rules in Step 1. To better understand how to set up your rules, it is important to analyze the underlying data. A useful exercise can be to set up your matching filters to look for exact matches of just First Name and Last Name. 

When you click the Find button, these rules can show you a broad overview of what duplicates are potentially in your database, and what fields might be useful to include in your matching fields. These settings are just for discovery and should not be used for a final merge operation; many people can have the same first and last names and are not duplicates. 

To get further context, in Step 2, click the gear icon on the right side of the title bar. Here, you can add any field in your database as a column to the Record Viewer to better understand the data inside of these records. These will also appear in your CSV report.

It can take a while for Insycle to find and match duplicates if the fields being used to identify them have very long values. The longer the values, the longer it takes Insycle to process the data and generate the results. This might come up when looking for matches based on long ID numbers, LinkedIn bio links, or other URLs with long strings attached (ex, https://www.linkedin.com/in/svadin%C3%ADr-n%C4%9Bmec-1234b31a3/).

You can speed this up by limiting how much of the value Insycle looks at.

If the beginning or ending portion of the values are all unique, you can limit the comparison to the first or last several characters using the Match Parts parameter under Step 1. 

Or use the Ignore Text (Substrings) parameter, then click the Terms button.

On the Ignored Text tab of the popup, add the common portion of the URL or text string.

Yes, Insycle does cross-audience deduplication in Mailchimp by default. See the Cross-Audience Deduplication in Mailchimp article to learn more.

Currently, there are two ways to make sure that the records that you are merging are indeed duplicate records.

First, always run your deduplication templates in Preview Mode before running them in Update Mode. This produces a CSV that shows you how your records would have been merged. Then you can ensure that your Merge Duplicates template is working as expected and not merging non-duplicate records together.

Additionally, you can reduce the risk when merging duplicates by narrowing your duplicate matching settings in Step 1. Try the Exact Match Comparison Rule instead of Similar Match. Then make sure that you are using actual uniquely identifying fields—first name, last name, email, and phone number are popular choices. The more tightly defined your filter is, the less likely you are to merge non-duplicate records.

If the Message column of the CSV report displays this text:

Change rules in Step 4 'Master Selection'. Failed to pick master record because multiple records (X) meet the selection criteria. In 'Master Selection', change, add, or reorder the rules such that only one record matches (if cannot determine master based on field values, use 'Record ID is lowest' as the last rule).

This means that based on all the rules, Insycle could not figure out which record in the duplicate group would be the master. None of the records meet more of the rules than others.

There are a couple of things you can try to resolve this:

1. Under Step 4, on the Master tab, experiment with reordering or adding additional fields that are likely to have unique values.
2. At the bottom of the Master tab under Step 4, check to ensure that you have By Priority selected and not Absolute.
   
   With By Priority, your master record only has to match one rule. Using Absolute, your master record would have to meet all of the rule criteria. The majority of the time, it is best to select By Priority.
   If By Priority was used, then none of the records in the duplicate group meet any of the criteria on the list more than the others. In this case, you'll need to experiment, reordering or adding additional rules for fields likely to have unique values.

Yes. You can use a customized list of duplicates and use the Magical Import module to tag duplicates in Mailchimp, then use the Merge Duplicates module to deduplicate in bulk. Include ID numbers from Mailchimp in your CSV.

Yes, Insycle allows you to select which field data is retained in the master record using the Fields tab under Step 4. See the Step 4: Control What Field Data is Retained section above for more details.

Yes, there are several ways to share details and get approval before merging duplicates.

You can manually approve master records and mark them in a CSV, then use Insycle to bulk deduplicate down to those master records. Consult with this Customize Bulk Deduplication Using Exclusions and Pre-Defined Masters article to learn more.

Or, you can run the Merge Duplicates module in Preview Mode, then deliver the preview CSV that Insycle generates. The CSV report that Insycle generates includes your entire merge operation down to individual duplicate groups but does not update your live data. Then your team can approve the merge based on this report, before running Merge Duplicates in Update Mode.

Additionally, team members can review duplicates and manually select the master for each record under Step 4. Review the Manually Merge Duplicates article for more detail.

No. The Similar Match Comparison Rule found in Step 1 looks for values that may be close but with a one-character difference (maybe a typo) and broadens the search. This search behaves like when Google shows results for a slightly different term or says, “Did you mean...” For example, if a Company Name of “Acme” is found, it could include records with the Company Name values “Akme, acm, Acma,” etc., as a match.

You should be careful when using Similar Match as the looser criteria can incorrectly identify non-duplicates as duplicates. 

Review the Similar Matching best practices for more details.

Insycle shows 50 records on the module screen as a preview, this isn't the entire list of records. Include All records when you view the Preview CSV report to see everything.

Insycle can process thousands of duplicate groups in one operation. Potentially, you could deduplicate your entire database in one operation. 

You can merge up to 100 duplicate records into a single master record. 

If you have duplicate groups that contain more than five records, you may want to change the value in Skip duplicate groups with more than 5 records per group under Step 3 to make sure you can get them all.

This is a precaution to ensure that if you use a duplicate matching filter that is too broad in Step 1, you do not accidentally merge many non-duplicate records together. If you are going to set this number at a high level, it is a good idea to run Preview Mode first to make sure your deduplication template is operating as you intend.

All plans include unlimited usage, unlimited users, and unlimited operations. See the pricing page for more details. During the free trial, there is a cap of 500 records updated, cleansed, or merged.