Cross-Audience Deduplication in Mailchimp

Deduplicate leads and contacts

How to Merge Duplicates Across Audiences for Streamlined Email Marketing

Email marketing effectiveness relies on delivering tailored messages to the right audiences. However, when Mailchimp contacts are spread across multiple audiences, duplication can occur, leading to redundant messages or missing context from previous communications. This can negatively impact complex customer journeys involving different business aspects or sales funnel stages.

Insycle's Merge Duplicates module solves this problem by allowing you to deduplicate within a single audience or across Mailchimp audiences effortlessly. It offers flexible deduplication using any data field, not just email addresses, and automatically deduplicates across audiences by default unless you specify to work within a specific audience. This streamlines your email marketing efforts, ensuring contacts receive relevant communications without redundancies or missing context.

Process Summary

  1. Set rules to identify duplicates.
  2. Review and analyze the identified duplicates.
  3. Choose Bulk Mode.
  4. Set rules to select the master record and determine what field data is retained.
  5. Deduplicate your records.

 

Step-by-Step Instructions

Step 1: Configure Rules to Identify Duplicates

Navigate to Data Management > Merge Duplicates, pick the record type, and explore the default templates for a pre-built solution.

Insycle offers a flexible way to identify duplicates using any data field, not just the email address. Step 1 looks through the records in your database, examining the fields that you specify for a match.

Click the + Field button to add fields you want to look at for duplicates, along with some parameters on what to look for. You want to choose fields that, in combination, give a high degree of certainty that the matched records are duplicates. See the Advanced How-Tos for more details.

For example, you can try and identify duplicate contacts using the First Name, and Last Name, and Email Address fields. Contacts will need to match all three criteria to be considered duplicates. 

This will identify duplicates across audiences while ignoring the subdomain and top-level domain. For example, it matches jane@acme.com with jane@acme.co.uk and jane@acme.io.

merge-duplicates-mailchimp-contacts-step-1-name-email.png

To look for duplicates in a single audience, include the List ID (Audience ID in Mailchimp) as a rule in Step 1.

You can use the template, Same email, ignoring top-level domain (within same audience) as a starting point.

merge-duplicates-mailchimp-contacts-step-1-email-list-ID-646x378.png

Step 2: Analyze the Identified Duplicates

When two or more records represent the same contact based on your matching rules, they are clustered together into duplicate groups. Each duplicate group shows the total number of records that were identified as duplicates based on your settings. For example, if you had four records for the same contact, it would count as one duplicate group with four duplicate records.

Under Step 2, each row represents a duplicate group, with the number indicating how many duplicate records were found. When you click a row, it expands and shows the duplicate records in the group.

merge-duplicates-mailchimp-contacts-step-2-name-email.png

Step 3: Choose Whether to Merge in Bulk or Manually

The most efficient and sustainable way to merge duplicates is in Bulk mode. This allows you to set rules for determining the master record automatically across all records in your database. You'll be able to use saved templates and recipes to repeat the process on a regular basis. 

In Manual mode, you have complete control over which records are merged together by selecting them from the Record Viewer. Manual mode should be reserved only for cases where you need a careful, controlled process. Learn more about merging duplicates in Manual mode.

dup3.png

Step 4: Set Rules for Master Record Selection and Data Retention

After selecting Bulk mode in Step 3, define how all of the matching duplicate groups should be merged at scale in Step 4.

Configure Rules to Automatically Select the Master Record

On the Master tab, you define how the duplicate groups should be merged at scale by creating rules that tell Insycle how to select the record from each group to become the master. The master is the record that will remain after the merge.

For example, if you had four records representing the same contact, they would make up one duplicate group with four records, all of which would be merged into one master record. The other three records would no longer exist.

For each duplicate group, Insycle will look at each Master rule to see which of the records in the group meet the criteria. Rules are read in order, from top to bottom, and as soon as a record is the only one to meet a rule, it is selected as the master record. The subsequent rules are ignored. 

Select the processing method—By Priority or Absolute. Most de-duplication operations should use By Priority. Learn more about these options in the Advanced How-Tos below.

merge-duplicates-intercom-users-step-4-master-rules.png

If none of the records in the duplicate group match any of the rules, the automatic master selection for the group fails. See the Troubleshooting section below for more details.

Configure Rules That Determine Values to Keep

Duplicates may be exact match versions of another record, but often, there is only partial data overlap between them. When data is split between two different records, both may contain unique and important information you'd like to keep.

Under Step 4, click the Fields tab. For each field you want to control the data retention for, you need to select a Field and tell it where the data for the field should be taken from. This is merged into the master. Any data that is not in the master or not copied to the master is removed.

merge-duplicates-intercom-step-4-field-rules.png

What happens to fields if I don’t create custom rules?

There is no need to create rules for every field in your CRM - Insycle automatically handles fields without specific rules using a "fill in the blanks" approach. When the master record has empty fields, Insycle copies values from the most recently updated record in the duplicate group where that data exists. For example, if the master record's Industry field is empty but another record in the duplicate group has an Industry value, that value will automatically be copied to the master record. This means you only need to create custom retention rules for the handful of fields that require special handling rather than setting up rules for all your fields.

Learn more about configuring data retention and master record selection.

Step 5: Preview Report and Update CRM

Preview Merged Changes in CSV Report

Now with the filters and master record set up, you can preview the changes you are making to your data. That way, you can check to ensure your deduplication configuration is working as expected before those changes are pushed to your live database. The CSV report that Insycle generates includes records from all the duplicate groups.

Under Step 5, click the Review button and select Preview mode.

merge-duplicates-step-5-review-preview-mode-your-CRM.png

Click the Next button to go to the Notify screen, where you can select recipients for the email report and add additional context.

On the When tab, click the Run Now tab, and select which records to apply the change to (in most cases this will be All), then click the Run Now button.

dup8.png

Insycle will generate a preview CSV and send it to your email. Open the CSV file from your email in a spreadsheet application and review the data.

merge-duplicates-mailchimp-contacts-csv.png

The Duplicate Group ID indicates which records will be merged together.

The Status column indicates:

  • Duplicate – The record is part of a duplicate group.
  • Master The master record is chosen for the duplicate group based on default behavior and your Master rules. Review the selections in this row to determine whether the appropriate records are being chosen.
  • Master (After) – For each duplicate group, a special row –Master (After)– will appear in the CSV report. The Master (After) row shows the values the final record will contain based on your Field rules and the default behavior.
  • Error – If Insycle is not able to determine which record would be the master, an error message will appear here. See the Troubleshooting section below for more details.

If everything looks good, return to Insycle and move forward with applying the changes.

Save Templates and Setup Automation

After you've seen the results in your CRM and are satisfied with how the operation runs, you can save your configuration as a template, and set up automation so this merge operation runs on a set schedule. If you have several templates you'd like to run together automatically, you can create a Recipe.

By automating with a template, you'll save time and ensure that your duplicates are merging consistently on an ongoing basis.

Return to the Template menu at the top of the page and click Copy to save your configurations as a new version of whatever template you started with. Then click the pencil to edit your new template name.

save-template-copy-and-rename.png

Under Step 5, click the Review button, and select Update mode.

On the Notify tab, select the send option appropriate for your automation: Always send, Send when errors, or Do not email.

Add any additional recipients who should receive the CSV (and make sure to hit Enter after each address). You can also provide additional context in the message subject or body.

merge-duplicates-step-5-update-notify-tab-always-send.png

On the When tab, select Automate, and configure the frequency you'd like the template to run. When finished, click Schedule.

merge-duplicates-step-5-review-update-automate-daily.png

You can view all your scheduled automations at any time on the Operations > Automations page.

Insycle-Mailchimp Merge Logic

When merging duplicates, Insycle goes through the following steps for all Mailchimp record types:

  1. Select the master record according to the rules set in Step 4 on the Master tab.
  2. Determine field values to retain in the master record based on rules configured in Step 4 on the Fields tab.
  3. For fields without specific rules, identify non-empty values using a "fill in the blanks" method:

    • Check whether the master record has any empty fields.

    • If empty, copy values from the most recently updated record in the duplicate group.

  4. Update the master record by:

    • Applying the field values chosen in step 2 (from Field rules).

    • Adding the non-empty values identified in step 3 (using "fill in the blanks").

    • Preserving all other existing values in the master record.

  5. Merge the duplicate records and reassign all related objects to the master record.
  6. Store an audit trail report accessible from the Activity Tracker and send an email notification to the designated recipients.

If you're uncertain about how conflicting field values will merge, you can add those fields to the Preview CSV report by including them in the Layout via the icon-gear-18x18.png gear icon in Step 2.

merge-duplicates-hubspot-contacts-job-title-layout-646w.png

This is also useful if there are fields you're worried about backing up; including them in the CSV report allows you to refer back to the values in the CSV sent to your email or downloaded from the Activity Tracker.

merge-duplicates-hubspot-contacts-job-title-csv-646w.png

Advanced How-Tos

Step 1: Setting Up the Fields to Find Duplicates

Each row in your matching fields setup is cumulative, so records must meet all of the criteria. For example, looking for records that have the same First Name and Last Name and Phone Number returns only results where all three values are the same.

  The minimum required length for the matching values is four characters or more. Values such as "Joe" or "Ace" will be disregarded.

Field Name Comparison Rule Ignored Match Parts

Pick a field that you think has some duplicate values.

Running a very simple match operation like just First and Last Name is okay for giving you an idea of what you have, but it is too broad to use for reliable analysis and deduplication. There may be legitimate duplicate names–different people with the same first and last name. You need additional, unique criteria to narrow it down.

Choosing Unique Identifiers

Matching duplicates requires unique identifiers—data that is unlikely to be shared by any other record unless it is a duplicate. If you don't use unique identifiers, you are likely to identify unrelated records as duplicates and may accidentally merge them.

Many CRMs match first names, last names, and email addresses. If all of those match, or are similar, you can confidently determine that the record is a duplicate.

Other unique identifying fields that are commonly used in deduplication include:

    • Phone number
    • Domain name
    • Mailing address
    • ID number
Step 4: Selecting the Processing Method: Priority vs. Absolute

At the bottom of the Master tab, under Step 4, select the processing method that will be applied to your master selection rules.

merge-duplicates-step-4-priority-match-w-arrow-606px.png

By Priority: Look through the master selection rules in order, one by one. As soon as a record meets one of the criteria, Insycle makes the master selection and skips the rest of the rules on the list. The vast majority of duplicate templates should use By Priority.

Absolute: The master record must meet all of the listed master selection rules. If a record does not match every rule listed, no master record will be identified. Absolute is appropriate for less flexible master selection.

For example, if a company wants to ensure the chosen master record is in their sales pipeline and already has a sales rep working the record, they can choose Absolute and set the Master rules:

  • Lifecycle Stage is lead
  • Contact Owner active user

Choosing Absolute can often result in no master record being identified since the record has to match every rule listed, so in most cases, you should select By Priority.

Step 4: Understanding Master Record Selection

Let's say we have four records that represent the same person—Marta Vaskovitch. The Merge Duplicates module will identify this as one duplicate group consisting of four records.

Here is the data that we have for this duplicate group (viewed here in the Grid Edit module):

mceclip11.png

Here are the master selection rules we have set up:

step-4-record-mktg-eml-high-low-not-gmail.png

We haven't sent any emails to Marta yet, so when Insycle processes the first three rules—Marketing emails clicked, emails bounced, and emails opened—Insycle cannot eliminate any record because they all have the same value of zero.

None of the record email addresses use a work domain, so there isn't a single choice for the master using the Email rule either.

In the next rule about contact owner, records 61301, 61201, and 61251 are eliminated since no contact owner exists for those records. Now, only one record remains, 61351, therefore that's the master record.

mceclip13.png

Step 4: Data Retention Setup Examples

The master record can use values from several different records from the duplicate group based on the rules that you set in the Fields tab under Step 4

By default, any fields not specified here will use the master record values. However, if the master field is blank, the value from the most recently updated duplicate will be used.

In this first example, the Ownership value from the record with the most recent Modified Date will be kept, and all the Account Owner values from the records in the duplicate group will be saved to the Merged Owners custom field.

merge-duplicates-step-4-field-rules-owner.png

In this example, the most recent interaction data for several fields will be used in the merged record.

merge-duplicates-intercom-step-4-field-rules-latest.png

In the example below, six master field rules have been set up, including two different rules for the Lifecycle Stage. Insycle will look at the first of the two, and if it finds a record that matches the criteria, the second Lifecycle Stage rule will be ignored. In the example, if a record in the duplicate group with the "Lifecycle Stage" of "Customer" is found, then the next rule looking for the "Lifecycle Stage" of "Opportunity" would be ignored.

merge-duplicates-companies-step-4-fields-two-lifecycle-stage-rules.png

Granular Control for Picking Duplicate Records

For situations where there are no common rules you can apply for identifying duplicates for all or some of the records, you may need more granular control for picking records to include or exclude from the process.

To do this in bulk, you can create a CSV and use the Magical Import module.

In these cases, you can use CSV files to customize your bulk merging, designate master records, and exclude records from deduplication. Then you can import the CSV from the Magical Import, and use the Merge Duplicates module for complete control over the final merge operation. Learn how to customize merging Duplicates in bulk using a CSV.

To do this one record at a time, you can use Manual mode of the Merge Duplicates module.

In Manual mode, you have complete control over which records are merged together by selecting them from the Record Viewer. Manual mode should be reserved only for cases needing a careful, controlled process. Learn more about merging duplicates in Manual mode.

Explore other advanced approaches to merging duplicates in our Deduplication Scenarios article.

Tips for Bulk Merging Duplicates

  • Begin with easy-to-find duplicates. Iterate through fields and rules you know will surface duplicates. Don’t expect to resolve all your duplicates by setting up and running this process once. You will need to run this process multiple times for different fields or nuanced variations.
  • Save templates. Each time you get a Merge Duplicates process to run the way you want in your database, save it as a template. When you have a solid set of templates that reliably resolve most of your dupes, you can put them together as a Recipe that can run on a regular, automated schedule.
  • Look for edge cases that fall outside your standard rules. These may be templates you run manually so you can make adjustments based on what you find.
  • Do some experimentation. Use the Preview mode CSV report to analyze patterns in the duplicates. Add additional fields to the CSV by clicking the gear icon in Step 2 and including them in the Layout. You may learn what is causing the duplicates and learn how to avoid having them in the first place. You may also want to explore your data in the Grid Edit module to understand what you have so you can design templates that catch all potential variations.

Troubleshooting

If you're not seeing the results you expect when merging duplicates, consider these issues:

Not all identified duplicates are merging into the master

You have duplicate records that have been identified by Insycle, but not all of them are merging into the master. Under Step 2, check to see how many duplicates are in the affected duplicate groups. If you have duplicate groups that contain more than five records, you may want to change the value in Skip duplicate groups with more than 5 records per group to make sure you can get them all.

merge-duplicates-step-3-bulk.png

This setting is intended to protect against the accidental merging of non-duplicate records if the filter in Step 1 is too broad.

"Failed" Result in CSV with "Change rules in Step 4 'Master Selection'" Message 

If the Message column of the CSV report displays this text:

Change rules in Step 4 'Master Selection'. Failed to pick master record because multiple records (X) meet the selection criteria. In 'Master Selection', change, add, or reorder the rules such that only one record matches (if cannot determine master based on field values, use 'Record ID is lowest' as the last rule).

merge-duplicates-intercom-users-csv-w-error.png

This means that based on all the rules, Insycle could not figure out which record in the duplicate group would be the master. None of the records meet more of the rules than others.

There are a few things you can try to resolve this:

  1. Under Step 4, on the Master tab, experiment with reordering or adding additional fields that are likely to have unique values.
  2. Also on the Step 4, Master tab, check to ensure that you have By Priority selected and not Absolute.

    merge-duplicates-step-4-priority-match-w-arrow-606px.png
    With By Priority, your master record only has to match one rule. Using Absolute, your master record would have to meet all of the rule criteria. The majority of the time, it is best to select By Priority.

    If By Priority was used, this indicates none of the records in the duplicate group meet any of the criteria on the list more than the others. In this case, you'll need to experiment on the Master tab, reordering or adding additional rules for fields likely to have unique values.

  3. As a last resort, you can add a rule on the Master tab of Step 4 that says Last Changed Date is latest, or Signup Timestamp is latest.
    merge-duplicates-mailchimp-contacts-step-4-master-tab-last-resort-rules-646px.png
Non-duplicate records are being merged together

There are a couple of things to look at that may be misidentifying records as duplicates.

First, you may need a better unique identifier. Under Step 1, if you only use fields that could correctly contain the same values in multiple records, these aren't unique identifiers. In this case, you are likely to identify unrelated records as duplicates and may accidentally merge them.

Unique identifiers are data that is unlikely to be shared by any other record unless it represents the same underlying entity. Fields that are commonly used in deduplication include phone numbers, email, mailing addresses, or ID numbers.

Second, this may indicate the Comparison Rule under Step 1 is too broad. Try using the Exact Match comparison rule instead of Similar Match. Similar Match looks for values that may be close but with a one-character difference (maybe a typo) which broadens the search. 

Remember, always run your deduplication in Preview Mode to confirm things are working as expected before running them in Update Mode and applying the changes to your Mailchimp records.

Insycle isn't finding any duplicates

Most of the time, when Insycle can't find duplicates, it is due to your matching rules in Step 1. It is important to analyze the underlying data to better understand how to set up your rules. A useful exercise can be to set up your matching filters to look for exact matches of just First Name and Last Name

bulk-merge_3.png

When you click Find, these rules can show you a broad overview of what duplicates are potentially in your database and what fields might be useful to include in your matching fields. These settings are just for discovery and should not be used for a final merge operation; many people can have the same first and last names and are not duplicates. 

To get further context, in Step 2, click the icon-gear-18x18.png gear icon on the right side of the title bar. Here, you can add any field in your database as a column to the duplicate group review to better understand the data inside these records.

dup15.png

For more help troubleshooting issues with Insycle, refer to our Troubleshooting Issues article.

Frequently Asked Questions

What happens to field data that isn't in the master record? Will I lose data?

By default, Insycle will keep the master record values; if a master record has empty fields, Insycle copies values from the most recently updated record in the duplicate group where that data exists.

merge-duplicates-most-recent-value-ani-644x374.gif

For example, if the master record's Industry field is empty but another record in the duplicate group has an Industry value, that value will automatically be copied to the master record.

Additionally, you can control what values are saved in the master record after the merge, regardless of the default merge behavior. By adding each field you want to control the data retention for in the Fields tab under Step 4 and selecting a Criteria, you can tell Insycle where the data for the field should be taken from and how to handle it.

Any data that is not in the master or not copied to the master is removed.

Learn more about configuring data retention and master record selection.

How do I ensure that I am not merging non-duplicate records together?

There are two ways to make sure that the records that you are merging are indeed duplicate records.

First, always run your deduplication templates in Preview Mode before running them in Update Mode. This produces a CSV that shows you how your records would have been merged. Then you can ensure that your Merge Duplicates template is working as expected and not merging non-duplicate records together.

Additionally, to ensure a smooth merge process, consider narrowing down the matching settings in Step 1. Try the Exact Match Comparison Rule instead of Similar Match. Then make sure that you are using actual uniquely identifying fields—first name, last name, email, and phone number are popular choices. The more tightly defined your filter is, the less likely you are to merge non-duplicate records.

I already have a list of duplicates, can Insycle bulk merge them?

Yes. You can use an existing CSV with duplicate record details. The file needs to include the record IDs and a "Deduplication Master" column, specifying which records should be the master, kept after the merge. Next, create a custom field "Deduplication Master" in your CRM to facilitate the merging. Use the Magical Import module to import the edited CSV file into the CRM, populating the new custom field. Finally, utilize this custom field to merge the duplicate records in the Merge Duplicates module.

Learn more about customizing bulk deduplication from a CSV.

Can I select which data is retained in my master record on a field-by-field basis?

Yes, Insycle allows you to select which field data is retained in the master record using the Fields tab under Step 4. See the section, Step 4: Set Rules for Master Record Selection and Data Retention of this article for more details.

merge-duplicates-mailchimp-contacts-step-4-fields-tab-4-rules-646px.png

Do my matching fields have to match each other exactly?

No. The Similar Match Comparison Rule found in Step 1 looks for values that may be close but with a one-character difference (maybe a typo) and broadens the search. This search behaves like when Google shows results for a slightly different term, or says “Did you mean...” For example, if a Company Name of, “Acme” is found, it could include records with the Company Name values “Akme, acm, Acma,” etc., as a match.

You should be careful when using Similar Match as the looser criteria can incorrectly identify non-duplicates as duplicates. 

Review the Understanding Similar Matching best practices for more detail.

Why can I only process 50 duplicate groups at a time?

Insycle shows 50 records on the module screen as a preview; this isn't the entire list of records. To see everything, include all records when you view the Preview CSV report.

Insycle can process thousands of duplicate groups in one operation. Potentially, you could deduplicate your entire database in one operation. 

How many duplicates can I merge into one master record?

You can merge up to 100 duplicate records into a single master record. 

If you have duplicate groups that contain more than five records, you may want to change the value in Skip duplicate groups with more than 5 records per group under Step 3 to make sure you can get them all.

merge-duplicates-step-3-bulk.png

This is a precaution to ensure that if you use a duplicate matching filter that is too broad in Step 1, you do not accidentally merge many non-duplicate records together. If you are going to set this number at a high level, it is a good idea to run Preview Mode first to make sure your deduplication template is operating as you intend.

Are there any limits on the number of records that can be identified and merged with my paid subscription?

All plans include unlimited usage, unlimited users, and unlimited operations. See the pricing page for more details. During the free trial, there is a cap of 500 records updated, cleansed, or merged.

Additional Resources

Related Help Articles

Related Blog Posts