Cross-Audience Deduplication in Mailchimp

Navigate to Data Management Merge Duplicates, pick the record type, and explore the default templates for a pre-built solution.

Insycle provides a flexible way to identify duplicates using any data field, not just email addresses. Step 1 scans your database records, checking the fields you specify for a match.

Click the + Field button to add the fields you want to examine for duplicates, along with parameters on what to search for. Choose fields that, together, provide a high level of certainty that the matched records are duplicates. See the Setting the Criteria for Finding Duplicate Matches article for complete details.

For example, you can try to identify duplicate contacts using the First Name, and Last Name, and Email Address fields. Contacts must match all three criteria to be considered duplicates.

You can specify parts of the values to ignore, such as whitespace, special characters, URL paths, and more. For example, if you choose to ignore the top-level domain in the email field, jane@acme.com, jane@acme.co.uk, and jane@acme.io will be considered matches.

To identify duplicates within a single audience, include the List ID (Audience ID in Mailchimp) as a rule in Step 1.

You can use the template, Same email, ignoring top-level domain (within same audience) as a starting point.

When two or more records represent the same contact based on your matching rules, they are clustered together into duplicate groups. Each duplicate group shows the total number of records that were identified as duplicates based on your settings. For example, if you had four records for the same contact, it would count as one duplicate group with four duplicate records.

Under Step 2, each row represents a duplicate group, with the number indicating the number of duplicate records found. When you click the chevron   at the right end of a duplicate group row to expand it and examine the individual records within that group. 

To exclude a duplicate group from all future deduplication analysis, click the X on the group row. Learn more about adding and removing duplicate groups from the Exclusions list in the Advanced How-Tos below.

Explore the record data in the duplicate groups. Double-check that the fields you set up in Step 1 are showing what you expected. 

Looking at the record details across several duplicate groups, decide the best way to determine which record is the master record that the other records will be merged into.

Add more columns to the view using the gear icon on the right to get more context.

In Step 3, you define how your duplicates are merged. You'll choose your operation mode, set rules for which record becomes the master, and optionally control which field values are retained after the merge. 

Choose Your Operation Mode

At the top of Step 3, select Bulk or Manual mode:

• Bulk mode (recommended) — Automatically merges all duplicate groups according to your master selection rules. This is the right choice for most deduplication operations and is required for Templates, Recipes, and Automation.
• Manual mode — Lets you select specific records to merge one at a time from the Record Viewer. Reserve this for high-value records or complex situations that require hands-on review of each merge.

Most users should select Bulk mode and proceed.

Configure Rules to Automatically Select the Master Record (Master Tab)

On the Master tab, you define how all of the matching duplicate groups should be merged at scale. To do this, you need to create a series of rules that tell Insycle which record in each group should be selected as the master. The master is the record that will remain after the merge.

For example, if you had four records representing the same company, they would form a duplicate group of four records, which would be merged into a single master record. The other three records no longer exist.

For each duplicate group, Insycle will examine each Master rule to determine which records in the group meet the criteria. Rules are read in order, from top to bottom, and as soon as a record is the only one to meet a rule, it is selected as the master record. The subsequent rules are ignored. 

Select the processing method—By Priority or Absolute. Most de-duplication operations should use By Priority. 

Learn more about these options in the Choosing Which Record Remains After a Merge article.

Setting Field Retention Rules (Fields Tab)

The Fields tab lets you control which field values end up in the master record after the merge. By default, Insycle keeps the master record's values and fills any empty fields with values from the most recently updated duplicate. You can override this field-by-field for any fields that need special handling.  

For example, you might want to retain the earliest created "Company Name" while keeping the "Phone Number" that appears most frequently.

For each field you want to control, select the field and specify where its data should be sourced. This is merged into the master. Insycle evaluates your rules from top to bottom and stops at the first match for each field. This means rule order matters—place your most specific rules first, with broader fallback rules below.

The Criteria dropdown gives you various options for choosing the data to keep, and the Group Fields lets you keep values of multiple fields from the same record.

Learn more about these options in the Keeping the Right Data When Records Merge article.

Step 4 is where you preview and apply your merge changes.

Preview Merged Changes in CSV Report

Now with the filters and master record set up, you can preview the changes you are making to your data. That way, you can check to ensure your deduplication configuration is working as expected before those changes are pushed to your live database. The CSV report that Insycle generates includes records from all the duplicate groups.

Under Step 4, click the Review button and select Preview mode.

Click the Next button to go to the Notify screen, where you can select recipients for the email report and add additional context.

On the When tab, click the Run Now tab, and select which records to apply the change to (in most cases this will be All), then click the Run Now button.

Insycle will generate a preview CSV and send it to your email. Open the CSV file from your email in a spreadsheet application and review the data.

The CSV report includes records from all duplicate groups. It shows the current state of the master record, excluding reassignments of associated contacts, deals, or activities. (To validate how these reassociations occur, run a small-scale test in Update mode.) This ensures that the CSV accurately represents the merging setup before executing in Update mode.

There's a row for each record, which includes:

• The Result of the action
• A Message with details
• The Duplicate Group ID indicates which records will be merged together
• A record Status that identifies which were picked as master and which were identified as duplicates and merged into the master. See below for more details
• All fields used to identify the duplicates in Step 1
• The record ID, record name, or email, and Deeplink to the CRM record
• All fields selected for the Master and Fields rules in Step 3. Note that if a field is used on both tabs, it will only appear once in the CSV. 
• Any additional fields added to the Layout tab in Step 2.

The Duplicate Group ID indicates which records will be merged together.

The Status column indicates:

• Duplicate – The record is part of a duplicate group.
• Master – The master record is chosen for the duplicate group based on default behavior and your Master rules. Review the selections in this row to determine whether the appropriate records are being chosen.
• Master (After) – For each duplicate group, a special row –Master (After)– will appear in the CSV report. The Master (After) row shows the values the final record will contain based on your Field rules and the default behavior.
• Error – If Insycle is not able to determine which record would be the master, an error message will appear here. See the Merge Duplicates Troubleshooting article for next steps.

If everything looks good, return to Insycle and move forward with applying the changes.Apply Changes to Your CRM Records

When you're satisfied with the preview results, you can apply the merge changes to your CRM.

Under Step 4, click the Review button again, and this time select Update mode.

On the When tab, you should use Run Now the first time you apply these changes to the CRM. If you have a large number of records, consider doing a smaller batch to assess the results in your CRM.

When merging duplicates, Insycle goes through the following steps for all Mailchimp record types:

1. Select the master record according to the rules set in Step 3 on the Master tab.
2. Determine field values to retain in the master record based on rules configured in Step 3 on the Fields tab.
3. For fields without specific rules, identify non-empty values using a "fill in the blanks" method:
   • Check whether the master record has any empty fields.
   • If empty, copy values from the most recently updated record in the duplicate group.
4. Update the master record by:
   • Applying the field values chosen in step 2 (from Field rules).
   • Adding the non-empty values identified in step 3 (using "fill in the blanks").
   • Preserving all other existing values in the master record.
5. Merge the duplicate records and reassign all related objects to the master record.
6. Store an audit trail report accessible from the Activity Tracker and send an email notification to the designated recipients.

If you're uncertain about how conflicting field values will merge, you can add those fields to the Preview CSV report by including them in the Layout via the gear icon in Step 2.

This is also useful if there are fields you're worried about backing up; including them in the CSV report allows you to refer back to the values in the CSV sent to your email or downloaded from the Activity Tracker.

If your database is large or you're getting an overwhelming number of duplicate groups, use the Filter button in Step 1 to narrow down the records Insycle analyzes. Filtering to a subset — for example, contacts created in the last 30 days, or companies in a specific region — makes it easier to validate your configuration before running it across your full database. A filtered dataset also processes faster.

To add a filter, click the Filter button in Step 1, choose a field, select a condition, and set a value. The filter is applied before the matching step runs.

Most of the options in the Field dropdown match the fields that are found in your HubSpot records, and for contact records, there are three additional options related to the Email value: 

• Email Username: The portion of the email address before the “@.” For example, if the email address were “maria@acmewidgets.com,” the username value would be “maria.” 
• Free Email Provider Domain: Choose True to filter out records where the email domain is Gmail, Hotmail, Yahoo, or any of about 10,000 other free email providers. This filter helps ensure these are real clients, or can determine which record is the legitimate one, because it’s most likely that customer companies aren't using free Gmail accounts (though a contact may have accidentally emailed us from one at some point). 
• Email Top-Level Domain: The top-level domain (TLD) is everything that follows the final dot of a domain name. For example, in the domain name acmewidgets.com, '.com' is the TLD. Some other popular TLDs include '.org', '.uk', and '.edu'. 

When setting up your matching fields for the first time, start with your highest-confidence criteria — fields that, together, are very unlikely to match unless the records are true duplicates. Exact Match on email address plus first and last name is a reliable starting point for contacts. Domain Name is a strong starting point for companies.

Once you're confident your configuration is catching true duplicates without pulling in false positives, you can broaden your criteria — for example, by adding Similar Match to catch typos and slight variations: 

Or, by using Related Fields to compare values across two fields that contain similar data (such as Email and Additional Email).

Insycle evaluates your Master tab rules from Step 3 in order, eliminating records that don't match each rule until only one remains. If multiple records still match after all rules are evaluated, no master can be determined, and the group will show an error in the CSV.

For example, imagine having four duplicate records for the same contact. (In this image, we are examining the records in the Grid Edit module.)

In the Merge Duplicates module, you have configured the first three Master rules based on email engagement metrics, but all four records have identical values of zero, so no records are eliminated. Your fourth rule checks for an active Contact Owner — three records have no owner, so they're eliminated. The one remaining record becomes the master.

This is why rule order matters. Place your most reliably differentiating rules — like record owner, lifecycle stage, or engagement activity — where they're most likely to yield a clear winner. If you're frequently seeing errors in your CSV, revisit your rule order and consider adding a tiebreaker rule, such as earliest Create Date or latest Last Modified Date, as a final fallback.

In this CSV report example, you can see that the one record with an active owner was chosen as the master.

Even though this was the fifth rule, it is the first one that matched only one record:

• When Insycle processed the first three rules—Marketing emails clicked, emails bounced, and emails opened—Insycle couldn't eliminate any records because they all had a value of 0.

• None of the record email addresses are "role-based," so there isn't a single choice for the master using the Email rule either.

• In the next rule about contact owner, there is one record with an active contact owner; therefore, that is the master record.

By default, all duplicate groups identified in Step 1 are candidates for merging. If there are specific groups you want to permanently prevent from being merged — regardless of which matching rules or templates you use — you can add them to the Duplicate Groups Exclusion List directly in Step 2.

 Note that this feature applies only to Bulk mode. If you are using Manual mode in Step 3, use the record checkboxes to deselect individual records instead.

To exclude a duplicate group:

In Step 2, click the X on a duplicate group row. The group will be added to the Exclusion List and will no longer appear in duplicate analysis or be included in merges, even when using different duplicate detection templates.

To review and manage excluded groups:

Click the Exclusions button in the Step 2 header. This opens a dialog showing all excluded groups. 

You can expand each group to view the individual records and their values, and use the Layout tab to configure which fields are displayed. To remove a group from the Exclusion List, click the X next to it — those records will once again be considered during duplicate analysis.

Important considerations:

• Exclusions apply to Bulk mode only and to the entire duplicate group — they cannot be applied to individual records within a group. If you need to exclude a single record rather than the entire group, switch to Manual mode in Step 3 and uncheck the records that should not be merged.
• Exclusions are based on a specific set of record IDs. If a new duplicate record is later added to your CRM, the system may detect a new group containing that record alongside previously excluded records. Because the group now includes a different set of record IDs, it will appear again during duplicate analysis and can be evaluated or excluded as needed.