duplicates

How to Deduplicate People and Companies in Bulk and Automatically

You have multiple records that represent the same companies in Intercom. Contacts and conversations may end up associated with a duplicate company record, making them difficult to find, track, and analyze. Duplicate company records can also inhibit your support team from effectively segmenting and personalizing your communications.

Insycle's Merge Duplicates module can identify duplicate companies and merge them flexibly, in bulk, using advanced rules to select the master record and control data retention.

Process Summary

  1. Set rules to identify duplicates.
  2. Review and analyze the identified duplicates.
  3. Choose Bulk Mode.
  4. Set rules to select the master record and determine what field data is retained.
  5. Deduplicate your records in Intercom.

 

Step 1: Configure Rules to Identify Duplicates

Navigate to Data Management > Merge Duplicates, pick the Companies record type, and explore the default templates for a pre-built solution.

This template below matches duplicate Intercom companies using the exact match for the company name.

merge-duplicates-intercom-companies-step-1-simple-tab-company-name-646w.png

Click the "Find" button and Insycle will analyze your Intercom data to find duplicate companies using these rules.

Step 2: Analyze the Identified Duplicates

When two or more records represent the same company based on your rules from Step 1, they are clustered together into duplicate groups. Each duplicate group shows the total number of records that were identified as duplicates based on your settings. For example, if you had four records for the same company, it would count as one duplicate group with four duplicate records.

Under Step 2, each row represents a duplicate group, with the number indicating how many duplicate records were found. 

merge-duplicates-companies-step-2-company-name-collapsed-groups-646w.png

When you click a row, it expands and shows the duplicate records in the group.

merge-duplicates-companies-step-2-company-name-graso-expanded-646w.png

Explore the record data in the duplicate groups. Double-check to make sure that the fields you set up in Step 1 are showing what you expected. 

Looking at the record details in several duplicate groups, decide the best way to determine which is the master record the other records will be merged into.

Add more columns to the view using the gear button on the right to get more context.

Step 3: Configure the Merge Logic

Step 3 is where you configure how duplicate groups are consolidated—from choosing your operation mode to defining master selection rules and field retention logic. This step combines operation selection with the merge logic configuration through multiple tabs.

Bulk or Manual Mode

At the top of Step 3, choose between Bulk and Manual modes in the header:

  • Bulk mode (recommended) automatically merges all duplicate groups according to the master record selection rules you define in the Master tab. This is the right choice for most deduplication processes, and the only mode that supports templates, Recipes, and automation.
  • Manual mode lets you review and select specific records to merge individually from the Record Viewer. Reserve this for high-value records or complex scenarios where you want hands-on control over each merge.

Most users should select Bulk mode and proceed.

merge-duplicates-intercom-step-3-bulk-manual-mode-highlight.png

Step 3 operates through two tabs that work together:

1) Master tab

On the Master tab, you define how all of the matching duplicate groups should be merged at scale. To do this, you need to create a series of rules that tell Insycle which record in each group should be selected as the master. The master is the record that will remain after the merge.

For example, if you had four records representing the same company, they would form a duplicate group of four records, which would be merged into a single master record. The other three records no longer exist.

For each duplicate group, Insycle will examine each Master rule to determine which records in the group meet the criteria. Rules are read in order, from top to bottom, and as soon as a record is the only one to meet a rule, it is selected as the master record. The subsequent rules are ignored. 

At the bottom of the Master tab, you'll also configure:

  • Processing method — Choose between By Priority (evaluates rules in sequence, stops at first unique match) or Absolute (master must match all rules). Most operations should use By Priority.
  • Exclusions — By default, Insycle skips any duplicate group containing more than 5 records—this prevents overly broad matching criteria from accidentally merging unrelated records. You can adjust this threshold up to 100 records per group.

Refer to Choosing Which Records to Merge Using a CSV for configuration details.

master record

2) Fields tab

The Fields tab lets you control which field values end up in the master record after the merge. By default, Insycle keeps the master record's values and fills any empty fields with values from the most recently updated duplicate. You can override this field-by-field for any fields that need special handling.  

For example, you might want to retain the earliest created "Company Name" while keeping the "Phone Number" that appears most frequently.

For each field you want to control, select the field and specify where its data should be sourced. This is merged into the master. Insycle evaluates your rules from top to bottom and stops at the first match for each field. This means rule order matters—place your most specific rules first, with broader fallback rules below.

The Criteria dropdown gives you various options for choosing the data to keep, and the Group Fields lets you keep values of multiple fields from the same record.

See Keeping the Right Data When Records Merge for full details on configuring field data retention rules.

merge-duplicates-intercom-companies-step-4-field-rules.png

What happens to field data if I don’t create custom rules?

There is no need to create rules for every field in your CRM - Insycle automatically handles fields without specific rules using a "fill in the blanks" approach. When the master record has empty fields, Insycle copies values from the most recently updated record in the duplicate group where that data exists. For example, if the master record's Industry field is empty but another record in the duplicate group has an Industry value, that value will automatically be copied to the master record. This means you only need to create custom retention rules for the handful of fields that require special handling rather than setting up rules for all your fields.

Step 4: Review Changes and Merge Records

Preview Merged Changes in CSV Report

Now with the filters and master record set up, you can preview the changes you are making to your data. That way, you can check to ensure your deduplication configuration is working as expected before those changes are pushed to your live database. The CSV report that Insycle generates includes records from all the duplicate groups.

Under Step 4, click the Review button and select Preview mode.

merge-duplicates-step-4-review-preview-mode-tab.png

Click the Next button to go to the Notify screen, where you can select recipients for the email report. You can also add additional context on this screen.

On the When tab, click the Run Now tab, and select which records to apply the change to (you could do All, but if you have a large number of duplicates, you may just want to do a chunk for your preview), then click the Run Now button.

merge-duplicates-step-5-review-preview-run-now.png

Insycle will generate a preview CSV and send it to your email. Open the CSV file from your email in a spreadsheet application.

merge-duplicates-intercom-companies-csv.png

The Duplicate Group ID indicates which records will be merged together

The Status column indicates:

  • Duplicate – The record is part of a duplicate group.
  • Master – The master record chosen for the duplicate group based on default behavior and your Master rules. Review the selections in this row to determine whether the appropriate records are being chosen.
  • Master (After) – For each duplicate group, a special row –Master (After)– will appear in the CSV report. The Master (After) row shows the values the final record will contain based on your Field rules and the default behavior. 
  • Error – If Insycle is not able to determine which record would be the master, an error message will appear here. See the Merge Duplicates Troubleshooting article for next steps.

If everything looks good, return to Insycle and move forward with applying the changes.

Apply Changes to Your Intercom Records

If everything in your CSV preview looks correct, return to Insycle and move forward with applying the changes to the live Intercom data.

Under Step 4, click the Review button again, and this time select Update mode.

On the When tab, you should use Run Now the first time you apply these changes to Intercom. If you have a large number of records, you may want to do a smaller batch to review the results in your live CRM database.

merge-duplicates-step-4-review-update-run-now-chunk-646w.png

Save as a Template and Set Up Automation

After you've seen the results in Intercom and are satisfied with how the merge runs, you can save your configuration as a template.

Return to the Template menu at the top of the page and click Copy to save your configurations as a new version of whatever template you started with. Then click the pencil to edit your new template name.

save-template-copy-and-rename.png

Now you can set up automation so this merge operation runs on a set schedule.

Under Step 4, click the Review button, and select Update mode.

On the When tab, select Automate, and configure the frequency you'd like the template to run. When finished, click Schedule.

merge-duplicates-step-4-review-update-automate-daily-646w.png

If you have several templates you'd like to automatically run together, you can create a Recipe.

Audit Trail and History

The Activity Tracker lets you review all changes made through Insycle. At any time you can download a CSV report of the operation and records affected.

Tips for Bulk Merging Duplicates

  • Begin with easy-to-find duplicates. Iterate through fields and rules you know will surface duplicates. Don’t expect to resolve all your duplicates by setting up and running this process once. You will need to run this process multiple times for different fields or nuanced variations.
  • Save templates. Each time you get a Merge Duplicates process to run the way you want in your database, save it as a template. When you have a solid set of templates that reliably resolve most of your dupes, you can put them together into a Recipe that runs on a regular, automated schedule.
  • Look for edge cases that fall outside your standard rules. These may be merges you perform manually so you can make adjustments based on what you find.
  • Do some experimentation. Use the Preview mode CSV report to analyze patterns in the duplicates. Add additional fields to the CSV by clicking the gear icon in Step 2 and including them in the Layout. You may learn what is causing the duplicates and learn how to avoid having them in the first place. You may also want to explore your data in the Grid Edit module to understand what you have so you can design templates that catch all potential variations.

Advanced How-Tos

Use a filter to work with a subset of your data

If your database is large or you're getting an overwhelming number of duplicate groups, use the Filter button in Step 1 to narrow down the records Insycle analyzes. Filtering to a subset — for example, contacts created in the last 30 days, or companies in a specific region — makes it easier to validate your configuration before running it across your full database. A filtered dataset also processes faster.

To add a filter, click the Filter button in Step 1, choose a field, select a condition, and set a value. The filter is applied before the matching step runs.

dup10.png

Most of the options in the Field dropdown match the fields that are found in your HubSpot records, and for contact records, there are three additional options related to the Email value: 

  • Email Username: The portion of the email address before the “@.” For example, if the email address were “maria@acmewidgets.com,” the username value would be “maria.” 
  • Free Email Provider Domain: Choose True to filter out records where the email domain is Gmail, Hotmail, Yahoo, or any of about 10,000 other free email providers. This filter helps ensure these are real clients, or can determine which record is the legitimate one, because it’s most likely that customer companies aren't using free Gmail accounts (though a contact may have accidentally emailed us from one at some point). 
  • Email Top-Level Domain: The top-level domain (TLD) is everything that follows the final dot of a domain name. For example, in the domain name acmewidgets.com, '.com' is the TLD. Some other popular TLDs include '.org', '.uk', and '.edu'. 

Start narrow, then broaden

When setting up your matching fields for the first time, start with your highest-confidence criteria — fields that, together, are very unlikely to match unless the records are true duplicates. Exact Match on email address plus first and last name is a reliable starting point for contacts. Domain Name is a strong starting point for companies.

merge-duplicates-hubspot-companies-step-1-similar-name-domain-exact-phone.png

Once you're confident your configuration is catching true duplicates without pulling in false positives, you can broaden your criteria — for example, by adding Similar Match to catch typos and slight variations: 

merge-duplicates-hubspot-companies-step-1-comparison-rules-646w.png

Or, by using Related Fields to compare values across two fields that contain similar data (such as Email and Additional Email).

merge-duplicates-hubspot-contacts-step-1-advanced-first-last-email+additional-646w.png

Understanding how master selection works

Insycle evaluates your Master tab rules from Step 3 in order, eliminating records that don't match each rule until only one remains. If multiple records still match after all rules are evaluated, no master can be determined, and the group will show an error in the CSV.

For example, imagine having four duplicate records for the same contact. (In this image, we are examining the records in the Grid Edit module.)

mceclip11.png

In the Merge Duplicates module, you have configured the first three Master rules based on email engagement metrics, but all four records have identical values of zero, so no records are eliminated. Your fourth rule checks for an active Contact Owner — three records have no owner, so they're eliminated. The one remaining record becomes the master.

This is why rule order matters. Place your most reliably differentiating rules — like record owner, lifecycle stage, or engagement activity — where they're most likely to yield a clear winner. If you're frequently seeing errors in your CSV, revisit your rule order and consider adding a tiebreaker rule, such as earliest Create Date or latest Last Modified Date, as a final fallback.

merge-duplicates-hubspot-contacts-step-3-master-tab-6-rules.png

In this CSV report example, you can see that the one record with an active owner was chosen as the master.

Even though this was the fifth rule, it is the first one that matched only one record:

  • When Insycle processed the first three rules—Marketing emails clicked, emails bounced, and emails opened—Insycle couldn't eliminate any records because they all had a value of 0.

  • None of the record email addresses are "role-based," so there isn't a single choice for the master using the Email rule either.

  • In the next rule about contact owner, there is one record with an active contact owner; therefore, that is the master record.

mceclip13.png

Exclude duplicate groups from deduplication 

By default, all duplicate groups identified in Step 1 are candidates for merging. If there are specific groups you want to permanently prevent from being merged — regardless of which matching rules or templates you use — you can add them to the Duplicate Groups Exclusion List directly in Step 2.

 Note that this feature applies only to Bulk mode. If you are using Manual mode in Step 3, use the record checkboxes to deselect individual records instead.

To exclude a duplicate group:

In Step 2, click the X on a duplicate group row. The group will be added to the Exclusion List and will no longer appear in duplicate analysis or be included in merges, even when using different duplicate detection templates.

merge-duplicates-hubspot-contacts-step-2-exclude-group-button-646px.png

To review and manage excluded groups:

Click the Exclusions button in the Step 2 header. This opens a dialog showing all excluded groups. 

merge-duplicates-hubspot-contacts-step-2-header-exclusions-button-646px.png

You can expand each group to view the individual records and their values, and use the Layout tab to configure which fields are displayed. To remove a group from the Exclusion List, click the X next to it — those records will once again be considered during duplicate analysis.

merge-duplicates-hubspot-contacts-step-2-manage-exclusions-popup-remove-exclusion-646px.png

Important considerations:

  • Exclusions apply to Bulk mode only and to the entire duplicate group — they cannot be applied to individual records within a group. If you need to exclude a single record rather than the entire group, switch to Manual mode in Step 3 and uncheck the records that should not be merged.
  • Exclusions are based on a specific set of record IDs. If a new duplicate record is later added to your CRM, the system may detect a new group containing that record alongside previously excluded records. Because the group now includes a different set of record IDs, it will appear again during duplicate analysis and can be evaluated or excluded as needed.

Need More Control Over Which Records Are Merged?

If your scenario requires selecting specific records to include or exclude that can't be handled with matching rules alone, you have three options: 

  1. If you already have a list of known duplicates, such as an export from HubSpot's manage duplicates tool, you can upload a CSV file of duplicate record ID pairs directly into the Merge Duplicates module using the CSV tab in Step 1

  2. For more complex scenarios — such as designating master records or excluding specific records using custom attributes — you can use the Merge Duplicates module in combination with Magical Import. Learn how to choose which records to merge using a CSV

  3. To handle records one at a time, use Manual mode in the Merge Duplicates module. Learn more about merging duplicates one at a time.

Additional Resources

Frequently Asked Questions

Check out the Merging Duplicates FAQ for help with questions about merging Intercom duplicates.

Related Help Articles

Related Blog Posts