Deduplicate HubSpot Contacts, Companies, and Deals in Bulk

 

Duplicate data in HubSpot poses serious problems for any company.

Duplicate HubSpot records inhibit your marketing team from effectively segmenting and personalizing your communications. Sales teams step on each other's toes and lack vital context in conversations. Support teams miss important information, and analysis and reporting are skewed.

Insycle helps you to merge duplicate contacts, companies, and deals—flexibly and powerfully.

How It Works

Insycle analyzes your HubSpot database and identifies duplicates with flexible matching rules, using any field in your database, to help you identify and merge more duplicates.

Once Insycle identifies the duplicate records, you set rules for determining the master record that other duplicates will be merged into—such as the first record created, the record with the most email opens, or any other attribute that would be relevant. You can also set merging logic on a field-by-field basis.

You can merge HubSpot duplicates in bulk, and Insycle provides a complete report of what was identified as a duplicate, what was merged, and what the outcome was in your master record.

Summary

With Insycle, you can identify and merge duplicate contacts, companies, and deals in flexible and powerful ways.

Insycle can identify duplicates using any field in your database, with flexible matching rules. Once Insycle identifies the duplicate records, you then set rules for determining the master record that other duplicates will be merged into—such as the first record created, record with the most email opens, or any other field that would be relevant for determining the right master record. You can also set merging logic on a field-by-field basis.

With Insycle, you can deduplicate in bulk, then automate the process using templates to keep your database free from duplicates at all times.

Step 1: Find Duplicates

Insycle deduplication takes place in the Merge Duplicates module. Navigate to there and from the top of the screen, choose whether you will be deduplicating contacts, companies, deals, or custom objects.

For a quick start, explore the default templates. There might be a template that already solves exactly what you need. Alternatively, you can use an existing template as a starting point and adjust it to your requirements.

Deduplication Templates


In order to find duplicates, you need to define how to match records.

For example, to find duplicate people you may use the "First Name", "Last Name", and "Email Domain" fields. In that example, people with the same first name AND same last name AND email domain will show as possible duplicates.

Matching rules


Insycle allows for deep control in defining how to match the values of those fields including exact vs. similar matching, ignored phrases, or matching only portions of the field.

You can add fields by clicking the "+" button, or remove fields by clicking the X button.

When you click "Find", Insycle will identify duplicate records based on the matching filter that you have set in this step.

Step 2: Review Duplicates

In this step, Insycle allows you to preview the identified duplicates and double-check to make sure that your filter is producing accurate results. That way, you can avoid merging any records that are not actually duplicated.

Once you click the "Find" button in Step 1, Insycle will generate a list of the identified duplicates for you to review.

When two or more records represent the same entity (person, company, or other), we refer to them as a Duplicate Group. For example, if you had 4 records for the same person, it would count as 1 Duplicate Group with 4 records.

In this view, each row represents a Duplicate Group. When you click a row, it expands and shows the records that belong to that group. You can add more columns to the view using the ⚙ button on the right, in order to get more context and analyze the records.

Review duplicates


Clicking the checkbox on a record expands your view and allows you to look at all identified duplicates within that Duplicate Group.

Duplicate group


Here, you can dig deep into your identified duplicate groups, into the individual records, to check for accuracy and get a better idea of the best way to determine the Master Record that other records will be merged into, which we will do in Step 4.

Additionally, this is where you can select records that you would like to merge manually, which can be chosen in Step 3.

Step 3: Choose Operation

In Step 3, you can merge duplicates in bulk using rules, or manually one by one.

When merging in bulk, Insycle limits the duplicate group size to 5 and skips groups that contain more records. For example, Insycle will not merge a group of 7 records that represent the same person. You can adjust the default value of 5 as needed from 2 and up to 100 records per Duplicate Group.

This is a safety precaution to make sure that if you use a duplicate matching filter that is too broad in Step 1, so you do not accidentally merge many non-duplicate records together.

Maximum records in duplicate group bulk merge


If you select Manual Mode, you'll have complete control over which records are merged together, by selecting them from the Record Viewer in Step 2.

duplicate group selections


In Manual Mode, you can also individually select the Master Record that other records will be merged into on a case-by-case basis.

master record


In Bulk Mode, you set rules for determining the master record automatically in Step 4.

Step 4: Master Selection

Insycle offers extremely flexible controls from determining the master record that all records in a duplicate group will merge into.

You can make your master selection on the record level, such as merging all duplicates in a group into the earliest created record.

Or, you can customize your master selection down to the field level. For example, maybe you want the "Email" field to come from the last modified record, but you want First Name, Last Name, and Phone Number to come from the earliest created record.

Insycle gives you complete control over how data is merged into your master record using advanced rules.

Master Selection By Record

The goal is to pick one record from each Duplicate Group as the master, and the remaining records in that group will get merged into it. For example, if you had 4 records that represent the same person, it would count as 1 Duplicate Group with 4 records, out of which 3 are duplicates that will be merged into 1 master record.

When merging in bulk, the master record from each Duplicate Group is determined using rules via an elimination process. For each Duplicate Group, Insycle eliminates records that do not meet the rule criteria. Rules are read in order, from top to bottom. So if a Master Record meets the first criteria (Highest marketing emails clicked in the example below) the others on the list will not impact the selected Master Record.

bulk master selection


For example, let's say that we have 4 records that represent the same person — Britany Calvert. That is 1 Duplicate Group that consists of 4 records.

Here is the data that we have for this duplicate group:

Record ID

Name

Email

Contact Owner

1000

Britany Calvert

britany@acme.com

 

1001

Britany Calvert

bcalvert@acme.com

 

1002

Britany Calvert

britany.alvert@acme.co.uk

Jane Adams

1003

Britany Calvert

britany.alvert@gmail.com

 


We haven't sent any emails to Britany yet, so when Insycle processes the first 3 rules — emails clicked, email opened, and the email bounced — Insycle cannot eliminate any record because they all have the same value of zero.

In the next rule about contact owner, records 1000, 1001, and 1003 are eliminated since no contact owner exists for those records. Now there is only one record remaining 1002 and therefore that's the master record.

You can add or remove rules using standard and custom fields. And since the order of the rules is important, you can also reorder the rules as needed using the arrow buttons.

Priority Match vs. Absolute Match

On the right-hand side of Step 4, you will find the option to select Priority and Absolute Match.

priority absolute match
  • Priority Match. Go down the master selection rules in order, one by one. As soon as a record meets one of the criteria, it becomes the master record. A vast majority of duplicate templates should use Priority Match.
  • Absolute Match. The master record must meet all of the listed rules in the "Record" tab in Step 4. If a record does not match every rule listed, no master record will be identified. Absolute match is less commonly used, but can be useful in specific use cases.

Please Note

Choosing Absolute Match can often result in no master record being identified since the record has to match every rule listed.

The vast majority of the time, you should select Priority Match.

Master Selection By Field

Insycle also gives you the ability to pick which data to retain in the master record on a field-by-field basis. Then, the master record will be decided based on the rules that you set but may pull values in fields from several different records from the duplicate group, based on the rules that you set in the Fields tab.

For instance, you might want the First Name and Last Name fields to be retained from the earliest created record in the duplicate group. But, you may want the email retained to come from the most recently updated record. With Insycle, you can. In this situation, the master record will keep all values for all additional fields, but the First Name, Last Name, and Email will be pulled from other records in the group based on your settings.

Or, you may want to retain the "owner" from the record that was created first.

Let's take a look:

By-field duplicate feature


You have multiple options for determining criteria:

  • From master record: The value for this field will always be retained from the master record that you set in the Record tab in this step.
  • From record where value: For example, retain the data in the Employee # field that is associated with the record with the most employees.
  • From record based on other field: Retain a field based on criteria for a secondary field. In the example above, we retain the data in the Contact Owner field from the record with the earliest create date.
  • Combine and append all values: Additionally, you can append data into master selection fields as well. This is useful for Notes fields or multi-select picklist fields.

In this example, we have set by-field master selection rules for 5 different fields; Annual Revenue, Employees, Contact owner, Lifecycle Stage, and My Custom Field.

In the case of Lifecycle Stage, there are two rules. Insycle will look at the first rule in the list first, and if it finds a record that matches the criteria, the second rule will be ignored. In the example above, if Insycle were to find a record in the duplicate group that had the "Lifecycle Stage" of "Customer," then the following rule looking for "SQL" would be ignored.

Using field-by-field master selection rules gives you unparalleled control over what data is retained in your master record during a bulk merge.

Step 5: Preview Or Update

Now, with our Filter and Master Record selection in place, we can run the Merge Duplicates module to update our selected records.

Click the Review button which will open up a pop, where you can choose between "Preview" mode which simulates the deduplication process for analysis purposes without modifying any data in HubSpot CRM, and "Update" mode which updates the data in the CRM.

merge duplicates preview and review


Initially, it is a good idea to run in Preview Mode and ensure that your deduplication template is working as you would expect. Once you have confirmed that it is, then you can update the live data in your database.

Click the "Next" button to go to the Notify screen, where you can select who will receive an email report of the Preview or Update. You can also add additional context here.

share report

 

Finally, click the "Next" button to choose whether you will schedule this deduplication process to run automatically on a set schedule, or one-time.

merge duplicates automation


Here, you can choose to schedule this operation to run hourly, daily, weekly, or monthly. When scheduled, the Module will automatically run on your desired schedule and email the reports after it has run. You can view all scheduled automations on the “Automations” page on your dashboard.

Reporting

The CSV report generated provides a complete look at your entire merge operation, down to individual duplicate groups.

In the "Result" column, the "Master (After)" for each duplicate group will show the data the final master record will contain, including master selection settings both for by-record and by-field configurations. You can see the entire duplicate group and the data retained in the master record.

To correlate merged records, you can see the "Duplicate Group ID" field in any report generated in the Merge Duplicates module. This is also the Record ID that all other records were merged into. The Duplicate Group ID will be identical for the entire duplicate group that was successfully merged into the master record.

CSV file


When a field says "(Default)," it means that the CRM will be using its default processes for dealing with the field—this is typically seen for blank fields, system IDs, and other specific situations.

Pro Tip

Every field that is included in Step 4: Master Selection is also included in the CSV report, and so you may want to specify additional fields to get more context about the records in the CSV report. For example, add "Industry exists" as the last rule to have the "Industry" field included in the CSV report.

Once you are satisfied with how the operation runs, you can save your settings as a template for future use by clicking on the "+" and "Save" buttons on the Template Menu.

templates

 

HubSpot Merge Logic

Here is what happens when you bulk merge HubSpot duplicates in Insycle.

Contacts

  1. Email: The email address from the master record becomes the primary, and the duplicate email addresses are added as additional email addresses.
  2. Activities (notes, emails, tasks, etc): Reassigned from the duplicates to the master.
  3. Deals: Reassigned from the duplicates to the master.
  4. Fields: Use Master Selection By Field to determine what data is retained in the master record on a field-by-field basis. By default, the most recently updated value becomes the present value, all other values are available in the history, see HubSpot's merge contacts help article).

Companies, Deals, Custom Objects

  1. Contacts: Reassigned from the duplicates to the master.
  2. Deals: Reassigned from the duplicates to the master.
  3. Activities (notes, emails, tasks, etc): Reassigned from the duplicates to the master
  4. Domains: Copied from the duplicates into the master and appended as secondary domains, to avoid future duplicates with the same domain.
  5. Fields: Use Master Selection By Field to determine what data is retained in the master record on a field-by-field basis. By default, retained from the master. When a value is empty in the master it picks a non-empty value from the most recently updated duplicate.

Customized Merge Logic

For situations where you need more granular customization for picking duplicate records to include - or exclude - from the deduplication process, or for picking the master record, and there are no common rules you can apply for all or some of the records, you can customize bulk deduplication using exclusions and pre-defined masters.

Separately, it is also possible to customize the merge logic behavior. For example, to instruct Insycle to copy values from field A to field B as part of the merge, or combine multiple fields into one. Customized merge logic requires an Enterprise plan. Please reach out via chat to discuss your specific requirements.

To learn more, see Customize Bulk Deduplication Using Exclusions and Pre-Defined Masters.

Preview Changes Before They Go Live

With Insycle's Merge Duplicates module, you can preview the changes that you are making to your data before those changes are pushed to your live database. That way, you can check to ensure your deduplication template is working as expected.

Automation

You can set up ongoing data maintenance automation with Insycle on the module level, using Recipes, or integrating with Workflows. You can view all scheduled automations on the “Automations” page on your dashboard.

Audit Trail and History

The Activity Tracker lets you review all changes made through Insycle. At any time you can download a CSV report of the operation and records affected.

Frequently Asked Questions

I already have a list of duplicates, can Insycle bulk merge them?

Yes. You can use a customized list of duplicates, as detailed in this help article, and use Insycle to deduplicate the list in bulk. Include ID numbers from your connected CRM in your CSV.

Why can I only process 50 duplicate groups at a time?

Insycle can process thousands of duplicate groups in one operation. Potentially, you could deduplicate your entire database in one operation. Insycle shows 50 records on the module screen as a preview. You do not necessarily need to select the duplicate groups individually because you will be able to select all relevant records after clicking the "Review" button to begin the operation.

deduplicate in bulk


How do I ensure that I am not merging non-duplicate records together?

Currently, there are two ways to make sure that the records that you are merging are indeed duplicate records.

First, always run your deduplication templates in Preview Mode before running them in Update Mode. This produces a CSV that shows you how your records would have been merged. Then you can ensure that your Merge Duplicates template is working as expected and not merging non-duplicate records together.

Additionally, you can reduce risk when merging duplicates by narrowing your filter in step 1. Use Exact Match instead of Similar Match. Make sure that you are using true uniquely identifying fields—first name, last name, email, and phone number are popular choices. The more tightly defined your filter is, the less likely you are to merge non-duplicate records.

How many duplicates can I merge into one master record?

You can merge as many duplicates as you would like into one single master record. This setting is available in Step 3 of the Merge Duplicates module.

maximum merge


This setting is here to ensure that if you set too broad of a filter, you do not accidentally merge many non-duplicate records together on accident. This is precautionary, but you can change the setting as needed to match the size of your duplicate groups. If you are going to set this number at a high level, it is a good idea to run in Preview Mode first to make sure your deduplication template is operating as you intend.

Does the amount of identified and merged records have any limits in the paid subscription?

All plans include unlimited usage, unlimited users, and unlimited operations, see the pricing page. Only during the free trial is the number of records updated, cleansed, or merged capped at 500 records.

Can I select which data is retained in my master record on a field-by-field basis?

Yes, Insycle allows you to select which fields are retained in the master record based on pre-determined rules. To learn more please see this help article.

By-field duplicate feature


I need to exclude some records from deduplication. Can I do that?

Yes. You can exclude records from deduplication by including a "Deduplication Exclude" field in your CSV, as detailed in this help article.

My team needs to review and approve the master, can I accommodate that with Insycle?

Yes. You can manually approve master records and mark them in a CSV, then use Insycle to bulk deduplicate down to those master records. Consult with this help article to learn more.

Alternatively, you can run the Merge Duplicates module in preview mode, then deliver the preview CSV that Insycle generates. This CSV shows exactly how the merge would take place but does not update your live data. Then they can approve the merge based on this document, before running Merge Duplicates in Update Mode.

Additionally, you can choose the master manually by selecting the "Manual" tab in Step 4.

master record


Do my matching fields have to match each other exactly?

No, Insycle does allow you to use similar matching to identify duplicates. Please read more about that in this help article.

Insycle is having trouble determining a master record. What could be causing this issue?

First, check Step 4 to ensure that you have "Priority Match" selected and not "Absolute Match."

priority vs absolute match

With Priority Match, the rules are processed in order and your master record only has to match one rule. In Absolute Match, your master record would have to meet all of the criteria in the "Records" tab in Step 4.

The vast majority of the time it is best to be in Priority Match.

How can I find duplicates when one field is empty?

Insycle requires the values to match in the "Step 1: Find Duplicates" step, and if the value for the matching fields listed in step 1 is blank, Insycle will not include those records in the deduplication.

The workaround to having an empty matching value is to remove that condition/field from Step 1 and add it in Step 4. Then, the field will be included in the report generated for you to review and identify when duplicates weren't merged due to a missing field.

For example, let's say in Step 1 you have the matching fields: First Name, Last Name, Phone Number. Sometimes, the Phone Number field may be empty.

Remove the Phone Number Field, and add the "Phone Number Exists" as the last rule in Step 4. Now, Insycle will only use First Name and Last Name to identify the duplicates, but the Phone Number field will be included in the report for those contacts. Then you can use a customized approach to deduplication to merge records that are missing the phone number effectively.

Additional Resources

Related Blog Articles

Related Help Articles

Additional Resources