Duplicate data poses serious problems for companies of any size.

Duplicate records inhibit your marketing team from effectively segmenting and personalizing your communications. Sales teams step on each other's toes and lack vital context in conversations. Support teams miss important information, and analysis and reporting are skewed.

Insycle helps you to merge duplicate contacts, companies, and deals—flexibly and powerfully.

How It Works

Insycle analyzes your database and identifies duplicates with flexible matching rules, using any field in your database, to help you identify and merge more duplicates in bulk.

Once Insycle identifies the duplicate records, you set rules for determining the master record that other duplicates will be merged into—such as the first record created, record with the most email opens, or any other attribute that would be relevant. You can also set merging logic on a field-by-field basis.

Insycle provides a complete report of what was identified as a duplicate, what was merged, and what the outcome was in your master record.

Quick Summary

Insycle guides you through the process of deduplication in five easy steps.

  • Step 1: Use a filter to identify duplicates, such as contacts with matching names and companies.
  • Step 2: Review those duplicates and set advanced rules for determining what data is retained in the master record after the duplicates are merged.
  • Step 3: Choose whether to bulk merge automatically or manually merge.
  • Step 4: Set rules for choosing your master record and retaining data during the merge.
  • Step 5: Run deduplication processes in bulk, and automate your deduplication templates to keep your database free from duplicates moving forward.

Step 1: Find Duplicates

First, navigate to the Merge Duplicates module. From the top of the screen, choose whether you will be deduplicating contacts, companies, deals, or custom objects.

For a quick start, explore the default templates. There might be a template that already does exactly what you need. You can also use an existing template as a starting point and adjust it to your requirements.

Deduplication Templates

In order to find duplicates, you need to define how to match records.

For example, to find duplicate people you might match using First Name, Last Name, and Email Domain fields. In that example, people with the same first name AND last name AND email domain will show as possible duplicates.

Insycle gives you deep control over how to match the values of those fields:

 Exact Match vs. Similar Match

  • Exact Match looks for records that match exactly, with no differences from one record to the next. This is great for identifying surface-level duplicates while remaining highly accurate.
  • Similar Match analyzes data and identifies customer records that are similar, but not exact matches. It works by analyzing the “closeness” of two different data points.

Closeness is determined by measuring the number of changes necessary to make the two data points match.

  • insertion: bar → barn
  • deletion: barnbar
  • substitution: barnbark

Ignored

Ignored allows you to pick parts of the value to ignore. For example, ignore symbols and whitespace when comparing phone numbers, or ignore HTTP, www, subdomain, or top-level domain (.com vs co.uk) when comparing websites or email domains. This is great for catching more advanced duplicates.

Match Parts

Match Parts allows you to compare specific portions of a particular field. For instance, you can pick whether to compare the entire value, any two words, just the first five letters, or a certain number of final characters, etc. This is great for specific use cases.

Adding Match Fields

You can add fields by clicking the "+" button, or remove fields by clicking the X button. When you click Find, Insycle will identify duplicate records based on the matching filter.

field-button

When finished, click the Find button and Insycle will generate a list of duplicates for you to review.

Step 2: Review Duplicates

In this step, Insycle allows you to preview the identified duplicates and double-check to make sure that your filter is producing accurate results. That way, you can avoid merging any records that are not actually duplicates.

Review duplicates

After clicking the Find button, the Record Viewer shows the identified duplicate groups. A duplicate group is when two or more records represent the same entity (person, company, or other). For example, if you had four records for the same person, it would count as one duplicate group with four records.

In this view, each row represents a duplicate group. When you click a row, it expands and shows the records that belong to that group.

Duplicate group

Here, you can dig deep into your identified duplicate groups, down into the individual records. This allows you to check for accuracy and decide the best way to determine which master record other records will be merged into in Step 4.

You can add more columns to the view using the ⚙ button on the right, in order to get more context and analyze the records.

Step 3: Choose Operation

In Step 3, you can merge duplicates in bulk using rules, or manually.

When merging in bulk, Insycle limits the duplicate group size to five and skips groups that contain more records. You can adjust the default value of five records as needed, up to 100 per duplicate group. This is a safety precaution to ensure that if you use a duplicate matching filter that is too broad in Step 1, you do not accidentally merge many non-duplicate records together.

Maximum records in duplicate group bulk merge

If you select Manual Mode, you'll have complete control over which records are merged together by selecting them from the Record Viewer in Step 2. Here, you get a full preview of the record and can choose which records will be merged and which will not.

duplicate group selections

In Bulk Mode, you set rules for determining the master record automatically in Step 4.

Step 4: Master Selection

In Bulk Mode, you set rules for determining the master record automatically when merging. Insycle offers extremely flexible controls for determining the master record when bulk merging.

The goal is to designate one master record from each Duplicate Group, and merge the remaining records in that group into it. For example, if you had four records that represent the same person, they would make up one Duplicate Group with four records, all of which would be merged into one master record.

Insycle gives you complete control over how data is merged into your master record using advanced rules. You can make your master selection on the record level, or on the field level.

If you are merging in Manual Mode, you can also select the master record that other records will be merged into on a case-by-case basis.

master record

Master Selection By Record

When merging in bulk, the master record from each Duplicate Group is determined using rules via an elimination process. For each Duplicate Group, Insycle eliminates records that do not meet the rule criteria.

Rules are read in order, from top to bottom. So if a master record meets the first criteria, the others on the list will not impact the selected master record. In the example below, the first criteria is highest marketing emails clicked.

bulk master selection

Here we have four records that represent the same person—Britany Calvert. That is one Duplicate Group.

Here is the data that we have for this duplicate group:

Record ID

Name

Email

Contact Owner

1000

Britany Calvert

britany@acme.com

 

1001

Britany Calvert

bcalvert@acme.com

 

1002

Britany Calvert

britany.alvert@acme.co.uk

Jane Adams

1003

Britany Calvert

britany.alvert@gmail.com

 

We haven't sent any emails to Britany yet. That means that when Insycle processes the first three rules—emails clicked, emails opened, and emails bounced—it cannot eliminate any record because they all have no value in the field.

In the next rule about contact owner, records 1000, 1001, and 1003 are eliminated since no contact owner exists for those records. Now only record 1002 remains, making it the master record.

You can add or remove rules using standard and custom fields. And since the order of the rules is important, you can also reorder the rules as needed using the arrow buttons.

Priority Match vs. Absolute Match

On the right hand side of Step 4, you will find the option to select Priority and Absolute Match.

  • Priority Match: Go down the master selection rules in order, one-by-one. As soon as a record meets one of the criteria, it becomes the master record. The vast majority of duplicate templates should use Priority Match.
  • Absolute Match: The master record must meet all of the listed rules in the Record tab in Step 4. If a record does not match every rule listed, no master record will be identified. Absolute match is less commonly used, but is ideal for specific use cases.

Choosing Absolute Match can often result in no master record being identified, since the record has to match every rule listed. The vast majority of the time, you should select Priority Match.

Master Selection By Field

Insycle also gives you the ability to pick which data to retain in the master record on a field-by-field basis.

For instance, you might want the First Name and Last Name fields to be retained from the earliest created record in the duplicate group. But, you may want the email to come from the most recently updated record. Insycle makes this possible. In this situation, the First Name, Last Name, and Email will be pulled from records in the group based on your settings, but the master record will retain its data for all additional fields.

Let's take a look at another example:

By-field duplicate feature

 

The image above shows that you have multiple options for determining criteria:

  • From master record: The value for this field will always be retained from the master record that you set in the Record tab in this step.
  • From record where value: For example, retain the data in the Employee # field that is associated with the record with the most employees.
  • From record based on other field: Retain a field based on criteria for a secondary field. In the example above, we retain the data in the Contact Owner field from the record with the earliest create date.
  • Combine and append all values: Additionally, you can append data into master selection fields as well. This is useful for Notes fields or multi-select picklist fields.

In this example, we have set master selection rules for five different fields; Annual Revenue, Employees, Contact Owner, Lifecycle Stage, and My Custom Field.

In the case of Lifecycle Stage, there are two rules. Insycle will look at the first rule in the list first, and if it finds a record that matches the criteria, the second rule will be ignored. In the example above, if Insycle were to find a record in the duplicate group that had the Lifecycle Stage of Customer, then it would ignore the following rule looking for SQL.

Using field-by-field master selection rules gives you unparalleled control over what data is retained in your master record during a bulk merge.

 Tip: Add Fields to Report Adding fields to this section also ensures that these fields show up on your report, whether you run in Preview Mode or Update Mode. For instance, if you want to show the Lifecycle Stage of each contact, you could include the Lifecycle Stage field and instruct Insycle to retain data from the master record. Then, your template will run as normal and the field will be included in your run report.

Step 5: Preview Or Update

Now, with our Filter and Master Record selection in place, we can run the Merge Duplicates module to update our selected records.

Click the Review button which will open up a pop up window where you can choose between Preview Mode and Update Mode.

  • Preview Mode simulates the deduplication process for analysis purposes without modifying any data in the CRM, and
  • Update mode runs the operation and updates the data in your CRM.
merge duplicates preview and review

 

You can preview the changes that you are making to your data before those changes are pushed to your live database. That way, you can check to ensure your deduplication template is working as expected.

Click the Next button to go to the Notify screen, where you can select recipients for the email report. You can also add additional context on this screen.

share report

 

Finally, click the Next button to choose whether you will schedule this deduplication process to run automatically on a set schedule, or one time.

merge duplicates automation

 

You can choose to schedule this operation to run hourly, daily, weekly, or monthly. When scheduled, you'll receive emailed reports after each run. You can view all scheduled automations on the Automations page on your dashboard.

Reporting

The CSV report that Insycle generates provides a complete look at your entire merge operation, down to individual duplicate groups.

In the Result column of the report, the Master (After) for each duplicate group will show the data the final master record will contain, including master selection settings both for by-record and by-field configurations. You can see the entire duplicate group and the data retained in the master record.

To correlate merged records, you can see the Duplicate Group ID field in any report generated in the Merge Duplicates module. The Duplicate Group ID matches the Master Record ID that all other records were merged into. The Duplicate Group ID will be identical for the entire duplicate group that was successfully merged into the master record.

CSV file

When a field says "(Default)," it means that the CRM will be using its default processes for dealing with the field. This is typically done for blank fields, system IDs, and other specific situations.

Every field that is included in Step 4: Master Selection is also included in the CSV report, and so you may want to specify additional fields to get more context about the records in the CSV report. For example, add "Industry Exists" as the last rule to have the "Industry" field included in the CSV report.

Once you are satisfied with how the operation runs, you can save your settings as a template for future use by clicking on the + and Save buttons on the Template Menu.

templates

Customizations

For situations where you need more granular customization for picking duplicate records to include—or exclude—from the deduplication process, you can customize bulk deduplication using exclusions and pre-defined masters. Additionally, you can use this process when there are no common rules you can apply to choose the master record.

Separately, it is also possible to customize the merge logic behavior. For example, you can instruct Insycle to copy values from field A to field B as part of the merge, or combine multiple fields into one. Customized merge logic requires an Enterprise plan. Please reach out via chat to discuss your specific requirements.

Preview Changes Before They Go Live

With Insycle's Merge Duplicates module, you can preview the changes that you are making to your data before those changes are pushed to your live database. That way, you can check to ensure your deduplication template is working as expected.

Automation

Insycle enables you to set up automation for ongoing deduplication for HubSpot companies and Salesforce accounts on the module level, using Recipes, or integrating with HubSpot Workflows. You can view all scheduled automations on the Automations page on your dashboard.

You can schedule your Merge Duplicates templates to run on an automated schedule. First, click the Review button at the bottom of the module page. Then, you will go through a three-step process to run the operation. In the third step, you can choose the Automate tab, and schedule your template to run on a set schedule.

merge duplicates salesforce automation

By setting this automated schedule, duplicates will be consistently identified and merged in your database.

Audit Trail and History

The Activity Tracker lets you review all changes made through Insycle. At any time you can download a CSV report of the operation and records affected.

Frequently Asked Questions

I already have a list of duplicates, can Insycle bulk merge them?

Yes. You can make a customized list of duplicates and use Insycle to deduplicate the list in bulk. Include ID numbers from your connected CRM in your CSV.

Why can I only process 50 duplicate groups at a time?

Insycle can process thousands of duplicate groups in one operation. Potentially, you could deduplicate your entire database in one operation. Insycle shows 50 records on the module screen as a preview. You do not necessarily need to select the duplicate groups individually, because you will be able to select all relevant records after clicking the Review button to begin the operation.

deduplicate in bulk

How do I ensure that I am not merging non-duplicate records together?

Currently, there are two ways to make sure that the records that you are merging are indeed duplicate records.

First, always run your deduplication templates in Preview Mode before running them in Update Mode. This produces a CSV that shows you how your records would have been merged. Then you can ensure that your Merge Duplicates template is working as expected and not merging non-duplicate records together.

Additionally, you can reduce risk when merging duplicates by narrowing your filter in Step 1. Use Exact Match instead of Similar Match. Make sure that you are using true uniquely identifying fields—first name, last name, email, and phone number are popular choices. The more tightly defined your filter is, the less likely you are to merge non-duplicate records.

How many duplicates can I merge into one master record?

You can merge as many duplicates as you would like into one single master record. This setting is available in Step 3 of the Merge Duplicates module.

maximum merge

This setting ensures that if you set too broad of a filter, you will not accidentally merge many non-duplicate records together by accident. This is precautionary, but you can change the setting as needed to match the size of your duplicate groups. If you are going to set this number at a high level, it is a good idea to run Preview Mode first to make sure your deduplication template is operating as you intend.

Does the amount of identified and merged records have any limits in the paid subscription?

All plans include unlimited usage, unlimited users, and unlimited operations. See the pricing page for more details. During the free trial, there is a cap of 500 records updated, cleansed, or merged.

Insycle is having trouble determining a master record. What could be causing this issue?

First, check Step 4 to ensure that you have Priority Match selected and not Absolute Match.

priority vs absolute match

With Priority Match, the rules are processed in order and your master record only has to match one rule. In Absolute Match, your master record would have to meet all of the criteria in the Records tab in Step 4.

The vast majority of the time it is best to select Priority Match.

Can I select which data is retained in my master record on a field-by-field basis?

Yes, Insycle allows you to select which fields are retained in the master record based on predetermined rules. To learn more, please see this Bulk Merge Duplicate People, Companies article.

By-field duplicate feature

I need to exclude some records from deduplication. Can I do that?

Yes. You can exclude records from deduplication by including a Deduplication Exclude field in your CSV, as detailed in this Customize Bulk Deduplication Using Exclusions and Pre-Defined Masters article.

My team needs to review and approve the master, can I accommodate that with Insycle?

Yes. You can manually approve master records and mark them in a CSV, then use Insycle to bulk deduplicate down to those master records. Consult with this Customize Bulk Deduplication Using Exclusions and Pre-Defined Masters article to learn more.

Alternatively, you can run the Merge Duplicates module in preview mode, then deliver the preview CSV that Insycle generates. This CSV shows exactly how the merge would take place, but does not update your live data. Then they can approve the merge based on this document, before running Merge Duplicates in Update Mode.

Additionally, you can choose the master manually choosing the "Manual" tab in Step 4.

master record

Do my matching fields have to match each other exactly?

No, Insycle does allow you to use similar matching to identify duplicates, although you should be careful not to catch records that are not actually duplicates using similar matching. Please read more about that in this help article and more about deduplication best practices.

How can I find duplicates when one field is empty?

Insycle requires the values to match Step 1. If the value for the matching fields listed in Step 1 are blank, Insycle will not include those records in the deduplication.

You can work around an empty matching value by removing that condition/field from Step 1 and adding it in Step 4. Then, the field will be included in the report generated for you to review and identify when duplicates weren't merged due to a missing field.

For example, in Step 1 you may have the matching fields: First Name, Last Name, Phone Number. But sometimes, the Phone Number field may be empty.

Remove the Phone Number Field, and add Phone Number Exists as the last rule in Step 4. Now, Insycle will only use First Name and Last Name to identify the duplicates, but the Phone Number field will be included in the report for those contacts. Then you can use a customized approach to deduplication to merge records that are missing the phone number.

Insycle isn't finding any duplicates. What's wrong?

This is a complex question without further context. Most of the time when Insycle is able to find duplicates, it is because of how the matching template is setup. To better understand how to setup your template, it is important that analyze the underlaying data. 

A useful exercise can be to setup your match filters to look at exact match First Name and Last Name. 

first-last-name-duplicate-analyze.png

Of course, you never want to merge duplicates using this template. Many people have the same first and last name but are not duplicates.

Click the Find button, and this template let's you see a broad overview of what duplicates are potentially in your database, and what fields might be useful to include in your matching fields. 

Additional Resources

Related Blog Articles

Related Help Articles

Additional Resources