Merge Duplicates with Blank Fields

Navigate to Data Management > Merge Duplicates, and select the database and record type from the top menu. Then explore the templates for an existing solution that may be close to what you need.

Configure Matching Rules

To find duplicates, you need to define how to match records. Step 1 looks through the records in your database, examining the fields that you specify for matches. Each row is for a field you want to look at for repeated values.

Configure the match rules in Step 1 under the Simple tab:

1. Field Name - Select a combination of unique identifier fields—data that is unlikely to be shared by any other record unless it is a duplicate. You want fields that, in combination, give a high degree of certainty that the matched records really represent the same entity.
2. Comparison Rule - Define what kind of likeness to look for when deciding if field values should be considered a match. It's a good idea to start with Exact Match and begin with easy-to-find duplicates.
3. Ignored - Specify parts of a field value to ignore, such as specific text, whitespace, or characters. These won’t be considered as part of the matching process.
4. Match Parts - Define specific portions of the field value to compare.

See the Advanced How-Tos for more details on using these fields.

The example above will identify duplicates by looking for records with the exact same values in the First Name, Last Name, and Phone Number fields. People with the same first name AND last name AND phone number will show as possible duplicates.

Commonly used matching fields in deduplication include:

• First name and last name
• Company name
• Email
• Email domain
• Company website
• Phone number
• ID numbers

Allowing a Field to Have an Empty Value

When using two or more fields to identify duplicates, records can still be considered matches even if one or all of the field values are blank. You just need to specify which field(s) allow a blank value.

Under Step 1, click the Conditions tab.

All the matching fields you included will automatically appear with the Value Required condition selected. Change the condition to Empty Allowed in Any Record to allow empty values for certain fields. Make sure at least one of the fields you keep as Value Required is a reliable unique identifier to ensure the records are really duplicates.

For example, on the Simple tab, you may have the matching fields: First Name, Last Name, and Phone Number. But on some of your records, the Phone Number field may be empty. You want all records with the same name, same phone number, and no phone number to be considered duplicates.

Click the Find button, and Insycle will generate a list of duplicates for you to review. 

Records that have the same values in the fields specified in Step 1 are considered matches. When two or more records represent the same entity (person, company, or other) based on your matching rules, they are clustered together into duplicate groups. Each duplicate group shows the total number of records that were identified as duplicates based on your settings. For example, if you had four records for the same person, it would count as one duplicate group with four duplicate records.

In this view, each row represents a duplicate group. When you click a row, it expands and shows the records that belong to that group. Records that have blank values in the fields you allowed will be included. 

Explore the record data in the duplicate groups. Double-check to make sure that the fields you set up in Step 1 are showing what you expected. 

Using the gear button on the right, add more fields to the view to get more context for analyzing the records.

Looking at the record details in several duplicate groups, decide the best way to determine which is the master record the other records will be merged into. Learn more about common practices for picking a master record.

In Step 3, you define how your duplicates are merged. You'll choose your operation mode, set rules for which record becomes the master, and optionally control which field values are retained after the merge. 

Choose Your Operation Mode

At the top of Step 3, select Bulk or Manual mode:

• Bulk mode (recommended) — Automatically merges all duplicate groups according to your master selection rules. This is the right choice for most deduplication operations and is required for Templates, Recipes, and Automation.
• Manual mode — Lets you select specific records to merge one at a time from the Record Viewer. Reserve this for high-value records or complex situations that require hands-on review of each merge.

Most users should select Bulk mode and proceed.

Configure Rules to Automatically Select the Master Record (Master Tab)

On the Master tab, define which record becomes the master in each duplicate group. Insycle evaluates your rules in order from top to bottom — the first record that uniquely matches a rule becomes the master.

For example, you might prioritize records with active owners, or select the contact with the highest engagement score, or the company with the most associated deals.

Learn more about these options in the Master Selection Rules reference in the Module Overview article.

Customize Field Data Retention (Fields Tab)

On the Fields tab, control which field values end up in the master record after the merge. By default, Insycle keeps the master record's values and fills any empty fields with values from the most recently updated duplicate. 

You only need to visit the Fields tab if you have fields that require special handling—for example, combining notes from all duplicates, preserving a specific lifecycle-stage value, or retaining record IDs in a custom tracking field. Otherwise, the default behavior is a reasonable starting point for most merges.

Learn more about these options in the Field Data Retention Rules reference in the Module Overview article.

Select the Merge Method (HubSpot & Salesforce Only)

The Merge tab lets you choose between Native and Synthetic merge logic, and control whether empty master fields are automatically filled. See the Method Tab Reference for details.

Before applying any changes to your HubSpot database, Insycle lets you preview exactly what will happen. Always run a preview first.

Preview Merged Changes in CSV Report

Under Step 4, click the Review button and select Preview mode.

Click the Next button to go to the Notify screen, where you can select recipients for the email report and add additional context.

On the When tab, click the Run Now tab. If you have a large number of duplicate groups, consider limiting the preview to a smaller batch to review more quickly. Then click Run Now to generate the report.

Insycle will generate a preview CSV and send it to your email. Open the CSV file from your email in a spreadsheet application and review the data.

There's a row for each record, which includes:

• The Result of the action
• A Message with details
• The Duplicate Group ID indicates which records will be merged together
• A record Status that identifies which were picked as master and which were identified as duplicates and merged into the master. See below for more details
• All fields used to identify the duplicates in Step 1
• The record ID, record name, or email, and Deeplink to the CRM record
• All fields selected for the Record and Fields rules in Step 4. Note that if a field is used on both tabs, it will only appear once in the CSV. 
• Any additional fields added to the Layout tab in Step 2.

 The Status column indicates:

• Duplicate – The record is part of a duplicate group.
• Master – The master record chosen for the duplicate group based on default behavior and your Master rules. Review the selections in this row to determine whether the appropriate records are being chosen.
• Master (After) – For each duplicate group, the Master (After) row shows the values the final record will contain based on your Field rules and the default behavior. 
• Error – If Insycle is not able to determine which record would be the master, an error message will appear here. See the Troubleshooting section below for more details.

If everything looks good, return to Insycle and move forward with applying the changes.

Apply Changes to Your CRM Records

When you're satisfied with the results in your preview, you can apply the merge changes to your CRM.

Under Step 4, click the Review button again, then select Update mode.

On the When tab, you should use Run Now the first time you apply these changes to the CRM. If you have a large number of records, you may want to do a smaller batch to review the results in your CRM.

After you've seen the results in your CRM and are satisfied with how the operation runs, you can save your configuration as a template and set up automation so this merge operation runs on a set schedule. If you have several templates you'd like to automatically run together, you can create a Recipe.

By automating with a template, you'll save time and ensure your records merge consistently on an ongoing basis.

Save Your Configuration as a Template

Return to the Template menu at the top of the page and click Copy to save your configurations as a new version of whatever template you started with. Then click the pencil to edit your new template name.

Set Up Automation

Under Step 4, click the Review button, and select Update mode.

On the Notify tab, select the send option appropriate for your automation: Always send, Send when errors, or Do not email.

Add any additional recipients who should receive the CSV (and make sure to hit Enter after each address). You can also provide additional context in the message subject or body.

On the When tab, select Automate and configure the frequency you'd like the template to run. When finished, click Schedule.

You can view all your scheduled automations at any time on the Operations > Automations page.

Bundle into a Recipe

When you have multiple templates that work reliably together — for example, separate templates for contacts, companies, and deals — you can combine them into a Recipe. A Recipe runs your templates in sequence, in the order you define, on a consistent schedule.

To back up select fields during your merge setup, include additional fields in the CSV report. This ensures you have the data later for undoing changes or for general review.

Under Step 2 in the Merge Duplicates module, click the gear icon in the header.

On the Layout tab, add any extra fields to the Visible Fields list.

The fields will be included in the CSV report.

Alternatively, to capture all field data before running a large merge operation, you can export records directly from your CRM to a CSV file. This can preserve all the record details, which can help later for undo and/or just for general review.

If your database is large or you're getting an overwhelming number of duplicate groups, use the Filter button in Step 1 to narrow down the records Insycle analyzes. Filtering to a subset — for example, contacts created in the last 30 days, or companies in a specific region — makes it easier to validate your configuration before running it across your full database. A filtered dataset also processes faster.

To add a filter, click the Filter button in Step 1, choose a field, select a condition, and set a value. The filter is applied before the matching step runs.

Most of the options in the Field dropdown match the fields that are found in your HubSpot records, and for contact records, there are three additional options related to the Email value: 

• Email Username: The portion of the email address before the “@.” For example, if the email address were “maria@acmewidgets.com,” the username value would be “maria.” 
• Free Email Provider Domain: Choose True to filter out records where the email domain is Gmail, Hotmail, Yahoo, or any of about 10,000 other free email providers. This filter helps ensure these are real clients, or can determine which record is the legitimate one, because it’s most likely that customer companies aren't using free Gmail accounts (though a contact may have accidentally emailed us from one at some point). 
• Email Top-Level Domain: The top-level domain (TLD) is everything that follows the final dot of a domain name. For example, in the domain name acmewidgets.com, '.com' is the TLD. Some other popular TLDs include '.org', '.uk', and '.edu'. 

When setting up your matching fields for the first time, start with your highest-confidence criteria — fields that, together, are very unlikely to match unless the records are true duplicates. Exact Match on email address plus first and last name is a reliable starting point for contacts. Domain Name is a strong starting point for companies.

Once you're confident your configuration is catching true duplicates without pulling in false positives, you can broaden your criteria — for example, by adding Similar Match to catch typos and slight variations: 

Or, by using Related Fields to compare values across two fields that contain similar data (such as Email and Additional Email).

Insycle evaluates your Master tab rules from Step 3 in order, eliminating records that don't match each rule until only one remains. If multiple records still match after all rules are evaluated, no master can be determined, and the group will show an error in the CSV.

For example, imagine having four duplicate records for the same contact. (In this image, we are examining the records in the Grid Edit module.)

In the Merge Duplicates module, you have configured the first three Master rules based on email engagement metrics, but all four records have identical values of zero, so no records are eliminated. Your fourth rule checks for an active Contact Owner — three records have no owner, so they're eliminated. The one remaining record becomes the master.

This is why rule order matters. Place your most reliably differentiating rules — like record owner, lifecycle stage, or engagement activity — where they're most likely to yield a clear winner. If you're frequently seeing errors in your CSV, revisit your rule order and consider adding a tiebreaker rule, such as earliest Create Date or latest Last Modified Date, as a final fallback.

In this CSV report example, you can see that the one record with an active owner was chosen as the master.

For situations where there are no common rules you can apply for identifying duplicates for all or some of your records, you may need more granular control over which records are included or excluded from the process.

Bulk Solutions

There are two options for doing this in bulk. You can upload a CSV file of known duplicate record ID pairs directly into the Merge Duplicates module using the CSV tab in Step 1. This lets you bypass field-based duplicate detection entirely and work from a list of specific record pairs you've already identified. From there, you continue through Steps 2–4 to configure master selection rules and field retention settings, and then run the merge. 

For more complex scenarios — such as designating master records or excluding specific records from deduplication using custom attributes — you can also use the Magical Import module in combination with Merge Duplicates for complete control over the process. 

Learn how to customize merging duplicates in bulk using a CSV.

Single Record Solution

To do this one record at a time, you can use Manual mode of the Merge Duplicates module.

In Manual mode, you have complete control over which records are merged together by selecting them from the Record Viewer. Manual mode should be reserved for cases that require a careful, controlled process. Learn more about merging duplicates in Manual mode.

You have duplicate records identified by Insycle, but not all are merging into the master. Check how many duplicates are in the affected groups. If you have duplicate groups with more than 5 records, you may want to change the Exclusions value at the bottom of Step 3 on the Master tab: Skip duplicate groups with more than 5 records per group to ensure you can get them all.

This setting is intended to prevent overly broad matching criteria from accidentally merging unrelated records. You can adjust this threshold up to 100 records per group.

If the Result is "Failed" and the Message column of the CSV report displays this text:

Change rules in Step 3 'Master Selection'. Failed to pick master record because multiple records (X) meet the selection criteria. In 'Master Selection', change, add, or reorder the rules such that only one record matches (if cannot determine master based on field values, use 'Record ID is lowest' as the last rule).

This means Insycle could not determine which record in the duplicate group should become the master — none of the records meet more of the rules than the others.

There are a few things you can try to resolve this:

1. Under Step 3, on the Master tab, experiment with reordering or adding rules based on fields that are likely to have unique values across your records. Think about what matters most to your business — the fields that would make one record clearly more valuable or authoritative than another. For example:

   • A field that stores an ID linked to an external system, such as an ERP, data warehouse, or proprietary internal tool — the record with that ID populated is likely the one your other systems depend on
   • The record with the highest number of associated records (contacts, deals, activities) — indicating it has accumulated more history and relationships
   • A field indicating CRM ownership or account status — for example, a record marked as "Customer" vs. "Lead," or one with an active owner assigned
   • A field that tracks revenue, subscription tier, or engagement level — the record with higher values may represent the more complete or current entry
   • A custom field your team uses to flag record quality or data source reliability

   The goal is to surface the fields that reflect record importance for your specific operations, not just default system fields.

2. Also on the Step 3 Master tab, confirm that By Priority is selected rather than Absolute.

   With By Priority, a record only needs to match one rule to be selected as master. With Absolute, a record must meet all rule criteria, which makes it much harder for any single record to qualify.

   In most cases, By Priority is the right choice. If By Priority is already selected and the error persists, none of the records in the group meet any of the criteria more than the others. Return to the Master tab and continue adding or reordering rules, focusing on the business-critical fields described in #1 above.

   

3. As a last resort, add a rule on the Master tab that uses Record ID is lowest, or Create Date is earliest. These rules will always produce a unique winner, ensuring master selection can complete even when other field-based rules cannot differentiate the records.

There are a couple of things to look at that may be causing records to be misidentified as duplicates.

First, you may need a better unique identifier. Under Step 1, if you use only fields that can correctly contain the same values across multiple records, they aren't unique identifiers. In this case, you are likely to identify unrelated records as duplicates and may accidentally merge them.

Unique identifiers are data unlikely to be shared by any other record unless they represent the same underlying entity. Common fields used for deduplication include phone numbers, email addresses, mailing addresses, and ID numbers.

Second, this may indicate the Comparison Rule under Step 1 is too broad. Try using the Exact Match comparison rule instead of the Similar Match rule. Similar Match looks for values that may be close, even with a one-character difference (e.g., a typo), broadening the search. 

Remember to always run your deduplication in Preview Mode to confirm it's working as expected before running it in Update Mode and applying the changes to your CRM records.

Most of the time, when Insycle can't find duplicates, it is due to your matching rules in Step 1. It is important to analyze the underlying data to better understand how to set up your rules. A helpful exercise for contacts is to set up your matching filters to find exact matches on only First Name and Last Name. For companies, you can use Company Name.

When you click Find, these rules can show you a broad overview of what duplicates are potentially in your database and what fields might be useful to include in your matching fields. These settings are just for discovery and should not be used for a final merge operation; many people can have the same first and last names and are not duplicates. 

To get further context, on Step 2, click the layout gear button on the right side of the title bar. Here, you can add any field from your database as a column in the duplicate group review to better understand the data in these records. These will also appear in your CSV report. 

It can take Insycle a while to find and match duplicates when the fields used to identify them have very long values. The longer the values, the longer it takes Insycle to process the data and generate the results. This might come up when looking for matches based on long ID numbers, LinkedIn bio links, or other URLs with long strings attached (ex, https://www.linkedin.com/in/svadin%C3%ADr-n%C4%9Bmec-1234b31a3/).

You can speed this up by limiting how much of the value Insycle looks at.

If the beginning or ending portion of the values are all unique, you can limit the comparison to the first or last several characters using the Match Parts parameter under Step 1. 

Or use the Ignore Text (Substrings) parameter, then click the Terms button.

On the Ignored Text tab of the popup, add the common portion of the URL or text string.

Field rules in Step 3 can only update writable fields. Read-only fields are not shown in the Fields tab dropdown because the CRM prevents these properties from being written back. Insycle excludes them from the field selection options to prevent errors.

While you cannot create field rules for read-only properties in the Fields tab, you can use them in the Master tab. Read-only fields, such as "Last Activity Date," can be used to make merge decisions without attempting to modify those properties.

You can also use read-only properties for filtering and matching in Step 1. Read-only fields are also available when reviewing data in Step 2 or in the CSV report.

To identify which fields are read-only, use the Cleanse Data module. Navigate to Data Management > Cleanse Data, then select a database and record type from the top menu. You can search for a specific field by name or click the Writable heading in the table to sort and review all writable or non-writable fields. A checked box in the Writable column indicates the field is writable.

In cases where two different phone numbers and empty values from the duplicates are present, they will match as different duplicate groups. Insycle decides how to match the blank field automatically.

In this example (viewed in the Grid Edit module), Record 1 has the phone number 888-555-1200, Record 2 has no phone, and Records 3 and 4 have the phone number 888-555-1212.

When Insycle matches the duplicates, it will either create two duplicate groups:

• The first duplicate group could be made up of Records 1 and 2
• The second duplicate group could be made up of Records 3 and 4

Or it will match Records 2, 3, and 4 into a single duplicate group, while Record 1 remains on its own.

Currently, neither HubSpot nor Salesforce provides an automated way to prioritize active owners during the merge process. You'd need to verify owner status manually for each merge operation.

However, Insycle's Merge Duplicates module includes an option to prioritize an active owner. 

First, you could add a Master rule under Step 3 to tell Insycle to select the record from each group with an active owner as the master record.

Add a rule with the following parameters:

• Field: Record owner
• Condition: active user

Second, you could create a Field rule to retain the owner who is an active user.

Add a rule with the following parameters:

• Field: Owner
• Criteria: From record where value
• Condition: active user

If the Message column of the CSV report displays this error:

Change rules in Step 3 'Master Selection'. Failed to pick master record because multiple records (X) meet the selection criteria. In 'Master Selection', change, add, or reorder the rules such that only one record matches (if cannot determine master based on field values, use 'Record ID is lowest' as the last rule).

None of the records meet more of the rules than the others do.

There are a couple of things you can try to resolve this:

1. On the Master tab in Step 3, experiment with reordering or adding additional fields that are likely to have unique values.
2. At the bottom of the Master tab in Step 3, ensure By Priority is selected, not Absolute.
   With By Priority, your master record only has to match one rule. Using Absolute, your master record would have to meet all of the rule criteria. In most cases, it is best to select By Priority.
   If By Priority was used, then none of the records in the duplicate group meet any of the criteria on the list more than the others. In this case, you'll need to experiment, reordering or adding additional rules for fields likely to have unique values.
3. As a last resort, you can add a rule on the Master tab in Step 3 that says Record ID is lowest, or Create Date is earliest.

There are two ways to ensure the records you are merging are indeed duplicates.

First, always run your deduplication templates in Preview Mode before running them in Update Mode. This produces a CSV file showing how your records would have been merged. Then you can ensure that your Merge Duplicates template is working as expected and not merging non-duplicate records together.

Additionally, to ensure a smooth merge, consider narrowing the matching settings in Step 1. Try the Exact Match Comparison Rule instead of Similar Match. Then make sure that you are using actual, uniquely identifying fields—first name, last name, email, and phone number are popular choices. The more tightly defined your filter is, the less likely you are to merge non-duplicate records.

Yes, Insycle allows you to select which field data to retain in the master record using the Fields tab in Step 3. See the Merge Duplicates module field data retention rules reference for more details.

When you initiate a merge operation in Insycle, you will see the estimated time to complete. If you notice that the operation is taking longer than expected, there are several factors to consider:

• If you are using a synthetic merge, this will take longer to run than a native merge.
• Complex processes, like merges and associations, can take a while. This is because each backend task—from updating associations to changing properties—contributes to the total processing time. For instance, when merging three deals into one, 20-40 operations may be required to update and reparent all related activities from the duplicates into the master.
• Throughput is measured across all operations; when multiple operations run in parallel, they increase processing load and slow each other down. To optimize processing time, run only one operation at a time.

There are two main ways to monitor the progress of a long-running operation:

Option One: Periodically check the Activity Tracker

The operation will appear in the Activity Tracker once it is complete.

Option Two: Periodically refresh the template for the operation

Open the template for the operation. Under Step 1, click the Find button.  As the merge continues, you should see the number of duplicate records gradually decrease in the Record Viewer, which indicates the process is actively working through them.

Look at the number of results in the Record Viewer.

Wait a few minutes and refresh the page. You should see the number of results decrease as Insycle processes the operation.

Yes, there are several ways to share details and get approval before merging duplicates.

You can manually approve master records and mark them in a CSV file, then use Insycle to bulk deduplicate into those master records. See the Customize Bulk Deduplication Using Exclusions and Pre-Defined Masters article to learn more.

Or, you can run the Merge Duplicates module in Preview Mode and then deliver the preview CSV that Insycle generates. The CSV report includes your entire merge operation down to individual duplicate groups, but does not update your live data. Then your team can approve the merge based on this report, before running Merge Duplicates in Update Mode.

Additionally, team members can review duplicates and manually select the master for each record under Step 3 by selecting Manual mode. Review the Manually Merge Duplicates article for more details.

No, matching fields do not need to match exactly. The Similar Match Comparison Rule in Step 1 acts as a fuzzy match, looking for values that may be close but differ by only one character (e.g., a typo), and broadens the search.

This search behaves like when Google shows results for a slightly different term or says, “Did you mean...” For example, if an Email of “huey@coahulldu.co” is found, it could include records with the values “hueyy@coahulldu.co" or "hue.y@coahulldu.co” as a match.

You should be careful when using Similar Match as the looser criteria can incorrectly identify non-duplicates as duplicates. 

Review the Understanding Similar Matching best practices for more details.

By default, merged records retain the value from the master record. If you want to keep a specific value from a read-only field that's not in the master, you have two options:

1. Copy to a writable custom property: In your CRM, create a custom field to copy the read-only field value into. Then run a Copy operation in the Transform Data module before performing the merge. Finally, set up Merge Duplicates rules on the Fields tab of Step 3 to retain the value of that custom property that meets your criteria.

2. Use master selection rules: Set the master selection rules on the Master tab of Step 3 to retain the record with the read-only value you want to keep. This guarantees the correct record becomes the master without trying to change the read-only property itself.

You can merge up to 100 duplicate records into a single master record.

By default, Insycle skips any duplicate group containing more than 5 records. If you have duplicate groups with more than 5 records, you may want to adjust the Exclusions setting at the bottom of the Master tab in Step 3. The Skip duplicate groups with more than X records per group setting can be increased up to 100 records per group to ensure all duplicates in large groups are processed.

This default limit is a precaution to prevent duplicate matching criteria that are too broad in Step 1 from accidentally merging many non-duplicate records. If you plan to set this number higher, it is a good idea to run Preview Mode first to ensure your deduplication template is operating as you intend.

All plans include unlimited usage, unlimited users, and unlimited operations. During the free trial, there is a cap of 500 records updated, cleansed, or merged. See the pricing page for more details.