Pipedrive Merge Duplicates Overview

Insycle supports merging the following Pipedrive record types:

• Contacts
• Organizations
• Deals

You can select the record type that you would like to deduplicate at the top of the module.

When merging duplicates, Insycle goes through the following steps for all Pipedrive record types:

1. Select the master record according to the rules set in Step 3 on the Master tab.
2. Determine field values to retain in the master record based on rules configured in Step 3 on the Fields tab.
3. For fields without specific rules, identify non-empty values using a "fill in the blanks" method:
   • Check whether the master record has any empty fields.
   • If empty, copy values from the most recently updated record in the duplicate group.
4. Update the master record by:
   • Applying the field values chosen in step 2 (from Field rules).
   • Adding the non-empty values identified in step 3 (using "fill in the blanks").
   • Preserving all other existing values in the master record.
5. Merge the duplicate records and reassign all related objects to the master record.
6. Store an audit trail report accessible from the Activity Tracker and send an email notification to the designated recipients.

If you're uncertain about how conflicting field values will merge, you can add those fields to the Preview CSV report by including them in the Layout via the gear icon in Step 2.

This is also useful if there are fields you're worried about backing up; including them in the CSV report allows you to refer back to the values in the CSV sent to your email or downloaded from the Activity Tracker.

Navigate to Data Management > Merge Duplicates, pick the record type, and explore the default templates for a pre-built solution.

Each row in Step 1 is for a field you want to look at for duplicates, along with some parameters on what to look for. You want to choose fields that, in combination, give a high degree of certainty that the matched records are duplicate records. See the Advanced How-Tos below for more details on configuration options.

Step 2 displays all duplicate groups identified in Step 1, allowing you to review and verify the matches before proceeding. Records that match your Step 1 criteria are automatically clustered into duplicate groups — each group represents a single entity and shows how many duplicate records it contains. For example, three records for the same person would appear as a single duplicate group containing three records.

Click the chevron  at the right end of a duplicate group row to expand it and examine the individual records within that group. You can also add fields as columns to provide more context when evaluating the matches by clicking the gear icon.

To exclude a duplicate group from all future deduplication analysis, click the X on the group row. Excluded groups will not appear as duplicates or be included in merges, regardless of the duplicate detection template used. To review and manage excluded groups, click the Exclusions button in the Step 2 header.

Step 3 is where you configure how duplicate groups are consolidated—from choosing your operation mode to defining master selection rules and field retention logic. This step combines operation selection with the merge logic configuration through multiple tabs.

At the top of Step 3, choose between Bulk and Manual modes in the header:

• Bulk mode (recommended) automatically merges all duplicate groups according to the master record selection rules you define in the Master tab. This is the right choice for most deduplication processes, and the only mode that supports templates, Recipes, and automation.
• Manual mode lets you review and select specific records to merge individually from the Record Viewer. Reserve this for high-value records or complex scenarios where you want hands-on control over each merge. Learn more about merging duplicates in Manual mode.

Next, define how all matching duplicate groups should be merged at scale.

Configure Rules to Automatically Select the Master Record

On the Master tab, you define how the duplicate groups should be merged at scale by creating rules that tell Insycle how to select the record from each group to become the master. The master is the record that will remain after the merge.

For example, if you had four records representing the same contact, they would form a duplicate group of four records, which would be merged into a single master record. The other three records no longer exist.

Here, you also select the processing method—By Priority or Absolute. Most de-duplication operations should use By Priority. Learn more about these options in the Advanced How-Tos below.

Configure Rules That Determine Values to Keep

Duplicates may be exact match versions of another record, but often, there is only partial data overlap between them. When data is split between two different records, both may contain unique and important information you'd like to keep.

Under Step 3, click the Fields tab. For each field you want to control the data retention for, select the field and specify where the data for that field should be taken from. This is merged into the master. Any data that is not in the master or not copied to the master is removed.

Learn more about configuring data retention and master record selection.

Preview Merged Changes in CSV Report

Now with the filters and master record set up, you can preview the changes you are making to your data. That way, you can check to ensure your deduplication configuration is working as expected before those changes are pushed to your live database. The CSV report that Insycle generates includes records from all the duplicate groups.

Under Step 4, click the Review button and select Preview mode.

Click the Next button to go to the Notify screen, where you can select recipients for the email report and add additional context.

On the When tab, click the Run Now tab, select which records to apply the change to (in most cases, this will be All), and then click the Run Now button.

Insycle will generate a preview CSV and send it to your email. Open the CSV file from your email in a spreadsheet application and review the data.

There's a row for each record, which includes:

• The Result of the action
• A Message with details
• The Duplicate Group ID which indicates which records will be merged together
• A record Status that identifies which were picked as master and which were identified as duplicates and merged into the master. See below for more details
• All fields used to identify the duplicates in Step 1
• The record ID, record name or email, and Deeplink to the CRM record
• All fields selected for the Master and Fields rules in Step 3. Note that if a field is used on both tabs, it will only appear once in the CSV.
• Any additional fields added to the Layout tab in Step 2

 The Status column indicates:

• Duplicate – The record is part of a duplicate group.
• Master – The master record is chosen for the duplicate group based on default behavior and your Master rules. Review the selections in this row to determine whether the appropriate records are being chosen.
• Master (After) – For each duplicate group, a special row –Master (After)– will appear in the CSV report. The Master (After) row shows the values the final record will contain based on your Field rules and the default behavior.
• Error – If Insycle is not able to determine which record would be the master, an error message will appear here. See the Troubleshooting section below for more details.

If everything looks good, return to Insycle and move forward with applying the changes.

Apply Changes to Your CRM Records

When you're satisfied with the results in your preview, you can apply the merge changes to your CRM.

Under Step 4, click the Review button again, and this time select Update mode.

On the When tab, you should use Run Now the first time you apply these changes to the CRM. If you have a large number of records, you may want to do a smaller batch to review the results in your CRM.

After you've seen the results in the CRM and you are satisfied with how the operation runs, you can save your configuration as a template and set up automation so this merge operation runs on a set schedule. If you have several templates you'd like to run together automatically, you can create a Recipe. 

Return to the Template menu at the top of the page and click Copy to save your configurations as a new version of whatever template you started with. Then click the pencil to edit your new template name.

Under Step 4, click the Review button, and select Update mode.

On the Notify tab, select the send option appropriate for your automation: Always send, Send when errors, or Do not email.

Add any additional recipients who should receive the CSV (and make sure to hit Enter after each address). You can also provide additional context in the message subject or body.

On the When tab, select Automate, and configure the frequency you'd like the template to run. When finished, click Schedule.

You can view all your scheduled automations at any time on the Operations > Automations page.

With the Activity Tracker, you have a complete audit trail and history of changes made through Insycle, including processes run in Preview mode or data syncs. At any time, you can download a CSV report that shows all the changes made in a given operation run.

Navigate to Operations > Activity Tracker, search by module, app, or template name, then click the Run ID for the operation.

If you've configured the Insycle Run ID property in Pipedrive, you can cross-reference these operations in your CRM records.

When working with Pipedrive, for every Insycle operation that updates or creates a record, the Run ID is updated in the custom property Insycle Run ID. 

When using an Insycle Recipe that includes templates for multiple object types, such as organizations and people, the same Run ID will appear in both Pipedrive records.

If the Insycle Run ID doesn't already exist in your Pipedrive database, add it as a custom field with the label "Insycle Run ID" to each object type as needed. 

This object label is the only requirement for the integration to work properly. It's not dependent on the internal name or underlying API field name.

Each row in your matching fields setup is cumulative, so records must meet all of the criteria. For example, looking for records that have the same Name and Email and Phone returns only results where all three values are the same.

  The minimum required length for the matching values is four characters or more. Values such as "Joe" or "Ace" will be disregarded.

At the bottom of the Master tab, under Step 3, select the processing method that will be applied to your master selection rules:

• By Priority: Look through the master selection rules in order, one by one. As soon as a record meets one of the criteria, Insycle makes the master selection and skips the rest of the rules on the list. The vast majority of duplicate templates should use the By Priority option.
• Absolute: The master record must meet all listed selection rules; failure to match any rule means no record is identified. Absolute is suited to less flexible selections but may return no match, so it is usually best to choose By Priority.

Let's say you have four records that represent the same person—Marta Vaskovitch. The Merge Duplicates module will identify this as a single duplicate group containing four records.

Here is the data that we have for this duplicate group (viewed here in the Grid Edit module):

Here are the master selection rules we have set up:

We haven't sent any emails to Marta yet, so when Insycle processes the first three rules—Marketing emails clicked, emails bounced, and emails opened—Insycle cannot eliminate any record because they all have the same value of zero.

None of the record email addresses use a work domain, so there isn't a single choice for the master using the Email rule either.

In the next rule about contact owner, records 61301, 61201, and 61251 are eliminated since no contact owner exists for those records. Now, only one record remains, 61351, therefore that's the master record.

Though duplicate records may be identical, often there is only partial data overlap between them. When data is split across two records, both may contain unique and important information about the customer you'd like to keep.

By default, Insycle will keep the master record values; if the master field is blank, the value from the most recently updated duplicate will be used. The Merge Duplicates module allows you to control the values saved in the master record after the merge, regardless of the default merge behavior. By adding each field you want to control the data retention for in the Fields tab under Step 3 and selecting a Criteria, you can tell Insycle where the data for that field should be taken from and how to handle it.

On the Fields tab under Step 3, the Criteria dropdown gives you various options for choosing the data to keep, and Group Fields lets you keep values of multiple fields from the same record:

• From master record – Use the value that exists in the master record.
• From master record (even empty) – If the field on the selected master record is blank, keep it that way. Don’t automatically fill it in with a value from the most recently updated record.
• Most frequent value* – If the same value appears in multiple records, use the one that appears most frequently.
• From record where value – Select data from one of the records in the duplicate group based on the values. These options vary depending on the field type. For example, retain the data in the Employee # field that is associated with the record with the most employees.
• From record based on other field value – Look at the value in a different field to decide which value from the duplicate group should be kept. The example above highlights how a Last Modified Date value can be used to determine which Lifecycle Stage value to use.
• Combine and append all values – You can merge the values from the selected field for all records in the group. For example, if there is some type of Notes field, you could keep the notes from all of the records in the duplicate group.
• Rollup numbers* – On number fields, you can add the values from all records (sum), calculate the average, or keep only the largest (max) or smallest (min) values.
• Collect all values from other field* – Select a destination field to copy and combine values into, then select what field the data should come from. For example, this could be used to keep the record Owner values of all duplicates and combine them into a custom field.
• Collect non-master values from other field* – Aggregate the values of all the duplicates that are not the master and not the same as the master, meaning all instances of that value are excluded from collection. This can be especially helpful if you want a record of the object IDs that were removed, so you can also remove them from another system and keep the master. Select a destination field to copy and combine values into, then select what field the data should come from.

* Indicates Criteria options that do not allow you to Group Fields.

For instance, if you want to preserve the record IDs of the duplicates that are merged into the master record, you can create a custom field, "Merged Record IDs,” in your CRM.

Then, add a Fields rule to override the default merge behavior. Select the Merged Record IDs field you created, the Collect non-master values from other field criteria, and ID as the other field. 

You can use the Preview CSV to see how this will preserve the record IDs of all the duplicates in each duplicate group.

As another example, if you want to determine what value to keep based on its priority, you can set multiple criteria for the same field. In this example, there are two different rules for the Buying Role. Insycle will look at the first of the two, and if it finds a record that matches the criteria, Buying Role is Budget Holder, the second Buying Role rule will be ignored.

You have duplicate records that have been identified by Insycle but not all of them are merging into the master. 

Look at the Exclusions at the bottom of Step 3 to see how many duplicates are in the affected duplicate groups. If you have duplicate groups with more than 5 records, you may want to change the Skip duplicate groups with more than [5] records per group setting to ensure you get all of them.

This 5-record default is intended to prevent accidental merging of non-duplicate records when the filter in Step 1 is too broad.

If the Result is "Failed" and the Message column of the CSV report displays this text:

Change rules in Step 3 'Master Selection'. Failed to pick master record because multiple records (X) meet the selection criteria. In 'Master Selection', change, add, or reorder the rules such that only one record matches (if cannot determine master based on field values, use 'Record ID is lowest' as the last rule).

This means Insycle could not determine which record in the duplicate group should become the master — none of the records meet more of the rules than the others.

There are a few things you can try to resolve this:

1. Under Step 3, on the Master tab, experiment with reordering or adding rules based on fields that are likely to have unique values across your records. Think about what matters most to your business — the fields that would make one record clearly more valuable or authoritative than another. For example:

   • A field that stores an ID linked to an external system, such as an ERP, data warehouse, or proprietary internal tool — the record with that ID populated is likely the one your other systems depend on
   • The record with the highest number of associated records (contacts, deals, activities) — indicating it has accumulated more history and relationships
   • A field indicating CRM ownership or account status — for example, a record marked as "Customer" vs. "Lead," or one with an active owner assigned
   • A field that tracks revenue, subscription tier, or engagement level — the record with higher values may represent the more complete or current entry
   • A custom field your team uses to flag record quality or data source reliability

   The goal is to surface the fields that reflect record importance for your specific operations, not just default system fields.

2. Also on the Step 3 Master tab, confirm that By Priority is selected rather than Absolute.

   With By Priority, a record only needs to match one rule to be selected as master. With Absolute, a record must meet all rule criteria, which makes it much harder for any single record to qualify.

   In most cases, By Priority is the right choice. If By Priority is already selected and the error persists, none of the records in the group meet any of the criteria more than the others. Return to the Master tab and continue adding or reordering rules, focusing on the business-critical fields described in #1 above.

   

3. As a last resort, add a rule on the Master tab that uses Record ID is lowest, or Create Date is earliest. These rules will always produce a unique winner, ensuring master selection can complete even when other field-based rules cannot differentiate the records.

There are a couple of things to look at that may be causing records to be misidentified as duplicates.

First, you may need a better unique identifier. Under Step 1, if you use only fields that can correctly contain the same values across multiple records, they aren't unique identifiers. In this case, you are likely to identify unrelated records as duplicates and may accidentally merge them.

Unique identifiers are data unlikely to be shared by any other record unless they represent the same underlying entity. Common fields used for deduplication include phone numbers, email addresses, mailing addresses, and ID numbers.

Second, this may indicate the Comparison Rule under Step 1 is too broad. Try using the Exact Match comparison rule instead of the Similar Match rule. Similar Match looks for values that may be close, even with a one-character difference (e.g., a typo), broadening the search. 

Remember to always run your deduplication in Preview Mode to confirm it's working as expected before running it in Update Mode and applying the changes to your CRM records.

Most of the time, when Insycle can't find duplicates, it is due to your matching rules in Step 1. It is important to analyze the underlying data to better understand how to set up your rules. A helpful exercise for contacts is to set up your matching filters to find exact matches on only First Name and Last Name. For companies, you can use Company Name.

When you click Find, these rules can show you a broad overview of what duplicates are potentially in your database and what fields might be useful to include in your matching fields. These settings are just for discovery and should not be used for a final merge operation; many people can have the same first and last names and are not duplicates. 

To get further context, on Step 2, click the layout gear button on the right side of the title bar. Here, you can add any field from your database as a column in the duplicate group review to better understand the data in these records. These will also appear in your CSV report. 

It can take Insycle a while to find and match duplicates when the fields used to identify them have very long values. The longer the values, the longer it takes Insycle to process the data and generate the results. This might come up when looking for matches based on long ID numbers, LinkedIn bio links, or other URLs with long strings attached (ex, https://www.linkedin.com/in/svadin%C3%ADr-n%C4%9Bmec-1234b31a3/).

You can speed this up by limiting how much of the value Insycle looks at.

If the beginning or ending portion of the values are all unique, you can limit the comparison to the first or last several characters using the Match Parts parameter under Step 1. 

Or use the Ignore Text (Substrings) parameter, then click the Terms button.

On the Ignored Text tab of the popup, add the common portion of the URL or text string.

There are two ways to ensure the records you are merging are indeed duplicates.

First, always run your deduplication templates in Preview Mode before running them in Update Mode. This produces a CSV file showing how your records would have been merged. Then you can ensure that your Merge Duplicates template is working as expected and not merging non-duplicate records together.

Additionally, to ensure a smooth merge, consider narrowing the matching settings in Step 1. Try the Exact Match Comparison Rule instead of Similar Match. Then make sure that you are using actual, uniquely identifying fields—first name, last name, email, and phone number are popular choices. The more tightly defined your filter is, the less likely you are to merge non-duplicate records.

If the Message column of the CSV report displays this error:

Change rules in Step 3 'Master Selection'. Failed to pick master record because multiple records (X) meet the selection criteria. In 'Master Selection', change, add, or reorder the rules such that only one record matches (if cannot determine master based on field values, use 'Record ID is lowest' as the last rule).

None of the records meet more of the rules than the others do.

There are a couple of things you can try to resolve this:

1. On the Master tab in Step 3, experiment with reordering or adding additional fields that are likely to have unique values.
2. At the bottom of the Master tab in Step 3, ensure By Priority is selected, not Absolute.
   With By Priority, your master record only has to match one rule. Using Absolute, your master record would have to meet all of the rule criteria. In most cases, it is best to select By Priority.
   If By Priority was used, then none of the records in the duplicate group meet any of the criteria on the list more than the others. In this case, you'll need to experiment, reordering or adding additional rules for fields likely to have unique values.
3. As a last resort, you can add a rule on the Master tab in Step 3 that says Record ID is lowest, or Create Date is earliest.

Yes, Insycle allows you to select which field data to retain in the master record using the Fields tab in Step 3. See the Merge Duplicates module field data retention rules reference for more details.

No, matching fields do not need to match exactly. The Similar Match Comparison Rule in Step 1 acts as a fuzzy match, looking for values that may be close but differ by only one character (e.g., a typo), and broadens the search.

This search behaves like when Google shows results for a slightly different term or says, “Did you mean...” For example, if an Email of “huey@coahulldu.co” is found, it could include records with the values “hueyy@coahulldu.co" or "hue.y@coahulldu.co” as a match.

You should be careful when using Similar Match as the looser criteria can incorrectly identify non-duplicates as duplicates. 

Review the Understanding Similar Matching best practices for more details.

Yes. You can use an existing list of duplicates with Insycle to deduplicate it in bulk, following these steps:

1. Prepare a CSV file with columns for the record IDs and a "Merge Master" column. In the "Merge Master" column, mark which record should be kept after merging.
2. Create a custom field called "Merge Master" in your CRM.
3. Use the Magical Import module to import your CSV file into the CRM, updating the "Merge Master" field for the relevant records.
4. Go to the Merge Duplicates module and set up a filter to select records based on the "Merge Master" field.

Learn more about deduplicate records using a CSV.

If the Message column of the CSV report displays this error:

Change rules in Step 3 'Master Selection'. Failed to pick master record because multiple records (X) meet the selection criteria. In 'Master Selection', change, add, or reorder the rules such that only one record matches (if cannot determine master based on field values, use 'Record ID is lowest' as the last rule).

None of the records meet more of the rules than the others do.

There are a couple of things you can try to resolve this:

1. On the Master tab in Step 3, experiment with reordering or adding additional fields that are likely to have unique values.
2. At the bottom of the Master tab in Step 3, ensure By Priority is selected, not Absolute.
   With By Priority, your master record only has to match one rule. Using Absolute, your master record would have to meet all of the rule criteria. In most cases, it is best to select By Priority.
   If By Priority was used, then none of the records in the duplicate group meet any of the criteria on the list more than the others. In this case, you'll need to experiment, reordering or adding additional rules for fields likely to have unique values.
3. As a last resort, you can add a rule on the Master tab in Step 3 that says Record ID is lowest, or Create Date is earliest.

To help you analyze and determine which records to merge, you can change the fields that appear in the preview under Step 2 by clicking the gear button to adjust the layout.