Deduplicate Across Salesforce Leads and Contacts

Navigate to Data Management > Merge Duplicates, and select the Salesforce database and Contacts record type from the top menu. Then explore the templates for an existing solution that may be close to what you need.

Step 1 looks through the records in your database examining the fields that you specify for a match. 

Add a row for each field you want to compare to find a matching value. Choose fields that, in combination, give a high degree of certainty that the matched records are duplicate records.

Check the Include Leads checkbox. With the box checked, Insycle will look at leads and contacts for duplicates, and allow you to merge both down into one singular record.

Click the Find button and Insycle will generate a list of duplicates for you to review.

Commonly used matching fields in lead and contact deduplication include:

• First Name + Last Name
• Company Name
• Email Username
• Email Domain
• Company Website
• Phone Numbers

For details on configuring the comparison rules, see the Bulk Merge Duplicate People, Companies article.

Expand Criteria for Matching Duplicates

If you'd like to look at the data in two different fields (that contain similar data) as if it were one, you can set up Related Fields under the Advanced tab. For example, you might want to look at both the Business Phone and Mobile Phone fields for duplicate values.

The Conditions tab provides rules that one or more of the records in a duplicate group must meet. These options let you choose required fields, fields that can be empty, or specify values that must be included.

• At Least One Record With Non-Empty - At least one record in the duplicate group must contain a value.
• Value Required in All Records - Each record must contain a value in this field to be considered a duplicate.
• Empty Allowed in Any Record - A record can still be considered a duplicate if this field is blank. Allowing empty values requires using two or more fields to identify duplicates.
• At Least One Record Match - At least one record in the duplicate group must match the specified value, and the other records cannot be blank. If none of the records have the specified value, the duplicate group will not be merged.
• Only One Record Match - If more than one record in a duplicate group contains the specified field value, the duplicate group is skipped (not merged).
• Within Timeframe - Set a time parameter that can find duplicates created or modified within a specific timeframe, such as the last 30 minutes.
• Values Don't Match - All records in the duplicate group must have a value, and it doesn't match the other records within the group.

Records that have the same values in the fields from Step 1 are considered matches, whether they are all leads, all contacts, or a mix. When two or more records represent the same person based on your matching rules, they are clustered together into duplicate groups.

Under Step 2, each duplicate group shows the total number of records that were identified as duplicates based on your settings. For example, if you had four records for the same person, it would count as one duplicate group with four duplicate records.

In this view, each row represents a duplicate group. When you click a row, it expands and shows the records that belong to that group. The Type column indicates whether the duplicate record is a contact or lead.

Explore the record data in the duplicate groups. Double-check to ensure that the fields you set up in Step 1 show what you expected.

Looking at the record details in several duplicate groups, decide the best way to determine which is the master record the other records will be merged into. Learn common practices for picking a master record.

To get more context for analyzing the records, add more columns to the groups layout using the gear button on the right end of the header.

The most efficient and sustainable way to merge duplicates is in Bulk mode. This allows you to set rules for automatically determining the master record across all lead and contact records in your database. You can then use saved templates and recipes to repeat the process.

Under Step 3: Choose Operation, the Bulk operation tab is automatically selected. Leave this as is.

If you have duplicate groups that contain more than five records, you may want to change the value in Skip duplicate groups with more than 5 records per group to make sure you can get them all.

Note: When merging in bulk, Insycle limits the duplicate group size to 5 records and skips groups that contain more. You can adjust the value as needed, up to 100 per duplicate group. This is a precaution to ensure that if you use a duplicate matching filter that is too broad in Step 1, you do not accidentally merge many non-duplicate records together.

In Manual mode, you have complete control over which records are merged together by selecting them from the Record Viewer. Manual mode should be reserved only for cases needing a careful, controlled process. Learn more about manually merging duplicates.

Configure Rules to Automatically Select the Master Record

On the Master tab of Step 4, define how all of the duplicate groups should be merged at scale. To do this, you need to create a series of rules that specify which record from each group should become the master. The master is the record that will remain after the merge.

For example, if you had four records representing the same person in one duplicate group, all four would be merged into one master record, and the other three records would no longer exist.

For each duplicate group, Insycle will look at each Master rule to see which of the records in the group meet the criteria. Rules are read in order, from top to bottom, and as soon as a record is the only one to meet a rule, it is selected as the master record. The subsequent rules are ignored. 

Select the processing method—By Priority or Absolute. Most de-duplication operations should use By Priority. When a record meets one of the criteria, Insycle makes the master selection and skips the rest of the rules on the list. Learn more about these options in the Advanced How-To, Step 4: Selecting Priority vs Absolute below.

If none of the records in the duplicate group match any of the rules, the automatic master selection for the group fails. See the Troubleshooting section below for more details.

After Insycle has identified the master record, it will use the Field selection rules to automatically pick which values from a duplicate group will be used in the master record.

Under Step 4, click the Fields tab. 

By default, Insycle will keep the most recent values. For each field you want to control data retention for, you need to tell Insycle where the data should be taken from. This information is merged into the master. Any other data not in the master or copied to the master is deleted.

The Criteria dropdown gives you various options for choosing the data to keep, and the Group Fields let you keep values of multiple fields from the same record:

• From record where value – Select data from one of the records in the duplicate group based on the values. These options vary depending on the field type. For example, with Salesforce, you can select an Owner field, the “From record where value” Criteria, and the Condition “active user.”
  Or, with an email field, you could retain the value that is using a professional domain rather than a free one. 
• From master record – Use the value that exists in the master record.
• From master record (even empty) – If the field on the selected master record is blank, keep it that way. Don’t automatically fill it in with a value from the most recently updated record.
• Most frequent value* – If the same value appears in multiple records, use the one that appears most frequently.
• From record based on other field value – Look at the value in a different field to decide which value from the duplicate group should be kept. The example above highlights how a Last Modified Date value can be used to determine which Level value to use.
• Combine and append all values* – You can merge the values from the selected field for all records in the group. For example, if there is some type of Notes field, you could keep the notes from all the records in the duplicate group, merging them into the master record.
• Rollup numbers* – On number fields, you can add the values from all records (sum), calculate the average, or keep only the largest (max) or smallest (min) values.
• Collect non-master values from other field* – Aggregate the values of all the duplicates that are not the master and not the same as the master, meaning all instances of that value are excluded from collection. This can be especially helpful if you want a record of the object IDs that were removed, so you can also remove them from another system and keep the master. Select a destination field to copy and combine values into, then select what field the data should come from.
• Collect all values from other field* – Select a destination field to copy and combine values into, then select what field the data should come from. For example, this could be used to keep the record Owner values of all the duplicates and combine them into a custom field.

* Indicates Criteria options that do not allow you to Group Fields.

  For each duplicate group, a special row –Master (After)– will appear in the CSV report. The Master (After) row shows the values the final record will contain based on your Field rules and the default behavior.

Preview Merge Changes in CSV Report

Now with the filters and master record set up, you can preview how the merging would be applied. That way, you can check to ensure your configuration works as expected before the changes are pushed to your live Salesforce database. 

Under Step 5, click the Review button and select Preview mode.

Click the Next button to go to the Notify screen to select recipients for the email report. You can also add additional context to the email message.

On the When tab, click the Run Now tab, and select which records to apply the change to (if you have many records, you might want to do only a few to test), then click the Run Now button.

Insycle will generate a preview CSV and send it to your email. Open the CSV file from your email in a spreadsheet application and review the data.

There's a row for each record, which includes:

• The Result of the action
• A Message with details
• The Duplicate Group ID which indicates which records will be merged together
• A record Status that identifies which were picked as master and which were identified as duplicates and merged into the master. See below for more details
• All fields used to identify the duplicates in Step 1
• The record ID, record name or email, and Deeplink to the CRM record
• The Type, indicating whether the record is a Contact or Lead
• All fields selected for the Master and Fields rules in Step 4. Note that if a field is used on both tabs, it will only appear once in the CSV. 
• Any additional fields added to the Layout tab in Step 2

The Status column indicates:

• Duplicate – The record is part of a duplicate group.
• Master – The master record chosen for the duplicate group based on default behavior and your Master rules. Review the selections in this row to determine whether the appropriate records are being chosen.
• Master (After) – For each duplicate group, the Master (After) row shows the values the final record will contain based on your Field rules and the default behavior. 
• Error – If Insycle is not able to determine which record would be the master, an error message will appear here. See the Troubleshooting section below for more details.

If everything looks good, return to Insycle and move forward with applying the changes.

Apply Changes to Your CRM Records

If everything in your CSV preview looks good, return to Insycle and move forward with applying the changes to your live Salesforce data.

Under Step 5, click the Review button, and this time select Update mode.

On the When tab, you should use Run Now the first time you apply these changes to Salesforce.

After you've seen the results in Salesforce and are satisfied with how the operation runs, you can save your configuration as a template and set up automation so this merge operation runs on a set schedule. If you have several templates you'd like to run together automatically, you can create a recipe.

By automating with a template, you'll save time and ensure that your leads and contacts are merging consistently on an ongoing basis.

Return to the Template menu at the top of the page and click Copy to save your configurations as a new version of the template you started with. Then click the pencil to edit your new template name.

Under Step 5, click the Review button and select Update mode.

On the Notify tab, select the send option appropriate for your automation: Always send, Send when errors, or Do not email.

Add any additional recipients who should receive the CSV (and make sure to hit Enter after each address). You can also provide additional context in the message subject or body.

On the When tab, select Automate and configure the frequency you'd like the template to run. When finished, click Schedule.

You can view all your scheduled automations at any time on the Operations > Automations page.

When you have a solid set of templates that reliably merge your records, you can put them together into a longer, ordered sequence as a Recipe. Then, you can schedule that Recipe to run on a consistent, set schedule. Your templates will run one after another in the order that you set.

To add your Recipe to a Salesforce Flow, the Automate setting must be Salesforce Flow Integration.

Learn more about integrating Insycle Recipes with Salesforce flows.

Each row in your matching fields setup is cumulative, so records must meet all of the criteria. For example, looking for records that have the same First Name and Last Name and Phone Number returns only results where all three values are the same.

  The minimum required length for the matching values is four characters or more. Values such as "Joe" or "Ace" will be disregarded.

Use the filter to work with a segment or smaller pool of records. Then Insycle will only analyze the remaining records for duplicates. To add filters, click the Filter button, then choose the field to look at, select the condition, and set the value to look for. The filter is applied before the matching step runs. 

Most of the options in the Field dropdown match the fields that are found in your CRM, and for Contact records, there are three additional options related to the Email value: 

• Email Username: The portion of the email address before the “@.” For example, if the email address were “maria@acmewidgets.com,” the username value would be “maria.” 
• Free Email Provider Domain: Choose True to filter out records where the email domain is Gmail, Hotmail, Yahoo, and about 10,000 other free email providers. This filter helps ensure these are real clients, or can determine which record is the legitimate one because it’s most likely customer companies aren't using free Gmail accounts (though a contact may have accidentally emailed us from it at some point). 
• Email Top-Level Domain: The top-level domain (TLD) is everything that follows the final dot of a domain name. For example, in the domain name acmewidgets.com', '.com' is the TLD. Some other popular TLDs include '.org', '.uk', and '.edu'. 

At the bottom of the Master tab under Step 4, select the processing method that will be applied to your master selection rules.

By Priority: Looks through the master selection rules in order, one by one. As soon as a record meets one of the criteria, Insycle makes the master selection and skips the rest of the rules on the list. The vast majority of duplicate templates should use By Priority.

Absolute: The master record must meet all of the listed master selection rules. If a record does not match every rule listed, no master record will be identified. Absolute is appropriate for less flexible master selection.

For example, if a company wants to ensure the chosen master record is in their sales pipeline and already has a sales rep working the record, they can choose Absolute and set the Master rules:

• Lifecycle Stage is lead
• Contact Owner active user

Choosing Absolute can often result in no master record being identified since the record has to match every rule listed, so in most cases, you should select By Priority.

For contacts and leads, it's often useful to pick master records based on activity and engagement.  You can also use other dates to pick a master record, such as the earliest created or the most recently updated record.

Though it's possible that duplicate records may be exactly the same, often there is only partial data overlap between them. When data is split between two different records, both records may contain unique and important information about the customer you'd like to keep.

By default, Insycle will keep the master record values; if the master field is blank, the value from the most recently updated duplicate will be used. The Merge Duplicates module allows you to control the values saved in the master record after the merge, regardless of the default merge behavior. By adding each field you want to control the data retention for in the Fields tab under Step 4 and selecting a Criteria, you can tell Insycle where the data for the field should be taken from and how to handle it.

If you want to preserve data from several fields based on one Criteria, select them under Group Fields. Note that this is not available for all Criteria.

For example, if merging Salesforce accounts, you may want to save all of the Account IDs from records that are merged together. You can add a new custom field, “Merged Account IDs” to your CRM.

Then, in the Merge Duplicates module, under the Fields tab of Step 4, add a rule to override the default merge behavior. Select the "Merged Account IDs" field, the "Collect non-master values from other field" criteria, and "Account ID" as the other field. 

You can use the Preview to see how this will preserve the Account IDs of all the duplicates in each duplicate group.

You could set multiple criteria for the same field, which can be useful when establishing a hierarchy for selecting the value so "Value A" takes precedence over "Value B."

In this example, two different rules are defined for the Customer Status field. The first rule states that if a record has the value "Customer," that value will be merged into the master record. However, if no records match the "Customer" criteria, the second rule for "Prospect" comes into play.

For situations where there are no common rules you can apply for identifying duplicates for all or some of the records, you may need more granular control for picking records to include or exclude from the process.

To do this in bulk, you can create a CSV and use the Magical Import module.

In these cases, you can use CSV files to customize your bulk merging, designate master records, and exclude records from deduplication. Then you can import the CSV from the Magical Import, and use the Merge Duplicates module for complete control over the final merge operation. Learn how to customize merging Duplicates in bulk using a CSV.

To do this one record at a time, you can use Manual mode of the Merge Duplicates module.

In Manual mode, you have complete control over which records are merged together by selecting them from the Record Viewer. Manual mode should be reserved only for cases needing a careful, controlled process. Learn more about merging duplicates in Manual mode.

When setting up your merge operation in the Merge Duplicates module, add extra fields to be included in the CSV report so you have the data later for undoing changes and/or just for general review.

Under Step 2, click the gear icon in the header.

On the Layout tab, add any extra fields to the Visible Fields list.

The fields will be included in the CSV report.

Before running a large merge operation, you can export all records directly from Salesforce to a CSV file. This can preserve all the record details, which can help later for undo and/or just for general review.

Create a Targeted Salesforce Report

To select only the necessary fields and filter out inapplicable records, you can create a Salesforce Report. This gives you a targeted export, omitting unnecessary data. Once configured and saved, you can reuse this report before each bulk merge operation.

In Salesforce, if the Reports tab doesn't appear at the top, click the App Launcher icon (nine squares) in the top left corner and search for "Reports."

Create a new report based on the specific data type and add all the desired fields. Add filters to narrow the results. Save and Run the report.

On the results page, click the arrow next to the Edit button and select Export. In the Export popup, select Details and CSV as the format.

Export All Data for a Record Type

To get all field data for all records of the same type, you can use Salesforce's Data Export Service. There are no filter/segmenting options, so if you have a large dataset, this can take a while to run.

In Salesforce, navigate to Setup > Data > Data Export.

Click Export Now, check the box for the record type, and then click Start Export. You'll receive an email when the file is ready for download. 

You have duplicate records that Insycle has identified, but not all of them are merging into the master. Check to see how many duplicates are in the affected duplicate groups. If you have duplicate groups that contain more than five records, you may want to change the value in Skip duplicate groups with more than 5 records per group to make sure you can get them all.

This setting is intended to protect against the accidental merging of non-duplicate records if the filter in Step 1 is too broad.

If the Message column of the CSV report displays this text:

Change rules in Step 4 'Master Selection'. Failed to pick master record because multiple records (X) meet the selection criteria. In 'Master Selection', change, add, or reorder the rules such that only one record matches (if cannot determine master based on field values, use 'Record ID is lowest' as the last rule).

This means that based on all the rules, Insycle could not figure out which record in the duplicate group would be the master. None of the records meet more of the rules than others.

There are a few things you can try to resolve this:

1. Under Step 4, on the Master tab, experiment with reordering or adding additional fields that are likely to have unique values.
2. Also on the Step 4, Master tab, check to ensure that you have By Priority selected and not Absolute.

   
   With By Priority, your master record only has to match one rule. Using Absolute, your master record would have to meet all of the rule criteria. The majority of the time, it is best to select By Priority.

   If By Priority was used, this indicates none of the records in the duplicate group meet any of the criteria on the list more than the others. In this case, you'll need to experiment on the Master tab, reordering or adding additional rules for fields likely to have unique values.

3. As a last resort, you can add a rule on the Master tab of Step 4 that says Record ID is lowest, or Created Date is earliest.
   

There are a couple of things to look at that may be misidentifying records as duplicates.

First, you may need a better unique identifier. Under Step 1, if you only use fields that could correctly contain the same values in multiple records, these aren't unique identifiers. In this case, you are likely to identify unrelated records as duplicates and may accidentally merge them.

Unique identifiers are data that is unlikely to be shared by any other record unless it represents the same underlying entity. Fields that are commonly used in deduplication include phone numbers, email, mailing addresses, or ID numbers.

Second, this may indicate the Comparison Rule under Step 1 is too broad. Try using the Exact Match comparison rule instead of Similar Match. Similar Match looks for values that may be close but with a one-character difference (maybe a typo) which broadens the search. 

Remember, always run your deduplication in Preview Mode to confirm things are working as expected before running them in Update Mode and applying the changes to your CRM records.

Most of the time, when Insycle can't find duplicates, it is due to your matching rules in Step 1. It is important to analyze the underlying data to better understand how to set up your rules. A useful exercise can be to set up your matching filters to look for exact matches of just First Name and Last Name.

When you click Find, these rules can show you a broad overview of what duplicates are potentially in your database and what fields might be useful to include in your matching fields. These settings are just for discovery and should not be used for a final merge operation; many people can have the same first and last names and are not duplicates. 

To get further context, on Step 2, click the layout gear button on the right side of the title bar. Here, you can add any field in your database as a column to the duplicate group review to better understand the data inside these records. The fields will also be included in the CSV report.

It can take a while for Insycle to find and match duplicates if the fields being used to identify them have very long values. The longer the values, the longer it takes Insycle to process the data and generate the results. This might come up when looking for matches based on long ID numbers, LinkedIn bio links, or other URLs with long strings attached (ex, https://www.linkedin.com/in/svadin%C3%ADr-n%C4%9Bmec-1234b31a3/).

You can speed this up by limiting how much of the value Insycle looks at.

If the beginning or ending portion of the values are all unique, you can limit the comparison to the first or last several characters using the Match Parts parameter under Step 1. 

Or use the Ignore Text (Substrings) parameter, then click the Terms button.

On the Ignored Text tab of the popup, add the common portion of the URL or text string.

Currently, Salesforce doesn't have an automated way to prioritize active owners during the merge process. You'll need to verify owner status manually for each merge operation.

However, Insycle's Merge Duplicates module has two options that enable you to prioritize an active owner. 

First, you could add a Master rule under Step 4 to tell Insycle to select the record from each group with an active owner to become the master record.

Add a rule with the following parameters:

• Field: Record owner
• Condition: active user

Second, you could create a Field rule to retain the owner who is an active user.

Add a rule with the following parameters:

• Field: Owner
• Criteria: From record where value
• Condition: active user

There are two ways to make sure that the records that you are merging are indeed duplicate records.

First, always run your deduplication templates in Preview Mode before running them in Update Mode. This produces a CSV that shows you how your records would have been merged. Then you can ensure that your Merge Duplicates template is working as expected and not merging non-duplicate records together.

Additionally, to ensure a smooth merge process, consider narrowing down the matching settings in Step 1. Try the Exact Match Comparison Rule instead of Similar Match. Then make sure that you are using actual uniquely identifying fields—first name, last name, email, and phone number are popular choices. The more tightly defined your filter is, the less likely you are to merge non-duplicate records.

Yes. You can merge specific records using a CSV file containing the records you want to combine. Here's how:

• Prepare a CSV file with columns for the record IDs and a "Merge Master" column. In the "Merge Master" column, mark which record should be kept after merging.
• Create a custom field called "Merge Master" in your CRM.
• Use the Magical Import module to import your CSV file into the CRM, updating the "Merge Master" field for the relevant records.
• Go to the Merge Duplicates module and set up a filter to select records based on the "Merge Master" field.

Learn more about customizing bulk deduplication from a CSV.

You can merge up to 100 duplicates into one single master record. 

If you have duplicate groups that contain more than five records, you may want to change the value in Skip duplicate groups with more than 5 records per group under Step 3 to make sure you can get them all.

This is a precaution to ensure that if you use a duplicate matching filter that is too broad in Step 1, you do not accidentally merge many non-duplicate records together. If you are going to set this number at a high level, it is a good idea to run Preview Mode first to make sure your deduplication template is operating as you intend.

Yes, Insycle allows you to select which field data is retained in the master record using the Fields tab under Step 4. See Step 4: Control What Field Data is Retained in this article for more details.

Yes, there are several ways to share details and get approval before merging duplicates.

You can manually approve master records and mark them in a CSV, then use Insycle to bulk deduplicate down to those master records. See the Customize Bulk Deduplication Using Exclusions and Pre-Defined Masters article to learn more.

Or, you can run the Merge Duplicates module in Preview Mode and then deliver the preview CSV that Insycle generates. The CSV report includes your entire merge operation down to individual duplicate groups but does not update your live data. Then your team can approve the merge based on this report, before running Merge Duplicates in Update Mode.

Additionally, team members can review duplicates and manually select the master for each record under Step 4. Review the Manually Merge Duplicates article for more detail.

No, your field data does not need to match exactly. The Similar Match comparison rule in Step 1 looks for values that may be close but with a one-character difference (maybe a typo) and broadens the search. 

This search behaves like when Google shows results for a slightly different term or says, “Did you mean...” For example, if an Email of “huey@coahulldu.co” is found, it could include records with the values “hueyy@coahulldu.co" or "hue.y@coahulldu.co” as a match.

You should be careful when using Similar Match as the looser criteria can incorrectly identify non-duplicates as duplicates. 

Review the Deduplication Best Practices article for more details.

All paid plans include unlimited usage, users, and operations. See Insycle's pricing page for more details. During the free trial, there is a cap of 500 records updated, cleansed, or merged.