How to Merge Records When Some Matching Fields Are Empty

You know you have duplicates in your database with a variety of different fields filled in or left blank. Some of these fields are normally helpful when looking to match duplicate records, such as phone numbers or a name. Just because one field is blank doesn't mean the record doesn't represent the same underlying entity; you still want to merge a record as long as it matches two out of three matching fields.

Insycle's Merge Duplicates module can identify duplicate records even if records are missing some of the values. Then you can merge them flexibly, in bulk, using advanced master record selection logic and field data retention rules.

Process Summary

  1. Set rules to identify duplicates and allow empty fields.
  2. Review and analyze the identified duplicates.
  3. Choose Bulk Mode.
  4. Set rules to select the master record and determine what field data is retained.
  5. Deduplicate your records.

 

Step-by-Step Instructions

Step 1: Configure Rules to Identify Duplicates

Navigate to Data Management > Merge Duplicates, and select the database and record type from the top menu. Then explore the templates for an existing solution that may be close to what you need.

Configure Matching Rules

To find duplicates, you need to define how to match records. Step 1 looks through the records in your database, examining the fields that you specify for matches. Each row is for a field you want to look at for repeated values.

Configure the match rules in Step 1 under the Simple tab:

bulk-merge-_3.png

  1. Field Name - Select a combination of unique identifier fields—data that is unlikely to be shared by any other record unless it is a duplicate. You want fields that, in combination, give a high degree of certainty that the matched records really represent the same entity.
  2. Comparison Rule - Define what kind of likeness to look for when deciding if field values should be considered a match. It's a good idea to start with Exact Match, and begin with easy-to-find duplicates.
  3. Ignored - Specify parts of a field value to ignore, such as specific text, whitespace, or characters. These won’t be considered as part of the matching process.
  4. Match Parts - Define specific portions of the field value to compare.

See the Advanced How-Tos for more detail on using these fields.

The example above will identify duplicates by looking for records with the exact same values in the First Name, Last Name, and Phone Number fields. People with the same first name AND last name AND email domain will show as possible duplicates.

Commonly used matching fields in deduplication include:

  • First Name + Last Name
  • Company Name
  • Email
  • Email Domain
  • Company Website
  • Phone Number

Allowing a Field to Have an Empty Value

When using two or more fields to identify duplicates, records can still be considered matches even if one of the field values is blank. You just need to specify which field(s) allow a blank value.

Under Step 1, click the Conditions tab.

allow-empty_2.png

All the matching fields you included will automatically appear with the Value Required condition selected. Change the condition to Empty Allowed to allow empty values for certain fields. Make sure at least one of the fields you keep as Value Required is a reliable unique identifier to ensure the records are really duplicates.

bulk-merge-_4.png

For example, on the Simple tab, you may have the matching fields: First Name, Last Name, and Phone Number. But on some of your records, the Phone Number field may be empty. You want all records with the same name, same phone number, and no phone number to be considered duplicates.

Click the Find button and Insycle will generate a list of duplicates for you to review. 

Step 2: Analyze the Identified Duplicates

Records that have the same values in the fields specified in Step 1 are considered matches. When two or more records represent the same entity (person, company, or other) based on your matching rules, they are clustered together into duplicate groups. Each duplicate group shows the total number of records that were identified as duplicates based on your settings. For example, if you had four records for the same person, it would count as one duplicate group with four duplicate records.

In this view, each row represents a duplicate group. When you click a row, it expands and shows the records that belong to that group. Records that have blank values in the fields you allowed will be included. 

allow-empty-review.png

Explore the record data in the duplicate groups. Double-check to make sure that the fields you set up in Step 1 are showing what you expected. 

Add more fields to the view to get more context for analyzing the records using the gear button on the right.

step-2-gear-2023-02-06.png

Looking at the record details in several duplicate groups, decide the best way to determine which is the master record the other records will be merged into. Learn more about common practices for picking a master record.

Step 3: Choose Bulk Mode

The most efficient and sustainable way to merge duplicates is in Bulk mode. This allows you to set rules for determining the master record automatically across all records in your database. With Bulk mode, you'll be able to use saved templates and recipes to repeat the process on a regular basis. 

Under Step 3: Choose Operation, the Bulk operation tab is automatically selected. Leave this as is.

dup3.png

If you have duplicate groups that contain more than five records, you may want to change the value in Skip duplicate groups with more than 5 records per group to make sure you can get them all.

Note: When merging in bulk, Insycle limits the duplicate group size to five and skips groups that contain more records. You can adjust the default value of five records as needed, up to 100 per duplicate group. This is a precaution to ensure that if you use a duplicate matching filter that is too broad in Step 1, you do not accidentally merge lots of non-duplicate records together.

In Manual mode, you have complete control over which records are merged together by selecting them from the Record Viewer. Manual mode should be reserved only for cases where you need a careful, controlled process. Learn more about Manually Merging Duplicates.

Step 4: Set Rules for Master Record Selection and Data Retention

Configure Rules to Automatically Select the Master Record

Now you’ll define how all of the matching duplicate groups should be merged at scale. To do this you need to create a series of rules that specify which record from each group should become the master.

The master is the record that will remain after the merge. For example, if you had four records representing the same person in one duplicate group, all of them would be merged into one master record. The other three records would not exist anymore.

On Step 4, select the matching type—Priority Match, or Absolute Match. 

Priority Match looks through the master selection rules one by one, in order. As soon as a record meets one of the criteria, this becomes the master and the rest of the rules are skipped. Most de-duplication operations should use Priority Match.

With Absolute Match, a record must meet all of the listed rules in the Record tab in Step 4 to become the master. If a record does not match every rule listed, no master record will be identified. Absolute Match is appropriate for less flexible master selection. Learn more about these options in the Advanced How-Tos below.

priority-or-absolute.png

For each duplicate group, Insycle will look at each master selection rule to see which of the records in the group meet the criteria. 

master-selection.png

If none of the records in the duplicate group match any of the rules, the automatic master selection for the group fails. See the Troubleshooting section below for more detail.

Configure Rules That Determine Values to Keep

After Insycle has identified the master record, it will use the Field selection rules to automatically pick which values from a duplicate group will be used in the master record.

Duplicates may be exact match versions of another record, but often there is only partial data overlap between the records. When data is split between two different records, both records may contain unique and important information about the customer you'd like to keep.

Under Step 4, click the Fields tab. For each field you want to control the data retention for, you need to select a field and tell it where the data for the field should be taken from. This is merged into the master. Any data that is not in the master or not copied to the master is deleted.

The Criteria dropdown gives you various options for choosing the data to keep:

merge-duplicates-hubspot-contacts-step-4-field-rules-9.png

  • From master record – Use the value that exists in the record Insycle selects as the master.
  • From master record (even empty) – If the field on the selected master record is blank, keep it that way. Don’t automatically fill it in with a value from the most recently updated record.
  • Most frequent value – If the same value appears in multiple records, use the one that appears most frequently.
  • From record where value – Select data from one of the records in the duplicate group based on the values. These options vary depending on the field type. For example, retain the data in the Email field that is using a professional domain rather than a free one.
  • From record based on other field value – Look at the value in a different field to decide which value from the duplicate group should be kept. The example above highlights how the Last Modified Date value can be used to determine which Lifecycle Stage value to use.
  • Combine and append all values – You can merge the values from the selected field for all records in the group. For example, if there is some type of Notes field, you could keep the notes from all of the records from the duplicate group.
  • Collect all values from other field – Select a destination field to copy and combine values into, then select what field the data should come from for each record in the duplicate group. For example, this could be used to keep the record Owner values of all duplicates and combine them into a custom field.
  • Collect non-master values from other field - Aggregate the values of all the duplicates that are not the master and not the same as the master, meaning all instances of that value are excluded from collection. This can be especially helpful if you want a record of the object IDs that were removed, so you can also remove them from another system. Select a destination field to copy and combine values into, then select what field the data should come from.

Remember: The Fields tab is a separate set of rules for automatically selecting what data to keep. You still need to configure the rules for automatically selecting the master record in the Record tab.

Step 5: Review and Merge

Preview Merged Changes in CSV Report

Now with the filters and master record set up, you can preview the changes you are making to your data. That way, you can check to ensure your deduplication configuration is working as expected before those changes are pushed to your live database. The CSV report that Insycle generates includes records from all the duplicate groups.

Under Step 5, click the Review button and select Preview mode.

review_4.png

Click the Next button to go to the Notify screen, where you can select recipients for the email report. You can also add additional context on this screen.

On the When tab, click the Run Now tab, and select which records to apply the change to (in most cases this will be All), then click the Run Now button.

review_3.png

Insycle will generate a preview CSV and send it to your email. Open the CSV file from your email in a spreadsheet application.

bulk-merge-_1.png

The rightmost Result column identifies which were picked as master, and which were identified as duplicates and merged into the master. You'll see the values:

  • Duplicate – The record is part of a duplicate group.
  • Master The master record that was chosen for the duplicate group based on your rules.
  • Master (After) – For each duplicate group, the Master (After) row will show the data the final record will contain, based on master selection and field data retention settings.
  • Error – If Insycle was not able to determine which record would be the master, an error message will appear here. See the Troubleshooting section below for more detail.

When a field value in the CSV says "(Default)," it means that the CRM will be using its default processes for dealing with the field. This is typically done for blank fields, system IDs, and other specific situations.

If everything in the Result column looks good, return to Insycle and move forward with applying the changes.

Apply Changes to Your CRM Records

When you're satisfied with the results in your preview, you can apply the merge changes to your CRM.

Under Step 5, click the Review button, then select Update mode.

Click the Next button to go to the Notify screen, where you can select recipients for the email report. You can also add additional context on this screen.

Click the Next button. As a best practice, the first time you are applying these changes to the CRM you should click Run Now.

review_2.png

Save Templates and Setup Automation

After you've seen the results in your CRM and are satisfied with how the operation runs, you can save your configuration as a template, and set up automation so this merge operation runs on a set schedule. If you have several templates you'd like to automatically run together, you can create a recipe.

By automating with a template, you'll save time and ensure that your leads and contacts are merging consistently on an ongoing basis.

Return to the Template menu at the top of the page and click Copy to save your configurations as a new version of whatever template you started with. Then click the pencil to edit your new template name.

save-template-copy-and-rename.png

Under Step 5, click the Review button, and select Update mode.

On the Notify tab, select the send option appropriate for your automation: Always send, Send when errors, or Do not email.

Add any additional recipients who should receive the CSV (and make sure to hit Enter after each address). You can also provide additional context in the message subject or body.

merge-duplicates-step-5-update-notify-tab-always-send.png

On the When tab, select Automate and configure the frequency you'd like the template to run. When finished, click Schedule.

review_1.png

Tips for Merging Duplicates

  • Begin with easy-to-find duplicates. Iterate through fields and rules you know will surface duplicates. Don’t expect to resolve all your duplicates by setting up and running this process once. You will need to run this process multiple times for different fields or nuanced variations.
  • Each time you get a Merge Duplicates process to run the way you want in your database, save it as a template. When you have a solid set of templates that reliably resolve most of your dupes, you can put them together as a Recipe that can run on a regular, automated schedule.
  • You may also need to look for edge cases that fall outside your standard rules. These may be templates you run manually, so you can adjust based on what you find.
  • Do some experimentation. Use the Preview mode and CSV report to analyze patterns in the duplicates. You may learn what is causing the duplicates and learn how to avoid having them in the first place.

Advanced How-Tos

Step 1: Narrowing Down the Records with a Filter

Use the filter to work with a segment or smaller pool of records. Then Insycle will only analyze the remaining records for duplicates. To add filters, click the Filter button, then choose the field to look at, select the condition, and set the value to look for. The filter is applied before the matching step runs. 

dup10.png

You may want to use a filter if:

  • You know you only want to work with a subset of your data, so there’s no need to run the operation on your whole database.
  • There are an overwhelming number of duplicate results. Add a filter to work with a reasonably sized subset while you work to get the configuration right. 
  • You want the operation to run faster. A refined segment can speed things up since there are fewer records to analyze.

Most of the options in the Field dropdown match the fields that are found in your CRM, and for Contact records, there are three additional options related to the Email value: 

  • Email Username: The portion of the email address before the “@.” For example, if the email address were “maria@acmewidgets.com,” the username value would be “maria.” 
  • Free Email Provider Domain: Choose True to filter out records where the email domain is Gmail, Hotmail, Yahoo, and about 10,000 other free email providers. This filter helps ensure these are real business contacts, or can determine which record is the legitimate one because it’s most likely customer companies aren't using free Gmail accounts (though a contact may have accidentally emailed us from a private account at some point). 
  • Email Top-Level Domain: The top-level domain (TLD) is everything that follows the final dot of a domain name. For example, in the domain name acmewidgets.com', '.com' is the TLD. Some other popular TLDs include '.org,' '.uk,' and '.edu.' 
Step 1: Setting Up the Fields
Field Name Comparison Rule Ignored Match Parts

Pick a field that you think has some duplicate values.

Running a very simple match operation like just First and Last Name can be helpful in giving you an idea of what you have, but it is too broad to use for reliable analysis and deduplication. There may be legitimate duplicate names–different people with the same first and last name. You need additional, unique criteria to narrow it down.

Choosing Unique Identifiers

Matching duplicates requires unique identifiers—data that is unlikely to be shared by any other record unless it is a duplicate. If you don't use unique identifiers, you are likely to identify unrelated records as duplicates and may accidentally merge them.

Many CRMs match first names, last names, and email addresses. If all of those match, or are similar, you can confidently determine that the record is a duplicate.

Other unique identifying fields that are commonly used in deduplication include:

    • Phone number
    • Mailing address
    • ID numbers

Each row in your matching fields setup is cumulative, so records must meet all of the criteria. For example, looking for records that have the same First Name, AND Last Name, AND Phone Number returns only results where all three values are the same.

To match against one field value OR another, you will need to run two different templates. For example, if you want to use fields like Phone Number OR Mobile Phone Number, you’ll run one template for Phone Number, then a second configured the same except with the Mobile Phone Number field.

The searched value must have four or more characters. For example, values of “Joe” will be ignored.

Step 1: Matching Using Two Different Fields 

Sometimes, you might want to match duplicates using data in two separate fields. For example, you might want to compare your Phone Number field to a Mobile Phone Number field to identify duplicates.

Using the Related Fields feature, you can use two different fields (that contain similar data) as matching fields to catch more duplicates.

You can set up Related Fields in the Advanced tab.

bulk-merge_2.png

Common Examples of Related Field Matching

Matching Field Related Fields
Phone Number Mobile Phone Number, Company Phone
Email Domain Website, Company Domain
Address Company Address
Step 4: Considerations When Picking a Master Record

For contacts, it's often useful to pick master records based on engagement. For example, the highest number of email clicks, or the most recent email opened. You can also use other statuses to pick a master record such as the furthest along in your sales lifecycle, or the most recently updated record. 

For companies, it's often useful to use associated records to determine the master record. For example, the highest number of associated contacts or deals. 

If you have a connected app, like Salesforce or an ERP system, pick the master record that is syncing with the other apps.

Step 4: Selecting Priority Match vs Absolute Match

step-4-priority-match-no-arrow-2023-06-01.png

Priority Match: Looks through the master selection rules in order, one by one. As soon as a record meets one of the criteria, Insycle makes the master selection and skips the rest of the rules on the list. The vast majority of duplicate templates should use Priority Match.

Absolute Match: The master record must meet all of the listed rules in the Record tab in Step 4. If a record does not match every rule listed, no master record will be identified. Absolute Match is appropriate for less flexible master selection.

For example, if a company wanted to ensure the chosen master record is in their sales pipeline and already has a sales rep working the record, they can choose Absolute Match and set the Record rules:

  • Lifecycle Stage is lead
  • Contact Owner exists

Choosing Absolute Match can often result in no master record being identified since the record has to match every rule listed, so in most cases, you should select Priority Match.

Step 4: Control What Field Data is Retained

The Merge Duplicates module allows you to control the values saved in the master record after the merge regardless of the default merge behavior. By adding each field you want to control the data retention for and selecting a Condition, you can tell Insycle where the data for the field should be taken from and how to handle it.

For example, if merging HubSpot companies, by default the HubSpot field “Merged Company IDs” would not be populated with the Record IDs of the duplicates that were merged into the master record. 

Say you want to save all of the Record IDs from records that are merged together and deleted. You can add a new custom field, “Insycle Merged Record IDs” to your CRM.

hubspot-field-insycle-merged-record-ids.png

Then in the Merge Duplicates module under the Fields tab of Step 4, add a rule to override the default merge behavior. Select the "Insycle Merged Record IDs" field, the "Collect all values from other field" criteria, and "Record ID" as the other field. 

step-4-fields-collect-all-values-from-other-field.png

You can use the Preview to see how this will preserve the Record IDs of all the duplicates in each duplicate group.

step-4-collect-all-values-from-other-field-CSV-example.png

Step 4: Customizing Merge Logic

For situations where you need more granular customization for picking duplicate records to include—or exclude—from the deduplication process, you can customize bulk deduplication using exclusions and pre-defined masters. Additionally, you can use this process when there are no common rules you can apply to choose the master record.

Separately, it is also possible to customize the merge logic behavior. For example, you can instruct Insycle to copy values from field A to field B as part of the merge, or combine multiple fields into one. Customized merge logic requires an Enterprise plan. Please reach out via chat to discuss your specific requirements.

Troubleshooting

If you're not seeing the results you expect when merging duplicates, consider these issues:

Not all identified duplicates are merging into the master

You have duplicate records that have been identified by Insycle but not all of them are merging into the master. Check to see how many duplicates are in the affected duplicate groups. If you have duplicate groups that contain more than five records, you may want to change the value in Skip duplicate groups with more than 5 records per group to make sure you can get them all.

merge-duplicates-step-3-bulk.png

This setting is intended to protect against the accidental merging of non-duplicate records if the filter in Step 1 is too broad.
"Cannot determine master record..." Result in CSV

If the Result column of the CSV report displays this error:

Cannot determine master record because multiple records (#) satisfy the master selection rules. In ‘Master Selection’, change/add/reorder the rules such that only one record satisfies them (if cannot determine master based on field values, use ‘ID is lowest’ as the last rule).

merge-duplicates-salesforce-contacts-csv-w-error.png

This means that based on all the rules, Insycle could not figure out which record in the duplicate group would be the master. None of the records meet more of the rules than others.

There are a couple of things you can try to resolve this:

  1. Under Step 4, experiment with reordering or adding additional fields that are likely to have unique values.
  2. In the Step 4 heading, check to ensure that you have Priority Match selected and not Absolute Match.
    With Priority Match, your master record only has to match one rule. Using Absolute Match, your master record would have to meet all of the rule criteria. The majority of the time it is best to select Priority Match.
    If Priority Match was used, then none of the records in the duplicate group meet any of the criteria on the list more than the others. In this case, you'll need to experiment, reordering or adding additional rules for fields likely to have unique values.
Non-duplicate records are being merged together

There are a couple of things to look at that may be misidentifying records as duplicates.

First, you may need a better unique identifier. Under Step 1, if you only use fields that could correctly contain the same values in multiple records, these aren't unique identifiers. In this case, you are likely to identify unrelated records as duplicates and may accidentally merge them.

Unique identifiers are data that is unlikely to be shared by any other record unless it represents the same underlying entity. Fields commonly used in deduplication include phone numbers, email, mailing addresses, and ID numbers.

Second, this may indicate the Comparison Rule under Step 1 is too broad. Try using the Exact Match comparison rule instead of Similar Match. Similar Match looks for values that may be close but with a one-character difference (maybe a typo) which broadens the search. 

Remember, always run your deduplication in Preview Mode to confirm things are working as expected before running them in Update Mode and applying the changes to your CRM records.

Insycle isn't detecting any duplicates

Most of the time, when Insycle can't find duplicates, it is due to your matching rules in Step 1. It is important to analyze the underlying data to better understand how to set up your rules. A useful exercise can be to set up your matching filters to look for exact matches of just First Name and Last Name. 

step-1-fname-Lname-only.png

When you click Find, these rules can show you a broad overview of what duplicates are potentially in your database, and what fields might be useful to include in your matching fields. These settings are just for discovery and should not be used for a final merge operation; many people can have the same first and last names and are not duplicates. 

To get further context, click the gear button on the right side of the Record Viewer pane. Here, you can add any field in your database as a column to the Record Viewer to better understand the data inside of these records. 

dup15.png

It's taking a long time for Insycle to find duplicates

It can take a while for Insycle to find and match duplicates if the fields being used to identify them have very long values. The longer the values, the longer it takes Insycle to process the data and generate the results. This might come up when looking for matches based on long ID numbers, LinkedIn bio links, or other URLs with long strings attached (ex, https://www.linkedin.com/in/svadin%C3%ADr-n%C4%9Bmec-1234b31a3/).

You can speed this up by limiting how much of the value Insycle looks at.

If the beginning or ending portion of the values are all unique, you can limit the comparison to the first or last several characters using the Match Parts parameter under Step 1

merge-duplicates-linkedin-bio-step-1-match-parts-last-9-chars.png

merge-duplicates-linkedin-bio-step-2-last-9-chars.png

Or use the Ignore Text (Substrings) parameter, then click the Terms button.

merge-duplicates-linkedin-bio-step-1-ignored-text-terms-button.png

On the Ignored Text tab of the popup, add the common portion of the URL or text string.

merge-duplicates-linkedin-bio-step-1-ignored-text-popup.png

For more help troubleshooting issues with Insycle, refer to our Troubleshooting Issues article.

Frequently Asked Questions

What duplicate group will records with blank values appear in?

In cases where two different phone numbers and empty values from the duplicates are present, they will match as different duplicate groups. Insycle decides how to match the blank field automatically.

In this example (viewed in the Grid Edit module), Record 1 has the phone number 888-555-1200, Record 2 has no phone, and Records 3 and 4 have the phone number 888-555-1212.

duplicates-in-grid-edit.png

When Insycle matches the duplicates, it will either create two duplicate groups:

  • The first duplicate group could be made up of Records 1 and 2
  • The second duplicate group could be made up of Records 3 and 4

duplicate-groups-one-w-empty-phone.png

Or, it will match Records 2, 3, and 4 in one duplicate group while Record 1 is left on its own.

duplicate-group-w-empty-phone.png

How do I ensure that I am not merging non-duplicate records together?

Currently, there are two ways to make sure that the records that you are merging are indeed duplicate records.

First, always run your deduplication templates in Preview Mode before running them in Update Mode. This produces a CSV that shows you how your records would have been merged. Then you can ensure that your Merge Duplicates template is working as expected and not merging non-duplicate records together.

Additionally, you can reduce the risk when merging duplicates by narrowing your duplicate matching settings in Step 1. Try the Exact Match Comparison Rule instead of Similar Match. Then make sure that you are using actual uniquely identifying fields—first name, last name, email, and phone number are popular choices. The more tightly defined your filter is, the less likely you are to merge non-duplicate records.

Insycle is having trouble determining a master record. What could be causing this issue?

If the Result column of the CSV report displays this error:

Cannot determine master record because multiple records (#) satisfy the master selection rules. In ‘Master Selection’, change/add/reorder the rules such that only one record satisfies them (if cannot determine master based on field values, use ‘ID is lowest’ as the last rule).

merge-duplicates-salesforce-contacts-csv-w-error.png

This means that based on all the rules, Insycle could not figure out which record in the duplicate group would be the master. None of the records meet more of the rules than others.

There are a couple of things you can try to resolve this:

  1. Under Step 4, experiment with reordering or adding additional fields that are likely to have unique values.
  2. In the Step 4 heading, check to ensure that you have Priority Match selected and not Absolute Match.
    With Priority Match, your master record only has to match one rule. Using Absolute Match, your master record would have to meet all of the rule criteria. The majority of the time it is best to select Priority Match.
    If Priority Match was used, then none of the records in the duplicate group meet any of the criteria on the list more than the others. In this case, you'll need to experiment, reordering or adding additional rules for fields likely to have unique values.
Can I select which data is retained in my master record on a field-by-field basis?

Yes, Insycle allows you to select which field data is retained in the master record using the Fields tab under Step 4. See Step 4: Set Rules for Master Record Selection and Data Retention section of this article for more details.

dup19.png

I need to exclude some records from deduplication. Can I do that?

Yes. You can exclude records from deduplication by including a Deduplication Exclude field in your CSV, as detailed in this Customize Bulk Deduplication Using Exclusions and Pre-Defined Masters article.

Can I match duplicates using two different fields?

No. It is not possible to match duplicates using data in two separate fields when using the Empty Value condition. These matching features don't work together.

The Advanced tab of Step 1 does include a Related Fields feature that allows you to use two different fields that contain similar data as matching fields to catch more duplicates. If you prefer to use the Related Field instead of allowing an Empty Value, you can learn more about matching using related fields.

My team needs to review and approve the master, can I accommodate that with Insycle?

Yes, there are several ways to share details and get approval before merging duplicates.

You can manually approve master records and mark them in a CSV, then use Insycle to bulk deduplicate down to those master records. Consult with this Customize Bulk Deduplication Using Exclusions and Pre-Defined Masters article to learn more.

Or, you can run the Merge Duplicates module in Preview Mode, then deliver the preview CSV that Insycle generates. The CSV report that Insycle generates includes your entire merge operation down to individual duplicate groups but does not update your live data. Then your team can approve the merge based on this report, before running Merge Duplicates in Update Mode.

Additionally, team members can review duplicates and manually select the master for each record under Step 4. Review the Manually Merge Duplicates article for more detail.

dup20.png

Do the field values I use to match need to be exactly the same?

No. The Similar Match Comparison Rule found in Step 1 looks for values that may be close but with a one-character difference (maybe a typo) and broadens the search.

step-1-email-only.png

This search behaves like when Google shows results for a slightly different term or says, “Did you mean...” For example, if an Email of “huey@coahulldu.co” is found, it could include records with the values “hueyy@coahulldu.co" or "hue.y@coahulldu.co” as a match.

similar-match-_1.png

You should be careful when using Similar Match as the looser criteria can incorrectly identify non-duplicates as duplicates. 

Review the Similar Matching best practices for more detail.

I already have a list of duplicates, can Insycle bulk merge them?

Yes. You can create a customized list of duplicates, tag them in your CRM with the Magical Import module, and deduplicate in bulk using the Merge Duplicates module. In your CSV, include ID numbers from your connected CRM.

Why can I only process 50 duplicate groups at a time?

Insycle shows 50 records on the module screen as a preview; this isn't the entire list of records. View the Preview CSV report to see the results for all records.

Insycle can process thousands of duplicate groups in one operation. Potentially, you could deduplicate your entire database in one operation. 

How many duplicates can I merge into one master record?

You can merge up to 100 duplicates into one single master record. 

If you have duplicate groups that contain more than five records, you may want to change the value in Skip duplicate groups with more than 5 records per group under Step 3 to make sure you can get them all.

merge-duplicates-step-3-bulk.png

This is a precaution to ensure that if you use a duplicate matching filter that is too broad in Step 1, you do not accidentally merge many non-duplicate records together. If you are going to set this number at a high level, it is a good idea to run Preview Mode first to make sure your deduplication template is operating as you intend.

Are there any limits on the number of records that can be identified and merged with my paid subscription?

All paid plans include unlimited usage, unlimited users, and unlimited operations. See the pricing page for more details. During the free trial, there is a cap of 500 records updated, cleansed, or merged.

Additional Resources

Related Help Articles

Related Blog Posts