Customize How Duplicates Are Merged Using a CSV

Rather than create a CSV file from scratch, you can use a Preview CSV generated by Insycle as a starting point for either method.

Navigate to Data Management > Merge Duplicates, and select the database and record type in the top menu. Then explore the templates for solutions that may be close to what you need.

Next, go through Steps 1-4 of the module to create a Preview CSV. Make sure to pause there; you don't want to update your records yet. Follow the instructions found in the Module Overview: Merge Duplicates article through to Step 4: Review.

Insycle will generate a preview CSV and send it to your email.

Pro Tip

To add all the fields you want to see and work with to the CSV, click the gear icon to the right of the Step 2 heading.

On the Layout tab of the popup, find the desired field in the right column, and drag it left to the Visible Fields column. Drag-and-drop fields to reorder them in the Visible Fields column.

The CSV to be used in Step 1 of the Merge Duplicates module must include a header row with exactly these two column names:

• ID_1
• ID_2

Each row should contain the record IDs of one duplicate pair. Insycle will process only these two columns; any additional columns in the file will be ignored.

You can create a CSV from scratch, or use a Preview Mode CSV report as outlined in Optional: Create a CSV Report of Duplicates above. This lets you use the matches identified by Insycle as a starting point.

If you are starting from a Preview Mode CSV, you will need to reformat it before uploading. Rename the record ID column to ID_1, then add a new ID_2 column and enter the corresponding duplicate record ID for each pair. You can delete all other columns, leaving only ID_1 and ID_2 with a header row.

Navigate to Data Management > Merge Duplicates and select your database and record type.

In Step 1 of the module, click the CSV tab. Then click the Upload button and select your CSV file.

Once uploaded, Insycle validates that the record IDs in the file exist in your CRM. Rows with invalid or missing record IDs will be skipped and will not appear in Step 2.

Once the upload is complete, proceed to Step 2 to review the duplicate pairs Insycle has identified before configuring your merge settings.

In Step 3. Merge Logic, configure how your duplicates will be merged. For detailed guidance on all the Merge Logic options, see the Module Overview: Merge Duplicates.

On the Master tab, set the rules that determine which record in each pair becomes the master—the record that will remain after the merge. 

On the Fields tab, control which field values are retained in the master record after the merge. For details on what happens if you don't create Field rules, see the Frequently Asked Questions below.

The Merge tab (HubSpot and Salesforce only) lets you choose the merge logic, and control whether empty master fields are automatically filled.

For complete instructions on all Merge Logic options, refer to the Module Overview: Merge Duplicates.

When you're satisfied with the results, return to Step 4. Review and select Update mode to apply the changes to your CRM.

In Step 4, use Preview mode first to confirm your merge configuration is working as expected.

On the When tab, click the Run Now tab, select which records to apply the change to (if you have many records, you might want to test only a few), and then click the Run Now button.

Insycle will generate a Preview CSV file and send it to your email. Review it to verify that the correct records are being selected as masters.

The CSV includes records from all duplicate groups. It shows the current state of the master record, excluding reassignments of associated contacts, deals, or activities. 

 The Status column indicates:

• Duplicate – The record is part of a duplicate group.
• Master – The master record is chosen for the duplicate group based on default behavior and your Master rules. Review the selections in this row to determine whether the appropriate records are being chosen.
• Master (After) – For each duplicate group, the Master (After) row shows the values the final record will contain based on your Field rules and the default behavior.
• Error – If Insycle cannot determine which record is the master, an error message will appear here. See the Troubleshooting section below for more details.

Apply Changes to Your CRM Records

When you're satisfied with the preview results, you can apply the merge changes to your CRM.

Under Step 4, click the Review button again, and select Update mode this time.

On the When tab, you should use Run Now the first time you apply these changes to the CRM. If you have a large number of records, consider doing a smaller batch to assess the results in your CRM.

If the Merge API is set to Synthetic on the Module tab in Step 3, clicking Run Now in Update Mode displays a Please confirm dialog before the operation begins. This dialog shows the number of records being merged and an estimated time for the operation. 

Click Yes to proceed. 

You can use a Preview Mode CSV report as outlined in Create a CSV Report of Duplicates above. Open the file that was emailed to you in a spreadsheet application. 

The CSV will have a row per record and include the following columns:

• The presumed Result of the action if you were to merge right now
• A Message with details
• The Duplicate Group ID indicates which records will be merged together
• A record Status that identifies which were picked as master and which were identified as duplicates and merged into the master. See below for more details
• All fields used to identify the duplicates in Step 1
• The record ID, record name, or email, and Deeplink to the CRM record
• All fields selected for the Master and Fields rules in Step 3. Note that if a field is used on both tabs, it will only appear once in the CSV
• Any additional fields added to the Layout tab in Step 2

Add two new columns to the CSV:

• Deduplication Exclude
  and
• Deduplication Master

Now, given the context from all the fields available in the report, update the CSV as needed:

• Enter TRUE in the Deduplication Exclude column for any row you'd like to exclude from the merge.
• Enter TRUE in the Deduplication Master column for the row you'd like to designate as the master from within a duplicate group.

When you're done, delete all columns except Record ID, Deduplication Exclude, and Deduplication Master, then save the file as a new CSV to your computer.

In your CRM, create two custom fields:

• Deduplication Exclude
  and
• Deduplication Master

For the field type, pick "Boolean" or "Checkbox."

The underlying field names for these fields in the CRM are: 

• deduplication_exclude
• deduplication_master

After creating the CRM fields, refresh the data in Insycle by navigating to Settings > Sync Status, and next to the account name, click the Sync changes from last day button (lightning bolt icon).

Open the Magical Import module and load the saved CSV file with your new custom fields.

Under Data Mapping, map the CRM's ID column to the CSV Record ID field. The two Deduplication fields will map automatically.

Select the ID field for the Matching Criteria field.

At the bottom of the Magical Import page, under Action, select the Records Mode, Only create net new, because those records already exist in the CRM. You're just "tagging" those existing records for deduplication.

Then click Import.

Now that records in the CRM have deduplication tags, you can use the custom fields to tailor the cleanup process.

Return to the Merge Duplicates module. In Step 1, click the Filter button to exclude records from deduplication.

For the filter settings in the popup, select the Deduplication Exclude field, the "not any of" Condition, and enter "Yes" in the value field.

To designate your chosen records as the masters, in Step 3, remove all rules and add just one rule: Deduplication Master "is "Yes."

Preview Your Changes in a New CSV Report, Then Apply the Updates to Your CRM

Under Step 4, use Preview mode to confirm that the deduplication works as expected with all the customizations. 

If everything in the CSV looks correct, return to Insycle and move forward with applying the changes to your CRM using Update mode.

Yes, Insycle allows you to select which field data to retain in the master record using the Fields tab in Step 3. See the Merge Duplicates module field data retention rules reference for more details.

No, matching fields do not need to match exactly. The Similar Match Comparison Rule in Step 1 acts as a fuzzy match, looking for values that may be close but differ by only one character (e.g., a typo), and broadens the search.

This search behaves like when Google shows results for a slightly different term or says, “Did you mean...” For example, if an Email of “huey@coahulldu.co” is found, it could include records with the values “hueyy@coahulldu.co" or "hue.y@coahulldu.co” as a match.

You should be careful when using Similar Match as the looser criteria can incorrectly identify non-duplicates as duplicates. 

Review the Understanding Similar Matching best practices for more details.

There is no need to create rules for every field in your CRM - Insycle automatically handles fields without specific rules using a "fill in the blanks" approach. If you don't specify any rule, the master values will be preserved. When a value is empty in the master, it picks a non-empty value from the most recently updated duplicate. 

For example, if the master record's Industry field is empty but another record in the duplicate group has an Industry value, that value will automatically be copied to the master record. This means you only need to create custom retention rules for the handful of fields that require special handling, rather than for all your fields.

HubSpot and Salesforce users can override this by disabling the Blank Fields autofill setting on the Method tab of Step 3.

You can merge up to 100 duplicate records into a single master record.

By default, Insycle skips any duplicate group containing more than 5 records. If you have duplicate groups with more than 5 records, you may want to adjust the Exclusions setting at the bottom of the Master tab in Step 3. The Skip duplicate groups with more than X records per group setting can be increased up to 100 records per group to ensure all duplicates in large groups are processed.

This default limit is a precaution to prevent duplicate matching criteria that are too broad in Step 1 from accidentally merging many non-duplicate records. If you plan to set this number higher, it is a good idea to run Preview Mode first to ensure your deduplication template is operating as you intend.