Choosing Which Records to Merge Using a CSV

Rather than creating a CSV file from scratch, you can use a Preview CSV generated by Insycle as a starting point for either method.

Navigate to Data Management > Merge Duplicates, and select the database and record type in the top menu. Then explore the templates for solutions that may be close to what you need.

Next, go through steps 1-4 of the module to create a Preview CSV. Pause there; don't update your records yet. Follow the instructions found in the Module Overview: Merge Duplicates article through to  4. Review.

Insycle will generate a preview CSV and email it to you.

The image above shows the Mode tab of the Review dialog in the 4. Review step of the Merge Duplicates module. Two radio button options are listed. “Preview - For analysis purposes. No data is modified.” is selected, with a yellow highlighted radio button. “Update - Data in your CRM will be updated.” is not selected. A Cancel button appears at the bottom left and a yellow Next button at the bottom right.

Pro Tip

To add all the fields you want to see and work with to the CSV, click the gear icon to the right of the 2. Review Duplicates heading.

On the Layout tab of the popup, find the desired field in the right column, and drag it left to the Visible Fields column. Drag and drop fields to reorder them in the Visible Fields column.

The image above shows the 2. Review Duplicates step of the Merge Duplicates module. A teal arrow points to the gear icon, which opens the Layout settings where you add fields to the Visible Fields column.

The CSV to be used in 1. Find Duplicates of the Merge Duplicates module must include a header row with exactly these two column names:

• ID_1
• ID_2

Each row should contain the record IDs of one duplicate pair. Insycle will process only these two columns; it will ignore any additional columns in the file.

The image above shows a CSV file open in a spreadsheet application, formatted for uploading a list of duplicate record ID pairs in 1. Find Duplicates of the Merge Duplicates module. The header row has three columns: ID_1, ID_2, and Company name. Each of the six data rows is populated. A teal callout box reading “This column will be ignored” has an arrow pointing to the Company name column header, indicating that Insycle does not process that column.

You can create a CSV from scratch, or use a Preview Mode CSV report as outlined in Optional: Create a CSV Report of Duplicates above. This lets you use the matches Insycle identifies as a starting point.

If you are starting from a Preview Mode CSV, you will need to reformat it before uploading. Rename the record ID column to ID_1, then add a new ID_2 column and enter the corresponding duplicate record ID for each pair. You can delete all other columns, leaving only ID_1 and ID_2 with a header row.

Navigate to Data Management > Merge Duplicates and select your database and record type.

Under 1. Find Duplicates of the module, click the CSV tab. Then click the Upload button and select your CSV file.

The image above shows the 1. Find Duplicates step of the Merge Duplicates module with the CSV tab selected. The CSV tab is active, and a yellow Upload button with a cloud-and-upward-arrow icon appears at the left.

Once uploaded, Insycle validates that the record IDs in the file exist in your CRM. Rows with invalid or missing record IDs will be skipped and will not appear in 2. Review Duplicates.

Once the upload is complete, proceed to 2. Review Duplicates and review the duplicate pairs Insycle has identified before configuring your merge settings.

In 3. Merge Logic, configure how your duplicates will be merged. For detailed guidance on all the Merge Logic options, see Choosing Which Record Remains After a Merge.

On the Master tab, set the rules that determine which record in each pair becomes the master—the record that will remain after the merge. 

The image above shows the Master tab of the 3. Merge Logic step in the Merge Duplicates module, with Bulk selected. Four rules are listed: Company owner is active user, Number of Associated Cont is highest, Phone Number is regex $[0-9]{3}$.*, and Create Date is earliest. Processing is set to By Priority, and Exclusions is set to duplicate groups with more than 5 records.

On the Fields tab, control which field values are retained in the master record after the merge. For details on what happens if you don't create Field rules, see the What Happens to Field Data After a Merge FAQs.

The image above shows the Fields tab of the 3. Merge Logic step in the Merge Duplicates module, with Bulk selected. Four rules are listed: Contact owner is From record where value is active user; Phone Number is Most frequent value; State/Region is From record where value is WA|OR|CA; and Membership Notes is Combine and append all values.

The Method tab (HubSpot and Salesforce only) lets you choose the merge logic and control whether empty master fields are automatically filled. See Native vs. Custom Merge: Two Ways to Combine Records in HubSpot and Salesforce for details.

Under 4. Review, select Preview mode first to confirm your merge configuration works as expected. 

On the When tab, click the Run Now tab, select which records to apply the change to (if you have a lot of records, you might want to test a smaller batch), and then click the Run Now button.

The image above shows the Run Now tab under 3. WHEN in the Review dialog of the 4. Review step in the Merge Duplicates module. Four options are listed: Selected, Chunk, Custom (20, selected), and All. A “Preview Mode” label appears beside the Run Now button, with Cancel and Back buttons on the left.

Insycle will generate a Preview CSV file and email it to you. Review it to confirm the correct records are selected as masters.

The CSV includes records from all duplicate groups. It shows the current state of the master record, excluding reassignments of associated contacts, deals, or activities. 

The image above shows a CSV report from the Merge Duplicates module in a spreadsheet application, with columns Result, Message, Row, Duplicate Group ID, Status, Record ID, Record Name, Deeplink, and Lifecycle Stage. Two duplicate groups, Graso and Coahulldu, each have three rows with Status values Duplicate, Master, and Master (After). Every row shows Merged in Result, Succeeded in Message, and Lead in Lifecycle Stage.

 The Status column indicates:

• Duplicate – The record is part of a duplicate group.
• Master – The master record is chosen for the duplicate group based on default behavior and your Master rules. Review the selections in this row to determine whether the appropriate records are being chosen.
• Master (After) – For each duplicate group, the Master (After) row shows the values the final record will contain based on your Field rules and the default behavior.
• Error – If Insycle cannot determine which record is the master, an error message will appear here. See the Merge Duplicates Troubleshooting article for more details.

Apply Changes to Your CRM Records

When you're satisfied with the preview results, you can apply the merge changes to your CRM.

Under 4. Review, click the Review button again, and select Update mode this time.

On the When tab, you should use Run Now the first time you apply these changes to the CRM. If you have a large number of records, consider doing a smaller batch to assess the results in your CRM.

The image above shows the Run Now tab under 3. WHEN in the Review dialog of the 4. Review step in the Merge Duplicates module. Four options are listed: Selected, Chunk (50, selected), Custom, and All. An “Update Mode” label appears beside the Run Now button, with Cancel and Back buttons on the left.

If the Merge API is set to Synthetic on the Module tab in 3. Merge Logic, clicking Run Now in Update Mode displays a Please confirm dialog before the operation begins. This dialog shows the number of records being merged and an estimated operation time. 

Click Yes to proceed. 

You can use a Preview Mode CSV report as outlined in Create a CSV Report of Duplicates above. Open the file that was emailed to you in a spreadsheet application. 

The CSV will have a row per record and include the following columns:

• The presumed Result of the action if you were to merge right now
• A Message with details
• The Duplicate Group ID indicates which records will be merged together
• A record Status that identifies which were picked as master and which were identified as duplicates and merged into the master. See below for more details
• All fields used to identify the duplicates, those selected for the Master and Fields rules, and any additional fields added to the Layout

The image above shows a CSV report from the Merge Duplicates module in a spreadsheet application, with columns Result, Message, Duplicate Group ID, Status, First Name, Last Name, Record ID, and Record Name. Two duplicate groups appear: Marta Vaskovitch, with Duplicate, Master, and Master (After) rows, and Shu Wong, with Duplicate and Master rows. Every row shows Merged in Result and Succeeded in Message.

To prepare your CSV for importing:

1. Delete all rows where the Status column has the value "Master (After)". These can't be used to define the exclusion or master and may cause matching issues.

2. Add two new columns to the CSV:

   • Deduplication Exclude
   • Deduplication Master

3. Given the context from all the fields available in the report, update the CSV as needed:

   • Enter TRUE in the Deduplication Exclude column for any row you'd like to exclude from the merge.

   • Enter TRUE in the Deduplication Master column for the row you'd like to designate as the master from within a duplicate group.

     

     The image above shows a CSV report from the Merge Duplicates module in a spreadsheet application, with two added columns, Deduplication Exclude and Deduplication Master, whose headers are highlighted in yellow. Group 61351 has two Duplicate rows for Marta Vaskovitch, with TRUE in Deduplication Master on the first row. Group 85151 has one Master row for Vaskovitch. Group 90501 has two Duplicate rows and one Master row for Wong, each with TRUE in Deduplication Exclude.

4. When you're done, delete all columns except Record ID, Deduplication Exclude, and Deduplication Master, then save the file as a new CSV to your computer.

   

   The image above shows a CSV file in a spreadsheet application with three columns: Record ID, Deduplication Exclude, and Deduplication Master. The Deduplication Exclude and Deduplication Master headers are highlighted in yellow. Six rows are listed: record 61301 has TRUE in Deduplication Master; records 87051, 87001, and 90501 have TRUE in Deduplication Exclude; and records 61201 and 61251 have no values in either column.

In your CRM, create two custom fields:

• Deduplication Exclude
  and
• Deduplication Master

For the field type, pick "Boolean" or "Checkbox."

The image above shows the Create new property panel in HubSpot, opened from the Settings area for Contacts, with the Create manually tab selected. The Property label field contains “Deduplication Master,” and the Field type dropdown is set to “Single checkbox.”

The underlying field names for these fields in the CRM are: 

• deduplication_exclude
• deduplication_master

After creating the CRM fields, refresh the data in Insycle by navigating to Settings > Sync Status, and next to the account name, click the Sync changes from last day button (lightning bolt icon).

The image above shows the Sync Status page in Insycle Settings, with a search box and a table with columns App, Plan, Account Name, Actions, and Account ID. The first row is a HubSpot account named Amaltri on the Business plan. An arrow points to the lightning bolt icon in that row’s Actions column, which has a tooltip reading “Sync changes from last day.” A trash can icon appears beside it.

1. Open the Magical Import module and load the saved CSV file with your new custom fields.

2. Under Data Mapping, map the CRM's ID column to the CSV Record ID field. The two Deduplication fields will map automatically.

3. Select the ID field for the Matching Criteria field.

   

   The image above shows the Data Mapping section of the Magical Import module, with the Matching tab selected. The CSV columns Record ID, Email, Deduplication Exclude, and Deduplication Master are mapped to HubSpot fields of the same names, each with Field Logic set to Update. Under Matching Criteria, the Matching Fields dropdown is set to Record ID, with icons to add another rule and duplicate the current one.

4. At the bottom of the Magical Import page, under Action, select the Records Mode, Only update existing, because those records already exist in the CRM. You're just "tagging" those existing records for deduplication.
5. Then click Import.

The image above shows the “Action to take with the CSV” section of the Magical Import module, with the Import tab selected. The Records Mode is set to Only update existing, and the other options are Update existing and create net new, and Only create net new. Add to List is empty, Email CSV report to user is checked, and the yellow button reads Import 10 Contacts.

Now that records in the CRM have deduplication tags, you can use the custom fields to tailor the cleanup process.

Return to the Merge Duplicates module. In 1. Find Duplicates, click the Filter button to exclude records from deduplication.

The image above shows the Simple tab of the 1. Find Duplicates step in the Merge Duplicates module. Three matching rules are listed: First Name with Exact Match, Last Name with Exact Match, and Email with Similar Match. Each rule has Ignored set to Symbols and Match Parts set to Entire Value. Find, Field, Filter, and Terms buttons appear at the bottom, and the Filter button has a badge showing 1.

For the filter settings in the popup, select the Deduplication Exclude field, the "not any of" Condition, and enter "True" (or Yes, depending on your custom field) in the value field.

The image above shows the Filter tab of the Advanced Settings dialog in the 1. Find Duplicates step of the Merge Duplicates module. One filter rule is set: the Deduplication Exclude field, the condition “not any of,” and the value True. Field and Clear buttons appear below, along with a yellow Search button.

To designate your chosen records as the masters, in 3. Merge Logic, remove all rules and add just one rule: Deduplication Master is "True".

The image above shows the Master tab of the 3. Merge Logic step in the Merge Duplicates module, with Bulk selected. One rule is set: the Deduplication Master Record field, the condition “is,” and the value True. Processing is set to By Priority, and Exclusions is set to duplicate groups with more than 5 records per group. Rule and Clear buttons appear below the rule.

Preview Your Changes in a New CSV Report, Then Apply the Updates to Your CRM

Under 4. Review, use Preview mode to confirm the deduplication works as expected. 

If everything in the CSV looks correct, return to 4. Review in Insycle and apply the changes to your CRM using Update mode.

The image above shows the Mode tab of the Review dialog in the 4. Review step of the Merge Duplicates module. Two options are listed: “Preview - For analysis purposes. No data is modified.” and “Update - Data in your CRM will be updated.” Update is selected, with a yellow-highlighted radio button. Cancel and Next buttons appear at the bottom.