Small SEO Tools

Why Removing Duplicate Lines Can Break a CSV File

A CSV note can contain a line break inside one quoted field. Removing repeated physical lines can delete part of that field and damage a record that should remain. Check the record structure before using line cleanup on an export.

Identify the complete records

This sample contains a header and three data records. Each note is the two-line value call followed by today. The final Aster record repeats the first one.

name,notes
Aster,"call
today"
Vale,"call
today"
Aster,"call
today"

The repeated physical line today" closes a quoted field in each record. Its repetition alone does not make it removable. RFC 4180 describes quoting fields that contain line breaks in its CSV format description.

Reproduce the failure from line removal

For a controlled test, paste the sample into the Duplicate Line Remover, without a final newline. Uncheck all five options, including Trim whitespace before comparing and Ignore blank lines. Press Remove Duplicates.

The tool reports seven lines in, four unique lines, three duplicates removed and four lines out.

name,notes
Aster,"call
today"
Vale,"call

The Vale field now has no closing quotation mark. Python’s CSV reader with strict=True raises unexpected end of data for this output. Its documentation distinguishes physical line counts from record counts because records can span several lines. See the Python CSV documentation.

Deduplicate parsed records from the original

Return to the saved original. Decide what makes two records equal. For this example, compare every parsed field exactly, so Aster and Vale remain separate even though their notes match. Keep the first occurrence of each complete record and preserve the header separately.

The following complete Python 3 example reads the original sample, removes its one repeated record and writes new CSV text. It uses commas, double quotes and explicit LF record terminators.

import csv
from io import StringIO

source = '''name,notes
Aster,"call
today"
Vale,"call
today"
Aster,"call
today"'''

reader = csv.reader(StringIO(source, newline=""), strict=True)
header = next(reader)
seen = set()
unique_rows = []
for row in reader:
    key = tuple(row)
    if key not in seen:
        seen.add(key)
        unique_rows.append(row)

output = StringIO(newline="")
writer = csv.writer(output, lineterminator="\n")
writer.writerow(header)
writer.writerows(unique_rows)
cleaned = output.getvalue()
print(cleaned, end="")

Check fields after serialization

The result contains the original header and two data records.

name,notes
Aster,"call
today"
Vale,"call
today"

Read the result with a CSV parser again and check the field values. Both notes should still contain their internal newline, with Aster followed by Vale. Physical line counts cannot establish those facts.

For an actual export, identify its header, delimiter, quoting rules and intended record key before adapting the example. Preserve the original while checking the result in its receiving application. Our plain-list deduplication guide covers lists where each complete item occupies one line.