Cleaning Product Catalog Data Without Losing Rare Attributes

Preserve long-tail categories while removing duplicate and contradictory catalog records. The useful boundary is a repeatable workflow with named inputs, explicit checks, and a release decision that another contributor can inspect. The examples below are illustrative procedure, not a claim about a particular customer or production result.

Set the operating boundary

Define a compact input schema for titles, categories, attributes, and source IDs. Treat that requirement as a recorded engineering decision rather than an informal cleanup step. Start with a small, versioned input set and keep the original source identifier beside every transformed record. That lets a reviewer trace an unexpected output back to the data, configuration, or evaluation rule that produced it.

The implementation should separate deterministic checks from judgment calls. Deterministic checks can reject missing fields, malformed files, invalid ranges, and inconsistent identifiers. Judgment calls need a documented review rule, a sample of borderline cases, and a reason for keeping or excluding them. Mixing the two creates silent changes that are hard to reproduce later.

Normalise whitespace, units, casing, and empty values without erasing meaningful distinctions

Normalise whitespace, units, casing, and empty values without erasing meaningful distinctions. Treat that requirement as a recorded engineering decision rather than an informal cleanup step. Start with a small, versioned input set and keep the original source identifier beside every transformed record. That lets a reviewer trace an unexpected output back to the data, configuration, or evaluation rule that produced it.

Record the result in a small manifest: source version, transformation version, input digest, output digest, reviewer, and timestamp. A compact manifest is more useful than a long narrative when a later run needs to be compared with this one.

Detect exact and near duplicates while retaining rare attribute combinations

Detect exact and near duplicates while retaining rare attribute combinations. Treat that requirement as a recorded engineering decision rather than an informal cleanup step. Start with a small, versioned input set and keep the original source identifier beside every transformed record. That lets a reviewer trace an unexpected output back to the data, configuration, or evaluation rule that produced it.

Stratify train and validation sets by category frequency

Stratify train and validation sets by category frequency. Treat that requirement as a recorded engineering decision rather than an informal cleanup step. Start with a small, versioned input set and keep the original source identifier beside every transformed record. That lets a reviewer trace an unexpected output back to the data, configuration, or evaluation rule that produced it.

Summary checks for missingness and category coverage

Show summary checks for missingness and category coverage. Treat that requirement as a recorded engineering decision rather than an informal cleanup step. Start with a small, versioned input set and keep the original source identifier beside every transformed record. That lets a reviewer trace an unexpected output back to the data, configuration, or evaluation rule that produced it.

Connect the cleaned output to the `qwen7b-product-taxonomy` use case

Connect the cleaned output to the qwen7b-product-taxonomy use case. Treat that requirement as a recorded engineering decision rather than an informal cleanup step. Start with a small, versioned input set and keep the original source identifier beside every transformed record. That lets a reviewer trace an unexpected output back to the data, configuration, or evaluation rule that produced it.

Practical check

Working Python script using only the standard library. The following minimal check is intentionally small enough to run before a longer workflow. Expand it only after its output identifies the missing condition or failure category.

import json
import sys
from pathlib import Path

path = Path(sys.argv[1])
valid = 0
with path.open(encoding="utf-8") as handle:
    for line_number, line in enumerate(handle, 1):
        if not line.strip():
            raise SystemExit(f"line {line_number}: blank record")
        json.loads(line)
        valid += 1
print(f"validated {valid} records")

Review and release criteria

A result is ready for the next stage only when the required fields, split boundaries, and evaluation evidence agree. Keep a rejected sample set as well as accepted output; otherwise the same edge case will return as an unexplained regression. Depending on a private taxonomy or proprietary dataset. The correct response to uncertainty is to label the limitation and add a focused validation case, not to hide it behind a blended score.

Before publishing, verify the data digest, configuration, expected output shape, and the most important rare-case slice. The final registry note should state what changed, what was checked, and what remains outside the adapter's intended scope. That makes the next iteration a comparison instead of a guess.

A small review sample should include ordinary records, rare records, and inputs designed to fail cleanly. Each slice needs a stable identifier so that changes can be compared across releases without rebuilding the entire test set.

Keep preprocessing idempotent. Running the same transformation twice on the same input should not progressively remove detail, append labels, or alter identifiers. Idempotence makes retries safe and exposes hidden state in the pipeline.

Use a separate holdout set for release decisions. Training examples can guide implementation, but they cannot demonstrate that the workflow generalises to new wording, new sources, or changed input order.

When a check fails, preserve the smallest reproducible example. A concise failing record and its expected result are usually more valuable than a large unlabelled export because they can become a permanent regression test.