Product Enrichment

What it is

Product Enrichment turns a bare distributor SKU — a part number, a brand, a cost — into a product page: a marketing description, a grouped specification sheet, an image gallery, and the manufacturer’s manuals and datasheets. The content comes from Icecat, and once it is fetched the images and PDFs are copied into Solidlio’s own storage and served from Solidlio’s own URLs.

Enrichment operates on the global catalog. There is one shared CatalogProduct table that every tenant reads from; enrichment writes to it once, and every organization, MSP and buyer sees the result. It is not tenant-scoped and there is no per-tenant copy.

---

Concepts

The two Icecat channels

Getting them mixed up is the most common configuration failure.

All bulk paths request level4, which is the Full Icecat subscription tier.

---

Roles and permissions

Two floors do the work:

• GET is open to any authenticated user, because the catalog is shared and every tenant reads it.
• These trigger a ~26M-entry sync and paid bulk operations against the shared catalog.

---

Walkthrough 1 — Keeping the Icecat index current

This is the routine that matters most, because matching can only find products that are in the index.

Where: Platform Admin → Catalog → Icecat.

1. Reference data first. Categories, suppliers, features and measures each show a row count and a “Last sync” time on the Icecat page. Use Sync All (or the per-type buttons) if any of them is empty. Reference data must exist before product-index rows can be written — the index has database foreign keys onto icecat_suppliers and icecat_categories (migration 20260210211343_portal_enablement, lines 3480 and 3483), so an index row for a supplier we have never synced is rejected outright.

2. Full index. Press Full Sync. It is a long run — the two-stage download-then-parse design exists because Icecat drops the connection if database writes slow the stream down. Progress is reported 0–49% for the download and 50–100% for the parse.

3. Daily delta. Press Daily Sync, or leave the scheduled job to it.

4. Check it landed. The Icecat page shows the index row count and a “Last sync” time; Recent Jobs lists syncs you queued from this page with their state and progress.

Read Limits and known behaviour below before you rely on step 3 — the daily delta has no gap recovery, and the page’s freshness display cannot distinguish “nothing changed” from “the job stopped running”.

What each file contains

Matching only ever considers onMarket: true rows, so a withdrawn product stops being matchable from that point.

---

Walkthrough 2 — Matching and enriching in bulk

See Limits and known behaviour.

A match sets icecatId, icecatQuality, icecatEnrichmentStatus = PENDING and the index row’s image URL as a provisional thumbnail.

2. Each product’s outcome is recorded as ENRICHED, FORBIDDEN, NOT_FOUND or ERROR, so a re-run does not repeat work that already succeeded.

Queuing a job whose type is already waiting or active returns 409 Conflict rather than starting a second one.

---

Walkthrough 3 — Enriching one product

Where: Platform Admin → Catalog → Products → (open a product).

1. Open the product. Its Icecat ID, quality and last sync are shown in the detail panel when it is matched.
2. If it has no MPN, EAN or UPC there is nothing to match on and the call returns 400.
   • the raw API response is stored in IcecatEnrichmentData, one row per product per language;
   • name, short and long description, quality and primary image are updated on the product;
   • features are replaced wholesale and attached to their Icecat feature groups;
   • images are replaced, except any that already have stored bytes;
   • documents are replaced, except any that already have stored bytes;
   • a media-download job is queued so the images and PDFs move onto Solidlio’s own URLs.
3. The response reports how many features, images, documents and videos the record contained.

The staff-preview trigger

There is one place where enrichment happens without anyone pressing a button. The spec sheet, gallery and datasheets are then cached for every buyer who opens the page afterwards.

Buyers never trigger it.

See Limits and known behaviour for the two conditions under which this trigger misbehaves.

---

Walkthrough 4 — Working the restricted-access queue

Not every Icecat product is inside every subscription. When a lookup is refused, Solidlio records it instead of discarding it.

Where: Platform Admin → Catalog → Icecat → Access.

2. Refusals land as PENDING. A 404 is recorded as UNAVAILABLE (the product does not exist in Icecat at all), which is a different problem from a 403 (it exists but is outside the subscription).
3. Sort the queue by priority, set a priority on the rows that matter, and mark the rest IGNORED.
4. Export CSV produces icecat-access-request.csv with MPN, brand, GTIN, Icecat ID, priority, source and request date — the file to send Icecat when asking for the products to be opened up.
5. When access is granted, re-run enrichment on those products; the log rows flip to GRANTED with a resolvedAt timestamp on the next successful call.

---

Walkthrough 5 — How media reaches the browser

1. Enrichment writes originalUrl on each image and sourceUrl on each document. No bytes have moved yet; the product renders from Icecat’s URLs.
2. The bytes are tamper-evident fingerprinted.
3. The file is written to the private assets container under platform/catalog/{productId}/{uuid}-{filename}, and storageDriver + storageKey are written back onto the image or document row. Keys contain a UUID, so the immutable cache header is safe — a replaced image gets a new key.

The container is private; the anonymous route is the only way in, and it will only serve keys containing a catalog segment.

---

Configuration

Scheduled jobs

---

Plan tiers

Product enrichment carries no plan gate. The catalog is global and platform-owned: enrichment is run by the platform, and every tenant on every plan reads the same enriched product data.

Access is controlled by role, not by plan. See the permissions matrix above.

---

Troubleshooting

These are the actual strings the services return.

---

Limits and known behaviour

Read this section before depending on any of it.

The daily delta has no gap recovery

Icecat replaces that file each day.

If the daily job does not run for N days, those N days of changes are gone. The next run collects only the most recent 24 hours.

The only recovery from a missed day is a manual Full Sync. After any interruption to the daily schedule, run one.

Whether the daily sync ran is answerable — but not from this page

They move when rows are written, and stay still when a run legitimately finds nothing to change. A healthy quiet day and a job that stopped running a week ago look identical.

Read it at Platform → Background Jobs, which reports each job’s last run, its last success and its processed count, and flags a job that has not run within its expected cadence as STALE (or STUCK, FAILING, DEAD_LETTERED, NEVER_RUN).

That page — not the “Last sync” figures here — is where you answer “did the daily sync run last night”. A gap in the daily schedule is also detected by the sync itself: because Icecat replaces daily.index.xml.gz every day, a missed day is permanently lost, so a run that finds a gap fails, names the days missed and reports that a full sync is required rather than reporting a clean delta.

Reference data is weekly; the product index is daily

A daily-delta row for a supplier or category Icecat added since the last weekly reference sync cannot be inserted. The failure is caught per entry, counted into errors, and the product is dropped — and the next day’s file will not contain it again, so it stays missing until the next full index sync.

Practically: after Icecat onboards a new brand, products from that brand may not appear in the index for up to a week. Run a reference sync before a full index sync, not after.

Re-runs are safe; interrupted runs leave the index partial

Both index syncs upsert on icecatProductId, so re-running converges. But neither has a resume cursor: an interrupted full sync restarts from the first entry. Batches already committed are kept, which means a failed run leaves the index partially updated, not rolled back — correct as far as it went, silently incomplete beyond that point.

The full sync has no reconciliation pass. Products present locally but absent from a newer full file are never removed or marked off-market. Withdrawals are only applied when a daily file carries an explicit REMOVED entry and the daily sync runs that day.

The daily parser may be reading the wrong element (verify before trusting the delta)

The full index is read by a streaming SAX parser that matches <file> elements wherever they appear in the document.

Run against both shapes:

A daily sync that silently applies nothing is indistinguishable, at every layer, from a genuinely quiet day.

Confirm this against one real daily.index.xml before relying on the delta. The fastest check is to compare updated + removed + newProducts from a manual Daily Sync against the entry count in the downloaded file; a non-trivial file that yields exactly zero on all three is the symptom.

The daily parser also discards fields the streaming parser keeps — GTINs, alternate MPN, model name, image URL and the index path. Products first seen in a daily file therefore land as minimal rows with no GTINs, which makes them invisible to the highest-confidence matching strategy until the next full sync fills them in.

Both parseIndexXml and decompressGzip hold the whole decompressed file in memory, so the daily path’s footprint scales with the size of the delta.

The Icecat page’s bulk enrichment panel calls paths that do not exist

The three buttons surface an error toast; the enrichment-stats funnel simply never populates.

Everything else on the Icecat page — reference syncs, full and daily index syncs, the index browser, the access queue — targets the right paths and works.

Single-product enrich and match have no button

The only ways to enrich one product today are the API directly, the bulk job, or the staff-preview trigger below.

Staff-preview enrichment does nothing for a technician

A technician’s preview therefore forwards a call that returns 403.

Its condition is also features === 0, not the enrichment status. A product whose enrichment came back FORBIDDEN, NOT_FOUND or ERROR still has no features, so every staff view re-fires a paid lookup that will fail the same way. The call is awaited inline with no timeout, so a slow Icecat response delays the page.

Enrichment has no cache expiry

It does not check icecatEnrichmentStatus or the age of icecatLastSync first. Only the bulk path filters to PENDING products; the single-product path re-fetches on every invocation. Treat each call as a billable request.

What enrichment stores, and what it does not

Enrichment parses videos, 360° view sets, reasons-to-buy, product stories, feature logos, energy labels and related products, and records counts of each on IcecatEnrichmentData. Only features, images and documents are written to queryable tables.

Six of the eleven CatalogDocumentType values are ever produced by enrichment (MANUAL, DATASHEET, SAFETY_SHEET, ENERGY_LABEL, WARRANTY, QUICK_START); anything else Icecat sends is filed as DATASHEET.

Media

• Images are stored at whatever resolution Icecat published. No thumbnail is generated, and CatalogProduct.thumbnailUrl is set to the same full-size URL as primaryImageUrl.
• Only the isMain image updates those denormalised URLs. If the main image fails to download but gallery images succeed, the product keeps pointing at Icecat’s URL.
• A permanently broken source URL is retried on every bulk sweep. There is no attempt cap.
• Deleting a product or an image does not delete its blob. There is no orphan reaper for catalog media.

Configuration and wiring traps

• Both pull the same seven files an hour apart. Disable one.

• Nothing in a browser can reach it.

Scope

• Enrichment is global, not per-tenant. There is no tenant-level override of an enriched description, spec or image.
• One language at a time: ICECAT_LANGUAGE governs both the bulk exports and the live lookups, and is stored as dataLanguage.
• IcecatProductIndex.limited — Icecat’s flag for “full data is restricted for this subscription” — is parsed but never written and never read. Enrichment cannot skip products it already knows will be refused.