Architecture & limitations

What dedup4j is, how it is put together, and — more usefully — what it does not promise.

The idea

Content is identified by its bytes, not by its name. Two uploads with the same bytes are the same content, stored once.

your application                    dedup4j
────────────────────────────────────────────────────────────
Attachment #1 ──┐
Attachment #2 ──┼──► assetContentId ──► one row  ──► one object
Document   #7 ──┘      refCount: 3      (your DB)   (your store)

dedup4j owns the row and the object. Your application owns the meaning — which user uploaded it, who may read it, what it is for. dedup4j deliberately knows none of that.

Content identity

Identity is (algorithm, hash, sizeBytes) — SHA-256 plus the byte count.

Not the filename, content type, metadata, uploader, or timestamp. Rename a file and it is the same content. Change one byte and it is different content.

This is what makes deduplication safe to do automatically: identity is a property of the data, not a claim by the caller.

Modules

Dependencies point inward. dedup4j-core depends on no cloud SDK and no Spring; each storage adapter depends only on core and its own SDK. You can implement BlobStorage without inheriting anything else.

Transaction ownership

dedup4j does not own transactions. You do.

It binds to your DataSource, and its writes join whatever transaction is active. A dedup4j metadata write and your own write can commit or roll back together.

retain and release take a pessimistic row lock held for the caller's transaction, so concurrent operations on the same content serialise instead of racing.

The limit that matters: two systems, one operation

What that means in practice

The write order is deliberate: object store first, database second.

1. hash the content
2. look for existing content by identity
3. if new: write the object      ← object store
4. then: write the metadata row  ← database

A crash between steps 3 and 4 leaves bytes in the object store with no row pointing at them. That is wasted storage, not corruption — the reverse ordering would leave a row referencing an object that does not exist, and every download of it would fail.

The failure mode was chosen. Orphaned bytes cost money and can be reconciled; a dangling reference is a broken user-facing feature.

Deletion has the same seam

The final release deletes the object and decrements the row in one transaction. If that transaction rolls back after the delete, the row returns and the bytes do not — see Retrieval, retain & release.

Watch dedup4j.storage.delete.failures; nothing retries it.

Concurrency

Two concurrent uploads of the same new content can both reach step 3. Both write the same object key — which is safe, because the key derives from content identity, so they write identical bytes to the same place.

At step 4 the database resolves it: one creates the row, the other retains the existing one, enforced by a unique constraint on content identity.

This is why ObjectKeyStrategy must be deterministic. A strategy with randomness in it turns a harmless duplicate write into two divergent objects.

The upload history ledger

Totals and failures live in an append-only history in your application's database: three tables, dedup4j_history_dataset (one row, the dataset identity), dedup4j_upload_event (the upload ledger) and dedup4j_failure_event.

• Immutability is enforced by the database. PostgreSQL uses row triggers and a TRUNCATE trigger. H2 uses row triggers, and foreign-key references block TRUNCATE. Entities are immutable and repositories can only append and read. This is why history supports only PostgreSQL and H2.
• A dataset-row lock gives a gapless sequence. Each upload locks the dataset row, reads the latest committed ledger row and appends the next one with running totals, holding the lock until commit. Reading totals is one row read.
• Failures are recorded after the failed transaction. The upload's own transaction rolled back, so the failure is appended in a separate transaction. Micrometer meters likewise increment only after commit.
• READ COMMITTED is required. The gapless sequence relies on it, so the history appends request it. If the transaction manager rejects a custom isolation level, dedup4j logs event=history.isolation.unsupported once and appends at the database default, which is correct on the PostgreSQL and H2 defaults.
• Corrections are compensating rows. A mistake is fixed by appending a row that negates and references the original. Nothing updates or deletes history.

The cost is that uploads serialise on the dataset row for the few statements between the ledger insert and commit, negligible at thousands of uploads per day.

What dedup4j does not do

Known limitations

• Content is read into memory to hash it. max-upload-size defaults to 25 MB. Large files are a memory decision, not just a policy one.
• The local provider assumes one node, or a shared mount.
• Reference counts can drift from your records after a crash or a manual database edit. Reconciliation finds drift; it does not prevent it.
• Deletion at zero is unconditional and cannot currently be disabled.

Design records

Decisions and their reasoning live in the repository rather than on this site: the upload facade and batch behaviour, provider configuration, the single generic starter, and Spring Boot auto-wiring.

Next

• Public API
• Retrieval, retain & release
• Contributing