Observability & dashboards

Deduplication makes a claim — you are storing less — and a claim worth making is worth measuring. dedup4j exposes metrics, optional management endpoints, and two different dashboards for two different jobs.

Metrics

Registered against Micrometer, so they reach whatever backend your application already uses.

Upload history

Micrometer counters live in one process and reset when it restarts. For numbers that survive restarts, dedup4j keeps an append-only upload history in your application's database. It is on by default.

• All-time totals per application dataset. Uploads, duplicates, physical writes, logical bytes and storage avoided belong to the dataset (datasetId), not to a process. Restarts do not reset them, and replicas sharing a database report the same totals.
• Totals start at zero on upgrade. There is no backfill from existing content.
• Failures are persisted. Failed uploads and failed physical deletes are recorded and shown under "Recent failures" in the management endpoint and both dashboards. Failed uploads are never counted in the totals. With history disabled, failures are neither recorded nor shown, and "Recent failures" stays empty.

Storage avoided is the number to quote. It is the bytes that did not become an additional stored copy: the sum of the sizes of duplicate outcomes. A duplicate that lost an insert race still counts, even though its bytes were sent to storage once, because the same key was overwritten rather than duplicated. It measures capacity saved, not upload traffic saved.

Management endpoints

Add dedup4j-spring-boot-management and opt in:

dedup4j:
  management:
    enabled: true              # default: false
    base-path: /dedup4j/management
    instance-name: orders-api

The routes are versioned under {base-path}/v1. With the default base path:

/dedup4j/management/metrics, without /v1, returns 404.

/metrics now reports datasetId, totalsScope (all-time or since-start) and physicalWrites alongside the existing totals, and /info reports datasetId.

/failures has two modes:

limit applies to both modes: it defaults to 100 and is clamped to 1..500. afterSequence below 0 is treated as 0.

Embedded dashboard

dedup4j-spring-boot-dashboard serves a read-only view inside your application.

dedup4j:
  dashboard:
    enabled: true                  # default: true when the module is present
    base-path: /dedup4j/dashboard
    failure-lookback: 7d

Note the default: enabled is true. Adding the dependency is the decision; the property is how you turn it back off. Same security caveat as above — read-only means it will not change your data, not that it is safe to expose.

The page is served at {base-path} and reads its data from {base-path}/api/v1: /overview, /instances/status, /instances/{id}/history and /failures. With the default base path, that is /dedup4j/dashboard/api/v1/overview and so on.

The embedded dashboard shows the history totals and the persisted failures even when the management endpoints are disabled. A caption under the headline number says whether the totals are all time or since application start.

failure-lookback bounds how far back the failure view reads. Longer windows mean heavier queries.

The aggregate starter

dedup4j-spring-boot-observability pulls in management and the embedded dashboard together, for when you want both and would rather not list both. It is an aggregate starter: it carries no code of its own, only dependencies.

See Installation for its coordinates.

Standalone dashboard

dedup4j-dashboard is a separate executable application, not a library. It polls the management endpoints of one or more running applications and shows them in one place.

Use the embedded dashboard to look at one application. Use the standalone one to watch a fleet.

It is distributed as an executable JAR on GitHub Releases and is deliberately not published to Maven Central. Do not add it as a dependency.

java -jar dedup4j-dashboard.jar

It binds to loopback on purpose

It keeps its own SQLite database for polled history and does not write to your application's database.

With upload history in place the dashboard:

• counts each dataset once, so replicas sharing a database are not added together. They show a "shared dataset" badge;
• starts from a zero baseline after a dashboard restart, rather than re-adding all-time totals as new activity;
• ingests failures per dataset through a cursor over afterSequence, attributed to the instance where each happened;
• captions the headline as all time, since application start, or some datasets since application start when the two are mixed;
• treats an older instance that reports no datasetId as its own dataset.

Registration

An application can announce itself to a standalone dashboard:

dedup4j:
  dashboard-registration:
    enabled: true
    dashboard-url: http://localhost:8080
    advertised-url: http://orders-api.internal:8080
    instance-name: orders-api

advertised-url is how the dashboard reaches back, which is not always the address the application sees for itself behind a proxy or in a container.

Choosing

Next

• Configuration properties — every key
• Retrieval, retain & release — what drift means
• Troubleshooting — reading the failure signals