Storage providers

dedup4j ships three storage adapters and an SPI for writing your own. All three adapters arrive with the starter; configuration selects one, and the others stay inert.

dedup4j:
  storage:
    provider: local   # local | s3 | azure

Local filesystem

For development, tests, and single-node deployments.

dedup4j:
  storage:
    provider: local
    local:
      root-directory: ./dedup4j-storage

Amazon S3

Also works with S3-compatible stores — MinIO, Cloudflare R2, Backblaze B2 — via endpoint.

dedup4j:
  storage:
    provider: s3
    s3:
      bucket: my-application-blobs
      region: eu-west-1

Credentials

There is no credentials property, deliberately. dedup4j uses the AWS SDK's default credential provider chain — environment variables, system properties, the shared credentials file, then container and instance metadata.

That means in production you attach an IAM role and configure nothing. A library that accepted access-key and secret-key as configuration would invite them into application.yaml, and from there into version control.

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_REGION=eu-west-1

MinIO or other S3-compatible

dedup4j:
  storage:
    provider: s3
    s3:
      bucket: blobs
      region: us-east-1
      endpoint: http://localhost:9000
      path-style: true

path-style: true is required by most S3-compatible stores, which do not implement virtual-host-style bucket addressing.

Azure Blob Storage

dedup4j:
  storage:
    provider: azure
    azure:
      container: application-blobs
      endpoint: ${AZURE_BLOB_SAS_URL}   # or connection-string

container plus one of endpoint or connection-string is required. With only container and account-name, startup fails with connectionString or endpoint must be configured.

Authentication

dedup4j 0.1.0 does not configure a managed identity or any other Azure credential for you. Three ways work:

1. connection-string: simplest; fine against Azurite.
2. endpoint with a SAS token in its query string.
3. Your own client. Declare a BlobContainerClient bean and dedup4j uses it instead of building one. This is how to use a managed identity today:

@Bean
BlobContainerClient blobContainerClient() {
    return new BlobServiceClientBuilder()
            .endpoint("https://mystorageaccount.blob.core.windows.net")
            .credential(new DefaultAzureCredentialBuilder().build()) // azure-identity
            .buildClient()
            .getBlobContainerClient("application-blobs");
}

container is still required when you supply your own client.

Custom providers

Implement BlobStorage from dedup4j-core — four methods:

public interface BlobStorage {
    StoredBlob put(PutBlobRequest request);
    BlobResource get(String objectKey);
    void delete(String objectKey);
    boolean exists(String objectKey);
}

Register it as a bean and the built-in adapters step aside:

@Configuration
public class GcsStorageConfiguration {

    @Bean
    BlobStorage blobStorage() {
        return new GcsBlobStorage(...);
    }
}

Your adapter handles bytes only. Content hashing, deduplication, reference counting, and metadata stay in dedup4j — an adapter is a transport, not a participant in the dedup decision.

Contract notes:

• get returns a BlobResource holding an open InputStream; the caller closes it.
• exists should be cheap — a metadata probe, not a download.
• put must be safe to call with content that already exists at that key. Deduplication reduces how often that happens; it does not guarantee it never will.

Comparison

Next

• Uploading & deduplication — what happens on store
• Configuration properties — every key
• Troubleshooting — misconfiguration symptoms