JFrog Artifactory

Why this page exists

Running Artifactory HA on Kubernetes in production teaches you things the docs imply but never quite say out loud. This is the field guide I wish I had before deploying a multi-cloud artifact platform — the mechanics and gotchas from a real HA, cross-cloud deployment. This pairs with the multi-cloud artifact platform case study, which tells the story; this page tells the engineering.

Storage: checksum-based means “no versioning”

Artifactory stores binaries as immutable, content-addressed objects — the object is the checksum. Once the filestore is backed by S3 or GCS with a checksum-based provider, the bucket is effectively write-once: enabling bucket versioning buys nothing and costs money on every upload. Encryption at rest is free on both clouds; a lifecycle rule aborting incomplete multipart uploads after a few days is the only cleanup you need.

The local cache-fs layer is a read-through cache in front of the bucket, sized by maxCacheSize. Two behaviors worth knowing before sizing anything:

• The quota reaper runs on schedule, not mid-serve — a single huge in-flight stream can push the cache far past the configured limit. Size the backing PVC with real headroom.
• Warm serving backfills the cache on the fly, so a “warm” read path is really bucket-or-cache, then RAM. If your serving is memory-resident, disk throughput numbers stop mattering — and benchmarks against them are red herrings.

The binarystore is pluggable — pick the direct chains

The direct provider chains (s3-storage-v3-direct, google-storage-v2-direct) talk straight to the object store. Authentication:

• AWS: IRSA (IAM Roles for Service Accounts) — no static cloud keys anywhere in the cluster.
• GCP: the provider wants a credential file mounted next to binarystore.xml; the org-standard path is a dedicated service account whose keyfile lives in Vault, materialized by the Vault Secrets Operator into a Kubernetes secret.

The cheapest end-to-end auth probe is the boot log line Bucket exists = true — that’s an authenticated call to the bucket, not a config check.

HA on Kubernetes: what actually matters

• All HA nodes share one external PostgreSQL and one object-store bucket. The redundancy parameter controls binary replication across nodes.
• Chart default anti-affinity is soft — it prefers, it doesn’t prevent. If two nodes must never share a Kubernetes node, set the hard rule (requiredDuringSchedulingIgnoredDuringExecution with a hostname topology key).
• Fresh boots run database migrations that can exceed chart-default probe windows. Tune the startupProbe failureThreshold, not just the initial delay — a kubelet SIGTERM mid-migration can leave a stale “migration in progress” claim row that wedges every subsequent boot until someone notices.

The chart will lie to you (in nice ways)

• Dead config keys exist. In some chart versions the documented pod-count key is replicaCount, not replicas; persistence.binaries is old HA-chart syntax that silently does nothing; storage class keys are named storageClassName in one subchart and storageClass in another.
• Nesting is load-bearing. Artifactory subchart keys live under a double nesting (artifactory.artifactory.*). A root-level block is silently ignored — the pods will run as the default service account, and everything downstream (cloud auth, mesh registration) breaks in confusing ways.
• The discipline that catches all of this: grep the vendored chart templates and render-verify, never trust documentation alone. Render locally (kustomize/helm template) and diff before every apply.

Secrets: Vault to Kubernetes, with restart semantics

The Vault Secrets Operator materializes Vault KV paths into Kubernetes Secrets; rolloutRestartTargets stamps an annotation that rolls exactly the consuming workloads. Three gotchas that bite in production:

• Kubelet secret-mount propagation lags — a pod restarted immediately after a secret change may still see the old value. If a config change isn’t taking effect after one restart, restart again.
• Adding a key to a watched path is a planned rollout of everything watching it. Model Vault paths so tooling-only credentials never share a path with runtime secrets.
• Schema-critical consumers are picky about URL shapes (see next section) — materialize exactly the keys each service expects, with transformations if needed.

PostgreSQL: three dialects of “connect to the database”

• The Java services speak jdbc:postgresql://.... Xray’s Go services parse the same env var and reject the jdbc prefix (their URL parser treats it as an unknown scheme) — they need plain postgres://host:5432/db. One convention does not fit all microservices; per-service override secrets exist for exactly this reason.
• Poolers change lock semantics. Boot-time migration coordination often uses session-scoped advisory locks. Through a pooling endpoint, a dead client’s locks persist server-side until the session closes — the next boot deadlocks. If migrations deadlock on pooled endpoints, suspect the pooler before the code; session pooling (or a dedicated endpoint) plus sane pool sizing fixes the class.
• Startup probes must outlast migration windows; a probe that kills a pod mid-migration turns a slow boot into a wedged database.

Federation (RTFS): identity is URL-keyed

Federation mirror pairing matches on each site’s identity URL, which derives from the platform’s base-URL configuration. If both sites claim the same identity — which happens the moment you point both at a shared entrypoint — every replication stream wedges with mirrorNotFound, and the only place the reject reason appears is the artifactory-side federation log, not the replication service’s logs. The playbook:

1. Set per-site identity URLs (an undocumented config element sits next to the base-URL knob).
2. The native re-binding machinery propagates the change to the peer’s registry — but does not rewrite the peer’s member lists; update member URLs explicitly.
3. Restart order matters: pods re-mint deleted registry rows from in-memory state, so normalize the registry before restarting anything.
4. Verify with canaries in both directions, including deletes.

Huge artifacts: the Xet protocol is not optional

Plain HTTPS pulls of very large single binaries (tens of GB) stall deterministically on the upstream-fetch path — the fetch is tracked and resumable across pod restarts, but the single-stream HTTPS transfer itself has a practical ceiling. The Xet protocol (chunk-based, deduplicated) makes huge pulls reliable end-to-end: expect ~110–130 MB/s per client cold, ~190 MB/s warm from the chunk store.

The capacity rule that actually holds: the per-process pipeline is the pacing unit. Concurrency knobs and newer client versions barely move the per-process ceiling; N processes on N machines scale linearly, multiple processes on one machine barely aggregate. Plan fleet pulls with N × per-process math, and keep per-file session caps in mind when an artifact ships as one giant blob.

Package managers have their own opinions

PyPI’s simple-index serialization changed upstream (PEP 503 metadata v1.4 with absolute hrefs and spaced anchors), and certain Artifactory versions extract zero file anchors from it — while classic serialization parses perfectly. Escape hatch: a tiny in-cluster nginx re-serializer in front of the remote repository — rewrite file URLs to relative paths, tighten the anchor spacing, downgrade the metadata version marker. Cheap, reversible, and it buys time for the vendor fix. Remember that index fetching and download redirecting consult different configuration knobs; flipping one without the other leaves the cache pointed at the old upstream.

Observability: OpenMetrics with expectations

• OpenMetrics is license-gated — enable it via the system YAML override; the chart’s own metrics knob lands under the wrong subtree in some versions.
• Scrape discovery requires the correct labels on the ServiceMonitor, auth requires a scoped token (metrics read-only is enough), and cross-site series need a static per-cluster label — pod names are identical across independent clusters, so anything aggregated by (pod) silently mixes sites.
• Verify alert metric names against the live exposition, not the docs tables — some documented series simply do not exist in a given version, and an alert on a nonexistent metric is a permanently silent alert.

Wrapping up

Most of these lessons are the same lesson wearing different clothes: every layer works in isolation, and the failures live in the interactions — identity vs. entrypoints, locks vs. poolers, docs vs. vendored templates, client pipelines vs. proxy paths. Make every layer verifiable (render-diff, canaries, checksum audits, live-exposure checks), and production stops being a surprise generator.