Real Experience — Running Production Java Microservices

Architecture books are full of elegant diagrams. Production is full of 3 AM pages, heap dumps, and incident docs that all end with “we didn’t expect this.” This article is the unvarnished version — the lessons from running Java microservices at real scale, the mistakes that teach the most, and the small habits that make the difference between a system you trust and a system you tolerate.

No new patterns, no new frameworks. Just the stuff you only learn by being on-call for it.

Lesson 1 — Most outages are boring

You’d think production goes down because of exotic race conditions or subtle distributed-systems edge cases. It almost never does. After hundreds of incidents, the ranked list of causes is remarkably mundane:

A deploy went wrong. New version had a bug, config drift between environments, missing migration.
A slow query crept in. New feature added a query without an index; fine at 100 rows, deadly at 10M.
A dependency changed behavior. Upstream API started returning 429s, a library upgrade changed defaults.
Capacity ran out. Traffic spike, autoscaler lagged, a pod OOM’d and the rest tipped over.
A human made a mistake. Wrong config applied, wrong DB connection string, wrong kubectl context.

The takeaway: invest more in prevention of boring failures and less in elegant solutions to exotic ones. Good migration hygiene, automated rollback, query-plan checks in CI, and careful capacity planning prevent more incidents than any clever piece of code.

Lesson 2 — The deploy pipeline is your most important service

At scale, every change ships through the pipeline. A brittle pipeline means either slow shipping or risky shipping. Investments here pay back quickly:

Canary deploys with SLO-based gating — 1% traffic to new version, auto-rollback if error rate or p99 regresses
Progressive rollout across zones/regions — never ship to all regions simultaneously
Feature flags decouple deploy from release — ship code dark, enable by flag, disable by flag
One-click rollback — must work without humans typing commands; the UI button matters
Every deploy emits a marker event — visible on every dashboard, queryable in logs

The number to optimize: time from “this deploy caused the incident” to “deploy rolled back”. If it’s over 5 minutes, your pipeline is the bottleneck. Aim for under 60 seconds.

A concrete habit that prevents a lot of pain: never deploy on Friday afternoon. It’s not superstition — it’s acknowledging that your response time to problems is worst when everyone’s heading home.

Lesson 3 — Memory management still matters in 2026

“Java handles memory automatically” is true until it isn’t. The OOM incidents I’ve seen most often:

Unbounded caches. A HashMap that never evicts, inside a long-running service. Six weeks after deploy, it fills the heap. Fix: always use a bounded cache (Caffeine with maximumSize), never a raw Map.

Large response payloads. A service loads 500 MB of data into a List<> to return as JSON. Works for one request; OOMs when five come in parallel. Fix: stream the response, use pagination, never materialize more than you need.

ThreadLocal leaks with thread pools. ThreadLocal set in a thread pool is never cleaned up — every request accumulates state. Fix: use ThreadLocal.remove() in a finally, or avoid ThreadLocal entirely in favor of explicit context objects.

Off-heap memory. Direct ByteBuffers, Netty allocations, JNI libraries. The heap looks fine; the container still OOMs. Fix: set -XX:MaxDirectMemorySize, monitor RSS not just heap.

The monitoring that catches these early:

// Expose JVM memory metrics via Micrometer
new JvmMemoryMetrics().bindTo(registry);
new JvmGcMetrics().bindTo(registry);
new ProcessMemoryMetrics().bindTo(registry);

Alert on container_memory_working_set_bytes approaching the limit, not just heap. Pods get killed based on RSS, not heap.

Lesson 4 — Hikari sizing is counterintuitive

Every Java team I’ve joined has at least one service with a 200-connection database pool and terrible performance. The instinct to “give it more connections” is wrong. The right pool size is almost always smaller than you think.

The heuristic that actually works:

connections = ((cores × 2) + effective_spindles)

For a modern cloud Postgres with SSD, that’s typically 10–20 connections per service instance, period. More connections create contention on the DB side — more locks, more context switches, worse throughput.

If you need more concurrency than that supports, the answer isn’t a bigger pool. It’s one of:

A connection pooler (PgBouncer in transaction mode) between services and DB
Async or reactive request handling (or virtual threads, which is easier)
Horizontal scaling of the DB via read replicas or sharding

The HikariCP config that survives production:

spring:
  datasource:
    hikari:
      maximum-pool-size: 15
      minimum-idle: 5
      connection-timeout: 2000
      validation-timeout: 1000
      idle-timeout: 300000
      max-lifetime: 1200000
      leak-detection-threshold: 20000

The leak-detection-threshold is the underrated one — it logs a stack trace for any connection held longer than 20 seconds. First time you enable it, you’ll find at least one bug.

Lesson 5 — Timeouts are load-bearing

Every incident post-mortem I’ve been in where a slow dependency took down the caller has the same root cause: a missing or too-generous timeout somewhere. The hierarchy of pain:

  no timeout                →  service hangs forever, thread pool exhausted
  30s timeout               →  users wait 30 seconds, then get an error
  timeout with infinite retry →  retry storm amplifies the outage
  budget-based timeout      →  fails fast, retries bounded, degrades gracefully

The right approach — timeout budgets that shrink as the call graph deepens:

Gateway has 5s total budget
Service A called from gateway gets 4s
Service B called from A gets 3s
DB query from B gets 2s

Passed in headers or computed from a deadline. Every layer respects the remaining budget, so nothing waits longer than makes sense.

Concretely in Spring with the declarative HTTP client:

@Configuration
public class HttpClientConfig {
    @Bean
    RestClient paymentsClient(RestClient.Builder builder) {
        return builder
            .baseUrl("http://payments")
            .requestFactory(new SimpleClientHttpRequestFactory() {{
                setConnectTimeout(1000);
                setReadTimeout(3000);
            }})
            .build();
    }
}

And for the call itself, wrap with Resilience4j timelimiter so a stuck socket doesn’t leak:

@TimeLimiter(name = "payments")
public CompletableFuture<PaymentResult> charge(...) { ... }

Lesson 6 — Kafka is not a magic queue

Teams adopt Kafka expecting it to solve messaging forever. It does, but not the way most people assume. Real production Kafka experience teaches a few specific things:

Consumer lag is the #1 thing to monitor. Throughput is fine; lag is the indicator. Alert when it starts growing, not when it’s an hour deep.

At-least-once delivery means your consumers must be idempotent. There is no “exactly once” in the network. Design for duplicates. Every handler checks “have I processed this already?” against a durable store.

Rebalance storms are painful. When one consumer joins or leaves, the whole group rebalances, pauses, and starts over. With cooperative rebalancing (enabled by default since Kafka 2.4) this is less dramatic, but still real. Keep consumer counts stable; don’t autoscale consumers aggressively.

Schema evolution will bite you. The first time someone renames a field without a default, every consumer downstream crashes. Put Avro or Protobuf schemas in a registry (Confluent, Apicurio), enforce backward compatibility in CI.

Partition count is a one-way door. Increasing partitions on a live topic changes hash distribution, breaks ordered processing for keys that moved. Decide partition count once, right.

The consumer config that actually survives production:

spring:
  kafka:
    consumer:
      auto-offset-reset: earliest
      enable-auto-commit: false
      max-poll-records: 100
      max-poll-interval-ms: 300000
      session-timeout-ms: 30000
      isolation-level: read_committed
    listener:
      ack-mode: manual
      concurrency: 3

Manual acknowledgment is the most important one. Auto-commit silently drops messages when the consumer crashes mid-processing.

Lesson 7 — The database is the single hardest thing to scale

Everything else — services, caches, gateways — scales horizontally with money. The database doesn’t, until you put in serious work. Reality:

Postgres on a single node handles more than you’d think. Well-indexed Postgres on a decent machine handles tens of thousands of transactions per second. Before sharding, verify you’ve actually hit that ceiling.

Long-running transactions are catastrophic. A transaction open for 10 seconds holds locks, blocks vacuum, bloats tables, and eventually kills write performance. Monitor pg_stat_activity for transactions older than a few seconds.

Connection pooling is about managing concurrency, not connection cost. Even with PgBouncer, you’re protecting the DB from too much concurrent work, not just avoiding TCP setup. The bottleneck is the DB’s ability to do work, not its ability to accept connections.

Indexes aren’t free. Every index you add slows every write. Production DBs often have 50% of their write cost going to index maintenance. Periodically audit pg_stat_user_indexes for indexes that are never read — drop them.

Online schema changes need planning. Adding a nullable column is fine. Adding a NOT NULL column with a default rewrites the whole table. Use pg_repack or gh-ost-style online migrations for big changes; never ALTER TABLE a 100 GB table blind.

Lesson 8 — Graceful shutdown is often broken

Most Java services die badly on SIGTERM. Kubernetes sends SIGTERM, the JVM exits, in-flight requests fail, Kafka consumers don’t commit their offsets, DB connections leak. Users see errors that didn’t need to happen.

A correct shutdown sequence:

Stop accepting new traffic — health check returns 503, load balancer drains
Let in-flight requests finish — with a reasonable maximum (e.g. 30s)
Drain background workers — stop Kafka consumers, finish current message, commit offsets
Close connections cleanly — flush HTTP clients, close DB pools, flush log buffers
Exit

Spring Boot gives you most of this with one flag:

server:
  shutdown: graceful
spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s
management:
  endpoint:
    health:
      probes:
        enabled: true
  health:
    livenessstate:
      enabled: true
    readinessstate:
      enabled: true

Combined with separate readiness/liveness probes in Kubernetes, the pod goes unready first (traffic drains), then terminates after grace period. Done right, users see zero errors during a rolling deploy.

Lesson 9 — Profiling beats speculation

Every Java team has an intuitive theory of where CPU goes. Every Java team is wrong, sometimes hilariously. Real example: a service was spending 40% of CPU on JSON date formatting because someone used SimpleDateFormat in a hot loop. Nobody would have guessed; the profiler showed it in ten seconds.

Tools worth knowing:

async-profiler — low-overhead CPU and allocation profiler. Attach to a running process, get a flame graph.
Java Flight Recorder — built into the JDK, near-zero overhead, invaluable in production.
Micrometer method timing — @Timed on key methods gives p99 tracking per method.

Running JFR continuously in prod is free-ish and gives you exactly the data you need when something goes wrong:

-XX:StartFlightRecording=name=prod,filename=/var/log/jfr/recording.jfr,maxage=1h,maxsize=100m,settings=profile

Rotate the recording, keep an hour of history. When something looks odd, you have a JFR file to analyze instead of a shrug.

Lesson 10 — The humans are the load-bearing layer

The tech matters less than how the team operates. Patterns I’ve seen distinguish reliable teams:

Blameless post-mortems that change the system. Every incident produces a document with root cause, contributing factors, and specific action items — each with an owner and a deadline. Items are tracked in the backlog like any other work. No finger-pointing, no “someone should be more careful.”

On-call is a shared responsibility. Same engineers who write the code respond to its incidents. Nothing aligns incentives like being woken up by your own code at 3 AM.

Runbooks per service, not per incident. A checked-in markdown file covering: how to deploy, how to rollback, how to restart, top 5 known failure modes, who to escalate to. Updated after every incident.

A rule: one person can bring the system down in under 30 seconds. That person is the on-call. They have the authority to roll back deploys, disable feature flags, scale services, regardless of who owns what. Democratic debates during outages are how outages get worse.

Capacity is planned, not assumed. Every quarter: review load growth, run load tests at 2× current peak, adjust autoscaling bounds. Catching a capacity problem in a load test costs nothing. Catching it in production during a traffic spike is an outage.

Lesson 11 — Things I stopped doing

After long enough in production, some habits reverse. Things that looked smart but caused more pain than they prevented:

Custom caching layers. Every hand-rolled cache eventually has a bug. Use Caffeine for in-process, Redis for distributed, and stop there.

Complex Kafka consumer frameworks. @KafkaListener plus manual acks plus a DLQ is enough. I’ve spent weeks debugging abstractions that added nothing.

Shared client libraries across services. A library that ten services depend on is a coordination bottleneck disguised as reuse. Share specs (OpenAPI, Protobuf), not code.

Tight JPA ↔ API coupling. Returning JPA entities from REST endpoints turns every schema change into an API change. Separate DTOs, always.

Clever async code without a reason. Reactive Mono/Flux is powerful but harder to debug. Unless you’ve proved you need it, virtual threads + blocking code is faster to write and easier to reason about.

Multi-cluster Kafka for redundancy. Complicated to operate, rarely better than a single well-run cluster with good replication. If you really need it, MirrorMaker 2 is the tool — but question the requirement first.

Lesson 12 — Starting a new service right

The 15 minutes before a new service’s first commit matter more than the next 3 months. Defaults that pay off forever:

JDK 21 or later with virtual threads enabled
Gradle or Maven with reproducible builds — byte-identical outputs for the same inputs
Containerized from day one — Jib or buildpacks, no hand-written Dockerfile
Flyway migrations committed to the repo — schema is source-controlled
OpenAPI spec as part of the repo — contract is explicit and reviewable
Micrometer + OpenTelemetry configured in the starter module — observability is never “added later”
Graceful shutdown configured — before you need it
Readiness and liveness probes distinct — before you need them
Sample application-local.yml — one command to run the service locally against Testcontainers
A README with deploy, rollback, debug, and on-call pointers — not aspirational, real

None of this is glamorous. All of it compounds.

Checklist: is your Java microservice production-ready?

Closing thought

Running Java microservices in production isn’t about knowing the most patterns or using the latest framework. It’s about a thousand small defaults that, together, mean the system stays up while people sleep. The engineers I’ve worked with who were best at production weren’t the ones who knew the fanciest tools — they were the ones who cared about boring things: timeouts set everywhere, metrics on the right signals, graceful shutdown tested, runbooks actually useful. None of it looks impressive in a design review. All of it is what the on-call engineer thanks you for at 3 AM.