From Stateful to Stateful: Distributed Domain Event Management on Huawei Cloud Container Engine (CCE)

Moving a stateful domain to the cloud is not merely about writing data to disk; it is about redefining the time, language, and proof of truth.

When we think about a payment or insurance system running on Huawei Cloud’s Cloud Container Engine (CCE), what makes the architecture strong is not the number of CPUs or the amount of RAM, but how state is modeled and replicated, how domain events are produced, and whether replaying those events proves the same truth every single time.

Here, the word “deterministic” is not a performance metric; it is a covenant.

A system that must always produce the same output for the same input does not merely store state – it breaks it down into provable events, relates them to time, and preserves the integrity of that relationship.

An architecture built on CCE must therefore progress along three axes:

keeping state distributed yet safe, producing domain events with causal consistency, and being able to reconstruct the same truth through rehydration when necessary.

To achieve determinism, the first step is distinguishing the types of state.

Operational state is verified at runtime and stored with low latency – things like order reservations, policy statuses, or risk decisions belong here.

Derived state is read-heavy projections derived from operational events – risk reports, customer 360 views, or portfolio scores.

Persistent proof state represents legally defensible sentences – ledger records, audit trails, and settlement outcomes.

If you merge these three types of state into a single storage layer, carried and replicated by one service, you may preserve determinism on paper, but in practice, you expand the surface area of failure.

The correct approach is to separate state according to the domain language and let each context own only its own proof sentences.

Each context’s storage lifecycle should be governed by the context itself, not by Kubernetes.

That means using tools like StatefulSet and PersistentVolume merely for how data is kept, while answering the what and why at the domain level.

On the CCE side, the lifecycle of stateful workloads requires more than loyalty to Kubernetes principles – it demands a discipline of time that prevents misunderstanding.

When using StatefulSets, ordinal identities, headless services, and stable DNS records are not only for reachability but for idempotent side effects.

When a node enters maintenance or fails, defining thresholds with PodDisruptionBudget, ensuring Readiness and Liveness probes produce meaningful operational signals, and publishing graceful shutdown markers like “last processed event offset” in the preStop hook all dramatically reduce rehydration cost.

Anti-affinity and topology spread constraints prevent replicas from clustering in the same failure domain, while the choice of storage class directly affects determinism:

for operational state, use block-based classes with high IOPS and low latency;

for derived state, use object-like strategies optimized for throughput;

for persistent proof, rely on append-friendly, write-order-preserving channels.

An architect, therefore, does not first choose a storage engine but a concept of time and sequence, because the proof power of a domain event depends not only on the disk it lives on but on which clock measured it.

Domain event generation must be designed together with CCE’s world of signals and queues.

Treating the outbox approach purely as an integration pattern misses the point of determinism.

The outbox should be the only place inside a service where proof sentences are declared; that is, a service’s write operation must produce both its internal state and the proof of that state within the same causal boundary.

The critical element here is fencing and the single-writer discipline: in replicated environments that use leader election, simultaneous write attempts without fencing tokens must be strictly rejected, and event IDs must increase monotonically within the domain’s sequence rather than rely on timestamps, ensuring that “idempotent consumption” is genuinely idempotent.

Kubernetes leadership can be handled by a sidecar or coordination library, but the real leadership question is: who has the right to form a sentence of truth?

That authority is not tied to the Pod’s runtime fate; it is bound to the domain’s definition of a single source of truth.

Eventual consistency should not remain a slogan on CCE; it must become a visible calendar.

The Order context, after receiving a PaymentPromiseReceived from the Payment context, must within N seconds either produce OrderCommitted or publish OrderCancelled.

This contract is not hidden behind queue latency or consumer scaling parameters; on the contrary, it is connected directly to Readiness probes and HPA (Horizontal Pod Autoscaler) signals.

HPA policy changes are triggered not only by CPU or latency thresholds but by the proof-generation deadline.

Scaling, therefore, no longer means deferring truth for speed; it becomes the allocation of resources to ensure truth is announced on time.

Observability plays a central role here: correlation IDs, causation IDs, and event offsets must be recorded along a timeline specific to the domain flow, separate from application logs.

That timeline must not vanish when Pods are rescheduled; it should persist somewhere external as the system’s collective memory.

Rehydration is the true test of determinism.

When a stateful service crashes or a new replica joins, rebuilding state by replaying the domain event log sounds simple in theory.

In practice, however, one must distinguish between what is being reconstructed and what is merely being projected.

Instead of replaying the entire operational state from zero each time, capturing snapshots – contextually signed points of time – at regular intervals reduces rehydration time from minutes to seconds.

But snapshots alone are not enough; the snapshot must carry evidence of which time contract it represents, which moment the next replay will continue from, and whether there were any gaps in the event stream.

On CCE, the safety of this process is not guaranteed merely by file locks; it requires cryptographic or at least fencing-based proof that the Pod taking the snapshot truly held the writing authority at that time.

Otherwise, a “fast rehydration” quickly turns into the rapid reconstruction of a false truth.

When determining the replication topology, one must not forget the principle of “From Stateful to Stateful.”

Most teams find comfort in stacking stateless readers on top of a stateful core, but they overlook the fact that derived state is, in another context, also stateful.

A risk projection is not a single SQL row; it is a temporal relationship between multiple contexts, and thus it cannot simply be accelerated by a cache layer.

The service managing derived state must understand the causal order of original events.

In situations where multiple events fall into the same “moment” within a context, deterministic binding rules must be established – for instance, assigning contextual priority when two events arrive in the same millisecond – so that a repeatable truth can be produced later.

That is why clock drift between Kubernetes nodes is not an operational issue but an architectural one.

The discipline of time synchronization begins not with PTP/NTP settings, but with the answer to this question:

“Which context’s clock determines which decision?”

During recovery and supervision stages, the preferred approach should not be automatic correction but forced re-articulation.

The Reconciliation service receives external files and compares them to the Ledger context’s sentences.

When a mismatch is found, instead of invoking a “repair script,” it compels the responsible context to rebuild its sentence.

This design is managed through domain contracts, though it may be integrated with Kubernetes Jobs or CronJobs.

Thus, correction is not a database column update – it is the reproduction of proof.

At the CCE scale, this approach shifts observability from operational logs to domain event maps; postmortems evolve from “which pod died?” to “which context spoke late or spoke wrongly?”

Team topology is the silent law of architecture.

Ten services deployed seamlessly on CCE produce only “ten different speeds” in an organization where one word carries multiple meanings – but not correctness.

The Authorization team is accountable for the correctness of “approval” sentences;

the Order team, for “commitment” sentences;

and the Ledger team, for “financial truth” sentences.

This accountability extends beyond code ownership to include ownership of time contracts and event contracts.

OKRs should not revolve around pod count or CPU cost but around whether contextual sentences are announced accurately and on time.

Kubernetes is the execution engine of this discipline; it does not create the discipline itself.

The discipline comes from a clear, defensible decision about who holds the authority to form sentences of truth within each context.

When all these components come together, the outline of a deterministic stateful architecture on CCE becomes clear.

The operational state protected by single-writer and fencing principles announces its sentences through the outbox.

These sentences travel through order-preserving queues to related contexts.

Derived state is rehydrated using a combination of replay and snapshot strategies.

Reconciliation resolves inconsistencies by forcing re-articulation rather than silent correction.

Observability prioritizes causality and time over metrics.

Scaling policies depend not on performance demand but on the calendar of proof production.

Storage class and topology decisions are made based on event order and causal durability, not on IOPS alone.

Most importantly, each context declares its own clock;

microservice independence is not achieved by avoiding data sharing, but by not being enslaved to someone else’s time.

Ultimately, “From Stateful to Stateful” means not just keeping state alive, but ensuring it speaks the same truth every time it does.

Huawei Cloud CCE provides a strong execution environment for this discipline, but it does not decide whether truth is deterministic – you do.

You decide by drawing clear boundaries, granting exclusive authority to form sentences, and turning time into a contract.

When correctness is built this way, speed ceases to be a right and becomes a reward;

a cache never replaces truth;

an outbox is not integration but the declaration of proof;

rehydration is not recovery but the rewriting of reality.

So it is best to end this architecture not with a diagram but with a principle:

If time is not deterministic, intelligence can never be safe.

HuaweiCloud #CloudArchitecture #StatefulSystems #Microservices #DomainDrivenDesign