The Challenge: Unexpected Failure in Ordinary Process  

A user reported that a process failed after an hour of execution. Initial logs showed that only about 200 items were processed before the authorization token expired, even though users expected to process several thousand.

At first glance, the system architecture appeared straightforward:  

• For each item, four API  — driven actions were executed.
• All items and actions were processed sequentially.
• Azure Durable Functions handled orchestration and asynchronous activity execution.

Yet the total execution time made no sense. Logs showed no unusually slow API calls, suggesting the performance issue lay elsewhere.

Understanding the Architecture

The system used Azure Durable Functions, which rely on three core components:

• Trigger: Initiates the Orchestrator (in this case, via Service Bus).
• Orchestrator: Contains the process logic.
• Activity: Wraps asynchronous operations such as API calls.

However, the codebase was implemented in Node.js, running on a framework based on the .NET ecosystem. This introduced important behavioral characteristics: 

• The Orchestrator cannot run long-running processes continuously when using Node.js.
• When an Activity is invoked:
• The Orchestrator schedules the Activity and exits.
• When the Activity completes, the Orchestrator is replayed from the beginning up to that Activity.
• Activity results are stored in dedicated storage and loaded on every replay.

This replay mechanism became a crucial clue.

Early Hypothesis: Orchestrator replays

Our initial suspicion was that repeated Orchestrator replays were accumulating significant overhead.

A small proof‑of‑concept showed that while replays added overhead, it was not enough to explain the multi‑second delays observed during user workflows. Something else was growing out of control.

To investigate further, the scenario was reproduced locally, and detailed timestamp logs were added around:

• Orchestrator executions
• Activity start/end times
• Storage access during Activity result retrieval

The pattern that emerged was surprising: Each time more Activities completed, the delay before the Orchestrator replay increased.

Discovering the Root Cause: A Growing Execution State

Durable Functions stores Activity results as part of the Orchestrator execution history. When NGINX proxy was added to inspect storage calls, the issue became clear:

• The payload containing execution history grew with each Activity call.
• Over time, this history reached multiple megabytes.
• Parsing these large blobs before each replay introduced multi‑second delays.

What caused the bloated state?

By the initial design of the system made by our team previously, every Activity returned an entire “context object”, which accumulated results from all previous Activities. This “context object” played a major part in the process. Activities used it to transfer data between each other. Even other processes could have relied on it.

This caused linear, but extremely costly, growth:

• Activity 1: 10 bytes
• Activity 2: +10 bytes = 20 bytes
• Activity 3: +10 bytes = 30 bytes
• …
• Activity 100: ~50 KB total
• Activity ~2,800 (real scenario): several MBs

Durable Functions were never intended to store such large intermediate states, so this pattern severely degraded performance.

Evaluating Possible Fixes

We identified two viable approaches:

1. Minimize Data Returned by Activities

• Each Activity would return only its result.
• Orchestrator would assemble the full context.
• Reduces execution state growth but keeps sequential processing.
• Still too slow for thousands of items.

Estimated processing time:

• ~7 minutes for 1,000 items
• ~13 minutes for 2,000 items

2. Implement Bulk Processing

• New “bulk Activity” types process many items at once.
• Backend APIs expanded to support batch operations.
• Greatly reduces Activity count, execution overhead, and state size.

This second approach required more engineering effort but promised significantly better performance and reusability.

Both technical and long-term architectural benefits made this the clear path forward.

Results: Dramatic Performance Improvements

After implementing bulk processing:

• The system successfully processed 10,000 items in under 5 minutes.
• However, another issue emerged: The final result message sent through Service Bus exceeded 1 MB, hitting the message size limit around 9,000 items.

The solution involved introducing a hard limit on maximum items per process, based on user needs and platform constraints.

Key Learnings

This incident highlighted several important engineering lessons:

1. Cloud frameworks have implicit limits
   Durable Functions’ execution history behavior is powerful but can cause exponential overhead when misused.
2. Language-platform combinations matter
   Node.js is supported, but not the most efficient environment for complex Durable Functions orchestrations.
3. Logs and simple tools remain invaluable
   Using simple tools like “grep”, “awk”, and NGINX proxy inside Docker was crucial for diagnosing storage-level performance issues.
4. Bulk processing is often essential for scalability
   Sequential workflows, even when optimized, cannot scale to thousands of items without batching.
5. Real-world requirements evolve
   The system was originally built without a requirement around item count. User expectations created unforeseen workloads and new architectural constraints.

Conclusion

This investigation turned a routine support ticket into a deep exploration of Durable Functions internals, performance tuning, and architectural redesign. By rethinking execution patterns and implementing bulk processing, we transformed a slow, failure‑prone workflow into a scalable, user-ready solution. 

The experience reinforced that powerful debugging often relies on simple tools, clear reasoning, and willingness to challenge architectural assumptions, skills essential for any Senior Software Engineer.