During my internship, I helped investigate a billing workflow that had stopped making reliable progress.

The database was busy. Workflows were piling up. Throughput was low. The system was doing a lot of work, but very little of that work appeared to be finishing.

At first, the mutex looked like the obvious problem. Workflows needed to acquire a lock for a billing account before performing certain operations, and the incident report identified that lock as the direct bottleneck behind the queue. But a mutex by itself did not explain the rest of what I was seeing.

It did not explain why database usage kept climbing. It did not explain why throughput stayed low. It did not explain why scaling the database from 4x to 12x did not restore the system.

Finding a starting point in systems fundamentals

What helped me investigate was not prior experience with that workflow engine. It was the vocabulary I had built in the CS 143 series, my operating systems courses.

Words like mutex, scheduling, fairness, starvation, preemption, deadlock, and forward progress gave me a way to ask better questions. Those questions moved the investigation from the visible bottleneck to a larger pattern: queueing, deadlines, partial progress, and recovery behavior may have been interacting in a way that amplified failure.

I want to be careful about scope. I did not establish the root cause of the incident, and I never received a definitive answer about the original trigger. What I developed was a working hypothesis for how an initial slowdown or timeout could grow into a degraded state. My contribution was to trace the relevant workflow behavior, connect it to systems concepts, and identify mechanisms worth investigating further.

High utilization is not high throughput

The symptoms looked contradictory at first:

• Database usage was high and continued to rise.
• Workflow throughput was low.
• Charging and finalization workflows were no longer making reliable progress.
• Work accumulated behind an account-level mutex.
• Increasing database capacity did not restore throughput.

The database scaling initially looked promising because it gave the system more room to operate. But the workload steadily consumed the additional capacity. We had increased how much work the system could attempt at once, not necessarily how much useful work it could finish.

That distinction became central to the investigation:

High utilization is not the same as high throughput.

A system can be extremely busy while repeatedly performing work that is later discarded, reset, or attempted again. In that state, adding capacity can give the failure mode more room to expand.

The mutex was only the beginning

The mutex existed for a good reason. Some operations on the same billing account could not safely run in parallel, so a workflow acquired an account-level lock before entering that part of the workflow.

Seeing a mutex immediately brought me back to operating systems. I started asking questions that sounded like they came straight out of class:

• Who currently owns the resource?
• What does fairness mean in this queue?
• What happens when a workflow holds the mutex too long?
• Does losing the lock stop the workflow, or can it keep running?
• Can a workflow starve even if the queue is moving?
• What state is left behind when work is interrupted?

Those questions did not produce an answer by themselves. They gave me a map for reading unfamiliar code.

I learned that lock requests were queued and that the mutex had an aging mechanism for waiters. Aging is a scheduling technique that increases priority the longer work waits, reducing the chance that newer work continually jumps ahead of older work. That gave the mutex a form of fairness and supported forward progress through the queue.

Queue fairness and workflow completion are different guarantees.

A waiting workflow could eventually acquire the lock and still have too little time left to finish its work. The queue could be fair while the overall workflow system still failed to make durable progress.

The lock and the workflow had different lifetimes

The lock and the workflow had separate time limits:

For example, suppose one workflow acquired the account lock and was still processing when its one-hour lock limit expired. Another workflow could then acquire the same lock, even though the first workflow had not reached its three-hour deadline. If the first workflow continued its protected operations, both could be working on the same billing account at once—the overlap the mutex was meant to prevent.

This is lease expiration: permission to hold a resource ends, but the previous holder is not necessarily stopped. I initially connected the lock timeout to preemption from operating systems, where the scheduler pauses a process so another can use the CPU. Here, expiration alone did not pause the workflow.

Losing permission to continue is not the same as being stopped.

That gave me a question to investigate: what happens to the previous holder after it loses the lock? This showed me that the mutex was not reliably protecting the critical section: the previous holder could continue executing after another workflow acquired the lock. I stopped pursuing this as a root-cause lead because the temporary workaround the team had applied was to disable the mutexes.

A senior developer explained that the configuration had worked for a long time and that the three-hour deadline was a generous buffer, not a calculated execution budget.

Waiting consumed the deadline

The next detail changed how I understood the failure mode: the three-hour workflow deadline started when the workflow started, not when it acquired the lock.

Time spent waiting in the mutex queue consumed part of the workflow’s execution budget.

If a workflow waited for one hour, it did not receive three fresh hours to complete its work. It had roughly two hours left.

That became the beginning of the possible snowball effect:

1. More contention meant longer waits.
2. Longer waits left less time to complete work after acquiring the lock.
3. Less execution time made a timeout after partial work more likely.

How a backfill could use the whole budget

This mattered during large backfills. A full-month finalization could process as many as 720 hours of usage, with activity calls performed for each hour. Even a small average cost per hour adds up quickly. As an illustrative calculation, at 15 seconds per hourly unit, 720 units take three hours before accounting for queueing, database slowdown, retries, variability, or overhead.

The system combined several reasonable-looking clocks into one risky budget:

time consumed from the workflow deadline
    = time waiting for the mutex
    + time performing the backfill
    + time lost to slower database operations

Under normal conditions, the workflow might complete. Under contention, queueing left less time for execution. Under database load, each unit of work took longer. A workflow that normally fit inside its deadline could cross that deadline in a degraded state.

Retrying removed the work already done

After waiting for the lock, a workflow could time out with its work only partially completed.

What the reset removed

The three-hour timeout ended the workflow, but it did not behave like rolling back a database transaction. The workflow could leave partial progress behind. When the failed work was retried, however, the reset behavior removed all the work completed by that attempt and started over.

There was no explicit retry policy on the workflow itself. The surrounding trigger and reset behavior allowed a later invocation to retry the failed work. That distinction mattered when I was tracing the code: the absence of a workflow retry policy did not mean failed work would never run again.

The cost of starting over

The retry did not resume from the previous attempt’s progress. It discarded that progress and required the database to do the work again. That made me look beyond whether another invocation could eventually produce the right result. I also needed to understand how much work repeated failures would force the system to redo.

This did not look like a classic deadlock. The system was active, but work could be performed, removed on retry, and performed again without the workflow reaching completion.

The failure-amplification loop

Once I connected waiting, timeouts, and the removal of previous work on retry, I could build a model that made the high database usage and low throughput less contradictory:

flowchart TD
    A[Workflow waits for the account lock]
    B[Waiting consumes part of its deadline]
    C[Workflow times out after partial progress]
    D[Retry removes the previous attempt's work]
    E[Work starts over]
    F[Repeated work increases database load]
    G[Database operations become slower]
    H[More workflows time out]

    A --> B --> C --> D --> E --> F --> G --> H
    H --> D

This remained a working hypothesis for how failure could amplify, not a proven root cause or an explanation of the original trigger. A missed deadline could lead to a retry that discarded previous progress. Starting over would add load to the database, slower operations could push more workflows past their deadlines, and those failures could create still more repeated work.

The model gave me one explanation for how the database could remain extremely busy while useful throughput stayed low: the system may have been spending increasing amounts of capacity redoing work from earlier failed attempts rather than completing new work.

It also gave me a way to interpret the unsuccessful database scaling. More capacity provided room to operate, but it did not change how waiting consumed the deadline or how retries discarded progress. If this loop was occurring, repeated work could consume the new headroom while throughput remained low.

What the class actually gave me

The operating systems concepts did not give me the answer. They gave me familiar pieces inside an unfamiliar system.

I did not understand the whole workflow architecture when I began investigating it. But I understood mutual exclusion and scheduling. Those concepts gave me questions about ownership and waiting; the answers led me to lifetimes, deadlines, and what happened when failed work ran again. Each question helped me build outward from the part I recognized toward a useful model of the system.

That changed how I think about learning fundamentals. Their value extends beyond encountering the same textbook problem later. When I encounter a system I do not understand, they give me somewhere to start.