Clarify required error information in lmcontrol API

[NicolasFussberger]

What

Currently, the lmcontrol API will only provide an enum value as error information indicating if a RunTarget change happed due to a RecoveryAction or a StateManager request.

It is not possible to identify which components have failed in case the transition is triggered by a RecoveryAction.
It is not possible identify intermediate transitions that lead to the final RunTarget.
Knowing in which active RunTarget the error occurred requires tracking of the active RunTarget by the user of the API.

We shall clarify the use cases that need to be supported and extend the interface with the required information.

See also discussion here:
#478 (comment)

Acceptance Criteria (DoD)

• Decide if and how to extend the interface with this error information

How

No response

[hskang-amelia]

Thanks for opening this. Since the DoD asks for the use cases first, here's what SM actually needs.

1. Picking a recovery policy. When SM sees a RunTarget change with kRecoveryAction, it has to decide what to do next: retry, degrade further, or escalate. That depends on which element failed and how. Without it every recovery ends up with the same worst-case response, whether a non-critical component died or a core one did.

2. Telling apart "fell back after one hop" and "fell back after exhausting the chain". In your example (A, C1+C2 fail, B, C3 fails, fallback), C1 alone doesn't separate those two. They need different handling once we're sitting on the fallback target, because the second case means the damage is wider than one component. So one error per attempted RunTarget, not the whole per-component chain.

3. Matching a result to a request. This is your third bullet but I'd put it differently. Tracking the active RunTarget on the SM side is fine. What we can't do right now is tell whether an incoming result is the answer to the request we just sent. If SM asks for A and a recovery kicks off at the same time, activation_source separates the two causes but not two requests in a row from the same source, so we can't safely throw away a late result.

4. SM restart. A fresh SM needs the current RunTarget, but it shouldn't be handed an error from an activation that finished long ago. That's the validity question below.

Rough shape, names aside:

struct ActivationFailure {
    RunTargetName    attempted_run_target;
    ExecutionError   execution_error;
    ProcessGroupName process_group;
    std::uint8_t     additional_failures;  // further failures in the same attempt
};

struct ActivationResult {
    RunTargetName              activated_run_target;
    RunTargetActivationSource  activation_source;
    std::uint64_t              transition_id;
    FixedVector<ActivationFailure, N> failures;
};

N is the recovery chain length, which the manifest fixes, so this doesn't grow with the number of failing components. Inside a single attempt, the first failure plus a count of the rest keeps the payload flat and still avoids the case where a harmless C1 hides a serious C2.

Two things around this.

executionError granularity: ExecutionErrorEvent{executionError, processGroup} is already in the tree at process group level. If executionError is meant to identify the failing element, that only works when the manifest assigns distinct values per process. Otherwise everything shows up as the same default and there's nothing to branch on. Whichever way it goes, probably worth writing the expectation down rather than leaving integrators to find out.

Validity window: if the error ends up inside a Field (the discussion in #478), it gets last-value semantics and stays readable indefinitely, so a late subscriber reads an error from an activation that's already superseded. We'd need to say when it's cleared, either on the next successful activation or via the transition id.

One question. Does a recovery transition that LM starts on its own, with no external request, still publish activation_result? 1 and 4 depend on that and I couldn't tell from the code.

Happy to turn this into a PR if it helps. Also, could you send me the lifecycle meeting schedule?