Most generally, the outcome of a channel is some data that says what should happen outside the channel when it is closed. The most minimal, specific example would be an array of balances for the participants of that channel. There are a large number of steps in between (for example we at FunFair take part of the channel balance and pay to (channel-specific) third parties before the remaining funds are distributed).
In an ideal world, State Channels end by mutual consent; this means that the outcome of a channel can be “clean” - for example at the end of a game where there are no funds at stake.
Regardless of your approach to finalisability of a channel however, the core tenet of a Dispute(/Challenge) is that it can be made at any state transition, whether or not the State Machine(/app) thinks this is a good idea or not.
Consequently we need the concept of a default outcome for each state as the channel progresses; where you start from in the case of a dispute before any penalties are applied.
Add to this thought the concept of validating outcomes. YMMV depending on what the outcome actually specifies, but in the simple case of a list of balances it would seem self evident that the balances should always add up to the channel balance. You could apply rules where this isn’t necessarily the case (and I know Tom has some, although I don’t really understand what they are). However these are still rules.
I posit that the validity of the default outcome should be verified as part of each state transition. If a state machine “invents” funds during the course of a channel, then it’s going to cause problems when the participants try and close it. We do this religiously in our code.
This then asks the question as to where this validation should take place. The State Machine knows the individual participant balances but may not (and often doesn’t need to) know the channel balance. The State Channel(/Force Move) code might know the channel balance, but it might not if it’s being hooked into a separate ledger channel or a separate funding contract.
Finally, participants need to validate that state transitions from their counterparties are valid progressions of the state according to the protocol. I would suggest that this would ideally be able to be acheived in a single call() function. As I’ve been refactoring my code, this call has moved further and further out so it now sits on the top level. It’s in the code that knows the channel funding and associated rules. It calls the code that validates that the state transition is actually appropriate for this channel (ID’s, nonces, signatures etc), which in turn calls the State Machine to process the state transition logic. This returns a default outcome, which in turn is returned to the top level where it can be validated against the channel funding rules.
Does this make sense? I’ve spent that last 15 years writing code that processes virtual currency, and have a lifetime of paranoia about accidentally inventing or destroying it!