Price discrepancy between oracle and the AMM causes liquidations and auto-rebalancing to pause

[turbolent]

Issue To Be Solved

Pause liquidations and rebalancing involving asset X automatically if no trustworthy price is available for asset X.

This is an important disaster prevention mechanism, i.e. a hard rule that FCM must always abide to:
If no trustworthy price is available for asset X, user positions (both collateral and debt) in asset X are not to be touched!

Suggested Solution

The Oracle should abort [..] if "the price seems weird".

Reasons to conclude that "the price seems weird":

• At least one of our price sources (including, perhaps, multiple off-chain oracles and multiple on-chain dexes) deviates by more than X% from the other price sources
• At least one of our price sources (or the median price itself) has changed by more than Y% in the last Z seconds

Further reading

slack thread where this is discussed https://flow-foundation.slack.com/archives/C08QF29F7TK/p1769540691604539 (in addition, this slack thread also discusses an independent governance mechanism for pausing liquidations, as described in issue #126)

[AlexHentschel]

Context

For resilience, FCM utilizes diverse (different) price oracles for the same asset:

• AMM provides a price estimate that is solely bases on the AMM's internal reserve balances. Depending on the reserve volume and a user's financial reserves, it is possible to manipulate the AMM's price
• third party oracles entail risks as well, where in extreme cases an oracle might be byzantine

We want to avoid (intentional or accidental) attack / fault vectors, where users' positions get liquidated or rebalances at incorrect prices, if one of the price sources is off.

---

Def of done:

Conceptually, The software design and documentation must clearly differentiate between

1. an FCM-internal notion of a trustworthy price for asset X

2. Mostly reliable but not universally trusted oracles that provide price estimates, which are usually quite accurate. However, in edge case scenarios, price estimates might be byzantine, i.e. (i) incorrect or (ii) unavailable.

   While scenario (ii) is relatively trivial to detect (depending on the implementation), (i) can be more subtle to notice, as colluding adversarial price inputs should be considered for the mature protocol.

FCM (to the best of my understanding) proceeds as common in blockchain:

• We take (typically a group) of individual mostly reliable oracles from the set 2. We axiomatically assume that those oracles are reasonably diverse and therefore, in their "collective behaviour" considered trustworthy. Note that this is a very prevalent axiom in web3, however the soundness of this axiom highly depends on the problem at hand.
• Alternatively, among the class of 2., a specific oracle might be specified as trustworthy by governance.
• Alternative approaches for implementing a trustworthy oracle should be possible. Important: FCM defines which oracles it considers trustworthy, not the caller (think: it is your responsibility that nobody is giving your fake money as change, not the person handing you the change in a potentially adversarial environment 😅 ).

• It would significantly simplify the formal analysis if all oracles in set 2. (incl. trustworthy oracles) would be allowed to return $\perp$ (undefined value), often represented as nil (pointer) in many languages. Not sure what the best approach is in cadence. Neither the trusted nor mostly reliable oracle should abort the transaction. There are many plausible scenarios where we might want to see if a price is available and if not proceed with an alternative.

• So we have multiple different implementations of a trustworthy oracle. Conceptually, the kind of trustworthy oracle that FCM uses is a container, referencing m mostly reliable oracles for the asset X. Furthermore, our 'trustworthy oracle container' has some logic and tuneable parameters, upon which it can return a trustworthy price or default to returning $\perp$.

  We should document the broad high-level idea ☝️ in the code. Though, for the first iteration, I think we should implement only the 'trustworthy oracle's that FCM initially needs: potentially a governance-authorized price oracle (maybe an attachment (?) that the governance can issue? Adding the credibility to a mostly reliable oracle to be considered trusted?).

• ease of tuning and updates:
  • to me, it would make sense for FCM to maintain one trusted oracle for each asset it is managing, being collateral or yield tokens (debt).
  • under the hood, each trusted oracle decides how to compute it's return. While each asset in FCM references one (only one) trusted oracle, for different assets, the trusted oracles can can work differently. (Formally, the map from asset type to trusted oracles is surjective [wiki].)

  It should be easy from an API perspective and anticipated from a system design perspective that the trusted oracles can change over time, e.g. for one specific asset, governance decides that the trusted price should be computed by trusted oracle $\Omega_1$ going forward as opposed to the existing $\Omega_0$. Alternatively, governance could decide to tune the parameters of an existing trusted oracle for asset X. Those updates would happen via isolated governance transactions, hence be atomic and happen either before or after any FCM logic running.

• For the first FCM release, we'll need the following specific implementation of 'trusted oracle' (rough, to be refined, please scrutinize):
  • We have a resource type such as MinDecentralizedTrustedOracle. An instance of which references an AMM as one price oracle and a third-party oracle as a second source.
  • Upon a price request, if any source return $\perp$, MinDecentralizedTrustedOracle returns $\perp$
  • If the relative price difference is too significant, MinDecentralizedTrustedOracle returns $\perp$
  • Otherwise (only remaining happy case): MinDecentralizedTrustedOracle returns some aggregate of both price signals. Formally, the most accurate approach would probably be the weighted geometric mean of both prices. The weights being the liquidity that is available at these prices. As the third party oracle does provide no liquidity in practise, its weight is zero and the geometric mean collapses to the AMM price

• Let's make sure we have proper documentation of what is supposed to happen.

• The details of how FCM derives trusted oracles (1) from (2) is relevant for the security of the system. So the details of this are important for the white paper. Please confirm / discuss with Alex.

Suggestions

I would like to provide the following architectural suggestions based on my experience engineering high-assurance systems. However, please take them with a grain of salt that I neither know the existing FCM codebase nor cadence mostly at all 😅. The following is meant as a starting point for further refinement; please don't hesitate to disagree, deviate, etc ... please be creativity 💡 beyond just my suggestion if you feel like it 🙂

• The software design and API should be architected that incorrectly conflating 1. and 2. does not produce runnable code. A safe-by-default design is a must-have requirement for high-assurance software. In my experience, pushing this as technical debt into the future has an unacceptably high cost: accidental bugs can result in bugs, code that needs to be refactored grows over time. Where appropriate, please refactor existing code ❤️. That work is improving on core safety aspects of the code, so time well spent imho! If we are talking about more than a week of work (as per Dete's recommendation), come and talk to me or Dete, please.

  Interfaces:

  • From the perspective of a high-assurance system, you should 1. and 2. should not share a common interface. This is because they differ in core aspects: they provide fundamentally different security guarantees.

  Let's unpack this a bit: we have the large set 2. of "Mostly reliable but not universally trusted oracles". As part of that large set, we have the subset of oracles that are actually trustworthy - they provide strictly stronger guarantees. In terms of software design: interface TrustworthyOracle" would be _derived_ from the interface MostlyReliablePriceOracle` (potentially currently called 'PriceOracle'?). This design conforms with our expectations: if some code is fine with a mostly mostly reliable oracle, the caller can provide a trusted oracle. However, the reverse does not hold: if the called code needs a trustworthy oracle, providing a mostly reliable oracle is not an acceptable input.

  I think Dete also commented on this aspect on slack:

  Yeah. I think Claude [AI Review of FCM] is getting confused there because “Price Oracle” in the code that it is looking at isn’t a singular price oracle. It’s an object that implements the “PriceOracle” interface that should include the logic I listed above. (And will return a median price if it is NOT the case that “the price seems weird”.)

  Although, looking at the code, it seems like we are using the Band Oracle directly and are not doing any of what I described above… 😕

[jordanschalm]

Pause liquidations and rebalancing involving asset X automatically if no trustworthy price is available for asset X.

This is implemented for manual liquidation in the form of a pre-liquidation check. We get a quote from the DEX, compare against the oracle price, and revert if the difference is too large. If the oracle returns nil, indicating no price is available, we will also revert. Obviously this assumes the oracle works satisfactorily.

I think there are two points to consider here:

1. FCM <-> Oracle Interface From the perspective of FCM (as in FlowCreditMarket smart contract), what does an "Oracle" look like? Currently it uses the DeFiActions.PriceOracle interface: it accepts a token type and returns either a price or nil. FCM holds one PriceOracle reference (not one per token type).

I think a single oracle interface and reference is ideal in the context of FCM, even if the implementation of that oracle elsewhere contains more details.

2. How should the Oracle be implemented Everything you said above, @AlexHentschel. I'm sure there are more details to consider too. I would suggest creating a separate issue for this discussion, once we align on 1.

Let's make sure we are aligned on 1. first, because it allows us to implement functions within FCM that depend on the oracle as a dependency, before the oracle is implemented. We should be able to implement this issue based on an interface definition and documented behaviour of "An Oracle" from the perspective of FCM.

[AlexHentschel]

Update & summary from technical sync:

Kan and Dete both disagreed that an API-level differentiation between trusted and mostly reliable oracles should be required, while diligent documentation is universally agreed to be a hard requirement. The argument is that documentation provides the necessary information while misuse remain still possible even with an API-level differentiation.

While I think that argument is true in theory, the practical difference is still quite significant and important in practise based on my experience engineering high-assurance systems for several years. We are tying to push our confidence of correct code from something like 90% to 95% and in order to do so, we should be worrying about rare but plausible sources of bugs as well, including accidentally supplying an untrusted oracle in place of a trusted one. We have numerous empirical evidence that this kind of "safe by default" APIs make a significant dent on the bug probability in the long run.

Anyway, risks have been discussed and are understood - though differently weighted. The final call whether or not to differentiate trustworthiness of oracles on an API layer should be made by the engineer and their technical leads. From my side, I disagree and commit, dropping my requirement of API-level differentiation between trusted and mostly reliable oracles.