Behind the scenes of running a DeFi project
There is a lot more to running a DeFi project than meets the eye of the typical end user. Most people only see & interact with the Dapp and website and never give much thought to what lies beneath.
In this post, we are going to pull back the curtain and show you all the moving parts that need to come together to create a platform like YOP Finance, how we maintain situational awareness about the services we run, the services we depend on and the crypto markets we are invested in.
The diagram below shows the YOP Finance Ecosystem architecture — that is the main YOP Finance components we have developed, the external services we integrate with and the monitoring / alerting tools we use to ensure things are working as expected.
In the diagram above, component groups are denoted with a dotted line around a set of components. Clockwise from top left, the component groups are:
Blockchain / HTTP Gateway
This component group includes components that provide gateways (or bridges) between the language of the internet (HTTP Requests) and the language of the blockchain (Transactions). These services expose HTTP APIs which can be invoked to read from and write to the blockchain. These services are used by the YOP Finance services to read on chain data and are used by the end user to write transactions to the blockchain. Unlike traditional Web 2 applications, where writes are sent to the application server and persisted to a database, with Web 3, the user interacts directly with the blockchain (via the gateway configured in their wallet), meaning that no transactions are ever routed via the YOP Finance Servers.
This component group also includes all the YOP Finance Smart Contracts (as detailed here: https://docs.yop.finance/security/smart-contracts) and the Smart Contracts of the DeFi Protocols that YOP Finance interacts with (e.g. Curve, Convex, Uniswap etc…)
Core YOP Finance Services
This component Group include all of the core services required to provide the YOP Finance Platform to end users. It includes components such as the Website, Docs Site, Dapp and backend APIs. These APIs are responsible for merging data from multiple sources (both on and off chain) to create useful data sets for the front end Dapp.
Price Feed Data
Real time price feed data is required to provide up to date dollar values against crypto assets. Price feed data for all supported tokens are handled by Chainlink (see here for a full list of their supported tokens). For any tokens which are not supported by Chainlink, we use a combination of Coin Gecko and Coin Market Cap. Having integrations to both of these services gives us a level of redundancy, so that if one service is unavailable, we can still source the data from the other service.
Monitoring & Alerting
This component group is critical to the smooth operation of the YOP Finance Platform as it provides the situational awareness required to manage the platform.
Monitoring and Alerting are (or should be) two sides of the same coin. You need them both working properly for effective situational awareness.
- Monitoring is the process of observing a systems behaviour over time
- Alerting is the process of triggering a notification in response to some event, threshold or other observed event.
Without monitoring in place, you have nothing to alert on.
Without alerting, you have no situational awareness.
Without a response plan (or Standard Operating Procedure) your alert is not actionable.
We will discuss more on Monitoring, Alerting and response plans in the section below on Administration.
This component group includes services that we use to protect the YOP user, protocol and tokens.
- Gnosis Safe is used for managing multi-sig transaction on the YOP Finance protocol related to governance
- Fireblocks is used for managing the YOP Finance Treasury and various other token pools including community, marketing and reserves
- Elliptic is used to screen user wallets to ensure that no illegal or illicit funds are deposited into the YOP Finance protocol.
Keeping all of the above components running smoothly together and working as expected is an ongoing task. For proactive administration (e.g. harvesting strategies, changing vault caps or rebalancing emissions), we schedule these in advance and execute during the working day.
For reactive administration, we rely on our monitoring & alerting tools as well as our standard operating procedures.
At YOP Finance we use a selection of different monitoring tools for different purposes. This includes the following tool set:
- Sentry.io — Error monitoring for failed API requests between the Dapp and the backend API
- Pingdom.com — real-time, actionable insights into your site’s uptime and performance. Used for Website, Docs Site and Dapp
- DataDog — log file monitoring & for front end Dapp and back end APIs
- Peg & Pool Monitoring — home grown system for monitoring stable coin and stETH pegs as well as the ratio of tokens in Curve pools. Alerts when specific thresholds are breeched (e.g. a de-pegging on 1% or a pool balance above 7-/30)
While many of the monitoring tools we use can do their own alerting, we find it far more efficient to have all the monitoring tools sending their alerts and notifications to Pager Duty, where we can configure on call rosters, escalation procedures and much more.
Standard Operating Procedures
Standard Operating Procedures (SOPs) are the step by step diagnostic and recovery documents which detail what to do in the event of a specific alert being triggered. For each alert, there needs to be an associated procedure which is executed in response to that alert. These procedures define what systems are affected, where to access them, what permissions are required and what specific steps to take to resolve an alert.
We have been putting all these SOPs in place over the last number of months and will continue to build them out as well as new monitoring probes and alert thresholds as the protocol develops.
For on chain SOPs, the complexity increases because of multi-signature governance. All YOP Finance Governance wallets have multi-key signing requirements meaning that multiple people need to be available for signing in order to execute an on chain step in an SOP. At the highest level of multi-sig is the main governance wallet, which requires 4 signers (out of a total of 9). This means that if we need to execute a core governance transaction out of hours, we need to make contact with at least 4 keyholders to get a transaction approved. In these situations (e.g. token depeg, protocol instability), time is of the essence, which means that it is vital to get signers activated as soon as possible. We are still working on the final multi-sig notification and alerting and aim to have this in place soon.
As you will hopefully have seen, there is quite a lot that goes into building and running a DeFi platform. It is not all about pushing out new features, partnerships and announcements. Spending time to ensure that the foundations are stable and that you have sufficient situational awareness is absolutely vital.