Minor Tweaks to Scoring and Penalties
Currently, the scoring mechanism looks at the percentile rank of each miner in each category, using a product of these ranks between all metrics to determine the final score. This forces some degree of balance between the metrics and has led to some well-balanced strategies scoring well on our system.
We have removed most of the penalties in the system with the incentive V3 update, which has greatly simplified scoring on the system. With the introduction of more standard scoring metrics, a minimum 60-day period, and daily returns, this is now possible. We have also historically been using a percentile-based max drawdown calculation, which has given miners some degree of flexibility in their strategies.
Problem Statement
The current state of the system lacks clarity for a number of reasons. A product-based penalty obscures the influence of each category on the overall score. This makes it more challenging to understand the trajectories the miner might take to improve their score.
The abnormality score is also difficult to understand. Its intention is to give the system some level of protection against spiraling strategies prior to their detection in the main scoring system. However, with the introduction of a specific martingale penalty, much of this pure cascading risk is either captured explicitly in the martingale penalty or passively in other scoring categories.
While martingales are already limited in capacity due to leverage constraints and likely only work on mean-reverting assets like forex pairs, this style of trading behavior is really not the target of the system and in general is not something we want to even implicitly encourage. We will introduce a specific martingale penalty to capture this behavior.
Proposed Adjustments
We are looking to make a few different simplifications to make our system easier to interpret and penalties more straightforward:
- The main scoring system will now be a summation of percentiles on scores, instead of a product.
- Max drawdown normalization for returns will switch back from the 90th percentile to the max overall drawdown.
- The abnormality penalty will be removed, and replaced with a specific martingale penalty.
Sum-based Scoring
Our new scoring system will be a summation of percentiles on scores, instead of a product. Instead of iteratively multiplying a final score of 1 by the percentile rank of each metric, we will now start from 0 and add to this final score based on the weighted average of each subcategory. This will make it substantially easier to understand the influence of each category on the final score and help us build clean visualizations as a final output. Here is the equation before penalties are applied:
Max Drawdown Normalization
With a more balanced overall scoring system, there is not as much influence from the overall scoring function. To simplify the interpretation of our return normalization and use industry standard metrics, we will switch back from a 90th percentile max drawdown to the max overall drawdown. This will make it easier to understand the influence of drawdowns on the final returns as a ratio between return and max drawdown.
Martingale Penalty
The martingale penalty will start in a relatively straightforward form, and it will likely be the first part of a suite of specific metrics meant to similarly proactively protect against harmful strategies.
For each position, we will count the position as a potential martingale if it increases the maximum leverage on a losing position at least twice. This will allow us to categorize each position into a list of 0s and 1s. We then run a weighted average on this list based on the positional returns, such that positions with larger returns will have relatively more weight in the final score. This will give us a final estimated “score” ranging between 0 and 1. Inputting this score into a sigmoid will then allow us to smoothly penalize strategies with a high proportion of martingale positions, with anything larger than 5% infiltration becoming a target of the penalty.
The graph below displays a losing position for a miner with each of the 5 orders being represented by a dot. Only the orders that surpass the magnitude of the maximum cumulative leverage already seen are counted as martingale steps. Since this example position has at least 2 martingale steps, this would be classified as a possible martingale position and would be penalized.
Timeline
Proposal Delivery: Dec 13, 2024
PR: Dec 13, 2024
Integration: Dec 16, 2024