Ytdyklly

I have a csv-file that consists of all match outcome probabilities for soccer matches. Each math can be result in a win, draw or loss. I also included the actual outcome. In order to test how accurate my predictions are I want to use the Ranked Probability Score (RPS). Basically, the RPS compares the cumulative probability distributions of the predictions and the outcome:

$ RPS = frac{1}{r-1} sumlimits_{i=1}^{r}left(sumlimits_{j=1}^i p_j - sumlimits_{j=1}^i e_j right)^2, $

where $r$ is the number of potential outcomes, and $p_j$ and
$e_j$ are the forecasts and observed outcomes at position $j$.

For additional information, see the following link.

import numpy as np

import pandas as pd



def RPS(predictions, observed):

   ncat = 3

   npred = len(predictions)

   RPS = np.zeros(npred)



   for x in range(0, npred):

      obsvec = np.zeros(ncat)

      obsvec[observed.iloc[x]-1] = 1

      cumulative = 0

      for i in range(1, ncat):

          cumulative = cumulative + (sum(predictions.iloc[x, 1:i]) - sum(obsvec[1:i])) ** 2

          RPS[x] = (1/(ncat-1)) * cumulative



    return RPS



df = pd.read_csv('test.csv', header=0)

predictions = df[['H', 'D', 'L']]

observed = df[['Outcome']]

RPS = RPS(predictions, observed)

The first argument (predictions) is a matrix with the predictions and the corresponding probabilities. Each row is one prediction, laid out in the proper order (H, D, L), where each element is a probability and each row sum to 1. The second argument (observed) is a numeric vector that indicates which outcome that was actually observed (1, 2, 3)

Feel free to give any feedback!
Thank you

Edit:

For some reason I am not able to reproduce the results of Table 3 of the link. I use Table 1 as input for predictions and observed. Any help is much appreciated!

Edit #2:

Hereby the small sample of the paper:

predictions = {'H': [1, 0.9, 0.8, 0.5, 0.35, 0.6, 0.6, 0.6, 0.5, 0.55],

               'D': [0, 0.1, 0.1, 0.25, 0.3, 0.3, 0.3, 0.1, 0.45, 0.1],

               'L': [0, 0, 0.1, 0.25, 0.35, 0.1, 0.1, 0.3, 0.05, 0.35]}



observed = {'Outcome': [1, 1, 1, 1, 2, 2, 1, 1, 1, 1]}

Tidy up your math

cumulative = cumulative + (sum(predictions.iloc[x, 1:i]) - sum(obsvec[1:i])) ** 2

can be

cumulative += (sum(predictions.iloc[x, 1:i]) - sum(obsvec[1:i])) ** 2

and

RPS[x] = (1/(ncat-1)) * cumulative

should be

RPS[x] = cumulative / (ncat-1)

Make a main method

This is a very small script, but still benefits from pulling your global code into a main.

PEP8

By convention, method names (i.e. RPS) should be lowercase.

That's all I see for now.

Thanks for all the replies. In the end it was relatively simple. My code is based on R, matrices start with 1 in R. Python, however, starts with 0. Adjusting my original code with this insight (sorry..) I am able to reproduce the output. I also included the remarks of Reinderien.

import numpy as np

import pandas as pd



def rps(predictions, observed):

ncat = 3

npred = len(predictions)

rps = np.zeros(npred)



for x in range(0, npred):

   obsvec = np.zeros(ncat)

   obsvec[observed.iloc[x]-1] = 1

   cumulative = 0

   for i in range(1, ncat):

       cumulative += (sum(predictions.iloc[x, 0:i]) - sum(obsvec[0:i])) ** 2

       rps[x] = cumulative / (ncat-1))



   return rps



df = pd.read_csv('test.csv', header=0)

predictions = df[['H', 'D', 'L']]

observed = df[['Outcome']]

rps = RPS(predictions, observed)