Ytdyklly

I have written an ensemble using Scikit Learn VotingClassifier.

I have set a seed in the cross validation section. However, it does not appear to 'hold'. Meaning, If I re-run the code block I get different results. (I can only assume each run of the code block is dividing the dataset into folds with different constituents instead of 'freezing' the random state.

Here is the code:

#Voting Ensemble of Classification

#Create Submodels

num_folds = 10

seed =7

kfold = KFold(n_splits=num_folds, random_state=seed)

estimators = 

model1 =LogisticRegression()

estimators.append(('LR',model1))

model2 = KNeighborsClassifier()

estimators.append(('KNN',model2))

model3 = GradientBoostingClassifier()

estimators.append(('GBM',model3))

#Create the ensemble

ensemble = VotingClassifier(estimators,voting='soft')

results = cross_val_score(ensemble, X_train, Y_train,cv=kfold)

print(results) 

The results printed are the results of the 10 CV fold training. If I run this code block several times I get the following results:

1:

[0.70588235 0.94117647 1.         0.82352941 0.94117647 0.88235294

 0.8125     0.875      0.8125     0.9375    ]

2:

[0.76470588 0.94117647 1.         0.82352941 0.94117647 0.88235294

 0.8125     0.875      0.8125     0.875     ]

3:

[0.76470588 0.94117647 1.         0.82352941 0.94117647 0.88235294

 0.8125     0.875      0.8125     0.875     ]

4:

[0.76470588 0.94117647 1.         0.82352941 1.         0.88235294

 0.8125     0.875      0.625      0.875     ]

So it appears my random_state=seed isn't holding.

What is incorrect?

Thanks in advance.

Random seed of models (LogisticRegression, GradientBoostingClassifier) needs to be fixed too, so that their random behavior becomes reproducible. Here is a working example that produces the same result over multiple runs:

import sklearn

from sklearn.model_selection import KFold, cross_val_score

from sklearn.linear_model import LogisticRegression

from sklearn.neighbors import KNeighborsClassifier

from sklearn.ensemble import GradientBoostingClassifier, VotingClassifier

import numpy as np



#Voting Ensemble of Classification

#Create Submodels

num_folds = 10

seed =7



# Data

np.random.seed(seed)

feature_1 = np.random.normal(0, 2, 10000)

feature_2 = np.random.normal(5, 6, 10000)

X_train = np.vstack([feature_1, feature_2]).T

Y_train = np.random.randint(0, 2, 10000).T



kfold = KFold(n_splits=num_folds, random_state=seed)

estimators = 

model1 =LogisticRegression(random_state=seed)

estimators.append(('LR',model1))

model2 = KNeighborsClassifier()

estimators.append(('KNN',model2))

model3 = GradientBoostingClassifier(random_state=seed)

estimators.append(('GBM',model3))

#Create the ensemble

ensemble = VotingClassifier(estimators,voting='soft')

results = cross_val_score(ensemble, X_train, Y_train,cv=kfold)

print('sklearn version', sklearn.__version__)

print(results)

Output:

sklearn version 0.19.1

[0.502 0.496 0.483 0.513 0.515 0.508 0.517 0.499 0.515 0.504]