tanzania_classifier_training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Training Classification Models\n",
    "\n",
    "The following models will be tested:\n",
    "1. Logistic Regression\n",
    "2. K-Nearest-Neighbors\n",
    "3. Complement Naive-Bayes\n",
    "4. Random Forest\n",
    "5. XGBoost\n",
    "6. SVM with Kernel Trick\n",
    "\n",
    "This dataset is a part of a competition of *drivendata.org*, so the test inputs are provided without their targets.\n",
    "A submission can be made on their platform on order to evaluate a classifier, however only 3 submissions per account\n",
    "are allowed each day. For that reason, 10% of the training samples will be chosen as test samples, in order to pick\n",
    "the best-performing classifiers. Then, these classifiers will be tuned using 10-Fold Cross-Validation and **Optuna** library."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Importing Train-Test Datasets"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "outputs": [
    {
     "data": {
      "text/plain": "((59400, 42), (59400, 2), (14850, 42))"
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "train_inputs = pd.read_csv('train_inputs.csv')\n",
    "train_targets = pd.read_csv('targets.csv')\n",
    "test_inputs = pd.read_csv('test_inputs.csv')\n",
    "\n",
    "train_inputs.shape, train_targets.shape, test_inputs.shape"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Dropping Columns\n",
    "\n",
    "All extra columns that contain no useful information are dropped from the dataset."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "outputs": [
    {
     "data": {
      "text/plain": "((59400, 21), (14850, 21))"
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_input_ids = test_inputs['id']\n",
    "\n",
    "columns_to_drop = [\n",
    "    'id', 'date_recorded', 'longitude', 'latitude', 'wpt_name', 'num_private', 'subvillage', 'region', 'ward', 'recorded_by',\n",
    "    'scheme_name', 'construction_year', 'extraction_type', 'extraction_type_class', 'management_group', 'payment', 'quality_group',\n",
    "    'quantity_group', 'source_type', 'source_class', 'waterpoint_type_group',\n",
    "]\n",
    "\n",
    "train_inputs = train_inputs.drop(columns=columns_to_drop)\n",
    "test_inputs = test_inputs.drop(columns=columns_to_drop)\n",
    "\n",
    "train_inputs.shape, test_inputs.shape"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Encoding Targets\n",
    "\n",
    "There are 3 classes:\n",
    "\n",
    "1. `functional`: The pump is functional\n",
    "2. `functional needs repair`: The pump is function but it is damaged, so it needs immidiate repair\n",
    "3. `non functional`: The pump is damaged and not operating at all"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "outputs": [
    {
     "data": {
      "text/plain": "0        0\n1        0\n2        0\n3        2\n4        0\n        ..\n59395    0\n59396    0\n59397    0\n59398    0\n59399    0\nName: status_group, Length: 59400, dtype: int32"
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_targets = train_targets['status_group'].replace({'functional': 0, 'functional needs repair': 1, 'non functional': 2}).astype(int)\n",
    "train_targets"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Encoding Inputs\n",
    "\n",
    "Inputs of this dataset are both numerical and categorical, so inputs must first be encoded before fed into the classifiers.\n",
    "The following encoders will be used:\n",
    "1.   `Ordinal (Label) Encoding`: Converts categorical variables to ordered numerical values\n",
    "2.   `Count Encoder`: Replaces each categorical variable with its frequency count in the column\n",
    "3.   `Hashing Encoding`: Hashes categorical variables to unique one-hot vectors with configurable dimensionality\n",
    "4.   `Backward Difference Encoder`: The mean of the $k_{th}$ categorical variable is compared with the mean of the $k_{th} - 1$\n",
    "5.    `Helmert Encoding`: The mean of the $k_{th}$ categorical variable is compared with the means of all previous categorical variables\n",
    "6.   `Cat Boost Encoding` Similar to `Target Encoder`, which converts each categorical value with the mean appearance of its target, but also involves an ordering principle, in order to overcome the `\"Target Leakage\"` problem\n",
    "7.   `Generalized Linear Mixed Model Encoder`: A supervised encoder similar to `Target Encoder`, but solid statistical theory is also included. The advantage over other supervised encoders is that no parameter tuning is needed"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "outputs": [],
   "source": [
    "from category_encoders import OrdinalEncoder\n",
    "from category_encoders import CountEncoder\n",
    "from category_encoders import HashingEncoder\n",
    "from category_encoders import BackwardDifferenceEncoder\n",
    "from category_encoders import HelmertEncoder\n",
    "from category_encoders import CatBoostEncoder\n",
    "from category_encoders import GLMMEncoder\n",
    "\n",
    "random_state = 0\n",
    "\n",
    "encoding_methods = {\n",
    "    'ordinal': OrdinalEncoder(),\n",
    "    'count': CountEncoder(),\n",
    "    'hashing': HashingEncoder(n_components=32, drop_invariant=True),\n",
    "    'backward_difference': BackwardDifferenceEncoder(),\n",
    "    'Helmert': HelmertEncoder(),\n",
    "    'CatBoost': CatBoostEncoder(random_state=random_state),\n",
    "    'GLMM': GLMMEncoder(random_state=random_state)\n",
    "}"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Building Classifier Configurations\n",
    "--------------------------\n",
    "\n",
    "* Logistic Regression (`sklearn.linear_model`)\n",
    "* K-Nearest Neighbors (`sklearn.neighbors`)\n",
    "* Gaussian Naive Bayes (`sklearn.naive_bayes`)\n",
    "* Random Forest (`sklearn.ensemble`)\n",
    "* SVM (`sklearn.svm`)\n",
    "* XGBoost (`xgboost`)"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.svm import SVC\n",
    "from xgboost import XGBClassifier\n",
    "\n",
    "random_state = 0\n",
    "\n",
    "classifiers = {\n",
    "    'Logistic Regression': LogisticRegression(n_jobs=-1, random_state=random_state),\n",
    "    'K-Nearest Neighbors': KNeighborsClassifier(n_neighbors=35, weights='distance', n_jobs=-1),\n",
    "    'Naive Bayes': GaussianNB(),\n",
    "    'Random Forest': RandomForestClassifier(random_state=random_state, n_jobs=-1),\n",
    "    'XGBoost': XGBClassifier(random_state=random_state, n_jobs=-1),\n",
    "    'Support Vector Machine': SVC(random_state=random_state)\n",
    "}"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Training Classifiers"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "----- Encoding data using ordinal Encoder -----Done in 0.29s\n",
      "Training Logistic Regression ClassifierDone in 2.13s\n",
      "Training K-Nearest Neighbors ClassifierDone in 0.3s\n",
      "Training Naive Bayes ClassifierDone in 0.02s\n",
      "Training Random Forest ClassifierDone in 0.76s\n",
      "Training XGBoost ClassifierDone in 2.7s\n",
      "Training Support Vector Machine ClassifierDone in 108.61s\n",
      "\n",
      "----- Encoding data using count Encoder -----Done in 0.5s\n",
      "Training Logistic Regression ClassifierDone in 1.53s\n",
      "Training K-Nearest Neighbors ClassifierDone in 0.26s\n",
      "Training Naive Bayes ClassifierDone in 0.02s\n",
      "Training Random Forest ClassifierDone in 0.79s\n",
      "Training XGBoost ClassifierDone in 2.65s\n",
      "Training Support Vector Machine ClassifierDone in 99.19s\n",
      "\n",
      "----- Encoding data using hashing Encoder -----Done in 14.3s\n",
      "Training Logistic Regression ClassifierDone in 2.06s\n",
      "Training K-Nearest Neighbors ClassifierDone in 0.31s\n",
      "Training Naive Bayes ClassifierDone in 0.04s\n",
      "Training Random Forest ClassifierDone in 0.89s\n",
      "Training XGBoost ClassifierDone in 4.01s\n",
      "Training Support Vector Machine ClassifierDone in 156.68s\n",
      "\n",
      "----- Encoding data using backward_difference Encoder -----Done in 2.32s\n",
      "Training Logistic Regression ClassifierDone in 17.85s\n",
      "Training K-Nearest Neighbors ClassifierDone in 2.19s\n",
      "Training Naive Bayes ClassifierDone in 0.53s\n",
      "Training Random Forest ClassifierDone in 4.22s\n",
      "Training XGBoost ClassifierDone in 39.51s\n",
      "Training Support Vector Machine ClassifierDone in 961.81s\n",
      "\n",
      "----- Encoding data using Helmert Encoder -----Done in 2.52s\n",
      "Training Logistic Regression ClassifierDone in 20.74s\n",
      "Training K-Nearest Neighbors ClassifierDone in 2.25s\n",
      "Training Naive Bayes ClassifierDone in 0.55s\n",
      "Training Random Forest ClassifierDone in 4.66s\n",
      "Training XGBoost ClassifierDone in 41.36s\n",
      "Training Support Vector Machine ClassifierDone in 970.52s\n",
      "\n",
      "----- Encoding data using CatBoost Encoder -----Done in 0.35s\n",
      "Training Logistic Regression ClassifierDone in 2.0s\n",
      "Training K-Nearest Neighbors ClassifierDone in 0.28s\n",
      "Training Naive Bayes ClassifierDone in 0.03s\n",
      "Training Random Forest ClassifierDone in 2.63s\n",
      "Training XGBoost ClassifierDone in 10.88s\n",
      "Training Support Vector Machine ClassifierDone in 119.37s\n",
      "\n",
      "----- Encoding data using GLMM Encoder -----Done in 5.46s\n",
      "Training Logistic Regression ClassifierDone in 1.58s\n",
      "Training K-Nearest Neighbors ClassifierDone in 0.25s\n",
      "Training Naive Bayes ClassifierDone in 0.02s\n",
      "Training Random Forest ClassifierDone in 0.85s\n",
      "Training XGBoost ClassifierDone in 3.17s\n",
      "Training Support Vector Machine ClassifierDone in 104.02s\n",
      "\n",
      "Classifier: Logistic Regression\tBest Score: 0.6648148148148149 on Test Data\t Using Helmert Encoder\n",
      "\n",
      "Classifier: K-Nearest Neighbors\tBest Score: 0.7728956228956229 on Test Data\t Using count Encoder\n",
      "\n",
      "Classifier: Naive Bayes\tBest Score: 0.7126262626262626 on Test Data\t Using CatBoost Encoder\n",
      "\n",
      "Classifier: Random Forest\tBest Score: 0.8042087542087543 on Test Data\t Using ordinal Encoder\n",
      "\n",
      "Classifier: XGBoost\tBest Score: 0.7986531986531986 on Test Data\t Using backward_difference Encoder\n",
      "\n",
      "Classifier: Support Vector Machine\tBest Score: 0.6861952861952862 on Test Data\t Using count Encoder\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "import time\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(\n",
    "    train_inputs, train_targets, test_size=0.1, random_state=random_state\n",
    ")\n",
    "\n",
    "classifier_best_results = {classifier_name: (0, 'encoder') for classifier_name in classifiers.keys()}\n",
    "\n",
    "\n",
    "for encoding_method, encoder in encoding_methods.items():\n",
    "    print(f'\\n----- Encoding data using {encoding_method} Encoder -----', end='')\n",
    "\n",
    "    start_time = time.time()\n",
    "\n",
    "    encoded_x_train = encoder.fit_transform(x_train, y_train)\n",
    "    encoded_x_test = encoder.transform(x_test)\n",
    "\n",
    "    end_time = time.time()\n",
    "    print(f'Done in {round(end_time-start_time, 2)}s')\n",
    "\n",
    "    for classifier_name, clf_algorithm in classifiers.items():\n",
    "        print(f'Training {classifier_name} Classifier', end='')\n",
    "\n",
    "        start_time = time.time()\n",
    "\n",
    "        clf_algorithm.fit(encoded_x_train, y_train)\n",
    "        y_pred = clf_algorithm.predict(encoded_x_test)\n",
    "        acc = accuracy_score(y_test, y_pred)\n",
    "\n",
    "        end_time = time.time()\n",
    "        print(f'Done in {round(end_time-start_time, 2)}s')\n",
    "\n",
    "        previous_acc, _ = classifier_best_results[classifier_name]\n",
    "        if previous_acc < acc:\n",
    "            classifier_best_results[classifier_name] = (acc, encoding_method)\n",
    "\n",
    "\n",
    "for classifier_name, (score, encoding_method) in classifier_best_results.items():\n",
    "    print(f'\\nClassifier: {classifier_name}\\tBest Score: {score} on Test Data\\t Using {encoding_method} Encoder')"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Detailed Classification Report\n",
    "\n",
    "The goal of this competition is to build a classifier with the highest accuracy. However,\n",
    "since this is an imbalanced dataset, it is more appropriate to rely on **Precision & Recall**."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "-----Classifier: Logistic Regression\tEncoding: Helmert-----\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.67      0.84      0.74      3253\n",
      "           1       0.21      0.01      0.02       424\n",
      "           2       0.66      0.54      0.60      2263\n",
      "\n",
      "    accuracy                           0.66      5940\n",
      "   macro avg       0.51      0.46      0.45      5940\n",
      "weighted avg       0.63      0.66      0.63      5940\n",
      "\n",
      "\n",
      "-----Classifier: K-Nearest Neighbors\tEncoding: count-----\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.78      0.86      0.82      3253\n",
      "           1       0.52      0.29      0.37       424\n",
      "           2       0.79      0.74      0.76      2263\n",
      "\n",
      "    accuracy                           0.77      5940\n",
      "   macro avg       0.70      0.63      0.65      5940\n",
      "weighted avg       0.76      0.77      0.77      5940\n",
      "\n",
      "\n",
      "-----Classifier: Naive Bayes\tEncoding: CatBoost-----\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.71      0.87      0.78      3253\n",
      "           1       0.25      0.10      0.14       424\n",
      "           2       0.76      0.61      0.67      2263\n",
      "\n",
      "    accuracy                           0.71      5940\n",
      "   macro avg       0.57      0.52      0.53      5940\n",
      "weighted avg       0.70      0.71      0.69      5940\n",
      "\n",
      "\n",
      "-----Classifier: Random Forest\tEncoding: ordinal-----\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.81      0.87      0.84      3253\n",
      "           1       0.55      0.38      0.45       424\n",
      "           2       0.82      0.78      0.80      2263\n",
      "\n",
      "    accuracy                           0.80      5940\n",
      "   macro avg       0.73      0.68      0.70      5940\n",
      "weighted avg       0.80      0.80      0.80      5940\n",
      "\n",
      "\n",
      "-----Classifier: XGBoost\tEncoding: backward_difference-----\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.78      0.91      0.84      3253\n",
      "           1       0.61      0.27      0.37       424\n",
      "           2       0.84      0.74      0.79      2263\n",
      "\n",
      "    accuracy                           0.80      5940\n",
      "   macro avg       0.75      0.64      0.67      5940\n",
      "weighted avg       0.79      0.80      0.79      5940\n",
      "\n",
      "\n",
      "-----Classifier: Support Vector Machine\tEncoding: count-----\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.68      0.87      0.76      3253\n",
      "           1       0.00      0.00      0.00       424\n",
      "           2       0.71      0.56      0.62      2263\n",
      "\n",
      "    accuracy                           0.69      5940\n",
      "   macro avg       0.46      0.47      0.46      5940\n",
      "weighted avg       0.64      0.69      0.65      5940\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\kochlis\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\sklearn\\metrics\\_classification.py:1327: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "C:\\Users\\kochlis\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\sklearn\\metrics\\_classification.py:1327: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "C:\\Users\\kochlis\\AppData\\Local\\Programs\\Python\\Python39\\lib\\site-packages\\sklearn\\metrics\\_classification.py:1327: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "import seaborn as sns\n",
    "\n",
    "scores = []\n",
    "pipelines = []\n",
    "\n",
    "for classifier_name, metric in classifier_best_results.items():\n",
    "    score, encoding_method = metric\n",
    "    encoder = encoding_methods[encoding_method]\n",
    "    encoded_x_train = encoder.fit_transform(x_train, y_train)\n",
    "    encoded_x_test = encoder.transform(x_test)\n",
    "\n",
    "    clf = classifiers[classifier_name]\n",
    "    clf.fit(encoded_x_train, y_train)\n",
    "    y_pred = clf.predict(encoded_x_test)\n",
    "\n",
    "    scores.append(score)\n",
    "    pipelines.append(classifier_name + ' - ' + encoding_method)\n",
    "\n",
    "    print(f'\\n-----Classifier: {classifier_name}\\tEncoding: {encoding_method}-----')\n",
    "    print(classification_report(y_test, y_pred))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "outputs": [
    {
     "data": {
      "text/plain": "Text(0, 0.5, 'Pipeline')"
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": "<Figure size 432x288 with 1 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAg8AAAEWCAYAAADhFHRsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA0O0lEQVR4nO3dd7wcVd3H8c83IUgLoQQFgRCMCIZ2IaF3RVFEQAEBQwn6gCjSFESFB2N7pKM0JSAEImAogkGlSSfUBNIREAhFkF4CBAjh9/xxzpJhuXt3J7l77yb5vl+vfWXmzJkzvzN7YX5z5uyuIgIzMzOzRvXo7gDMzMxs3uLkwczMzEpx8mBmZmalOHkwMzOzUpw8mJmZWSlOHszMzKwUJw9mZtYyJA2RdH13x2Edc/JgZjVJukXSK5I+1t2xdBZJQyWFpN27O5ZmkvRNSWMlvSHpWUnXSNq8u+OqJyIuiogvdncc1jEnD2bWLkn9gS2AAHZsQvsLdXabDdoXeBnYpysP2pX9lfQD4LfA/wGfAPoBZwE7dVUMc6Ib/yasJCcPZlbLPsDdwAjSBRdJH5P0qqS1KpUkLSdphqSP5/UdJI3P9e6UtE6h7jRJR0maCLwpaSFJP5b0qKTpkqZK+lqhfk9JJ0t6UdLjkr6fRw0Wytv7SPpjvrP+j6RfSepZq0OSVgG2Ag4AtpO0fNWxflqIZZyklfO2NSXdIOllSc9J+mkuHyHpV4U2tpb09Jz2N++zv6QHC9vXl3SkpCuq6p0m6Xft9LEP8AvgoIj4S0S8GREzI+LqiDiy8D7+VtIz+fXbyuhSpQ+SfiTp+Xxud5a0vaSH8zn4aeF4wyRdLmlUjvl+SesWtnf0/g6VNEbSqZJeAoblsjvyduVtz0t6XdKkyt9efu8vlPSCpCckHSOpR6HdOySdpDRy9rikL9f6u7A5EBF++eWXXx95Af8GvgcMAmYCn8jl5wG/LtQ7CLg2L68HPA9sBPQkJR3TgI/l7dOA8cDKwKK5bDfgk6Sbmd2BN4EV8rYDganASsDSwD9JIyEL5e1XAmcDiwMfB+4FvtNBn/4XuDcvTwJ+WNh2ZC5bHRCwLrAs0Bt4FvghsEhe3yjvMwL4VaGNrYGnC+tl+7sb8B9ggxzDp4FVgBVyvaVyvYXyeR7UTh+/BLxXOUc1zsMvSInhx4HlgDuBXxb68B5wLNAL2B94Abg4931NYAawaq4/LP997JrrHwE8DvRqoL9D87EOzn1aNJfdkbdvB4wDlsrn47OFfS8E/ppj6g88DHy70O7MHHtP4LvAM4C6+7+r+eXV7QH45ZdfrfcCNs//8+2b1/8FHJ6XtwUeLdQdA+yTl39fuQgVtj8EbJWXpwHfqnPs8cBOefkmCslAPnbkC80ngHfIF+W8fU/g5g7afgQ4LC//BJhQFedO7eyzJ/BAjfZGUD95KNPf64BDa9S7Btg/L+8ATK1Rbwjw3zrHfBTYvrC+HTCt0IcZQM+83juf840K9ccBO+flYcDdhW09SMnWFg30dyjwZNX2ocxOHj5HSgo2BnoU6vQE3gUGFsq+A9xSaOPfhW2L5T4s3xX//SwILz+2MLP27AtcHxEv5vWLcxnAzcBikjZSmhfRRhoBgHSX/MP8yOJVSa+S7ro/WWj7qeKBJO1TeMzxKrAW0Ddv/mRV/eLyKqQ73WcL+55Nupv+CEmbAasCfy70aW1JbXl9ZdJFtVqt8kaV6W9Hx7oA2Csv7wWMrFHvJaCvOp4/8EngicL6E3z4PXopImbl5Rn53+cK22cASxTWP+hjRLwPPF1pr05/P7RvtYi4CTgDOBN4XtJwSUvm/Xu104cVC+v/LbTzVl4sxmxzwcmDmX2IpEWBbwBbSfqvpP8ChwPrSlo3X1QuJd2R7wn8LSKm592fIj3SWKrwWiwiLikcIgrHWgU4B/g+sGxELAVMJg1RQ7qDXamw78qF5adIIw99C8daMiLWrNG1fXO743Of7imUV9ob0M5+TwGfqtHmm6S72orl26lTpr+1YgC4ClgnP/PfAbioRr27SOdl5xrbIQ3hr1JY75fL5tQH70ued7AS8EwD/YXC+WlPRJwWEYOAgcBnSI+XXiSNjFX34T9z0QcrwcmDmVXbGZhF+p91W359Frid2Z9QuJj0/HpIXq44Bzgwj0pI0uKSviKpd41jLU66eLwAIGk/0p1pxaXAoZJWlLQUcFRlQ0Q8C1wPnCxpSUk9JA2QtFX1QSQtQkqIDij0qY30rP2b+S79XOCXklbLsa8jaVngb8AKkg7LEw17S9ooNz0e2F7SMkqTLw+r0c9G+3sucISkQTmGT+cLMBHxNnA56XzfGxFPtneAiHiNNF/hzDzRcTFJvSR9WdIJudolwDFKk1375vp/qhN7RwZJ+no+j4eRkpe7G+hvhyRtkP+WepEStbeB9wsJ7K/z+7EK8IO57IOV4OTBzKrtC5wfEU9GxH8rL9Lw8RBJC0XEPaT/mX+S9CwegIgYS5qkdgbwCmnS5dBaB4qIqcDJpLvl54C1SXMoKs4hJQgTgQeAf5Am2FWG1PcBFiZNqnyFdHFdoZ1D7Uwaar+wqk/nkeZPfAk4hXRBuh54HfgjaT7FdOALwFdJQ+GPANvkdkcCE0hzG64HRtXqayP9jYjLgF+TEoTppNGGZQpNXJD3qfXIotLOyaSL6TGkC/dTpLv/q3KVXwFjSed1EnB/LptTfyUlk68AewNfj/QJj3rvbz1Lkv4GXiE9lngJODFvO5j0N/gYcAfpnJ03F32wEpQnk5iZtbz8cbs/RMQqdSvPhyT1I01eXT4iXu/ueCB9VBP4dETsVa+uzT888mBmLUvSokrfL7CQpBWBnzF7cuYCJc8l+AHw51ZJHGzB5W/zMrNWJuDnpMcBM4C/k57PL1AkLU4a9n+C9IjFrFv5sYWZmZmV4scWZmZmVoofW9h8r2/fvtG/f//uDsPMbJ4ybty4FyNiufa2OXmw+V7//v0ZO3Zsd4dhZjZPkfRErW1+bGFmZmalOHkwMzOzUvzYwuZ7Dz79EoOOvLC7wzCzFjTuxH3qV7KP8MiDmZmZleLkwczMzEpx8mBmZmalOHkwMzOzUpw8mJmZWSlOHszMzKwUJw9mZmZWipMHMzMzK8XJg5mZmZXi5MHMzMxKaYnkQdIbndDGYEmndbC9v6RvNlq/nf1vkfSQpAmS7pPUNpchdxpJO0r6cZOPsbWkv1WVjZC0a5396taZi5h+2ox2zcysYy2RPHSGiBgbEYd0UKU/8EHy0ED99gyJiHWBs4ATy0f5UZJ6zm0bETE6Io7rjHjmBUp6AE4ezMy6QcsmD5LaJN0taaKkKyUtncs3yGXjJZ0oaXIu/+DOWNJWeft4SQ9I6g0cB2yRyw6vqr+EpPMlTcpt71InvLuAFfO+i0s6T9K9+Vg75fLFJF0qaWqO/x5Jg/O2NySdLGkCsImkvfL+4yWdLalnfo2QNDnHdXje95Dc5kRJf85lQyWdkZf7S7opb79RUr9cPkLSaZLulPRYZ48GSBok6VZJ4yRdJ2mFdupMk/Sb3M+xktbPdR+VdGCh3pF5dGeipJ8X+vWQpAuBycAfgUVzWxd1Zl/MzKxjLZs8ABcCR0XEOsAk4Ge5/HzgOxHRBsyqse8RwEG5zhbADODHwO0R0RYRp1bV/1/gtYhYOx/vpjqxfQm4Ki8fDdwUERsC2wAnSloc+B7wSkQMzO0PKuy/OHBPHsV4Cdgd2KzQpyFAG7BiRKwVEWvnfpP7sV6O80A+6nTggrz9IqD4aGYFYHNgB1IyVVYl+RovaTywI4CkXvm4u0bEIOA84Nc12ngy9/N2YASwK7AxUEkSvgisBmxIOgeDJG2Z910NOCsi1oyI/YAZ+f0cUn0QSQfkBGXse29Nn4OumplZLS35k9yS+gBLRcStuegC4DJJSwG9I+KuXH4x6UJYbQxwSr4j/UtEPC2po0NuC+xRWYmIV2rUu0jSwsASpAsbwBeBHSUdkdcXAfqRLtK/y+1NljSx0M4s4Iq8/HlSYnFfjnFR4HngauBTkk4H/g5cn+tPzHFcxewEpmgT4Ot5eSRwQmHbVRHxPjBV0idq9LEjt0fEB+db0oi8uDqwFnBD7kNP4NkabYzO/04CloiI6cB0Se/k9/eL+fVArrcEKWl4EngiIu5uJNCIGA4MB1h8+VWjkX3MzKwxLZk8zK2IOE7S34HtgTGStuukpocA40jzHU4nXaQF7BIRDxUr1klW3o6IyqiJSCMFP6muJGldYDvSCMM3gG8BXwG2BL4KHC1p7RLxv1Nsvp3jHQTsn1e3j4hnGmxXwJSI2KREDO9XxfM+6e9RwG8i4uyq2PoDbzYYj5mZNVFLPraIiNeAVyRtkYv2Bm6NiFdJd6kb5fI92ttf0oCImBQRxwP3AWsA04HeNQ55A3BQYf+lO4gtSI8hNpa0BnAdcLBytiBpvVx1DOmCj6SBQK2L/I3ArpI+nusuI2kVSX2BHhFxBXAMsL7SJMGVI+Jm4CigD+nOvOhOZp+XIaTHAw2JiDPzY4C2EokDwEPAcpI2yX3oJWnNEvsXXQd8S9ISua0VK+emHTPzIxMzM+tCrTLysJikpwvrpwD7An+QtBjwGLBf3vZt4BxJ7wO3Aq+1095hkrYh3c1OAa7Jy7PyJMURzB4WB/gVcKbS5MtZpOfvf6kVbETMkHQycCTwfeC3wMR8cX+c9CjlLOACSVOBf+U4PhJrREyVdAxwfd5/JimRmQGcn8sAfkJ6HPCn/FhHwGkR8WrVKMfBeb8jgRcK561pIuLdPAHztBzbQqRzMmUO2rpe0meBu3K/3gD2ov35LcNJ5/3+9uY9mJlZcyjdSM87JC0REW/k5R8DK0TEod0c1kcofQSzV0S8LWkA8E9g9Yh4t5tDW+AsvvyqscbeP+/uMMysBY07cZ/uDqFlSRoXEYPb29YqIw9lfEXST0ixPwEM7d5waloMuDkPqwv4nhMHMzObH8xzyUNEjAJGdXcc9eRPEbSbsZmZmc3LWnLCpJmZmbUuJw9mZmZWipMHMzMzK8XJg5mZmZXi5MHMzMxKcfJgZmZmpTh5MDMzs1Lmue95MCvrsysty1h/i5yZWafxyIOZmZmV4uTBzMzMSnHyYGZmZqU4eTAzM7NSnDyYmZlZKU4ezMzMrBR/VNPme+8+O4Unf7F2d4dhZvOgfsdO6u4QWpJHHszMzKwUJw9mZmZWipMHMzMzK8XJg5mZmZXi5MHMzMxKcfJgZmZmpTh5MDMzs1KcPJiZmVkpTh7MzMysFCcPZmZmVoqTBzMzMyulacmDpKMlTZE0UdJ4SRs161gNxHKYpMXaKf+ZpN9UlbVJerBk+0tJ+l4nxDlN0u1VZeMlTZ7D9m6RNLid8sGSTpvTOFtFrffVzMyaqynJg6RNgB2A9SNiHWBb4KlmHKuBWHoChwHtXWQuAXavKtsjl5exFFAqeZBU60fJektaOdf5bMk4GhIRYyPikGa03cUOo/331czMmqhZIw8rAC9GxDsAEfFiRDwDH9xd983LgyXdkpeHSRop6S5Jj0jaP5dvLek2SX+X9JCkP0jqkbftKWmSpMmSjq8cXNIbkk6WNAE4GvgkcLOkm4tBRsTDwCtVoyLfAC6RNEDStZLGSbpd0hq57U9IulLShPzaFDgOGJBHCU5UcmKOa5Kk3Qt9uV3SaGBqjXN3KbMTmj0pJDKS+uf978+vTQvbjsrHmiDpuEJ7u0m6V9LDkrYoxPG3wnk/L49SPCbpkEKbe+V9x0s6OydiDZO0Tx55miBpZKEPN+XyGyX1y+UjJO1a2PeNQqy3SLpc0r8kXZTP7yHUeF/NzKy5mvWT3NcDx0p6GPgnMCoibm1gv3WAjYHFgQck/T2XbwgMBJ4ArgW+LulO4HhgEPAKcL2knSPiqrz/PRHxQwBJ3wK2iYgX2znmJaTRhnskbQy8HBGPSLoRODAvbwScBXwOOA24NSK+li+mSwA/BtaKiLZ8vF2ANmBdoC9wn6Tb8vHWz3Ufr3EOrgDOB04CvgoMAfbO254HvhARb0taLcc+WNKXgZ2AjSLiLUnLFNpbKCI2lLQ98DPSKFC1NYBtgN7AQ5J+D3yalMRsFhEzJZ2VY7mwRtwfImlN4Bhg04h4sRDT6cAFEXFBfl9OA3au09x6wJrAM8CYHNNpkn5AjfdV0gHAAQAr9unVSMhmZtagpow8RMQbpIv6AcALwChJQxvY9a8RMSNfDG4mJQ0A90bEYxExi3TB3BzYALglIl6IiPeAi4Atc/1ZpItwI0YBu+bRjD1Iow5LAJsCl0kaD5xNGk2BlED8PvdzVkS81k6bmwOX5O3PAbfmeCt9qZU4ALxEGg3ZA3gQeKuwrRdwjqRJwGWkhApSQnB+RLyV43q5sM9f8r/jgP41jvn3iHgnn/fngU8Anye9h/flc/B54FMdxF3tc8BllQt7IaZNgIvz8kjSuarn3oh4OiLeB8Z30I8PRMTwiBgcEYOXWbzUgImZmdXRrJEH8oX+FuCWfLHbFxgBvMfspGWR6t1qrNcqr+XtfPxG4nxK0uPAVsAupItbD+DVykhCJ3uzgTqjgDOBoVXlhwPPkUY0egBvN9DWO/nfWdR+v98pLFfqiTRC8JNaDecRmbPz6rERMbqBeGr54O8iJ3IL14nPzMy6SbMmTK6eh9Ur2kiPHACmke5oIV2si3aStIikZYGtgfty+YaSVs0Xld2BO4B7ga0k9c2PD/Yk3eG3ZzppSL6WS4BTgcfyHe7rwOOSdsv9kaR1c90bge/m8p6S+rTT/u3A7nn7cqQRkXs7OH61K4ETgOuqyvsAz+Y78L2Byi31DcB+yp88qHpsMaduJI3IfLzSpqRVihUi4p6IaMuv6sThJtJ8i2WrYrqTNMID6TFI5dMl05j9d7EjaZSlnnrvq5mZNUGzJkwuAVwgaaqkiaTh9WF528+B30kaS7qLLJpIelxxN/DLyiRLUhJxBmkY/3Hgyoh4ljTX4GZgAjAuIv5aI57hwLUdTKy7jPRMvfgpiyHAt5UmXU4hzSkAOBTYJo+mjAMGRsRLwJg8QfJE0sV/Yo7rJuBHEfHfGsf+iIiYHhHHR8S7VZvOAvbNMa1BHsWIiGuB0cDY/IjhiEaP1UEMU0lzFq7P7+ENzH5008j+U4BfA7fmeE/Jmw4mJToTSQnQobn8HFIyOIE0+tPICE2999XMzJpAEfWeAHQNScOANyLipKryrYEjImKHbgjL5gPrrLho/O07n+7uMMxsHtTv2EndHUK3kTQuIj7yXUHgb5g0MzOzklpm4llEDKtRfgtp4qWZmZm1AI88mJmZWSlOHszMzKwUJw9mZmZWipMHMzMzK8XJg5mZmZXi5MHMzMxKaZmPapo1y8IrrEm/Y8d2dxhmZvMNjzyYmZlZKU4ezMzMrBQnD2ZmZlaKkwczMzMrxcmDmZmZleLkwczMzErxRzVtvvev5//FZqdv1t1hmNl8bszBY7o7hC7jkQczMzMrxcmDmZmZleLkwczMzEpx8mBmZmalOHkwMzOzUpw8mJmZWSlOHszMzKwUJw9mZmZWipMHMzMzK8XJg5mZmZXi5MHMzMxKWaCTB0kh6eTC+hGShtXZZ0dJP+6EYw+V9IKk8ZKmSLpc0mJz225nkrShpNskPSTpAUnndhSjpDZJ2xfWm9rH6uOZmVnXWKCTB+Ad4OuS+ja6Q0SMjojjOun4oyKiLSLWBN4Fdu+kdueapE8AlwFHRcTqEbEecC3Qu4Pd2oDqi3kz+9je8czMrMkW9OThPWA4cHj1BklflXRPvuP+Z76YVu6mz5DUR9ITknrk8sUlPSWpl6QBkq6VNE7S7ZLW6CgISQsBiwOv1Dq2pB6SHpG0XK7TQ9K/JS2XX1dIui+/Nst1tsp3/eNzWx1d+KsdBFwQEXdVCiLi8oh4Lo9I3JXbvFPS6pIWBn4B7J6P96EkoZ0+9pd0k6SJkm6U1K9O+W6SJkuakEdDOjyemZk1z4KePACcCQyR1Keq/A5g43zH/WfgR8WNEfEaMB7YKhftAFwXETNJCcnBETEIOAI4q8axd5c0HvgPsAxwda1jR8T7wJ+AIbnOtsCEiHgB+B1wakRsAOwCnJvrHAEcFBFtwBbAjEZOSLYWMK7Gtn8BW+T4jgX+LyLezcuVkYZRdfp4Oik5WQe4CDitTvmxwHYRsS6wYwfHA0DSAZLGSho7842ZJbptZmb1LPDJQ0S8DlwIHFK1aSXgOkmTgCOBNdvZfRSzh+H3AEZJWgLYFLgsXzTPBlaocfhR+cK+PFA5TkfHPg/YJy9/Czg/L28LnJGPNxpYMscxBjhF0iHAUhHxXu0zUUofUv8mA6fS/rmpqNXHTYCL8/JIYPM65WOAEZL2B3rWCzAihkfE4IgY3GuJXg11yszMGtNQ8iBpMUn/K+mcvL6apB2aG1qX+i3wbdKwesXpwBkRsTbwHWCRdvYbDXxJ0jLAIOAm0jl9Nd8NV16f7ejgERGkO/ItOzp2RDwFPCfpc8CGwDW5fg/SSEXleCtGxBt5bsb/AIsCY6ofn0g6qPBY45NVYU3JfWrPL4GbI2It4Ks1zk29PpYSEQcCxwArA+MkLTsn7ZiZ2dxrdOThfNLkwk3y+n+AXzUlom4QES8Dl5ISiIo+pH4C7FtjvzeA+0iPDf4WEbPySMbjknYDULJuA2FsDjzawLHPJT2+uCwiZuWy64GDKxUkteV/B0TEpIg4Psf5oeQhIs4sJBzPVB3nDGBfSRsV2v16nvtRjG9oYZ/pdDyhstjHO0mjNZAexdzeUXnuyz0RcSzwAimJqHc8MzNrgkaThwERcQIwEyAi3gLUtKi6x8lA8VMXw0hD8+OAFzvYbxSwV/63YgjwbUkTSHfwO9XYtzLZbyKwHumOvt6xRwNLMPuRBaRHLoPzJMOpwIG5/LA8yXAi6b27hgZFxHOki/hJSh/VfBDYjnTBPgH4jaQHgIUKu90MDKyawFirjwcD++XyvYFD65SfKGlSflRyJzChxvHMzKzJlEaT61SS7gQ+D4yJiPUlDQAuiYgNmx2gfZikwaTJkVt0dyzziiX6LRHrHtnI4I+Z2Zwbc/CY7g6hU0kaFxGD29u2UHuF7fgZ6TP+K0u6CNiMDw9XWxdQ+nKq7zL7ExdmZmZdrqHkISJukHQ/sDHpccWhEdHRUL41QZ4A2VlfUGVmZjZHGh15gDSj/pW8z0BJRMRtzQnLzMzMWlVDyYOk40nfZzAFeD8XB+DkwczMbAHT6MjDzsDqEfFOE2MxMzOzeUCjH9V8DPDX9JmZmVnDIw9vAeMl3Uj6sigAIqL6K53NzMxsPtdo8jA6v8zMzGwB19CXRJnNywYPHhxjx47t7jDMzOYpc/wlUZIujYhv5F93/EiWkX822czMzBYg9R5bVH5XYH76BU0zMzObCx0mDxHxbP73ia4Jx8zMzFpdvccW05n9uKLyK5qRlyMilmxibGZmZtaC6o089O6qQMzMzGze0OiXRCFpc0n75eW+klZtXlhmZmbWqhpKHiT9DDgK+EkuWhj4U7OCMjMzs9bV6JdEfQ1YD7gfICKekeRHGjZPmP7QQ9y65VbdHYaZ2VzZ6rZbuzuEDzT62OLdSN8mFQCSFm9eSGZmZtbKGk0eLpV0NrCUpP2BfwLnNC8sMzMza1UNPbaIiJMkfQF4HfgMcGxE3NDUyMzMzKwlNTrnAWASsCjp0cWk5oRjZmZmra7RT1v8D3Av8HVgV+BuSd9qZmBmZmbWmhodeTgSWC8iXgKQtCxwJ3BeswIzMzOz1tTohMmXgOmF9em5zMzMzBYwjY48/Bu4R9JfSXMedgImSvoBQESc0qT4zMzMrMU0mjw8ml8Vf83/+ouizMzMFjCNflTz580OxMzMzOYNHc55kPTb/O/VkkZXv5odnKQ3CsvbS3pY0ipVdaZJuqKwvqukEc2OrT2SftrBttJxShos6bQ6dfpLmlxj2y2SBtcJe57V0fk2M7PmqTfyMDL/e1KzA+mIpM8DpwHbRcQT7VQZJGlgREztxGMuFBHvldztp8D/dbC9VJwRMRYYWzKGTjGH/e9q9c63mZk1Qb1PW0yRdBiwG7AGMCYibq28mh4dIGlL0ldh7xARj9aodjJwdDv7Li7pPEn3SnpA0k65vL+k2yXdn1+b5vKtc/loYKqknpJOlHSfpImSvpPrrSDpNknjJU2WtIWk44BFc9lFnRTn1pL+lpeXk3SDpCmSzpX0hKS+uYmeks7J266XtGih+b0LcW6Y21pG0lW5T3dLWieXD5M0UtIYYKSkNXNM43Pd1Wq+Ue2QdJSkSZIm5PODpLZ8zImSrpS0dC7/YJQk/+T7tLw8VNJfJF0r6RFJJ+TyRs63mZk1Qb3k4QJgMOkbJb9Muvh1pY8BVwE7R8S/Oqh3KbC+pE9XlR8N3BQRGwLbACfmH/V6HvhCRKwP7E4a1ahYHzg0Ij4DfBt4LSI2ADYA9pe0KvBN4LqIaAPWBcZHxI+BGRHRFhFDOinOop/lOmsClwP9CttWA87M214FdilsWyzH+T1mfy/Hz4EHImId0t37hYX6A4FtI2JP4EDgd3n/wcDTNfr1EZK+TPpUzkYRsS5wQt50IXBUPvak3K962kjv09rA7pJWrne+JR0gaayksa/NnNlo2GZm1oB6jy0GRsTaAJL+SPqWya40k/RlVN8GDu2g3izgROAnwDWF8i8CO0o6Iq8vQrroPgOcIakt7/uZwj73RsTjhf3XkbRrXu9DulDfB5wnqRdwVUSMb7A/ZeMs2pz00+hExLWSXilse7wQwzigf2HbJXmf2yQtKWmp3NYuufwmSctKWjLXHx0RM/LyXcDRklYC/hIRjzTYT4BtgfMj4q18nJcl9QGWKoxaXQBc1kBbN0bEawCSpgKrAE91tENEDAeGA6zeu3eUiNvMzOqoN/LwwS1bNz3/fh/4BrChpJ/mxwjj8+sXVXVHAlsCKxfKBOyS707bIqJfRDwIHA48Rxo1GAwsXNjnzar9Dy7sv2pEXB8Rt+Vj/QcYIWmfEn0qE2ej3iksz+LDSWH1hbPehfSD/kfExcCOwAzgH5I+V6wo6WuF92NuJ2a+x+y/x0WqtnXUPzMz62L1kod1Jb2eX9NJd+GvS5ou6fWuCDDfuX4FGAIMLVxgj62qNxM4lZQYVFwHHCxJAJLWy+V9gGcj4n1gb6BnjcNfB3w3jzAg6TN5fsIqwHMRcQ5wLulRB8DMSt0O+lMmzqIxpEQKSV8Elu7oOAW75302Jz2CeQ24nXQ+kbQ18GJEfOT9lPQp4LGIOI303R7rVPXlysL7UT2x8wZgP0mL5baWycd+RdIWuc7eQGUUYhowKC/vSmPqnm8zM+t8HSYPEdEzIpbMr94RsVBhecmO9u1MEfEy8CXgGEk7dlD1j3z4rvSXQC/St2FOyesAZwH7SppAmgj6Ju07F5gK3K/0ccizc/tbAxMkPUC6OP8u1x+ej1VvAl+jcRb9HPhijmM34L98+CvDa3k7x/kH0uMfgGGkT35MBI4D9q2x7zeAyZLGA2vx4bkRHYqIa4HRwNi8f+WRzL6kOR0TSXMZKiNIJ5EStQeAvjSm0fNtZmadSBF+HDwvkPQxYFZEvCdpE+D3eSKj1bF6794xfL3161c0M2thW93WJR9y/ICkcRHR7iNpPzued/QDLpXUA3gX2L+b4zEzswWUk4d5RP6kQ3tzIczMzLpUoz/JbWZmZgY4eTAzM7OSnDyYmZlZKU4ezMzMrBQnD2ZmZlaKkwczMzMrxcmDmZmZleLvebD5Xu/VV+/yb2YzM5ufeeTBzMzMSnHyYGZmZqU4eTAzM7NSnDyYmZlZKU4ezMzMrBQnD2ZmZlaKP6pp873nn36NM354dXeHYWbWJb5/8lebfgyPPJiZmVkpTh7MzMysFCcPZmZmVoqTBzMzMyvFyYOZmZmV4uTBzMzMSnHyYGZmZqU4eTAzM7NSnDyYmZlZKU4ezMzMrBQnD2ZmZlZK05IHSStLelzSMnl96bzeP6+vJulvkh6VNE7SzZK2zNuGSnpB0nhJUyRdLmmxToytTdL2c7DfCEm7dsLxb5E0eG7bafBY/SVNLlH/g9gk/UPSUnn5EEkPSrpI0sck/TO/P7s3KXQzM2tRTUseIuIp4PfAcbnoOGB4REyTtAjw97w+ICIGAQcDnyo0MSoi2iJiTeBdoDMvUm1A6eRhXiCp037sLCK2j4hX8+r3gC9ExBBgvby9LSJGdXVcZmbWvZr92OJUYGNJhwGbAyfl8iHAXRExulIxIiZHxIjqBvJFZ3HglbzeX9JNkiZKulFSvzrlu0maLGmCpNskLQz8Ath9Du+ct5U0VtLDknYoHPt2Sffn16aF+I+SNCkf/7hiQ5J65NGMX+U4T8nlh0p6LC9/StKYvHyspPtyf4ZLUi6/RdJvJY0FDpU0KB9vAnBQR52RtKikP+dRhSuBRQvbpknqK+kPpMTuGklHAX8CNsjnb0A+3q15BOk6SSt0EFetesdLujef1y1yeU9JJ+X+TpR0cC5vt52qfh2Q36exb7z1WuPvrpmZ1dXUu8GImCnpSOBa4IsRMTNvWhO4v87uu0vaHFgBeBio/Kby6cAFEXGBpG8BpwE7d1B+LLBdRPxH0lIR8a6kY4HBEfH9OehWf2BDYABws6RPA8+T7srflrQacAkwWNKXgZ2AjSLiLeVHONlCwEXA5Ij4taTlgR/lbVsAL0laMS/flsvPiIhfAEgaCexQOC8LR0TlccNE4PsRcZukE+v057vAWxHxWUnr0M77EhEHSvoSsE1EvCjpHuCIiNhBUi9gJLBTRLyQk7FfA98qxpXr3dpBvYUiYkOlx0k/A7YFDsjnuy0i3pO0TG7n9A7aqcQ8HBgO0G/51aLOOTAzsxK6YsLkl4FngbVqVZB0Zb67/EuheFREtAHLA5OAI3P5JsDFeXkkaUSjo/IxwAhJ+wM9564rAFwaEe9HxCPAY8AaQC/gHEmTgMuAgbnutsD5EfEWQES8XGjnbHLikLf9F1hCUm9g5dyXLUnJw+15n20k3ZOP8zlSElYxCkBpjsJSEVFJOEbW6c+WpJEEImIiMLHRE5GtTnpvb5A0HjgGWKk6rgbqVd77caSEAdL5Ozsi3svxvdxAO2Zm1mRNHXmQ1AZ8AdgYuEPSnyPiWWAK6aIFQER8TWmS3knVbURESLqaNCfiuOrt9eS75o2ArwDjJA2qE/P5pGf6z0REe/Miqu9iAzgceA5Yl5SQvd1AaHeSkoGTI+LtQtl+wEOkhOFbpKToh0rzRM4ijZg8JWkYsEihvTcbOGYzCJgSEZvU2P5mg/Xeyf/OouO/y3rtmJlZkzXz0xYiTZg8LCKeBE5kdnJwMbCZpB0Lu3T0aYrNgUfz8p3AHnl5CLPvytstlzQgIu6JiGOBF0h39dOB3u0dKCL2yxMBa02o3C3PVRhAmgfwENAHeDYi3gf2ZvYIxw3AfsqfFKl6bPFH4B/ApZo9mfB24AjSY4oHgG2AdyLiNWYnCi9KWgJo91MfeYLjq/mRT+VcdOQ24Js5vrWAderUr/YQsJykTXIbvSStORf1im4AvlM5P/n8zUk7ZmbWiZr52GJ/4MmIuCGvnwV8VtJWETGD9Lz+QEmPSbqLNPz8q8L+lQmNE0kjAb/M5QeTLsgTSRfqQ+uUn6g0YXEyKcGYANwMDNScTZh8ErgXuAY4MI8anAXsqzRBcQ3y3XZEXAuMBsbmIfYjig1FxCmkJGGkpB6k5GFl4LaImAU8BdyR674KnANMBq4D7usgxv2AM/MxVac/vyc9LnmQNJF0XN0z8OE+vEtKZI7P/R8PbDqn9aqcSzrfE/M+35zDdszMrBMpwnPJbP7Wb/nV4kdDTunuMMzMusT3T/5qp7QjaVxlIn41f8OkmZmZleIv7llASNoOOL6q+PGI+Fp3xGNmZvMuJw8LiIi4jjRXwszMbK74sYWZmZmV4uTBzMzMSnHyYGZmZqU4eTAzM7NSnDyYmZlZKf60hc33Pr5Sn0770hQzM/PIg5mZmZXk5MHMzMxKcfJgZmZmpTh5MDMzs1KcPJiZmVkpTh7MzMysFH9U0+Z7zz7+KL/ea9fuDsPMrEsd/afLm9a2Rx7MzMysFCcPZmZmVoqTBzMzMyvFyYOZmZmV4uTBzMzMSnHyYGZmZqU4eTAzM7NSnDyYmZlZKU4ezMzMrBQnD2ZmZlaKkwczMzMrZYFLHiTNkjRe0mRJV0taqpPaHSrpjM5oq6rdWyQ9lGMeL6kpP9Igqb+kbzaj7RrHmyapb16+cy7aGdGsc2JmZu1b4JIHYEZEtEXEWsDLwEHdHVADhuSY2yKioV86kVT2R8/6A01JHurFEhGbNuO4ZmbWHAti8lB0F7AigKQNJd0l6QFJd0paPZcPlfQXSddKekTSCZWdJe0n6WFJ9wKbFcr7S7pJ0kRJN0rql8tHSPq9pLslPSZpa0nnSXpQ0ohGg5a0jKSrcvt3S1onlw+TNFLSGGCkpOUkXSHpvvzaLNfbqjCS8YCk3sBxwBa57PAmxLKspOslTZF0LqBCG2/kf7fOIy2XS/qXpIskKW87NvdhsqThlXIzM+t6C2zyIKkn8HlgdC76F7BFRKwHHAv8X6F6G7A7sDawu6SVJa0A/JyUNGwODCzUPx24ICLWAS4CTitsWxrYBDg8H/tUYE1gbUltNcK9qHCxXzYf94Hc/k+BCwt1BwLbRsSewO+AUyNiA2AX4Nxc5wjgoIhoA7YAZgA/Bm7Poxun1jxxH9VoLD8D7oiINYErgX412lsPOCzv+ylmJ2VnRMQGecRoUWCHjoKSdICksZLGvvn2OyW6Y2Zm9ZQd2p4fLCppPGnE4UHghlzeB7hA0mpAAL0K+9wYEa8BSJoKrAL0BW6JiBdy+SjgM7n+JsDX8/JI4IRCW1dHREiaBDwXEZPy/lNIjw7GtxPzkIgYW1mRtDkpGSAibsp39UvmzaMjYkZe3hYYWLhJX1LSEsAY4BRJFwF/iYin5+JGvtFYtiSfk4j4u6RXarR3b0Q8nfs5nnRO7gC2kfQjYDFgGWAKcHWtoCJiODAcYMVll4457ZyZmX3UgjjyMCPfca9CGjqvzHn4JXBzvrP9KrBIYZ/iress5i7pqrT1flW7789luxVvFpZ7ABsX5kusGBFvRMRxwP+Q7uDHSFqjowYl/boy8jEXsTTqI+da0iLAWcCuEbE2cA4ffn/MzKwLLYjJAwAR8RZwCPDDPKGvD/CfvHloA03cA2yV77R7AbsVtt0J7JGXhwC3d0rQs92e20XS1sCLEfF6O/WuBw6urFQei0gaEBGTIuJ44D5gDWA60Lu9g0XE0ZUEZC5iuY08IVPSl0mPbxpVSRRezCMn/nSFmVk3WmCTB4CIeACYCOxJerTwG0kP0MAIQEQ8CwwjTbocQ3oEUnEwsJ+kicDewKGdGznDgEG5/eOAfWvUOwQYnCczTgUOzOWH5YmHE4GZwDWk8zBL0oQyEyZLxPJzYMv8eObrwJONHiAiXiWNNkwGriMlPGZm1k0U4cfBNn9bcdml43tf/nx3h2Fm1qWO/lNDn+yvSdK4iBjc3rYFeuTBzMzMynPyYGZmZqU4eTAzM7NSnDyYmZlZKU4ezMzMrBQnD2ZmZlaKkwczMzMrxcmDmZmZlbIg/jCWLWBWWHXAXH9ZipmZzeaRBzMzMyvFyYOZmZmV4uTBzMzMSvEPY9l8T9J04KHujqMDfYEXuzuIDrRyfK0cGzi+udHKscGCEd8qEbFcexs8YdIWBA/V+mW4ViBprOObM60cGzi+udHKsYHj82MLMzMzK8XJg5mZmZXi5MEWBMO7O4A6HN+ca+XYwPHNjVaODRbw+Dxh0szMzErxyIOZmZmV4uTBzMzMSnHyYPMNSV+S9JCkf0v6cTvbPyZpVN5+j6T+LRbflpLul/SepF1bLLYfSJoqaaKkGyWt0mLxHShpkqTxku6QNLCV4ivU20VSSOqyj/g1cO6GSnohn7vxkv6nq2JrJL5c5xv572+KpItbKT5JpxbO3cOSXm2h2PpJulnSA/m/3e077eAR4Zdf8/wL6Ak8CnwKWBiYAAysqvM94A95eQ9gVIvF1x9YB7gQ2LXFYtsGWCwvf7cFz92SheUdgWtbKb5crzdwG3A3MLhVYgOGAmd01fmag/hWAx4Als7rH2+l+KrqHwyc1yqxkSZNfjcvDwSmddbxPfJg84sNgX9HxGMR8S7wZ2Cnqjo7ARfk5cuBz0tSq8QXEdMiYiLwfhfFVCa2myPirbx6N7BSi8X3emF1caArZ4I38rcH8EvgeODtFoytuzQS3/7AmRHxCkBEPN9i8RXtCVzSJZE1FlsAS+blPsAznXVwJw82v1gReKqw/nQua7dORLwHvAYs2yXRNRZfdykb27eBa5oa0Yc1FJ+kgyQ9CpwAHNJFsUED8UlaH1g5Iv7ehXFB4+/tLnlY+3JJK3dNaEBj8X0G+IykMZLulvSlLouuxH8b+VHeqsBNXRAXNBbbMGAvSU8D/yCNjHQKJw9m1jBJewGDgRO7O5ZqEXFmRAwAjgKO6e54KiT1AE4BftjdsdRwNdA/ItYBbmD26FyrWIj06GJr0p39OZKW6s6AatgDuDwiZnV3IAV7AiMiYiVge2Bk/nuca04ebH7xH6B4x7RSLmu3jqSFSMN4L3VJdI3F110aik3StsDRwI4R8U4XxQblz92fgZ2bGVCVevH1BtYCbpE0DdgYGN1FkybrnruIeKnwfp4LDOqCuCoaeW+fBkZHxMyIeBx4mJRMtEp8FXvQdY8soLHYvg1cChARdwGLkH4wa645ebD5xX3AapJWlbQw6T/k0VV1RgP75uVdgZsizyRqkfi6S93YJK0HnE1KHLrymXOj8RUvJl8BHmmV+CLitYjoGxH9I6I/ac7IjhExtrtjA5C0QmF1R+DBLoir4fiAq0ijDkjqS3qM8VgLxYekNYClgbu6KK5GY3sS+HyO8bOk5OGFTjl6V81a9cuvZr9Iw3IPk2YgH53LfkH6HzX5P5zLgH8D9wKfarH4NiDdZb1JGhGZ0kKx/RN4DhifX6Nb7Nz9DpiSY7sZWLOV4quqewtd9GmLBs/db/K5m5DP3RqtdO4AkR77TAUmAXu0Unx5fRhwXFfG1eC5GwiMye/teOCLnXVsfz21mZmZleLHFmZmZlaKkwczMzMrxcmDmZmZleLkwczMzEpx8mBmZmalOHkwM2uQpJ3zr2Ku0d2xmHUnJw9mZo3bE7gj/9sUkno2q22zzuLkwcysAZKWADYnfeXvHrmsp6STJE3OPyx1cC7fQNKdkiZIuldSb0lDJZ1RaO9vkrbOy29IOlnSBGATScdKui+3O7zy66+SPi3pn7nd+yUNkHShpJ0L7V4kqZV+OdPmQ04ezMwasxNwbUQ8DLwkaRBwANAfaIv0w1IX5a8KHgUcGhHrAtsCM+q0vThwT0SsGxF3AGdExAYRsRawKLBDrncR6eep1wU2BZ4F/ggMBZDUJ5d39a932gLGyYOZWWP2JP3oFvnfPUmJwdmRfuKdiHgZWB14NiLuy2WvV7Z3YBZwRWF9G0n3SJoEfA5YU1JvYMWIuDK3+3ZEvBURt5J+42C5HNMVDRzPbK4s1N0BmJm1OknLkC7ia0sKoCcQpB8natR7fPiGbZHC8tuRf8pZ0iLAWaTfv3hK0rCquu25ENiL9DhlvxIxmc0RjzyYmdW3KzAyIlaJ9OuYKwOPk35w6Dv5J94rScZDwAqSNshlvfP2aUCbpB6SVgY2rHGsSqLwYp5nsStAREwHnq7Mb5D0MUmL5bojgMNyvamd1muzGpw8mJnVtydwZVXZFcAKpJ89npgnO34zIt4FdgdOz2U3kBKCMaSEYypwGnB/eweKiFeBc4DJwHV8eHRjb+AQSROBO4Hl8z7PkX5K+/y57ahZI/yrmmZm87g8AjEJWD8iXuvueGz+55EHM7N5mKRtSaMOpztxsK7ikQczMzMrxSMPZmZmVoqTBzMzMyvFyYOZmZmV4uTBzMzMSnHyYGZmZqX8P8v0jtHbJzlzAAAAAElFTkSuQmCC\n"
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "scores, pipelines = zip(*sorted(zip(scores, pipelines)))\n",
    "scores = list(scores)\n",
    "pipelines = list(pipelines)\n",
    "\n",
    "ax = sns.barplot(y=pipelines, x=scores)\n",
    "ax.set_title('Average Accuracy Comparison')\n",
    "ax.set_xlabel('Accuracy')\n",
    "ax.set_ylabel('Pipeline')"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Hyper-Parameter Optimization - K-Nearest Neighbors"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001B[32m[I 2022-07-13 13:19:57,878]\u001B[0m A new study created in memory with name: no-name-7b8c2f87-1199-41fc-b383-18ef2da0cc20\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:20:00,413]\u001B[0m Trial 0 finished with value: 0.6994444444444445 and parameters: {'n_neighbors': 101, 'weights': 'uniform'}. Best is trial 0 with value: 0.6994444444444445.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:20:04,188]\u001B[0m Trial 1 finished with value: 0.766952861952862 and parameters: {'n_neighbors': 169, 'weights': 'distance'}. Best is trial 1 with value: 0.766952861952862.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:20:07,578]\u001B[0m Trial 2 finished with value: 0.6944949494949495 and parameters: {'n_neighbors': 120, 'weights': 'uniform'}. Best is trial 1 with value: 0.766952861952862.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:20:10,640]\u001B[0m Trial 3 finished with value: 0.7247811447811449 and parameters: {'n_neighbors': 40, 'weights': 'uniform'}. Best is trial 1 with value: 0.766952861952862.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:20:13,345]\u001B[0m Trial 4 finished with value: 0.7438552188552189 and parameters: {'n_neighbors': 16, 'weights': 'uniform'}. Best is trial 1 with value: 0.766952861952862.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:20:16,941]\u001B[0m Trial 5 finished with value: 0.7101178451178451 and parameters: {'n_neighbors': 76, 'weights': 'uniform'}. Best is trial 1 with value: 0.766952861952862.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:20:19,698]\u001B[0m Trial 6 finished with value: 0.7089562289562289 and parameters: {'n_neighbors': 78, 'weights': 'uniform'}. Best is trial 1 with value: 0.766952861952862.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:20:23,161]\u001B[0m Trial 7 finished with value: 0.7669865319865321 and parameters: {'n_neighbors': 176, 'weights': 'distance'}. Best is trial 7 with value: 0.7669865319865321.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:20:26,182]\u001B[0m Trial 8 finished with value: 0.7187373737373738 and parameters: {'n_neighbors': 50, 'weights': 'uniform'}. Best is trial 7 with value: 0.7669865319865321.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:20:29,829]\u001B[0m Trial 9 finished with value: 0.6779124579124579 and parameters: {'n_neighbors': 210, 'weights': 'uniform'}. Best is trial 7 with value: 0.7669865319865321.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:20:32,961]\u001B[0m Trial 10 finished with value: 0.7667676767676768 and parameters: {'n_neighbors': 171, 'weights': 'distance'}. Best is trial 7 with value: 0.7669865319865321.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:20:37,017]\u001B[0m Trial 11 finished with value: 0.7668686868686868 and parameters: {'n_neighbors': 164, 'weights': 'distance'}. Best is trial 7 with value: 0.7669865319865321.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:20:40,799]\u001B[0m Trial 12 finished with value: 0.7666666666666667 and parameters: {'n_neighbors': 243, 'weights': 'distance'}. Best is trial 7 with value: 0.7669865319865321.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:20:43,734]\u001B[0m Trial 13 finished with value: 0.7669528619528619 and parameters: {'n_neighbors': 163, 'weights': 'distance'}. Best is trial 7 with value: 0.7669865319865321.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:20:46,904]\u001B[0m Trial 14 finished with value: 0.7668686868686868 and parameters: {'n_neighbors': 200, 'weights': 'distance'}. Best is trial 7 with value: 0.7669865319865321.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:20:49,818]\u001B[0m Trial 15 finished with value: 0.767037037037037 and parameters: {'n_neighbors': 154, 'weights': 'distance'}. Best is trial 15 with value: 0.767037037037037.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:20:52,684]\u001B[0m Trial 16 finished with value: 0.7670538720538721 and parameters: {'n_neighbors': 138, 'weights': 'distance'}. Best is trial 16 with value: 0.7670538720538721.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:20:55,517]\u001B[0m Trial 17 finished with value: 0.7673569023569022 and parameters: {'n_neighbors': 123, 'weights': 'distance'}. Best is trial 17 with value: 0.7673569023569022.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:20:58,329]\u001B[0m Trial 18 finished with value: 0.7672222222222221 and parameters: {'n_neighbors': 130, 'weights': 'distance'}. Best is trial 17 with value: 0.7673569023569022.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:01,083]\u001B[0m Trial 19 finished with value: 0.7677272727272728 and parameters: {'n_neighbors': 113, 'weights': 'distance'}. Best is trial 19 with value: 0.7677272727272728.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:03,806]\u001B[0m Trial 20 finished with value: 0.7684175084175084 and parameters: {'n_neighbors': 103, 'weights': 'distance'}. Best is trial 20 with value: 0.7684175084175084.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:06,436]\u001B[0m Trial 21 finished with value: 0.7688383838383838 and parameters: {'n_neighbors': 93, 'weights': 'distance'}. Best is trial 21 with value: 0.7688383838383838.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:09,035]\u001B[0m Trial 22 finished with value: 0.7685521885521885 and parameters: {'n_neighbors': 87, 'weights': 'distance'}. Best is trial 21 with value: 0.7688383838383838.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:11,621]\u001B[0m Trial 23 finished with value: 0.7691919191919192 and parameters: {'n_neighbors': 79, 'weights': 'distance'}. Best is trial 23 with value: 0.7691919191919192.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:14,593]\u001B[0m Trial 24 finished with value: 0.7691919191919192 and parameters: {'n_neighbors': 79, 'weights': 'distance'}. Best is trial 23 with value: 0.7691919191919192.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:17,093]\u001B[0m Trial 25 finished with value: 0.7702861952861954 and parameters: {'n_neighbors': 54, 'weights': 'distance'}. Best is trial 25 with value: 0.7702861952861954.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:19,579]\u001B[0m Trial 26 finished with value: 0.7702356902356902 and parameters: {'n_neighbors': 53, 'weights': 'distance'}. Best is trial 25 with value: 0.7702861952861954.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:21,984]\u001B[0m Trial 27 finished with value: 0.7702693602693602 and parameters: {'n_neighbors': 41, 'weights': 'distance'}. Best is trial 25 with value: 0.7702861952861954.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:24,261]\u001B[0m Trial 28 finished with value: 0.7651851851851852 and parameters: {'n_neighbors': 7, 'weights': 'distance'}. Best is trial 25 with value: 0.7702861952861954.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:26,741]\u001B[0m Trial 29 finished with value: 0.7702693602693602 and parameters: {'n_neighbors': 47, 'weights': 'distance'}. Best is trial 25 with value: 0.7702861952861954.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:29,132]\u001B[0m Trial 30 finished with value: 0.769983164983165 and parameters: {'n_neighbors': 31, 'weights': 'distance'}. Best is trial 25 with value: 0.7702861952861954.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:31,587]\u001B[0m Trial 31 finished with value: 0.7702356902356902 and parameters: {'n_neighbors': 53, 'weights': 'distance'}. Best is trial 25 with value: 0.7702861952861954.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:34,088]\u001B[0m Trial 32 finished with value: 0.7699999999999999 and parameters: {'n_neighbors': 62, 'weights': 'distance'}. Best is trial 25 with value: 0.7702861952861954.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:36,492]\u001B[0m Trial 33 finished with value: 0.769983164983165 and parameters: {'n_neighbors': 31, 'weights': 'distance'}. Best is trial 25 with value: 0.7702861952861954.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:39,011]\u001B[0m Trial 34 finished with value: 0.769814814814815 and parameters: {'n_neighbors': 63, 'weights': 'distance'}. Best is trial 25 with value: 0.7702861952861954.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:41,343]\u001B[0m Trial 35 finished with value: 0.7705892255892255 and parameters: {'n_neighbors': 23, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:43,703]\u001B[0m Trial 36 finished with value: 0.7703030303030304 and parameters: {'n_neighbors': 21, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:46,178]\u001B[0m Trial 37 finished with value: 0.735 and parameters: {'n_neighbors': 26, 'weights': 'uniform'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:48,524]\u001B[0m Trial 38 finished with value: 0.7696969696969698 and parameters: {'n_neighbors': 16, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:50,960]\u001B[0m Trial 39 finished with value: 0.7414814814814815 and parameters: {'n_neighbors': 19, 'weights': 'uniform'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:53,384]\u001B[0m Trial 40 finished with value: 0.7701346801346801 and parameters: {'n_neighbors': 40, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:55,648]\u001B[0m Trial 41 finished with value: 0.7612289562289563 and parameters: {'n_neighbors': 5, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:21:58,180]\u001B[0m Trial 42 finished with value: 0.7702693602693602 and parameters: {'n_neighbors': 41, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:00,766]\u001B[0m Trial 43 finished with value: 0.7695622895622896 and parameters: {'n_neighbors': 64, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:03,405]\u001B[0m Trial 44 finished with value: 0.725117845117845 and parameters: {'n_neighbors': 38, 'weights': 'uniform'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:06,123]\u001B[0m Trial 45 finished with value: 0.7703030303030304 and parameters: {'n_neighbors': 21, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:08,575]\u001B[0m Trial 46 finished with value: 0.7705723905723906 and parameters: {'n_neighbors': 22, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:11,263]\u001B[0m Trial 47 finished with value: 0.73493265993266 and parameters: {'n_neighbors': 25, 'weights': 'uniform'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:14,063]\u001B[0m Trial 48 finished with value: 0.7683501683501683 and parameters: {'n_neighbors': 13, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:16,469]\u001B[0m Trial 49 finished with value: 0.7703030303030304 and parameters: {'n_neighbors': 21, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:18,933]\u001B[0m Trial 50 finished with value: 0.7703030303030304 and parameters: {'n_neighbors': 21, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:21,479]\u001B[0m Trial 51 finished with value: 0.7703030303030304 and parameters: {'n_neighbors': 21, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:24,302]\u001B[0m Trial 52 finished with value: 0.7657239057239056 and parameters: {'n_neighbors': 9, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:26,902]\u001B[0m Trial 53 finished with value: 0.7702188552188552 and parameters: {'n_neighbors': 30, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:29,800]\u001B[0m Trial 54 finished with value: 0.7695959595959596 and parameters: {'n_neighbors': 17, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:32,489]\u001B[0m Trial 55 finished with value: 0.77010101010101 and parameters: {'n_neighbors': 32, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:35,351]\u001B[0m Trial 56 finished with value: 0.7625925925925926 and parameters: {'n_neighbors': 6, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:38,101]\u001B[0m Trial 57 finished with value: 0.7693771043771045 and parameters: {'n_neighbors': 70, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:41,105]\u001B[0m Trial 58 finished with value: 0.7391245791245791 and parameters: {'n_neighbors': 21, 'weights': 'uniform'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:44,477]\u001B[0m Trial 59 finished with value: 0.7666835016835017 and parameters: {'n_neighbors': 240, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:46,995]\u001B[0m Trial 60 finished with value: 0.7704713804713804 and parameters: {'n_neighbors': 46, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:49,891]\u001B[0m Trial 61 finished with value: 0.7697474747474747 and parameters: {'n_neighbors': 33, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:52,392]\u001B[0m Trial 62 finished with value: 0.7704713804713804 and parameters: {'n_neighbors': 46, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:55,108]\u001B[0m Trial 63 finished with value: 0.7700841750841751 and parameters: {'n_neighbors': 48, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:22:57,656]\u001B[0m Trial 64 finished with value: 0.7704713804713805 and parameters: {'n_neighbors': 43, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:00,506]\u001B[0m Trial 65 finished with value: 0.7698148148148147 and parameters: {'n_neighbors': 59, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:03,387]\u001B[0m Trial 66 finished with value: 0.7702693602693602 and parameters: {'n_neighbors': 47, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:05,920]\u001B[0m Trial 67 finished with value: 0.7703367003367003 and parameters: {'n_neighbors': 42, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:09,007]\u001B[0m Trial 68 finished with value: 0.7703030303030303 and parameters: {'n_neighbors': 37, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:12,117]\u001B[0m Trial 69 finished with value: 0.7693602693602692 and parameters: {'n_neighbors': 71, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:14,901]\u001B[0m Trial 70 finished with value: 0.7702188552188552 and parameters: {'n_neighbors': 45, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:18,058]\u001B[0m Trial 71 finished with value: 0.7702861952861954 and parameters: {'n_neighbors': 54, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:20,951]\u001B[0m Trial 72 finished with value: 0.7678114478114477 and parameters: {'n_neighbors': 12, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:23,831]\u001B[0m Trial 73 finished with value: 0.7701010101010101 and parameters: {'n_neighbors': 28, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:26,678]\u001B[0m Trial 74 finished with value: 0.7683164983164983 and parameters: {'n_neighbors': 107, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:29,798]\u001B[0m Trial 75 finished with value: 0.7671548821548821 and parameters: {'n_neighbors': 150, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:32,744]\u001B[0m Trial 76 finished with value: 0.7685690235690236 and parameters: {'n_neighbors': 88, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:36,162]\u001B[0m Trial 77 finished with value: 0.7669528619528619 and parameters: {'n_neighbors': 183, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:38,791]\u001B[0m Trial 78 finished with value: 0.7262121212121213 and parameters: {'n_neighbors': 36, 'weights': 'uniform'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:41,271]\u001B[0m Trial 79 finished with value: 0.7698148148148147 and parameters: {'n_neighbors': 59, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:43,731]\u001B[0m Trial 80 finished with value: 0.7703367003367003 and parameters: {'n_neighbors': 44, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:46,232]\u001B[0m Trial 81 finished with value: 0.7704713804713805 and parameters: {'n_neighbors': 43, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:48,691]\u001B[0m Trial 82 finished with value: 0.7704713804713805 and parameters: {'n_neighbors': 43, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:51,160]\u001B[0m Trial 83 finished with value: 0.7704713804713805 and parameters: {'n_neighbors': 43, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:53,704]\u001B[0m Trial 84 finished with value: 0.7701346801346801 and parameters: {'n_neighbors': 51, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:56,372]\u001B[0m Trial 85 finished with value: 0.7693602693602692 and parameters: {'n_neighbors': 71, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:23:58,957]\u001B[0m Trial 86 finished with value: 0.7703367003367003 and parameters: {'n_neighbors': 55, 'weights': 'distance'}. Best is trial 35 with value: 0.7705892255892255.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:24:01,443]\u001B[0m Trial 87 finished with value: 0.7707407407407407 and parameters: {'n_neighbors': 57, 'weights': 'distance'}. Best is trial 87 with value: 0.7707407407407407.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:24:03,934]\u001B[0m Trial 88 finished with value: 0.7693265993265993 and parameters: {'n_neighbors': 66, 'weights': 'distance'}. Best is trial 87 with value: 0.7707407407407407.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:24:06,355]\u001B[0m Trial 89 finished with value: 0.7699663299663299 and parameters: {'n_neighbors': 36, 'weights': 'distance'}. Best is trial 87 with value: 0.7707407407407407.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:24:08,906]\u001B[0m Trial 90 finished with value: 0.7336363636363636 and parameters: {'n_neighbors': 27, 'weights': 'uniform'}. Best is trial 87 with value: 0.7707407407407407.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:24:11,420]\u001B[0m Trial 91 finished with value: 0.7702693602693602 and parameters: {'n_neighbors': 41, 'weights': 'distance'}. Best is trial 87 with value: 0.7707407407407407.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:24:13,972]\u001B[0m Trial 92 finished with value: 0.7707407407407407 and parameters: {'n_neighbors': 57, 'weights': 'distance'}. Best is trial 87 with value: 0.7707407407407407.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:24:16,666]\u001B[0m Trial 93 finished with value: 0.7701010101010101 and parameters: {'n_neighbors': 58, 'weights': 'distance'}. Best is trial 87 with value: 0.7707407407407407.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:24:19,235]\u001B[0m Trial 94 finished with value: 0.7690404040404041 and parameters: {'n_neighbors': 82, 'weights': 'distance'}. Best is trial 87 with value: 0.7707407407407407.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:24:21,698]\u001B[0m Trial 95 finished with value: 0.7701178451178452 and parameters: {'n_neighbors': 49, 'weights': 'distance'}. Best is trial 87 with value: 0.7707407407407407.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:24:24,263]\u001B[0m Trial 96 finished with value: 0.7691582491582492 and parameters: {'n_neighbors': 68, 'weights': 'distance'}. Best is trial 87 with value: 0.7707407407407407.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:24:26,624]\u001B[0m Trial 97 finished with value: 0.7704713804713805 and parameters: {'n_neighbors': 26, 'weights': 'distance'}. Best is trial 87 with value: 0.7707407407407407.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:24:29,001]\u001B[0m Trial 98 finished with value: 0.7683501683501683 and parameters: {'n_neighbors': 13, 'weights': 'distance'}. Best is trial 87 with value: 0.7707407407407407.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:24:31,610]\u001B[0m Trial 99 finished with value: 0.7693939393939394 and parameters: {'n_neighbors': 75, 'weights': 'distance'}. Best is trial 87 with value: 0.7707407407407407.\u001B[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": "    number     value             datetime_start          datetime_complete  \\\n0        0  0.699444 2022-07-13 13:19:57.880970 2022-07-13 13:20:00.412597   \n1        1  0.766953 2022-07-13 13:20:00.414589 2022-07-13 13:20:04.188689   \n2        2  0.694495 2022-07-13 13:20:04.189659 2022-07-13 13:20:07.578683   \n3        3  0.724781 2022-07-13 13:20:07.579682 2022-07-13 13:20:10.640665   \n4        4  0.743855 2022-07-13 13:20:10.641665 2022-07-13 13:20:13.345806   \n..     ...       ...                        ...                        ...   \n95      95  0.770118 2022-07-13 13:24:19.236398 2022-07-13 13:24:21.698574   \n96      96  0.769158 2022-07-13 13:24:21.699576 2022-07-13 13:24:24.263477   \n97      97  0.770471 2022-07-13 13:24:24.264478 2022-07-13 13:24:26.624897   \n98      98  0.768350 2022-07-13 13:24:26.625781 2022-07-13 13:24:29.001553   \n99      99  0.769394 2022-07-13 13:24:29.002550 2022-07-13 13:24:31.610706   \n\n                 duration  params_n_neighbors params_weights     state  \n0  0 days 00:00:02.531627                 101        uniform  COMPLETE  \n1  0 days 00:00:03.774100                 169       distance  COMPLETE  \n2  0 days 00:00:03.389024                 120        uniform  COMPLETE  \n3  0 days 00:00:03.060983                  40        uniform  COMPLETE  \n4  0 days 00:00:02.704141                  16        uniform  COMPLETE  \n..                    ...                 ...            ...       ...  \n95 0 days 00:00:02.462176                  49       distance  COMPLETE  \n96 0 days 00:00:02.563901                  68       distance  COMPLETE  \n97 0 days 00:00:02.360419                  26       distance  COMPLETE  \n98 0 days 00:00:02.375772                  13       distance  COMPLETE  \n99 0 days 00:00:02.608156                  75       distance  COMPLETE  \n\n[100 rows x 8 columns]",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>number</th>\n      <th>value</th>\n      <th>datetime_start</th>\n      <th>datetime_complete</th>\n      <th>duration</th>\n      <th>params_n_neighbors</th>\n      <th>params_weights</th>\n      <th>state</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0</td>\n      <td>0.699444</td>\n      <td>2022-07-13 13:19:57.880970</td>\n      <td>2022-07-13 13:20:00.412597</td>\n      <td>0 days 00:00:02.531627</td>\n      <td>101</td>\n      <td>uniform</td>\n      <td>COMPLETE</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>0.766953</td>\n      <td>2022-07-13 13:20:00.414589</td>\n      <td>2022-07-13 13:20:04.188689</td>\n      <td>0 days 00:00:03.774100</td>\n      <td>169</td>\n      <td>distance</td>\n      <td>COMPLETE</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2</td>\n      <td>0.694495</td>\n      <td>2022-07-13 13:20:04.189659</td>\n      <td>2022-07-13 13:20:07.578683</td>\n      <td>0 days 00:00:03.389024</td>\n      <td>120</td>\n      <td>uniform</td>\n      <td>COMPLETE</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>3</td>\n      <td>0.724781</td>\n      <td>2022-07-13 13:20:07.579682</td>\n      <td>2022-07-13 13:20:10.640665</td>\n      <td>0 days 00:00:03.060983</td>\n      <td>40</td>\n      <td>uniform</td>\n      <td>COMPLETE</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>4</td>\n      <td>0.743855</td>\n      <td>2022-07-13 13:20:10.641665</td>\n      <td>2022-07-13 13:20:13.345806</td>\n      <td>0 days 00:00:02.704141</td>\n      <td>16</td>\n      <td>uniform</td>\n      <td>COMPLETE</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>95</th>\n      <td>95</td>\n      <td>0.770118</td>\n      <td>2022-07-13 13:24:19.236398</td>\n      <td>2022-07-13 13:24:21.698574</td>\n      <td>0 days 00:00:02.462176</td>\n      <td>49</td>\n      <td>distance</td>\n      <td>COMPLETE</td>\n    </tr>\n    <tr>\n      <th>96</th>\n      <td>96</td>\n      <td>0.769158</td>\n      <td>2022-07-13 13:24:21.699576</td>\n      <td>2022-07-13 13:24:24.263477</td>\n      <td>0 days 00:00:02.563901</td>\n      <td>68</td>\n      <td>distance</td>\n      <td>COMPLETE</td>\n    </tr>\n    <tr>\n      <th>97</th>\n      <td>97</td>\n      <td>0.770471</td>\n      <td>2022-07-13 13:24:24.264478</td>\n      <td>2022-07-13 13:24:26.624897</td>\n      <td>0 days 00:00:02.360419</td>\n      <td>26</td>\n      <td>distance</td>\n      <td>COMPLETE</td>\n    </tr>\n    <tr>\n      <th>98</th>\n      <td>98</td>\n      <td>0.768350</td>\n      <td>2022-07-13 13:24:26.625781</td>\n      <td>2022-07-13 13:24:29.001553</td>\n      <td>0 days 00:00:02.375772</td>\n      <td>13</td>\n      <td>distance</td>\n      <td>COMPLETE</td>\n    </tr>\n    <tr>\n      <th>99</th>\n      <td>99</td>\n      <td>0.769394</td>\n      <td>2022-07-13 13:24:29.002550</td>\n      <td>2022-07-13 13:24:31.610706</td>\n      <td>0 days 00:00:02.608156</td>\n      <td>75</td>\n      <td>distance</td>\n      <td>COMPLETE</td>\n    </tr>\n  </tbody>\n</table>\n<p>100 rows × 8 columns</p>\n</div>"
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "import optuna\n",
    "\n",
    "n_trials = 100\n",
    "\n",
    "encoder = CountEncoder()\n",
    "x_train = encoder.fit_transform(train_inputs)\n",
    "\n",
    "def knn_objective(trial):\n",
    "    n_neighbors = trial.suggest_int('n_neighbors', 5, int(train_inputs.shape[0]**(1/2)))\n",
    "    weights = trial.suggest_categorical('weights', ['uniform', 'distance'])\n",
    "\n",
    "    knn_clf = KNeighborsClassifier(n_neighbors=n_neighbors, weights=weights)\n",
    "    avg_acc = cross_val_score(knn_clf, x_train, train_targets, cv=10, n_jobs=-1).mean()\n",
    "    return avg_acc\n",
    "\n",
    "\n",
    "knn_study = optuna.create_study(direction='maximize')\n",
    "knn_study.optimize(knn_objective, n_trials=n_trials)\n",
    "knn_study.trials_dataframe()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "data": [
        {
         "colorbar": {
          "title": {
           "text": "Objective Value"
          }
         },
         "colorscale": [
          [
           0,
           "rgb(5,10,172)"
          ],
          [
           0.35,
           "rgb(40,60,190)"
          ],
          [
           0.5,
           "rgb(70,100,245)"
          ],
          [
           0.6,
           "rgb(90,120,245)"
          ],
          [
           0.7,
           "rgb(106,137,247)"
          ],
          [
           1,
           "rgb(220,220,220)"
          ]
         ],
         "connectgaps": true,
         "contours": {
          "coloring": "heatmap"
         },
         "hoverinfo": "none",
         "line": {
          "smoothing": 1.3
         },
         "reversescale": false,
         "x": [
          -6.9,
          5,
          6,
          7,
          9,
          12,
          13,
          16,
          17,
          19,
          21,
          22,
          23,
          25,
          26,
          27,
          28,
          30,
          31,
          32,
          33,
          36,
          37,
          38,
          40,
          41,
          42,
          43,
          44,
          45,
          46,
          47,
          48,
          49,
          50,
          51,
          53,
          54,
          55,
          57,
          58,
          59,
          62,
          63,
          64,
          66,
          68,
          70,
          71,
          75,
          76,
          78,
          79,
          82,
          87,
          88,
          93,
          101,
          103,
          107,
          113,
          120,
          123,
          130,
          138,
          150,
          154,
          163,
          164,
          169,
          171,
          176,
          183,
          200,
          210,
          240,
          243,
          254.9
         ],
         "y": [
          "distance",
          "uniform"
         ],
         "z": [
          [
           null,
           0.7612289562289563,
           0.7625925925925926,
           0.7651851851851852,
           0.7657239057239056,
           0.7678114478114477,
           0.7683501683501683,
           0.7696969696969698,
           0.7695959595959596,
           null,
           0.7703030303030304,
           0.7705723905723906,
           0.7705892255892255,
           null,
           0.7704713804713805,
           null,
           0.7701010101010101,
           0.7702188552188552,
           0.769983164983165,
           0.77010101010101,
           0.7697474747474747,
           0.7699663299663299,
           0.7703030303030303,
           null,
           0.7701346801346801,
           0.7702693602693602,
           0.7703367003367003,
           0.7704713804713805,
           0.7703367003367003,
           0.7702188552188552,
           0.7704713804713804,
           0.7702693602693602,
           0.7700841750841751,
           0.7701178451178452,
           null,
           0.7701346801346801,
           0.7702356902356902,
           0.7702861952861954,
           0.7703367003367003,
           0.7707407407407407,
           0.7701010101010101,
           0.7698148148148147,
           0.7699999999999999,
           0.769814814814815,
           0.7695622895622896,
           0.7693265993265993,
           0.7691582491582492,
           0.7693771043771045,
           0.7693602693602692,
           0.7693939393939394,
           null,
           null,
           0.7691919191919192,
           0.7690404040404041,
           0.7685521885521885,
           0.7685690235690236,
           0.7688383838383838,
           null,
           0.7684175084175084,
           0.7683164983164983,
           0.7677272727272728,
           null,
           0.7673569023569022,
           0.7672222222222221,
           0.7670538720538721,
           0.7671548821548821,
           0.767037037037037,
           0.7669528619528619,
           0.7668686868686868,
           0.766952861952862,
           0.7667676767676768,
           0.7669865319865321,
           0.7669528619528619,
           0.7668686868686868,
           null,
           0.7666835016835017,
           0.7666666666666667,
           null
          ],
          [
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           0.7438552188552189,
           null,
           0.7414814814814815,
           0.7391245791245791,
           null,
           null,
           0.73493265993266,
           0.735,
           0.7336363636363636,
           null,
           null,
           null,
           null,
           null,
           0.7262121212121213,
           null,
           0.725117845117845,
           0.7247811447811449,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           0.7187373737373738,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           0.7101178451178451,
           0.7089562289562289,
           null,
           null,
           null,
           null,
           null,
           0.6994444444444445,
           null,
           null,
           null,
           0.6944949494949495,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           0.6779124579124579,
           null,
           null,
           null
          ]
         ],
         "type": "contour"
        },
        {
         "marker": {
          "color": "black",
          "line": {
           "color": "Grey",
           "width": 0.5
          }
         },
         "mode": "markers",
         "showlegend": false,
         "x": [
          101,
          169,
          120,
          40,
          16,
          76,
          78,
          176,
          50,
          210,
          171,
          164,
          243,
          163,
          200,
          154,
          138,
          123,
          130,
          113,
          103,
          93,
          87,
          79,
          79,
          54,
          53,
          41,
          7,
          47,
          31,
          53,
          62,
          31,
          63,
          23,
          21,
          26,
          16,
          19,
          40,
          5,
          41,
          64,
          38,
          21,
          22,
          25,
          13,
          21,
          21,
          21,
          9,
          30,
          17,
          32,
          6,
          70,
          21,
          240,
          46,
          33,
          46,
          48,
          43,
          59,
          47,
          42,
          37,
          71,
          45,
          54,
          12,
          28,
          107,
          150,
          88,
          183,
          36,
          59,
          44,
          43,
          43,
          43,
          51,
          71,
          55,
          57,
          66,
          36,
          27,
          41,
          57,
          58,
          82,
          49,
          68,
          26,
          13,
          75
         ],
         "y": [
          "uniform",
          "distance",
          "uniform",
          "uniform",
          "uniform",
          "uniform",
          "uniform",
          "distance",
          "uniform",
          "uniform",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "uniform",
          "distance",
          "uniform",
          "distance",
          "distance",
          "distance",
          "distance",
          "uniform",
          "distance",
          "distance",
          "uniform",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "uniform",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "uniform",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "uniform",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance",
          "distance"
         ],
         "type": "scatter"
        }
       ],
       "layout": {
        "title": {
         "text": "Contour Plot"
        },
        "template": {
         "data": {
          "histogram2dcontour": [
           {
            "type": "histogram2dcontour",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "choropleth": [
           {
            "type": "choropleth",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            }
           }
          ],
          "histogram2d": [
           {
            "type": "histogram2d",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "heatmap": [
           {
            "type": "heatmap",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "heatmapgl": [
           {
            "type": "heatmapgl",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "contourcarpet": [
           {
            "type": "contourcarpet",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            }
           }
          ],
          "contour": [
           {
            "type": "contour",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "surface": [
           {
            "type": "surface",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "mesh3d": [
           {
            "type": "mesh3d",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            }
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "parcoords": [
           {
            "type": "parcoords",
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scatterpolargl": [
           {
            "type": "scatterpolargl",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "scattergeo": [
           {
            "type": "scattergeo",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scatterpolar": [
           {
            "type": "scatterpolar",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "scattergl": [
           {
            "type": "scattergl",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scatter3d": [
           {
            "type": "scatter3d",
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scattermapbox": [
           {
            "type": "scattermapbox",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scatterternary": [
           {
            "type": "scatterternary",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scattercarpet": [
           {
            "type": "scattercarpet",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ]
         },
         "layout": {
          "autotypenumbers": "strict",
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "hovermode": "closest",
          "hoverlabel": {
           "align": "left"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "bgcolor": "#E5ECF6",
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "ternary": {
           "bgcolor": "#E5ECF6",
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "sequential": [
            [
             0.0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1.0,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0.0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1.0,
             "#f0f921"
            ]
           ],
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ]
          },
          "xaxis": {
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "automargin": true,
           "zerolinewidth": 2
          },
          "yaxis": {
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "automargin": true,
           "zerolinewidth": 2
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white",
            "gridwidth": 2
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white",
            "gridwidth": 2
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white",
            "gridwidth": 2
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "geo": {
           "bgcolor": "white",
           "landcolor": "#E5ECF6",
           "subunitcolor": "white",
           "showland": true,
           "showlakes": true,
           "lakecolor": "white"
          },
          "title": {
           "x": 0.05
          },
          "mapbox": {
           "style": "light"
          }
         }
        },
        "xaxis": {
         "title": {
          "text": "n_neighbors"
         },
         "range": [
          -6.9,
          254.9
         ]
        },
        "yaxis": {
         "title": {
          "text": "weights"
         },
         "range": [
          -0.05,
          1.05
         ],
         "type": "category"
        }
       },
       "config": {
        "plotlyServerURL": "https://plot.ly"
       }
      },
      "text/html": "<div>                            <div id=\"94ba6da0-4ee9-4058-b016-b6bbd09080be\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"94ba6da0-4ee9-4058-b016-b6bbd09080be\")) {                    Plotly.newPlot(                        \"94ba6da0-4ee9-4058-b016-b6bbd09080be\",                        [{\"colorbar\":{\"title\":{\"text\":\"Objective Value\"}},\"colorscale\":[[0,\"rgb(5,10,172)\"],[0.35,\"rgb(40,60,190)\"],[0.5,\"rgb(70,100,245)\"],[0.6,\"rgb(90,120,245)\"],[0.7,\"rgb(106,137,247)\"],[1,\"rgb(220,220,220)\"]],\"connectgaps\":true,\"contours\":{\"coloring\":\"heatmap\"},\"hoverinfo\":\"none\",\"line\":{\"smoothing\":1.3},\"reversescale\":false,\"x\":[-6.9,5,6,7,9,12,13,16,17,19,21,22,23,25,26,27,28,30,31,32,33,36,37,38,40,41,42,43,44,45,46,47,48,49,50,51,53,54,55,57,58,59,62,63,64,66,68,70,71,75,76,78,79,82,87,88,93,101,103,107,113,120,123,130,138,150,154,163,164,169,171,176,183,200,210,240,243,254.9],\"y\":[\"distance\",\"uniform\"],\"z\":[[null,0.7612289562289563,0.7625925925925926,0.7651851851851852,0.7657239057239056,0.7678114478114477,0.7683501683501683,0.7696969696969698,0.7695959595959596,null,0.7703030303030304,0.7705723905723906,0.7705892255892255,null,0.7704713804713805,null,0.7701010101010101,0.7702188552188552,0.769983164983165,0.77010101010101,0.7697474747474747,0.7699663299663299,0.7703030303030303,null,0.7701346801346801,0.7702693602693602,0.7703367003367003,0.7704713804713805,0.7703367003367003,0.7702188552188552,0.7704713804713804,0.7702693602693602,0.7700841750841751,0.7701178451178452,null,0.7701346801346801,0.7702356902356902,0.7702861952861954,0.7703367003367003,0.7707407407407407,0.7701010101010101,0.7698148148148147,0.7699999999999999,0.769814814814815,0.7695622895622896,0.7693265993265993,0.7691582491582492,0.7693771043771045,0.7693602693602692,0.7693939393939394,null,null,0.7691919191919192,0.7690404040404041,0.7685521885521885,0.7685690235690236,0.7688383838383838,null,0.7684175084175084,0.7683164983164983,0.7677272727272728,null,0.7673569023569022,0.7672222222222221,0.7670538720538721,0.7671548821548821,0.767037037037037,0.7669528619528619,0.7668686868686868,0.766952861952862,0.7667676767676768,0.7669865319865321,0.7669528619528619,0.7668686868686868,null,0.7666835016835017,0.7666666666666667,null],[null,null,null,null,null,null,null,0.7438552188552189,null,0.7414814814814815,0.7391245791245791,null,null,0.73493265993266,0.735,0.7336363636363636,null,null,null,null,null,0.7262121212121213,null,0.725117845117845,0.7247811447811449,null,null,null,null,null,null,null,null,null,0.7187373737373738,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0.7101178451178451,0.7089562289562289,null,null,null,null,null,0.6994444444444445,null,null,null,0.6944949494949495,null,null,null,null,null,null,null,null,null,null,null,null,0.6779124579124579,null,null,null]],\"type\":\"contour\"},{\"marker\":{\"color\":\"black\",\"line\":{\"color\":\"Grey\",\"width\":0.5}},\"mode\":\"markers\",\"showlegend\":false,\"x\":[101,169,120,40,16,76,78,176,50,210,171,164,243,163,200,154,138,123,130,113,103,93,87,79,79,54,53,41,7,47,31,53,62,31,63,23,21,26,16,19,40,5,41,64,38,21,22,25,13,21,21,21,9,30,17,32,6,70,21,240,46,33,46,48,43,59,47,42,37,71,45,54,12,28,107,150,88,183,36,59,44,43,43,43,51,71,55,57,66,36,27,41,57,58,82,49,68,26,13,75],\"y\":[\"uniform\",\"distance\",\"uniform\",\"uniform\",\"uniform\",\"uniform\",\"uniform\",\"distance\",\"uniform\",\"uniform\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"uniform\",\"distance\",\"uniform\",\"distance\",\"distance\",\"distance\",\"distance\",\"uniform\",\"distance\",\"distance\",\"uniform\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"uniform\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"uniform\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"uniform\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\",\"distance\"],\"type\":\"scatter\"}],                        {\"title\":{\"text\":\"Contour Plot\"},\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"xaxis\":{\"title\":{\"text\":\"n_neighbors\"},\"range\":[-6.9,254.9]},\"yaxis\":{\"title\":{\"text\":\"weights\"},\"range\":[-0.05,1.05],\"type\":\"category\"}},                        {\"responsive\": true}                    ).then(function(){\n                            \nvar gd = document.getElementById('94ba6da0-4ee9-4058-b016-b6bbd09080be');\nvar x = new MutationObserver(function (mutations, observer) {{\n        var display = window.getComputedStyle(gd).display;\n        if (!display || display === 'none') {{\n            console.log([gd, 'removed!']);\n            Plotly.purge(gd);\n            observer.disconnect();\n        }}\n}});\n\n// Listen for the removal of the full notebook cells\nvar notebookContainer = gd.closest('#notebook-container');\nif (notebookContainer) {{\n    x.observe(notebookContainer, {childList: true});\n}}\n\n// Listen for the clearing of the current output cell\nvar outputEl = gd.closest('.output');\nif (outputEl) {{\n    x.observe(outputEl, {childList: true});\n}}\n\n                        })                };                });            </script>        </div>"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "optuna.visualization.plot_contour(knn_study, params=['n_neighbors', 'weights'])"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "data": [
        {
         "cliponaxis": false,
         "hovertemplate": [
          "n_neighbors (IntUniformDistribution): 0.08213298963722387<extra></extra>",
          "weights (CategoricalDistribution): 0.9178670103627761<extra></extra>"
         ],
         "marker": {
          "color": "rgb(66,146,198)"
         },
         "orientation": "h",
         "text": [
          "0.08213298963722387",
          "0.9178670103627761"
         ],
         "textposition": "outside",
         "texttemplate": "%{text:.2f}",
         "x": [
          0.08213298963722387,
          0.9178670103627761
         ],
         "y": [
          "n_neighbors",
          "weights"
         ],
         "type": "bar"
        }
       ],
       "layout": {
        "showlegend": false,
        "title": {
         "text": "Hyperparameter Importances"
        },
        "xaxis": {
         "title": {
          "text": "Importance for Objective Value"
         }
        },
        "yaxis": {
         "title": {
          "text": "Hyperparameter"
         }
        },
        "template": {
         "data": {
          "histogram2dcontour": [
           {
            "type": "histogram2dcontour",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "choropleth": [
           {
            "type": "choropleth",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            }
           }
          ],
          "histogram2d": [
           {
            "type": "histogram2d",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "heatmap": [
           {
            "type": "heatmap",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "heatmapgl": [
           {
            "type": "heatmapgl",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "contourcarpet": [
           {
            "type": "contourcarpet",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            }
           }
          ],
          "contour": [
           {
            "type": "contour",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "surface": [
           {
            "type": "surface",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "mesh3d": [
           {
            "type": "mesh3d",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            }
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "parcoords": [
           {
            "type": "parcoords",
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scatterpolargl": [
           {
            "type": "scatterpolargl",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "scattergeo": [
           {
            "type": "scattergeo",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scatterpolar": [
           {
            "type": "scatterpolar",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "scattergl": [
           {
            "type": "scattergl",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scatter3d": [
           {
            "type": "scatter3d",
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scattermapbox": [
           {
            "type": "scattermapbox",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scatterternary": [
           {
            "type": "scatterternary",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scattercarpet": [
           {
            "type": "scattercarpet",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ]
         },
         "layout": {
          "autotypenumbers": "strict",
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "hovermode": "closest",
          "hoverlabel": {
           "align": "left"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "bgcolor": "#E5ECF6",
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "ternary": {
           "bgcolor": "#E5ECF6",
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "sequential": [
            [
             0.0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1.0,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0.0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1.0,
             "#f0f921"
            ]
           ],
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ]
          },
          "xaxis": {
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "automargin": true,
           "zerolinewidth": 2
          },
          "yaxis": {
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "automargin": true,
           "zerolinewidth": 2
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white",
            "gridwidth": 2
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white",
            "gridwidth": 2
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white",
            "gridwidth": 2
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "geo": {
           "bgcolor": "white",
           "landcolor": "#E5ECF6",
           "subunitcolor": "white",
           "showland": true,
           "showlakes": true,
           "lakecolor": "white"
          },
          "title": {
           "x": 0.05
          },
          "mapbox": {
           "style": "light"
          }
         }
        }
       },
       "config": {
        "plotlyServerURL": "https://plot.ly"
       }
      },
      "text/html": "<div>                            <div id=\"0b48af5e-8c8f-47da-9537-f64c04dc31f1\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"0b48af5e-8c8f-47da-9537-f64c04dc31f1\")) {                    Plotly.newPlot(                        \"0b48af5e-8c8f-47da-9537-f64c04dc31f1\",                        [{\"cliponaxis\":false,\"hovertemplate\":[\"n_neighbors (IntUniformDistribution): 0.08213298963722387<extra></extra>\",\"weights (CategoricalDistribution): 0.9178670103627761<extra></extra>\"],\"marker\":{\"color\":\"rgb(66,146,198)\"},\"orientation\":\"h\",\"text\":[\"0.08213298963722387\",\"0.9178670103627761\"],\"textposition\":\"outside\",\"texttemplate\":\"%{text:.2f}\",\"x\":[0.08213298963722387,0.9178670103627761],\"y\":[\"n_neighbors\",\"weights\"],\"type\":\"bar\"}],                        {\"showlegend\":false,\"title\":{\"text\":\"Hyperparameter Importances\"},\"xaxis\":{\"title\":{\"text\":\"Importance for Objective Value\"}},\"yaxis\":{\"title\":{\"text\":\"Hyperparameter\"}},\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}}},                        {\"responsive\": true}                    ).then(function(){\n                            \nvar gd = document.getElementById('0b48af5e-8c8f-47da-9537-f64c04dc31f1');\nvar x = new MutationObserver(function (mutations, observer) {{\n        var display = window.getComputedStyle(gd).display;\n        if (!display || display === 'none') {{\n            console.log([gd, 'removed!']);\n            Plotly.purge(gd);\n            observer.disconnect();\n        }}\n}});\n\n// Listen for the removal of the full notebook cells\nvar notebookContainer = gd.closest('#notebook-container');\nif (notebookContainer) {{\n    x.observe(notebookContainer, {childList: true});\n}}\n\n// Listen for the clearing of the current output cell\nvar outputEl = gd.closest('.output');\nif (outputEl) {{\n    x.observe(outputEl, {childList: true});\n}}\n\n                        })                };                });            </script>        </div>"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "optuna.visualization.plot_param_importances(knn_study)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Hyper-Parameter Optimization - Random Forest"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001B[32m[I 2022-07-13 13:31:14,772]\u001B[0m A new study created in memory with name: no-name-75fa07dd-a656-4e3f-a4c1-bc370682a64c\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:31:50,249]\u001B[0m Trial 0 finished with value: 0.7983164983164983 and parameters: {'n_estimators': 335, 'optimizer': 'entropy'}. Best is trial 0 with value: 0.7983164983164983.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:32:38,592]\u001B[0m Trial 1 finished with value: 0.7992424242424243 and parameters: {'n_estimators': 454, 'optimizer': 'entropy'}. Best is trial 1 with value: 0.7992424242424243.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:33:09,829]\u001B[0m Trial 2 finished with value: 0.799141414141414 and parameters: {'n_estimators': 288, 'optimizer': 'entropy'}. Best is trial 1 with value: 0.7992424242424243.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:33:36,366]\u001B[0m Trial 3 finished with value: 0.7998653198653198 and parameters: {'n_estimators': 247, 'optimizer': 'log_loss'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:34:12,700]\u001B[0m Trial 4 finished with value: 0.7990235690235691 and parameters: {'n_estimators': 336, 'optimizer': 'entropy'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:34:20,647]\u001B[0m Trial 5 finished with value: 0.7974915824915825 and parameters: {'n_estimators': 73, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:34:43,022]\u001B[0m Trial 6 finished with value: 0.7985521885521887 and parameters: {'n_estimators': 235, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:35:28,392]\u001B[0m Trial 7 finished with value: 0.7989898989898989 and parameters: {'n_estimators': 419, 'optimizer': 'entropy'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:36:18,202]\u001B[0m Trial 8 finished with value: 0.7985185185185185 and parameters: {'n_estimators': 464, 'optimizer': 'log_loss'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:36:55,991]\u001B[0m Trial 9 finished with value: 0.7986195286195286 and parameters: {'n_estimators': 353, 'optimizer': 'entropy'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:37:12,506]\u001B[0m Trial 10 finished with value: 0.7987542087542089 and parameters: {'n_estimators': 152, 'optimizer': 'log_loss'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:37:35,858]\u001B[0m Trial 11 finished with value: 0.7986026936026935 and parameters: {'n_estimators': 213, 'optimizer': 'log_loss'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:38:25,164]\u001B[0m Trial 12 finished with value: 0.7987542087542088 and parameters: {'n_estimators': 438, 'optimizer': 'log_loss'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:38:44,597]\u001B[0m Trial 13 finished with value: 0.7985185185185186 and parameters: {'n_estimators': 168, 'optimizer': 'log_loss'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:39:34,458]\u001B[0m Trial 14 finished with value: 0.7989562289562289 and parameters: {'n_estimators': 495, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:40:19,411]\u001B[0m Trial 15 finished with value: 0.7988215488215488 and parameters: {'n_estimators': 386, 'optimizer': 'entropy'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:40:49,716]\u001B[0m Trial 16 finished with value: 0.7985185185185185 and parameters: {'n_estimators': 279, 'optimizer': 'log_loss'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:41:01,474]\u001B[0m Trial 17 finished with value: 0.797946127946128 and parameters: {'n_estimators': 106, 'optimizer': 'entropy'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:41:25,745]\u001B[0m Trial 18 finished with value: 0.7986868686868688 and parameters: {'n_estimators': 216, 'optimizer': 'log_loss'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:41:56,846]\u001B[0m Trial 19 finished with value: 0.798939393939394 and parameters: {'n_estimators': 306, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:42:42,836]\u001B[0m Trial 20 finished with value: 0.7990909090909091 and parameters: {'n_estimators': 386, 'optimizer': 'entropy'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:43:13,953]\u001B[0m Trial 21 finished with value: 0.799057239057239 and parameters: {'n_estimators': 268, 'optimizer': 'entropy'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:43:33,527]\u001B[0m Trial 22 finished with value: 0.7981313131313132 and parameters: {'n_estimators': 163, 'optimizer': 'entropy'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:44:05,869]\u001B[0m Trial 23 finished with value: 0.7989898989898989 and parameters: {'n_estimators': 289, 'optimizer': 'entropy'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:44:30,505]\u001B[0m Trial 24 finished with value: 0.7983838383838384 and parameters: {'n_estimators': 228, 'optimizer': 'entropy'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:45:13,077]\u001B[0m Trial 25 finished with value: 0.7994444444444445 and parameters: {'n_estimators': 380, 'optimizer': 'log_loss'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:46:08,877]\u001B[0m Trial 26 finished with value: 0.7985521885521887 and parameters: {'n_estimators': 485, 'optimizer': 'log_loss'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:46:56,344]\u001B[0m Trial 27 finished with value: 0.7989898989898989 and parameters: {'n_estimators': 400, 'optimizer': 'log_loss'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:47:49,430]\u001B[0m Trial 28 finished with value: 0.7990909090909091 and parameters: {'n_estimators': 447, 'optimizer': 'log_loss'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:48:32,323]\u001B[0m Trial 29 finished with value: 0.799057239057239 and parameters: {'n_estimators': 358, 'optimizer': 'log_loss'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:49:06,716]\u001B[0m Trial 30 finished with value: 0.7988047138047139 and parameters: {'n_estimators': 325, 'optimizer': 'log_loss'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:49:33,289]\u001B[0m Trial 31 finished with value: 0.7988552188552188 and parameters: {'n_estimators': 249, 'optimizer': 'entropy'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:50:05,988]\u001B[0m Trial 32 finished with value: 0.799074074074074 and parameters: {'n_estimators': 310, 'optimizer': 'entropy'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:50:39,356]\u001B[0m Trial 33 finished with value: 0.7995791245791246 and parameters: {'n_estimators': 358, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:51:13,909]\u001B[0m Trial 34 finished with value: 0.7988552188552188 and parameters: {'n_estimators': 371, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:51:52,083]\u001B[0m Trial 35 finished with value: 0.7982323232323232 and parameters: {'n_estimators': 409, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:52:34,671]\u001B[0m Trial 36 finished with value: 0.7991582491582492 and parameters: {'n_estimators': 457, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:53:07,365]\u001B[0m Trial 37 finished with value: 0.7983333333333335 and parameters: {'n_estimators': 342, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:53:47,255]\u001B[0m Trial 38 finished with value: 0.7985185185185185 and parameters: {'n_estimators': 427, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:54:39,838]\u001B[0m Trial 39 finished with value: 0.7988383838383839 and parameters: {'n_estimators': 475, 'optimizer': 'log_loss'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:54:57,816]\u001B[0m Trial 40 finished with value: 0.7988383838383839 and parameters: {'n_estimators': 188, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:55:40,492]\u001B[0m Trial 41 finished with value: 0.7992929292929293 and parameters: {'n_estimators': 451, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:56:19,610]\u001B[0m Trial 42 finished with value: 0.799057239057239 and parameters: {'n_estimators': 416, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:57:02,802]\u001B[0m Trial 43 finished with value: 0.7985690235690235 and parameters: {'n_estimators': 441, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:57:39,913]\u001B[0m Trial 44 finished with value: 0.7993602693602694 and parameters: {'n_estimators': 372, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:58:14,645]\u001B[0m Trial 45 finished with value: 0.7991750841750842 and parameters: {'n_estimators': 370, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:58:44,616]\u001B[0m Trial 46 finished with value: 0.7987542087542089 and parameters: {'n_estimators': 321, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:59:17,765]\u001B[0m Trial 47 finished with value: 0.7993602693602695 and parameters: {'n_estimators': 347, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 13:59:49,353]\u001B[0m Trial 48 finished with value: 0.7996969696969697 and parameters: {'n_estimators': 339, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:00:24,987]\u001B[0m Trial 49 finished with value: 0.798148148148148 and parameters: {'n_estimators': 338, 'optimizer': 'log_loss'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:00:49,176]\u001B[0m Trial 50 finished with value: 0.7984006734006734 and parameters: {'n_estimators': 255, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:01:25,412]\u001B[0m Trial 51 finished with value: 0.7989057239057239 and parameters: {'n_estimators': 388, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:01:52,924]\u001B[0m Trial 52 finished with value: 0.7991919191919192 and parameters: {'n_estimators': 295, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:02:26,530]\u001B[0m Trial 53 finished with value: 0.7988047138047139 and parameters: {'n_estimators': 359, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:02:58,376]\u001B[0m Trial 54 finished with value: 0.7988215488215489 and parameters: {'n_estimators': 339, 'optimizer': 'gini'}. Best is trial 3 with value: 0.7998653198653198.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:03:37,757]\u001B[0m Trial 55 finished with value: 0.8000841750841751 and parameters: {'n_estimators': 375, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:04:19,320]\u001B[0m Trial 56 finished with value: 0.7989562289562289 and parameters: {'n_estimators': 395, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:04:56,840]\u001B[0m Trial 57 finished with value: 0.7989225589225588 and parameters: {'n_estimators': 354, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:05:25,684]\u001B[0m Trial 58 finished with value: 0.7992929292929293 and parameters: {'n_estimators': 269, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:06:00,850]\u001B[0m Trial 59 finished with value: 0.7989393939393941 and parameters: {'n_estimators': 306, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:06:15,581]\u001B[0m Trial 60 finished with value: 0.7980976430976432 and parameters: {'n_estimators': 132, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:06:50,798]\u001B[0m Trial 61 finished with value: 0.7996464646464647 and parameters: {'n_estimators': 375, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:07:28,670]\u001B[0m Trial 62 finished with value: 0.7989057239057239 and parameters: {'n_estimators': 324, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:08:09,107]\u001B[0m Trial 63 finished with value: 0.7992255892255892 and parameters: {'n_estimators': 382, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:08:53,926]\u001B[0m Trial 64 finished with value: 0.7981986531986532 and parameters: {'n_estimators': 426, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:09:00,055]\u001B[0m Trial 65 finished with value: 0.7970875420875421 and parameters: {'n_estimators': 63, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:09:44,404]\u001B[0m Trial 66 finished with value: 0.798973063973064 and parameters: {'n_estimators': 407, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:10:04,606]\u001B[0m Trial 67 finished with value: 0.7988047138047139 and parameters: {'n_estimators': 202, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:10:36,613]\u001B[0m Trial 68 finished with value: 0.7989898989898989 and parameters: {'n_estimators': 285, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:11:14,421]\u001B[0m Trial 69 finished with value: 0.7987205387205387 and parameters: {'n_estimators': 364, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:11:55,284]\u001B[0m Trial 70 finished with value: 0.7991077441077442 and parameters: {'n_estimators': 345, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:12:31,669]\u001B[0m Trial 71 finished with value: 0.7992929292929293 and parameters: {'n_estimators': 384, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:13:07,132]\u001B[0m Trial 72 finished with value: 0.7987710437710438 and parameters: {'n_estimators': 376, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:13:44,985]\u001B[0m Trial 73 finished with value: 0.7991414141414142 and parameters: {'n_estimators': 398, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:14:18,679]\u001B[0m Trial 74 finished with value: 0.7988215488215489 and parameters: {'n_estimators': 328, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:14:43,525]\u001B[0m Trial 75 finished with value: 0.7984680134680134 and parameters: {'n_estimators': 235, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:15:20,339]\u001B[0m Trial 76 finished with value: 0.7991077441077443 and parameters: {'n_estimators': 351, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:16:03,167]\u001B[0m Trial 77 finished with value: 0.7985858585858586 and parameters: {'n_estimators': 368, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:16:35,094]\u001B[0m Trial 78 finished with value: 0.7989225589225589 and parameters: {'n_estimators': 312, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:17:04,714]\u001B[0m Trial 79 finished with value: 0.7988888888888889 and parameters: {'n_estimators': 300, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:17:54,639]\u001B[0m Trial 80 finished with value: 0.7987710437710438 and parameters: {'n_estimators': 427, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:18:27,906]\u001B[0m Trial 81 finished with value: 0.7994444444444444 and parameters: {'n_estimators': 271, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:18:59,755]\u001B[0m Trial 82 finished with value: 0.799006734006734 and parameters: {'n_estimators': 261, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:19:29,425]\u001B[0m Trial 83 finished with value: 0.7982828282828283 and parameters: {'n_estimators': 246, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:19:56,619]\u001B[0m Trial 84 finished with value: 0.7989562289562289 and parameters: {'n_estimators': 220, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:20:26,550]\u001B[0m Trial 85 finished with value: 0.7992087542087541 and parameters: {'n_estimators': 281, 'optimizer': 'entropy'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:21:10,935]\u001B[0m Trial 86 finished with value: 0.7988552188552189 and parameters: {'n_estimators': 407, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:21:42,796]\u001B[0m Trial 87 finished with value: 0.7986363636363636 and parameters: {'n_estimators': 331, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:22:19,647]\u001B[0m Trial 88 finished with value: 0.7988888888888889 and parameters: {'n_estimators': 352, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:22:56,210]\u001B[0m Trial 89 finished with value: 0.7988383838383839 and parameters: {'n_estimators': 376, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:23:35,193]\u001B[0m Trial 90 finished with value: 0.7994444444444444 and parameters: {'n_estimators': 394, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:24:13,140]\u001B[0m Trial 91 finished with value: 0.7988215488215488 and parameters: {'n_estimators': 390, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:24:48,236]\u001B[0m Trial 92 finished with value: 0.7988215488215488 and parameters: {'n_estimators': 362, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:25:30,110]\u001B[0m Trial 93 finished with value: 0.7987373737373737 and parameters: {'n_estimators': 416, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:26:02,744]\u001B[0m Trial 94 finished with value: 0.799074074074074 and parameters: {'n_estimators': 346, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:26:32,541]\u001B[0m Trial 95 finished with value: 0.7991919191919192 and parameters: {'n_estimators': 318, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:27:14,583]\u001B[0m Trial 96 finished with value: 0.7986700336700336 and parameters: {'n_estimators': 376, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:27:53,656]\u001B[0m Trial 97 finished with value: 0.7989225589225589 and parameters: {'n_estimators': 394, 'optimizer': 'gini'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:28:30,758]\u001B[0m Trial 98 finished with value: 0.7990404040404041 and parameters: {'n_estimators': 338, 'optimizer': 'entropy'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n",
      "\u001B[32m[I 2022-07-13 14:29:18,065]\u001B[0m Trial 99 finished with value: 0.7988047138047137 and parameters: {'n_estimators': 438, 'optimizer': 'log_loss'}. Best is trial 55 with value: 0.8000841750841751.\u001B[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": "    number     value             datetime_start          datetime_complete  \\\n0        0  0.798316 2022-07-13 13:31:14.773330 2022-07-13 13:31:50.249579   \n1        1  0.799242 2022-07-13 13:31:50.250580 2022-07-13 13:32:38.592176   \n2        2  0.799141 2022-07-13 13:32:38.593176 2022-07-13 13:33:09.829821   \n3        3  0.799865 2022-07-13 13:33:09.831814 2022-07-13 13:33:36.366150   \n4        4  0.799024 2022-07-13 13:33:36.367152 2022-07-13 13:34:12.700284   \n..     ...       ...                        ...                        ...   \n95      95  0.799192 2022-07-13 14:26:02.745816 2022-07-13 14:26:32.541169   \n96      96  0.798670 2022-07-13 14:26:32.542130 2022-07-13 14:27:14.583398   \n97      97  0.798923 2022-07-13 14:27:14.584424 2022-07-13 14:27:53.656808   \n98      98  0.799040 2022-07-13 14:27:53.657803 2022-07-13 14:28:30.758131   \n99      99  0.798805 2022-07-13 14:28:30.759131 2022-07-13 14:29:18.065809   \n\n                 duration  params_n_estimators params_optimizer     state  \n0  0 days 00:00:35.476249                  335          entropy  COMPLETE  \n1  0 days 00:00:48.341596                  454          entropy  COMPLETE  \n2  0 days 00:00:31.236645                  288          entropy  COMPLETE  \n3  0 days 00:00:26.534336                  247         log_loss  COMPLETE  \n4  0 days 00:00:36.333132                  336          entropy  COMPLETE  \n..                    ...                  ...              ...       ...  \n95 0 days 00:00:29.795353                  318             gini  COMPLETE  \n96 0 days 00:00:42.041268                  376         log_loss  COMPLETE  \n97 0 days 00:00:39.072384                  394             gini  COMPLETE  \n98 0 days 00:00:37.100328                  338          entropy  COMPLETE  \n99 0 days 00:00:47.306678                  438         log_loss  COMPLETE  \n\n[100 rows x 8 columns]",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>number</th>\n      <th>value</th>\n      <th>datetime_start</th>\n      <th>datetime_complete</th>\n      <th>duration</th>\n      <th>params_n_estimators</th>\n      <th>params_optimizer</th>\n      <th>state</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0</td>\n      <td>0.798316</td>\n      <td>2022-07-13 13:31:14.773330</td>\n      <td>2022-07-13 13:31:50.249579</td>\n      <td>0 days 00:00:35.476249</td>\n      <td>335</td>\n      <td>entropy</td>\n      <td>COMPLETE</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>0.799242</td>\n      <td>2022-07-13 13:31:50.250580</td>\n      <td>2022-07-13 13:32:38.592176</td>\n      <td>0 days 00:00:48.341596</td>\n      <td>454</td>\n      <td>entropy</td>\n      <td>COMPLETE</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2</td>\n      <td>0.799141</td>\n      <td>2022-07-13 13:32:38.593176</td>\n      <td>2022-07-13 13:33:09.829821</td>\n      <td>0 days 00:00:31.236645</td>\n      <td>288</td>\n      <td>entropy</td>\n      <td>COMPLETE</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>3</td>\n      <td>0.799865</td>\n      <td>2022-07-13 13:33:09.831814</td>\n      <td>2022-07-13 13:33:36.366150</td>\n      <td>0 days 00:00:26.534336</td>\n      <td>247</td>\n      <td>log_loss</td>\n      <td>COMPLETE</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>4</td>\n      <td>0.799024</td>\n      <td>2022-07-13 13:33:36.367152</td>\n      <td>2022-07-13 13:34:12.700284</td>\n      <td>0 days 00:00:36.333132</td>\n      <td>336</td>\n      <td>entropy</td>\n      <td>COMPLETE</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>95</th>\n      <td>95</td>\n      <td>0.799192</td>\n      <td>2022-07-13 14:26:02.745816</td>\n      <td>2022-07-13 14:26:32.541169</td>\n      <td>0 days 00:00:29.795353</td>\n      <td>318</td>\n      <td>gini</td>\n      <td>COMPLETE</td>\n    </tr>\n    <tr>\n      <th>96</th>\n      <td>96</td>\n      <td>0.798670</td>\n      <td>2022-07-13 14:26:32.542130</td>\n      <td>2022-07-13 14:27:14.583398</td>\n      <td>0 days 00:00:42.041268</td>\n      <td>376</td>\n      <td>log_loss</td>\n      <td>COMPLETE</td>\n    </tr>\n    <tr>\n      <th>97</th>\n      <td>97</td>\n      <td>0.798923</td>\n      <td>2022-07-13 14:27:14.584424</td>\n      <td>2022-07-13 14:27:53.656808</td>\n      <td>0 days 00:00:39.072384</td>\n      <td>394</td>\n      <td>gini</td>\n      <td>COMPLETE</td>\n    </tr>\n    <tr>\n      <th>98</th>\n      <td>98</td>\n      <td>0.799040</td>\n      <td>2022-07-13 14:27:53.657803</td>\n      <td>2022-07-13 14:28:30.758131</td>\n      <td>0 days 00:00:37.100328</td>\n      <td>338</td>\n      <td>entropy</td>\n      <td>COMPLETE</td>\n    </tr>\n    <tr>\n      <th>99</th>\n      <td>99</td>\n      <td>0.798805</td>\n      <td>2022-07-13 14:28:30.759131</td>\n      <td>2022-07-13 14:29:18.065809</td>\n      <td>0 days 00:00:47.306678</td>\n      <td>438</td>\n      <td>log_loss</td>\n      <td>COMPLETE</td>\n    </tr>\n  </tbody>\n</table>\n<p>100 rows × 8 columns</p>\n</div>"
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "import optuna\n",
    "\n",
    "n_trials = 100\n",
    "\n",
    "encoder = OrdinalEncoder()\n",
    "x_train = encoder.fit_transform(train_inputs)\n",
    "\n",
    "def rf_objective(trial):\n",
    "    n_estimators = trial.suggest_int('n_estimators', 50, 500)\n",
    "    optimizer = trial.suggest_categorical('optimizer', ['gini', 'entropy', 'log_loss'])\n",
    "    rf_clf = RandomForestClassifier(n_estimators=n_estimators, criterion=optimizer)\n",
    "    avg_acc = cross_val_score(rf_clf, x_train, train_targets, cv=10, n_jobs=-1).mean()\n",
    "    return avg_acc\n",
    "\n",
    "\n",
    "rf_study = optuna.create_study(direction='maximize')\n",
    "rf_study.optimize(rf_objective, n_trials=n_trials)\n",
    "rf_study.trials_dataframe()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "data": [
        {
         "colorbar": {
          "title": {
           "text": "Objective Value"
          }
         },
         "colorscale": [
          [
           0,
           "rgb(5,10,172)"
          ],
          [
           0.35,
           "rgb(40,60,190)"
          ],
          [
           0.5,
           "rgb(70,100,245)"
          ],
          [
           0.6,
           "rgb(90,120,245)"
          ],
          [
           0.7,
           "rgb(106,137,247)"
          ],
          [
           1,
           "rgb(220,220,220)"
          ]
         ],
         "connectgaps": true,
         "contours": {
          "coloring": "heatmap"
         },
         "hoverinfo": "none",
         "line": {
          "smoothing": 1.3
         },
         "reversescale": false,
         "x": [
          41.4,
          63,
          73,
          106,
          132,
          152,
          163,
          168,
          188,
          202,
          213,
          216,
          220,
          228,
          235,
          246,
          247,
          249,
          255,
          261,
          268,
          269,
          271,
          279,
          281,
          285,
          288,
          289,
          295,
          300,
          306,
          310,
          312,
          318,
          321,
          324,
          325,
          328,
          331,
          335,
          336,
          338,
          339,
          342,
          345,
          346,
          347,
          351,
          352,
          353,
          354,
          358,
          359,
          362,
          364,
          368,
          370,
          371,
          372,
          375,
          376,
          380,
          382,
          384,
          386,
          388,
          390,
          394,
          395,
          398,
          400,
          407,
          409,
          416,
          419,
          426,
          427,
          438,
          441,
          447,
          451,
          454,
          457,
          464,
          475,
          485,
          495,
          516.6
         ],
         "y": [
          "entropy",
          "gini",
          "log_loss"
         ],
         "z": [
          [
           null,
           null,
           null,
           0.797946127946128,
           null,
           null,
           0.7981313131313132,
           null,
           null,
           null,
           null,
           null,
           null,
           0.7983838383838384,
           null,
           null,
           null,
           0.7988552188552188,
           null,
           null,
           0.799057239057239,
           null,
           null,
           null,
           0.7992087542087541,
           null,
           0.799141414141414,
           0.7989898989898989,
           null,
           null,
           null,
           0.799074074074074,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           0.7983164983164983,
           0.7990235690235691,
           0.7990404040404041,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           0.7986195286195286,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           0.7990909090909091,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           0.7989898989898989,
           null,
           null,
           null,
           null,
           null,
           null,
           0.7992424242424243,
           null,
           null,
           null,
           null,
           null,
           null
          ],
          [
           null,
           0.7970875420875421,
           0.7974915824915825,
           null,
           null,
           null,
           null,
           null,
           0.7988383838383839,
           0.7988047138047139,
           null,
           null,
           null,
           null,
           0.7984680134680134,
           null,
           null,
           null,
           0.7984006734006734,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           null,
           0.7991919191919192,
           0.7988888888888889,
           0.798939393939394,
           null,
           0.7989225589225589,
           0.7991919191919192,
           0.7987542087542089,
           null,
           null,
           0.7988215488215489,
           0.7986363636363636,
           null,
           null,
           null,
           0.7988215488215489,
           0.7983333333333335,
           null,
           0.799074074074074,
           0.7993602693602695,
           0.7991077441077443,
           null,
           null,
           null,
           0.7995791245791246,
           0.7988047138047139,
           0.7988215488215488,
           0.7987205387205387,
           null,
           0.7991750841750842,
           0.7988552188552188,
           0.7993602693602694,
           0.7996464646464647,
           0.7988383838383839,
           null,
           0.7992255892255892,
           0.7992929292929293,
           null,
           0.7989057239057239,
           0.7988215488215488,
           0.7989225589225589,
           null,
           0.7991414141414142,
           null,
           null,
           0.7982323232323232,
           0.7987373737373737,
           null,
           null,
           0.7985185185185185,
           null,
           0.7985690235690235,
           null,
           0.7992929292929293,
           null,
           0.7991582491582492,
           null,
           null,
           null,
           0.7989562289562289,
           null
          ],
          [
           null,
           null,
           null,
           null,
           0.7980976430976432,
           0.7987542087542089,
           null,
           0.7985185185185186,
           null,
           null,
           0.7986026936026935,
           0.7986868686868688,
           0.7989562289562289,
           null,
           null,
           0.7982828282828283,
           0.7998653198653198,
           null,
           null,
           0.799006734006734,
           null,
           0.7992929292929293,
           0.7994444444444444,
           0.7985185185185185,
           null,
           0.7989898989898989,
           null,
           null,
           null,
           null,
           0.7989393939393941,
           null,
           null,
           null,
           null,
           0.7989057239057239,
           0.7988047138047139,
           null,
           null,
           null,
           null,
           0.798148148148148,
           null,
           null,
           0.7991077441077442,
           null,
           null,
           null,
           0.7988888888888889,
           null,
           0.7989225589225588,
           0.799057239057239,
           null,
           null,
           null,
           0.7985858585858586,
           null,
           null,
           null,
           0.8000841750841751,
           0.7986700336700336,
           0.7994444444444445,
           null,
           null,
           null,
           null,
           null,
           null,
           0.7989562289562289,
           null,
           0.7989898989898989,
           0.7988552188552189,
           null,
           null,
           null,
           0.7981986531986532,
           0.7987710437710438,
           0.7988047138047137,
           null,
           0.7990909090909091,
           null,
           null,
           null,
           0.7985185185185185,
           0.7988383838383839,
           0.7985521885521887,
           null,
           null
          ]
         ],
         "type": "contour"
        },
        {
         "marker": {
          "color": "black",
          "line": {
           "color": "Grey",
           "width": 0.5
          }
         },
         "mode": "markers",
         "showlegend": false,
         "x": [
          335,
          454,
          288,
          247,
          336,
          73,
          235,
          419,
          464,
          353,
          152,
          213,
          438,
          168,
          495,
          386,
          279,
          106,
          216,
          306,
          386,
          268,
          163,
          289,
          228,
          380,
          485,
          400,
          447,
          358,
          325,
          249,
          310,
          358,
          371,
          409,
          457,
          342,
          427,
          475,
          188,
          451,
          416,
          441,
          372,
          370,
          321,
          347,
          339,
          338,
          255,
          388,
          295,
          359,
          339,
          375,
          395,
          354,
          269,
          306,
          132,
          375,
          324,
          382,
          426,
          63,
          407,
          202,
          285,
          364,
          345,
          384,
          376,
          398,
          328,
          235,
          351,
          368,
          312,
          300,
          427,
          271,
          261,
          246,
          220,
          281,
          407,
          331,
          352,
          376,
          394,
          390,
          362,
          416,
          346,
          318,
          376,
          394,
          338,
          438
         ],
         "y": [
          "entropy",
          "entropy",
          "entropy",
          "log_loss",
          "entropy",
          "gini",
          "gini",
          "entropy",
          "log_loss",
          "entropy",
          "log_loss",
          "log_loss",
          "log_loss",
          "log_loss",
          "gini",
          "entropy",
          "log_loss",
          "entropy",
          "log_loss",
          "gini",
          "entropy",
          "entropy",
          "entropy",
          "entropy",
          "entropy",
          "log_loss",
          "log_loss",
          "log_loss",
          "log_loss",
          "log_loss",
          "log_loss",
          "entropy",
          "entropy",
          "gini",
          "gini",
          "gini",
          "gini",
          "gini",
          "gini",
          "log_loss",
          "gini",
          "gini",
          "gini",
          "gini",
          "gini",
          "gini",
          "gini",
          "gini",
          "gini",
          "log_loss",
          "gini",
          "gini",
          "gini",
          "gini",
          "gini",
          "log_loss",
          "log_loss",
          "log_loss",
          "log_loss",
          "log_loss",
          "log_loss",
          "gini",
          "log_loss",
          "gini",
          "log_loss",
          "gini",
          "log_loss",
          "gini",
          "log_loss",
          "gini",
          "log_loss",
          "gini",
          "gini",
          "gini",
          "gini",
          "gini",
          "gini",
          "log_loss",
          "gini",
          "gini",
          "log_loss",
          "log_loss",
          "log_loss",
          "log_loss",
          "log_loss",
          "entropy",
          "log_loss",
          "gini",
          "log_loss",
          "gini",
          "gini",
          "gini",
          "gini",
          "gini",
          "gini",
          "gini",
          "log_loss",
          "gini",
          "entropy",
          "log_loss"
         ],
         "type": "scatter"
        }
       ],
       "layout": {
        "title": {
         "text": "Contour Plot"
        },
        "template": {
         "data": {
          "histogram2dcontour": [
           {
            "type": "histogram2dcontour",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "choropleth": [
           {
            "type": "choropleth",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            }
           }
          ],
          "histogram2d": [
           {
            "type": "histogram2d",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "heatmap": [
           {
            "type": "heatmap",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "heatmapgl": [
           {
            "type": "heatmapgl",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "contourcarpet": [
           {
            "type": "contourcarpet",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            }
           }
          ],
          "contour": [
           {
            "type": "contour",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "surface": [
           {
            "type": "surface",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "mesh3d": [
           {
            "type": "mesh3d",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            }
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "parcoords": [
           {
            "type": "parcoords",
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scatterpolargl": [
           {
            "type": "scatterpolargl",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "scattergeo": [
           {
            "type": "scattergeo",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scatterpolar": [
           {
            "type": "scatterpolar",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "scattergl": [
           {
            "type": "scattergl",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scatter3d": [
           {
            "type": "scatter3d",
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scattermapbox": [
           {
            "type": "scattermapbox",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scatterternary": [
           {
            "type": "scatterternary",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scattercarpet": [
           {
            "type": "scattercarpet",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ]
         },
         "layout": {
          "autotypenumbers": "strict",
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "hovermode": "closest",
          "hoverlabel": {
           "align": "left"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "bgcolor": "#E5ECF6",
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "ternary": {
           "bgcolor": "#E5ECF6",
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "sequential": [
            [
             0.0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1.0,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0.0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1.0,
             "#f0f921"
            ]
           ],
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ]
          },
          "xaxis": {
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "automargin": true,
           "zerolinewidth": 2
          },
          "yaxis": {
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "automargin": true,
           "zerolinewidth": 2
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white",
            "gridwidth": 2
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white",
            "gridwidth": 2
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white",
            "gridwidth": 2
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "geo": {
           "bgcolor": "white",
           "landcolor": "#E5ECF6",
           "subunitcolor": "white",
           "showland": true,
           "showlakes": true,
           "lakecolor": "white"
          },
          "title": {
           "x": 0.05
          },
          "mapbox": {
           "style": "light"
          }
         }
        },
        "xaxis": {
         "title": {
          "text": "n_estimators"
         },
         "range": [
          41.4,
          516.6
         ]
        },
        "yaxis": {
         "title": {
          "text": "optimizer"
         },
         "range": [
          -0.1,
          2.1
         ],
         "type": "category"
        }
       },
       "config": {
        "plotlyServerURL": "https://plot.ly"
       }
      },
      "text/html": "<div>                            <div id=\"4bfeec3d-5061-4419-927f-7865d9161ba3\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"4bfeec3d-5061-4419-927f-7865d9161ba3\")) {                    Plotly.newPlot(                        \"4bfeec3d-5061-4419-927f-7865d9161ba3\",                        [{\"colorbar\":{\"title\":{\"text\":\"Objective Value\"}},\"colorscale\":[[0,\"rgb(5,10,172)\"],[0.35,\"rgb(40,60,190)\"],[0.5,\"rgb(70,100,245)\"],[0.6,\"rgb(90,120,245)\"],[0.7,\"rgb(106,137,247)\"],[1,\"rgb(220,220,220)\"]],\"connectgaps\":true,\"contours\":{\"coloring\":\"heatmap\"},\"hoverinfo\":\"none\",\"line\":{\"smoothing\":1.3},\"reversescale\":false,\"x\":[41.4,63,73,106,132,152,163,168,188,202,213,216,220,228,235,246,247,249,255,261,268,269,271,279,281,285,288,289,295,300,306,310,312,318,321,324,325,328,331,335,336,338,339,342,345,346,347,351,352,353,354,358,359,362,364,368,370,371,372,375,376,380,382,384,386,388,390,394,395,398,400,407,409,416,419,426,427,438,441,447,451,454,457,464,475,485,495,516.6],\"y\":[\"entropy\",\"gini\",\"log_loss\"],\"z\":[[null,null,null,0.797946127946128,null,null,0.7981313131313132,null,null,null,null,null,null,0.7983838383838384,null,null,null,0.7988552188552188,null,null,0.799057239057239,null,null,null,0.7992087542087541,null,0.799141414141414,0.7989898989898989,null,null,null,0.799074074074074,null,null,null,null,null,null,null,0.7983164983164983,0.7990235690235691,0.7990404040404041,null,null,null,null,null,null,null,0.7986195286195286,null,null,null,null,null,null,null,null,null,null,null,null,null,null,0.7990909090909091,null,null,null,null,null,null,null,null,null,0.7989898989898989,null,null,null,null,null,null,0.7992424242424243,null,null,null,null,null,null],[null,0.7970875420875421,0.7974915824915825,null,null,null,null,null,0.7988383838383839,0.7988047138047139,null,null,null,null,0.7984680134680134,null,null,null,0.7984006734006734,null,null,null,null,null,null,null,null,null,0.7991919191919192,0.7988888888888889,0.798939393939394,null,0.7989225589225589,0.7991919191919192,0.7987542087542089,null,null,0.7988215488215489,0.7986363636363636,null,null,null,0.7988215488215489,0.7983333333333335,null,0.799074074074074,0.7993602693602695,0.7991077441077443,null,null,null,0.7995791245791246,0.7988047138047139,0.7988215488215488,0.7987205387205387,null,0.7991750841750842,0.7988552188552188,0.7993602693602694,0.7996464646464647,0.7988383838383839,null,0.7992255892255892,0.7992929292929293,null,0.7989057239057239,0.7988215488215488,0.7989225589225589,null,0.7991414141414142,null,null,0.7982323232323232,0.7987373737373737,null,null,0.7985185185185185,null,0.7985690235690235,null,0.7992929292929293,null,0.7991582491582492,null,null,null,0.7989562289562289,null],[null,null,null,null,0.7980976430976432,0.7987542087542089,null,0.7985185185185186,null,null,0.7986026936026935,0.7986868686868688,0.7989562289562289,null,null,0.7982828282828283,0.7998653198653198,null,null,0.799006734006734,null,0.7992929292929293,0.7994444444444444,0.7985185185185185,null,0.7989898989898989,null,null,null,null,0.7989393939393941,null,null,null,null,0.7989057239057239,0.7988047138047139,null,null,null,null,0.798148148148148,null,null,0.7991077441077442,null,null,null,0.7988888888888889,null,0.7989225589225588,0.799057239057239,null,null,null,0.7985858585858586,null,null,null,0.8000841750841751,0.7986700336700336,0.7994444444444445,null,null,null,null,null,null,0.7989562289562289,null,0.7989898989898989,0.7988552188552189,null,null,null,0.7981986531986532,0.7987710437710438,0.7988047138047137,null,0.7990909090909091,null,null,null,0.7985185185185185,0.7988383838383839,0.7985521885521887,null,null]],\"type\":\"contour\"},{\"marker\":{\"color\":\"black\",\"line\":{\"color\":\"Grey\",\"width\":0.5}},\"mode\":\"markers\",\"showlegend\":false,\"x\":[335,454,288,247,336,73,235,419,464,353,152,213,438,168,495,386,279,106,216,306,386,268,163,289,228,380,485,400,447,358,325,249,310,358,371,409,457,342,427,475,188,451,416,441,372,370,321,347,339,338,255,388,295,359,339,375,395,354,269,306,132,375,324,382,426,63,407,202,285,364,345,384,376,398,328,235,351,368,312,300,427,271,261,246,220,281,407,331,352,376,394,390,362,416,346,318,376,394,338,438],\"y\":[\"entropy\",\"entropy\",\"entropy\",\"log_loss\",\"entropy\",\"gini\",\"gini\",\"entropy\",\"log_loss\",\"entropy\",\"log_loss\",\"log_loss\",\"log_loss\",\"log_loss\",\"gini\",\"entropy\",\"log_loss\",\"entropy\",\"log_loss\",\"gini\",\"entropy\",\"entropy\",\"entropy\",\"entropy\",\"entropy\",\"log_loss\",\"log_loss\",\"log_loss\",\"log_loss\",\"log_loss\",\"log_loss\",\"entropy\",\"entropy\",\"gini\",\"gini\",\"gini\",\"gini\",\"gini\",\"gini\",\"log_loss\",\"gini\",\"gini\",\"gini\",\"gini\",\"gini\",\"gini\",\"gini\",\"gini\",\"gini\",\"log_loss\",\"gini\",\"gini\",\"gini\",\"gini\",\"gini\",\"log_loss\",\"log_loss\",\"log_loss\",\"log_loss\",\"log_loss\",\"log_loss\",\"gini\",\"log_loss\",\"gini\",\"log_loss\",\"gini\",\"log_loss\",\"gini\",\"log_loss\",\"gini\",\"log_loss\",\"gini\",\"gini\",\"gini\",\"gini\",\"gini\",\"gini\",\"log_loss\",\"gini\",\"gini\",\"log_loss\",\"log_loss\",\"log_loss\",\"log_loss\",\"log_loss\",\"entropy\",\"log_loss\",\"gini\",\"log_loss\",\"gini\",\"gini\",\"gini\",\"gini\",\"gini\",\"gini\",\"gini\",\"log_loss\",\"gini\",\"entropy\",\"log_loss\"],\"type\":\"scatter\"}],                        {\"title\":{\"text\":\"Contour Plot\"},\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"xaxis\":{\"title\":{\"text\":\"n_estimators\"},\"range\":[41.4,516.6]},\"yaxis\":{\"title\":{\"text\":\"optimizer\"},\"range\":[-0.1,2.1],\"type\":\"category\"}},                        {\"responsive\": true}                    ).then(function(){\n                            \nvar gd = document.getElementById('4bfeec3d-5061-4419-927f-7865d9161ba3');\nvar x = new MutationObserver(function (mutations, observer) {{\n        var display = window.getComputedStyle(gd).display;\n        if (!display || display === 'none') {{\n            console.log([gd, 'removed!']);\n            Plotly.purge(gd);\n            observer.disconnect();\n        }}\n}});\n\n// Listen for the removal of the full notebook cells\nvar notebookContainer = gd.closest('#notebook-container');\nif (notebookContainer) {{\n    x.observe(notebookContainer, {childList: true});\n}}\n\n// Listen for the clearing of the current output cell\nvar outputEl = gd.closest('.output');\nif (outputEl) {{\n    x.observe(outputEl, {childList: true});\n}}\n\n                        })                };                });            </script>        </div>"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "optuna.visualization.plot_contour(rf_study, params=['n_estimators', 'optimizer'])"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "data": [
        {
         "cliponaxis": false,
         "hovertemplate": [
          "optimizer (CategoricalDistribution): 0.019533725768447825<extra></extra>",
          "n_estimators (IntUniformDistribution): 0.9804662742315522<extra></extra>"
         ],
         "marker": {
          "color": "rgb(66,146,198)"
         },
         "orientation": "h",
         "text": [
          "0.019533725768447825",
          "0.9804662742315522"
         ],
         "textposition": "outside",
         "texttemplate": "%{text:.2f}",
         "x": [
          0.019533725768447825,
          0.9804662742315522
         ],
         "y": [
          "optimizer",
          "n_estimators"
         ],
         "type": "bar"
        }
       ],
       "layout": {
        "showlegend": false,
        "title": {
         "text": "Hyperparameter Importances"
        },
        "xaxis": {
         "title": {
          "text": "Importance for Objective Value"
         }
        },
        "yaxis": {
         "title": {
          "text": "Hyperparameter"
         }
        },
        "template": {
         "data": {
          "histogram2dcontour": [
           {
            "type": "histogram2dcontour",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "choropleth": [
           {
            "type": "choropleth",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            }
           }
          ],
          "histogram2d": [
           {
            "type": "histogram2d",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "heatmap": [
           {
            "type": "heatmap",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "heatmapgl": [
           {
            "type": "heatmapgl",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "contourcarpet": [
           {
            "type": "contourcarpet",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            }
           }
          ],
          "contour": [
           {
            "type": "contour",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "surface": [
           {
            "type": "surface",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0.0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1.0,
              "#f0f921"
             ]
            ]
           }
          ],
          "mesh3d": [
           {
            "type": "mesh3d",
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            }
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "parcoords": [
           {
            "type": "parcoords",
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scatterpolargl": [
           {
            "type": "scatterpolargl",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "scattergeo": [
           {
            "type": "scattergeo",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scatterpolar": [
           {
            "type": "scatterpolar",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "scattergl": [
           {
            "type": "scattergl",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scatter3d": [
           {
            "type": "scatter3d",
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scattermapbox": [
           {
            "type": "scattermapbox",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scatterternary": [
           {
            "type": "scatterternary",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "scattercarpet": [
           {
            "type": "scattercarpet",
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            }
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ]
         },
         "layout": {
          "autotypenumbers": "strict",
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "hovermode": "closest",
          "hoverlabel": {
           "align": "left"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "bgcolor": "#E5ECF6",
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "ternary": {
           "bgcolor": "#E5ECF6",
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "sequential": [
            [
             0.0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1.0,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0.0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1.0,
             "#f0f921"
            ]
           ],
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ]
          },
          "xaxis": {
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "automargin": true,
           "zerolinewidth": 2
          },
          "yaxis": {
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "automargin": true,
           "zerolinewidth": 2
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white",
            "gridwidth": 2
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white",
            "gridwidth": 2
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white",
            "gridwidth": 2
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "geo": {
           "bgcolor": "white",
           "landcolor": "#E5ECF6",
           "subunitcolor": "white",
           "showland": true,
           "showlakes": true,
           "lakecolor": "white"
          },
          "title": {
           "x": 0.05
          },
          "mapbox": {
           "style": "light"
          }
         }
        }
       },
       "config": {
        "plotlyServerURL": "https://plot.ly"
       }
      },
      "text/html": "<div>                            <div id=\"39789cf6-5e29-4114-ac2a-ada6650ae211\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                require([\"plotly\"], function(Plotly) {                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"39789cf6-5e29-4114-ac2a-ada6650ae211\")) {                    Plotly.newPlot(                        \"39789cf6-5e29-4114-ac2a-ada6650ae211\",                        [{\"cliponaxis\":false,\"hovertemplate\":[\"optimizer (CategoricalDistribution): 0.019533725768447825<extra></extra>\",\"n_estimators (IntUniformDistribution): 0.9804662742315522<extra></extra>\"],\"marker\":{\"color\":\"rgb(66,146,198)\"},\"orientation\":\"h\",\"text\":[\"0.019533725768447825\",\"0.9804662742315522\"],\"textposition\":\"outside\",\"texttemplate\":\"%{text:.2f}\",\"x\":[0.019533725768447825,0.9804662742315522],\"y\":[\"optimizer\",\"n_estimators\"],\"type\":\"bar\"}],                        {\"showlegend\":false,\"title\":{\"text\":\"Hyperparameter Importances\"},\"xaxis\":{\"title\":{\"text\":\"Importance for Objective Value\"}},\"yaxis\":{\"title\":{\"text\":\"Hyperparameter\"}},\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}}},                        {\"responsive\": true}                    ).then(function(){\n                            \nvar gd = document.getElementById('39789cf6-5e29-4114-ac2a-ada6650ae211');\nvar x = new MutationObserver(function (mutations, observer) {{\n        var display = window.getComputedStyle(gd).display;\n        if (!display || display === 'none') {{\n            console.log([gd, 'removed!']);\n            Plotly.purge(gd);\n            observer.disconnect();\n        }}\n}});\n\n// Listen for the removal of the full notebook cells\nvar notebookContainer = gd.closest('#notebook-container');\nif (notebookContainer) {{\n    x.observe(notebookContainer, {childList: true});\n}}\n\n// Listen for the clearing of the current output cell\nvar outputEl = gd.closest('.output');\nif (outputEl) {{\n    x.observe(outputEl, {childList: true});\n}}\n\n                        })                };                });            </script>        </div>"
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "optuna.visualization.plot_param_importances(rf_study)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Default Random Classifier vs Tuned Random Forest"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "outputs": [],
   "source": [
    "clf = RandomForestClassifier(random_state=random_state, n_jobs=-1)\n",
    "tuned_clf = RandomForestClassifier(n_estimators=375, criterion='log_loss', random_state=random_state, n_jobs=-1)\n",
    "\n",
    "encoder = OrdinalEncoder()\n",
    "x_train = encoder.fit_transform(train_inputs)\n",
    "x_test = encoder.transform(test_inputs)\n",
    "\n",
    "trials = {'Default-RF': clf, 'Tuned-RF': tuned_clf}\n",
    "target_labels = {0: 'functional', 1: 'functional needs repair', 2: 'non functional'}\n",
    "\n",
    "for clf_name, rf_clf in trials.items():\n",
    "    rf_clf.fit(x_train, train_targets)\n",
    "    y_pred = rf_clf.predict(x_test)\n",
    "\n",
    "    submission_df = pd.DataFrame({\n",
    "        'id': test_input_ids,\n",
    "        'status_group': y_pred\n",
    "    })\n",
    "    submission_df['status_group'] = submission_df['status_group'].replace(target_labels)\n",
    "    submission_df.to_csv(f'submission-{clf_name}.csv', index=False)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Test Accuracy\n",
    "\n",
    "The classifiers were evaluated on the \"*drivendata.org*\" platform"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "| Classifier            | Accuracy   |\n",
    "|-----------------------|------------|\n",
    "| Default Random Forest | 0.7977     |\n",
    "| Tuned Random Forest   | **0.8001** |"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [],
   "metadata": {
    "collapsed": false
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}