{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a554392b",
   "metadata": {},
   "source": [
    "# Lesson 8 Exercise\n",
    "\n",
    "This exercise will be short and sweet to top off the series of exercise sheets for this Python tutorial series.\n",
    "\n",
    "We will be comparing the performance of two classifiers that you learned about (k-Nearest Neighbor and Logistic Regression) on a new dataset that you haven't worked with before: the 8x8 pixel images of handwritten digits (0-9) dataset.\n",
    "\n",
    "Then, at the end, we will create an interesting visualization that show the Logistic Regression classifier's learned weights for each pixel in the images, hopefully providing you with a more intuitive understanding of how this classifier works."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "859f293c",
   "metadata": {},
   "source": [
    "# Part 0: Setup and necessary Python modules"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "597cca7b",
   "metadata": {},
   "source": [
    "Please be sure to use the Python virtual environment you've set up for this course. Pandas, Numpy, Matplotlib, and Scikit-learn should already be installed if you followed the tutorials.\n",
    "\n",
    "No additional packages are necessary for this exercise. Just make sure you have the aforementioned packages installed. You can install all of them by running the code block below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fd0141f",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "!pip install pandas numpy matplotlib scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45829df5",
   "metadata": {},
   "source": [
    "If that code block didn't work, just run the following command in your terminal:\n",
    "```\n",
    "pip install pandas numpy matplotlib scikit-learn\n",
    "```\n",
    "\n",
    "Next, import the necessary Python modules by running the following code block:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abc7ef47",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69c885fa",
   "metadata": {},
   "source": [
    "# Part 1: Comparing classifiers on 8x8 digits dataset \n",
    "\n",
    "## Loading and splitting the dataset\n",
    "\n",
    "**Dataset:** Run the code block below to load the 8x8 digits dataset from Scikit-learn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb47b588",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits\n",
    "digits = load_digits()\n",
    "X, y = digits.data, digits.target"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22c8529c",
   "metadata": {},
   "source": [
    "Great, now that we have the dataset loaded, let's take a look at its structure.\n",
    "\n",
    "**Task:** Use the code block below to understand the datatype, shape, and structure of the input data `X` and the target labels `y`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e56a3b1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO task"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ae108fd",
   "metadata": {},
   "source": [
    "As you know, it is always crucial to split the data into a training set and a test set before training any machine learning model. This allows us to evaluate the model's performance on unseen data.\n",
    "\n",
    "**Task:** Split the dataset into training and test sets using an 80-20 split. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4bcf6f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# HINT BLOCK - Run this to show a hint if you're struggling with this part of the exercise sheet\n",
    "print(\"\".join(chr(ord(c) - 1) for c in \"Vtf!uif!usbjo`uftu`tqmju!gvodujpo!xjui!uif!lfzxpse!bshvnfou!uftu`tj{f>1/3/\\x0bUif!sftvmujoh!pvuqvu!tipvme!cf!nbqqfe!up!Y`usbjo-!Y`uftu-!z`usbjo-!boe!z`uftu-!jo!uibu!psefs/\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa9dd3f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO task"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72d0f370",
   "metadata": {},
   "source": [
    "### Training the two models and comparing performance\n",
    "\n",
    "With the dataset in hand, let's compare the performance of the k-Nearest Neighbors and the Logistic Regression classifiers. These are two simple machine learning models for classification that you learned about in the lesson.\n",
    "\n",
    "> **Supplementary note: How are we able to classify 10 digits with Logistic Regression?**  \n",
    "> You may remember that we discussed that logistic regression models only work with binary classification (classification of samples into one of two classes). How then, are we able to use a logistic regression model to classify digits into one of 10 classes (one class for each digit between 0 and 9)?\n",
    ">\n",
    "> The fact that logistic regression models can only categorize samples into two classes remains true. The answer to utilizing more than two classes lies in a strategy called **One-vs-Rest (OvR)**. Instead of trying to solve a 10-way problem all at once, the algorithm essentially trains 10 separate binary classifiers simultaneously—one for each digit. For instance, the \"Digit 0\" classifier is trained to classify a digit as \"zero\" or \"something other than zero\", and so on for each digit.\n",
    ">\n",
    "> Scikit-learn automatically applies this one-vs-rest approach when we create a model with the `LogisticRegression` class and train it on a dataset with more than two classes.\n",
    "\n",
    "**Run:** Simply run the code block below to import the predefined models from Scikit-learn. You will still need to call the model classes to create the model objects later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7721c01e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.neighbors import KNeighborsClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d43244e",
   "metadata": {},
   "source": [
    "As you recall, both of these models have a `fit()` method for training on the training data and a `score()` method for evaluating performance on the test data.\n",
    "\n",
    "**Task:** Train both classifiers on the training data and evaluate their accuracy on the test data. Print out the accuracy scores for both models. Play around with different values of `n_neighbors` for the k-NN classifier to give it the best possible performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5395f65",
   "metadata": {},
   "outputs": [],
   "source": [
    "# HINT BLOCK - Run this to show a hint if you're struggling with this part of the exercise sheet\n",
    "print(\"\".join(chr(ord(c) - 1) for c in \"Uifsf!bsf!uisff!tufqt!up!tpmwjoh!uijt!qspcmfn;\\x0b\\x0b2/!Dsfbuf!bo!jotubodf!pg!fbdi!npefm!)f/h/!mphjtujd`npefm!>!MphjtujdSfhsfttjpo)**\\x0b3/!Usbjo!fbdi!npefm!vtjoh!npefm/gju)Y`usbjo-!z`usbjo*\\x0b4/!Fwbmvbuf!fbdi!npefm(t!qfsgpsnbodf!po!votffo!ebub!vtjoh!npefm/tdpsf)*-!xijdi!xpslt!tjnjmbsmz!cvu!zpv!offe!up!gjhvsf!pvu!ipx!up!qbtt!uif!uftu!ebub!jotufbe!pg!usbjo!ebub/\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90205853",
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO task"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "120e1316",
   "metadata": {},
   "source": [
    "**Note:** If you got an error about the logistic regression not converging, try setting the `max_iter` keyword argument when creating the Logistic Regression model object. For example, you can set `max_iter=200` or even higher if necessary. When a model doesn't converge, it essentially means that the optimization algorithm used to find the best parameters for the model did not reach a solution within the default number of iterations. Increasing `max_iter` allows the algorithm more iterations to converge.\n",
    "\n",
    "Now that you've trained the models, consider the following questions:\n",
    "\n",
    "**Question:** Which of your models reached the highest accuracy on the test set? Did they perform similarly? Does running the entire notebook using the \"Run All Cells\" option yield different results each time?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a302340c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ANSWER BLOCK - Run this to reveal the correct answer\n",
    "print(\"\".join(chr(ord(c) - 1) for c in \"Uif!npefmt!tipvme!cpui!dpotjtufoumz!ibwf!cfuufs!uibo!:6&!qfsgpsnbodf-!bmuipvhi!uif!fybdu!bddvsbdz!wbmvft!xjmm!gmvduvbuf!b!cju!xifo!zpv!sf.svo!bmm!dfmmt!jo!uif!Opufcppl/\\x0bUiftf!gmvduvbujpot!bsf!dbvtfe!cz!ejggfsfou!tqmjut!jo!uif!ebub-!bt!ejtdvttfe!jo!uif!mfttpo/\\x0bJo!hfofsbm-!uif!l.OO!dmbttjgjfs!tipvme!fyijcju!tmjhiumz!ijhifs!bddvsbdz!uibo!uif!mphjtujd!sfhsfttjpo!dmbttjgjfs/\\x0bJg!opu-!zpv!nbz!ibwf!epof!tpnfuijoh!xspoh!jo!zpvs!dpef/\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ab0bef0",
   "metadata": {},
   "source": [
    "# Part 2: Visualizing involving the 8x8 digits dataset\n",
    "\n",
    "In this section, we will create two visualizations related to the 8x8 digits dataset and the Logistic Regression classifier. We will not use the k-Nearest Neighbors classifier for this part of the exercise sheet.\n",
    "\n",
    "## Visualizing entries in the dataset itself\n",
    "\n",
    "So far, you've trained and compared classifiers on the 8x8 handwritten digits dataset. But we still don't really have an idea of what the data looks like. It would be nice if we could actually see some of the handwritten digits so we know what the model is working with.\n",
    "\n",
    "**Task:** You don't have to edit the logic in the code block below. Simply edit the model variable name below to match the name you gave your Logistic Regression models in the previous section. Change the name of the `X_test` and `y_test` variables in as well if you happened to use different names when you split the data.\n",
    "\n",
    "**Run:** Run the code block below to display a random digit from the test data and the model's prediction for that digit. Feel free to run multiple times to see different digits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1eca11e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO task (just change the variable names, no need to modify the logic)\n",
    "\n",
    "# model prediction for random sample\n",
    "import random\n",
    "random_sample_index = random.randint(0, X_test.shape[0] - 1)\n",
    "random_sample = X_test[random_sample_index: random_sample_index + 1]\n",
    "prediction = logistic_model.predict(random_sample)[0]  # change model name here if necessary\n",
    "print(f\"Target label: {y_test[random_sample_index]}\")\n",
    "print(f\"Model prediction: {prediction}\")\n",
    "\n",
    "# visualization of actual digit\n",
    "sample_image = random_sample.reshape(8, 8)\n",
    "print(f\"\\n Visualization of digit given to the model for classification:\")\n",
    "plt.imshow(sample_image, cmap=\"gray\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7778e5b",
   "metadata": {},
   "source": [
    "## Visualizing the parameters learned by the model\n",
    "\n",
    "In the previous section, we saw some of the 8x8 handwritten digits that were fed to the model for classification. In this part, we will visualize the what the model has learned to look for in each digit.\n",
    "\n",
    "> **Supplementary note: the mathematics behind Logistic Regression models**\n",
    ">\n",
    "> This is a quick refresher on the mathematical foundations behind **Logistic Regression** model.\n",
    "> As you may remember from the lesson, they constantly adjust weights $w_0$ to $w_n$ and a bias $b$ during training in order to provide the best classification for a sample of $n$ features.\n",
    "> The classification is issued using the following equation, in which $\\hat{y}$ represents the probability in the range (0, 1) of a sample belonging to a particular class:  \n",
    "> $\\hat{y} = \\sigma(b + w_1x_1 + w_2x_2 + ... + w_nx_n)$\n",
    ">\n",
    "> In this instance, those $n$ features are the values of pixels in the 8x8 digits.\n",
    "> This is to say, the model you trained is learning 64 weights $w_0$ to $w_{64}$ and a bias $b$ to best fit the data, and each of those weights is explicitly associated with one of the 64 pixels in the handwritten digits.\n",
    ">\n",
    "> In this exercise, we will visualize those weights graphically by showing positive weights as blue and negative weights as red. Specifically, we will visualize the learned weights of each of the ten logistic regression models trained using the **one-vs-rest** strategy discussed in the previous supplementary note."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0715698",
   "metadata": {},
   "source": [
    "**Task:** As before, change the name of the model variable to match what you used in your code when you created your Logistic Regression model. Change any other variable names like `X_train`, etc. if you used a different naming convention when you defined your variables.\n",
    "\n",
    "**Run:** Once you change the variable names to match yours, just run the code block below to view the visualizations of the learned model weights for each digit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0cf9ac28",
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO task (just change the variable names, no need to modify the logic)\n",
    "\n",
    "fig, axes = plt.subplots(2, 5, figsize=(12, 6))\n",
    "for i, ax in enumerate(axes.ravel()):\n",
    "    coef_image = logistic_model.coef_[i].reshape(8, 8)  # change the model name here if necessary\n",
    "    \n",
    "    ax.imshow(coef_image, cmap='RdBu', interpolation='nearest', vmin=-0.5, vmax=0.5)\n",
    "    ax.set_title(f'Digit: {i}')\n",
    "    ax.set_xticks([])\n",
    "    ax.set_yticks([])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c371031e",
   "metadata": {},
   "source": [
    "And there you have it! Looking at the heatmap for each digit, do you see any correlation between that heatmap and what the actual digit looks like?\n",
    "\n",
    "For most of the digits -- not really. This is because the model is understanding the numbers in a different way than we do as humans. Instead of looking for patterns that define the shape of each digit (like loops, curves, and angles), it is literally just calculating the influence of a particular pixel's being filled-in on the probability of the whole sample being a particular number.\n",
    "\n",
    "> **Supplementary note: Convolutional Neural Networks**  \n",
    "> The approach to classifying samples described above is actually a major limitation and one of the reasons we just trained our model on 8x8 digits. When the numbers are in a 8x8 space, there is only a small amount of flexibility on how a number can be draw, so this \"brute force\" approach to analyzing the pixels is acceptable. But as soon as we work with, say, 16x16 grids of pixels, a logistic regression model becomes a terrible choice. It would not be able to, for example, recognize a number if it is drawn in just the corner of a larger grid.\n",
    ">\n",
    "> For more complex image classification tasks, we would definitely want to use a **Convolutional Neural Network**, which is much more complicated (i.e. outside the scope of this course) and which uses clever tricks to look for patterns that make up the shapes of numbers (loops, curves, angles, etc.) with less dependency on the exact locations of those patterns on a pixel grid.\n",
    "\n",
    "I hope this shorter exercise sheet was engaging for you and that you learned a thing or two about machine learning beyond what was covered in the lesson!\n",
    "\n",
    "If you have any feedback, send it to:\n",
    "[py.ldv@xcit.tum.de](mailto:py.ldv@xcit.tum.de)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "adlr",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}