{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00bf530",
   "metadata": {},
   "source": [
    "# Homework 4\n",
    "\n",
    "This homework asks you to perform various experiments with k-nearest neighbors and related algorithms. \n",
    "\n",
    "The dataset is the same real estate dataset we previously used from:\n",
    "\n",
    "https://www.kaggle.com/datasets/mirbektoktogaraev/madrid-real-estate-market\n",
    "\n",
    "You will write code and discussion into code and text cells in this notebook. \n",
    "\n",
    "If a code block starts with TODO:, this means that you need to write something there. \n",
    "\n",
    "There are also markdown blocks with questions. Write the answers to these questions in the specified locations.\n",
    "\n",
    "Some code had been written for you to guide the project. Don't change the already written code.\n",
    "\n",
    "## Grading\n",
    "The points add up to 10. Extensive partial credit will be offered. Thus, make sure that you are at least attempting all problems. \n",
    "\n",
    "Make sure to comment your code, such that the grader can understand what different components are doing or attempting to do. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a440a191",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import sklearn.linear_model\n",
    "import sklearn.metrics\n",
    "import sklearn.neighbors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "691d1a9f",
   "metadata": {},
   "source": [
    "## A. Setup for the single variable part\n",
    "\n",
    "In this part we are creating training data and test data in a single variable setting. The explanatory variable ``sq_mt_built`` is used to predict the price of the house ``buy_price``. \n",
    "\n",
    "This part is had been done for you, such that the work does not depend on you importing parts from the previous projects. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "902ccefe",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"houses_Madrid.csv\")\n",
    "#print(f\"The lenght {len(df.index)}\")\n",
    "#print(f\"The columns of the database {df.columns}\")\n",
    "dfsel = df[[\"sq_mt_built\", \"buy_price\"]]\n",
    "dfselnona = dfsel.dropna()\n",
    "dfselnona.plot.scatter(x=\"sq_mt_built\", y=\"buy_price\")\n",
    "df_shuffled = dfselnona.sample(frac=1) # shuffle the rows\n",
    "x = df_shuffled[[\"sq_mt_built\"]].to_numpy(dtype=np.float64)\n",
    "y = df_shuffled[\"buy_price\"].to_numpy(dtype=np.float64)\n",
    "training_data_x = x[:16000]\n",
    "training_data_y = y[:16000]\n",
    "test_data_x = x[16000:]\n",
    "test_data_y = y[16000:]\n",
    "print(f\"Training data is composed of {len(training_data_x)} samples.\")\n",
    "print(f\"Test data is composed of {len(test_data_x)} samples.\")\n",
    "\n",
    "print(f\"Number of NaN-s in training data x: {np.count_nonzero(np.isnan(training_data_x))}\")\n",
    "print(f\"Number of NaN-s in training data y: {np.count_nonzero(np.isnan(training_data_y))}\")\n",
    "print(f\"Number of NaN-s in test data x: {np.count_nonzero(np.isnan(test_data_x))}\")\n",
    "print(f\"Number of NaN-s in test data y: {np.count_nonzero(np.isnan(test_data_y))}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59b7fd82",
   "metadata": {},
   "source": [
    "# B. Establishing an error baseline for linear regression\n",
    "\n",
    "Here we are going to implement the regresssion using linear regression. We also check the performance of the resulting regressor, and print the error. \n",
    "\n",
    "This part is had been done for you, such that the work does not depend on you importing parts from the previous projects. \n",
    "\n",
    "You will need to adapt this for the other models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2a89694",
   "metadata": {},
   "outputs": [],
   "source": [
    "# training the linear regressor\n",
    "regressor = sklearn.linear_model.LinearRegression().fit(training_data_x, training_data_y)\n",
    "# We will create the predictions yhat for every x from the training data. We will do this one at a time. This is not an efficient way to do it, but it allows you to write and debug functions that return a scalar number\n",
    "yhats = []\n",
    "for x in test_data_x:\n",
    "    yhat = regressor.predict([x])\n",
    "    yhats.append(yhat[0])\n",
    "\n",
    "# Now, print some examples of the quality of the classifier\n",
    "examples = [45, 67, 170, 189, 207]\n",
    "for i in examples:\n",
    "    x = test_data_x[i]\n",
    "    y = test_data_y[i]\n",
    "    yhat = regressor.predict([x])[0]\n",
    "    print(f\"House {i} with {x} sqmt was sold for {y} euros, but our system predicted {yhat:.2f}\")\n",
    "\n",
    "# Now calculate the root mean square error on the resulting arrays\n",
    "error = sklearn.metrics.mean_squared_error(yhats, test_data_y, squared=False)\n",
    "print(f\"The mean square error of the linear regression is {error:.2f} euro\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9c67e1e",
   "metadata": {},
   "source": [
    "# P1: K-nearest neighbors (5 points)\n",
    "\n",
    "Implement the k-nearest neighbors algorithm in a function called k_nearest_neighbor_predict.\n",
    "\n",
    "You do not necessarily need to do anything for training, but if you choose to do something, then return the created data structures from the training function, and pass it to the prediction function. \n",
    "\n",
    "Experiment with the function you have implemented for at least the k values 1, 3 and 15. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46998eda",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO implement the nearest neighbor algorithm here\n",
    "\n",
    "def k_nearest_neighbor_training(k, training_data_x, training_data_y):\n",
    "    # you don't necessarily need to implement anything here. \n",
    "    pass\n",
    "\n",
    "def k_nearest_neighbor_predict(x, k, training_data_x, training_data_y):\n",
    "    return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3e778ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO experiment with the function you implemented similar to the way we did in section B. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8542f6a",
   "metadata": {},
   "source": [
    "# Questions: \n",
    "* Q: Do you find that kNN has a higher accuracy than linear regression?\n",
    "* A: << Fill in your answer here >>\n",
    "* Q: How do you find the performance of the implementation? Is it faster or slower than linear regression?\n",
    "* A: << Fill in your answer here >>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9daa82b",
   "metadata": {},
   "source": [
    "# P2: K nearest neighbors using sklearn (2 points)\n",
    "Use the KNeighborsRegressor function from sklearn to solve the problem. Print the resulting error and samples, similar to the way in Section B. \n",
    "\n",
    "Experiment with at least with the k (number of neighbors) values 1, 3 and 15. \n",
    "\n",
    "Answer the questions in the markdown section. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a12d838",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO implement here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2b8ee62",
   "metadata": {},
   "source": [
    "# Questions: \n",
    "* Q: Do you find that the sklearn version has a higher accuracy than your implementation? \n",
    "* A: << Fill in your answer here >>\n",
    "* Q: How do you find the performance of the sklearn implementation? Is it faster or slower than your implementation?\n",
    "* A: << Fill in your answer here >>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39cba7e7",
   "metadata": {},
   "source": [
    "# C. Setup for the multi-variable part. \n",
    "\n",
    "For the next part of the project, we are going to work in a multi-variable setting. \n",
    "\n",
    "This time, there are 7 explanatory variables: ``sq_mt_built``, ``n_rooms``, ``n_bathrooms``, ``is_renewal_needed``, ``is_new_development`` and ``has_fitted_wardrobes``. \n",
    "\n",
    "We will first create the training and test data while doing some minimal data cleaning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f68a641",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"houses_Madrid.csv\")\n",
    "#print(f\"The columns of the database {df.columns}\")\n",
    "\n",
    "xfields = [\"sq_mt_built\", \"n_rooms\", \"n_bathrooms\", \"has_individual_heating\", \\\n",
    "           \"is_renewal_needed\", \"is_new_development\", \"has_fitted_wardrobes\"]\n",
    "yfield = [\"buy_price\"]\n",
    "# print (xfields + yfield)\n",
    "dfsel = df[xfields + yfield]\n",
    "dfselnona = dfsel.dropna()\n",
    "df_shuffled = dfselnona.sample(frac=1) # shuffle the rows\n",
    "x = df_shuffled[xfields].to_numpy(dtype=np.float64)\n",
    "y = df_shuffled[yfield].to_numpy(dtype=np.float64)\n",
    "print(x.shape)\n",
    "training_data_x = x[:8000]\n",
    "training_data_y = y[:8000]\n",
    "test_data_x = x[8000:]\n",
    "test_data_y = y[8000:]\n",
    "print(f\"Training data is composed of {len(training_data_x)} samples.\")\n",
    "print(f\"Test data is composed of {len(test_data_x)} samples.\")\n",
    "# print(test_data_x[45])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "395f2cfb",
   "metadata": {},
   "source": [
    "# D. Creating a linear regression multi-variable baseline. \n",
    "\n",
    "In this section we make a linear regression predictor for the multi-variable case. We also check the performance of the resulting regressor, and print the error. \n",
    "\n",
    "This part is had been done for you, such that the work does not depend on you importing parts from the previous projects. \n",
    "\n",
    "You will need to adapt this for the other models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b2cc6cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# training the linear regressor\n",
    "regressor = sklearn.linear_model.LinearRegression()\n",
    "regressor.fit(training_data_x, training_data_y)\n",
    "# We will create the predictions yhat for every x from the training data. We will do this one at a time. This is not an efficient way to do it, but it allows you to write and debug functions that return a scalar number\n",
    "yhats = []\n",
    "for x in test_data_x:\n",
    "    yhat = regressor.predict([x])\n",
    "    yhats.append(yhat[0])\n",
    "\n",
    "# Now, print some examples of the quality of the classifier\n",
    "examples = [45, 67, 170, 189, 207]\n",
    "for i in examples:\n",
    "    x = test_data_x[i]\n",
    "    y = test_data_y[i]\n",
    "    yhat = regressor.predict([x])[0][0]\n",
    "    print(f\"House {i} with {x[0]} sqmt was sold for {y} euros, but our system predicted {yhat:.2f}\")\n",
    "\n",
    "# Now calculate the root mean square error on the resulting arrays\n",
    "error = sklearn.metrics.mean_squared_error(yhats, test_data_y, squared=False)\n",
    "print(f\"The mean square error of the linear regression is {error:.2f} euro\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85113c6b",
   "metadata": {},
   "source": [
    "# P3: K nearest neighbors using sklearn (3 points)\n",
    "\n",
    "Use the KNeighborsRegressor function from sklearn to predict the prices of the house. Print the resulting error and samples, similar to the way in Section D. \n",
    "\n",
    "* Experiment with at least with the k (number of neighbors) values 1, 3 and 15. \n",
    "* Experiment with different values for the weights parameter (uniform vs distance)\n",
    "* Experiment with different values for the metric\n",
    "    * Euclidean\n",
    "    * Manhattan\n",
    "    * Some other Minkowski metric parameter (explain which)\n",
    "* Answer the questions in the markdown section. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61701b71",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9dcd129",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO implement here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbe6a914",
   "metadata": {},
   "source": [
    "# Questions: \n",
    "* Q: Do you find that the kNN implementation for multiple input variables performs better that linear regression? Discuss. \n",
    "* A: << Fill in your answer here >>\n",
    "* Q: Do you find that the kNN implementation for multiple input variables performs better than the version for a single variable? Discuss.\n",
    "* A: << Fill in your answer here >>\n",
    "* Q: Discuss how sensitive the kNN implementation is to the setting of the hyperparameters. \n",
    "* A: << Fill in your answer here >>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}