{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00bf530",
   "metadata": {},
   "source": [
    "# Homework 3\n",
    "\n",
    "This project asks you to perform various experiments with regression. The dataset we are using is taken from a real estate dataset (also used in HW2).\n",
    "\n",
    "You will write code and discussion texts into code and text cells in this notebook. \n",
    "\n",
    "If a block starts with TODO:, this means that you need to write something there. \n",
    "\n",
    "Some code had been written for you to guide the project. Don't change the already written code.\n",
    "\n",
    "## Grading\n",
    "Each subproblem is worth 1 pt. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a440a191",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import scipy as sp\n",
    "import sklearn as sk"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "691d1a9f",
   "metadata": {},
   "source": [
    "## Setup for the first part of the homework\n",
    "\n",
    "For problems P1 to P6 we are using a simple dataset where we extract one \n",
    "explanatory variable ``sq_mt_built`` to predict the price of the house ``buy_price``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "902ccefe",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"houses_Madrid.csv\")\n",
    "print(f\"The lenght {len(df.index)}\")\n",
    "print(f\"The columns of the database {df.columns}\")\n",
    "df[[\"sq_mt_built\", \"buy_price\"]].plot.scatter(x=\"sq_mt_built\", y=\"buy_price\")\n",
    "## FIXME: add here the creation of the training data and test data\n",
    "\n",
    "df_shuffled = df.sample(frac=1) # shuffle the rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54bf2ec2",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = df_shuffled[\"sq_mt_built\"].to_numpy(dtype=np.float64)\n",
    "y = df_shuffled[\"buy_price\"].to_numpy(dtype=np.float64)\n",
    "training_data_x = x[:16000]\n",
    "training_data_y = y[:16000]\n",
    "test_data_x = x[16000:]\n",
    "test_data_y = y[16000:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d8bfd4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_data_y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c63bf75",
   "metadata": {},
   "source": [
    "## P1: Loss function\n",
    "Implement a root-mean-square error (RMSE) loss function between the prediction $\\hat{y}$ and $y$ value using Python operations. Run some experiments to validate that this works as expected. \n",
    "Then, look up the same in the sklearn library\n",
    "https://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics\n",
    "and implement it based on what is there. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66d0d477",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: implement the loss function here using Python math ops and sklearn\n",
    "def loss_RMSE(y, yhat):\n",
    "    return 0\n",
    "\n",
    "def loss_RMSE_sk(y, yhat):\n",
    "    return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a723623d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Now, run some experiments with your function, with the one taken with sklearn \n",
    "# Compare their outputs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0548dea",
   "metadata": {},
   "source": [
    "## P2: Implement a linear predictor\n",
    "Implement a function of type ``predict(x, theta) --> y_hat`` which implements a linear model of the type $\\hat{y} = \\theta_1 \\cdot x + \\theta_0$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b91956b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: implement the predictor function here\n",
    "def predict(x, theta):\n",
    "    y_hat = 0\n",
    "    return y_hat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7af53924",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: now, run some experiments with it"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5fcd1c6",
   "metadata": {},
   "source": [
    "## P3: Grid search\n",
    "Implement a function ``grid_search()`` which returns an estimate of the best $\\theta$ by trying out all the combinations of possibilities on a grid and returning the values that give you the most values. \n",
    "gridx and gridy define the range of numbers that we want to explore. For instance, grid0 might be [0, 0.25, 0.5, 0.75, 1.0] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9663e31d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: implement the grid search function here \n",
    "def grid_search(training_data_x, training_data_y, grid0, grid1):\n",
    "    theta = [0, 0]\n",
    "    return theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65e10363",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: run some experiments with grid_search\n",
    "# Define some grid values. Train it on the data set. Test it on the test set. \n",
    "# Print the loss on the data set and the test set. Measure and print how long the training takes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "716ed9d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: repeat the experimentation from above with different grids. \n",
    "# Finally, print the grid that provides the best value while still running faster \n",
    "# than 10 seconds."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c46a9f4",
   "metadata": {},
   "source": [
    "## P4: Random search\n",
    "Implement a function that returns the estimate for the best $\\theta$ by trying out random \n",
    "$\\theta=[\\theta_0, \\theta_1]$ values, and returning the one that minimizes the error on the training set passed to it. The number of tries is described in the ``trials`` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc373403",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: implement the random search function here \n",
    "def grid_search(training_data_x, training_data_y, trials):\n",
    "    theta = [0, 0]\n",
    "    return theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "294dde6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: run some experiments with random_search\n",
    "# Choose some value for trial. Train it on the data set. Test it on the test set. \n",
    "# Print the loss on the data set and the test set. Measure and print how long the training takes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85e169df",
   "metadata": {},
   "source": [
    "## P5: Improvements\n",
    "Propose an improvement to the algorithms you have implemented for 4 and 5 and show that with your improvement, the performance is better than the original. Some examples of what you might try:\n",
    "* Choose values for $\\theta_0$ and $\\theta_1$ on a non-uniform grid\n",
    "* First find one of them, and fix it, and then refine on the other one\n",
    "* For random: sample according to a non-uniform distribution\n",
    "* First use a low resolution search to find the approximate values of  $\\theta_0$ and $\\theta_1$, then search for a more precise value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb552fcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: implement your improvements here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d1dd74e",
   "metadata": {},
   "source": [
    "TODO: Describe in one paragraph the conclusions you have drawn from your improvement experiments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5c0d4e8",
   "metadata": {},
   "source": [
    "## P6: Using the sklearn library for linear regression\n",
    "\n",
    "Use ``sklearn.linear_model.LinearRegression`` to solve the same problem you previously solved using the grid search and random search.\n",
    "\n",
    "Compare the returned values with what you have achieved. Compare the parameters that had been found to the parameters you have found. Compare the speed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "136e4f63",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Implement here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1268d33c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Run performance experiments here. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b2dc65e",
   "metadata": {},
   "source": [
    "TODO: discuss the performance of the sklearn library implementation, compared to what you implemented."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39cba7e7",
   "metadata": {},
   "source": [
    "# Setup for the second part of the project\n",
    "For the questions P7-P10 we use linear regression on a multivariate setting. This time, there are 7 explanatory variables: ``sq_mt_built``, ``n_rooms``, ``n_bathrooms``, ``is_renewal_needed``, ``is_new_development`` and ``has_fitted_wardrobes``. \n",
    "\n",
    "We will first create the training and test data while doing some minimal data cleaning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f68a641",
   "metadata": {},
   "outputs": [],
   "source": [
    "# replacing the NA values with some sensible defaults\n",
    "# the way I was investigating these is by printing \n",
    "#    df[\"has_individual_heeating\"].value_counts(dropna=False) etc\n",
    "df_shuffled[\"has_individual_heating\"] = df_shuffled[\"has_individual_heating\"].fillna(False)\n",
    "df_shuffled[\"n_bathrooms\"] = df_shuffled[\"n_bathrooms\"].fillna(1)\n",
    "df_shuffled[\"has_individual_heating\"] = df_shuffled[\"has_individual_heating\"].fillna(False)\n",
    "df_shuffled[\"is_new_development\"] = df_shuffled[\"is_new_development\"].fillna(False)\n",
    "df_shuffled[\"has_fitted_wardrobes\"] = df_shuffled[\"has_fitted_wardrobes\"].fillna(False)\n",
    "\n",
    "xfields = [\"sq_mt_built\", \"n_rooms\", \"n_bathrooms\", \"has_individual_heating\", \\\n",
    "           \"is_renewal_needed\", \"is_new_development\", \"has_fitted_wardrobes\"]\n",
    "\n",
    "x = df_shuffled[xfields].to_numpy(dtype=np.float64)\n",
    "y = df_shuffled[\"buy_price\"].to_numpy(dtype=np.float64)\n",
    "training_data_x = x[:16000]\n",
    "training_data_y = y[:16000]\n",
    "test_data_x = x[16000:]\n",
    "test_data_y = y[16000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "848d0c5b",
   "metadata": {},
   "source": [
    "## P7: Grid search for multiple variables\n",
    "Implement the linear predictor model for multiple variables. Note that this time ``x`` will be an array of 7 values, and ``theta`` will be an array of 8 values. \n",
    "\n",
    "Then, implement a grid search function (similar to P3) but this time for the 7 explanatory variables. Pass the grids as an array into the grid variable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bde6d4ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: implement the predictor function here\n",
    "def predict_multi(x, theta):\n",
    "    y_hat = 0\n",
    "    return y_hat\n",
    "\n",
    "# TODO: implement the grid search function here \n",
    "def grid_search(training_data_x, training_data_y, grids):\n",
    "    theta = [0, 0]\n",
    "    return theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "080d821a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: run experiments with your implementation for the grid search "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab609edb",
   "metadata": {},
   "source": [
    "TODO: describe here your experiences with implementing this problem, conclusions you draw. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "020b814a",
   "metadata": {},
   "source": [
    "## P8: Random search for multiple variables\n",
    "Implement the random search technique for the multiple variables. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37f3b832",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: implement the random seeach function here \n",
    "def grid_search(training_data_x, training_data_y):\n",
    "    theta = [0, 0]\n",
    "    return theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c553a29d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: run experiments with your implementation for the"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba6f2531",
   "metadata": {},
   "source": [
    "TODO: describe here your experiments with the implementation for the random search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f8cd1c2",
   "metadata": {},
   "source": [
    "## P9: Use sklearn for linear regression in multiple variables\n",
    "Use ``sklearn.linear_model.LinearRegression`` to solve the same problem you previously solved using the grid search and random search.\n",
    "\n",
    "Compare the returned values with what you have achieved. Compare the parameters that had been found to the parameters you have found. Compare the speed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4f85619",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: implement here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b7fbdca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: run experiments here. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f3fb028",
   "metadata": {},
   "source": [
    "TODO: describe in one paragraph your experiences with implementing the multiple variable linear regression. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5761a993",
   "metadata": {},
   "source": [
    "TODO: describe in one paragraph your experiences with the data wrangling process."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22ba6a81",
   "metadata": {},
   "source": [
    "## P10: Explore other linear regression techniques\n",
    "\n",
    "Explore the use of other models provided from the sklearn library for linear regression. \n",
    "\n",
    "https://scikit-learn.org/stable/modules/classes.html#module-sklearn.linear_model\n",
    "\n",
    "Try out two of them of your choice. Explain the results you obtained and compare them with other approaches. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11946c5e",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}