{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c00bf530",
   "metadata": {},
   "source": [
    "# Homework 5\n",
    "\n",
    "This homework asks you to perform various experiments with ensemble methods. \n",
    "\n",
    "The dataset is the same real estate dataset we previously used from:\n",
    "\n",
    "https://www.kaggle.com/datasets/mirbektoktogaraev/madrid-real-estate-market\n",
    "\n",
    "You will write code and discussion into code and text cells in this notebook. \n",
    "\n",
    "If a code block starts with TODO:, this means that you need to write something there. \n",
    "\n",
    "There are also markdown blocks with questions. Write the answers to these questions in the specified locations.\n",
    "\n",
    "Some code had been written for you to guide the project. Don't change the already written code.\n",
    "\n",
    "## Grading\n",
    "The points add up to 10. Extensive partial credit will be offered. Thus, make sure that you are at least attempting all problems. \n",
    "\n",
    "Make sure to comment your code, such that the grader can understand what different components are doing or attempting to do. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a440a191",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import sklearn.linear_model\n",
    "import sklearn.metrics\n",
    "import sklearn.ensemble\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39cba7e7",
   "metadata": {},
   "source": [
    "# A. Setup. \n",
    "\n",
    "In this project we are going to work in a multi-variable setting. \n",
    "\n",
    "This time, there are 7 explanatory variables: ``sq_mt_built``, ``n_rooms``, ``n_bathrooms``, ``is_renewal_needed``, ``is_new_development`` and ``has_fitted_wardrobes``. \n",
    "\n",
    "We will first create the training and test data while doing some minimal data cleaning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f68a641",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"houses_Madrid.csv\")\n",
    "#print(f\"The columns of the database {df.columns}\")\n",
    "\n",
    "xfields = [\"sq_mt_built\", \"n_rooms\", \"n_bathrooms\", \"has_individual_heating\", \\\n",
    "           \"is_renewal_needed\", \"is_new_development\", \"has_fitted_wardrobes\"]\n",
    "yfield = [\"buy_price\"]\n",
    "# print (xfields + yfield)\n",
    "dfsel = df[xfields + yfield]\n",
    "dfselnona = dfsel.dropna()\n",
    "df_shuffled = dfselnona.sample(frac=1) # shuffle the rows\n",
    "x = df_shuffled[xfields].to_numpy(dtype=np.float64)\n",
    "y = df_shuffled[yfield].to_numpy(dtype=np.float64)\n",
    "print(x.shape)\n",
    "training_data_x = x[:8000]\n",
    "training_data_y = y[:8000]\n",
    "test_data_x = x[8000:]\n",
    "test_data_y = y[8000:]\n",
    "print(f\"Training data is composed of {len(training_data_x)} samples.\")\n",
    "print(f\"Test data is composed of {len(test_data_x)} samples.\")\n",
    "# print(test_data_x[45])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "395f2cfb",
   "metadata": {},
   "source": [
    "# B. Creating a linear regression multi-variable baseline. \n",
    "\n",
    "In this section we make a linear regression predictor for the multi-variable case. We also check the performance of the resulting regressor, and print the error. \n",
    "\n",
    "This part is had been done for you, such that the work does not depend on you importing parts from the previous projects. \n",
    "\n",
    "You will need to adapt this for the other models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b2cc6cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# training the linear regressor\n",
    "regressor = sklearn.linear_model.LinearRegression()\n",
    "regressor.fit(training_data_x, training_data_y)\n",
    "# We will create the predictions yhat for every x from the training data. We will do this one at a time. This is not an efficient way to do it, but it allows you to write and debug functions that return a scalar number\n",
    "yhats = []\n",
    "for x in test_data_x:\n",
    "    yhat = regressor.predict([x])\n",
    "    yhats.append(yhat[0])\n",
    "\n",
    "# Now, print some examples of the quality of the classifier\n",
    "examples = [45, 67, 170, 189, 207]\n",
    "for i in examples:\n",
    "    x = test_data_x[i]\n",
    "    y = test_data_y[i]\n",
    "    yhat = regressor.predict([x])[0][0]\n",
    "    print(f\"House {i} with {x[0]} sqmt was sold for {y} euros, but our system predicted {yhat:.2f}\")\n",
    "\n",
    "# Now calculate the root mean square error on the resulting arrays\n",
    "error = sklearn.metrics.mean_squared_error(yhats, test_data_y, squared=False)\n",
    "print(f\"The mean square error of the linear regression is {error:.2f} euro\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85113c6b",
   "metadata": {},
   "source": [
    "# P1: Random Forest using sklearn (5 points)\n",
    "\n",
    "Use the RandomForestRegressor function from sklearn to predict the prices of the house. Print the resulting error and samples, similar to the way in Section B. \n",
    "\n",
    "Experiment with the settings of the hyperparameters: n_estimators (try at least values 10, 25, 100, 200) and max_depth (try at least values 1, 2, 4, 8, 16 and None).\n",
    "\n",
    "Retain the hyperparameter value that gives you the best result. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9dcd129",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO implement here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbe6a914",
   "metadata": {},
   "source": [
    "# Questions: \n",
    "* Q: Do you find that Random Forest performs better than the previous approaches you implemented? Discuss. \n",
    "* A: << Fill in your answer here >>\n",
    "* Q: Explain the impact of the number of estimators and max tree depth hyperparameters on the accuracy. Which hyperparameter setting gives you the best value? Is this the same as the default settings in sklearn?\n",
    "* A: << Fill in your answer here >>\n",
    "* Q: Explain the impact of the hyperparameters on the training time. \n",
    "* A: << Fill in your answer here >>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e98e415f",
   "metadata": {},
   "source": [
    "# P2: AdaBoost using sklearn (5 points)\n",
    "\n",
    "Use the AdaBoost function from sklearn to predict the prices of the house. Print the resulting error and samples, similar to the way in Section B. \n",
    "\n",
    "Experiment with the settings of the hyperparameters: loss (try \"linear\", \"square\" and \"exponential) and learning_rate (try at least values 0.2, 0.5, 1 and 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa7b135a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO implement here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "165b9e5c",
   "metadata": {},
   "source": [
    "# Questions: \n",
    "* Q: Do you find that Adaboost performs better than the previous approaches you implemented? Discuss. \n",
    "* A: << Fill in your answer here >>\n",
    "* Q: Explain the impact of the loss and the learning_rate hyperparameters on the accuracy. Which hyperparameter setting gives you the best value? Is this the same as the default settings in sklearn?\n",
    "* A: << Fill in your answer here >>\n",
    "* Q: Explain the impact of the hyperparameters on the training time. \n",
    "* A: << Fill in your answer here >>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}