From f104ed49eec865da6c1d0a9cdb455d4266d73c6b Mon Sep 17 00:00:00 2001 From: Nerea Larrachea Date: Mon, 7 Nov 2022 17:51:45 +0100 Subject: [PATCH 1/2] uploading finished lab --- ... Statistical Significance-checkpoint.ipynb | 221 +++++ .../.ipynb_checkpoints/main-checkpoint.ipynb | 895 ++++++++++++++++++ ...Testing and Statistical Significance.ipynb | 4 +- your-code/main.ipynb | 703 +++++++++++++- 4 files changed, 1782 insertions(+), 41 deletions(-) create mode 100755 your-code/.ipynb_checkpoints/Hypothesis Testing and Statistical Significance-checkpoint.ipynb create mode 100755 your-code/.ipynb_checkpoints/main-checkpoint.ipynb diff --git a/your-code/.ipynb_checkpoints/Hypothesis Testing and Statistical Significance-checkpoint.ipynb b/your-code/.ipynb_checkpoints/Hypothesis Testing and Statistical Significance-checkpoint.ipynb new file mode 100755 index 0000000..1fc4ad0 --- /dev/null +++ b/your-code/.ipynb_checkpoints/Hypothesis Testing and Statistical Significance-checkpoint.ipynb @@ -0,0 +1,221 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Hypothesis Testing and Statistical Significance\n", + "\n", + "\n", + "Lesson Goals\n", + "\n", + "In this lesson we will learn the fundamentals of hypothesis testing and statistical significance.\n", + "Introduction\n", + "\n", + "Many times in the real world we would like to resolve a question that requires a comparison of two quantities. For example, does leaving the light on cause people to take longer to fall asleep, or do books with more pages sell more copies. Using a few basic assumptions, we can use statistical inference to come to a conclusion and determine an answer for these questions. Hypothesis allow us to compare two samples and using certain assumptions we can either reject or not reject our hypothesis (as true statisticians we never say that we accept a hypothesis only reject or do not reject).\n", + "Key Concepts\n", + "Population\n", + "\n", + "The population includes all elements in a dataset. For example, the population of all students of a certain university includes all students enrolled at that university.\n", + "Sample\n", + "\n", + "A sample is a subset of the population. For example, a sample of university students from a certain university can be a randomly selected group of students that are enrolled in the university.\n", + "Null Hypothesis\n", + "\n", + "A null hypothesis is what we consider the current status quo. We are trying to disprove this null hypothesis. For example, a null hypothesis might be that all students enrolled in a university have a similar grade point average. We denote the null hypothesis H0.\n", + "\n", + "The typical notation for a null hypothesis would be: H0: μ > 85\n", + "\n", + "Where Ој is the mean and our initial assumption is that it is less than or equal to 85.\n", + "Alternative Hypothesis\n", + "\n", + "An alternative hypothesis is a rival hypothesis to the null hypothesis. We compare this hypothesis with the null hypothesis to decide whether or not we reject the null hypothesis.\n", + "\n", + "We denote the alternative hypothesis with H1 or Ha.\n", + "\n", + "The typical notation for an alternative hypothesis is H1: μ > 85.\n", + "\n", + "We would like to prove that Ој is in fact less than 85.\n", + "Types of Hypotheses\n", + "\n", + "There are 3 types of hypotheses:\n", + "\n", + " H0: μ ≤ k\n", + "\n", + " H1: μ > k\n", + "\n", + " H0: μ ≥ k\n", + "\n", + " H1: μ < k\n", + "\n", + " H0: μ = k\n", + "\n", + " H1: μ ≠ k\n", + "\n", + "The first two types are one-sided hypothesis tests while the third is two-sided. A one-sided hypothesis test only looks at whether the test statistic has a very large positive value or a very large negative value. A two-sided test looks at both. In the two-sided case, we will reject the null hypothesis in the case that the test statistic is either a very large positive value or a very large negative value.\n", + "Test Assumptions\n", + "\n", + "In order to conduct a hypothesis test we need to meet certain assumptions:\n", + "\n", + " Our observations must be independent of each other. For example, if we have people who live in the same household participating in a medical trial, they might be exposed to the same environmental conditions or eat the same food. This can bias our results.\n", + "\n", + " Normality of data - We assume that the sample is derived from a normally distributed data.\n", + "\n", + " Adequate sample size. In order to perform a test using the normal distribution and not approximate to the t distribution, our sample size must be greater than 30.\n", + "\n", + " In order to use the normal distribution for our hypothesis test, we must assume the population standard deviation is known. If the population standard deviation is not known, then we use the t-distribution for the hypothesis test.\n", + "\n", + "Test Statistic\n", + "\n", + "Once we determine the type of hypothesis test and that our assumptions have been met, we use our data to decide whether to reject or not reject the null hypothesis." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This means that our test statistic is equal to the difference between the sample mean and our constant divided by the standard deviation over the square root of the sample size.\n", + "\n", + "We compare the z or t value found in the test with the boundary that we determine for the test using the significance that we chose for the test and determine the result of our test.\n", + "\n", + "\n", + "# Test significance\n", + "\n", + "Typically when we perform a hypothesis test, we cannot be 100% sure of the result. However, we can determine the percent of certainty in our result. For 95% confidence we look at a 5% significance in our test.\n", + "\n", + "Having 95% confidence means that we start off with the null hypothesis as the status quo. We compute our test statistic given that we think the null hypothesis is true. The significance that we produce from the test statistic is the probability that we obtained our result due to random chance. If there is less than a 5% chance that we obtained our result due to random chance, then that is a very small probability of this outcome being random and it is most likely happening due to something in the experiment. Then we can reject the null hypothesis.\n", + "\n", + "\n", + "# Confidence Intervals\n", + "\n", + "Using the critical value that we have computed for our test statistic, we can construct a confidence interval. We compute the confidence interval for a 2 sided test:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "What do confidence intervals mean?\n", + "\n", + "For a 95% confidence interval it means that if we take our population and sample it 100 times, 95 times out of those hundred times, we will produce a confidence interval that contains the true mean of the population (remember that we don't know the true mean of the population, we are estimating it using the sample). If a large portion of our confidence intervals do not contain what our hypothesis claims to be the mean, then we can reject the null hypothesis.\n", + "Performing a Hypothesis Test\n", + "\n", + "Exploring a more concrete example will help tie these concepts together. A pharmaceutical company is trying out a medication for lowering blood sugar and managing diabetes. It is known that any level of Hemoglobin A1c below 5.7% is considered normal. The drug company has treated 100 study volunteers with this medication and would like to prove that after treatment their mean A1c is below 5.7%.\n", + "\n", + "Our hypothesis will be:\n", + "\n", + " H0: μ ≥ 5.7\n", + "\n", + " H1: μ < 5.7\n", + "\n", + "The pharmaceutical company reports a mean A1c of 5.1% with a sample standard deviation of 1.6.\n", + "\n", + "We would like to use our sample to make inference regarding the population. If we reject the null hypothesis, this could provide us with some evidence that the average levels of A1c are within the normal range.\n", + "\n", + "In order to perform a hypothesis test, first we check our assumptions. We do not know anything about the population standard deviation, so even though the sample size is large enough, we will use the t test. We can assume that the pharmaceutical company used proper practices when selecting their study subjects in such a way to avoid dependence and that they are also aware that the data is derived from a normally distributed population." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Since we have 100 subjects, we use 99 degrees of freedom to compute the test statistic. The test statistic for 99 degrees of freedom translates to a p-value of less than 0.0005. This is smaller than 0.05 and therefore we reject the null hypothesis. This is great news for our patients!\n", + "\n", + "\n", + "\n", + "# Hypothesis Tests in Python\n", + "\n", + "We can use the Scipy library to perform hypothesis tests. The Scipy library has a function for one sample hypothesis tests called ttest_1samp. This test takes a dataset and a constant for comparison and returns the test statistic and the p value for a 2 sided test.\n", + "\n", + "Our test is a one-sided test so we will only look at the test statistic. In order to use the p-value we have to divide the p-value by 2.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Ttest_1sampResult(statistic=-2.124917532126503, pvalue=0.03608308764211481)" + ] + }, + "execution_count": 1, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import numpy as np\n", + "from scipy.stats import ttest_1samp\n", + "patients = np.random.normal(5.1, 1.6, 100)\n", + "ttest_1samp(patients, 5.7)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In this example, we generated random data with mean 5.1 and standard deviation 1.6 in order to simulate our patients. Our test statistic is close but not exactly the same since the mean of the sample is not exactly 5.1 like in the example but in fact:\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "5.379469677682324" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "np.mean(patients)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This explains the small discrepancy in the test statistic. However, the result is the same - we reject the null hypothesis." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.6" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/your-code/.ipynb_checkpoints/main-checkpoint.ipynb b/your-code/.ipynb_checkpoints/main-checkpoint.ipynb new file mode 100755 index 0000000..827ad50 --- /dev/null +++ b/your-code/.ipynb_checkpoints/main-checkpoint.ipynb @@ -0,0 +1,895 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Before your start:\n", + "- Read the README.md file\n", + "- Comment as much as you can and use the resources (README.md file)\n", + "- Happy learning!" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "# import numpy and pandas\n", + "import pandas as pd\n", + "import numpy as np" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Challenge 1 - Exploring the Data\n", + "\n", + "In this challenge, we will examine all salaries of employees of the City of Chicago. We will start by loading the dataset and examining its contents" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "# Run this code:\n", + "salaries = pd.read_csv('../Current_Employee_Names__Salaries__and_Position_Titles.csv')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Examine the `salaries` dataset using the `head` function below." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
NameJob TitlesDepartmentFull or Part-TimeSalary or HourlyTypical HoursAnnual SalaryHourly Rate
0AARON, JEFFERY MSERGEANTPOLICEFSalaryNaN101442.0NaN
1AARON, KARINAPOLICE OFFICER (ASSIGNED AS DETECTIVE)POLICEFSalaryNaN94122.0NaN
2AARON, KIMBERLEI RCHIEF CONTRACT EXPEDITERGENERAL SERVICESFSalaryNaN101592.0NaN
3ABAD JR, VICENTE MCIVIL ENGINEER IVWATER MGMNTFSalaryNaN110064.0NaN
4ABASCAL, REECE ETRAFFIC CONTROL AIDE-HOURLYOEMCPHourly20.0NaN19.86
\n", + "
" + ], + "text/plain": [ + " Name Job Titles \\\n", + "0 AARON, JEFFERY M SERGEANT \n", + "1 AARON, KARINA POLICE OFFICER (ASSIGNED AS DETECTIVE) \n", + "2 AARON, KIMBERLEI R CHIEF CONTRACT EXPEDITER \n", + "3 ABAD JR, VICENTE M CIVIL ENGINEER IV \n", + "4 ABASCAL, REECE E TRAFFIC CONTROL AIDE-HOURLY \n", + "\n", + " Department Full or Part-Time Salary or Hourly Typical Hours \\\n", + "0 POLICE F Salary NaN \n", + "1 POLICE F Salary NaN \n", + "2 GENERAL SERVICES F Salary NaN \n", + "3 WATER MGMNT F Salary NaN \n", + "4 OEMC P Hourly 20.0 \n", + "\n", + " Annual Salary Hourly Rate \n", + "0 101442.0 NaN \n", + "1 94122.0 NaN \n", + "2 101592.0 NaN \n", + "3 110064.0 NaN \n", + "4 NaN 19.86 " + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Your code here:\n", + "salaries.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We see from looking at the `head` function that there is quite a bit of missing data. Let's examine how much missing data is in each column. Produce this output in the cell below" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "33183" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Your code here:\n", + "len(salaries[salaries.isnull().any(axis=1)])" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(33183, 8)" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "salaries.shape" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's also look at the count of hourly vs. salaried employees. Write the code in the cell below" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Salary 25161\n", + "Hourly 8022\n", + "Name: Salary or Hourly, dtype: int64" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Your code here:\n", + "salaries[\"Salary or Hourly\"].value_counts()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "What this information indicates is that the table contains information about two types of employees - salaried and hourly. Some columns apply only to one type of employee while other columns only apply to another kind. This is why there are so many missing values. Therefore, we will not do anything to handle the missing values." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "There are different departments in the city. List all departments and the count of employees in each department." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "POLICE 13414\n", + "FIRE 4641\n", + "STREETS & SAN 2198\n", + "OEMC 2102\n", + "WATER MGMNT 1879\n", + "AVIATION 1629\n", + "TRANSPORTN 1140\n", + "PUBLIC LIBRARY 1015\n", + "GENERAL SERVICES 980\n", + "FAMILY & SUPPORT 615\n", + "FINANCE 560\n", + "HEALTH 488\n", + "CITY COUNCIL 411\n", + "LAW 407\n", + "BUILDINGS 269\n", + "COMMUNITY DEVELOPMENT 207\n", + "BUSINESS AFFAIRS 171\n", + "COPA 116\n", + "BOARD OF ELECTION 107\n", + "DoIT 99\n", + "PROCUREMENT 92\n", + "INSPECTOR GEN 87\n", + "MAYOR'S OFFICE 85\n", + "CITY CLERK 84\n", + "ANIMAL CONTRL 81\n", + "HUMAN RESOURCES 79\n", + "CULTURAL AFFAIRS 65\n", + "BUDGET & MGMT 46\n", + "ADMIN HEARNG 39\n", + "DISABILITIES 28\n", + "TREASURER 22\n", + "HUMAN RELATIONS 16\n", + "BOARD OF ETHICS 8\n", + "POLICE BOARD 2\n", + "LICENSE APPL COMM 1\n", + "Name: Department, dtype: int64" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Your code here:\n", + "salaries[\"Department\"].value_counts()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Challenge 2 - Hypothesis Tests\n", + "\n", + "In this section of the lab, we will test whether the hourly wage of all hourly workers is significantly different from $30/hr. Import the correct one sample test function from scipy and perform the hypothesis test for a 95% two sided confidence interval." + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Ttest_1sampResult(statistic=20.6198057854942, pvalue=4.3230240486229894e-92)" + ] + }, + "execution_count": 25, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Your code here:\n", + "from scipy import stats\n", + "stats.ttest_1samp(salaries[salaries['Hourly Rate'].isnull() == False]['Hourly Rate'], 30)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We are also curious about salaries in the police force. The chief of police in Chicago claimed in a press briefing that salaries this year are higher than last year's mean of $86000/year a year for all salaried employees. Test this one sided hypothesis using a 95% confidence interval.\n", + "\n", + "Hint: A one tailed test has a p-value that is half of the two tailed p-value. If our hypothesis is greater than, then to reject, the test statistic must also be positive." + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "3.081997005712994 0.0010301701775482569\n" + ] + } + ], + "source": [ + "# Your code here:\n", + "tstat, pvalue = stats.ttest_1samp(salaries[(salaries['Department'] == 'POLICE') & (salaries['Annual Salary'].isnull() == False)]['Annual Salary'], 86000)\n", + "print(tstat, pvalue/2)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Using the `crosstab` function, find the department that has the most hourly workers. " + ] + }, + { + "cell_type": "code", + "execution_count": 40, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
Salary or HourlyHourlySalary
Department
ADMIN HEARNG0.0000001.000000
ANIMAL CONTRL0.2345680.765432
AVIATION0.6642110.335789
BOARD OF ELECTION0.0000001.000000
BOARD OF ETHICS0.0000001.000000
BUDGET & MGMT0.0434780.956522
BUILDINGS0.0000001.000000
BUSINESS AFFAIRS0.0409360.959064
CITY CLERK0.0000001.000000
CITY COUNCIL0.1557180.844282
COMMUNITY DEVELOPMENT0.0193240.980676
COPA0.0000001.000000
CULTURAL AFFAIRS0.1076920.892308
DISABILITIES0.0000001.000000
DoIT0.0000001.000000
FAMILY & SUPPORT0.4666670.533333
FINANCE0.0785710.921429
FIRE0.0004310.999569
GENERAL SERVICES0.7806120.219388
HEALTH0.0061480.993852
HUMAN RELATIONS0.0000001.000000
HUMAN RESOURCES0.0506330.949367
INSPECTOR GEN0.0000001.000000
LAW0.0982800.901720
LICENSE APPL COMM0.0000001.000000
MAYOR'S OFFICE0.0941180.905882
OEMC0.6056140.394386
POLICE0.0007450.999255
POLICE BOARD0.0000001.000000
PROCUREMENT0.0217390.978261
PUBLIC LIBRARY0.2945810.705419
STREETS & SAN0.8471340.152866
TRANSPORTN0.6359650.364035
TREASURER0.0000001.000000
WATER MGMNT0.8052160.194784
\n", + "
" + ], + "text/plain": [ + "Salary or Hourly Hourly Salary\n", + "Department \n", + "ADMIN HEARNG 0.000000 1.000000\n", + "ANIMAL CONTRL 0.234568 0.765432\n", + "AVIATION 0.664211 0.335789\n", + "BOARD OF ELECTION 0.000000 1.000000\n", + "BOARD OF ETHICS 0.000000 1.000000\n", + "BUDGET & MGMT 0.043478 0.956522\n", + "BUILDINGS 0.000000 1.000000\n", + "BUSINESS AFFAIRS 0.040936 0.959064\n", + "CITY CLERK 0.000000 1.000000\n", + "CITY COUNCIL 0.155718 0.844282\n", + "COMMUNITY DEVELOPMENT 0.019324 0.980676\n", + "COPA 0.000000 1.000000\n", + "CULTURAL AFFAIRS 0.107692 0.892308\n", + "DISABILITIES 0.000000 1.000000\n", + "DoIT 0.000000 1.000000\n", + "FAMILY & SUPPORT 0.466667 0.533333\n", + "FINANCE 0.078571 0.921429\n", + "FIRE 0.000431 0.999569\n", + "GENERAL SERVICES 0.780612 0.219388\n", + "HEALTH 0.006148 0.993852\n", + "HUMAN RELATIONS 0.000000 1.000000\n", + "HUMAN RESOURCES 0.050633 0.949367\n", + "INSPECTOR GEN 0.000000 1.000000\n", + "LAW 0.098280 0.901720\n", + "LICENSE APPL COMM 0.000000 1.000000\n", + "MAYOR'S OFFICE 0.094118 0.905882\n", + "OEMC 0.605614 0.394386\n", + "POLICE 0.000745 0.999255\n", + "POLICE BOARD 0.000000 1.000000\n", + "PROCUREMENT 0.021739 0.978261\n", + "PUBLIC LIBRARY 0.294581 0.705419\n", + "STREETS & SAN 0.847134 0.152866\n", + "TRANSPORTN 0.635965 0.364035\n", + "TREASURER 0.000000 1.000000\n", + "WATER MGMNT 0.805216 0.194784" + ] + }, + "execution_count": 40, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Your code here:\n", + "# first rows, then columnns\n", + "hourly_workers = pd.crosstab(salaries['Department'], salaries['Salary or Hourly'], normalize='index')\n", + "hourly_workers" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
Salary or HourlyHourlySalary
Department
STREETS & SAN1862336
\n", + "
" + ], + "text/plain": [ + "Salary or Hourly Hourly Salary\n", + "Department \n", + "STREETS & SAN 1862 336" + ] + }, + "execution_count": 36, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "hourly_workers[hourly_workers['Hourly'] == hourly_workers['Hourly'].max()]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The workers from the department with the most hourly workers have complained that their hourly wage is less than $35/hour. Using a one sample t-test, test this one-sided hypothesis at the 95% confidence level." + ] + }, + { + "cell_type": "code", + "execution_count": 54, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "Ttest_1sampResult(statistic=-9.567447887848152, pvalue=1.6689265282353859e-21)" + ] + }, + "execution_count": 54, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Your code here:\n", + "# workers STREETS & SAM: hourly wage < 35\n", + "# t-test, one-sided...95% confidence\n", + "stats.ttest_1samp(salaries[(salaries['Department'] == 'STREETS & SAN') & (salaries['Hourly Rate'].isnull() == False)]['Hourly Rate'], 35, alternative = 'less' )" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Challenge 3 - Constructing Confidence Intervals\n", + "\n", + "While testing our hypothesis is a great way to gather empirical evidence for accepting or rejecting the hypothesis, another way to gather evidence is by creating a confidence interval. A confidence interval gives us information about the true mean of the population. So for a 95% confidence interval, we are 95% sure that the mean of the population is within the confidence interval. \n", + ").\n", + "\n", + "To read more about confidence intervals, click [here](https://en.wikipedia.org/wiki/Confidence_interval).\n", + "\n", + "\n", + "In the cell below, we will construct a 95% confidence interval for the mean hourly wage of all hourly workers. \n", + "\n", + "The confidence interval is computed in SciPy using the `t.interval` function. You can read more about this function [here](https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.t.html).\n", + "\n", + "To compute the confidence interval of the hourly wage, use the 0.95 for the confidence level, number of rows - 1 for degrees of freedom, the mean of the sample for the location parameter and the standard error for the scale. The standard error can be computed using [this](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.sem.html) function in SciPy." + ] + }, + { + "cell_type": "code", + "execution_count": 69, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(32.52345834488529, 33.05365708767727)" + ] + }, + "execution_count": 69, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Your code here:\n", + "# 95% CI for the mean hourly wage of all hourly workers\n", + "# t.interval, len - 1 degrees of freedom\n", + "# scipy.stats.sem --> standard error \n", + "from scipy.stats import t\n", + "hourly = salaries.loc[(salaries['Hourly Rate'].isnull() == False), 'Hourly Rate']\n", + "mean_ = np.mean(hourly)\n", + "error_ = stats.sem(hourly)\n", + "stats.t.interval(.95, loc = mean_, scale = error_, df = len(hourly)-1)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now construct the 95% confidence interval for all salaried employeed in the police in the cell below." + ] + }, + { + "cell_type": "code", + "execution_count": 62, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(86177.05631531784, 86795.77269094894)" + ] + }, + "execution_count": 62, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Your code here:\n", + "police = salaries.loc[(salaries['Department'] == \"POLICE\") & (salaries['Annual Salary'].isnull() == False), 'Annual Salary']\n", + "mean_ = np.mean(police)\n", + "error_ = stats.sem(police)\n", + "stats.t.interval(.95, loc = mean_, scale = error_, df = len(police)-1)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Challenge 4 - Hypothesis Tests of Proportions\n", + "\n", + "Another type of one sample test is a hypothesis test of proportions. In this test, we examine whether the proportion of a group in our sample is significantly different than a fraction. \n", + "\n", + "You can read more about one sample proportion tests [here](http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/SAS/SAS6-CategoricalData/SAS6-CategoricalData2.html).\n", + "\n", + "In the cell below, use the `proportions_ztest` function from `statsmodels` to perform a hypothesis test that will determine whether the number of hourly workers in the City of Chicago is significantly different from 25% at the 95% confidence level." + ] + }, + { + "cell_type": "code", + "execution_count": 66, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(-8294.874987757015, 0.0)" + ] + }, + "execution_count": 66, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Your code here:\n", + "from statsmodels.stats.proportion import proportions_ztest\n", + "hourly = [salaries[salaries['Hourly Rate'].isnull() == False]['Hourly Rate']]\n", + "lenght = len(salaries)\n", + "hourly_work = len(hourly)\n", + "proportions_ztest(hourly_work, lenght, value=.25)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.6" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/your-code/Hypothesis Testing and Statistical Significance.ipynb b/your-code/Hypothesis Testing and Statistical Significance.ipynb index d958da9..1fc4ad0 100755 --- a/your-code/Hypothesis Testing and Statistical Significance.ipynb +++ b/your-code/Hypothesis Testing and Statistical Significance.ipynb @@ -199,7 +199,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -213,7 +213,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.7.2" + "version": "3.10.6" } }, "nbformat": 4, diff --git a/your-code/main.ipynb b/your-code/main.ipynb index 4128c1e..827ad50 100755 --- a/your-code/main.ipynb +++ b/your-code/main.ipynb @@ -12,11 +12,13 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 2, "metadata": {}, "outputs": [], "source": [ - "# import numpy and pandas\n" + "# import numpy and pandas\n", + "import pandas as pd\n", + "import numpy as np" ] }, { @@ -30,12 +32,11 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# Run this code:\n", - "\n", "salaries = pd.read_csv('../Current_Employee_Names__Salaries__and_Position_Titles.csv')" ] }, @@ -48,11 +49,131 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 4, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
NameJob TitlesDepartmentFull or Part-TimeSalary or HourlyTypical HoursAnnual SalaryHourly Rate
0AARON, JEFFERY MSERGEANTPOLICEFSalaryNaN101442.0NaN
1AARON, KARINAPOLICE OFFICER (ASSIGNED AS DETECTIVE)POLICEFSalaryNaN94122.0NaN
2AARON, KIMBERLEI RCHIEF CONTRACT EXPEDITERGENERAL SERVICESFSalaryNaN101592.0NaN
3ABAD JR, VICENTE MCIVIL ENGINEER IVWATER MGMNTFSalaryNaN110064.0NaN
4ABASCAL, REECE ETRAFFIC CONTROL AIDE-HOURLYOEMCPHourly20.0NaN19.86
\n", + "
" + ], + "text/plain": [ + " Name Job Titles \\\n", + "0 AARON, JEFFERY M SERGEANT \n", + "1 AARON, KARINA POLICE OFFICER (ASSIGNED AS DETECTIVE) \n", + "2 AARON, KIMBERLEI R CHIEF CONTRACT EXPEDITER \n", + "3 ABAD JR, VICENTE M CIVIL ENGINEER IV \n", + "4 ABASCAL, REECE E TRAFFIC CONTROL AIDE-HOURLY \n", + "\n", + " Department Full or Part-Time Salary or Hourly Typical Hours \\\n", + "0 POLICE F Salary NaN \n", + "1 POLICE F Salary NaN \n", + "2 GENERAL SERVICES F Salary NaN \n", + "3 WATER MGMNT F Salary NaN \n", + "4 OEMC P Hourly 20.0 \n", + "\n", + " Annual Salary Hourly Rate \n", + "0 101442.0 NaN \n", + "1 94122.0 NaN \n", + "2 101592.0 NaN \n", + "3 110064.0 NaN \n", + "4 NaN 19.86 " + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ - "# Your code here:\n" + "# Your code here:\n", + "salaries.head()" ] }, { @@ -64,11 +185,43 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 7, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "33183" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ - "# Your code here:\n" + "# Your code here:\n", + "len(salaries[salaries.isnull().any(axis=1)])" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(33183, 8)" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "salaries.shape" ] }, { @@ -80,12 +233,25 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 10, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "Salary 25161\n", + "Hourly 8022\n", + "Name: Salary or Hourly, dtype: int64" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "# Your code here:\n", - "\n" + "salaries[\"Salary or Hourly\"].value_counts()" ] }, { @@ -104,11 +270,58 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 11, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "POLICE 13414\n", + "FIRE 4641\n", + "STREETS & SAN 2198\n", + "OEMC 2102\n", + "WATER MGMNT 1879\n", + "AVIATION 1629\n", + "TRANSPORTN 1140\n", + "PUBLIC LIBRARY 1015\n", + "GENERAL SERVICES 980\n", + "FAMILY & SUPPORT 615\n", + "FINANCE 560\n", + "HEALTH 488\n", + "CITY COUNCIL 411\n", + "LAW 407\n", + "BUILDINGS 269\n", + "COMMUNITY DEVELOPMENT 207\n", + "BUSINESS AFFAIRS 171\n", + "COPA 116\n", + "BOARD OF ELECTION 107\n", + "DoIT 99\n", + "PROCUREMENT 92\n", + "INSPECTOR GEN 87\n", + "MAYOR'S OFFICE 85\n", + "CITY CLERK 84\n", + "ANIMAL CONTRL 81\n", + "HUMAN RESOURCES 79\n", + "CULTURAL AFFAIRS 65\n", + "BUDGET & MGMT 46\n", + "ADMIN HEARNG 39\n", + "DISABILITIES 28\n", + "TREASURER 22\n", + "HUMAN RELATIONS 16\n", + "BOARD OF ETHICS 8\n", + "POLICE BOARD 2\n", + "LICENSE APPL COMM 1\n", + "Name: Department, dtype: int64" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ - "# Your code here:\n" + "# Your code here:\n", + "salaries[\"Department\"].value_counts()" ] }, { @@ -122,11 +335,24 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 25, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "Ttest_1sampResult(statistic=20.6198057854942, pvalue=4.3230240486229894e-92)" + ] + }, + "execution_count": 25, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ - "# Your code here:\n" + "# Your code here:\n", + "from scipy import stats\n", + "stats.ttest_1samp(salaries[salaries['Hourly Rate'].isnull() == False]['Hourly Rate'], 30)" ] }, { @@ -140,11 +366,21 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 29, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "3.081997005712994 0.0010301701775482569\n" + ] + } + ], "source": [ - "# Your code here:\n" + "# Your code here:\n", + "tstat, pvalue = stats.ttest_1samp(salaries[(salaries['Department'] == 'POLICE') & (salaries['Annual Salary'].isnull() == False)]['Annual Salary'], 86000)\n", + "print(tstat, pvalue/2)" ] }, { @@ -156,11 +392,329 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 40, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
Salary or HourlyHourlySalary
Department
ADMIN HEARNG0.0000001.000000
ANIMAL CONTRL0.2345680.765432
AVIATION0.6642110.335789
BOARD OF ELECTION0.0000001.000000
BOARD OF ETHICS0.0000001.000000
BUDGET & MGMT0.0434780.956522
BUILDINGS0.0000001.000000
BUSINESS AFFAIRS0.0409360.959064
CITY CLERK0.0000001.000000
CITY COUNCIL0.1557180.844282
COMMUNITY DEVELOPMENT0.0193240.980676
COPA0.0000001.000000
CULTURAL AFFAIRS0.1076920.892308
DISABILITIES0.0000001.000000
DoIT0.0000001.000000
FAMILY & SUPPORT0.4666670.533333
FINANCE0.0785710.921429
FIRE0.0004310.999569
GENERAL SERVICES0.7806120.219388
HEALTH0.0061480.993852
HUMAN RELATIONS0.0000001.000000
HUMAN RESOURCES0.0506330.949367
INSPECTOR GEN0.0000001.000000
LAW0.0982800.901720
LICENSE APPL COMM0.0000001.000000
MAYOR'S OFFICE0.0941180.905882
OEMC0.6056140.394386
POLICE0.0007450.999255
POLICE BOARD0.0000001.000000
PROCUREMENT0.0217390.978261
PUBLIC LIBRARY0.2945810.705419
STREETS & SAN0.8471340.152866
TRANSPORTN0.6359650.364035
TREASURER0.0000001.000000
WATER MGMNT0.8052160.194784
\n", + "
" + ], + "text/plain": [ + "Salary or Hourly Hourly Salary\n", + "Department \n", + "ADMIN HEARNG 0.000000 1.000000\n", + "ANIMAL CONTRL 0.234568 0.765432\n", + "AVIATION 0.664211 0.335789\n", + "BOARD OF ELECTION 0.000000 1.000000\n", + "BOARD OF ETHICS 0.000000 1.000000\n", + "BUDGET & MGMT 0.043478 0.956522\n", + "BUILDINGS 0.000000 1.000000\n", + "BUSINESS AFFAIRS 0.040936 0.959064\n", + "CITY CLERK 0.000000 1.000000\n", + "CITY COUNCIL 0.155718 0.844282\n", + "COMMUNITY DEVELOPMENT 0.019324 0.980676\n", + "COPA 0.000000 1.000000\n", + "CULTURAL AFFAIRS 0.107692 0.892308\n", + "DISABILITIES 0.000000 1.000000\n", + "DoIT 0.000000 1.000000\n", + "FAMILY & SUPPORT 0.466667 0.533333\n", + "FINANCE 0.078571 0.921429\n", + "FIRE 0.000431 0.999569\n", + "GENERAL SERVICES 0.780612 0.219388\n", + "HEALTH 0.006148 0.993852\n", + "HUMAN RELATIONS 0.000000 1.000000\n", + "HUMAN RESOURCES 0.050633 0.949367\n", + "INSPECTOR GEN 0.000000 1.000000\n", + "LAW 0.098280 0.901720\n", + "LICENSE APPL COMM 0.000000 1.000000\n", + "MAYOR'S OFFICE 0.094118 0.905882\n", + "OEMC 0.605614 0.394386\n", + "POLICE 0.000745 0.999255\n", + "POLICE BOARD 0.000000 1.000000\n", + "PROCUREMENT 0.021739 0.978261\n", + "PUBLIC LIBRARY 0.294581 0.705419\n", + "STREETS & SAN 0.847134 0.152866\n", + "TRANSPORTN 0.635965 0.364035\n", + "TREASURER 0.000000 1.000000\n", + "WATER MGMNT 0.805216 0.194784" + ] + }, + "execution_count": 40, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ - "# Your code here:\n" + "# Your code here:\n", + "# first rows, then columnns\n", + "hourly_workers = pd.crosstab(salaries['Department'], salaries['Salary or Hourly'], normalize='index')\n", + "hourly_workers" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
Salary or HourlyHourlySalary
Department
STREETS & SAN1862336
\n", + "
" + ], + "text/plain": [ + "Salary or Hourly Hourly Salary\n", + "Department \n", + "STREETS & SAN 1862 336" + ] + }, + "execution_count": 36, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "hourly_workers[hourly_workers['Hourly'] == hourly_workers['Hourly'].max()]" ] }, { @@ -172,11 +726,25 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 54, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "Ttest_1sampResult(statistic=-9.567447887848152, pvalue=1.6689265282353859e-21)" + ] + }, + "execution_count": 54, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ - "# Your code here:\n" + "# Your code here:\n", + "# workers STREETS & SAM: hourly wage < 35\n", + "# t-test, one-sided...95% confidence\n", + "stats.ttest_1samp(salaries[(salaries['Department'] == 'STREETS & SAN') & (salaries['Hourly Rate'].isnull() == False)]['Hourly Rate'], 35, alternative = 'less' )" ] }, { @@ -200,11 +768,30 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 69, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "(32.52345834488529, 33.05365708767727)" + ] + }, + "execution_count": 69, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ - "# Your code here:\n" + "# Your code here:\n", + "# 95% CI for the mean hourly wage of all hourly workers\n", + "# t.interval, len - 1 degrees of freedom\n", + "# scipy.stats.sem --> standard error \n", + "from scipy.stats import t\n", + "hourly = salaries.loc[(salaries['Hourly Rate'].isnull() == False), 'Hourly Rate']\n", + "mean_ = np.mean(hourly)\n", + "error_ = stats.sem(hourly)\n", + "stats.t.interval(.95, loc = mean_, scale = error_, df = len(hourly)-1)" ] }, { @@ -216,11 +803,26 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 62, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "(86177.05631531784, 86795.77269094894)" + ] + }, + "execution_count": 62, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ - "# Your code here:\n" + "# Your code here:\n", + "police = salaries.loc[(salaries['Department'] == \"POLICE\") & (salaries['Annual Salary'].isnull() == False), 'Annual Salary']\n", + "mean_ = np.mean(police)\n", + "error_ = stats.sem(police)\n", + "stats.t.interval(.95, loc = mean_, scale = error_, df = len(police)-1)" ] }, { @@ -238,17 +840,40 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 66, "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "(-8294.874987757015, 0.0)" + ] + }, + "execution_count": 66, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ - "# Your code here:\n" + "# Your code here:\n", + "from statsmodels.stats.proportion import proportions_ztest\n", + "hourly = [salaries[salaries['Hourly Rate'].isnull() == False]['Hourly Rate']]\n", + "lenght = len(salaries)\n", + "hourly_work = len(hourly)\n", + "proportions_ztest(hourly_work, lenght, value=.25)" ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] } ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, @@ -262,7 +887,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.7.3" + "version": "3.10.6" } }, "nbformat": 4, From 7c138690a61878440d23598d7b9ad2c08e1460ea Mon Sep 17 00:00:00 2001 From: Nerea Larrachea Date: Tue, 8 Nov 2022 10:54:42 +0100 Subject: [PATCH 2/2] updated challenge 4 --- your-code/.ipynb_checkpoints/main-checkpoint.ipynb | 11 ++++++----- your-code/main.ipynb | 11 ++++++----- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/your-code/.ipynb_checkpoints/main-checkpoint.ipynb b/your-code/.ipynb_checkpoints/main-checkpoint.ipynb index 827ad50..1ec3df9 100755 --- a/your-code/.ipynb_checkpoints/main-checkpoint.ipynb +++ b/your-code/.ipynb_checkpoints/main-checkpoint.ipynb @@ -840,27 +840,28 @@ }, { "cell_type": "code", - "execution_count": 66, + "execution_count": 75, "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "(-8294.874987757015, 0.0)" + "(-3.5099964213703005, 0.0004481127249057967)" ] }, - "execution_count": 66, + "execution_count": 75, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Your code here:\n", + "# We reject the null hypothesis, because the p-value is smaller than 0.05\n", "from statsmodels.stats.proportion import proportions_ztest\n", - "hourly = [salaries[salaries['Hourly Rate'].isnull() == False]['Hourly Rate']]\n", + "hourly = salaries[salaries['Salary or Hourly'] == 'Hourly']\n", "lenght = len(salaries)\n", "hourly_work = len(hourly)\n", - "proportions_ztest(hourly_work, lenght, value=.25)" + "proportions_ztest(hourly_work, lenght, value = .25)" ] }, { diff --git a/your-code/main.ipynb b/your-code/main.ipynb index 827ad50..1ec3df9 100755 --- a/your-code/main.ipynb +++ b/your-code/main.ipynb @@ -840,27 +840,28 @@ }, { "cell_type": "code", - "execution_count": 66, + "execution_count": 75, "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "(-8294.874987757015, 0.0)" + "(-3.5099964213703005, 0.0004481127249057967)" ] }, - "execution_count": 66, + "execution_count": 75, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Your code here:\n", + "# We reject the null hypothesis, because the p-value is smaller than 0.05\n", "from statsmodels.stats.proportion import proportions_ztest\n", - "hourly = [salaries[salaries['Hourly Rate'].isnull() == False]['Hourly Rate']]\n", + "hourly = salaries[salaries['Salary or Hourly'] == 'Hourly']\n", "lenght = len(salaries)\n", "hourly_work = len(hourly)\n", - "proportions_ztest(hourly_work, lenght, value=.25)" + "proportions_ztest(hourly_work, lenght, value = .25)" ] }, {