Learn By Example: Statistics and Data Science in R

This course is a gentle yet thorough introduction to Data Science, Statistics and R using real life examples.

Q. How do companies make decisions? 

A. Using data

We talk about what it takes to go from data to making a decision from data. This sets the agenda for the rest of the course - each of the things on this journey is covered in the upcoming sections

Get setup with R and Rstudio. All the examples that follow in this course will have source code attached. Download and run them in Rstudio

Bosses are impatient. They often want you to cut to the chase, and give them an answer that's ok, but in a short amount of time. Descriptive statistics are the first place to start - they are often the 10s answer to any question about the data. 

Computing a frequency distribution using R

A histogram is a good visual summary of your data. 

Computing Mean, Median, Mode in R

The mean, median and mode are point estimates to represent your data. IQR is a measure that explains the spread of the data.

What is IQR (Inter-quartile Range)? 

Visualize the IQR and outliers using box and whisker plots

The standard deviation measures the spread of a dataset, and it so happens, the standard deviation is actually very profound.

Computing IQR and Standard Deviation in R

Drawing inferences from data is key to being able to take decisions using data. There is a science to this, whose foundation is in random variables, probability distributions, and performing tests of statistical significance. 

Random variables are everywhere. Any data that you'll study is a random variable whose behaviour is determined by a probability distribution.

The Normal Distribution is arguably the most well-known and commonly seen probability distribution. It is characterized by its probability density function, mean and standard deviation.

Sampling is a little like fishing. Sampling is crucial to induction - drawing conclusions about something by looking at some evidence.

A sample is described by sample statistics like the sample mean. The sampling distribution is the probability distribution of sample means. 

Find a point estimate for the average weight of all football players using a sample of football players in 1 college team.

Find a point estimate for the % of voters in favor of a candidate.

A test of significance is an important step in building support for your findings and inferences. Here is the first example of a test of significance - is the population mean equal to a given value? 

Perform a test of significance to check whether the population % is equal to a certain value

Perform a test of significance to compare 2 population means. The example used is A/B Testing - which is pretty widely used in internet companies to test out product features.

Perform a test of significance to compare two population proportions

The next few sections dive deep into all the data processing, slicing and dicing ability that R provides. The wide variety of R packages available is one reason why R is popular among many data scientists. 

Let's start with the basics. What are variables and how do we assign variables in R? 

print(), show(), message(), cat() are different ways to print something to screen. 

Numbers are of type numeric 

R has built-in datatypes for dates and timestamps. 

Logical is a datatype that is the result of conditional tests in R

The wide variety of built-in data structures are what makes R different from other standard programming languages. These include vectors, arrays, matrices, data frames and lists. 

Creating a Vector

The mode of a vector is the datatype of all its elements. 

Vectors are Atomic

Doing something with each element of a Vector

Finding the sum, product, or mean of a vector

Operations between vectors of the same length

Operations between vectors of different length

Generate sequences using the : operator, rep() and seq()

Using conditions with Vectors

Find the lengths of multiple strings using Vectors

Generate a complex sequence (using recycling)

Access elements based on their position in the vector.

Access elements based on whether they pass a conditional test. 

Assign names to the elements of a vector

Creating an array can be done by using a vector and then arranging it along dimensions.

Indexing an Array

Operations between 2 Arrays

Operations between an Array and a Vector

Outer products are complex operations that operate on every pair of elements from two arrays.

A Matrix is a 2 Dimensional array. But it has special meaning and can be interpreted in a bunch of different ways.

Creating a Matrix

Matrix Multiplication

rbind() and cbind() to merge matrices.

Solving a set of linear equations

A factor is a special type of vector used to represent categorical variables

Find the distinct values in a dataset (using factors)

Replace the levels of a factor

Aggregate factors with table()

Aggregate factors with tapply()

Lists are fundamentally different from vectors, arrays and matrices - which are all homogenous data structures.

Data Frames are how R stores data read from files and databases.

Reading Data from files

Indexing a Data Frame

Using the aggregate() and order() function

Merge data frames based on one or more common columns

Regression is the process of finding a model that describes the relationship between variables. 

Linear regression is the process of fitting a line or a linear model that best explains the relationship between 2 variables. Understand what residuals are, the ordinary least squares method and R-Squared

The Capital Asset Pricing Model describes a relationship between risk and return. Use it with regression to either find the risk or returns of a given stock. Regression is one of the ways to estimate the Beta in CAPM.

Find the Beta of Google by regressing Google returns against NASDAQ returns. We describe how to find, and prepare the data for fitting a linear model. 

LINEST() is a function in excel that fits a linear model for a given set of variables. However LINEST() has a bunch of issues, including its inability to deal with missing values.

Find the Beta of Google by regressing Google returns against NASDAQ returns. We describe how process data frames and prepare the data for fitting a linear model. 

lm() is used to build linear models in R. The results of lm() can be parsed using summary(). Building the linear model in R has a bunch of advantages over doing the same in Excel.

Build a linear model with multiple independent variables : Regress the returns of an oil stock against S&P 500 and the returns of an exchange traded oil fund. 

We describe how categorical variables can be built into a linear model, and how to do this in R specifically

rlm() helps you build Robust linear models that downweight the influence of outliers.

lm() returns a bunch of diagnostic plots that are used to validate the assumptions underlying linear regression - Q-Q plots, Scale-location and Cook's distance plots

Data Visualization gives you the power to effectively get your point across and to deeply understand your data.

The plot() function in R

Control color palettes with RColorbrewer

Drawing barplots

Drawing a heatmap

Drawing a Scatterplot Matrix

ggplot2 is a pretty cool R package for complex 2D graphics. Plot the time series of 4 different stocks in the same graph.