Thinking About Data

Example

Representing a function as a Power Series

• Geometric Series
  

  First take the Series:

  \[\begin{aligned} S_n &= \sum_{n=0}^{n} r^k \\ &= 1 + r + r^2 + r^3 \ldots + r^{n-1} + r^n \\ \implies r \cdot S_n &= r + r^2 + r^3 + r^4 \ldots r^n + r^{n+ 1} \\ \implies S_n - r \cdot C_n &= 1 + r^{n+ 1} \\ \implies S_n &= \frac{1+ r^{n+ 1}}{1- r} \end{aligned}\]

  So now consider the geometric series:

  \[\begin{aligned} \sum^{\infty}_{k= 0} \left[ x^k \right] &= \lim_{n \rightarrow \infty}\left[ \sum^{n}_{k= 0} x^k \right]\\ &= \lim_{n \rightarrow \infty}\left[ \frac{1+ x^{n+ 1}}{1- x} \right]\\ &= \frac{1+ \lim_{n \rightarrow \infty}\left[ x^{n+ 1} \right]}{1 - x} \\ &= \frac{1+ 0}{1- x}\\ &= \frac{1}{1- x} \end{aligned}\]

• Using The Geometric Series to Create a Power Series
  

  Take for example the function:

  \[\begin{aligned} g\left( x \right)&= \frac{1}{1 + x^2} \end{aligned}\]

  This could be represented as a power series by observing that:

  \[\begin{aligned} \frac{1}{1- \#_1} = \sum_{n=0}^{\infty} \left[ \#_1^n \right] \end{aligned}\]

  And then simply putting in the value of \(\#_1 = \left( - x^2 \right)\) :

  \[\begin{aligned} \frac{1}{1- \left( - x^2 \right) } = \sum_{n=0}^{\infty} \left[ \left( - x^2 \right) ^n \right] \end{aligned}\]

Calculus Rules and Series

• Differentiation
  

  \[\frac{\mathrm{d} }{\mathrm{d} x}\left( \sum_{n=1}^{\infty} c_n\left( z- a \right) ^n \right) = \sum_{n=1}^{\infty} \left[ \frac{\mathrm{d} }{\mathrm{d} x}\left( c_n\left( z- a \right) ^n \right) \right] \]

• Integration
  

  \[\int \left( \sum_{n=1}^{\infty} c_n \left( z- a \right) ^n\right) \mathrm{d}x = \sum_{n=1}^{\infty} \left[ c_n \left( z- a \right) ^n \right] \]

Taylor Series

• Power Series Expansion of \(e\)
  

  \[\begin{aligned} f\left( z \right)= e^z&= \sum^{\infty}_{n= 0} \left[ \frac{f^{\left( n \right)}\left( 0 \right)}{n!}\cdot x^n \right]\\ &= \sum^{\infty}_{n= 0} \left[ \frac{e^0}{n!}x^n \right] \\ &= \sum^{\infty}_{n= 0} \left[ \frac{x^n}{n!} \right] \end{aligned}\]

what is the $y$-axis in a Density curve?   ggplot2 ATTACH

     # layout(mat = matrix(1:6, nrow = 3))
      layout(matrix(1:6, 3, 2, byrow = TRUE))


      x <- rnorm(10000, mean = 0, sd = 1)
      sd(x)
      hist(rnorm(10000), breaks = 5, freq = FALSE)
      ## curve(dnorm(x, 0, 1), add = TRUE, lwd = 3, col = "royalblue")

      hist(rnorm(10000), breaks = 10, freq = FALSE)
      ## curve(dnorm(x, 0, 1), add = TRUE, lwd = 3, col = "royalblue")

      hist(rnorm(10000), breaks = 15, freq = FALSE)
     ##  curve(dnorm(x, 0, 1), add = TRUE, lwd = 3, col = "royalblue")

      hist(rnorm(10000), breaks = 20, freq = FALSE)
       curve(dnorm(x, 0, 1), add = TRUE, lwd = 3, col = "royalblue")

      hist(rnorm(10000), breaks = 25, freq = FALSE)
       curve(dnorm(x, 0, 1), add = TRUE, lwd = 3, col = "royalblue")

      hist(x, breaks = 30, freq = FALSE, col = "lightblue")
      curve(dnorm(x, 0, 1), add = TRUE, lwd = 3, col = "royalblue")

   library(tidyverse)
   library(gridExtra)
   x <- rnorm(10000)
   x <- tibble::enframe(x)
   head(x)
   PlotList <- list()
   for (i in seq(from = 5, to = 30, by = 5)) {
     PlotList[[i/5]] <- ggplot(data = x, mapping = aes(x = value)) +
       geom_histogram(aes(y = ..density..), col = "royalblue", fill = "lightblue", bins = i) +
       stat_function(fun = dnorm, args = list(mean = 0, sd = 1))+
       theme_classic()
   }

   # arrangeGrob(grobs = PlotList, layout_matrix = matrix(1:6, nrow = 3))
   grid.arrange(grobs = PlotList, layout_matrix = matrix(1:6, nrow = 3))

Defining the Normal Distribution

Modelling only distance from the mean

Incorporating Proportional to Frequency

Putting both Conditions together

• IC, Probability Adds to 1
  

  The area bound by the curve must be 1 because it represents probability, hence:

  \[\begin{aligned} 1&= \int_{-\infty}^{\infty} f \mathrm{d}f \\ 1&= - C \int_{-\infty}^{\infty} e^{\frac{k}{2} \left( x- \mu \right)^2} \mathrm{d}f \\ \end{aligned}\]

  Using integration by substitution:

  \[\begin{aligned} \text{let:} \quad u^2&= \frac{k}{2}\left( x - \mu \right)^2\\ u&= \sqrt{\frac{k}{2}} \left( x- \mu \right) \\ \frac{\mathrm{d}u }{\mathrm{d} x}&= \sqrt{\frac{k}{2}} \end{aligned}\]

  hence:

  \[\begin{aligned} 1&= - C \int_{-\infty}^{\infty} e^{\frac{k}{2} \left( x- \mu \right)^2}\\ 1&= \sqrt{\frac{2}{k}} \cdot C \int_{-\infty}^{\infty} e^{- u^2} \mathrm{d}u \\ 1^2&= \left( \sqrt{\frac{2}{k}} \cdot C \int_{-\infty}^{\infty} e^{- u^2} \mathrm{d}u \right)^2\\ 1^2&= \left( \sqrt{\frac{2}{k}} \cdot C \int_{-\infty}^{\infty} e^{- u^2} \mathrm{d}u \right) \times \left( \sqrt{\frac{2}{k}} \cdot C \int_{-\infty}^{\infty} e^{- u^2} \mathrm{d}u \right) \end{aligned}\]

  Because this is a definite integral \(u\) is merely a dummy variable and instead we can make the substitution of \(x\) and \(y\) for clarity sake.

  \[\begin{aligned} 1^2&= \left( \sqrt{\frac{2}{k}} \cdot C \int_{-\infty}^{\infty} e^{- x^2} \mathrm{d}x \right) \times \left( \sqrt{\frac{2}{k}} \cdot C \int_{-\infty}^{\infty} e^{- y^2} \mathrm{d}y \right) \end{aligned}\]

  Now presume that the definite integral is equal to some real constant \(\beta \in \mathbb{R}\):

  \[\begin{aligned} 1&= \frac{2}{k}\cdot C^2 \int_{-\infty}^{\infty} e^{- y^2} \mathrm{d}y \times \beta \\ &= \frac{2}{k}\cdot C^2 \int_{-\infty}^{\infty} \beta\cdot e^{- y^2} \mathrm{d}y \\ &= \frac{2}{k}\cdot C^2 \cdot \int_{-\infty}^{\infty} \left( \int_{-\infty}^{\infty} e^{- x^2} \mathrm{d}x \right)e^{- y^2} \mathrm{d}y\\ &= \frac{2}{k}\cdot C^2 \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} e^{- \left( x^2+ y^2 \right)} \mathrm{d}x \mathrm{d}y \end{aligned}\]

  This integral will be easier to evaluate in polar co-ordinates, a double integral may be evaluated in polar co-ordinates using the following relationship: ³

  \[\begin{aligned} \iint_D f\left( x,y \right) dA = \int_{\alpha}^{\beta} \int_{h_1\left( \phi \right)}^{h_2\left( \phi \right)} f\left( r \cdot \cos{\left( \phi \right)} , r \cdot \sin \left( \phi \right)\right) \mathrm{d}r \mathrm{d}\phi \end{aligned}\]

  hence this simplifies to:

  \[\begin{aligned} 1&= \frac{2}{k} c ^2 \int_{0}^{2 \pi } \int_{0}^{r} r \cdot e^{\left( r \cdot \cos{\theta}\right)^2+ \left( r \cdot \sin{\theta} \right)^2} \mathrm{d}r \mathrm{d}\theta \\ 1&= \frac{2}{k} c ^2 \int_{0}^{2 \pi} \int_{0}^{r} r \cdot e^{r^2} \mathrm{d}r \mathrm{d}\theta \end{aligned}\]

  Because the integrand is of the form \(f'\left( x \right)\times g\left( f\left( x \right) \right)\) we may use integration by substitution:

  \[\begin{aligned} \text{let:} \quad u&= - r^2\\ \frac{\mathrm{d}u }{\mathrm{d} r} &= - 2 r \\ \mathrm{d} r &= - \frac{1}{2r} \mathrm{d} u \end{aligned}\]

  and hence:

  \[\begin{aligned} 1&= \frac{2}{k} c ^2 \int_{0}^{2 \pi} \int_{0}^{r} r \cdot e^{r^2} \mathrm{d}r \mathrm{d}\theta \\ \implies 1&= - \frac{2}{k}c^2 \int_{0}^{2 \pi} \int_{0}^{\infty} r \cdot e^{r^2} \mathrm{d}r \mathrm{d}\theta \end{aligned}\]

  \[\begin{aligned} 1&= \frac{2}{k} c ^2 \int_{0}^{2 \pi} \int_{0}^{r} r \cdot e^{r^2} \mathrm{d}r \mathrm{d}\theta \\ \implies 1&= - \frac{2}{k}c^2 \int_{0}^{2\pi} \int_{0}^{\infty} - \frac{1}{2}e^{- u} \mathrm{d}u \mathrm{d}\theta \\ &= \frac{1}{k}c^2 \int_{0}^{2\pi} \int_{0}^{\infty} e^{- u} \mathrm{d}u \mathrm{d}\theta \\ &= \frac{1}{k} c^2 \int_{0}^{2\pi} \left[ - e^{- u} \right]_{0}^{\infty}\mathrm{d}\theta \\ 1&= \frac{1}{k}c^2 2 \pi \\ \implies C^2&= \frac{k}{2\pi} \end{aligned}\]

  So from before:

  \[\begin{aligned} f&= - C \cdot e^{k\cdot \frac{\left( x- \mu \right)^2}{2}} \\ &= - \sqrt{\frac{k}{2\pi}} \cdot e^{k\cdot \frac{\left( x- \mu \right)^2}{2}} \end{aligned}\]

  so now we simply need to apply the next initial condition.

• IC, Mean Value and Standard Deviation
  
  • Definitions
    

    The definition of the expected value, where \(f(x)\) is a probability function is: ⁴

    \[\begin{aligned} \mu= E(x) = \int^{b}_{a} x \cdot f\left( x \right) \mathrm{d}x \end{aligned}\]

    That is, roughly, the sum of the expected proportion of occurence.

    The definition of the variance is:

    \[\begin{aligned} V\left( x \right)= \int_{a}^{b} \left( x- \mu \right)^2 f\left( x \right) \mathrm{d}x \end{aligned}\]

    which can be roughly interpreted as the sum of the proportion of squared distance units from the mean. The standard deviation is \(\sigma = \sqrt{V(x)}\).

  • Expected Value of the Normal Distribution
    

    The expected value of the normal distribution is \(\mu\), this can be shown rigorously:

    \[\begin{aligned} \text{let:} \quad v&= x- \mu \\ \implies \mathrm{d} v&= \mathrm{d} x \end{aligned}\]

    Observe that the limits of integration will also remain as \(\pm \infty\) following the substitution:

    \[\begin{aligned} E\left( v \right)&= \int_{-\infty}^{\infty} v\times f\left( v \right) \mathrm{d}v \\ &= k\cdot \int_{-\infty}^{\infty} v\cdot e^{v^2} \mathrm{d}v \\ &= \frac{1}{2} \left[ e^{x^2} \right]^{\infty}_{\infty}\\ &= \frac{1}{2} \lim_{b \rightarrow \infty} \left[\left[ e^{x^2} \right]^{b}_{-b} \right] \\ &= \frac{1}{2} \lim_{b \rightarrow \infty} \left[ e^{b^2} - e^{\left( - b \right)^2} \right]\\ &= \lim_{b \rightarrow \infty}\left[ 0 \right] \times \frac{1}{2}\\ &= \frac{1}{2} \times 0 \\ &= 0 \end{aligned}\]

    Hence the Expected value of the standard normal distribution is \(0=x-\mu\) and so \(E(x)=\mu\).

  • Variance of the Normal Distribution
    

    Now that the expected value has been confirmed, consider the variance of the distribution:

    \[\begin{aligned} \sigma^2 &= \int_{-\infty}^{\infty} \left( x- \mu \right) ^2 \times f \left( x \right) \mathrm{d}x \\ \end{aligned}\]

    Now observe that \(\left( x- \mu \right)\) appears as an exponential and as a factor if this is redefined as \(w= x- \mu \implies \mathrm{d} x= \mathrm{d} w\) we have:

    \[\begin{aligned} \sigma^2&= \sqrt{\frac{k}{2}} \int_{-\infty}^{\infty} w^2e^{-\frac{k}{2}w^2} \mathrm{d}w \end{aligned}\]

    Now the integrand is of the form \(f\left( x \right)\times g\left( x \right)\) meaning that the only strategy to potentially deal with it is integration by parts:

    \[\begin{aligned} \int u \mathrm{d}v&= u\cdot v- \int v \mathrm{d}u \end{aligned}\] where:

    • \(u\) is a function that simplifies with differentiation
    • \(\mathrm{d} v\) is something that can be integrated

    \begin{matrix} u= w & \enspace & \mathrm{d}v = w \cdot e^{- \frac{k}{2}w^2}\mathrm{d} w \\ \implies \mathrm{d}u = \mathrm{d}w & \enspace & \implies v= \int w\cdot e^{\frac{k}{2} w^2} \mathrm{d}w \\ & \enspace & \implies v=\frac{1}{k} e^{\frac{k}{2}w^2} \end{matrix}

    Hence the value of the variance may be solved:

    Now that the expected value has been confirmed, consider the variance of the distribution:

    \[\begin{aligned} \sigma^2&= \int_{-\infty}^{\infty} \left( x- \mu \right)^2\times f\left( x \right) \mathrm{d}x \\ &= \int_{-\infty}^{\infty} \left( x- \mu \right)^2\times \left( \sqrt{\frac{k}{2\pi}}e^{- \frac{k}{2}\left( x- \mu \right)^2} \right)\mathrm{d}x \\ &= \sqrt{\frac{k}{2\pi}} \int_{-\infty}^{\infty} \left( x- \mu \right)^2\times \left( e^{- \frac{k}{2}\left( x- \mu \right)^2} \right)\mathrm{d}x \end{aligned}\]

    Now observe that \(\left( x- \mu \right)\) appears as an exponential and as a factor if this is redefined as \(w= x- \mu \implies \mathrm{d} x= \mathrm{d} w\) we have:

    \[\begin{aligned} \sigma^2&= \sqrt{\frac{k}{2}} \int_{-\infty}^{\infty} w^2e^{-\frac{k}{2}w^2} \mathrm{d}w \end{aligned}\]

    Now the integrand is of the form \(f\left( x \right)\times g\left( x \right)\) meaning that the only strategy to potentially deal with it is integration by parts:

    \[\begin{aligned} \int u \mathrm{d}v&= u\cdot v- \int v \mathrm{d}u \end{aligned}\] where:

    • \(u\) is a function that simplifies with differentiation
    • \(\mathrm{d} v\) is something that can be integrated

    \begin{matrix} u= w & \enspace & \mathrm{d}v = w \cdot e^{- \frac{k}{2}w^2}\mathrm{d} w \\ \implies \mathrm{d}u = \mathrm{d}w & \enspace & \implies v= \int w\cdot e^{\frac{k}{2} w^2} \mathrm{d}w \\ & \enspace & \implies v=\frac{1}{k} e^{\frac{k}{2}w^2} \end{matrix}

    Hence the value of the variance may be solved:

    \[\begin{aligned} \sigma^2&= \sqrt{\frac{k}{2\pi}} \int_{-\infty}^{\infty} w^2e^{-\frac{k}{2}w^2} \mathrm{d}w \\ &= \sqrt{\frac{k}{2\pi}} \left[ u\cdot v - \int v \mathrm{d}u \right]^{\infty}_{\infty}\\ &= \sqrt{\frac{k}{2\pi}} \left( \left[ \frac{-w}{k}\cdot e^{-\frac{k}{2}w^2} \right]^{\infty}_{\infty} - \frac{1}{k} \int^{\infty}_{-\infty} e^{\frac{k}{2}w^2} \mathrm{d}w \right) \\ &= \sqrt{\frac{k}{2\pi}} \left[ \frac{-w}{k}\cdot e^{-\frac{k}{2}w^2} \right]^{\infty}_{\infty} - \frac{1}{k} \left( \sqrt{\frac{k}{2\pi}} \int^{\infty}_{-\infty} e^{\frac{k}{2}w^2} \mathrm{d}w \right) \\ \end{aligned}\]

    The left term evaluates to zero and the right term is the area beneath the bell curve with mean value 0 and so evaluates to 1:

    \[\begin{aligned} \sigma^2&= 0- \frac{1}{k}\\ \implies k&= \frac{1}{\sigma^2} \end{aligned}\]

    So the function for the density curve can be simplified:

    \[\begin{aligned} &= - \sqrt{\frac{k}{2\pi}} \cdot e^{k\cdot \frac{\left( x- \mu \right)^2}{2}}\\ &= \sqrt{\frac{1}{2\pi \sigma^2}} \cdot e^{ \frac{1}{2}\cdot \frac{\left( x- \mu \right)^2}{\sigma^2}} \end{aligned}\]

    now let \(z= \frac{ x- \mu }{\sigma} \implies \mathrm{d}z= \frac{\mathrm{d} x}{\sigma}\), this then simplifies to:

    \[\begin{aligned} f\left( x \right)= \sqrt{\frac{1}{2\pi}}\cdot e^{- \frac{1}{2}z^2} \end{aligned}\]

    Now using the power series identity from BEFORE :

    \[\begin{aligned} e^{- \frac{1}{2}z^2}= \sum^{\infty}_{n= 0} \left[\frac{ \left( - \frac{1}{2}z^2 \right)^n}{n!} \right] \end{aligned}\]

    We can solve the integral of \(f\left( x \right)\) (which has no elementary integral.

    \[\begin{aligned} f\left( x \right)&= \sqrt{\frac{1}{2\pi}}\cdot \sum^{\infty}_{n= 0} \left[\frac{ \left( - \frac{1}{2}z^2 \right)^n}{n!} \right] \\ \int f\left( x \right) \mathrm{d}x &= \frac{1}{\sqrt{2\pi} }\int \sum^{\infty}_{n= 0} \left[ \frac{\left( - \frac{1}{2}z^2 \right)^n}{n!} \right] \mathrm{d}z \\ &= \frac{1}{\sqrt{2\pi} }\cdot \sum^{\infty}_{n= 0} \left[ \int \frac{\left( - 1 \right)^{- 1}z^{2n}}{2^n\cdot n!} \mathrm{d}z \right] \\ &= \frac{1}{\sqrt{2\pi} }\cdot \sum^{\infty}_{n= 0} \left[ \frac{\left( - 1 \right)^n \cdot z^{2n+ 1}}{2^n\left( 2n+ 1 \right)n!} \right] \end{aligned}\]

    Although this is a power series it still gives a method to solve the area beneath the curve of the density function of the normal distribution.

Problem

Solution

• Hypotheses
  
  1. \(H_0 \quad :\) The Null Hypothesis that the average internet usage is 10 hours per week
  2. \(H_a \quad :\) The Alternative Hypothesis that the average internet usage exceeds 10 hours a week
• Data
  

  Value	Description
  \(n=100\)	The Sample Size
  \(\sigma= 5.2\)	The Standard Deviation of internet usage of the population
  \(\mu=10\)	The alleged average internet usage.
  \(\bar{x}=11\)	The average of the sample
  \(\mu_True=12\)	The actual average from the population
  \(\alpha=0.05\)	The probability of a type 1 error at which the null hypothesis is rejected
  \(\beta=??\)	The probability of a type 2 error

Step 1; Find the Critical Sample Mean (\(\bar{x}_\mathsf{crit}\))

Step 2: Find the Difference between the Critical and True Means as a Z-Value (prob of Type II)

Step 3: State the value of \(\beta\)

Step 4: State the Power Value

The Reasoning

Lecture

• Boxplots
  

  The delimiting marks in box plots correspond to the median and interquarlite range (which is basically the median of all data below the median):

       library("ggplot2")
       ggplot(iris, aes(x = Sepal.Width, y = Sepal.Length, color = Species)) +
       geom_boxplot() +
       theme_bw()
  

  

  Figure 5: Bar Plots Generated in ggplot2

• Lecture Announcements
  

  Everything is online now. We’ll be using Zoom a lot.

  • DONE Finish Quiz 1
    
  • TODO Finish Quiz 1
    

    30 minutes to finish it, Test your computer First.

  • Post pacman on the mailing list.
    
• Naming Variables
  

  Attribute … Data Base

• DONE Review Chi Distribution,
  
  • Is it in VNote?
    
  • Should I put it in org-mode?
    
• DONE Fix YAML Headers in rmd to play ball with Notable
  
  • DONE Post this use-case to Reddit
    
  • DONE Fix YAMLTags and TagFilter and Post to Reddit   bash
    

    The Bash Script is here

    • TODO Post an easy way to use this to the mailing list
      

      Here is a start that I’ve made

    • Should I have all the lists and shit in /tmp
      

      Pros	Cons
      Less Mess	Harder to Directly watch what’s happening
      Easier to manage 5

      should I use /tmp or /tmp/Notes or somting?

    • DONE Is there an easy way to pass the md off to vnote
      

      Or should I just use the ln -s ~/Notes/DataSci /tmp/notes trick?

      Follow the instructions here, it has to be done manually and then symlinked SCHEDULED: <2020-03-21 Sat>

    • Should I have all the lists and shit in /tmp
      

      Pros	Cons
      Less Mess	Harder to Directly watch what’s happening
      Easier to manage 5

      should I use /tmp or /tmp/Notes or somting?

  • DONE Is there a way to fix the Text Size of Code in emacs when I zoom out?
    

    Yeah just disable M-x mixed-pitch-mode

• Calculalating mean
  

   library(tidyverse)
   bwt <- c(3429, 3229, 3657, 3514, 3086, 3886)
   (bwt <- sort(bwt))
   mean(bwt)
   mean(c(3429, 3514))
   median(bwt)
   max(bwt)-min(bwt)
  

  
   [1] 3086 3229 3429 3514 3657 3886
  
   [1] 3466.833
  
   [1] 3471.5
  
   [1] 3471.5
  
   [1] 800
  

  The mean value is nice in that it has good mathematical properties, so for predictions and classifications (like gradient descent), if the model contains the mean the model will be smooth and the mean will lead to a well behaved model with respect to the derivative.

  The Median value, however is more immune to large outliers, for example:

   library(tidyverse)
   x <- c(rnorm(10), 9) * 10 %>% round(1)
   mean(x); median(x)
  

   ── Attaching packages ────────────────────────────────── tidyverse 1.3.0.9000 ──
   ✔ ggplot2 3.2.1     ✔ purrr   0.3.3
   ✔ tibble  2.1.3     ✔ dplyr   0.8.4
   ✔ tidyr   1.0.2     ✔ stringr 1.4.0
   ✔ readr   1.3.1     ✔ forcats 0.5.0
   ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
   ✖ dplyr::filter() masks stats::filter()
   ✖ dplyr::lag()    masks stats::lag()
  
   [1] 5.337263
   [1] 0.6294542
  

• Calculating Range
  

   range(bwt)
   bwt %>% range %>% diff
  

  [1] 3086 3886
  
  [1] 800
  

• Calculating Variance
  

     (var <- (bwt-mean(bwt))^2 %>%  mean)
     var(bwt)
     (sd <- (bwt-mean(bwt))^2 %>%  %>% sqrt) # Not using n-1 !!
     (sd <- sqrt(sum((bwt-mean(bwt))^2)/(length(bwt) -1)))
     sd(bwt)
     mean(sum((bwt-mean(bwt))^2))
  

   [1] 69519.81
  
   [1] 83423.77
  
   Error: unexpected SPECIAL in "(sd <- (bwt-mean(bwt))^2 %>%  %>%"
  
   [1] 288.8317
  
   [1] 288.8317
  
   [1] 417118.8
  

• InterQuartile Data
  

Tutorial

DONE Quizz for t Distribution Material

Lecture

• Combinatorics
  

  The Counting Formulas are:

  selection	ordered	unordered
  With Repetition	\(n^m\)	\(\binom{m+n-1}{n}\)
  Without Repetition	\(n_{(m)}\)	\(\binom{n}{m}\)

  Where:

  • \(\binom{n}{m} =\frac{m_(n)}{n!}=\frac{m!}{n!(m-n)!}\)
  • \(n_{(m)}=\frac{n!}{(n-m)!}\)
  • \(n! = n \times (n-1) \times (n-2) \times 2 \times 1\)
• Binomial Distribution
  

  A Binomial experiment requires the following conditions:

  1. We have n independent events.
  2. Each event has the same probability p of success.
  3. We are interested in the number of successes from the \(n\) trials (referred to as size in R.
  4. The probability of k successes from the n trials is a Binomial distribution with

  probabilities:

  \[\begin{aligned} P\left( k \right)= \binom{n}{k} p^k \left( 1- p \right)^{n- k} \end{aligned}\]

  • Problem
    

    Hard drives have an annual failure rate of 10%, what is the probability of 2 hard drives failing after 3 years?

    This means that:

    • The number of repetitions is 3 (\(n\) = size = 3)

    • The statistic we are interested in is 2 (\(k\) = x = 2)

              dbinom(x = 2, size = 3, prob = 0.1)
      
              ## For Hard Drives
                  ## k/x....is the number of years
                  ## n/ ....size is the number of failures
                  ## p  ....is the probability of failure
      
              ## choose(n,k)*p^k*(1-p)^(n-k)
              ## dbinom(x = 2, size = 4, prob = p)
      

      [1] 0.027
      

      So in this case there would only a 2% chance.

• Poisson
  

  An interesting thing with the poisson distribution is that the mean value and the variance are both equal.

  The expected value is the limit that the mean value would approach if the sample was made arbitrarily large, the value is denoted \(\lambda\)

  Poisson is French for fish so sometimes people call the distribution the fishribution.

• Binomial and Poisson
  

  The Poisson distribution is derived from the Binomial distribution.

  If \((1-p)\) is close to 1 then \(np(1-p) \approx np\), so for very large sample sizes we have the expected value equal to the variance, and a Poisson distribution.

  This has something to do with widely increasing the number of trials, like say if we hade an infinite number of trials with the probability of success in a given hour as 30%.

  For a binomial distribution there are a set number of trials, let’s say 8 trials with a 20% probability of Success:

  1	2	3	4	5	6	7	8
  F	S	F	F	S	S	F	F

  In this case there are 3 successes, so let’s set \(k=3\) and instead however the region was divided into smaller spaces (\(n \rightarrow \infty\)):

  1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
  X	S	X	X	S	S	F	F	X	S	X	X	S	S	F	F

  If we kept dividing this up we would have

  The poisson is the limit as we increase the number of trials but try to keep \(k\) constant??

• Confidence Intervals
  
  • Binomial
    

    Just use bootstrapping, but also you can just use an approximate standardisation:

    \[\begin{aligned} \sigma &= \sqrt{p\left( 1- p \right) } \\ \sigma_{\overline{x}}&= \frac{\sigma}{\sqrt{n} } \\ &= \frac{\sqrt{p\left( 1- p \right) }}{\sqrt{n} } \ \\ \implies p-z_{0.025}\times \sigma_{\overline{x}} < &p < p-z_{0.975}\times \sigma_{\overline{x}} \\ \implies p-1.96 \sigma_{\overline{x}} < &p < p-z_{0.975} 1.96 \sigma_{\overline{x}} \end{aligned}\]

    So an approximate 95% confidence interval could be

    This is the normal data, but remember that this is estimating binomial by standard normal and so will only be good for large values of n because binomial is discrete by nature.

    See the Correlation Notes and Khan Academy and this section section [[]]

    This means that for any sample drawn from the population, the true population value would be found within this interval for 0.95 of those samples

  • Poisson
    

    This can also be done with Poisson by bootstrapping or using the same trick of \(\lambda\) as the variance:

    \[\begin{aligned} \sigma &= \sqrt{\lambda} \\ \sigma_{\overline{x}}&= \frac{\sigma}{\sqrt{n} } \\ &= \frac{\sqrt{\lambda}}{\sqrt{n} } \ \\ \implies \lambda-z_{0.025}\times \sigma_{\overline{x}} < &p < \lambda+z_{0.975}\times \sigma_{\overline{x}} \\ \lambda - 1.96 \sigma_{\overline{x}} < &p < \lambda + 1.96 \sigma_{\overline{x}} \\ p-1.96 \sqrt{\frac{\lambda}{n}} < &p < p-z_{0.975} 1.96 \sqrt{\frac{\lambda}{n}} \end{aligned}\]

• Summary
  
  • Binomial for independent trials
    • mean: np
    • variance: np(1-p)
      • standard error roughly is \(\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}\)
  • Poisson for number of events in a given period
    • mean λ
    • variance λ
    • Standard error roughly is \(\sqrt{\frac{\hat{\lambda}}{n}}\)
  • Choropleth maps are useful for visualising changes over area.

Tutorial

DONE Lecture

• DONE How to Derive the Correlation Coefficient
  

  Refer to this paper The Correlation coefficient can be interpreted in one of two ways:

  • The covariance scaled relative to the \(x\) and \(y\) variance
    • \(\rho = \frac{S_{x, y}}{s_x \cdot s_y}\)
  • The rate of change of the line of best fit of the standardised data
    • This is equivalent to the rate of change of the line of best fit divided by \((\frac{s_y}{s_x})\):
      • \(\rho = b \cdot \frac{s_x}{s_y} \quad \iff \quad \hat{y_i}=bx_i + c\)

  This can be seen by performing linear regression in R:

    head(cars)
    cars_std <- as.data.frame(scale(cars))
    y <- cars$dist
    x <- cars$speed
  
    ## Correlation Coefficient
    cor(x = cars$speed, y = cars$dist)
  
    ## Covariance
    cov(x = cars$speed, y = cars$dist)/sd(cars$speed)/sd(cars$dist)
  
    ## Standardised Rate of Change
    lm(dist ~ speed, data = cars_std)$coefficients[2]
  
    ### Using Standardised Rate of change
    lm(dist ~ speed, data = cars)$coefficients[2] / (sd(y)/sd(x))
  

    speed dist
  1     4    2
  2     4   10
  3     7    4
  4     7   22
  5     8   16
  6     9   10
  
  [1] 0.8068949
  
  [1] 0.8068949
  
      speed 
  0.8068949
  
      speed 
  0.8068949
  

  #+BEGIN_SRC R :cache yes :exports both :results output graphics :file ./test.png
    library("ggplot2")
    cars_std <- as.data.frame(scale(cars))
  
    ggplot(cars_std, aes(x = speed, y = dist)) +
    geom_point() +
    geom_smooth(method = "lm") +
    theme_bw()
  

  

  This shows that despite noise data can still have a correlation coefficient of 1 if the noise is evenly distributed in a way that the correlation coefficient can have a rate of change of 1.

• TODO Prove the Correlation Coefficient and email Laurence
  
• Bootstrapping
  

  The big assumption with bootstrapping is that the population can be seen as equiv to an infinite repetition of the sample size.

  So assume that a population that is an infinite repetition of the sample, then take a sample of that infinite population and you have a bootstrap.

  So we could either create a population from the sample with a size of \(\infty\), which might be difficult, or, we could instead just resample the observation for each replication.

• Confidence Intervals
  

    load("Notes/DataSci/ThinkingAboutData/TAD.rdata ")
    r = cor(crabsmolt$postsz, crabsmolt$presz)
    a <- crabsmolt
    N <- nrow(crabsmolt)
    pos <- sample(N, size = N, replace = TRUE)
    aboot <- a[pos,]
  
    cor(aboot$postsz, aboot$presz)
  
  
  #  replicate(10^4, {})
  

  • Attached RMD
• TODO Questions
  

  In this part would it simply be equivalent to take the mean of all observations?

TODO Lecture

TODO Tutorial

• Measuring \(p\) -value for a particular slope
  

  Say we had two data sets as shown in 1:

  Listing 1: Generating Sample Data

  x    <- 1:10
  y    <- round(1:10 * 3 + rnorm(10), 2)
  (exampleData <- data.frame(x, y))
  

  1	2.66
  2	5.56
  3	10.19
  4	11.99
  5	15.85
  6	16.15
  7	19.78
  8	24.64
  9	26.98
  10	29.72

  Suppose that we wanted to know:

  The probability of incorrectly assuming that there was a non-zero slope for the linear regression between \(x\) and \(y\) when in fact the slope of the linear regression is zero and this observation was made by chance.

  then we could easily measure that by using summary and lm as shown in listing 2:

  Listing 2: Summarising a Linear Model

  summary(lm(y ~ x, exampleData))
  

  
  Call:
  lm(formula = y ~ x, data = exampleData)
  
  Residuals:
      Min      1Q  Median      3Q     Max
  -1.0919 -0.4014 -0.2601  0.2745  1.6065
  
  Coefficients:
              Estimate Std. Error t value Pr(>|t|)
  (Intercept)  -1.5033     0.6205  -2.423   0.0417 *
  x             3.2917     0.1000  32.917 7.91e-10 ***
  ---
  Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
  
  Residual standard error: 0.9083 on 8 degrees of freedom
  Multiple R-squared:  0.9927,	Adjusted R-squared:  0.9918
  F-statistic:  1084 on 1 and 8 DF,  p-value: 7.912e-10
  

  this provides a very low \(p\) value for a non-zero slope at 7.91e-10, which is too be expected because we set the data up in listing 2 to have a slope of 3.

  Now let’s modify our question, instead of testing for a slope of 0, say we wanted to test for a slope of 3, then our question would be:

  What is the probability of incorrectly accepting that there is a slope with a value different from 3, when in fact the slope of the data is actually 3?

  There’s nothing built in that just works, so we need to perform a boot strap like this:

  ## Calculate the current slope (offset by 3)
  slope <- lm((y-3*x) ~ x, data = exampleData)[[1]][2]
  
  ## Simulate null hypothesis
  
  sim <- replicate(10^3, {
      ## Shuffle the data
      x_perm <- sample(exampleData$x)
      ## Recalculate the slope
      slope_sim <- lm((y-x) ~ x_perm, data = exampleData)[[1]][2]
      ## Is the slope more extreme
      abs(slope_sim) > abs(slope)
  })
  
  (pval <- mean(sim))
  

  0.967
  

  This provides that the p-value is actually quite high, indicating a high probability of a Type I Error (Incorrectly rejecting the null hypothesis when it was true), so might conclude something like this:

  There is no evidence to support rejecting the hypotheis that the slope is 3

TODO Get the old ANOVA Notes and transcribbe them

• Pooled Variance
  

  Pooled variance is not the same as the variance of all the data, because, it uses the means of both groups, so it will be smaller than the variance of all the data put together.

• The sum of squares
  

  nil or better yet This PDF Export

  • Between groups
    

    Add the differences between the blobs of groups

  • Within groups
    

    The difference between Category Values and the category mean.

    Add the differences within the groups

  • TODO Add an inkscape image
    

    So we ratio SSB and SSW to measure how seperate the groups are.

    Via bootstrapping we can measure the probability of these groupbs being so blurry by chance.

    For the Hair Colour Data we have:

    	Dark Blonde	Dark Brunette	Light Blonde	Light Brunette
    ns	5	5	5	4
    menas	51.2	37.4	59.2	42.5
    vars	86.2	69.3	72.7	29.67

    xbark <- c(51.2, 37.4, 59.2, 42.5)
    n <- c(5,5,5,4)
    xbar = 47.85
    N <- sum(n)
    k <- length(n)
    
    ## Between SS
    SSB <- sum(n*(xbark-xbar)^2)
    
    ## Within SS
    vark <- c(86.2, 69.3, 72.7, 29.67)
    SSW <- sum(vark*(n-1))
    
    ## F Stat
    
    (F <- ((SSB)/(k-1)) / ((SSW)/(N-k)))
    

    
    [1] 6.791345
    

  • The FStatistic
    

    Now the question is:

    Is the F statistic large enough to reject the Null Hypothesis?

    or rather:

    What’s the probability of incorrectly rejecting the null hypothesis assuming that the null hypothesis is true.

    We can just do this in R:

    load("~/Notes/DataSci/ThinkingAboutData/TAD.rdata")
    oneway.test(pain ~ hair, data = hair2, var.equal = TRUE)
    names(hair2)
    

    If you use the coin library

    hair <- hair2
    names(hair2)
    
    ns = table (hair$hair)
    
    x = replicate (1000,{
        # permute the categories (to satisfy H0)
        hair.perm <- sample(hair$hair)
        fit0 = aov (pain ~ hair.perm, data=hair)
        # compute MSE and means on permuted categories
        MSE = summary(fit0)[[1]][2,3] ## Extract the residual MSE
        means = aggregate(pain ~ hair.perm, data=hair, mean)[,2]
        # compute t statistic for all pairs
        ## Plus and Minus do weird things
        Ts = outer(means, means, "-")/ sqrt(outer(1/ns,1/ns, "+"))
        Ts = Ts/ sqrt (MSE)
        # keep only the largest t statistic
        max ( abs (Ts))
    })
    
    
    c(1,2,3) %*% c(4,5,6)
    c(1,2,3) %o% c(4,5,6)
    

  • R Prob Functions
    
    • p is cummulative probability
    • d is the area
    • r is a random number

Lecture

Tutorial