1 

     2 =============

     3  Probability

     4 =============

     5 .. contents::

     6    :local:

     7 

     8 .. role:: def

     9    :class: def

    10 

    11 PMF

    12 ===

    13 

    14 :def:`PMF` or :def:`probability mass function` or :def:`probability law` or

    15 :def:`probability discribuion` of discrete random variable is a function that

    16 for given number give probability of that value.

    17 

    18 To denote PMF used notations:

    19 

    20 .. math::

    21 

    22    PMF(X = x) = P(X = x) = p_X(x) = P({ω ∈ Ω: X(ω) = x})

    23 

    24 where :math:`X` is a random variable on space :math:`Ω` of outcomes which mapped

    25 to real number via :math:`X(ω)`.

    26 

    27 Expected value

    28 ==============

    29 

    30 :def:`Expected value` of PMF is:

    31 

    32 .. math::

    33 

    34   E[X] = Σ_{ω∈Ω} Χ(x) * p(ω) = Σ_{x} x * p_X(x)

    35 

    36 We write :math:`a ≤ X ≤ b` for :math:`∀ ω∈Ω a ≤ X(ω) ≤ b`.

    37 

    38 If :math:`X ≥ 0` then :math:`E[X] ≥ 0`.

    39 

    40 if :math:`a ≤ X ≤ b` then :math:`a ≤ E[X] ≤ b`.

    41 

    42 If :math:`Y = g(X)` (:math:`∀ ω∈Ω Y(ω) = g(X(ω))`) then:

    43 

    44 .. math::

    45 

    46   E[Y] = Σ_{x} g(x) * p_X(x)

    47 

    48 **Proof** TODO:

    49 

    50 .. math::

    51 

    52   E[Y] = Σ_{y} y * p_Y(y)

    53 

    54   = Σ_{y∈ℝ} y * Σ_{ω∈Ω: Y(ω)=y} p(ω)

    55 

    56   = Σ_{y∈ℝ} y * Σ_{ω∈Ω: g(X(ω))=y} p(ω)

    57 

    58   = Σ_{y∈ℝ} y * Σ_{x∈ℝ: g(x)=y} Σ_{ω∈Ω: X(ω) = x} p(ω)

    59 

    60   = Σ_{y∈ℝ} y * Σ_{x∈ℝ: g(x)=y} p_X(x)

    61 

    62   = Σ_{y∈ℝ} Σ_{x∈ℝ: g(x)=y} y * p_X(x)

    63 

    64   = Σ_{x∈ℝ} Σ_{y∈ℝ: g(x)=y} y * p_X(x)

    65 

    66   = Σ_{x} g(x) * p_X(x)

    67 

    68 .. math::

    69 

    70   E[a*X + b] = a*E[X] + b

    71 

    72 Variance

    73 ========

    74 

    75 :def:`Variance` is a:

    76 

    77 .. math::

    78 

    79   var[X] = E[(X - E[X])^2] = E[X^2] - E^2[X]

    80 

    81 :def:`Standard deviation` is a:

    82 

    83 .. math::

    84 

    85   σ_Χ = sqrt(var[X])

    86 

    87 Property:

    88 

    89 .. math::

    90 

    91   var(a*X + b) = a²̇ · var[X]

    92 

    93 

    94 Total probability theorem

    95 =========================

    96 

    97 Let :math:`A_i ∩ A_j = ∅` for :math:`i ≠ j` and :math:`∑_i A_i = Ω`:

    98 

    99 .. math::

   100 

   101   p_X(x) = Σ_i P(A_i)·p_{X|A_i}(x)

   102 

   103 Conditional PMF

   104 ===============

   105 

   106 :def:`Conditional PMF` is:

   107 

   108 .. math::

   109 

   110   p_{X|A}(x) = P(X=x | A)

   111 

   112   E[X|A] = ∑_x x·p_{X|A}(x)

   113 

   114 Total expectation theorem

   115 =========================

   116 

   117 .. math::

   118 

   119   E[X] = Σ_i P(A_i)·E[X|A_i]

   120 

   121 To prove theorem just multiply total probability theorem by :math:`x`.

   122 

   123 Joint PMF

   124 =========

   125 

   126 :def:`Joint PMF` of random variables :math:`X_1,...,X_n` is:

   127 

   128 .. math::

   129 

   130    p_{X_1,...,X_n}(x_1,...,x_n) = P(AND_{x_1,...,x_n}: X_i = x_i)

   131 

   132 Properties:

   133 

   134 .. math::

   135 

   136   E[X+Y] = E[X] + E[Y]

   137 

   138 Conditional PMF

   139 ===============

   140 

   141 :def:`Conditional PMF` is:

   142 

   143 .. math::

   144 

   145   p_{X|Y}(x|y) = P(X=x | Y=y) = P(X=x \& Y=y) / P(Y=y)

   146 

   147 So:

   148 

   149 .. math::

   150 

   151   p_{X,Y}(x,y) = p_Y(y)·p_{X|Y}(x|y) = p_X(x)·p_{Y|X}(y|x)

   152 

   153   p_{X,Y,Z}(x,y,z) = p_Y(y)·p_{Z|Y}(z|y)·p_{X|Y,Z}(x|y,z)

   154 

   155   ∑_{x,y} p_{X,Y|Z}(x,y|z) = 1

   156 

   157 Conditional expectation of joint PMF

   158 ====================================

   159 

   160 :def:`Conditional expectation of joint PMF` is:

   161 

   162 .. math::

   163 

   164   E[X|Y=y] = ∑_x x·p_{X|Y}(x|y)

   165 

   166   E[g(X)|Y=y] = ∑_x g(x)·p_{X|Y}(x|y)

   167 

   168 Total probability theorem for joint PMF

   169 =======================================

   170 .. math::

   171 

   172   p_X(x) = ∑_y p_Y(y)·p_{X|Y}(x|y)

   173 

   174 Total expectation theorem for joint PMF

   175 =======================================

   176 .. math::

   177 

   178   E[X] = ∑_y p_Y(y)·E[X|Y=y]

   179 

   180 Independence of r.v.

   181 ====================

   182 

   183 r.v. :math:`X` and :math:`Y` is :def:`independent` if:

   184 

   185 .. math::

   186 

   187   ∀_{x,y}: p_{X,Y}(x,y) = p_X(x)·p_Y(y)

   188 

   189 So if two r.v. are independent:

   190 

   191 .. math::

   192 

   193   E[X·Y] = E[X]·E[Y]

   194 

   195   var(X+Y) = var(X) + var(Y)

   196 

   197 Well known discrete r.v.

   198 ========================

   199 

   200 Bernoulli random variable

   201 -------------------------

   202 

   203 :def:`Bernoulli random variable` with parameter :math:`p` is a random variable

   204 that have 2 outcomes denoted as :math:`0` and :math:`1` with probabilities:

   205 

   206 .. math::

   207 

   208   p_X(0) = 1 - p

   209 

   210   p_X(1) = p

   211 

   212 This random variable models a trial of experiment that result in success or

   213 failure.

   214 

   215 :def:`Indicator` of r.v. event :math:`A` is function::

   216 

   217    I_A = 1 iff A occurs, else 0

   218 

   219 .. math::

   220 

   221   P_{I_A} = p(I_A = 1) = p(A)

   222 

   223   I_A*I_B = I_{A∩B}

   224 

   225 .. math::

   226 

   227   E[bernoulli(p)] = 0*(1-p) + 1*p = p

   228 

   229   var[bernoulli(p)] = E[bernoulli(p) - E[bernoulli(p)]]

   230 

   231    = (0-p)²·(1-p) + (1-p)²·p = p²·(1-p) + (1 - 2p + p²)·p

   232 

   233    = p² - p³ + p - 2·p² + p³ = p·(1-p)

   234 

   235 Discret uniform random variable

   236 -------------------------------

   237 

   238 :def:`Discret uniform random variable` is a variable with parameters :math:`a`

   239 and :math:`b` in sample space :math:`{x: a ≤ x ≤ b & x ∈ ℕ}` with equal

   240 probability of each possible outcome:

   241 

   242 .. math::

   243 

   244   p_{unif(a,b)}(x) = 1 / (b-a+1)

   245 

   246 .. math::

   247 

   248   E[unif(a,b)] = Σ_{a ≤ x ≤ b} x * 1/(b-a+1)

   249   = 1/(b-a+1) * Σ_{a ≤ x ≤ b} x

   250 

   251   = 1/(b-a+1) * (Σ_{a ≤ x ≤ b} a + Σ_{0 ≤ x ≤ b-a} x)

   252 

   253   = 1/(b-a+1) * ((b-a+1)*a + (b-a)*(b-a+1)/2)

   254 

   255   = a + (b-a)/2

   256   = (b+a)/2

   257 

   258 

   259 .. math::

   260 

   261   var[unif(a,b)] = E[unif²(a,b)] - E²[unif(a,b)]

   262 

   263   = ∑_{a≤x≤b} x²/(b-a+1) - (b+a)²/4

   264 

   265   = 1/(b-a+1)·(∑_{0≤x≤b} x² - ∑_{0≤x≤a-1} x²) - (b+a)²/4

   266 

   267   = 1/(b-a+1)·(b+3·b²+2·b³ - (a-1)+3·(a-1)²+2·(a-1)³)/6 - (b+a)²/4

   268 

   269   = (2·b² + 2·a·b + b + 2·a² - a)/6 - (b+a)²/4

   270 

   271   = (b - a)·(b - a + 2) / 12

   272 

   273 .. NOTE::

   274 

   275    From Maxima::

   276 

   277      sum(i^2,i,0,n), simpsum=true;

   278 

   279               2      3

   280        n + 3 n  + 2 n

   281        ---------------

   282              6

   283 

   284      factor(b+3*b^2+2*b^3 - (a-1)-3*(a-1)^2-2*(a-1)^3);

   285 

   286                        2                  2

   287        (b - a + 1) (2 b  + 2 a b + b + 2 a  - a)

   288 

   289      factor((2*b^2 + 2*a*b + b + 2*a^2 - a)/6 - (b+a)^2/4), simp=true;

   290 

   291        (b - a) (2 - a + b)

   292        -------------------

   293                12

   294 

   295 Binomial random variable

   296 ------------------------

   297 

   298 :math:`Binomial random variable` is a r.v. with parameters :math:`n` (positive

   299 integer) and p from interval :math:`(0,1)` and sample space of positive integers

   300 from inclusive region :math:`[0, n]`:

   301 

   302 .. math::

   303 

   304   p_{binom(n,p)}(x) = n!/(x!*(n-x)!) p^x p^{n-x}

   305 

   306 Binomial random variable models a number of success of :math:`n` independent

   307 trails of Bernoulli experimants.

   308 

   309 .. math::

   310 

   311   E[binom(n,p)] = E[∑_{1≤x≤n} bernoulli(p)] = ∑_{1≤x≤n} E[bernoulli(p)] = n·p

   312 

   313   var[binom(n,p)] = var[∑_{1≤x≤n} bernoulli(p)] = ∑_{1≤x≤n} var[bernoulli(p)] = n·p·(1-p)

   314 

   315 Geometric random variable

   316 -------------------------

   317 

   318 :def:`Geometric random variable` is a r.v. with parameter :math:`p` from

   319 half open interval :math:`(0,1]`, sample space is all positive numbers:

   320 

   321 .. math::

   322 

   323   p_{geom(p)}(x) = p (1-p)^(x-1)

   324 

   325 This random variable models number of tosses of biased coin until first success.

   326 

   327 .. math::

   328 

   329   E[geom(p)] = ∑_{x=1..∞} x·p·(1-p)^(x-1)

   330 

   331   = p·∑_{x=1..∞} x·(1-p)^(x-1)

   332 

   333   = p/(1-p)·∑_{x=0..∞} x·(1-p)^x

   334 

   335   = p/(1-p)·(1-p)/(1-p - 1)² = p/p² = 1/p

   336 

   337 .. NOTE::

   338 

   339    Maxima calculation::

   340 

   341      load("simplify_sum");

   342      simplify_sum(sum(k * x^k, k, 0, inf));

   343        Is abs(x) - 1 positive, negative or zero?

   344        negative;

   345        Is x positive, negative or zero?

   346        positive;

   347        Is x - 1 positive, negative or zero?

   348        negative;

   349             x

   350        ------------

   351         2

   352        x  - 2 x + 1