Derivative#

The linear function

\[ f(x) = ax + b \]

is nice and simple, and its graph is a straight line. Using differentiation and differentials one can in some way reduce any smooth function to a linear one!

In a similar manner any smooth function is linear in a small neighborhood of the tangent point:

\[ f(x + h) \approx f(x) + f'(x)h \text{ for small } h. \]

A more scientific definition:

(60)#\[ f(x + h)= f(x) + f'(x)h + O(h^2), \quad h\to 0.\]

Strict definition

The derivative of \(f \colon (x-\delta, x+\delta) \to \mathbb R\), \(\delta > 0\), at point \(x\) is

\[ f'(x) = \lim\limits_{h\to 0} \frac{f(x+h)-f(x)}h. \]

Existence of derivative \(f'(x)\) is equivalent to differentiability of \(f\) at point \(x\):

\[ f(x + h) - f(x) = f'(x)h + o(h), \quad h\to 0. \]

Note that (60) imply this equality but the converse is false.

Differential#

The function \(df(x ,h) = f'(x)h\) is called differential of \(f\) at point \(x\). Note that it is a function of two variables \(x\) and \(h\), and the dependency on \(h\) is linear.

Important

Due to historical reasons, the increment \(h\) is often denoted as \(dx\); then the formula for the differential is

\[ df = f'(x)dx. \]

Differential is the main linear part of the increment \(\Delta f = f(x + h) - f(x)\).

Rules of differentiation#

Derivative:

  1. \(f'(x) \equiv 0\) if \(f(x)\equiv \mathrm{const}\)

  2. \((\alpha f(x) + \beta g(x))' = \alpha f'(x) + \beta g'(x)\)

  3. \((f(x)g(x))' = f'(x) g(x) + f(x) g'(x)\)

  4. \(\big(\frac{f(x)}{g(x)}\big)' = \frac{f'(x) g(x) - f(x) g'(x)}{g^2(x)}\) if \(g(x) \ne 0\)

  5. \(((f \circ g)(x))' = f'(g(x)) g'(x)\) (chain rule)

Differential:

  1. \(d(\alpha f + \beta g) = \alpha df + \beta dg\)

  2. \(d(fg) = fdg + g df\)

  3. \(d\big(\frac fg\big) = \frac{gdf - f dg}{g^2}\)

  4. \(d(f \circ g)(x) = df(g(x)) = f'(g(x)) \cdot dg(x)\) (chain rule)

Derivatives of higher orders#

If the function \(f'(x)\) is also differentiable, then its derivative is called the second derivative of \(f\): \(f''(x) =\frac d{dx}(f'(x))\). By indtuction, \(n\)-th derivative is defined as

\[ f^{(n)}(x) = \frac d{dx}\big(f^{(n-1)}(x)\big). \]

To find the second differential of \(f\) just differentiate \(df(x) = f'(x) dx\) with respect to \(x\) assuming that \(dx\) is a constant:

\[ d^2 f(x) = f''(x) dx^2, \quad d^n(x) = d(d^{n-1}f)(x) = f^{(n)}(x) dx^n. \]

Applications of derivatives#

  1. If \(f'(x) > 0\) (\(f'(x) < 0\)) for all \(x\in (a, b)\), then \(f\) is increasing (decreasing) on \((a, b)\).

  2. If \(f'(x) = 0\) and \(f''(x) > 0\) (\(f''(x) < 0\)), then \(x\) is a local minimum (maximum) of \(f\).

  3. If \(f''(x) > 0\) (\(f''(x) < 0\)) for all \(x\in (a, b)\), then \(f\) is strictly convex (concave) on \((a, b)\)

https://i.stack.imgur.com/GNBZ4.png

Exercises#

  1. Show that \(\sigma'(x) = \sigma(x) (1 - \sigma(x))\) where

    \[ \sigma(x) = \frac 1{1 + e^{-x}} \]

    — sigmoid function.

  2. Find \(\max\limits_{x\in\mathbb R}\sigma'(x)\).

  3. Give an example of a function \(f\) which is differentiable at point \(x\) but (60) does not hold.

  4. Find the first and the second differential of \(f(x) = \sin x\) at point \(x = \frac \pi 3\).

  5. Find global maximium of

    \[ f(x) = \prod\limits_{i=1}^n \exp\Big(-\frac{(x - a_i)^2}{2 \sigma_i^2}\Big), \quad \sigma_i > 0. \]

    What if \(\sigma_1 = \ldots = \sigma_n = \sigma > 0\)? Does this function has global minimum?