1-d calculus

The crucial concepts of calculus are derivative and integral.

Derivative

The linear function

\[ f(x) = ax + b \]

is nice and simple, and its graph is a straight line. Using differentiation and differentials one can in some way reduce any smooth function to a linear one!

In a similar manner any smooth function is linear in a small neighborhood of the tangent point:

\[ f(x + h) \approx f(x) + f'(x)h \text{ for small } h. \]

A more scientific definition:

(60)\[ f(x + h)= f(x) + f'(x)h + \mathcal O(h^2), \quad h\to 0.\]

Strict definition

The derivative of \(f \colon (x-\delta, x+\delta) \to \mathbb R\), \(\delta > 0\), at point \(x\) is

\[ f'(x) = \lim\limits_{h\to 0} \frac{f(x+h)-f(x)}h. \]

Existence of derivative \(f'(x)\) is equivalent to differentiability of \(f\) at point \(x\):

\[ f(x + h) - f(x) = f'(x)h + o(h), \quad h\to 0. \]

Note that (60) imply this equality but the converse is false.

Differential

The function \(df(x ,h) = f'(x)h\) is called differential of \(f\) at point \(x\). Note that it is a function of two variables \(x\) and \(h\), and the dependency on \(h\) is linear.

Important

Due to historical reasons, the increment \(h\) is often denoted as \(dx\); then the formula for the differential is

\[ df = f'(x)dx. \]

Differential is the main linear part of the increment \(\Delta f = f(x + h) - f(x)\).

Rules of differentiation

Derivative:

1. \(f'(x) \equiv 0\) if \(f(x)\equiv \mathrm{const}\)

2. \((\alpha f(x) + \beta g(x))' = \alpha f'(x) + \beta g'(x)\)

3. \((f(x)g(x))' = f'(x) g(x) + f(x) g'(x)\)

4. \(\big(\frac{f(x)}{g(x)}\big)' = \frac{f'(x) g(x) - f(x) g'(x)}{g^2(x)}\) if \(g(x) \ne 0\)

5. \(((f \circ g)(x))' = f'(g(x)) g'(x)\) (chain rule)

Differential:

1. \(d(\alpha f + \beta g) = \alpha df + \beta dg\)

2. \(d(fg) = fdg + g df\)

3. \(d\big(\frac fg\big) = \frac{gdf - f dg}{g^2}\)

4. \(d(f \circ g)(x) = df(g(x)) = f'(g(x)) \cdot dg(x)\) (chain rule)

Derivatives of higher orders

If the function \(f'(x)\) is also differentiable, then its derivative is called the second derivative of \(f\): \(f''(x) =\frac d{dx}(f'(x))\). By indtuction, \(n\)-th derivative is defined as

\[ f^{(n)}(x) = \frac d{dx}\big(f^{(n-1)}(x)\big). \]

To find the second differential of \(f\) just differentiate \(df(x) = f'(x) dx\) with respect to \(x\) assuming that \(dx\) is a constant:

\[ d^2 f(x) = f''(x) dx^2, \quad d^n(x) = d(d^{n-1}f)(x) = f^{(n)}(x) dx^n. \]

A function \(f \colon [a, b] \to \mathbb R\) is called continuously differentiable \(n\) times (denoted as \(f \in C^n[a, b]\)) if its \(n\)-th derivative is continuous: \(f^{(n)} \in C[a, b]\). If \(f \in C^{n+1}[a, b]\) then the Taylor formula holds:

\[ f(x) = f(a) + f'(a)(x-a) + \frac 12 f''(a)(x-a)^2 + \ldots + \frac {f^{n}}{n!}(x-a)^n + r_n, \]

where remainder term \(r_n\) is

\[ r_n = \frac {f^{n+1}(c)}{(n+1)!}(c-a)^{n+1}, \quad a < c < x. \]

If \(\lim\limits_{n\to\infty} r_n = 0\), the function \(f\) can be expanded into Taylor series:

\[ f(x) = \sum\limits_{k=0}^\infty \frac{f^{(k)}(x)}{k!}(x-a)^k. \]

If \(a=0\) the Taylor series if called Maclaurin series.

Example

If \(f(x) = e^x\) then \(f^{(k)}(x) = e^x\) for all \(k\in\mathbb N\). Also,

\[ r_n = \frac{e^c}{(n+1)!}c^n \to 0, \quad n\to\infty. \]

Hence,

\[ e^x = \sum\limits_{k=0}^\infty \frac{x^k}{k!}, \quad x \in \mathbb R. \]

Applications of derivatives

1. If \(f'(x) > 0\) (\(f'(x) < 0\)) for all \(x\in (a, b)\), then \(f\) is increasing (decreasing) on \((a, b)\).

2. If \(f'(x) = 0\) and \(f''(x) > 0\) (\(f''(x) < 0\)), then \(x\) is a local minimum (maximum) of \(f\).

3. If \(f''(x) > 0\) (\(f''(x) < 0\)) for all \(x\in (a, b)\), then \(f\) is strictly convex (concave) on \((a, b)\)

Exercises

1. Find derivative of \(f(x) = \tanh x = \frac{\sinh x}{\cosh x} = \frac {e^x - e^{-x}}{e^x + e^{-x}}\).

2. Show that \(\sigma'(x) = \sigma(x) (1 - \sigma(x))\) where

   \[ \sigma(x) = \frac 1{1 + e^{-x}} \]

   — sigmoid function.

3. Find \(\max\limits_{x\in\mathbb R}\sigma'(x)\).

4. Give an example of a function \(f\) which is differentiable at point \(x\) but (60) does not hold.

5. Find the first and the second differential of \(f(x) = \sin x\) at point \(x = \frac \pi 3\).

6. Find Maclaurin series of \(f(x) = \frac 1{1 - x}\) and \(g(x) = \frac 1{(1-x)^2}\).

7. Find global maximium of

   \[ f(x) = \prod\limits_{i=1}^n \exp\Big(-\frac{(x - a_i)^2}{2 \sigma_i^2}\Big), \quad \sigma_i > 0. \]

   What if \(\sigma_1 = \ldots = \sigma_n = \sigma > 0\)? Does this function has global minimum?