Notation

This document provides the formal notation used in this site. The main goal is to elegantly differentiate between notational clashes across different math topics. For example, $X$ in linear algebra is a matrix, while $X$ in probability theory is a random variable.

We mostly follow the notations in the Deep Learning Book, which is based on this LaTex file. Differences between our notation and theirs will be highlighted in $\text{blue}$ for clarity.

This notation elegantly differentiates between scalars $x$ and scalar random variable $\mathbf{x}$, where the original random variable notation $X$ is only used with subscripts $X_{i,j}$ to denote an element of matrix $\mathit{X}$.

Numbers and Arrays

Notation	Description
${\displaystyle a}$	A scalar (integer or real)
${\displaystyle \mathit{a}}$	A vector
${\displaystyle \mathit{A}}$	A matrix
${\displaystyle \mathsf{A}}$	A tensor
${\displaystyle {\mathit{I}}_n}$	Identity matrix with $n$ rows and $n$ columns
${\displaystyle \mathit{I}}$	Identity matrix with dimensionality implied by context
${\displaystyle {\mathit{e}}^{(i)}}$	Standard basis vector $[0,\dots ,0,1,0,\dots ,0]$ with a 1 at position $i$
${\displaystyle \text{diag}(\mathit{a})}$	A square, diagonal matrix with diagonal entries given by $\mathit{a}$
${\displaystyle \text{a}}$	A scalar random variable
${\displaystyle \mathbf{a}}$	A vector-valued random variable
${\displaystyle \mathbf{A}}$	A matrix-valued random variable

Sets and Graphs

Notation	Description
${\displaystyle \mathbb{A}}$	A set
${\displaystyle \mathbb{R}}$	The set of real numbers
${\displaystyle \{0,1\}}$	The set containing 0 and 1
${\displaystyle \{0,1,\dots ,n\}}$	The set of all integers between $0$ and $n$
${\displaystyle [a,b]}$	The real interval including $a$ and $b$
${\displaystyle (a,b]}$	The real interval excluding $a$ but including $b$
${\displaystyle \mathbb{A}\mathrm{\setminus }\mathbb{B}}$	Set subtraction, i.e., the set containing the elements of $\mathbb{A}$ that are not in $\mathbb{B}$
${\displaystyle \mathcal{G}}$	A graph
${\displaystyle P{a}_{\mathcal{G}}({\text{x}}_i)}$	The parents of ${\text{x}}_i$ in $\mathcal{G}$

Indexing

Notation	Description
${\displaystyle a_i}$	Element $i$ of vector $\mathit{a}$, with indexing starting at 1
${\displaystyle a_{-i}}$	All elements of vector $\mathit{a}$ except for element $i$
${\displaystyle A_{i,j}}$	Element $i,j$ of matrix $\mathit{A}$
${\displaystyle {\mathit{A}}_{i,:}}$	Row $i$ of matrix $\mathit{A}$
${\displaystyle {\mathit{A}}_{:,i}}$	Column $i$ of matrix $\mathit{A}$
${\displaystyle {\mathsf{A}}_{i,j,k}}$	Element $(i,j,k)$ of a 3-D tensor $\mathsf{A}$
${\displaystyle {\mathsf{A}}_{:,:,i}}$	2-D slice of a 3-D tensor
${\displaystyle {\text{a}}_i}$	Element $i$ of the random vector $\mathbf{a}$

Linear Algebra Operations

Notation	Description
${\displaystyle {\mathit{A}}^{\mathrm{\top }}}$	Transpose of matrix $\mathit{A}$
${\displaystyle {\mathit{A}}^{+}}$	Moore-Penrose pseudoinverse of $\mathit{A}$
${\displaystyle \mathit{A}\odot \mathit{B}}$	Element-wise (Hadamard) product of $\mathit{A}$ and $\mathit{B}$
${\displaystyle \det (\mathit{A})}$	Determinant of $\mathit{A}$

Calculus

Notation	Description
${\displaystyle \frac{dy}{dx}}$	Derivative of $y$ with respect to $x$
${\displaystyle \frac{\partial y}{\partial x}}$	Partial derivative of $y$ with respect to $x$
${\displaystyle {\mathrm{\nabla }}_{\mathit{x}}y}$	Gradient of $y$ with respect to $\mathit{x}$
${\displaystyle {\mathrm{\nabla }}_{\mathit{X}}y}$	Matrix derivatives of $y$ with respect to $\mathit{X}$
${\displaystyle {\mathrm{\nabla }}_{\mathsf{X}}y}$	Tensor containing derivatives of $y$ with respect to $\mathsf{X}$
${\displaystyle \frac{\partial f}{\partial \mathit{x}}\text{ or }\mathit{J}(f)(\mathit{x})}$	Jacobian matrix $\mathit{J}\in {\mathbb{R}}^{m\times n}$ of $f:{\mathbb{R}}^n\to {\mathbb{R}}^m$
${\displaystyle {\mathrm{\nabla }}_{\mathit{x}}^{2}f(\mathit{x})\text{ or }\mathit{H}(f)(\mathit{x})}$	The Hessian matrix $\mathit{H}\in {\mathbb{R}}^{n\times n}$ of $f:{\mathbb{R}}^n\to \mathbb{R}$ at input point $\mathit{x}$
${\displaystyle \int f(\mathit{x})d\mathit{x}}$	Definite integral over the entire domain of $\mathit{x}$; clash of notation with indefinite integrals, will strive to avoid this notation
${\displaystyle {\int }_{\mathbb{S}}f(\mathit{x})d\mathit{x}}$	Definite integral with respect to $\mathit{x}$ over the set $\mathbb{S}$

Probability and Information Theory

Notation	Description
${\displaystyle \text{a}\mathrm{\perp }\text{b}}$	The random variables $\text{a}$ and $\text{b}$ are independent
${\displaystyle \text{a}\mathrm{\perp }\text{b}\mid \text{c}}$	They are conditionally independent given $\text{c}$
${\displaystyle P(\text{a})}$	A probability distribution over a discrete variable
${\displaystyle p(\text{a})}$	A probability distribution over a continuous variable, or over a variable whose type has not been specified
${\displaystyle \text{a}\sim P}$	Random variable $\text{a}$ has distribution $P$
${\displaystyle {\mathbb{E}}_{\text{x}\sim P}[f(x)]\text{ or }\mathbb{E}f(x)}$	Expectation of $f(x)$ with respect to $P(\text{x})$; abuse of notation, will strive to use ${\mathbb{E}}_{\text{x}\sim P}[f(\text{x})]$ instead
${\displaystyle \mathrm{Var}(f(x))}$	Variance of $f(x)$ under $P(\text{x})$; abuse of notation, will strive to use $\mathrm{Var}(f(\text{x}))$ instead
${\displaystyle \mathrm{Cov}(f(x),g(x))}$	Covariance of $f(x)$ and $g(x)$ under $P(\text{x})$; abuse of notation, will strive to use $\mathrm{Cov}(f(\text{x}),g(\text{y}))$ instead
${\displaystyle H(\text{x})}$	Shannon entropy of the random variable $\text{x}$
${\displaystyle D_{\mathrm{KL}}(P\Vert Q)}$	Kullback-Leibler divergence of P and Q
${\displaystyle \mathcal{N}(\mathit{x};\mathit{\mu },\mathbf{\Sigma })}$	Gaussian distribution over $\mathit{x}$ with mean $\mathit{\mu }$ and covariance $\mathbf{\Sigma }$

Functions

Notation	Description
${\displaystyle f:\mathbb{A}\to \mathbb{B}}$	The function $f$ with domain $\mathbb{A}$ and range $\mathbb{B}$
${\displaystyle f\circ g}$	Composition of the functions $f$ and $g$
${\displaystyle f(\mathit{x};\mathit{\theta })}$	A function of $\mathit{x}$ parametrized by $\mathit{\theta }$. (Sometimes we write $f(\mathit{x})$ and omit the argument $\mathit{\theta }$ to lighten notation)
${\displaystyle \log x}$	Natural logarithm of $x$
${\displaystyle \sigma (x)}$	Logistic sigmoid, ${\displaystyle \frac{1}{1+\exp (-x)}}$
${\displaystyle \zeta (x)}$	Softplus, $\log (1+\exp (x))$
${\displaystyle ||\mathit{x}|{|}_p}$	$L^p$ norm of $\mathit{x}$
${\displaystyle ||\mathit{x}||}$	$L^2$ norm of $\mathit{x}$
${\displaystyle x^{+}}$	Positive part of $x$, i.e., $max(0,x)$
${\displaystyle {\mathbf{1}}_{\mathrm{condition}}\text{ or }[\mathrm{condition}]}$	is 1 if the condition is true, 0 otherwise

Sometimes we use a function $f$ whose argument is a scalar but apply it to a vector, matrix, or tensor: $f(\mathit{x})$, $f(\mathit{X})$, or $f(\mathsf{X})$. This denotes the application of $f$ to the array element-wise. For example, if $\mathsf{C}=\sigma (\mathsf{X})$, then ${\mathsf{C}}_{i,j,k}=\sigma ({\mathsf{X}}_{i,j,k})$ for all valid values of $i$, $j$ and $k$.

Datasets and Distributions

Notation	Description
${\displaystyle p_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}}$	The data generating distribution
${\displaystyle {\hat{p}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}}$	The empirical distribution defined by the training set
${\displaystyle \mathbb{X}}$	A set of training examples
${\displaystyle {\mathit{x}}^{(i)}}$	The $i$-th example (input) from a dataset
${\displaystyle y^{(i)}\text{ or }{\mathit{y}}^{(i)}}$	The target associated with ${\mathit{x}}^{(i)}$ for supervised learning
${\displaystyle \mathit{X}}$	The $m\times n$ matrix with input example ${\mathit{x}}^{(i)}$ in row ${\mathit{X}}_{i,:}$

Comments