But in the case of data ordered in time, say intraday stock prices, the points in the sample are often neither independent nor identically distributed. So if we let the collection of random variables $\{X_t\}_{t=0}^T$ denote the data, a complete description is given by the joint distribution $F_{X_0, \ldots, X_T}$. And that is impossible to estimate given we have a sample with a single point ($N = 1$).

To make the problem well-posed, we must impose some regularity conditions. For this we need the following definitions.

Let $\{X_t\}$ be a collection of time-indexed, real-valued random variables with the joint distribution

Def (Mean function): The mean function is defined as

for all $t$, if the expectations exist.

Def (Autocovariance function): The autocovariance function is defined as

for all $s$ and $t$.

Def (Autocorrelation function): The autocorrelation function is defined as

for all $s$ and $t$.

Now we can talk about the regularity conditions.

A series is strongly/strictly stationary if the joint distribution of any subcollection is the same as that of the shifted subcollection. Meaning the joint distribution depends only on the time differences rather than the actual times.

Def (Strong/Strict stationarity): A time series $\{X_t\}_{t=0}^T$ is said to be strongly/strictly stationary iff

for all $0 \leq t_1 \ldots \leq t_k \leq T$ and $h \geq 0$.

Note that this means the $X_i$'s are (marginally) identically distributed, but not that they're necessarily independent. For example, let ${X \sim \mathcal{N}(0, 1)}$ and let ${X_k = (-1)^kX}$ for $k \in \mathbb{N}$. Then $X_k$'s are identically distributed but not independent, and the series is strongly stationary. So iid implies strong stationarity but it's not true the other way around.

This theoretical constraint is too strong to apply to the real world. Plus,it's hard to test. So we need a weaker, more practical condition.

Def (Weak stationarity): A time series $\{X_t\}_{t=0}^T$ is weakly stationary if the mean is constant and autocovariance depends only on the lag. So

for all $t$, and

for all $s, t$.

Instead of constraining the marginal distributions, we only control their first and second moments.

Note that strong stationarity implies weak stationarity. Also, when we talk about stationarity, it's weak stationarity in particular. Now let's consider a few classic examples.

Eg. (White Noise): Let $W_t$ be independent random variables with mean $0$ and variance $\sigma^2$. Then $\mu(t) = 0$ for all $t$ and

Clearly this series is (weakly) stationary. Instead, if we suppose that $W_t$ are i.i.d. standard normal, then

and the series becomes strongly stationary too.

Eg. (Moving Average) Let $W_t$ be mean $0$ variance $\sigma^2$ white noise variates and let $X_t = \frac{W_{t-1} + W_t}{2}$. Then

for all $t$, and

for all $s$, $t$. Since $\gamma$ depends only on $|s - t|$, this series is also stationary.

Eg. (Random Walk) Let $W_t$ be mean $0$ variance $\sigma^2$ white noise variates and let $X_t = X_{t-1} + W_t$ with $X_0 = 0$, which gives $X_t = \sum_{i=1}^t W_i$. Then the mean function is

for all $t$, and

for all $s$, $t$. Since $\gamma$ depends on both $s$ and $t$ and not just their difference, this series is not weakly stationary. In fact in some sense this is a quintessential example of a non-stationary process since it's used to test data for stationarity — c.f. Augmented Dickey-Fuller test.

References

[1] Shumway, R. H., & Stoffer, D. S. (2017). Time Series Analysis and Its Applications: With R Examples (4th ed.). Springer, Cham, Switzerland.

Basic Idea

Let's consider the usual setup of a supervised learning problem with $n$ data points and $p$ predictors.

It's intuitive that points close to one another in the predictor space should have similar outputs. This is the idea behind $k$-NN (k-Nearest Neighbours). Take a distance metric, pick a $k$ (how many nearest neighbours to look at), and decide how to predict (for example, by averaging or a majority vote).

Another reasonable claim is that there's a partition of the predictor space into regions such that points in a region have similar values of the response. This is similar to $k$-NN in that it's a local method, except that closeness isn't measured by distance from points but instead by belonging in a region. Tree-based methods take this approach.

More concretely, let's consider an out-of-sample point $\mathbf{x}$. The idea is (1) to partition the predictor space into regions $R_1, \ldots, R_m$, (2) check which region $\mathbf{x}$ belongs to, and then (3) guess the value based on this region. Steps (1) and (3) make use of the training data.

These regions can be arbitrary making the problem ill-posed. In fact, even if we only consider the partition into hyperrectangles, they can be staggered and hard to define. So we simplify the problem further by allowing only recursive binary partitions. I.e. any region can only be split further into two rectangular regions — the hyperrectangles are always neatly stacked.

Now, more choices need to be made based on the problem, so we consider regression and classification separately.

Regression

We want to answer: how to make the regions and how to predict values within them. In supervised learning this reduces to choosing a cost function and finding the parameters (the regions and guess values in this case) that minimize it on the observed data.

Suppose we already have the partition into regions, and for a region $R_j$ we decide to predict the constant value $c_j$ to minimize the mean squared error (MSE). Then our estimate of the underlying function is

and the MSE is

Note that the MSE can be split into sums over the regions:

and hence, the $c_j$s that minimize the MSE are easily obtained by the mean of responses in the regions:

It turns out that finding the best possible partition that gives the lowest MSE is also computationally infeasible. So we take a greedy, iterative approach. At each step, we look at all possible splits for all possible predictors in a region, and choose the predictor and split that gives the largest reduction in MSE. The two new regions are added to the queue of all regions and the process can continue (in principle) till there's only one data point left in each region.

Pros and cons

Although this algorithm is easy to implement and gives interpretable decision trees, they have a high variance and generalize poorly. I.e. the fully-grown trees overfit the training data and have low predictive power.

At the level of individual trees this is tackled by specifying growth constraints, or by cost complexity pruning. But there are ways to combine multiple weak trees to get a strong estimator that are much more effective — namely bagging and boosting.

References

[1] Hastie, Trevor, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed. Springer Series in Statistics. New York: Springer, 2009. https://doi.org/10.1007/978-0-387-84858-7.

[2] James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani. An Introduction to Statistical Learning: with Applications in Python. 2nd ed. Springer Texts in Statistics. Cham: Springer, 2023. https://doi.org/10.1007/978-3-031-38747-0.

How can we then compare different bonds? The answer is: using Mathematics.

Keeping only the most necessary and useful properties, and ignoring the rest: A bond is simply a promise to pay back certain amounts at certain times, and it can be traded in the market for a price.

Now, consider a bond: let $F$ be its face value, $C$ be the coupon payment, $T$ be the time periods to maturity and $P$ be the market price. The yield to maturity $r$ is defined as the interest rate that solves the equation:

It is the discount rate which equates the present value of the bond's cash flows to its market price. This assumes that (1) the bond is held to maturity, and (2) the issuer makes the coupon and principal payments as promised.

To think of $r$ as the return on the bond, it is also assumed that (3) the coupons are (or can be) reinvested at the yield:

The left hand side is the future value of an investment of $P$ at the yield $r$ for $T$ periods, and the right hand side is the future value of the cash flows from the bond with reinvestment of coupons at the yield.

At the cost of a few assumptions, a single meaningful number $r$ can be used to compare very different bonds.

What's left now is actually solving for $r$. Not long after staring at the equations, you might realize the good news and the bad news. There are no weird $\sin$, $\cos$ or $\log$ terms, it's only a polynomial equation. But, unlike the quadratics from high school, it has no general closed-form solution (formula).

So, we turn to Numerical Analysis. The idea is to keep trying values of $r$ such that: (1) every attempt is better than the previous one, (2) ultimately the LHS and the RHS are very close (if not equal), and (3) all this happens reasonably fast.

Two such methods are the Bisection Method and the Newton's Method.

But before diving into them, note that solving for $r$ in equation (1) is the same as solving for $r$ in

Also, this $f(r)$ is decreasing in $r$. It looks like:

Bisection Method

In spirit, it's just like binary search. Start with $r_1$ and $r_2$ such that $f(r_1) > 0$ and $f(r_2) < 0$. Then the interval $[r_1, r_2]$ definitely contains the root. Guessing initial values could be challenging, but in this case, $r_1 = 0$ and $r_2 = 1$ are easy choices.

1. Now, check the sign of $f\left(\frac{r_1 + r_2}{2}\right)$.
2. If it's positive, set $r_1:=\frac{r_1 + r_2}{2}$; if it's negative, set $r_2:=\frac{r_1 + r_2}{2}$.
3. Repeat steps 1 and 2 till $\left|f\left(\frac{r_1 + r_2}{2}\right)\right| < \epsilon$.

Here, $\epsilon$ is an error tolerance parameter decided by the user.

Newton's Method

Since $f''(r) > 0$, $f(r)$ is a convex function. The Newton's method exploits this artfully. Start with an arbitrary value of $r=r_0$, then

1. Get the equation of the tangent to $f$ at $r_n$: ${y = f(r_n) + f'(r_n)(r - r_n)}$.
2. Get the r-intercept of this line; set $y := 0$, then ${r_{n+1} = r_n - \frac{f(r_n)}{f'(r_n)}}$.
3. Repeat steps 1 and 2 with $r = r_{n+1}$ till $\left|f(r_n)\right| < \epsilon$.

Again, $\epsilon$ is an error tolerance parameter.

Newton's method is much faster (quadratic convergence) than bisection and hence used in practice:

You can play around with the implementations using numanpy, the code for which is here. You can also look at the bond yield computation in this Jupyter Notebook.