This section recalls numerical induction and introduces structural induction. These techniques will be crucial to design and analysis of algorithms throughout the course, particularly with Divide and Conquer, Graph Traversal, and Dynamic Programming.
Objectives. After learning this material, you should be able to:
In past courses, you have seen numerical induction. This is a method to prove a statement of the form "for all $n$, $P(n)$ is true." Here is an example.
Recall that the Fibonacci numbers are defined as $F_0 = 0$, $F_1 = 1$, and $F_n = F_{n-1} + F_{n-2}$ for $n \geq 2$. The first few are 0,1,1,2,3,5,8,13,...
Prove by induction that $F_n \leq 2^n$.
The first base case is $n=0$. In this case, $F_0 = 0 \leq 1 = 2^0$. The second base case is $n=1$. In this case, $F_1 = 1 \leq 2 = 2^1$.
For the inductive step, let $n \geq 2$. The inductive hypothesis (IH) is that for all $k < n$, we have $F_k \leq 2^k$. Using the IH:
We have proven that $F_n \leq 2^n$. This completes the proof by induction.
Note: strong induction. The previous example is actually called "strong" induction. This is because, in order to prove the statement for $n$, we needed to assume it was true for all $k < n$. In contrast, weak induction only uses the inductive hypothesis that the statement is true for $k=n-1$. However, the distinction between strong and weak induction will not be very important for this class; they are both proofs by induction.
Checking your familiarity. In order to tackle the next portion of the class, you should be comfortable with the following type of examples. Can you solve it?
Prove by induction that the sum of the first $n$ odd numbers is equal to $n^2$.
You probably also have used induction to prove facts that are not just numerical equalities or inequalities. Here is an example. Recall that a prime factorization of an integer is a list of prime numbers whose product equals the integer.
Prove by induction that every integer $n \geq 2$ has a prime factorization.
The base case is $n=2$. This has the prime factorization $2$.
For the inductive step, assume that every $k < n$ has a prime factorization. We must prove this holds for $n$ as well. There are two cases: $n$ is prime, or $n$ is composite. If $n$ is prime, then it has the prime factorization $n$, so we are done. If $n$ is compsite, then $n = ab$ for $a,b \in \{2,\dots,n-1\}$. By inductive hypothesis, $a$ and $b$ each have a prime factorization. Multiplying them together gives a prime factorization of $n$.
We will often deal with "structures" such as strings, graphs, and trees. Many of our algorithms' correctness and runtime guarantees will be proven by a type of induction. We will begin with an example.
Given a finite set $X$, called the alphabet, whose elements are called characters, a string is either:
For example, we can take the alphabet to the be 26 lowercase English letters, in which case a string would be all lists consisting of these letters, including the empty string.
The length $len(s)$ of a string $s$ is defined inductively: if $s$ is empty, its length is zero. If $s$ consists of $c$ followed by a string $s'$, its length is $1 + len(s')$.
Let $m = |X|$, the size of the alphabet. Then there are $m^t$ different strings of length $t$.
The base case is when $t=0$, i.e. the string has length zero. In this case, it is the empty string, and there is only one empty string, so the number of strings is $1 = m^0$, so the formula is correct.
For the inductive case, fix $t \geq 1$ and suppose the claim is true for all $k < t$. A string of length $t$ is of the form $cs'$, where $c$ is a character and $s'$ is a string of length $t-1$. By inductive hypothesis, there are $m^{t-1}$ possible strings $s'$. For each $s'$, we get a different string of length $t$ by prepending each of the $m$ possible characters. So there are $m \cdot m^{t-1} = m^t$ possible strings of length $t$.
Here is another example of structural induction.
A binary tree is either:
The root of a binary tree is the node that is not a child of any other node. The height of a binary tree is the length of the longest path from the root to any leaf.
A binary tree of height $h$ has at most $2^{h+1} - 1$ nodes.
The base case is a leaf node, i.e. a tree of height $0$. The number of nodes is $1 = 2^{0+1} - 1$, as needed.
Now let $h \geq 1$ and suppose the claim holds for all $k < h$. Consider a binary tree of height $h$. The root has at most two children. Each child is the root of a binary tree of height at most $h-1$. So by inductive hypothesis, there are most $2^{h} - 1$ nodes in each subtree. Adding them together gives an upper bound of $2(2^{h} - 1) = 2^{h+1} - 2$ nodes. Adding the root node gives an upper bound of $2^{h+1} - 1$ nodes, proving the claim.
This section introduces a powerful algorithmic paradigm, divide-and-conquer. This paradigm is closely related to recursion, and we introduce recursive algorithms as well.
Objectives. After learning this material, you should be able to:
You have probably seen recursive algorithms before, so we will recall them briefly. An algorithm is recursive if, in order to compute its output, it calls itself on a modified input. An example of a recursive algorithm is this naive implementation of the Fibonacci numbers:
// Algorithm 1: Fibonacci numbers
1 fib(n):
2 if n == 0:
3 return 0
4 elseif n == 1:
5 return 1
6 else:
7 return fib(n-1) + fib(n-2)
We will be looking at recursive algorithms that fall into the following algorithmic paradigm.
An algorithm that splits the input into parts, solves a useful subproblem on each part, then combines the solutions together to produce an answer to the original problem.
Often, divide-and-conquer algorithms are recursive in nature because the subproblems are of the same form as the original problem. We will focus on such examples here.
For our first example, we'll look at a very important algorithm: binary search. In this problem, we are given a sorted list of elements coming from some universe. Given an element, we need to find its location in the list, if present. The elements could be numbers, strings, or other objects. For simplicity, we will assume they are numbers, but the algorithm easily generalizes to other cases.
The Search Problem:
The binary search algorithm is as follows. We will 0-index A for this algorithm. We use A[i:j] to denote the subarray of A from element i to j inclusive. If i > j, such as A[0:-1] or A[5:4], this produces an empty array. We write A[i:end] to denote the subarray starting at index i and going to the end of the array. We write len(A) for the length of A. We assume that a subarray can be created, or referenced, in constant time.
// Algorithm 2: Binary Search
1 binary_search(A, x):
2 if len(A) == 0:
3 return 0
4 let i = int(len(A)/2) // int() rounds down
5 if x == A[i]:
6 return i
7 elseif x < A[i]:
8 return binary_search(A[0:i-1], x)
9 elseif x > A[i]:
10 return i + 1 + binary_search(A[i+1:end], x)
Binary search correctly solves the binary search problem, i.e. given a valid input, it produces a correct output.
We prove correctness by induction on the length of A. The base case is when A has length 0. In this case, the algorithm returns 0, which is the correct index.
For the inductive step, let the length of A be at least 1 and assume the algorithm is correct on all arrays shorter than A. Let $i = int(len(A)/2)$, where int() rounds down. We note that because $len(A) \geq 1$, we have $i < len(A)$, which means that the array $A[0:i-1]$ and the array $A[i+1:end]$ are both shorter than A.
There are three cases. First, if $x == A[i]$, then the algorithm correctly returns index $i$, and we are done.
Second, suppose $x < A[i]$. By inductive hypothesis, our recursive call on line 8 correctly returns the index to insert $x$ in the sublist A[0:i-1]. We claim this index is also correct for inserting $x$ into A. This follows immediately if the return value is in $0,\dots,i-1$, which implies that $x$ should be inserted somewhere before the end of the sublist. The other case is if the return value is $i$, meaning an insertion at the end of the sublist, implying $x > A[i-1]$. This is still correct, because we have supposed $x < A[i]$, so $i$ is the correct position.
Third, suppose $x > A[i]$. By inductive hypothesis, our recursive call on line 10 correctly returns the index to insert $x$ in the sublist A[i+1:end]. If this index is some $j > 0$, then x should be inserted somewhere after the beginning of this sublist, which means the same location, shifted by $i+1$, is correct to insert x into A. If the index is $0$, this implies that $x \leq A[i+1]$. On the other hand, we have $x > A[i]$, so $i+1$ is correct.
Note. The exact code of binary search and its correctness proof are notoriously fiddly due to several edge cases. As Donald Knuth famously wrote, "Beware this code. I have not tested it, only proven it correct."
Binary search runs in time $O(\log(n))$, where $n = len(A)$.
First, we prove by induction that, when we call binary_search(A,x), it makes at most $\log(n) + 1$ recursive calls total, where the logarithm is base 2. The first base case is when n == 0. In this case, we make $0$ recursive calls. Since $\log(0)$ is not defined, we will define this case as being satisfied. The other base case is when n == 1. In this case, we either make no recursive calls, because $x == A[0]$, or we make one recursive call on a length-0 subarray. The maximum number of recursive calls is therefore $1 = \log(1) + 1$, as claimed.
For the inductive step, let $n \geq 2$. In this case, binary_search(A,x) either makes no recursive calls, or makes a call in line 8, or makes a call in line 10 (but not both). In either case, the call is for a subarray of length at most n/2. By inductive hypothesis, after that call, at most $\log(n/2) + 1$ more calls are made. This makes the total number at most $1 + \log(n/2) + 1 = 1 + \log(n) - \log(2) + 1 = 1 + \log(n)$, as claimed.
Now we look at each call to binary_search(). By counting the operations, we can see that any call does at most a constant number of steps, about 10, plus any steps done by its own recursive call. Since we proved that we visit the function binary_search() at most $\log(n) + 1$ times total, this gives a total running time bound of $10(\log(n) + 1) = 10\log(n) + 10 = O(\log(n))$.
As with binary search, Mergesort is a recursive divide-and-conquer problem. It also applies to lists of elements, and as with binary search above, we will focus on the case where the elements are numbers, but the ideas can easily extend to other types of list elements.
The Sorting Problem:
The idea of Mergesort is to split the list in half and sort the left and right halves separately. Then, we need to run the merge subroutine to combine the two sorted sublists into our final sorted list.
// Algorithm 3: Mergesort
1 mergesort(A):
2 if len(A) <= 1:
3 return A
4 let i = int(len(A)/2)
5 let B = mergesort(A[0:i])
6 let C = mergesort(A[i+1:end])
7 return merge(B, C)
// Subroutine 1: Merge two sorted arrays
1 merge(B, C):
2 let D = []
3 let i = 0
4 let j = 0
5 while i < len(B) or j < len(C):
6 if j >= len(C) or (i < len(B) and B[i] < C[j]):
7 append B[i] to D
8 i += 1
9 else:
10 append C[j] to D
11 j += 1
12 return D
First, we need a lemma that the "merge" subroutine is correct.
The subroutine merge(B,C), given two sorted lists, outputs a sorted list containing all of the elements in B and C.
All elements of B and C eventually get added to D because we only increment i when we add B[i] to D and similarly for j and C[j]. Now we just need to show that D is sorted. To do so, we claim that in the while loop of lines 5-11, each iteration, the smallest remaining element of B[i:end] and C[j:end] is added to D. This follows because B is sorted, so its minimum remaining element is located at B[i], and similarly for C[j], and we add the smaller of these to D. (If we have already added all elements of B, then $i > len(B)$, so we add C[j]; and vice versa.) Because we always append the smallest remaining element to D, D is sorted.
Mergesort correctly sorts the input.
By induction. For base cases, if len(A) is 0 or 1, mergsort(A) returns A, which is correct.
For the inductive case, let $len(A) \geq 2$. The elements of A are partitioned into B and C in lines 7 and 8. Each of these is shorter than A, so by inductive hypothesis, B and C are correctly sorted by the recursive calls. By the lemma, merge(B,C) returns a sorted version of A, so mergesort is correct.
We again assume that subarrays can be referenced in constant time.
merge(B,C) runs in time at most 10(len(B) + len(C)) + 4.
We first analyze the while loop, lines 5-11. Inside the loop, we take a constant number of steps, upper-bounded by about 10. How many iterations does the while loop take? Each iteration, we either increase i or j, but not both. We stop incrementing i when it reaches len(B) and we stop incrementing j when it reaches len(C). When both happen, the while loop completes. Therefore, there are len(B) + len(C) iterations of the while loop, so lines 5-11 take a total of at most 10(len(B) + len(C)) time. Adding 4 steps for lines 2-4 and line 12 gives a running time bound of 10(len(B) + len(C)) + 4.
mergesort(A) runs in time $O(n \log(n))$, where $n = len(A)$.
We will use a similar strategy to binary search: we calculate how much work happens in each call to mergesort(), then we calculate the number of calls. However, we will see that this situation is a bit more complex.
It will be helpful to have an expression for the running time of mergesort on a given input size. Define T(n) to be the running time of mergesort on lists of length n.
First, how much work is done in mergesort(A) itself, ignoring recursive calls? The answer is about 6 steps plus the work done in merge(B,C), which gives a total (using the Lemma) of 10 + 10(len(B) + len(C)). We can observe that len(B) + len(C) = len(A), so this runtime is 10 + 10 len(A). To simplify analysis, let's consider the case $len(A) \geq 1$, so we can upper-bound this work by 10 + 10 len(A) $\leq$ 20 len(A).
Now, we need to consider the recursive calls. To simplify the analysis, we will pretend that $len(A) = 2^k$ for some integer $k$, i.e. the length is a power of $2$. That implies that, because we divide the array in half each time, every subarray we create throughout the algorithm has even length until the length reaches $1$.
We recursively call mergesort twice, each on a list of length n/2. This gives the following recurrence relation:
Here is how it relates to the code:
// Algorithm 3: Mergesort, analysis
1 mergesort(A): // T(n) time total
2 if len(A) <= 1:
3 return A
4 let i = int(len(A)/2)
5 let B = mergesort(A[0:i]) // T(n/2)
6 let C = mergesort(A[i+1:end]) // T(n/2)
7 return merge(B, C) // O(n)
Verbally, the running time of mergesort on a list of length $n$ is equal to twice its running time on lists of length $n/2$, plus an extra $20n$ steps needed to merge the lists and complete the rest of the algorithm.
Solving the recurrence. Now the question is: what formula T(n) solves the recurrence? The method we will use is called the recursion tree method. We create a tree where each node represents a call to mergesort() and its two children represent the two recursive calls it makes. We can make the following key observations.
We are now ready to add up the total time complexity by summing over the "layers" of the recursion tree. At each layer $t=0,\dots,k$, there are $2^t$ nodes. Each one does a total amount of work 10 + 10 len(input) = $10 + 10 2^{k-t}$. Summing, the total time complexity is:
Recalling that $n = len(A)$ and $k = \log(n)$, we can rewrite this running time bound as $T(n) = 20 n \log(n) = O(n \log(n))$.
As a note, we can check that $T(n) = 20n \log(n)$ satisfies our original recurrence:
Let's consider one more concrete example before we discuss how to analyze divide-and-conquer algorithms in general.
The Integer Multiplication Problem:
The "grade-school" algorithm is to multiply each bit of $x$ by each bit of $y$ and take the results, suitably shifted, and add them all up. We won't go into to the analysis, but this takes $O(n^2)$ time, as we must consider all pairs of bits.
Can we multiply faster than that? Famous mathematicians thought no. But in fact, we can, using a divide-and-conquer algorithm. First, we'll look at an approach that is not faster, but paves the way.
// Algorithm 4: Divide-and-Conquer Multiplication
1 multiply(x, y): // assume both have n bits
2 let n = len(x)
3 if n <= 1:
4 return x*y
5 let x0,x1 = split(x)
6 let y0,y1 = split(y)
7 a = multiply(x1,y1)
8 b = multiply(x1,y0)
9 c = multiply(x0,y1)
10 d = multiply(x0,y0)
11 return a*2^n + (b+c)*2^(n/2) + d
// Subroutine for dividing an integer into upper and lower halves
1 split(x):
2 let x0 = x % 2^(n/2) // % is the remainder operation
3 let x1 = int(x/2^(n/2)) // int() rounds down
4 return x0,x1
Let's take correctness as given an analyze this algorithm using the recurrence-tree method. Let $T(n)$ be the running time of multiply(). The split() subroutine is $O(n)$ time using bit operations. Line 11 is actually $O(n)$ as well, because adding is $O(n)$ and multiplying by a power of two is just a bit shift, so also $O(n)$. That leaves lines 7-10. Each calls multiply() on inputs of half the original size, so they take $T(n/2)$ time each.
Let $n = 2^k$. The tree method gives that at each level $t=0,1,\dots,\log(n)$, there are $4^t$ nodes and each node does $n/2^t = 2^{k-t}$ work.
A more rigorous proof wouldn't hurt, but we conclude the algorithm is $O(n^2)$. No faster! However, a clever observation allows us to speed it up asymptotically. We'll see this next.
We know that addition and subtraction of $n$-bit numbers are $O(n)$ time operations. Asymptotically, these are much, much cheaper than multiplication, which is $O(n^2)$ as far as we know. Therefore, we should be willing to do a few extra additions and subtractions if they can help us save even one multiplication. That is the idea behind the next algorithm.
Remember that $x = 2^n (x1) + x0$ and $y = 2^n (y1) + y0$, where $x1,x0,y1,y0$ are $n/2$ bits. We want to compute $xy = (2^n x1 + x0)(2^n y1 + y0) = 2^{2n} x1y1 + 2^{n}(x1y0 + x0y1) + x0y0$. Let's take for granted that we'll need to compute $x1y1$ and $x0y0$. Can we get the middle term any easier?
The key observation is that we can compute the following product of $n/2$ bit numbers: $(x1 + x0)(y1 + y0) = x1y1 + x0y1 + x1y0 + x0y0$. Subtracting off $x1y1$ and $x0y0$ leaves us with the middle term we need, $x0y1 + y0x1$.
// Algorithm 5: Karatsuba Multiplication
1 multiply(x, y): // assume both have n bits
2 let n = len(x)
3 if n <= 1:
4 return x*y
5 let x0,x1 = split(x)
6 let y0,y1 = split(y)
7 a = multiply(x1,y1)
8 d = multiply(x0,y0)
9 x_sum = x1 + x0
10 y_sum = y1 + y0
11 product = multiply(x_sum, y_sum)
12 b = a + d - product
13 return a*2^n + b*2^(n/2) + d
Karatsuba multiplication runs in time $O(n^c)$ where $c = \log_2(3) \approx 1.58.$
We first need to write the recurrence. We have:
// Algorithm 5: Karatsuba Multiplication
1 multiply(x, y): // T(n) steps
2 let n = len(x)
3 if n <= 1: // O(1)
4 return x*y // O(1)
5 let x0,x1 = split(x) // O(n)
6 let y0,y1 = split(y) // O(n)
7 a = multiply(x1,y1) // T(n/2)
8 d = multiply(x0,y0) // T(n/2)
9 x_sum = x1 + x0 // O(n)
10 y_sum = y1 + y0 // O(n)
11 product = multiply(x_sum, y_sum) // T(n/2)
12 b = a + d - product // O(n)
13 return a*2^n + b*2^(n/2) + d // O(n)
We can write this recurrence as:
It turns out that the constants on the $O(n)$ don't matter for the big-O solution to the recurrence. We can analyze it with the tree method. Just as with mergesort, there are $k = \log_2(n)$ levels of the tree, because we cut the input size in half at each level. But now, there are $3^t$ nodes at level $t=0,1,\dots,\log_2(n)$. In each node at level $t$, we have an input size of $n/2^t$, so we do at most $C n/2^t$ work at that node for some $C$. The total is:
We used that the sum of an increasing geometric series, in this case $\frac{3}{2} + \left(\frac{3}{2}\right)^2 + \cdots + \left(\frac{3}{2}\right)^k$, is bounded by a constant times its final element.
In this section, we look at cases where D&C might seem like it works, but doesn't.
Objectives: After learning this material, you should be able to:
Sometimes, a tempting D&C solution doesn't actually work. Other times, there are variants of correct solutions that have a mistake causing them to fail. By practicing to spot these errors, we can get a deeper understanding of Divide and Conquer as well as practice with an important Algorithms skill: constructing counterexamples.
Consider the following version of mergesort. What is the problem?
// Algorithm 6: Try to Mergesort
1 try_to_mergesort(A):
2 if len(A) <= 1:
3 return A
4 let i = int(len(A)/2)
5 let B = try_to_mergesort(A[0:i])
6 let C = try_to_mergesort(A[i+1:end])
7 return (B,C) // list B followed by list C
The difference is in the last line: instead of using merge(B,C), we simply concatenate B and C together.
If you understood mergesort completely, you might feel that it's obvious that this code will fail. But can you prove that it's wrong?
The property of correctness is a "for all" statement: for every input, the algorithm produces a correct output. Therefore, the property of "not correct" is a "there exists" statement: there exists an example where the algorithm produces an incorrect output. To prove incorrectness, all you need to do is give an single input and show why the algorithm produces an incorrect output.
The "Try to Mergesort" algorithm is incorrect.
Consider the input A = [8,3]. We claim that the algorithm outputs [8,3], which is not sorted, so it is an incorrect output.
On input A = [8,3], the algorithm first splits [8] and [3] and recursively calls itself on each, resulting in B = [8] and C = [3]. It then concatenates them together, resulting in B followed by C, which is [8,3].
Let's try again. What is the problem here?
// Algorithm 7: Try Harder to Mergesort
1 try_harder_to_mergesort(A):
2 if len(A) <= 1:
3 return A
4 let i = int(len(A)/2)
5 let B = try_harder_to_mergesort(A[0:i])
6 let C = A[i+1:end]
7 return merge(B,C)
Here, the problem is that We only recursively call on B, but not C. That means that C may not be correctly sorted, leading to an incorrect result. But again, to prove it, we have to give a concrete example.
The "Try Harder to Mergesort" algorithm is incorrect.
Consider the input A = [1,2,4,3]. We claim the output will be [1,2,4,3], which is not correctly sorted.
First, the algorithm divides A into [1,2] and [4,3]. Then, it recursively calls itself on B, which we can check will return B = [1,2]. Then it will merge B = [1,2] and C = [4,3]. Recalling how merge works, it will start from the beginning of each array and take the smaller element, so it will produce [1,2,4,3].
In this section, we learn a rule of thumb for solving most recurrences we will encounter.
Objectives. After learning this material, you should be able to:
We will now see how to analyze the time complexity of divide-and-conquer algorithms in general. We can turn the tree method used above into a general rule-of-thumb that makes analysis much easier, as long as we can write the recurrence for a given D&C algorithm. Many divide-and-conquer algorithms are very similar to the algorithms above, so the general framework looks familiar:
// Divide-and-Conquer Framework
1 divide_and_conquer(L):
2 if is_base_case(L):
3 return solve_base_case(L)
4 (L1,...,Lt) = divide(L) // each Li is of size L/k for some k
5 A1 = divide_and_conquer(L1)
6 A2 = divide_and_conquer(L2)
7 // ...
8 Am = divide_and_conquer(Lm)
9 return combine(A1,...,Am)
We can see four steps:
To write a recurrence, we can analyze the framework:
// Divide-and-Conquer Framework, recurrence
1 divide_and_conquer(L): // T(n) time complexity total
2 if is_base_case(L): // O(1)
3 return solve_base_case(L) // O(1)
4 (L1,...,Lm) = divide(L) // complexity depends on the problem
5 A1 = divide_and_conquer(L1) // T(n/k)
6 A2 = divide_and_conquer(L2) // T(n/k)
7 // ... // ...
8 Am = divide_and_conquer(Lm) // T(n/k)
9 return combine(A1,...,At) // complexity depends on the problem
If divide() and combine() take time $O(n^c)$ for some constant $c \geq 0$, then we get the following general recurrence:
If an algorithm satisfies the recurrence $T(n) = m T(n/k) + O(n^c)$, then its time complexity is:
We won't prove it here (see e.g. Algorithms by Dasgupta, Papadimitriou, and Vazirani), but the proof follows the tree method.
Here is the general approach to analyzing the time complexity of Divide-and-Conquer algorithms:
For binary search, the recurrence is $T(n) = T(n/2) + O(1)$. Here $m=1$, $k=2$, and $c=0$. We have $\log_k(m) = 0 = c$. So we are in the third case of the master theorem, so the complexity is $O(n^c \log(n)) = O(\log(n))$.
For mergesort, the recurrence is $T(n) = 2T(n/2) + O(n)$. Here $m=2$, $k=2$, and $c=1$. We have $\log_k(m) = 1 = c$. So we are in the third case again, so the complexity is $O(n^c \log(n)) = O(n \log(n))$.
For Karatsuba multiplication, the recurrence is $T(n) = 3T(n/2) + O(n)$. Here $m=3$, $k=2$, and $c=1$. We have $\log_k(m) \approx 1.58 > c$. So we are in the second case of the master theorem, so the complexity is $O(n^{\log_k(m)}) = O(n^{\log_2(3)})$.