# CSE347 Analysis of Algorithms (Lecture 1)

## Greedy Algorithms

* Builds up a solution by making a series of small decisions that optimize some objective.
* Make one irrevocable choice at a time, creating smaller and smaller sub-problems of the same kind as the original problem.
* There are many potential greedy strategies and picking the right one can be challenging.

### A Scheduling Problem

You manage a giant space telescope.

* There are $n$ research projects that want to use it to make observations.
* Only one project can use the telescope at a time.
* Project $p_i$ needs the telescope starting at time $s_i$ and running for a length of time $t_i$.
* Goal: schedule as many as possible

Formally

Input:

* Given a set $P$ of projects, $|P|=n$
* Each request $p_i\in P$ occupies interval $[s_i,f_i)$, where $f_i=s_i+t_i$

Goal: Choose a subset $\Pi\sqsubseteq P$ such that

1. No two projects in $\Pi$ have overlapping intervals.
2. The number of selected projects $|\Pi|$ is maximized.

#### Shortest Interval 

Counter-example: `[1,10],[9,12],[11,20]`

#### Earliest start time

Counter-example: `[1,10],[2,3],[4,5]`

#### Fewest Conflicts

Counter-example: `[1,2],[1,4],[1,4],[3,6],[7,8],[5,8],[5,8]`

#### Earliest finish time

Correct... but why

#### Theorem of Greedy Strategy (Earliest Finishing Time)

Say this greedy strategy (Earliest Finishing Time) picks a set $\Pi$ of intervals, some other strategy picks a set $O$ of intervals.

Assume sorted by finishing time

* $\Pi=\{i_1,i_2,...,i_k\},|\Pi|=k$
* $O=\{j_1,j_2,...,j_m\},|O|=m$

We want to show that $|\Pi|\geq|O|,k>m$

#### Lemma: For all $r<k,f_{i_r}\leq f_{j_r}$

We proceed the proof by induction.

* Base Case, when r=1.
    $\Pi$ is the earliest finish time, and $O$ cannot pick a interval with earlier finish time, so $f_{i_r}\leq f_{j_r}$

* Inductive step, when r>1.
    Since $\Pi_r$ is the earliest finish time, so for any set in $O_r$, $f_{i_{r-1}}\leq f_{j_{r-1}}$, for any $j_r$ inserted to $O_r$, it can also be inserted to $\Pi_r$. So $O_r$ cannot pick an interval with earlier finish time than $Pi$ since it will also be picked by definition if $O_r$ is the optimal solution $OPT$.

#### Problem of “Greedy Stays Ahead” Proof

* Every problem has very different theorem.
* It can be challenging to even write down the correct statement that you must prove.
* We want a systematic approach to prove the correctness of greedy algorithms.

### Road Map to Prove Greedy Algorithm

#### 1. Make a Choice

Pick an interval based on greedy choice, say $q$

Proof: **Greedy Choice Property**: Show that using our first choice is not "fatal" – at least one optimal solution makes this choice.

Techniques: **Exchange Argument**: "If an optimal solution does not choose $q$, we can turn it into an equally good solution that does."

Let $\Pi^*$ be any optimal solution for project set $P$.
- If $q\in \Pi^*$, we are done.
- Otherwise, let $x$ be the optimal solution from $\Pi^*$ that does not pick $q$. We create another solution $\bar{\Pi^*}$ that replace $x$ with $q$, and prove that the $\bar{\Pi^*}$ is as optimal as $\Pi^*$

#### 2. Create a smaller instance $P'$ of the original problem

$P'$ has the same optimization criteria.

Proof: **Inductive Structure**: Show that after making the first choice, we're left with a smaller version of the same problem, whose solution we can safely combine with the first choice.

Let $P'$ be the subproblem left after making first choice $q$ in problem $P$ and let $\Pi'$ be an optimal solution to $P'$. Then $\Pi=\Pi^*\cup\{q\}$ is an optimal solution to $P$.

$P'=P-\{q\}-\{$projects conflicting with $q\}$

#### 3. Solution: Union of choices that we made

Union of choices that we made.

Proof: **Optimal Substructure**: Show that if we solve the subproblem optimally, adding our first choice creates an optimal solution to the *whole* problem.

Let $q$ be the first choice, $P'$ be the subproblem left after making $q$ in problem $P$, $\Pi'$ be an optimal solution to $P'$. We claim that $\Pi=\Pi'\cup \{q\}$ is an optimal solution to $P$.

We proceed the proof by contradiction.

Assume that $\Pi=\Pi'+\{q\}$ is not optimal.


By Greedy choice property $GCP$. we already know that $\exists$ an optimal solution $\Pi^*$ for problem $P$ that contains $q$. If $\Pi$ is not optimal, $cost(\Pi^*)<cost(\Pi)$. Then since $\Pi^*-q$ is also a feasible solution to $P'$. $cost(\Pi^*-q)>cost(\Pi-q)=\Pi'$ which leads to contradiction that $\Pi'$ is an optimal solution to $P'$.

#### 4. Put 1-3 together to write an inductive proof of the Theorem

This is independent of problem, same for every problem.

Use scheduling problem as an example:

Theorem: given a scheduling problem $P$, if we repeatedly choose the remaining feasible project with the earliest finishing time, we will construct an optimal feasible solution to $P$.

Proof: We proceed by induction on $|P|$. (based on the size of problem $P$).

- Base case: $|P|=1$.
- Inductive step.
  - Inductive hypothesis: For all problems of size $<n$, earliest finishing time (EFT) gives us an optimal solution.
  - EFT is optimal for problem of size $n$.
  - Proof: Once we pick q, because of greedy choice. $P'=P=\{q\} -\{$interval that conflict with $q\}$. $|P'|<n$, By Inductive hypothesis, EFT gives us an optimal solution to $P'$, but by inductive substructure, and optimal substructure. $\Pi'$ (optimal solution to $P'$), we have optimal solution to $P$.

_this step always holds as long as the previous three properties hold, and we don't usually write the whole proof._

```python
# Algorithm construction for Interval scheduling problem
def schedule(p):
  # sorting takes O(n)=nlogn
  p=sorted(p,key=lambda x:x[1])
  res=[P[0]]
  # O(n)=n
  for i in p[1:]:
    if res[-1][-1]<i[0]:
      res.append(i)
  return res
```

## Extra Examples:

### File compression problem

You have $n$ files of different sizes $f_i$.

You want to merge them to create a single file. $merge(f_i,f_j)$ takes time $f_i+f_j$ and creates a file of size $f_k=f_i+f_j$.

Goal: Find the order of merges such that the total time to merge is minimized.

Thinking process: The merge process is a binary tree and each of the file is the leaf of the tree.

The total time required =$\sum^n_{i=1} d_if_i$, where $d_i$ is the depth of the file in the compression tree.

So compressing the smaller file first may yield a faster run time.

Proof:

#### Greedy Choice Property

Construct part of the solution by making a locally good decision.

Lemma: $\exist$ some optimal solution that merges the two smallest file first, lets say $[f_1,f_2]$

Proof: **Exchange argument**

* Case 1: Optimal choice already merges $f_1,f_2$, done. Time order does not matter in this problem at some point.
  * eg: [2,2,3], merge 2,3 and 2,2 first don't change the total cost
* Case 2: Optimal choice does not merges $f_1$ and $f_2$.
  * Suppose the optimal solution merges $f_x,f_y$ as the deepest merge.
  * Then $d_x\geq d_1,d_y\geq d_2$. Exchanging $f_1,f_2$ with $f_x,f_y$ would yield a strictly less greater solution since $f_1,f_2$ already smallest.

#### Inductive Structure

* We can combine feasible solution to the subproblem $P'$ with the greedy choice to get a feasible solution to $P$
* After making greedy choice $q$, we are left with a strictly smaller subproblem $P'$ with the same optimality criteria of the original problem
* 
Proof: **Optimal Substructure**: Show that if we solve the subproblem optimally, adding our first choice creates an optimal solution to the *whole* problem.

Let $q$ be the first choice, $P'$ be the subproblem left after making $q$ in problem $P$, $\Pi^*$ be an optimal solution to $P'$. We claim that $\Pi=\Pi'\cup \{q\}$ is an optimal solution to $P$.

We proceed the proof by contradiction.

Assume that $\Pi=\Pi^*+\{q\}$ is not optimal.

By Greedy choice property $GCP$. we already know that $\Pi^*$ is optimal solution that contains $q$. Then $|\Pi^*|>|\Pi|$ $\Pi^*-q$ is also feasible solution to $P'$. $|\Pi^*-q|>|\Pi-q|=\Pi'$ which is an optimal solution to $P'$ which leads to contradiction.

Proof: **Smaller problem size**

After merging the smallest two files into one, we have strictly less files waiting to merge.

#### Optimal Substructure

* We can combine optimal solution to the subproblem $P'$ with the greedy choice to get a optimal solution to $P$

Step 4 ignored, same for all greedy problems.

### Conclusion: Greedy Algorithm

* Algorithm
* Runtime Complexity
* Proof
  * Greedy Choice Property
    * Construct part of the solution by making a locally good decision.
  * Inductive Structure
    * We can combine feasible solution to the subproblem $P'$ with the greedy choice to get a feasible solution to $P$
    * After making greedy choice $q$, we are left with a strictly smaller subproblem $P'$ with the same optimality criteria of the original problem
  * Optimal Substructure
    * We can combine optimal solution to the subproblem $P'$ with the greedy choice to get a optimal solution to $P$
* Standard Contradiction Argument simplifies it

## Review:

### Essence of master method

Let $a\geq 1$ and $b>1$ be constants, let $f(n)$ be a function, and let $T(n)$ be defined on the nonnegative integers by the recurrence

$$
T(n)=aT(\frac{n}{b})+f(n)
$$

where we interpret $n/b$ to mean either ceiling or floor of $n/b$. $c_{crit}=\log_b a$ Then $T(n)$ has to following asymptotic bounds.

* Case I: if $f(n) = O(n^{c})$ ($f(n)$ "dominates" $n^{\log_b a-c}$) where $c<c_{crit}$, then $T(n) = \Theta(n^{c_{crit}})$

* Case II: if $f(n) = \Theta(n^{c_{crit}})$, ($f(n), n^{\log_b a-c}$ have no dominate) then $T(n) = \Theta(n^{\log_b a} \log_2 n)$

  Extension for $f(n)=\Theta(n^{critical\_value}*(\log n)^k)$

  * if $k>-1$

    $T(n)=\Theta(n^{critical\_value}*(\log n)^{k+1})$

  * if $k=-1$

    $T(n)=\Theta(n^{critical\_value}*\log \log n)$

  * if $k<-1$

    $T(n)=\Theta(n^{critical\_value})$

* Case III: if $f(n) = \Omega(n^{log_b a+c})$ ($n^{log_b a-c}$ "dominates" $f(n)$) for some constant $c >0$, and if a $f(n/b)<= c f(n)$ for some constant $c <1$ then for all sufficiently large $n$, $T(n) = \Theta(n^{log_b a+c})$