San Diego State University

Introaduction to Algorithms, Chapter16

Assume:

- We have a list of n items: a1, a2, ..., an

- Key(ak) = ak

- Probability of accessing item ak is known in advance and is P(ak)

- The list is ordered by keys, a1 <= a2 <= ... an

How to produce the BST that has the least search cost given the access probability for each key?

Example:

- list = a, b, c, d

- P(a) = .1, P(b) = .2, P(c) = .3, P(d) = .4
- Average Cost = 1*0.4 + 2*0.3 + 3*0.2 + 4*0.1 =
**2.0**

- Optimal BST

- Average Cost = 1*0.3 + 2*0.4 + 2*0.2 + 3*0.1 =
**1.8**

Let A[j, k] = minimum average search time for a binary search tree with items aj <= aj+1 <= ... ak

j\k | 0 | 1 | 2 | 3 | 4 | 5 |

1 | 0 | P(a1) | ||||

2 | 0 | 0 | P(a2) | A[2,4] | ||

3 | 0 | 0 | 0 | P(a3) | ||

4 | 0 | 0 | 0 | 0 | P(a4) | |

5 | 0 | 0 | 0 | 0 | 0 | P(a5) |

6 | 0 | 0 | 0 | 0 | 0 | 0 |

Time Complexity for finding Optimal BST Theta(n

"Dynamic programming algorithm stores the results for small subproblems and looks them up, rather than recomputing them, when it needs them later to solve larger subproblems"

Baase

"Dynamic programming algorithm is very useful approach to many optimization problems. ... Usually, we can find a tailor-made algorithm which is more efficient than the straight-forward algorithm based on dynamic programming"

T. C. Hu

Characterize the structure of an optimal solution

Recursively define the value of an optimal solution

Compute the value of an optimal solution in a bottom-up fashion

Construct an optimal solution from computed information

Let A be a 10 * 100 matrix,

- B be a 100 * 5 matrix
- C be a 5 * 50 matrix

- A * B is a 10 * 5 matrix
- B * C is a 100 * 50 matrix

Computing

- A * B takes 10 * 100 * 5 = 5,000 multiplications
- (A * B) * C takes 10 * 5 * 50 = 2,500 additional mult.
- So (A * B) * C requires total of 7,500 multiplications

Computing

- B * C takes 100 * 5 * 50 = 25,000 multiplications
- A * (B * C) takes 10 * 100 * 50 = 50,000 additional mult.
- So A * (B * C) requires total of 75,000 multiplications

But (A * B) * C = A * (B * C)

Given a chain <A1, A2, ..., An> matrices, where matrix Ak has dimension pk-1 * pk fully parenthesize the product A1* A2*... *An, in a way that minimizes the number of scalar multiplications

Optimal solution is of the form:

- (A1* ... *Ak) * (Ak+1* ... *An)

not showing the parentheses in (A1* ... *Ak) or in (Ak+1* ... *An)

We must use the optimal parenthesization of A1* ... *Ak and the optimal parenthesization of Ak+1* ... *An

Thus the optimal solution to an instance of the matrix-chain multiplication problem contains within it optimal solutions to subproblem instances

That is it is recursive

Recall matrix Ak has dimension pk-1 * pk for k = 1,..., n

Let m[k, w] be the minimum the number of scalar multiplications needed to compute Ak* ... *Aw

If (Ak* ... *Az) * (Az+1* ... *Aw) is the optimal solution then

- m[k, w] = m[k, z] + m[z+1, w] + pk-1 * pz * pk

since (Ak* ... *Az) is a pk-1 * pz matrix

and (Az+1* ... *Aw) is a pz * pw matrix

Example

Matrix dimension r pr A1 10*20 0 10 A2 20*3 1 20 A3 3*5 2 3 A4 5*30 3 5 4 30

1 | 2 | 3 | 4 | |

1 | 0 | 600 | 750 | 1950 |

2 | 0 | 300 | 2250 | |

3 | 0 | 450 | ||

4 | 0 |

Example

Matrix dimension r pr A1 10*20 0 10 A2 20*3 1 20 A3 3*5 2 3 A4 5*30 3 5 4 30

1 | 2 | 3 | 4 | |

1 | A1 | A1A2 | (A1A2)A3 | (A1A2)(A3A4) |

2 | A2 | A2A3 | A2(A3A4) | |

3 | A3 | A3A4 | ||

4 | A4 |

We have a list of n items: a1, a2, ..., an

Probability of accessing item ak is P(ak)

Let A[j, k] = minimum average search time for a binary search tree with items aj <= aj+1 <= ... ak

Let root[j,k] = p that gave the minimum value for A[j, k]

That is root[j,k] = root of OBST for items aj, a2, ..., ak

Let w[j,k] = P(aj) + P(aj+1) + ... + P(ak)

for k = 1 to n do

- A[k, k] = P(ak)
- A[k, k-1] = 0
- root[k,k] = k
- w[k, k] = P(ak)

A[n+1, n] = 0

for diagonal = 1 to n -1 do

- for j = 1 to n - diagonal do
- k = j + diagonal
- w[j, k] = w[j, k - 1] + P(ak)
- let p, j <= p <= k, be the value minimizes:
- root[j,k] = p
- A[j, k] = A[j, p-1] + A[p+1, k] + w[j, k]
- end for

k 1 2 3 4 5 6 ak a b c d e f P(ak)'s = 0.4 0.05 0.15 0.05 0.1 0.25

root

1 1 1 1 1 3 0 2 3 3 3 5 0 0 3 3 3 5 0 0 0 4 5 6 0 0 0 0 5 6 0 0 0 0 0 6

A

0 0.4 0.5 0.85 1 1.35 2.1 0 0 0.05 0.25 0.35 0.6 1.2 0 0 0 0.15 0.25 0.5 1.05 0 0 0 0 0.05 0.2 0.6 0 0 0 0 0 0.1 0.45 0 0 0 0 0 0 0.25 0 0 0 0 0 0 0

P(ak)'s = (0.15 0.025 .05 .025 .05 .125 .025 .075 0.075 .05 .15 .075 .05 .025 .05)

Root

1 1 1 1 1 3 3 6 6 6 6 6 6 6 6 0 2 3 3 3 5 6 6 6 6 6 9 9 9 11 0 0 3 3 3 5 6 6 6 6 9 9 9 9 11 0 0 0 4 5 6 6 6 6 6 9 9 9 9 11 0 0 0 0 5 6 6 6 6 8 9 9 9 11 11 0 0 0 0 0 6 6 6 8 8 9 9 11 11 11 0 0 0 0 0 0 7 8 8 9 9 11 11 11 11 0 0 0 0 0 0 0 8 8 9 9 11 11 11 11 0 0 0 0 0 0 0 0 9 9 11 11 11 11 11 0 0 0 0 0 0 0 0 0 10 11 11 11 11 11 0 0 0 0 0 0 0 0 0 0 11 11 11 11 11 0 0 0 0 0 0 0 0 0 0 0 12 12 12 13 0 0 0 0 0 0 0 0 0 0 0 0 13 13 13 0 0 0 0 0 0 0 0 0 0 0 0 0 14 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 15

A[1,15] = 2.925

for diagonal = 1 to n -1 do

- for j = 1 to n - diagonal do
- k = j + diagonal
- w[j, k] = w[j, k - 1] + P(ak)
- let p,
**root[j,k-1] <= p <= root[j+1,k],**be the value minimizes: - root[j,k] = p
- A[j, k] = A[j, p-1] + A[p+1, k] + w[j, k]
- end for

Let H(i, j) be a real number for 1 <= i < j <= n

Let c(i, j) be defined by:

- c(i, i) = 0
- c(i, j) = H(i, j) +

H(i, j) is

- H(j, k) <= H(x, y) if j <= x < y <= k

H(i, j) satisfies the

- H(j, k) + H(x, y) <= H(x, k) + H(j, y) if j <= x < k <= y

- K(i, j) <= K(i, j+ 1) <= K(i+1, j+1)

Recall in OBST we have:

- We have a list of n items: a1, a2, ..., an and leaves b0, b1, ..., bn
- Probability of accessing item ak is P(ak) =
**Alpha**k - Let
**Beta**k be the probability of accessing a key that is between ak and ak+1, that is leaf bk

In splay tree we have

Theorem 9: Let Popt be the be the weighted path length of optimum BST. We have:

- where H = H(
**Beta**0 ,**Alpha**1 ,**Beta**1 ,**Alpha**2,**Beta**2,... ,**Alpha**n,**Beta**n) is entropy

Thus we have:

- for some x

Now:

So m*H =

Thus: m*Popt <= m*H + 1 = + m

We have a list of n items: a1, a2, ..., an

Probability of accessing item ak is P(ak)

k 1 2 3 4 5 6

ak a b c d e f

P(ak)'s = 0.4 0.05 0.15 0.05 0.1 0.25

Average Access Cost = 2.2

ak a b c d e f

P(ak)'s = 0.4 0.05 0.15 0.05 0.1 0.25

Average Access Cost = 2.1

An alphabetic tree is a binary search tree in which all data is in the leaves. Internal nodes are used in search for the data

Let V1, V2,... Vn be the order of the leaves

Let wk be the weight, or frequency of access, of leaf Vk

Combining Vk and Vp, denote their parent node by Vkp and it weight wkp = wk+ wp

All leaves are square nodes, all parents are round nodes

Two nodes are a compatible pair if they are adjacent or if all nodes between them are round nodes

The weight of a pair is the weight of the parent of the two nodes

A pair with minimum weight over all pairs is a minimum pair

Minimum compatible pair is the compatible pair with the least weight over all compatible pairs

- Ties are broken by taking the pair with the left most left node
- If two compatible pairs have the same left node and the same weight, then pick the pair with the leftmost right node

1. Construction

- Find the minimum compatible pair
- Replace the left node of the pair by the pair's parent
- Remove right node of the pair
- Repeat n-1 times
- Call the resultant tree T'

2. Level Assignment

- Determine the level number Lk of every leaf Vk in T'

- We have the level numbers L1, L2,... Ln of all leaves
- Find the leftmost maximum level number, say Lk = q
- Then Lk+1 = q
- Replace Lk and Lk+1 with a parent node with level q - 1
- Repeat n - 1 times

Theorem. The Hu-Tucker algorithm can be implemented to produce the optimal alphabetic tree with N leaves in O(Nlg(N)) time and O(N) space

We have a list of n items: a1, a2, ..., an and leaves b0, b1, ..., bn

Probability of accessing item ak is known in advance and is P(ak) =

Let

The list is ordered by keys, b0< a1 <b1< a2 < ... < an < bn

(

Let

Let L(ak) be the level of the node ak

Let

P is the weighted path length of a tree

Let (

H(

is the

Let TBB be the tree resulting from the nearly optimal BST algorithm 1

Theorem 7[2]: Let L(ak) be the depth of node ak and let L(bk) be the depth of leaf bk in tree TBB Then

- bk <= floor (log 1/bk) + 2
- ak <= floor (log 1/ak)

Theorem 8[3]: Let PBB be the weighted path length of the tree TBB. Then

Theorem 9[4]: Let PBB be the weighted path length of tree TBB. Let Popt be the be the weighted path length of optimum BST. We have:

- where H = H(
**Beta**0 ,**Alpha**1 ,**Beta**1 ,**Alpha**2,**Beta**2,... ,**Alpha**n,**Beta**n) and

This gives us

In English, the probability of occurrence of the i-th most frequent word is approximately[5]

This yields

H(

The weighted path length of an optimum binary search tree for

We have a list of n items: a1, a2, ..., an

Probability of accessing item ak is P(ak)

k 1 2 3 4 5 6

ak a b c d e f

P(ak)'s = 0.4 0.05 0.15 0.05 0.1 0.25

Problem:

k 1 2 3

ak a b c

P(ak)'s = 0.45 0.1 0.45

Given the ordered set {a} of names, such that a1 <= a2 <= ... an , and two parameters, F and N0

(1) If N <= N0 , use dynamic programming.

(2) If N> N0, let W[k, l] be the weight of the subtree with frequencies

**Beta**k ,**Alpha**k+1 ,**Beta**k+1 , ... ,**Alpha**l,**Beta**l- F a parameter and AC the centroid.
- Form the ordered set of names {AF} = {AL} union AC,
- where the members of the set {AL} satisfy

(3) Find an index, max, such that

(4) If in the set {AF} there is at least one name preceding or equal to AC with associated frequency

- If there is no such p, let {AQ} be the null set and go to Step 6.

(5) If ap is the first member of {AF} and

- if ap is not the first member of {AF}, let {AQ} be the null set.

(6) If in the set {AF} there is at least one name following or equal to AC with associated frequency

- if there is no such r, let {AS} be the null set and go to Step 8.

(7) If ar is the last member of {AF} and

- If ar is not the last member of {AF}, let {AS} be the null set.

(9) Go to Step 1 and repeat the algorithm for the subtrees a1, a2, ... aroot-1 and aroot+1, ... aN, where N is root-I and N-root for the two cases.

if

if

if

if

if

The tree takes O(Nlog(N)) to construct

Average search time for NOBST is within 2% of the average search time of the OBST

N0= 15, F = 5

N | Alpha freq | Beta freq | OBST | NOBST |

5 | 19,846 | 982,497 | 3.4114 | 3.4114 |

15 | 42,653 | 959,690 | 4.2864 | 4.2864 |

25 | 60,087 | 942,256 | 5.0638 | 5.1033 |

50 | 92,117 | 910,226 | 6.0483 | 6.1461 |

100 | 138,975 | 863,368 | 7.0007 | 7.0437 |

150 | 173,157 | 829,186 | 7.4885 | 7.5503 |

200 | 200,412 | 801,931 | 7.8795 | |

500 | 305,266 | 697,077 | 8.9606 | |

1000 | 401,288 | 601,055 | 9.6490 | |

3000 | 561,956 | 440,387 | 10.6220 | |

6000 | 655,538 | 346,805 | 11.1177 | |

12000 | 740,022 | 262,321 | 11.1592 | |