Time and Space Complexity

Table of Contents

Adapted from Hunter Schafer: https://courses.cs.washington.edu/courses/cse373/20au/lectures/4.pptx

1 Reading

Read Section 5.1.1 of Bailey (p. 81–85 of the text, 97–101 of the pdf). See the rest of Section 5.1 for examples and additional discussion.

2 Introduction

  • Time and space analysis help us choose between data structures
    • ArrayList
      • Zero extra space (overhead) per element (internal array just stores the elements)
      • But extra capacity is wasted
    • LinkedList
      • One or two extra references per element (next and previous for each node)
      • But exactly as many nodes as elements (no wasted capacity)
    • If ArrayList is managed precisely, it will be more space efficient
      • But both structures take linear space
  • We need a tool that is
    • Simple: we don't care about tiny differences in implementation, we want the big picture result
    • Mathematically rigorous: use mathematical functions as a precise, flexible basis
    • Decisive: produce a clear comparison indicating which code takes "more work"

3 Algorithmic Analysis

analysis.png

  • Code Modeling: go from code to a function describing the code's runtime (the amount of work it does when run)
  • Asymptotic Analysis: go from this function to a complexity class describing the function's asymptotic behavior (what happens when n gets really huge)

3.1 Complexity Class

Complexity Class Big-O Runtime if you double \(n\)
constant \(O(1)\) unchanged
logarithmic \(O(\log_2 n)\) increases slightly
linear \(O(n)\) doubles
log-linear (linearithmic) \(O(n \log_2 n)\) slightly more than doubles
quadratic \(O(n^2)\) quadruples
… … …
exponential \(O(2^n)\) multiplies drastically

complexity-classes.png

Does complexity really matter? Yes! The following table presents several hypothetical algorithm runtimes as an input size \(n\) grows, assuming that each algorithm required 100ms to process 100 elements. Notice that even if they all start at the same runtime for a small input size, the ones in higher complexity classes become so slow as to be impractical.

runtimes.png

3.2 Code Modeling

Code modeling is the process of mathematically representing how many operations a piece of code will perform in relation to the input size \(n\).

3.2.1 What is an operation?

We don't know exact time every operation takes, but for now let;s try simplifying assumption: all basic operations take the same time.

  • Basics:
    • +, -, /, *, %, ==
    • Assignment
    • Returning
    • Variable/array access
  • Function Calls
    • Total time from the operations in the code for that function
  • Conditionals
    • Total time for the condition abd time for the followed branch
  • Loops
    • Number of iterations * total time for the condition and code inside the loop

3.2.2 Example 1

code-modeling-1.png

3.2.3 Example 2

code-modeling-2.png

3.3 Asymptotic Analysis

We have an expression for \(f(n)\). How do we get the Big-O complexity?

  1. Find the dominating term and delete all others.
    • The dominating term is the one that is largest as \(n\) gets bigger. In this class, often the largest power of \(n\).
  2. Remove any constant factors.

big-O-1.png

Is it okay to throw away all that information?

  • Think of asymptotic analysis as the analysis of function's behavior as its input approaches infinity
    • We only care about what happens when \(n\) approaches infinity
    • For small inputs, doesn't really matter: all code is "fast enough"
    • Since we’re dealing with infinity, constants and lower-order terms don't meaningfully add to the final result. The highest-order term is what drives growth!
  • Remember our goals: a Simple and Decisive analysis tool

No seriously, this is really okay?

  • There are tiny variations in these functions (\(2n\) vs. \(3n\) vs. \(3n+1\))
    • But at infinity, will be clearly grouped together
    • We care about which group a function belongs in
  • Let's convince ourselves this is the right thing to do:

4 Timing Demo

import java.util.ArrayList;
import java.util.LinkedList;

import edu.princeton.cs.algs4.StopwatchCPU;

public class Timing {
    public static void main(String[] args) {
        StopwatchCPU watch = new StopwatchCPU();

        // compare O(n) with O(1)
        int n = 100000;
        int[] nums = new int[n];
        double start = watch.elapsedTime();
        for (int i = 0; i < nums.length; i++) {
            nums[i] = i;
        }
        double end = watch.elapsedTime();
        System.out.println("loop took " + (end - start) + "s");

        start = watch.elapsedTime();
        nums[0] = 100;
        end = watch.elapsedTime();
        System.out.println("one assignment took " + (end - start) + "s");

        ArrayList<Integer> arraylist = new ArrayList<>();
        LinkedList<Integer> linkedlist = new LinkedList<>();
        start = watch.elapsedTime();
        // O(n^2)
        for (int i = 0; i < nums.length; i++) {
            // linear time insertion into arraylist
            arraylist.add(0, i);
        }
        end = watch.elapsedTime();
        System.out.println("linear time insertion took " + (end - start) + "s");
        start = watch.elapsedTime();
        // O(n)
        for (int i = 0; i < nums.length; i++) {
            // constant time insertion into linkedlist
            linkedlist.add(0, i);
        }
        end = watch.elapsedTime();
        System.out.println("constant time insertion took " + (end - start) + "s");
    }
}

5 Practice Problems1

  1. Model the number of operations involved in the following code in terms of \(n\): 

    public int method1(int n) {
    int sum = 0;
    for (int i = n; i > 0; i -= 2) {
        for (int j = 0; j < n; j++) {
            sum++;
        }
    }
    return sum;
}

  2. Model the number of operations involved in the following code in terms of \(n\): 

    public int method2(int n) {
    int sum = 0;
    for (int i = 1; i <= n * 2; i++) {
        for (int j = 1; j <= n; j++) {
            sum++;
        }
    }
    for (int j = 1; j < 100; j++) {
        sum++;
        sum++;
    }
    return sum;
}

  3. Model the number of operations involved in running bar in terms of \(n\): 

    public boolean foo(int x) {
    return x % 2 == 0 && x > 0;
}

public int bar(int n) {
    int sum = 0;
    for (int i = 0; i < n; i++) {
        if (foo(i)) {
            sum += i;
        }
    }
    return sum;
}
  4. Give the big-O complexity for problems 1, 2, and 3.
  5. Suppose an algorithm takes exactly the given number of statements for each value below, in terms of an input size \(n\). Give the big-O bound for each algorithm, representing the closest complexity class for that algorithm based on that runtime.
    1. \(3n\)
    2. \(n + 900\)
    3. \(\frac{1}{2}n\log n + \log n\)
    4. \(n^2 - (n + n\log n + 1000)\)
    5. \(n^2 \log n + 2n\)
    6. \(\frac{1}{2}(3n + 5 + n)\)
    7. \((2n + 5 + n^4)/n\)
    8. \(n! + 2n\) (\(n!\) is \(n\)-factorial, meaning the product \(n \times (n - 1) \times (n - 2) \cdots 2 \times 1\))

Footnotes:

1

Solutions:

  1. \(f(n) = (3n + 3)\frac{n}{2} + 3 = \frac{3n^2 + 3n}{2} + 3\) 

    public int method1(int n) {
    int sum = 0;                      // +1
    for (int i = n; i > 0; i -= 2) {  // +1 before loop (i = n), +2 inside (i > 0, i -= 2) |
        for (int j = 0; j < n; j++) { // +1 before, +2 inside |                            | (3n + 3)(n/2) total
            sum++;                    // +1                   | 3n total                   |
        }
    }
    return sum;                       // +1
}

  2. \(f(n) = 6n^2 + 8n + 400\) 

    public int method2(int n) {
    int sum = 0;                       // +1
    for (int i = 1; i <= n * 2; i++) { // +1, +3            |
        for (int j = 1; j <= n; j++) { // +1, +2 |          | (3n + 4)2n
            sum++;                     // +1     | 3n       |
        }
    }
    for (int j = 1; j < 100; j++) {    // +1, +2 |
        sum++;                         // +1     | 99*4
        sum++;                         // +1     |
    }
    return sum;                        // +1
}

  3. \(f(n) = 7n + 3\) 

    public boolean foo(int x) {
    return x % 2 == 0 && x > 0;   // +4
}                                           

public int bar(int n) {                     
    int sum = 0;                  // +1        
    for (int i = 0; i < n; i++) { // +1, +2  |
        if (foo(i)) {             // see foo | 7n, assume sum += i always happens (overly pessimistic worst case)
            sum += i;             // +1      |
        }
    }
    return sum;                   // +1
}
  4. \(O(n^2)\), \(O(n^2)\), \(O(n)\)
    1. \(O(n)\)
    2. \(O(n)\)
    3. \(O(n\log n)\)
    4. \(O(n^2)\)
    5. \(O(n^2 \log n)\)
    6. \(O(n)\)
    7. \(O(n^3)\)
    8. \(O(n!)\)