Computational complexity

Authors: Sarang Harshey (sh2669) (SysEn 5800 Fall 2024)

Stewards: Nathan Preuss, Wei-Han Chen, Tianqi Xiao, Guoqing Hu

Introduction

Computational complexity is a study of resources, especially time and space, required to solve a computational problem. It provides an understanding of how the resource requirement scales-up as the problem gets bigger and bigger. Another objective is to know which problems are solvable using algorithms and which problems are truly difficult and practically impossible to solve.

Complexity Classes

Problems can be grouped into sets that share a common property and require similar amount of computational resources. These sets or groups are known as Complexity Classes. Some basic types of complexity classes are mentioned below.

P (Polynomial Time) Class

If a problem can be solved in polynomial time, then it is considered as a Class P problem. Problems in this class are efficiently solvable, however, the time required to solve grows polynomially with the size of input. Linear programming problems are considered under P class.

NP (Non-deterministic Polynomial Time) Class

If the solution of a problem can be verified in polynomial time, then the problem is considered as a Class NP problem. These problems require a solution that can be verified, but finding a solution itself may take more than polynomial time.

NP-Hard Class

NP-Hard problems may or may not fall under NP Class. A problem H is NP-Hard if for every problem L in NP, there exists a polynomial time reduction from L to H. These are usually optimization problems.

NP-Complete Class

NP-Complete problems are a subset of NP class. If all the problems in a NP can be reduced to problem H in polynomial time, then problem H is considered as NP-Complete. NP Complete problems are the hardest problems to solve. These are usually decision problems.

History

The history of computational complexity can be traced back to the paper, On computable numbers with an application to the Entsheidungs problem, published in 1936 by A M Turing. In this paper, Turing introduced his famous Turing Machine and provided a convincing formalization of computable function. In following years, once the theory was established to determine which problem can and cannot be solved, questions regarding relative computational difficulty came up. This can be considered as the birth of computational complexity. Rabin M O in his paper, Degree of difficulty of computing a function and partial ordering or recursive sets, in 1960 made an attempt to measure the amount of work inherent in computing a function along with defining the degree of difficulty of computing a function. Another influential paper that was published in 1965 by Hartmanis and Stearns, On the computational complexity of algorithms, discussed the method of categorizing problems according to how hard they are to compute. As part of this paper, a variety of theorems were established. It also provided a way to classify functions or recognition problems according to their computational complexity and laid out definitions of time complexity and space complexity. Paper by Alan Cobham in 1965, The intrinsic computational difficulty of functions, discussed the relative difficulty between addition and multiplication and characterised an important class of functions computable in time bounded by a polynomial in the length of input. These papers from Cobhem, Hartemanis and Rabin can be considered as the foundation for the field of computational complexity. Manuel Blum in 1967, formulated axioms known as Blum Axioms, that specify desirable properties of complexity measures on the set of computation function. Field of computational complexity started to grow from 1971 onwards, with studies from Steven Cook, Leonid Levin and Richard Karp which expanded the study of problems belonging to class NP-Complete.

Need for Computational Complexity

Computational complexity provides a method to analyse an algorithm in terms of complexity and provides information on the performance that can be expected. In a complex algorithm, through computational complexity, costliest steps (in terms of space and time) can be identified and efforts can be made for improving efficiency by tuning these steps before implementation.  It also helps in selecting the best algorithms and eliminate inefficient algorithms.

Algorithm Discussion

Complexity of an algorithm can be calculated by establishing a relationship between the length of the input and the number of steps (time complexity) or amount of space that algorithm takes (space complexity) to compute the problem. This relationship is usually represented as a function. The smaller the value of this function, the higher the efficiency of the algorithm. Function value also indicates the growth rate with respect to size of the input.

Asymptotic Notations

Mathematical notations, usually known as Asymptotic Notations, are used to describe the complexity of any algorithm. There are broadly three notations used to describe complexity, namely – Big-O (O - Notation), Big-Theta (Θ - Notation) and Big-Omega (Ω - Notation).

Big-O Notation

O - notation represents the upper bound or the worst case, i.e. maximum time (or space) required for an algorithm. For functions f and t,

$f(n)=O(t(n))$,

if $f(n)\leq c.t(n)$ for all $n\geq d$, where c and d are positive integers

Big-Omega Notation

Ω - notation represents lower bound or the best case, i.e. minimum time (or space) required for an algorithm. For functions, f and t,

$f(n)=\Omega (t(n))$,

if $f(n)\geq c.t(n)$ for all $n\geq d$, where c and d are positive integers

Big-Theta Notation

Θ - notation represents average-case as it gives both upper and lower bounds of the algorithm. For functions, f and t,

$f(n)=\Theta (t(n))$,

if $c_{1}.t(n)\leq f(n)\leq c_{2}.t(n)$ for all $n\geq d$, where c₁, c₂ and d are positive integers

Common Big-O Notations

Big-O (O-notation) is commonly used to describe the complexity of algorithms, as it provides the worst-case estimate. It considers only the number of steps involved for the problem resolution and not the hardware configuration that’s being used. Some of the common complexities are mentioned below.

Constant Time Complexity $O(1)$: computation time for the problem is independent of size of the input data.

Linear Time Complexity $O(n)$: computation time for the problem is proportional to the size of the input n and increase linearly.

Polynomial Time Complexity $O(n^{c})$: computation time for the problem is polynomial times of size of the input n where c is a constant depending on the problem.

Exponential Time Complexity $O(2^{n})$: computation time for the problem doubles with every addition of input n.

Logarithmic Time Complexity $O(logn)$: computation time for the problem is proportional to the logarithm of input size n.

Numerical Example

Complexity is defined based on the number of steps involved. Below are few examples demonstrating complexity calculation by analysing the algorithm.

Example 1: Complexity calculation of a time function

Let, $f(n)=3n^{2}+2n+6$, where, $f(n)$ represents the time it takes for an algorithm to solve the problem of size $n$.

Function $f(n)$ is a sum of three terms $3n^{2},2n,6$. The term with the highest growth rate is $3n^{2}$(product of coefficient $3$ and $n^{2}$) and the growth does not depend on coefficient $3$. Thus omitting the coefficient, results in simplified term $n^{2}$.

Thus we can say that $f(n)=O(n^{2})$

Above result can be verified using the formal definition: $f(n)=O(t(n))$, if $f(n)\leq c.t(n)$, for all $n\geq d$, where c and d are positive integers.

Let's consider a function $t(n)=n^{2}$, and $c=11$, $d=1$

Then, $3n^{2}+2n+6\leq 11.n^{2}$, for all $n\geq 1$

Example 2: Complexity calculation for sorting algorithm

Let’s consider an array in descending order (6,4,3,1). We will use different sorting methods and compute the complexity for each algorithm. Since the array is in descending order, it will be the worst-case for the algorithms in terms of number of operations or time. Based on the number of operations performed or the time taken by each algorithm, we can calculate the worst-case complexity and determine the best suited algorithm for this sorting.

Bubble Sort Complexity Analysis

Bubble sort starts at the beginning of the array, compares each adjacent pair, and swaps their position if they are not in order. The algorithm stops when there is no more swap required during the pass, indicating that the array is now sorted. For the array (6,4,3,1), below operations will take place.

First Pass,

(6 4 3 1) → (4 6 3 1), here 6 and 4 were swapped

(4 6 3 1) → (4 3 6 1), here 6 and 3 were swapped

(4 3 6 1) → (4 3 1 6), here 6 and 1 were swapped

Second Pass,

(4 3 1 6) → (3 4 1 6), here 4 and 3 swapped

(3 4 1 6) → (3 1 4 6), here 4 and 1 swapped

(3 1 4 6) → (3 1 4 6), no swapping as 4 and 6 are already in the correct order

Third Pass,

(3 1 4 6) → (1 3 4 6), here 1 and 3 swapped

(1 3 4 6) → (1 3 4 6), no swapping as 2 and 4 are in correct order

(1 3 4 6) → (1 3 4 6), no swapping as 4 and 6 are in correct order

Fourth Pass,

(1 3 4 6) → (1 3 4 6), no swapping as 1 and 3 are in correct order

(1 3 4 6) → (1 3 4 6), no swapping as 3 and 4 are in correct order

(1 3 4 6) → (1 3 4 6), no swapping as 4 and 6 are in correct order

Even though the array was already sorted after third pass, the stopping condition of no swaps during the pass was not met. Hence the algorithm made the fourth pass through all the array elements, comparing the adjacent pairs. Since, no swaps were made during fourth pass, algorithm stopped.

Summarizing the operations performed by the algorithm for the array size of 4, during each pass;

• First Pass  : 3 comparisons; 3 swaps
• Second Pass: 3 comparisons; 2 swaps
• Third Pass  : 3 comparisons; 1 swap
• Fourth Pass  : 3 comparisons; 0 swap

By generalising above scenario and considering n as the input size, we can compute the complexity for Bubble Sort;

• Number of passes required to sort the array: $n-1$
• Number of swaps in each pass: $(n-1),(n-2),...,2,1$
• Using formula for sum of $(n-1)$ natural numbers, we get total swaps = $(n-1)*((n-1)+1)/2=n*(n-1)/2=n^{2}/2-n/2$
• Omitting the lower terms and coefficients of the highest growth term $n^{2}/2$, we get worst time complexity of $O(n^{2})$

Merge Sort Complexity Analysis

Merge sort divides the unsorted list in n sub-lists, each sub-list contains one element and is considered sorted. Then it repeatedly merges the sub-lists to produce new sorted sub-lists until there is only one sub-list remaining. This last sub-list will be the sorted array. Sorting of the array (6,4,3,1) will be as per the diagram below

Merge Sort

Summarizing the operations performed by the algorithm for an array of size 4

• Number of splits to reach level 3 where each sub-list contains single element: 3
• Number of merges to reach the final sorted list: 3
• Number of comparisons made: 2 comparisons at first merging level, 3 comparisons at second merging level

By generalising above scenario and considering n as the input size, we can compute the complexity for merge sort;

• Number of splits to reach level k where each sub-list contains single element is ($n-1$)
• Number of merges required from level k to reach final sorted list is ($n-1$)
• Assuming $(n-1)\approx n$, when $n$ is very large
• Number of comparisons made at each level is $n$
• Number of levels, $k=log_{2}n$ (as $n=2^{k}$)
• Therefore, total computation performed is equal to summation of total number of splits and total number of merges with comparisons
• Total computations are $(n-1)+n.log_{2}n$
• Omitting the lower growth terms, we get worst case complexity of $O(n.log_{2}n)$

Complexity Comparison

Complexity for Bubble sort is $O(n^{2})$,

Complexity of Merge sort is $O(n.log_{2}n)$,

Since, $O(n.log_{2}n)$ < $O(n^{2})$, Merge Sort is more efficient algorithm for sorting this array.

Applications

Computational complexity helps developers to determine the most efficient algorithm that consumes least number of resources (time and space). It is widely used in various fields. Some of the major application areas are discussed below.

Computer science and algorithm efficiency analysis

It helps in designing efficient algorithms and aids in software/hardware selection. It enables comparison of multiple algorithms that can be used for solving a specific problem. Computational complexity helps in optimizing database queries and information retrieval processes.

Quantum Computing

Quantum computing is majorly used for cybersecurity, artificial intelligence, data management, data analytics and optimization. With the help of quantum complexity theory, quantum algorithms can be made exponentially efficient compared to classical algorithms. This enables businesses to solve problems which cannot be solved by conventional computing devices.

Game Theory

Game theory deals with optimal decision making in any strategic setting. It is extensively used for analysing customer behaviour, market trend, develop effective business strategies, competitive product pricing etc. Computational complexity helps in defining the most efficient algorithm for approaching the game and to come up with best outcome while considering all the possible variations.

Healthcare system

Various aspects of healthcare system such as patient management, continuity of patient care, nursing etc requires efficient decision-making processes. These complex tasks are broken down into fairly smaller and manageable tasks using computational models while giving due consideration to interrelatedness of systems.

Conclusion

Computational complexity provides a framework for analysing, comparing and selecting the most efficient algorithm and also enables the selection of best suited hardware and software combination. The algorithms or problems are categorised in classes P, NP, NP-Complete, NP-Hard and the worst-case complexity is represented using Big-O notation. Computational complexity has been evolving continuously for past few decades and remains a major area of study. Computational complexity is extensively used in the field of game theory, quantum mechanics, health system where the number of computations are huge and cannot be solved using conventional methods.