!newssinet!sinetnews!eclnews!nd-tk001.ocn.ad.jp!nntp-egw.ocn.ad.jp!newssvt04.tk!newssvt05.tk!news-stock.gsl.net!news-peer.gsl.net!news.gsl.net!gip.net!cpk-news-hub1.bbnplanet.com!news.bbnplanet.com!howland.erols.net!cs.utexas.edu!utnut!kwon!watserv3.uwat
erloo.ca!undergrad.math.uwaterloo.ca!alopez-o
Subject: Comp.theory FAQ
Date: Mon, 8 Sep 1997 18:05:27 GMT
Nntp-Posting-Host: daisy.uwaterloo.ca

Archive-Name: theory-faq-intro
Version: 0.5

http://daisy.uwaterloo.ca:80/~alopez-o/comp-faq/faq.html



Comp.Theory FAQ

---------------------------------------------------------------------------

Table of Contents

  1. Introduction
  2. Purely Theoretical Models of Computation
  3. Theoretical Models of More Practical Importance
  4. Kolmogorov Complexity
  5. Sorting and related topics
  6. Permutations and Random number generators
  7. P vs NP
  8. Complexity Theory
  9. Logic in Computer Science
 10. Open Problems.
 11. Electronic Resources
 12. Bibliography

---------------------------------------------------------------------------

  1. Introduction

     History of (Theoretical) Computer Science

          "[W]e so readily assume that discovering, like seeing or
          touching, should be unequivocally attributable to an
          individual and to a moment in time. But the latter
          attribution is always impossible, and the former often is as
          well. [...] discovering [...] involves recognizing both that
          something is and what it is."
          -- Thomas S. Kuhn, The Structure of Scientific Revolutions,
          2nd Ed, 1970.

     In the same manner, there is no particular time or person who should
     be credited with the discovery or creation of theoretical computer
     science. However, there are important steps along the way towards the
     consolidation of the systematic resolution of problems as a science.

     Timeline:

     EARLY ALGORITHMS

     300 B.C. Greatest Common Divisor Algorithm proposed by Euclid.

     250 B.C. Sieve of Eratosthenes for prime numbers.

     Compass and straight edge (ruler) constructions

     EARLY ANALYSIS

     1910 H. C. Pocklington describes the complexity of an algorithm as
     polynomial on the number of bits.

     COMPUTABILITY

     1900 David Hilbert proposes the famous tenth problem asking for a
     general procedure to solve diophantine equations (polynomial equations
     with integer unknowns), setting the background for a formal definition
     of computable and computability.

     LOGIC FOUNDATIONS

     1920-30 Post proposes a simple unary model of computation known as the
     Post machine.

     1930-35 Goedel proves that any set of axioms containing the axioms of
     integer numbers has undecidable propositions.

     1930 Alonzo Church introduces lambda-calculus.

     1936 Alan Turing publishes the seminal paper where he presents the
     Turing Machine, a formal model of computability which is also
     physically realizable.

     References:

     E. Bach and J. Shallit. Algorithmic Number Theory : Efficient
     Algorithms. MIT Press, 1996.

  2. Purely Theoretical Models of Computation
        o The Turing Machine
        o Other equivalent models
          Lambda Calculus, Post Machines.
        o Weaker Models
          DFA's, NFA's. PDAs. CFLs.
        o Universal Turing Machine and Church's Thesis
        o Variations on the Turing Machine
          NDTMs, ATMs. Oracles.
  3. Theoretical Models of More Practical Importance
        o RAM
        o PRAM and other models of parallel computation (NC, AC).
        o Other Models (DNA, Quantum Computers)
        o Circuits
  4. Kolmogorov Complexity Motivation

     Consider the following sequences of coin tosses:

     1) head, head, tail, head, tail, tail, tail, head, head

     2) head, tail, head, tail, head, tail, head, tail, head

     3) tail, tail, tail, tail, tail, tail, tail, tail, tail

     Now, if you had bet a hunderd dollars on heads, it is likely that you
     would see outcomes (2) and (3) with suspicion, due to their
     regularity.

     However, standard probability theory argues that each of the three
     outcomes above is equally (un)likely, and thus there is no reason why
     you should complain.

     In the same manner, if the sequences above had been generated by a
     pseudo-random generator for, say a Monte Carlo algorithm, you would
     consider (2) and (3) to be fairly poor sequences.

     In other words, we have an intuitive notion of randomness --applicable
     to outcomes of random trials-- which is not properly captured by
     classic probability.

     Definition

     In words of A.N. Kolmogorov:

          In everyday language we call random those phenomena where we
          cannot find a regularity allowing us to predict precisely
          their results. gernerally speaking, there is no ground to
          believe that random phenomena should possess any definite
          probability. Therefore, we should distinguish between
          randomness proper (as absence of any regularity) and
          stochastic randomness (which is the subject of probability
          theory). There emerges the problem of finding reasons for
          the applicability of the mathematical theory of probability
          to the real world.

     As indicated above, we tend to identify randomness with lack of
     discernable patterns, or irregularity.

     Definition A sequence of numbers is non-stochastically random if it is
     irregular.

     Further, we can narrow down the definition of irregularity.

     Definition A sequence of numbers is irregular if there is no
     discernible pattern in it.

     While it might seem that this is not much progress, we are now in fact
     very close to a formal definition.

     There three key observations to be made:

       1. Some sequences seem more regular than others, which suggests the
          need for a measure of randomness (as opposed to a strict yes/no
          criterion).
       2. We can use Church's thesis to define "discernible" as anything
          that can be computed or discovered by a computer, such as a
          Turing Machine.
       3. A "pattern" is anything that allows us to make a succint
          description of a sequence.

     For example, a sequence such as 00000000000000000000000000000000 can
     be described by a short program such as

      for i=1,30 print 0

     while a sequence with no pattern can only be described by itself (or
     some equally long sequence), e.g. 01001101110111011000001011

      print 01001101110111011000001011

     Definition The Kolmogorov Complexity K(x)of a string x is the length
     of the shortest program that on outputs it.

     One can actually show that the choice of programming language affects
     the Kolmogorov Complexity by at most a constant constant additive
     factor (i.e. such factor does not depend on x) for all but a finite
     number of strings x.

     Definition A string is said to be incompressible (i.e. random or
     irregular) if
     K(x) > length(x)+constant

     Reference:

     Li, M and Vitanyi, P. An Introduction to Kolmogorov Complexity and its
     applications. New York, Springer-Verlag, 1993.

  5. Sorting and related topics

     (a) What is the fastest sort?

     The answer to this question depends on (i) what you are sorting and
     (ii) which tools you have.

     To be more precise, the parameters for (i) are the (a) size of the
     universe from which you select the elements to be sorted, (b) the
     number of elements to be sorted, (c) whether the comparison function
     can be applied over parts of the keys, (d) information on the
     distribution of the input.

     For (ii) we have the amount of available primary and secondary storage
     as a function of (i.a) and (i.b), and the power of the computational
     model (sorting network, RAM, Turing Machine, PRAM).

     To illustrate with a simple example, sorting n different numbers in
     the interval [1,n] can be done trivially in linear time on a RAM.

     A well studied case is sorting n elements from an infinite universe
     with a sequence of comparators which only accept two whole elements
     from the universe as input and produce as output the sorted pair.

     These comparators may be arranged in a predetermined manner or the
     connections can be decided at run time.

     It turns out that this apparently unrealistic setting (after all we
     sort with von Neumann RAM machines running programs which use
     arithmetic operations) models a large class of sorting algorithms for
     RAMs which are used in practice, and thus the importance of the next
     section.

     (b) n log n information bound

          Theorem. Any comparison based sorting program must use at
          least ceil(lg N!) > N lg N - N/ ln 2 comparisons for some
          input.

     The main two reasons for using this model are that (a) it is amenable
     to study and (b) it produces bounds and timings that were generally
     sufficiently close to practical applications.

     However, nowadays servers come routinely equipped with up to 1
     Gigabyte of memory. Under this configuration some methods which are
     memory intensive (such as radix sort or bucket sort) become practical.
     In fact, a method such as bucket sort on N = 10,000,000 records and
     100,000 buckets takes time 27 N. In contrast, the best comparison
     based sorting algorithms take time ~ 40 N.

     Recently, Andersson has proposed a promising algorithm that takes time
     O(n log log n) which takes the advantage of the fact that RAM
     computers can operate on many bits (usally 32 or 64) bits on a single
     instruction. You can find more information in Stefan Nilsson's Home
     Page.

          Theorem. It is possible to sort n keys each occupying L
          words in O(nL) time using indirect addressing on a RAM
          machine.

          Theorem. A set of n keys of length w = word size can be
          sorted in linear space in O(n log n/log log n) time

     Other sorts:

        o Adaptive sorting
        o Sorting Networks

     References:

     R. Sedgewick, P. Flajolet. An introduction to the anlysis of
     algorithms. Addison Wesley, 1996.

     A. Andersson. Sorting and Searching Revisited. Proceedings of the 5th
     Scandinavian Workshop on Algorithm Theory. Lecture Notes in Computer
     Science 1097. Springer-Verlag, 1996.

     Electronic Resources:

     Sedgewick's Shell Sort home page.

     Pat Morin's sorting Java Applets

  6. Permutations and Random number generators The pLab page on the "Theory
     and Practice of Random Number Generation" is a very good start

  7. P vs NP

     Historical Context

     If one goes back a couple of hundred years, we can see that the
     historical motivation for the study of complexity of algorithms is the
     desire to identify, under a formal framework, those problems that can
     be solved "fast".

     To achieve this, we need to formally define what we mean by "problem",
     "solve" and "fast".

     Let's postpone the issue of what "problem" and "solve" is by
     restricting ourselves to well-defined mathematical problems such as
     addition, multiplication, and factorization.

     One of the first observations that can be made then, is that even some
     "simple" problems may take a long time if the question is long enough.
     For example, computing the product of two numbers seems like a fast
     enough problem, Nevertheless one can easily produce large enough
     numbers that would bog down a fast computer for a few minutes.

     We can conclude then that we must consider the size of the "question"
     as a parameter for time complexity. Using this criterion, we can
     observe that constant time answers as a function of the size of the
     question are fast and exponential time are not. But what about all the
     problems that might lie in between?

     It turns out that even though digital computers have only been around
     for fifty years, people have been trying for at least thrice that long
     to come up with a good definition of "fast". (For example, Jeff
     Shallit from the University of Waterloo, has collected an impressive
     list of historical references of mathematicians discussing time
     complexity, particularly as it relates to Algorithmic Number Theory).

     As people gained more experience with computing devices, it became
     apparent that polynomial time algorithms were fast, and that
     exponential time were not.

     In 1965, Jack Edmonds in his article Paths, trees, and flowers
     proposed that "polynomial time on the length of the input" be adopted
     as a working definition of "fast".

     So we have thus defined the class of problems that could be solved
     "fast", i.e. in polynomial time. This class is commonly denoted by P.

     By the late 1960s it had become clear that there were some seemingly
     simple problems that resisted polynomial time algorithmic solutions.
     In an attempt to classify this family of problems, Steve Cook came up
     with a very clever observation: for a problem to be solved in
     polynomial time, one should be able --at the very least-- to verify a
     given correct solution in polynomial time. This is called certifying a
     solution in polynomial time.

     Because, you see, if we can solve a problem in polynomial time and
     somebody comes up with a proposed solution S, we can always rerun the
     program, obtain the correct solution C and compare the two, all in
     polynomial time.

     Thus the class NP of problems for which one can verify the solution in
     polynomial time was born. Cook also showed that among all NP problems
     there were some that were the hardest of them all, in the sense that
     if you could solve any one of those in polynomial time, then it
     followed that all NP problems can be solved in polynomial time. This
     fact is known as Cook's theorem, and the class of those "hardest"
     problems in NP is known as NP-complete problems. This result was
     independently discovered by Leonid Levin and published in the USSR at
     about the same time.

     A year later, Richard Karp showed that some very interesting problems
     that had eluded polynomial time solutions could be shown to be
     NP-complete, and in this sense, while hard, they were not beyond hope.
     This list grew quite rapidly as others contributed, and it now
     includes many naturally occuring problems for which we do not know yet
     how to solve in polynomial time.

     References

     S. Cook.
     J. Edmonds. Paths, trees, and flowers.
     M. R. Garey, D. S. Johnson. Computers and Intractability, W.H.Freeman
     &Co, 1979.
     L. Levin. Universal Search Problems, Probl.Pered.Inf. 9(3), 1973. A
     translation appears in B. A. Trakhtenbrot. A survey of Russian
     approaches to Perebor (brute-force search) algorithms, Annals of the
     History of Computing, 6(4):384-400, 1984

     (a) What is NP?

     First we need to define formally what we mean by a problem. Typically
     a problem consists of a question and an answer. Moreover we group
     problems by general similarities.

     Again using the multiplication example, we define that a
     multiplication problem as a pair of numbers, and the answer is their
     product. An instance of the multiplication problem is a specific pair
     of numbers, to be multiplied.

     Problem. Multiplication
     Input. A pair of numbers x and y
     Output. The product x times y

     A clever observation is that we can convert a multiplication problem
     into a yes/no answer by joining together the original question and the
     answer and asking if they form a correct pair. In the case of
     multiplication, we can convert a question like
     4 x 9 = ??

     into a yes/no statement such as

     "is it true that 4 x 9 = 36?" (yes), or
     "is it true that 5 x 7 = 48?" (no).

     In general we can apply this technique to most (if not all) problems,
     simplifying formal treatment of problems.

     Definition A decision problem is a language L of strings over an
     alphabet. A particular instance of the problem is a question of the
     form "is x in L?" where x is a string. The answer is yes or no.

     (b) NP complete list

     Pierluigi Crescenzi and Viggo Kann mantain a good list of NP
     optimization problems

     (d) Other complete problems (PSPACE, P).

  8. Complexity Theory

     (a) Lower Bounds

     (b) YACC (Yet Another Complexity Class)

  9. Logic in Computer Science

 10. Open Problems.

     The are several important open problems within theoretical computer
     science. Among them

     P =? NP

     AC != P

     Find RAM problem with time complexity T(n) = \omega(n log n). T(n) =
     O(n^k).

     Show that sorting is n log n on a RAM with constant word size.

     Find exact time complexity of prime decomposition.

 11. Electronic Resources
        o SIGACT Home Page
        o Fundamentals of Computing By Leonid Levin.
        o Computer Science Bibliography Collection
        o Average-Case Complexity Forum
        o The Steiner Tree Web Page
        o The Alonzo Church Archive
        o Theoretical Computer Science On The Web
        o Electronic Colloqium on Computational Complexity
          http://www.eccc.uni-trier.de/eccc/index.html
        o An Introduction to the Theory of Computation by Eitan Gurari,
          Ohio State University Computer Science Press, 1989, ISBN
          0-7167-8182-4
        o Online search systems for CS publications:
             + Bibnet
             + Technical Report Research Service
             + Unified Computer Science Index Technical Reports Library
             + The Hypertext Bibliography Project
             + On-line CS Techreports
             + The New Zealand Digital Library
             + Computer Science Technical Reports Archive Sites
             + Networked Computer Science Technical Reports Library
             + Computer Science Bibliography Glimpse Server

 12. Bibliography

     Among the truly few FAQs in this newsgroup are recommendations for a
     Data Structures book and a Complexity Theory book. Here are some of
     titles in brackets I've added the number of e-mail recommendations
     that I get plus any comments.

     Data Structures and Analysis of Algorithms

     Baase, Sara. Computer algorithms : introduction to design and analysis
     of algorithms. 2nd ed. Addison-Wesley Pub. Co., c1988.

     Flajolet, Philippe and Sedgewick, Robert. An introduction to the
     analysis of algorithms. Addison-Wesley, c1996.

     Lewis, H and Denenberg, L. Data Structures and their Algorithms.
     Harper-Collins, 1991.

     Cormen, Thomas; Leiserson, Charles; Rivest, Ronald. Introduction to
     algorithms. MIT Press, 1989.

     Goodman and Hedetniemi. Introduction to the Design and Analysis of
     Algorithms. McGraw-Hill.

     Hopcroft and Ullman. Introduction to Authomata Theory, Languages and
     Computation. Addison-Wesley.

     All this books has excelent topics on computation and computational
     complexity.

     Complexity Theory

     Papadimitriou, Christos H. Computational complexity. Addison-Wesley,
     c1994.

     D.Bovet and P. Crescenzi. Introduction to the Theory of Complexity.
     Prentice Hall.

     C.-K. Yap: "Theory of Complexity Classes". Via FTP. General Interest

     Hofstadter, D. Escher, Godel and Bach: An Eternal Golden Braid,
     Penguin Books.



---
Alex Lopez-Ortiz                               alopez-o@daisy.UWaterloo.ca
http://daisy.uwaterloo.ca/~alopez- o                    Research Scientist
Open Text Corp.            185 Columbia St W Waterloo, Ont N2L 5Z5  Canada
-- 
Alex Lopez-Ortiz                               alopez-o@daisy.UWaterloo.ca
http://daisy.uwaterloo.ca/~alopez-o                     Research Scientist
Open Text Corp.            185 Columbia St W Waterloo, Ont N2L 5Z5  Canada
