TSE Querier

Goals

• to introduce the TSE Querier
• to learn another form of testing: fuzz testing
• to learn about expressions and operator precedence

the Querier

The third component of the Tiny Search Engine is the Querier, which reads the index produced by the Indexer and the page files produced by the Crawler, to interactively answer written queries entered by the user.

Our Querier loads the index into memory (a data structure we developed for the Indexer) and then prompts the user for queries. Queries are comprised of words, with optional and/or operators. For example,

computer science
computer and science
computer or science
baseball or basketball or ultimate frisbee

The first two examples are treated identically, matching only documents that have both words - not necessarily together (as in the phrase “computer science”). The third picks up documents that have either word. The fourth matches documents that mention baseball, or basketball, or both “ultimate” and the word “frisbee” (not necessarily together).

Here’s an example run with the provided crawled pages and index file with the letters seed URL at depth 6. The TSE output is also available at the plank server under /thayerfs/courses/22spring/cosc050/cs50tse/tse-output. These pages were crawled on CS servers with the old-www domain, so the output URLs start with old-www.

$ tse_dir=/thayerfs/courses/22spring/cosc050/cs50tse/tse-output
$ ./querier $tse_dir/letters-depth-6/ $tse_dir/letters-index-6 
Query? first and search
Query: first and search 
Matches 2 documents (ranked):
score   1 doc   3: http://old-www.cs.dartmouth.edu/~cs50/data/tse/letters/B.html
score   1 doc   8: http://old-www.cs.dartmouth.edu/~cs50/data/tse/letters/D.html
-----------------------------------------------
Query? tiny search engine
Query: tiny search engine 
No documents match.
-----------------------------------------------
Query? NOTE we LOWERcase the query first
Query: note we lowercase the query first 
No documents match.
-----------------------------------------------
Query? spaces      do    not    matter
Query: spaces do not matter 
No documents match.
-----------------------------------------------
Query? non-letter characters are disallowed
Error: bad character '-' in query.
Query? even digits as in cs50
Error: bad character '5' in query.
Query? and
Query: and 
Error: 'and' cannot be first
Query? or
Query: or 
Error: 'or' cannot be first
Query? what about and
Query: what about and 
Error: 'and' cannot be last
Query? what about or
Query: what about or 
Error: 'or' cannot be last
Query? ^D

Let’s study the Requirements Spec (located under querier/ directory of your tse repo) for the Querier, and run some demos.

Fuzz Testing

In a recent lecture we talked about unit testing, and the difference between glass-box testing and black-box testing. Usually, these tests are based on a carefully constructed series of test cases, devised to test all code sequences and push on the “edge cases”.

However, such tests are only as good as the test writer - who must logically study the code (for glass-box testing) or the specs (for black-box testing) to think of the suitable test cases. It’s possible they will miss some important cases.

Another solution, therefore, is fuzz testing, a form of black-box testing in which you fire thousands of random inputs at the program to see how it reacts. The chances of triggering an unconsidered test case is far greater if you try a lot of cases!

Here is a fuzz-testing program for our querier. It generates a series of random queries on stdout, which it then pipes to the querier on stdin. Here’s the core of the fuzz tester:

/**************** generate_query ****************/
/* generate one random query and print to stdout.
 * pull random words from the wordlist and from the dictionary.
 */
static void
generate_query(const wordlist_t *wordlist, const wordlist_t *dictionary)
{
  // some parameters that affect query generation
  const int max_words = 6;        // generate 1..max_words
  const float or_probability = 0.3;   // P(OR between two words)
  const float and_probability = 0.2;  // P(AND between two words)
  const float dict_probability = 0.2; // P(draw from dict instead of wordlist)

  int qwords = random() % max_words + 1; // number of words in query
  for (int qw = 0; qw < qwords; qw++) {
    // draw a word either dictionary or wordlist
    if ((random() % 100) < (dict_probability * 100)) {
      printf("%s ", dictionary->words[random() % dictionary->nwords]);
    } else {
      printf("%s ", wordlist->words[random() % wordlist->nwords]);
    }

    // last word?
    if (qw < qwords-1) {
      // which operator to print?
      int op = random() % 100;
      if (op < (and_probability * 100)) {
        printf("AND ");
      }
      else if (op < (and_probability * 100 + or_probability * 100)) {
        printf("OR ");
      }
    }
  }
  printf("\n");
}

With the following setup,

$ cd tse
$ mkdir data/letters/
$ crawler/crawler http://cs50tse.cs.dartmouth.edu/tse/letters/index.html data/letters 3
$ indexer/indexer data/letters data/index.letters

And here’s the output of 10 random queries:

$ querier/fuzzquery 
usage: querier/fuzzquery indexFile numQueries randomSeed
$ querier/fuzzquery data/index.letters 10 0
querier/fuzzquery: generating 10 queries from 15 words
search AND biology 
tse OR for breadth OR the first 
tse this OR biology 
twice-damaged search page OR home OR world-paralyzing OR computational 
breadth OR eniac this page AND computational OR eniac 
algorithm playground computational computational OR for 
breadth 
home depth OR first 
for 
tse OR biology OR this AND this AND for 

And here’s what happens when we pipe it to our querier (output abbreviated a little):

$ querier/fuzzquery data/index.letters 10 0 | querier/querier data/letters data/index.letters
querier/fuzzquery: generating 10 queries from 15 words
Query: search and biology 
No documents match.
-----------------------------------------------
Query: tse or for breadth or the first 
Matches 2 documents (ranked):
score   1 doc   1: http://cs50tse.cs.dartmouth.edu/tse/letters/index.html
score   1 doc   3: http://cs50tse.cs.dartmouth.edu/tse/letters/B.html
-----------------------------------------------
Query: tse this or biology 
Matches 2 documents (ranked):
score   1 doc   1: http://cs50tse.cs.dartmouth.edu/tse/letters/index.html
score   1 doc   6: http://cs50tse.cs.dartmouth.edu/tse/letters/C.html
-----------------------------------------------
Query: timestamp search page or home or faraday or computational 
Matches 6 documents (ranked):
score   2 doc   1: http://cs50tse.cs.dartmouth.edu/tse/letters/index.html
score   2 doc   6: http://cs50tse.cs.dartmouth.edu/tse/letters/C.html
score   1 doc   2: http://cs50tse.cs.dartmouth.edu/tse/letters/A.html
score   1 doc   3: http://cs50tse.cs.dartmouth.edu/tse/letters/B.html
score   1 doc   4: http://cs50tse.cs.dartmouth.edu/tse/letters/E.html
score   1 doc   5: http://cs50tse.cs.dartmouth.edu/tse/letters/D.html
-----------------------------------------------
Query: breadth or eniac this page and computational or eniac 
Matches 2 documents (ranked):
score   1 doc   3: http://cs50tse.cs.dartmouth.edu/tse/letters/B.html
score   1 doc   4: http://cs50tse.cs.dartmouth.edu/tse/letters/E.html
-----------------------------------------------
Query: algorithm playground computational computational or for 
Matches 6 documents (ranked):
score   1 doc   2: http://cs50tse.cs.dartmouth.edu/tse/letters/A.html
score   1 doc   1: http://cs50tse.cs.dartmouth.edu/tse/letters/index.html
score   1 doc   3: http://cs50tse.cs.dartmouth.edu/tse/letters/B.html
score   1 doc   4: http://cs50tse.cs.dartmouth.edu/tse/letters/E.html
score   1 doc   5: http://cs50tse.cs.dartmouth.edu/tse/letters/D.html
score   1 doc   6: http://cs50tse.cs.dartmouth.edu/tse/letters/C.html
-----------------------------------------------
Query: breadth 
Matches 1 documents (ranked):
score   1 doc   3: http://cs50tse.cs.dartmouth.edu/tse/letters/B.html
-----------------------------------------------
Query: home depth or first 
Matches 2 documents (ranked):
score   2 doc   5: http://cs50tse.cs.dartmouth.edu/tse/letters/D.html
score   1 doc   3: http://cs50tse.cs.dartmouth.edu/tse/letters/B.html
-----------------------------------------------
Query: for 
Matches 6 documents (ranked):
score   1 doc   1: http://cs50tse.cs.dartmouth.edu/tse/letters/index.html
score   1 doc   2: http://cs50tse.cs.dartmouth.edu/tse/letters/A.html
score   1 doc   3: http://cs50tse.cs.dartmouth.edu/tse/letters/B.html
score   1 doc   4: http://cs50tse.cs.dartmouth.edu/tse/letters/E.html
score   1 doc   5: http://cs50tse.cs.dartmouth.edu/tse/letters/D.html
score   1 doc   6: http://cs50tse.cs.dartmouth.edu/tse/letters/C.html
-----------------------------------------------
Query: tse or biology or this and this and for 
Matches 2 documents (ranked):
score   2 doc   1: http://cs50tse.cs.dartmouth.edu/tse/letters/index.html
score   1 doc   6: http://cs50tse.cs.dartmouth.edu/tse/letters/C.html
-----------------------------------------------

We could generate a different series of random queries by changing the random seed, and we can run a lot more queries, too!

The fuzz tester does not test all aspects of the querier; in particular, it will not generate syntactically incorrect inputs. Those should be tested by another program, perhaps another fuzz tester. Furthermore, it does not verify whether the querier actually produces the right answers!

For regression testing, we might save the querier output in a file, and then compare the output of a fresh test run against the saved results from earlier runs. If we had earlier believed those results to be correct, then seeing unchanged output would presumably indicate the results (and thus the new code) are still correct.

Expressions and accumulators

Thinking ahead to the querier, let’s think about how one evaluates an expression involving operators. We’ll work with an arithmetic analogy.

Arithmetic expressions

Consider the following arithmetic expression:

sum = a + b + c + d

Since addition is a left-associative operator, this means the same thing as

sum = (((a + b) + c) + d)

This means we can scan the expression from left to right, accumulating a sum as we go, effectively like this:

sum = 0
sum = sum + a
sum = sum + b
sum = sum + c
sum = sum + d

Here, the sum acts as an accumulator. (Indeed, many early hardware architectures include an explicit register called an ‘accumulator’.)

We often see this approach generalized in code:

int n = 5;
int array[n] = {42, 34, 12, -5, 19};
int sum = 0;
for (int i = 0; i < n; i++)
	sum += array[i];
printf("sum = %d; average = %f\n", sum, (float) sum / n);

Precedence

What if you have a mixture of operators, with precedence?

Consider the following arithmetic expression:

sum = a + b * c + d

Both addition and multiplication are left-associative operators, but multiplication has precedence over addition. Thus, we implicitly rewrite the above expression as follows:

sum = ((a + (b * c)) + d)

or, in sequence,

sum = 0
sum = sum + a 
prod = 1
prod = prod * b
prod = prod * c
sum = sum + prod
sum = sum + d

Notice how we ‘step aside’ from the sum for a moment while we compute the product b * c … using an exactly analogous process. prod is an accumulator for the product; it is initialized to the multiplicative identity (1) instead of the additive identity (0), for reasons I hope are obvious. But then we just multiply in each of the successive items, one at a time.

This generalizes to longer expressions like

sum = a * b + c * d * e + f + g * h * i

becomes

sum = 0
prod = 1
prod = prod * a
prod = prod * b
sum = sum + prod
prod = 1
prod = prod * c
prod = prod * d
prod = prod * e
sum = sum + prod
prod = 1
prod = prod * f
sum = sum + prod
prod = 1
prod = prod * g
prod = prod * h
prod = prod * i
sum = sum + prod

Let’s add some indentation to make this a little easier to read:

sum = 0
	prod = 1
	prod = prod * a
	prod = prod * b
sum = sum + prod
	prod = 1
	prod = prod * c
	prod = prod * d
	prod = prod * e
sum = sum + prod
	prod = 1
	prod = prod * f
sum = sum + prod
	prod = 1
	prod = prod * g
	prod = prod * h
	prod = prod * i
sum = sum + prod

Notice what I did with f, and that I never add anything to sum other than prod.

This structure should give you a hint about how you might write code to evaluate such expressions… if you have a product function to scan the expression left to right from a given starting point, accumulating a product of individual items until it sees a + or the end of the expression, you can then write a function sum that scans the expression left to right from the start, accumulating a sum of products by calling product at the start and after each +.

Activity

In today’s activity your group discusses how you would parse a query while dealing with the precedence of operators – it’s very analogous to dealing with expressions described earlier.