Remember a program or a function should do one thing and one thing well. You should start to write good readable code from the start. Break your programs into meaningful functions. Be defensive when dealing with user input. Exit gracefully and if an error has been encountered – IO failed – inform the user in a meaningul but not verbose way – remember the flip side: silence is goldern.
Defensive coding: you should always assume that the user will enter incorrect input. So how are you going to deal with it? Tips: make sure you check the number of parameters entered against what you expect and their correctness.
Modular code: beakdown your code in to meaningful functions, e.g., convertAddress(). Make it more reabale. Reusable, easier to test and debug. The functional decomposition of crawler covers: file processing, (url) list processing, dictionary processing, html processing (e.g., parsing). Make sure that the major functional (processing) decomposition is reflected in meaningful source files (e.g., dictionary.[ch]) and functions (e.g., getDataWithKey()).
Debugging trick bugs with valgrind and gdb: You encounter many different bugs while coding crawler. Use valgrind and gdb to clear bugs associated with dynamic memory management and pointers.
Force tricky bugs into the open, now swat, splat!: Change system parameters such as MAX_HASH_SLOTS to force the crawler to always operate in collision processing forcing edge cases.
Exit gracefully: clean up and inform the user, assert conditions.
There is the detailed rubric:
Correctness (1-10) [70%]
WARNING: DO NOT SUBMIT SEGFAULTED CODE:
In all cases below there is no grade given for a program that segfaults. A student will be asked to resubmit their segfaulted code. The resubmitted code will be graded out of 50% of the original grade. This is a significant penalty so make sure that you double check that your code compiles correctly and runs correctly without segfaults. Make your code is defensive to incorrect user input. NOTE, because the crawler can run for a period of time double check that you get no segfaults at DEPTH 0, 1, 2 and 3 at the SEED URL http://www.cs.dartmouth.edu/ campbell/. Check that your crawler˙test.sh script implements all the required tests.
– Is the program correct (i.e., does it work) and error free?
1. write a makefile to build crawler [6%]
– make -4%
– make clean - 1%
– make debug -1%
2. write a bash test script (crawler˙test.sh) [19%]
to test command line arguments and run crawler on the SEED directory at level 0, 1, 2 and 3. Your script should:
– Create a log file to save all output of the test script -2%
– Do a clean build (make clean followed by a make) -2%
– Do input argument testing on crawler
– TEST1: check for wrong number of arguments -2%
– TEST2: check for wrong directory -2%
– TEST3: check for depth exceeds maximum -2%
– TEST4: incorrect SEED URL -2%
In all cases exit the script if you get incorrect $? For example, if you call crawler with the wrong path you expect it to fail and give usage. You can check $? from the called crawler to determine this.
– TEST5: Run crawler at depth 0 check return status $? -1%
– TEST6: Run crawler at depth 1 check return status $? -2%
– TEST7: Run crawler at depth 2 check return status $? -2%
– TEST8: Run crawler at depth 3 check return status $? -2%
3. Your crawler should crawl the SEED URL http://www.cs.dartmouth.edu/ campbell/ [20%]
[DEPTH, FILES]
0 1 -5%
1 19 -5%
2 90 -5%
3 599 -5%
Note for 1, 2, 3 we will accept +/- 10% in each case for example 1, 19 – 17, 18, 19 will be acceptable.
4. does it meet the the crawler REQUIREMENTS. Check on each requirement, for each missing one, [5%].
5. File decomposition and modular design. [20%]
We discussed decomposition in class: crawler processing, file processing, list processing, dictionary processing. Within each of those modules or files there should be a decomposition into functions (e.g., addElement(), getDataWithKey(), fileLength(), etc). While you do not have to follow this proposed design and decomposition you have to deliver some breakdown and clustering of functions into meaningful source files.
– little file and function decomposition - 10%
– no file decomposition - 20%
Clarity (1-10) [10%]
– Is the code easy to read, well commented, use good names
– for variables and functions. In essence, is it easy to understand and use
– [K&P] Clarity makes sure that the code is easy to understand for people and machines.
– Too much mysterious variable names with comments -
Simplicity (1-10) [10%]
– Is the code as simple as possible.– [K&P] Simplicity keeps the program short and manageable
– a program or function should do one thing and one thing well.
Generality (1-10) [10%]
– Can the program easily adapt to changes or modifications.
– [K&P] Generality means the code can work well in a broad range of situations and adapt
Extra Credit [15%]
http://www.cs.dartmouth.edu/ 0 1
http://www.cs.dartmouth.edu/ 1 74
http://www.cs.dartmouth.edu/ 2 3813
http://www.cs.dartmouth.edu/ 3 8534