Introduction

The final assignment of this course is a competition. The challenge is to build a classification model for a given dataset, and the most accurate model wins. You are given a labeled dataset on which you can try out several algorithms, and an unlabeled dataset for which you are asked to provide the labels (predictions) using your model.

You participate in this challenge by yourself. You can ask questions to classmates, but the final process and report should be done individually. Your submitted solution will count for 2 out of 10 points on your final grade (the remaining 8 points are given based on the written exam). Since this is a competition, the best solutions will be rewarded: the solutions will be ranked on predictive accuracy, and the top 30% of submissions will gain an additional bonus point.

Experimenting

The labeled dataset can be found here. Use it to preprocess the data, select algorithms, optimize parameters and build models, using the WEKA Explorer or Experimenter. Note that this a rather large dataset, and some classifiers may require a lot of memory. Therefore, it is good to start WEKA with additional memory, e.g., using 'java -Xmx1000M -jar weka.jar'.

Submitting your predictions

The unlabeled dataset can be found here. When you have done all preprocessing and have selected your classifier and parameter settings, you should use the generated model to generate predictions for this unlabeled dataset. Guidelines can be found here. For instance, you can do the following:

• Use the Explorer to build a model on the labeled dataset. You will get performance results as usual.
• Left, under 'Supplied test set', set the unlabeled dataset.
• Under 'More Options', make sure that 'Output predictions' is checked. If you use WEKA 3.7, select CSV as output format.
• In the result list (bottom left), right-click your model and choose 'Re-evaluate model on current test set'.
• In the output window (right), you will find the predictions under the header '=== Predictions on user test set ==='

Using this method, you will get an output that looks like this:

These are the instance number, actual label (unknown), the prediction (pos or neg), the error (unknown) and the probability of each prediction. If you use WEKA 3.7, the output can be slightly different.

Finally, send in the entire prediction list.

Timing

• Start: November 4, 2013
• Questions and answers: November 11, 2013
• Deadline: December 2, 2013
• Results published: Mid December, 2013

What to hand in

• Your predictions, see above.
• A report (2-4 pages) describing the process, the classifiers tried, the pre-processing tried, and the final choices that led to you prediction.

Both can be sent to jvrijn@liacs.nl. The predictions and the report together will count for 2/10 of your final grade, and the top 30% most accurate predictions will gain an additional bonus point. In case of a tie the report counts as a tie breaker.

Questions

If you have questions about the assignment, you can contact the course lecturer. Also, there will be a question and answer session on 11-11-13.