Homework Five

Due: Wednesday 06/11/2014 at 11:59 p.m.

Please follow this link to see the submission instructions.

This set of assignment is designed for you to get familiar with various evaluation mechanisms in IR.

1. Given the set of data in the table,
   1. Compute the recall/precision rate.
   2. Graph the result for the recall/precision rate.
   3. Compute the corresponding F-measure for each of the recall points (0.0 to 1.0). That is, the parameters α = 0.5 and β = 1.

   The total relevant documents in a document set is 10. The top 10 retrieved documents are listed with the ones that are relevant marked.

   Order	Doc ID	Relevant
   1	586	X
   2	357
   3	358	X
   4	108	X
   5	345
   6	114
   7	555	X
   8	888
   9	167
   10	666	X

2. Compute the Kappa statistics for the following data.

   		Judge 2 Relevance
   		Yes	No	Total
   Judge 1	Yes	200	15	215
   Relevance	No	20	50	70
   	Total	220	65	285