Caltech Challenge 2007

Instructions | Development Kit | Caltech-256 Dataset

This challenge explores visual recognition algorithm performance over broad statistics of image categories. You will train on Caltech-256 images, and test on both Caltech-256 and a fresh set of images. We hope to identify 'difficult' categories and determine where different algorithm's strengths lie.

Final Results (Updated: 10/14/2007) 

View the final results here!

Update (Updated: 10/2/2007) 

We are no longer requiring contestants to re-run their results on a new set of test image - the existing challenge seems to already be using up everybody's available CPU time! If you are running short of time, just submit as many confusion matrices as you can. Your results will still be posted.

Any and all submissions accepted - but please try to submit within 24 hours of the workshop, so that I have time to update the web page.
-Greg

Registration (Updated: 9/6/2007)

Please register by sending mail to: Greg Griffin. Apologies for delayed replies: a response should come within 24 hours.

Schedule (Updated: 10/1/2007)

  • Oct 15 2007 - Workshop: results presented here
  • Oct 14 2007 - Final results sent to us
  • Sept 30 2007 - Preliminary results sent to us.   DEADLINE EXTENDED
  • Sept 7-29 2007 - Participants register for the competition DEADLINE EXTENDED
  • May 5 2007 - Development kit published

    • Challenge Description

    You will perform a 1-out-of-256 classification task.

    The Challenge scores performance based on three standards:
    Benchmark Description
    1. Standard Correctly classified fraction of images (i.e. the mean of the confusion matrix diagonal).
    2. Originality Rewards algorithms that are good on categories that other algorithms found difficult.
    3. Similarity Confusing a dog with a goat is less egregious than confusing a tree with a kitchen appliance.
    details on 2. and 3. will be posted here.

    How to submit

    You will send us 256x256 confusion matrices for 5, 10, 20, and 50 training examples. Instructions.

    Development Kit

    The development kit includes Matlab scripts to show images and generate training sets.