CSP Bencmark Repository: Call for Benchmarks
       ----------------------------------------------

Recently, a significant effort was made in developing heuristics for
solving various problems using Constraint Satisfaction Problem (CSP)
formulations. Furthermore, it was suggested that because various
heuristics may be applicable to different real-world problem
distributions, one should try to develop an expert CSP solver that
determines which heuristic is preferable in a given domain. It is
clear that common evaluation benchmarks are needed, especially in the
domain of scheduling.

As a further motivation, we would like to mention the Machine Learning
Repository here, at UCI, which is widely used for evaluating machine
learning systems. It has proven to be very successful and has made a
significant contribution to the machine learning community.

The purpose of this message is to direct the attention of people
working in related fields to the need for a CSP benchmark repository.
In particular we are interested in:

1. Establishing a common language with which problems can be
   described. This will allow numerous algorithms to be compared
   on a large common set of problem distributions.

2. Characterizing attributes of benchmarks that affect performance of
   algorithms such as measures of size, variance of difficulty etc.
   Successful characterization will allow prediction of performance of
   algorithms in real-world domains based on experiments made on
   artificially generated problems.

3. Collecting real-world problems from industrial environments.

4. Collecting generators people use to evaluate their algorithms.

We have been looking at two approaches for developing a common
language: 1) ALICE (AIJ 10:29-127, 1978), 2) Constraint Logic
Programming (CLP) languages. The major concern here is not to be
dragged into expressivness and semantical issues. Whichever language
is chosen, only a subset of it that is expressive enough for our
purposes, should be used. This subset can be obtained either by
restricting its syntax or by other approaches.

Anyone who is interested, please send opinions, suggestions and (of
course) benchmarks to eschwalb@ics.uci.edu. The replies will
be summarized and posted.