COLIEE-2018 CALL FOR TASK PARTICIPATION

Competition on Legal Information Extraction/Entailment (COLIEE)

COLIEE-2018 Workshop: Nov 12-13, 2018 (collocated with JURISIN 2018)
Venue: Raiosha, Hiyoshi Campus in Keio University, Yokohama, Japan

COLIEE competition Registration Due: June 30, 2018

Those who wish to use previous COLIEE data for a trial, please contact miyoung2(at)ualberta.ca .

Sponsored by
Albera Machine Intelligence Institute (AMII)
University of Alberta
National Institute of Informatics (NII)
Compass Law
Ross Intelligence
Intellicon

Download Call for Participation


As an associated event of JURISIN 2018, we are happy to announce the 5th Competition on Legal Information Extraction and Entailment (COLIEE-2018), which will include tasks on both a statue law and case law.

Four tasks are included in the 2018 competition: Tasks 1 and 2 are about case law competition, and tasks 3 and 4 are statute law competition. Task 1 is a legal case retrieval task, and it involves reading a new case Q, and extracting supporting cases S1, S2â¦, Sn from the provided case law corpus, to support the decision for Q. Task 2 is the legal case entailment task, which involves the identification of a paragraph from existing cases that entails the decision of a new case. As in previous COLIEE competitions, Task 3 is to consider a yes/no legal question Q and retrieve relevant statutes from a database of Japanese civil code statutes; Task 4 is to confirm entailment of a yes/no answer from the retrieved civil code statutes.

1. Task Description

1.1 (COLIEE Case Law Competition) Task 1:The Legal Case Retrieval Task

This legal case competition focuses on two aspects of legal information processing related to a database of predominantly Federal Court of Canada case law, provided by Compass Law.

The legal case retrieval task involves reading a new case Q, and extracting supporting cases S1, S2, ... Sn for the decision of Q from the entire case law corpus. Through the document, we will call the supporting cases for the decision of a new case 'noticed cases'.

1.2 (COLIEE Case Law Competition) Task 2: The Legal Case Entailment task

This task involves the identification of a paragraph from existing cases that entails the decision of a new case.

Given a decision Q of a new case and a relevant case R, a specific paragraph that entail the decision Q needs to be identified. We confirmed that the answer paragraph can not be identified merely by information retrieval techniques using some examples. Because the case R is a relevant case to Q, many paragraphs in R can be relevant to Q regardless of entailment.

This task requires one to identify a paragraph which entails the decision of Q, so a specific entailment method is required which compares the meaning of each paragraph in R and Q in this task.

1.3 (COLIEE Statute Law Competition) Task 3: The Statute Law Retrieval Task

The COLIEE statute law competition focuses on two aspects of legal information processing related to answering yes/no questions from Japanese legal bar exams (the relevant data sets have been translated from Japanese to English).

Task 3 of the legal question answering task involves reading a legal bar exam question Q, and extracting a subset of Japanese Civil Code Articles S1, S2,..., Sn from the entire Civil Code are those appropriate for answering the question such that

Entails(S1, S2, ..., Sn , Q) or Entails(S1, S2, ..., Sn , not Q).

Given a question Q and the entire Civil Code Articles, we have to retrieve the set of "S1, S2, ..., Sn" as the answer of this track.

1.4 (COLIEE Statute Law competition) Task 4: The Legal Question Answering Data Corpus

Task 4 of the legal question answering task involves the identification of an entailment relationship such that

Entails(S1, S2, ..., Sn , Q) or Entails(S1, S2, ..., Sn , not Q).

Given a question Q, we have to retrieve relevant articles S1, S2, ..., Sn through phase one, and then we have to determine if the relevant articles entail "Q" or "not Q". The answer of this track is binary: "YES"("Q") or "NO"("not Q").

2. Data Corpus

2.1 Case Law Competition Data Corpus(Task 1 and Task 2)

COLIEE-2018 data is drawn from an existing collection of predominantly Federal Court of Canada case law.

Participants can choose which phase they will apply for, amongst the two sub-tasks as follows:

1) Task 1: legal information retrieval task. Input is an unseen legal case Q, and output should be relevant cases in the given legal case corpus that support the decision of the input case, which are 'noticed cases'.

2) Task 2: Recognizing entailment between the decision of a new case and a relevant case. Input is a decision paragraph from an unseen case and a relevant case. Output should be a specific paragraph from the relevant case, which entails the decision of the unseen case.

2.2 Statute Law Competition Data Corpus(Task 3 and Task 4)

The corpus of legal questions is drawn from Japanese Legal Bar exams, and also all the Japanese Civil Law articles are also provided (file format and access described below).

Participants can choose which phase they will apply for, amongst the two sub-tasks as follows:

1) Task 3: legal information retrieval Task. Input is a bar exam 'Yes/No' question and output should be relevant civil law articles.

2) Task 4: Recognizing Entailment between law articles and queries. Input is a bar exam 'Yes/No' question. After retrieving relevant articles using your method, you have to determine 'Yes' or 'No' as the output.

3. Measuring the Competition Results

TBA

4. Submission details

We require participants to submit a paper on their method and experimental results to JURISIN 2018 workshop in accordance with an instruction specified at http://research.nii.ac.jp/jurisin2018/ and to present the paper at a special session of JURISIN 2018.

These papers will be reviewed in the same way of usual submission to JURISIN 2018 and extended versions could be submitted to an LNAI post-proceeding of JSAI-IsAI2018 (the name of symposium in which JURISIN 2018 is included). Selected papers will be published in the LNAI post-proceedings.

5. Schedules


   Jun 30, 2018 Task Registration Due.
Jun 30, 2018 Dry run data release.
Jul 31, 2018 Formal run data release.
Aug 15, 2018 Formal run submission due.
Aug 21, 2018 Notification of Evaluation results and Release of gold standard answers.
Aug 30, 2018 Paper submission due.
Sep 30, 2018 Notification of acceptence.
Oct 10, 2018 Camera ready papers due.
Nov 12-13, 2018 COLIEE 2018 Workshop.

6. Details of Each Task

6.1 Task 1 Details

Our goal is to explore and evaluate legal document retrieval technologies that are both effective and reliable.

The task investigates the performance of systems that search a set of case laws that support the unseen case law. The goal of the task is to return noticed cases in the given collection to a query. We call a case is ¡®noticed¡¯ to a query iff the case supports the decision of the query case. In this task, the query case does not include the decision. Because our goal is how accurately a machine can capture decision-supporting cases for a new case (with no decision yet).

Federal Court of Canada case law corpus will be provided which include around 1,000 cases. The process of executing the new query cases over the existing cases and generating the experimental runs should be entirely automatic. In training data, each query case will be given noticed cases. In test data only query cases will be included with no noticed cases.

There should be no human intervention at any stage, including modifications to your retrieval system motivated by an inspection of the test queries. You should not peek the test data before you submit your runs.

At most three runs from each group will be assessed. The submission format and evaluation methods are described below.

6.2 Task 2 Details

Our goal is to predict the decision of a new case by entailment from previous relevant cases.

As a simpler version of predicting a decision, a decision of a new case and a noticed (what is a noticed case?) case will be given as a query. Then, your legal textual entailment system identifies which paragraph in the noticed case entails the decision, by comparing the meanings between queries and the paragraphs.

The task investigates the performance of systems that identifies a paragraph that entails the decision of an unseen case.

Training data consists of triples of a query, noticed case, and a paragraph number of the noticed case by which the decision of the query is entailed. The process of executing the queries over the noticed cases and generating the experimental runs should be entirely automatic. Test data will include only queries and noticed cases, but no paragraph numbers.

There should be no human intervention at any stage, including modifications to your retrieval system motivated by an inspection of the test queries.

6.3 Task 3 Details

Our goal is to explore and evaluate legal document retrieval technologies that are both effective and reliable.

The task investigates the performance of systems that search a static set of civil law articles using previously-unseen queries. The goal of the task is to return relevant articles in the collection to a query. We call an article is "Relevant" to a query iff the query sentence can be answered Yes/No, entailed from the meaning of the article. If combining the meanings of more than one article (e.g., "A", "B", and "C") can answer a query sentence, then all the articles ("A", "B", and "C") are considered "Relevant". If a query can be answered by an article "D", and it can be also answered by another article "E" independently, we also consider both of the articles "D" and "E" are "Relevant". This task requires the retrieval of all the articles that are relevant to answering a query.

Japanese civil law articles (English translation besides Japanese) will be provided, and training data consists of pairs of a query and relevant articles. The process of executing the queries over the articles and generating the experimental runs should be entirely automatic. Test data will include only queries but no relevant articles.

There should be no human intervention at any stage, including modifications to your retrieval system motivated by an inspection of the queries. You should not materially modify your retrieval system between the time you downloaded the queries and the time you submit your runs.

At most three runs from each group will be assessed. The submission format and evaluation methods are described below.

6.4 Task 4 Details

Our goal is to construct Yes/No question answering systems for legal queries, by entailment from the relevant articles.

If a 'Yes/No' legal bar exam question is given, your legal information retrieval system retrieves relevant Civil Law articles. Then, the task investigates the performance of systems that answer 'Y' or 'No' to previously unseen queries by comparing the meanings between queries and your retrieved Civil Law articles. Training data consists of triples of a query, relevant article(s), a correct answer "Y" or "N". Test data will include only queries, but no 'Y/N' label, no relevant articles.

There should be no human intervention at any stage, including modifications to your retrieval system motivated by an inspection of the queries. You should not materially modify your retrieval system between the time you downloaded the queries and the time you submit your runs.

At most three runs for each group should be assessed. The submission format and evaluation methods are described below.

7. Corpus Structure

The structure of the test corpora is derived from a general XML representation developed for use in RITEVAL, one of the tasks of the NII Testbeds and Community for Information access Research (NTCIR) project, as described at the following URL:

http://sites.google.com/site/ntcir11riteval/

The RITEVAL format was developed for the general sharing of information retreival on a variety of domains.

7.1 Case Law Competition Corpus Structure (Tasks 1 and 2)

The format of the COLIEE competition corpora derived from an NTCIR representation of confirmed relationships between questions and the cases, as in the following example:

<pair id="t1-1">
<query>
Summary: A land developer paid redevelopment charges to a regional municipality before it began physical construction on the land. The developer sought to deduct the redevelopment payments from income earned in the taxation year. Revenue Canada disallowed the deduction. The Tax Court of Canada upheld Revenue Canada's decision. The developer appealed.
The essential facts are as follows. Urbandale paid the Regional Municipality of Ottawa-Carleton "redevelopment" charges of $2,908,100 on December 31, 1986 and sought to deduct that amount from income earned in its 1986 taxation year. These redevelopment charges were imposed pursuant to the authority conferred upon the Regional Municipality of Ottawa-Carleton by s. 50(6) of the Planning Act , R.S.O. 1983, and by ss. 31(1) and 33 of the Regional Municipality of Ottawa-Carleton , R.S.O. 1980. These charges are imposed for a public purpose, namely the financing costs of various municipal services and are not earmarked for the installation and upgrading of services in respect of the land for which the charges are levied.
Urbandale carries on business as a land developer. It buys raw land, subdivides it into building lots, services the lots and sells them to construction companies. The stage of development of Urbandale's lands in question in 1986 was limited to a draft plan of subdivision filed with the region. No physical construction had yet been built on the lands. The charges were only payable on the issuance of a building permit but Urbandale decided to pay the charges on the last day of 1986 anticipating an increase of the charges in 1987. In the 1986 taxation year, Urbandale had land sales of $7,410,489 but the development charges in question were not related to those sales.
Both parties agreed that the instant appeal would be heard solely on the basis of evidence adduced before the Tax Court and that no other witnesses would be called.
On behalf of the plaintiff, Alan Cohen, a lawyer specializing in municipal development and planning law, identified the policies and practices of the various levels of municipal government involved in the development of land process. He testified that the regional development charges were a one-time payment designed to reduce the impact on the existing tax base of the indirect capital costs of new development. Thus, the purpose of the charges was to raise funds to finance capital expenditures related to general urban expansion and was not linked directly to a single newly developed subdivision. Urbandale paid the development charges well before the building permit deadline so as to avoid anticipated increases in the rate. In its audited statements for 1986 Urbandale treated the charges as part of its "administrative-overhead" costs in computing its income for the year.
Leonard Cogan, C.A., a member of the firm of accountants which audited Urbandale's financial statements for the year 1986 stated that in accordance with generally accepted accounting principles, Urbandale deducted interests, taxes and other carrying costs on land held for development as an expense on its financial statement. He felt this was a conservative approach. In his view, the development charges amount to nothing more than a tax on developers calculated by reference to the number of lots for which building permits are to be issued. The payment of the charges does nothing to enhance the value of the land: thus capitalizing of such charges would overstate the profit from land development.
</query>
<cases_noticed>
24
35
327
580
32
385
194
292
452
</cases_noticed>
</pair>

The above is an example of Task 1 training data where query id "t1-1" has 9 supporting cases, and the supporting case IDs are given. The test corpora will include only the query field, but no cases_noticed.

<pair id="t2-1">
<query>
<case_description>
Summary: A land developer paid redevelopment charges to a regional municipality before it began physical construction on the land. The developer sought to deduct the redevelopment payments from income earned in the taxation year. Revenue Canada disallowed the deduction. The Tax Court of Canada upheld Revenue Canada's decision. The developer appealed.
The essential facts are as follows. Urbandale paid the Regional Municipality of Ottawa-Carleton "redevelopment" charges of $2,908,100 on December 31, 1986 and sought to deduct that amount from income earned in its 1986 taxation year. These redevelopment charges were imposed pursuant to the authority conferred upon the Regional Municipality of Ottawa-Carleton by s. 50(6) of the Planning Act , R.S.O. 1983, and by ss. 31(1) and 33 of the Regional Municipality of Ottawa-Carleton , R.S.O. 1980. These charges are imposed for a public purpose, namely the financing costs of various municipal services and are not earmarked for the installation and upgrading of services in respect of the land for which the charges are levied.
Urbandale carries on business as a land developer. It buys raw land, subdivides it into building lots, services the lots and sells them to construction companies. The stage of development of Urbandale's lands in question in 1986 was limited to a draft plan of subdivision filed with the region. No physical construction had yet been built on the lands. The charges were only payable on the issuance of a building permit but Urbandale decided to pay the charges on the last day of 1986 anticipating an increase of the charges in 1987. In the 1986 taxation year, Urbandale had land sales of $7,410,489 but the development charges in question were not related to those sales.
Urbandale claims that the redevelopment charges constituted a property tax and were therefore deductible pursuant to s. 18(2) of the Act . In the alternative, it alleges that these charges were expenditures under s. 9 of the Act . On the other hand, the Minister submits that these charges cannot be treated as a current expense for the purpose of computing Urbandale's income under s. 9 of the Act and that these charges do not constitute a property tax within the meaning of s. 18(2) and, in any event, that s. 18(2) does not authorize the current deduction of property taxes. In the alternative, the Minister alleges that the amount paid by Urbandale was in respect of a period posterior to 1986 with the consequence that it would be deductible only in the years after 1986, as prescribed by s. 18(9) of the Act .
Both parties agreed that the instant appeal would be heard solely on the basis of evidence adduced before the Tax Court and that no other witnesses would be called.
On behalf of the plaintiff, Alan Cohen, a lawyer specializing in municipal development and planning law, identified the policies and practices of the various levels of municipal government involved in the development of land process. He testified that the regional development charges were a one-time payment designed to reduce the impact on the existing tax base of the indirect capital costs of new development. Thus, the purpose of the charges was to raise funds to finance capital expenditures related to general urban expansion and was not linked directly to a single newly developed subdivision. Urbandale paid the development charges well before the building permit deadline so as to avoid anticipated increases in the rate. In its audited statements for 1986 Urbandale treated the charges as part of its "administrative-overhead" costs in computing its income for the year.
Leonard Cogan, C.A., a member of the firm of accountants which audited Urbandale's financial statements for the year 1986 stated that in accordance with generally accepted accounting principles, Urbandale deducted interests, taxes and other carrying costs on land held for development as an expense on its financial statement. He felt this was a conservative approach. In his view, the development charges amount to nothing more than a tax on developers calculated by reference to the number of lots for which building permits are to be issued. The payment of the charges does nothing to enhance the value of the land: thus capitalizing of such charges would overstate the profit from land development.
</case_description>
<decision>
It is not desirable that the tax returns always be in conformity with the financial statements prepared by the accountant. The key "matching principle" calls for annual profits from a business to be determined by setting the expenses incurred in earning the revenues from the business of the year against those revenues.
</decision>
<case_noticed>
25
</case_noticed>
</query>
<entailing_paragraph>
22
</entailing_paragraph>
</pair>

The above is an example of Task 2 training data where the noticed case of the query id "t2-1" is case ID 25, and a decision in the query was entailed from the paragraph No. 22 in the noticed case. For your information, the noticed case 25 and the paragraph No. 22 can be accessed at the bottom of this document. The decision in the query does not mean the whole decision of the case. This is a decision for a part of the case, and a relevant paragraph that supports this decision should be identified. The test corpora will not include the paragraph No. that entails the decision.

7.2 Statute Law Competition Corpus Structure (Tasks 3 and 4)

The format of the COLIEE competition corpora derived from an NTCIR representation of confirmed relationships between questions and the articles and cases relevant to answering the questions, as in the following example:

<pair label="Y" id="H18-1-2">
<t1>
(Seller's Warranty in cases of Superficies or Other Rights)Article 566 (1)In cases where the subject matter of the sale is encumbered with for the purpose of a superficies, an emphyteusis, an easement, a right of retention or a pledge, if the buyer does not know the same and cannot achieve the purpose of the contract on account thereof, the buyer may cancel the contract. In such cases, if the contract cannot be cancelled, the buyer may only demand compensation for damages. (2)The provisions of the preceding paragraph shall apply mutatis mutandis in cases where an easement that was referred to as being in existence for the benefit of immovable property that is the subject matter of a sale, does not exist, and in cases where a leasehold is registered with respect to the immovable property.(3)In the cases set forth in the preceding two paragraphs, the cancellation of the contract or claim for damages must be made within one year from the time when the buyer comes to know the facts.
(Seller's Warranty in cases of Mortgage or Other Rights)Article 567(1)If the buyer loses his/her ownership of immovable property that is the object of a sale because of the exercise of an existing statutory lien or mortgage, the buyer may cancel the contract.(2)If the buyer preserves his/her ownership by incurring expenditure for costs, he/she may claim reimbursement of those costs from the seller.(3)In the cases set forth in the preceding two paragraphs, the buyer may claim compensation if he/she suffered loss.
</t1>
<t2>
There is a limitation period on pursuance of warranty if there is restriction due to superficies on the subject matter, but there is no restriction on pursuance of warranty if the seller's rights were revoked due to execution of the mortgage.
</t2>
</pair>

The above is an example where query id "H18-1-2" is confirmed to be answerable from article numbers 566 and 567 (relevant to Task 3). The pair label "Y" in this example means the answer of query is "Yes", which is entailed from the relevant articles (relevant to Task 4).

For the Tasks 3 and 4, the training data will be the same. The groups who participate in the only Task 3 can disregard the pair label.


Competition Results Submission Format

For Tasks 1 and 3, a submission should consist of a single ASCII text file. Use as single space to separate columns as follows, with three columns per line as follows:

H18-1-2 566 univABC
H18-1-2 567 univABC
H18-5-A 322 univABC
H19-19-I 433 univABC
H21-5-3 110 univABC
.
.
.
where:

1. The first column is the query id.
2. The second column is the official article(case) number of the retrieved article(case).
3. The third column is called the "run tag" and should be a unique identifier for the submitting group, i.e., each run should have a different tag that identifies the group. Please restrict run tags to 12 or fewer letters and numbers, with no punctuation.
In this example of a submission, you can see that H18-1-2 has multiple relevant articles (566 and 567).

For Task 4, again a submission should consist of a single ASCII text file. Use as single space to separate columns as follows, with three columns per line as follows:

H18-1-2 Y univABC
H18-5-A N univABC
H19-19-I Y univABC
H21-5-3 N univABC
.
.
.
where:

1. and 3 as for Phase One,
2. "Y" or "N" indicating whether the Y/N question was confirmed to be true ("Y") by the relevant articles, or confirmed to be false ("N").

Program Chairs

Ken Satoh, ksatoh(at)nii.ac.jp, National Institute of Informatics and Sokendai, Japan
Randy Goebel, rgoebel(at)ualberta.ca, University of Alberta, Canada
Mi-Young Kim, miyoung2(at)ualberta.ca, Department of Computing Science, University of Alberta, Canada
Yoshinobu Kano, kano(at)inf.shizuoka.ac.jp, Shizuoka University, Japan
Masaharu Yoshioka, yoshioka(at)ist.hokudai.ac.jp, Hokkaido University, Japan

Program Committee Members

Yusuke Miyao, yusuke(at)nii.ac.jp, National Institute of Informatics, Japan
Bernardo Magnini, magnini(at)fbk.eu, Fondazione Bruno Kessler, Italy
Nguyen Le Minh, nguyenml(at)jaist.ac.jp, JAIST, Japan
Satoshi Tojo, tojo(at)jaist.ac.jp, JAIST, Japan
Livio Robaldo, livio.robaldo(at)uni.lu, University of Luxembourg, Luxembourg
Adam Wyner, azwyner(at)abdn.ac.uk, University of Aberdeen, UK
Douglas Oard, oard(at)umd.edu, University of Maryland, USA
Akira Shimazu, shimazu(at)jaist.ac.jp, JAIST, Japan
Kentaro Inui, inui(at)ecei.tohoku.ac.jp, Tohoku University, Japan
Katsumasa Yoshikawa, KATSUY(at)jp.ibm.com, IBM Research, Tokyo, Japan
Marie-Francine Moens, sien.moens(at)cs.kuleuven.be, KU Leuven, Belgium
Jack G. Conrad, Jack.G.Conrad(at)ThomsonReuters.com, Thomson Reuters

Questions and further information

miyoung2(at)ualberta.ca

Application Details

Potential participats to COLIEE-2018 should respond to this call for participation by submitting an application. To apply, submit the application and memorandums of the following URL to miyoung2(at)ualberta.ca:

Application:
http://www.ualberta.ca/~miyoung2/COLIEE2018/application.pdf
Memorandum for Tasks 1 and/or 2 (Case law competition)
http://www.ualberta.ca/~miyoung2/COLIEE2018/Case_law_competition_memorandum.pdf
Memorandum for Tasks 3 and/or 4 (Statute law competition, English data)
http://www.ualberta.ca/~miyoung2/COLIEE2018/Statute_law_competition_memorandum_EN.pdf
Memorandum for Tasks 3 and/or 4 (Statute law competition, Japanese data)
http://www.ualberta.ca/~miyoung2/COLIEE2018/Statute_law_competition_memorandum_JA.pdf

We will send an acknowledgement to the email address supplied in the form once we have processed the form.

Previous COLIEE

COLIEE 2017
COLIEE 2016
COLIEE 2015
Last updated: Apr., 2018