Link Search Menu Expand Document

Mapping BITs RNN framework and dedicated tool

Contract: analytics/drafting/reviewContract: automated creation
mappinginvestmenttreaties.com/specials/rnn-expe...
Main research: March 2022

Contents

  1. What does it claim to do?
  2. Substantiation of claims & potential issues
  3. How might the end-user assess effectiveness?
  4. What form does it take?
  5. Is it currently in use?
  6. The creators
  7. Jurisdiction
  8. License

What does it claim to do?

The Mapping BITs RNN framework is a legal text generation and assembly system. The system incorporates a Recurrent :Neural Network (RNN) and offers a means to automatically generate the text of international investment treaties. The generated text is a prediction of the text from a model that was trained on actual treaties or treaties in the process of negotiation. A dedicated tool (archived) demonstrates an application of the framework. The application takes account of the respective bargaining power of the parties to the treaty in the generation process.

Claimed essential features

  • Provides a system for text generation and assembly in the domain of international investment law.
  • Provides a tool to predict treaty text based on user-defined specifications.
  • Simulates bargaining behaviour.

“This paper constructs a legal text generation and assembly system in the domain of international investment law.” (Alschner and Skougareveskiy, 2017A)

“As a practical application of our improved RNN framework, we created a dedicated tool that allows users to predict treaties based on self-set specifications at http://mappinginvestmenttreaties.com/specials/rnn-experiment.” (Alschner and Skougareveskiy, 2017A)

“We simulate the bargaining behavior in negotiations through our filter mechanism [in the dedicated tool]” (Alschner and Skougareveskiy, 2017A)

The framework offers a means of “predicting treat[ies] under negotiation” (Alschner and Skougareveskiy, 2017A)

”[It] gives the user more control over the process by letting her select the legal building blocks of her choice for automated text production thereby limiting the “black box” aspect otherwise present in the practical applications of RNNs.” (Alschner and Skougareveskiy, 2017A)

Claimed rationale and benefits

  • Can be used to produce drafts of international investment treaties.
  • Draft texts include compromises derived from past treaty practices of the contracting states.

“we … create a user-driven application aimed at facilitating consensus-building at the bargaining stage” (Alschner and Skougareveskiy, 2017A)

“AI can facilitate consensus building by generating compromise drafts between states based on their prior treaty practice to avoid a battle-of-forms at the outset of negotiations. AI can also consolidate thousands of bilateral texts into a single multilateral treaty draft that can serve as the basis to harmonize the divergent practice of two hundred countries.” (Alschner and Skougareveskiy, 2017A)

“RNNs [might] facilitate bilateral or multilateral negotiations by distilling a first draft from existing practice guided by human input and conditioned by human-imposed filters.” (Alschner and Skougarevskiy, 2016)

“RNN-based treaty text prediction … promises a range of possible future applications” (Alschner and Skougarevskiy, 2016)

Claimed design choices

The claimed design choices include those relating to the choice of algorithms, training corpora (the algorithm is trained on clauses relating to specific issues rather than the treaty text as a whole), methods for addressing imbalanced data (imposition of priors), and selection (filtering) of the texts generated by the system. 

  • Using issue-specific corpora
  • Post-processing of the documents
  • Using prior probability distribution during training to reduce repetition.

“By generating texts from RNNs trained on issue-specific corpora we are able to apply … post-processing to the semantically close texts dealing with one legal concept. We, therefore, are able to produce a structured document provided that individual segments are concatenated following a document organization familiar to the user. Second, we can avoid the duplication of issue elements … when predicting entire treaties.” (Alschner and Skougareveskiy, 2017A)

“the introduction of priors [a prior probability distribution]” at the training stage  “leads to a significant reduction in repetitiveness of predicted texts” (Alschner and Skougareveskiy, 2017A)

“… we generate a large pool of predicted texts for each treaty clause type. The generated element are then compared to corresponding elements in a user-selected set of actual treaty texts that serve to filer the pool of predicted clauses …” (Alschner and Skougareveskiy, 2017A)

Top

Substantiation of claims & potential issues

  • The system assumes that past treaty practice is a good indicator of future treaty practice. It cannot take account of a state’s current negotiating policy and practice. Its utility as a means of forecasting future treaties may therefore be limited.
  • Treaties, like other agreements, must operate as a coherent whole. It is not clear how the system achieves this when it issues predictions on a clause-by-clause basis.
  • The authors assume a linear correlation between the bargaining power of a state and its treaty outcomes. While some correlation may exist, we doubt it is linear. These assumptions have implications for the utility of the system.

The Mapping Bits system is described in great detail in Wolfgang Alschner and Dmitriy Skougarevskiy, ‘Towards an automated production of legal texts using recurrent :neural networks’ in Proceedings of the 16th edition of the International Conference on Artificial Intelligence and Law (ICAIL ‘17) ACM, New York, 229 (Alschner and Skougareveskiy, 2017A).

Data

“… we have built a corpus of 1628 English-language bilateral investment treaties, which we split into their 22 632 constituent articles.” (Alschner and Skougareveskiy, 2017A)

  • The data is split into clause-specific subcorpora. A text generation system is trained on each subcorpus:

    “… we rely on an automated rule-based procedure … to annotate each of these clauses with meta information as to the issue area to which they relate. We thus know for each clause whether it concerns ‘definitions’, ‘expropriation’, ‘dispute settlement’ or another legal concept. Furthermore, we also code for sub-issues within issue areas to capture variation within clauses on the same subject. As a third step, we either select the issue areas of a target treaty to mimic its design or let a user choose the elements they want in their predicted treaty and create the issue-specific corpus”. (Alschner and Skougareveskiy, 2017A)

  • In previous research, the authors remarked an uneven temporal distribution in the training data which negatively impacted the performance of the system. They therefore include a step of upsampling more recent training data:

    “We circumvent this problem [of ‘backwards bias’] by repeating each text in the training data with the following frequency: freq(year) = discretize(½ (max # treaties - # treatiesyear)), where year is the year of signature of the given treaty, # treatiesyear is the count of treaties in a given year, discretize is the function that maps the obtained weight to an integer sequence. We give utmost weight in our training set to the most recent treaties, as well as the treaties concluded in years with few signed agreements. However, every treaty is still featured in our training set at least once. (Alschner and Skougareveskiy, 2017A)

Training and architecture

“We employed a 2-layer LSTM [long short-term memory] with 512 nodes per layer, sequence length of 200 characters and a dropout factor of 0.5 to train it on 80% of the data (10% were used for validation and test sets) for each issue area corpus separately.[..] Corpus texts were constructed by concatenating split article texts back to one large text file, preserving the article numbers and names in headers, which precede each article text within each treaty. We trained each model for 10 epochs (weighted case) or 50 epochs (unweighted case). Validation set loss was typically lower than train set loss due to a high dropout factor specified, signalling little overfitting.” (Alschner and Skougareveskiy, 2017A)

  • The systems then generates text using a trained model:

    “… we specified the starting sequence of “#Article” (signifies a new article delimiter in train data) and generated 150 strings of 100,000 characters from the trained model with a temperature of 0.5 (a factor between 0 and 1 by which the predicted character probabilities are divided to supply more innovative results) for each issue area”. (Alschner and Skougareveskiy, 2017A)

  • It then filters the generated texts:

    “The generated elements are … compared to corresponding elements in a user-selected set of actual treaty texts that serve to filter the pool of predicted clauses to identify the text that is closest to the user-selected benchmark.

    We adopt two measures to establish distance between the real and generated clauses. First, … we compute 5-character-long q-grams of the actual and generated element strings and compute their Jaccard distances. Second, … we compute the cosine distance between term-frequency-weighted mean of word embeddings vectors from the GloVe model trained on the entire corpus of BIT texts.

    When the distance between each real and generated treaty element is defined, the legal text generation problem is reduced to finding an RNN-simulated treaty with minimum distance to a set of treaty elements having observable features specified by a user”. (Alschner and Skougareveskiy, 2017A)

    This feature allows for the simulation of ‘bargaining power’ in the tool. By sliding the bar in the demo page to one signatory vs. another, the user is changing the set of treaty elements that the generated samples are compared to.

Evaluation

  • Evaluation of the RNN model was done by calculating its performance on a test set.

  • The authors also evaluate the ‘repetitiveness’ of the generated output, that is they measure the diversity of subjects of the generated texts and compare it to that of the actual treaties:

    “For each actual or simulated text we apply the rule-based categorization procedure of [2] and gather its results in a binary vector. Then we compare Shannon’s entropy of vectors for actual and generated texts”. A second external evaluation is the propensity of articles about a specific topic: “we observe that the weighting of training data improves the predicted output as the share of actual and predicted sub-issues, which we aggregated for convenience by issue, converges towards the 45-degree line.”

Resources

Academic Papers

  • Wolfgang Alschner and Dmitriy Skougarevskiy, ‘Towards an automated production of legal texts using recurrent neural networks’ in Proceedings of the 16th edition of the International Conference on Artificial Intelligence and Law (ICAIL ‘17) Association for Computing Machinery, New York, NY, USA, 229. (Alschner and Skougareveskiy, 2017A)
  • Wolfgang Alschner and Dmitriy Skougarevskiy, ‘Can Robots Write Treaties? Using Recurrent Neural Networks to Draft International Investment Agreements’ in F. Bex and S. Villata (eds.), JURIX: Legal Knowledge and Information Systems (IOS Press, 2016) 119 available at SSRN: https://ssrn.com/abstract=2984935 
  • Wolfgang Alschner and Dmitriy Skougarevskiy, ‘Convergence and Divergence in the Investment Treaty Universe — Scoping the Potential for Multilateral Consolidation’ Trade, Law and Development 8, 2 (2017) (Alschner and Skougareveskiy, 2017B)

Github

  • In (Alschner and Skougareveskiy, 2017A) the authors state that they use a “character-level recurrent neural network (char-RNN)” and that they “employed a 2-layer LSTM” of Justin Johnson, ‘torch-rnn’ (archived)  (2016).
Top

How might the end-user assess effectiveness?

A dedicated tool demonstrates an application of the authors’ RNN framework.

Top

What form does it take?

Form

Application

Details

Describing the RNN Experiment, a demo of which is featured on the M apping Bits website, the authors state that “As a practical application of our improved RNN framework, we created a dedicated tool … “ (Alschner and Skougareveskiy, 2017A)

Top

Is it in current use?

The dedicated tool which demonstrates an application of the RNN framework is available on the Mapping Bits website. The RNN-based legal text generation and assembly system constructed by Alschner and Skougarevskiy does not appear to be otherwise available for use.

Top

The creators

Created by

Academics

Details

Alschner and Skougarevskiy are legal scholars. Alschner is also a member of the Californian bar. The RNN framework is based on the code of Justin Johnson, Assistant Professor, Computer Science and Engineering at University of Michigan.

Top

Jurisdiction

Background of developers

Alschner was formerly a researcher at The Graduate Institute, Geneva and World Trade Institute, Bern and is now an Associate Professor, University of Ottawa. He is a member of the California Bar. (Wolfgang Alschner ; archived)

Skougarevskiy was formerly a researcher at The Institute for the Rule of Law and the World Trade Institute. He is now an Associate Professor of Empirical Legal Studies at the European University at St Petersburg. (Dmitriy Skougarevskiy; archived)

Alschner and Skougarevskiy received funding support from the Swiss National Science Foundation and the Swiss Network for International Studies (Alschner and Skougareveskiy, 2017A).

Target jurisdiction

International

Target legal domains

International investment law

Top

License

The code for the RNN framework and application is not available.

The authors state that they employed a 2-layer LSTM based on Johnson’s torch-rnn. Johnson’s code appears on github at https://github.com/jcjohnson/torch- rnn. Johnson’s code is made available under a permissive MIT licence (jcjohnson/torch-rnn; archived) It is not clear how much of the functionality of the system depends on Johnson’s torch-rnn and how much depends on other code.

Top

Updated Top