Compare Query

Using Compare Queries

Compare is particularly useful for reviewing:

• Documents against company policies.

• New documents from a client against a set of documents containing previously acceptable positions.

• Documents across a standard set of clauses or precedents.

• Summarising relevant sections side-by-side.

Compare works best when:

• You are reviewing a document against a known standard or previous version.

• You want a side-by-side summary of relevant sections from multiple documents.

• You are exploring changes or inconsistencies without needing Playbooks to make a judgement.

Adding Compare Queries

1. You must upload the document(s) you want to compare against as a dataset. This should be the base document(s) you will be comparing to, NOT the document you are reviewing.

2. Follow the steps outlined here

3. The name field is a human-friendly label. Add a label which makes sense to you (and anyone else using this Playbook), e.g. “Limitation of Liability”

4. In the dataset section, choose the dataset you created in step 1.

5. The query section describes the query you are trying to perform. You can phrase it as a question or a statement.

Limitations of Compare Queries

1. No external knowledge

   Compare only evaluates information in your document and your Dataset. It cannot check compliance against external legislation, standards or databases. Unless the Dataset you provide includes that external information, it cannot answer:

   • “Does this document infringe the Australian Consumer Law in a different way to the comparison Dataset?”

   • “If the terms of this document differ from the comparison Dataset, the client must have sent us an email instructing us to agree to those terms.”

   • “Does this document contain any terms that are off-market that the client hasn’t agreed to previously?”

   This means that you can trust that every part of an answer from a Compare query is grounded in the actual document that you are reviewing. There is no risk of hallucination.

2. Not a replacement for human judgement
   
   Compare highlights differences and similarities but is not well-suited to qualitatively assessing which version is better or advise on legal or commercial preference. Playbooks will attempt to answer the question with relevant information and provide a judgement, however, Playbooks is not designed for human decision-making, so this judgement may be incorrect.

3. Requires sufficient relevance in the new document
   
   Compare relies on the new document containing meaningful content to analyse. If the new document doesn’t include relevant information, Compare will return no result, even if the Dataset contains that information.

   This doesn’t necessarily mean there’s nothing relevant in the document, it may simply mean that the question was too broad or relied too heavily on the comparison documents.

   To get meaningful results, you need to give Compare enough clues about what to look for in the document itself. The more precisely you describe what’s relevant in the document, the better the comparison will be.

   If you find yourself already outlining the exact information that will be relevant, you likely have enough knowledge of the topic to move beyond Compare. In that case, Extract or Test queries may serve you better, they let you check or summarise those specific points directly.

Best Practice for Best Results

1. Open-ended queries work best
   Compare pulls information from both documents and summarises similarities and differences, rather than checking a requirement.