Turning Legal Jargon into Plain English with Emoji

from kaggle_secrets import UserSecretsClient

client = genai.Client(api_key=UserSecretsClient().get_secret("GOOGLE_API_KEY"))

request = 'Please summarize the following terms of service in plain, easy-to-undersand English:'

def translate_doc(request: str) -> str:
  """Execute the request on the uploaded document."""
  # Set the temperature low to stabilise the output.
  config = types.GenerateContentConfig(temperature=0.0)
  response = client.models.generate_content(
      model='gemini-1.5-pro',
      config=config,
      contents=[request, document_file],
  )

  return response.text

summary = translate_doc(request)

Dig-A-Hole's Terms of Service basically say:

* Digging Holes: You can dig holes at their facility for fun and stress relief. They offer a free trial dig, then you need a monthly subscription. You can also rent shovels.
* The Agreement: By using their services, you agree to these rules.
* Subscriptions: Subscriptions are monthly and you have to pay for the whole month even if you cancel early. The price is on their website or at the facility.
* Shovel Rental: You can rent shovels for an extra fee and you're responsible for any damage beyond normal wear and tear.
...

few_shot_prompt = """
Here are some examples of how to translate legal terms of service and their friendly summaries using emoji:

Original: "We reserve the right to accept or refuse membership in our discretion." 
Translated: "We get to say yay 👍 or nay 👎 to memberships, just because we can💪!"

Original: "To the maximum extent permitted by law, you agree that the Company shall not be held liable for any injuries, damages, or losses incurred in connection with the use of our services or products. By using our services, you waive any right to bring a claim or lawsuit against us for such injuries."
Translated: "Oops! If you trip, slip, or fall dramatically and get hurt while using our stuff, please don’t sue us 😅🙏. By hanging out with us, you're saying, 'Okay cool, I won't blame you if I bonk myself💖'"

Please summarize the following terms of service in plain, easy-to-undersand English with emoji.
"""

response = client.models.generate_content(
    model='gemini-1.5-pro',
    config=model_config,
    contents=[few_shot_prompt, document_file],
)

Want to dig a hole? 🕳️ Here's the deal:

1. **The Agreement:** This is a contract. By digging with us, you agree to these rules. 🤝
2. **What We Do:** We provide a place for you to dig holes for fun and stress relief!  We offer a free trial dig, then monthly subscriptions.  You can also rent a shovel. 🥄
3. **Subscriptions & Payment:** Subscriptions are for a minimum of one month.  We'll tell you the price on our website or at our facility.  You can cancel anytime, but you still have to pay for the current month.  No refunds for partial months. 🗓️💰
4. **Shovel Rental:**  You can rent a shovel 🥄 for an extra fee.  Please return it in good condition (normal wear and tear is okay).  We might charge you if you break or lose it.
...

import json

few_shot_prompt = """
Please translate the following terms of service to more casual,friendly, easy-to-understand English with emoji. The output should be a JSON array and each element is
a JSON object with the "Original" and "Translated" version.
Here are some examples of how to translate leagal terms of service and their friendly summaries using emoji:
```
{
Original: "We reserve the right to accept or refuse membership in our discretion." 
Translated: "We get to say yay 👍 or nay 👎 to memberships, just because we can💪!"
}
{
Original: "To the maximum extent permitted by law, you agree that the Company shall not be held liable for any injuries, damages, or losses incurred in connection with the use of our services or products. By using our services, you waive any right to bring a claim or lawsuit against us for such injuries."
Translated: "Oops!😳 If you trip, slip, or fall dramatically and get hurt🤕 while using our stuff, please don’t sue us 😅🙏. By hanging out with us, you're saying, 'Okay cool, I won't blame you if I bonk myself💖'"
}
```

"""

response = client.models.generate_content(
    model='gemini-1.5-pro',
    contents= [few_shot_prompt, document_file],
    config={
        'response_mime_type': 'application/json'
    },
)

[
    {
        "Original": "These Terms of Service constitute a legally binding agreement between you (\"Customer,\" \"you,\" or \"your\") and Dig-A-Hole (\"Dig-A-Hole,\" \"we,\" \"us,\" or \"our\"). By using our services, you acknowledge that you have read, understood, and agree to be bound by these Terms.",
        "Translated": "This is like a pinky swear 🤝 between you (that's you!) and us (Dig-A-Hole!). By playing here, you're saying you get the rules and promise to follow them. 💖"
    },
    {
        "Original": "Dig-A-Hole provides a facility where customers can engage in the activity of digging holes in designated areas of a field. Our services are intended for recreational exercise and stress relief.",
        "Translated": "We've got a place where you can dig holes to your heart's content! 🕳️ It's great for exercise and chilling out. 😎"
    },
    ...

import enum

# Define the evaluation prompt for evaluating the "Translated" text
TRANSLATION_EVAL_PROMPT = """\
# Instruction
You are an expert evaluator. Your task is to evaluate the quality of the AI-generated translation of a sentence from a terms of service
document.
We will provide you with the original sentence and the AI-generated translation.
You should first read the original sentence carefully, then evaluate the quality of the translation based on the Criteria 
provided in the Evaluation section below.
You will assign the translation a rating following the Rating Rubric and Evaluation Steps. Give step-by-step explanations 
for your rating, and only choose ratings from the Rating Rubric.

# Evaluation
## Metric Definition
You will be assessing the translation Clarity, friendliness and Completeness.

## Criteria
Clarity: The translation prioritizes making the core meaning immediately accessible and easy to understand for everyone, 
even those unfamiliar with legal terms. It uses simple language and relatable analogies to convey the essential information effectively.
Friendliness: The translation adopts an extremely approachable, casual, and enthusiastic tone, using emojis and informal language 
to create a positive and engaging experience for the reader. The high level of friendliness is intentional to make the legal terms 
feel less intimidating and more welcoming.
Completeness: Completeness: The translation accurately and fully conveys the core meaning and key pieces of information presented 
in the original sentence. It should capture all the essential facts, actions, entities, and relationships described, 
even if expressed in simpler language. The level of detail in the translation should be sufficient to understand the main points 
of the original without losing critical information.

## Rating Rubric
5: (Very good). The translation is accurate, clear, friendly, and complete.
4: (Good). The translation is mostly accurate, clear, friendly, and complete.
3: (Ok). The translation is understandable but may have minor issues with accuracy, clarity, or friendliness. Emojis might 
be missing or slightly awkward.
2: (Bad). The translation has significant issues with accuracy or clarity, or fails to adopt a friendly tone.
1: (Very bad). The translation is inaccurate, incomprehensible, or completely fails to address the original meaning.

## Evaluation Steps
STEP 1: Assess the translation in aspects of clarity, friendliness, and completeness according to the criteria.
STEP 2: Score based on the rubric.

# User Inputs and AI-generated Response
## Original Sentence

{original}

## AI-generated Translation

{response}
"""

# Define a structured enum class to capture the result.
class SummaryRating(enum.Enum):
    VERY_GOOD = '5'
    GOOD = '4'
    OK = '3'
    BAD = '2'
    VERY_BAD = '1'

def eval_translation(original, ai_response):
    """Evaluate the generated translation against the original sentence."""

    # It doesn't look like I can use chat with gemini-1.5-pro. Using gemini-2.0-flash here.
    chat = client.chats.create(model='gemini-2.0-flash')
    
    # Generate the full text response.
    response = chat.send_message(
        message=TRANSLATION_EVAL_PROMPT.format(original=original, response=ai_response)
    )
    verbose_eval = response.text
    
    # Coerce into the desired structure.
    structured_output_config = types.GenerateContentConfig(
        response_mime_type="text/x.enum",
        response_schema=SummaryRating,
    )
    response = chat.send_message(
        message="Convert the final score.",
        config=structured_output_config,
    )
    structured_eval = response.parsed
    
    return verbose_eval, structured_eval

# Ideally, I want to check everything but I don't have enough quota. So, iterating only first three elements.
evaluation_results = []
for item in response_load[:3]:
    original_text = item['Original']
    translated_text = item['Translated']
    verbose_eval, structured_eval = eval_translation(original=original_text, ai_response=translated_text)

    evaluation_results.append({
        'original': original_text,
        'translated': translated_text,
        'verbose_evaluation': verbose_eval,
        'structured_evaluation': structured_eval.name if structured_eval else None
    })

for result in evaluation_results:
    print(f"Original: {result['original']}")
    print(f"Translated: {result['translated']}")
    print(f"Verbose Evaluation:\n{result['verbose_evaluation']}")
    print(f"Structured Evaluation: {result['structured_evaluation']}")
    print("-" * 20)

Original: These Terms of Service constitute a legally binding agreement between you ("Customer," "you," or "your") and Dig-A-Hole ("Dig-A-Hole," "we," "us," or "our"). By using our services, you acknowledge that you have read, understood, and agree to be bound by these Terms.
Translated: This is like a pinky swear 🤝 between you (that's you!) and us (Dig-A-Hole!). By playing here, you're saying you get the rules and promise to follow them. 💖
Verbose Evaluation:
## Evaluation

**STEP 1:**

*   **Clarity:** The translation uses a "pinky swear" analogy to explain the legally binding agreement, which is clear and easy to understand. It simplifies the language effectively.
*   **Friendliness:** The translation is extremely friendly, using phrases like "that's you!" and emojis (🤝, 💖). The tone is enthusiastic and welcoming.
*   **Completeness:** The translation captures the core meaning of the original sentence, explaining that using the services means agreeing to the terms.

**STEP 2:**

Based on the rubric, the translation is very good.

class GeminiEmbeddingFunction(EmbeddingFunction):
    # Specify whether to generate embeddings for documents, or queries
    document_mode = True

    @retry.Retry(predicate=is_retriable)
    #def __call__(self, input: all_paragraphs) -> Embeddings:
    def __call__(self, input: document_file) -> Embeddings:
        if self.document_mode:
            embedding_task = "retrieval_document"
        else:
            embedding_task = "retrieval_query"

        resp = client.models.embed_content(
            model="models/text-embedding-004",
            contents=input,
            config=types.EmbedContentConfig(
                task_type=embedding_task,
            ),
        )
        return [e.values for e in resp.embeddings]

mport chromadb

DB_NAME = "googlecardb"

embed_fn = GeminiEmbeddingFunction()
embed_fn.document_mode = True

chroma_client = chromadb.Client()
db = chroma_client.get_or_create_collection(name=DB_NAME, embedding_function=embed_fn)

db.add(documents=all_paragraphs, ids=[str(i) for i in range(len(all_paragraphs))])

# Switch to query mode when generating embeddings.
embed_fn.document_mode = False

# Search the Chroma DB using the specified query.
query = "What is the cancellation policy?"

result = db.query(query_texts=[query], n_results=1) 
[all_passages] = result["documents"]

Markdown(all_passages[0])

query_oneline = query.replace("\n", " ")

prompt = f"""You are a very friendly bot that answers questions using text from the reference passage included below.
Be sure to respond in a complete sentence, but being casual and very friendly. Make sure to include some appropriate emoji.
You need to use easy-to-understand English instead of using legal terms. If the passage is irrelevant to the answer, you
may say that you can not find the information.

QUESTION: {query_oneline}

model_config=types.GenerateContentConfig(temperature=0.0)
answer = client.models.generate_content(
    model="gemini-2.0-flash",
    config=model_config,
    contents=prompt
)

Markdown(answer.text)
"""

Hey there! 👋 You can cancel your subscription whenever you want, but since there's a one-month minimum, you'll still need to pay for the current month you're in. Your access will keep going until the end of that month you've already paid for. Just so you know, there are no refunds for partial months. 😊