Quick answer
Compare models using the same Arabic tasks and score factual accuracy before style. Record model versions and test dates; the result applies to your workflow rather than every Arabic use case.
The best model for a short formal email may not be the best for Darija, technical translation or a mixed Arabic–French report. Compare the work you actually need. This guide provides a test method, not a claim that one current model wins.
Before you begin
Choose two or three models you can access and prepare the same small set of fictional tasks. Record the model name, version when available, date, settings and applicable cost. Do not send confidential text to unapproved services.
Method guide · not a product benchmark01Build a representative task set
Include a formal Arabic email, a concise summary and a task with Arabic and French terminology. Add a Darija example only if your readers need it. Write the required facts and desired tone before running the comparison. Avoid judging a model from a single impressive answer.
02Keep inputs consistent
Use the same task, source text and instructions for every model. Ask for the same length and output format. Record whether web access or other tools were enabled. If one model has extra source material, you are comparing workflows rather than the models alone.
03Score what matters
Rate factual fidelity, natural Arabic, terminology, instruction following and editing effort separately. A simple 1–5 scale works if each score has a written definition. Have a fluent reader review the output. Beautiful phrasing should not compensate for incorrect dates or invented facts.
04Check layout and repetition
Paste the output into your actual document workflow. Inspect right-to-left paragraphs, mixed-direction numbers, punctuation and table alignment. Repeat a few tasks to see whether results are stable. Distinguish model errors from formatting problems introduced by your editor or PDF export.
05Choose for the specific workflow
Compare quality with cost, speed, privacy requirements and the amount of correction needed. Keep examples of failures and rerun the same tasks after major model changes. Report your method and limits if you publish the comparison. Avoid declaring a universal winner from a small personal test.
Try this prompt
Rewrite the following fictional business email in clear Modern Standard Arabic. Preserve every name, number and date. Use a professional, natural tone and no more than 120 words. Do not add promises or facts. After the email, list any ambiguous phrases requiring clarification. Source: [fictional email]
Worked example
Fictional teaching example — not a product test or a real customer record.
Starting information
Source facts: Meeting on 21 October at 10:00. Location unconfirmed. Task: write a short formal Arabic invitation without adding facts.
Expected result
Acceptable example: ندعوكم إلى اجتماع يوم 21 أكتوبر الساعة 10:00. سيُؤكَّد المكان لاحقًا. A response that invents a city or changes the time fails factual fidelity, even if its Arabic sounds polished.
Before you use the result
- Mark factual fidelity as failed if a required fact changes or an unsupported fact appears.
- For naturalness: 1 = hard to understand; 3 = understandable but needs edits; 5 = natural and ready to use.
- Record editing minutes separately; repeat tasks before choosing a model.
Worksheet
Editable text file. Fill it with your own evidence; blank fields are not results.
Download the worksheet (.txt)
