AI Models Tested on Israeli Rabbinical Exams, Showing Strong Knowledge
Translated & summarized from NEWSru Israel by baba
AI models were tested on Israeli Chief Rabbinate exams covering Jewish law, with Claude Fable 5.1 scoring nearly 80%. The models operated offline, simulating closed-book conditions. Researchers believe AI could become a useful tool for those studying Jewish law, though it cannot make independent rulings.
The story in 5 lines · by baba
- AI models were tested on Israeli Chief Rabbinate exams covering Jewish law.
- Claude Fable 5.1 achieved the highest score, nearing 80% on the HalachaBench test.
- The AI models operated offline, simulating a closed-book exam environment.
- The evaluation used AI algorithms due to the lack of official answer keys.
- Researchers suggest AI could become a useful tool for studying Jewish law.
Researchers have evaluated the ability of artificial intelligence models to answer questions on Jewish law (Halakha) by creating a project called HalachaBench. The AI chatbots were tasked with responding to real questions from the Israeli Chief Rabbinate's exams, covering topics such as family purity laws, wedding ceremonies, prayers, blessings, and agricultural commandments observed in Israel. The models operated offline, simulating a closed-book exam environment. Claude Fable 5.1 achieved the highest score, nearing 80 out of 100, followed by GPT-6 Astra and GPT-6.1 Sol. The evaluation was conducted using AI algorithms, as the Chief Rabbinate's exams lack official answer keys. The researchers noted that the top-performing model, Claude Fable 5.1, was involved in developing the scoring criteria, which could potentially affect the objectivity of the results. While the experiment does not suggest AI can independently make halakhic rulings, it demonstrates that current language models possess substantial knowledge of Jewish law and could serve as valuable tools for rabbis, educators, and students in the future.