Read July 2026

SciKnowEval

My takeaway

Problem

Scientific benchmark dataset

Solution

This paper contains knowledge memory, comprehension, reasoning, safety, application - 5 categories which include knowledge application, but our paper only has application level QA, which is the most challenging. This paper has 4 domains with balanced QA distribution, whereas our paper has 9+ domains with imbalanced QA distribution. Here QA are curated using i) Direct QA scraping like we did but from other OS benchmark datasets ii) Literature from text books to QA generation using LLM iii) scientific DBs to QA generation using LLM. QA types mcqa, generation QAs, true/false QAs. not images. 20 models evaluated - open source, closed source, scientific domain-specific open sourced models. Insights from Evaluation metrics are very simple - just accuracy and f1 score, but ours has more insights.

benchmark
dataset