ScholarEval Evaluation Framework

Detailed criteria, rubrics, and quality indicators for each ScholarEval dimension.

1. Problem Formulation & Research Questions

Strong (4–5):

• Research question is specific, measurable, and bounded in scope
• Theoretical or practical significance is explicitly stated
• Hypothesis or objective is falsifiable and testable

Adequate (3):

• Research question is present but could be more precise
• Significance implied rather than argued

Weak (1–2):

• Research question is absent, vague, or overly broad
• No articulation of why this question matters

Critical issues (score 1):

• No discernible research question
• Question is trivially answered or unanswerable

2. Literature Review

Strong (4–5):

• Covers foundational and recent (≤5 years) literature
• Synthesises themes, contradictions, and research gaps
• Critically evaluates sources rather than summarising

Adequate (3):

• Coverage of major works but gaps in recent literature
• Describes sources without deep synthesis

Weak (1–2):

• Missing major foundational works
• Chronological listing with no synthesis

3. Methodology & Research Design

Strong (4–5):

• Design clearly matches research questions
• Validation procedures stated and justified
• Limitations explicitly acknowledged
• Reproducible: code, data, or protocol available

Adequate (3):

• Design appropriate but validation incomplete
• Limitations mentioned briefly

Weak (1–2):

• Design mismatched to research questions
• No validation or reproducibility information

4. Data Collection & Sources

Strong (4–5):

• Data source clearly described and justified
• Sample size appropriate with power analysis or justification
• Preprocessing steps fully documented

Weak (1–2):

• Data source unclear or inappropriate
• No justification for sample size

5. Analysis Rigor

Strong (4–5):

• Statistical tests appropriate to data type and distribution
• Effect sizes and confidence intervals reported
• Multiple comparison corrections applied where needed
• Assumptions explicitly tested

Weak (1–2):

• Inappropriate tests for data type
• p-values reported without effect sizes or CI
• Assumptions untested

6. Presentation & Scholarly Writing

Strong (4–5):

• Logical structure with clear transitions
• Figures and tables are self-contained with informative captions
• Conclusions follow directly from results
• Appropriate academic register throughout

Weak (1–2):

• Disorganised structure
• Conclusions overclaim beyond the data

7. Contribution & Significance

Strong (4–5):

• Clear articulation of what is new relative to prior work
• Practical or theoretical implications stated
• Limitations of generalisability acknowledged

Scoring Scale Reference

Search Patterns

• “Problem Formulation criteria” → Section 1
• “Literature Review rubric” → Section 2
• “Methodology assessment” → Section 3
• “Data quality indicators” → Section 4
• “Analysis rigor standards” → Section 5
• “Writing quality checklist” → Section 6