18 min read

Building a NEET Question Ingestion and Semantic Validation Pipeline: From Raw CSVs to Model-Ready MCQs

A reliable AI-powered examination platform depends on the quality of the questions it consumes. Before questions can be used for semantic analysis, recommendation, validation, or intelligent examinati...

#Data Analyst#Technical Guide
DBERT Verified Contributor Dossier

Arijit Shaw

Data Analyst Fellow • Data Analyst

Verified Credential: CERT-GE-DBERT-2026-PA-021 →
Source Code on GitHub ↗ ₹5,000 Production Milestone Disbursed

Want to build production projects like Arijit Shaw?

Work 1:1 with practicing Delhi NCR engineers, build verified GitHub codebases, and earn production stipends up to ₹5,000.

Abstract

A reliable AI-powered examination platform depends on the quality of the questions it consumes. Before questions can be used for semantic analysis, recommendation, validation, or intelligent examination workflows, they must be collected, normalized, checked, and stored in a consistent format.

This article explores the design and implementation of a Python-based question ingestion and validation pipeline developed for a NEET question bank. The system processes multiple CSV files containing previous-year questions, standardizes question records, validates mandatory fields, identifies duplicate questions, and prepares a structured question bank for downstream model-based validation.

The implementation combines Python-based data processing, structured validation with Pydantic, duplicate detection, and semantic similarity using Sentence Transformers. It also introduces an optional AI-assisted judging stage to extend validation beyond basic structural checks.

The central objective is to transform heterogeneous question data into a consistent, validated, and reusable dataset that can support intelligent examination systems.


1. Introduction

Competitive examinations such as the National Eligibility cum Entrance Test (NEET) require large and well-organized question banks. These question banks may contain questions collected from multiple years, examination papers, and sources. Although the questions may be available in CSV format, their structures and data quality are not always consistent.

A raw dataset can contain:

  • Inconsistent question formatting.
  • Different representations of correct answers.
  • Missing options or explanations.
  • Inconsistent subject and chapter metadata.
  • Duplicate or near-duplicate questions.
  • Variations in mathematical expressions and special characters.
  • Records that do not follow the expected question schema.

Directly feeding such data into an AI model can reduce reliability and introduce incorrect or misleading information.

To address these challenges, a dedicated ingestion and validation pipeline was developed. It acts as an intermediate processing layer between raw question datasets and the downstream examination system.

The pipeline has two major responsibilities:

  1. Question ingestion: Collecting, parsing, normalizing, validating, and storing questions in a consistent structure.
  2. Question validation: Examining processed questions using structured checks and semantic similarity techniques, with an optional AI-assisted judging stage.

2. Project Objectives

The primary goal is to create a reusable pipeline that can process questions from multiple CSV files without requiring every dataset to follow an identical internal structure.

The specific objectives are:

  1. Read question data from multiple CSV files.
  2. Extract question statements, options, answers, and metadata.
  3. Normalize records into a common MCQ format.
  4. Validate the presence of essential question fields.
  5. Identify duplicate questions and avoid unnecessary repetition.
  6. Generate a structured JSON question bank.
  7. Use semantic embeddings to identify potentially similar questions.
  8. Provide an optional AI-assisted validation stage for further examination of candidate questions.
  9. Maintain a modular architecture so that parsing, normalization, importing, and validation can be maintained independently.

The pipeline is designed to be reusable as the question bank grows.


3. Technology Stack

The implementation uses Python and several libraries for data processing, schema validation, and semantic analysis.

TechnologyPurpose
PythonCore programming language and pipeline implementation
CSVReading source question datasets
JSONStoring normalized and processed questions
PydanticDefining and enforcing structured question schemas
NumPyNumerical operations on embedding vectors
Sentence TransformersGenerating semantic embeddings
all-MiniLM-L6-v2Sentence embedding model used for similarity analysis
pathlibCross-platform file and directory handling
loggingRecording validation and processing information
argparseSupporting command-line execution options

The architecture separates data processing from semantic validation, allowing each component to be developed and tested independently.


4. System Architecture

The project follows a modular, multi-stage architecture.

TEXT
             RAW QUESTION DATA
                     |
                     v
              CSV DATASETS
                     |
                     v
             QUESTION PARSER
                     |
                     v
           QUESTION NORMALIZER
                     |
                     v
            INITIAL VALIDATION
                     |
                     v
           NORMALIZED JSON FILE
                     |
                     v
             IMPORT PROCESSOR
                     |
          +----------+----------+
          |                     |
          v                     v
    FIELD VALIDATION      DUPLICATE CHECK
          |                     |
          +----------+----------+
                     |
                     v
             PROCESSED JSON
                     |
                     v
             QUESTION VALIDATOR
                     |
          +----------+----------+
          |                     |
          v                     v
    STRUCTURED CHECKS     SEMANTIC ANALYSIS
          |                     |
          +----------+----------+
                     |
                     v
             VALIDATION RESULT
                     |
                     v
          OPTIONAL AI JUDGING STAGE

Each stage has a specific responsibility. This modular separation makes it easier to identify processing errors, update individual components, and extend the pipeline without rewriting the entire system.

4.1 Project Structure

The main implementation is organized as follows:

TEXT
DBERT_NEET/
│
├── data/
│   └── Question_Bank/
│       ├── normalized/
│       │   └── questions_normalized.json
│       │
│       └── processed/
│           └── questions.json
│
├── dbert/
│   └── neet_pyqs_csv/
│       ├── AIPMT_2014.csv
│       ├── NEET_2016.csv
│       ├── NEET_2017.csv
│       └── ...
│
├── ingestion/
│   ├── __init__.py
│   ├── question_parser.py
│   ├── question_normalizer.py
│   ├── process_questions.py
│   ├── import_questions.py
│   ├── main.py
│   └── WORKFLOW.md
│
└── validator.py

The architecture avoids mixing every operation into one large Python file.

The ingestion package handles the transformation of raw datasets into structured question records. The root-level validator.py handles subsequent validation and semantic analysis.


5. Question Ingestion Pipeline

The ingestion pipeline is responsible for converting raw CSV records into a standardized question bank.

It is divided into several stages.

5.1 CSV Parsing

The first stage reads the available CSV files and extracts the question data.

The parser is implemented in:

TEXT
ingestion/question_parser.py

The datasets contain fields such as:

  • Question statement.
  • Subject.
  • Chapter.
  • Difficulty.
  • Answer.
  • Option A.
  • Option B.
  • Option C.
  • Option D.
  • Examination year.

The parser also identifies the source dataset and records useful information about where a question originated.

A common representation is created regardless of minor differences between source files.

For example, a parsed question may look like:

PYTHON
{
    "text": "Which organelle is the primary site of aerobic respiration?",
    "option_a": "Endoplasmic reticulum",
    "option_b": "Mitochondria",
    "option_c": "Ribosome",
    "option_d": "Golgi apparatus",
    "correct_option": "B",
    "explanation": None,
    "difficulty": "Easy",
    "subject": "Biology",
    "chapter": "Cell Structure",
    "year": 2020,
    "source": "NEET 2020"
}

The exact content varies depending on the original examination paper, but the target structure remains consistent.

5.2 Question Normalization

Raw question records may contain inconsistent whitespace, answer formats, or textual representations.

The normalization module is implemented in:

TEXT
ingestion/question_normalizer.py

It is responsible for standardizing records before they enter the processed question bank.

Key operations include:

  • Cleaning and normalizing text.
  • Standardizing correct-option representations.
  • Normalizing difficulty values.
  • Normalizing examination years.
  • Ensuring consistent field names.
  • Preparing records for validation.

One important challenge is that the answer field may contain the actual answer text rather than a direct option label.

For example:

CODE
Answer: Mitochondria

Option A: Endoplasmic reticulum Option B: Mitochondria Option C: Ribosome Option D: Golgi apparatus ```

The correct answer must be resolved to the corresponding option:

TEXT
Correct option: B

This allows different source answer formats to be converted into a consistent representation.

Normalization is especially important because later validation stages depend on predictable field names and values.

5.3 Initial Question Validation

After normalization, each question is checked for essential information.

The validation functions identify missing mandatory fields such as:

  • Question text.
  • Option A.
  • Option B.
  • Option C.
  • Option D.
  • Correct option.

Questions that do not satisfy these checks are recorded as invalid rather than being treated as complete MCQs.

This stage provides an initial quality filter before the import process.

5.4 Generating the Normalized JSON Dataset

Once parsing and normalization are completed, the records are written to:

TEXT
data/Question_Bank/normalized/questions_normalized.json

JSON provides a convenient structured representation for later processing.

Each question is represented as an individual object, allowing the import process to inspect, validate, and transform records independently.


6. Import Processing and Duplicate Detection

The next stage takes the normalized JSON file and prepares the final processed question bank.

The implementation is located in:

TEXT
ingestion/import_questions.py

It performs three major operations:

  1. Loading normalized questions.
  2. Validating question records.
  3. Identifying and removing duplicate questions.

6.1 Loading Questions

The normalized JSON file is loaded into Python.

Each record is evaluated independently so that invalid questions can be tracked without stopping the entire import process.

This makes the pipeline more resilient when processing large datasets.

6.2 Field Validation

The import processor checks whether the essential question fields are present.

Valid and invalid questions are collected separately.

The processing logic follows this general pattern:

CODE
valid_questions = []
invalid_questions = []

for question in questions: missing_fields = validate_question(question)

if missing_fields: invalid_questions.append(question) else: valid_questions.append(question) ```

The purpose is to prevent incomplete question records from entering the final processed dataset.

6.3 Duplicate Detection

Large question banks can contain repeated questions across datasets.

Duplicates may occur because:

  • A question appears in multiple source files.
  • The same examination question is collected more than once.
  • Different datasets contain repeated records with similar wording.

The normalizer includes duplicate-detection functionality.

During the current import implementation, questions are grouped by their source before duplicate detection is applied.

This means duplicate removal is performed within each source group rather than globally across every examination year.

This is a deliberate scope distinction: identical questions appearing in different years may remain in the processed bank unless a separate cross-source deduplication policy is introduced.

6.4 Saving the Processed Question Bank

After validation and duplicate removal, the clean records are written to:

TEXT
data/Question_Bank/processed/questions.json

This file acts as the input dataset for the semantic validator.

The ingestion process therefore produces two important artifacts:

OutputPurpose
questions_normalized.jsonStandardized records after parsing and normalization
questions.jsonValidated and duplicate-filtered question bank

7. Structured Question Schema

A consistent question schema is necessary for reliable downstream processing.

The validator uses a Pydantic-based question model to define the expected structure.

The schema contains the following fields:

FieldData typeDescription
textStringMain question statement
option_aStringFirst answer option
option_bStringSecond answer option
option_cStringThird answer option
option_dStringFourth answer option
correct_optionStringCorrect option label
explanationOptional StringExplanation of the answer
difficultyLiteralEasy, Medium, or Hard
subjectOptional StringSubject classification
chapterOptional StringChapter classification
yearOptional IntegerExamination year
sourceStringOrigin of the question

Pydantic provides a structured way to describe the expected data and validate values against the schema.

For example, the correct-option field is constrained to the expected option labels, while difficulty is restricted to the supported categories.

A clearly defined schema reduces inconsistencies between ingestion and validation components.


8. Semantic Question Validation

Basic field validation can determine whether a question contains the required information, but it cannot establish whether two questions have similar meanings.

For example:

Question 1:

Which organelle is responsible for aerobic respiration in a cell?

Question 2:

In which cellular organelle does most ATP synthesis take place?

Although the wording differs, the questions are conceptually related.

Traditional exact-text comparison may fail to recognize this relationship.

To address this limitation, the validator uses semantic embeddings.

8.1 Sentence Transformers

The semantic validation component is implemented in:

TEXT
validator.py

It uses the Sentence Transformers library with:

TEXT
sentence-transformers/all-MiniLM-L6-v2

The model converts question text into numerical vectors called embeddings.

Conceptually:

TEXT
Question Text
     |
     v
Sentence Transformer
     |
     v
Numerical Embedding
     |
     v
Semantic Similarity Comparison

These vectors represent aspects of the textual meaning and allow the system to compare questions based on semantic similarity.

8.2 Generating Question Embeddings

During initialization, the validator loads the processed question bank.

It then extracts the question text and generates embeddings for the bank.

The implementation uses normalized embeddings:

PYTHON
embeddings = model.encode(
    question_texts,
    normalize_embeddings=True,
    show_progress_bar=False
)

Normalized embeddings make cosine similarity calculations straightforward.

The resulting vectors are stored in memory for subsequent comparisons.

8.3 Duplicate Similarity Detection

The validator can encode a candidate question and compare its embedding against the existing question-bank embeddings.

For normalized vectors, the dot product corresponds to cosine similarity:

\[ \operatorname{Similarity}(A,B)=A\cdot B \]

A similarity value closer to 1 indicates stronger semantic similarity.

The validator uses a similarity threshold to identify potentially duplicate questions.

A high similarity score does not automatically prove that two questions are duplicates. Questions from the same topic may naturally share vocabulary or concepts.

Therefore, the similarity threshold should be calibrated using representative questions from the actual dataset.

Semantic similarity is best treated as a method for identifying candidates that deserve further inspection, rather than as a substitute for final correctness verification.

8.4 Embedding Cache

Generating embeddings for an entire question bank can take time, particularly when the validator is executed repeatedly.

To avoid unnecessary recomputation, the implementation uses a NumPy compressed cache:

TEXT
questions.json.emb.npz

The cache stores previously generated embeddings.

When the question bank and cache metadata satisfy the implementation's validity checks, the validator can reuse the cached vectors instead of encoding the entire dataset again.

This improves repeated execution efficiency.

However, the cache must be invalidated or regenerated whenever the underlying question bank or embedding configuration changes.


9. Optional AI-Assisted Judging

Semantic similarity helps detect related questions, but it does not independently establish whether a question is factually correct or whether its answer is valid.

For additional examination, the validator supports an optional judging stage.

The command-line workflow distinguishes between a regular validation run and one that enables the judge.

A regular run can be executed using:

BASH
python3 validator.py

The optional judging mode is enabled with:

BASH
python3 validator.py --judge

Without the judge option, the system can return:

TEXT
Verdict: NEEDS_JUDGE

This indicates that the optional judging stage was skipped and that further evaluation is required.

The validator can also generate a prompt for an answer-blind solver pass.

The purpose of an answer-blind approach is to assess the question independently rather than simply exposing the supplied answer and asking the evaluator to agree with it.

This creates a useful separation between the supplied dataset answer and the independently generated assessment.

The judging stage should be treated as an additional validation layer. It does not replace official answer keys, subject-matter verification, or source-paper inspection.


10. End-to-End Execution Workflow

The ingestion and validation workflow is organized into two main execution stages.

Stage A: Run the ingestion pipeline

From the project root, execute:

BASH
python3 -m ingestion.main

The main module coordinates the ingestion operations:

  1. Start question ingestion.
  2. Process the CSV files.
  3. Parse and normalize questions.
  4. Generate the normalized JSON file.
  5. Import normalized questions.
  6. Validate essential fields.
  7. Remove detected duplicates.
  8. Save the processed question bank.
  9. Print the final processing summary.

Stage B: Run the semantic validator

After ingestion is complete, execute:

BASH
python3 validator.py

The validator:

  1. Loads the processed question bank.
  2. Initializes the embedding model.
  3. Loads or generates question embeddings.
  4. Performs the configured validation operations.
  5. Produces a validation verdict.
  6. Indicates whether optional AI judging is required or was performed.

When the additional judging stage is needed, use:

BASH
python3 validator.py --judge

The two-stage approach keeps dataset preparation separate from downstream semantic validation.


11. Processing Results and Dataset Statistics

During one of the recorded ingestion runs, the pipeline discovered 13 CSV files containing a total of 2,440 question records.

The processing output reported:

MetricRecorded result
CSV files discovered13
Total questions parsed2,440
Invalid questions reported1,289
Duplicate detections reported during processing17
Duplicate records reported as removed in the final import summary10
Final questions written1,141

These figures represent a particular recorded run, not a guaranteed final dataset count after subsequent corrections.

The difference between the duplicate detections reported during CSV processing and the duplicates removed during import reflects the fact that the pipeline has separate duplicate-processing stages and scopes.

The results demonstrate that the pipeline can process multiple source files in a single execution, produce structured output, and report records that fail validation.

More importantly, the invalid-question count highlights why data validation is necessary: a successful CSV import does not automatically mean every question is suitable for model consumption.


12. Challenges and Engineering Considerations

Developing this pipeline involves several challenges that are important for maintaining dataset reliability.

12.1 Inconsistent Source Data

Different examination-year datasets may follow different conventions for answers, metadata, and text formatting.

A common normalization layer is therefore necessary to reduce variation.

12.2 Incorrect or Missing Fields

A question may contain a valid statement but still be unusable because one of its options or the correct answer is missing.

Structural validation helps identify these records before they enter the processed bank.

12.3 Exact Duplicate vs. Semantic Duplicate

Two records can be duplicates even when their text formatting differs.

Conversely, two questions can be semantically similar while testing different concepts or requiring different answers.

For this reason, exact-text matching and semantic similarity should not be treated as interchangeable operations.

12.4 Model Initialization and Embedding Time

Sentence Transformers must load model weights and generate embeddings.

For a large question bank, the first execution may take longer than subsequent runs.

Embedding caching helps reduce this repeated workload.

12.5 Text-Only Semantic Analysis

The current embedding approach operates on question text. It does not fully interpret diagrams, graphs, chemical structures, or image-based answer options.

A text embedding model cannot reliably understand a diagram merely because the question text contains the word picture.

Questions whose meaning depends on visual material require an additional visual-processing strategy or a separate review process.

12.6 Separating Data Quality from Model Validation

The semantic validator can identify textual similarity, but it cannot guarantee that the original question or answer is correct.

The original examination paper and official answer key remain important references when verifying source content.


13. Current Scope and Future Improvements

The existing implementation establishes the foundation for large-scale question ingestion and semantic validation.

Several improvements can make it more reliable and suitable for more advanced AI workflows.

13.1 Stronger Data Quality Validation

The validation process can be extended to check all schema fields consistently, rather than relying only on a minimal set of mandatory-field checks during ingestion.

Additional checks can identify:

  • Empty or meaningless option text.
  • Invalid correct-option labels.
  • Unexpected difficulty values.
  • Invalid year values.
  • Malformed question records.
  • Suspicious placeholder text.

13.2 Improved Duplicate Detection

The similarity threshold should be evaluated against manually reviewed examples.

The system can also be extended with cross-source duplicate detection and subject-aware comparisons.

This would help avoid incorrectly grouping questions from unrelated subjects or treating conceptually related but distinct questions as duplicates.

13.3 More Reliable Embedding Cache Management

A content-based cache key can improve reliability by ensuring that changes in the question bank invalidate stale embeddings.

The cache should also account for the embedding model and relevant encoding configuration.

13.4 Better Validation Logging

The validator should provide clear diagnostic information when model initialization, embedding generation, or cache loading fails.

Silent failure handling makes debugging difficult and can hide problems that affect validation quality.

13.5 Source-Based Question Correction

For records containing corrupted text, missing options, or image placeholders, a separate dataset-repair process can compare the CSV against the original examination-paper PDFs.

This should be considered an upstream data-quality activity, not something semantic similarity can automatically solve.

The PDF should remain the source of truth when reconstructing missing or corrupted question content.

13.6 Multimodal Question Support

Future versions can preserve references to diagrams, graphs, molecular structures, circuit images, and image-based answer options.

A richer question representation could include:

  • Question text.
  • Textual answer options.
  • Mathematical expressions.
  • Chemical formulas and reactions.
  • Image asset references.
  • Visual content metadata.

This would allow the dataset to support multimodal models instead of restricting every question to plain text.


14. Conclusion

A reliable AI examination platform begins with reliable data.

The question ingestion and validation pipeline provides a structured method for transforming multiple raw CSV datasets into a common question-bank format. Through parsing, normalization, field validation, duplicate detection, and JSON storage, it prepares examination questions for downstream processing.

The semantic validation component extends this functionality using Sentence Transformers and the all-MiniLM-L6-v2 embedding model. It allows questions to be compared by semantic similarity rather than relying entirely on exact textual matches. An optional AI-assisted judging stage provides another layer of evaluation when additional assessment is required.

The most important engineering principle behind the system is the separation of responsibilities:

  • Ingestion prepares the data.
  • Normalization creates consistency.
  • Structural validation checks completeness.
  • Duplicate detection reduces repeated records.
  • Semantic analysis identifies potentially related questions.
  • AI-assisted judging provides optional additional evaluation.
  • Source-based verification establishes whether the original question and answer are correct.

Together, these components create a foundation for building a more reliable and scalable NEET question bank that can support intelligent examination workflows and future AI-powered educational applications.

The objective is not simply to import more questions. It is to ensure that the questions entering the system are structured, traceable, and suitable for meaningful downstream model processing.

DBERT Verified Contributor Dossier

Arijit Shaw

Data Analyst Fellow • Data Analyst

Verified Credential: CERT-GE-DBERT-2026-PA-021 →
Source Code on GitHub ↗ ₹5,000 Production Milestone Disbursed

Want to build production projects like Arijit Shaw?

Work 1:1 with practicing Delhi NCR engineers, build verified GitHub codebases, and earn production stipends up to ₹5,000.

🛡️ DBERT Institutional Quality Guarantee • Academic Peer Review

This technical publication represents an authenticated production milestone engineered during the DBERT Incubation Fellowship. The underlying codebase, automated test suites, and system architecture have undergone independent rubric evaluation with Turnitin originality validation (< 3.0% similarity index).

For Aspiring Engineers

Build Production Systems

Join the next cohort, receive 1:1 mentorship from industry staff, and publish verified capstone case studies.

Apply for Fellowship →
For Tech Recruiters & Employers

Hire Pre-Vetted Fellows

Discover engineers with verified code rubrics, live deployment URLs, and zero recruiter markup fees.

Browse Pre-Vetted Directory →