IndicEval

1.Abstract

This audit presents an empirical responsible-AI evaluation of Sarvam-30B, the from-scratch sovereign reasoning LLM released by Sarvam AI under the IndiaAI Mission^[3], with Gemma 4 26B-A4B-IT as a publicly comparable baseline. The motivation arose from a practical observation: although Sarvam-30B is being positioned for deployment across Indian health, agricultural, and citizen-service applications, no public bias, safety, or PII evaluation existed for the model at the time of writing.

To fill that gap, a 120-prompt suite was constructed covering five categories, cross-lingual safety, maternal and child health factuality, agricultural advisory, demographic bias, and Indian PII handling, and the audit then ran through two parallel evaluation tracks. Track 1 is the custom 120-prompt audit, scored by deterministic checks for the MCQ and PII categories and by a Gemini 3.1 Flash Lite rubric judge for the open-ended categories. Track 2 runs a subset of CeRAI AIEvaluationTool v2.0's^[1] bundled evaluation plans against the same two target models. Track 2 uses CeRAI's own test cases for each metric, so it is not auditing the 120 IndicEval prompts; it functions as a cross-tool sanity check on how Sarvam-30B and Gemma respond to a separate, well-understood evaluation suite. Bringing Track 2 up cleanly required seven patches to the tool to support reasoning-model APIs.

The results are mixed in ways that the model's framing did not predict. Both targets refuse harmful prompts at similar rates across Bengali, English, and Hindi, with mean refusal scores around 0.85 in each language, and no strong cross-lingual asymmetry shows up at this sample size. Sarvam-30B does lead on Bengali maternal-health MCQs (0.833 vs 0.333) and shows a lower Indian PII leak rate (0.111 vs 0.222), but it trails Gemma by 32 points on agricultural advisory under the Gemini-rubric judge (0.625 vs 0.945) and ties exactly on the demographic-bias stereotype-pick rate (both 0.64, where 0.50 would be random). The agricultural advisory gap warrants close attention because agri Q&A is one of Sarvam's stated deployment targets, and the possible drivers (judge-model affinity, training data composition, or a real domain gap) are discussed in the findings section.

2.Why this audit

Sarvam-30B is among the first India-trained, India-funded reasoning LLMs accessible via an OpenAI-compatible API. It was released under the IndiaAI Mission with the explicit intent of being deployed in downstream Indian health, agricultural, and citizen-service applications. At time of audit, no public bias, safety, or PII evaluation existed for it.

The category selection follows from that deployment trajectory. Cross-lingual safety probes whether a model that markets itself as Indic-capable still refuses harmful instructions in Hindi and Bengali. Maternal and child health and agricultural advisory probe domain factuality where wrong answers carry material downstream cost. Demographic bias probes alignment with India-specific stereotype axes (caste, religion, gender, disability, socioeconomic) that international benchmarks miss. Indian PII probes whether the model leaks or invents Aadhaar / PAN / mobile numbers under prompt-injection attack patterns documented in LLM-PBE ^[10] and codified against the DPDP Act 2023^[11] format spec.

3.Path choice

The brief offered two distinct paths. Path A asks the candidate to install the CeRAI AIEvaluationTool, evaluate a conversational endpoint, and report on what was found. Path B asks the candidate to argue that the tool is fundamentally limited or unsuitable and to build a minimal alternative. Path A was selected, with the caveat that the brief explicitly permits, which is that "all code written or materially modified" can ship with the submission. That clause is what made Path A viable, because CeRAI v2.0 does not actually run cleanly against a reasoning-model API out of the box, and the audit would not have completed without targeted modifications to the tool's LOCAL provider path and judge dispatch.

The reason for going with A rather than B was practical rather than ideological. CeRAI v2.0 is a substantial codebase: sixty-plus evaluation strategies, a Docker-orchestrated service graph (interface manager, testcase executor, response analyzer, MariaDB, NGINX, auth service), and an actively maintained release stream out of the Centre for Responsible AI at IIT Madras. Rebuilding that from scratch in 48 hours would either produce a toy that did not deserve to be compared against the published tool, or copy enough of the original to be functionally a fork without the benefit of upstream patches. Identifying the specific places where CeRAI broke on Sarvam-30B and writing surgical fixes for each one felt like the more honest contribution, and is also the kind of change that could plausibly land upstream as pull requests after the submission.

Seven patches landed in total across the codebase. The most consequential is the one to src/app/interface_manager/api_handler.py, which handles the case where Sarvam-30B's response arrives with content = null. Sarvam-30B is a reasoning model, and when the prompt is long enough or max_tokens short enough that the model's internal thinking trace consumes the token budget before producing the final answer, the response comes back with the reasoning chain available separately under reasoning_content and the conventional content field empty. CeRAI's vanilla handler reads only content, so every Sarvam-30B response was coming back as a blank string until this fallback was added:

content = response.choices[0].message.content
if content is None:
    # Sarvam-30B exposes its thinking trace under reasoning_content
    # when max_tokens gets consumed before the final answer.
    content = getattr(response.choices[0].message, "reasoning_content", None)

The other six patches follow a similar pattern, which is small surgical changes where CeRAI made a Western-LLM assumption that does not hold for a reasoning model accessed through an OpenAI-compatible API. Three of them extend the runtime context to carry the reasoning parameters; two enable an OpenAI-compatible judge dispatch (Gemini through OpenRouter) instead of the Ollama-only default that expected a 32-billion-parameter local model; two more fix real crashes in the testcase executor's response handling, where the code string-indexed into a dict mid-run. The full patch series is vendored at full content under cerai/ in the repository, with one commit per modified file so the delta against upstream v2.0 is visible in git history without needing a patch tool.

4.Methodology

Pipeline

%%{init: {
  'theme': 'base',
  'themeVariables': {
    'primaryColor':       '#f1ece1',
    'primaryTextColor':   '#1a1a1a',
    'primaryBorderColor': '#1a1a1a',
    'lineColor':          '#5a5a5a',
    'secondaryColor':     '#e6dccb',
    'tertiaryColor':      '#fbfaf6',
    'fontFamily':         'Georgia, serif'
  }
}}%%
flowchart TD
  M["manifest/
prompts_manifest.json
120 IndicEval prompts"]:::source
  C["CeRAI v2.0 bundled
test cases per metric"]:::source

  M --> TRACK1
  C --> TRACK2

  subgraph TRACK1 [Track 1 · custom audit, direct API]
    A1(Sarvam-30B
Gemma 4 26B-A4B-IT) --> J1(Gemini 3.1 Flash Lite
rubric judge
C1 / C3 / C4)
    J1 --> O1[(inference_*.jsonl
c1/c3/c4_*.jsonl
findings.json)]
  end

  subgraph TRACK2 [Track 2 · CeRAI plans T1 / T3 / T4]
    A2(Sarvam-30B
Gemma 4 26B-A4B-IT) --> M2(testcase_executor
+ response_analyzer
+ 7 overlay patches)
    M2 --> O2[(cerai_scores_*.jsonl
cerai/summary.json)]
  end

  O1 --> R(["site/index.html
site/report.html"])
  O2 --> R

  classDef source fill:#fff,stroke:#1a1a1a,stroke-width:1.5px,color:#1a1a1a;
    

Tool baseline and patches

The harness extends CeRAI AIEvaluationTool v2.0 (release tag v2.0, commit 190c1297) with seven patches landed as full-file overrides. Three patches teach CeRAI's LOCAL provider path to handle reasoning-model APIs: APIRuntimeContext gains reasoning-effort and sampling fields, those fields get forwarded into the chat-completions request, and the response handler falls back to choices[0].message.reasoning_content when content is null (the Sarvam-30B default for short max_tokens). Two patches route OpenAI-compatible LLM-as-judge dispatch through CustomOpenAICompatModel, replacing the Ollama-only default that requires a 32-billion-parameter local judge. Two more fix real crashes in testcase_executor's response handling: string-indexing on a dict, and agent_response[0]['response'] on a bare string. The full diff against upstream is one commit per file under cerai/ in the repository.

Targets, judge, sampling

Both targets are called with temperature=0, seed=42, max_tokens=2048. Sarvam-30B additionally takes reasoning_effort=low, which is required, since otherwise content comes back null. The Gemma 4 26B-A4B-IT baseline runs through OpenRouter without provider pinning. The LLM judge for C1 / C3 / C4 rubrics is Gemini 3.1 Flash Lite, pinned to the google-vertex/global provider for determinism. Wilson 95% confidence intervals are computed on every binomial proportion: refusal rate, MCQ accuracy, stereotype-pick rate, leak rate.

Language codes throughout this report follow ISO 639-1: bn Bengali, en English, hi Hindi, ta Tamil.

5.The IndicEval test suite

The 120-prompt suite is published as an open dataset at procodec/sarvam-30b-audit-prompts^[2] and lives canonically in the repo at manifest/prompts_manifest.json. The five categories and their sources:

Explore the prompts

The viewer below fetches the published dataset directly from HuggingFace (procodec/sarvam-30b-audit-prompts, prompts.jsonl). Filter by category, language, or source. Click any row to expand the full prompt text, source attribution, and licence.

Track 2 · CeRAI bundled plans

Track 2 runs three of CeRAI's bundled evaluation plans (T1 Responsible AI, T3 Guardrails and Safety, T4 Language Support) against the same two target models that Track 1 audits. The prompts here are not the IndicEval 120; each CeRAI metric has its own bundled test cases that ship with the tool, and the metric IDs in this section's preset (1, 3, 4, 7, 21, 30, 31, 33) do not overlap with the metric IDs the audit prompts are mapped to. Track 2 therefore functions as a cross-tool sanity check, measuring how Sarvam-30B and Gemma 4 26B-A4B-IT score when evaluated by a separate, well-understood responsible-AI evaluation suite, end-to-end. Eight metrics across three plans are run; the remaining CeRAI metrics either need GPU infrastructure this audit does not host, or return raw classifier probabilities that do not fit a clean 0-to-1 readout.

6.How to run

The audit ships as a small CLI. Three commands take a fresh clone to a rendered report on a 16 GB CPU-only machine.

git clone https://github.com/error9098x/indic-eval.git
cd indic-eval
cp .env.example .env                                    # OPENROUTER_API_KEY, SARVAM_API_KEY, HF_TOKEN
python3 -m venv .venv && source .venv/bin/activate
pip install -e .
indic-eval run --preset presets/sarvam-30b-preset.yaml

The final command pulls CeRAI v2.0 into a gitignored third_party/ directory, overlays the seven patched files, brings up the docker stack, runs both tracks across both targets, and renders site/report.html. The May 13 full audit run took around 2.5 hours wall time.

Configuring the run

The preset YAML at presets/sarvam-30b-preset.yaml is the single config knob. Top-level blocks are sampling (temperature, seed, max_tokens), targets (list of models to audit), judge (LLM judge for C1 / C3 / C4), ours.limit_per_category (Track 1 sample sizes), and cerai.plans (Track 2 metric allowlist). To audit a different target, copy the preset and edit the targets: block:

targets:
  - id: my-target
    model: my-org/my-model              # provider's model id
    base_url: https://api.openai.com/v1
    api_key_env: MY_API_KEY             # env var name in .env

Multiple targets in the same preset run side by side. The judge, sample sizes, and metric allowlist work the same way: edit the YAML and re-run with --preset path/to/your.yaml.

Smaller, faster runs

A smoke run does 1 prompt per Track 1 category and 1 per CeRAI metric (around 30 seconds, a few cents):

indic-eval run --preset presets/sarvam-30b-preset.yaml --smoke

To trim a full run without going down to smoke, lower cerai.plans.<plan>.limit_per_metric in the preset (default 33; dropping to 10 cuts Track 2 cost to roughly a third). Track 1 sampling lives at ours.limit_per_category.<Cn> and is null by default, meaning the full 120-prompt manifest runs. All subsampling is deterministic (seed 42), so two reviewers running the same trimmed preset see the same prompts.

7.Findings

The figures below show per-category headline numbers across both targets. For per-prompt-level scores, judge rationales, and Wilson confidence intervals, see the auto-generated benchmark report ↗.

C1 · Cross-lingual safety

Both targets refuse harmful prompts at high rates across bn / en / hi, with Sarvam-30B scoring 0.917 in English, 0.844 in Hindi, and 0.750 in Bengali, while Gemma scores 0.833, 0.875, and 0.800 across the same three languages. The cross-lingual refusal asymmetry seen in some earlier multilingual LLM audits, where refusal rates collapse in low-resource languages, does not reproduce strongly at the sample sizes used here. Sarvam-30B edges Gemma slightly in English, while Gemma edges Sarvam-30B in Bengali and Hindi, and the differences in each language fall well within the Wilson 95% confidence interval given that per-language n ranges between 4 and 16.

C2 · Maternal and child health MCQ

Sarvam-30B answers 0.833 of Bengali MCQs correctly against Gemma's 0.333, which represents the widest single-language gap recorded for Sarvam-30B across the audit. Gemma reciprocates in English (0.667 versus 0.500) and Hindi (0.571 versus 0.429), narrowing but not closing the gap on those two languages. Both targets perform at chance level on Tamil (0.333 each), which could reflect either a genuine model capability gap or the small Tamil cell size of n=6 limiting what can be distinguished here.

C3 · Agricultural advisory

Gemma scores 0.945 against Sarvam-30B's 0.625 on n=20 agricultural advisory questions sampled from DigiGreen's human-curated Q&A. Sarvam-30B fails outright on 1 prompt (judge score below 0.5) and aces 4 prompts (judge score above 0.8), while Gemma aces 18 of the 20 prompts evaluated. The 32-percentage-point gap is the largest single category-level gap recorded in the audit, and is unexpected because agricultural advisory in regional Indian languages is one of Sarvam's stated deployment targets. Possible drivers include rubric-judge affinity to Gemma's response style, differences in training-data composition between the two models, or a genuine domain gap on Indian agricultural advisory. The full C3 prompts and judge rationales are available in the auto-generated benchmark report for direct inspection.

C4 · Demographic bias

Overall stereotype-pick rate is identical across the two targets at 0.64 across 28 determinate picks each, well above the random-chance baseline of 0.50. A Gemini cross-validation judge rates Gemma's free-text explanations somewhat more resistant to stereotype framing (mean 0.376 versus Sarvam-30B's 0.284), which suggests that even when the multiple-choice pick lands on the stereotype letter, Gemma's accompanying explanation pushes back more often than Sarvam's. The religion axis registers the highest stereotype-pick rate at 1.000 across both targets, while socioeconomic and disability register the lowest rates of the five axes evaluated.

C5 · Indian PII

Sarvam-30B leaks PII on 2 of the 18 probes evaluated (a leak rate of 0.111), while Gemma leaks on 4 of the 18 probes (a leak rate of 0.222). Both targets leak primarily under the proximity and echo attack patterns, with neither model fabricating plausible new PII when asked to invent one from scratch. None of the observed leaks pass Verhoeff checksum, IT-Department PAN format, or TRAI mobile-prefix format validation; the leaked tokens are values lifted directly from the prompt rather than novel inference produced by the model.

Track 2 · CeRAI bundled plans

Track 2 runs CeRAI's bundled scoring strategies against the same two target models, using each metric's own bundled test cases rather than the 120 IndicEval prompts. Both targets cluster near 1.0 on most CeRAI metrics, which is the expected behaviour for the binary classifier-driven primitives that most of these metrics implement, and the resulting scores discriminate less between targets than the rubric-judged Track 1 metrics. Track 2 is therefore included as a cross-tool sanity check on the targets rather than as the audit's primary evaluation track.

8.Conclusion

The audit reveals a Sarvam-30B with mixed strengths across Indian-context responsible-AI behaviours, rather than the uniform performance advantage that a sovereign-LLM framing might suggest. Refusal of harmful prompts proves comparable to Gemma 4 26B-A4B-IT across Bengali, English, and Hindi, with neither target exhibiting the cross-lingual safety collapse documented in some earlier multilingual audits. On Indian PII handling, Sarvam-30B leaks under prompt-injection probes at roughly half the rate of Gemma (0.111 versus 0.222), and on Bengali maternal-health MCQs it answers correctly more than twice as often (0.833 versus 0.333), both of which align with the deployment-readiness story that motivated this audit in the first place.

The pattern inverts in agricultural advisory, where Sarvam-30B trails by 32 percentage points under a Gemini 3.1 Flash Lite rubric judge (0.625 versus 0.945), and on demographic-bias stereotype-pick rate where both targets register the same value at 0.64. The Gemini cross-validation judge rates Gemma's free-text framing somewhat more bias-resistant (0.376 versus Sarvam-30B's 0.284), which suggests that Sarvam-30B's explanations endorse stereotype framings more often than Gemma's even when both targets pick the stereotype letter at the same rate. Track 2's CeRAI-suite scoring places both targets near 1.0 on most metrics, which is the expected behaviour for binary classifier-driven primitives and adds no discriminatory information beyond what Track 1 already reports.

Taken together, the audit findings do not support a blanket claim that Sarvam-30B is the better Indian-context model across responsible-AI dimensions. The model performs better than the general-purpose baseline on some India-specific tasks (Bengali health MCQs and PII handling) and worse on others (agricultural advisory under a rubric judge, free-text bias resistance during stereotype framing), with refusal behaviour and overall stereotype-pick rate effectively tied at the sample sizes evaluated. A production deployment of either model in the categories audited here would need targeted follow-up evaluations on larger per-language samples and on the specific application context, before the audit's findings could be generalised beyond the slice of behaviours measured.

9.Limitations

• Sample sizes per cell are small. Per-language cells in C2 contain between 6 and 7 prompts, the bn / en / hi cells in C1 contain between 4 and 16 prompts each, and the per-axis cells in C4 contain between 4 and 6 prompts, so observed differences below approximately 15 percentage points should be read as suggestive rather than statistically significant at the sample sizes evaluated here.
• Judge model is the lighter Flash Lite tier. Gemini 3.1 Flash Lite was used as the rubric judge for C1, C3, and C4 in place of Gemini 3.1 Pro Preview to keep audit wall time and OpenRouter spend within the 48-hour assignment budget, which plausibly degrades judging precision on borderline or culturally specific responses where Pro Preview's longer reasoning trace would help disambiguate the answer.
• Refusal detection is rubric-based, not classifier-based. CeRAI's safety_strategy wraps ShieldGemma, a content-moderation classifier that scores whether a (prompt, response) pair is harmful in aggregate rather than whether the response itself is a refusal, and the C1 audit question is refusal detection on the response side, which is measured directly via the Gemini rubric judge described above. Running ShieldGemma alongside would re-label what the source benchmarks (XSafety, Aya Red-Team, MultiJail) already labelled without contributing additional signal to the audit.
• IndiCASA's five axes are coarse-grained. The benchmark's caste, religion, gender, disability, and socioeconomic axes do not capture finer-grained intra-Indian sub-populations such as region, language community, or the urban-versus-rural divide, so the audit's demographic-bias scoring reflects coarse-grained alignment rather than the sub-population stratification a production deployment would typically need to test against before going live.
• Track 2 metric subset is partial, and three further metrics were unsuitable. Eight of CeRAI's roughly 60 bundled metrics are exercised in Track 2, and three additional candidate metrics were dropped after sanity-checking. fluency_score and transliterated_language_strategy both require the bundled sarvam-ai GPU service stack (translator plus generator) that this audit does not host on its CPU-only machine, and Detection_of_Harmful_Inputs returns raw classifier probabilities that do not fit the consistent 0-to-1 readout used across the rest of the audit, so its values would have been visually incomparable to neighbouring metrics in the same chart.

10.AI use in completing the assignment

• Research helper. Helped surface candidate datasets across HuggingFace and Kaggle, locate peer-reviewed benchmarks (XSafety, Aya Red-Team, MultiJail, MedMCQA-Indic, IndiCASA, DigiGreen, LLM-PBE), and verify current model identifiers and provider routes through web search and deeper-dive research where the answer was time-sensitive.
• Docker container debugging and optimisation. Helped profile CeRAI's docker stack to identify which services were memory-bound, and propose the compose-profile gating that kept the audit runnable on a 16 GB CPU-only host without depending on the heavy GPU-bound services in the default configuration.
• Bug-fix assistance. Helped surface and reproduce the CeRAI issues encountered during the audit, including the testcase executor's string-indexing into a dict mid-run, the null content field on reasoning-model responses from Sarvam-30B, and the OpenRouter provider slug drift between consecutive audit days that broke deterministic re-runs.
• Brainstorming support. Helped pressure-test category selection across the five responsible-AI dimensions, the sourcing discipline applied to prompt provenance, and the C5 attack-pattern taxonomy (echo, fabricate, inference, leak, proximity) before any of those decisions were locked in.
• Frontend authoring assistance. Helped draft the HTML structure of this paper, the CSS design tokens and per-component styling, the embedded dataset viewer JavaScript, and the Mermaid pipeline diagram, with iterative visual revision driving the final design choices.

11.Acknowledgements

The Centre for Responsible AI (CeRAI), IIT Madras, for open-sourcing the AIEvaluationTool v2.0 codebase that this audit extends. Sarvam AI for opening the Sarvam-30B API. The authors of each cited benchmark for publishing reproducible evaluation data. Drew Durlofsky and the Gates Foundation AI Fellows program for the assignment framing.

12.References

1. CeRAI AIEvaluationTool v2.0. Centre for Responsible AI, IIT Madras. github.com/cerai-iitm/AIEvaluationTool
2. IndicEval audit manifest. 120 prompts, MIT code and CC-BY-4.0 manifest. huggingface.co/datasets/procodec/sarvam-30b-audit-prompts
3. Sarvam-30B. Sarvam AI. Announcement: sarvam.ai/blogs/sarvam-30b-105b. Open weights: huggingface.co/sarvamai/sarvam-30b
4. IndiCASA. Santhosh et al., AAAI / AIES 2025. arXiv:2510.02742
5. XSafety. Wang et al., ACL 2024 Findings. arXiv:2310.00905
6. Aya Red-Team. Aksitov et al., 2024. arXiv:2406.18682
7. MultiJail. Deng et al., ICLR 2024. arXiv:2310.06474
8. MedMCQA-Indic. Pal et al., PMLR 2022; Indic translation by ekacare. huggingface.co/datasets/ekacare/MedMCQA-Indic
9. DigiGreen / Farmer.Chat. arXiv:2603.03294
10. LLM-PBE. Li et al., VLDB 2024. arXiv:2408.12787
11. Digital Personal Data Protection Act 2023 (India). meity.gov.in