AI Function Validator

AI Function Validator is a sandbox for stress-testing LLM tool-calling pipelines. You give it a natural-language task and a registry of allowed functions, and it walks the request through structured JSON tool calls, schema validation, type checking, registry enforcement, and SQL injection detection before executing against a SQLite database. The analytics dashboard then surfaces per-function success rates, schema-violation patterns, and average tool-call latency so you can compare models on real, deterministic tasks instead of vibes.

Highlights

• Multi-layer validation engine: JSON schema, type check, registry enforcement, SQL injection detection
• Function-call analytics dashboard with success rates, latency, and schema-violation breakdowns
• Model-agnostic: works with any OpenAI-compatible endpoint (OpenAI, Ollama, vLLM, etc.)
• Dataset quality scoring: every run produces a reproducibility hash and a quality delta vs. ground truth

Challenges

• Designing a registry DSL that is restrictive enough to catch injection but flexible enough for real-world tool definitions
• Surfacing LLM errors as actionable analytics instead of opaque stack traces

What I learned

• Treat tool calls like database queries: validate, log, and isolate them — never let raw model output touch the DB
• Deterministic validation pipelines make model comparison honest