Software developer · Istanbul
I'm Mert Efe Kurt. I build deterministic, CI-friendly Python tooling for LLM evaluation, agent reliability, and RAG quality — checks that run where your data lives and fail loudly when something regresses.
Selected work
Each one runs locally, needs no API keys, and returns exit codes a CI pipeline can act on. Python standard library at runtime — nothing to break at 2 a.m.
Finds the regressions your averages are hiding.
Pairs baseline and candidate evaluation records by ID, then slices the deltas by any metadata field — language, tier, topic. Seeded bootstrap confidence intervals decide what counts as a real regression, and the gate fails the build when one shows up.
Static analysis for agent tool-call traces.
Inspects recorded agent conversations for protocol violations, malformed arguments, leaked secrets, risky shell commands, call loops, and blown latency budgets — before they surface as production incidents. Every rule is deterministic and explainable.
Do the citations actually support the claims?
Splits RAG answers into claims, resolves their inline citations, and scores each claim against the chunks it cites using explainable content-token recall. Uncited claims, invalid references, and fabricated quotes all get flagged — no model calls, no network.
Contract tests for MCP servers.
Define expected tool behavior in a JSON scenario; mcp-probe spawns the stdio server, validates the handshake and advertised schemas, runs assertion-based tool calls with latency budgets, and reports in terminal, JSON, or JUnit — ready for any CI pipeline.
Also on GitHub
Audits Markdown links, anchors, and external URLs across documentation-heavy repos. Concurrent checks, CI-friendly exit codes.
Groups JSONL logs by trace, surfaces slow requests, and exports incident evidence — without shipping data to a SaaS.
A compact FastAPI backend with JWT auth, role management, and clear validation boundaries. SQLAlchemy underneath.
Approach
Four constraints I hold every tool to — they're what make the results worth trusting.
No API keys, no network calls, no data leaving the machine. Evaluation data is often the most sensitive thing a team owns — tools should come to it, not the other way around.
Same input, same output, same exit code — every run. Seeded resampling, explainable scoring, and no "the linter felt different today." If a check can't explain itself, it doesn't ship.
Exit codes that gate merges. SARIF for code scanning, JUnit for test dashboards, JSON for everything else. A report nobody reads is decoration; a failing pipeline gets fixed.
Python standard library at runtime, minimal dependencies elsewhere. Fewer things to install, fewer things to patch, and nothing that quietly breaks when an SDK bumps a major version.
Stack
Experience
Owned database structure, SQL performance, and automation for operational workflows.
Studying where software, data, and organizations meet — while shipping open-source tooling on the side.
Contact
I'm open to backend, tooling, and AI-evaluation work — and always happy to talk about making systems easier to inspect.