How we watch Nehemvix

We can follow every guide from your request to each model call and see what it cost, where it slowed down and why it failed. Your code, your prompts and your key are never part of that record.

What we record for each guide

Every model call a guide makes is written to our own database as it happens. We keep which provider and model answered, the tokens in and out, the cost, how long the call took, any wait for rate-limit room, how many attempts it needed and, when it failed, what kind of failure it was. Every tool that runs as code, such as fetching the commit or removing secrets, is recorded with how long it took and whether it worked. Each guide also keeps its progress through the stages, any pause and the reason for it, and its grounding score.

That is enough to answer the questions that matter when you run AI in production: what did this guide cost, where did the time go, and which provider is failing today. None of it is sent to an outside monitoring service.

Traces, when we switch them on

For a closer look, we switch on OpenTelemetry traces for 4, 24 or 48 hours, for a demonstration or an investigation. They are off the rest of the time and switch themselves off when the period ends.

A traced guide shows as one tree: your request, then each stage, and inside each stage every model call and every tool that runs as code, with tokens and cost on each call. We view them in our own Arize Phoenix. If you run Nehemvix yourself, you can send them to Phoenix, LangSmith, Langfuse or any OTLP backend.

What is never recorded

Traces hold names, ids, counts, costs and times. They never hold your code, prompts, answers, keys or e-mail addresses, and we test that the hard way: a live test runs a whole guide into a real Arize Phoenix and fails if code, prompt text, a credential or an e-mail address appears anywhere in it.

Your model key is never stored at all, and a Git access token is used for one run and dropped. Every log line also passes through a filter that removes keys, tokens and passwords before it is written. What we do keep, such as your account and the guides you make, is listed on the privacy page.

If something goes wrong

When a request fails on our side, the message ends with a short reference, such as Something went wrong (ref ab12cd34). Send that reference to support@aelsha.com with a line on what you were doing. Every log line written for your request carries the same random id, so the reference takes us straight to what happened. It says nothing about you.

Signals, objectives and alerts

An admin dashboard in the app, built from the same database, shows the signals we run Nehemvix by: speed, traffic, errors, load, quality, and the cost of each useful guide. It also counts what the guardrails did, such as secrets removed before a model saw them and text in a repository addressed to AI tools.

We hold the service to two objectives, how many guides finish and how long they take, and each has an error budget. Alerts on the dashboard flag a drop in quality, a spike in failures or cost, slow guides and model prices due for review, each with the next step to take. Only administrators signed in with two-step verification can open it, and it shows counts, models and scores, never repository content, prompts or keys.

Signals like these are where running an agent starts, and we wrote separately about keeping an agent's quality up after release.

Questions the dashboard doesn't answer

Administrators can ask them of the same numbers in a read-only query console, and every query is written to the audit log.