Find the lowest-cost model that meets your requirements

Shoreheld Lab tests open models on your own tasks and reports quality, speed and cost per task beside the model you use now. Then we size the capacity the winner needs.

How an evaluation works

  1. You share a sample of real tasks and tell us how you judge a good result.
  2. We run each candidate model, and the model you use now, on the same tasks.
  3. You get the pass rate, response time and cost per successful task for each one.
  4. We find the smallest set-up that still passes: model size, precision and number of GPUs.

The smaller the model that passes, the less you pay to run it.

What an evaluation reportsAn illustration, not real data. Four models are tested on your tasks: the model you use now and three open models. Each gets a quality score against your pass mark, a speed, and a cost per successful task. Open C is fastest and cheapest but fails the pass mark. Open B passes, runs faster than the current model and costs a third as much, so it is chosen.QualitySpeedCostCurrentOpen AOpen BChosenOpen CFailsPass markIllustration only. Cost is per taskthat passes. The Lab reports yours.

What the Lab does

ServiceWhat you get
Evaluation on your tasksA test set built from your own work, run on each candidate beside the model you use now. You get quality, response time and cost per successful task, and you keep the test.
SizingWe replay your real traffic to measure peak demand, then work out how many GPUs you need and how full they will run.
Model fittingWe find the smallest model, at the lowest precision, that still passes your test, then build that version and test it again.
Routing designWe write the gateway policy that decides which requests stay on open models, as a file your security team owns.
Security engineeringA threat model and a red-team suite for your set-up, with each gateway control mapped to the rules you answer to.

What you keep

Every result comes with the test that produced it, and you keep the test to rerun when models change.

We build on open-source software. What we build for you runs in your account or on your capacity, and you can run it without us.

Discuss an evaluation

Tell us the work you want to move, how you judge it today and the budget you have for it.