Using a Decision Model to Decide When to Block Supplier Offers

Introduction

Today, the decision “should we block this supplier offer?” is made by a person who looks at the complaint and the sales history. I checked whether that decision can be handed to a small model that is cheap to run – without losing decision quality.

What sparked it was the fact that a decision model built on Bielik – basal – took first place in a benchmark for decision models. I decided to test it in a more business-oriented setting by asking: “Should offer X be switched off because it is faulty?”

What’s inside?

Decision models evaluate the context of a situation, provided as input (evidence), and produce a decision, a probability distribution or a decision score as output. As a rule, decision models decide “here and now”: they have no memory and do not call external tools. They rely entirely on the picture they are given (the context) and on the matrix of possible decisions.

Our case: axe or pardon for a supplier offer

At one of our clients – an online store selling digital products, which buys the same product from many suppliers – one of the problems we deal with is faulty supplier offers. The causes vary: human error in mapping, missing or wrong supplier data (e.g. region), and quite often the batch we receive is simply faulty.

The goal is to check whether a decision model, based on the evidence provided, can decide to take a specific offer off sale.

Show me the evidence!

For the model to make what it considers the best decision, its context needs information. For the question “Should offer X be switched off because it is faulty?”, the features I included in the context are:

Offer
- supplier: name, sub-seller and supplier offer ID
- product name
The complaint that triggers the decision
- date and time received
- reason
- content
Sales of this offer in the last 24h
- units sold
- number of complaints
- complaint rate (%)
The same product from other suppliers in the last 24h
- units sold
- number of complaints
- complaint rate (%)
Day-by-day offer history (last 7 days, days with no activity skipped)
- units sold (completed transactions)
- failed purchases from the supplier
- number of complaints
Earlier complaints about this offer within those 7 days
- count
- for each (max. 8): date, reason and content (up to 400 characters)

The final context sent to the model looks like this (an anonymized example – the supplier, product, complaint text and dates are changed, the structure is 1:1). It is translated here; the model receives it in Polish, as shown in the raw request at the end:

Offer: supplier Supplier B/Seller 07 #OF-10482, product Software Subscription - 3 Months - GLOBAL.



New complaint (2026-10-14 11:20+00:00), reason: invalid. Content: code not working



Sales of this offer in the last 24h: 1 unit, complaints: 1 (100.0%).



Same product from other suppliers in 24h: 0 units, complaints: 1 (no sales).



History of this offer over the last 7 days (days with no activity skipped):



- 2026-10-14: sold 1, failed supplier purchases 0, complaints 1



Earlier complaints about this offer in this period (0):

This set of features gives the model a clear picture of the situation and a comparison with competing offers, especially for “busy” products.

Show me the meat!

Diagram: complaint facts, business context and questions with a cost matrix go into the basal model, which returns a decision distribution, a cause distribution and call metadata — *Solution architecture (labels in Polish). The output on the right comes from the original call, before the data was anonymized.*

For the test, I considered 3 ways of answering the question posed at the start of the article:

calling the model with the context described in the section above,
calling the model without the built context – judging only the complaint itself,
a simple heuristic – 3 complaints in a row within 24h (rolling window) – ban.

Since I was working retrospectively, it was tempting to give the context the full picture, including events that happened after the complaint was filed, but that would not reflect the real situation.

So the test method called the decision model at the moment the complaint came in (t) and fed it historical data from before t. In other words, even though we call the model after the fact, it knows nothing about the future.

The sample consisted of the last 300 complaints, 170 of which concerned an offer with an earlier complaint.

How much does a mistake hurt?

For the model to make the right decision, we need to feed it a so-called loss matrix. It defines how painful specific decisions are for us when the model is wrong or right. It is how the business sets its appetite for risk.

For our scenario, I assumed 3 cost classes:

action	offer actually bad (`"true"`)	offer actually fine (`"false"`)
`block` – Block the offer	0: justified block	3: unnecessary block, we lose sales until it is unblocked
`keep` – Keep the offer on sale	10: a bad offer keeps selling, so refunds, support work and marketplace ratings are our biggest pain	0: rightly kept
`human` – Let a person decide	1: a human review is a minor cost	1: same, a review costs the same regardless of the outcome

Rows are the actions we can take; columns are the true state of the offer, which we do not know at decision time. Each cell says how much we lose if we take that action.

Since the unit is arbitrary, I used a 0–10 scale.

Results

The distribution of decisions (block / keep / human) made by each of the 3 methods across all 300 cases:

Approach	`block`	`human`	`keep`
Model + context	177	121	2
Model without context	218	82	0
3-in-24h rule	115	0	185

As you can see, context clearly tempers how aggressively the model blocks offers, which shows that feature engineering will be the key step in a production rollout.

The simple heuristic is the most lenient – with no analytical component, it cannot make an aggressive decision unless the law of series (in Polish) kicks in.

Hardware

Since the model is self-hosted, I ran it on two machines I had at hand:

Parameter	PC with NVIDIA RTX 3060	MacBook Air M1
CPU	AMD Ryzen 5 8400F	Apple M1, 8 CPU cores (4 performance + 4 efficiency)
GPU	NVIDIA GeForce RTX 3060, 12 GB VRAM	Apple M1, 8 GPU cores (integrated)
RAM	32 GB	16 GB (memory shared by CPU and GPU)
OS	Ubuntu 25.10	macOS 27.0
Model	basal-2.0 4.5B	basal-2.0 4.5B, MLX port
Weight precision	bf16, approx. 9 GB	8-bit, approx. 5 GB
basal engine	2.3.1	2.1.0
Python	3.12	3.12
Average time to evaluate one scenario	~450 ms	~730 ms

As you can see, the model is fast – much faster than any LLM. That keeps evaluation costs relatively low. The model can also be run on well-known online model runners, such as Hugging Face.

Conclusions

This is definitely a direction worth exploring. The deployment cost and the predictability of the solution show promising potential for this way of making decisions when cutting off questionable offers. The key to a rollout will be building good features and wrapping them in context, so that the model has as much knowledge as possible when it makes a decision.

Sources

basal.si5.pl – basal model website
JevBench v1 – benchmarkheaven.com/jev-models – ranking and description of the decision model benchmark (typed decision accuracy, calibration, latency, cost)
fstandhartinger/jevbench on GitHub – JevBench code and methodology
benchmarkheaven/jevbench on Hugging Face – benchmark dataset
rkinas/basal on GitHub – model license and authorship (Remigiusz Kinas, co-creator of Bielik)
Cost-loss model – Wikipedia – theoretical basis for decisions based on a cost matrix
The Foundations of Cost-Sensitive Learning (Elkan) – a classic paper on learning with a cost matrix
Law of series (in Polish)

Raw model request and response

The same anonymized example as above, in the original Polish. The response comes from the original call, before names and text were replaced.

Request – POST http://localhost:8000/v1/systemone:

{



  "state": "Oferta: dostawca Dostawca B/Sprzedawca 07 #OF-10482, produkt Software Subscription - 3 Months - GLOBAL.\nNowa reklamacja (2026-10-14 11:20+00:00), powód: invalid. Treść: code not working\nSprzedaż tej oferty w ostatnich 24h: 1 szt., reklamacji: 1 (100.0%).\nTen sam produkt u innych dostawców w 24h: 0 szt., reklamacji: 1 (brak sprzedaży).\nHistoria tej oferty z ostatnich 7 dni (dni bez ruchu pominięte):\n- 2026-10-14: sprzedane 1, nieudane zakupy u dostawcy 0, reklamacje 1\nWcześniejsze reklamacje tej oferty z tego okresu (0):",



  "questions": {



    "block": {



      "type": "act",



      "instructions": "Czy oferta dostawcy Dostawca B/Sprzedawca 07 #OF-10482 dla produktu Software Subscription - 3 Months - GLOBAL jest wadliwa (dostawca dostarcza niedziałające, użyte lub złe regionowo kody) i trzeba ją zablokować?",



      "criteria": {



        "true": "Tak, problem leży po stronie tego dostawcy",



        "false": "Nie, to pojedynczy przypadek, błąd klienta albo problem całego produktu"



      },



      "costs": {



        "block": { "true": 0, "false": 3 },



        "keep": { "true": 10, "false": 0 },



        "human": { "true": 1, "false": 1 }



      }



    },



    "cause": {



      "type": "choice",



      "instructions": "Jaka jest najbardziej prawdopodobna przyczyna reklamacji?",



      "criteria": {



        "supplier": "Wadliwe kody od tego dostawcy (nieważne, użyte, zły region)",



        "product": "Problem całego produktu, występuje też u innych dostawców",



        "customer": "Błąd klienta (zła platforma, konto, region klienta)",



        "other": "Inne (dostawa, płatność, pytanie)"



      }



    }



  }



}

Response:

{



  "model": "basal-2.0-4.5B",



  "answers": {



    "block": {



      "type": "act",



      "action": "block",



      "expected_costs": {



        "block": 0.880595,



        "keep": 7.064682,



        "human": 1.0



      },



      "answer": "true",



      "probabilities": {



        "true": 0.7064682245254517,



        "false": 0.29353171586990356



      },



      "confidence": 0.7064682245254517,



      "calibration": "validated"



    },



    "cause": {



      "type": "choice",



      "choice": "supplier",



      "probabilities": {



        "supplier": 0.7587137818336487,



        "product": 0.029148070141673088,



        "customer": 0.1687118411064148,



        "other": 0.04342631623148918



      },



      "confidence": 0.7587137818336487



    }



  },



  "usage": {



    "input_tokens": 741,



    "output_tokens": 0,



    "questions": 2,



    "branches": 2,



    "latency_ms": 445.73,



    "longest_prompt_tokens": 383



  }



}