How to Evaluate a Plant Operating System: A Buyer's Checklist
DBR77 Team7 min read
On this page
Evaluate a plant operating system by running your own shop-floor scenarios through it, not by comparing feature lists. Ask each vendor to show one real event, such as a machine stop at night, as it moves through production, maintenance, quality and the warehouse to a closed record. Then check the data model, integration, security, AI controls and full cost in writing.
Every vendor says it covers MES, CMMS, QMS and WMS. The difference is how the modules connect and what happens when nobody acts.
Why feature lists mislead
A feature list answers "can the software do X?", and the answer is almost always yes. It skips the questions that decide whether a plant runs better a year later:
- Does a stop in production reach maintenance without retyping?
- Does a failed inspection block the right lot in the warehouse?
- Does a task have one owner, a deadline and a check that it was done?
- Can a shift lead see everything open on a line at handover?
These are questions about flow between modules. A system can tick every box and still leave each department with its own record, so test how events move between them. If the plant has not yet decided which current tools to keep, sort that out first with How to Unify MES, WMS, QMS and CMMS Without Replacing Everything.
- Red flag
- We sync every night
- Good answer
- One ID per object, shown live in the demo
- Red flag
- Slides or a recorded video only
- Good answer
- Done live, in a few clicks, no retyping
- Red flag
- Tasks are just notifications
- Good answer
- Named owner, deadline, escalation, closure check
A week before the demo, send three real scenarios and ask to see them live.
Eight criteria and the questions to ask
| Criterion | What to ask | Good answer | Red flag |
|---|---|---|---|
| Data model | Do all modules share one record of machines, orders, lots and people? | One ID per object, shown live in the demo | "We sync every night" |
| Cross-module events | Show a stop becoming a work order and a quality check | Done live, in a few clicks, no retyping | Slides or a recorded video only |
| Task ownership | Who owns each task, and what happens if nobody acts? | Named owner, deadline, escalation, closure check | Tasks are just notifications |
| KPI definitions | How are OEE, scrap and downtime defined and calculated? | Written formulas, the same for every department | Each dashboard calculates its own |
| Integration | How do you connect our ERP and the tools we keep? | Open interfaces and a named integration scope | Everything needs custom work later |
| Security | How do you align with OT security standards? | Clear answers on network, access and audit logs | No OT security documentation |
| AI controls | Where does a person approve, and what is logged? | Approval rules per decision type, full audit trail | "The AI handles it" |
| Cost and phasing | What does one module on one line cost, and what is extra? | Module price plus a quoted implementation | A single price for "the platform" |
For KPI definitions, ISO 22400 specifies an industry-neutral framework for defining, composing, exchanging, and using key performance indicators for manufacturing operations management. Ask vendors how their formulas compare.
Script the demo with your own scenarios
Do not let the vendor choose the demo path. Send three scenarios from real events a week before, and ask to see them live.
Illustrative example: a plant making painted metal parts sends these three.
- Night-shift stop. At 02:10 the paint line conveyor stops for 25 minutes. Show who is told, how the stop reason is captured, how a work order is opened, and what the morning shift lead sees at 06:00.
- Supplier lot fails incoming inspection. Powder coat lot 4471 fails a gloss check at receiving. Show the hold, how planning learns that the material is not available, and how the supplier gets a corrective action.
- Rush order. Sales adds an order for 800 parts due in two days. Show how the plan changes, which orders move, and who approves the new plan.
Score each scenario on the same points: clicks, systems touched, anything typed twice, and whether the final record shows the full story. Ask a shift lead and a technician to sit in; they spot a weak screen fast.
Questions for IT/OT and security
Plant software touches machines and networks, so IT/OT should run its own review.
The ISA/IEC 62443 series defines requirements and processes for implementing and maintaining electronically secure industrial automation and control systems. It sets requirements for asset owners, service providers and product suppliers, so ask the vendor which parts it follows. NIST SP 800-82 Rev. 3 is a guide to securing operational technology while addressing their unique performance, reliability, and safety requirements.
Ask:
- Where is our data stored, and can we choose the region or run on site?
- How do devices and users authenticate, and is every action logged?
- What happens on the line if the network or the cloud is down?
- How is our data separated from other customers' data?
- If AI is included, where does it run, and does our data train other models?
For AI, the NIST AI Risk Management Framework is a voluntary guide to incorporate trustworthiness considerations into the design, development, use, and evaluation of AI systems. Use it to frame questions on how recommendations are checked and approved; the levels are set out in When AI Should Recommend and When People Should Decide.
How to score and decide
Weight each criterion before the first demo, so the best presenter does not set the weights. A simple split: half the weight on the scripted scenarios, a quarter on integration and security, a quarter on cost and phasing.
Then narrow to two vendors and run a pilot on one line with your own data. A pilot shows what a demo cannot: how your operators actually work, which stop reasons occur and where your data has gaps. Decide in advance which numbers you will compare before and after. How to Build a Business Case for a Plant Operating System shows how to turn those numbers into a case for the budget.
How this works in DBR77 IRIS
DBR77 IRIS runs production, warehouse, quality, maintenance, planning, KPIs and safety as modules on one shared data model, so cross-module events can be shown live in the Atelier Toy demo. IRIS KPI keeps one definition per measure, with targets, thresholds and alert rules, so every department reads the same number.
For the AI controls criterion: the IRIS assistant only reads data and documents, answers within the permissions of the user's role, and takes every number from a tool that read your data. Data & AI recommendations are rule-based and show the basis for their score. A person applies, approves, rejects or defers each one, and IRIS records the outcome. The Compliance screen keeps a register of AI uses and high-risk decisions, and Decision Traces show how a decision was reached. IRIS runs as SaaS with each factory's data kept separate, or in a private cloud or your own data center.
On cost, pricing is per module, so a plant can start with one module and add more later. The monthly price includes hosting, updates and standard support; implementation, custom integrations and premium training are quoted separately. See the IRIS platform and security for details.
FAQ
How many vendors should we shortlist?
Three for the scripted demo, two for a pilot. More, and the team stops comparing like with like.
Who should be on the evaluation team?
Operations, maintenance, quality, IT/OT and finance, plus a shift lead and a technician. Purchasing runs the process.
What if we already have an MES?
Include it. Ask each vendor how it would connect to it, and compare keeping it with replacing it.
Conclusion
Write your three scenarios from real events and send them to each vendor a week before the demo. Weight the criteria first, score every demo on clicks, systems touched and anything typed twice, and let IT/OT run its security review in parallel. Then take two vendors into a one-line pilot and agree in advance which numbers decide it.