What Is an AI System in the Regulatory Context?

By Harun İncegül
Information Security Manager and Data Protection Officer

AI Governance & Compliance for Practitioners · Part I · Article 1 · Art. 2 & 3 EU AI Act · ISO/IEC 22989 | 42001 | 23894

---

1. The problem: why correct classification is decisive

In corporate practice, the term artificial intelligence is often used loosely. Systems are either prematurely labelled AI or deliberately not classified as such in order to avoid regulatory obligations. With the entry into force of the EU AI Act, that imprecision now carries concrete legal consequences.¹

Classifying a system as an AI system is the operational precondition for any further risk analysis, for the choice of conformity-assessment route and, ultimately, for the liability question if something goes wrong. In practice I encounter two opposing errors. The first is under-classification — a system with genuine inference and learning mechanisms is dismissed as conventional software. The second is over-classification — every automation tool is precautionarily labelled AI, which ties up resources and paralyses compliance processes. Both errors stem from the same cause: an imprecise idea of what counts, in regulatory terms, as AI.

What this means in practice

A misclassification leads either to statutory requirements being missed or to unnecessary compliance measures being implemented. The definition is not a formality — it is the starting point of every regulatory obligation.

---

2. Scope (Art. 2): who falls under the EU AI Act?

Before a system is assessed as an AI system, the prior question is whether the organisation falls within the scope of the regulation at all. Art. 2(1) EU AI Act² addresses this explicitly:

• Providers: anyone who develops an AI system and places it on the market under their own name — irrespective of where the company is established.
• Deployers: anyone who uses an AI system within their own area of responsibility — typically organisations integrating third-party AI solutions.
• Importers and distributors: anyone bringing AI systems in from third countries or further distributing them on the internal market.
• Product manufacturers: anyone placing an AI system on the market together with a product under their own name.
• Affected persons: natural persons in the EU who are affected by an AI system.

Particularly relevant in practice is the extraterritorial reach of the EU AI Act. Art. 2(1)(c) makes it explicit: providers and deployers established outside the EU also fall within the regulation when the output of their AI systems is used in the EU. A US SaaS provider whose AI-driven HR tool is deployed by a German company is therefore on regulatory equal footing with an EU-established provider.²

Caveat — the Open-Source exemption has teeth

Art. 2(12) EU AI Act provides an exemption for open-source AI systems. It does not apply, however, where the system is classified as high-risk, or where it falls under Art. 5 (prohibited practices) or Art. 50 (transparency obligations). Anyone integrating an open-source foundation model into a high-risk application falls fully under the high-risk requirements — irrespective of the open-source licence.

What this means in practice

Scope is not limited to where a company is established. What matters is whether an AI system is used in the EU context or produces effects there. This also captures companies without an EU establishment.

---

3. The definition of an AI system (Art. 3): the core criteria

Art. 3(1) EU AI Act³ defines an AI system as a machine-based system that operates with varying levels of autonomy, may exhibit adaptiveness after deployment, and infers — for explicit or implicit objectives — from the input it receives how to generate outputs such as predictions, content, recommendations or decisions that can influence physical or virtual environments. The definition is deliberately technology-neutral: it does not bind itself to any specific method, framework or architecture.

Four elements are decisive for practical classification:

• Machine-based: the system must run on hardware or software. A human expert giving recommendations is not within scope.
• Varying levels of autonomy: full autonomy is not required. Even a system that merely produces decision proposals can fall under the definition.
• Inference from inputs: the system derives outputs that are not fully determined by explicit programming.
• Influence on physical or virtual environments: the output must have a real or digital effect.

Particularly important is the element of implicit objectives: many AI systems do not optimise on an explicitly coded goal, but on latent patterns derived from training data. This is not an edge case — it is the operating principle of most modern machine-learning models. The EU AI Act definition is built precisely to capture them.³

From practice

In a project automating supplier-rating, the deployed system consisted of two components: a rule-based filter (hard thresholds) and an ML model trained on historical rating data. The rule-based filter is not an AI system — its logic is fully determined in code. The ML model unambiguously is — it derives risk parameters from data in a way that cannot be reconstructed from the code itself. The first operational governance decision was to separate both components and classify them independently.

What this means in practice

A system qualifies as an AI system in particular when its behaviour on new inputs is not fully predictable, because it operates on learned patterns — and not exclusively on explicitly programmed rules.

---

4. Drawing the line: when is a system not an AI system?

Not every data-processing system falls under the definition. ISO/IEC 22989:2022⁴ adds precision: AI systems are characterised by their capacity for generalisation — they draw inferences about unknown cases from known data. Typical non-AI systems include:

• Deterministic rule sets (if-then logic) without a learning component
• Conventional database systems without pattern recognition
• Systems with fully pre-defined decision trees
• Simple statistical analyses (means, sums, percentages)

Feature	Conventional software	AI system (Art. 3 EU AI Act / ISO 22989)
Logic	Deterministic — fully in code	Probabilistic — inference-based from data
Specification	Explicit programming instructions	Learning objectives and training data
Generalisation	None — only known cases	Yes — including unknown cases
Autonomy	None — pure command execution	Variable — discretionary scope present
Testability	Complete (unit / integration tests)	Statistical (validation, not verification)
Governance	Process and code integrity	Data- and model-centric control

What this means in practice

The line is drawn along one simple question: does the system execute what was explicitly programmed — or does it derive its own decision logic from data? In the first case: not an AI system. In the second case: classification as an AI system is highly likely.

---

5. The paradigm shift: from code-centric to model-centric

The distinction between deterministic software and probabilistic AI systems has direct consequences for information-security management. Conventional control mechanisms systematically fall short with AI systems:

• Code review: a static code review does not capture the actual source of risk in an ML model — the model itself and the data it learned from.
• Penetration test: a conventional pen test examines interfaces and infrastructure, but not the adversarial robustness of a model.
• Change management: a process that only captures software changes overlooks model drift — a creeping performance decay without any code change.

For governance structures this means: the existing control framework must be complemented with data- and model-centric controls — not replaced. Art. 9 EU AI Act⁵ requires a risk management system across the entire lifecycle for high-risk AI systems. ISO/IEC 23894:2023⁶ provides the methodological foundation.

AI-specific risk	Description	Operational control (ISO/IEC 42001)
Model drift	Performance decay due to changing data distributions without code change	Continuous monitoring; defined retraining triggers
Algorithmic bias	Discrimination caused by bias in training or validation data	Bias audits; diversified datasets; impact analysis
Hallucinations (generative AI)	Factually incorrect but plausibly worded outputs	Human-in-the-loop (HITL) for decision-relevant outputs
Adversarial attacks	Targeted manipulation of the model through crafted inputs	AI red teaming; robustness testing

A note on RAG architectures

RAG architectures (retrieval-augmented generation) reduce hallucinations — they do not eliminate them. Anyone treating RAG as a complete control underestimates the residual risk. HITL controls remain mandatory for decision-relevant outputs — irrespective of the architecture deployed.⁷

What this means in practice

Conventional governance instruments remain necessary — but no longer sufficient. AI systems additionally require model-centric controls: monitoring of model performance, bias analysis of training data, adversarial robustness testing.

---

6. ISO/IEC 42001 as an organisational integration framework

Art. 17 EU AI Act⁸ requires providers of high-risk AI systems to operate a quality management system. ISO/IEC 42001:2023⁹ — the first internationally recognised management system for AI (AIMS) — addresses precisely this requirement. Three components are immediately relevant:

• AI policy: ethical and operational guardrails for the entire AI deployment within the organisation.
• Data governance: control over the quality, provenance and lawfulness of training, validation and test data.
• Lifecycle management: documentation from design through operation and monitoring to decommissioning.

ISO/IEC 42001 does not prescribe a specific maturity level — the standard is scalable. What matters is the demonstrable systematics of the approach, not a bureaucratic completeness claim.¹⁰

What this means in practice

Certification under ISO/IEC 42001 is not an automatic proof of EU AI Act conformity. It can, however, serve as structured evidence of organisational maturity towards supervisory authorities and customers — and considerably reduces the effort required during conformity assessment.

---

7. Three operational test questions for first-pass classification

Building on Art. 3(1) EU AI Act, three test questions can be derived as a structured entry point for any classification decision:¹¹

• Question 1 — Inference: does the system derive outputs that cannot be fully reconstructed from the source code? If yes: AI-system suspicion.
• Question 2 — Training data: has the system learned its parameters or decision boundaries from data? If yes: suspicion confirmed.
• Question 3 — Autonomy: does the system make assessments or recommendations whose concrete derivation has not been deterministically prescribed? If yes: classification as an AI system under the EU AI Act.

These three questions are not a legal opinion — they are the documented entry into a structured analysis. The result must be set down in writing. In case of doubt, a legal review is unavoidable.

What this means in practice

The classification decision must be documented and reasoned. It is the starting point of every further compliance measure and must be reassessed in the case of substantial system changes (Art. 3(23) EU AI Act: substantial modification).

---

8. Regulatory consequences: why the definition cannot be viewed in isolation

Classification as an AI system is not an end in itself. It determines whether — and to what extent — regulatory requirements apply. Under the EU AI Act, further assessment follows a risk-based approach:

• Prohibition of certain applications (Art. 5): specific AI practices are forbidden without exception.
• High-risk classification (Art. 6, Annex III): extensive requirements for documentation, risk management and oversight.
• Transparency obligations (Art. 50): labelling and disclosure obligations vis-à-vis users.

What this means in practice

The definition of an AI system is the first step of a multi-stage process that flows directly into concrete compliance obligations. This article lays the foundation — the subsequent articles in this book build on it systematically.

---

9. Conclusion: classification is not a formality

An AI system in the regulatory context is a system whose behaviour is determined not exclusively by code but materially by data and inference. The EU AI Act definition is functional and technology-neutral — it forces analysis of the system, not its label.

For organisations, three central tasks follow:

• Clarify scope: does the organisation — as provider, deployer or importer — fall under the EU AI Act?
• Clean classification: is the system actually an AI system within the meaning of the regulation? Documented and reasoned.
• Derive obligations: which regulatory requirements follow from this classification? Risk-based and proportionate.

Only when definition, scope and regulatory consequences are considered together does a robust foundation for governance and compliance emerge. Classification is the first step — and as with any first step, it determines the direction of every subsequent one.

---

1. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 (EU AI Act), OJ L 2024/1689. Art. 1 (subject matter) and Recital 1. ↩

2. EU AI Act, Art. 2(1)(a)–(g): scope. Art. 2(1)(c): extraterritorial effect. Art. 2(12): open-source exemption with limitations. ↩ ↩²

3. EU AI Act, Art. 3(1): definition of an AI system. Recital 12: distinction from rule-based systems. OECD (2019, rev. 2023): Recommendation on Artificial Intelligence. OECD/LEGAL/0449. ↩ ↩²

4. ISO/IEC 22989:2022: Artificial intelligence — Concepts and terminology. Geneva: ISO. Sections 3.1 (AI system), 3.4 (machine learning), 3.6 (generalisation). ↩

5. EU AI Act, Art. 9: risk management system. Requires a continuous, iterative process across the entire lifecycle of high-risk AI systems. ↩

6. ISO/IEC 23894:2023: Information technology — Artificial intelligence — Guidance on risk management. Geneva: ISO. ↩

7. Hacker, P., Engel, A. & Mauer, M. (2023): Regulating ChatGPT and other Large Language Models. ACM FAccT 2023, pp. 1112–1123. ↩

8. EU AI Act, Art. 17: quality management system for providers of high-risk AI systems. ↩

9. ISO/IEC 42001:2023: Information technology — Artificial intelligence — Management system. Geneva: ISO. Compatible with ISO 27001 and ISO 9001. ↩

10. European Commission (2021): Impact Assessment Report accompanying the AI Act Proposal. SWD(2021) 84 final. Section 3.3: proportionality for SMEs. ↩

11. BSI (2023): KI-Systeme kompetent einschätzen und sicher nutzen. Bonn: BSI. Recommends documented classification decisions as a governance obligation. ↩