Towards Trustworthy AI-Assisted Software Engineering

Abstract

Large language models (LLMs) are increasingly integrated into software engineering activities including code generation, testing, debugging, documentation, and security analysis. Although these capabilities can improve development speed and reduce implementation effort, they also introduce challenges concerning correctness, security, verification, reproducibility, human oversight, and accountability. This paper examines the question: How should humans, AI assistants, testing, and verification be combined to produce trustworthy software? A synthesis of recent literature suggests that trustworthy AI-assisted software engineering cannot be achieved by treating an LLM as an autonomous programmer or by relying on the apparent correctness of generated code. Instead, generated code should be treated as a candidate artefact whose trustworthiness must be established through evidence.

The paper develops an evidence-based, risk-proportionate model in which humans define requirements, constraints, risk levels, and acceptance criteria; AI assists with specification and implementation; automated mechanisms provide testing, static analysis, security analysis, and targeted verification; and humans evaluate the resulting evidence and retain responsibility for acceptance. Runtime monitoring provides further evidence after deployment, while provenance mechanisms support reproducibility, traceability, auditing, and accountability. The literature also demonstrates that nondeterminism, security vulnerabilities, inadequate test coverage, and limitations in LLM-based verification prevent model capability from being treated as a sufficient indicator of software trustworthiness.

The proposed model therefore reframes AI-assisted development from AI-as-generator to AI-within-assurance. Trustworthy software emerges from the interaction of human judgement, AI capability, independent verification, risk-proportionate assurance, provenance, and accountability. The paper identifies research gaps in end-to-end evaluation, multidimensional trustworthiness benchmarks, human verification behaviour, security-aware development environments, and verification-aware AI agents. The central conclusion is that AI should generate possibilities, verification should generate evidence, and humans should determine whether that evidence is sufficient to justify trust and deployment.

Keywords: large language models; generative AI; software engineering; AI coding assistants; software testing; verification; software security; human oversight; trustworthy AI; code generation

1. Introduction

Large language models (LLMs) are becoming increasingly embedded in software engineering, extending their role beyond natural-language interaction to activities including code generation, code comprehension, test generation, defect identification, program repair and software documentation. The breadth of this transformation is reflected in the systematic literature review by Hou et al. (2024), which examined 395 studies and identified applications of LLMs across a wide range of software-engineering activities. LLMs are therefore no longer best understood as specialised code-completion tools; they are increasingly being investigated as general-purpose assistants operating across multiple stages of the software development lifecycle.

The principal motivation for this adoption is partly productivity. Experimental evidence from Peng et al. (2023) indicates that developers using GitHub Copilot can complete programming tasks substantially faster than developers working without the tool. Such findings support the proposition that LLM-based assistants can augment developers by reducing the effort required to produce software. However, productivity cannot be considered independently of software quality. Faster code production does not necessarily constitute improved software engineering if the resulting artefacts are incorrect, insecure, difficult to maintain or insufficiently verified. The relevant question is therefore not simply whether LLMs enable developers to produce code more quickly, but whether they enable developers to produce trustworthy software more efficiently.

This distinction is particularly important because LLM-generated code is probabilistic rather than deterministically derived from a formal specification. The same natural-language request can produce different implementations, meaning that generation itself cannot be treated as a stable or reproducible transformation from requirements to software. Ouyang et al. (2025) provide empirical evidence of this problem, demonstrating substantial non-determinism in ChatGPT-based code generation across 829 programming problems. Importantly, variability persisted even when the temperature parameter was set to zero. Different generated solutions can consequently exhibit different functional behaviour and different testing outcomes. For software engineering, this raises questions not only about correctness but also about reproducibility, traceability, maintenance and accountability.

The security implications are equally significant. The fact that generated code is syntactically plausible or functionally executable does not establish that it is secure. Negri-Ribalta et al. (2024), in a systematic review of research on the security of AI-generated code, identify evidence of vulnerabilities across generated software and emphasise the need for dedicated security assessment rather than assuming that LLM-generated code is secure by default. This concern extends beyond the security of generated artefacts to the ability of LLMs themselves to identify security weaknesses. Sajadi et al. (2025), for example, found considerable limitations in the ability of leading LLMs to recognise vulnerabilities in programming-related questions, with detection performance varying substantially across models and datasets. The implication is consequential: an LLM cannot necessarily be treated as a sufficiently reliable security authority for evaluating code generated by an LLM.

These limitations expose a fundamental tension within AI-assisted software engineering. LLMs can reduce the cost of producing software while simultaneously increasing the need for mechanisms capable of establishing whether that software is correct, secure and fit for purpose. The resulting assurance problem becomes particularly acute as LLMs move from isolated coding assistance towards more autonomous and agentic forms of software development. If AI-generated code is produced at increasing scale, manual inspection alone may become an inadequate mechanism for maintaining software quality.

Testing and verification consequently become central to the adoption of LLMs in software engineering. Rather than treating testing as a downstream activity performed after code generation, recent research increasingly explores the integration of testing into the generation process itself. Fakhoury et al. (2024), for example, investigate test-driven interactive code generation in which tests are used to help formalise developer intent and constrain the generation process. Their findings suggest that integrating testing into the interaction between developer and LLM can improve the effectiveness of code generation while also supporting developers in evaluating generated solutions. Similarly, research by Shang et al. (2025) demonstrates the growing use of LLMs for automated unit-test generation, while Wang, Guo and Tan (2025) investigate the integration of code generation and software testing through context-based retrieval-augmented generation.

However, testing alone cannot establish software trustworthiness. A program may pass an inadequate test suite while failing to satisfy requirements that have not been represented in the tests. This limitation motivates a broader conception of verification in which automated testing is combined with complementary techniques such as static analysis, security analysis, property-based testing and, where appropriate, formal verification. Bose (2025) argues for precisely this shift from prompt-based generation towards property-based specification, while Dolcetti and Iotti (2025) identify the combination of LLMs with established verification techniques as an important direction for trustworthy AI-assisted software engineering. Yang et al. (2026) similarly demonstrate the potential of combining static and dynamic analysis to detect reliability problems in LLM-generated code.

The role of the human developer must therefore also be reconsidered. Human oversight remains essential because software requirements are often contextual, ambiguous and dependent on domain knowledge that cannot be reduced to code-generation accuracy. At the same time, human review is neither perfectly reliable nor infinitely scalable. Klemmer et al. (2024) show that software professionals recognise significant security concerns associated with AI assistants while nevertheless incorporating them into development practices. This creates a critical socio-technical problem: developers must evaluate AI-generated output, but the effectiveness of that evaluation depends on their expertise, workload, trust calibration and ability to recognise subtle defects. Human oversight should consequently not be conceptualised simply as a final approval step; rather, it should be integrated throughout the specification, generation, testing and verification process.

The problem becomes more demanding in high-assurance domains. In safety-critical software, failures can have consequences substantially more serious than those associated with ordinary application development. Liu et al. (2024) demonstrate the importance of evaluating LLM-based code generation specifically within safety-critical contexts rather than assuming that performance on general programming benchmarks transfers directly to high-assurance software. This suggests that the appropriate level of verification should be proportionate to the potential consequences of software failure. A coding assistant used to develop a low-risk prototype may reasonably operate under different assurance requirements from an AI system contributing to medical, automotive, aviation or industrial-control software.

Taken together, this literature indicates that the central challenge is shifting from code generation to assurance. The important research question is therefore no longer simply whether LLMs can generate executable or functionally correct code. It is:

How should humans, AI assistants, testing and verification be combined to produce trustworthy software?

This paper argues that answering this question requires a conceptual shift from an AI-as-generator paradigm to an AI-within-assurance paradigm. In an AI-as-generator model, the LLM produces a candidate implementation and the developer subsequently inspects the result. Assurance is therefore largely positioned after generation. In an AI-within-assurance model, by contrast, generation is embedded within a continuous process of specification, testing, verification, security assessment and human judgement. The LLM is treated not as an autonomous source of trusted software, but as one component within a broader socio-technical assurance system.

This distinction provides the organising principle for the discussion that follows. First, the paper considers the productivity benefits and expanding role of LLMs in software development. It then examines the limitations of code generation, including non-determinism and weaknesses in functional and security assurance. The discussion subsequently considers test-driven generation, property-based testing and hybrid verification as mechanisms for producing evidence about generated software. Finally, it examines the role of human oversight, security and safety, before identifying research gaps concerning the design and empirical evaluation of end-to-end trustworthy AI-assisted development processes.

The central proposition is that trustworthiness should not be treated as an inherent property of an LLM or of the code it generates. Rather, trustworthiness should be established through an evidence-producing development process in which AI-generated artefacts are subjected to appropriate testing, independent verification and human judgement. The future of AI-assisted software engineering therefore depends not only on improving the capabilities of LLMs, but also on developing engineering practices capable of constraining, evaluating and governing those capabilities.

2. From AI-Assisted Productivity to Trustworthy Software

2.1 The productivity proposition

Productivity is one of the strongest initial motivations for adopting large language models (LLMs) in software engineering. Empirical evidence suggests that AI assistance can reduce the time required to complete programming tasks. For example, Peng et al. (2023) reported substantially faster task completion among developers using GitHub Copilot than among developers in a control condition. Although such evidence should be interpreted cautiously, particularly given that this study was conducted on a specific programming task, it is important because it moves the productivity debate beyond anecdotal claims and towards experimentally measurable effects.

However, development speed is not equivalent to engineering productivity. Software engineering is concerned not simply with producing code, but with producing software that is correct, secure, maintainable and fit for its intended purpose. A developer who produces twice as much code in half the time has not necessarily become more productive if the additional output requires proportionally greater effort to test, debug, secure or maintain. The relevant question is therefore not simply how quickly code can be generated, but how efficiently trustworthy software can be produced.

This distinction is particularly important in the context of LLM-based development. Hou et al. (2024) show that LLMs are being investigated across a wide range of software-engineering activities, including code generation, testing, debugging, maintenance and other development tasks. The breadth of these applications suggests that the potential productivity gains extend well beyond code completion. At the same time, the review identifies substantial challenges in evaluating these systems consistently, indicating that measures of speed or task completion alone provide an incomplete account of their engineering value.

The productivity proposition should therefore be reformulated. Rather than asking:

How much faster can developers write code?

the more consequential question is:

How much faster can developers produce software that has been adequately tested, verified, secured and judged fit for purpose?

This is a substantially more demanding proposition. It shifts productivity from a narrow measure of code-production speed towards a broader measure of trustworthy engineering output relative to the effort required to produce it. Under this interpretation, testing, verification and human review are not external costs that detract from AI productivity. They are integral components of the productivity outcome itself.

2.2 The productivity–assurance tension

The shift from code-generation speed to trustworthy engineering exposes a fundamental tension in AI-assisted development. LLMs can reduce the cost of producing software while simultaneously increasing the volume and velocity of software artefacts that require evaluation. The productivity benefit may therefore create a corresponding assurance burden.

The resulting dynamic can be expressed conceptually as:

AI-assisted generation → increased development velocity → increased software output → increased assurance demand → potential verification bottleneck.

This does not imply that AI assistance necessarily reduces overall productivity. Rather, it highlights a condition under which its benefits may be constrained. If generation becomes substantially faster while testing, security analysis, verification and human review remain predominantly manual, the assurance activities may become the limiting factor in the development process.

The problem is particularly significant because generated code cannot be assumed to be correct simply because it is produced quickly or appears plausible. As discussed later, LLM-generated code can exhibit nondeterminism, security weaknesses and context-dependent errors (Negri-Ribalta et al., 2024; Ouyang et al., 2025; Sajadi et al., 2025). Consequently, increasing the rate of generation without increasing the capacity to establish correctness and security risks shifting effort downstream rather than eliminating it.

This creates an important distinction between generation scalability and assurance scalability. An AI assistant may allow a developer to generate hundreds of lines of code in the time previously required to write dozens, but this provides limited engineering value if every additional artefact requires equally intensive manual inspection. The long-term productivity potential of AI-assisted software engineering therefore depends not only on improving generation, but also on developing assurance mechanisms capable of operating at comparable scale.

Recent research increasingly points towards this integration. Rather than treating testing and verification as activities performed only after code has been generated, researchers are investigating workflows in which tests, specifications, static analysis and other forms of verification are incorporated into the generation process itself. Fakhoury et al. (2024), for example, investigate test-driven interactive code generation in which tests help structure the interaction between the developer and the LLM. Shang et al. (2025) examine the use of LLMs for unit-testing tasks, while Wang et al. (2025) explore the use of contextual retrieval to support AI-assisted testing. These approaches suggest a transition from a sequential model—generate first, assure afterwards—towards a more integrated model in which generation and assurance are mutually constraining activities.

This distinction is central to the argument of this chapter. If AI is to deliver sustainable productivity gains, the objective cannot be to maximise the amount of code generated per unit of time. The objective must be to maximise the amount of trustworthy software produced per unit of engineering effort. This requires assurance mechanisms to evolve alongside generation capabilities. The key research challenge is therefore not simply how to make AI-assisted development faster, but how to make it faster without weakening the evidence required to trust the resulting software.

3. Code Generation Requires Specification, Not Merely Prompting

3.1 The specification problem

A fundamental limitation of natural-language code generation is that software requirements are rarely specified with the precision required to determine a unique implementation. Human requirements are often incomplete, ambiguous or dependent on contextual assumptions that are obvious to a developer but absent from the prompt. A request such as “write a function that validates user input”, for example, does not specify which inputs are valid, how boundary cases should be handled, what security constraints apply, what error behaviour is expected, or what performance requirements must be satisfied.

An LLM must therefore infer aspects of the intended behaviour that have not been explicitly specified. This creates an important distinction between generating code from a prompt and generating code from a specification. In the former case, the model is required to resolve ambiguity through probabilistic inference; in the latter, the intended behaviour is constrained by explicit, testable requirements.

This distinction is particularly important because plausible code is not necessarily correct code. An LLM may produce an implementation that is syntactically valid, internally coherent and apparently appropriate while nevertheless implementing an interpretation of the requirement that differs from the developer's actual intent. The problem is therefore not solely one of model capability. It is also a problem of requirements representation: the quality of the generated implementation is constrained by the extent to which the intended behaviour has been made explicit.

Fakhoury et al. (2024) address this problem through TiCoder, an interactive workflow that uses tests to help formalise user intent during code generation. In their user study with 15 programmers, participants using the workflow were better able to evaluate generated code and reported lower cognitive load. Their larger-scale evaluation also demonstrated substantial improvements in pass@1 performance across multiple LLMs and datasets. These findings are significant because they suggest that improving the interaction between the developer and the model can be as important as improving the model itself.

The conceptual contribution is particularly important. In a conventional AI-assisted workflow, testing is often treated as an activity that follows generation:

natural-language intention → code generation → testing → human inspection

In a specification-oriented workflow, however, tests can participate in defining what the generated program is expected to do:

natural-language intention → clarification → executable tests → code generation → execution → human evaluation

The latter workflow establishes a stronger connection between requirements and implementation. Instead of asking the LLM to infer all relevant behaviour from an unconstrained natural-language request, the developer progressively externalises that behaviour into artefacts against which generated code can be evaluated.

This suggests that the role of the AI assistant should not be understood solely as code generation. The assistant can instead participate in a broader process of requirements clarification, specification construction and implementation. Such a process reduces, although does not eliminate, the semantic gap between what the developer intends and what the generated program actually implements.

3.2 Tests as executable specifications

The significance of this approach extends beyond conventional unit testing. Tests can serve as a form of executable specification: an operational representation of selected aspects of the behaviour that the software is expected to exhibit. They provide a concrete artefact against which both the developer and the LLM can reason, allowing implementation decisions to be evaluated against explicit expected outcomes.

This creates two important benefits. First, constructing tests forces previously implicit assumptions to become explicit. Questions about valid inputs, invalid inputs, boundary conditions and expected behaviour must be resolved sufficiently to express them as test cases. In this sense, testing can function as a mechanism for requirements clarification, rather than merely defect detection.

Second, executable tests provide an automated basis for rejecting at least some incorrect implementations. Instead of relying exclusively on a developer's judgement that generated code looks correct, the implementation can be executed against predefined behavioural expectations. This is particularly valuable in LLM-assisted development because generated code can be plausible enough to escape superficial inspection.

Nevertheless, tests constitute only a partial specification. A program can satisfy every test in a test suite while still violating requirements that the suite does not represent. The adequacy of the resulting assurance therefore depends not only on whether the generated code passes the tests, but also on whether the tests adequately capture the relevant behaviour and risks.

This limitation is fundamental to software testing more generally: successful execution against a finite set of test cases provides evidence of conformance to those cases, but does not establish that all possible executions satisfy the intended specification. In the context of LLM-generated software, the limitation is particularly important because the model may generate implementations that satisfy the visible examples while failing on untested inputs, edge cases or security-relevant conditions.

Consequently, passing tests should be understood as evidence of correctness, not proof of correctness. The distinction becomes even more important when software operates in security-sensitive or safety-critical contexts, where untested behaviours may have consequences that are disproportionate to their frequency.

Trustworthy AI-assisted development therefore requires a hierarchy of assurance rather than reliance on a single mechanism. Executable tests can constrain and evaluate generated behaviour, but they should be complemented where appropriate by broader forms of testing, static and security analysis, property-based testing and formal verification. Bose (2025), for example, argues for the use of property-based testing to move beyond fixed examples towards specifications expressed as properties that generated programs should satisfy. Dolcetti and Iotti (2025) similarly emphasise the complementary relationship between LLM-based code generation and established verification techniques.

The resulting principle is therefore stronger than the claim that LLMs should be prompted more carefully. The deeper requirement is that LLM-based code generation should be constrained by increasingly explicit specifications and accompanied by evidence-producing assurance mechanisms. Prompting can initiate the development process, but trustworthy software requires the developer's intent to be progressively transformed into artefacts that can be tested, analysed and, where necessary, formally verified.

In this sense, the central transition is from prompt-to-code towards intent-to-specification-to-code. The LLM remains an important component of the process, but it is no longer treated as the sole mechanism through which human intent is translated into software behaviour.

4. Testing as the First Layer of AI-Code Assurance

4.1 Automated testing

Testing represents one of the most accessible and scalable mechanisms for evaluating LLM-generated software. As LLMs have become increasingly integrated into software engineering activities, their role has expanded beyond the generation of implementation code to include the generation of unit tests, assertions, and test suites. Shang et al. (2025), for example, investigate LLM-based approaches to unit-test generation, assertion generation, and test evolution. Their work illustrates an important extension of AI-assisted development: the same class of models used to produce implementation code can also assist in producing artefacts that provide evidence about that code.

This capability is potentially important because increased AI-assisted code generation can create additional demand for testing and verification. If LLMs can assist with producing relevant tests alongside implementation code, some of the resulting assurance workload can potentially be automated (Shang et al., 2025). However, the existence of generated tests does not by itself establish their adequacy. Liu et al. (2025) highlight the importance of evaluating whether benchmark tests provide sufficient evidence of code-generation performance, while Shang et al. (2025) demonstrate that LLM-based testing itself is an empirical problem requiring evaluation. Generated tests should therefore be treated as assurance artefacts that require assessment, rather than as inherently reliable evidence.

This limitation is particularly important when the same model is used to generate both the implementation and its tests. If both artefacts are produced from the same incomplete understanding of the requirements, the tests may fail to challenge the assumptions embedded in the implementation. This is consistent with the broader distinction between specification and implementation established by Fakhoury et al. (2024): tests are more valuable as assurance evidence when they express or derive from explicit behavioural requirements rather than merely reproducing assumptions already present in the generated code. Consequently, testing can provide stronger evidence when it is grounded in explicit specifications, diverse test inputs, relevant contextual information, or complementary verification techniques (Fakhoury et al., 2024; Bose, 2025).

Wang, Guo and Tan (2025) similarly demonstrate the value of integrating LLMs more directly into software-testing workflows. Their context-based retrieval-augmented approach emphasises the importance of supplying the model with relevant information about the software under test. Repository context and other available software artefacts can therefore be important inputs when generating tests that meaningfully evaluate a target implementation (Wang, Guo and Tan, 2025). This supports a broader principle that AI-assisted testing should be context-aware rather than isolated from the software and requirements it is intended to evaluate.

The literature therefore suggests a potentially useful division of responsibilities within AI-assisted development:

AI programmer → proposes implementation

AI tester → proposes, executes, or evaluates tests

This division should not be interpreted as establishing independent verification simply because different AI roles are assigned to different stages. Where the same model, context, or assumptions influence both generation and testing, the resulting evidence may not be genuinely independent. The distinction is nevertheless valuable because it encourages an assurance-oriented workflow in which generated implementations are systematically challenged rather than accepted on the basis of their apparent plausibility.

Testing can also be strengthened by moving beyond collections of example-based unit tests towards more general behavioural properties. Bose (2025) argues for the use of property-based testing in the evaluation of LLM-generated code, highlighting the potential to assess broader classes of inputs and behaviours rather than relying solely on predefined examples. This reinforces the view that testing should be understood as a spectrum of evidence-producing techniques rather than as a single mechanism.

The broader principle is therefore that AI should be used not only to increase the production of software, but also to increase the production of evidence about software. This distinction is central to trustworthy development. An implementation generated by an LLM is a candidate claim about how a system should behave; tests and other verification mechanisms provide evidence about whether that behaviour holds under the conditions being evaluated (Fakhoury et al., 2024; Bose, 2025; Shang et al., 2025).

Testing is consequently best understood as the first layer of AI-code assurance, rather than the complete assurance process. Its value lies not simply in identifying failures, but in creating observable evidence that can be combined with static analysis, security analysis, formal verification where appropriate, and human judgement to establish whether an AI-generated artefact is sufficiently trustworthy for its intended use.

4.2 Property-based testing

Conventional unit tests typically express expected behaviour through individual examples:

input X → expected output Y

Such examples are useful but inherently limited. A finite collection of test cases can cover only a small subset of the possible inputs and execution states of a program. This limitation becomes particularly relevant for LLM-generated code, where implementations may appear correct on familiar examples while failing on boundary conditions, unusual inputs or states that were not represented in the prompt or benchmark tests.

Property-based testing addresses this limitation by specifying general properties that should hold across classes of inputs, rather than defining expected outputs only for individual examples. Instead of specifying that a particular input should produce a particular result, a property might state that an operation should preserve an invariant, satisfy a mathematical relationship, or behave consistently under a defined transformation. Automated generators can then explore many inputs in an attempt to find counterexamples to those properties.

Bose (2025) argues that this approach is particularly relevant to LLM-generated code because properties can provide a richer specification of expected behaviour than a collection of fixed examples. In the context of AI-assisted development, property-based testing therefore has a dual role: it can act as a testing technique and as a more expressive representation of requirements against which generated implementations can be evaluated.

However, property-based testing does not eliminate the specification problem. The properties themselves must be correctly identified and sufficiently comprehensive. An incorrect or incomplete property can provide misleading assurance just as an inadequate unit-test suite can. The key issue is therefore not simply whether more tests are generated, but whether the testing process captures the behaviours that matter for the intended system.

4.3 Complementary assurance layers

The limitations of any individual testing technique suggest that AI-generated software should be assessed through complementary assurance mechanisms rather than a single testing method. Example-based tests can establish expected behaviour for representative scenarios; boundary and negative tests can probe failure conditions; property-based testing can explore broader classes of behaviour; fuzzing and generative testing can expose unexpected inputs and states; static analysis can identify classes of defects without requiring execution; and formal verification can provide stronger guarantees for properties that can be expressed and proved within an appropriate formal framework.

These techniques should not be understood as a simple ladder in which each successive method universally replaces the previous one. They answer different assurance questions. A unit test may establish that a particular scenario behaves as expected, property-based testing may search for violations across a broad input space, static analysis may identify structural or semantic problems without executing the program, and formal verification may establish that a specified property holds under explicitly defined assumptions. The appropriate combination depends on the software's complexity, risk, domain and consequences of failure.

This distinction is especially important for security- and safety-critical software. Increasing the number of unit tests does not necessarily provide proportionally stronger security assurance, just as passing a large benchmark suite does not demonstrate that an implementation is free from vulnerabilities. Assurance must therefore be risk-sensitive: the greater the consequences of failure, the stronger and more diverse the evidence required to justify confidence.

For AI-generated software, this leads to a broader assurance model:

generation → testing → analysis → verification → human judgement

with feedback between these activities rather than a strictly sequential process. Test failures can prompt regeneration; static-analysis findings can constrain implementation; security analysis can generate additional tests; and verification results can identify ambiguities in the underlying specification. Assurance therefore becomes an iterative process in which evidence is continuously fed back into generation and refinement.

This perspective also changes the role of the LLM. Rather than viewing the model as an autonomous programmer whose output is inspected only after completion, the model becomes one participant in an evidence-producing development process. Its outputs can include implementation proposals, tests, properties and explanations, but each remains subject to independent evaluation. The objective is not to make the LLM itself perfectly reliable; it is to construct a development workflow in which the consequences of model error are detected and constrained before they become software defects.

Testing is therefore best understood as the first layer of AI-code assurance, rather than the final authority on correctness. It provides scalable and valuable evidence, but trustworthy software requires that evidence to be complemented by increasingly targeted forms of analysis and verification.

5. Testing Is Necessary but Insufficient

Testing is an essential component of trustworthy AI-assisted software development, but passing a test suite should not be interpreted as evidence that software is universally correct, secure or fit for purpose. Tests establish evidence about the behaviours they specify and execute; they do not, by themselves, establish the absence of defects outside those behaviours. This distinction is particularly important for LLM-generated software, where plausible implementations may contain errors that are not exposed by the available tests.

At least four limitations are particularly relevant.

5.1 Incomplete coverage

The first limitation is coverage. Tests can evaluate only those behaviours that have been represented in the test suite and exercised during execution. Even a large collection of tests may leave important combinations of inputs, execution states, environmental conditions or failure modes unexplored.

This problem is amplified when tests are generated from the same prompt or assumptions used to generate the implementation. If the underlying requirement omits an important edge case, an AI-generated test suite may reproduce that omission rather than expose it. Consequently, increasing the number of tests does not necessarily produce a proportional increase in assurance. What matters is whether the tests provide meaningful coverage of the behaviours and risks relevant to the system.

5.2 Incorrect or incomplete specifications

The second limitation is more fundamental: tests can faithfully verify the wrong requirement. A test suite may correctly determine that an implementation produces the expected output while the expected output itself does not reflect what the system is actually required to do.

This is particularly significant in natural-language-driven development. Requirements may contain ambiguities, implicit assumptions or unstated constraints that are difficult to capture in executable tests. If these assumptions are incorrect, the resulting test suite can provide strong evidence of conformance to an incorrect specification.

This creates an important distinction between verification of an implementation and validation of the intended system behaviour. Testing primarily addresses whether the implementation behaves as specified under the tested conditions. It does not, on its own, establish that the specification accurately represents the user's, organisation's or system's actual requirements. Human judgement therefore remains important in determining whether the specification itself is appropriate.

5.3 Security vulnerabilities

The third limitation is that functional correctness does not imply security. Software can pass its functional tests while remaining vulnerable to security failures such as injection, authentication bypass, insecure cryptographic practices, improper authorisation or unsafe handling of untrusted data.

This distinction is especially important for AI-generated code. Negri-Ribalta et al. (2024) conclude from their systematic review that AI-generated code cannot be assumed to be secure by default. Sajadi et al. (2025) similarly demonstrate limitations in LLMs' ability to identify vulnerabilities in programming-related responses. These findings indicate that an LLM should not automatically be treated as a reliable security authority for evaluating its own output.

Security assurance therefore requires dedicated analysis in addition to functional testing. Depending on the application, this may include static security analysis, dependency analysis, vulnerability scanning, threat modelling, penetration testing and security-focused test generation. The appropriate combination depends on the system's threat model and the consequences of compromise.

5.4 Emergent system behaviour

The fourth limitation concerns system-level behaviour. Software is rarely composed of isolated functions operating independently. Components interact with databases, networks, libraries, external services, operating systems and other components. Correctness at the unit level therefore does not necessarily imply correctness at the system level.

An individual component may pass its unit tests while an interaction between components produces unexpected behaviour. Problems can arise from incorrect assumptions about interfaces, shared state, concurrency, error propagation, configuration or external dependencies. For AI-assisted development, this issue is particularly relevant because LLMs may generate components independently without possessing a complete and persistent model of the architecture in which those components will operate.

System-level assurance therefore requires integration testing, system testing and, where appropriate, architectural analysis and runtime monitoring. The objective is to evaluate not only whether individual generated components behave correctly in isolation, but whether their interactions satisfy the requirements of the overall system.

5.5 From testing to layered assurance

These limitations demonstrate why trustworthy AI-assisted development cannot be reduced to a question of whether generated code passes its tests. Testing provides valuable empirical evidence, but that evidence is necessarily bounded by the adequacy of the tests, the correctness of the underlying specification, the security properties being considered and the system context in which the software operates.

The resulting assurance model should therefore be layered rather than monolithic. Functional tests can provide evidence of expected behaviour; property-based and generative testing can explore broader input spaces; static analysis can identify classes of defects without execution; security analysis can target vulnerabilities that functional tests may overlook; formal verification can provide stronger guarantees for properties that can be specified and proved; and human review can assess contextual requirements and assumptions that automated techniques cannot fully capture.

The layers should also interact rather than operate as isolated checkpoints. A security analysis may reveal a missing test; a failed test may expose an ambiguity in the specification; static analysis may identify an architectural concern; and human review may identify a requirement that has not been represented in any automated check. Trustworthy development therefore emerges from the combined evidence produced by multiple assurance mechanisms, rather than from any single pass/fail criterion.

This leads to a central principle for AI-assisted software engineering:

A passing test suite is evidence that software behaves correctly under tested conditions; it is not, by itself, evidence that the software is trustworthy.

The distinction matters because trustworthiness encompasses more than functional correctness. It includes properties such as security, reliability, maintainability and fitness for purpose, together with confidence that these properties have been established through an appropriate and sufficiently independent assurance process.

For LLM-generated software, this means that testing should be treated as a foundational layer of assurance rather than as the final gate before deployment. The critical question is not whether the AI-generated code passes the available tests, but whether the overall development process has produced sufficient evidence to justify trusting the software for its intended context and level of risk.

6. Static Analysis, Dynamic Analysis and Formal Verification

The limitations of testing suggest that trustworthy AI-assisted software engineering requires a broader verification architecture. Recent research increasingly explores hybrid approaches in which LLMs are combined with established software-engineering techniques rather than being treated as substitutes for them. This reflects a fundamental distinction between the capabilities of probabilistic language models and those of deterministic or mathematically grounded verification tools.

Dolcetti and Iotti (2025) provide a useful conceptual framework for understanding this emerging field by distinguishing between LLM4Verification and Verification4LLM. LLM4Verification refers to the use of LLMs to assist with verification activities, for example by generating specifications, analysing code, explaining verification results or assisting with proofs. Verification4LLM, by contrast, concerns the verification and validation of code produced by LLMs. The distinction is important because it separates two related but different research questions: how LLMs can improve verification and how verification can constrain the risks introduced by LLM-based code generation.

Their review identifies combinations of LLMs with static analysis, formal verification, retrieval-augmented generation, prompt engineering and other tool integrations as important directions. This emerging architecture reflects a broader principle: LLMs need not replace deterministic software-engineering tools in order to provide substantial value. Instead, their probabilistic capabilities can be combined with tools that provide forms of analysis and evidence that language models cannot reliably provide on their own.

6.1 Complementary assurance mechanisms

Different assurance techniques address different questions about software. LLMs are particularly useful for generating code, explanations, candidate specifications and reasoning assistance, but their outputs remain probabilistic and may contain factual or technical errors. Automated testing provides concrete evidence about observed behaviour, but its conclusions are bounded by the tests that are executed. Property-based testing can explore broader behavioural constraints, but useful properties must first be formulated accurately.

Static analysis provides a complementary perspective because it examines program structure and semantics without requiring every relevant execution to occur. It can systematically identify predefined classes of defects, coding violations, suspicious data flows or security weaknesses. However, static analysis can produce false positives and false negatives, and its effectiveness depends on the analyses and rules being applied. It also does not necessarily resolve whether the program implements the intended high-level requirements.

Dynamic analysis provides another form of evidence by observing actual program behaviour during execution. It can expose failures that are difficult to identify through static inspection and can provide information about performance, resource use, memory behaviour or interactions with external components. Its fundamental limitation, however, is that execution-based techniques cannot exhaustively explore all possible inputs and execution paths in non-trivial programs.

Formal verification provides yet another form of assurance. Where software requirements can be expressed as formal properties and the relevant assumptions can be adequately modelled, mathematical techniques can establish that an implementation satisfies specified properties. This can provide substantially stronger evidence than testing for the properties that have actually been formalised. However, formal verification is not universally applicable or cost-effective. The effort required to construct formal specifications and proofs can be considerable, and a formally verified implementation may still fail to satisfy requirements that were omitted from the formal model.

Human review occupies a different position within this architecture. Developers contribute contextual knowledge, architectural understanding, domain expertise and the ability to assess whether the specification itself reflects the intended purpose of the system. Human judgement can therefore address questions that are difficult to reduce to automated checks. At the same time, human review is costly, difficult to scale and vulnerable to cognitive limitations, particularly when reviewers are confronted with large volumes of AI-generated code.

The resulting picture is not one of a single technique being universally superior. Each mechanism produces a different type of evidence and has different assumptions and failure modes. The objective should therefore be to combine mechanisms whose weaknesses are not perfectly correlated.

6.2 From sequential verification to hybrid assurance

This perspective challenges a simple sequential model in which an LLM generates code and a developer subsequently chooses one verification technique to assess it. A more robust architecture is hybrid and iterative.

An LLM might first generate an implementation and candidate tests. Automated execution can then identify concrete failures, while static analysis can identify structural or security-related concerns that tests do not expose. Property-based testing can explore broader behavioural assumptions, and formal verification can be applied to particularly critical properties where the required specification and verification infrastructure exist. Human developers can then interpret the evidence, resolve ambiguities and determine whether the assurance obtained is appropriate for the system's intended use.

The process can consequently be viewed as a feedback loop:

LLM generation → automated testing → static/dynamic analysis → targeted verification → human evaluation → refinement

The arrows should not be interpreted as a rigid sequence. Findings from any stage can feed back into earlier stages. A static-analysis warning may lead to a new test; a failed property may reveal an ambiguity in the specification; a verification failure may require the implementation or specification to be revised; and human review may identify a requirement that none of the automated mechanisms currently capture.

This feedback is particularly important for AI-assisted development because verification can become part of the interaction with the model itself. Rather than asking an LLM to produce a final implementation in a single step, developers can provide analysis results, failed tests or verification feedback and ask the model to propose a revised implementation. In this configuration, automated verification becomes not merely a gatekeeper but a source of structured feedback that constrains subsequent generation.

6.3 Verification should be risk-sensitive

The appropriate combination of assurance techniques should also depend on the consequences of failure. A small internal utility and a safety-critical control system should not be subject to identical assurance requirements simply because both contain AI-generated code.

For low-risk software, conventional tests, static analysis and human review may provide an appropriate level of evidence. For security-sensitive systems, additional security analysis and adversarial testing may be necessary. For safety-critical or otherwise high-consequence systems, stronger forms of verification may be justified for critical components, together with explicit traceability between requirements, implementation and verification evidence.

This risk-sensitive perspective avoids two opposite mistakes. The first is over-reliance on the LLM, in which generated code is accepted because it appears plausible or because it passes a limited test suite. The second is over-verification, in which expensive assurance techniques are applied indiscriminately regardless of risk or feasibility. Trustworthy engineering requires an assurance strategy proportionate to the consequences of failure.

The central implication is therefore that LLMs should be understood as components within a verification ecosystem, not as replacements for that ecosystem. Their ability to generate, summarise, explain and reason about software can substantially improve engineering workflows, but these capabilities become more trustworthy when combined with tools that provide independent and complementary forms of evidence.

The strongest architecture is consequently complementary rather than substitutive. LLMs can accelerate the production of code and assurance artefacts; testing can provide empirical evidence of behaviour; static and dynamic analysis can identify additional classes of defects; formal verification can establish specified properties where feasible; and human experts can determine whether the resulting evidence is sufficient for the intended context.

The goal is therefore not to find a single technique capable of proving that AI-generated software is trustworthy. It is to construct an assurance process in which different techniques compensate for one another's limitations, while maintaining sufficient traceability between human intent, generated artefacts, verification evidence and the final engineering decision.

7. Hybrid Verification Is Emerging as a Central Paradigm

A particularly important development in trustworthy AI-assisted software engineering is the emergence of hybrid verification architectures that combine LLM-based generation with established static and dynamic analysis techniques. These approaches recognise that the strengths of language models in generating and reasoning about software do not remove the need for independent mechanisms capable of challenging their outputs.

Yang et al. (2026), for example, propose a hybrid static/dynamic approach for detecting hallucinated code. Their work addresses a particularly important class of LLM failure: code that is syntactically plausible and superficially convincing but nevertheless contains semantic or logical errors. Evaluated on MBPP, CodeHaluEval and HalluCode, their approach reported average precision, recall and F1 scores of 0.771, 0.783 and 0.776, respectively. Although these results should not be interpreted as demonstrating complete detection of hallucinated code, they provide empirical evidence that combining complementary analysis techniques can identify problems that may not be apparent from the generated code itself.

The significance of this research is therefore conceptual as well as technical. It challenges a potentially problematic assumption underlying some AI-assisted development workflows: that the LLM that generates an implementation can also be relied upon to determine whether that implementation is correct.

An alternative principle is:

The LLM generates a candidate artefact; independent mechanisms attempt to falsify, analyse or verify it.

This represents an important shift in the role assigned to the model. Rather than treating the LLM as both producer and authority, the model becomes a source of candidate solutions whose outputs are subjected to additional forms of evidence. The distinction is analogous to established engineering practices in which independent validation is used because the process responsible for producing an artefact should not necessarily be regarded as the final authority on its correctness.

7.1 The importance of independence

Independence is particularly important because errors in AI-assisted development may be correlated. If the same LLM generates an implementation, generates the tests used to evaluate it, interprets the test results and ultimately declares the implementation correct, the apparent assurance process may provide considerably less independence than its multiple stages suggest.

For example, an LLM may misunderstand a requirement and subsequently generate both code and tests that reflect the same misunderstanding. The implementation can then pass its generated tests while remaining inconsistent with the actual system requirement. Similarly, an LLM asked to review its own code may reproduce the assumptions that led to the original defect rather than identifying them.

This does not mean that LLM-based self-review is without value. On the contrary, an LLM can be highly useful for identifying potential defects, generating additional tests, explaining analysis results and suggesting alternative implementations. The important distinction is that such outputs should be treated as candidate evidence or recommendations, rather than as definitive proof of correctness.

Independent mechanisms can provide a different perspective. Static analysers operate according to explicitly defined rules and analyses; dynamic testing observes actual execution behaviour; security tools target known vulnerability patterns; property-based testing searches for violations across generated inputs; and formal verification can establish specified properties under explicit assumptions. None of these mechanisms is infallible, but their failure modes need not coincide with those of the LLM.

The objective is therefore not to maximise the number of verification tools, but to introduce complementary sources of evidence whose assumptions and failure modes differ.

7.2 From verification after generation to verification within generation

Hybrid verification also changes the temporal relationship between generation and assurance. In a conventional development model, verification is often treated as a downstream activity:

requirements → implementation → verification → deployment

For AI-assisted development, a more effective architecture may be iterative:

requirements → specification → LLM generation → automated analysis → feedback → revised generation → verification → human judgement

Here, verification is not merely a final gate. Its results can influence subsequent generations of the implementation. A static-analysis warning can become a prompt constraint; a failed test can trigger code revision; a detected security weakness can generate additional security tests; and a failed formal property can require either the implementation or the underlying specification to be reconsidered.

This creates a closed assurance loop in which generation and verification continuously interact. The LLM remains responsible for proposing solutions, but the surrounding assurance infrastructure constrains which solutions can be accepted.

Such an architecture also provides a more realistic interpretation of AI-assisted productivity. If verification results can be automatically incorporated into subsequent generation cycles, the increased volume of AI-generated code need not necessarily produce a proportional increase in manual review effort. Instead, automated assurance mechanisms can filter, challenge and refine generated artefacts before they reach human reviewers.

7.3 Independent verification as a trust boundary

The distinction between generation and verification can therefore be understood as a trust boundary. The LLM is not assumed to be inherently trustworthy; rather, its output crosses into a verification process in which additional evidence must be produced before the artefact is accepted.

This is especially important for security- and safety-sensitive systems. In such contexts, accepting an LLM's own assessment of its output would create a circular assurance argument: the same probabilistic system would be responsible for producing the artefact and establishing the evidence for trusting it. Independent analysis breaks this circularity by introducing mechanisms that do not rely exclusively on the model's own judgement.

However, independence should not be interpreted as absolute. Static analysers, test suites and formal verification systems also depend on specifications, assumptions, configurations and models that may themselves be incomplete or incorrect. Independent verification therefore does not guarantee trustworthiness automatically. Rather, it reduces reliance on a single potentially correlated source of error.

This leads to a more defensible conception of trustworthy AI-assisted software engineering. Trust should not be placed in the LLM as an isolated component. Instead, confidence should emerge from the interaction of multiple assurance mechanisms, each contributing different forms of evidence and each subject to appropriate scrutiny.

The emerging paradigm can therefore be summarised as:

LLM generation + independent analysis + iterative verification + human judgement = evidence-based assurance

The central shift is from trusting the generator to trusting a process that systematically challenges the generator's output. Hybrid verification is consequently not merely a technical strategy for improving defect detection. It represents a broader architectural principle for trustworthy AI-assisted development: AI should generate and assist, while independent assurance mechanisms provide the evidence required for acceptance.

8. Security Requires an Independent Assurance Layer

Security provides one of the clearest arguments for treating AI-assisted software development as a layered assurance problem. Functional testing can establish that software behaves as expected under selected conditions, but it does not establish that the software is secure against adversarial inputs, misuse or unforeseen attack paths. For LLM-generated code, this distinction is particularly important because an implementation can appear functionally correct while containing vulnerabilities that are difficult to identify through conventional testing.

Negri-Ribalta et al. (2024) systematically review the security implications of AI-generated code and identify vulnerabilities across common classes of software weakness. Their findings challenge the assumption that code generated by AI systems is secure by default. Rather, they conclude that dedicated security measures and verification processes remain necessary. The implication is not that LLM-generated code is inherently insecure, but that its origin provides no sufficient basis for assuming security.

Hamer, d'Amorim and Williams (2024) similarly investigate the security characteristics of ChatGPT-generated code in comparison with code obtained from Stack Overflow. Their work is important because it frames AI-generated code as an empirical security question rather than assuming either that LLMs produce unusually insecure code or that they can be trusted to reproduce established secure-development practices. Security properties must instead be evaluated through appropriate evidence.

More recent research raises a related concern about the ability of LLMs to identify security problems in code. Sajadi et al. (2025) evaluated three prominent LLMs on programming questions containing vulnerabilities and found relatively low and variable vulnerability-detection rates, ranging from 12.6% to 40% across the datasets and models examined. These results indicate that an LLM's ability to generate plausible security-related explanations should not be confused with reliable vulnerability detection.

Taken together, these findings support an important principle:

An LLM should not be treated as the sole security authority for code generated by an LLM.

The problem is fundamentally one of assurance independence. If an LLM generates an implementation and the same or a similar model is subsequently asked whether that implementation is secure, the two judgements may share the same assumptions and blind spots. A vulnerability that arises from a model's misunderstanding of an authentication requirement, input-validation boundary or trust relationship may therefore remain undetected when the model is asked to review its own output.

This does not make LLM-based security assistance useless. LLMs can help identify potential vulnerabilities, explain security findings, generate security-focused tests, suggest mitigations and assist developers in understanding unfamiliar code. However, these capabilities should complement rather than replace security mechanisms capable of providing independent evidence.

8.1 Security as a dedicated assurance activity

A trustworthy AI-assisted development pipeline should therefore incorporate security assurance explicitly rather than treating security as an incidental outcome of functional testing. Depending on the system and its threat model, relevant mechanisms may include static security analysis, dependency and package scanning, secret detection, dynamic security testing, threat modelling, secure code review, vulnerability intelligence and penetration testing.

Static security analysis can identify suspicious patterns, unsafe data flows and known vulnerability classes without requiring every execution path to be exercised. Dependency analysis can identify vulnerable third-party components, an important consideration for LLM-generated code because models may suggest packages, APIs or libraries without guaranteeing that their versions, configurations or security characteristics are appropriate. Secret detection can identify credentials or sensitive material inadvertently introduced into source code or configuration.

Dynamic security testing provides a complementary perspective by examining the behaviour of the running system under potentially malicious or unexpected inputs. Depending on the application, this may include fuzzing, adversarial input testing and other forms of security testing. Penetration testing can provide a further layer of evidence for systems where the potential consequences of exploitation justify the additional effort.

Threat modelling occupies a different but equally important role. It requires developers to reason about assets, trust boundaries, attackers, attack surfaces and potential misuse scenarios before or during implementation. This is an area where human expertise remains particularly important because security requirements are strongly dependent on system architecture and operational context.

Secure code review can then integrate the results of these mechanisms with domain and architectural knowledge. The reviewer is not simply asking whether the code passes its functional tests, but whether the security assumptions underlying the implementation are appropriate for the system's threat model.

8.2 Security assurance should be independent and risk-sensitive

Not every software system requires the same level of security assurance. A low-risk internal tool and an internet-facing service handling sensitive information have fundamentally different threat models and consequences of failure. Security assurance should therefore be proportionate to risk rather than implemented as a fixed checklist.

Nevertheless, the underlying principle remains constant: security should be established through evidence that does not depend solely on the model that produced the code. The greater the potential consequences of compromise, the greater the justification for multiple and complementary forms of independent analysis.

This leads to an important distinction between AI-assisted security and security assurance for AI-generated software. In the first case, the LLM is used as a security assistant. In the second, the security of the LLM's own output becomes an object of independent evaluation. Both are useful, but they serve different purposes. An LLM can help a developer reason about a vulnerability without being sufficiently reliable to establish that no relevant vulnerabilities remain.

The resulting security pipeline can therefore be conceptualised as:

LLM generation → security analysis → adversarial testing → remediation → independent review → acceptance

with feedback between each stage. Security findings should not merely be recorded after generation; they can be fed back into subsequent code generation and testing. In this way, security assurance becomes part of the development loop rather than a final inspection performed immediately before deployment.

The broader implication is that security cannot be inferred from the apparent quality of generated code. A program may be readable, idiomatic, well-tested and functionally correct while still exposing exploitable attack paths. Trustworthy AI-assisted software engineering therefore requires security to be treated as a distinct assurance dimension with its own evidence, tools and review processes.

The principle can be stated succinctly:

Functional correctness asks whether the software does what it is expected to do; security assurance asks whether it remains safe when confronted with conditions it was not merely expected to encounter.

For AI-generated software, both questions must be answered. LLMs can contribute to answering them, but neither correctness nor security should be accepted solely on the authority of the model that generated the implementation.

9. Human Oversight Remains Necessary—but Must Be Redesigned

9.1 Humans are not perfect verification mechanisms

The most immediate response to concerns about unreliable AI-generated software is to place a human at the end of the process and require them to inspect the generated output. Human oversight is undoubtedly important, but treating human review as a universal solution is insufficient. Humans are themselves limited assurance mechanisms: they have finite attention, limited time, varying levels of domain expertise and imperfect ability to detect subtle defects.

The problem becomes particularly acute when AI assistance substantially increases the volume of generated software. If an LLM enables developers to produce considerably more code, requiring developers to inspect every generated line with the same level of attention may simply transfer the bottleneck from code production to code review. The resulting process may preserve safety at the cost of much of the productivity benefit that motivated AI adoption in the first place.

There is also a risk of automation bias. When AI-generated suggestions are presented as plausible, coherent and authoritative, developers may become more willing to accept them, particularly when the output is difficult to evaluate independently. The danger is not necessarily that developers blindly trust AI. A more subtle problem is that repeated exposure to apparently successful suggestions may gradually alter the level of scrutiny applied to subsequent outputs.

Klemmer et al. (2024) provide important empirical evidence concerning this relationship between trust, use and verification. Their study involved 27 software professionals, including engineers, team leads and security testers, complemented by an analysis of 190 Reddit discussions. Participants expressed significant concerns about the security and reliability of AI-generated code, yet nevertheless reported using AI assistants for security-relevant activities. Their concerns also motivated verification practices, with participants often checking AI-generated suggestions in ways broadly similar to how they assessed code written by human developers.

These findings illustrate an important paradox:

limited trust in AI → increased verification → increased verification workload → potential reduction in realised productivity gains

AI assistance can therefore simultaneously increase the speed of code generation and increase the amount of work required to establish confidence in that code. The productivity benefit depends partly on whether the surrounding assurance process can scale with the increased generation rate.

9.2 From “human in the loop” to risk-sensitive human oversight

The solution should therefore not be framed simply as “keep a human in the loop.” That formulation says little about what the human should do, when they should intervene, or which decisions should remain under human control.

A more useful principle is:

The objective is the right human involvement at the right point in the assurance process.

This implies a redistribution rather than elimination of human responsibility. Automated testing, static analysis, security analysis and other verification mechanisms can handle large volumes of routine evidence generation. Humans can then concentrate their limited attention on activities that require contextual judgement, such as interpreting ambiguous requirements, assessing architectural consequences, evaluating security and safety risks, resolving conflicts between requirements, and determining whether the available evidence is sufficient for the intended use.

This distinction is particularly important because not all generated code presents the same level of risk. A trivial formatting utility does not warrant the same level of human scrutiny as authentication logic, cryptographic code, safety-critical control software or code handling sensitive information. Human review should therefore be risk-sensitive.

Low-risk outputs may be accepted after automated tests and analysis satisfy predefined criteria. Higher-risk outputs may require targeted expert review, additional testing or formal verification. Critical components may require traceability from requirements through implementation and verification evidence before they can be accepted.

Such an approach transforms human review from a universal inspection task into a decision-making and assurance activity. The human reviewer is not expected to independently rediscover every defect in AI-generated code. Instead, the reviewer evaluates the evidence produced by the broader assurance process and determines whether it is adequate for the context.

9.3 Human oversight as contextual assurance

Human involvement remains particularly important because software requirements cannot always be reduced to executable checks. Developers and domain experts possess knowledge about users, operational environments, organisational constraints, architectural trade-offs and potential consequences of failure that may not be represented in the source code or test suite.

This means that human oversight should begin before code generation rather than being restricted to final approval. Humans should participate in defining requirements, clarifying ambiguities, identifying risks and determining what properties must be demonstrated. They should also remain involved in interpreting verification results and deciding whether unresolved uncertainty is acceptable.

The resulting workflow can be represented as:

human intent and risk assessment → AI-assisted specification and generation → automated assurance → targeted human review → acceptance or refinement

This is fundamentally different from a workflow in which an LLM generates code autonomously and a developer performs a final superficial inspection. In the former, human judgement shapes the assurance process from the beginning; in the latter, human judgement is treated primarily as a final error-detection mechanism.

9.4 Humans as governors of the assurance process

This perspective suggests a broader conception of human oversight. The human's most important role may not be to act as the final checker of every line of code, but to govern the conditions under which AI-generated software can be trusted.

This includes deciding which tasks are appropriate for AI assistance, defining acceptable risk thresholds, selecting appropriate verification techniques, determining when independent review is required and evaluating whether the evidence generated by automated mechanisms is sufficient. In high-consequence systems, humans may also retain explicit authority to reject AI-generated solutions even when they pass all automated checks.

Such governance is essential because automated verification itself operates within assumptions and specifications established by humans. A test suite can be inadequate, a static analyser can miss a vulnerability, and a formal model can omit an important requirement. Human oversight is therefore required not only to inspect the software but also to assess the adequacy of the assurance process itself.

The resulting model is neither human-only nor AI-only. It is a socio-technical division of responsibility:

AI generates and assists; automated tools test and analyse; humans specify, contextualise, challenge and decide.

This provides a more realistic foundation for trustworthy AI-assisted software engineering than the generic concept of a “human in the loop.” The goal is not to maximise human intervention, nor to eliminate it. The goal is to allocate human attention where it provides the greatest assurance value while using AI and automated verification to handle the scale and volume of modern software development.

Ultimately, human oversight should be understood not as a compensatory mechanism for an unreliable AI system, but as a deliberately designed component of the assurance architecture. The question is therefore no longer whether a human should remain involved, but which decisions require human judgement, which evidence can be generated automatically, and how responsibility should be allocated between them.

10. Human–AI Division of Labour

The preceding discussion suggests that trustworthy AI-assisted software engineering should not be organised around a simple choice between human development and autonomous AI development. A more productive approach is to treat software development as a distributed assurance process in which LLMs, automated verification tools and human developers contribute different capabilities.

The underlying principle is one of comparative advantage. LLMs are highly effective at producing and transforming textual and programming artefacts; automated tools are effective at repeatedly applying explicit checks at scale; and human developers are better positioned to interpret context, resolve ambiguity, assess risk and accept responsibility for engineering decisions. Trustworthiness emerges when these capabilities are combined rather than when any one component is expected to perform the entire development process.

10.1 The role of LLMs

LLMs are particularly valuable for tasks involving generation, transformation, explanation and exploration. They can rapidly produce code scaffolding and boilerplate, propose alternative implementations and explain unfamiliar code. They can also assist with documentation, test generation, interpretation of test results, identification of potential defects and suggestions for refactoring.

These capabilities make LLMs well suited to expanding the developer's solution space. Instead of manually constructing a single implementation, a developer can ask an LLM to propose several alternatives, explain their trade-offs or generate candidate tests and then subject those artefacts to further evaluation.

However, generation should not be confused with authority. An LLM's ability to produce a plausible implementation or convincing explanation does not establish that the implementation is correct or appropriate. Generated artefacts should therefore enter the assurance process as candidate solutions, rather than as automatically accepted outputs.

10.2 The role of automated verification

Automated verification mechanisms provide a complementary capability: they can apply explicit checks repeatedly, consistently and at a scale that is difficult to achieve through manual review alone. Compilation and execution can establish whether an implementation satisfies particular operational conditions. Unit and regression testing can detect concrete behavioural failures, while property-based testing can explore broader classes of inputs.

Static analysis can identify structural and semantic patterns associated with defects or security weaknesses. Dependency analysis can examine third-party components, while security scanning can target known vulnerability patterns and insecure configurations. Performance testing can evaluate behaviour under defined workloads, and formal verification can provide stronger guarantees for selected properties where appropriate specifications and verification methods are available.

The important point is that these mechanisms produce evidence, rather than merely opinions about whether code appears correct. Their outputs can be incorporated into the development process and used to constrain subsequent AI-generated implementations.

At the same time, automated verification is not infallible. Each technique operates within its own assumptions and coverage limitations. A test suite may omit important behaviours, a static analyser may miss a vulnerability, and a formal proof may establish a property that does not capture the actual system requirement. Automated verification should therefore be understood as a collection of complementary evidence-producing mechanisms rather than a universal substitute for human judgement.

10.3 The role of humans

Human developers retain responsibility for decisions that depend heavily on context, intent, risk and accountability. These include defining and interpreting requirements, making architectural decisions, classifying risk, establishing acceptance criteria and determining which assurance mechanisms are appropriate.

Human involvement is particularly important when requirements are ambiguous or when different objectives must be balanced. Questions such as whether a performance optimisation is acceptable at the expense of maintainability, whether a security trade-off is appropriate for a particular deployment context, or whether the available verification evidence is sufficient cannot always be answered through automated checks alone.

Humans should also retain particular responsibility for security- and safety-critical decisions and for final acceptance of systems where the consequences of failure are significant. This does not mean that humans should manually inspect every generated artefact. Rather, their role should be concentrated on decisions where contextual expertise, ethical judgement, risk assessment or organisational accountability are required.

10.4 Distributed responsibility rather than rigid task allocation

These roles should not be interpreted as a rigid division in which each activity belongs exclusively to one actor. In practice, responsibilities will overlap.

An LLM may help formulate requirements, propose architectural alternatives or explain a static-analysis finding. Automated tools may generate information that humans use to refine requirements. Developers may use LLMs to interpret verification results or generate additional tests in response to detected failures. Formal verification tools may themselves require human assistance to construct specifications and interpret proof obligations.

The important distinction is therefore not who performs a task, but who or what provides the evidence and who retains responsibility for accepting the result.

For example, an LLM can generate a security test, but the test should be evaluated for adequacy. An LLM can propose an architectural design, but the design should be assessed against system requirements and risks. An automated analyser can report that code satisfies a particular property, but humans must still determine whether that property is relevant to the intended system.

This leads to a more useful model of distributed responsibility:

LLMs generate and assist → automated tools test, analyse and verify → humans specify, contextualise, challenge and decide.

The model is iterative rather than sequential. Evidence generated by automated tools can be fed back to the LLM for refinement; human decisions can change the specification; and new requirements can trigger additional analysis. The development process therefore becomes a continuous interaction between generation, verification and judgement.

10.5 Accountability must remain explicit

The distribution of technical tasks must not result in the distribution of accountability to the point where no actor remains responsible for the final system. AI-generated code does not remove the developer's or organisation's responsibility for deploying that code.

This is particularly important when failures arise from interactions between multiple components. If an LLM generated the implementation, an automated system generated the tests and another tool performed security analysis, responsibility cannot simply be attributed to whichever component happened to produce the defective artefact. The engineering organisation remains responsible for determining whether the assurance process was appropriate and whether the resulting evidence justified deployment.

Accordingly, a trustworthy AI-assisted workflow should preserve traceability of responsibility alongside traceability of software artefacts. It should be possible, where appropriate, to identify which requirements motivated an implementation, which AI-generated artefacts were used, which tests and analyses were performed, which findings were produced, which issues were resolved and who authorised acceptance.

The resulting principle is therefore stronger than a simple division of labour:

AI should expand human engineering capability, automated tools should expand assurance capacity, and humans should retain responsibility for the decisions that determine whether the resulting evidence is sufficient for trust.

This represents a shift from the traditional developer–tool relationship towards a human–AI–verification ecosystem. The objective is not to assign every software-engineering task permanently to one actor, but to design the workflow so that each capability is used where it provides the greatest value while critical decisions remain subject to appropriate evidence and accountable human judgement.

11. Trust Should Be Evidence-Based Rather Than Confidence-Based

A central implication of the literature is that trust in AI-generated software should be based on evidence rather than apparent confidence. LLMs are highly capable of producing code that is syntactically coherent, well structured, and professionally presented. However, the fluency and plausibility of an output provide limited evidence that the resulting software is correct, secure, reliable, or fit for purpose. This creates an important distinction between apparent correctness and demonstrated correctness.

The problem is compounded by the probabilistic and nondeterministic nature of LLM-based code generation. Ouyang et al. (2025) demonstrate that repeated requests for the same programming problem can produce materially different outputs, including when temperature is set to zero. Consequently, confidence derived from inspecting or successfully testing a single generated response should not automatically be generalised to the model's behaviour or to future outputs. Trust must therefore attach not simply to the model or to an individual generated response, but to the artefact and the assurance process through which it has been evaluated.

This suggests that trust should be calibrated according to accumulated evidence. A generated implementation may initially be treated as a candidate solution rather than an accepted one. It can then be subjected to increasingly relevant forms of assurance, such as automated tests, property-based testing, static analysis, security analysis, targeted verification, human review, and, where appropriate, monitoring after deployment. Conceptually, this can be represented as:

model output → testing → analysis → security assurance → human review → deployment monitoring

Each stage contributes a different form of evidence and addresses different potential failure modes. Importantly, the sequence should not be interpreted as a universal linear pipeline in which every project requires every technique. Rather, the appropriate combination and depth of assurance should depend on the software's intended use, potential consequences of failure, and uncertainty surrounding the generated artefact. Higher-risk systems should require stronger and more diverse evidence before acceptance.

This evidence-based view also changes the meaning of AI capability. A more capable model may generate better code on average, but model capability alone does not establish that any particular output is trustworthy. Trustworthiness is instead a property of the relationship between the generated artefact, its intended requirements, the evidence collected about its behaviour, and the human and organisational processes responsible for accepting it. In this sense, verification is not merely a technical post-processing activity; it is the mechanism through which plausible AI output is converted into justified engineering confidence.

The resulting principle can therefore be expressed as:

Trustworthiness ≠ model capability

Rather:

Trustworthiness = capability + evidence + verification + human judgement + accountability

The equation is conceptual rather than mathematical. Its purpose is to emphasise that trustworthy AI-assisted software cannot be inferred from the capabilities of the generating model alone. Trust must be earned through evidence, challenged through verification, contextualised through human judgement, and ultimately supported by clear responsibility for the decision to deploy.

12. Nondeterminism and Reproducibility

Nondeterminism introduces an additional challenge for trustworthy AI-assisted software engineering because the same development task may not reliably produce the same output. Ouyang et al. (2025) found substantial nondeterminism in ChatGPT-generated code across repeated attempts at identical programming problems. Under default settings, more than half of the studied problems across their benchmarks produced no identical test output among repeated candidates, while setting the temperature to zero did not guarantee deterministic behaviour. These findings demonstrate that reproducibility cannot be assumed simply because the same model and programming task are used.

This has important implications for software engineering. If an AI assistant produces different candidate implementations in response to repeated requests, then reproducing how a particular piece of code was generated may require more than preserving the final source code. Nondeterminism can complicate debugging, auditing, incident investigation, regulatory compliance, software maintenance, and research replication. It can also make it more difficult to determine why a particular implementation was selected, what alternatives were considered, and whether a later model interaction would produce materially different behaviour.

In conventional software development, reproducibility is primarily concerned with preserving the artefacts and environments required to reconstruct a particular software behaviour. AI-assisted development introduces an additional layer of provenance because the generated artefact may depend on the state and configuration of the AI interaction itself. Meaningful reconstruction may therefore require recording factors such as:

• the model and model version;

• the prompt and system instructions;

• contextual files and project information supplied to the model;

• retrieved information or external context;

• tool, library, and dependency versions;

• generation parameters, including temperature where applicable;

• generated alternatives and intermediate outputs;

• tests, verification results, and analysis findings;

• accepted, modified, and rejected AI suggestions.

This suggests the need for a more explicit concept of AI provenance: a record of the relevant inputs, configurations, generated artefacts, and assurance activities that contributed to the final software artefact. AI provenance should not be understood as requiring every interaction to be preserved indefinitely or as eliminating nondeterminism. Rather, its purpose is to provide sufficient traceability to reconstruct, understand, evaluate, and audit the development process when this is necessary.

AI provenance also strengthens the evidence-based trust model developed in the previous section. If trust depends on evidence, then that evidence must itself be traceable. A test result, security finding, or verification outcome has greater value when the artefact and environment to which it relates can be identified and reconstructed. Similarly, when AI-generated code is modified by a developer, provenance can help distinguish what was generated by the model from what was subsequently changed, reviewed, tested, or verified by humans and automated tools.

The significance of provenance therefore extends beyond reproducibility. It supports accountability and maintenance by preserving a connection between the software artefact, the AI process that contributed to it, and the assurance evidence used to accept it. In higher-risk systems, such traceability may become an important component of governance and compliance rather than merely a desirable development practice.

The resulting principle is that trustworthy AI-assisted development requires not only reliable verification of the final artefact, but also sufficient provenance to understand how that artefact came to exist and under what conditions its assurance evidence was produced. As AI-generated software becomes increasingly integrated into professional development workflows, provenance should therefore be treated as a core component of trustworthy engineering rather than an optional record of model interaction.

13. From Code Provenance to AI-Assisted Software Provenance

Traditional software engineering already relies on mechanisms such as version control, issue tracking, continuous integration, code review, and build systems to preserve the history of software artefacts. These mechanisms provide traceability between changes, requirements, defects, tests, and released versions. AI-assisted development does not remove the need for such provenance; rather, it introduces additional dependencies that may need to be captured within the existing engineering infrastructure.

For significant AI-generated or AI-modified artefacts, provenance could therefore extend beyond the final source code to include the relevant stages of the AI-assisted development process. A conceptual provenance chain might be represented as:

requirement → prompt and context → model/version → generated candidate → tests → verification results → human modifications → review → deployment

Such a chain establishes a traceable relationship between the intended behaviour, the AI-assisted implementation process, and the evidence used to justify acceptance. It can also make explicit where responsibility and human judgement entered the process. For example, a developer may accept an AI-generated implementation, substantially modify it following review, or reject the initial candidate after testing reveals unexpected behaviour. Recording these transitions provides a richer account of how the final artefact was produced than source-code history alone.

AI-assisted software provenance is particularly important because the generated output may depend on contextual information and model configurations that are not represented in the resulting source code. A later developer who examines only the final implementation may therefore be unable to reconstruct why a particular solution was generated or what alternatives were considered. Preserving relevant provenance can support debugging, maintenance, auditing, incident investigation, and the evaluation of assurance evidence. It can also provide a basis for demonstrating that required development and verification processes were followed.

This becomes especially significant in safety-critical, security-sensitive, and regulated systems, where organisations may need to demonstrate not only that software satisfies specified requirements, but also that appropriate development, testing, verification, and review processes were followed. Provenance can provide an evidential link between these processes and the released artefact. However, provenance should not be confused with correctness: knowing how code was produced does not establish that the code is correct or secure. Rather, provenance makes the evidence and decisions surrounding the artefact more traceable and auditable.

The principle is therefore analogous to reproducible builds, but extends beyond build reproducibility to encompass the broader AI-assisted development process:

A trustworthy software artefact should have a traceable path from requirement to verified implementation.

AI assistance does not eliminate this requirement. Instead, it makes the path from intention to implementation more complex and therefore increases the importance of preserving the relevant provenance. In a trustworthy AI-assisted engineering process, provenance becomes part of the assurance infrastructure that connects requirements, AI generation, human decisions, verification evidence, and the final deployed software.

14. Safety-Critical Software Presents a Higher Threshold

The consequences of unreliable AI-generated code become particularly significant in safety-critical and high-consequence domains. In such environments, the question is not simply whether an LLM can generate code that passes a set of functional tests, but whether the resulting software can satisfy the substantially higher assurance requirements associated with potential harm, system failure, and regulatory responsibility.

Liu et al. (2024) empirically examined LLM-based code generation for safety-critical software and demonstrate why results from general-purpose code-generation benchmarks cannot simply be transferred to high-assurance environments. Performance on conventional programming tasks provides useful evidence about a model's coding capability, but it does not establish that the model can reliably satisfy the domain-specific requirements, constraints, and assurance obligations of safety-critical systems. This distinction reinforces the broader argument that generation capability and software assurance are separate properties.

The appropriate level of assurance is therefore inherently risk-dependent. A developer may reasonably tolerate a minor defect in a prototype, personal script, or low-consequence internal automation tool, particularly where the software is easily inspected, replaced, or corrected. The same defect may be unacceptable when software contributes to the operation of medical devices, aircraft, industrial control systems, automotive safety systems, financial infrastructure, or other critical services. The relevant question is consequently not whether AI-generated code is universally reliable, but whether the level of assurance is proportionate to the consequences of failure.

This leads to the principle of risk-proportionate assurance. Low-risk applications may permit relatively lightweight AI assistance combined with basic testing and human review, whereas higher-risk applications should require stronger and more diverse assurance evidence. Depending on the domain, this may include extensive testing, static and dynamic analysis, security analysis, formal verification, independent review, traceability, provenance, and compliance with applicable engineering or regulatory processes. Human involvement should also become more deliberate as risk increases, particularly for requirements interpretation, architectural decisions, acceptance of residual risk, and final approval.

A useful principle is:

The higher the potential consequence of failure, the less acceptable LLM-only generation and evaluation become.

This does not imply that LLMs have no role in safety-critical development. They may still assist with documentation, test generation, code explanation, requirements analysis, defect investigation, or other activities where their outputs can be independently evaluated. However, the acceptable role of the model should be determined by the assurance demands of the system, rather than by the apparent competence or productivity of the model alone.

This principle also suggests a direction for future AI-assisted development environments. Rather than treating verification as a fixed process, development tools could incorporate risk-aware assurance policies that adapt the required evidence to the characteristics of the software being developed. A system classified as low risk might trigger standard testing and review, while a safety-critical component could automatically require additional analysis, stronger verification techniques, independent review, enhanced provenance, and explicit human approval before deployment.

Risk-proportionate assurance therefore provides a practical mechanism for reconciling the scalability of AI-assisted development with the responsibilities of high-assurance engineering. The objective is not to prevent the use of AI in safety-critical contexts, but to ensure that the strength of the assurance process grows with the potential consequences of failure.

15. A Proposed Trustworthy AI-Assisted Development Model

Synthesising the literature reviewed in this chapter suggests a risk-sensitive, evidence-based model of trustworthy AI-assisted software development. Rather than treating the LLM as an autonomous programmer, the proposed model positions AI generation within a broader socio-technical assurance process in which humans establish intent, AI assists with specification and implementation, automated mechanisms generate verification evidence, humans evaluate context and residual risk, and runtime operation provides further evidence about real-world behaviour.

The model consists of six complementary layers.

Layer 1: Human Requirements

The process begins with human-defined intent. Humans establish the system's purpose, requirements, constraints, risk level, security expectations, and acceptance criteria. This layer is fundamental because an LLM cannot independently determine what a system should do merely from its ability to generate plausible implementations. Requirements therefore provide the basis against which subsequent AI-generated artefacts can be evaluated.

Human involvement at this stage also establishes the assurance context. The consequences of failure, intended operating environment, security requirements, and regulatory constraints should influence the level and type of verification applied later in the process. This implements the principle of risk-proportionate assurance developed earlier in the chapter.

Layer 2: AI-Assisted Specification

The LLM can then assist humans in transforming natural-language requirements into more explicit and testable representations. This may include candidate specifications, examples, edge cases, test cases, assertions, and behavioural properties. The objective is not to delegate requirements definition to the model, but to use the model to expose ambiguity and help make human intent more precise.

This extends the principle demonstrated by TiCoder in Fakhoury et al. (2024), where interactive test generation was used to help formalise user intent during code generation. In this model, specification becomes an intermediate assurance layer between human intention and implementation:

human intention → clarification → executable examples and properties → implementation

This reduces the risk of moving directly from an ambiguous natural-language prompt to code whose correctness is difficult to assess.

Layer 3: AI-Assisted Generation

Once requirements and specifications have been sufficiently clarified, the LLM can generate one or more implementation candidates. These may include alternative designs, code fragments, tests, documentation, or complete implementations depending on the task.

The critical principle is that generated code is treated as a candidate artefact rather than a verified artefact. LLM capability can increase the speed and breadth of implementation, but it does not by itself establish correctness, security, or fitness for purpose. Multiple generated candidates may also be considered where alternative implementations provide useful design choices or expose uncertainty.

This distinction is essential to the overall architecture: generation produces something that must subsequently be challenged by independent forms of evidence.

Layer 4: Automated Verification

Candidate implementations are then subjected to an appropriate combination of automated assurance mechanisms. These may include compilation, unit and regression testing, property-based testing, static analysis, dependency checking, security analysis, dynamic testing, and, where justified by system risk, more rigorous verification techniques.

These mechanisms should not be understood as interchangeable or as forming a simple hierarchy. Each provides different evidence and has different assumptions and failure modes. Testing provides evidence about observed behaviour; static analysis can identify structural and semantic problems without executing the program; security analysis targets adversarial and vulnerability-related properties; and formal verification can provide stronger guarantees for explicitly formalised properties under stated assumptions.

The appropriate combination should therefore depend on the risk and characteristics of the software. The objective is not to maximise the number of verification techniques, but to obtain sufficiently diverse and relevant evidence to justify acceptance.

Layer 5: Human Review and Acceptance

Automated verification does not eliminate the need for human judgement. Developers and other responsible engineers must evaluate whether the implementation satisfies the intended requirements and whether the available evidence is sufficient for acceptance.

Human review should consider correctness, architectural fit, security, maintainability, requirements compliance, unexplained behaviour, and any residual risks identified during verification. Humans are particularly important where requirements are ambiguous, evidence is incomplete or contradictory, or the consequences of failure are significant.

This layer also establishes accountability. AI-generated code does not transfer responsibility for deployment to the model. Humans and organisations remain responsible for determining whether the available evidence is sufficient to justify release.

Layer 6: Runtime Assurance

Assurance does not end at deployment. Software can encounter inputs, environments, dependencies, interactions, and operating conditions that were not fully represented during development. Runtime telemetry, monitoring, anomaly detection, security monitoring, incident response, and regression monitoring can therefore provide additional evidence about real-world behaviour.

Runtime assurance also creates a feedback mechanism. Operational failures, unexpected behaviours, security events, and newly identified requirements can feed back into testing, specification, verification, and future development. The resulting process is therefore better represented as a continuous assurance loop than as a one-time sequence.

The overall model can be represented conceptually as:

Human intent
↓
AI-assisted specification
↓
AI-generated candidate
↓
Automated testing + static analysis + security analysis + targeted verification
↓
Human review and acceptance
↓
Deployment
↓
Runtime monitoring and operational evidence
↓
Feedback into requirements, specification and development

Importantly, these layers should not be interpreted as a rigid pipeline in which every activity occurs once and in exactly the same order. Verification findings may require changes to the specification; human review may reject generated implementations; runtime incidents may introduce new requirements; and higher-risk systems may require additional verification or independent review before deployment. The model is therefore iterative, risk-sensitive, and evidence-driven.

The proposed architecture can be summarised by the division of responsibilities established throughout this chapter: humans define intent and accept responsibility; AI assists with specification and generation; automated mechanisms produce scalable verification evidence; and runtime systems provide continuing evidence after deployment. No individual component is assumed to be sufficient on its own.

This is fundamentally different from treating an LLM as an autonomous programmer. The objective is not to maximise the autonomy of the AI system, but to maximise the amount of trustworthy software produced per unit of engineering effort while preserving appropriate verification, human judgement, traceability, and accountability.

The central principle of the proposed model is therefore:

AI should generate possibilities, verification should generate evidence, humans should determine whether that evidence is sufficient, and runtime operation should continuously challenge the resulting assumptions.

Trustworthy AI-assisted software engineering is consequently best understood not as AI replacing software engineering, but as the integration of human intent, AI capability, automated assurance, and operational feedback into a single evidence-producing development process.

16. Research Gaps

Despite rapid progress in AI-assisted software engineering, the literature remains fragmented across individual capabilities such as code generation, test generation, vulnerability detection, and automated repair. The proposed trustworthy development model highlights a broader research challenge: understanding how these capabilities interact when combined into an end-to-end engineering process. Several important gaps therefore remain.

16.1 End-to-End Empirical Evaluation

A large proportion of existing research evaluates individual components of AI-assisted development rather than complete development workflows. Studies have examined code-generation performance, test generation, vulnerability detection, and verification techniques independently, but considerably less is known about how these components interact when integrated into a realistic software-engineering process.

Future research should therefore compare complete development strategies under controlled conditions, for example:

human-only development

against

LLM generation + human review

against

LLM generation + automated testing

against

LLM generation + testing + static/security analysis + human review

Such comparisons should use the same realistic development tasks and should evaluate not only development speed but also the quality and assurance of the resulting software. Relevant measures should include functional correctness, security, maintainability, technical debt, defect escape rate, review effort, developer cognitive load, reproducibility, and overall assurance effort.

This is important because an AI workflow that produces code more quickly may not produce trustworthy software more efficiently if the additional testing, debugging, review, and remediation effort offsets the initial productivity gain. End-to-end evaluation is therefore necessary to determine whether AI assistance improves trustworthy engineering productivity, rather than merely code-generation speed.

16.2 Multi-Dimensional Definitions of Trustworthiness

A second gap concerns how trustworthy software is evaluated. Functional correctness and pass@k remain important measures in LLM code-generation research, but they capture only part of what it means for software to be trustworthy. Liu et al. (2025) highlight limitations in the adequacy of existing benchmarks for evaluating LLM code generation, reinforcing the need to consider whether benchmark tests sufficiently represent the properties that matter in real software development.

Future benchmarks should therefore move beyond a predominantly functional view of correctness and evaluate multiple dimensions of software quality and assurance. A more comprehensive framework could consider:

correctness + security + reliability + maintainability + performance + reproducibility + human comprehensibility

These dimensions should not necessarily be collapsed into a single numerical score. Different software systems have different risk profiles, and a security-critical component may require substantially stronger security evidence than a low-risk prototype requires. The research challenge is therefore to develop multi-dimensional and risk-sensitive evaluation frameworks capable of representing the trade-offs and assurance requirements of real software systems.

16.3 Human Factors and Verification Behaviour

The human role in AI-assisted development remains another significant research gap. Klemmer et al. (2024) provide important qualitative evidence about how software professionals use AI assistants and the security concerns surrounding their use, but larger controlled studies are needed to determine how AI assistance changes developer behaviour under different conditions.

In particular, research should examine when developers:

• accept or reject AI-generated suggestions;

• detect subtle functional defects;

• identify security vulnerabilities;

• over-trust plausible but incorrect output;

• under-trust correct AI-generated solutions;

• alter their verification effort in response to perceived AI capability.

An especially important question is whether AI assistants improve or degrade developers' verification behaviour. An assistant that increases generation speed but causes developers to reduce scrutiny could produce a net reduction in software assurance. Conversely, an assistant that makes testing, analysis, and verification easier could increase the amount of assurance performed. Understanding this relationship is essential for evaluating the real impact of AI-assisted development.

16.4 Developer Expertise, AI Reliance and Verification Performance

AI assistance may affect developers with different levels of expertise in substantially different ways. Less experienced developers may benefit from AI-generated explanations, examples, and implementation scaffolding, but may lack the domain knowledge required to recognise subtle errors in plausible-looking output. More experienced developers may be better equipped to identify defects and assess architectural consequences, but expertise does not eliminate the possibility of automation bias or excessive reliance on apparently sophisticated suggestions.

The relationship between:

developer expertise × AI reliance × verification performance

therefore warrants systematic investigation.

Future studies should examine not only whether developers can produce better code with AI assistance, but whether they can evaluate AI-generated code effectively. This distinction is critical because trustworthy AI-assisted development depends on the ability of humans to challenge generated output rather than simply accept it. Expertise should therefore be studied as a factor influencing both generation performance and assurance performance.

16.5 Security-Aware Development Environments

Security should increasingly be investigated as an integrated property of AI-assisted development rather than as a final review activity. The literature indicates that AI-generated code cannot be assumed to be secure by default, while LLMs themselves may have limited reliability when identifying vulnerabilities (Negri-Ribalta et al., 2024; Sajadi et al., 2025). This creates a need for development environments in which security analysis is continuously incorporated into the generation and refinement process.

A future workflow could take the form:

LLM suggestion → security analysis → warning → revised suggestion → developer approval

Such closed-loop systems could allow security findings to influence code generation before vulnerabilities become embedded in later development stages. However, research should also examine whether repeated automated remediation introduces new defects, removes legitimate functionality, or creates false confidence. Security-aware generation should therefore remain subject to independent verification rather than treating the AI's ability to respond to security feedback as evidence that the resulting code is secure.

16.6 Verification-Aware AI Agents

The emergence of increasingly agentic coding systems creates a further research challenge. An AI agent capable of generating code, executing tests, analysing failures, modifying implementations, and repeating the process could substantially reduce human effort. However, greater autonomy also increases the potential scale of undetected errors. An agent that controls both implementation and parts of its own verification process may repeatedly reinforce an incorrect assumption rather than independently challenge it.

Future systems should therefore be designed around the principle of verification-aware agency. An agent should operate within explicit constraints concerning:

• acceptable classes of changes;

• required test and verification evidence;

• security requirements;

• confidence or evidence thresholds;

• conditions requiring human escalation;

• provenance and auditability;

• restrictions on autonomous deployment.

Research should investigate how such constraints affect both productivity and assurance. In particular, agentic systems should be evaluated not only according to how much human effort they eliminate, but according to whether they preserve or improve the reliability of the overall assurance process.

The central research challenge is consequently to determine how much autonomy can be introduced without weakening the independence, diversity, and human accountability of verification. This extends the broader argument of the chapter: trustworthy AI-assisted development is not simply a question of making AI systems better at writing code. It requires designing development environments in which increasing AI capability is matched by appropriate increases in specification quality, verification strength, human oversight, provenance, and accountability.

These gaps suggest that the next stage of research should move from evaluating AI capabilities in isolation toward evaluating AI-assisted software engineering as an integrated socio-technical assurance system.

17. Discussion

The literature suggests that the most promising future for LLMs in software engineering is neither full automation nor simple human supervision.

Full automation is problematic because LLMs remain probabilistic, non-deterministic and capable of generating incorrect or insecure code.

Simple human supervision is also insufficient because human review is expensive and itself subject to error.

The more defensible model is therefore human–AI–tool collaboration.

The LLM contributes speed, flexibility and broad pattern knowledge.

Automated verification contributes repeatability and systematic checking.

Humans contribute contextual understanding, requirements interpretation, architectural judgement and accountability.

These components compensate for one another's weaknesses.

The relationship can therefore be understood as:

Component

Main contribution

Main risk

Human

Intent, context, judgement

Cognitive limitations

LLM

Generation and reasoning assistance

Hallucination, nondeterminism

Testing

Behavioural evidence

Incomplete coverage

Static analysis

Systematic defect detection

False positives/negatives

Security tools

Vulnerability detection

Limited semantic/contextual understanding

Formal verification

Strong assurance

Cost and scalability

Runtime monitoring

Real-world evidence

Detects problems after deployment

The implication is that trustworthiness is an emergent property of the development process, not a property that can be assigned to the LLM alone.

18. Conclusion

LLMs are changing software engineering by making the generation, transformation, explanation, and testing of code increasingly accessible and scalable. The literature reviewed in this paper demonstrates, however, that increased generation capability does not automatically produce trustworthy software. LLM-generated code can be plausible yet incorrect, functionally adequate yet insecure, or apparently well engineered yet poorly aligned with the intended requirements. Nondeterminism further complicates reproducibility and auditing, while the use of AI for both generation and evaluation can introduce correlated errors. These limitations mean that trust cannot reasonably be inferred from the apparent quality of an AI-generated response or from the capability of the underlying model alone.

The central argument of this paper is therefore that trustworthiness should be established through an evidence-producing engineering process. AI-generated code should be treated as a candidate implementation rather than a verified artefact. Human-defined requirements and acceptance criteria provide the basis for evaluation; AI can assist in transforming ambiguous intentions into specifications, examples, tests, and candidate implementations; automated techniques can provide scalable evidence through testing, property-based testing, static analysis, security analysis, dependency checking, dynamic analysis, and, where appropriate, formal verification; and humans must interpret this evidence, evaluate contextual risks, and retain responsibility for acceptance.

This leads to a shift from AI-as-generator to AI-within-assurance. The distinction is important because it changes the role of AI in the development process. The objective is not to determine how much of software engineering can be delegated to an LLM, but how AI capability can be integrated without weakening the mechanisms through which software quality and trust are established. In this model, AI increases generation and engineering capacity, automated verification increases assurance capacity, and human judgement provides contextual interpretation, risk assessment, and accountability.

The analysis also demonstrates that testing is necessary but insufficient. A passing test suite provides evidence about tested behaviours, but does not establish complete correctness, security, maintainability, or fitness for purpose. Trustworthy development therefore requires complementary forms of evidence whose failure modes are not identical. This is particularly important for security, where functional correctness does not imply resistance to adversarial behaviour, and for safety-critical systems, where the consequences of failure require substantially stronger assurance. The appropriate level of verification should consequently be risk-proportionate: as the potential consequences of failure increase, the required diversity and strength of assurance evidence should also increase.

Human involvement remains central, but it should not be reduced to the principle that developers simply “check the AI's work”. Humans have limited attention and expertise, and increased AI-generated output can itself create an assurance bottleneck. Effective human oversight should therefore be concentrated where human judgement provides the greatest comparative advantage: defining requirements, resolving ambiguity, assessing architecture and risk, interpreting conflicting or incomplete evidence, challenging assumptions, and making acceptance decisions. The resulting division of labour is not human versus AI, but a human–AI–verification ecosystem in which each component contributes different capabilities while responsibility remains explicit.

Nondeterminism and provenance add a further dimension to trustworthy AI-assisted development. Because repeated interactions with an LLM may produce materially different outputs, reproducibility may require preserving not only source code but also relevant model versions, prompts, contextual information, tools, generated alternatives, and verification results. AI-assisted software provenance can therefore extend traditional version control and development histories, providing a traceable relationship between requirements, AI interactions, implementation, verification, human modifications, and deployment. Such traceability becomes particularly important for safety-critical, security-sensitive, and regulated systems.

The proposed six-layer model brings these findings together:

human requirements → AI-assisted specification → AI-generated candidate → automated verification → human review → runtime assurance

The model is deliberately iterative rather than strictly linear. Verification can expose problems in requirements or implementation; human review can trigger further generation and testing; and runtime monitoring can provide evidence that feeds back into subsequent development. The model therefore treats software assurance as a continuous process rather than a final gate applied immediately before deployment.

Several research gaps remain. Future work should move beyond isolated evaluations of code generation, test generation, or vulnerability detection towards controlled studies of complete AI-assisted development pipelines. Evaluation should also become multidimensional, incorporating security, reliability, maintainability, reproducibility, human comprehensibility, and other properties alongside functional correctness. Further research is needed to understand how AI changes developer verification behaviour, how expertise influences reliance and error detection, how security can be integrated continuously into AI-assisted development environments, and how increasingly autonomous coding agents can operate under explicit verification, escalation, provenance, and accountability constraints.

Ultimately, trustworthy AI-assisted software engineering should not be understood as the search for an LLM that can be trusted to write software correctly on its own. The more defensible objective is to construct development processes in which AI-generated software can be systematically challenged, evaluated, traced, and accepted according to evidence. The key principle can therefore be stated simply:

AI should generate possibilities, verification should generate evidence, and humans should determine whether that evidence is sufficient to justify trust.

Trustworthiness is consequently not an inherent property of the model or its output. It is an emergent property of the software, the assurance evidence surrounding it, the humans responsible for evaluating that evidence, and the organisational processes that govern its deployment and continued operation.

References

Bose, D.B. (2025) ‘From prompts to properties: Rethinking LLM code generation with property-based testing’, Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, pp. 1660–1665. doi: 10.1145/3696630.3728702.

Dolcetti, G. and Iotti, E. (2025) ‘A dual perspective review on large language models and code verification’, Frontiers in Computer Science, 7, 1655469. doi: 10.3389/fcomp.2025.1655469.

Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S. and Lahiri, S. (2024) ‘LLM-based test-driven interactive code generation: User study and empirical evaluation’, IEEE Transactions on Software Engineering, 50, pp. 2254–2268.

Hamer, S., d'Amorim, M. and Williams, L. (2024) ‘Just another copy and paste? Comparing the security vulnerabilities of ChatGPT generated code and StackOverflow answers’.

Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J. and Wang, H. (2024) ‘Large language models for software engineering: A systematic literature review’, ACM Transactions on Software Engineering and Methodology, 33(8), Article 220. doi: 10.1145/3695988.

Huang, X. et al. (2024) ‘A survey of safety and trustworthiness of large language models through the lens of verification and validation’, Artificial Intelligence Review, 57, 175. doi: 10.1007/s10462-024-10824-0.

Klemmer, J.H., Horstmann, S.A., Patnaik, N., Ludden, C., Burton, C., Powers, C., Massacci, F., Rahman, A., Votipka, D., Lipford, H.R., Rashid, A., Naiakshina, A. and Fahl, S. (2024) ‘Using AI assistants in software development: A qualitative study on security practices and concerns’, Proceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security, pp. 2726–2740. doi: 10.1145/3658644.3690283.

Liu, M., Wang, J., Lin, T., Ma, Q., Fang, Z. and Wu, Y. (2024) ‘An empirical study of the code generation of safety-critical software using LLMs’, Applied Sciences, 14(3), 1046. doi: 10.3390/app14031046.

Liu, et al. (2025) ‘Evaluating the test adequacy of benchmarks for LLMs on code generation’, Journal of Software: Evolution and Process. doi: 10.1002/smr.70034.

Negri-Ribalta, C., Geraud-Stewart, R., Sergeeva, A. and Lenzini, G. (2024) ‘A systematic literature review on the impact of AI models on the security of code generation’, Frontiers in Big Data, 7, 1386720. doi: 10.3389/fdata.2024.1386720.

Ouyang, S., Zhang, J.M., Harman, M. and Wang, M. (2025) ‘An empirical study of the non-determinism of ChatGPT in code generation’, ACM Transactions on Software Engineering and Methodology, 34(2), Article 42. doi: 10.1145/3697010.

Peng, S., Kalliamvakou, E., Cihon, P. and Demirer, M. (2023) ‘The impact of AI on developer productivity: Evidence from GitHub Copilot’.

Sajadi, A., Le, B., Nguyen, A., Damevski, K. and Chatterjee, P. (2025) ‘Do LLMs consider security? An empirical study on responses to programming questions’, Empirical Software Engineering, 30, 101. doi: 10.1007/s10664-025-10658-6.

Shang, Y., Zhang, Q., Fang, C., Gu, S., Zhou, J. and Chen, Z. (2025) ‘A large-scale empirical study on fine-tuning large language models for unit testing’, Proceedings of the ACM on Software Engineering, 2(ISSTA), Article ISSTA074, pp. 1678–1700. doi: 10.1145/3728951.

Wang, Y., Guo, S. and Tan, C.W. (2025) ‘From code generation to software testing: AI Copilot with context-based retrieval-augmented generation’, IEEE Software, 42(4), pp. 34–42. doi: 10.1109/MS.2025.3549628.

Yang, B., Dang, J., Liu, H. and Jin, Z. (2026) ‘Advancing LLM-generated code reliability: A hybrid approach for hallucination detection’, IEEE Transactions on Software Engineering, 52(2), pp. 578–594. doi: 10.1109/TSE.2025.3640641.

Contact

Reach out via email for inquiries.

Email

Subscribe to newsletter

info@grcadvisory.ch

© 2025. All rights reserved.