Productivity, Quality, Security and the Transformation of the Software Engineering Lifecycle

Abstract

The rapid development of large language models (LLMs) is transforming software engineering by enabling increasingly capable forms of code generation, code understanding, testing, debugging and software maintenance. Although these capabilities may increase development speed and reduce the cost of routine programming activities, their broader implications for software quality, security, developer roles and organisational governance remain uncertain. This paper critically examines the impact of AI-assisted software engineering by considering the relationship between productivity, software quality, security, human–AI collaboration and organisational control.

The analysis argues that the value of AI in software engineering cannot be adequately measured through code-generation speed or functional correctness alone. AI-generated software may introduce security vulnerabilities, maintenance costs, architectural inconsistencies and additional verification effort, while the non-deterministic nature of large language models creates further challenges for reproducibility, provenance and configuration management. Consequently, meaningful evaluation must extend beyond isolated coding benchmarks towards repository-level, multidimensional and longitudinal assessment of software quality and total engineering effort.

The paper further argues that AI is more likely to recompose software engineering work than eliminate software engineers. As AI assumes a greater proportion of routine implementation, human expertise becomes increasingly concentrated on requirements interpretation, architecture, verification, security, contextual judgement and accountability. Effective human–AI collaboration therefore requires iterative interaction, independent verification and calibrated trust rather than uncritical acceptance of generated output.

At an organisational level, successful adoption requires AI to be embedded within a broader engineering and governance system. Model and prompt management, repository context, automated testing, security controls, provenance, human approval and risk-based governance become increasingly important as AI systems gain greater autonomy. The paper concludes that the central challenge is not whether AI can generate code, but whether organisations can consistently transform AI-generated output into secure, maintainable, reliable and economically valuable software.

Keywords: artificial intelligence, large language models, software engineering, code generation, software security, human–AI collaboration, software quality, developer productivity, AI governance, autonomous software development

1. Introduction

Artificial intelligence (AI), and particularly large language models (LLMs), is becoming increasingly embedded within software engineering (SE), transforming how software is specified, designed, implemented, tested, understood, secured and maintained. Rather than representing a narrow advance in automated code completion, LLM-based systems are developing into general-purpose software engineering capabilities that can interact with developers through natural language and support activities across the software development lifecycle. A systematic literature review by Hou et al. (2024), analysing 395 studies published between 2017 and January 2024, identifies applications of LLMs across six broad areas of software engineering: requirements engineering, software design, software development, software quality assurance, software maintenance and software management. This breadth is significant because it indicates that the impact of generative AI is not confined to programming itself; instead, it has the potential to reshape the wider system through which software is conceived, produced and governed.

The emergence of LLMs changes the interface through which developers interact with software systems and development tools. Historically, software engineers have translated requirements and design decisions into formal programming languages through a largely manual process. LLM-based tools introduce a different interaction model in which natural-language intent can become an intermediate interface between human reasoning and executable artefacts. Developers can ask an AI system to generate code, explain an unfamiliar implementation, suggest an API, construct tests, identify defects, propose fixes or document existing functionality. Nam et al. (2024), for example, demonstrate that an LLM integrated directly into the development environment can assist developers with code explanation, API understanding, domain-specific terminology and usage examples. Their study of 32 participants found that the conversational tool could support task completion more effectively than conventional web search in the studied tasks. AI therefore has relevance not only to code production but also to the cognitive work involved in understanding and navigating software systems.

This development has important implications for the economics and organisation of software development. Controlled evidence suggests that AI assistance can produce substantial productivity gains for particular programming activities. In an experiment involving GitHub Copilot, Peng et al. (2023) found that developers with access to the AI pair programmer completed an HTTP-server implementation task 55.8% faster than developers in the control group. Such findings demonstrate the potential of AI to reduce the time required for specific coding tasks and to lower the effort associated with translating relatively well-defined intentions into working implementations. However, productivity should not be equated with the quantity or speed of code generation. Software engineering involves requirements interpretation, architectural reasoning, integration, testing, security, maintenance and communication, meaning that faster code production does not automatically imply faster or better software delivery.

Indeed, the introduction of generative AI may redistribute rather than eliminate software engineering work. As AI becomes more capable of producing implementation artefacts, the relative importance of activities such as specifying requirements, clarifying intent, evaluating alternatives, validating behaviour and managing system-level constraints may increase. The developer's role can consequently shift from being primarily a producer of source code towards being a specifier, evaluator, integrator and governor of machine-generated artefacts. This represents a fundamental change in the division of labour between humans and computational systems. The key engineering capability is no longer simply the ability to write code efficiently, but the ability to determine whether a generated implementation correctly expresses the intended behaviour and is appropriate within its technical and organisational context.

Fakhoury et al. (2024) provide particularly strong evidence for the importance of this distinction. Their TiCoder workflow addresses a central problem of natural-language-driven code generation: natural-language requirements are inherently informal and therefore difficult to verify directly. By using tests as a mechanism for clarifying and partially formalising user intent, the approach creates an interactive process in which developers and LLMs iteratively refine the desired behaviour. In a mixed-methods study involving 15 programmers, participants using the workflow were significantly more likely to evaluate generated code correctly and reported lower task-induced cognitive load. At scale, the approach produced an average absolute improvement of 45.97 percentage points in pass@1 code-generation accuracy across four LLMs and two Python datasets within five user interactions (Fakhoury et al., 2024). These findings suggest that the most effective use of AI may not be unrestricted automation, but structured human-AI interaction in which human intent is progressively formalised and machine-generated outputs are continuously evaluated.

This has wider implications for how software quality should be understood in an AI-assisted environment. Code generation is only one stage in the transformation from requirements to reliable software. An LLM can produce syntactically valid, plausible and locally functional code while nevertheless misunderstanding the underlying requirement, violating architectural constraints, introducing a security weakness or creating maintenance problems. Hou et al. (2024) identify hallucinations, security vulnerabilities, limited generalisability and interpretability challenges among the continuing limitations of LLMs for software engineering. The growing capability of AI therefore creates an apparent paradox: as the marginal cost of producing code decreases, the relative importance of verifying that code increases.

The distinction between code generation and software engineering is particularly important because generated code is probabilistic rather than inherently deterministic. Ouyang et al. (2024) demonstrate that ChatGPT can produce substantially different implementations in response to the same coding problem. Examining 829 code-generation problems across CodeContests, APPS and HumanEval, they report considerable non-determinism in generated outputs and show that setting temperature to zero does not necessarily guarantee deterministic behaviour. Such variability has implications for correctness, consistency, developer trust and reproducibility. In conventional software development, identical source code is expected to produce the same behaviour under equivalent conditions. In LLM-assisted development, by contrast, the process that produced the code may itself be variable. Consequently, prompts, model versions, parameters, contextual information and interaction histories can become relevant components of the engineering process.

Non-determinism also complicates the notion of reproducibility. If two developers provide ostensibly identical instructions to an AI system but receive different implementations, the resulting software may differ in functionality, security characteristics or maintainability. This creates a requirement for stronger engineering controls around AI-assisted development, including versioning, testing, review, provenance and documentation of significant AI interactions. It also challenges organisations to distinguish between reproducibility of the generated artefact and reproducibility of the generation process. As Ouyang et al. (2024) argue, this issue is relevant not only to industrial development but also to empirical research, where unstable outputs can affect the validity and replicability of reported findings.

The security implications of AI-assisted software development provide an even stronger illustration of why productivity cannot be evaluated independently of quality and risk. Hamer, d'Amorim and Williams (2024) compared security vulnerabilities in ChatGPT-generated Java code with vulnerabilities in corresponding Stack Overflow answers. Across 108 snippets from each source, they identified 248 vulnerabilities in ChatGPT-generated code compared with 302 in the Stack Overflow material. Although the AI-generated code contained fewer vulnerabilities in their study, it still exhibited substantial security weaknesses and covered 19 different Common Weakness Enumeration categories. Their findings therefore do not support the conclusion that AI-generated code is inherently less secure; rather, they reinforce the broader point that code obtained from an AI system should not be trusted merely because it is generated rather than copied from a human-authored source (Hamer, d'Amorim and Williams, 2024).

Wang et al. (2024) reach a complementary conclusion through CodeSecEval, a benchmark designed to evaluate secure code generation and repair across 44 vulnerability types and 180 samples. Their results indicate that contemporary LLMs frequently overlook security issues during both code generation and code repair, producing vulnerable implementations in cases where security properties should be explicitly considered. The difficulty is not necessarily that LLMs are incapable of producing secure code, but that security is not reliably preserved as an implicit property of generated implementations. This suggests that secure AI-assisted development requires security requirements, automated analysis, testing and human review to remain explicit parts of the engineering process.

The practical significance of these findings is reinforced by Klemmer et al. (2024), who examined how software professionals actually use AI assistants in security-sensitive development. Their study combined 27 semi-structured interviews with software professionals and an analysis of 190 relevant Reddit posts and comments. Despite widespread concerns about the quality and security of AI-generated suggestions, participants reported using AI assistants for security-related activities including code generation, threat modelling and vulnerability detection. At the same time, their mistrust of AI outputs led them to scrutinise suggestions in ways similar to other imperfect sources of software engineering knowledge. This indicates that professional adoption is neither unconditional trust nor complete rejection. Instead, AI is becoming another source of engineering assistance whose outputs require contextual evaluation.

This finding is important because it places AI-assisted software development within a broader socio-technical system. The risk associated with an AI-generated component is determined not only by the model that produced it, but also by how developers use the model, what contextual information is supplied, what verification mechanisms are available and what organisational controls govern its deployment. An insecure suggestion that is rejected during code review may have little practical impact; the same suggestion, if accepted without validation and deployed in a security-sensitive application, may create substantial organisational and technical risk. AI governance must therefore address the complete development workflow rather than treating the underlying model as the sole source of risk.

The issue becomes even more consequential in safety-critical software. Liu et al. (2024) investigate LLM-based code generation in the context of safety-critical software, where failures can have consequences beyond conventional software defects. Their work highlights both the potential of LLMs to improve development efficiency and the additional constraints associated with domains such as nuclear energy, aviation, automotive systems and rail transportation. In such environments, software must satisfy stringent requirements concerning correctness, reliability, traceability, validation and assurance. The possibility of using AI to accelerate implementation therefore cannot be separated from the need to demonstrate that the resulting software satisfies safety requirements. The more consequential the software, the less appropriate it becomes to treat AI-generated code as an authoritative implementation simply because it appears plausible or passes limited tests.

Taken together, these studies suggest that the central problem is not whether AI can generate code. It clearly can. The more consequential question is under what conditions AI-generated software can be considered trustworthy. Trustworthiness requires a chain of evidence connecting requirements to implementation, implementation to tests, tests to expected behaviour, and software behaviour to the relevant security, safety and operational constraints. This places testing and verification at the centre of AI-assisted development rather than treating them as downstream activities performed after code generation.

The implications extend beyond implementation to software maintenance and comprehension. Much of the economic cost of software is associated not with initially writing code but with understanding, modifying and maintaining existing systems. Nam et al. (2024) demonstrate that LLMs can support this cognitive dimension by helping developers understand code, APIs and domain concepts directly within the IDE. This is particularly significant for legacy systems, unfamiliar codebases and complex development environments where documentation may be incomplete or difficult to navigate. AI could therefore reduce some of the knowledge barriers associated with software maintenance and developer onboarding. However, the same risks of hallucination, incomplete context and misplaced confidence remain relevant: an incorrect explanation of legacy code can be particularly dangerous because it may influence subsequent modifications to systems whose original design rationale is already poorly understood.

The evidence therefore points towards a transformation of the software development lifecycle rather than a simple substitution of programmers with AI. Hou et al. (2024) show that LLM applications span requirements, design, development, quality assurance, maintenance and management. Fakhoury et al. (2024) demonstrate the value of interactive intent clarification and test-driven generation; Nam et al. (2024) show the potential of AI for code comprehension; Peng et al. (2023) provide evidence of productivity gains; and the studies by Hamer, d'Amorim and Williams (2024), Wang et al. (2024), Klemmer et al. (2024), Liu et al. (2024) and Ouyang et al. (2024) demonstrate why security, safety, human oversight and reproducibility remain essential. The emerging evidence is consequently best interpreted as a transition towards human-AI software engineering, rather than fully autonomous software development.

Within this model, the distribution of responsibility between humans and machines becomes a central design question. AI systems are particularly well suited to activities that are repetitive, pattern-based, information-intensive or amenable to rapid generation and comparison. Humans remain comparatively important for resolving ambiguity, establishing priorities, making architectural trade-offs, interpreting organisational and ethical constraints, judging whether evidence is sufficient and accepting responsibility for consequential decisions. This does not imply a fixed division of labour. As AI capabilities evolve, the boundary between machine-supported and human-led activities will continue to shift. Nevertheless, the need for human judgement is unlikely to disappear because software engineering ultimately concerns the relationship between technical artefacts and human, organisational and societal objectives.

This shift also raises questions of accountability and governance. In traditional development, responsibility for software can generally be traced through developers, reviewers, requirements and organisational processes. AI-assisted development introduces additional actors and dependencies: foundation models, prompts, model versions, retrieval systems, plugins, automated agents, external tools and generated artefacts may all contribute to the final system. Organisations must therefore establish mechanisms for determining who is responsible for AI-generated code, how its provenance should be recorded, which components require human review, how security and quality should be assessed, and when AI use is inappropriate. These requirements are particularly important where software is safety-critical, security-sensitive, regulated or operationally consequential.

The emerging evidence also suggests that organisations should avoid framing AI adoption primarily as a procurement or productivity initiative. The central organisational question is not simply which AI coding assistant should be deployed, but how software engineering processes should change when AI becomes a routine participant in development. This may require changes to development standards, code-review practices, testing strategies, secure development lifecycles, documentation, developer training, architecture governance and accountability structures. AI literacy consequently becomes a software engineering competency rather than merely a technical skill associated with prompt writing. Developers need to understand not only how to obtain useful outputs from LLMs but also how to recognise uncertainty, test generated behaviour, identify security weaknesses and determine when AI assistance should not be relied upon.

The research literature also reveals important limitations in the evidence base. Hou et al. (2024) identify a substantial imbalance between academic and industrial evidence: among the 395 studies reviewed, only a very small number used industrial datasets. This raises questions about how well results obtained from curated benchmarks and public repositories generalise to large, heterogeneous and continuously evolving software systems. Controlled productivity experiments, such as that of Peng et al. (2023), provide valuable evidence of potential efficiency gains, but they do not necessarily capture the cumulative costs of review, debugging, integration, security analysis and maintenance in production environments. Similarly, benchmark-based evaluations of code generation may measure whether a generated program passes predefined tests without fully capturing architectural quality, maintainability, security or alignment with organisational requirements.

Consequently, the evaluation of AI in software engineering should move beyond narrow measures such as lines of code generated, completion time or benchmark accuracy. A more meaningful evaluation framework should consider productivity, correctness, security, maintainability, developer cognitive load, reproducibility, system reliability and organisational risk. Fakhoury et al. (2024), for instance, demonstrate that an intervention can improve both generation accuracy and the developer's ability to evaluate generated code, while Klemmer et al. (2024) show that actual professional use is shaped by trust and security concerns. These findings imply that AI effectiveness is multidimensional: an AI system that generates code rapidly but increases review burden or introduces unacceptable security risk may produce little net organisational value.

The evidence therefore supports a more nuanced understanding of AI's impact on software engineering. AI can substantially accelerate selected activities, broaden access to programming assistance, reduce some forms of cognitive effort and support developers in navigating unfamiliar code. At the same time, AI-generated outputs remain probabilistic, potentially insecure, context-sensitive and difficult to reproduce. The appropriate response is neither uncritical adoption nor categorical resistance. Instead, software engineering must evolve towards development processes in which AI-generated artefacts are treated as candidate solutions requiring evidence-based validation, rather than authoritative solutions requiring minimal scrutiny.

This paper therefore examines the transformation through four research questions:

1. How is AI being applied across the software development lifecycle?

2. What empirical evidence exists regarding its impact on software developer productivity and software engineering performance?

3. What technical, security, organisational and governance risks arise from the increasing use of AI-generated software?

4. How should software engineering practices, roles and governance mechanisms evolve in response to the increasing integration of AI?

The central argument is that AI should not be understood simply as an automation technology or as a substitute for software developers. It is more appropriately understood as a general-purpose software engineering capability that changes how software work is distributed between humans and machines. Its value depends not only on model capability but on the quality of the surrounding socio-technical system: requirements, architecture, development practices, testing, security controls, developer expertise, organisational governance and mechanisms of accountability.

The fundamental impact of AI on software development therefore lies not in the automated production of more code, but in the transformation of the software engineering system itself. As the marginal cost of generating implementations falls, the relative importance of specifying what software should do, determining whether it does it correctly, establishing whether it is secure and maintainable, and deciding whether it should be deployed becomes greater. The strategic challenge for software engineering is consequently to move from a model in which developers primarily create software to one in which developers direct, evaluate, verify and govern increasingly capable machine-generated software.

The future of AI-assisted software engineering will therefore depend less on whether machines can generate code—which the existing evidence increasingly establishes—and more on whether organisations can develop reliable processes for integrating machine-generated outputs into human-centred engineering practices. The most consequential transformation may ultimately be a change in what it means to be a software engineer: from producing every implementation detail manually towards defining intent, managing complexity, interrogating machine-generated solutions, validating evidence and exercising engineering judgement. AI may consequently reduce the amount of code humans must write while increasing the importance of the decisions humans must make.

2. AI Across the Software Development Lifecycle

The influence of artificial intelligence (AI) on software development extends well beyond the generation of source code. Large language models (LLMs) are increasingly being applied to activities spanning requirements engineering, software design, implementation, quality assurance, maintenance and software management. Hou et al. (2024), in a systematic literature review of 395 primary studies, identify these six areas as the principal domains of LLM application within software engineering. Notably, software development accounts for approximately 56.65% of the studies reviewed, demonstrating the field's strong concentration on code-related applications, while also revealing that LLM research increasingly encompasses activities before and after implementation.

This breadth matters because software development is not a sequence of independent tasks. Requirements shape architectural decisions; architecture constrains implementation; implementation determines what must be tested; and all of these decisions influence maintainability, security and operational performance. An error introduced at an early stage can therefore propagate through subsequent stages, becoming increasingly expensive to identify and correct. AI should consequently be evaluated not as an isolated coding technology, but as a capability that can influence the entire engineering workflow.

The central issue is therefore not simply where AI can be inserted into the software lifecycle, but how AI-mediated decisions interact across lifecycle stages. If AI is used to interpret requirements, propose an architecture and subsequently generate the implementation, an incorrect assumption can potentially be carried forward through the entire development chain. Conversely, if AI-generated outputs are continuously constrained by explicit requirements, tests, architectural rules and human review, AI may support a more integrated and iterative engineering process. The distinction between these two models—uncontrolled generation and continuously validated augmentation—is fundamental to understanding the role of AI in software engineering.

This chapter focuses on three particularly consequential areas: requirements engineering, software design and architecture, and code generation and completion. Together, these represent a progression from defining what a system should do, through determining how it should be structured, to producing the implementation itself.

2.1 Requirements Engineering

Requirements engineering represents the first major point at which AI can influence the software development lifecycle. Requirements establish the intended functionality, constraints and quality attributes of a system and consequently provide the basis for subsequent design, implementation and testing. Hou et al. (2024) identify a range of LLM applications in requirements engineering, including requirements classification, analysis and evaluation, specification generation, requirements elicitation, ambiguity treatment, formalisation and traceability.

The potential value of LLMs in this domain derives partly from the nature of requirements themselves. Unlike source code, requirements are frequently expressed in natural language. They may contain ambiguity, incomplete assumptions, inconsistent terminology or statements whose meaning depends on domain knowledge. This creates a fundamental translation problem: human intentions are generally expressed informally, whereas software must ultimately satisfy precise behavioural and technical constraints.

LLMs are well positioned to assist with this translation. They can summarise large requirements documents, identify potentially ambiguous statements, classify requirements, suggest missing information, convert informal descriptions into structured specifications and generate candidate acceptance criteria or test cases. In principle, such capabilities can reduce the cognitive and administrative effort associated with requirements analysis while helping developers and stakeholders identify inconsistencies earlier in the lifecycle.

However, the same capability creates a fundamental risk. An LLM may produce a coherent and technically sophisticated interpretation of a requirement that is nevertheless wrong. The problem is therefore not necessarily whether an AI system can transform natural language into a technical artefact, but whether it has correctly understood the intended meaning behind the language.

This distinction is particularly important because requirements frequently contain information that is implicit rather than explicitly stated. Domain experts may assume regulatory, operational or business constraints without documenting them in detail. A human engineer can sometimes infer such constraints from organisational knowledge and prior experience; an LLM may instead generate a plausible interpretation based primarily on linguistic patterns. Consequently, fluency should not be confused with requirements correctness.

Fakhoury et al. (2024) provide an important response to this problem through their TiCoder workflow, which uses tests as a mechanism for guided intent clarification and partial formalisation. Rather than treating natural-language code generation as a one-step process, TiCoder introduces an interactive workflow in which tests help clarify the intended behaviour and constrain the generated implementation. In a mixed-methods study involving 15 programmers, the authors found that participants using the workflow were significantly more likely to evaluate AI-generated code correctly and reported lower task-induced cognitive load. Their larger-scale evaluation across four LLMs and two Python datasets found an average absolute improvement of 45.97 percentage points in pass@1 code-generation accuracy within five user interactions.

The significance of this finding extends beyond code generation. It demonstrates the value of making requirements observable and testable before treating an AI-generated implementation as correct. Tests can act as an intermediate representation of intent: instead of relying solely on a natural-language statement such as "process customer orders correctly", the development process can specify observable conditions that distinguish correct from incorrect behaviour.

This suggests that the relationship between requirements engineering and AI-assisted development should be viewed as a process of progressive formalisation:

Natural-language intent → clarification → executable constraints → AI-generated implementation → verification

Within this model, AI does not replace requirements engineering. Instead, it can accelerate the movement from informal intent towards representations that can be evaluated mechanically. This is an important distinction because the quality of the resulting software remains dependent on whether the underlying intent has been correctly established.

The principle is therefore broader than prompt quality alone:

The reliability of AI-generated software depends substantially on the quality with which human intent is specified, constrained and made testable.

Human involvement remains essential because ambiguity is not simply a linguistic problem. Determining what a requirement should mean involves business priorities, domain knowledge, risk tolerance and organisational responsibility. AI can identify potentially ambiguous statements or propose interpretations, but stakeholders and engineers must determine which interpretation is actually correct.

This has an important implication for requirements engineering practice. If AI becomes increasingly capable of generating software from natural language, requirements may need to become more explicit rather than less explicit. The apparent convenience of expressing requirements conversationally could otherwise create a false sense that ambiguity has been eliminated. In practice, the easier it becomes to generate an implementation from an informal statement, the easier it may become to conceal unresolved ambiguity behind apparently functional software.

AI therefore has considerable potential to augment requirements engineering, but its most valuable contribution may be not the automatic production of requirements artefacts, but the systematic identification, clarification and formalisation of uncertainty.

2.2 Software Design and Architecture

The influence of AI can then extend from defining what software should do to determining how it should be structured. LLMs can generate candidate designs, suggest design patterns, propose interface definitions, compare implementation approaches, produce data models and explain architectural alternatives. Hou et al. (2024) place software design among the six major areas of LLM application identified in their systematic review.

This capability potentially changes the economics of design exploration. Architectural decisions have traditionally required substantial engineering effort because each alternative must be conceptualised, documented and evaluated. An LLM can generate multiple candidate solutions rapidly, enabling developers to explore a wider design space before committing to a particular implementation. AI can consequently function as a design exploration and reasoning assistant, particularly during the early stages of system development.

The potential benefit is not necessarily that AI will produce the optimal architecture automatically. Rather, it can lower the cost of considering alternatives. A developer might ask an LLM to propose several approaches to data storage, service decomposition, API design or integration and then use those alternatives as inputs to a structured architectural evaluation. The value lies partly in expanding the range of options available for human consideration.

However, architecture differs fundamentally from generating an isolated function or code fragment. Architectural decisions involve interconnected technical, organisational and economic trade-offs. They must account for performance, scalability, reliability, security, maintainability, cost, team capabilities, existing infrastructure, regulatory obligations and long-term operational requirements.

An architecture can therefore be technically valid yet organisationally inappropriate. For example, an LLM might recommend a sophisticated microservices architecture for an enterprise application because it satisfies conventional scalability and modularity principles. Yet if the organisation lacks the operational maturity, observability infrastructure or engineering capacity required to manage distributed services, the recommendation may increase rather than reduce system risk. Similarly, an AI-generated architecture might fail to account adequately for data-residency requirements, auditability, identity management or the constraints imposed by an organisation's existing technology estate.

The limitation is therefore fundamentally one of context and accountability. LLMs can generate plausible architectural alternatives from the information supplied to them, but the information available to a model may not represent the complete organisational context in which an architectural decision must operate. Institutional knowledge, strategic priorities, risk appetite and operational experience are often only partially documented.

This creates an important distinction between architectural generation and architectural judgement. AI can increasingly assist with the former; the latter remains a responsibility requiring human expertise, contextual knowledge and accountability.

The role of the software architect may therefore evolve rather than disappear. Instead of manually producing every design artefact, architects may increasingly define constraints, interrogate AI-generated alternatives, evaluate trade-offs and establish architectural principles against which machine-generated proposals are assessed. AI can widen the design space, but humans remain responsible for deciding which part of that space is acceptable.

This suggests a useful model of AI-assisted architecture:

AI generates alternatives → human defines constraints → alternatives are evaluated → architectural decision is justified → implementation is governed against the decision

Such a model is preferable to treating the LLM as an architectural authority. It also creates a direct connection between architecture and governance: if AI-generated designs become increasingly common, organisations will need explicit mechanisms for ensuring that generated proposals remain consistent with security, regulatory, operational and strategic requirements.

The architectural implications are particularly important because errors at this stage can propagate widely. A poorly chosen data model or service boundary can affect thousands of subsequent implementation decisions. Consequently, AI assistance at the design stage arguably requires more, rather than less, human scrutiny than assistance with low-level coding.

2.3 Code Generation and Completion

Code generation and completion remain the most visible applications of generative AI in software engineering. Modern LLM-based development assistants can generate functions, classes, API calls, database queries, configuration files and tests from natural-language instructions, while code-completion systems can predict and generate subsequent sections of source code within an integrated development environment.

The attraction is straightforward. Software development contains substantial amounts of implementation work that follows recognisable patterns. If AI can automate part of the translation from an established intention into implementation details, developers can potentially spend less time on repetitive coding and more time on activities requiring contextual judgement.

Empirical research provides evidence that these benefits can be substantial under appropriate conditions. Peng et al. (2023), in a controlled experiment involving GitHub Copilot, asked software developers to implement an HTTP server in JavaScript as quickly as possible. Participants with access to the AI pair programmer completed the task 55.8% faster than those in the control group.

This is important evidence that AI assistance can reduce the time required to complete specific programming tasks. However, the result should not be interpreted as evidence that AI makes software developers universally 55.8% more productive. The experiment concerned a defined programming task performed under controlled conditions. The productivity effect in production software development is likely to vary with task complexity, developer experience, familiarity with the repository, requirements clarity, model capability and the effort required to validate generated code.

The distinction between coding speed and engineering productivity is therefore critical. If an AI assistant enables a developer to generate an implementation in half the time but the developer subsequently spends substantial additional effort reviewing, debugging, testing or restructuring the output, the net productivity gain may be much smaller. Conversely, where generated code is reliable and readily verifiable, faster generation may translate into genuine delivery improvements.

This distinction becomes more important as the scope of generated code increases. A developer using AI to complete a short, well-understood function is operating in a relatively constrained problem space. Generating an entire application component, however, introduces interactions among requirements, dependencies, architectural conventions, security controls and existing code. The larger the generated artefact, the greater the potential for errors that are locally plausible but globally inconsistent.

Hou et al. (2024) provide useful context for interpreting the emphasis on code generation. Their review found that software development accounted for approximately 56.65% of the 395 primary studies analysed, substantially exceeding the other lifecycle categories. This concentration demonstrates both the maturity of interest in AI-assisted programming and the relative ease with which code-generation outcomes can be benchmarked. Code can be measured through compilation, test execution and benchmark accuracy, whereas effects on architecture, maintainability, organisational performance and long-term software quality are substantially more difficult to quantify.

This creates a potential research and industry bias: what is easiest to measure may receive disproportionate attention. If AI research focuses heavily on whether models can generate code that passes predefined tests, organisations may overestimate the significance of coding performance relative to less measurable engineering activities. Software systems must remain understandable, secure, adaptable and operationally viable long after the initial generation event.

Consequently, code generation should be viewed as one component of a broader engineering process:

Requirement → design → implementation → verification → integration → maintenance

AI can accelerate implementation within this chain, but it cannot remove the dependencies between stages. A faster implementation of an incorrectly specified requirement remains incorrect software.

2.4 AI for Code Understanding and Maintenance

The lifecycle impact of AI also extends beyond creating new code. One of the most important but less visible applications is helping developers understand existing software. This is particularly significant because substantial software engineering effort is devoted to maintaining and modifying systems that developers did not originally create.

Nam et al. (2024) investigated an LLM-based conversational interface embedded directly within an IDE to support code understanding. Their tool provided explanations of selected code, information about API calls, explanations of domain-specific terminology and usage examples, while also allowing open-ended questions contextualised by the code being edited. In a study involving 32 participants, the authors found that the LLM-supported interface could aid task completion more than conventional web search, although the ways in which participants used the system and perceived its usefulness varied between students and professionals.

This finding broadens the conception of AI-assisted development. The value of an LLM is not limited to producing new artefacts; it can also reduce the knowledge cost of navigating existing ones. This may be particularly valuable in large or poorly documented repositories, legacy systems and unfamiliar technical domains.

However, code understanding introduces a distinctive risk. A developer may accept an incorrect AI explanation not because the code itself is wrong, but because the explanation appears coherent and authoritative. In maintenance environments, this can be particularly consequential: an inaccurate explanation of a legacy component may inform subsequent modifications and gradually introduce defects into a system whose original design is already poorly understood.

AI-supported comprehension should therefore be treated as assistance with interpretation rather than an authoritative source of truth. Developers should be able to validate explanations against source code, tests, documentation and runtime behaviour. The same principle that applies to generated code applies to generated understanding: plausibility is not equivalent to correctness.

This capability nevertheless has considerable strategic potential. If AI can reduce the time required to understand unfamiliar code, organisations may be able to reduce some of the costs associated with onboarding, knowledge transfer and legacy-system maintenance. The long-term significance of AI in software engineering may therefore extend beyond making developers faster at writing code to making them faster at understanding and changing systems they did not create.

2.5 From Lifecycle Automation to AI-Augmented Engineering

Taken together, the evidence suggests that AI is beginning to influence software development at multiple levels:

• Requirements engineering: AI can help interpret, structure, analyse and formalise human intent.

• Software design: AI can generate and compare candidate architectures and design alternatives.

• Implementation: AI can generate and complete source code and other technical artefacts.

• Code understanding and maintenance: AI can explain unfamiliar implementations, APIs and domain concepts.

These capabilities suggest a progression:

Human intent → AI-assisted requirements → AI-assisted design → AI-generated implementation → automated verification → human validation

The important feature of this model is not automation at any individual stage, but the increasing interconnection between stages. AI can potentially carry information from requirements into design and from design into implementation, while tests and other verification mechanisms provide feedback into the process.

This creates both an opportunity and a risk.

The opportunity is a more integrated development process in which AI can help translate intent into increasingly concrete artefacts while rapidly generating alternatives and verification material. The risk is that errors can propagate across the same chain. An incorrect interpretation of a requirement can produce an inappropriate architectural recommendation, which can generate incorrect implementation code and ultimately a system that appears functional while failing to meet its intended purpose.

The appropriate response is therefore not to prevent AI from participating across the lifecycle, but to establish continuous validation between lifecycle stages. Requirements should be explicit and testable; architectural proposals should be evaluated against defined constraints; generated implementations should undergo automated testing and security analysis; and consequential decisions should receive appropriate human review.

This suggests a transition from AI-assisted coding to AI-augmented engineering. In the former model, AI is primarily a tool used by programmers to generate code. In the latter, AI becomes an integrated participant in a broader engineering system, contributing to analysis, design, implementation, understanding and verification while humans retain responsibility for intent, contextual judgement and accountability.

The distinction is consequential because the more stages of the lifecycle in which AI participates, the more important the interfaces between those stages become. Software engineering may therefore increasingly depend on mechanisms that preserve traceability from requirements to design, from design to implementation and from implementation to verification. AI should not merely generate artefacts; it should operate within a process in which each artefact can be challenged against explicit evidence.

The emerging model can consequently be represented as:

Intent → Specification → Architecture → Implementation → Verification → Human acceptance → Operation and maintenance

with AI potentially supporting each stage, but with human judgement and organisational controls remaining embedded throughout the process.

This reframes the central question from “Can AI perform this software engineering task?” to a more consequential question:

“How should AI perform this task within a system of requirements, constraints, verification and human accountability?”

That shift is essential. The evidence demonstrates that AI can accelerate implementation and support developers across multiple lifecycle activities, but it does not establish that autonomous generation produces trustworthy software. Instead, the emerging literature points towards a model in which AI increases the speed and breadth of engineering activity while simultaneously increasing the importance of validation, integration and governance.

The software development lifecycle is therefore not being replaced by AI; it is being reconfigured around human-AI collaboration. AI can increasingly perform or support activities that are repetitive, pattern-based and information-intensive, while human engineers retain responsibility for defining objectives, resolving ambiguity, evaluating trade-offs, validating outcomes and accepting consequential decisions. This provides the foundation for the next chapter, which examines whether the productivity gains associated with AI-assisted development translate into measurable improvements in software engineering performance—and whether those gains persist once quality, verification and organisational costs are taken into account.

3. The Productivity Question

Productivity is arguably the most immediate and measurable justification for the adoption of artificial intelligence (AI) in software development. Whereas claims about AI-driven organisational transformation can be difficult to operationalise, productivity can be examined through observable measures including task completion time, developer effort, cycle time, defect rates, rework and software delivery throughput. The rapid adoption of AI coding assistants reflects an expectation that these systems can reduce the effort required for routine development activities while allowing software engineers to devote more time to complex problem-solving, architectural reasoning and decision-making.

Large language models (LLMs) can support a broad range of development activities, including code generation and completion, boilerplate implementation, API and library integration, documentation, test generation, debugging, code comprehension, routine refactoring and learning unfamiliar technologies. Hou et al. (2024) demonstrate the breadth of this application landscape, identifying software development as the largest category in their systematic review of LLMs in software engineering, accounting for approximately 56.65% of the 395 primary studies examined. This concentration reflects both the maturity of AI-assisted programming and the relative ease with which coding outcomes can be measured.

However, the productivity proposition requires careful qualification. AI should not simply be conceptualised as a mechanism for replacing developer labour. A more useful interpretation is that AI changes the relative cost of different software engineering activities. Tasks that previously required a developer to search documentation, formulate an implementation, write repetitive code and troubleshoot an initial solution can potentially be compressed into a shorter interaction involving AI generation followed by human evaluation.

This distinction matters because software development productivity cannot be equated with the amount of code produced. Additional lines of code do not necessarily create additional value, and faster implementation can be counterproductive if it produces defects, security vulnerabilities, architectural inconsistency or technical debt. The appropriate productivity question is therefore not how much code AI can generate, but how much effort AI can remove from the complete process of producing trustworthy, maintainable and valuable software.

The productivity debate consequently needs to move from a narrow focus on generation speed towards a broader consideration of total engineering effort. This includes the effort required to understand requirements, generate and integrate solutions, verify behaviour, conduct security analysis, review code, correct defects and maintain the resulting system.

3.1 Evidence of Productivity Gains

Empirical evidence demonstrates that AI coding assistants can produce substantial productivity improvements for particular development tasks. Peng et al. (2023), in a controlled experiment involving GitHub Copilot, found that developers using the AI assistant completed an HTTP-server programming task 55.8% faster than participants in the control group. The result provides important experimental evidence that generative AI can materially reduce the time required for certain programming activities.

The finding is significant because it moves the productivity argument beyond speculation. It demonstrates that, under controlled conditions, AI assistance can reduce the time developers require to transform a programming requirement into a working implementation. The result is particularly relevant to tasks in which the problem is sufficiently well specified and the implementation follows established programming patterns.

Nevertheless, the finding should not be interpreted as evidence that AI universally increases software developer productivity by 55.8%. The experiment involved a particular task, technology and development context. Production software engineering introduces considerably greater complexity, including large and unfamiliar repositories, legacy systems, evolving requirements, architectural constraints, organisational coding standards, security considerations and collaboration across teams.

The magnitude of productivity gains is therefore likely to vary according to several interacting factors:

• task complexity, novelty and ambiguity;

• developer experience and programming expertise;

• familiarity with the relevant language, framework and codebase;

• capability and reliability of the underlying AI model;

• quality of the prompt, specification and contextual information;

• integration of the AI assistant into the developer's existing tools;

• availability of tests and automated verification; and

• the amount of review, debugging and rework required after generation.

The productivity question should therefore be reformulated from:

Can AI write code faster?

to:

Under what technical, organisational and workflow conditions does AI reduce the total effort required to produce high-quality software?

This distinction is central to assessing the economic value of AI in software engineering. A tool that reduces coding time but increases verification or maintenance costs may deliver limited net productivity improvement. Conversely, an AI system that reduces both implementation and information-search costs while operating within an effective testing and review process may generate substantially greater value.

3.2 From Code Generation to Developer Productivity

The productivity contribution of AI extends considerably beyond source-code generation. Developers routinely spend significant amounts of time understanding unfamiliar systems, searching documentation, determining how APIs should be used, diagnosing errors, constructing tests and identifying dependencies between components. AI can potentially reduce these cognitive and information-search costs.

Nam et al. (2024), for example, investigated an LLM-based conversational interface integrated into an IDE to assist developers with code understanding. The system could explain selected code, answer questions about APIs and domain-specific terminology, and provide usage examples. In their study of 32 participants, the authors found that the AI-supported interface could help participants complete code-understanding tasks more effectively than conventional web search in the studied context. The finding illustrates an important dimension of AI productivity: the reduction of the time required to acquire and apply technical knowledge.

This is particularly relevant to software maintenance. Developers frequently modify systems they did not originally design and may need to reconstruct the rationale and behaviour of unfamiliar components before making even relatively small changes. If an AI assistant can rapidly explain code, identify relevant dependencies or summarise system behaviour, it may reduce the time required to establish the context necessary for a safe modification.

The same principle applies to technical learning. Developers working with unfamiliar frameworks, APIs or programming languages can use AI to obtain explanations and examples without repeatedly searching across documentation, forums and other sources. This can reduce the friction associated with acquiring technical knowledge and potentially shorten the time required to become productive in an unfamiliar environment.

AI can therefore be understood as a form of cognitive augmentation. Its productivity value lies not only in producing software artefacts but also in reducing the time required to understand, search, reason about and manipulate those artefacts.

This distinction is important because some of the most consequential productivity gains may be difficult to capture through conventional coding metrics. A developer who spends ten minutes asking an AI system to explain a complex legacy component rather than an hour reconstructing its behaviour manually has achieved a productivity improvement even though the amount of newly generated code may be negligible.

3.3 Productivity Depends on Workflow Integration

AI capability alone does not determine organisational productivity. A highly capable model may provide limited value if it is poorly integrated into the developer's working environment. Conversely, a somewhat less capable model may produce substantial value when it has appropriate access to relevant context, development tools and verification mechanisms.

A useful conceptual model is:

AI productivity = generation capability + developer judgement + workflow integration

This is not a mathematical productivity equation. Rather, it emphasises three interdependent conditions through which model capability becomes practical engineering value.

Generation capability determines what an AI system can produce, analyse or explain. More capable models may be able to handle more complex tasks, larger contexts and a broader range of programming activities.

Developer judgement determines whether the resulting output is appropriate. Developers must assess whether generated code satisfies requirements, conforms to architectural principles, introduces security weaknesses and remains understandable and maintainable.

Workflow integration determines how easily AI capabilities can be incorporated into the actual software engineering process. An assistant embedded within an IDE, repository, testing environment and continuous integration pipeline can potentially provide considerably more value than an isolated chatbot that requires developers to transfer requirements, context and generated code manually between systems.

This suggests that organisations should not evaluate AI coding tools solely by asking which model generates the strongest code. They should also ask how the capability fits into the organisation's software engineering operating model.

An integrated AI environment could potentially provide access to relevant repositories, coding standards, architectural documentation, dependency information, tests, issue-tracking data and security policies. Such contextual integration reduces the distance between an AI-generated suggestion and the actual environment in which that suggestion must operate.

The implication is significant: productivity gains depend partly on context quality. A model generating code without knowledge of the repository's conventions or architectural constraints may require extensive correction. The same model operating with relevant contextual information may produce outputs that are more readily integrated and evaluated.

AI adoption should therefore be considered partly as a workflow-design problem. Organisations that simply add a chatbot to an existing process may obtain relatively modest benefits, whereas organisations that redesign the interaction between AI, developers, repositories, testing and deployment may capture considerably greater value.

3.4 The Role of Testing in AI Productivity

A central constraint on AI productivity is that generated code still requires verification. If developers spend substantial amounts of time inspecting, correcting and testing AI-generated output, some of the initial time savings can be offset.

Fakhoury et al. (2024) provide evidence that integrating testing and user feedback directly into code generation can mitigate this problem. Their test-driven interactive approach combines LLM generation with explicit tests and iterative developer interaction. The study found improvements not only in code-generation performance but also in participants' ability to evaluate generated code and in task-induced cognitive load.

The importance of this finding extends beyond test-driven generation. It suggests that verification should not be conceptualised as a separate stage that begins after AI has finished producing software. Instead, verification can become part of the interaction through which the AI and developer jointly construct the solution.

The conventional model might be represented as:

AI generates code → developer inspects code

A more effective AI-assisted model is:

Developer specifies intent → AI generates candidate solution → tests evaluate behaviour → developer reviews evidence → AI and developer iterate → validated software is produced

The second model changes the role of testing from a downstream quality-control activity into an active mechanism for guiding generation.

This has important implications for organisational investment. If AI increases the speed of software generation, organisations may need to increase their investment in automated testing, static analysis, continuous integration, security analysis and other verification capabilities. These systems enable developers to determine more rapidly whether generated output is fit for purpose.

The productivity of AI-assisted development is therefore partly dependent on the speed and reliability of verification. If generation becomes dramatically faster while verification remains largely manual, the bottleneck simply moves from writing code to checking it.

3.5 The Productivity Paradox

The increasing speed of code generation creates a potential productivity paradox: AI may reduce the cost of producing software without proportionally increasing the amount of valuable software delivered.

Consider a developer who can implement a feature twice as quickly using an AI assistant. If the feature is correct, secure and maintainable, the time saving represents genuine productivity. If the generated implementation subsequently requires extensive debugging, security remediation or architectural restructuring, however, the initial gain may be substantially reduced.

The distinction can be represented as:

Gross productivity gain = reduced generation effort

but:

Net productivity gain = reduced generation effort − verification − rework − integration − maintenance costs

The second formulation better reflects the economic reality of software engineering. Although these terms should not be interpreted as a literal accounting equation applicable to every organisation, they illustrate why generation speed alone is an incomplete productivity measure.

This creates the possibility of a verification bottleneck. Historically, a significant constraint in software development was the production of implementation code. As AI lowers the cost of generating that code, constraints may increasingly emerge elsewhere:

• requirements clarification;

• architectural evaluation;

• code review;

• testing and verification;

• security analysis;

• integration;

• deployment; and

• long-term maintenance.

The resulting dynamic can be expressed as:

Lower generation cost → greater generation capacity → greater verification demand → increased importance of governance

This does not mean that AI productivity gains are illusory. Rather, it means that their magnitude depends on whether the rest of the software engineering system can absorb the additional volume and complexity of generated output.

Hou et al. (2024) provide an important contextual warning here. The concentration of research on software development reflects the relative ease with which code generation can be evaluated, whereas the longer-term effects on maintainability, organisational performance and software quality are more difficult to measure. This creates a risk that both academic research and organisational adoption may overemphasise easily observable improvements in coding speed.

More meaningful organisational measures should therefore include:

• end-to-end delivery cycle time;

• defect and escaped-defect rates;

• security vulnerabilities;

• rework and rollback rates;

• code-review effort;

• deployment frequency and reliability;

• maintainability;

• developer cognitive load;

• incident rates; and

• business value delivered.

The appropriate objective is consequently not more code per developer, but more valuable software per unit of total engineering effort.

3.6 From Individual Productivity to Workflow Transformation

The strongest productivity implications may ultimately occur at the workflow level rather than at the level of individual programming tasks.

If AI is introduced merely as a chatbot alongside an otherwise unchanged development process, productivity gains may remain constrained. Developers may still need to copy requirements into the AI tool, transfer generated code into the IDE, execute tests manually and document resulting changes.

A more significant transformation occurs when AI capabilities are integrated across the development lifecycle. Requirements can be analysed and clarified; specifications can inform implementation; AI can generate code and tests; automated systems can evaluate resulting artefacts; and validated changes can progress through existing continuous integration and deployment processes.

This creates a potential AI-augmented software engineering workflow:

Requirements → AI-assisted analysis → Design → AI-assisted implementation → Automated verification → Human review → Deployment → Monitoring → Feedback

Within such a workflow, AI is no longer merely a tool used by individual developers. It becomes part of the software production system.

This distinction is strategically important. Organisations that focus primarily on acquiring increasingly powerful models may achieve incremental task-level improvements. Organisations that redesign their development processes around AI, automation, contextual integration and continuous verification may capture substantially greater value.

The difference can be understood as the distinction between tool adoption and workflow transformation.

Tool adoption asks:

How can developers use AI to perform their existing tasks faster?

Workflow transformation asks:

How should software development itself be redesigned when AI can participate in multiple stages of the lifecycle?

The second question has considerably broader organisational implications because it affects team structures, process design, quality assurance, technical governance and the distribution of engineering responsibility.

3.7 The Changing Role of the Developer

The productivity question ultimately leads to a broader question about the future role of software developers.

If AI increasingly performs routine implementation, developers may spend proportionally less time writing individual lines of code and more time defining requirements, evaluating AI-generated solutions, designing architectures, resolving ambiguity, reviewing security and making system-level decisions.

This does not diminish the importance of software engineering expertise. It may instead increase the value of expertise in evaluating and constraining machine-generated software. As the volume of AI-generated code increases, the ability to distinguish an apparently plausible solution from a genuinely appropriate one becomes increasingly important.

The developer is consequently evolving from a role centred primarily on implementation towards one that combines programming, design, evaluation, orchestration and decision-making.

The distribution of software engineering work is consequently beginning to shift. Traditionally, developers have devoted substantial effort to manually writing implementation code, searching documentation, producing tests, debugging individual problems and implementing software components. With AI-assisted development, these activities are increasingly being supplemented or partially replaced by specifying intent and generating candidate implementations, retrieving and synthesising technical knowledge through conversational interfaces, generating and refining tests with AI assistance, using AI to diagnose problems and suggest corrections, and evaluating and integrating machine-generated components. The emphasis is also shifting from local optimisation of individual pieces of code towards system-level orchestration, in which developers coordinate AI systems, development tools, data, software components and engineering workflows. In this emerging model, the developer's contribution is less centred on producing every individual software artefact and more focused on directing, evaluating and integrating the outputs of an increasingly AI-enabled development environment.

This should not be interpreted as evidence that traditional programming skills are becoming obsolete. Programming knowledge remains essential precisely because developers need sufficient technical understanding to evaluate AI output, recognise incorrect assumptions, identify security weaknesses and constrain the system effectively.

Indeed, AI may increase the value of deep software engineering knowledge. A developer who understands architecture, algorithms, security, testing and system behaviour is better positioned to distinguish a useful AI suggestion from one that merely appears convincing.

The emerging developer is therefore not simply an AI operator. The role is closer to that of an engineering orchestrator and validator: someone who defines objectives, provides context, evaluates alternatives, coordinates tools and accepts responsibility for the resulting system.

3.8 From Productivity Tool to Development-System Capability

The evidence suggests that AI productivity should be understood at several interconnected levels.

At the task level, AI can accelerate specific programming activities, as demonstrated by Peng et al. (2023).

At the developer level, AI can reduce information-search and cognitive costs by supporting activities such as code understanding, as demonstrated by Nam et al. (2024).

At the workflow level, AI can connect requirements, implementation, testing and documentation more efficiently, particularly when supported by interactive approaches such as that studied by Fakhoury et al. (2024).

At the organisational level, AI can potentially change how software teams allocate effort, structure processes and govern software production.

The greatest productivity gains may therefore not arise from simply asking AI to generate more code. They may emerge from redesigning the development system around continuous human-AI collaboration, contextual integration and automated verification.

This leads to a central proposition:

AI productivity in software development is fundamentally a systems problem rather than merely a model-capability problem.

A highly capable model can accelerate individual tasks, but sustainable productivity requires the surrounding organisation to provide appropriate context, developer expertise, workflow integration, testing, security and governance. The model is one component of a larger socio-technical system.

This perspective also resolves an apparent tension in the literature. Peng et al. (2023) provide compelling evidence that AI can accelerate specific coding tasks, while Fakhoury et al. (2024) demonstrate that structured interaction and testing can further improve the reliability of generated solutions. Nam et al. (2024) show that productivity extends beyond generation into code understanding, while Hou et al. (2024) demonstrate that LLM applications span the wider software engineering lifecycle. Taken together, these findings suggest that AI's productivity contribution is best understood as the reduction and redistribution of engineering effort across a network of interdependent activities, rather than simply the acceleration of programming.

The critical counterweight, however, is that increased speed and reduced effort do not automatically produce better software. AI-generated outputs can contain functional defects, security vulnerabilities and inappropriate assumptions, while the probabilistic nature of LLMs can introduce variability between outputs. Consequently, the productivity benefits identified in controlled settings must be evaluated alongside the costs of verification, security review, rework and governance.

The central challenge is therefore not whether AI can make individual developers faster. The evidence increasingly establishes that it can, under appropriate conditions. The more consequential question is whether organisations can convert task-level acceleration into sustainable improvements in end-to-end software delivery without proportionally increasing technical and organisational risk.

This provides the transition to the next chapter. If AI lowers the marginal cost of producing software, the critical issue becomes whether the resulting increase in software production can be achieved while maintaining correctness, security, maintainability and accountability. The productivity promise of AI is therefore inseparable from its quality and risk profile. The faster software can be generated, the more important it becomes to establish whether that software can be trusted.

4. Software Quality and Reliability

aProductivity gains provide an important rationale for adopting artificial intelligence (AI) in software development, but speed alone cannot establish the value of AI-assisted engineering. The objective of software engineering is not simply to produce software rapidly; it is to produce software that is correct, reliable, secure, maintainable and fit for its intended purpose. Any meaningful assessment of AI-assisted development must therefore consider both the efficiency with which software is generated and the quality of the resulting artefacts, including the effort required to test, review, integrate and maintain them.

This distinction is particularly important for large language models (LLMs). Generated code can be syntactically valid, stylistically polished and highly plausible while nevertheless implementing the wrong behaviour. The fluency of an LLM's output can therefore create an illusion of correctness that is not necessarily supported by evidence. This produces a distinctive assurance challenge: the more convincing AI-generated software appears, the more important independent verification becomes.

The literature consequently points towards a fundamental principle of AI-assisted development: AI should be treated as a mechanism for generating and analysing candidate solutions, rather than as an authoritative source of correct implementations. The quality of those solutions must be established through testing, static analysis, security analysis, integration, human review and operational evidence.

4.1 The Plausibility Problem

Traditional software defects arise from many sources, including incorrect requirements, design errors, programming mistakes, inadequate testing and failures to account for exceptional conditions. AI-generated software introduces these same categories of failure but adds an important characteristic: the system can produce an implementation that appears carefully reasoned even when the underlying interpretation is incorrect.

This can be described as the plausibility problem. LLMs generate outputs by modelling patterns in language and code; they do not provide an intrinsic guarantee that the resulting implementation satisfies the intended requirements. Consequently, generated code may be coherent and professionally structured while containing fundamental semantic or functional errors.

Empirical research demonstrates the breadth of this problem. Tambon et al. (2024), in an analysis of 333 bugs in LLM-generated code, identified recurring defects including requirement misinterpretation, syntax errors, missing corner cases, hallucinated objects, incorrect attributes and incomplete implementations. The significance of these findings is that AI-generated defects cannot be reduced to obvious programming mistakes. Some originate from a failure to understand the semantics, assumptions or contextual constraints of the task.

This distinction becomes particularly important in production environments. A generated function may compile successfully and pass a small set of ordinary test cases while failing under boundary conditions, unusual inputs or interactions with other components.

The central problem can therefore be expressed succinctly:

AI-generated code can be syntactically correct, semantically plausible and functionally wrong.

Software quality cannot consequently be inferred from the apparent sophistication of generated code. It must be demonstrated through evidence.

4.2 Why Visual Inspection Is Insufficient

Human code review remains an important component of software quality assurance. Experienced developers can identify suspicious patterns, inconsistent logic, inappropriate dependencies and violations of coding standards through inspection. However, AI-generated code presents a particular challenge because it often conforms closely to established programming conventions.

An LLM can produce descriptive variable names, appropriate abstractions, comments and familiar structural patterns. These characteristics can make generated code appear trustworthy even when its behaviour is incorrect. A superficial review may therefore assess the appearance of quality rather than quality itself.

Effective review of AI-generated code requires developers to determine whether the implementation:

• satisfies the original requirements;

• handles boundary and exceptional conditions;

• interacts correctly with surrounding components;

• uses libraries and dependencies appropriately;

• satisfies security requirements;

• conforms to architectural constraints; and

• remains understandable and maintainable over time.

This creates a potential verification burden. If AI substantially increases the volume of software that developers can generate, it may simultaneously increase the amount of software that requires validation.

The productivity implications are therefore closely connected to quality assurance:

If AI accelerates software generation faster than an organisation can validate the resulting software, apparent productivity gains may be accompanied by increased quality risk.

The relevant objective is consequently not maximum AI-generated output, but maximum validated and useful software.

4.3 Testing as a Core AI-Assurance Mechanism

Automated testing becomes particularly important when software is increasingly produced with AI assistance. Tests provide an external mechanism for evaluating whether generated code exhibits the behaviour specified by the requirements.

This is especially important because the AI system that generates an implementation cannot automatically be treated as an independent validator of that implementation. Asking an LLM to generate code and then explain why its code is correct does not constitute independent assurance. The same assumptions, contextual gaps or errors that influenced the original generation may also influence the subsequent explanation.

A stronger approach is therefore to establish a separation between generation and verification.

An AI-assisted quality pipeline can be represented as:

Requirement → AI-generated implementation → automated testing → static analysis → security analysis → human review → integration testing → deployment

Each mechanism addresses different classes of failure. Unit tests evaluate individual functions or components; integration tests assess interactions between components; static analysis identifies structural and coding problems without executing the software; security analysis identifies potentially vulnerable patterns; and human review addresses requirements, architecture and contextual issues that automated tools may not capture.

Fakhoury et al. (2024) provide empirical support for integrating testing directly into the generation process. Their test-driven interactive approach combines LLM-generated code with explicit tests and iterative user interaction. Rather than treating testing as an activity performed after generation has concluded, the approach uses tests to clarify intent and constrain the generation process. The resulting workflow improved both code-generation performance and participants' ability to evaluate generated code.

The implication is significant: testing can function not merely as a quality gate, but as a mechanism through which human intent is progressively communicated to and evaluated against AI-generated implementations.

The stronger development model is therefore:

Intent → candidate implementation → behavioural evidence → refinement → validated implementation

rather than:

Prompt → code → acceptance

4.4 Static Analysis and Type Systems

Static analysis, compilation and type checking provide additional layers of assurance around AI-generated software. These techniques can identify classes of problems before software is executed, including unreachable code, inappropriate variable use, type errors, potentially unsafe operations, code-quality violations and certain insecure programming patterns.

Their importance arises partly from the different characteristics of deterministic software-analysis tools and probabilistic generative models. An LLM may produce a candidate implementation probabilistically, whereas a compiler, type checker or static analyser can apply defined rules consistently to that implementation.

This creates a useful principle for AI-assisted engineering:

Probabilistic generation should be surrounded by deterministic verification wherever practical.

The principle does not imply that static analysis can prove software correct. Static tools have their own limitations and cannot capture every functional, architectural or contextual requirement. They can, however, provide an independent source of evidence that complements AI generation.

The combination is therefore more robust than relying on either technology independently: AI expands the speed and breadth of software production, while deterministic analysis provides repeatable checks around defined properties of the generated artefact.

4.5 Integration and System-Level Reliability

Correctness at the component level does not establish correctness at the system level. AI-generated components must operate appropriately within the architecture, data model, security mechanisms and dependencies of the wider application.

A generated function may behave correctly in isolation but fail when interacting with an existing database schema, external API, authentication mechanism, message queue, legacy component or another service. These failures arise because software behaviour is often dependent on context rather than solely on the internal logic of an individual function.

Integration testing is consequently essential in AI-assisted development. It provides evidence that individually plausible components work together as intended.

Software quality should therefore be assessed across multiple levels:

Code → Component → Service → Application → Enterprise system

This hierarchy reinforces a central limitation of AI-generated code: local correctness does not imply global correctness. An implementation can satisfy its immediate unit tests while violating an architectural assumption elsewhere in the system.

This issue is particularly significant in enterprise environments, where software often incorporates legacy systems, third-party services, complex data dependencies and organisational standards that may not be fully represented in the context available to an LLM.

Consequently, AI-generated artefacts should be evaluated not only according to whether they work independently, but also according to whether they remain consistent with the system into which they are introduced.

4.6 Reliability and Edge Cases

One recurring challenge in generated software concerns boundary conditions and uncommon system states. LLMs are highly effective at reproducing common programming patterns, but unusual combinations of conditions may be more difficult to capture reliably.

A generated implementation may perform correctly during normal operation while failing when input is missing or malformed, values fall outside expected ranges, network connections fail, dependencies become unavailable, concurrent requests occur or users perform unexpected sequences of actions.

Such cases are not peripheral to software reliability. They are often precisely the conditions under which production systems fail.

AI-assisted development therefore requires deliberate attention to boundary, adversarial and failure-state testing. Developers should not evaluate generated implementations solely against the most obvious or representative scenarios. They must also consider how the system behaves when assumptions are violated.

AI can itself assist in generating test cases, but this introduces a further challenge: generated tests may reproduce the same assumptions embedded in the generated implementation. A test suite that merely confirms the model's interpretation of the requirement may provide a misleading impression of coverage.

Test quality must therefore be assessed independently of code-generation quality. Requirements, domain knowledge, mutation testing, property-based testing and human-designed edge cases can all help expose weaknesses that an AI-generated test suite might otherwise miss.

4.7 AI Can Strengthen Quality as Well as Create Risk

The relationship between AI and software quality is not exclusively negative. The same capabilities that enable LLMs to generate software can also support quality assurance activities.

AI systems can assist developers with defect identification, refactoring, unit-test generation, code explanation, duplicate-code detection, code-review support and the diagnosis of potential problems. Consequently, AI can contribute to improving software written by both humans and machines.

This creates the possibility of an AI-assisted quality feedback loop:

Generate → Test → Analyse → Identify defect → Correct → Retest

The effectiveness of this model depends on the independence and reliability of the verification mechanisms involved. Organisations should be cautious about allowing the same AI system to generate an implementation and then determine, without external evidence, that the implementation is correct.

A stronger model combines AI-based analysis with independent tests, static analysis, security tools and human judgement. AI can accelerate the search for potential problems, but consequential quality decisions should remain grounded in evidence that is not dependent solely on the model's own assessment.

This distinction is particularly important because AI can improve the efficiency of quality assurance without necessarily making the underlying verification problem disappear. If AI generates both the code and the tests, for example, the organisation may gain speed but not necessarily gain independent assurance.

4.8 Continuous Integration and Continuous Verification

The increased speed of AI-assisted development strengthens the case for continuous integration and automated verification. If developers can generate and modify software more rapidly, manual validation processes may increasingly become a bottleneck.

Continuous integration allows changes to be automatically compiled, tested and analysed as they enter a shared repository. A mature AI-assisted development pipeline could therefore incorporate:

1. AI-assisted code generation or modification;

2. compilation and type checking;

3. unit testing;

4. static analysis;

5. dependency and security scanning;

6. integration testing;

7. performance testing where appropriate;

8. human review; and

9. controlled deployment.

Such a pipeline effectively creates quality gates around AI-generated software.

The importance of these gates increases with the scale and speed of AI adoption. If generation becomes significantly faster while verification remains predominantly manual, organisations may either fail to realise the potential productivity benefits or increase the amount of unverified software entering production.

Continuous verification therefore becomes a strategic complement to AI generation. The objective is not simply to automate more of the development process, but to ensure that the speed of assurance increases sufficiently to keep pace with the speed of generation.

4.9 Runtime Monitoring and Operational Feedback

Software quality cannot be established entirely before deployment. Some defects and reliability problems only become visible under real-world operating conditions, where applications encounter data, workloads and user behaviour that cannot be completely reproduced during development.

Runtime monitoring can provide evidence concerning application errors, performance degradation, unusual behaviour, security events, resource consumption and unexpected interactions between components. This operational evidence is particularly valuable when software is being generated or modified at high frequency.

Quality assurance should therefore extend beyond deployment:

Generate → Test → Deploy → Observe → Learn → Improve

This represents a shift from quality assurance as a discrete stage towards continuous quality management across the software lifecycle.

Operational feedback can also create a feedback loop into future development. Production incidents can inform new tests; observed performance problems can inform architectural decisions; and recurring failures can reveal weaknesses in requirements or generated implementations. In this model, software quality becomes an evolving property supported by evidence gathered throughout the system's lifecycle rather than a one-time certification performed before release.

4.10 From Software Generation to Software Assurance

The evidence ultimately suggests that AI does not remove the need for conventional software engineering discipline. Instead, it may make that discipline more important.

AI can reduce the time and cost required to produce software, but it does not eliminate the requirements for correctness, testing, security, maintainability, reliability, architectural consistency or operational monitoring. Indeed, the ability to generate software rapidly may increase the importance of these activities because organisations can potentially produce executable artefacts at a rate that exceeds the capacity of conventional manual review.

The resulting relationship can be expressed as:

AI increases the speed of software generation; engineering assurance determines whether that speed creates value or risk.

This suggests that the future of AI-assisted software development should not be framed as a choice between AI automation and traditional engineering practice. The two are complementary. AI can provide rapid generation, explanation, analysis and assistance, while established engineering practices provide independent mechanisms for verification, control and accountability.

The central implication is therefore that AI-assisted software development requires an assurance architecture alongside its generation architecture. Testing, static analysis, security scanning, code review, continuous integration and runtime monitoring should not be regarded as secondary safeguards added after AI adoption. They are fundamental components of a development model in which AI increasingly participates in producing executable software.

The relationship can be conceptualised as two interconnected layers. The generation layer uses AI to transform requirements and developer intent into candidate designs, code and tests. The assurance layer independently evaluates those artefacts through deterministic analysis, automated testing, security controls, human review and operational evidence. The value of AI emerges when these layers operate together rather than when generation is allowed to bypass established engineering controls.

This perspective also changes how software quality should be measured. Traditional metrics such as defect density and test coverage remain relevant, but AI-assisted development creates a need to examine additional factors, including the proportion of generated code requiring rework, verification effort per change, escaped defects associated with AI-generated components, reproducibility of generated artefacts and the effectiveness of human review.

The central challenge is therefore shifting from how quickly software can be written towards how reliably software can be generated, verified, governed and maintained at scale. AI may dramatically lower the cost of producing implementation artefacts, but it does not lower the standard to which production software must be held.

Indeed, the more capable AI becomes at generating software, the less defensible it becomes to treat successful generation as evidence of successful engineering. A generated program that compiles is not necessarily correct; one that passes unit tests is not necessarily secure; one that satisfies its functional requirements is not necessarily maintainable; and one that performs correctly in testing is not necessarily reliable in production.

Software engineering assurance must therefore become an integral part of AI adoption rather than a compensating control added afterwards. The emerging development model is not:

AI generates software → traditional engineering checks it afterwards

but rather:

AI-assisted generation + continuous verification + human judgement + operational evidence → trustworthy software

This provides the foundation for the next chapter, which examines one of the most consequential dimensions of this assurance challenge: the security risks associated with AI-generated software and the extent to which increased development speed may introduce new vulnerabilities into the software supply chain.

5. Security Risks

sSecurity is one of the most consequential challenges associated with the increasing use of artificial intelligence (AI) in software development. The ability of large language models (LLMs) to generate functional code does not imply that the resulting software is secure. Security depends on properties that extend beyond functional correctness, including appropriate authentication and authorisation, secure input handling, protection of sensitive information, safe dependency management, correct cryptographic implementation and resistance to known attack techniques.

This distinction is fundamental because a security vulnerability can remain invisible while an application continues to perform its intended function. A program may correctly process a transaction while simultaneously exposing sensitive information, accepting malicious input or permitting unauthorised access. Functional correctness and security correctness are therefore distinct dimensions of software quality. Hou et al. (2024) identify security as an important limitation within the broader application of LLMs to software engineering, reinforcing the need to evaluate AI-generated software not only according to what it does, but also according to how safely it performs those functions.

Recent empirical research provides more direct evidence of this problem. Wang et al. (2024) introduced CodeSecEval, a benchmark containing 180 samples covering 44 vulnerability types, to evaluate LLMs on secure code generation and vulnerability repair. Their experiments found that contemporary models frequently overlooked security weaknesses in both tasks. This demonstrates that the problem extends beyond the generation of insecure code to the reliable identification and remediation of existing vulnerabilities.

The security implications of AI-assisted development are therefore broader than the question of whether an LLM occasionally produces vulnerable code. They concern the interaction between model capability, developer behaviour, software context, security expertise, development workflows and organisational governance. The central challenge is consequently not simply to determine whether AI can write secure code, but to establish how organisations can reliably integrate AI-generated software into secure engineering processes.

5.1 Vulnerable Generated Code

The most immediate security risk is that an AI coding assistant may generate code containing exploitable weaknesses. Potential problems include insecure authentication and authorisation, inadequate input validation, unsafe database queries, insecure file handling, weak session management, inappropriate cryptographic operations, incorrect access-control mechanisms, insecure API endpoints and improper handling of sensitive information.

For example, an AI-generated database query may successfully return the expected data during normal testing while remaining vulnerable to SQL injection. Similarly, generated authentication logic may correctly authenticate legitimate users while containing weaknesses that allow controls to be bypassed under particular conditions. Such vulnerabilities may remain invisible during ordinary functional testing.

The difficulty is compounded by the fact that security failures frequently depend on interactions between components and assumptions about how an application will be attacked. A generated function may therefore appear secure when considered independently while becoming vulnerable when integrated into a wider system.

Code that works is consequently not necessarily code that is secure.

Wang et al. (2024) provide particularly relevant evidence. Their CodeSecEval evaluation found that current LLMs frequently failed to account for security issues during both code generation and vulnerability repair, with some vulnerability categories proving especially challenging for models. This suggests that general coding capability cannot safely be treated as evidence of secure software generation.

The underlying problem is that LLMs are generally asked to satisfy a stated programming objective. Unless security requirements, constraints and threat assumptions are explicitly represented in the task and subsequently verified, there is no guarantee that the generated implementation will satisfy the security requirements of the wider system.

Security must therefore be incorporated into requirements and architecture, rather than appended only during code review.

5.2 Security Is a Context-Dependent Property

Security is particularly difficult for general-purpose LLMs because secure implementation depends heavily on context. A request such as “implement an authentication system” is radically underspecified from a security perspective. A robust implementation depends on the application's threat model, identity architecture, authentication protocol, session-management requirements, password policies, privileged-user model, regulatory obligations and security controls already implemented elsewhere.

An LLM may nevertheless produce a technically convincing implementation from such an incomplete request.

This creates a fundamental limitation:

The model can only reliably incorporate security constraints that are available within its effective context.

If critical assumptions remain implicit, the model may satisfy the visible functional requirement while violating requirements that exist at the system, organisational or regulatory level.

Security should therefore shape the context in which AI-generated implementations are produced and evaluated. Threat modelling, security requirements and architectural constraints should be represented before or alongside code generation rather than being treated exclusively as post-generation checks.

This is consistent with Klemmer et al. (2024), whose qualitative study of software professionals found that AI assistants are already being used for security-relevant activities such as code generation, threat modelling and vulnerability detection despite concerns about the reliability of AI-generated suggestions.

The implication is therefore not that AI should be excluded from security engineering, but that it should operate within a security process rather than substitute for one.

5.3 False Confidence and Trust Calibration

A further risk arises from how developers interpret AI-generated output. LLMs can produce source code that is polished, well structured and accompanied by convincing explanations. Meaningful variable names, comments, familiar programming patterns and apparently sophisticated abstractions can create an impression of competence and reliability.

This presentation quality can produce automation-induced confidence. Developers may place greater trust in an AI-generated implementation because it appears systematic and authoritative, even though the output has not necessarily undergone meaningful security validation.

The fluency of the output provides no guarantee of its security.

This issue is particularly important where developers have limited security expertise. An experienced security engineer may recognise an insecure authentication pattern or identify an unsafe assumption about input handling, whereas a less experienced developer may interpret the same implementation as evidence of a competent solution.

Klemmer et al. (2024) provide useful evidence concerning this human dimension. Their study involved 27 semi-structured interviews with software professionals and an analysis of 190 relevant Reddit discussions. Despite significant concerns about the security and quality of AI-generated output, participants continued to use AI assistants for security-critical activities. At the same time, their concerns led many participants to scrutinise AI suggestions rather than treating them as authoritative.

The resulting challenge is therefore one of trust calibration. Developers must understand when AI output is sufficiently reliable to inform a decision and when independent evidence or specialist expertise is required.

AI may consequently lower the barrier to producing sophisticated-looking software without lowering the level of expertise required to determine whether that software is safe.

5.4 Propagation of Insecure Patterns

Another concern relates to the software practices represented in the data from which LLMs learn. Models are trained using large and diverse collections of text and code, including publicly available software repositories and programming examples. These sources contain both secure and insecure implementations, including outdated practices and vulnerable patterns.

Wang et al. (2024) identify the use of potentially unsanitised open-source data as a concern because it creates the possibility that models may reproduce security weaknesses present in their training material. Their findings further demonstrate that contemporary models can overlook security weaknesses when generating or repairing code.

Potentially problematic patterns include insecure input handling, deprecated APIs, weak authentication mechanisms, unsafe dependency use and inappropriate cryptographic practices. This does not mean that LLMs simply reproduce insecure source code verbatim. Models can learn broader programming and security concepts and may produce improvements over individual examples. Nevertheless, the possibility of reproducing insecure patterns means that AI-generated code cannot be assumed to be secure merely because it was produced by a modern model.

The risk becomes more significant when AI-generated patterns are reused at scale. A flawed implementation that might previously have appeared in one codebase can potentially be reproduced across multiple projects, repositories or organisational templates.

This creates the possibility of security debt propagation. AI can reduce the marginal cost of software production and reuse, but that same efficiency can allow insecure practices to spread more rapidly if they are not detected.

The security problem is therefore partly a question of scale. AI does not necessarily create entirely new vulnerability classes; it may increase the speed and volume at which existing insecure practices are introduced, reproduced and distributed.

5.5 AI-Generated Code Compared with Human Alternatives

The evidence should nevertheless be interpreted carefully. It would be misleading to conclude that AI-generated code is inherently less secure than code produced by humans.

Hamer, d'Amorim and Williams (2024), for example, compared security vulnerabilities in ChatGPT-generated Java code with corresponding Stack Overflow answers. Their findings showed that the ChatGPT-generated snippets contained fewer vulnerabilities than the Stack Overflow examples examined.

This is an important finding because it challenges the simplistic assumption that AI assistance necessarily reduces software security.

The result should not, however, be interpreted as evidence that AI-generated code is inherently secure. Vulnerabilities were still present in the AI-generated material, and the comparison was conducted between particular sources under particular experimental conditions.

The appropriate conclusion is therefore comparative rather than absolute:

AI-generated code may be safer than some commonly used human-generated alternatives without being sufficiently secure for production use.

For organisations, the relevant benchmark should therefore not be whether an LLM performs better than an average online code example. The relevant question is whether the resulting implementation satisfies the organisation's security requirements, threat model, architectural constraints and regulatory obligations.

5.6 Code Generation and Vulnerability Repair

Security risks also arise when AI is used to repair existing vulnerabilities. Automated vulnerability remediation is attractive because developers can identify a weakness and ask an LLM to modify the relevant implementation, potentially reducing the time required to produce a candidate fix.

Security remediation, however, is not simply a matter of changing the line of code associated with a vulnerability. A valid remediation must address the underlying security weakness without introducing new vulnerabilities, breaking legitimate functionality, creating inconsistent behaviour elsewhere, bypassing another security control or leaving related instances of the same weakness unresolved.

This is why the CodeSecEval findings of Wang et al. (2024) are particularly important. Their evaluation considers both code generation and code repair, demonstrating that current models can struggle not only to avoid vulnerabilities when writing new code but also to remediate them reliably when explicitly asked to do so.

AI-assisted repair should therefore be understood as candidate remediation rather than autonomous remediation. A model can accelerate vulnerability analysis and suggest potential fixes, but those fixes should be independently tested and reviewed before being incorporated into production systems.

This also highlights an important distinction between syntactic remediation and security remediation. Removing a static-analysis warning does not necessarily demonstrate that the underlying threat has been eliminated. Effective remediation must be evaluated against the security property represented by the vulnerability.

5.7 Security Verification in AI-Assisted Development

The risks associated with AI-generated software reinforce the need to integrate security controls directly into the development lifecycle. Security should not be treated as a final inspection performed after AI-generated code has already been integrated.

A robust AI-assisted development pipeline can incorporate multiple complementary mechanisms, including:

• Static Application Security Testing (SAST) to identify potentially insecure coding patterns;

• Software Composition Analysis (SCA) to assess third-party dependencies;

• dependency vulnerability scanning to identify known weaknesses in libraries and packages;

• Dynamic Application Security Testing (DAST) to evaluate applications during execution;

• secret detection to identify credentials and other sensitive information;

• threat modelling to identify attack paths and security requirements;

• secure code review to assess implementation and architectural decisions;

• penetration testing for higher-risk applications; and

• runtime security monitoring to identify suspicious behaviour after deployment.

These mechanisms create multiple layers of defence around probabilistically generated software.

The principle is therefore:

AI should generate candidate implementations; independent security controls should determine whether those implementations are acceptable.

This separation is important because an LLM should not be treated as an independent validator of its own security decisions. Asking a model to generate an implementation and subsequently confirm that the same implementation is secure provides weaker assurance than subjecting the artefact to independent testing, analysis and review.

The objective is not to eliminate AI from security-sensitive development. It is to ensure that AI generation does not become the final authority on security.

5.8 Security Expertise Remains Essential

The increasing availability of AI coding assistants changes the distribution of software engineering work, but it does not eliminate the need for security expertise. Developers still require an understanding of authentication, authorisation, confidentiality, integrity, secure input handling, cryptography, threat modelling, common vulnerability classes and secure architecture.

Indeed, increased use of AI may make security expertise more important.

If AI lowers the cost of producing software, developers may produce substantially more software and make changes more frequently. This can increase the number of components, interfaces, dependencies and attack surfaces requiring assessment.

Klemmer et al. (2024) found that software professionals already use AI assistants for security-critical tasks despite concerns about their reliability. Participants generally recognised the need to scrutinise AI suggestions rather than treating them as authoritative.

This suggests an evolution in the role of security professionals. Rather than manually inspecting every line of AI-generated code, security specialists may increasingly focus on establishing security policies, architectural constraints, automated controls and governance mechanisms that operate across the development lifecycle.

Security expertise therefore shifts, at least in part, from inspecting individual artefacts towards designing the assurance system within which those artefacts are produced.

5.9 Security as an Organisational and Governance Problem

The security implications of AI-generated code extend beyond individual vulnerabilities. Organisations must establish governance over how AI is incorporated into software development.

Key questions include:

• Which AI tools are approved for development?

• What source code or proprietary information may be submitted to external AI services?

• How should sensitive data be protected?

• How should AI-generated code be identified and reviewed?

• What security testing is required?

• Who is accountable for vulnerabilities introduced through AI assistance?

• How should AI-generated dependencies and components be governed?

These questions are not purely technical. They concern risk ownership, accountability, data governance, procurement, compliance and organisational policy.

Klemmer et al. (2024) provide evidence of this governance challenge. Participants expressed concerns about the security implications of AI-assisted development and, in some cases, the need for standardised verification processes before AI-generated code is merged into production codebases. The study also identified concerns relating to insecure model behaviour and potential model poisoning.

Organisations adopting AI coding assistants therefore require policies that connect software development with information security, data protection, risk management and accountability. Such policies should establish not only which AI tools developers may use, but also how their outputs must be validated before becoming organisational software assets.

Governance should consequently address at least four dimensions:

Tool governance — which models and AI-enabled development environments are approved.

Data governance — what source code, credentials, customer information and proprietary material may be exposed to AI systems.

Engineering governance — what testing, review and security controls are required for AI-assisted changes.

Accountability governance — who remains responsible for software produced with AI assistance.

The final dimension is particularly important. AI may participate in software production, but organisational responsibility cannot simply be transferred to the model.

5.10 From AI Generation to Secure AI-Assisted Engineering

The evidence does not support either of two extreme conclusions. AI-generated software should not be considered inherently insecure, but neither should it be considered inherently trustworthy.

A more defensible position is that AI introduces another source of software artefacts whose security properties must be independently established.

The emerging security model can therefore be represented as:

AI-assisted generation → security analysis → testing → human review → controlled integration → runtime monitoring → feedback

This model recognises both the opportunities and limitations of generative AI. AI can accelerate implementation, assist with vulnerability identification and propose remediation strategies. At the same time, it can generate vulnerable code, reproduce insecure patterns, overlook security requirements and create misplaced confidence.

The appropriate organisational response is therefore not necessarily to prohibit AI-generated code, but to establish strong assurance mechanisms around its use.

The central argument of this chapter is:

AI-generated code is neither inherently insecure nor inherently trustworthy; its security must be established through explicit requirements, independent verification, automated security controls and appropriate human oversight.

This conclusion reinforces the broader argument developed throughout the paper. AI changes the economics and distribution of software production, but increased generation speed creates corresponding demands for security engineering, quality assurance and governance. The key issue is not whether AI can produce functional software, but whether organisations can integrate AI into development processes without weakening the security properties on which trustworthy software depends.

There is also a deeper systems-level implication. As AI becomes embedded across requirements, design, implementation, testing and maintenance, security can no longer be treated as a property of individual code fragments alone. It becomes a property of the human–AI development system: the models used, the data supplied to them, the prompts and specifications provided by developers, the tools through which generated code is integrated, the verification mechanisms surrounding it and the organisational controls governing its use.

The security challenge therefore extends beyond:

“Is this AI-generated code vulnerable?”

to the broader question:

“Can an organisation establish sufficient visibility, verification and accountability to manage the security consequences of software generated at AI-enabled speed and scale?”

This reframing is important because the principal risk may not be that AI creates entirely new categories of vulnerability. Rather, it may increase the speed, volume and reach of software production, allowing existing vulnerabilities or insecure practices to be introduced and propagated more rapidly.

Ultimately, secure AI-assisted software engineering requires a balance between automation and assurance. AI can increase the speed at which software is generated and contribute to security analysis, but human expertise, independent testing, automated controls and governance remain essential for establishing trust.

The challenge is therefore not to prevent AI from participating in software engineering, but to ensure that its growing role is bounded by controls capable of preserving security at scale.

This provides the foundation for the next chapter, which examines human–AI collaboration, including changing developer roles, verification responsibilities, trust, accountability and the transition from AI assistance towards increasingly autonomous development.

6. Human-AI Collaboration

The evidence reviewed throughout this paper suggests that the most plausible near-term model of AI-assisted software development is not the replacement of software engineers, but the emergence of structured human–AI collaboration. Large language models (LLMs) can generate code, explain unfamiliar systems, propose solutions, produce tests and assist with debugging, but these capabilities do not eliminate the need for human judgement. Instead, they alter where that judgement is applied. As AI systems become increasingly capable of producing implementation artefacts, the relative importance of defining objectives, supplying context, evaluating alternatives, validating outputs and accepting responsibility for consequential decisions is likely to increase.

This interpretation is consistent with the broader literature on LLMs in software engineering. Hou et al. (2024), in their systematic literature review, identify applications of LLMs across requirements engineering, design, implementation, testing, maintenance and other software engineering activities. Their findings indicate that AI is increasingly relevant across the software engineering lifecycle rather than being limited to code generation alone. Nam et al. (2024) similarly demonstrate the potential of LLM-based tools to support developers with code-understanding tasks, showing that AI can contribute to the cognitive work involved in navigating existing software systems.

The emerging development model can therefore be understood as a system in which humans establish intent and constraints, AI generates and transforms candidate solutions, automated mechanisms provide independent feedback, and humans retain responsibility for consequential decisions and outcomes. This differs both from traditional manual programming and from the concept of completely autonomous software development.

A useful representation is:

Human intent and requirements → AI-assisted generation and analysis → iterative human–AI refinement → automated verification → human judgement and approval → controlled deployment and monitoring

The significance of this model is that AI should not be considered merely as an isolated productivity application. Instead, it is increasingly becoming a component of a broader human–AI software engineering system in which model capability, developer expertise, repository context, verification mechanisms and organisational controls interact.

6.1 From Programmer to Orchestrator

The increasing capability of generative AI has implications for how software engineering work is distributed. Traditional development requires engineers to translate requirements into designs and subsequently express those designs through source code. AI-assisted development increasingly allows developers to describe desired behaviour, constraints and examples in natural language and ask an AI system to produce candidate implementations.

This does not make programming expertise obsolete. Instead, it changes the economic and cognitive value of programming expertise. Developers may spend less time manually expressing every implementation detail and more time determining whether a proposed implementation is appropriate, identifying hidden assumptions, comparing alternatives and integrating generated components into a larger system.

The developer's responsibilities consequently expand toward requirements interpretation, constraint definition, architectural evaluation, code review, testing, debugging, security assessment, integration and technical decision-making. In this sense, the developer increasingly becomes an orchestrator of software production rather than solely a producer of source code.

This distinction is important because the ability to generate software is not equivalent to the ability to engineer a software system. An LLM may generate several technically plausible implementations, but selecting the appropriate solution requires knowledge of the system's architecture, operational environment, performance requirements, security posture and long-term maintenance objectives. Hou et al. (2024) emphasise the breadth of software engineering activities in which LLMs are being investigated, reinforcing the distinction between code generation and broader engineering capability.

The growing importance of orchestration is also evident in research on code understanding. Nam et al. (2024) found that developers could use an LLM-based IDE interface to obtain explanations of code, API usage and domain-specific concepts, demonstrating that AI can assist not only with producing new code but also with understanding existing software.

The resulting shift is therefore better characterised as from code production toward software-production orchestration. Developers increasingly coordinate requirements, AI systems, repositories, tests, security controls, architectural constraints and deployment processes to produce a desired outcome.

6.2 Complementary Capabilities

Human–AI collaboration is most valuable when the capabilities of humans and machines are treated as complementary rather than interchangeable.

AI systems are particularly useful for rapidly generating candidate implementations, recognising recurring programming patterns, transforming existing code, summarising information, generating documentation and tests, explaining unfamiliar code and exploring alternative solutions. The breadth of these applications is reflected in the systematic literature reviewed by Hou et al. (2024), which identifies LLM applications across multiple stages of the software engineering lifecycle.

Human engineers, by contrast, remain essential where software development involves ambiguity, contextual interpretation, competing objectives and consequential trade-offs. Developers and architects must understand organisational priorities, business requirements, regulatory obligations, risk tolerance, existing infrastructure and the capabilities of the teams responsible for operating the resulting system.

The distinction can therefore be expressed as a division between machine-scale generation and human judgement. AI can rapidly explore what could be built; humans must determine what should be built, under what constraints and with what level of acceptable risk.

This distinction is particularly important because software engineering rarely has a single objectively optimal solution. An AI system may propose several technically valid architectures, but the preferred solution may depend on cost, organisational expertise, regulatory requirements, operational maturity, scalability expectations or strategic priorities. These considerations are not necessarily contained within the source code or immediate technical specification supplied to the model.

Human judgement therefore remains necessary even where AI can produce technically sophisticated alternatives.

6.3 Iterative Human–AI Interaction

Human–AI collaboration should not be conceptualised as a single prompt followed by acceptance of the resulting code. A more realistic model is iterative co-development, in which developers progressively constrain, evaluate and refine AI-generated solutions.

A typical interaction may begin with a developer specifying an objective and providing relevant context. The AI then proposes an implementation, which the developer evaluates. The developer may identify missing requirements, incorrect assumptions or architectural constraints and provide additional information. The AI produces a revised solution, after which automated tests and other verification mechanisms provide further evidence. The process can continue until the implementation satisfies the relevant requirements and quality criteria.

This model is supported particularly strongly by Fakhoury et al. (2024). Their TiCoder approach incorporates tests into an interactive process of intent clarification rather than treating natural-language code generation as a one-shot activity. In their user study, participants using the interactive approach were more successful in evaluating generated code and experienced lower task-induced cognitive load. Their broader empirical evaluation also reported substantial improvements in pass@1 performance across the datasets and models examined.

The significance of this research extends beyond test generation. It demonstrates that interaction itself can become a mechanism for improving the reliability of AI-assisted development. Human feedback supplies information that may be absent from the original prompt, while tests provide an executable representation of expected behaviour. The interaction between developer, model and verification mechanism consequently becomes part of the engineering process (Fakhoury et al., 2024).

Human–AI collaboration is therefore better understood as iterative problem-solving than autonomous code generation. The AI generates candidates; humans provide context and judgement; automated systems provide evidence; and successive iterations progressively narrow the gap between intended and implemented behaviour.

6.4 Verification as a Central Human Responsibility

As AI becomes more capable, verification becomes more—not less—important. AI-generated code can appear coherent and professionally structured while nevertheless containing functional errors, inappropriate assumptions or security weaknesses. Human developers must therefore determine whether an output is appropriate before it becomes part of a production system.

Klemmer et al. (2024) provide useful evidence concerning how software professionals approach this problem. Their qualitative study examined the experiences of software professionals using AI assistants and found that practitioners expressed significant concerns regarding the reliability and security of AI-generated output. At the same time, participants reported using AI assistants for security-related activities including code generation, threat modelling and vulnerability detection. Their findings indicate that developers do not necessarily treat AI output as authoritative; instead, they continue to apply verification and judgement when incorporating AI-generated recommendations into their work.

This finding challenges the assumption that AI assistance automatically eliminates expert review. Instead, AI appears to be becoming another source of engineering artefacts that developers must assess.

Verification should therefore be regarded as a central component of the human role, rather than as a final administrative step. Developers need sufficient understanding of generated code to determine whether it satisfies requirements, interacts correctly with surrounding components, introduces security risks and remains consistent with architectural principles.

The importance of this verification role is reinforced by research demonstrating that LLM-generated code can contain security weaknesses. Hamer, d'Amorim and Williams (2024) found that ChatGPT-generated code exhibited security vulnerabilities when compared with Stack Overflow answers, while Wang et al. (2024), through CodeSecEval, demonstrated challenges in secure code generation and vulnerability remediation across multiple vulnerability categories. AI-generated code therefore requires independent assessment rather than implicit trust.

As AI-generated software becomes more prevalent, the value of engineering expertise may consequently shift from the ability to produce every line of code manually toward the ability to evaluate, constrain and validate machine-generated software.

6.5 The Risk of Automation Bias

Human oversight, however, is not automatically effective. One of the principal risks of AI-assisted development is automation bias, whereby users place excessive confidence in suggestions generated by an automated system.

This risk is amplified by the presentation of LLM output. AI-generated code can be accompanied by fluent explanations, comments and apparently rational justifications. The combination of polished implementation and confident language can create an impression that the underlying solution has been more rigorously validated than it actually has.

The problem becomes particularly acute when developers lack expertise in the relevant domain, operate under time pressure or encounter output that passes basic tests. A developer may infer that a solution is reliable because it compiles, appears well structured and satisfies a limited set of test cases, even though important assumptions or edge cases remain unexamined.

The issue is therefore not simply whether a human remains in the development loop. The more important question is whether the human remains meaningfully engaged in the loop.

Meaningful oversight requires sufficient technical expertise, adequate review time, independent verification mechanisms and organisational processes that make questioning AI output both possible and expected. Klemmer et al. (2024) provide evidence that software professionals remain concerned about the reliability and security of AI assistance, reinforcing the importance of active verification rather than passive acceptance.

The issue is further complicated by the non-deterministic nature of LLM code generation. Ouyang et al. (2024) found substantial variability in ChatGPT-generated solutions to the same coding problems and demonstrated that setting the temperature parameter to zero did not guarantee deterministic generation. Their findings indicate that apparently similar interactions can produce materially different outputs.

Non-determinism therefore matters for human–AI collaboration because developers may not always be interacting with a stable computational process. If similar requests can result in different implementations, reproducibility, review and accountability become more complicated. AI-assisted engineering consequently requires processes that capture sufficient context, generated artefacts, tests and approvals to make consequential changes traceable (Ouyang et al., 2024).

6.6 Human Expertise Becomes More, Not Less, Important

The increasing capability of AI does not necessarily reduce the importance of software engineering expertise. In some respects, it may increase it.

If AI assumes more responsibility for routine implementation, the relative importance of higher-order skills—including requirements analysis, architecture, security, testing, debugging, integration, risk assessment and AI-output evaluation—may increase. A developer who can generate code quickly but cannot assess its correctness may be less effective than an engineer who can critically evaluate and integrate machine-generated solutions.

Consider a system in which an AI assistant proposes three possible implementations of a complex service. The value of receiving three alternatives depends on the developer's ability to understand their respective trade-offs. Without adequate architectural and domain knowledge, greater AI-generated choice can simply produce greater uncertainty.

This suggests that AI may augment expertise rather than eliminate it. Experienced developers may be able to use AI more effectively because they can provide better specifications, identify inappropriate assumptions and recognise subtle defects. This interpretation is consistent with the broader evidence reviewed by Hou et al. (2024), which shows that LLM applications span diverse software engineering activities whose effective use depends on substantial contextual understanding.

The implication for software engineering education is substantial. Future engineers will need to learn not only how to write software but also how to specify software for AI systems, evaluate generated implementations, design effective tests, identify security weaknesses and govern increasingly capable development tools.

Programming knowledge therefore remains important, but its function may increasingly include constraining and evaluating AI-generated software rather than exclusively producing software manually.

6.7 Human–AI Collaboration in Security

The need for human judgement becomes particularly pronounced in security-sensitive development.

AI can assist with vulnerability identification, secure coding suggestions, security-test generation, preliminary threat modelling, explanation of security weaknesses and remediation proposals. Klemmer et al. (2024) found that software professionals were already using AI assistants for several security-related activities while simultaneously expressing concerns about their reliability and security.

The appropriate interpretation is not that AI should become an autonomous security authority. Security decisions are inherently contextual and frequently depend on threat models, organisational risk tolerance, regulatory requirements, system architecture and the consequences of failure.

The limitations of AI-generated security advice are supported by empirical research. Hamer, d'Amorim and Williams (2024) identified security vulnerabilities in ChatGPT-generated code, while Wang et al. (2024) found limitations in secure code generation and vulnerability remediation across multiple vulnerability classes. Liu et al. (2024) similarly demonstrate the importance of careful evaluation when LLMs are applied to safety-critical software.

An AI system may therefore identify a potentially useful security improvement while failing to understand the complete context in which the decision must be made. AI can contribute analysis without possessing the authority to determine whether a system is acceptably secure.

A robust model therefore combines AI assistance with independent security analysis, automated controls and human security expertise. In this model, AI expands the analytical capacity available to developers and security teams, while humans retain responsibility for security decisions.

6.8 From Human-in-the-Loop to Human-on-the-Loop

The increasing autonomy of AI systems raises an important question concerning the form of human involvement. A traditional human-in-the-loop model requires human approval of individual AI actions. This approach provides direct oversight but may become impractical when AI systems perform large numbers of routine tasks.

An alternative is a human-on-the-loop model. Here, humans establish objectives, permissions, policies, constraints and escalation mechanisms, while AI systems perform routine activities within those boundaries. Human intervention is required at defined decision points rather than after every individual action.

For example, an AI development agent might be authorised to create a branch, inspect a repository, modify code, generate tests, execute a test suite, analyse failures and prepare a pull request. However, merging that change into a protected production branch could require human approval.

The resulting architecture can be represented as:

AI acts within defined boundaries → automated controls verify behaviour → humans intervene at consequential decision points

This approach may provide a more scalable model of AI autonomy. Rather than requiring a human to observe every low-level action, organisations can establish controls around permissions, repositories, testing, security and deployment while reserving human judgement for decisions with significant technical, financial, security or regulatory consequences.

The need for such controls becomes increasingly relevant as the scope of LLM applications expands across the software lifecycle (Hou et al., 2024). Increased capability should therefore be accompanied by appropriately increased governance.

6.9 Accountability Cannot Be Delegated to AI

One principle remains unchanged regardless of the degree of AI autonomy: accountability cannot be delegated to the model.

If AI-generated software causes a security incident, operational failure, financial loss or regulatory breach, stating that “the AI generated the code” does not establish meaningful organisational responsibility. The organisation must still be able to determine who defined the requirements, who authorised the AI's use, which controls were applied, who reviewed the output and who approved deployment.

This makes traceability increasingly important. AI-assisted development processes should, where proportionate, retain information about generated artefacts, relevant prompts or specifications, test results, security checks, review decisions and deployment approvals. Such records can support incident investigation and help organisations understand how a consequential change was produced.

The security literature reinforces the importance of this principle. Klemmer et al. (2024) show that practitioners recognise security and reliability concerns associated with AI assistance, while Wang et al. (2024) and Hamer, d'Amorim and Williams (2024) demonstrate empirically that generated code cannot simply be assumed to be secure.

Accountability also becomes more important as AI systems gain greater agency. The more actions an AI system is permitted to take, the clearer the organisation must be about who authorised those permissions, what constraints apply and where human intervention is mandatory.

The principle can therefore be stated simply:

Autonomy may be delegated; accountability may not.

6.10 A New Software Engineering Operating Model

Human–AI collaboration ultimately suggests a transformation in the software engineering operating model itself.

Rather than treating AI as an additional application used alongside conventional development tools, organisations can increasingly embed AI capabilities throughout the lifecycle:

Human intent and requirements → AI-assisted analysis and design → AI-generated implementation → AI-assisted and conventional testing → automated quality and security controls → human architectural and risk review → controlled deployment → runtime monitoring → continuous improvement

This represents a shift from AI as a development tool toward AI as a component of the software production system.

The organisational implications are significant. Development teams may require revised code-review processes, stronger automated testing, improved repository controls, clearer AI-use policies and new governance mechanisms for determining when AI can act autonomously. Security teams may increasingly focus on establishing automated policies and assurance controls that operate continuously rather than manually inspecting every AI-generated change.

The need for stronger verification is supported by research demonstrating limitations in AI-generated security and safety-critical code (Hamer, d'Amorim and Williams, 2024; Liu et al., 2024; Wang et al., 2024). At the same time, Peng et al. (2023) provide evidence that AI coding assistance can generate substantial productivity benefits under controlled conditions. Taken together, these findings suggest that organisations need to capture productivity gains without allowing generation capacity to outpace verification and assurance capacity.

This also changes the economics of engineering management. If AI increases the rate at which code can be generated, organisations may need to invest proportionally more in automated testing, observability, security analysis and governance. Otherwise, software production may accelerate faster than the organisation's ability to assess and control what is being produced.

6.11 From Assistance to Agency

The progression from AI assistance to AI agency represents a further development of the collaboration model.

A conventional coding assistant generally responds to explicit developer requests:

Prompt → Code

An increasingly autonomous development agent can instead operate toward a higher-level objective:

Objective → Plan → Inspect repository → Modify code → Run tests → Analyse failures → Revise implementation → Submit change

The distinction is fundamental. In the first model, the developer remains responsible for decomposing the problem into individual interactions. In the second, the AI system can perform substantial portions of that decomposition itself.

This increases potential productivity, but it also increases the importance of permissions, observability, reproducibility, testing and governance. Hou et al. (2024) identify the increasing breadth of LLM applications across software engineering, while Ouyang et al. (2024) demonstrate that LLM-generated code can exhibit substantial variability. As AI systems become more autonomous, these characteristics make controlled execution and traceability increasingly important.

The transition from assistance to agency should therefore not be understood simply as a progression in model capability. It is also a progression in governance requirements. Greater autonomy should be accompanied by stronger controls.

6.12 The Emerging Role of the Software Engineer

The evidence ultimately suggests that AI is unlikely to make software engineers irrelevant in the near term. Instead, it is likely to change what engineering expertise is used for.

Developers may spend proportionally less time producing routine implementation code and more time defining system objectives, specifying constraints, evaluating AI-generated alternatives, validating system behaviour, managing security and risk, coordinating AI agents and development tools, and accepting responsibility for the resulting system.

The most valuable capability may therefore become the ability to translate organisational intent into precise constraints and testable expectations that can guide both human and AI participants, and then determine whether the resulting system satisfies those expectations.

This also reinforces the importance of code understanding. Nam et al. (2024) demonstrate that LLM-based assistance can support developers in understanding unfamiliar code and APIs, suggesting that AI can contribute to the cognitive work required to navigate complex software systems as well as to software production itself.

The resulting engineer is therefore not simply a programmer who uses AI, but a professional who coordinates intent, architecture, AI systems, verification mechanisms and operational constraints.

The central proposition is consequently that the most sustainable model of AI-assisted software development is not human versus AI, but human judgement combined with machine-scale generation, analysis and automation. AI can expand the amount of software that engineers can generate, understand and modify, but humans remain essential for intent, context, judgement and accountability.

The trajectory toward increasingly autonomous AI agents makes this distinction more—not less—important. As machines become capable of performing larger portions of the development process, the central engineering challenge shifts from determining whether AI can perform a task to determining under what conditions it should be permitted to perform that task, what evidence is required before its output is accepted, and who remains responsible for the outcome.

Thus, the future of software engineering is unlikely to be defined simply by increasingly powerful coding models. It will be defined by the quality of the socio-technical system surrounding those models: the developers who direct them, the architectures that constrain them, the tests and security controls that evaluate them, the workflows through which their outputs are integrated, and the governance structures that determine the boundaries of their autonomy.

This provides the foundation for the next chapter, which examines the organisational and governance implications of embedding AI more deeply within software engineering, including accountability, policy, process redesign and the controls required to ensure that increasing AI autonomy does not outpace engineering assurance.

7. The Problem of Non-Determinism

One of the more distinctive challenges introduced by large language models (LLMs) into software engineering is non-determinism. Conventional software engineering is built around the expectation that, given the same program, inputs and relevant execution environment, software behaviour can be reproduced with a high degree of consistency. This property underpins testing, debugging, defect investigation, configuration management and operational reliability. AI-assisted software development introduces an additional probabilistic layer into this otherwise relatively deterministic process. The same or substantially similar instruction can produce different implementations across interactions, even when the requested functionality remains unchanged.

This distinction is important because variation in AI-generated output is not necessarily limited to superficial differences in formatting or naming. Different generations may employ different algorithms, dependencies, error-handling strategies, architectural patterns or security mechanisms. They may therefore differ in performance, maintainability, readability and security even when they satisfy the same basic functional requirement. Ouyang et al. (2024) provide empirical evidence that non-determinism in ChatGPT code generation can affect the consistency and reproducibility of generated solutions and can consequently complicate the evaluation of LLM-based programming systems.

The issue should therefore be understood as more than a technical characteristic of LLMs. It represents a software engineering and governance challenge because organisations must manage both the software artefact produced by AI and the variable process through which that artefact was produced. This becomes increasingly important as AI moves from generating isolated code fragments towards participating in requirements analysis, code generation, testing, debugging and increasingly autonomous development workflows (Hou et al., 2024).

The central challenge can be expressed simply:

Traditional software engineering seeks predictable behaviour from deterministic artefacts; AI-assisted engineering must additionally govern a probabilistic generation process.

The objective, however, should not necessarily be to eliminate all variability. A more practical goal is to establish controlled variability, in which different AI-generated implementations are acceptable provided that they satisfy clearly defined functional, security, architectural and operational constraints.

7.1 Understanding Non-Determinism in AI-Assisted Development

In conventional software development, the relationship between a program and its execution can be represented approximately as:

Program + input + defined environment → reproducible behaviour

AI-assisted development introduces a different relationship:

Prompt + model + context + generation conditions → candidate implementation

Repeating the second process does not necessarily result in the same implementation. A developer asking an AI system to implement a transaction-processing function may receive one solution during an initial interaction and a materially different solution during a subsequent interaction. Both may be syntactically valid and pass the same basic tests, while differing in their dependencies, computational complexity, error handling or security characteristics.

Ouyang et al. (2024) demonstrate why this matters for software engineering: variability in LLM-generated code can affect evaluation outcomes and complicate assumptions about the consistency of AI-assisted programming. Their findings indicate that repeated generations cannot necessarily be treated as interchangeable evidence of model behaviour.

This leads to an important distinction:

Functional equivalence does not imply engineering equivalence.

Two implementations may produce the same outputs for the tested inputs while differing substantially in maintainability, efficiency, security, scalability or architectural suitability. Consequently, evaluating AI-assisted development requires more than establishing whether an LLM can generate a correct solution. It requires understanding whether the development process can consistently produce software that satisfies the organisation's broader engineering requirements.

7.2 Non-Determinism and Reproducibility

Reproducibility is fundamental to both software engineering and empirical research. When a defect is discovered, engineers need to reconstruct the circumstances under which it occurred, understand the relevant implementation and determine whether the problem can be reproduced. Conventional version-control systems provide much of this capability by recording source-code changes, authorship and historical versions.

AI-assisted development complicates this model because the source code may be only one part of the relevant development record.

Consider a situation in which an AI-generated component is subsequently found to contain a security vulnerability. Reconstructing its origin may require knowledge of:

• the exact prompt and system instructions;

• the model and model version;

• model configuration and generation parameters;

• the repository and contextual information supplied to the model;

• documentation or external material retrieved during generation;

• tools or agents available to the AI system;

• the original generated output;

• subsequent developer modifications; and

• the tests and security checks applied before integration.

Without such information, reproducing the generation process may be difficult or impossible.

This creates an important extension to conventional configuration management:

Reproducible AI-assisted development requires traceability of both the software artefact and the conditions under which the artefact was generated.

The requirement becomes particularly important when AI systems are updated. A change to the underlying model or AI-assisted development environment can alter generated outputs, meaning that a workflow that previously produced an acceptable implementation may behave differently at a later point.

This concern is consistent with the broader literature reviewed by Hou et al. (2024), which identifies the increasing integration of LLMs into diverse software engineering activities and highlights the importance of evaluating these systems within their broader application context rather than as isolated code generators.

For organisations using AI in production engineering, significant model changes should therefore be considered potentially relevant changes to the development environment rather than necessarily being treated as invisible service updates.

7.3 Model Versioning as Configuration Management

Traditional software engineering treats dependencies, libraries, compilers and build environments as configuration-controlled components. AI-assisted development suggests that the AI model itself should increasingly be treated as part of the development configuration.

A meaningful record of an AI-assisted change may therefore need to capture the model identity and version alongside conventional source-code and dependency information. Where appropriate, organisations may also need to record relevant generation parameters, system instructions, prompts, contextual inputs and retrieved information.

This is analogous to dependency management. If an application depends on a particular library version, silently replacing that library with a different version can alter application behaviour. Similarly, changing the model used within an AI-assisted development workflow can alter the software that the workflow produces.

Model versioning therefore becomes particularly important for:

• regulated software development;

• safety-critical systems;

• security-sensitive applications;

• reproducible research;

• long-lived enterprise applications; and

• automated AI development pipelines.

The implication is not that every casual interaction with an LLM must be exhaustively archived. Rather, the degree of provenance and configuration control should be proportional to the consequences of the software being produced.

This risk-based approach is particularly appropriate in safety-critical contexts, where empirical research has already raised questions about the reliability and suitability of LLM-generated software (Liu et al., 2024).

7.4 Prompts as Engineering Artefacts

Non-determinism also raises the status of prompts within software engineering. A prompt is not necessarily just an informal instruction to an AI system. In a sufficiently structured development workflow, it can contain requirements, constraints, coding standards, architectural assumptions and security expectations that materially influence the resulting implementation.

Consider the difference between:

“Write a function that validates user input.”

and:

“Implement secure input validation for an API endpoint. Reject malformed input, prevent injection attacks, return structured validation errors and conform to the project's existing validation framework.”

The second instruction provides substantially more information about the intended behaviour and constraints. The generated implementation is consequently more strongly bounded by explicit engineering requirements.

This suggests that prompts can function as a form of machine-interpretable specification. As organisations increasingly use standardised prompts to generate production artefacts, important prompts may warrant version control, review and change management alongside other engineering specifications.

The emerging configuration relationship can therefore be represented as:

Prompt + context + model + constraints → generated software

Managing these inputs becomes particularly important when generated artefacts are subsequently reused or regenerated. A change to the prompt may produce different software even if the underlying model remains unchanged.

7.5 Context as Part of the Generation Process

The model alone is not sufficient to explain an AI-generated implementation. Modern AI-assisted development increasingly depends on contextual information, including repository contents, documentation, coding standards, issue descriptions, dependency information and existing tests.

Consequently, the effective generation environment can be represented more completely as:

Model + prompt + repository context + tools + retrieved information + generation conditions → output

This has significant implications for reproducibility. Two developers using the same model and prompt may obtain different results if the AI system has access to different repository information or retrieved documentation.

The problem is therefore not simply model non-determinism. It is also contextual variability.

This reinforces the broader findings of Hou et al. (2024), whose systematic review demonstrates that LLM applications in software engineering extend across activities including code generation, maintenance, testing and other lifecycle tasks. As AI becomes embedded across the software lifecycle, the development context supplied to the model becomes an increasingly important part of the engineering environment.

Research on LLM-assisted code understanding similarly illustrates the importance of providing models with appropriate software context rather than treating generated responses as independent from the surrounding codebase (Nam et al., 2024).

7.6 Validation Rather Than Exact Reproduction

Non-determinism does not necessarily mean that AI-generated software must be rejected. An important distinction exists between requiring identical output and requiring equivalent engineering properties.

Suppose an AI system produces five different implementations of the same requirement. If all five satisfy the same functional tests, security requirements, performance thresholds and architectural constraints, their differences may be acceptable.

The engineering objective can therefore shift from:

“Did the model generate exactly the same code?”

to:

“Does the generated implementation satisfy the required properties?”

This distinction is particularly important because software engineering has always permitted multiple valid implementations of the same requirement. Developers can choose different algorithms, data structures or design patterns while producing functionally equivalent systems.

AI non-determinism therefore becomes problematic primarily when it causes generated solutions to fall outside acceptable engineering boundaries.

The appropriate response is consequently constraint and validation rather than an unrealistic pursuit of absolute determinism.

7.7 Testing as a Stability Mechanism

Automated testing provides one of the most important mechanisms for controlling variability in AI-generated software. A comprehensive test suite establishes a relatively stable reference against which different generated implementations can be evaluated.

The process can be represented as:

Variable AI generation → fixed requirements and tests → validated implementation

This approach is consistent with the test-driven and interactive generation model investigated by Fakhoury et al. (2024). Their empirical study examines LLM-based interactive code generation and demonstrates the importance of incorporating testing and developer interaction into the generation process rather than treating AI generation as a single isolated step.

This is particularly important in a non-deterministic environment. If the AI produces different implementations, automated tests can establish whether those differences remain within acceptable behavioural boundaries.

Testing can therefore act as a stability mechanism around probabilistic generation.

The same principle applies to other automated assurance mechanisms. Static analysis, type checking, dependency analysis, security scanning and policy-based validation can establish constraints that generated implementations must satisfy regardless of the precise form of the generated code.

7.8 Non-Determinism and Developer Trust

Variability also has implications for how developers perceive and trust AI systems.

If an AI system generates materially different answers to the same request, developers may question which output should be trusted. This can create two opposing risks.

The first is over-trust, where developers accept a plausible response without sufficient independent validation. The second is under-trust, where excessive variability causes developers to disregard AI assistance even when it could provide useful productivity benefits.

The appropriate objective is therefore calibrated trust.

Developers need to understand both the capabilities and limitations of AI systems and should be able to determine when generated output requires additional verification. Klemmer et al. (2024) identify both the usefulness of AI assistants and significant security-related concerns among software professionals, reinforcing the importance of integrating AI into established development practices rather than treating it as an authoritative source of engineering decisions.

Trust should therefore be grounded in evidence, process and verification, rather than in the apparent confidence or sophistication of an AI-generated response.

7.9 Provenance and Change Tracking

Non-deterministic generation also expands the concept of software provenance.

Conventional version control can establish:

• who changed the software;

• what was changed;

• when the change occurred; and

• which previous version was modified.

AI-assisted development introduces additional provenance questions:

• Was the original artefact generated by AI?

• Which model and version were used?

• What prompt and context influenced generation?

• Which external sources or tools were involved?

• How substantially did the developer modify the output?

• Which automated checks were performed?

• Who reviewed and approved the resulting change?

These questions become particularly important where software failures have regulatory, financial, safety or security consequences.

An expanded provenance model can therefore be expressed as:

Origin → generation → modification → verification → approval → deployment

Maintaining this chain provides organisations with a stronger basis for investigating defects, assessing responsibility and demonstrating compliance.

This principle is also relevant to security. Klemmer et al. (2024) report that developers have concerns about how AI assistants interact with security practices, suggesting that organisational processes must account for how AI-generated artefacts enter and are subsequently reviewed within development workflows.

7.10 Non-Determinism and Security

The implications of non-determinism become particularly important when security is considered. AI-generated code can contain security vulnerabilities even when it appears functionally correct. Wang et al. (2024), through the CodeSecEval evaluation, demonstrate that LLMs can struggle with secure code generation and vulnerability remediation across a broad range of vulnerability categories. Hamer, d'Amorim and Williams (2024) similarly investigate the security vulnerabilities present in ChatGPT-generated code and provide evidence that AI-generated implementations should not automatically be assumed to be secure simply because they are syntactically valid or functionally successful.

Non-determinism adds another dimension to this problem. Different generations of the same requested implementation may not have identical security properties. One implementation may use a safer API or stronger validation mechanism than another.

Security validation must therefore be applied to the actual artefact being integrated, rather than inferred from the fact that an earlier generation was considered secure.

This reinforces the importance of independent security controls such as static application security testing, dependency analysis, vulnerability scanning and security-focused testing. AI generation should be treated as variable, while the security requirements imposed upon its output should remain stable.

The principle is therefore:

Variable generation requires invariant security constraints.

This principle is especially important in high-consequence environments. Liu et al. (2024), for example, examine LLM-generated code in safety-critical software and illustrate why conventional functional success cannot by itself establish suitability for safety-sensitive development.

7.11 Non-Determinism and Software Quality

The same principle applies to broader software quality. Ouyang et al. (2024) demonstrate that non-determinism can influence code-generation outcomes and therefore complicate evaluation of LLM performance. If two runs of the same development workflow produce different implementations, measuring AI productivity or quality using a single generated result may provide an incomplete picture.

This has methodological implications for both organisations and researchers. Evaluations should account for variability across multiple generations rather than assuming that one output represents stable model behaviour.

For organisations, this means that AI tools should be evaluated not only according to their best-case output, but also according to the consistency and reliability of their outputs across representative tasks.

For researchers, it means that claims about model capability should distinguish between isolated successful generations and reproducible performance across repeated trials.

Non-determinism therefore affects both software production and the evaluation of software-generation technologies.

7.12 Implications for AI Agents

The problem becomes substantially more consequential as AI systems evolve from assistants into autonomous development agents.

A conventional coding assistant might perform:

Prompt → code suggestion

An AI development agent may instead perform:

Objective → planning → repository inspection → code modification → test execution → failure analysis → further modification → pull request

Each additional autonomous action creates another opportunity for variation.

An agent may take a different implementation path, modify different files, interpret test failures differently or perform a different sequence of corrective actions when given the same objective. As autonomy increases, the importance of observability, action logging, state management, permissions, rollback and human intervention increases correspondingly.

This connects non-determinism directly to the broader human–AI collaboration model. Hou et al. (2024) document the expanding scope of LLM applications across software engineering, while Klemmer et al. (2024) demonstrate that developers' perceptions, practices and security concerns remain important factors in AI-assisted development.

The principle is straightforward:

The greater the autonomy of an AI system, the greater the need to record what it did, constrain what it can do and establish how its actions can be reproduced or reversed.

7.13 Engineering Controls for Managing Non-Determinism

Non-determinism can be managed through a combination of conventional software engineering practices and AI-specific controls.

First, model versioning should be used for workflows where reproducibility or assurance is important. Organisations should know which model generated a significant software artefact and should control model changes where necessary.

Second, prompt versioning should be considered for important or repeatedly used generation workflows. Changes to prompts can materially alter the resulting software and should therefore be treated as changes to the development process.

Third, context management should be implemented where reproducibility matters. Relevant repository state, documentation, retrieved information and tool access may need to be recorded or controlled.

Fourth, automated validation should establish stable acceptance criteria through testing, static analysis, type checking, security analysis and other quality controls. Fakhoury et al. (2024) provide empirical support for interactive, test-driven approaches in which testing becomes part of the generation and validation process rather than merely a final-stage activity.

Fifth, provenance and change tracking should establish how an AI-generated artefact was created and subsequently modified.

Sixth, human approval should remain mandatory for changes whose consequences exceed the authority delegated to the AI system.

Together, these mechanisms transform non-determinism from an uncontrolled source of variability into a managed characteristic of the engineering process.

7.14 Controlled Variability Rather Than Absolute Determinism

A key conceptual distinction is therefore required between determinism and reproducibility.

Absolute determinism would require an AI system to generate exactly the same output every time it receives the same input. This is neither necessarily realistic nor always desirable. Software engineering routinely permits multiple valid solutions to the same problem.

Reproducibility instead requires that an organisation can reconstruct the relevant development conditions sufficiently to understand, evaluate and govern the resulting artefact.

For example, two AI-generated implementations may differ significantly at the source-code level while both satisfying:

• the same functional requirements;

• the same security constraints;

• the same performance thresholds;

• the same architectural principles; and

• the same testing criteria.

In such circumstances, variability may be an acceptable feature of the development process rather than a defect.

The objective should therefore be:

Controlled variability, not absolute determinism.

This distinction provides a more useful foundation for integrating probabilistic AI systems into established software engineering practices.

7.15 From Configuration Management to AI Configuration Management

The implications of non-determinism ultimately extend conventional configuration management.

Traditional configuration management primarily controls source code, dependencies, build tools, environments and deployment configurations. AI-assisted development introduces additional variables into this configuration space: models, model versions, prompts, contextual data, retrieval mechanisms, tools and generation parameters.

The resulting development environment can therefore be understood as:

Traditional configuration + AI configuration + continuous verification

This represents an important extension of software engineering practice. Organisations do not necessarily need to control every interaction with an AI assistant, but critical AI-assisted workflows should have sufficient configuration control to support reliability, security and accountability.

The more consequential the software, the stronger these controls should become.

7.16 The Broader Engineering Implication

The problem of non-determinism illustrates a broader principle underlying AI-assisted software engineering.

AI introduces probabilistic generation into a discipline that has historically relied heavily on deterministic artefacts and repeatable processes. This does not make conventional engineering practices obsolete. Instead, it increases the importance of surrounding probabilistic generation with stable requirements, automated verification, configuration management and human accountability.

The resulting model can be represented as:

Probabilistic AI generation + deterministic engineering controls + continuous verification

This model also connects the major themes developed throughout the paper. AI can accelerate code production and support developers across the software lifecycle (Hou et al., 2024; Peng et al., 2023). However, generated output remains subject to functional and security limitations (Wang et al., 2024; Hamer, d'Amorim and Williams, 2024). Interactive, test-driven approaches can improve the reliability of generation (Fakhoury et al., 2024), while effective use of AI depends on developers retaining sufficient expertise and exercising meaningful oversight (Klemmer et al., 2024).

Non-determinism therefore does not represent a reason to exclude AI from software engineering. Rather, it demonstrates why AI must be integrated within an engineering system capable of controlling and evaluating probabilistic behaviour.

7.17 Conclusion: Governing the Generation Process

The central challenge posed by LLM non-determinism is not simply that AI can produce different answers. Software engineering has always permitted multiple solutions to the same problem. The deeper issue is that the process by which AI-generated solutions are produced may itself vary, potentially affecting correctness, security, quality, reproducibility and developer trust (Ouyang et al., 2024).

As a result, organisations must increasingly manage two related artefacts: the software that is produced and the conditions under which it was produced.

This requires an expanded approach to software engineering in which model versions, prompts, context, generated outputs, validation results and human modifications can form part of the development record where the risk profile warrants it. Automated testing, static analysis, security controls and continuous integration provide stable mechanisms for evaluating variable AI outputs, while human engineers remain responsible for requirements, context, judgement and accountability.

The central argument can therefore be stated as follows:

Non-determinism does not prevent AI from becoming part of software engineering; it changes the engineering controls required to use AI responsibly.

The objective is not to force probabilistic systems into behaving like conventional deterministic programs. It is to surround probabilistic generation with sufficiently strong requirements, constraints, validation, provenance, configuration management and human oversight that variability remains within acceptable boundaries.

This becomes even more important as AI progresses from code assistants towards autonomous development agents. The greater the ability of AI to plan, modify, test and integrate software independently, the greater the need for organisations to establish clear permissions, observable actions, reproducible workflows, rollback mechanisms and accountable decision points.

The resulting transformation is therefore not simply a transition from human-written code to AI-generated code. It is a transition towards a software engineering environment in which probabilistic generation must coexist with deterministic assurance. The central engineering question consequently shifts from whether AI can consistently generate the same software to whether organisations can consistently ensure that whatever software AI generates is acceptable, secure, testable, maintainable and accountable.

This provides a direct bridge to the next chapter: if AI-generated software is variable, increasingly autonomous and embedded throughout the development lifecycle, then organisations require not only technical controls but also governance structures that define responsibility, permissible AI use, oversight, accountability and the boundaries of machine autonomy.

8. Evaluation: Beyond “Does the Code Run?”

A major limitation of current research on AI-assisted software development is that evaluation often remains centred on narrow measures of functional correctness. Benchmarks such as HumanEval and MBPP have made an important contribution by providing standardised methods for assessing whether large language models (LLMs) can generate code that satisfies predefined programming tasks. Such benchmarks are useful for establishing reproducible measures of code-generation capability, but systematic reviews of the field indicate that software engineering encompasses a substantially broader range of activities and quality dimensions than isolated code generation alone (Hou et al., 2024).

These benchmarks typically involve relatively small, self-contained problems and therefore capture only one dimension of software engineering performance. They provide useful evidence of a model's ability to generate functionally correct code under controlled conditions, but they do not establish whether that model can contribute reliably to the development and maintenance of production-quality software. Hou et al. (2024) consequently emphasise the importance of evaluating LLMs across a broader range of software engineering activities rather than treating code-generation performance as a sufficient proxy for overall engineering capability.

Real-world software engineering is considerably more complex. Production systems typically consist of multiple interacting components, evolve over time and operate within technical, organisational, security and regulatory constraints. Developers must work with existing codebases, dependencies, architectural standards, legacy systems, changing requirements and operational environments. Research into LLM-assisted code understanding similarly demonstrates that effective use of AI in software engineering requires models to work with and reason about existing code rather than simply generate isolated implementations (Nam et al., 2024). A model that can generate a function that passes a predefined test may therefore demonstrate useful coding capability without demonstrating the broader competencies required for professional software engineering.

The evaluation of AI-assisted software development should therefore move beyond the question of whether generated code executes successfully. A more meaningful assessment must consider whether the resulting software is secure, maintainable, efficient, reliable, understandable, architecturally appropriate and operationally suitable. Equally important, evaluation should consider the human effort required to understand, review, validate and maintain AI-generated artefacts. The relevant unit of analysis is consequently not simply the generated code, but the software system and engineering process within which that code is produced and used. This broader perspective is consistent with the systematic research agenda proposed by Hou et al. (2024) and with empirical studies examining security, code understanding, testing and developer interaction with AI systems (Hamer et al., 2024; Klemmer et al., 2024; Nam et al., 2024; Fakhoury et al., 2024).

8.1 The Limitations of Functional-Correctness Benchmarks

Benchmarks such as HumanEval and MBPP remain valuable because they provide reproducible measurements of code-generation capability. They allow researchers to compare models under relatively standardised conditions and determine whether generated implementations satisfy predefined tests. Such benchmarks are particularly useful for measuring progress in basic programming competence and for establishing whether models can translate natural-language specifications into executable solutions.

Their limitation is not that they are inappropriate, but that they measure a relatively narrow aspect of software engineering. Hou et al. (2024), in their systematic literature review, identify the breadth of software engineering activities that must be considered when evaluating LLM-based systems. Professional software development rarely involves producing a single function in isolation. Instead, developers must understand how new code interacts with existing systems, dependencies, interfaces, configuration, documentation and architectural decisions.

They must also account for non-functional requirements such as security, performance, reliability and maintainability. Research examining safety-critical software generation illustrates the importance of this distinction: functional code generation alone is insufficient when software operates under stringent safety and reliability requirements (Liu et al., 2024).

A model can therefore perform exceptionally well on an isolated function-generation benchmark while remaining poorly suited to modifying a large enterprise repository. The benchmark may establish that the model can solve the programming problem presented to it, but it does not establish whether the model can safely determine where the solution belongs, how it should interact with existing components or what unintended consequences it might create.

This distinction is increasingly important as AI systems move from generating isolated snippets towards modifying existing repositories and participating in larger development workflows. Research on LLM-assisted code understanding provides evidence that understanding existing software is itself an important AI-assisted engineering task (Nam et al., 2024). Benchmark performance should consequently be interpreted as evidence of capability under defined conditions rather than as a comprehensive measure of software engineering competence (Hou et al., 2024).

8.2 From Code Correctness to System Quality

The fundamental limitation of narrow coding benchmarks can be expressed through a simple distinction:

“Does the code work?” is not equivalent to “Is this good software?”

A generated function may pass all available unit tests while introducing a security vulnerability, increasing technical debt, violating architectural principles, reducing system performance or making future maintenance more difficult. It may also introduce unnecessary dependencies or create operational complexity that is not visible at the level of an individual function. Research evaluating the security of AI-generated code demonstrates precisely this problem: generated implementations can be functionally usable while still containing security weaknesses (Wang et al., 2024; Hamer et al., 2024).

Software quality is therefore inherently multidimensional. A meaningful evaluation should consider functional correctness alongside security, maintainability, performance, reliability, resilience, architectural consistency, developer experience and operational suitability. The systematic literature review by Hou et al. (2024) supports this broader conception of LLM-based software engineering, while studies of safety-critical code further demonstrate why correctness must be considered alongside other quality attributes when evaluating AI-generated software (Liu et al., 2024).

This broader perspective is particularly important because AI changes the economics of software production. If AI substantially reduces the cost of generating code, organisations may produce more software and make changes more rapidly. The consequences of those changes, however, may only become visible over longer periods. An evaluation framework that measures only whether code passes an immediate test may therefore systematically underestimate the long-term costs associated with poor architectural decisions, security weaknesses or difficult-to-maintain implementations.

The evaluation problem is consequently not simply whether AI can generate correct code, but whether it can contribute to the production of high-quality software over time.

8.3 Repository-Level Evaluation

One of the most important developments in AI software engineering research is the movement towards repository-level evaluation. Rather than asking an AI system to generate an isolated function from a short description, repository-level tasks assess whether the system can understand and modify an existing software project.

This represents a considerably more realistic test of engineering capability. The AI must understand existing code, dependencies, naming conventions, architectural patterns, configuration, testing frameworks, documentation and interfaces between components. In many cases, it must also infer design intentions that are not explicitly stated in the immediate task description. Research by Nam et al. (2024) is particularly relevant here because it examines the use of LLMs to assist developers with understanding existing code, highlighting that comprehension of software artefacts is an important component of AI-assisted engineering.

The challenge therefore becomes one of software comprehension, modification and integration, rather than simple code generation.

This distinction matters because professional developers rarely begin with an empty file. A substantial proportion of software engineering involves maintaining and modifying systems that already exist. Developers must determine how a proposed change interacts with previous design decisions, identify the appropriate components to modify, preserve existing behaviour and avoid introducing regressions. These broader engineering requirements are consistent with the lifecycle-wide perspective identified in the systematic review by Hou et al. (2024).

Future evaluation should therefore increasingly focus on realistic repository-level tasks such as feature implementation, bug fixing, refactoring, dependency upgrades, performance optimisation and security remediation. Such tasks provide a stronger indication of whether AI systems can operate within the complexity of real software environments.

8.4 Maintainability and Long-Term Software Quality

Maintainability represents another dimension that conventional coding benchmarks frequently overlook. Two implementations may satisfy exactly the same functional tests while producing substantially different long-term consequences for an organisation.

One implementation may be simple, readable, modular, well documented and consistent with the existing architecture. Another may duplicate functionality, introduce unnecessary abstractions, increase complexity or rely on inappropriate dependencies. Both may be judged equally successful by a functional benchmark, despite creating very different maintenance costs.

This issue becomes particularly important if AI substantially increases the volume of software that organisations can produce. The short-term reduction in implementation effort may be accompanied by a longer-term increase in the effort required to understand, modify, test and maintain the resulting code. Hou et al. (2024) identify maintainability and broader software quality as important areas for continued investigation precisely because conventional code-generation evaluations do not fully capture these consequences.

The central question therefore becomes whether AI reduces the cost of software creation while increasing the cost of software maintenance. This cannot be answered through short-term coding benchmarks alone. It requires longitudinal research examining how AI-generated software evolves over time, how frequently it requires rework, how technical debt develops and how easily subsequent developers can understand and modify it.

Maintainability should consequently be treated as an outcome in its own right rather than as an implicit consequence of functional correctness.

8.5 Security, Reliability and Resilience

Security evaluation must similarly extend beyond functional testing. Software can satisfy all of its stated functional requirements while remaining vulnerable to exploitation. AI-generated software should therefore be assessed against relevant vulnerability classes, security requirements and threat models. Wang et al. (2024), through the CodeSecEval evaluation, demonstrate that LLM-generated code can exhibit security weaknesses across a broad range of vulnerability categories. Hamer et al. (2024) likewise found vulnerabilities in both ChatGPT-generated code and Stack Overflow examples, reinforcing the importance of evaluating security independently of functional correctness.

The same principle applies to reliability and resilience. Production software must operate correctly not only under expected conditions but also when systems encounter failures, unexpected inputs or unusual operating environments. Research on safety-critical software generation reinforces the importance of evaluating generated implementations against requirements that extend beyond basic functional behaviour (Liu et al., 2024).

Evaluation should therefore consider questions such as what happens when a dependency becomes unavailable, how the application handles malformed or unexpected input, whether partial failures can be contained and whether the system can recover gracefully.

Performance under realistic workloads is also important. A solution that is functionally correct but inefficient may create unacceptable infrastructure costs or degrade the user experience when deployed at scale.

These considerations demonstrate why software evaluation should increasingly incorporate adversarial, abnormal and operational conditions, rather than testing only successful execution under normal inputs. The quality of AI-generated software should be assessed according to how it behaves when the environment is imperfect, not merely when every assumption holds.

8.6 The Human Dimension of Evaluation

AI software engineering cannot be evaluated adequately without considering the interaction between AI systems and developers. The usefulness of an AI-generated solution depends partly on whether developers can understand, review and safely incorporate it into their work.

A technically correct implementation may still impose substantial costs if developers struggle to understand its logic, identify its assumptions or determine whether it is appropriate for the surrounding system. Research on AI-assisted code understanding demonstrates the importance of evaluating how AI affects developers' ability to comprehend software (Nam et al., 2024). Similarly, qualitative research involving software professionals shows that developers use AI assistants across development and security activities while also expressing concerns about the reliability and security of AI-generated outputs (Klemmer et al., 2024).

Evaluation should therefore consider factors such as developer comprehension, review effort, cognitive load, debugging effort, trust calibration and the time required to validate generated changes. Research into LLM-based test-driven interactive code generation also demonstrates the importance of studying the interaction between generated code, testing and developer activity rather than evaluating generation as an isolated event (Fakhoury et al., 2024).

This is particularly important because raw generation speed can create a misleading impression of productivity. An AI system that produces a solution in seconds may appear highly efficient, but if a developer subsequently spends significant time checking, debugging and restructuring that solution, the net productivity benefit may be considerably smaller.

The appropriate measure is therefore total engineering effort, rather than generation time alone. This connects the evaluation problem directly to the productivity question discussed earlier: the relevant outcome is the amount of high-quality software produced per unit of overall engineering effort.

8.7 Towards a Multidimensional Evaluation Framework

The limitations of current benchmarks suggest the need for a broader evaluation framework that measures AI performance across multiple dimensions of software engineering.

Functional correctness should remain a fundamental measure because generated software must satisfy its stated requirements. However, evaluation should also determine whether the implementation is secure, maintainable, performant and reliable. It should assess whether the software conforms to the architecture of the surrounding system and whether it remains resilient when exposed to unexpected conditions.

The framework should also incorporate the developer experience. Researchers should examine whether AI reduces or increases the total effort required to complete a task, how much review is required, whether developers can understand the generated implementation and whether AI affects cognitive load and decision-making. The importance of this human dimension is supported by research examining developer interaction with AI coding assistants (Klemmer et al., 2024) and AI-assisted code understanding (Nam et al., 2024).

Finally, evaluation should consider operational suitability. Software that performs well in a benchmark but cannot be safely deployed, monitored or maintained in a production environment has limited organisational value.

A comprehensive evaluation framework can therefore be understood as assessing several interconnected dimensions. Functional correctness asks whether the software satisfies its requirements. Security asks whether it resists relevant threats and vulnerabilities. Maintainability considers whether developers can understand and modify it efficiently over time. Performance assesses whether it meets defined resource and responsiveness requirements. Reliability examines whether it behaves consistently under expected and unexpected conditions, while resilience considers whether it can tolerate and recover from failures. Architectural evaluation determines whether generated changes conform to system-level design principles. Developer-experience measures assess the human effort required to understand, review and validate the output. Explainability considers whether developers can establish why the generated implementation behaves as it does. Finally, operational evaluation determines whether the resulting software can be deployed, monitored and maintained safely within the organisation.

This broader framework changes the fundamental question of AI software engineering evaluation. Instead of asking only:

“Does the generated code pass the test?”

research should increasingly ask:

“Does the resulting software provide sustainable value within the real-world system?”

This is a considerably more demanding standard, but it is also more representative of the actual purpose of software engineering.

8.8 From Benchmark Performance to Engineering Capability

The distinction between coding performance and engineering capability is ultimately critical to interpreting the rapid progress of AI systems.

A model can demonstrate impressive performance on controlled programming benchmarks without necessarily possessing the contextual understanding required to operate safely within a production environment. Conversely, a model that performs less impressively on isolated coding tasks may still provide substantial value when integrated effectively into a broader engineering workflow that supplies repository context, automated testing, security controls and human oversight. This systems-level interpretation is consistent with the broader research synthesis provided by Hou et al. (2024).

The evaluation of AI should therefore move from measuring what the model can generate in isolation towards measuring what the human–AI system can reliably accomplish together.

This represents an important conceptual shift. The relevant system of analysis is no longer simply the model. It is the combination of the AI system, developer, repository, development tools, verification mechanisms and organisational constraints. Research into AI-assisted development already indicates that developer interaction, verification and contextual understanding are important components of the overall system (Klemmer et al., 2024; Nam et al., 2024; Fakhoury et al., 2024).

In this sense, AI software engineering evaluation should become increasingly systems-oriented. Model capability remains important, but organisational value emerges from the interaction between model capability and the engineering environment in which that capability is deployed.

8.9 Towards Real-World Evidence

Future research should consequently place greater emphasis on empirical evidence from realistic software development environments. Controlled experiments remain valuable because they allow researchers to isolate particular effects, but they should increasingly be complemented by repository-level studies, longitudinal assessments, enterprise deployments and field experiments. Hou et al. (2024) identify the need for broader empirical investigation across the software engineering lifecycle, while existing controlled studies demonstrate both the value and limitations of evaluating AI in narrower settings (Peng et al., 2023; Fakhoury et al., 2024).

Such research could examine not only whether developers complete tasks more quickly, but also whether AI-assisted teams produce fewer defects, create less technical debt, resolve vulnerabilities more rapidly, maintain software more effectively and deliver greater business value over extended periods.

Longitudinal research is particularly important because many consequences of AI-assisted development may not become visible immediately. A development team may initially experience substantial productivity gains, while the associated increase in code volume, architectural complexity or dependency usage may create costs months or years later. The productivity experiment conducted by Peng et al. (2023), for example, provides evidence of substantial short-term productivity benefits under particular experimental conditions, but such evidence does not by itself establish the long-term organisational consequences of increased AI-assisted code production.

The true impact of AI therefore cannot be established solely through short-duration experiments. It requires evidence about the lifecycle consequences of AI-assisted software production.

8.10 Evaluation as a Measure of Sustainable Software Value

The broader argument is that AI software engineering should ultimately be evaluated according to the value it creates rather than the volume of code it produces.

Code generation is an intermediate activity. The final objective of software engineering is to create systems that solve problems reliably, securely and economically over time.

This means that the most meaningful evaluation framework must connect technical outcomes with organisational outcomes. An AI system that generates more code but increases defects, security incidents, maintenance costs or developer review effort may provide little net value. Conversely, a system that produces less code but enables developers to solve complex problems more reliably may generate substantially greater value. The distinction between generation speed and broader engineering outcomes is consistent with evidence from productivity, security and developer-interaction research (Peng et al., 2023; Wang et al., 2024; Klemmer et al., 2024).

The central proposition is therefore:

AI software engineering should be evaluated as a system of sustainable software production, not as a contest of code-generation capability.

This requires evaluation to move beyond isolated functional correctness towards a multidimensional assessment of quality, security, maintainability, reliability, performance, developer experience and operational value. It also requires researchers to examine the interaction between AI capabilities and the broader engineering environment.

The transition from benchmark-oriented evaluation to systems-oriented evaluation is consequently not merely a methodological refinement. It reflects a deeper change in the nature of the technology being evaluated. As AI moves from generating isolated code fragments towards participating in repositories, workflows and increasingly autonomous development processes, the appropriate object of evaluation increasingly becomes the AI-augmented software engineering system itself.

The key question for organisations is therefore no longer simply whether AI can write code. It is whether AI, embedded within an appropriate engineering and governance environment, can help organisations produce better software with lower total cost and acceptable levels of risk. This conclusion is supported by the broader body of evidence across productivity, security, code understanding, testing and software engineering research (Peng et al., 2023; Wang et al., 2024; Nam et al., 2024; Fakhoury et al., 2024; Hou et al., 2024).

This provides the foundation for the final discussion of the paper, which considers the broader organisational, economic and governance implications of adopting AI as an increasingly embedded capability within software engineering.

9. AI and the Future Software Developer

he widespread adoption of generative AI in software engineering is unlikely to result simply in the disappearance of software developers. A more plausible outcome is the recomposition of software engineering work, in which the relative importance of different activities changes as AI systems become capable of performing an increasing proportion of routine implementation tasks (Hou et al., 2024).

AI is particularly well suited to activities that can be expressed through established programming patterns, including code completion, boilerplate generation, documentation, test generation, code transformation and aspects of code understanding (Nam et al., 2024; Fakhoury et al., 2024). Controlled evidence also demonstrates that AI assistance can reduce the time required for particular coding tasks, although the magnitude of these benefits depends on the task and development context (Peng et al., 2023).

Other activities remain considerably more dependent on context, judgement and organisational knowledge. These include requirements interpretation, architectural design, system-level reasoning, trade-off analysis, security decisions, stakeholder communication and accountability. Consequently, AI may reduce the amount of code that developers personally write while simultaneously increasing the importance of engineering expertise, judgement and oversight.

This produces an important paradox:

AI may reduce the amount of programming performed manually without reducing the need for software engineering expertise.

The future of the profession is therefore better understood as a shift in the distribution of engineering work rather than its elimination.

9.1 From Code Writer to Code Reviewer and Validator

As AI generates an increasing proportion of implementation code, developers may spend relatively less time translating specifications directly into source code and more time evaluating, validating and integrating machine-generated implementations.

The traditional relationship can be characterised as:

Developer effort → code production

The emerging relationship is increasingly:

AI effort → candidate code production

Developer effort → specification + validation + integration + judgement

This represents a significant change in the economics of engineering work. The developer's responsibility shifts from producing every implementation manually toward determining whether candidate implementations are correct, secure, maintainable and appropriate for the broader system.

This does not make programming knowledge less important. On the contrary, the ability to evaluate generated software requires an understanding of algorithms, programming languages, architecture, dependencies, testing and failure modes. Research on LLM-assisted code understanding similarly demonstrates that AI can support developers in comprehending existing software, suggesting that future workflows may involve AI supporting both the production and interpretation of code (Nam et al., 2024).

The distinction is therefore between writing less code and understanding less software. AI may reduce the former without necessarily reducing—and potentially increasing—the importance of the latter.

9.2 Expertise as a Force Multiplier

The benefits of AI are unlikely to be distributed equally across developers.

Consider two developers receiving the same AI-generated implementation. An experienced engineer may recognise an inappropriate architectural decision, an inefficient algorithm, a hidden security weakness, an unnecessary dependency or an unhandled edge case. A less experienced developer may accept the same implementation because it appears professionally written and passes a limited set of tests.

AI can therefore potentially amplify both expertise and inexperience.

This is particularly significant because the plausibility of generated code can make incorrect solutions difficult to recognise without sufficient technical knowledge. The ability to generate multiple candidate implementations is valuable only if the developer can meaningfully compare their trade-offs and identify their limitations.

The evidence on AI-assisted development therefore does not support a simple substitution model in which AI capability eliminates the need for expertise. Rather, it suggests that AI may act as a force multiplier for developers who possess the knowledge required to direct and evaluate it (Hou et al., 2024).

This has important implications for organisations and education. Investment in AI adoption should be accompanied by continued development of foundational software engineering capabilities rather than treating AI as a substitute for them.

9.3 The AI-Enabled Software Architect and Orchestrator

The future software developer may increasingly resemble an AI-enabled engineer, architect and orchestrator.

Instead of manually implementing every component, the developer may define the desired outcome, establish technical and organisational constraints, ask AI systems to explore alternative solutions, evaluate those alternatives, direct implementation and coordinate automated verification.

A typical workflow could therefore involve the developer defining the objective and constraints, an AI system proposing alternative designs or implementations, automated systems testing and analysing those proposals, and the developer resolving architectural and risk-related issues before approving deployment.

This represents a movement from implementation-centric engineering toward intent- and system-centric engineering.

The developer increasingly operates at a higher level of abstraction: rather than specifying every line of implementation directly, they specify the properties the resulting system must satisfy and orchestrate the tools required to achieve them.

This is consistent with the broader evolution identified in the literature from isolated code generation toward repository-level development, code understanding, testing and increasingly complex software engineering tasks (Hou et al., 2024; Nam et al., 2024; Fakhoury et al., 2024).

The changing distribution of work can consequently be understood as a progression from manually writing implementation code toward specifying intent, generating candidate implementations, evaluating generated components, coordinating AI and development tools, and governing the resulting system.

9.4 Verification Becomes a Core Engineering Skill

One of the most important consequences of this transition is that verification may become a central component of professional software engineering.

If AI systems produce increasingly large quantities of code, developers must be capable of determining whether that output is fit for purpose. This involves understanding requirements, constructing effective tests, interpreting failures, reviewing architectural decisions and identifying security and reliability risks.

Research demonstrates that AI-generated code can contain functional and security defects, while studies of AI-assisted development indicate that developers continue to check and evaluate AI-generated suggestions rather than treating them as inherently reliable (Wang et al., 2024; Klemmer et al., 2024).

The developer therefore becomes, in part, a validator of machine-generated software.

This may change the relative value of different engineering skills. The ability to write a routine function from scratch may become less differentiating as AI becomes better at implementation. By contrast, the ability to identify whether a proposed implementation is correct, secure, efficient and architecturally appropriate may become increasingly valuable.

In this environment, testing, debugging, code comprehension and critical review are not secondary activities. They become central mechanisms through which human expertise controls AI-generated output.

9.5 Continuous Learning and AI Literacy

The changing role of the software developer also creates a requirement for continuous professional development.

Future developers may need competence in AI-assisted programming, prompt and context engineering, AI-output evaluation, software architecture, security, testing, AI governance and agent orchestration. However, this does not imply that every software engineer must become an AI researcher.

Rather, developers need sufficient AI literacy to understand the capabilities, limitations and failure modes of the systems they use.

This includes understanding that AI output may be non-deterministic, that apparently confident answers can be incorrect, that generated code may contain security vulnerabilities and that model behaviour depends substantially on the context supplied to it (Ouyang et al., 2024; Wang et al., 2024).

Developers must therefore learn not only how to prompt AI systems effectively, but also when not to trust their output without independent verification.

The most valuable engineers may consequently be those who combine strong traditional software engineering expertise with the ability to collaborate effectively with AI systems.

9.6 Security Expertise Remains Critical

Security represents a particularly important area in which human expertise remains difficult to delegate.

AI systems can assist with vulnerability identification, secure coding suggestions, security testing and remediation. However, secure software engineering depends on understanding the broader threat model, system architecture, regulatory environment and organisational risk tolerance (Klemmer et al., 2024; Wang et al., 2024).

A generated security recommendation may therefore be technically plausible while remaining inappropriate for the particular system in which it is deployed.

This makes security expertise increasingly important as AI expands the volume of software that can be produced. The challenge is not simply to generate secure code, but to establish whether the resulting system satisfies its actual security requirements.

The future developer therefore needs to be capable of using AI as a security-assistance capability without treating it as an autonomous security authority.

9.7 The Developer as an Accountability Boundary

Regardless of how capable AI systems become, accountability for production software cannot simply be delegated to a model.

Organisations still require clearly identifiable humans and teams responsible for architectural decisions, security, quality, compliance, operational risk and business outcomes. This is particularly important in regulated, safety-critical and security-sensitive environments, where the consequences of incorrect decisions may extend beyond technical failure (Liu et al., 2024).

The principle becomes increasingly important as AI systems move toward greater autonomy. An agent may be capable of inspecting a repository, modifying files, running tests and proposing a change, but the organisation must still determine what authority the agent has, what constraints apply and who is accountable for its actions.

Human accountability is therefore not necessarily equivalent to manually performing every development task. Instead, accountability requires humans to retain authority over objectives, constraints, consequential decisions and acceptance of risk.

9.8 The Future Software Engineer

The evidence ultimately suggests that the future software engineer will not simply be a person who writes code with the assistance of an AI chatbot. The role is likely to become broader and more system-oriented.

A developer may increasingly spend time defining what a system should achieve, specifying constraints, evaluating AI-generated alternatives, validating behaviour, managing security and risk, coordinating AI agents and development tools, and taking responsibility for the resulting system.

This represents a movement from code production toward software orchestration and assurance.

The progression can be expressed as:

Manual implementation → AI-assisted implementation → AI-augmented engineering → AI-orchestrated engineering

At each stage, programming remains relevant, but the locus of human effort moves progressively upward: from writing individual instructions toward defining objectives, architecture, constraints, verification criteria and acceptable outcomes.

This does not imply that routine programming will disappear. Nor does it suggest that all developers will immediately become architects or AI orchestrators. Rather, it indicates a gradual recomposition of engineering work as AI systems become capable of performing a larger proportion of implementation and analytical tasks (Hou et al., 2024).

9.9 Overall Implication

The central implication is that AI is unlikely to make software engineers irrelevant in the near term. Instead, it is likely to change what engineering expertise is used for.

The most valuable developer may increasingly be the person who can translate organisational intent into precise technical constraints, use AI to explore and implement candidate solutions, and then determine whether the resulting system actually satisfies requirements for correctness, security, maintainability, performance and reliability.

The future software engineer is therefore best understood not as an AI operator, but as a technical decision-maker working within an AI-augmented engineering system.

The central argument of this chapter can consequently be expressed as follows:

AI may automate increasing amounts of software production, but the need for human judgement, expertise and accountability remains. The role of the software engineer is therefore likely to shift from primarily producing code toward specifying, evaluating, orchestrating and governing software systems.

This distinction is critical to understanding the broader impact of AI on the profession. The fundamental transformation may not be that machines replace developers, but that software generation becomes increasingly abundant while engineering judgement becomes comparatively scarce and valuable. The organisations most likely to benefit will therefore be those that combine AI's capacity for rapid generation with human expertise, automated assurance and clear accountability.

10. Implications for Software Engineering Organisations

The adoption of AI coding tools should not be understood simply as the procurement of another developer productivity application. The evidence reviewed throughout this paper suggests that generative AI has implications for the software engineering operating model itself. AI changes not only how code is produced, but also how software must be specified, reviewed, tested, secured, integrated, governed and maintained (Hou et al., 2024).

The organisational objective should therefore not be to maximise the volume of AI-generated code. Rather, organisations should seek to optimise the relationship between software generation, engineering assurance, security, cost, maintainability and business value. This is particularly important because productivity gains can increase the volume of software requiring validation, while non-deterministic generation and AI-assisted autonomy introduce additional requirements for provenance, governance and oversight (Ouyang et al., 2024).

A mature organisational model should consequently treat AI as one component of a broader socio-technical engineering system, combining AI capabilities, repository context, automated verification, human expertise and organisational governance.

10.1 AI Governance and Accountability

AI adoption requires explicit governance over how AI systems are used throughout the software development lifecycle. Organisations should establish policies governing which AI tools and models are approved, what information may be submitted to external services, how confidential source code and proprietary information are protected, and which categories of software development require additional human oversight.

Governance should also establish responsibility for AI-generated artefacts. The use of an AI system does not transfer accountability from the organisation or developer to the model provider. Responsibility for requirements, architecture, security, testing, deployment and operational outcomes must remain clearly assigned to human actors and organisational processes.

This becomes particularly important as AI systems move from conversational assistants toward agents capable of modifying repositories, executing tests and proposing or implementing changes autonomously. Greater autonomy requires corresponding controls over permissions, action logging, escalation and approval.

Governance should therefore be proportionate to risk. An experimental internal prototype should not necessarily be subject to the same controls as software responsible for financial transactions, critical infrastructure or sensitive personal information. Nevertheless, the underlying principle remains consistent: AI-generated software must operate within clearly defined technical, security and organisational boundaries (Klemmer et al., 2024; Hou et al., 2024).

10.2 Secure Development Practices

AI-generated code should be subject to the same security standards as human-generated code and, for higher-risk applications, may require additional controls. The fact that an implementation was produced by a sophisticated model provides no evidence that it is secure.

This is particularly important because research demonstrates that LLMs can generate and fail to adequately repair security vulnerabilities across a broad range of vulnerability classes (Wang et al., 2024). At the same time, comparative evidence indicates that AI-generated code is not necessarily less secure than code obtained from conventional developer resources: Hamer, d'Amorim and Williams (2024), for example, found fewer vulnerabilities in their ChatGPT-generated Java snippets than in the Stack Overflow examples they examined. The appropriate conclusion is therefore neither that AI-generated code is inherently insecure nor that it is inherently trustworthy.

Organisations should instead establish security controls that operate independently of the generation mechanism. These should include automated vulnerability scanning, software composition and dependency analysis, static application security testing, secret detection, threat modelling, security-focused testing and human security review for high-risk changes. Research involving software professionals also indicates that developers already use AI assistants for security-related activities while expressing concerns about the reliability and security of their outputs (Klemmer et al., 2024).

Security must consequently be embedded into the AI-assisted development lifecycle rather than added after code generation.

10.3 Automated Verification as Organisational Infrastructure

As AI reduces the marginal cost of producing software, automated verification becomes increasingly important. The productivity experiment conducted by Peng et al. (2023) demonstrates that AI assistance can substantially accelerate particular development tasks, but higher generation capacity also raises the amount of software that organisations need to validate.

Development environments should therefore integrate automated testing, compilation, type checking, static analysis, security scanning, dependency analysis, integration testing and continuous integration wherever appropriate. Generated changes should pass defined quality gates before they can enter production systems.

This creates a critical organisational principle:

The faster software can be generated, the faster it must be capable of being verified.

Testing infrastructure therefore becomes more than a quality-control mechanism. It becomes a potential strategic capability that determines how safely an organisation can exploit AI-generated development velocity. Research on AI-assisted test-driven development similarly suggests that AI can participate in test generation and interactive development, creating opportunities to strengthen the relationship between generation and verification (Fakhoury et al., 2024).

The organisational challenge is consequently to prevent AI from accelerating production while leaving verification dependent on slow, predominantly manual processes.

10.4 Developer Training and the Changing Skills Profile

AI adoption should be accompanied by substantial investment in developer education. Developers need sufficient AI literacy to understand hallucination, non-determinism, context limitations, model failure modes and the difference between plausible output and verified correctness (Ouyang et al., 2024; Hou et al., 2024).

However, AI literacy should complement rather than replace conventional software engineering expertise. Developers still require strong capabilities in programming, algorithms, architecture, testing, debugging, security, system design and requirements analysis. Indeed, the increasing ability of AI systems to produce plausible implementations may make these skills more important because developers must be capable of evaluating whether generated solutions are appropriate.

Research into developer interaction with AI also suggests that understanding and reviewing generated code remains an important part of effective use. Nam et al. (2024), for example, demonstrate the potential of LLMs to assist with code understanding, reinforcing the broader shift in which AI can support both production and comprehension rather than simply replacing manual implementation.

The most valuable skill may therefore increasingly be critical evaluation: the ability to determine what should be delegated to AI, how generated output should be constrained, and whether the resulting software actually satisfies the requirements.

10.5 Repository and Architectural Context

The effectiveness of AI coding systems depends not only on model capability but also on the quality and relevance of the context available to them.

Generating an isolated function without understanding the surrounding application can produce code that is locally plausible but inconsistent with the existing system. Professional software development typically involves modifying established repositories containing architectural conventions, dependencies, interfaces, tests, documentation and historical design decisions. AI systems must therefore increasingly operate with access to relevant repository-level context.

This includes the existing source code, repository structure, dependency information, coding standards, architectural patterns, documentation, testing frameworks and organisational policies.

The importance of this contextual dimension is consistent with the broader software-engineering literature, which identifies a movement away from isolated code generation toward more complex tasks involving code understanding, modification and repository-level engineering (Hou et al., 2024; Nam et al., 2024).

The implication is significant:

The future value of AI coding systems is likely to depend as much on context integration as on model capability.

A highly capable model operating without sufficient context may produce inferior results to a somewhat less capable model embedded within a well-engineered development environment.

10.6 Architecture as a Constraint on AI-Generated Software

As AI increases the speed at which individual components can be produced, architectural governance becomes increasingly important.

Without appropriate architectural constraints, rapid generation can lead to duplicated functionality, inconsistent interfaces, unnecessary dependencies and fragmented design decisions. These problems may not be immediately visible at the level of an individual function but can accumulate into significant technical debt at system level.

Organisations should therefore establish explicit architectural constraints covering areas such as approved technologies, service boundaries, interface standards, security patterns, data architectures, integration mechanisms and non-functional requirements.

AI can then operate within these boundaries, generating candidate implementations that conform to established architectural principles.

This represents a potential change in the role of architecture. Rather than specifying every implementation detail manually, architects may increasingly focus on defining the constraints, interfaces and principles within which human developers and AI systems operate. This is consistent with the broader shift from individual artefact production toward system-level orchestration identified across the literature on LLM-supported software engineering (Hou et al., 2024).

10.7 Measuring Outcomes Rather Than Output

AI adoption also requires organisations to reconsider how software engineering productivity is measured.

Traditional measures such as lines of code, number of completed functions or raw development speed become increasingly problematic when AI can generate large quantities of software. Such measures capture production volume, but not necessarily engineering value.

The evidence from Peng et al. (2023) is particularly relevant here. Their controlled study demonstrates productivity gains associated with Copilot under particular experimental conditions, but such findings should not be interpreted as evidence that generating more code automatically produces greater organisational value.

More meaningful organisational measures should therefore include defect rates, security vulnerabilities, review effort, rework, deployment frequency, technical debt, maintainability, incident rates, developer experience and ultimately business outcomes.

The principle can be expressed simply:

Productivity should measure valuable, validated outcomes rather than the volume of AI-generated artefacts.

An AI system that enables developers to produce twice as much code but simultaneously doubles defects, review effort or long-term maintenance costs may provide little or no net productivity benefit.

10.8 Redesigning the Software Engineering Operating Model

Taken together, these implications suggest that organisations should move beyond treating AI as an isolated coding assistant. AI should instead become integrated into an end-to-end engineering workflow:

Human intent and requirements → AI-assisted analysis and design → context-aware generation → automated testing and verification → security analysis → human review → controlled deployment → runtime monitoring → organisational learning

Such an operating model allows organisations to exploit AI's strengths in generation, transformation, information retrieval and pattern recognition while retaining independent mechanisms for quality, security and accountability.

This also changes the economics of software engineering. If AI makes code generation abundant, then the scarce resources may increasingly become engineering judgement, architectural capacity, testing infrastructure, security expertise and organisational ability to govern software at scale.

The organisational challenge is therefore not simply to increase generation capacity. It is to ensure that assurance capacity grows alongside it.

10.9 From AI Tool Adoption to AI Engineering Capability

The broader evidence suggests that successful AI adoption will depend less on selecting the most capable model in isolation and more on constructing an effective system around that model.

Such a system combines:

AI models + repository context + development tools + automated testing + security controls + observability + governance + human expertise

Each component addresses a different limitation. Repository context helps the model understand the actual system. Automated verification tests its output. Security controls identify vulnerabilities. Governance establishes acceptable use and accountability. Human expertise provides contextual judgement and handles decisions that cannot be safely delegated.

This means that the competitive advantage associated with AI may increasingly arise from the quality of the surrounding engineering system, rather than from model capability alone.

An organisation with a slightly less capable model but excellent testing, repository integration, architecture, security controls and governance may therefore achieve greater practical value than an organisation using a more capable model without these supporting capabilities.

10.10 Overall Implications for Organisations

The evidence reviewed throughout this paper therefore supports a shift from AI as a coding tool toward AI as a component of the software engineering operating model.

Organisations seeking to adopt AI effectively should develop AI-aware workflows, provide appropriate repository and architectural context, automate verification, embed security-by-design, manage models and prompts, establish meaningful human oversight, strengthen observability, train developers and evaluate outcomes rather than raw output (Fakhoury et al., 2024; Hou et al., 2024; Klemmer et al., 2024; Ouyang et al., 2024).

The strategic question should consequently not be:

“How much code can our developers generate with AI?”

It should instead be:

“How can we increase the speed and scale of software creation without reducing the reliability, security, maintainability and strategic value of the resulting systems?”

This reframes AI adoption from a procurement decision into an organisational engineering challenge.

The successful organisation will not necessarily be the one that generates the most software. It will be the one that can most effectively connect intent, AI-assisted generation, automated assurance, human judgement, deployment, monitoring and organisational learning.

In this sense, the future of software engineering is unlikely to be a choice between AI and developers. It is more accurately understood as developers, AI systems and automated assurance mechanisms operating together within an engineered socio-technical system. The central organisational capability will therefore be the ability to make AI-generated software trustworthy, governable and maintainable at scale.

11. From AI-Assisted Coding to AI-Augmented Software Engineering

a The evidence reviewed throughout this paper suggests that artificial intelligence is already capable of generating meaningful improvements across a broad range of software engineering activities, including code generation and completion, software maintenance, code understanding, documentation, testing and security-related tasks. The systematic review by Hou et al. (2024), which examined 395 studies across 85 software engineering tasks, demonstrates that the impact of large language models (LLMs) extends considerably beyond programming alone and encompasses requirements engineering, design, development, quality assurance, maintenance and software management.

The empirical evidence nevertheless indicates that these benefits are highly dependent on context. Peng et al. (2023), for example, found that developers using GitHub Copilot completed a controlled HTTP-server programming task 55.8% faster than those in the control group. This provides strong evidence that AI assistance can reduce the time required for particular programming activities, but it does not establish a universal 55.8% improvement in software engineering productivity. The experiment concerned a specific task under controlled conditions, and the authors also observed heterogeneous effects across participants (Peng et al., 2023).

The broader evidence therefore supports a more cautious conclusion: AI can increase the productive capacity of software developers, but the organisational value of that additional capacity depends on the quality of the engineering system surrounding it.

This distinction is fundamental because productivity cannot be equated with the quantity of code generated. AI can create a chain in which higher development speed leads to greater code production, which in turn creates greater requirements for testing, security analysis, review and maintenance. If the capacity to generate software grows faster than the capacity to verify and govern it, the initial productivity improvement may be partially offset by defects, rework, security vulnerabilities and technical debt. Evidence from studies of AI-generated code, security and developer interaction reinforces this need for verification rather than unconditional acceptance of generated output (Fakhoury et al., 2024; Hamer, d'Amorim and Williams, 2024; Klemmer et al., 2024; Wang et al., 2024).

The central conclusion of this paper is therefore that the impact of AI should be understood not simply in terms of how much software AI can generate, but in terms of how effectively organisations can transform increased generation capacity into reliable, secure and maintainable software.

11.1 The Verification Bottleneck

Traditional software development is constrained to a significant degree by the availability of human development capacity. Developers must understand requirements, design solutions, write code, construct tests and resolve defects. Increasing the number of developers can therefore increase an organisation's capacity to produce software, although coordination and other constraints also affect the relationship.

Generative AI changes this relationship by reducing the marginal effort associated with many implementation activities. A single developer can potentially use an AI assistant to generate, explain, transform and test substantially more code than would be practical through manual development alone. Peng et al. (2023) provide controlled evidence of this effect, while Hou et al. (2024) demonstrate that AI assistance is already being applied across a much wider range of software engineering activities than code generation alone.

As generation becomes faster, however, other stages of the software lifecycle may become relatively more constrained. The bottleneck can shift towards code review, testing, security analysis, architecture, integration, deployment and operational monitoring. This is particularly significant because generated software cannot simply be assumed to be correct or secure. Wang et al. (2024) demonstrate that LLMs can struggle with secure code generation and repair across a broad range of vulnerability types, while Hamer, d'Amorim and Williams (2024) found vulnerabilities in both ChatGPT-generated code and Stack Overflow examples. The latter finding is especially important because it demonstrates that the problem is not unique to AI: software artefacts obtained from either humans or AI require appropriate engineering assurance.

The emerging economic constraint may therefore shift from the capacity to produce code towards the capacity to establish that generated code is fit for purpose.

This can be expressed conceptually as:

Greater generation capacity → greater software output → greater assurance requirements → potential verification bottleneck

The implication is not that AI reduces the importance of software engineers. Rather, it changes where engineering capacity is most valuable. As implementation becomes cheaper, judgement, validation, architecture, security and system integration may become comparatively scarcer resources.

11.2 The Risk of Accelerating Technical Debt

One of the most important risks associated with increased generation capacity is the potential acceleration of technical debt. Technical debt arises when short-term implementation decisions create additional costs in future development and maintenance. AI may contribute to this problem if developers accept solutions that are immediately functional but unnecessarily complex, poorly integrated or difficult to maintain.

Generated code may, for example, duplicate existing functionality, introduce unnecessary abstractions, add inappropriate dependencies or depart from established architectural patterns. The problem is not necessarily that individual AI-generated changes are obviously poor. Rather, relatively small compromises can accumulate across a large number of changes.

The scale of AI-assisted development makes this possibility particularly significant. A development team that previously made a limited number of manually implemented changes may be able to generate and modify software much more rapidly with AI assistance. If architectural review and quality controls do not scale at the same rate, organisations could accumulate what might be described as high-velocity technical debt.

This conclusion is consistent with the broader literature's emphasis on the limitations and risks of LLM-based software engineering. Hou et al. (2024) identify quality, reliability and broader software engineering challenges alongside the growing range of successful LLM applications, while Liu et al. (2024) demonstrate the particular importance of careful evaluation when LLMs are applied to safety-critical software.

However, AI does not inevitably create technical debt. The outcome depends substantially on the engineering environment in which AI is deployed. Strong architectural standards, automated testing, security controls, code review and experienced developers can constrain generated output and prevent local productivity improvements from becoming long-term maintenance liabilities.

The critical variable is therefore not generation capacity alone, but the relationship between generation capacity and assurance capacity.

11.3 Testing as a Strategic Capability

If AI makes software generation increasingly abundant, automated testing becomes more than a conventional quality-assurance mechanism. It becomes a strategic capability that determines how rapidly an organisation can safely exploit AI-generated software.

Fakhoury et al. (2024) provide particularly relevant evidence. Their test-driven interactive approach integrates testing into the interaction between developers and LLMs, helping clarify user intent and evaluate generated solutions. In their user study, participants using the approach were significantly more likely to evaluate AI-generated code correctly and reported lower task-induced cognitive load. Their larger-scale evaluation also found substantial improvements in pass@1 code-generation accuracy across multiple LLMs when iterative user feedback and tests were incorporated into the process.

This suggests that testing should not be regarded simply as an activity performed after AI has generated code. Instead, testing can become part of the mechanism through which human intent constrains and improves AI generation.

The strategic implication is significant. Organisations with comprehensive automated testing, continuous integration, security analysis and deployment controls may be able to adopt AI more aggressively because they can evaluate generated changes quickly and consistently. Organisations that rely heavily on manual validation may find that increased generation speed simply transfers the bottleneck from development to quality assurance.

The ability to verify software at machine speed may therefore become almost as strategically important as the ability to generate it at machine speed.

11.4 Architecture Becomes More Important

Architecture represents another area in which the importance of human expertise may increase rather than decline.

AI systems can generate individual components rapidly, but rapid component generation does not automatically produce a coherent system. Without appropriate architectural constraints, developers may accumulate inconsistent interfaces, duplicated functionality, unnecessary dependencies and incompatible design decisions.

Architecture therefore provides the structural boundaries within which AI-generated implementation can operate safely.

These boundaries may include approved technologies, service boundaries, interface standards, data architectures, security patterns, dependency policies and non-functional requirements. Rather than specifying every implementation detail manually, architects may increasingly focus on defining the constraints, principles and interfaces within which AI systems are permitted to generate solutions.

This interpretation is consistent with the broader evidence that LLM applications span the software lifecycle rather than being restricted to isolated coding tasks (Hou et al., 2024). It is also consistent with evidence that LLMs can assist with code comprehension, which is important for understanding and working within existing architectural structures (Nam et al., 2024). In a study involving 32 participants, Nam et al. found that an LLM-based IDE interface could support code-understanding tasks and aid task completion relative to web search.

The future architectural role may therefore shift from specifying implementation towards specifying the boundaries within which increasingly autonomous implementation can occur.

11.5 From AI Tools to AI Engineering Systems

The evidence reviewed in this paper suggests that simply adopting increasingly capable AI models is unlikely to be sufficient to capture the full value of AI in software engineering.

The more consequential opportunity lies in integrating AI into a broader AI software engineering system. Such a system combines model capability with repository context, development tools, automated testing, security controls, observability, architectural governance and human oversight.

The model is therefore only one component of the overall system. A highly capable model without adequate repository context may produce inappropriate changes. A capable model without automated testing may increase verification costs and operational risk. A sophisticated AI system without security controls may introduce vulnerabilities, while a system without governance may create uncertainty over accountability and acceptable use.

Conversely, a somewhat less capable model embedded within a well-designed development environment may produce greater organisational value because it has access to appropriate context, tools and feedback mechanisms.

This systems perspective is strongly supported by Hou et al. (2024), whose systematic review demonstrates that LLM-based software engineering encompasses numerous activities and that performance depends on application methods, data, optimisation strategies and evaluation approaches rather than model capability alone.

The strategic principle is therefore:

The value of AI in software engineering depends not only on the capability of the model, but on the quality of the engineering system surrounding it.

11.6 The Central Research Finding

Taken together, the evidence suggests that AI is transforming software development from a predominantly manual production activity into an increasingly AI-augmented engineering discipline.

The transition can be understood as a progression from manual programming, through AI-assisted programming, towards AI-augmented software engineering and, ultimately, increasingly autonomous software engineering.

Each stage changes the nature of engineering control. AI-assisted programming increases the need to verify generated output. More integrated AI-augmented workflows increase the importance of architectural and security governance. Increasingly autonomous AI agents introduce additional requirements for permissions, observability, traceability, rollback and human intervention.

The problem of non-determinism illustrates why these controls matter. Ouyang et al. (2025) found substantial variability in ChatGPT's code-generation outputs across repeated requests, with considerable differences in generated solutions even for the same coding tasks. Their study also found that setting temperature to zero did not guarantee deterministic output. Such variability has implications for correctness, consistency, developer trust and reproducibility.

The resulting transformation is therefore not the elimination of software engineering, but its reconfiguration.

As AI becomes more capable of producing implementation artefacts, software engineers increasingly need to establish the conditions under which those artefacts can be generated, evaluated, integrated and maintained safely.

11.7 The Strategic Question

The strategic question for organisations should therefore not be:

“How much code can our developers generate with AI?”

A more meaningful question is:

“How can we increase the speed and scale of software creation without reducing the reliability, security, maintainability and strategic value of the resulting systems?”

Answering this question requires more than adopting better models. It requires organisations to redesign the software engineering environment around AI.

This includes AI-aware development workflows, repository-level context, automated verification, security-by-design, architectural governance, model and prompt management, human oversight, observability, developer training and outcome-based measurement.

Security is particularly important within this model. Klemmer et al. (2024), based on interviews with software professionals and analysis of developer discussions, found that practitioners already use AI assistants for security-related activities such as code generation, threat modelling and vulnerability detection, while maintaining concerns about the reliability and security of AI-generated suggestions. Their findings indicate that developers tend to check AI suggestions rather than treating them as inherently trustworthy.

Similarly, Wang et al. (2024) demonstrate that security cannot simply be assumed from functional success: their CodeSecEval work evaluates LLMs across 44 vulnerability types and highlights limitations in both secure code generation and vulnerability repair. The implication is that security must be embedded within the development process rather than added after AI-generated code has already entered the system.

The organisational objective should therefore be to move from AI as a coding tool towards AI as an integrated capability within the software engineering operating model.

11.8 The Importance of Human Expertise

The findings also challenge the assumption that increasing AI capability necessarily reduces the importance of software engineering expertise.

AI can generate implementations, explain code, produce tests and suggest solutions, but it does not remove the need for developers to understand requirements, architecture, security, system behaviour and organisational constraints. Indeed, the value of expertise may increase as AI-generated software becomes more abundant.

Nam et al. (2024) demonstrate that LLMs can support developers in understanding unfamiliar code and APIs, while Klemmer et al. (2024) show that software professionals continue to apply human scrutiny to AI-generated security-related outputs. These findings support a model in which AI expands developers' analytical and productive capacity rather than simply replacing their expertise.

This is especially important in high-risk environments. Liu et al. (2024), examining LLM-based generation of safety-critical software, illustrate why increased generation capability does not remove the need for stringent engineering controls. In such environments, correctness is only one requirement among many, and the consequences of an inappropriate implementation can be severe.

The emerging division of labour is therefore better understood as machine-scale generation combined with human-scale judgement.

11.9 From Code Production to Engineering Capacity

The evidence ultimately suggests that the most important economic effect of AI may not be that developers simply write code faster. Instead, AI changes the relative scarcity of different software engineering activities.

When implementation becomes cheaper, activities such as requirements interpretation, architecture, verification, security analysis, integration and governance may become comparatively more valuable.

This can produce a fundamental shift in the economics of software engineering:

Code generation becomes increasingly abundant, while engineering judgement and assurance become comparatively scarce.

This helps explain why productivity gains should not be evaluated through code volume alone. Peng et al. (2023) demonstrate that AI can accelerate particular programming tasks, but Fakhoury et al. (2024), Klemmer et al. (2024), Wang et al. (2024) and Ouyang et al. (2025) collectively illustrate why generation speed must be considered alongside verification, security, human judgement and variability.

The organisation that benefits most from AI may therefore not be the one that generates the greatest amount of code. It may be the organisation that can generate, evaluate, integrate and govern software most effectively as a complete system.

11.10 Overall Implication

The evidence ultimately points towards a more nuanced understanding of AI's impact on software engineering.

AI is neither simply a productivity tool nor an autonomous replacement for developers. It is becoming a new engineering capability that changes the relationship between human expertise, software generation, verification and organisational governance. The systematic literature reviewed by Hou et al. (2024) supports this broader interpretation, showing that LLM applications already span a wide range of software engineering activities and that important research gaps remain around their limitations, evaluation and organisational use.

The most significant transformation may therefore not be that AI writes more code. It may be that AI changes the economics of software production sufficiently that generation becomes abundant while judgement, verification and governance become comparatively scarce.

Organisations that recognise this shift can design their engineering environments accordingly. Rather than maximising AI-generated output, they can optimise the entire lifecycle:

Intent → AI-assisted analysis and design → generation → automated verification → human judgement → controlled deployment → monitoring → organisational learning

Within this model, AI is neither an autonomous replacement for the developer nor merely a faster version of an existing coding tool. It becomes one component of a broader socio-technical software production system in which humans, AI models, development tools and automated assurance mechanisms work together.

The research therefore supports a shift from thinking about AI primarily as a code-generation technology towards understanding it as an engineering-system capability. Its value depends on the interaction between model capability, developer expertise, repository context, testing infrastructure, security controls, architectural governance and organisational processes.

The broader conclusion of this paper is consequently that the successful adoption of AI in software engineering will depend less on whether AI can write code—it clearly can—and more on whether organisations can build the technical, organisational and governance infrastructure required to make AI-generated software trustworthy at scale.

In this sense, the future of software engineering is unlikely to be “AI versus developers.” A more defensible interpretation is developers, AI systems and automated assurance mechanisms operating together within an engineered socio-technical system.

The strategic advantage will therefore belong not simply to organisations with access to the most capable models, but to those capable of building the most effective system around them.

12. Conclusion

The rapid development of AI-assisted software engineering represents a significant change in how software can be produced, understood and maintained. Large language models can already perform a wide range of activities across the software lifecycle, including code generation, code explanation, testing, debugging, documentation and vulnerability analysis. The evidence examined in this paper nevertheless suggests that the significance of these capabilities cannot be understood simply in terms of how much code AI can generate or how quickly it can complete isolated programming tasks.

The central argument of this paper is that AI changes the economics and distribution of software engineering work without eliminating the fundamental requirements of software engineering itself.

AI can substantially reduce the cost of routine implementation. However, lower generation costs do not automatically translate into higher overall productivity. The relevant outcome is the amount of high-quality software produced per unit of total engineering effort. If AI-generated code requires substantial review, debugging, security analysis, architectural correction or long-term maintenance, the apparent productivity gain may be considerably smaller than raw generation speed suggests.

This distinction is particularly important when evaluating AI systems. Conventional coding benchmarks such as HumanEval and MBPP provide useful evidence of functional code-generation capability, but they do not capture the full complexity of professional software engineering. Production software operates within repositories, architectures, dependency networks, organisational processes and security constraints. It must remain maintainable, reliable, performant and operationally suitable over time.

Evaluation should therefore move from code-generation capability towards engineering capability.

A more meaningful assessment of AI-assisted software development must consider functional correctness alongside security, maintainability, performance, reliability, architectural consistency, developer experience and operational suitability. It should also measure the total human effort required to specify, review, validate, debug and maintain AI-generated software. The appropriate object of evaluation is consequently not the model in isolation, but the human–AI engineering system in which the model operates.

The security analysis reinforces this conclusion. AI-generated code is neither inherently insecure nor inherently trustworthy. Models can generate vulnerable implementations, overlook security weaknesses and reproduce insecure patterns, but evidence also suggests that AI-generated code can sometimes compare favourably with commonly used human-generated alternatives. The relevant organisational question is therefore not whether AI is universally more or less secure than human developers. It is whether its outputs can be subjected to sufficiently strong requirements, independent verification, security controls and human oversight to satisfy the security requirements of the system.

This creates an important principle for AI-assisted development:

AI should generate candidate implementations; engineering controls should determine whether those implementations are acceptable.

Security therefore becomes a property not only of the generated code but of the development system surrounding it. Threat modelling, static analysis, dependency scanning, security testing, code review and runtime monitoring provide independent assurance mechanisms that prevent the AI system from becoming the final authority on security.

The same principle applies to the problem of non-determinism. Large language models introduce a probabilistic generation process into a discipline that has traditionally relied heavily on reproducible artefacts and controlled configuration. Different generations may produce different implementations even when the requested functionality remains unchanged. These differences can affect not only source code but also performance, maintainability, security and architectural suitability.

The appropriate response is not necessarily to demand absolute determinism. Software engineering has always permitted multiple valid implementations of the same requirement. Instead, organisations should seek controlled variability. Model versions, prompts, relevant context, generated artefacts, validation results and human modifications should be traceable where the consequences of the software justify such controls.

This leads to a broader principle:

Probabilistic generation requires deterministic assurance.

Automated tests, static analysis, security controls, configuration management and human approval can provide stable boundaries around variable AI generation. As AI systems become more autonomous, these controls become increasingly important because the number and significance of actions performed by the system also increase.

The changing role of the software developer follows directly from this transformation. The evidence does not suggest that AI will simply eliminate software engineering work. Instead, it is likely to recompose that work. Developers may spend less time manually producing routine implementation code and more time defining objectives, establishing constraints, evaluating alternatives, understanding system context, validating generated output, managing risk and making architectural decisions.

This does not reduce the importance of software engineering expertise. In many cases, it increases it.

An AI system can generate several plausible implementations, but determining which implementation is appropriate requires knowledge of architecture, security, performance, organisational priorities and long-term maintenance. Experienced developers may therefore gain greater leverage from AI because they are better positioned to specify problems, recognise inappropriate solutions and evaluate trade-offs.

The emerging developer is consequently better understood not simply as a code writer, but as an orchestrator and technical decision-maker who coordinates AI systems with repositories, development tools, verification mechanisms and organisational constraints.

This also changes the nature of human oversight. As AI systems move from conversational assistants towards development agents capable of inspecting repositories, modifying files, executing tests and preparing changes, continuous human approval of every action may become impractical. A more scalable model is human-on-the-loop governance, in which AI operates within defined permissions and automated controls while humans retain authority over consequential decisions.

However, increasing autonomy does not imply decreasing accountability.

Autonomy may be delegated; accountability may not.

Organisations remain responsible for the software they deploy, regardless of whether particular components were written by humans or generated by AI. This requires clear governance over approved tools, permitted data, model usage, review requirements, security controls, deployment authority and responsibility for resulting systems.

The organisational implications are therefore substantial. AI adoption should not be treated merely as the procurement of another developer productivity tool. It represents a change to the software engineering operating model. Organisations will need to integrate AI governance with existing practices for security, architecture, quality assurance, configuration management, compliance and risk management.

The central organisational objective should not be to maximise AI-generated code. It should be to establish an environment in which AI can increase the capacity of engineering teams without allowing generation speed to exceed the organisation's ability to verify, secure and maintain what is produced.

This leads to the broader conclusion of the paper:

The future of AI-assisted software engineering is not primarily a transition from human-written code to AI-written code. It is a transition towards a socio-technical system in which human judgement, machine-scale generation and deterministic engineering assurance operate together.

The most important competitive advantage may therefore not come from organisations that generate the greatest volume of AI-written software. It may come from organisations that are best able to integrate AI into disciplined engineering processes. Such organisations can use AI to accelerate routine work while preserving strong architecture, security, testing, maintainability and accountability.

The ultimate measure of successful AI adoption is consequently not the quantity of code produced, the number of tasks completed or the speed of individual generations. It is whether organisations can use AI to produce better software, more effectively, at acceptable levels of cost and risk.

AI-assisted software engineering should therefore be understood as an evolution of the software engineering system rather than simply an improvement in code-generation technology. As models become more capable and autonomous, the critical question will increasingly be not “Can AI write the software?” but:

“Can the organisation create the engineering, verification and governance environment necessary to ensure that whatever AI produces is valuable, secure, maintainable and accountable?”

The answer to that question will ultimately determine whether the rapid expansion of AI capabilities translates into sustainable improvements in software engineering productivity and organisational value.

References

Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S. and Lahiri, S. (2024) ‘LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation’, IEEE Transactions on Software Engineering, 50, pp. 2254–2268.

Hamer, S., d'Amorim, M. and Williams, L. (2024) ‘Just another copy and paste? Comparing the security vulnerabilities of ChatGPT generated code and StackOverflow answers’.

Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J. and Wang, H. (2024) ‘Large Language Models for Software Engineering: A Systematic Literature Review’, ACM Transactions on Software Engineering and Methodology, 33(8), Article 220. doi: 10.1145/3695988.

Klemmer, J.H., Horstmann, S.A., Patnaik, N., Ludden, C., Burton Jr., C., Powers, C., Massacci, F., Rahman, A., Votipka, D., Richter Lipford, H., Rashid, A., Naiakshina, A. and Fahl, S. (2024) ‘Using AI Assistants in Software Development: A Qualitative Study on Security Practices and Concerns’.

Liu, M., Wang, J., Lin, T., Ma, Q., Fang, Z. and Wu, Y. (2024) ‘An Empirical Study of the Code Generation of Safety-Critical Software Using LLMs’, Applied Sciences, 14(3), 1046. doi: 10.3390/app14031046.

Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B. and Myers, B.A. (2024) ‘Using an LLM to Help With Code Understanding’, Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE 2024)

Ouyang, S., Zhang, J.M., Harman, M. and Wang, M. (2024) ‘An Empirical Study of the Non-determinism of ChatGPT in Code Generation’, ACM Transactions on Software Engineering and Methodology. doi: 10.1145/3697010.

Peng, S., Kalliamvakou, E., Cihon, P. and Demirer, M. (2023) ‘The Impact of AI on Developer Productivity: Evidence from GitHub Copilot’.

Wang, J., Luo, X., Cao, L., He, H., Huang, H., Xie, J., Jatowt, A. and Cai, Y. (2024) ‘Is Your AI-Generated Code Really Safe? Evaluating Large Language Models on Secure Code Generation with CodeSecEval’.

Contact

Reach out via email for inquiries.

Email

Subscribe to newsletter

info@grcadvisory.ch

© 2025. All rights reserved.