부산시청 도서요약
국내도서 요약
국내도서 프리뷰 해외도서 프리뷰
글로벌 트렌드 미디어 브리핑스
   글로벌 트렌드 

책표지






    • Medical AI Can No Longer Pass on Accuracy Alone

    - The expansion of regulatory AI guidelines is making validation responsibility and explainability new conditions for market entry

    The competitive standards for medical AI are changing rapidly. Simply presenting high accuracy is no longer enough to demonstrate that a technology is safe and trustworthy for patients. Regulatory authorities have begun examining the quality and representativeness of training data, performance in clinical environments, model change histories, and the ways in which medical professionals intervene. The core capabilities required of medical AI companies are also expanding beyond algorithm development to include regulatory competence combining validation, documentation, monitoring, and accountability management.

    [Key Message]
    * Competition in medical AI is shifting from accuracy-driven performance to clinical evidence. High performance figures alone are no longer sufficient; companies must demonstrate that safety and effectiveness are maintained across diverse patient populations and real-world clinical environments.

    * Responsibility for validating medical AI is expanding beyond developers to hospitals and healthcare professionals. Developers must clearly define the technology?셲 limitations and intended use, while healthcare institutions must continuously assess its suitability and performance in clinical practice.

    * Explainability is not about exposing every internal mechanism of an algorithm but about providing the information needed for safe decisions. Healthcare professionals and patients must be able to understand the AI system?셲 purpose, scope of use, performance, limitations, and potential for error.

    * The safety of medical AI is maintained through continuous monitoring rather than a one-time authorization. Organizations must track performance deterioration caused by changing data and model updates and be prepared to revalidate the system or restrict its use when problems arise.

    * Regulatory capability is becoming a new source of competitive advantage for medical AI companies. Companies that systematically manage data, clinical validation, model changes, documentation, and post-market monitoring will be better positioned to earn market trust and expand globally.

    ***

    From Performance Competition to Evidence Competition
    In the early stages of the medical AI industry, competition focused on how high a level of accuracy a system could achieve. Accuracy in detecting lesions in medical images, sensitivity and specificity in predicting disease risk, and the speed of processing data faster than medical professionals were representative indicators used to demonstrate technological capability. Some companies promoted the potential of medical AI by emphasizing results showing that their systems had achieved performance comparable to or better than that of specialists in particular tests.

    In clinical practice, however, accuracy is a far more complicated figure than it may appear. High performance on test data secured during development does not guarantee that the same results will be reproduced across all hospitals and patient populations. If the age, sex, race, and disease severity of the patients used in training differ from those in the actual use environment, performance may decline. Differences in medical imaging equipment manufacturers and scanning conditions can also alter the characteristics of input data. Even for the same disease, hospitals may use different diagnostic standards and clinical procedures, which means that AI outputs may be used differently in practice.

    The way regulatory authorities view medical AI is also changing to reflect this reality. The central question is no longer what percentage of accuracy a model achieved on test data. Regulators now examine the purpose for which it was developed, the patient populations in which it was validated, the environments in which its performance may deteriorate, and the level of risk that errors could create for patients. The conditions and evidence behind a performance figure have become more important than the figure itself.

    The U.S. Food and Drug Administration is strengthening its approach to managing AI-enabled medical device software across the entire product lifecycle rather than evaluating it only during development and authorization. Developers are expected to explain not only the model architecture and training process, but also how training, validation, and test datasets were separated, whether those datasets adequately represent the intended users, and how the limitations and risks of the model will be managed. Plans for monitoring performance and reporting problems after the product enters the market are also becoming important areas of review.

    This change shows that competition in medical AI is shifting from performance competition to evidence competition. The ability to develop a sophisticated algorithm is no longer enough. Companies must demonstrate, through reproducible evidence, that the algorithm operates safely and effectively across diverse patients and clinical environments.

    Companies must therefore design their validation strategies from the earliest stages of development. It is difficult to secure the level of evidence required by regulators if a company completes product development first and only then begins collecting materials for authorization. The intended patient population, the risk the product is designed to reduce, and the position the AI will occupy within the medical decision-making process must all be defined in advance. Data should then be collected in accordance with those objectives, and differences in performance and potential failure modes across patient groups should be analyzed.

    High accuracy remains important in the medical AI market. However, accuracy is only the starting point for demonstrating a product?셲 value. A system can be recognized as a trustworthy medical technology only when the process by which its performance was achieved, the scope within which it can be used, its clinical effectiveness, and its risks can all be demonstrated together.

    Validation Responsibility Is Expanding Beyond Developers
    Medical AI does not operate as an independent clinician. It connects with hospital information systems, influences the judgments of medical professionals, and becomes part of patient testing and treatment processes. Even the same product may produce different outcomes depending on the hospital in which it is used, the person using it, and the way it is incorporated into practice. This is why the safety of medical AI cannot be viewed solely as a matter of the algorithm itself.

    Under traditional medical device regulation, manufacturers were primarily responsible for demonstrating the safety and effectiveness of their products. Developers and manufacturers still bear central responsibility in medical AI, but the roles of hospitals and healthcare professionals inevitably become greater during actual operation. If a hospital arbitrarily expands the intended use of a product or applies it to patient groups that were not included in validation, the safety established at the time of authorization may no longer be maintained. New risks may also arise if medical professionals treat AI outputs as absolute conclusions or, conversely, repeatedly ignore necessary warnings.

    The validation of medical AI therefore cannot end with a single premarket test conducted by the manufacturer. Hospitals adopting the technology must also determine whether it is suitable for their patient populations, equipment, and workflows. They must continuously assess whether the system actually improves the quality of care compared with existing methods, whether errors are concentrated in particular patient groups, and whether medical professionals correctly understand the outputs.

    The scope of responsibility is also expanding to data providers and external technology companies. The development of medical AI may involve a complex combination of clinical data generated by hospitals, externally purchased datasets, cloud infrastructure, and general-purpose AI models. If the source of data is unclear or labeling standards are inconsistent, the performance and fairness of the model are difficult to trust. When a general-purpose model is modified for medical use, organizations must also assess how changes and updates to the original model could affect medical functions.

    In such a structure, responsibility cannot be assigned to a single company or department. Development teams must explain the technical characteristics and limitations of the model, while clinical teams must assess its meaning in actual care. Quality management departments must record changes and errors, and regulatory teams must verify that submission materials and operational procedures satisfy regulatory requirements. Within hospitals, clinicians, information technology personnel, medical device management departments, and ethics and legal teams must jointly establish standards for adoption and operation.

    The boundaries of responsibility are becoming even more complex as generative AI enters healthcare. An AI system that summarizes medical records or drafts patient information may appear less risky than a diagnostic AI system. However, if it presents incorrect information in fluent language or omits important medical details, it can still influence clinical care. Even when the principle is established that medical professionals must review AI-generated output, heavy workloads may reduce that review to a merely formal procedure.

    The purpose of a medical AI accountability framework is not limited to deciding who is liable after a problem occurs. More importantly, it should clearly define what each participant must do to identify and control risks in advance. Developers must specify the intended scope and limitations of the product. Hospitals must ensure that actual use remains consistent with the authorized purpose. Medical professionals must evaluate AI outputs within the clinical context and be able to report suspicious results.

    The expansion of validation responsibility in medical AI does not mean that responsibility becomes so widely distributed that no one is accountable. It means that responsible parties and verification procedures must be defined more clearly at every stage, from development to use. As technology becomes more deeply integrated into healthcare systems, safety will be secured not through a one-time approval but through a management structure linking multiple participants.

    Explainability Is Clinical Information, Not a Technical Explanation
    Explainability is one of the most frequently used terms in medical AI regulation. Yet there are still differing interpretations of what explainability actually means. Some believe that AI should disclose every computational step that led to a conclusion, while others argue that it is sufficient to provide evidence that medical professionals can understand.

    Fully explaining the internal operation of a complex deep learning model is not easy in practice. Because millions or more parameters interact to generate an output, it is difficult to summarize the cause of a particular result in one or two sentences. Technologies can visualize internal calculations or indicate the importance of specific inputs, but this does not necessarily amount to a clinically valid explanation. Even if a model shows that it focused on a particular region of an image, that region must still be evaluated separately to determine whether it constitutes medically appropriate evidence.

    The explanation required in medical AI should focus less on revealing every internal detail of the model and more on providing the information users need to make safe judgments. Medical professionals must know which patients and diseases the AI was developed for, the situations in which it should not be used, and the input conditions under which its performance may decline. It must also be clear whether the output represents a definitive diagnosis or a reference opinion, and whether additional confirmation by a medical professional is required.

    The information patients need may differ from the information required by clinicians. Patients may want to know whether AI was used in their care, how much influence its judgment had on a treatment decision, and how they can raise an objection if an error occurs. Developers tend to focus on explaining the structure and training method of a model, but information about patient rights and the treatment process may be more important to patients.

    The transparency principles jointly presented by the U.S. Food and Drug Administration, Health Canada, and the United Kingdom?셲 Medicines and Healthcare products Regulatory Agency also emphasize delivering information at the appropriate time and in an appropriate form to users, including medical professionals, patients, and healthcare institutions, rather than merely disclosing it. Information affecting patient safety and decision-making, such as the product?셲 purpose, performance, limitations, data characteristics, updates, and potential errors, must be provided in a way that each audience can understand.

    Explainability is also not the same as increasing the volume of documentation. Even if hundreds of pages of technical materials are provided, it cannot be considered effective explanation if medical professionals cannot quickly locate the information they need. Conversely, displaying too many warnings on the screen may create alert fatigue, causing users to ignore repeated notices. The user experience must be designed so that necessary information is delivered at the moment when an actual decision is being made.

    For example, when a diagnostic imaging AI indicates the probability of an abnormality, it is more useful to provide not only a risk score but also an explanation of what the score was designed to measure, how the result changes depending on the threshold, and which patient populations have not been sufficiently validated. Medical professionals must be able to reject the AI?셲 recommendation, and there should also be a structure for recording that decision and using it to improve future performance.

    Explainability is a mechanism that protects not only patient safety but also the responsibility of healthcare professionals. It is not appropriate to place all decision-making responsibility on clinicians when they have not been provided with sufficient information about the conditions under which the AI operates or the level of reliability it offers. Companies supplying the product must provide information that is clear enough for medical professionals to make reasonable judgments.

    Some argue that explainability may hinder innovation in medical AI. However, not every technology that cannot be fully explained is necessarily dangerous, and a technology is not necessarily safe simply because it can be explained. What matters is providing a level and form of explanation appropriate to the intended use and level of risk. In medical AI, explainability is not a demand to simplify algorithms. It is the process of translating the performance and limitations of technology into information needed for clinical decision-making.

    Continuous Monitoring Is More Important Than a One-Time Authorization
    Traditional medical devices often remained functionally unchanged after authorization. AI software, however, can change in performance as data and algorithms change. A model may be retrained with new data or improved with additional functions, and changes in operating systems or hospital systems can also affect its results.

    The capacity for learning and improvement, often regarded as one of AI?셲 greatest strengths, therefore creates a new regulatory challenge. If the model used in practice differs from the model that was originally authorized, the earlier validation results may no longer apply. Yet requiring a complete reauthorization process for every minor change could excessively delay technological improvement.

    This is why the U.S. Food and Drug Administration has developed the concept of a predetermined change control plan. Developers may specify in advance which parts of a product may be changed after authorization, how those changes could affect performance and risk, and how the modified model will be validated and managed. If the regulatory authority considers the plan appropriate, the company may improve the model within the defined scope while continuing to manage safety and effectiveness.

    However, changes that could not be anticipated in advance may also occur. If a new disease spreads in a particular region, or patient lifestyles and healthcare environments change, the distribution of input data may shift. The introduction of new imaging equipment or changes in testing protocols can also expose a model to unfamiliar data. Data drift and model drift may emerge as performance that was stable during training gradually deteriorates over time.

    Post-market monitoring is therefore essential for medical AI. Organizations must examine not only overall performance but also whether error rates differ according to sex, age, race, region, and disease severity. In addition to user complaints and incident reports, the frequency with which medical professionals override AI recommendations and the circumstances in which use is discontinued may also provide important signals.

    Procedures must also be established for responding when monitoring identifies a problem. A simple software error may be resolved through an update, but if a particular patient group is consistently disadvantaged, the scope of use may need to be restricted or the model may need to undergo renewed validation. If a serious risk to patient safety is anticipated, it may be necessary to suspend use of the product temporarily.

    European Union AI regulation also treats many AI systems used for medical purposes as high-risk systems and identifies risk management, data governance, technical documentation, record keeping, user information, human oversight, and post-market monitoring as important requirements. The AI Act entered into force in August 2024, and a structure is emerging in which medical device regulation and AI regulation apply together to high-risk medical AI.

    Continuous monitoring also changes the way companies manage quality. Organizations must move away from a linear development process in which the development team completes the model and then hands it over to regulatory personnel. A circular structure is required in which data and user experience collected after the product enters the market are fed back into development and validation.

    Hospitals likewise cannot remain passive consumers that simply purchase and install AI systems. They must maintain internal inventories of AI products, manage versions and update histories, and collect information on performance anomalies and user feedback. The more important the medical decisions supported by a system, the greater the need for periodic reassessment and clearly defined criteria for discontinuing use.

    Authorization in the age of medical AI is not a permanent guarantee of safety. It is closer to a starting point confirming that safety and effectiveness were demonstrated at a particular time and under particular conditions. Because technology, data, and clinical environments continue to change, trust must also be continuously verified and renewed.

    Regulatory Capability Is Becoming a Competitive Advantage in Medical AI
    Stronger regulation may create a substantial burden for medical AI companies. Securing clinical evidence across diverse patient groups, managing the sources and quality of data, and documenting development processes and change histories require considerable time and expense. For smaller startups, regulatory costs may feel like a greater barrier than the cost of technology development itself.

    Nevertheless, viewing regulation only as an obstacle fails to reflect the characteristics of the medical AI market. Medical technologies directly affect patient health and survival. If an inadequately validated product spreads rapidly and later causes harm, the result may be not only patient injury but also a collapse of trust in hospitals and the industry as a whole. In a market that has lost trust, even excellent technologies will struggle to gain adoption.

    The competitiveness of medical AI companies is therefore unlikely to be determined by algorithmic accuracy alone. The ability to trace which data were used, validate performance across patient groups, control model changes, explain limitations to healthcare professionals and patients, and detect and respond to problems after market entry will all be evaluated together.

    The gap may also widen between companies that treat regulatory compliance as an administrative task after development and those that incorporate it into their development strategies. When documentation is prepared only after a product has been completed, the evidence behind data selection and development decisions may no longer be available. In contrast, companies that apply quality management and validation standards from the beginning can accumulate regulatory evidence naturally throughout the development process. Problems can also be identified and corrected earlier, reducing development costs and market entry risks over the long term.

    Regulatory capability also becomes a valuable asset in international expansion. The United States, European Union, United Kingdom, Canada, and other jurisdictions differ in their detailed requirements, but they share a common emphasis on lifecycle management, high-quality data, human oversight, transparency, and post-market monitoring. Even if authorization documents from one market cannot be submitted unchanged in another, companies that build quality systems around international principles can respond more quickly to the requirements of multiple countries.

    The criteria used by healthcare institutions to select products may also change. Functions, price, and accuracy have traditionally been major evaluation factors, but hospitals may increasingly focus on how transparently developers provide performance data, whether safety can be maintained after updates, and whether the company can respond quickly when an incident occurs. Insurers and investors may likewise regard regulatory risk and accountability structures as central indicators of corporate sustainability.

    Regulatory authorities must also avoid remaining merely as institutions that block innovation. They should differentiate requirements according to the level of technological risk and provide clear guidance so that companies can predict the evidence and documentation they need to prepare. If different countries apply excessively divergent standards to the same function, unnecessary costs will arise. This is why international regulatory alignment and shared principles are important.

    The next stage of competition in medical AI is not limited to building larger and more complex models. Companies must be able to demonstrate who can safely use the technology, in which environments, and within what boundaries. They must also possess the operational capability to continuously identify performance deterioration and errors and to modify or restrict use when problems arise.

    The expansion of regulatory AI guidelines is not simply an administrative change requiring the medical AI industry to produce more documents. It is a transformation that reorganizes technology-centered development culture around patient safety and clinical accountability. It may create additional burdens for companies, but it also provides a standard for distinguishing trustworthy products from those that are not.

    For medical AI to become an established part of healthcare, it needs more than high accuracy. Data and validation processes must be transparent, medical professionals must be able to understand the limitations of the technology, and patients must be able to know what role AI plays in their care. Systems must also be established to trace causes and respond responsibly when problems occur.

    Accuracy demonstrates the potential of medical AI. Validation, explanation, monitoring, and accountability allow that potential to become patient trust. The companies that lead the medical AI market in the future will not be those that present the most impressive technology first, but those that can demonstrate the safety and value of their technology for the longest period of time.

    Reference
    U.S. Food and Drug Administration, January 2025, Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations.
    U.S. Food and Drug Administration, June 2024, Transparency for Machine Learning-Enabled Medical Devices: Guiding Principles.
    U.S. Food and Drug Administration, August 2025, Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions.
    European Union, 2024, Regulation (EU) 2024/1689 Laying Down Harmonised Rules on Artificial Intelligence.
    European Commission, 2025, Interplay Between the Medical Devices Regulation, In Vitro Diagnostic Medical Devices Regulation and the Artificial Intelligence Act.