How AI and NLP are transforming the 80% of clinical data that EHRs cannot read
Approximately 80 per cent of healthcare data is unstructured, meaning it exists in formats that electronic health record systems and analytics platforms cannot directly read or use, including clinical notes, discharge summaries, radiology reports, and scanned documents. This unstructured data contains the richest clinical information in the healthcare system, yet it remains largely inaccessible, producing a data crisis that limits clinical decision-making, inflates administrative costs, and prevents the population health insights that structured data alone cannot deliver.
Quick Summary
In this article, you will learn what the unstructured data crisis in healthcare looks like, why it matters clinically and operationally, and how AI, NLP, and automation are solving it.
- What unstructured data is and why it makes up the majority of clinical records
- The specific challenges it creates: data silos, extraction difficulty, and compliance risk
- How structured and unstructured data differ, and why structured data alone is insufficient
- How AI and NLP convert unstructured clinical content into actionable structured insights
- Real-world use cases and best practices for managing unstructured healthcare data
What Is Unstructured Data in Healthcare?
Definition and Key Characteristics
Unstructured data in healthcare refers to any clinical or administrative information that does not reside in a predefined, machine-readable schema. It is not stored in rows and columns, does not carry a standard code or field label that a database query can directly retrieve, and cannot be aggregated or analysed by standard reporting tools without a processing step that converts it into a structured form. Unstructured data is characterised by variability in format, length, and vocabulary: every clinician’s note is different, every radiology report follows a different template, and every referral letter contains the information the referring physician chose to include, in the order they chose to present it.
The defining feature of unstructured clinical data is that it is extraordinarily rich in clinical content. A physician’s consultation note contains the patient’s reason for attendance, their reported symptoms, the relevant past history, the clinical findings, the physician’s diagnostic reasoning, the management plan, and the agreed follow-up, all expressed in the nuanced, contextual language of clinical practice. None of this information exists as a structured field that can be queried, aggregated, or used to trigger an automated workflow without first being extracted and coded.
Examples of Unstructured Clinical Data
- Clinical consultation notes: free-text documentation of the encounter, including history, examination findings, assessment, and plan
- Discharge summaries: narrative accounts of inpatient admissions, diagnoses, treatments, and discharge instructions
- Radiology reports: unformatted narrative descriptions of imaging findings from radiologists
- Pathology reports: free-text descriptions of tissue examination findings and diagnostic conclusions
- Referral letters: narrative communications between clinicians describing the patient’s condition and the reason for referral
- Operative notes: detailed free-text accounts of surgical procedures and intraoperative findings
- Nursing notes: clinical observations, care activities, and patient responses documented in narrative form
- Patient-reported history: verbatim or transcribed patient narratives from intake forms, telephone triage, and telemedicine encounters
- Scanned paper documents: historical records, consent forms, insurance cards, and external documents processed through OCR
- Medical imaging files: DICOM images from X-ray, CT, MRI, and ultrasound, which require AI interpretation to extract clinical findings
Why Unstructured Data Is a Growing Crisis in Healthcare
Explosion of Clinical Data Volume
The volume of clinical data generated per patient encounter has grown dramatically over the past two decades, driven by the expansion of EHR adoption, the proliferation of diagnostic tests, the growth of telemedicine generating new documentation streams, and the increasing complexity of patients with multiple chronic conditions requiring comprehensive documentation. Industry estimates consistently find that between 75 and 80 percent of all data in healthcare systems is unstructured, and this proportion is growing as ambient documentation tools, patient messaging platforms, and remote monitoring devices generate new unstructured data streams that compound the existing backlog.
Limitations of Traditional Data Systems
Traditional EHR and clinical data systems are designed to store and query structured data, including coded diagnoses, laboratory values, medication orders, and vital signs. They have limited capability to process, search, or analyse free-text content at scale. A query for all patients with a particular diagnosis finds the patients whose diagnosis was recorded as a structured ICD-10 code but misses the patients whose diagnosis exists only in a clinical note. A population health report based on structured fields alone reflects an incomplete and systematically biased picture of the patient population, because the richest clinical information, in the unstructured notes, is invisible to the reporting tool.
Impact on Clinical Decision-Making
When a clinician sees a patient whose complete clinical history is distributed across multiple unstructured documents in different systems, they are making clinical decisions on the basis of whatever information they can find and read in the time available. A referral letter containing a critical allergy, a discharge summary documenting a recent hospitalisation, or a previous specialist’s note describing a treatment failure all represent clinically significant information that may affect the management decision. If that information is in an unstructured document that is not indexed, not easily retrievable, and not integrated with the clinician’s view of the patient, the clinical decision is made without it, with the consequences that entails.
Challenges of Managing Unstructured Data in Healthcare
Data Silos and Fragmentation
Unstructured clinical data is particularly susceptible to siloing because it is stored in formats that are not interoperable between systems. A clinical note created in one EHR system may not be readable by another system even when both systems support FHIR, because the note is stored as an unstructured text attachment rather than as FHIR resource fields. Radiology reports live in the PACS system. Pathology reports are in the laboratory information system. Referral letters are in the document management system or arrive as PDF email attachments. Scanned historical records are in a separate archive. No single view of the patient aggregates all of this content, and no standard query can retrieve it.
Difficulty in Data Extraction and Analysis
Extracting meaningful clinical information from unstructured text requires capabilities that most healthcare organisations do not have readily available. Natural language processing models must be trained on large corpora of clinical text to understand medical vocabulary, abbreviations, and clinical context. Named entity recognition models must identify the specific entities, diagnoses, medications, procedures, and values, within highly variable free-text formats. Negation detection must distinguish between conditions the patient has and conditions that have been ruled out, denied, or mentioned only as a differential. These capabilities require specialist AI and NLP engineering that is not a standard component of EHR or clinical systems.
Murphi’s EHR integration platform automates the extraction and structuring of unstructured clinical data from connected EHR systems, applying clinical NLP to convert free-text content into coded, standardised data that downstream systems can use directly.
Compliance and Security Concerns
Unstructured data is more difficult to govern than structured data. Identifying and protecting personally identifiable information (PII) and protected health information (PHI) in free-text documents requires NLP-based PII detection, because the same information cannot be identified through field-level tagging as it can in a structured database. Retention and deletion obligations under HIPAA and applicable state privacy laws must be applied to unstructured documents as well as structured records. Access controls for sensitive unstructured content, such as mental health notes, substance use documentation, and reproductive health records, require document-level permissions that many document management systems do not natively support.
Structured vs Unstructured Data in Healthcare
Key Differences and Use Cases
| Characteristic | Structured Data | Unstructured Data |
|---|---|---|
| Format | Predefined schema: rows, columns, coded fields | Free text, images, audio, PDFs, or variable-format narrative |
| Examples | ICD-10 codes, lab values, vital signs, appointment dates, claim fields | Clinical notes, discharge summaries, radiology reports, referral letters, patient emails |
| Storage | Relational databases, EHR structured fields, claims systems | Document repositories, PACS systems, scanned file stores, unindexed EHR attachments |
| Searchability | Directly queryable using SQL or standard reporting tools | Requires NLP, OCR, or AI extraction before it is machine-searchable |
| Proportion of healthcare data | Approximately 20% of all healthcare data | Approximately 80% of all healthcare data |
| AI processing required | Minimal; directly usable by analytics and reporting tools | Significant; NLP, entity recognition, and code mapping needed before downstream use |
| Clinical richness | High precision but low context; captures coded values only | High context and narrative detail; captures clinical reasoning and nuance |
| Regulatory compliance | Straightforward to audit and track in structured databases | Requires additional governance to track access, retention, and de-identification |
Why Structured Data Alone Is Not Enough
Structured EHR data, including coded diagnoses, laboratory values, and medication records, provides an important but fundamentally incomplete picture of a patient’s health. It captures what was coded but not the clinical reasoning behind the coding decision. It captures a diagnosis code but not the severity, trajectory, or contextual factors that differentiate two patients with the same code. It captures a medication but not the patient’s adherence, their reported side effects, or the clinician’s assessment of treatment response. All of this critical information lives in the unstructured notes, and it is only accessible to clinical intelligence systems that can read and understand that content.
Population health management, predictive risk scoring, quality reporting, and clinical decision support all benefit substantially from incorporating unstructured data insights alongside structured fields. A readmission risk model that includes the content of the discharge note substantially outperforms a model that uses only structured EHR fields.
How AI Is Solving the Unstructured Data Problem
Role of Natural Language Processing
Natural language processing is the core technology that makes unstructured clinical data accessible to automated systems. Clinical NLP models, including transformer-based architectures such as BioBERT, PubMedBERT, and ClinicalBERT trained on large medical text corpora, understand the vocabulary, abbreviations, and semantic patterns of clinical language with sufficient accuracy to extract meaningful clinical information reliably at scale. Key NLP capabilities applied to healthcare unstructured data include named entity recognition, relation extraction, negation and uncertainty detection, and temporal reasoning.
Machine Learning for Data Extraction
Machine learning extends NLP capability beyond entity recognition to the full pipeline of converting unstructured content into structured, coded, usable data. Classification models assign document types and clinical categories to incoming documents. Code mapping models assign standardised clinical codes to extracted entities. Anomaly detection models identify clinically significant patterns in unstructured text. The combination of these capabilities produces a data extraction pipeline that can process thousands of unstructured documents per day with consistent, auditable accuracy.
Automating Clinical Documentation
One of the most impactful AI applications in healthcare unstructured data management is the automation of clinical documentation creation itself. Ambient AI scribes listen to clinical encounters and generate structured draft notes from the spoken content, converting the encounter from an unstructured audio event into a structured, coded clinical record before the clinician has left the room.
Murphi’s white-label automation platform enables healthcare technology companies to embed this documentation automation capability within their own clinical products, providing their customers with structured, EHR-ready clinical documentation without building the underlying AI infrastructure independently.
Benefits of Converting Unstructured Data into Structured Insights
Improved Clinical Decision-Making
When unstructured clinical data is converted into structured, searchable, and interoperable form, clinicians gain access to the complete clinical picture at the point of care. Medication histories from previous providers, documented allergies in referral letters, risk factors described in consultation notes, and treatment responses documented in specialist reports all become visible and actionable. This completeness directly improves clinical decision quality and reduces the risk of decisions made on partial information.
Enhanced Operational Efficiency
Structuring unstructured data at scale eliminates the manual extraction, transcription, and re-entry workflows that currently consume significant administrative staff time in healthcare organisations. Referral letter intake, prior authorisation documentation, discharge summary processing, and coding from clinical notes all involve humans reading unstructured documents and entering the relevant information into target systems. AI-powered unstructured data processing automates these workflows, reducing processing time from hours to seconds.
Better Patient Outcomes
The downstream clinical impact of making unstructured data accessible to analytics and decision support tools is measurable in patient outcomes. Risk stratification models that incorporate unstructured data identify high-risk patients earlier and with greater accuracy. Care gap identification that draws on the full clinical record surfaces gaps that structured-data-only approaches miss. Population health programmes informed by complete clinical data produce more targeted and effective interventions.
Real-World Use Cases of AI in Healthcare Data Management
Clinical Documentation Automation
AI-powered clinical documentation automation is the most widely deployed application of unstructured data processing in healthcare. Ambient AI scribes generate structured notes from clinical encounters. Discharge summary generation tools extract the relevant diagnoses, procedures, and follow-up instructions from the encounter record. Referral response automation generates structured responses from specialist consultations.
Predictive Analytics and Risk Assessment
Unstructured data significantly improves the accuracy of predictive analytics models in healthcare. Readmission risk models that incorporate the content of discharge notes substantially outperform models based solely on structured EHR fields. Sepsis early warning systems that process nursing notes and physician assessments in real time detect deterioration signals earlier than systems limited to vital sign trends and laboratory values.
Revenue Cycle Optimisation
In the revenue cycle, unstructured data processing improves coding accuracy and completeness by ensuring that the full clinical detail documented in notes and reports is reflected in the codes submitted for billing. AI extraction from the unstructured note surfaces the additional specificity, comorbidities, and complicating factors that support more accurate, higher-specificity coding, improving both revenue capture and compliance with payer documentation requirements.
Challenges in Implementing AI for Unstructured Data Management
Integration with Existing Systems
Integrating AI-powered unstructured data processing with existing EHR and clinical systems is the most common implementation challenge. Healthcare organisations typically have heterogeneous technology environments with multiple EHR systems, speciality applications, and document repositories, each with different integration interfaces and data formats. Building a processing pipeline that ingests unstructured content from all relevant sources requires significant integration engineering and ongoing maintenance.
Data Accuracy and Quality
AI NLP models for clinical text achieve high accuracy on well-formed documentation but performance degrades on low-quality scanned documents, highly abbreviated clinical shorthand, rare speciality vocabularies, and documentation styles that differ significantly from the training corpus. Ongoing accuracy monitoring, systematic collection and review of extraction errors, and regular model updates are essential to maintaining the accuracy levels required for clinical and billing use.
Regulatory Compliance
Processing unstructured healthcare data through AI systems introduces specific HIPAA compliance obligations. The AI processing pipeline must operate under a Business Associate Agreement with every vendor who handles PHI. PII and PHI must be identified and protected in unstructured text through NLP-based detection. Access controls must be applied at the document and field level. Audit logs must record every access, transformation, and routing event.
Best Practices to Manage Unstructured Data in Healthcare
Choosing the Right AI Tools
Selecting AI tools for unstructured data management requires evaluation against the specific document types and clinical domains the organisation needs to process. Purpose-built clinical NLP solutions trained on large medical corpora and validated on the specific document types relevant to the organisation provide higher out-of-the-box accuracy and require less customisation. Accuracy benchmarking on a representative sample of the organisation’s own documents, before deployment, is the most reliable evaluation method.
Ensuring Data Security and Compliance
A secure unstructured data processing implementation requires encryption of all document content at rest and in transit, role-based access controls, audit logging of every document ingestion and delivery event, PII and PHI detection and redaction before data is shared outside the controlled processing environment, and a signed HIPAA BAA with every vendor in the processing pipeline.
Continuous Monitoring and Optimisation
Unstructured data processing pipelines require continuous monitoring for accuracy and quality. NLP model performance drifts over time without systematic retraining. Monitoring programmes should track entity extraction accuracy, code mapping precision and recall, exception rates, and human correction patterns. Corrections made by clinical reviewers should be systematically captured and used to fine-tune the processing models, creating a continuous improvement loop.
Frequently Asked Questions
What is unstructured data in healthcare?
Unstructured data in healthcare is clinical or administrative information stored in formats without a predefined machine-readable schema, including free-text clinical notes, discharge summaries, radiology reports, referral letters, scanned documents, and medical images. It represents approximately 80 percent of all healthcare data and contains the richest clinical content in the health record, but it is inaccessible to standard analytics tools without AI-powered processing.
Why is unstructured data a problem in healthcare?
Unstructured data creates a crisis because the most clinically significant information in the health record is invisible to EHR reporting tools, analytics platforms, and automated workflows. This means clinical decisions are made on incomplete data, administrative processes require manual document reading and re-entry, and population health insights are systematically distorted by the 80 percent of data that structured queries cannot access.
How does AI help manage unstructured data in healthcare?
AI manages unstructured healthcare data through clinical NLP that identifies and extracts clinical entities from free text, OCR that converts scanned documents to machine-readable text, ML models that map extracted entities to standardised clinical codes, and validation pipelines that ensure accuracy before data is delivered to EHR and analytics systems.
What are examples of unstructured clinical data?
Examples include consultation notes, discharge summaries, radiology and pathology reports, operative notes, referral letters, nursing notes, patient-reported histories, scanned paper documents, medical images from PACS systems, and transcripts from telemedicine encounters or ambient AI scribe recordings.
What is the difference between structured and unstructured data in healthcare?
Structured data has a predefined schema: coded diagnoses, lab values, vital signs, and medication orders stored in searchable database fields. Unstructured data has no predefined schema: free-text clinical narratives, reports, and images that require NLP, OCR, or AI to extract their clinical content. Structured data makes up approximately 20 per cent of healthcare data; unstructured data makes up the remaining 80 per cent.