Process level lineage

Process-level lineage outlines the sequence of processing steps or transformations that data undergoes, without necessarily going into the details of field or table-level changes. It helps in understanding the overall data processing workflows and the dependencies between different processes.

 

Let's illustrate process-level lineage with an example:

 

Consider a healthcare organization that collects patient data from various sources such as hospitals, clinics, and medical devices. The organization wants to analyse this data to improve patient care and operational efficiency.

 

Data Collection: The process begins with data collection from different sources. This may involve extracting patient records, medical images, lab results, and sensor data from hospitals, clinics, and medical devices.

 

Data Ingestion: Once collected, the data is ingested into a centralized data repository or data lake. This step involves transferring the data from its source format to a format suitable for storage and analysis. For example, converting data from HL7 format (Health Level 7) for electronic health records into a structured format like JSON or Parquet.

 

Data Preprocessing: After ingestion, the data undergoes preprocessing to clean and prepare it for analysis. This may include removing duplicate records, handling missing values, standardizing data formats, and performing data quality checks. For instance, normalizing patient names and addresses, converting timestamps to a standard timezone, and validating medical codes against reference databases.

 

Feature Engineering: In this step, new features or variables are created from the raw data to enhance its predictive power or facilitate analysis. For example, deriving features such as patient age from date of birth, calculating body mass index (BMI) from height and weight measurements, or extracting text features from medical notes using natural language processing (NLP) techniques.

 

Data Analysis and Modelling: Once the data is preprocessed and feature-engineered, it is ready for analysis. Data scientists and analysts use statistical methods, machine learning algorithms, and data visualization techniques to gain insights from the data. This may involve identifying patterns in patient health records, predicting disease outcomes, or optimizing treatment protocols.

 

Reporting and Visualization: The insights derived from the data analysis are communicated to stakeholders through reports and visualizations. Dashboards, charts, and graphs are used to present key findings and trends in a clear and actionable format. This helps healthcare providers make informed decisions about patient care and resource allocation.

 

Decision-Making and Action: Finally, based on the insights obtained from the data analysis, healthcare organizations can make data-driven decisions to improve patient outcomes, optimize operations, and allocate resources effectively. This may involve adjusting treatment plans, implementing preventive measures, or optimizing staffing levels in hospitals and clinics.

 

In this example, process-level lineage provides an overview of the sequential steps involved in processing healthcare data, from collection and ingestion to analysis and decision-making. It helps stakeholders understand the overall data processing workflow and the dependencies between different processes, enabling them to identify opportunities for optimization and improvement.

Example

{
  "processes": [
    {
      "name": "data_collection",
      "description": "Collects patient data from hospitals, clinics, and medical devices.",
      "input_sources": ["hospitals", "clinics", "medical_devices"],
      "output": "raw_data"
    },
    {
      "name": "data_ingestion",
      "description": "Ingests collected data into a centralized data repository or data lake.",
      "input": "raw_data",
      "output": "processed_data"
    },
    {
      "name": "data_preprocessing",
      "description": "Preprocesses the ingested data to clean and prepare it for analysis.",
      "input": "processed_data",
      "output": "cleaned_data"
    },
    {
      "name": "feature_engineering",
      "description": "Creates new features from the preprocessed data to enhance analysis.",
      "input": "cleaned_data",
      "output": "feature_engineered_data"
    },
    {
      "name": "data_analysis_and_modeling",
      "description": "Analyzes the feature-engineered data using statistical methods and machine learning algorithms.",
      "input": "feature_engineered_data",
      "output": "insights"
    },
    {
      "name": "reporting_and_visualization",
      "description": "Communicates insights to stakeholders through reports and visualizations.",
      "input": "insights",
      "output": "reports_visualizations"
    },
    {
      "name": "decision_making_and_action",
      "description": "Uses insights to make data-driven decisions and take action to improve patient outcomes.",
      "input": "insights",
      "output": "improved_patient_outcomes"
    }
  ]
}

 

 

 

Previous                                                 Next                                                 Home