Fitti Weissglas

Principal Software and Data Engineer

Overview

The Data Use Facilitation Tool (DUFT) is an advanced platform designed to enhance data integration, management, and visualisation. Built with a modular and configuration-driven architecture, DUFT overcomes long-standing challenges in data-heavy environments, particularly in fragmented health information systems. Traditional reporting platforms struggle with scattered datasets, rigid workflows, and an over-reliance on manual ETL processes. DUFT addresses these limitations by offering a highly adaptable solution where dashboards, data queries, and workflows can be modified dynamically without requiring backend code changes. Currently in beta, DUFT is expected to be released in 2025.

At the core of DUFT is a separation of concerns between data processing, configuration, and presentation. Unlike conventional applications that hardcode dashboards and reporting workflows into their frontend and backend, DUFT introduces a fully decoupled design, ensuring flexibility at every layer.

DUFT

A Three-Tiered Architecture for Maximum Adaptability

DUFT is structured around three primary components: DUFT Server, DUFT Config, and DUFT UI. This architecture was deliberately chosen to maintain a high degree of adaptability while keeping deployment lightweight.

DUFT Server: A Configurable Backend for Data Execution

At its foundation, DUFT Server is a Django/Python-based execution engine responsible for query processing, ETL task execution, authentication, and API interactions. Unlike traditional monolithic backends that require extensive modifications for every change, DUFT Server is designed to be entirely configuration-driven, reducing deployment complexity.

Engineering Decisions

  • Implemented a modular service-oriented architecture, allowing new data processing capabilities to be added without affecting the core system.
  • Enabled asynchronous execution of long-running data transformations, ensuring system responsiveness by offloading tasks to background processes.

DUFT Config: A Git-Compatible Control Layer

To maximise flexibility, DUFT eliminates hardcoded logic by storing system behaviour in plain-text configuration files. These files define dashboards, data sources, scheduled ETL tasks, and navigation structures, allowing DUFT’s behaviour to be modified dynamically.

Engineering Decisions

  • Prioritised declarative configurations over procedural logic, allowing even non-developers to modify dashboards and workflows.
  • Introduced modular apps within DUFT Config, allowing multiple teams to manage separate configurations without interfering with each other.

DUFT UI: A React-Powered, Fully Configurable Frontend

Unlike conventional web applications where UI components are predefined and hardcoded, DUFT UI dynamically constructs dashboards based on DUFT Config. Built with React and Tailwind CSS, DUFT UI does not rely on fixed layouts; instead, it fetches configurations in real-time, rendering dashboards, charts, and navigation menus on demand.

Engineering Decisions

  • Developed a 3DL Parser that dynamically translates 3DL (DUFT Dashboard Design Language) into React components at runtime, eliminating the need for frontend modifications when updating dashboards.
  • Utilised React Context API to handle UI state management efficiently, ensuring seamless data filtering, navigation, and updates.

A Declarative Approach to Dashboards and Data Processing

A key innovation in DUFT is 3DL, which provides a declarative way to define dashboards, data sources, and visualisation components. Traditional dashboarding tools require extensive backend logic to manage UI changes, leading to tightly coupled systems that are difficult to maintain. DUFT’s 3DL introduces an XML-like syntax that allows dashboards to be structured hierarchically, with UI components dynamically binding to SQL queries, API responses, or external data sources.

Engineering Decisions

  • Designed 3DL as JSX-compatible, ensuring seamless React integration while maintaining human readability.
  • Implemented query binding mechanisms, allowing dashboards to reflect real-time data updates without requiring backend modifications.

To handle large-scale data processing, DUFT includes a Data Task Engine (DTE), which automates ETL processes and complex data transformations. Instead of embedding workflows into static SQL queries, DUFT allows implementers to define tasks using Python scripts or Jupyter notebooks, ensuring full flexibility in how data is processed.

Security, Role-Based Access, and Extensibility

Security within DUFT is designed to be both robust and flexible. Built on Django’s authentication framework, DUFT enforces Role-Based Access Control (RBAC), ensuring that different users have appropriate levels of access to dashboards, data tasks, and queries.

Engineering Decisions

  • Abstracted permissions at the action level rather than binding them to specific API endpoints, making security policies more maintainable.
  • Introduced a centralised Permission Service, ensuring a single source of truth for authorisation decisions, improving auditability and security enforcement.

DUFT’s extensibility is further enhanced by its batch-based data upload system, allowing structured datasets to be transmitted between different DUFT instances or national repositories. Unlike traditional APIs that transmit individual records, DUFT optimises performance by processing uploads in bulk, reducing network overhead and ensuring efficient data transfers.

A Future-Ready Platform for Data-Driven Decision Making

DUFT is more than just a reporting tool—it is a data engineering and visualisation platform designed for scalability, adaptability, and seamless integration. By prioritising a configuration-driven approach, DUFT eliminates many of the pain points associated with traditional software deployments, enabling organisations to modify dashboards, data pipelines, and security policies dynamically.

From an engineering perspective, DUFT demonstrates how modern software architecture principles—such as modular service-oriented design, declarative configuration, and real-time UI generation—can be leveraged to build a highly flexible and extensible system. Whether deployed in national health data systems, research institutions, or data-heavy enterprises, DUFT provides a foundation for seamless data management and actionable insights.