Title: A Hybrid Heuristic Framework for Resource-Efficient Querying of Scientific Experiments Data Show Chinese title

Title: 一种资源高效查询科学实验数据的混合启发式框架

Abstract: Scientific experiments and modern applications are generating large amounts of data every day. Most organizations utilize In-house servers or Cloud resources to manage application data and workload. The traditional database management system (DBMS) and HTAP systems spend significant time & resources to load the entire dataset into DBMS before starting query execution. On the other hand, in-situ engines may reparse required data multiple times, increasing resource utilization and data processing costs. Additionally, over or under-allocation of resources also increases application running costs. This paper proposes a lightweight Resource Availability &Workload aware Hybrid Framework (RAW-HF) to optimize querying raw data by utilizing existing finite resources efficiently. RAW-HF includes modules that help optimize the resources required to execute a given workload and maximize the utilization of existing resources. The impact of applying RAW-HF to real-world scientific dataset workloads like Sloan Digital Sky Survey (SDSS) and Linked Observation Data (LOD) presented over 90% and 85% reduction in workload execution time (WET) compared to widely used traditional DBMS PostgreSQL. The overall CPU, IO resource utilization, and WET have been reduced by 26%, 25%, and 26%, respectively, while improving memory utilization by 33%, compared to the state-of-the-art workload-aware partial loading technique (WA) proposed for hybrid systems. A comparison of MUAR technique used by RAW-HF with machine learning based resource allocation techniques like PCC is also presented. Show Chinese abstract

Abstract: 科学实验和现代应用每天都在生成大量数据。大多数组织利用内部服务器或云资源来管理应用程序数据和工作负载。传统的数据库管理系统（DBMS）和混合事务分析处理（HTAP）系统在开始查询执行之前会花费大量的时间和资源来加载整个数据集到DBMS中。另一方面，在位引擎可能会多次重新解析所需的数据，从而增加资源利用率和数据处理成本。此外，资源的过度分配或不足分配也会增加应用程序的运行成本。 本文提出了一种轻量级的资源可用性与工作负载感知混合框架（RAW-HF），以通过高效利用现有有限资源来优化原始数据的查询。RAW-HF 包含有助于优化执行给定工作负载所需的资源，并最大化现有资源利用率的模块。将 RAW-HF 应用于真实世界的科学数据集工作负载（如斯隆数字天空调查（SDSS）和链接观测数据（LOD））时，与广泛使用的传统 DBMS PostgreSQL 相比，工作负载执行时间（WET）减少了 90% 和 85%。与针对混合系统提出的最先进的工作负载感知部分加载技术（WA）相比，整体 CPU、IO 资源利用率和 WET 分别降低了 26%、25% 和 26%，同时提高了 33% 的内存利用率。还展示了 RAW-HF 中使用的 MUAR 技术与基于机器学习的资源分配技术（如 PCC）的比较。