|
Description
|
This dataset accompanies the paper “Orchestrating Schema Conversions Across Heterogeneous Schema Languages: A Graph-Based, Black-Box Approach” and provides source code snapshots, evaluation inputs and outputs, LLM-based conversion annotations, and derived analyses. It includes the Schema Conversion Orchestrator, a Python/Flask service that discovers, executes, and ranks paths through heterogeneous schema converters, and MetaConfigurator, the web application into which the orchestrator is integrated. The broad evaluation covers JSON Schema, XSD, SHACL, LinkML, and MD-Models, using three input schemas per language at simple, medium, and complex tiers. The medium tier represents the same EnzymeML v2 model in all five languages. Conversions between every distinct ordered language pair yield 60 source–target–input tasks, with alternative conversion paths evaluated for each task. Conversion quality is assessed through an LLM-based annotation pipeline using GPT-5 mini, a fixed comparison prompt, and a structured response format. Each annotation identifies preserved features and conversion issues with supporting source–target evidence, and estimates preserved, missing, incorrect, and extraneous information. Runtime failures and empty outputs receive zero retention deterministically. A sample of 30 model judgments was checked by a human for plausibility. These retained-information estimates support ranking and comparative analysis; they do not establish semantic equivalence. The package includes source code snapshots and evaluation source schemas; conversion outputs and execution metadata; structured annotations and CSV exports; language-pair summaries of conversion strengths and limitations; source-schema profiles and construct counts; benchmark results, plots, and machine-readable ranking scores; and analyses of ranking effectiveness, alternative paths, pruning, and caching. Annotation records include multiple rounds, distinguished by model and prompt-version metadata. The source-schema analysis characterizes input size, declarations, fields, references, depth, and coverage of ten conceptual construct groups. The conversion analyses document recurring losses, including dropped constraints, cardinality errors, lost references, changes in nesting, and loss of semantic identifiers. Language-pair summaries consolidate these findings into evidence-backed strengths and limitations, reviewed and corrected by a human. Plots summarize task-level conversion results and direct-edge quality. Per-edge measurements distinguish robustness (the fraction of serialization-valid output) from conditional retained information; their product estimates expected retained information. Separate SHACL–JSON Schema benchmark scores measure structural agreement with expected target schemas using F1. The benchmark is published separately on DaRUS: https://doi.org/10.18419/DARUS-5964). The included code supports extending the input corpus, registering additional converters, rerunning conversions and annotations, recomputing quality scores, and comparing ranking strategies. Published prompts, response schemas, and annotation provenance document the workflow, although individual LLM judgments may vary between runs. Selected third-party binaries, installed dependencies, and generated build artifacts are omitted from the source snapshot; replication notes describe how to obtain or rebuild them. (2026-09-09)
|
