Benchmarking LLM over roundtrip evolution of Ecore and SQL

FILES OVERVIEWThis repository evaluates round-trip Model Transformation-by-Example (MTBE) between Ecore (S) and SQL-XMI (T). Each experiment starts from a paired model (S, T), applies a single operator delta Δ to one side, then performs:Inverse step: Δ(T) → S′Forward step: S′ → T′We validate by comparing S′ vs Δ(S) and T′ vs Δ(T).KEY POINTERS WHEN RE-DOING THE EXPERIMENTSPrompt Composition (same formula):Prompt = Example Pair(s) (E) +  Δ-variant input (IM) + Stage TaskSemantic Completeness: Ignore minor formatting/whitespace. Prioritize structural and referential correctness (classes/tables, attributes/columns, types, multiplicities/keys, containment vs reference, FKs).Model-Specific Behavior: LLMs are sensitive to prompt length and example order. Keep the same example pair(s) across both stages of a run.Manual Review: Outputs may contain hallucinations or omissions. Always perform loadability and semantic checks before scoring.STEPS TO REPLICATE THE PROCESS1. Prompt GenerationPrepare two prompts per (pairId, op, k):A) Inverse (Δ(T) → S′)Include (in this order):Example Pair(s) (Eₙ) — S_[pairId].ecore + T_[pairId].sql.xmi (k = 1, 2, or 4)Input Artifact — Delta_[op]T[pairId].sql.xmiTask - “Provide only the Ecore/XMI serialization of S’, metamodel-conformant. If ∆(T ) contains elements not covered by the example mapping, briefly flag them in XML comments, but still produce a valid S’.”B) Forward (S′ → T′)Same structure, but the Input Artifact is the S′ you just produced.Task - “Generate only SQL-XMI for T′ (no prose/comments). Ensure metamodel conformance and Δ-awareness.”Keep E fixed across both stages to isolate round-trip effects.2. Running LLMsFor each of the four tested LLMs:GPT-5DeepSeek V3Qwen-3-Next-80B-A3BGemini 2.5Execution tips:Optional system prompt:“You are a transformation assistant that learns from example models and produces structurally valid models only.”3. Manual EvaluationCompare:Inverse: inverse.ecore (S′) vs Delta_[op]S[pairId].ecore (Δ(S))Forward: forwardsql.xmi (T′) vs Delta[op]T[pairId]_sql.xmi (Δ(T))Record per stage in a sheet with columns:LOC, Cr, InCr, Ad, Miss, %Correct, %WSuccess, DeltaDetected, LoadableMetrics (same style ):% Correct = Cr / (LOC + Miss − Ad)% Weighted Success = (Cr−InCr−Miss−Ad[harmful]−2[∆ undetected] / LOCDelta checks:S′ ≡ Δ(S) and T′ ≡ Δ(T) (after canonical whitespace/ID normalization)Semantic/structural equivalence (entities, properties, types, multiplicities, FKs, containment vs reference), even if IDs/order differ from the original files.Loadability sanity:Ensure XML namespaces (xmi:version, xmlns:*, nsURI) match the metamodel declarations.4. Repeat for All CombinationsLet:B = number of base pairs (here 10)D = number of delta operators (here 10)L = number of LLMs (here 4)Then you’ll have:Runs: B × D × L, each with two saved outputs (inverse + forward)Stage evaluations: 2 × B × D × L (here 800)