MLNX Job Placement Failure Dataset for Simulated Datacenter Clusters with Reconfigurable Optical Networks
Description
đ OverviewThis dataset contains cluster-level snapshots and job placement outcomes generated using a simulated large-scale datacenter environment.The data is intended for training and evaluating machine learning models that predict whether a job submission will succeed or fail given the current cluster state and job resource request.The dataset was produced as part of the MLSysOps project (EU Horizon Europe) and supports research on:job admission control,failure prediction,resource fragmentation,and network feasibility in modern datacenter architectures.Each data sample represents a single scheduling decision and includes both:detailed cluster state features, andthe observed outcome of the placement attempt.đ˘ System ContextSimulated Datacenter ArchitectureThe dataset is generated using a proprietary datacenter simulator modeling a hierarchical cluster composed of Scalable Units (SUs).Cluster configuration:32 Scalable Units (SUs)32 servers per SU (1024 servers total)8 leaf switches per SU8 GPUs per serverLeaf switches interconnected via a reconfigurable optical circuit switch (OCS)Failure Modes CapturedEach job placement attempt can result in:Successful placementFailure due to insufficient serversFailure due to insufficient or infeasible uplink connectivityWhile server insufficiency can be determined via simple capacity checks,uplink infeasibility is more complex, as it depends on:current optical circuit configurations,contention between jobs,and connectivity constraints of the OCS fabric.The dataset explicitly captures these outcomes to support learning-based approaches for failure prediction.đ Dataset StructureFormat: Apache ParquetGranularity: One row per scheduling decisionEach row contains:Job request featuresCluster state features (scalar + vector)Ground-truth placement outcome labelRows are treated as independent samples.đˇď¸ Ground-Truth LabelsThe dataset includes a label column encoding the observed outcome of the job placement:ValueMeaning0Job placement succeeded1Job placement failed due to insufficient servers2Job placement failed due to insufficient uplinks / infeasible network connectivityNotes:Labels 1 and 2 both indicate job failure, but with different root causes.This encoding allows:binary failure prediction,failure cause analysis,and future multi-class modeling.đ Feature DescriptionScalar Cluster FeaturesThese features summarize utilization, imbalance, and fragmentation across the cluster:ColumnDescriptionf1_event_typeThe recorded event: add, failed_server, failed_uplinkf2_mean_utilMean server utilizationf3_diff_max_min_utilUtilization imbalance across SUsf4_cv_utilCoefficient of variation of server utilizationf5_ratio_max_to_mean_workloadWorkload skew across SUsf6_mean_uplink_utilMean uplink utilizationf7_diff_max_min_uplink_utilUplink utilization imbalancef8_cv_uplink_utilCoefficient of variation of uplink utilizationf9_mean_combined_utilCombined compute and network utilizationf10_resource_imbalanceCompute vs network mismatchf11_bottleneck_ratioNetwork-to-compute utilization ratiof12_frag_spread_susFragmentation due to SU spreadf13_frag_wastedFragmentation due to wasted capacityf14_frag_su_sparsenessIntra-SU sparsenessf15_total_servers_usedTotal servers in usef16_total_sus_usedNumber of active SUsf17_total_uplink_utilizedTotal uplink usageVector FeaturesFeatureDescriptionf18_su_server_bitmapBinary vector (length 1024) indicating per-server usagef19_leaf_upVector (length 256) indicating leaf switch uplink utilizationJob Request FeatureColumnTypeDescriptionf20_requested_nodesint / floatNumber of nodes requested by the jobđ§Ş Data Collection MethodologyEnvironment: Simulated datacenterWorkloads: Synthetic job traces with varying sizes and arrival patternsPlacement policy: Simulator-internal scheduling logicLabeling: Determined by placement outcome (success or failure cause)The simulator executes job placement attempts under varying load, fragmentation, and network conditions to generate diverse training examples.â ď¸Â The simulator itself is not publicly released. Only the resulting dataset is provided.đ Statistical SummaryThe dataset contains a total of 1,062,943 rows, each corresponding to a single job placement attempt in the simulated cluster.The table below summarizes the distribution of all numeric columns, including the ground-truth label.Column Summary and Data TypesDescriptorTypeCountMeanStdMin25%50% (Median)75%Maxl1_failedint321,062,9430.61980.71270.00.00.01.02.0f2_mean_utilfloat321,062,9430.91290.10890.00780.89060.94040.97171.0f3_diff_max_min_utilfloat321,062,9430.58430.33320.00.28130.56251.01.0f4_cv_utilfloat321,062,9430.19820.30470.00.07120.14460.25005.5678f5_ratio_max_to_mean_workloadfloat321,062,9431.18651.22991.01.02911.06331.122832.0f6_mean_uplink_utilfloat321,062,9430.56370.11560.00.51950.58400.63670.9598f7_diff_max_min_uplink_utilfloat321,062,9430.90160.17170.00.81250.90630.96882.0f8_cv_uplink_utilfloat321,062,9430.46810.30220.00.35260.42580.50763.8730f9_mean_combined_utilfloat321,062,9430.73830.10020.00390.71390.75630.79150.9716f10_resource_imbalancefloat321,062,9430.34930.10100.00010.28030.33500.40430.8926f11_bottleneck_ratiofloat321,062,9430.61370.11370.00.56360.63450.68941.7378f12_frag_spread_susfloat321,062,9431.07130.08181.01.02611.05241.09224.0f13_frag_wastedfloat321,062,9430.07130.08180.00.02610.05240.09223.0f14_frag_su_sparsenessfloat321,062,9430.01770.01760.00.00650.01350.02350.2589f15_total_servers_usedint641,062,943934.86111.4789129639951024f16_total_sus_usedint641,062,94331.113.10131323232f17_total_uplink_utilizedint641,062,9434617.66946.6304256478452167863f20_requested_nodesint641,062,94354.9638.778204487128Ground-truth label distribution note:The l1_failed column encodes job outcomes as:0: success1: failure due to insufficient servers2: failure due to insufficient uplinks / infeasible connectivityBoth 1 and 2 correspond to job failures.đ§° Working with the DataLoading the Dataset (Python)import pandas as pddf = pd.read_parquet("final_merged.parquet")print(df.head())Loading Selected Columns cols = ["f20_requested_nodes", "f2_mean_util", "l1_failed"]df = pd.read_parquet("final_merged.parquet", columns=cols)Tools and DocumentationApache Parquet specification: https://parquet.apache.org/docsPandas Parquet I/O: https://pandas.pydata.org/docs/reference/api/pandas.read_parquet.htmlPyArrow Parquet support: https://arrow.apache.org/docs/python/parquet.htmlđŻ Intended UseThis dataset is intended for:machine learning research on job failure prediction,benchmarking admission-control models,studying resource fragmentation and network feasibility,offline evaluation of scheduling heuristics.It is not intended to represent any specific production datacenter.â ď¸ LimitationsData is generated from a simulator, not a production system.The cluster topology is fixed and may not generalize to other architectures.Temporal dependencies between jobs are not explicitly modeled.Network behavior is abstracted and may differ from real optical fabrics.đ CitationIf you use this dataset in your research, please cite it using the citation provided by Zenodo (available in the right sidebar of the dataset recordđ¤ Acknowledgements & FundingThis work is part of the MLSysOps project and is funded by the European Unionâs Horizon Europe research and innovation programme under grant agreement No. 101092912.More information: https://mlsysops.eu/
Citations (0)
No citations found
Mentions (0)
No mentions found
Metrics Over Time
Publication Details
DOI
Publisher
Zenodo
Subfield
Information Systems
Field
Computer Science
Domain
Physical Sciences
Confidence Score
58%
Source
Scholar Data Model