4 of 6 standards met
Summary: This diff decouples metric data availability from trial / orchestration status. Previously, marked trials as when optimization config metrics were missing. This conflated two concerns: whether a trial finished running (orchestration) and whether its data is complete (availability). Marking discarded useful partial data from model fitting, allowed re-suggestion of already-evaluated arms, and excluded trials from analysis. The choice to exclude trial data was made since we could not rely on the quality of the data collected from trials. However, by mixing orchestration and metric statuses together, we ended up also throwing away good quality data that just happened to be incomplete (maybe metric fetching failed upstream for unrelated reasons, or one of the metrics failed to compute but the other was fine). With this change: We go back to consuming all data that we believe to be reliable. For transitions, we still require data availability to ensure that we can still fit a model for candidate generation if we transition. This diff introduces a dedicated metric availability layer: Core types () enum (, , ) function that checks which opt config metrics have data for each trial, using and DataFrame groupby Trial completion (, ) Trials are always marked COMPLETED regardless of metric completeness A warning is logged when optimization config metrics are missing Pending points () COMPLETED trials with incomplete data are treated as pending to prevent re-suggestion of already-evaluated arms Transition criteria () Default criterion sets so COMPLETED trials without data don't count toward transition threshold Reviewed By: sdaulton Differential Revision: D93924193
Repository: facebook/Ax. Description: Adaptive Experimentation Platform Stars: 2711, Forks: 359. Primary language: Python. Languages: Python (95.1%), Jupyter Notebook (4.3%), JavaScript (0.3%), Makefile (0.1%), Batchfile (0.1%). License: MIT. Homepage: https://ax.dev Latest release: 1.2.3 (1w ago). Open PRs: 100, open issues: 81. Last activity: 1d ago. Community health: 75%. Top contributors: saitcakmak, mpolson64, lena-kashtelyan, esantorella, bernardbeckerman, sdaulton, mgarrard, ldworkin, Balandat, 2timesjay and others.
Last 12 weeks · 262 commits
Question I am fairly new with Ax and am doing some onboarding type stuff but it is all from an older build of Ax so the examples do not work anymore. What I am trying to do is take a discrete set of featurized data points and evaluate these in the acquisition function then select the one with the highest value to test on my actual function. The model is trained on some data. However I cannot figure out from the documentation how to access my model's acquisition function to evaluate my points. Any help? Please provide any relevant code snippet if applicable. Code of Conduct [x] I agree to follow this Ax's Code of Conduct
Summary: Changes the default black disclosure triangle (▶) used in showing subtitles to an info icon (ℹ️) when rendering analysis cards in IPython/Jupyter environments. This helps in achieveing consistency with how subtitles are rendered in the Ax UI. The icon change only affects notebook rendering via the HTML template; Ax UI uses its own frontend and is unaffected. Differential Revision: D94571402
Summary: Two changes to how pending observation features are computed: 1. Do not use individually abandoned arms as pending points. Previously, arms that were individually abandoned within a BatchTrial (via ) were explicitly added as pending observation features. This is removed — individually abandoned arms should not occupy the pending point space. Note: arms belonging to fully abandoned trials are still treated as pending, since the entire trial was abandoned and the model should avoid re-suggesting those configurations. 2. Treat completed trials without data as pending. Previously, completed trials were never considered pending, even if they had no data for some metrics. Now, checks completed trials for missing metric data and marks those arms as pending for the missing metrics, consistent with how deployed (STAGED/RUNNING) trials are handled. Differential Revision: D94531161