Skip to content

Running an experiment

The Run step is a live dashboard. Once you start an experiment, it shows progress as it goes, so you always know how far along it is and how it is doing.

The Run step after a finished run: the Experiment complete card reading 9 of 9 points evaluated and the best overall score, four progress cards, the chart Best Result So Far with the overall score on its vertical axis, and the panel What is moving the needle with three factors.

Four cards along the top track Points Tried (Design Points on a fixed design), Runs Completed, Elapsed, and Estimated Remaining, with a progress bar underneath showing how much of the run budget is used. Because the search knows its budget, the estimate is honest about how much is left.

The chart plots each configuration’s overall score as it finishes, in the order it ran. The score runs from 0 to 1: every ticked response at its weight, scaled over the points finished so far. A line traces the best score found so far, so you can watch it climb and then level off as the search settles. Feasible points are filled; points that broke a limit show as hollow rings and are excluded from the ranking.

Beside the chart, a panel ranks the factors by how much each one changed the lead response, between a factor’s two settings or, for a factor tried at three or more settings, between its lowest and highest averages, longest bar first. It fills in when the run completes, once the whole picture is in: while the run is still going, the panel says the answer is on its way. The same chart appears on Results as the main effects.

Below the chart, Live Leaderboard by Overall Score ranks the best points so far, six at a time. Its header says how many it shows, such as “Top 6 of 6 points by overall score, updated a moment ago”, or “No points ranked yet”. Each row gives the Rank, a Status (Best, Tied, 2nd, Tied 3rd and the rest, the same words as the ranked table on Results), the value of each factor, the Overall score with its 95 percent range, and a trend chip: New for a point that has just entered the board, Up n or Down n for a point that has moved up or down the board by n places, and Steady for one that has not moved. The point that just finished is highlighted and carries a “Point n” badge, and the line under the table says where it entered the board. If that point broke a limit, it is pinned unranked with a red Infeasible chip.

A row that reads “Baseline, today’s settings”, ranked Base and marked Vs today, is your model as it stands, so you can see what the points are measured against. It needs at least two replications; with one, a note under the table says why the row is missing. Points still running sit below the board, marked Running. A point that broke a limit is pinned with the rank None. Once the analysis behind the ranked table has answered, the places come from it; until then they come from the same rule worked out on the finished points.

The Run step of PM Optimizer for the experiment Queue size and who goes first, scrolled to the Live Leaderboard by Overall Score and All Points, with the footer reading Run in progress. The leaderboard header reads Top 6 of 7 points by overall score, updated a moment ago, and its columns are Rank, Status, Treatment Line: Capacity, Treatment Line: Queue Discipline, Nurse: Quantity, Overall score and Trend. The first row is Base, Baseline, today's settings, marked Vs today. Below it come six ranked points, Best then 2nd to 6th, with the chips Steady, New with Point 7 on the highlighted row at rank 4, and Down 1, then a Running row reading 3 of 20 reps. A line says the highlighted row is the point that just finished and entered the board at rank 4. Under it, All Points lists Baseline and the points P1 and P2 with Status Done, Reps 20 / 20 and the column Urgent: Completed count.
Figure 1. The Run step during a run, with the Live Leaderboard by Overall Score above All Points: the Baseline row marked Vs today, a Running row, and the point that just finished highlighted.

All Points lists every design point with its status: #, Point, Status, Reps and the lead response, the first one checked. Status reads Queued, Running, Done, Failed or Cut short. A Smart Optimizer run adds a Why column that reads Explored for a point’s first pass, Leader, refined for a leader that earned extra replications, and “Cut short” with the reason for a point the search stopped spending runs on.

By default, each configuration is tested against the same random conditions, replication for replication, and the engine compares them run by run when it decides ties. This paired comparison means the differences you see come from the settings you changed, not from one configuration happening to get an easier run. It makes the comparison much sharper at no cost to you.

Stop ends the run. The points gathered so far stay on screen for you to read, but a stopped run is not a finished experiment: it is not saved with the model, and the Results tab stays locked. Run the experiment again when you want a result you can rank and keep. A Smart Optimizer run also stops on its own when better results stop coming.

At completion, the top of the dashboard becomes a summary card: how many points were evaluated, how long it took, and the best overall score with its settings, or, when ties remain, “shared by N configurations tied for best”. Once the analysis has answered, it names the winner of the ranked table. The card has one button, View Results, and the window then opens the Results step for you, where the ranked table and the export live.

If the app closes in the middle of a run, nothing gathered is lost silently: the next time you open PM Optimizer, it offers the stranded results, to export as a spreadsheet or to discard.