9.1 – What is SimRunner?
SimRunner is a decision support tool used to help optimize simulated processes. SimRunner takes your existing ProcessModel models, evaluates them for you, then performs tests to find better ways to achieve the results you desire.
Typically, most people use simulation tools to predict and improve system performance or to establish the relationships between various system elements. By modeling the actual facility or process, you can conduct what-if analyses to determine the best way to improve system performance—this is optimization. Although SimRunner cannot guarantee it will identify the optimal solution for all processes every time, it will find better solutions than you would likely get on your own.
With each optimization project, SimRunner runs sophisticated optimization algorithms on your model to help you optimize multiple factors simultaneously. Each project requires a validated model, a standard by which to measure system performance (an objective function), and a group of factors that SimRunner may change to improve system performance.
9.1.1 – Benefits
Simulation allows you to find solutions to complex problems without incurring the tremendous costs associated with a trial and error approach. Optimization helps ensure that the solutions you implement are at or near their optimal values.
Unlike manual simulation experimentation which answers only specific what-if questions, SimRunner automatically seeks a course of action that will help optimize your entire system’s performance—essentially answering how to meet your objectives.
To help you present your findings, SimRunner allows you to output various types of reports:
• Data reports (for spreadsheets)
• Analysis reports (for text and word processing reports)
• Graphical reports (for charts and graphs)
You can print each of these reports or export them to other applications through common clipboard functions.
9.2 – Where Do I Begin?
Start with a validated model
Once you complete and validate your simulation model, you are ready to begin an optimization project. If you are not working with a valid model, you don’t need to perform an optimization until the output from the simulation is valid.
Identify your simulation type
It is important to properly identify the simulation type—terminating or nonterminating. What does it mean to refer to a simulation as terminating or nonterminating? A terminating system stops when some key event occurs like the end of the day. When you come back the next day, you start fresh again. A non-terminating system is not necessarily a system that never stops production, rather it is a system that resumes from the point it left off. For both terminating and nonterminating simulations, you need to determine the appropriate run length and number of replications. For non-terminating simulations it is also necessary to determine the warm-up period.
Determine if model is a true candidate for optimization
Not every simulation model is built with the express purpose of optimizing some particular element. Many simulation models are built to demonstrate the relationships that exist between various elements of your system. If optimization is appropriate, define the objective function—the output statistics used to measure the performance of proposed solutions.
Use simulation to identify and examine potential solutions
Simulation has always been trial and error when it comes to optimization. We have some sort of optimization method that we apply to the model and we examine the model’s output statistics to see if we achieved the desired outcome. This is not a bad approach if you have one decision variable you are trying to optimize—but what if you are trying to optimize multiple decision variables at once? Interaction becomes very complex and requires more advanced optimization methods like SimRunner.
Define scenario parameters
When you build your model, you must define a scenario parameter for any variable you want to optimize. This provides SimRunner with a series of values it can change as it seeks to optimize your model. In SimRunner, these values are called factors. For information on how to add and modify Scenario Parameters see Chapter 3, Section 3.13.1, Scenario Parameters.
Part of the simulation process is to evaluate the relationships that exist between model elements, or factors. Often, you will take the time to adjust some part of your model to find that the adjustment has no impact on system performance. Factor screening is the process of identifying which model elements (factors) do not affect the output of your model, and narrowing your search to include only those factors that affect the model’s output. Be discriminant in your selections.
Relax and wait
While some models contain relatively few factors that you can quickly optimize, others contain many. In a previous inventory reduction project, a high-end computer took approximately 24 hours to compute what it estimated to be the optimal value for the model. Although it took a long time to produce this result, the net savings were tremendous.
Consider the results
While there is no promise that SimRunner will identify the optimal solution to your process, it is possible. SimRunner will, however, find better solutions than you would likely get with your own trial and error experimentation. The surest way to know the optimal solution to any model is to run an infinite number of replications of all possible inputs. Since this is not an option, take into consideration the number of experiments you are able to perform and act accordingly.
9.2.1 – Concepts and Theories
SimRunner uses both genetic and evolution strategies algorithms with its primary algorithm being evolution strategies. The specific design of these algorithms for SimRunner is based on the work of Dr. Royce Bowden and other experts. Evolutionary algorithms have been extensively evaluated by academics and have proved reliable in optimizing simulated systems.
An evolutionary algorithm is a numerical optimization technique based on simulated evolution. Solutions generated from the evolutionary algorithm must adapt to their environment in order to survive. Since each potential solution returns a specific result, you must establish an objective function to measure the performance of each solution. If the returned value falls within the acceptable range defined by the objective function, SimRunner will continue to examine the value as it searches for the optimal solution.
To help you better understand how SimRunner’s optimization process works, consider the following analogy. If you and a group of explorers found yourselves on the slopes of a mountain, in the dark, with nothing but radios and altimeters, how would you find the summit?
The first step would be to establish the current altitude of everyone in your group by recording each person’s altimeter reading.
Next, you would direct your group members to wander out in any direction for various distances, then stop and review their new altimeter readings. At the second reading, you find that some of your group has moved higher and some have moved lower.
By comparing the results of each altimeter reading, you can determine the general direction in which to proceed. Since you want to climb to the summit, you proceed in the general direction that had the highest reading (in this case, 2). Again, after everyone moves various distances, you stop to take another altimeter reading.
By taking the average of the group’s new altimeter readings, you can see that the overall reading increased. This confirms that you are moving in the right direction. Just to be sure, you repeat the process and take another reading.
As you will notice from the example, your group is beginning to converge upon a single point. The more you repeat the data collection process, the closer the group will get to one another, and the more certain you will be that you are at or near the summit. Just to be on the safe side, however, you even send a few members of your group out to remote parts of the terrain to be sure that there is nothing higher than the point you already identified.
Once your average altimeter reading is equal to the best in your group, you have converged upon the summit and can be confident that you have achieved your goal.
Conceptually, this is how SimRunner works—only the terminology is different. Instead of altimeters, explorers, and tests, you will use objective functions, input factors, and replications. Each time the explorers move, SimRunner calls this a generation. The concepts are the same.
Not an exhaustive search
In spite of what some may think, you don’t want an algorithm that will do an exhaustive search. In a previous inventory reduction project, modelers estimated the number of possible solutions to be around 9.38 x 10^37 —it was a big project. To perform an exhaustive search for the optimum value would take a lifetime. More than likely, you will never get the answer.
What you want is an algorithm that can efficiently explore the response surface (the output from the model) and focus on those areas returning good answers—without evaluating everything.
9.2.2 – General Procedure
The following is an overview of the process you will use to perform an optimization of your system.
Step 1: Create, verify, and validate
The most important preparation you can make for an optimization project is a validated model. It is not enough to simply create a model—it will profit you nothing if the model does not reflect the real operation. Once you validate the model, you are ready to begin.
Step 2: Build a project
With your model prepared for evaluation, create a new SimRunner project and identify the response statistic you wish to target. Using these response statistics, define an objective function by which to gauge system performance. SimRunner will use this objective function to measure system improvement. Next, select the input factors you will allow SimRunner to use as it determines how best to achieve system improvement. When you optimize the model, SimRunner tests each input factor to seek the combination of factors that will result in the greatest improvement of model performance.
Step 3: Run experiments
Once you select the input factors and define the objective function, you can use SimRunner to automatically conduct a series of experiments on your model. SimRunner runs your model for you and tests a variety of possible combinations of values. After it completes the tests, SimRunner lists the test results in order of the most to the least successful combination of factor values.
Step 4: Evaluate suggestions
The fourth step is to consider and evaluate SimRunner’s suggestions. This is crucial because SimRunner will often identify several candidate solutions to your problem and you may, for reasons not addressed in the model, prefer one solution over another. You may also wish to make additional model runs (replications) and look at confidence intervals to further evaluate SimRunner’s list of possible solutions.
Step 5: Apply solution
Once you identify the solution that best fits your needs, implement the solution.
9.2.3 – Pitfalls
If you follow the general procedure stated previously, your chances of success are very good. Typically, projects fail because the:
• Model is not valid
• Analysis considers insignificant factors
• Analysis ignores significant factors
• Objective Function is inappropriately formulated
• Test results are not scrutinized
9.2.4 – Suggested Readings
If you would like more information on optimization, see the following references:
Bowden, R. O. and J. D. Hall. 1998. Simulation Optimization Research and Development. Proceedings of the 1998 Winter Simulation Conference 1693-1698.
Hall, J. D. and R. O. Bowden. 1997. Simulation Optimization by Direct Search: A Comparative Study. Sixth International Industrial Engineering Research Conference 298-303.
Hall, J. D.; R. O. Bowden; and J. M. Usher. 1996. Using Evolution Strategies and Simulation to Optimize a Pull Production System. Journal of Materials Processing Technology 61:47-52.
Goldberg, D. 1989. Genetic Algorithms in Search, Optimization, and Machine Learning. Addison Wesley, Massachusetts.
Schwefel, H. P. 1981. Numerical Optimization of Computer Models. Chichester: John Wiley and Son.
9.3 – The SimRunner Interface
The SimRunner interface provides you with easy access to every step necessary to create an optimization project. The project building process is divided into three phases, each containing a series of steps necessary to complete the phase. As you move from phase to phase (displayed at the top of the dialog), a list of steps for the phase appears in the left pane.
9.3.1 – Set Up a Project
The first thing you must do to create an optimization project is to start a ProcessModel simulation of your model. You don’t need to complete it. The simulation just needs to be started in order to load the data required by SimRunner. After ending the simulation and returning to the modeling window, click the Tools menu and select SimRunner.
The model data will be loaded automatically.
9.3.2 – Define Objectives
The objective of your project is the final outcome you want to achieve. SimRunner measures your progress toward this goal using an objective function. An objective function is composed of response statistics, a min/max or target range, and a specific weight you wish to apply to each response statistic.
The response category is the type of statistic you wish to use to evaluate your model. Response categories include model elements such as locations, entities, resources, and variables.
By clicking on one of the Response Categories, you will see a list of Response Statistics. Simply put, response statistics are those values you wish to improve. Once you define your targeted improvements, you are ready to define how you want them to perform—the objective for the response statistic.
Objective for Response Statistic
The objective for the response statistic refers to the way in which you want to effect change for that item. If you are trying to increase the overall output of a system, you would maximize the response statistic. Likewise, you could minimize the statistic or target a specific range within which you want the result. Finally, enter the reward or weight for each response statistic—larger numbers signify greater rewards in situations that require more than one objective.
Max Check this option if you want to maximize the final value of this statistic.
Min Check this option if you want to minimize the final value of this statistic.
Maximized objective functions return positive values. Minimized objective functions return negative values.
Target Range Check this option to enter a specific target range within which you want the final result.
Statistic’s weight Weights serve as a means of load balancing for statistics that might bias the objective function. Since most simulation models produce a variety of large and small values, it is often necessary to weight these values to ensure that the objective function does not unintentionally favor any particular statistic. For example, suppose that a run of your model returns a throughput of .72 and an average WIP of 4.56. If you maximize throughput and minimize WIP by applying the same weight to
both ( W 1 = W 2 ), you will bias the objective function in favor of WIP:
Maximize [(W 1 )*(Throughput)] = .72
Minimize[(W 2 )*(WIP)] = 4.56
In this case, since you want to ensure that both statistics carry equal weight in the objective function, you will apply a weight of 6.33 ( W 1 =6.33) to throughput and 1.0 ( W 2 =1.0) to WIP to make them of equal weight in the objective function.
Maximize[(W 1 )*(Throughput)] = 4.56
Minimize[(W 2 )*(WIP)] = 4.56
In situations where it is necessary to favor one statistic over another, balancing the statistics first will make it easier to control the amount of bias you apply. For example, if you apply a weight of 12.67 ( W 1 =12.67) to throughput and 1.0 ( W 2 =1.0) to WIP, the objective function will consider throughput to be twice as important as WIP (adapted from Harrell, Ghosh, and Bowden 2000).
Typically, you will need to experiment with your model to identify the weight ratio necessary to balance statistics.
Response statistics selected for objective function
After you define the objective for the response statistic, you may click the Add button to include the statistic as part of the objective function. SimRunner combines the statistics into a linear combination and displays the updated objective function for the project.
The objective function is an expression used to quantitatively evaluate a simulation model’s performance. By measuring various performance characteristics and taking into consideration how you weigh them, SimRunner can measure how well your system operates. However, SimRunner knows only what you tell it via the objective function. For instance, if your objective function measures only one variable, Total_Throughput, SimRunner will attempt to optimize that variable. If you do not include an objective function term to tell it that you also want to minimize the total number of operators used, SimRunner will assume that you don’t care how many operators you use. Since the objective function can include multiple terms, be sure to include all of the response statistics about which you are concerned.
SimRunner’s capacity to include many different response statistics in an objective function gives it tremendous capability. For example, the objective function below signifies that you wish to maximize the total ovens and cooktops processed while minimizing the total resource cost. The numeric weighting factors indicate that maximizing Total Ovens Processed is the most important, followed by maximizing Total Cooktops Processed, then minimizing Total Resource Cost.
Z =Max:10 * (Total Ovens Processed) +
Max:5 * (Total Cooktops Processed) +
Min:2 * (Total Resource Cost)
Perhaps the best way to define your objective function is in terms of cost or profit. When possible, this allows you to use a simple, single response statistic like maximize profit without concern over assigning meaningful weights to multiple response statistics.
SimRunner’s objective function is calculated by multiplying the result of each objective by its weighting factor, and then adding each product together. Maximized and Target Range objectives are positive and minimized objectives are negative.
For example, if you maximize the number of entities processed, minimize their cost, and minimize the number of resources used, the formula would be as follows:
(entities processed * wt) + (-cost * wt) + (-resources used * wt)
So using the following data for a single experiment:
Entities processed = 100
Entity weight factor = 10
Cost = 25
Cost weight factor = 5
Resources = 8
Resource weight factor = 1
Your result would be:
objective function = (100 * 10) + (-25 * 5) + (8 * 1) = 867
Example: Maximize up to a target
The following example shows how to achieve a throughput target level. Suppose you want to target a production rate of 300 to 325 units per day. In the target range fields, assign a range of 300 to 325 and enter a weight of 1.
The target range in SimRunner is a means of applying bounds to the output values. SimRunner calculates the mean of the range you specify. The closer an entity is to that mean, the higher the value that element of the objective function returns.
If the value returned is below the mean of the objective function, the formula is:
Objective Function Element = Weight * (Value – Min)
If the value returned is above the mean of the objective function, the formula is:
Objective Function Element = Weight * (-1) * (Value – Max)
If the value returned is equal to the mean of the objective function, the formula is:
Objective Function Element = Take any of the above (above or below mean) calculated value
For example, suppose you want to assign a target range to the total number of bicycles processed. You want your objective function to return a value nearest to 400 and your acceptable target range is 300 to 500. Assume your weighting factor is 1. If the number of bikes processed was 380, SimRunner would calculate the value of that element of the objective function as 1 * (380 – 300) = 80. If the number of bikes processed was 420, the value of that element of the objective function would be 1 * (-1) * (420 – 500) = 80. If the number of bikes processed was 400, the value of that element of the objective function would be 1 * 400 = 400.
The key to using the target range is that it must be a RANGE of possible values. If you defined the target range as 400 to 400, the range of possible values would be 0. Therefore you would never get a value besides 0 out of that element of the objective function.
The objective function, in its entirety, is the sum of all elements of the objective function. It calculates the value of each element by multiplying the weight by the returned value. Then, if you are minimizing that element, it is multiplied by -1. All of those elements are added together to calculate the total objective function value.
9.3.3 – Define Inputs
In every system, there are controllable and uncontrollable factors that determine the outcome of the process. Controllable factors include staffing, equipment, schedules, and facilities. Uncontrollable factors refer to such things as arrival rates.
Often, relationships exist between various controllable factors. How do you identify these relationships and use them to seek the best value for your objective function?— SimRunner. SimRunner allows you to target controllable model factors and determine which combination of values for those factors will elicit the behavior you desire.
With each test it performs, SimRunner examines the results to see if the results will produce the effect you require from the objective function. For information about how SimRunner performs these tests and evaluates data, see “Concepts and Theories” in Chapter 9.2.1.
Scenario Parameters listed in model
All Scenario Parameters defined in your model are displayed in SimRunner as Macros. In this manual macros will consistently be referred to as Scenario Parameters. SimRunner will use these Scenario Parameters to improve the value of your objective function. Typically, these are the controllable factors of your model.
Scenario Parameter Properties
Scenario Parameter properties describe the basic attributes of each Scenario Parameter used in your project.
Data Type: the data type refers to the numeric type of the data (integer or real) SimRunner will use. Typically, you will use integers to represent the number of resources (e.g., people) and real numbers to represent time values or percentages (e.g., machine processing times or product mix).
Default Value: Refers to SimRunner’s initial setting for the Scenario Parameter—the value SimRunner will use to analyze the model before conducting the optimization.
Lower and Upper Bound: The limits (constraints) within which this value must fall during subsequent tests.
Scenario Parameters Selected as Input Factors: Displays a list of all Scenario Parameters you selected for use as input factors. Typically, you will want to use as few inputs as possible—only those you anticipate will affect the value of the objective function.
Define Scenario Parameters
Before you can use a model element as an input factor, you must define a Scenario Parameter for that element. Scenario Parameters allow SimRunner to control the value of the factor and feed it back into the model.
If you want to use a variable from your model as an input factor, you must define a corresponding Scenario Parameter and set the variable equal to the Scenario Parameter at some point in your model. To do this, you can:
• Enter the name of the Scenario Parameter in the variable’s “initial value” field.
• Set the variable equal to the Scenario Parameter in the model’s processing logic.
Resource & Activity Capacities
In order to use a resource or activity capacity as an input factor, you will need to define a Scenario Parameter that represents the capacity or number of resources and place this Scenario Parameter in the appropriate field for the resource or activity. For instance, if you have a resource named Operator_1, you can create a Scenario Parameter named Number_Of_Operators to represent the “number of units” for the operator. When you run SimRunner, the Scenario Parameter Number_Of_Operators will appear on the list of input factors. Testing this factor will change the number of units of Operator_1 in the model.
SimRunner will not recognize any Scenario Parameter containing non-numeric characters (e.g., “N”, “)”, or “e+”) as a valid input from the scenario parameters “text” field.
Since Scenario Parameters must be completely numeric in the “text” field for SimRunner’s use, Scenario Parameters that represent distributions will not appear in the list of available input factors. A Scenario Parameter can be used to represent a distribution parameter. For example, suppose you have a normal distribution, N(10,2), with an average of 10 and a standard deviation of 2. You can create a Scenario Parameter, Dist_1_Average, with a value of 10 and use this Scenario Parameter as the text value of the distribution: N(Dist_1_Average, 2). This will allow you to manipulate the distribution from SimRunner.
9.4 – Analyze Model
The first step in conducting any analysis of your simulation model’s output is to make sure that your model produces output with the degree of accuracy you desire. If all of the aspects of your model are deterministic (with no random variance in any part of the model), you will get a precise output statistic with one replication. In most models, however, random variance is present in several places. The result of using random inputs in the model is that the output is also random. Because of this, you should think of a simulation experiment as a sampling technique. Since you don’t know the exact, true value or population statistic, the best you can do is collect a sample large enough to make a good estimate—you must run multiple replications. ref: Analyze Model.
In general, with each additional replication you run, you generate a more accurate estimate of the population statistic your are trying to measure in the objective function. For example, suppose you wish to maximize the utilization of a given resource by changing several factors in the model. For each experiment, you may need to run ten replications to get an accurate estimate of the utilization for a specific combination of settings. If your estimate isn’t accurate enough, you will not be able to tell if improvements to a system are the result of changes or simply an effect of random variance. SimRunner helps you select the appropriate number of replications for both terminating and nonterminating systems.
For non-terminating systems, SimRunner’s Objective Function Time Series graph shows the value of the objective function as it changed during the simulation based on several replications. At the very beginning of a simulation run, the simulation model is “empty.” There are no entities in the system and often all the statistics are set to zero. This is an artificial condition and, for non-terminating simulations, does not reflect reality. When you run a non-terminating simulation, you must run it long enough for the operation to stabilize—to reach steady state. The warm-up time is how long it takes the simulation to reach steady state. The end of the warm-up time is the point at which you want to begin sampling statistical data—data representative of how the system normally operates.
When you define a warm-up period, the model runs for a specific period of time at the beginning of each replication before results are recorded.
How do you know how long the warm-up period should be? SimRunner uses an approach based on Welch’s graphical method. SimRunner displays a time series graph of the objective function’s current value and moving average based on several replications of the simulation. As the objective function begins to stabilize, the moving average graph will appear to “flatten out” around a specific value or range. This behavior signals the end of the warm-up period.
It is quite possible that you may be unable to determine the warm-up period from just one look at the graph. If you are unable to make this determination, run the model again with a longer run length.
In the case of a non-terminating system, the warm-up period is the amount of time the model must run for the objective function to exhibit statistical regularity—to reach a steady-state. This occurs when the distribution of the objective function values is the same from one time period to the next. SimRunner helps you identify the length of the warm-up period.
Simulation Run Length: The total duration of the simulation. If you run a non-terminating simulation, the run length should be sufficient to allow the model to warm up and reach steady state. Start with a large initial value for the run length.
Output Recording Time Interval (period): SimRunner records output statistics during regular intervals of time. This sampling technique provides you with an observation of the output statistic for each time interval. The output recording time interval refers to the length of each time interval. The recording time interval should be large enough to ensure that simulated events occur, and that the model produces an output value for each response statistic listed as part of the objective function. For example, given a job arrives to a queue every 30 minutes, you do not want to select a recording time interval of less than 30 minutes if the average time jobs wait in the queue is part of the objective function. Be sure to set the recording time interval to observe at least one observation during each interval.
Number of Test Replications: The number of replications needed to conduct the analysis. The number of test replications should be five or more depending on how long you have to wait for results.
Percent Error in Objective Function Estimate: Given that you can only estimate the true value of the objective function, you have to decide how accurate your estimate needs to be. Entering a 10 indicates that you wish to estimate the required number of replications needed to obtain an estimate of the average value of the objective function to within 10% of its true value.
Confidence Level: The confidence level (90%, 95%, or 99%) used to approximate the number of replications needed to estimate the value of the objective function to within the stated “percent error” of its true value. The higher the confidence level, the larger the number of replications needed will be.
Run: Begin analyzing the model.
Final Report: Produces an analysis report which describes the results of each experiment.
Analysis Status: Displays the completion status of the analysis.
Warm-up: Detection for steady-state estimates.
Moving Average Window: Allows you to adjust the number of periods used to compute the moving average plot. The plot is automatically redrawn.
Estimate required number of replications
Warm-up Periods: Enter the number of time periods you wish to use as the warm-up period when estimating the required number of replications. Periods refer to the frequency with which you recorded data from the model. For example, if your output recording time interval is four hours, you may find that five periods will suffice (20 hours).
No. of Replications: This field is automatically updated as the warm-up periods field is defined.
9.5 – Optimize Model
9.5.1 – Introduction
Often, the reason for building a simulation model of a system is to answer questions such as: “What are the optimal settings for to minimize (or maximize) ?” You can think of the simulation model as a black box that imitates the actual system. When you present inputs to the black box, the box produces outputs that estimate how the actual system will respond. In the question above, the first blank represents the input(s) to the simulation model that you control. (These inputs are often called decision variables or factors.) The second blank represents the performance measure(s) of interest, computed from the output response of the simulation model when you set the factors to specific values (see example below).
In the question, “What is the optimal number of material handling devices needed to minimize the time that workstations wait for material?”, the input factor is the number of material handling devices and the performance measure is the time that workstations wait (computed from the output response of the simulation model). The goal is to locate the optimal value for each factor that minimizes or maximizes the performance measure of interest.
The following diagram shows the relationship between an optimization algorithm and a simulation model (adapted from Harrell, Ghosh, and Bowden 2000).
If you were to evaluate all combinations of the different values for the input factors (X 1 , X 2 , X 3, …) and plot the output re