Input-Output Validation: Using Historical Input Data

• In the previous bank example and other places, we used some artificial data as input to a simulation model.

• An alternative is to use past data as input (historical input data). Sometimes this is called trace-driven simulation.

• To conduct a validation test using historical input data, it is important that all the input data and all the system response data be collected during the same time period.

• Example 11.3 (The Candy Factory)
  • The Candy Factory has three machines, the Candy maker, the Candy packer, and the Box maker.
  • The three machines are connected by two conveyors.
  • The machines have random break-down time.
  • When a machine breaks down, it causes its incoming conveyors to be full and out-going conveyors to be empty.
  • The time-to-failure (operating time) and down-time of each machine is recorded. For machine i
    
    $$T_{i1}, D_{i1}, T_{i2}, D_{i2},...$$
    
    

  • The system responses: the production level $Z_1$, the number $Z_2$ and the time of occurence $Z_3$ of operator interventions were recorded.
  • The model responses ( $Y_i, i = 1,2,3$) for the same parameters were collected for comparison to the corresponding system responses.
  • The differences between $Y_i$ and $Z_i$ can be used to build test statistics (in this case the Student Test t) to see if the model is statistically the same as the real system.
  • If there is only one set of input data, only one set of output can be obtained. The result doesn't carry much statistical significance.
  • If K sets of input data are available, K runs of tests can be conducted. See Table 11.6 for an example.
  • Let $Z_{ij}, j = 1,2,...K$ be the system responses and $W_{ij}, j = 1,2,...K$ be the model responses, the differences $d_j = Z_{ij} - W_{ij}$ are approximately normally distributed with mean $\mu_d$ and variance $\sigma_d^2$
  • If the mean $\mu$ is zero, then statistically $W_{ij}$ are the same as $Z_{ij}$. A t test is conducted
    
    $$H_0 : \mu_d = 0$$
    
    
    and
    
    $$H_1 : \mu_d \ne 0$$
    
    

  • The t statistic is computed as
    
    $$t_0 = \frac{\overline{d} - \mu_d} {S_d/\sqrt{K}}$$
    
    

  • The critical value $t_{\alpha/2, K-1}$ where K is the degree of freedom. In our example, K is the number of input data sets (thus, the number of experiments run).
  • If $\vert t_0\vert \le t_{\alpha/2, K-1}$ do not reject $H_0$, otherwise reject $H_0$.
• Example 11.4 (The Candy Factory continued) Use actual numbers (e.g. K = 5, etc.) to show how the above example works.