Um estudo quantitativo sobre a variabilidade dos tempos de execução de programas em experimentos computacionais

Abstract

In experimental research in computer science many works depend on correct measurement and analysis of the execution times of computer programs. It is observed that not all hold that repeated executions of the same program can produce significantly different execution times in statistical terms. Because of this, there was a quantitative study on the variability of programs execution times, in order to establish a protocol for comparative quantitative analysis of execution times of programs in computational experiments. Subsequently, three experiments were set up and implemented so as to allow the observation of some factors present in the computing environment, in particular related to the operating system. The factors chosen were: Runlevel, compiler Optimization, the Size of an Environment Variable, the Number of Threads and Thread Allocation Strategies. The results obtained with the RTA protocol demonstrate that these factors can influence the program execution times, leading to erroneous conclusions. In some cases, the statistical difference between treatments in an experiment reached 100% of the comparisons. Furthermore, it was observed that the distribution of execution times is not always adhered to a Gaussian distribution. It was observed also that in significance analysis with multiple treatments, the problem known as familywise error rate should be considered and should be treated because it can also lead to wrong conclusions. Thus, these observations emphasize the use of a controlled computing environment and using a statistical method for statistical analysis of multiple comparisons.