Date Archives February 22, 2018

White Box, Black Box, and Gray Box


In software testing, there are three different types of testing depending on how much knowledge the testers have on the software being tested.

White Box Testing

  • With full knowledge of the software. (internal structures, source codes, etc.)
  • Usually tests how the software performs.
  • Suitable for software developers.

Black Box Testing

  • Without having any knowledge of the software. (internal structures, source codes, etc.)
  • Usually tests for functionality of the software.
  • Suitable for third-party testers.

Gray Box Testing

  • Test with partial knowledge of the software.

Testing Methods


These are currently popular methods to test software.

Fuzz Testing

  • Input massive invalid, unexpected, or random data(Fuzz) to find errors or vulnerabilities in the software.
  • Effective on blackbox testing
  • Pros
    • Simple
    • Provides results with little effort
  • Cons
    • May miss bugs that don’t trigger a full program crash
    • May be less likely to trigger bugs that are only triggered in highly specific circumstances


Stochastic Testing

  • Input random sequence of datas 


Model-based Testing

  • Generates meaningful test case models which can detect errors on the desired behavior.


Symbolic Testing

  • Replace inputs of program with symbolic variables.
  • Track symbols rather than concrete input
  • When execution path diverges, fork and add constraints on symbolic values
  • Find the feasibility of all paths in the program.
  • Quite expensive for finding feasibility of path.
x = read();
if (x > 3) {
  y = 1;
  if (x < 0)
    y = 2;
} else y = 3;

After changing with symbolic variables, it checks the feasibility of paths. On above example, the expression (y = 2; on line #5) will not be reached.

  • Challenges
    • path explosion
      • Exponential in branching structure
      • Loop on symbolic variables
    • Complex code and environment dependencies
      • System calls
      • Library calls
      • Recursive calls
  • Possible Solutions
    • For “path Explosion”
      • Random search (e.g., randomly restart in a while loop, pick next path randomly), but check coverage implementation is required
      • Combined search
    • For “complex code and environment dependencies”
      • Simulate system call
      • Build simple versions of library calls
      • Run library an system calls with a concrete value
      • Summarize the loops
  • Applications:
    • Detecting infeasible paths
    • Generating test inputs
    • Finding bugs and vulnerabilities
    • Proving two code segments are equivalent (Code Hunt)


Concolic Testing (Concrete + Symbolic)

  • Input actual concrete value to the program, then generate new input by changing the path condition.
  • Less expensive than Symbolic testing.


A/B (or split, bucket) Testing

  • Comparing two versions of software which performs better
  • Statistical analysis which performed better

e.g.) randomly showing two different versions to users, and see which one had more uses


Boundary Testing

  • Extreme ends like Start- End, Lower- Upper, Maximum-Minimum, Just Inside-Just Outside values are called boundary values and the testing is called “boundary testing”.
  • Input variables are selected at their:
    • Minimum
    • Just above the minimum
    • A nominal value
    • Just below the maximum
    • Maximum
  • Boundary Testing comes after the Equivalence Class Partitioning.


Equivalence partitioning or equivalence class partitioning (ECP)

  • Software testing technique that divides the input data of a software unit into partitions of equivalent data from which test cases can be derived
  • In principle, test cases are designed to cover each partition at least once.
  • This technique tries to define test cases that uncover classes of errors, thereby reducing the total number of test cases that must be developed.
    • Pros: reduction in the time required for testing software due to lesser number of test cases