Nowadays everyone throws around the term "GTO" without much thought as to what the term actually means. This led me to investigate the origin of this word from a purely mathematical point of view. For the sake of simplicity, let us consider a two person poker game. From a game theoretical perspective, poker is a zero-sum game which means that the winnings of one player equals the losses of the other. Now the strategic form of a two person zero-sum game is given by a triplet (X, Y, f) where X is the set of strategies of player I, Y is the set of strategies of player II, and f := f(x,y) is a real valued function on the product space (X,Y) that represents the winnings of player I and the losses of player II.
The great mathematician, John von Neumann proved a theorem that is known as the Minimax Theorem which states the following:
For every finite two-person zero sum game,
a). There exists a number V called the value of the game.
b). There exists a mixed strategy for player I such that player I's average gain is at least V no matter what player II does.
c). There exists a mixed strategy for player II such that player II's loss is at most V no matter what player II does.
The term GTO or game theory optimal, then, refers to the mixed strategies in von Neumanns theorem. The caveat is that the Minimax Theorem states that these mixed strategies exist but it does not tell us how to find them.