Both the Open Hole Compression (OHC) and Open Hole Tension (OHT) ply based analyses are longitudinal max strain failure methods. For each ply, HyperSizer calculates the strain in the 1 direction (the fiber direction) and then compares the strain to the allowable to calculate the margin. If the strain is positive, it compares to the OHT allowable, if negative, it compares to the OHC allowable.
While this is not an interaction equation at the ply level, if you enter a shear load for the laminate, that shear will have an effect on the 1 direction strain of any off axis plies, therefore laminate shear strain will have an effect on this margin.
You should be able to confirm this behavior. Set the OHT allowable equal to the etu1 strain allowable on the Strain Allowables I tab. Also set the OHC allowable equal to the ecu1 allowable. On the failure tab, turn on all of the max strain composite strength margins along with the Open Hole Tension and Open Hole Compression margins. No matter what you put in for a load on the composite, the Max Strain 1 direction margin should match either the OHC or the OHT margin (depending on whether the load is compressive or tensile).