I think the process you are using is correct. You size in HyperSizer with panel buckling turned on and then do a check in Nastran for panel buckling as a double check, is that right?
The automated buckling span calculation tries to get the correct spans from the FEM, but in some situations is may or may not get it right. That is why we allow you to type your own buckling length in. Remember that HyperSizer calculates buckling by creating an idealized rectangular panel so if you have something really complex shaped in the FEA, you have to make an engineering judgement about what is the correct buckling length so that the buckling of the idealized rectangle matches that of the actual geometry.
As far as how HyperSizer calculates the buckling lengths, the process is described in the Pro Users Manual in the Finite Element coupling reference, Page 121. The section should be titled “Buckling Span Length Calculation”.
Let me know if you have more questions.
Phil