if y is a random number between
-.866 and +.866
and x is a random number between
-1 and +1
you can check if
abs(x) < 1 - .5( abs(Y)/.866 )
if so, you can mark a point at X and Y.
otherwise, you can discard that point, because its within the bounding rectangle, but outside the hexagon.
so, as the absolute value of Y approaches 0.866, the bounds of the absolute value of X approaches 0.5, and as the absolute value of Y approaches 0, the bounds of the absolute value of X approaches 1, which describes a regular hexagon with a radius of 1. to scale this hexagon, you can just multiply the X and Y results by whatever radius of Hexagon you want to create.
this updated method will produce an even distribution of random points within a regular hexagon, but the total number of points will be randomized based on how many attempted points within the bounding rectangle were discarded because they missed the hexagon.