This is a limitation in our discount feature, sadly. A product specific discount will pay attention to only one line of the cart in terms of quantity, which means that an order of five spread out around different colors or sizes will not trigger the discount.
Rather than using a product specific discount I recommend using an "All cart items" discount. If the discount is focusing on everything in the cart rather than just one product you can apply a discount to an order of five items regardless of what sizes and colors are ordered. That does mean any five items rather than just five of that one product, but otherwise it will work as intended.