There are a lot of people who are queuing with their friends. when he and his friend didnt accept the same match, they just ignore the "confirm match" because it only gives 1 minute penalty
so my suggestion is:
consecutively not clicking "Confirm Match" will have increasing penalty.
first fail in accepting match = 1min
2nd = 5min
3rd = 10min
4th = 20min
5th = 40min