“Turn down” is the term for declining an offer/invitation.
Let’s see the sentence with the word included: We had to turn down their invitation to lunch as we had a previous engagement. It is basically saying that they declined an invitation to lunch as they had their previous engagement.
