Not enough details
What kind of documentation do you have? What kind of documentation do they have. Was it a loan? Did the grandfather himself ask you to pay it back? What were the circumstances under which you were given the money? What state are you in? Have you talked to the grandfather about this situation? A question like yours can't be answered without details.