You can just not git add the one file on the secondary drive when it is not connected. This schould not be that much overhead and (at least for me) it is much simpler than any other solution. Ps: Why do you want that one config file on the external drive and not on the main one?
That explains why it is easy for you. (joking)