Paying GST on Residential House Construction

Hi, I have a small business where we construct residential houses for clients on turnkey basis. We give a payment schedule to clients and they pay us in advance. We than use the money to buy materials for the construction. When client makes us payment we dont give them any gst bill as per my knowledge construction of residential building do not falls under gst regime. But we still have our own gst number where we fall under general contracts etc. moreover suppose 1. we add gst to the payment given to us by the client than the amount becomes huge and the client will never pay that extra gst but will himself construct the house Instead of giving the work to us. 2. Now if we pay gst to govt we will be at a huge loss as the client has refused to pay gst. Can somebody please explain if there is a easy and legal way to tackle this challenge.