Can a new landlord hold the tenant's money before ever giving them a key?

We recently signed a lease for a new rental, and paid the deposit, pet deposit, and first month's rent in cash (we got a receipt for it). We have Section 8, and the landlord told us that we could only have the key after the apartment ped inspection. Inspection failed today, and now the landlord is saying he no longer wants to rent to us and we can have neither the key or any of the money we gave him. He stated that it was in the contract that he could keep first month's rent, however the copies of the paperwork he gave us say nothing about this. Also, he has about $700 more than just the first month's rent because of the pet deposit and security deposit.