Because they had no influence over the budget and military.
In Hamilton's words that would be "no influence ever either the sword or the purse". He argued that because of this the judiciary would be the least threatening branch of the government and that it had to rely on both other branches to uphold its rulings and decisions. In Federalist No. 78 he also states what powers and responsibilities the judicial branch would have.
