The difference is one that is very important to the American experience. The Secretary of a branch (or department) is a civilian. The Chief of Staff is military.

In the United States, the military is under civilian control. The Commander in Chief is the President. Since a President has a LOT more to worry about than the day-to-day workings of the military, he delegates authority to the Secretary of Defense, who in turn delegates authority to the separate Secretaries of the respective branches (Navy, Army, Air Force). All of these people are civilians, and they are appointed by the President, with the advice and consent of the Senate.

The Chiefs of Staff of the respective branches (and the Commandant of the Marine Corps) are uniformed military and are the heads of their service. They hold the command authority, but that authority is subject to the directives and policies of the Secretary of that branch. The Chiefs of Staff are also appointed by the President, with the advice and consent of the Senate. Thus there is always ultimate civilian control. Helps to prevent military coups and such.