Not everything that lowers HDL is bad for you. If you change from a high-fat, high-cholesterol diet to a healthy low-fat, low-cholesterol diet, your HDL levels may stay the same or even decrease because there is less need for it. When you have less garbage, you need fewer garbage trucks to remove it, so your body may make less HDL.