06 Lincoln ls making grinding noise?

[Vet92]

So every time I startup my Lincoln first thing in the morning or after it sits for over a few hours it makes a grinding/loose noise (on startup). I thought it might be the starter so i replaced it with a new one starts up better now but, it still makes that noise! but after the first starts it starts up smoothly after that unless it sits too long. looking to find if anyone knows what this is or had same problem?