I have made my custom space marine chapter, currently I have only a generic apothecary and captain as my HQ. What I wanted to know is if I could use specific HQ units like the imperial fists Captain Lysander or the Whites Scars Kor'sarro Khan in my custom chapter.
Yes, since your chapter has the Adeptus Astartes keyword, units with that or imperial keyword can be included BUT these HQ units will function as written, their abilities won't change to reference your custom chapter.
Using Lysander as an example, Icon of Obstinacy will ONLY affect Imperial Fists units, For the Khan! from Kor'sarro will ONLY affect White Scars. When talking about casual play, house rules can change this, you can agree with your opponent to allow these abilities to work on your custom faction, but not in tournament.