- People who do not pray are one of two types
1) leaves the prayer ,the obligatory prayer while rejecting of its obligation --> murtad and shoud be executed by hadd. Off course after reminding him and asks the person to repent, make Tawbah etc
2) the one who does not pray because of laziness(for any reason), but believes that the prayer is wajib --> Should be executed, but hukm is that he is a muslim. "Off course after remining him and asks the person to repent, make Tawbah etc"

This is based on the shafii madhab, allah knows best. Accorrding to imam ahmed; the one who does not pray because of laziness is a kaafir. So both types is a kafir.
Just that your Islam being questioned should really be enough. perform your prayer!

- Also; No ghusl, No janazah and you will be buried with the kuffar.