Really scared

Hi, I'm a first time poster, just after some honesty and hopefully reassurance. I noticed some pain in my left breast last week, just before the bank holiday weekend. Over the weekend my nipple has inverted. I managed to get into the doctors on Tuesday. She has prescribed a weeks course of antibiotics and passed a referral to the breast clinic. I've now been on the antibiotics for two days.  The pain seems to have subsided but I'm left with a full breast with swelling that comes and goes and a nipple that sometimes inverts in or is otherwise flat. I'm hardly eating or sleeping as I've convinced myself it's something really bad like Inflammatory breast cancer. I have two children and I well up every time I think about leaving them without a mum. 
 

I just wondered if anyone could offer any advice. I initially thought it may be something to do with a milk duct due to my age (I'm 41) but now I'm not so sure.