Hydroxychloroquine, Palbociclib, and Letrozole Before Surgery in Treating Patients With Estrogen Receptor Positive, HER2 Negative Breast Cancer

Inclusion Criteria:

* Signed written informed consent
* Diagnosis of estrogen positive breast cancer, estrogen receptor-positive and HER2-negative by American Society of Clinical Oncology (ASCO)/College of American Pathologists (CAP) criteria
* Eastern Cooperative Oncology Group (ECOG) performance status 0-1
* Postmenopausal defined by: a. Age \>= 55 years and 1 year or more of amenorrhea b. Age \< 55 years and 1 year or more of amenorrhea with luteinizing hormone (LH) and/or follicle stimulating hormone (FSH) levels in the postmenopausal range c. Age \< 55 with prior hysterectomy but intact ovaries with LH and/or FSH levels in the postmenopausal range d. Chemotherapy or medically induced ovarian suppression with 1 year or more of amenorrhea and with LH and/or FSH levels in the postmenopausal range e. Status after bilateral oophorectomy (\>= 28 days prior to first study treatment)
* Absolute neutrophil count (ANC) \>= 1500 cells/ul
* Platelet count \>= 100,000/ul
* Serum creatinine concentration \< 1.5 x upper limit of normal (ULN)
* Bilirubin level \< 1.5 x ULN
* Aspartate aminotransferase (AST) or alanine aminotransferase (ALT) \< 3 x ULN
* Alkaline phosphatase =\< 2.5 ULN
* Metastatic cohorts (Phase I): Diagnosis of stage IV estrogen positive breast cancer, estrogen receptor-positive and HER2-negative by ASCO/CAP criteria
* Metastatic cohorts (Phase I): Must be a candidate for treatment with CDK4/6 inhibitor and hormonal therapy with an aromatase inhibitor as standard of care
* Metastatic cohorts (Phase I): No prior exposure to CDK 4/6 inhibitors
* Neoadjuvant cohorts (Phase II): Diagnosis of stage I-III estrogen positive breast cancer, estrogen receptor-positive and HER2-negative by ASCO/CAP criteria. If stage I, clinical tumor size must be \>= 1.5 cm
* Neoadjuvant cohorts (Phase II): Baseline tumor Ki67 \> 5%
* Neoadjuvant cohorts (Phase II): Surgical candidate and appropriate for pre-operative endocrine therapy

Exclusion Criteria:

* Prior exposure to CDK 4/6 inhibitor therapy
* History of retinal disease or active visual disturbances (normal baseline study-specified retinal exam required)
* Acute illness, including infections requiring medical therapy, known bleeding diathesis or need for anticoagulation
* Treatment with any of the following medications within 4 weeks before the baseline diagnostic biopsy is taken: a. Oral estrogens, including hormone replacement therapy (but prior depot estrogen use not allowed). b. Investigational agents (or 5 half-lives, whichever is longer)
* Required concomitant use of any drug that is a strong CYP3A inhibitor or inducer
* Psychological, familial, sociological or geographical conditions that do not permit compliance with the study protocol
* Life expectancy of less than 6 months
* Pregnancy, lactation or planning to be pregnant.
* Neo-adjuvant cohorts (Phase II): Prior therapy for breast cancer (medical, surgical or radiation therapy)
* Neo-adjuvant cohorts (Phase II): Clinical T4 disease
* Neo-adjuvant cohorts (Phase II): Inoperable or metastatic breast cancer based on standard evaluation