Pembrolizumab and Tamoxifen With or Without Vorinostat for the Treatment of Estrogen Receptor Positive Breast Cancer

Inclusion Criteria:

* Pre and postmenopausal women or men with stage IV ER+ breast cancer histological or cytological confirmation
* \> 10% expression of PD-1/Cytotoxic T-lymphocyte-associated protein 4 (CTLA-4) dual staining in Cluster of differentiation 8 (CD8) cells in tumor or blood or \>5% expression of PD-1/CTLA-4 dual staining in CD4 in blood (only).
* Eastern Cooperative Oncology Group (ECOG) performance status of =\< 1
* Understand and voluntarily sign an informed consent prior to any study-related assessments or procedures are conducted and are able to adhere to the study visit schedule and other protocol requirements
* Consent to paired tumor biopsy
* Measurable tumor by Response Evaluation Criteria in Solid Tumors (RECIST) criteria
* Per Good Clinical Practice, any toxicity related to prior therapies that, in the opinion of the investigator, would potentially be worsened with anti-PD1 therapy should be resolved to less than grade 1
* Absolute neutrophil count (ANC) \>= 1.5 X 10\^9/L
* Hemoglobin (Hgb) \>= 9 g/dL (may transfuse if clinically indicated)
* Platelets (plt) \>= 100 x 10\^9/L
* Potassium within normal range, or correctable with supplements
* Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) =\< 2.5 x upper limit normal (ULN) or =\< 5.0 x ULN if liver tumor is present
* Serum total bilirubin =\< 1.5 x ULN
* Serum creatinine =\< 1.5 x ULN, or 24-hr clearance \>= 60 ml/min
* Females of childbearing potential (defined as sexually mature women who):

  * Has not undergone a hysterectomy (the surgical removal of the uterus) or bilateral oophorectomy (the surgical removal of both ovaries) or,
  * Has not been naturally postmenopausal for at least 24 consecutive months (i.e., has had menses at any time during the preceding 24 consecutive months) must have

    * Negative serum pregnancy test within 14 days before starting study treatment in females of childbearing potential (FCBP) and willingness to adhere to acceptable forms or birth control (a physician-approved contraceptive method (oral, injectable, or implantable hormonal contraceptive; tubal ligation; intra-uterine device; barrier contraceptive with spermicide; or vasectomized partner)
* All female and male participants must agree to use approved contraception during the treatment period and for at least 18 weeks after the last dose of study treatment and refrain from donating sperm during this period

Exclusion Criteria:

* Prior treatment with pembrolizumab or other PD-(L)1
* Any significant medical condition, laboratory abnormalities, which places the subject at unacceptable risk if he/she were to participate in the study
* Has a history of (non-infectious) pneumonitis/interstitial lung disease that required steroids or current pneumonitis/ interstitial lung disease
* Has known active hepatitis B (e.g., hepatitis B surface antigen (HBsAg) reactive) or hepatitis C (e.g., hepatitis C virus (HCV) ribonucleic acid (RNA) \[qualitative\] is detected)
* Has a history of hepatitis B virus (HBV)
* Has a history or current evidence of any condition, therapy, or laboratory abnormality that might confound the results of the study, interfere with the subject's participation for the full duration of the study, or is not in the best interest of the subject to participate, in the opinion of the treating investigator
* Symptomatic central nervous system metastases. Subjects with brain metastases that have been previously treated and are stable for 6 weeks are allowed
* Persistent diarrhea or malabsorption \>= National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE) grade 2, despite medical management
* Unstable angina, significant cardiac arrhythmia, or New York Heart Association (NYHA) class 3 or 4 congestive heart failure
* Prior systemic cancer-directed treatments or investigational modalities =\< 5 half-lives or 4 weeks, whichever is shorter, prior to starting study drug or who have not recovered from side effects of such therapy (except alopecia)
* Active autoimmune disease except for vitiligo or hypothyroidism
* Active and ongoing steroid use, except for non-systemically absorbed treatments (such as inhaled or topical steroid therapy for asthma, chronic obstructive pulmonary disease (COPD), allergic rhinitis)
* Major surgery =\< 2 weeks prior to starting a study drug or who have not recovered from side effects of such therapy
* Pregnant or breastfeeding
* Known human immunodeficiency virus (HIV) infection
* Known history of tuberculosis
* Known allergic reaction or intolerability to tamoxifen
* Patients with prior history of deep vein thrombosis (DVT)s must be on therapeutic or preventive anticoagulation