MEDI4736 and Tremelimumab in Treating Patients With Metastatic HER2 Negative Breast Cancer

Inclusion Criteria:

* Patients must have a histologically documented (either primary or metastatic site) diagnosis of breast cancer that is HER2 non-overexpressing by immunohistochemistry, namely 0 or 1; if they have an equivocal immunohistochemistry, 2, the tumor must be non-gene amplified by fluorescence in situ hybridization (FISH) performed upon the primary tumor or metastatic lesion (ratio \< 2 and HER2 copy number \< 4); estrogen receptor (ER) positivity is defined as 1% or greater
* Patients must have measurable disease by Response Evaluation Criteria in Solid Tumors (RECIST) criteria
* Patients who are ER negative must have progressed through at least one prior chemotherapy regimen in the metastatic setting or within 12 months of their last adjuvant systemic treatment; patients who are ER positive must have progressed through standard hormone therapy options and have received at least one line of chemotherapy in the metastatic setting
* Completion of prior chemotherapy systemic anticancer therapy at least 2 weeks prior to study entry
* Radiation therapy must be completed at least 2 weeks prior to study entry; radiated lesions may not serve as measurable disease unless they have been radiated over 12 months prior to enrollment
* Patients may have parenchymal brain metastases if stable (no evidence of progression) for at least 1 month after local therapy (radiation or surgery); leptomeningeal disease is excluded; must have completed any prescribed steroid taper
* Patients may have had a prior diagnosis of cancer if it has been \> 5 years since their last treatment
* Patients must exhibit an Eastern Cooperative Oncology Group (ECOG) performance status of =\< 2
* Absolute neutrophil count \>= 1,000/mcL
* Platelets \>= 50,000/mcl
* Total bilirubin =\< 1.5 times the institutional upper limit of normal (ULN) (or =\< 3 times ULN in case of liver metastasis)
* Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SPGT\]) =\< 2.5 X institutional ULN (or =\< 5 times ULN in case of liver metastasis)
* Creatinine =\< 2 ng/ml
* Females of child-bearing potential (FOCBP) and males must agree to use 2 methods of adequate contraception prior to study entry, for the duration of study participation, and for number (#) days following completion of therapy; should a female patient become pregnant or suspect she is pregnant while participating in this study, she should inform her treating physician immediately

  * NOTE: A FOCBP is any woman (regardless of sexual orientation, having undergone a tubal ligation, or remaining celibate by choice) who meets the following criteria:
  * Has not undergone a hysterectomy or bilateral oophorectomy
  * Has had menses at any time in the preceding 12 consecutive months (and therefore has not been naturally postmenopausal for \> 12 months)
* FOCBP must have a negative pregnancy test within 7 days prior to registration on study
* Willingness to provide a fresh biopsy prior to study enrollment and after 2 cycles of treatment
* Patients must have the ability to understand and the willingness to sign a written informed consent prior to registration on study

Exclusion Criteria:

* Patients who have had chemotherapy or radiotherapy within 2 weeks prior to entering the study or those who have not recovered from adverse events due to agents administered more than 4 weeks earlier are not eligible.
* Current or prior use of immunosuppressive therapy within 2 weeks of starting investigational therapy
* Patients who are taking any herbal (alternative) medicines are NOT eligible for participation; patients must be off any such medications by the time of registration for at least 2 weeks; NOTE: Vitamin supplements are acceptable
* Patients may not have received any other investigational agents within 4 weeks prior to registration
* Prior treatment with immune therapy (including but not limited to cluster of differentiation \[CD\]137, OX40, programmed death \[PD\]-1, PD-L1 or cytotoxic T-lymphocyte antigen 4 \[CTLA4\] inhibitors)
* Prior severe infusion reaction to a monoclonal antibody
* Patients with a history of or active autoimmune disease within the past 3 years with the following exceptions:

  * Vitiligo or alopecia
  * Hypothyroidism on stable doses of thyroid replacement therapy
  * Psoriasis not requiring systemic therapy within the past 3 years
* History of primary immunodeficiency disease or tuberculosis
* Major medical conditions that might affect study participation (uncontrolled pulmonary, renal, or hepatic dysfunction, uncontrolled infection) are not eligible; other significant comorbid condition which the investigator feels might compromise effective and safe participation in the study
* Patients who have an uncontrolled intercurrent illness including, but not limited to any of the following, are not eligible:

  * Uncontrolled pulmonary, renal, or hepatic dysfunction
  * Ongoing or active infection requiring systemic treatment
  * Known active or chronic viral hepatitis or human immunodeficiency virus (HIV)
  * Psychiatric illness/social situations that would limit compliance with study requirements
  * Any other illness or condition that the treating investigator feels would interfere with study compliance or would compromise the patient's safety or study endpoints
* Female patients who are pregnant or nursing are not eligible