Ipilimumab, Nivolumab, and Talimogene Laherparepvec Before Surgery in Treating Participants With Localized, Triple-Negative or Estrogen Receptor Positive, HER2 Negative Breast Cancer-deleted

Inclusion Criteria:

* Written informed consent must be obtained from the subject/legal representative prior to performing any protocol-related procedures, including screening evaluations
* Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0-1
* Localized, palpable, biopsy proven triple negative or ER positive HER2 negative infiltrating ductal breast cancer with size \> 1.5 cm by palpation, excluding breast cancer where neoadjuvant chemotherapy is indicated by current guidelines (i.e. inflammatory subtype, etc.)
* Tumor that is palpable and injectable
* Hemoglobin \>= 9 g/dL
* Absolute neutrophil count \>= 1,500/mm\^3
* Platelet count \>= 100,000/mm\^3
* Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) =\< 2.5 x institutional upper limit of normal (ULN)
* Bilirubin =\< 1.5 x ULN; for subjects with documented/suspected Gilbert?s disease, bilirubin =\< 3 x ULN
* Albumin \>= 2.5 g/dl
* Prothrombin time (PT) / International normalized ratio (INR) and partial thromboplastin time (PTT) =\< 1.5 x institutional upper limit of normal unless the patient is on anticoagulant therapy within 28 days prior to enrollment (if the patient is receiving anticoagulant therapy, PT, and a PTT must be within therapeutic range of intended use of anticoagulants)
* Patients must be willing to submit blood and tissue specimens for translational medicine studies
* Patients must be offered the opportunity to participate in specimen banking for future research
* Women of childbearing potential (WOCBP) must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of human chorionic gonadotropin \[HCG\]) within 24 hours prior to the start of study drug
* Women must not be breastfeeding
* Women of childbearing potential (WOCBP) must be willing to use either two adequate barrier methods or a barrier method plus a hormonal method of contraception to prevent pregnancy, or to abstain from heterosexual activity (complete abstinence) throughout the study, starting with visit 1 through 5 months after the last dose of study therapy. Approved contraceptive methods include, for example, intrauterine device, diaphragm with spermicide, cervical cap with spermicide, male condoms, female condoms with spermicide, or oral contraceptives. Spermicides alone are not an acceptable method of contraception. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately
* Male patients must agree to use an adequate method of contraception, or to abstain from heterosexual activity (complete abstinence) starting with the first dose of study drug through 7 months after the last dose of study therapy

Exclusion Criteria:

* Contraindications to tumor biopsy and injections (coagulopathy, known history of keloid formation, etc.)
* Women who are pregnant or breastfeeding
* Sexually active subjects and their partners unwilling to use male or female latex condom to avoid potential viral transmission during sexual contact while on treatment and within 30 days after treatment with talimogene laherparepvec
* Inability to give informed consent
* History of malignancies except cured basal cell carcinoma, cutaneous squamous cell carcinoma, melanoma in situ, superficial bladder cancer or carcinoma in situ of the cervix; for other malignancies, must be documented to be free of cancer for \>= 2 years. All other cases can be considered on a case by case basis at the discretion of the principal investigator
* Any condition that might interfere with the subject?s participation in the study, safety, or in the evaluation of the study results
* Concurrent enrollment in another clinical study, unless it is an observational (non-interventional) clinical study or the follow-up period of an interventional study
* Prior exposure to any anti-PD-1 or anti-PD-L1 antibody, or any anti-CTLA 4 antibodies
* Patients must not have received prior treatment with talimogene laherparepvec or other oncolytic virus agents
* Patients must not have received any live vaccine within 30 days prior to registration. Seasonal flu vaccines that do not contain live virus are permitted
* Patients must not have an active infection requiring systemic therapy nor a viral infection requiring intermittent treatment with an antiherpetic drug, other than intermittent topical use
* Patients must not have active herpetic skin lesions or prior complications of herpetic infection (e.g., herpetic keratitis or encephalitis) which requires intermittent or chronic treatment with an anti-herpetic drug other than intermittent topical use
* Any concurrent chemotherapy, immunotherapy, biologic or hormonal therapy for cancer treatment. Concurrent use of hormones for non-cancer-related conditions (e.g., insulin for diabetes and hormone replacement therapy) is acceptable
* Patient must not have evidence of any clinically significant immunosuppression such as the following: primary immunodeficiency state such as severe combined immunodeficiency disease; concurrent opportunistic infection; receiving systemic immunosuppressive therapy within 28 days before enrollment with the exceptions of intranasal, topical, and inhaled corticosteroids or oral corticosteroids at physiologic doses not to exceed 10 mg/day of prednisone or equivalent
* Patients must not have known history human immunodeficiency virus (HIV)
* Clinical or laboratory evidence of an active herpetic infection and in patients who require daily antiviral therapy such as acyclovir
* Active or prior documented autoimmune disease within the past 3 years

  * NOTE: Subjects with active, known or suspected autoimmune disease such as vitiligo, type I diabetes mellitus, residual hypothyroidism due to autoimmune condition only requiring hormone replacement, psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger are permitted to enroll
* Active or prior documented inflammatory bowel disease
* History of (non-infectious) interstitial lung disease (ILD)/pneumonitis that required steroids or has current ILD/pneumonitis or where suspected ILD/pneumonitis cannot be ruled out by imaging at screening
* Exposure to any investigational drug within 7 days prior to screening visit or for which 5 half-lives have not elapsed
* Prisoners or subjects who are involuntarily incarcerated

  * Note: under certain specific circumstances a person who has been imprisoned may be included or permitted to continue as a subject
* Subjects who are compulsorily detained for treatment of either a psychiatric or physical (e.g., infectious disease) illness