Epacadostat and Pembrolizumab Before Surgery in Treating Participants With Stage II-III Esophageal or Gastroesophageal Cancer

Inclusion Criteria:

* Documented informed consent of the participant and/or legally authorized representative
* Agreement to allow the use of archival tissue from diagnostic tumor biopsies

  * If unavailable, exceptions may be granted only with study principal investigator (PI) approval
* Eastern Cooperative Oncology Group (ECOG) =\< 2
* Non-metastatic cancer of the esophagus OR esophagus and gastroesophageal junction (GEJ; tumor extending =\< 2 cm into the stomach)
* Confirmed stage II-III diagnosis of one of the following:

  * Squamous cell; OR
  * Adenocarcinoma; OR
  * Mixed adenosquamous carcinoma
* Deemed appropriate for neoadjuvant chemoradiation by the multidisciplinary team (surgeon, medical oncologist, and radiation oncologist)

  * Chemotherapy defined as weekly carboplatin/paclitaxel; AND
  * Radiation defined as external beam radiotherapy defined as: 50.4 gray (Gy) as per institutional and national treatment guidelines
* Deemed appropriate for esophagectomy or repeat endoscopic biopsies if non-operative management is pursued
* Absolute neutrophil count (ANC) \>= 1500/mm\^3 within 14 days prior to day 1 of study participation/1st endoscopic biopsy
* Platelets \>= 100,000/mm\^3 within 14 days prior to day 1 of study participation/1st endoscopic biopsy
* Serum total bilirubin =\< 1.5 x upper limit of normal (ULN) OR direct bilirubin =\< ULN if total bilirubin levels \> 1.5 x ULN within 14 days prior to day 1 of study participation/1st endoscopic biopsy
* Aspartate aminotransferase (AST) =\< 2 x ULN within 14 days prior to day 1 of study participation/1st endoscopic biopsy
* Alanine aminotransferase (ALT) =\< 2 x ULN within 14 days prior to day 1 of study participation/1st endoscopic biopsy
* Creatinine =\<1.5 x ULN OR for patients with Creatinine \>1.5 x ULN Creatinine clearance of \>=60 ml/min per 24 hour urine test or the Cockcroft-Gault formula
* International normalized ratio (INR) or prothrombin time (PT) =\< 1.5 x ULN within 14 days prior to day 1 of study participation/1st endoscopic biopsy
* Activated partial thromboplastin time (aPTT) =\< 1.5 x ULN
* Female of childbearing potential only: negative urine or serum pregnancy test; if the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required
* Agreement by women of childbearing potential (WOCBP) and males of childbearing potential to use an effective method of birth control or abstain from heterosexual activity for the course of the study through at least 120 days after the last dose of protocol therapy

  * Childbearing potential defined as not being surgically sterilized (men and women) or have not been free from menses for \> 1 year (women only)

Exclusion Criteria:

* Immune checkpoint inhibitor(s) (e.g. anti-PD-1, anti-CTLA4)
* Indoleamine-2,3-dioxygenase (IDO) inhibitors
* Radiotherapy within 21 days prior to day 1 of study participation
* Investigational agent within 21 days prior to Day 1 of study participation
* Live-virus vaccination within 30 days prior to Day 1 of study participation
* Systemic cytotoxic chemotherapy, antineoplastic biologic therapy, or major surgery within 21 days of study participation
* Chronic systemic steroid therapy or on any other form of immunosuppressive medication
* Monoamine oxidase inhibitors (MAOI) or any drug associated with MAOI activity
* Any UGT1A9 inhibitors (including acitretin, amitriptyline, androsterone, cyclosporine, dasatinib, diclofenac, diflunisal, efavirenz, erlotinib, estradiol (17-beta), flutamide, gefitinib, gemfibrozil, glycyrrhetinic acid glycyrrhizin, imatinib, imipramine, ketoconazole, linoleic acid supplements, mefenamic acid, and mycophenolic acid, niflumic acid, nilotinib, phenobarbital, phenylbutazone, phenytoin, probenecid)
* Coumarin-based anticoagulants
* Unstable or untreated brain/leptomeningeal metastasis
* Clinically active diverticulitis, intra-abdominal abscess, gastrointestinal (GI) obstruction, or abdominal carcinomatosis (known risks factors for bowel perforation)
* Severe hypersensitivity reaction to treatment with another antibody and/or hypersensitivity to epacadostat excipients
* Active autoimmune disease that has required systemic treatment in the past 2 years
* Known history of human immunodeficiency virus (HIV), hepatitis B or hepatitis C (with confirmation of negative hepatitis B surface antigen \[HBsAg\], hepatitis B virus \[HBV\] polymerase chain reaction \[PCR\], and hepatitis C virus \[HCV\] PCR)
* History of pneumonitis (non-infectious) that required steroids or current pneumonitis
* Known history of active tuberculosis
* Diagnosed with or treated for cancer within the previous two years, other than histologies listed in inclusion criteria or non-melanoma carcinoma of the skin
* Female only: pregnant or breastfeeding
* Any othr condition that would, in the Investigator's judgement, contraindicate the patient's participation in the clinical study due to safety concerns with clinical study procedures
* Prospective participants who, in the opinion of the investigator, may not be able to comply with all study procedures (including compliance issues related to feasibility/logistics)