A Phase II Study Evaluating an Organ Preservation Strategy Using Immune Checkpoint Blockade for Participants With Primary Colorectal or Gastroesophageal Cancer

* INCLUSION CRITERIA:
* Participants must have biopsy-proven stage I-III colorectal cancer (CRC) \[any MMR or Tumor Mutational Burden (TMB) status\] or stage I-III gastroesophageal cancer (GEC) (MMR deficient only).
* Participants with known mismatch repair protein expression by immunohistochemical staining and/ or known next-generation sequencing report of tumor mutational burden and/or microsatellite status. Note: For participants that come to NIH with an equivocal MMR status, next-generation sequencing (NGS) by TSO500 will be done at NIH.
* More than four weeks must have elapsed since completion of any prior systemic therapy or radiotherapy at the time of enrollment. Participants are permitted to have undergone prior treatment with systemic chemotherapy (e.g. FOLFOX, FOLFIRI, FLOT) and/or radiotherapy. Note: Participant may have undergone minor surgical procedures within the four weeks prior to enrollment, if related major organ toxicities have recovered to \<= grade 1.
* Participants must have endoscopically evaluable disease.
* Age \>=18 years.
* ECOG performance status =\<1.
* Participants must have adequate organ and marrow function as defined below:

  * White Blood Cell (WBC), \>=3,000/mm\^3
  * Hemoglobin \>8.0 d/dL, (transfusion permitted)
  * platelets, \>=100,000/mm\^3
  * total bilirubin, \< 1.5 mg/dL (except in participants with Gilbert s Syndrome, who must have a total bilirubin \< 3.0 mg/dL)
  * AST(SGOT)/ALT(SGPT), 5.0 X institutional upper limit of normal
  * serum creatinine, \< 1.6 mg/dL
* No pre-existing autoimmune or infectious conditions for which treatment with immune checkpoint blockade is contraindicated.
* Serology

  * Seronegative for HIV antibody.
  * Seronegative for hepatitis B surface antigen and seronegative for hepatitis C antibody. If hepatitis C antibody test is positive, then the participant must be tested for the presence of antigen by RT-PCR and be HCV RNA negative
* Must have a negative pregnancy test.
* Women of childbearing potential must be willing to must agree to use adequate contraception (surgical sterilization, partner vasectomy, hormonal or barrier method of birth control; abstinence) from the time of enrollment through 3 months after ipilimumab or for 5 months after nivolumab, whichever is later.
* Breastfeeding participants must be willing to discontinue breastfeeding from study treatment initiation through 3 months after ipilimumab or for 5 months nivolumab, whichever is later.
* Ability of participant to understand and the willingness to sign a written informed consent document.
* Participants with MMR proficient colon tumors must have extenuating circumstances that make surgical treatment an unacceptable option. This must be documented in the medical record. Some examples:

  * Religious or strong personal objections
  * Prior colorectal surgery, such that another resection could lead to short gut, permanent stoma or detriment to quality of life
* Participant must be co-enrolled on protocol 03-C-0277

EXCLUSION CRITERIA:

* Participants who are receiving any other investigational agents.
* Previous treatment with checkpoint inhibitors (anti-CTLA-4, anti-PD-1 or anti-PD-L1 antibodies) for the primary tumor in question.
* History of allergic reactions attributed to compounds of similar chemical or biologic composition to nivolumab and ipilimumab or other agents used in study.
* Concomitant medications, such as steroids or other immunosuppressive agents, that have the potential to affect the activity of the study agents.
* Any active or uncompensated major medical illness that would preclude major intraabdominal surgery.
* Any form of primary immunodeficiency (such as Severe Combined Immunodeficiency Disease) or any immune disorder that would be a contraindication to ICB treatment.
* Concurrent opportunistic infections
* Tumor is causing symptomatic bowel obstruction (participants with a diverting ostomy are eligible).
* History of significant autoimmune adverse events due to administration of anti-PD-1, anti-PD L1 or anti-CTLA-4 antibodies when given for prior indication.
* Participants who are medically unfit to undergo major abdominal surgery.
* Participants with tumors that are unable to be endoscopically evaluated.
* Uncontrolled intercurrent illness that would limit compliance with study requirements.