Autologous Tumor Infiltrating Lymphocytes MDA-TIL in Treating Patients With Recurrent or Refractory Ovarian Cancer, Colorectal Cancer, or Pancreatic Ductal Adenocarcinoma

Inclusion Criteria:

* Subjects must be willing and able to provide informed consent
* Clinical performance status of Eastern Cooperative Oncology Group (ECOG) 0 or 1 at enrollment and within 7 days of initiating lymphodepleting chemotherapy
* Subjects must have an area of tumor amenable to excisional biopsy (core biopsies may be allowed) for the generation of TIL separate from, and in addition to, a target lesion to be used for response assessment
* Any prior therapy directed at the malignant tumor, including radiation therapy, chemotherapy, and biologic/targeted agents must be discontinued at least 28 days prior to tumor resection for preparing TIL therapy
* Absolute neutrophil count (ANC) \>= 1000/mm\^3 (within 7 days of enrollment)
* Hemoglobin \>= 9.0 g/dL (transfusion allowed) (within 7 days of enrollment)
* Platelet count \>= 100,000/mm\^3 (within 7 days of enrollment)
* Alanine aminotransferase (ALT)/serum glutamate pyruvate transaminase (SGPT) and aspartate aminotransferase (AST)/serum glutamic-oxaloacetic transaminase (SGOT) \< 2.5 x the upper limit of normal (ULN)

  * Patients with liver metastases may have liver function test \[LFT\] =\< 5.0 x ULN (within 7 days of enrollment)
* Calculated creatinine clearance (Cockcroft-Gault) \>= 50.0 mL/min (within 7 days of enrollment)
* Total bilirubin =\< 1.5 x ULN (within 7 days of enrollment)
* Prothrombin time (PT) \& activated partial thromboplastin time (aPTT) =\< 1.5 x ULN (correction with vitamin K allowed) unless subject is receiving anticoagulant therapy (which should be managed according to institutional norms prior to and after excisional biopsy) (within 7 days of enrollment)
* Negative serum pregnancy test (female subjects of childbearing potential) (within 7 days of enrollment)
* Subjects must not have a confirmed human immunodeficiency virus (HIV) infection
* Subjects must have a 12-lead electrocardiogram (EKG) showing no active ischemia and Fridericia's corrected QT interval (QTcF) less than 480 ms
* Subjects must also have a negative dobutamine stress echocardiogram before beginning treatment
* Subjects of childbearing potential must be willing to practice an approved highly effective method of birth control starting at the time of informed consent and for 1 year after the completion of the lymphodepletion regimen. Approved methods of birth control are as follows: hormonal contraception (i.e. birth control pills, injection, implant, transdermal patch, vaginal ring); intrauterine device (IUD); tubal ligation or hysterectomy; subject/partner status post vasectomy; implantable or injectable contraceptives; and condoms plus spermicide
* Able to adhere to the study visit schedule and other protocol requirements
* Pulmonary function tests (spirometry) demonstrating forced expiratory value (FEV) 1 greater than 65% predicted or forced vital capacity (FVC) greater than 65% of predicted
* Ovarian cancer cohort only: Subjects must have high grade non-mucinous histology (carcinosarcomas are allowed)
* Ovarian cancer cohort only: Subjects must have failed at least two prior lines of chemotherapy (i.e. frontline adjuvant chemotherapy plus one additional line for recurrent/progressive disease), or have platinum resistant disease
* Colorectal cohort only: Subjects with colorectal adenocarcinoma must have metastatic disease that is considered incurable with currently available therapies and have derived maximal benefit from or have become refractory to frontline conventional therapy (e.g. leucovorin calcium \[calcium folinate\], 5-fluorouracil and oxaliplatin \[FOLFOX\], leucovorin calcium, 5-fluorouracil, and irinotecan \[FOLFIRI\]).
* Colorectal cohort only: Subjects should have low disease burden such that in the treating physician's opinion the patient would not require additional intervening treatment for 7-8 weeks (required for TIL harvest and manufacturing)
* Pancreatic adenocarcinoma cohort only: Subjects must have histologically or cytologically documented diagnosis of PDAC with oligo-metastatic disease
* Pancreatic adenocarcinoma cohort only: Subjects must have progressed on, or received maximal benefit from, front-line therapy
* Pancreatic adenocarcinoma cohort only: Patients may have received unlimited lines of prior standard of care therapy
* Pancreatic adenocarcinoma cohort only: Patients with ascites or carcinomatosis are not eligible for the study
* Pancreatic adenocarcinoma cohort only: Patients will need an albumin of \>= 3.0 mg/dL within 7 days of enrollment
* TREATMENT INCLUSION CRITERION: Within 24 h of starting lymphodepleting chemotherapy, subjects must meet the following laboratory criteria:

  * Absolute neutrophil count (ANC) \>= 1000/mm\^3
  * Hemoglobin \>= 9.0 g/dL (transfusion allowed)
  * Platelet count \>= 100,000/mm3
  * ALT/SGPT and AST/SGOT =\< 2.5 x the upper limit of normal (ULN)

    * Patients with liver metastases may have liver function tests (LFT) =\< 5.0 x ULN
  * Calculated creatinine clearance (Cockcroft-Gault) \>= 50.0 mL/min
  * Total bilirubin =\< 1.5 X ULN

Exclusion Criteria:

* Active systemic infections requiring intravenous antibiotics, coagulation disorders or other major medical illnesses of the cardiovascular, respiratory or immune system. Principal investigator (PI) or his/her designee shall make the final determination regarding appropriateness of enrollment
* Patients with active viral hepatitis
* Patients who have a left ventricular ejection fraction (LVEF) \< 45% at screening
* Patients with a history of prior adoptive cell therapies
* Persistent prior therapy-related toxicities greater than grade 2 according to Common Toxicity Criteria for Adverse Events (CTCAE) v. 4.03, except for peripheral neuropathy, alopecia, or vitiligo prior to enrollment
* Primary immunodeficiency
* History of organ or hematopoietic stem cell transplant
* Chronic steroid therapy, however prednisone or its equivalent is allowed at \< 10 mg/day
* Patients who are pregnant or nursing
* Presence of a significant psychiatric disease, which in the opinion of the principal investigator or his/her designee, would prevent adequate informed consent
* History of clinically significant autoimmune disease including active, known, or suspected autoimmune disease. Subjects with resolved side effects from prior checkpoint inhibitor therapy, vitiligo, psoriasis, type 1 diabetes or resolved childhood asthma/atopy would be an exception to this rule. Subjects that require intermittent use of bronchodilators or local steroid injections would not be excluded. Subjects with hypothyroidism stable on hormone replacement or Sjogren's syndrome will not be excluded
* History of clinically significant chronic obstructive pulmonary disease (COPD), asthma, or other chronic lung disease
* History of a second malignancy (diagnosed in the last 5 years). Exceptions include basal cell carcinoma of the skin, squamous cell carcinoma of the skin, or in situ cervical cancer that has undergone potentially curative therapy
* History of known active central nervous system metastases and/or carcinomatous meningitis. Subjects with previously treated brain metastases may participate provided they are stable (without evidence of progression by imaging for at least four weeks prior to the first dose of trial treatment and any neurologic symptoms have returned to baseline), have no evidence of new or enlarging brain metastases, and are not using steroids for at least 7 days prior to initiation of lymphodepletion
* Has received a live vaccine within 30 days prior to the initiation of lymphodepletion
* Patients who have a contraindication to or history of hypersensitivity reaction to any components or excipients of the TIL therapy or the other study drugs: NMA-LD (cyclophosphamide, mesna, and fludarabine); IL-2; any component of the TIL infusion product formulation including human serum albumin (HSA), IL-2, and dextran-40
* Any other condition that in the investigator's judgement would significantly increase the risks of participation
* Patient has any complication or delayed healing from excisional procedure that in the investigator's opinion would increase the risks of lymphodepletion, adoptive TIL therapy and adjuvant IL-2
* Patients has a decline in performance status to ECOG \> 1 (at the visit prior to admission for lymphodepletion)