Testing the Combination of Anetumab Ravtansine With Either Nivolumab, Nivolumab and Ipilimumab, or Gemcitabine and Nivolumab in Advanced Pancreatic Cancer

Inclusion Criteria:

* Patients must have histologically or cytologically confirmed pancreatic adenocarcinoma that is metastatic or unresectable or recurrent
* Only subjects with positive mesothelin expression (Ventana mesothelin \[MSLN\]- immunohistochemistry \[IHC\]; Negative=H-score =\< 10) are eligible. This is to be performed centrally. For dose escalation cohorts, patients with mesothelin expression in \>= 5% of tumor cells are eligible. For dose expansion, patients must have moderate or strong tumor mesothelin expression defined as \>= 30% of tumor cells with mesothelin expression of 2+/3 on immunohistochemical staining
* Patients must be \>= 18 years of age
* Patients must have received and either progressed or been intolerant to at least 1 systemic therapy
* Life expectancy of at least 3 months
* Eastern Cooperative Oncology Group (ECOG) performance status score 0-1 (Karnofsky \>= 80%)
* Prior anti-cancer treatments are permitted (i.e. chemotherapy, including gemcitabine and nab-paclitaxel; radiotherapy; hormonal, or immunotherapy with the exception of anti-CTLA4, anti-PD1/PD-L1, and combination of anti-CTLA4 and anti-PD1/PD-L1) providing toxicity (except for alopecia) related to prior anti-cancer therapy and/or surgery have either resolved, improved to baseline or G1
* At least one (1) measurable lesion at baseline by computed tomography (CT) or magnetic resonance imaging (MRI) as per RECIST version (v)1.1; measurable disease is a requirement in both dose escalation phase and dose expansion phase

  * Note: Measurable lesions may be in an irradiated field as long as there is documented progression and the lesion(s) can be reproducibly measured
* At least one lesion safely accessible for biopsy unless medically contraindicated; biopsies are mandatory both in dose escalation and in dose expansion; in dose escalation and in expansion the following biopsies are optional: at baseline and at progression; biopsy could be: core needle or excisional or punch biopsy. Irradiated lesions can be biopsied if tumor growth is confirmed
* Patients must have archival tumor tissue for mesothelin expression and correlative biomarker studies; subjects must consent to provide tumor blocks or slides and the availability of the tissue must be confirmed prior to subjects receiving study medication; if an archived tumor specimen is unavailable or unsuitable for correlative biomarker studies, a pre-treatment fresh tumor biopsy is required
* Women of childbearing potential (WOCBP) must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of human chorionic gonadotropin \[HCG\]) within 24 hours of study enrollment or randomization; WOCBP must agree to appropriate methods of contraception for the duration of treatment and for 6 months after completion of treatment; males who are sexually active with a partner of childbearing potential must agree to appropriate methods of contraception for the duration of treatment. For all male patients, prior to treatment, advice should be sought for conserving sperm due to the chance of irreversible infertility as a consequence of treatment; genetic consultation is recommended if the patient wishes to have children after ending treatment; the investigator or a designated associate is requested to advise the patient how to achieve highly effective birth control

  * Highly effective (failure rate of less than 1% per year) contraception methods include:

    * Combined (estrogen and progesterone containing: oral, intravaginal, transdermal) and progesterone-only (oral, injectable, implantable) hormonal contraception associated with inhibition of ovulation
    * Intrauterine device (IUD) or intrauterine hormone-releasing system (IUS)
    * Bilateral tubal occlusion or vasectomized partner (provided that partner is the sole sexual partner and has received medical assessment of the surgical success)
    * Sexual abstinence (reliability to be evaluated in relation to the duration of the clinical trial and the preferred and usual lifestyle of the patient)
  * Male patients with a female partner of childbearing potential must use a condom and ensure that an additional form of contraception is also used during

    * Note: a woman is considered WOCBP, i.e. fertile, following menarche and until becoming postmenopausal unless permanently sterile; permanent sterilization methods include but are not limited to hysterectomy, bilateral salpingectomy and bilateral oophorectomy
  * A postmenopausal state is defined as no menses for 12 months without an alternative medical cause; a high follicle stimulating hormone (FSH) level in the postmenopausal range may be used to confirm a postmenopausal state in women not using hormonal contraception or hormonal replacement therapy; a man is considered fertile after puberty unless permanently sterile by bilateral orchiectomy
* Leukocytes \>= 3,000/mcL
* Absolute neutrophil count (ANC) \>= 1,500/mcL
* Platelets \>= 100,000/mcL
* Hemoglobin \>= 9 g/dL

  * Patients must have not had a transfusion in the 2 weeks preceding this hemoglobin (Hb) measurement
* Total bilirubin =\< institutional upper limit of normal (ULN)
* Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 3 x institutional ULN
* Creatinine =\< institutional ULN OR glomerular filtration rate (GFR) \>= 60 mL/min/1.73 m\^2 unless data exists supporting safe use at lower kidney function values, no lower than 30 mL/min/1.73 m\^2
* Albumin \>= 2.5 mg/dL
* Ability to understand and the willingness to sign a written informed consent document

Exclusion Criteria:

* Patients not recovered from clinically significant adverse events of their most recent therapy/intervention prior to enrollment. Concurrent enrollment in a non-interventional clinical study or the follow-up period of an interventional study is allowed.
* Untreated central nervous system (CNS) metastatic disease, leptomeningeal disease, or cord compression; subjects with previously treated brain metastases may participate provided they are stable (without evidence of progression by imaging for at least six weeks prior to the first dose of trial treatment and any neurologic symptoms have returned to baseline), have no evidence of new or enlarging brain metastases, and are not using steroids for at least 14 days prior to trial treatment
* Evidence of uncontrolled, active infection, requiring parenteral anti-bacterial, anti-viral or anti-fungal therapy (washout: 7 days prior to cycle 1 day 1 \[C1D1\])
* Patients are prohibited from receiving the following therapies during the screening and treatment phase of this trial:

  * Antineoplastic systemic chemotherapy or biological therapy
  * Radiation therapy

    * Note: Radiation therapy to a symptomatic solitary lesion or to the brain may be considered on an exceptional case by case basis after consultation with Cancer Therapy Evaluation Program (CTEP); the patient must have clear measurable disease outside the radiated field; administration of palliative radiation therapy will be considered clinical progression for the purposes of determining progression free survival (PFS)
  * Live vaccines within 30 days prior to the first dose of trial treatment and while participating in the trial; examples of live vaccines include, but are not limited to, the following: measles, mumps, rubella, chicken pox, yellow fever, rabies, Bacillus Chalmette-Guerin (BCG), typhoid (oral) vaccine, and intranasal influenza vaccines (e.g., Flu-Mist)
* Current or prior use of systemic immunosuppressive medication (except corticosteroids at physiological doses, not exceeding 10 mg prednisone-equivalent day) within 10 days before the first dose of study medication; intranasal, inhaled, topical, or local steroid injections are allowed; steroids as premedication for hypersensitivity reactions (i.e. CT scan premedication) are allowed; systemic glucocorticoids used to modulate symptoms from an event of suspected immunologic etiology are permitted
* Any major surgery within 4 weeks of study drug administration
* Concomitant second malignancies (except adequately treated squamous cell carcinoma \[SCC\] or basal cell carcinoma \[BCC\] skin cancers or in situ bladder, breast or cervical cancers) within the last 3 years prior to study entry
* Uncontrolled or significant cardiovascular disease, including but not limited to ongoing or active symptomatic congestive heart failure, uncontrolled hypertension, unstable angina pectoris, unstable cardiac arrhythmia
* National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE) version (v)5 \>= grade (G)2 peripheral neuropathy (sensory or motor)
* Patients with corneal epitheliopathy and at the discretion of the ophthalmologist any other eye disorder

  * Note: Low grades of superficial punctate keratitis, within the range seen in the normal population, should not lead to the exclusion of the patient
* Active or prior documented inflammatory bowel disease (i.e. ulcerative colitis)
* Active or prior documented autoimmune disease within the past 2 years

  * Note: subjects with vitiligo, Grave's disease, psoriasis not requiring systemic treatment or hypothyroidism (i.e. following Hashimoto syndrome) stable on hormone replacement are not excluded
* Recent history or current evidence of bleeding disorder (i.e. any CTCAE G \>= 2 hemorrhage/bleeding event within 28 days before the start of treatment)
* Active human immunodeficiency virus (HIV), hepatitis B or C infection; HIV-positive patients on antiretroviral therapy with undetectable viral load will not be excluded from the trial; subjects with treated hepatitis B or C with unquantifiable viral loads and no organ compromise are not excluded
* Any condition that, in the opinion of the investigator, would interfere with evaluation of study treatment or interpretation of patient safety or study results
* Pregnant women are excluded from this study because of the potential for teratogenic or abortifacient effects; because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother, breastfeeding should be discontinued for at least 6 months after last dose of study drugs; these potential risks may also apply to other agents used in this study; should a patient become pregnant or suspect she is pregnant while she is participating in this study, the patient should inform the treating physician immediately
* Participants who have had prior organ transplants (i.e. renal, lung, heart) due to the potential for increased rejection with immunotherapy
* Patients taking strong CYP3A4 inhibitors or strong CYP3A4 inducers within 2 weeks before the start of study treatment are excluded; consumption of grapefruit or its juice, and other fruit/juices which are strong CYP3A4 inhibitors within 2 weeks of study treatment is also not permitted; examples of strong CYP3A4 inhibitors include the following: indinavir, ritonavir, clarithromycin, itraconazole, ketoconazole, nefazodone, and saquinavir; examples of strong CYP3A4 inducers include the following: carbamazepine, rifampin, phenytoin, St. John's wort, and phenobarbital; these lists are not exhaustive