Danvatirsen and Durvalumab in Treating Patients With Advanced and Refractory Pancreatic, Non-Small Cell Lung Cancer, and Mismatch Repair Deficient Colorectal Cancer

Inclusion Criteria:

* The patient/legal representative must be able to read and understand the informed consent form (ICF) and must have been willing to give written informed consent and any locally required authorization (e.g., Health Insurance Portability and Accountability Act in the United States of America \[USA\]; European Union Data Privacy Directive in the European Union \[EU\]) before any study-specific procedures, including screening evaluations, sampling, and analyses
* Has a histological confirmation of pancreatic cancer, mismatch deficient colorectal cancer, or non-small cell lung cancer (NSCLC) that is refractory to standard therapy or for which no standard of care regimen currently exists
* Has an Eastern Cooperative Oncology Group (ECOG) performance score (PS) score of 0 or 1
* Has measurable disease, defined as at least 1 lesion that can be accurately measured in at least 1 dimension (longest diameter to be recorded) with a minimum size of 10 mm by computerized tomography (CT) scan, except lymph nodes which must have minimum short axis size of 15 mm (CT scan slice thickness no greater than 5 mm in both cases). Indicator lesions must not have been previously treated with surgery, radiation therapy, or radiofrequency ablation unless there is documented progression after therapy
* Transfusions intended to elevate any parameters below solely for the intent of meeting study eligibility are not permitted
* Leukocytes \>= 3000 mcL
* Absolute neutrophil count \>= 1500 mcL
* Platelets \> = 100 000 mcL
* Hemoglobin \>= 9 g/dL
* Total bilirubin =\< 1.5 x upper limit of normal (ULN)
* Total bilirubin =\< 3 x ULN in patients with documented Gilbert's syndrome (unconjugated hyperbilirubinemia) or in the presence of liver metastases
* Alanine aminotransferase (ALT) and aspartate aminotransferase (AST) =\< 2.5 x ULN if no demonstrable liver metastases or =\< 5 x ULN in the presence of liver metastases
* Creatinine within normal limits OR, for patients with levels above institutional normal: creatinine clearance measured by 24-hour urine collection \>= 60 mL/min, OR calculated corrected creatinine clearance \>= 60 mL/min/1.73 m\^2 using the Cockcroft-Gault formula (Cockcroft and Gault 1976) corrected for the body surface area
* Women of childbearing potential and men who are sexually active with a female partner of childbearing potential must be surgically sterilized, practicing abstinence, or agree to use 2 birth control methods before study entry, for the duration of study participation, and for 20 weeks after the final dose of study drug; cessation of birth control after this point should be discussed with a responsible physician. Women of childbearing potential are defined as those who are not surgically sterile (i.e., bilateral tubal ligation, bilateral oophorectomy, or complete hysterectomy) or postmenopausal (defined as 12 months with no menses without an alternative medical cause). Two methods of contraception which are considered accurate per protocol must be combined. Periodic abstinence, the rhythm method, and the withdrawal method are not acceptable methods of birth control
* Women of childbearing potential also may not be breast feeding and must have a negative serum or urine pregnancy test within 72 hours before the start of study treatment
* The patient/legal representative must be willing to provide written consent for collection of formalin fixed paraffin-embedded blocks or slides from archival diagnostic histology samples, where available

Exclusion Criteria:

* Has a spinal cord compression unless asymptomatic, radiographically stable over the last 4 weeks, and not requiring steroids for at least 4 weeks before the start of study treatment
* Presently has a second malignancy other than squamous cell carcinoma of the head and neck (SCCHN), or history of treatment for invasive cancer other than SCCHN in the past 3 years. Exceptions are:

  * Previously treated in-situ carcinoma (i.e., noninvasive)
  * Cervical carcinoma stage 1B or less
  * Noninvasive basal cell and squamous cell skin carcinoma
  * Radically treated prostate cancer (prostatectomy or radiotherapy) with normal prostate-specific antigen, and not requiring ongoing antiandrogen hormonal therapy
* Patients must have completed previous cancer-related treatments before enrollment. Any concurrent chemotherapy, radiotherapy, immunotherapy, or biologic, or hormonal therapy for cancer excludes the patient (concurrent use of hormones for noncancer-related conditions \[e.g., insulin for diabetes or hormone replacement therapy\] is acceptable). The following intervals between end of the prior treatment and first dose of study drug must be observed:

  * Port-a-cath placement: no waiting required
  * Minor surgical procedures: \>= 7 postoperative days
  * Major surgery: \>= 4 weeks
  * Radiotherapy: \>= 4 weeks
  * Chemotherapy: \>= 4 weeks
  * Immunotherapy or investigational anticancer therapy with agents other than monoclonal antibodies (mAbs): \>= 4 weeks
  * Immunotherapy or investigational anticancer therapy with mAbs: \>= 6 weeks
  * Immunosuppressive medication: \>= 4 weeks with the exceptions of intranasal or inhaled corticosteroids or systemic corticosteroids at physiologic doses not to exceed 10 mg/day of prednisone or equivalent
* Is still experiencing toxicity related to prior treatment and assessed as Common Terminology Criteria for Adverse Events (CTCAE) grade \> 1. Exceptions are alopecia and/or anorexia. The eligibility of patients who are still experiencing irreversible toxicity that is not reasonably expected to be exacerbated by the study drugs in this study (e.g., hearing loss) must be reviewed and approved by both the principal investigator and medical monitor
* Has experienced immune-related adverse events (AEs) (irAEs) while receiving prior immunotherapy (including anti-CTLA4 treatment) and assessed as CTCAE grade \>= 3
* Has active or prior documented autoimmune disease within the past 2 years with the exceptions of vitiligo, Grave's disease, and/or psoriasis not requiring systemic treatment
* Has active or prior documented inflammatory bowel disease (e.g., Crohn's disease, ulcerative colitis)
* Has a history of primary immunodeficiency
* Has undergone an organ transplant that requires use of immunosuppressive treatment
* Has a history of interstitial lung disease or pneumonitis from any cause
* Has a history of allergic reactions attributed to the study treatments (AZD9150 or MEDI4736), their compounds, or agents of similar chemical or biologic composition (e.g., antibody therapeutics)
* Suffers from a comorbidity that in the opinion of the investigator renders the patient unsuitable for participation in the study. Such comorbidity may include, but is not limited to, uncontrolled intercurrent illness such as active infection, severe active peptic ulcer disease or gastritis, myocardial infarction within 6 months before entry, congestive heart failure, symptomatic congestive heart failure, active cardiomyopathy, unstable angina pectoris, cardiac arrhythmia, uncontrolled hypertension, or psychiatric illness/social situations that would limit compliance with study requirements
* As judged by the investigator, has any evidence of severe or uncontrolled systemic diseases such as active bleeding diatheses, is positive for human immunodeficiency virus (HIV), or has active hepatitis B virus (HBV) and/or hepatitis C virus (HCV)
* Has a known history of tuberculosis
* Has a condition that, in the opinion of the investigator, would interfere with the evaluation of the study drugs or the interpretation of patient safety or study results
* Has received a live attenuated vaccine within 28 days before the first dose of study drug
* Judgment by the investigator that the patient should not participate in the study if the patient is unlikely to comply with study procedures, restrictions, and requirements
* Patients with clinically active brain metastases (known or suspected) are excluded unless the brain metastases have been previously treated and are considered stable. Stable brain metastases are defined as no change on CT scan or magnetic resonance imaging (MRI) scan for a minimum of 2 months AND no change in steroid dose for a minimum of 4 weeks, unless change due to intercurrent infection or other acute event
* Female subjects who are pregnant, breast-feeding or male or female patients of reproductive potential who are not employing an effective method of birth control