Testing the Addition of Pembrolizumab, an Immunotherapy Cancer Drug to Olaparib Alone as Therapy for Patients With Pancreatic Cancer That Has Spread With Inherited BRCA Mutations

Inclusion Criteria:

* Patients must have a histologic or cytologic diagnosis of pancreatic adenocarcinoma. Patients with neuroendocrine tumors, acinar cell and adenosquamous carcinomas are excluded. All disease must be assessed and documented on the Baseline Tumor Assessment Form
* Patients must have one of the following mutations: germline mutation in BRCA 1 or 2 that was tested in a Clinical Laboratory Improvement Act (CLIA) certified lab defined as positive and/or deleterious (that is, pathogenic or likely pathogenic variant). (NOTE: Patients with tumor somatic mutations are not eligible)
* Patients must have metastatic disease and received first line platinum-based chemotherapy (i.e. fluorouracil, irinotecan, leucovorin and oxaliplatin \[FOLFIRINOX\], leucovorin calcium, 5-fluorouracil, and oxaliplatin \[FOLFOX\], gemcitabine + nab-paclitaxel + cisplatin or gemcitabine + cisplatin)
* Patients must have had a CT or MRI showing stable or responding disease on first line platinum-based chemotherapy within 30 days prior to registration
* Patients with known human immunodeficiency virus (HIV)-infection are eligible providing they are on effective anti-retroviral therapy and have undetectable viral load at their most recent viral load test and within 6 months prior to registration
* Patients with history of chronic hepatitis B virus (HBV) infection must have undetectable HBV viral load within 30 days prior to registration
* Patients with a history of hepatitis C virus (HCV) infection must have been treated and cured. For patients with HCV infection who are currently on treatment must have an undetectable HCV viral load within 30 days prior to registration
* Patients must have received at least 16 weeks of first line platinum-based chemotherapy for metastatic disease. Patients may have also received one cycle of treatment (no more than 4 weeks) with gemcitabine + nab-paclitaxel while waiting for germline test results, prior to platinum-based therapy
* Patients' last chemotherapy treatment must be within 30 days prior to registration
* Patients must have resolved or stable =\< grade 1 toxicity from prior administration of another investigational drug and/or prior anti-cancer treatment, excluding neuropathy and alopecia
* Patients must not have a known hypersensitivity to olaparib or any of the excipients of the product
* Patients must not be planning to receive strong or moderate CYP3A inhibitors or inducers while on olaparib treatment. Patients receiving strong or moderate CYP3A inhibitors must discontinue use at least 2 weeks prior to receiving olaparib. Patients receiving strong or moderate CYP3A inducers must discontinue use at least 5 weeks prior to receiving olaparib. Medications should be checked using a frequently updated medical reference for a list of drugs to avoid
* Patients must not have received live vaccines within 42 days prior to randomization and must not be planning to receive live virus or live bacterial vaccines while receiving study treatment and during the 30 day follow up period. Examples of live vaccines include, but are not limited to, the following: measles, mumps, rubella, chicken pox, shingles, yellow fever, rabies, Bacillus Calmette-Guerin (BCG), and typhoid (oral) vaccine. Seasonal influenza vaccines for injection are generally killed virus vaccines and are allowed; however, intranasal influenza vaccines (e.g., Flu-Mist) are live attenuated vaccines, and are not allowed. Coronavirus disease 2019 (COVID-19) messenger ribonucleic acid (mRNA) vaccine is allowed
* Patients must not have had prior therapy with an anti-PD-1, anti-PD-L1, or anti-PD-L2 agent, or any other immune checkpoint inhibitors
* Patients must not have had prior therapy with PARP inhibitors
* Patients must not have had a prior diagnosis of immunodeficiency or receiving systemic steroid therapy (defined as \>= 10 mg prednisone or equivalent) or any other form of immunosuppressive therapy within 7 days prior to the first dose of trial treatment
* Zubrod performance status of 0-1
* Patients must be \>= 18 years old
* Patients must have a complete medical history and physical exam within 28 days prior to registration
* Absolute neutrophil count \>= 1.5 x 10\^3/uL (within 14 days of registration)
* Platelets \>= 100 x 10\^3/uL (within 14 days of registration)
* Total bilirubin =\< 1.5 institutional upper limit of normal (ULN) (within 14 days of registration)
* Aspartate aminotransferase (AST)/alanine aminotransferase (ALT) =\< 3 x institutional ULN (within 14 days of registration)
* Albumin \>= 3.0 g/dL (within 14 days of registration)
* Hemoglobin \>= 9.0 g/dL (within 14 days of registration)
* Creatinine clearance (Cockcroft \_Gault) \> 50 mg/dL (within 14 days of registration)
* Participants must have a serum creatinine =\< the institutional (I)ULN OR measured OR calculated creatinine clearance \>= 50 mL/min using the following Cockcroft-Gault Formula. This specimen must have been drawn and processed within 14 days prior to registration
* Patients must have CA19-9 obtained within 42 days prior to registration
* Patients must be able to swallow and retain oral medications and have no known gastrointestinal disorders likely to interfere with absorption of the study medication
* Participants with a prior or concurrent malignancy whose natural history or treatment (in the opinion of the treating physician) does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial provided it does not require concurrent therapy
* Participants must not be pregnant or nursing due to the possibility of harm to the fetus or nursing infant from this treatment regimen. Women of childbearing potential must have a negative urine or serum pregnancy test within 28 days prior to registration. Women/men of reproductive potential must have agreed to use an effective contraceptive method for the course of the study through 6 months after the last dose of study medication. A woman is considered to be of "reproductive potential" if she has had menses at any time in the preceding 12 consecutive months. In addition to routine contraceptive methods, "effective contraception" also includes heterosexual celibacy and surgery intended to prevent pregnancy (or with a side-effect of pregnancy prevention) defined as a hysterectomy, bilateral oophorectomy or bilateral tubal ligation. However, if at any point a previously celibate participant chooses to become heterosexually active during the time period for use of contraceptive measures, he/she is responsible for beginning contraceptive measures. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately
* Patients must not have a history of (non-infectious) pneumonitis that required steroids or current pneumonitis
* Patients must not have an active infection requiring systemic therapy
* Patients must not have active autoimmune disease that has required systemic treatment in past 2 years (i.e., with use of disease modifying agents, corticosteroids or immunosuppressive drugs). Replacement therapy (e.g., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment
* Patients must be offered the opportunity to participate in specimen banking of formalin-fixed paraffin-embedded (FFPE) tissue and whole blood. If a patient is unable to submit archival tissue, should the patient need to undergo a standard of care biopsy per National Comprehensive Cancer Network (NCCN) guidelines, patients must then be offered the opportunity to submit the fresh tumor tissue from that biopsy. With participant consent, specimens must be collected and submitted via the Southwest Oncology Group (SWOG) Specimen Tracking System
* Patients must be informed of the investigational nature of this study and must sign and give informed consent in accordance with institutional and federal guidelines. For participants with impaired decision making capabilities, legally authorized representatives may sign and give informed consent on behalf of study participants in accordance with applicable federal, local, and Canada Industrial Relations Board (CIRB) regulations
* As a part of the Oncology Patient Enrollment Network (OPEN) registration process the treating institution's identity is provided in order to ensure that the current (within 365 days) date of institutional review board approval for this study has been entered in the system