Hypofractionated Radiation Therapy and Bintrafusp Alfa for the Treatment of Advanced Intrahepatic Cholangiocarcinoma

Inclusion Criteria:

* Patients must be male or female \>= 18 years of age
* Patients with histologically or cytologically confirmed intrahepatic cholangiocarcinoma. There must be at least two measurable tumors. One larger mass that will be radiated and a secondary metastatic site that is amenable to biopsies
* Patients must have received at least one standard first-line chemotherapy regimen or have refused chemotherapy
* Patients with measurable disease assessed at baseline by computed tomography (CT) (or magnetic resonance imaging \[MRI\] where CT is contraindicated) will be entered in this study
* Eastern Cooperative Oncology Group (ECOG) performance status 0-1 (Karnofsky \> 70)
* Leukocytes \>= 3,000 cells/mm\^3
* Absolute neutrophil count \>= 1,500 cells/mm\^3
* Platelets \>= 100,000 cells/mm\^3
* Hemoglobin \>= 9 g/dl (no blood transfusions within 4 weeks prior to enrollment)
* Total bilirubin \< 1.5 x institutional upper limit of normal (IULN)
* Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x IULN without liver metastasis, =\< 5 x IULN for patients with liver metastasis
* Creatinine within normal institutional limits OR creatinine clearance \>= 60 mL/min/1.73 m\^2 for patients with creatinine levels above institutional normal
* Women of childbearing potential (defined as not post-menopausal for 65 days or no previous surgical sterilization) and fertile men must agree to use adequate contraception for the duration of study participation
* All female patients of childbearing potential (defined as not post-menopausal for 65 days or no previous surgical sterilization) must agree to use adequate birth control during study treatment and for 90 days after the last dose of study drug and have a negative serum pregnancy test at screening
* Fertile males must be willing to employ adequate means of contraception during study treatment and for 3 months after the last dose of study drug
* Male subjects must agree to refrain from sperm donation during the study and for 125 days after the last dose of study drugs
* Ability to read and/or understand the details of the study and provide written evidence of informed consent as approved by Institutional Review Board (IRB)/ Scientific Review Committee (SRC)

Exclusion Criteria:

* Patients who are pregnant or lactating
* Patients with uncontrolled intercurrent illness including symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia and myocardial infarction (MI) within 3 months of initiation of therapy
* Patients with recent bacterial, viral, or fungal infection(s) requiring systemic antibiotic therapy within one month of enrolling
* Patient has undergone major surgery within 4 weeks prior to day 1 of treatment in this study. Diagnostic or minimally invasive surgery (i.e., done to obtain a biopsy for diagnosis without removal of an organ or to place an abdominal spacer) are acceptable at physician discretion
* Patient received radiotherapy, surgery, chemotherapy, or an investigational therapy within 2 weeks prior to study entry weeks
* Patient has serious medical risk factors involving any of the major organ systems such that the investigator considers it unsafe for the patient to receive an experimental research drug
* Serious psychiatric or medical conditions that could interfere with treatment
* Major bleeding in the last 4 weeks
* Receipt of prior immune checkpoint inhibitors (e.g., anti-CTLA-4, anti-programmed cell death receptor 1, anti-programmed cell death ligand 1 \[anti-PD-L1\], and any other antibody or drug specifically targeting T-cell costimulation)
* Receiving an immunologically based treatment for any reason, including chronic use of systemic steroid at doses \>= 7.5 mg/day prednisone equivalent within 14 days prior to the first dose of study treatment. Use of inhaled or topical steroids or systemic corticosteroids \< 7.5 mg is permitted
* Receipt of any anticancer medication in the 21 days prior to receiving the first dose of study medication or any unresolved toxicity (\> grade 1) from previous anticancer therapy, except for stable chronic toxicities not expected to resolve, such as peripheral neurotoxicity. Prior treatment with nitrosoureas (e.g., carmustine or lomustine) require a 6-week washout prior to the first dose of study treatment
* Untreated central nervous system (CNS) metastases, or CNS metastases that have progressed (e.g., evidence of new or enlarging CNS metastasis or new neurological symptoms attributable to CNS metastases). Subjects with treated and clinically stable CNS metastases and off all corticosteroids for at least 2 weeks are eligible
* Any active or inactive autoimmune process (e.g. rheumatoid arthritis, moderate or severe psoriasis, multiple sclerosis, inflammatory bowel disease) or who are receiving systemic therapy for an autoimmune or inflammatory disease

  * Note: Exceptions include subjects with vitiligo, hypothyroidism stable on hormone replacement, controlled asthma, type I diabetes, Graves' disease, or Hashimoto's disease, or with medical monitor approval