Anti-CTLA4-NF mAb (BMS986218), Nivolumab, and Stereotactic Body Radiation Therapy for the Treatment of Metastatic Solid Malignancies

Inclusion Criteria:

* Patients must have histological confirmation of solid metastatic cancer with at least one metastatic or primary lesion in the bone, adrenal, liver or lung/chest.
* Patients who have completed prior systemic anti-cancer therapies, an interval of 5 drug half-lives or 4-weeks whichever is shorter, is required, prior to enrollment on study. Note: patients with anaplastic thyroid carcinoma will be waived from this inclusion criteria given the rapid trajectory of their disease. In addition, patients with metastatic castration-resistant prostate cancer (CRPC) will be allowed to continue their maintenance therapy with gonadotropin-releasing hormone (GnRH) analogue agents and supportive bone therapy (e.g., denosumab \[XGEVA\], zoledronic acid \[Zometa\]).
* All patients must have at least one metastatic or primary lesion within the bone, lung/chest, liver or adrenal located in an anatomical location amenable to radiation treatment with either 50 Gy in 4 fractions, 60 Gy in 10 fractions, or 30 Gy in 5 fx. Patients may not have more then 80% liver displaced with cancer.
* Repeat radiation in fields previously radiated will be allowed at the discretion of the treating physician
* Age \>= 18 years
* Women of childbearing potential (WOCBP) must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of human chorionic gonadotropin) within 24 hours prior to the start of study treatment. Subjects of childbearing potential are those who have not been surgically sterilized or have not been in postmenopausal state. Females in postmenopausal state under the age of 55 years must have a serum follicle stimulating hormone, (FSH) level \> 40 mIU/mL to confirm menopause
* Women must not be breastfeeding for the duration of the study and 105 days AFTER completion of study treatment for BMS-986218 monotherapy treatment, 155 days AFTER completion of combination therapy (BMS-986218 + nivolumab) or 5 months AFTER completion of nivolumab maintenance treatment
* WOCBP receiving monotherapy or combination therapy treatment with BMS-986218 must agree to follow instructions for method(s) of contraception or be surgically sterile, or abstain from heterosexual activity for the duration of the study and 105 days AFTER completion of study treatment for BMS-986218 monotherapy treatment, 155 days AFTER completion of combination therapy (BMS-986218 + nivolumab) or 5 months AFTER completion of nivolumab maintenance treatment. WOCBP who are continuously not heterosexually active are exempt from contraceptive requirements, but should still undergo pregnancy testing as described in this section. In the case of male participants, during the course of treatment and 165 days AFTER the end of BMS-986218 monotherapy treatment or 215 days AFTER the end of combination therapy treatment (BMS-986218 + nivolumab) or 7 months AFTER completion of nivolumab maintenance treatment the participant should not father a child (condom use is mandatory, even if vasectomized) or donate sperm. Local laws and regulations may require use of alternative and/or additional contraception methods
* Male participants who are sexually active with WOCBP must agree to follow instructions for methods of contraception during monotherapy treatment with BMS-986218 plus 5 half-lives (75 days) of BMS-986218 plus 90 days for a total of 165 days after the end of BMS-986218 monotherapy treatment or 215 days after the end of combination therapy treatment (BMS-986218 + nivolumab), or 7 months AFTER completion of nivolumab maintenance treatment. In addition, male participants must be willing to refrain from sperm donation during this time
* Investigators shall counsel WOCBP, and male participants who are sexually active with WOCBP, on the importance of pregnancy prevention and the implications of an unexpected pregnancy. Investigators shall advise on the use of highly effective methods of contraception that have a failure rate of \< 1% when used consistently and correctly
* Eastern Cooperative Oncology Group (ECOG) performance status 0-1 (Karnofsky \> 60%)
* Patients must have normal organ and marrow function as defined below: (use of growth factors or blood transfusion to achieve these requirements is not allowed 2 weeks prior to study enrollment). The blood test for the following analytes will be performed at screening/baseline and at the first day of every BMS-986218 therapy cycle
* Total bilirubin =\< 2.0 x upper limit of normal (ULN), except participants with Gilbert's syndrome who must have normal direct bilirubin
* Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamic pyruvic transaminase \[SGPT\]) =\< 2.5 X institutional upper limit of normal
* Whole blood cell (WBC) \>= 2500/uL
* Absolute neutrophil count (ANC) \>= 1000/uL
* Platelets \>= 75K
* Hemoglobin \>= 9 g/dL
* Creatinine =\< 1.5 x ULN, or creatinine clearance (CrCl) \>= 40 mL/min (measured using the Cockcroft-Gault formula)
* Patients must be willing and able to review, understand, and provide written consent before starting therapy
* Patients with brain metastasis will be included as long as they are free of neurologic symptoms related to metastatic brain lesions and do not require or receive steroid therapy or brain metastasis. We will allow patients with stable brain metastases (no radiographic progression for at least 4 weeks following radiation and/or surgery or 4 weeks of observation), no longer taking steroids for at least 2 weeks prior to first dose of study treatment, and with no new neurological signs and symptoms (clinically and radiographically) for \>= 4 weeks to enroll on the protocol
* Patients that have previously progressed on immunotherapy such as ipilimumab, anti-PD-I, anti-PDL-1 or talimogene laherparepvec (T-VEC) will be eligible, but progression on immunotherapy is not required

Exclusion Criteria:

* Patients with active, known, or suspected with autoimmune disorders or those who are at risk for flare of auto-immunity or immune-related diseases
* Participants with well controlled asthma and/or mild allergic rhinitis (seasonal allergies) are eligible
* Participants with the following disease conditions are also eligible:

  * Vitiligo.
  * Type 1 diabetes mellitus.
  * Residual hypothyroidism due to autoimmune condition only requiring hormone replacement.
  * Euthyroid participants with a history of Grave's disease (participants with suspected autoimmune thyroid disorders must be negative for thyroglobulin and thyroid peroxidase antibodies and thyroid stimulating immunoglobulin \[Ig\] prior to the first dose of study treatment).
  * Psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger are permitted to enroll
* Active diverticulitis, intra-abdominal abscess, gastrointestinal (GI) obstruction, abdominal carcinomatosis or other known risk factors for bowel perforation
* Any underlying medical or psychiatric condition, which in the opinion of the Investigator, will make the administration of study drug hazardous or obscure the interpretation of adverse event (AE)s: e.g. a condition associated with frequent diarrhea or chronic skin conditions, recent surgery or colonic biopsy from which the patient has not recovered, or partial endocrine organ deficiencies
* Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, history of congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements
* Known human immunodeficiency virus (HIV) positive status or with positive test for hepatitis B surface antigen. Patients with hepatitis C antibody (Ab) positive will be considered for enrollment only if quantitative polymerase chain reaction (PCR) is negative
* Any non-oncology vaccine therapy used for prevention of infectious diseases (for up to one month prior to or after any dose of BMS-986218). The use of inactivated seasonal influenza vaccines (e.g., Fluzone), will be permitted on study without restriction
* Concomitant therapy with any of the following: IL-2, interferon or other non-study immunotherapy regimens; cytotoxic chemotherapy; immunosuppressive agents; other investigational therapies; or chronic use of systemic corticosteroids while receiving BMS-986218 (as long as steroid replacement is greater than what is required for physiologic replacement, i.e. in hypothyroidism, adrenal insufficiency)
* History of or current immunodeficiency disease or prior treatment compromising immune function at the discretion of the treating physician
* Prior allogeneic stem cell transplantation or organ allograft
* Patients who were intolerant to previous immuno-oncology (IO) drugs with side effects (grade 4 or greater) that were unable to be resolved with treatment, should be excluded
* Patients that have previously received or progressed on any anti-CTLA4-NF drug
* Absolute lymphocyte count below 0.4 x 10\^9/L