BMS-986156, Ipilimumab, and Nivolumab With or Without Stereotactic Body Radiation Therapy in Treating Patients With Advanced or Metastatic Lung/Chest or Liver Cancers

Inclusion Criteria:

* Patients must have histological confirmation of solid metastatic cancer with at least one metastatic or primary lesion in the liver or lung/chest, except for group 1.
* Patients who have completed prior systemic anti-cancer therapies, an interval of 5 drug half-lives or 4-weeks whichever is shorter, is required, prior to enrollment on study. Note: patients with anaplastic thyroid will be waived from this inclusion criteria given the rapid trajectory of their disease
* All patients must have at least one metastatic or primary lesion within the lung/chest or liver located in an anatomical location amenable to SBRT treatment with 50 Gy in 4 fractions or with 60 Gy in 10 fractions, except for group 1.
* Repeat radiation in fields previously radiated will be allowed at the discretion of the treating physician
* Eastern Cooperative Oncology Group (ECOG) performance status =\< 2 (Karnofsky \> 60%)
* Total bilirubin =\< 2.0 mg/dL (does NOT apply to patients with Gilbert's syndrome) (use of growth factors or blood transfusion to achieve these requirements is not allowed 2 weeks prior to study enrollment)
* Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) \< 2.5 x institutional upper limit of normal (use of growth factors or blood transfusion to achieve these requirements is not allowed 2 weeks prior to study enrollment)
* White blood count (WBC) \>= 2500/uL (use of growth factors or blood transfusion to achieve these requirements is not allowed 2 weeks prior to study enrollment)
* Absolute neutrophil count (ANC) \>= 1000/uL (use of growth factors or blood transfusion to achieve these requirements is not allowed 2 weeks prior to study enrollment)
* Platelets \>= 75K (use of growth factors or blood transfusion to achieve these requirements is not allowed 2 weeks prior to study enrollment)
* Hemoglobin \>= 9 g/dL (use of growth factors or blood transfusion to achieve these requirements is not allowed 2 weeks prior to study enrollment)
* Creatinine =\< 2.0 x upper limit of normal (ULN) (use of growth factors or blood transfusion to achieve these requirements is not allowed 2 weeks prior to study enrollment)
* Patients must be willing and able to review, understand, and provide written consent before starting therapy
* Patients with brain metastasis will be included as long as they are free of neurologic symptoms related to metastatic brain lesions and who do not require or receive systemic corticosteroid therapy, \> 10 mg/day in the 14 days prior to beginning the trial (=\< 10 mg steroid, e.g.: prednisone, is allowed). Patients with stable brain metastases (clinically and radiographically) for \>= 4 weeks to enroll on the protocol.
* Patients that have previously progressed on immunotherapy such as ipilimumab, anti-PD-I, anti-PDL-1 or talimogene laherparepvec (T-VEC) will be eligible.

Exclusion Criteria:

* Serious autoimmune disease at the discretion of the treating attending: patients with a history of active serious inflammatory bowel disease (including Crohn's disease and ulcerative colitis) and autoimmune disorders such as rheumatoid arthritis, systemic progressive sclerosis (scleroderma), systemic lupus erythematosus or autoimmune vasculitis (e.g., Wegener's granulomatosis) are excluded from this study
* Active diverticulitis, intra-abdominal abscess, gastrointestinal (GI) obstruction, abdominal carcinomatosis or other known risk factors for bowel perforation
* Any underlying medical or psychiatric condition, which in the opinion of the Investigator, will make the administration of study drug hazardous or obscure the interpretation of adverse events (AEs): e.g. a condition associated with frequent diarrhea or chronic skin conditions, recent surgery or colonic biopsy from which the patient has not recovered, or partial endocrine organ deficiencies
* Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, history of congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements
* Known active human immunodeficiency virus (HIV), hepatitis B, or hepatitis C that has not been documented to be stable
* Any non-oncology vaccine therapy used for prevention of infectious diseases (for up to one month prior to or after any dose of ipilimumab)
* Concomitant therapy with any of the following: IL-2, interferon or other non-study immunotherapy regimens; cytotoxic chemotherapy; immunosuppressive agents; other investigational therapies; or chronic use of systemic corticosteroids while receiving ipilimumab (as long as steroid replacement is significantly greater than what is required for physiologic replacement, i.e. in hypothyroidism)
* Pregnant women are excluded from this study. Women of child-bearing potential (WOCBP) must have a pregnancy test every 4 weeks and WOCBP must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of human chorionic gonadotropin \[HCG\]) within 24 hours prior to the start of immunotherapy drugs (ipilimumab/nivolumab/BMS-986156), during the course of the treatment and 160 days AFTER the last dose of study drug you should not get pregnant or breast feed. In the case of male participants, during the course of treatment and 220 days AFTER the last dose of immunotherapy you should not father a child (condom use is mandatory, even if vasectomized) or donate sperm. For contraception guidelines please see protocol
* History of or current immunodeficiency disease or prior treatment compromising immune function at the discretion of the treating physician
* Prior allogeneic stem cell transplantation
* Patients who were intolerant to previous immuno-oncology (IO) drugs should be excluded