Testing Nivolumab and Ipilimumab Immunotherapy With or Without the Targeted Drug Cabozantinib in Recurrent, Metastatic, or Incurable Nasopharyngeal Cancer

Inclusion Criteria:

* Patients must have histologically documented nasopharyngeal carcinoma (NPC) regardless of World Health Organization (WHO) classification (keratinizing squamous cell carcinoma, non-keratinizing, or basaloid squamous cell carcinoma) and regardless of association with Epstein-Barr virus (EBV) and/or human papillomavirus (HPV)
* Recurrent, metastatic and incurable disease treated with platinum-gemcitabine and prior PD-1/L1 blockade (as first or second-line therapy)
* Patients are eligible regardless of prior smoking history, p16 immunohistochemistry (IHC) status, PD-L1 expression status, EBV tumor status, EBV viral load at baseline, or tumor genomic alteration status
* Patients must have at least one measurable lesion (by RECIST v1.1) which has not been previously irradiated that can be accurately measured in at least one dimension (longest diameter to be recorded for non-nodal lesions as \>= 10 mm (\>= 1 cm) (and short axis for nodal lesions, LN \>= 15 mm) with CT scan, MRI, or calipers by clinical exam
* Patients may have had no more than 2 prior lines of prior systemic therapy for recurrent, metastatic NPC
* No prior VEGFR targeted therapy permitted
* Age \>= 18 years
* Eastern Cooperative Oncology Group Performance (ECOG) performance status 0-2
* Absolute neutrophil count (ANC) \>= 1,000/mm\^3
* Hemoglobin \>= 9 g/dL
* Platelet count \>= 100,000/mm\^3
* Creatinine or creatinine clearance =\< 1.5 mg/dL or \>= 30 Modification of Diet in Renal Disease (MDRD)
* Total bilirubin =\< 1.5 x institutional upper limit of normal (ULN); except subjects with Gilbert syndrome who can have a total bilirubin \< 3 mg/dL
* Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGT\]) =\< 3 x upper limit of normal (ULN)

  * Up to =\< 5 allowed with liver metastases
* Not pregnant and not nursing, because this study involves an investigational agent whose genotoxic, mutagenic and teratogenic effects on the developing fetus and newborn are unknown. Therefore, for women of childbearing potential only, a negative urine or serum pregnancy test, per institution standard, done =\< 7 days prior to registration is required.

  * Pregnant women are excluded from this study because nivolumab, ipilimumab, and cabozantinib are all Class C or D agents with the potential for teratogenic or abortifacient effects. Because there is an unknown but potential risk for adverse events in nursing infants, secondary to treatment of the mother with any of the study agents, breastfeeding should be discontinued if the mother is treated with as part of this study (in either arm)
* No active tumor bleeding: or radiographic evidence of major blood vessel infiltration as judged by the treating investigator
* Prior -anti-cancer therapy is allowed: Patients need to be recovered from adverse events due to prior anti-cancer therapy (i.e., have residual toxicities \> grade 1), with the exception of alopecia. Any life-threatening events clearly attributable to prior immunotherapy exposure that have a high possibility of recurring should warrant exclusion: including severe pneumonitis, grade 4 bullous dermatitis/drug reaction with eosinophilia and systemic symptoms (DRESS), neurologic events such as autoimmune encephalitis transverse myelitis, and/or myocarditis. Maintenance hormonal replacement or long-term hormonal therapy exposure is permitted.

  * No chemotherapy or radiotherapy within 3 weeks (6 weeks for nitrosoureas or mitomycin C) prior to registration. Palliative (limited-field) radiation therapy is permitted, if all of the following criteria are met:

    * Repeat imaging demonstrates no new sites of bone metastases.
    * The lesion being considered for palliative radiation is not a target lesion
* No patients with a prior malignancy whose natural history or treatment has the potential to interfere with the safety or efficacy assessment of the investigational regimen
* Brain metastases allowed: Patients with treated brain metastases are eligible if follow-up brain imaging 3 weeks after central nervous system (CNS)-directed therapy shows no evidence of progression. Patients with new or progressive brain metastases (active brain metastases) or leptomeningeal disease are eligible if the treating physician determines that immediate CNS specific treatment is not required and is unlikely to be required during the first cycle of therapy
* Human immunodeficiency virus (HIV)-infected patients on effective anti-retroviral therapy with undetectable viral load within 6 months prior to registration are eligible for this trial
* For patients with evidence of chronic hepatitis B (HBV) infection, the HBV viral load must be undetectable on suppressive therapy, if indicated

  * Patients with a history of hepatitis C virus (HCV) infection must have been treated and cured. For patients with HCV infection who are currently receiving treatment, they are eligible if they have an undetectable HCV viral load
* Solid organ or tissue transplant is allowed: - subsequent therapy with nivolumab increases the risk of organ/tissue rejection. Patients must be instructed that it is crucial they stay in touch with their transplant team during treatment
* No active autoimmune disease: or history of autoimmune disease that might recur, and which may affect vital organ function or require immune suppressive treatment including systemic corticosteroids. These include but are not limited to patients with a history of

  * Immune related neurologic disease,
  * Multiple sclerosis,
  * Autoimmune (demyelinating) neuropathy,
  * Guillain-Barre syndrome (GBS),
  * Myasthenia gravis;
  * Systemic autoimmune disease such as SLE,
  * Connective tissue diseases,
  * Scleroderma, inflammatory bowel disease (IBD),
  * Crohn's, ulcerative colitis,
  * Patients with a history of toxic epidermal necrolysis (TEN),
  * Stevens-Johnson syndrome, or phospholipid syndrome should be excluded because of the risk of recurrence or exacerbation of disease
* Patients with vitiligo, endocrine deficiencies including thyroiditis managed with replacement hormones including physiologic corticosteroids are eligible
* Patients with rheumatoid arthritis and other arthropathies, Sjogren's syndrome, and psoriasis controlled with topical medication and patients with only positive serology, such as antinuclear antibodies (ANA) or anti-thyroid antibodies, should be evaluated for the presence of target organ involvement and potential need for systemic treatment but should otherwise be eligible
* Pneumonitis should be evaluated for the nature of the disease process, need for treatment prior study treatment, and the risk of exacerbation with study treatment
* Able to swallow oral medication: No known medical condition causing an inability to swallow oral formulations of agents
* No condition requiring systemic treatment with either corticosteroids (\> 10 mg daily prednisone equivalent) or other immunosuppressive medications within 14 days of study registration. Patients are permitted the use of topical, ocular, intra-articular, intranasal, and inhalational corticosteroids (with minimal systemic absorption). Adrenal replacement steroid doses \> 10 mg daily prednisone are permitted. A brief (less than 3 weeks) course of corticosteroids for prophylaxis (e.g., contrast dye allergy) or for treatment of non-autoimmune conditions (e.g., delayed-type hypersensitivity reaction caused by a contact allergen) is permitted
* Concomitant anticoagulation with coumarin agents (e.g., warfarin), direct thrombin inhibitors (e.g., dabigatran), direct factor Xa inhibitor betrixaban, or platelet inhibitors (e.g., clopidogrel) is prohibited. Allowed anticoagulants are the following:

  * Prophylactic use of low-dose aspirin for cardio-protection (per local applicable guidelines) and low-dose low molecular weight heparins (LMWH).
  * Therapeutic doses of LMWH or anticoagulation with direct factor Xa inhibitors rivaroxaban, edoxaban, or apixaban in subjects without known brain metastases who are on a stable dose of the anticoagulant for at least 1 week before first dose of study treatment without clinically significant hemorrhagic complications from the anticoagulation regimen or the tumor
* Concomitant use of any medications or substances that are strong inhibitors or inducers of CYP3A4 is discouraged; if unavoidable, the dose of cabozantinib on study should be adjusted accordingly. Any complementary medications (e.g., herbal supplements or traditional Chinese medicines) intended to treat the disease under study are prohibited