Glembatumumab Vedotin, Nivolumab, and Ipilimumab in Treating Patients With Advanced Metastatic Solid Tumors That Cannot Be Removed by Surgery

Inclusion Criteria:

* Patients must have histologically confirmed malignancy that is metastatic or unresectable and for which standard curative measures do not exist or are no longer effective
* For TNBC and solid tumors other than melanoma: GPNMB expression as defined by at least 25% of malignant epithelial cells or tumor stromal cells expression GPNMB at any intensity via central immunohistochemistry on archived or biopsied tumor tissue (as per standard clinical care) from an advanced/metastatic disease site; for melanoma or uveal melanoma cohort: GPNMB testing results will not be required for eligibility assessment
* Patients must have measurable disease, in addition to at least one lesion in a reasonably safe location for study-related biopsies; measurable disease is defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded for non-nodal lesions and short axis for nodal lesions) as \>= 20 mm (\>= 2 cm) by chest x-ray or as \>= 10 mm (\>= 1 cm) with computed tomography (CT) scan, magnetic resonance imaging (MRI), or calipers by clinical exam
* Willing to undergo study-related biopsies as noted in study calendar: before treatment, after glembatumumab-only, and after combination therapy
* Prior treatment with immune checkpoint inhibitors or antibody drug conjugates are allowed.

  * For cutaneous melanoma cohort, patients should have received PD-1/CTLA-4 targeting therapy in combination and had disease progression
  * For all other cohorts including phase Ib dose-finding, patients should have received at least one line of standard non-immunotherapy treatment
* Eastern Cooperative Oncology Group (ECOG) performance status =\< 1 (Karnofsky \>= 80%)
* Life expectancy of greater than 12 weeks
* Leukocytes \>= 2,000/mcL
* Absolute neutrophil count \>= 1,500/mcL
* Platelets \>= 75,000/mcL
* Hemoglobin \>= 9.0 g/dL
* Total bilirubin =\< 1.5 x institutional upper limit of normal (except patients with Gilbert syndrome, who can have total bilirubin \< 3.0 mg/dL)
* Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 3.0 x institutional upper limit of normal (or =\< 5.0 x institutional upper limit of normal in presence of metastatic liver disease)
* Serum creatinine =\< 1.5 x institutional upper limit of normal OR creatinine clearance \>= 50 mL/min (if using the Cockcroft-Gault formula)
* Women of childbearing potential (WOCBP) and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation; WOCBP should use an adequate method to avoid pregnancy for 32 weeks after the last dose of investigational drug; women of childbearing potential must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of HCG) within 24 hours prior to the start of nivolumab; women must not be breastfeeding; men who are sexually active with WOCBP must use any contraceptive method with a failure rate of less than 1% per year; men receiving nivolumab and who are sexually active with WOCBP will be instructed to adhere to contraception for a period of 31 weeks after the last dose of investigational product; women who are not of childbearing potential (i.e., who are postmenopausal or surgically sterile as well as azoospermic men) do not require contraception; women of childbearing potential (WOCBP) is defined as any female who has experienced menarche and who has not undergone surgical sterilization (hysterectomy or bilateral oophorectomy) or who is not postmenopausal; menopause is defined clinically as 12 months of amenorrhea in a woman over 45 in the absence of other biological or physiological causes; in addition, women under the age of 55 must have a documented serum follicle stimulating hormone (FSH) level less than 40 mIU/mL; should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she (or the participating partner) should inform the treating physician immediately
* Ability to understand and the willingness to sign a written informed consent document

Exclusion Criteria:

* Patients who have had systemic therapy within 2 weeks (6 weeks for nitrosoureas or mitomycin C) or radiotherapy within 2 weeks prior to entering the study
* Patients who have not recovered from adverse events due to prior anti-cancer therapy (i.e., have residual toxicities \>= grade 2 other than alopecia)
* Patients who are receiving any other concurrent investigational agents; those with history of receiving investigational cancer treatment are allowed as long as washout period is fulfilled
* Symptomatic or untreated leptomeningeal or brain metastases or spinal cord compression; patients with treated and stable brain metastases will be allowed; controlled brain metastases are defined as no radiographic progression for at least 4 weeks following radiation and/or surgical treatment (or 4 weeks of observation if no intervention is clinically indicated), and off of steroids for at least 2 weeks, and no new or progressive neurological signs and symptoms
* History of allergic reactions attributed to compounds of similar chemical or biologic composition to nivolumab, glembatumumab vedotin, or other agents used in study; this includes sensitivity to the drugs dolastatin or auristatin
* History of severe hypersensitivity reaction to any monoclonal antibody
* Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements
* Participants must not have a history of life-threatening toxicity related to prior IO treatment (eg, anti-CTLA-4 or anti-PD-1/PD-L1 treatment or any other antibody or treatment specifically targeting T-cell co-stimulation or immune checkpoint pathways); i) participants with toxicities that are unlikely to recur with standard countermeasures (eg, hormone replacement treatment after adrenal crisis) are eligible
* Pregnant women are excluded from this study; breastfeeding should be discontinued if the mother is treated with nivolumab and glembatumumab vedotin
* Human immunodeficiency virus (HIV) patients should be allowed if otherwise eligible, with the following caveats:

  * Undetectable HIV viral load by standard clinical assay
  * Willing to remain adherent with antiretroviral therapy that has minimal overlapping toxicity or pharmacokinetic interactions with protocol therapy
  * CD4+ T cell counts of 200/mm\^3 or greater
  * No acquired immunodeficiency syndrome (AIDS)-defining events other within the past 12 months
  * Near normal life expectancy if not for presence of the cancer
  * Not taking antiviral agents that are otherwise prohibited because of CYP interactions
* Patients should be excluded if they have a positive test for hepatitis B virus surface antigen (HBV sAg) or hepatitis C virus ribonucleic acid (HCV RNA) indicating acute or chronic infection
* Patients with active autoimmune disease or history of autoimmune disease that might recur, which may affect vital organ function or require immune suppressive treatment including systemic corticosteroids, should be excluded; these include but are not limited to patients with a history of immune related neurologic disease, multiple sclerosis, autoimmune (demyelinating) neuropathy, Guillain-Barre syndrome, myasthenia gravis; systemic autoimmune disease such as systemic lupus erythematosus (SLE), connective tissue diseases, scleroderma, inflammatory bowel disease (IBD), Crohn's, ulcerative colitis, hepatitis; and patients with a history of toxic epidermal necrolysis (TEN), Stevens-Johnson syndrome, or phospholipid syndrome should be excluded because of the risk of recurrence or exacerbation of disease; patients with vitiligo, endocrine deficiencies including thyroiditis managed with replacement hormones including physiologic corticosteroids are eligible; patients with rheumatoid arthritis and other arthropathies, Sjogren's syndrome and psoriasis controlled with topical medication and patients with positive serology, such as antinuclear antibodies (ANA), anti-thyroid antibodies should be evaluated for the presence of target organ involvement and potential need for systemic treatment but should otherwise be eligible
* Patients are permitted to enroll if they have vitiligo, type I diabetes mellitus, residual hypothyroidism due to autoimmune condition only requiring hormone replacement, psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger (precipitating event)
* Patients should be excluded if they have a condition requiring systemic treatment with either corticosteroids (\> 10 mg daily prednisone equivalents) or other immunosuppressive medications within 14 days of study drug administration; inhaled or topical steroids and adrenal replacement doses =\< 10 mg daily prednisone equivalents are permitted in the absence of active autoimmune disease; patients are permitted to use topical, ocular, intra-articular, intranasal, and inhalational corticosteroids (with minimal systemic absorption); physiologic replacement doses of systemic corticosteroids are permitted, even if =\< 10 mg/day prednisone equivalents; a brief course of corticosteroids for prophylaxis (e.g., contrast dye allergy) or for treatment of non-autoimmune conditions (e.g., delayed-type hypersensitivity reaction caused by contact allergen) is permitted
* Patients who have had evidence of active or acute diverticulitis, intra-abdominal abscess, gastrointestinal (GI) obstruction and abdominal carcinomatosis which are known risk factors for bowel perforation should be evaluated for the potential need for additional treatment before coming on study
* Use of any vaccines against infectious diseases (e.g. influenza, varicella, pneumococcus) within 4 weeks of initiation of study treatment