Anti-SEMA4D Monoclonal Antibody VX15/2503 With Nivolumab or Ipilimumab in Treating Patients With Stage III or IV Melanoma

Inclusion Criteria:

* Patients with histologically confirmed unresectable stage III or stage IV metastatic melanoma, who have not been previously treated with a SEMA4D antibody and have had prior anti-PD1/PDL1 inhibitors with documented progression; patient may have or not have prior anti-CTLA4 treatments
* Measurable disease per RECIST v. 1.1 criteria using imaging scans, or peripheral lesions that can be adequately documented with a picture and a ruler even if they do not meet RECIST criteria.
* Patients must have non-target lesion accessible for sequential biopsy (core needle biopsy or excision preferred, fine needle aspiration not eligible)
* Eastern Cooperative Oncology Group (ECOG) performance status 0 or 1
* Women of child-bearing potential must have a negative serum pregnancy test within 24 hour of initiation of dosing and must agree to use an effective form of contraception during the study from the time of the negative pregnancy test up to 6 months after the last dose of study drug. Effective forms of contraception include abstinence, hormonal contraceptive in conjunction with a barrier method, or a double barrier method. Women of non-childbearing potential may be included if they are either surgically sterile or have been postmenopausal for \>= 1 year. Fertile men must also agree to use an effective method of birth control while on study drug and up to 6 months after the last dose of study drug
* Patients must have fully recovered from the effects of any major surgery or significant traumatic injury within 14 days of course 1 day 1 (C1D1)
* Absolute neutrophil count \>= 1 X 10\^9/L
* Hemoglobin (Hgb) \> 8 g/dL
* Platelet count \>= 75 X 10\^9/L
* Aspartate aminotransferase (AST)/alanine aminotransferase (ALT) =\< 3 X upper limit of normal (ULN) or \< 5 X ULN in the presence of liver metastases
* Bilirubin =\< 3 X ULN or \< 5 X ULN in the presence of liver metastases
* Creatinine =\< 3 X ULN or calculated creatinine clearance (CrCl) \> 30 mL/min using Cockcroft- Gault formula
* Willing and able to provide written informed consent prior to any study related procedures and to comply with all study requirements

Exclusion Criteria:

* Active secondary malignancy, unless the malignancy is not expected to interfere with the evaluation of safety and is approved by the Medical Monitor.
* Investigational drug use within 28 days of C1D1
* Chemotherapy, targeted therapy, growth factors or radiation therapy within 14 days of C1D1
* Systemic steroid therapy or any other form of immunosuppressive therapy within 7 days prior to registration
* History of any of the following toxicities associated with a prior immunotherapy:

  * Grade \>= 3 immune mediated adverse event that was considered related to previous immunotherapy and required immune suppressive therapy. It is acceptable to allow patients with Grade 3 ophthalmologic immune-mediated events that improved to Grade 1 within 2 weeks after topical therapy only, and patients with Grade 3 endocrine immune-mediated events that did not experience symptoms lasting \> 6 weeks and are not requiring \> 7.5mg prednisone or equivalent per day.
  * Immune mediated adverse event that was considered related to previous immunotherapy and is still \> grade 1 with immune suppressive therapy
  * • Any active immune-mediated adverse events requiring ongoing immune suppressive therapy (hormone replacement therapy is permitted).
* Patients with known active central nervous system (CNS) lesions are excluded (i.e., those with radiographically unstable, symptomatic lesions). However, patients treated with stereotactic therapy or surgery are eligible if they remain without evidence of disease progression in the brain for 14 days.
* Major surgery within 14 days of registration
* Has received a live vaccine within 28 days prior to registration
* A known active and clinically significant bacterial, fungal or viral infection including hepatitis B (HBV), hepatitis C (HCV), known human immunodeficiency virus (HIV) or acquired immunodeficiency syndrome (AIDS) related illness (human immunodeficiency virus \[HIV\] testing is not required), including patients who have an active infection requiring systemic therapy