ESK981 and Nivolumab for the Treatment of Metastatic Castration Resistant Prostate Cancer

Inclusion Criteria:

* Eastern Cooperative Group (ECOG) performance status =\< 1
* Recovery to baseline or =\< grade 1 Common Terminology Criteria for Adverse Events (CTCAE) version (v.)5 from toxicities related to any prior treatments, unless adverse event (AE)(s) are clinically non-significant and/or stable on supportive therapy
* Absolute neutrophil count (ANC) \>= 1.5 K/mm\^3
* Hemoglobin (Hgb) \>= 9 g/dL
* Platelets (Plt) \>= 100,000/mm\^3
* Serum creatinine =\< 1.5 times the upper limit of normal OR creatinine clearance \> 30 mL/min by Cockcroft-Gault formula
* Total bilirubin =\< 1.5 x upper limit of normal (ULN)
* Aspartate aminotransferase (AST) =\< 2.5 x ULN (=\< 5 x ULN with known hepatic metastases)
* Alanine aminotransferase (ALT) =\< 2.5 x ULN (=\< 5 x ULN with known hepatic metastases)
* Prothrombin time (PT) and activated partial thromboplastin time (aPTT) levels =\< 1.5 x ULN (If patient is receiving anticoagulation that is expected to alter these levels, should be in targeted therapeutic range for that agent)
* Patient must have progressive disease while receiving androgen deprivation therapy (ADT) defined by any one of the following as per the PCWG3 criteria for PSA, measurable or non-measurable (bone) disease and must have a castrate serum testosterone level (i.e. =\< 50 ng/dL) at screening:

  * PSA: At least two consecutive rises in serum PSA, obtained at a minimum of 1-week intervals, with the final value \>= 2.0 ng/mL
  * Measurable disease (by Response Evaluation Criteria in Solid Tumors \[RECIST\] 1.1): \>= 20% increase (with an absolute increase of at least 5 mm) in the sum of diameters of all measurable lesions or the development of one or more new lesions. The short axis of a target lymph node must be more than 15 mm to be assessed for change in size
  * Non-measurable (bone) disease: The appearance of two or more new areas of uptake on bone scan consistent with metastatic disease compared to previous imaging during castration therapy. The increased uptake of pre-existing lesions on bone scan will not be taken to constitute progression, and ambiguous results must be confirmed by other imaging modalities (e.g. X-ray, computed tomography \[CT\] or magnetic resonance imaging \[MRI\])
* Metastatic prostate cancer (M1) as documented by appropriate medical imaging (i.e. CT-Scan, positron emission tomography \[PET\] scan or bone scan)
* Progression on a hormonal agent (abiraterone/enzalutamide) and on a chemotherapy agent (docetaxel and/or cabazitaxel) in the metastatic castration resistant setting as per PCWG3 criteria
* Progression on chemotherapy (e.g. docetaxel, cabazitaxel) in the metastatic castration resistant setting. Progression of disease within 6 months of completing docetaxel in the metastatic castrate-sensitive setting is acceptable
* Have signed an informed consent document indicating that the subject understands the purpose of and procedures required for the study and are willing to participate in the study
* Be willing and able to adhere to the prohibitions and restrictions specified in this protocol
* Willingness to use contraception by a method that is deemed effective by the investigator throughout the treatment period and for at least 30 days following the last dose of therapy
* Willingness and ability to comply with study procedures and follow-up examination
* Able to swallow and retain oral medication
* Willingness and ability to undergo mandatory tumor biopsy at baseline and at the cycle 3 visit
* Willingness and ability to undergo mandatory whole blood sample collections at baseline, weeks 2-4 in the first cycle, and then monthly

Exclusion Criteria:

* Systemic therapy (other than a gonadotrophin releasing hormone \[GnRH\] agonist/antagonist) for CRPC within the past two weeks from cycle 1/day 1 including:

  * CYP-17 inhibitors (e.g. ketoconazole, abiraterone)
  * Antiandrogens (e.g. bicalutamide, nilutamide)
  * Second generation antiandrogens (e.g. enzalutamide, ARN-509, galeterone)
  * Immunotherapy (e.g. sipuleucel-T, ipilimumab)
  * Chemotherapy (e.g. docetaxel, cabazitaxel)
* Prior radiopharmaceutical therapy (e.g. radium-223, strontium-89, samarium-153, etc.) within the past year
* Have any condition that, in the opinion of the investigator, would compromise the well-being of the subject or the study or prevent the subject from meeting or performing study requirements
* The patient is currently on warfarin or heparin therapy
* The patient has any pre-existing coagulopathy, recent hemoptysis, gross hematuria or gastrointestinal bleeding
* The patient has a history of a clinically significant cardiovascular or cerebrovascular event within 3 months prior to study entry
* The patient has uncontrolled hypertension defined as a blood pressure measurement greater than 150 mm Hg systolic or 90 mm Hg diastolic with medication
* The patient has previously been enrolled in the study or received ESK981
* The patient has known hypersensitivity to gelatin or lactose monohydrate
* The patient has taken a medication known to be a potent inducer of CYP1A2, CYP2C8, or CYP3A4 within 4 weeks prior to the first dose of study drug
* Prior treatment with anti-PD/PD-L1/CTLA4/IDO antibody
* Untreated brain metastases or spinal cord compression
* Major surgical procedure or significant traumatic injury within 6 weeks prior to study registration. (\> 6 weeks prior to registration is permitted as long as they have fully recovered from any such procedure)
* History of another primary malignancy except for: malignancy treated with curative intent and no known active disease for \>= 5 years, adequately treated non-melanoma skin cancer without evidence of disease, adequately treated carcinoma in situ without evidence of disease
* Angina, myocardial infarction symptomatic congestive heart failure, cerebrovascular accident, transient ischemic attack, arterial embolism, pulmonary embolism, percutaneous angioplasty or Coronary arterial bypass surgery within the past 3 months
* The patient has received any investigational drug within 28 days prior to registration or 5 half-lives of the investigational drug, whichever is sooner