Enzalutamide With Venetoclax in Treating Patients With Metastatic Castration Resistant Prostate Cancer

Inclusion Criteria:

* Histological or cytological documentation of diagnosis of prostate cancer.
* Documented progressive metastatic castrate resistant prostate cancer (mCRPC) based on at least one of the following criteria:

  * Prostate specific antigen (PSA) progression defined as 25% increase over baseline value with an increase in the absolute value of at least 2 ng/mL that is confirmed by another PSA level with a minimum of a 1 week interval and a minimum PSA of 2 ng/mL.
  * Soft-tissue progression defined as an increase \>= 20% in the sum of the longest diameter (LD) of all target lesions based on the smallest sum LD since treatment started or the appearance of one or more new lesions.
  * Progression of bone disease (evaluable disease) or, new bone lesion(s), by bone scan.
  * If on an anti-androgen, must have documented progression 6 weeks after stopping anti-androgen therapy.
* Willing to undergo a biopsy, if readily available biopsy site present (i.e., nodal or visceral metastasis).
* Have an Eastern Cooperative Oncology Group (ECOG) performance status of 0 - 1.
* Have testosterone level of \< 50 ng/dL. Note: Patients must continue primary androgen deprivation with a luteinizing hormone-releasing hormone (LHRH) analogue (agonist or antagonist) if they have not undergone orchiectomy.
* White blood cells \>= 1.5 x 10\^9/L (obtained within 14 days prior to treatment start)
* Platelets (UNVPLT) \>= 100 x 10\^9/L (obtained within 14 days prior to treatment start)
* Hemoglobin (HGB) \>= 9 g/dL (obtained within 14 days prior to treatment start)
* Potassium (K), total calcium (CA) (corrected for serum albumin), magnesium, sodium (NA) and phosphorus within normal limits for the institution or corrected to within normal limits with supplements before first dose of study medication (obtained within 14 days prior to treatment start)
* Total calcium (CA) (corrected for serum albumin) within normal limits for the institution or corrected to within normal limits with supplements before first dose of study medication (obtained within 14 days prior to treatment start)
* Magnesium within normal limits for the institution or corrected to within normal limits with supplements before first dose of study medication (obtained within 14 days prior to treatment start)
* Sodium (NA) within normal limits for the institution or corrected to within normal limits with supplements before first dose of study medication (obtained within 14 days prior to treatment start)
* Phosphorus within normal limits for the institution or corrected to within normal limits with supplements before first dose of study medication (obtained within 14 days prior to treatment start)
* Institutional normalized ratio (INR) =\< 1.5 (obtained within 14 days prior to treatment start)
* Adequate renal function as demonstrated by a creatinine clearance \>= 30 mL/min; calculated by the Cockcroft Gault formula
* Aspartate aminotransferase (aspartate transaminase \[AST\]) and alanine aminotransferase (alanine transaminase \[ALT\]) =\< 3 X upper limit of normal (ULN). If the patient has liver metastases, ALT and AST must still be =\< 3 x ULN. Patients with liver metastases and AST/ALT above this limit will not be enrolled (obtained within 14 days prior to treatment start)
* Total serum bilirubin =\< 1.5 x ULN (unless bilirubin rise is due to Gilbert?s syndrome or of non-hepatic origin); or total bilirubin (TBILI) =\< 3.0 x ULN with direct bilirubin within normal range in patients with well documented Gilbert?s syndrome (obtained within 14 days prior to treatment start)
* Ability to swallow and retain oral medication (without crushing, dissolving or chewing tablets).
* Phase Ib only: Prior enzalutamide treatment and/or other approved treatments for CRPC are acceptable
* Phase II only: Participants MUST be treatment naive in the CRPC setting: i.e., no prior exposure to abiraterone acetate other specific CYP-17 inhibitors; no prior exposure to enzalutamide or investigational androgen receptor (AR) targeted agents; and no prior exposure to chemotherapy and or RAD-223.
* Sexually active males must agree to use a condom during intercourse while taking the study drug and for at least 3 months after stopping study treatment. Sexually active males should not father a child during this period. A condom is required to be used by vasectomized men in order to prevent delivery of the drug via seminal fluid. Should a woman become pregnant or suspect she is pregnant while her partner is participating in this study, she should inform her treating physician immediately.
* Participant or legal representative must understand the investigational nature of this study and voluntarily sign and date an Independent Ethics Committee/Institutional Review Board approved written informed consent form prior to receiving any study related procedure.

Exclusion Criteria:

* Phase II only: Prior exposure to abiraterone acetate.
* Phase II only: Prior exposure to BCL-2 inhibitors agents like venetoclax.
* Phase II only: Prior chemotherapy for castration resistant disease. Chemotherapy given in the castration-sensitive setting is permissible if stopped at least 4 weeks prior to treatment start.
* Phase II only: Prior isotope therapy with strontium-89, samarium or radium-223 within 12 weeks of treatment start.
* Subject has acute promyeloctyic leukemia
* Subject has known active CNS involvement with AML
* Participants with known symptomatic brain metastases.
* Participant has a concurrent malignancy or malignancy within 3 years of treatment start, with the exception of adequately treated, basal or squamous cell carcinoma, nonmelanomatous skin cancer or curatively resected cervical cancer.
* Participant has a known history of human immunodeficiency virus (HIV) infection (testing not mandatory);
* Chronic hepatitis B virus (HBV) or hepatitis C (HCV) requiring treatment. Note: subjects with serologic evidence of prior vaccination to HBV (i.e. hepatitis B surface (HBs) antigen negative-, anti-HBs antibody positive and anti-hepatitis B core (HBc) antibody negative) or positive anti-HBc antibody from intravenous immunoglobulins (IVIG) may participate
* Uncontrolled and/or active systemic infection (viral, bacterial or fungal).
* Participant has clinically significant, uncontrolled heart disease and/or recent events including any of the following:

  * History of acute coronary syndromes (including myocardial infarction, unstable angina, uncontrolled hypertension, uncontrolled arrhythmia, stroke, coronary artery bypass grafting, coronary angioplasty, or stenting) or symptomatic pericarditis within 12 months prior to treatment start
  * Cardiac history of CHF requiring treatment or Ejection fraction ≤ 50% or chronic stable angina
  * A cardiovascular disability status of New York Heart Association Class \> 2
  * Class 2 is - Defined as cardiac disease in which patients are comfortable at rest but ordinary physical activity results in fatigue, palpitations, dyspnea or angina pain
  * On screening 12 lead electrocardiography (ECG), any of the following cardiac parameters: bradycardia (heart rate \< 50 at rest), tachycardia (heart rate \> 90 at rest), PR interval \> 220 msec, QRS interval \> 109 msec or, Fridericia's correction formula (QTcF) \> 450 msec. Congenital long QT syndrome or family history of long QT syndrome.
* DLCO \<=65% or FEVI \<= 65\&
* Treatment with any of the following within 7 days prior to the first dose of study drug:

  * Steroid therapy for anti-neoplastic intent
  * Moderate or strong cytochrome P450 3A (CYP3A) inhibitors
  * Moderate or strong CYP3A inducers
  * Medications that have a known risk to prolong the QT interval or induce Torsades de Pointes
  * Herbal preparations/medications
* Participants receiving any medications or substances that are inhibitors or inducers of CYP2C8 enzymes are ineligible.
* Administration or consumption of any of the following within 3 days prior to the first dose of study drug:

  * Grapefruit or grapefruit products
  * Seville oranges (including marmalade containing Seville oranges)
  * Star fruit.
* Patient who has received radiotherapy =\< 4 weeks prior to start of treatment or limited field radiation for palliation =\< 2 weeks prior to treatment start and, who has not recovered to grade 1 or better from related side effects of such therapy (exceptions include alopecia) and/or in whom \>= 30% of the bone marrow was irradiated.
* Patients with central nervous system (CNS) involvement.
* Patients with seizure disorder.
* Patient has not recovered from all toxicities related to prior anticancer therapies to National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE) (version 5.0) grade \< =1 (Exception to this criterion: patients with any grade of alopecia are allowed to enter the study).
* Participant has any other concurrent severe and/or uncontrolled medical condition that would cause, in the investigator's judgment, an unacceptable safety risk.
* Unwilling or unable to follow protocol requirements.
* Any condition which in the investigator's opinion deems the participant an unsuitable candidate to receive study drug.