Erdafitinib and Abiraterone Acetate or Enzalutamide in Treating Patients With Double Negative Prostate Cancer

Inclusion Criteria:

* Patients must be \>= 18 years of age prior to signing informed consent
* History of histologically diagnosed prostatic adenocarcinoma
* Participants must have evidence of castration resistant prostate cancer as evidenced by a confirmed rising PSA or radiographic progression (per Prostate Cancer Working Group 3 \[PCWG3\] criteria) and a castrate serum testosterone level (i.e. =\< 50 mg/dL)
* Participants must have previously progressed on abiraterone acetate and/or enzalutamide, with PSA or radiographic progression on the most recent agent per PCWG3 criteria. If the most recent agent received was abiraterone or enzalutamide there should be no washout prior to initiating erdafitinib per protocol
* Measurable disease as defined per RECIST v1.1 criteria
* Subjects who have progressed on only one next-generation AR-directed therapy (e.g. abiraterone, enzalutamide) and who have not received taxane chemotherapy will be required to have evidence of double-negative prostate cancer as defined by immunohistochemistry on biopsy. A fresh metastatic biopsy within 8 weeks is preferred; however, any archival tissue showing a DNPC phenotype will be acceptable for determining eligibility. Patients who have received two prior lines of AR-directed therapy and at least one prior taxane do not require histologic confirmation of DNPC. Note: transcript profiling methods for defining DNPC may be accepted per the PI's discretion
* Eastern Cooperative Oncology Group (ECOG) performance status score =\< 2
* Hemoglobin \>= 8 g/dL (\>= 5 mmol/L) (must be without red blood cell \[RBC\] transfusion within 7 days prior to the laboratory test)
* Platelets \>= 75 x 10\^9/L
* Absolute neutrophil count (ANC) \>= 1.5 x 10\^9/L (prior growth factor support is permitted but must be without support in the 7 days prior to the laboratory test)
* Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) =\< 2.5 x upper limit of normal (ULN) or =\< 5 x ULN for subjects with liver metastases
* Creatinine clearance \>= 40 mL/min/1.73 m\^2 based upon modified diet in renal disease formula calculation
* Total bilirubin =\< 1.5 x ULN; except in subjects with congenital bilirubinemia, such as Gilbert syndrome
* Corrected QT interval (corrected QT interval by Fridericia \[QTcF\] or QT corrected interval by the Bazett's formula \[QTcB\]) =\< 480 msec based on the average of triplicate assessments performed approximately 5 minutes apart
* Subjects must agree to use acceptable contraception
* Must sign an informed consent form (ICF) (or their legally acceptable representative must sign) indicating that he or she understands the purpose of, and procedures required for, the study and is willing to participate in the study

Exclusion Criteria:

* Treatment with any other investigational agent or participation in another clinical trial with therapeutic intent within 14 days prior to randomization
* Active malignancies (i.e., requiring treatment change in the last 24 months) other than malignancy under study (except skin cancers within the last 24 months that is considered completely cured)
* Evidence of predominant small cell or neuroendocrine variant prostate cancer on most recent standard of care metastatic biopsy
* Symptomatic central nervous system (CNS) metastases. Treated CNS metastases will be allowed if these are stable for at least 8 weeks prior to enrollment
* Received prior FGFR inhibitor treatment or if the subject has known allergies, hypersensitivity, or intolerance to erdafitinib or its excipients
* Current central serous retinopathy (CSR) or retinal pigment epithelial detachment of any grade
* Has persistent phosphate level \> ULN during screening (on 2 consecutive assessments at least 1 week apart, within 14 days of treatment and prior to cycle 1 day 1) and despite medical management
* Has a history of or current uncontrolled cardiovascular disease including:

  * Unstable angina, myocardial infarction, ventricular fibrillation, Torsades de Pointes, cardiac arrest, or known congestive heart failure class III-V within the preceding 3 months; cerebrovascular accident or transient ischemic attack within the preceding 3 months
  * Pulmonary embolism or other VTE (venous thromboembolism) within the preceding 2 months
* Has known active acquired immune deficiency syndrome (AIDS) (human immunodeficiency virus \[HIV\] infection)
* Hepatitis B infection as defined according to the American Society of Clinical Oncology guidelines. In the event the infection status is unclear, quantitative levels are necessary to determine the infection status. Hepatitis C (anti-hepatitis C virus \[HCV\] antibody positive or HCV-ribonucleic acid \[RNA\] quantitation positive) or known to have a history of hepatitis C. If positive, further testing of quantitative levels to rule out positivity is required
* Has not recovered from reversible toxicity of prior anticancer therapy (except toxicities which are not clinically significant such as alopecia, skin discoloration, hot flashes, grade 1 neuropathy, grade 1-2 hearing loss)
* Has impaired wound healing capacity defined as skin/decubitus ulcers, chronic leg ulcers, known gastric ulcers, or unhealed incisions
* Major surgery within 2 weeks of the first dose, or will not have fully recovered from surgery, or has surgery planned during the time the subject is expected to participate in the study or within 2 weeks after the last dose of study drug administration. (Note: subjects with planned surgical procedures to be conducted under local anesthesia may participate
* Any serious underlying medical condition, such as:

  * Evidence of serious active viral, bacterial, or uncontrolled systemic fungal infection
  * Active autoimmune disease or a documented history of autoimmune disease
  * Psychiatric conditions (e.g., alcohol or drug abuse), dementia, or altered mental status
* Any other issue that would impair the ability of the subject to receive or tolerate the planned treatment at the investigational site, to understand informed consent or any condition for which, in the opinion of the investigator, participation would not be in the best interest of the subject (e.g., compromise the well-being) or that could prevent, limit, or confound the protocol-specified assessments
* Patient, who, in the opinion of their treating physician, requires immediate treatment (e.g. those with extensive liver metastases)