Dose Escalation and Dose Expansion Study of GSK525762 in Combination With Androgen Deprivation Therapy in Participants With Castrate-resistant Prostate Cancer

Inclusion Criteria:

* Written informed consent provided.
* Males \>=18 years of age (at the time written consent is obtained for screening).
* Histologically confirmed adenocarcinoma of the prostate: screening and on-treatment biopsy is mandatory. If adequate number of paired biopsy samples are collected (\>=20 paired samples for each dose level in each Arm, unless an Arm is closed early), then further biopsy sampling will be considered based on available data; screening biopsy can be waived if participant had a recent biopsy after failure of the most recent therapy (within 30 days) and the biopsy sample is secured to be sent as screening biopsy for this study.
* Surgically or medically castrated, with testosterone levels of less than or equal to (\<=)50 nanograms per deciliter (ng/dL) (\<2.0 nanometer \[nM\]). If the participant is being treated with luteinizing hormone-releasing hormone (LHRH) agonists/antagonists (participant who have not undergone orchiectomy) this therapy must have been initiated at least 4 weeks prior to Week 1 Day 1 and must be continued throughout the study.
* Participants must have failed prior therapy with abiraterone, enzalutamide, or both:

  1. Has completed at least 12 weeks of prior continuous therapy with abiraterone or enzalutamide in any prior line.
  2. Lead-in dosing period for enzalutamide only will be required under the following circumstance:

  (i) If the participant has enzalutamide discontinuation for \>7 days prior to dosing start with GSK525762 plus enzalutamide on trial, then a enzalutamide only lead-in dosing of 28 days is required.

(ii) If the participant has enzalutamide discontinuation for \<=7 days prior to dosing start with GSK525762 plus enzalutamide on trial, then an enzalutamide only lead-in dosing of 14 days is required.

(iii) If the participant is on continuous dosing with enzalutamide prior to dosing start with GSK525762 plus enzalutamide on trial, then participant can start on combined dosing at end of screening period.

(c) Lead-in dosing period for abiraterone only will be required: if the participant has abiraterone discontinuation for more than 3 days prior to dosing start with GSK525762 plus abiraterone on trial, then abiraterone only lead-in dosing of 7 days is required.

* One to two line(s) of prior taxane-based chemotherapy allowed. If docetaxel chemotherapy is used more than once, this will be considered as one regimen. Participants who have not received prior chemotherapy in any setting will qualify for study if they are ineligible for or refuse chemotherapy.
* Documented prostate cancer progression as assessed by the investigator with one of the following:

  1. PSA progression defined by a minimum of 3 rising PSA levels with an interval of \>=1 week between each determination. The PSA value at screening must be \>=5 microgram (µg)/Liter (L) (5 ng/mL) if PSA is the only indication of progression; participants on systemic glucocorticoids for control of symptoms must have documented PSA progression by PCWG3 while on systemic glucocorticoids prior to commencing Week 1 Day 1 treatment.
  2. Radiographic progression of soft tissue disease by PCWG3-modified RECIST 1.1 criteria or bone metastasis with 2 or more documented new bone lesions on a bone scan with or without PSA progression.
* ECOG performance status of 0 or 1.
* Life expectancy \>12 weeks.
* Able to swallow and retain orally administered medication.
* Must have adequate organ function.
* Male participants are eligible to participate if they agree to use contraceptive methods.

Exclusion Criteria:

* Surgery or local prostatic intervention (excluding a prostatic biopsy) less than 28 days of Week 1 Day 1.
* Participants with neuroendocrine and/or small cell CRPC.
* Recent prior therapy, defined as:

  1. Any investigational or approved non-biologic anti-cancer drug within 14 days prior to the first dose of GSK525762 and abiraterone/enzalutamide.
  2. Any nitrosoureas or mitomycin C within 42 days prior to the first dose of GSK525762 and abiraterone/enzalutamide.
  3. Any anti-cancer biologic agents within five half-lives prior to the first dose of GSK525762 and abiraterone/enzalutamide.
  4. If the participant received radiotherapy \<90 days prior to study treatment, the irradiated lesion cannot be the only lesion used for evaluating response.
  5. Any major surgery within 28 days prior to the first dose of GSK525762 and abiraterone/enzalutamide.
* Evidence of severe or uncontrolled systemic diseases (e.g., unstable or uncompensated respiratory, hepatic, renal, cardiac disease, or clinically significant bleeding episodes). Any serious and/or unstable pre-existing medical (aside from malignancy), psychiatric disorder, or other conditions that could interfere with participant's safety, obtaining informed consent or compliance to the study procedures, in the opinion of the Investigator; systolic blood pressure higher than 150 millimeter of mercury (mmHg) or diastolic blood pressure higher than 90 mmHg found on 2 separate occasions separated by 1 week, despite adequate therapy, will be defined as uncontrolled hypertension; uncontrolled diabetes mellitus (despite therapeutic, compliance intervention) as defined by a hemoglobin A1c (HbA1c) level more than 8% and/or occurrence of more than 2 episodes of ketoacidosis in the 12 months prior to the first dose of study drug.
* Cardiac abnormalities as evidenced by any of the following:

  1. Baseline QT interval corrected for heart rate by Fridericia's formula (QTcF) interval \>=480 milliseconds (msec).
  2. Clinically significant conduction abnormalities or arrhythmias, such as participants with second degree (Type II) or third degree atrio-ventricular block.
  3. History or evidence of current \>=Class II congestive heart failure as defined by New York Heart Association (NYHA).
  4. History of acute coronary syndromes (including unstable angina and myocardial infarction), coronary angioplasty, or stenting within the past 3 months. Participants with a history of stent placement requiring ongoing anti-coagulant therapy (e.g., clopidogrel, prasugrel) will not be permitted to enroll.
  5. Known cardiac metastasis.
* Participants with history of known bleeding disorder(s) or history of clinically significant hemorrhage (e.g., gastrointestinal, neurologic), within the past 6 months.
* Therapeutic-dose anticoagulation (e.g., warfarin, low-molecular weight heparin \[LMWH\], or novel oral anticoagulants) must be discontinued and coagulation parameters must be normalized prior to the first dose of GSK525762 and abiraterone/enzalutimide. Prophylactic anticoagulation, with low doses (per standard practice) of agents such as LMWH, direct thrombin inhibitors, or factor Xa inhibitors is permitted.
* Concurrent use of high dose aspirin (doses up to 81 milligrams (mg) oral dose daily allowed, or 100 mg, as per country standards) and non-steroidal anti-inflammatory drugs (NSAIDS), except for where NSAIDs provide documented benefit over other analgesics, and then to be used with caution including concomitant use of proton pump inhibitors).
* Any acute toxicities due to prior chemotherapy and / or radiotherapy that have not resolved to a National Cancer Institute-Common Terminology Criteria for Adverse Events (NCI-CTCAE) version 4.0 grade \<=1 with the exception of chemotherapy induced alopecia and grade 2 peripheral neuropathy.
* The participant has an active second malignancy other than curatively resected basal cell or squamous cell carcinoma of the skin, in situ carcinoma of the bladder, or other cancers for which they are treated with curative intent with no active disease in the 3 years prior to enrollment.
* Participants with known symptomatic brain metastasis are not suitable for enrolment. Participants with asymptomatic, stable, treated brain metastases are eligible for study entry.
* History of seizure within 6 months of study treatment initiation or any condition that may predispose participant to seizure (e.g., prior cortical stroke or significant brain trauma) or who are currently being treated with cytochrome P450 enzyme inducing anti-epileptic drugs for seizures (use of anti-epileptic drugs to control pain is allowed in participants not suffering from seizures unless drug is excluded due to Cytochrome (CY)P3A4 induction - phenytoin, carbamazepine, phenobarbital).
* History of loss of consciousness or transient ischemic attack within 12 months prior to enrollment.
* Participants with symptomatic or impending cord compression unless appropriately treated beforehand and clinically stable and asymptomatic.
* Current use of a prohibited medication or planned use of any forbidden medications during treatment with GSK525762 and abiraterone/enzalutamide. This includes medications that are potent inducers or inhibitors of CYP3A4 enzymes or strong inhibitors of CYP2C8.
* Participants with gastrointestinal disorders likely to interfere with absorption of the study medication.
* Participants with known bleeding diathesis.
* Current active liver or biliary disease (with the exception of Gilbert's syndrome or asymptomatic gallstones, liver metastases or otherwise stable chronic liver disease per investigator assessment).
* Initiating bisphosphonate or denosumab therapy or adjusting dose/regimen within 3 months prior to Week 1 Day 1. Participants on a stable bisphosphonate or denosumab therapy are eligible and may continue.
* Any serious known immediate or delayed hypersensitivity reaction to GSK525762 or idiosyncrasy to drugs chemically related to the investigational drugs. Additionally, any known hypersensitivity to either enzalutamide, abiraterone or any excipients would be excluded.
* Known history of human immunodeficiency virus (HIV).
* Presence of hepatitis B surface antigen (HBsAg) or positive hepatitis C antibody test result at screening.