Dose-finding Study of GSK2636771 When Administered in Combination With Enzalutamide in Male Subjects With Metastatic Castration-Resistant Prostate Cancer

Inclusion Criteria:

* Signed written informed consent provided
* Males \>=18 years of age (at the time consent is obtained)
* Histologically or cytologically confirmed diagnosis of prostate adenocarcinoma, surgically castrated or continuous medical castration (for \>=8 weeks prior to Screening)
* Serum testosterone \<50 nanogram per deciliter (ng/dL) (1.7 nanomole per liter \[nM/L\])
* PTEN deficient tumor as documented from archival or fresh (from biopsy) tumor tissue analyzed by GlaxoSmithKline selected laboratory
* Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1
* Completed at least 12 weeks of prior continuous therapy with enzalutamide. A 2 week or less treatment (enzalutamide) holiday will be permitted prior to initiating study treatment.
* Most recent enzalutamide dose received is 160 mg once daily with no change in dose for at least 2 weeks prior to Screening.
* Has progressive disease at time of enrollment defined as one or more of the following criteria: PSA progression defined by PCWG2 criteria or soft tissue disease progression defined by RECIST 1.1 or bone disease progression defined by PCWG2 criteria Able to swallow and retain orally administered medication.
* Adequate baseline organ function.
* Must have a QT interval corrected for heart rate according to Fridericia's formula (QTcF) \<470 milli seconds (msec) or \<480 msec with bundle branch block.
* Male subject with a female partner of childbearing potential must have either had a prior vasectomy or agree to use effective contraception from time of Screening until 3 months after the last dose of study treatment.

Exclusion Criteria:

* Prior treatment with: anti-cancer therapy (e.g., chemotherapy with delayed toxicity, immunotherapy, biologic therapy or chemoradiation) within 21 days (or within 42 days if prior nitrosourea or mitomycin C containing therapy) prior to enrollment and/or daily or weekly chemotherapy without the potential for delayed toxicity within 14 days prior to enrolment. Subjects may remain on luteinizing hormone releasing hormone (LHRH) agonists (i.e., leuprolide, goserelin, triptorelin or histrelin). Subjects must have prior enzalutamide treatment; Any PI3K, AKT or mammalian target of rapamycin (mTOR) inhibitors; Investigational drug(s) within 30 days or 5 half-lives, whichever is longer, prior to enrollment
* Prior malignancy other than CRPC. Subjects who have been disease-free for 5 years, or subjects with a history of completely resected non-melanoma skin cancer or successfully treated in situ carcinoma are eligible.
* Current use of or anticipated requirement during the study of prohibited medication(s) (any investigational drug(s), Other anti-cancer therapy (chemotherapy, radiation therapy, immunotherapy, biologic therapy, or hormone therapy other than for replacement), AR antagonists (e.g., bicalutamide, flutamide, nilutamide), 5-alpha reductase inhibitors (e.g., finasteride, dutasteride), Androgens (e.g., testosterone, dihydroepiandrosterone), Herbal medication(s) that may affect PSA levels (e.g., saw palmetto), Other herbal medications including, but not limited to: St. John's wort, kava, ephedra (ma huang), gingko biloba, yohimbe and ginseng)
* Any unresolved \>=Grade 2 (per CTCAE v 4.0) toxicity from previous anti-cancer therapy at the time of enrollment, except alopecia or Grade 2 anemia (if hemoglobin is \>9.0 g/dL)
* Any \>=Grade 2 hypophosphatemia (per CTCAE v4.0) at the time of enrolment
* Serum calcium \>=Grade 1 (per CTCAE v4.0) at time of enrolment, unless ionized calcium is within normal range
* Presence of any clinically significant gastrointestinal (GI) abnormality or other condition(s) that may alter absorption such as malabsorption syndrome or major resection of the stomach or substantial portion of the small intestine
* Active peptic ulcer disease or history of abdominal fistula, GI perforation, or intra abdominal abscess within 28 days prior to enrolment
* Previous major surgery within 28 days prior to enrolment
* Known active infection requiring intravenous (IV) or oral anti-infective treatment
* Presence of hepatitis B surface antigen (HBsAg), positive hepatitis C antibody test result at screening or within 3 months prior to first dose of study treatment.
* A positive pre-study drug/alcohol screening (testing at time of screening is not required).
* A positive test for human immunodeficiency virus (HIV) antibody (testing at time of screening is not required).
* Evidence of severe or uncontrolled systemic diseases (e.g., unstable or uncompensated respiratory, hepatic, renal or cardiac disease)
* History of seizure or any condition that may predispose subject to seizure (e.g., prior cortical stroke or significant brain trauma). History of loss of consciousness or transient ischemic attack within 12 months of randomization
* History or evidence of cardiovascular risk including any of the following: Clinically significant electrocardiogram abnormalities including second degree (Type II) or third degree atrioventricular block; history of myocardial infarction, acute coronary syndromes (including unstable angina), coronary angioplasty, stenting, or bypass grafting within the past 6 months prior to enrolment, Class III or IV heart failure as defined by the New York Heart Association functional classification system, LVEF below 50%; known cardiac metastases
* Poorly controlled hypertension (defined as systolic blood pressure of\>=150 millimeter of mercury (mmHg) or diastolic blood pressure of \>100 mmHg based on a mean of three measurements at approximately 2-minute intervals)
* History of congenital platelet function defect (e.g., Bernard-Soulier syndrome, Chediak-Higashi syndrome, Glanzmann thrombasthenia, storage pool defect)
* Have a known immediate or delayed hypersensitivity reaction or idiosyncrasy to drugs chemically related to GSK2636771or enzalutamide or excipients.
* Any serious and/or unstable pre-existing medical, psychiatric disorder or other conditions that could interfere with subject's safety, obtaining informed consent or compliance to the study procedures.
* Exposure to more than 4 investigational medicinal products within 12 months prior to the first dose of study treatment