GSK3174998 Alone and With Pembrolizumab in Participants With Advanced Solid Tumors (ENGAGE-1)

Inclusion Criteria:

* Provide signed, written informed consent.
* Male and female participants, age \>=18 years (at the time consent is obtained).
* Histological documentation of locally advanced, recurrent or metastatic solid malignancy that has progressed after standard therapy appropriate for the specific tumor type, or for which standard therapy has proven to be ineffective, intolerable, or is considered inappropriate. Participants should not have received more than 5 prior lines of therapy for advanced disease including both standards of care and investigational therapies. Participants whose cancers harbor molecular alterations for which targeted therapy is standard of care should have received health authority approved appropriate targeted therapy for their tumor types before enrollment.
* Participants with the following solid tumors are eligible for screening: Non-small cell lung cancer (NSCLC), Squamous cell carcinoma of the head and neck (SCCHN), Renal cell carcinoma (RCC), melanoma, bladder, Soft Tissue Sarcoma (STS), Triple-negative breast cancer (TNBC), and Colorectal carcinoma displaying high microsatellite instability (MSI CRC). In Part 2B (Cohort Expansion), specific subgroups of the above solid tumors will be studied. These subgroups may be defined by specific lines of treatment, types of prior treatment, histological subtypes, and may be enriched for selected biomarkers or participant characteristics. Populations to be studied in Amendment 3 include but are not limited to the following. Enrolment of additional populations will be communicated in writing: Participants with dedifferentiated liposarcoma who have not received prior treatment with a Programmed death ligand 1 (PD-L1) inhibitor; Participants with melanoma who have received a prior PD-L1 inhibitor, had a CR, PR or SD and subsequently progressed while on PD-L1 therapy. Participants who have received prior treatment with a PD-L1 inhibitor must have documented disease progression as defined by meeting all of the following criteria: Has received at least 2 doses of an approved PD-L1 inhibitor; has demonstrated disease progression as defined by Response Evaluation Criteria in Solid Tumors (RECIST) version 1.1. The initial evidence of disease progression is to be confirmed by a second assessment no less than four weeks from the date of the first documented PD , in the absence of rapid clinical progression; Progressive disease has been documented within 18 weeks from the last dose of the PD-L1 inhibitor.
* In Parts 1A and 2A, a biopsy of the tumor tissue obtained at anytime from the initial diagnosis to study entry. Although a fresh biopsy obtained during screening is preferred, archival tumor specimen is acceptable if it is not feasible to obtain a fresh biopsy.

Participants enrolled in Part 1A or Part 2A Pharmacodynamic Cohorts or in Part 2B of the study must provide a fresh biopsy of a tumor lesion not previously irradiated during the screening period and must agree to provide at least one additional on-treatment biopsy. In addition, an archived tumor tissue should be submitted for Participants in Part 2B, if available. The criterion for collection of fresh biopsies may be waived once GlaxoSmithKline (GSK)has determined an appropriate number of viable tissue samples have been analyzed For Part 1B and Part 2B, any archival tumor specimen must have been obtained within 3 months of starting study drug.

* Measurable disease as per RECIST v1.1
* Palpable lesions that are not measurable by radiologic or photographic evaluations may not be utilized as the only measurable lesion.
* Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0-1.
* Life expectancy of at least 12 weeks.
* Adequate organ function as defined by System Laboratory Values; Hematologic (Absolute neutrophil count \[ANC\] \>=1.5x10\^9/ liter \[L\], Lymphocyte count \>=800/cubic millimeter \[mm\^3\], Hemoglobin \>=9 grams/deciliter \[g/dL\], Platelets \>=100x10\^9/L), Hepatic (Total bilirubin \<=1.5x upper limit of normal \[ULN\] \[For participants with Gilbert's Syndrome, only if direct bilirubin \<=35 percent (%), \<=3.0xULN\], for Part 1A and 2A: alanine aminotransferase \[ALT\] \<=1.5xULN), Part 2B: ALT \<=2.5xULN; Renal (Serum Creatinine \<=1.5xULN OR Calculated creatinine clearance \[CrCl \>50 mL/min ) and Endocrine (Thyroid stimulating hormone \[TSH\]) within normal limits. If TSH is not within normal limits at baseline, the participant may still be eligible if total triiodothyronine (T3) or free T3 and free thyroxine (T4) are within the normal limits.
* QT duration corrected for heart rate by Fridericia's formula (QTcF) \<450 milliseconds (msec) or \<480 msec for participants with bundle branch block.
* In France, a participant will be eligible for inclusion in this study only if either affiliated to or a beneficiary of a social security category.
* Female participant: is eligible to participate if she is not pregnant (as confirmed by a negative serum beta-human chorionic gonadotrophin \[beta-hCG\] test), not lactating, and at least one of the following conditions applies: Non-reproductive potential defined as: Pre-menopausal females with one of the following: Documented tubal ligation; Documented hysteroscopic tubal occlusion procedure with follow-up confirmation of bilateral tubal occlusion ; Hysterectomy; Documented Bilateral Oophorectomy.

Postmenopausal defined as 12 months of spontaneous amenorrhea (in questionable cases a blood sample with simultaneous follicle stimulating hormone (FSH) and estradiol levels consistent with menopause). Females on hormone replacement therapy (HRT) and whose menopausal status is in doubt will be required to use one of the highly effective contraception methods if they wish to continue their HRT during the study. Otherwise, they must discontinue HRT to allow confirmation of post-menopausal status prior to study enrolment.

* Reproductive potential and agrees to follow one of the options listed below in the GSK Modified List of Highly Effective Methods for Avoiding Pregnancy in Females of Reproductive Potential (FRP) requirements from 30 days prior to the first dose of study medication and until 120 days after the last dose of study medication and completion of the follow-up visit. GSK Modified List of Highly Effective Methods for Avoiding Pregnancy in Females of Reproductive Potential (FRP).

This list does not apply to FRP with same sex partners, when this is their preferred and usual lifestyle or for participants who are and will continue to be abstinent from penile-vaginal intercourse on a long term and persistent basis: Contraceptive subdermal implant with a \<1% rate of failure per year, as stated in the product label; Intrauterine device or intrauterine system with a \<1% rate of failure per year, as stated in the product label; Oral Contraceptive, either combined or progestogen alone; Injectable progestogen; Contraceptive vaginal ring; Percutaneous contraceptive patches; Male partner sterilization with documentation of azoospermia prior to the female participant's entry into the study, and this male is the sole partner for that participant. These allowed methods of contraception are only effective when used consistently, correctly and in accordance with the product label. The investigator is responsible for ensuring that participants understand how to properly use these methods of contraception.

* Male Participants with female partners of child bearing potential must comply with the following contraception requirements from the time of first dose of study medication until 120 days after the last dose of study medication: Vasectomy with documentation of azoospermia; Male condom plus partner use of one of the contraceptive options below; Contraceptive subdermal implant with a \<1% rate of failure per year, as stated in the product label; Intrauterine device or intrauterine system with a \<1% rate of failure per year, as stated in the product label; Oral Contraceptive, either combined or progestogen alone Injectable progestogen; Contraceptive vaginal ring; Percutaneous contraceptive patches.

These allowed methods of contraception are only effective when used consistently, correctly and in accordance with the product label. The investigator is responsible for ensuring that participants understand how to properly use these methods of contraception.

Exclusion Criteria:

* Prior treatment with the following agents (from last dose of prior treatment to first dose of GSK3174998): Tumor necrosis factor receptor (TNFR) agonists, including OX40, CD27, CD137 (4-1BB), CD357 (GITR): at any time; Checkpoint inhibitors, including Programmed death receptor-1 (PD-1),

  1, and cytotoxic T-lymphocyte-associated antigen 4 (CTLA-4) inhibitors: within 4 weeks; other anticancer therapy, including chemotherapy, targeted therapy, and biological therapy: within 4 weeks or 5 half lives of the drug, whichever is shorter. Prior radiation therapy is permissible if at least one unirradiated measurable lesion is available for assessment via RECIST version 1.1. A wash out of at least two weeks before start of study drug for palliative radiation to the extremities for osseous bone metastases and 4 weeks for radiation to the chest, brain, or visceral organs is required; Investigational therapy: if the participant has participated in a clinical trial and has received an investigational product: within 30 days or 5 half-lives of the investigational product (whichever is shorter). At least 14 days must have passed between the last dose of prior investigational agent and the first dose of study drug.
* Prior allogeneic or autologous bone marrow transplantation or other solid organ transplantation.
* Toxicity from previous treatment: Participants with \>=Grade 3 toxicity related to prior immunotherapy leading to study treatment discontinuation are not eligible; participants whose toxicity related to prior treatment has not resolved to \<=Grade 1 (except alopecia, hearing loss, grade \<=2 neuropathy or endocrinopathy managed with replacement therapy) are not eligible.
* Malignancy other than disease under study, except as noted below: any other malignancy from which the participant has been disease-free for more than 2 years and, in the opinion of the principal investigators and GSK Medical Monitor, will not affect the evaluation of the effects of this clinical trial treatment on currently targeted malignancy, can be included in this clinical trial.
* Central nervous system (CNS) metastases, with the following exception: Participants who have previously-treated CNS metastases, are asymptomatic, and have had no requirement for steroids for 2 weeks prior to first dose of study drug.
* Has received transfusion of blood products (including platelets or red blood cells) or administration of colony stimulating factors (including granulocyte colony stimulating factor \[G-CSF\], granulocyte-macrophage colony-stimulating factor \[GMCSF\], recombinant erythropoietin) within 2 weeks before the first dose of study drug.
* Major surgery \<=4 weeks before the first dose of study treatment. Participants must have also fully recovered from any surgery (major or minor) and/or its complications before initiating study treatment.
* Active autoimmune disease that has required systemic treatment within the last 2 years (that is with use of disease modifying agents, corticosteroids or immunosuppressive drugs). Replacement therapy (example \[e.g.\], thyroxine or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment.
* Concurrent medical condition requiring the use of systemic immunosuppressive medications within 28 days before the first dose of study treatment. Physiologic doses of corticosteroids for treatment of endocrinopathies or steroids with minimal systemic absorption, including topical, inhaled, or intranasal corticosteroids may be continued if the participant is on a stable dose.
* Active infection, known human immunodeficiency virus infection, or positive test for hepatitis B surface antigen or hepatitis C.
* Current active liver or biliary disease (with the exception of Gilbert's syndrome or asymptomatic gallstones, liver metastases, or otherwise stable chronic liver disease per investigator assessment).
* Known, current drug or alcohol abuse.
* Recent history (within the past 6 months) of acute diverticulitis, inflammatory bowel disease, intra-abdominal abscess, or gastrointestinal obstruction.
* Receipt of any live vaccine within 4 weeks.
* Recent history of allergen desensitization therapy within 4 weeks of starting study Treatment.
* History of severe hypersensitivity to other mAbs.
* History or evidence of cardiovascular risk including any of the following: Recent (within the past 6 months) history of serious uncontrolled cardiac arrhythmia or clinically significant ECG abnormalities including second degree (Type II) or third degree atrioventricular block; Documented cardiomyopathy, myocardial infarction, acute coronary syndromes (including unstable angina pectoris), coronary angioplasty, stenting, or bypass grafting within the past 6 months before enrollment; documented congestive heart failure (Class II, III, or IV) as defined by the New York Heart Association functional classification system; recent (within the past 6 months) history of symptomatic pericarditis Current or history of idiopathic pulmonary fibrosis, interstitial lung disease, or organizing pneumonia.
* History of (non-infectious) pneumonitis that required steroids or current pneumonitis.
* Recent history (within 6 months) of uncontrolled symptomatic ascites or pleural effusions.
* Any serious and/or unstable pre-existing medical, psychiatric disorder, or other condition that could interfere with the participant's safety, obtaining informed consent, or compliance to the study procedures.
* Is or has an immediate family member (e.g., spouse, parent/legal guardian, sibling or child) who is investigational site or sponsor staff directly involved with this trial, unless prospective Institutional Review Board (IRB) approval (by chair or designee) is given allowing exception to this criterion for a specific participant.
* History of severe hypersensitivity (\>=Grade 3) to pembrolizumab and/or any of its excipients.