A Dose Exploration Study With Birabresib (MK-8628) in Participants With Selected Hematologic Malignancies (MK-8628-005)

Inclusion Criteria:

* Diagnosis of AML (AML de novo and post-MDS) or DLBCL
* AML participants must have the following malignancy criteria: measurable and evaluable disease per tumor response criteria; ≥ 5% bone marrow blasts without alternate causality; and \> 90 days since allogeneic stem cell transplantation relapse in participants relapsing after transplant
* AML participants who are Philadelphia chromosome positive must have received ≥ 2 lines of therapy, including 2 bcr-abl tyrosine-kinase (TK) inhibitors (among imatinib, nilotinib and dasatinib), or only 1 line including 1 TK inhibitor if the relapse/refractoriness is associated with the detection of a resistance mutation to these inhibitors
* AML participants \< 60 years old must be in second or further relapse or relapsing after allogeneic stem cell transplantation regardless of number of relapses
* AML participants ≥ 60 years old in first relapse with a disease-free interval \< 12 months, or further relapse. First relapse is also applicable to AML post-MDS patients who have received prior treatment for MDS, but have not received prior treatment for AML.
* DLBCL participants must have the following malignancy criteria: measurable and evaluable disease per tumor response criteria and ≥ 1 tumor mass that is ≥ 15 mm (long axis of lymph node) or ≥ 10 mm (short axis of lymph node or extranodal lesions) on spiral CT scan; failed 2 standard lines of therapy (at least one containing an anti-CD20 monoclonal antibody), or for whom such treatment is contraindicated.
* Eastern Cooperative Oncology Group (ECOG) Performance Status ≤1
* An interval of ≥3 weeks since chemotherapy (≥ 6 weeks for nitrosoureas or mitomycin C), immunotherapy, hormone therapy or any other anticancer therapy or surgical intervention resection, or ≥3 half-lives for monoclonal antibodies, or ≥ 5 half-lives for other non-cytotoxic agents (whichever is longer)
* Female participants must not be pregnant (negative urine or serum human chorionic gonadotropin test within 72 hours of study start)
* Female and male participants of reproductive potential must agree to use adequate contraception starting from the first dose of trial treatment through 90 days after the last dose of study medication

Exclusion Criteria:

* Known primary central nervous system (CNS) malignancy or symptomatic or untreated CNS metastases
* History of prior or concomitant malignancies within 3 years of study start
* Has other serious illness or medical condition, such as active infection, unresolved bowel obstruction, psychiatric disorders, or cerebrovascular accident within 1 year of study start
* Known history of human immunodeficiency virus (HIV) and/or active Hepatitis B or C infections
* Has one of the following cardiac-related conditions: Congestive heart failure; angina pectoris; myocardial infarction (within 1 year of study start); uncontrolled hypertension; or uncontrolled arrhythmias
* Is receiving other concomitant anticancer treatment
* Has received high dose chemotherapy followed by autologous stem cell transplantation less than 90 days prior to first dose of study treatment
* Is receiving concomitant therapy with strong CYP3A4 or CYP2A6 inhibitors or inducers
* Is pregnant or breast-feeding
* Participation in a clinical trial involving an investigational drug within 30 days of study start
* Known additional malignancy that is progressing or requires active treatment
* Has been previously treated with a Bromodomain and Extra-terminal (BET) inhibitor
* Has acute promyelocytic leukemia, clinically uncontrolled disseminated intravascular coagulation, or peripheral cytopenia
* Has chronic graft versus host disease (GVHD) or on immunosuppressive therapy for the control of GVHD
* Has uncontrolled disease-related metabolic disorder
* Unable to swallow oral medications, or has gastrointestinal condition deemed to jeopardize intestinal absorption.