Venetoclax and Decitabine in Treating Participants With Relapsed/Refractory Acute Myeloid Leukemia or Relapsed High-Risk Myelodysplastic Syndrome

Inclusion Criteria:

* Patients with AML, BPDCN, biphenotypic or bilineage leukemia (including a myeloid component) or mixed phenotype acute leukemia (MPAL) who have failed prior therapy; patients with AML should have failed prior therapy or have relapsed after prior therapy; patients with isolated extramedullary AML are eligible
* Elderly (\> 60 year old) patients with newly diagnosed AML, BPDCN, or mixed phenotype acute leukemia (MPAL) not eligible for intensive chemotherapy
* Patients with newly diagnosed AML with poor risk complex karyotype and/or TP53 deletions/mutations equal or younger than 60 year old
* AML or BPDCN patients with prior history of MDS or CMML who received any therapy or no therapy for the MDS or CMML and progressed to AML, are eligible at the time of diagnosis of AML regardless of any prior therapy for MDS; the World Health Organization (WHO) classification will be used for AML
* Patients with high-risk MDS with bone marrow blasts between 10% and 20%, relapsed or refractory to prior hypomethylating agent (HMA) therapy, defined as prior receipt of 4 cycles of HMA therapy with failure to attain a response, or relapse after prior response to HMA therapy; patients with high risk chronic myelomonocytic leukemia (CMML) with bone marrow blasts \>= 10% regardless of prior therapy
* Eastern Cooperative Oncology Group (ECOG) performance status =\< 3
* White blood cell count =\< 10,000
* Adequate renal function including creatinine \< 2 unless related to the disease
* Adequate hepatic function including total bilirubin \< 2 x upper limit of normal (ULN) unless increase is due to Gilbert's disease or leukemic involvement
* Alanine aminotransferase (ALT) \< 3 x ULN unless considered due to leukemic involvement
* Provision of written informed consent
* Oral hydroxyurea and/or one dose of cytarabine (up to 2 g/m\^2) for patients with rapidly proliferative disease is allowed before the start of study therapy and while the patient is on active study treatment through cycle 1, as needed, for clinical benefit and after discussion with the principal investigator (PI); concurrent therapy for central nervous system (CNS) prophylaxis or continuation of therapy for controlled CNS disease is permitted
* Females must be surgically or biologically sterile or postmenopausal (amenorrheic for at least 12 months) or if of childbearing potential, must have a negative serum or urine pregnancy test within 72 hours before the start of the treatment
* Women of childbearing potential must agree to use an adequate method of contraception during the study and until 3 months after the last treatment; males must be surgically or biologically sterile or agree to use an adequate method of contraception during the study until 3 months after the last treatment

Exclusion Criteria:

* Patients having received any prior BCL2 inhibitor therapy
* Patients with t(15;17) karyotypic abnormality or acute promyelocytic leukemia (French-American-British \[FAB\] class M3-AML)
* Patients with symptomatic CNS leukemia or patients with poorly controlled CNS leukemia
* Active and uncontrolled comorbidities including active uncontrolled infection, uncontrolled hypertension despite adequate medical therapy, active and uncontrolled congestive heart failure New York Heart Association (NYHA) class III/IV, clinically significant and uncontrolled arrhythmia as judged by the treating physician
* Patients with known infection with human immunodeficiency virus (HIV) or active hepatitis B or C
* Any other medical, psychological, or social condition that may interfere with study participation or compliance, or compromise patient safety in the opinion of the investigator
* Pregnant or breastfeeding