A Study of the Drugs Prexasertib, Irinotecan, and Temozolomide in People With Desmoplastic Small Round Cell Tumor and Rhabdomyosarcoma

Inclusion Criteria:

* Consent/Assent: all patients and/or their parents or legally authorized representatives must sign written informed consent; assent, when appropriate, will be obtained according to institutional guidelines
* Age: patients must be ≥12 months of age at the time of study enrollment
* Diagnosis: patients must have histologically documented locally advanced or metastatic desmoplastic small round cell tumor or rhabdomyosarcoma (confirmed at MSK)
* Therapeutic options: patient's current disease state must be one which has failed standard therapy and for which there is no known curative therapy
* Disease Status: patients must have measurable disease based on RECIST 1.1
* Performance level: Karnofsky ≥ 50% for patients \> 16 years of age and Lansky ≥ 50 for patients ≤ 16 years of age
* Prior Therapy: patients may have had any number of prior therapies, but must have recovered from the acute toxic effects of all prior anti-cancer therapy (other than alopecia) as described below and must meet the following minimum duration from prior anti-cancer directed therapy prior to enrollment

  °patients who have previously received irinotecan and/or temozolomide will be allowed
  * 21 days must have elapsed after the last dose of cytotoxic or myelosuppressive chemotherapy
  * 7 days must have elapsed after the last dose of anti-cancer agents not known to be myelosuppressive
  * 14 days must have elapsed after radiation therapy, and toxicity related to prior radiation therapy must be recovered to grade ≤ 1
  * 21 days must have elapsed after the last dose of antibody therapy, and toxicity related to prior antibody therapy must be recovered to grade ≤ 1
* Organ Function Requirements: Adequate bone marrow function defined as:

  * absolute neutrophil count (ANC) ≥ 1500/mm\^3
  * platelet count ≥ 100,000/ mm\^3
  * hemoglobin ≥ 8 g/dl
* Adequate renal function defined as:

  * Creatinine clearance or radioisotope glomerular filtration rate ≥ 70 mL/min/1.73m2 OR
  * Serum creatinine based on age/gender derived from the Schwartz formula for estimating GFR53
* Adequate liver function defined as:

  * Bilirubin (sum of conjugated + unconjugated) ≤ 1.5 x upper limit of normal for age
  * AST or ALT ≤ 5 x upper limit of normal for patients with liver metastases
  * Serum albumin ≥ 2.5 g/dl
* Adequate cardiac function defined as:

  * echocardiogram with left ventricular ejection fraction (LVEF) \>45%
  * QTc \< 470 ms on screening 12 lead electrocardiogram
* Pregnancy/Contraception

  * post-menarchal females must have a negative urine or serum pregnancy test at screening and ≤ 24 hours prior to study treatment
  * males or females of reproductive potential must be willing to use a barrier method of contraception throughout the course of the study and for 6 months after participation

Exclusion Criteria:

* Patients for whom the investigator deems that irinotecan and temozolomide are not appropriate are not eligible.
* Patients who have an uncontrolled infection are not eligible.
* Patients who are pregnant or breast feeding are not eligible.
* Patients who have a history of Torsades de Pointes, carry a diagnosis of congestive heart failure, or have a family history of prolonged QT syndrome are not eligible.
* Patients who, in the opinion of the investigator, may not be able to comply with the safety monitoring requirements of the study are not eligible.
* Patients with known hypersensitivity to irinotecan or its excipients are not eligible.