Alisertib (MLN8237) in Participants With Ovarian, Fallopian Tube or Peritoneal Cancer Preceded by Phase 1 Study of MLN8237 Plus Paclitaxel Treatment of Ovary or Breast Cancer

Inclusion Criteria:

Each participant must meet all of the following inclusion criteria to be enrolled in the study:

* Female participants 18 years or older
* Previously treated, metastatic or locally recurrent malignancy with 1 of the following diagnoses, which has been confirmed histologically or cytologically: adenocarcinoma of the breast (Phase 1 only), recurrent epithelial ovarian, fallopian tube, or primary peritoneal carcinoma (Phase 1 and 2)
* In the Phase 1 portion of the study, participants with breast cancer must have received treatment with at least 1 but no more than 4 prior chemotherapy regimens not including regimens received in the neoadjuvant and/or adjuvant setting
* Participants with breast cancer must have measurable disease per Response Evaluation Criteria in Solid Tumors (RECIST) 1.1
* No antineoplastic therapy or radiotherapy within 3 weeks before enrollment (2 weeks for regimens with recovery expected within 7 to 14 days) and recovered from toxicities of prior therapy (except alopecia); the participant must have recovered from all treatment-related toxicities and must have evidence of progressive disease (PD) or persistent disease
* Eastern Cooperative Oncology Group (ECOG) performance status 0 or 1
* Adequate bone marrow, liver and renal function
* Postmenopausal at least 1 year, OR Surgically sterile, OR If childbearing potential, agree to 2 effective methods of nonhormonal contraception, or agree to completely abstain from heterosexual intercourse
* Able to provide written informed consent
* Willing to comply with scheduled visits, treatment plan, laboratory tests and other trial procedures
* Suitable venous access

Specific Inclusion Criteria for participants with Recurrent Ovarian, Fallopian Tube or Peritoneal Cancer:

* Prior treatments must have included a platinum and a taxane; the most recent treatment need not be a platinum-containing or taxane-containing regimen
* Disease must have recurred ≤ 12 months after discontinuation of platinum therapy
* Participants who previously received weekly taxane are potentially eligible, provided that they did not progress during therapy or within 3 months of completing therapy
* Participants with platinum-refractory disease, as defined by progression during primary or subsequent platinum-based therapy or persistent radiographic disease after primary or subsequent platinum-based therapy, will be included
* Participants must have measurable disease in target lesions or assessable disease (defined by cancer antigen-125 - CA-125 per protocol), and disease progression per Response Evaluation Criteria in Solid Tumors (RECIST) version 1.1 or modified Gynecologic Cancer Intergroup (GCIG) CA-125 criteria

Exclusion Criteria:

Participants meeting any of the following exclusion criteria are not to be enrolled in the study:

* Prior treatment with an Aurora A-targeted agent (including MLN8237)
* Treatment with clinically significant enzyme inducers within 14 days prior to the first dose of MLN8237 and during the study
* Treatment with more than 4 cytotoxic chemotherapy regimens in the metastatic setting; prior therapy cannot include more than 2 prior taxane-containing regimen. Current use of tamoxifen, thalidomide, or any agent used as maintenance or consolidation therapy for OC.
* Known hypersensitivity to Cremophor® EL, paclitaxel or its components
* Prior history of ≥ Grade 2 neurotoxicity or any toxicity requiring discontinuation from taxane chemotherapy that is not resolved to ≤ Grade 1
* Comorbid or unresolved toxicity that would preclude administration of weekly paclitaxel
* Primary central nervous system malignancy or carcinomatous meningitis
* Symptomatic brain metastasis
* Inability to swallow oral medications or maintain a fast
* History of hemorrhagic or thrombotic cerebrovascular event in past 12 months
* Surgery within 3 weeks before study enrollment and not fully recovered
* Diagnosis or treatment of another malignancy within 2 years preceding first dose of MLN8237 and have any evidence of residual disease except nonmelanoma skin cancer or in situ malignancy completely resected
* Pregnant or lactating
* Serious illness that could interfere with protocol completion
* Investigational treatment 21 days prior to first dose of MLN8237
* Prior allogeneic bone marrow or organ transplantation
* Infection requiring systemic antibiotic therapy within 14 days prior to first dose of MLN8237
* Known history of human immunodeficiency virus (HIV) infection, hepatitis B, or hepatitis C
* Radiotherapy to \> 25% bone marrow or whole pelvic radiotherapy
* Requirement for constant administration of proton pump inhibitor, H2 antagonist, or pancreatic enzymes. Intermittent uses of antacids of H2 antagonists are allowed