Mirvetuximab Soravtansine and Gemcitabine Hydrochloride in Treating Patients With FRalpha-Positive Recurrent Ovarian, Primary Peritoneal, Fallopian Tube, Endometrial, or Triple Negative Breast Cancer

Inclusion Criteria:

* All patients must have one of the following pathologically documented recurrent tumor types with FRalpha positivity by the Ventana immunohistochemistry (IHC):

  * Ovarian, primary peritoneal, fallopian tube (with exclusion of low grade, clear cell or sarcomatoid histologies for ovarian cancer) \>= 50% of tumor staining \>= 2+ intensity
  * Endometrial \>= 50% of tumor staining \>= 2+ intensity
  * TNBC confirmed by medical history of HER2-negative (confirmed by IHC 0, 1+ regardless of fluorescence in situ hybridization \[FISH\] ratio; IHC 2+ with FISH ratio \< 2.0 or HER2 gene copy \< 6.0; FISH ratio of 0, indicating gene deletion; when positive and negative in situ hybridization controls are present); estrogen receptor (ER) negative (confirmed as ER expression =\< 1% positive tumor nuclei); progesterone receptor (PR) negative (confirmed as PR expression =\< 1% positive tumor nuclei): \>= 25% of tumor staining \>= 1+ intensity
* Patients must have measurable disease, defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded for non-nodal lesions \>= 10 mm and short axis for nodal lesions \>= 15 mm); patients with recurrent ovarian, primary peritoneal, fallopian tube cancer may have biochemical relapse only, with baseline values of CA-125 at least 2 X upper limit of normal (ULN)
* Treatment with targeted agents, immunotherapy, or hormones is allowed; patients are only eligible if they have received and failed, or have been intolerant to standard treatments known to confer clinical benefit
* Life expectancy of greater than 3 months
* Absolute neutrophil count \>= 1.5 x 10\^9/L, determined within 14 days of registration
* Platelets \>= 100 x 10\^9/L, determined within 14 days of registration
* Total bilirubin =\< 1.5 x upper limit of normal (ULN), determined within 14 days of registration
* Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 X institutional upper limit of normal, determined within 14 days of registration
* Alkaline phosphatase =\< 2.5 X institutional upper limit of normal, determined within 14 days of registration
* Creatinine clearance \>= 50 mL/min/1.73 m\^2 for patients with creatinine levels above institutional normal, determined within 14 days of registration
* Women of childbearing potential must have a negative pregnancy test at screening and must agree to use adequate contraception prior to study entry, for the duration of study participation, and for 3 months after the last dose of IMGN853 and gemcitabine
* Patients must consent to analysis on archival tissue
* Ability to understand and the willingness to sign a written informed consent document
* Patients must have resolution of toxic effect(s) of the most recent prior chemotherapy to grade 1 or less (except alopecia)
* Cohort A: Patients with TNBC must have received no more than 4 lines of systemic cytotoxic chemotherapy; patients must have received and failed, or have been intolerant to anthracycline, taxanes, capecitabine, eribulin or other agents known to confer clinical benefit; patients are not required to fail all these agents if, in the investigator's opinion, patients would benefit from treatment on current protocol
* Cohort B: Patients with recurrent endometrial cancer may have received up to 2 lines of cytotoxic chemotherapy (adjuvant and one line for recurrent disease, or 2 lines of chemotherapy for recurrent uterine cancer in patients who did not receive adjuvant chemotherapy); patients must have received and failed, or have been intolerant to platinum agents, taxanes, liposomal doxorubicin or other agents known to confer clinical benefit; patients are not required to fail all these agents if, in the investigator's opinion, patients would benefit from treatment on current protocol
* Cohort C: Eligible patients must have received no more than 4 lines of systemic cytotoxic chemotherapy and must have disease resistant to platinum therapy (disease that progressed during or within six months of completing subsequent platinum therapy); primary platinum refractory patients are eligible providing they meet other eligibility criteria; in addition to platinum agents, patients must have received and failed, or have been intolerant to taxanes, liposomal doxorubicin or other agents known to confer clinical benefit; patients are not required to fail all these agents if, in the investigator's opinion, patients would benefit from treatment on current protocol
* Cohort C: Patients in this cohort only will require a single tumor biopsy 48-72H after the first administration of IMGN853 and gemcitabine on day 1, cycle 1 of treatment, providing it is safe/feasible and confers non-significant risk to patient

Exclusion Criteria:

* Previous treatment with gemcitabine
* Prior treatment with FR-targeting investigational agents is not allowed
* Patients who have had chemotherapy (including investigational cytotoxic chemotherapy), biologic agents (e.g., cytokines or antibodies) within 3 weeks (6 weeks for nitrosoureas or mitomycin C) before the first dose of study treatment
* Patients who have received radiation within 14 days before the first dose of study treatment
* Any other prior malignancy from which the patient has been disease free for less than 3 years, with the exception of adequately treated and basal or squamous cell skin cancer, superficial bladder cancer, carcinoma in situ of any site
* Patients with known brain metastases
* Serious concurrent illness or clinically-relevant active infection, including, but not limited to the following:

  * Known active hepatitis B or C
  * Known human immunodeficiency virus (HIV) infection
  * Varicella-zoster virus (shingles)
  * Cytomegalovirus infection
  * Any other known concurrent infectious disease, requiring IV antibiotics with 2 weeks of study enrollment
* Other intercurrent illness including, but not limited to symptomatic congestive heart failure and/or QT interval \> 470 for females and \> 450 for males, unstable angina pectoris, cardiac arrhythmia, hemorrhagic or ischemic stroke within the last 6 months or psychiatric illness/social situations that would limit compliance with study requirements
* History of interstitial pneumonitis
* History of cirrhotic liver disease
* Presence of \> grade 1 peripheral neuropathy
* Active or chronic corneal disorder, including but not limited to the following: Sjogren's syndrome, Fuchs corneal dystrophy (requiring treatment), history of corneal transplantation, active herpetic keratitis, and also active ocular conditions requiring on-going treatment/monitoring such as wet age-related macular degeneration requiring intravitreal injections, active diabetic retinopathy with macular edema, presence of papilledema, and acquired monocular vision
* Major surgery within 2 months prior to enrollment or minor surgery within 7 days of the first day of treatment
* History or evidence of thrombotic or hemorrhagic disorders within 6 months before first study treatment
* History of allergic reactions attributed to compounds of similar chemical or biologic composition to gemcitabine or IMGN853
* Pregnant women are excluded from this study; breastfeeding should be discontinued if the mother is treated with gemcitabine or IMGN853
* Required used of folate-containing supplements (e.g. folate deficiency)