Cixutumumab and Temsirolimus in Treating Patients With Locally Recurrent or Metastatic Breast Cancer

Inclusion Criteria:

* Histologically confirmed diagnosis of breast cancer with diagnosis of metastatic or locally recurrent disease (locally recurrent disease should be stage IV e.g. chest wall involvement)
* Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0 or 1 (Karnofsky \>= 80%)
* Life expectancy of \> 12 weeks
* Capable of understanding investigational nature, potential risks and benefits of the study and able to provide written informed consent
* Negative serum pregnancy test =\< 7 days of registration for women of childbearing potential:

  * Women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) for the duration of study therapy and for 3 months after the last dose of IMC-A12 and CCI-779 (temsirolimus)
  * Should a woman become pregnant or suspect she is pregnant while participating in this study, she should inform her treating physician immediately
  * Nursing women must be willing to discontinue nursing; NOTE: breastfeeding should be discontinued if the mother is treated with CCI-779 and IMC-A12
* Absolute neutrophil count \>= 1,500/mcL
* Hemoglobin \>= 8.5 g/dL
* Platelets \>= 100,000/mcL
* Total bilirubin =\< 1.5 X institutional upper limit of normal (ULN)
* Aspartate aminotransferase (AST)(serum glutamic oxaloacetic transaminase \[SGOT\]) and alanine aminotransferase (ALT)(serum glutamate pyruvate transaminase \[SGPT\]) =\< 3 x institutional ULN (=\< 5 X institutional ULN if liver function test \[LFT\] elevations due to liver metastases)
* Creatinine =\< 1.5 X institutional ULN OR creatinine clearance \>= 60 mL/min/1.73\^2 for patients with creatinine \> institutional ULN
* Fasting serum cholesterol =\< 350 mg/dL (9.0 mmol/L)
* Fasting triglycerides =\< 400 mg/dL (4.56 mmol/L)
* Albumin \>= 3.4 mg/dL
* Fasting or non fasting serum glucose \< 120 mg/dL
* Hemoglobin A1c (HbA1c) (for all patients with a history of diabetes mellitus) \< 8%
* Phase I only: Any number of prior therapy regimens is allowed
* Phase II only: Measurable disease is required for the Phase II portion of the study; measurable disease is defined as at least one lesion that can be accurately measured in at least one dimension (longest diameter to be recorded) as \>= 20 mm with conventional techniques (computed tomography \[CT\], magnetic resonance imaging \[MRI\], x-ray) or as \>= 10 mm with spiral CT scan
* Phase II only: =\< two and at least one prior chemotherapy regimens in the setting of metastatic or locally recurrent (stage IV chest wall involvement) disease are required

Exclusion Criteria:

* Phase I patients only: Patients with base line diabetes requiring oral hypoglycemics or insulin
* Phase II patients only: Poorly controlled diabetes mellitus; NOTE: patients with a history of diabetes mellitus on oral hypoglycemics or insulin are allowed to participate, provided that their fasting blood glucose is \< 120 mg/dL and that they are on a stable dietary or therapeutic regimen for this condition
* Any of the following:

  * Pregnant women
  * Nursing women
  * Men or women of childbearing potential who are unwilling to employ adequate contraception (hormonal agents are not allowed and oral contraceptives are not acceptable for contraception)
* Receiving hormonal agents used for the treatment of breast cancer with the exception that premenopausal women who have been on a gonadotropin-releasing hormone (GnRH) agonist and subsequently progressed may, at the discretion of the treating physician, continue on the GnRH agonist
* Any of the following prior therapies:

  * Systemic anti-cancer therapy =\< 3 weeks prior to registration
  * Radiation therapy =\< 2 weeks prior to registration
* Prior invasive non-breast malignancy, except for adequately treated basal or squamous cell carcinoma of the skin or other cancer from which the patient has been disease free for \>= 5 years
* Known hypersensitivity reactions to macrolide antibiotics (such as erythromycin, clarithromycin, and azithromycin); allergic reactions attributed to compounds of similar chemical or biologic composition to IMC-A12, or temsirolimus
* Prior treatment with agents targeting the insulin-like growth factor-I receptor (IGF-IR)/insulin-like growth factors (IGFs) or phosphatidylinositol-4,5-bisphosphate 3-kinase, catalytic subunit alpha (PI3K)/v-akt murine thymoma viral oncogene homolog 1 (Akt)/mechanistic target of rapamycin (mTOR) pathway
* Receiving any other investigational agents or herbal preparations
* Patients may not be taking oral corticosteroids except for replacement for adrenal insufficiency
* Uncontrolled brain metastases; Note: brain metastases are not permitted on study unless the metastases have been treated by surgery or radiotherapy, and the patient has been neurologically stable and off of steroids for \>= 12 weeks
* Known human immunodeficiency virus (HIV)-positive patients who have cluster of differentiation (CD)4 counts below the normal range or who are on anti-retroviral therapy that may interfere with the metabolism of temsirolimus
* Uncontrolled intercurrent illness including, but not limited to:

  * Ongoing or active infection
  * Symptomatic congestive heart failure
  * Unstable angina pectoris
  * Uncontrolled symptomatic cardiac arrhythmia
  * Psychiatric illness/social situations that would limit compliance with study requirements
* Receiving enzyme-inducing antiepileptic drugs (EIAEDs; e.g., phenytoin, carbamazepine, phenobarbital) or any other cytochrome P450, family 3, subfamily A, polypeptide 4 (CYP3A4) inducer such as rifampin or St. John's wort