Temsirolimus and Brentuximab Vedotin in Treating Patients With Relapsed or Refractory Hodgkin Lymphoma

Inclusion Criteria:

* Patients must have histologically confirmed cluster of differentiation (CD)30 positive relapsed or refractory Hodgkin lymphoma
* Measurable disease, defined as at least one lesion \> 1.5 cm, in the greatest transverse diameter
* Patients must have failed autologous stem cell transplant or at least 2 prior cytotoxic regimens for Hodgkin lymphoma
* Eastern Cooperative Oncology Group (ECOG) performance status =\< 2 (Karnofsky \>= 60%)
* Life expectancy of greater than 3 months
* Leukocytes \>= 3,000/mcL
* Absolute neutrophil count \>= 1,500/mcL
* Platelets \>= 100,000/mcL
* Total bilirubin within normal institutional limits or \< 3 x the upper limit of normal in patients with Gilbert's disease
* Aspartate aminotransferase (AST) (serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 × institutional upper limit of normal
* Creatinine =\< 1.5 x institutional upper limit of normal OR creatinine clearance \>= 40 mL/min/1.73 m\^2 for patients with creatinine levels above institutional normal
* Patients must have no evidence of dyspnea at rest, no exercise intolerance, and a pulse oximetry \> 92% while breathing room air
* Patients must have forced expiratory volume in 1 second (FEV1)/forced vital capacity (FVC) \> 60% by pulmonary function test (PFT), unless due to large mediastinal mass from Hodgkin's lymphoma (HL); carbon monoxide diffusion capacity (DLCO), FEV1, and FVC all \> 50% predicted value
* Women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation; should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately; men treated or enrolled on this protocol must also agree to use adequate contraception prior to the study, for the duration of study participation, and 4 months after completion of temsirolimus and brentuximab vedotin administration
* Ability to understand and the willingness to sign a written informed consent document
* Fasting serum cholesterol =\< 300 mg/dL OR =\< 7.75 mmol/L AND fasting triglycerides =\< 2.5 x upper limit of normal (ULN); NOTE: In case one or both of these thresholds are exceeded, the patient can only be included after initiation of appropriate lipid lowering medication
* Patients with known human immunodeficiency virus (HIV) infection must have CD4 count greater than 200; anti-retroviral agents that induce or inhibit cytochrome P450 (CYP)3A4 activity are not allowed

Exclusion Criteria:

* Patients who are eligible for autologous stem cell transplant unless they refuse to receive autologous stem cell transplant
* Patients who have had chemotherapy or radiotherapy within 3 weeks of registration or those who have not recovered from adverse events due to agents administered more than 3 weeks earlier; Note: patients are considered enrolled on the study after protocol registration and not after signing consent
* Patients who have received brentuximab vedotin within 6 months of enrolling on the study
* Patients who are receiving any other investigational agents
* Patients with known cerebral or meningeal involvement by lymphoma should be excluded from this clinical trial
* History of allergic reactions attributed to compounds of similar chemical or biologic composition to temsirolimus or brentuximab vedotin
* Patients receiving any medications or substances that are inhibitors or inducers of CYP3A4 are ineligible
* Uncontrolled intercurrent illness including, but not limited to, ongoing or active infection, uncontrolled diabetes, clinically significant pneumonitis, symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, or psychiatric illness/social situations that would limit compliance with study requirements
* Pregnant women are excluded from this study; breastfeeding should be discontinued if the mother is treated with temsirolimus and brentuximab vedotin
* Previous primary progression or grade 3 toxicity on treatment with brentuximab vedotin
* Patients receiving chronic, systemic treatment with corticosteroids or another immunosuppressive agent, except corticosteroids with a daily dosage equivalent to prednisone =\< 20 mg; topical or inhaled corticosteroids are allowed