Nivolumab and Lenalidomide in Treating Patients With Relapsed or Refractory Non-Hodgkin or Hodgkin Lymphoma

Inclusion Criteria:

* PHASE I: Histologically confirmed B-cell NHL with any of the following subtypes: DLBCL, mantle cell lymphoma (MCL), FL, marginal zone lymphoma (MZL) and lymphoplasmacytic lymphoma/Waldenstrom's macroglobulinemia (LL/WM), Burkitt's lymphoma (BL); patients with histological transformation to DLBCL from indolent lymphoma, primary mediastinal lymphoma and grey zone lymphoma are eligible
* PHASE I: Histologically confirmed classical or lymphocyte predominant Hodgkin's disease that is relapsed or refractory after at least one prior chemotherapy
* PHASE I: Patients must have received at least one prior therapy; prior autologous stem cell transplant is permitted; patients with DLBCL who have not had prior high-dose therapy (HDT)/autologous stem cell transplant (ASCT) must be ineligible for transplant; prior lenalidomide is not permitted if patients have progressed on therapy
* PHASE IB DOSE EXPANSION: Histologically confirmed classical or lymphocyte predominant Hodgkin's disease
* PHASE IB DOSE EXPANSION: Patients must have received at least one prior therapy; prior autologous stem cell transplant is permitted; patients who have not had prior HDT/ASCT must be ineligible for transplant; prior lenalidomide is not permitted if patients have progressed on therapy
* PHASE II: Histologically confirmed B-cell NHL:

  * Cohort 1: with only de novo DLBCL or transformed indolent Non-Hodgkin's lymphoma (excludes Richter's syndrome).
  * Cohort 2: with only FL of grade 1, 2 or 3a
* PHASE II: Patients must have received at least one prior therapy (in patients with transformed DLBCL must have received at least one prior therapy; prior autologous stem cell transplant is permitted; patients with DLBCL who have not had prior HDT/ASCT must be ineligible for transplant; prior lenalidomide is not permitted if patients have progressed on therapy
* Be willing and able to provide written informed consent/assent for the trial
* Have evaluable disease
* Eastern Cooperative Oncology Group (ECOG) performance status of 0-2
* Absolute neutrophil count (ANC) \>= 1,500 /mcL
* Platelets \>= 100,000 /mcL in the absence of transfusion support within 7 days of determining eligibility
* Hemoglobin \>= 8 g/dL
* Serum creatinine =\< 1.5 X upper limit of normal (ULN) OR measured or calculated creatinine clearance (glomerular filtration rate \[GFR\] can also be used in place of creatinine or creatinine clearance \[CrCl\]) \>= 40 mL/min creatinine clearance
* Serum total bilirubin =\< 1.5 X ULN OR except subjects with Gilbert syndrome, who can have total bilirubin \< 3.0 mg/dL
* Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\]) and alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 3.0 X ULN
* Female subject of childbearing potential should have a negative urine or serum pregnancy at screening and within 24 hours prior to receiving the first dose of study medication; if the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required
* Female subjects of childbearing potential should be willing to use 2 methods of birth control or be surgically sterile, or abstain from heterosexual activity for the course of the study through 120 days after the last dose of study medication; subjects should agree to ongoing pregnancy testing during the course of the study and after the end of study therapy; female subjects of childbearing potential are those who have not been surgically sterilized or have not been free from menses for \> 1 year
* Male subjects should agree to use an adequate method of contraception starting with the first dose of study therapy through 120 days after the last dose of study therapy; males must refrain from donating sperm during study participation and for 120 days after the last dose of study medication
* Willing and able to receive adequate prophylaxis and/or therapy for thromboembolic events
* Be willing and able to understand and give written informed consent and comply with all study related procedures

Exclusion Criteria:

* Is currently participating and receiving study therapy or has participated in a study of an investigational agent and received study therapy or used an investigational device within 3 weeks of the first dose of treatment
* Has a diagnosis of immunodeficiency or is receiving systemic steroid therapy or any other form of immunosuppressive therapy within 7 days prior to the first dose of trial treatment; subjects may use topical or inhaled corticosteroids or low-dose steroids (=\< 20 mg of prednisone or equivalent per day) as therapy for comorbid conditions; during study participation, subjects may receive systemic or enteric corticosteroids as needed for treatment-emergent comorbid conditions
* Has a known history of active TB (bacillus tuberculosis)
* Hypersensitivity to nivolumab or lenalidomide or any of their excipients
* Has had a prior anti-cancer monoclonal antibody (mAb) within 2 weeks prior to study day 1 or who has not recovered (i.e., =\< grade 1 or at baseline) from adverse events due to agents administered more than 4 weeks earlier
* Has had prior chemotherapy or radiation therapy within 2 weeks prior to study day 1 or who has not recovered (i.e., =\< grade 1 or at baseline) from adverse events due to a previously administered agent

  * Note: subjects with =\< grade 2 neuropathy are an exception to this criterion and may qualify for the study
  * Note: if subject received major surgery, they must have recovered adequately from the toxicity and/or complications from the intervention prior to starting therapy; patients must be 4 weeks out from major procedures
* Has a known additional malignancy that is progressing or requires active treatment; exceptions include basal cell carcinoma of the skin or squamous cell carcinoma of the skin that has undergone potentially curative therapy or in situ cervical cancer
* Has known active central nervous system (CNS) lymphoma
* Has active autoimmune disease that has required systemic treatment in the past 2 years (i.e. with use of disease modifying agents, corticosteroids or immunosuppressive drugs); replacement therapy (thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency, etc.) is not considered a form of systemic treatment; conditions expected to not recur in the absence of an external trigger
* Has an active infection requiring intravenous systemic therapy
* Is unable to swallow capsules or malabsorption syndrome, disease or condition significantly affecting gastrointestinal function
* Clinically significant cardiovascular disease with uncontrolled arrhythmia, New York Association class 3 or 4 congestive heart failure, history of myocardial infarction within 6 months, or prolonged corrected QT (QTc) \> 500 msec
* Is pregnant or breastfeeding, or expecting to conceive or father children within the projected duration of the trial, starting with the pre-screening or screening visit through 120 days after the last dose of trial treatment
* Has progressed on prior therapy with an anti-PD-1, anti-PD-L1, or anti-PD-L2 agent
* Has progressed on prior therapy with lenalidomide
* Has a known history of human immunodeficiency virus (HIV) (HIV 1/2 antibodies)
* Has known active hepatitis B (e.g., hepatitis B surface antigen \[HBsAg\] reactive) or hepatitis C (e.g., hepatitis C virus \[HCV\] ribonucleic acid \[RNA\] \[qualitative\] is detected)
* Has a history of deep venous thrombosis/embolism, threatening thromboembolism or known thrombophilia or are at a high risk for a thromboembolic event in the opinion of the investigator and who are not willing/able to take venous thromboembolic event prophylaxis during the entire treatment period
* Has a history of prior allogeneic hematopoietic cell transplant with a history of prior Graft Versus Host Disease