Durvalumab With or Without Lenalidomide in Treating Patients With Relapsed or Refractory Cutaneous or Peripheral T Cell Lymphoma

Inclusion Criteria:

* Documented informed consent of the participant and/or legally authorized representative
* Registered into Revlimid REMS program
* Eastern Cooperative Oncology Group (ECOG) performance status of 0-2
* Fully recovered from acute toxicities (except alopecia) of all prior therapies to Common Terminology Criteria for Adverse Events (CTCAE) =\< grade 1
* Relapsed/refractory disease
* Failed \>= 2 prior systemic therapies \*NOTE: For systemic ALCL prior systemic therapy must also include progression on brentuximab vedotin

CUTANEOUS T-CELL LYMPHOMA (CTCL) ONLY

* Histologically confirmed mycosis fungoides (MF) or Sezary syndrome (SS); Phase 1: \>= stage IIB OR \>= stage IB-IIA folliculotropic/transformed MF; Phase 2: \>= stage IB

  * Stage of disease according to TNMB classification
  * Pathology report must be diagnostic or be consistent with MF/SS criteria
  * SS is defined as meeting T4 plus B2 criteria; where the biopsy of erythrodermic skin may only reveal suggestive but not diagnostic histopathological features, the diagnosis may be based on either node biopsy or fulfillment of B2 criteria
  * For MF where the histological diagnosis by light microscopic examination is not confirmed, diagnostic criteria that has been recommended by the International Society of Cutaneous Lymphomas (ISCL) should be used
* Measurable disease per either mSWAT, Sezary count, or Lugano Classification (Section 11.2)
* Baseline skin biopsy taken within 6 months available for central review submission

PERIPHERAL T-CELL LYMPHOMA (PTCL) ONLY

* Histologically confirmed PTCL as defined by World Health Organization (WHO) 2008 criteria
* Measurable and/or evaluable disease per Lugano Classification (Section 11.2)
* Absolute neutrophil count (ANC) \>= 1000/mm\^3

  \* Growth factor is not permitted within 14 days of ANC assessment unless cytopenia is secondary to disease involvement
* Platelets \>= 100,000/mm\^3

  \* Platelet transfusions are not permitted within 14 days of platelet assessment unless cytopenia is secondary to disease involvement
* Total serum bilirubin =\< 2.2 mg/dL
* Aspartate aminotransferase (AST) =\< 2 x upper limit of normal (ULN)
* Alanine aminotransferase (ALT) =\< 2 x ULN
* Creatinine clearance of \>= 60 mL/min per the Cockcroft-Gault formula
* If not receiving anticoagulants: international normalized ratio (INR) AND prothrombin (PT) =\< 1.5 x ULN

  \* If on anticoagulant therapy: PT must be within therapeutic range of intended used of anticoagulants
* Female of childbearing potential: negative urine or serum pregnancy test

  \* If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required
* Female of child bearing potential: willing to use 2 methods of birth control or be surgically sterile, or abstain from heterosexual activity for the course of the study through 90 days after the last dose of study medication

  \* Childbearing potential defined as not being surgically sterilized or have not been free from menses for \> 1 year
* Male: use an adequate method of contraception starting with the first dose of study therapy through 90 days after the last dose of study therapy

Exclusion Criteria:

* Immunotherapy with immune checkpoint inhibitors, cell-based therapies, or cancer vaccines
* Lenalidomide, thalidomide or other immunomodulatory drugs (IMiDs)
* Monoclonal antibody within 5 half-lives of the antibody prior to initiating protocol therapy
* Any systemic therapy, including monoclonal antibody within 28 days or 5 half-lives (whichever is shorter) of initiating protocol therapy
* Any skin-directed therapy within 14 days prior to initiating protocol therapy
* Any radiation therapy within 21 days prior to initiating protocol therapy
* Immunosuppressive medication within 14 days prior to the first dose of study treatment; the following are exceptions to this criterion:

  * Intranasal, inhaled, topical or local steroid injections (e.g., intra-articular injection) and are on stable dose for at least 28 days
  * Systemic corticosteroids at physiologic doses of \< 10 mg/day of prednisone or equivalent
* Live, attenuated vaccine within 30 days prior to the first dose of protocol therapy
* History of pneumonitis (non-infectious) that required steroids or current pneumonitis
* Disease free of prior malignancies for \>= 5 years with the exception of:

  * Currently treated squamous cell and basal cell carcinoma of the skin
  * Carcinoma in situ of the cervix, or
  * Surgically removed melanoma in situ of the skin (stage 0) with histological confirmed free margins of excision or
  * Prostate cancer (T1a or T1b using the TNM \[tumor, nodes, metastasis\] clinical staging system) that has/have been surgically cured, or
  * Any other malignancy that has/have been curatively treated with surgery and/or localized radiation
* Allergic reaction/ hypersensitivity to thalidomide or to the excipients contained in the formulation of durvalumab
* Female only: pregnant or lactating
* Prior stem cell transplantation
* Acute infection requiring systemic treatment
* Known history of human immunodeficiency virus (HIV) infection
* Active hepatitis B or C infection
* Conditions requiring chronic steroid or immunosuppressive treatment that likely need additional steroid or immunosuppressive treatments in addition to the protocol therapy
* Current peripheral neuropathy \>= grade 2
* Renal failure requiring hemodialysis or peritoneal dialysis
* Unstable cardiac disease as defined by one of the following:

  * Cardiac events such as myocardial infarction (MI) within the past 6 months
  * NYHA (New York Heart Association) heart failure class III-IV
  * Uncontrolled atrial fibrillation or hypertension
* Major surgery (as defined by the investigator) within the 28 days prior to the first dose of study treatment
* Active or prior documented autoimmune or inflammatory disorders requiring therapy within the past 3 years prior to the start of treatment; the following are exceptions to this criterion:

  * Vitiligo or alopecia;
  * Hypothyroidism (e.g., following Hashimoto syndrome) stable on hormone replacement; or
  * Psoriasis not requiring systemic treatment
* History of primary immunodeficiency
* Incidence of gastrointestinal disease that may significantly alter the absorption of lenalidomide
* Any other condition that would, in the investigator's judgement, contraindicate the patient's participation in the clinical study due to safety concerns or compliance with clinical study procedures, e.g., infection/inflammation, intestinal obstruction, unable to swallow medication, social/psychological issues, etc
* In the opinion of the investigator, may not be able to comply with all study procedures (including compliance issues related to feasibility/logistics)