Vaccine Therapy and Sargramostim With or Without Docetaxel in Treating Patients With Metastatic Lung Cancer or Metastatic Colorectal Cancer

Inclusion Criteria:

* Histologically confirmed lung OR colorectal cancer
* Incurable metastatic disease
* Currently available standard treatment not likely to offer a survival advantage or result in superior palliation
* Evaluable disease by radiograph
* Tumor must currently express carcinoembryonic antigen (CEA) by immunohistochemistry OR CEA \>= 10 ng/mL at any point during disease course
* No clinically active brain metastases
* Must have had first- and second-line treatment OR declined second-line treatment (part I only)
* Patients with colon cancer must have had or have been offered treatment with oxaliplatin (part I only)
* ECOG 0-1
* Life expectancy of at least 4 months
* Absolute neutrophil count \>= 1,500/mm\^3
* WBC \>= 3,000/mm\^3
* Platelet count \>= 100,000/mm\^3
* Bilirubin normal
* Meets 1 of the following criteria:

  * SGOT and SGPT =\< 2.5 times upper limit of normal (ULN) AND alkaline phosphatase normal
  * SGOT and SGPT =\< normal AND alkaline phosphatase =\<4.0 times ULN
* Hepatitis B and C negative by clinical history and physical exam
* Creatinine =\< 1.5 mg/dL OR creatinine clearance \>= 60 mL/min
* Proteinuria =\< grade 1
* No known or suspected history of impaired cardiac function as evidenced by baseline echocardiogram
* Adequate pulmonary function
* No history or clinical evidence of immune deficiency or autoimmunity
* HIV negative
* No history of or concurrent diagnosis of any of the following:

  * Altered immunodeficiency
  * Eczema or other eczematoid skin disorders
  * Acute, chronic, or exfoliative skin condition (e.g., atopic dermatitis, burns, impetigo, varicella zoster, severe acne, or other open rashes or wounds)
* No history of allergy or untoward reaction to prior vaccination with vaccinia virus
* No history of severe hypersensitivity reaction to docetaxel or other drugs formulated with polysorbate 80
* No history of allergy to eggs or egg products
* No frequent vomiting or severe anorexia
* No inflammatory bowel disease
* No Crohn's disease
* No ulcerative colitis
* No active diverticulitis
* Neuropathy =\< grade 1 (sensory neuropathy)
* No uncontrolled seizure disorder
* No encephalitis
* No multiple sclerosis
* Must be maintaining a reasonable state of nutrition (=\< 10 % weight loss in the past month)
* Must be able to avoid close household contact (defined as sharing housing or having close physical contact) for at least 3 weeks after recombinant vaccinia vaccination with individuals with active or a history of eczema or other eczematoid skin disorders
* Must be able to avoid close household contact (defined as sharing housing or having close physical contact) for at least 3 weeks after recombinant vaccinia vaccination with those with unresolved acute, chronic, or exfoliative skin conditions (e.g., atopic dermatitis, burns, impetigo, varicella zoster, severe acne, or other open rashes or wounds)
* Must be able to avoid close household contact (defined as sharing housing or having close physical contact) for at least 3 weeks after recombinant vaccinia vaccination with any of the following individuals: pregnant or nursing women; children =\< 5 years of age; immunodeficient or immunosuppressed individuals (by disease or therapy), including HIV infection
* Not pregnant or nursing
* Negative pregnancy test
* Fertile patients must use effective contraception during and for at least 6 months after study participation
* No other concurrent serious medical illness that would preclude study participation
* No concurrent biologic therapy
* No other concurrent immunotherapy
* At least 6 weeks since prior nitrosoureas or mitomycin
* Prior docetaxel allowed (part I only)
* No prior docetaxel (part II only)
* No other concurrent chemotherapy
* No concurrent systemic steroids except for the following:

  * physiologic doses for systemic steroid replacement therapy
  * local (topical, nasal, or inhaled) steroid use
  * no concurrent steroid eye drops
  * premedication prior to and after docetaxel
* No concurrent hormonal therapy
* No prior radiotherapy to \> 50 % of all nodal groups
* More than 21 days since prior major surgery
* No prior splenectomy
* Recovered from prior therapy
* At least 3-4 weeks since prior cytotoxic therapy