Erlotinib Hydrochloride in Treating Patients With Stage I-III Colorectal Cancer or Adenoma

Inclusion Criteria:

* Participants with one or more of the following criteria will be eligible to participate:

  * History of Stage I-III colorectal cancer, not treated in the past 6 months with no anticipated treatment in the next 3 months
  * Adenoma ≥ 1 cm in size
  * 3 or more adenomas (of any size) removed at one colonoscopy within past 6 years
  * Sessile serrated adenoma ≥ 5 mm in size
  * Adenoma (of any size) with villous features (villous, tubulovillous)
  * Adenoma (of any size) with high grade dysplasia
* Participants are eligible for randomization into the treatment phase of the trial if they are found to have ≥ 4 ACFs at either baseline colonoscopy or baseline flexible sigmoidoscopy
* Blood tests at screening which meet the following criteria:
* WBC \> 3000/mm\^3
* Platelets \> 100,000/mm\^3
* Hemoglobin \> 10g/dl
* Plasma creatinine of \< 1.6mg/dl
* Total bilirubin \< 1.5 x the upper limit of normal
* Serum ALT \< 1.5 x the upper limit of normal
* Serum AST \< 1.5 x the upper limit of normal
* ECOG performance status 0-1
* Women of child-bearing potential and men taking study drug must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry and for the duration of study participation
* Ability to understand, as well as sign the written informed consent document
* If a woman is of child-bearing potential, she must have a negative pregnancy test prior to study entry; should a woman become pregnant or suspect she is pregnant while participating in this study, she should inform her study physician immediately

Exclusion Criteria:

* History of Inflammatory Bowel Disease (IBD)
* History of interstitial lung disease or chronic lung disease
* Smoking within the past 3 months
* Increased bleeding risk from rectal biopsy (Patients receiving aspirin or plavix can be enrolled)
* Patients receiving warfarin or coumadin
* Uncontrollable diarrhea of any cause
* Patients, including rectal cancer patients, that have received prior radiation to the rectum or pelvis
* Participants taking a known significant CYP 3A4 inducer or inhibitor; known significant inducers/inhibitors include: amprenavir, aprepitant, atazanavir, carbamazepine, clarithromycin, conivaptan, diltiazem, darunavir/ritonavir, dronedarone, erythromycin, fluconazole, fosamprenavir, indinavir, itraconazole, ketoconazole, nefazodone, nelfinavir, phenytoin, posaconazole, rifampin, ritonavir, St. John's wort, saquinavir, telithromycin, tipranavir/ritonavir, verapamil, voriconazole
* Women who are pregnant or breast-feeding
* Active keratoconjunctivitis, or corneal surgery in the past three weeks
* Any medical or psychosocial condition that could jeopardize the subject's participation in and compliance to the study
* Participants who are taking any other investigational pharmaceutical agents
* Previous history of sensitivity to erlotinib, Iressa, or Erbitux, such as a rash that is uncontrollable by topical steroids and/or antibiotics