CTCF is a DNA-binding protein which plays critical roles in chromatin structure organization and transcriptional regulation; however, little is known about the functional determinants of different CTCF-binding sites (CBS). Using a conditional mouse model, we have identified one set of CBSs that are lost upon CTCF depletion (lost CBSs) and another set that persists (retained CBSs). Retained CBSs are more similar to the consensus CTCF-binding sequence and usually span tandem CTCF peaks. Lost CBSs are enriched at enhancers and promoters and associate with active chromatin marks and higher transcriptional activity. In contrast, retained CBSs are enriched at TAD and loop boundaries. Integration of ChIP-seq and RNA-seq data has revealed that retained CBSs are located at the boundaries between distinct chromatin states, acting as chromatin barriers. Our results provide evidence that transient, lost CBSs are involved in transcriptional regulation, whereas retained CBSs are critical for establishing higher-order chromatin architecture.