In mammals, chromatin organization undergoes drastic reorganization during oocyte development. However, the dynamics of three-dimensional chromatin structure in this process is poorly characterized. Using low-input Hi-C (genome-wide chromatin conformation capture), we found that a unique chromatin organization gradually appears during mouse oocyte growth. Oocytes at late stages show self-interacting, cohesin-independent compartmental domains marked by H3K27me3, therefore termed Polycomb-associating domains (PADs). PADs and inter-PAD (iPAD) regions form compartment-like structures with strong inter-domain interactions among nearby PADs. PADs disassemble upon meiotic resumption from diplotene arrest but briefly reappear on the maternal genome after fertilization. Upon maternal depletion of Eed, PADs are largely intact in oocytes, but their reestablishment after fertilization is compromised. By contrast, depletion of Polycomb repressive complex 1 (PRC1) proteins attenuates PADs in oocytes, which is associated with substantial gene de-repression in PADs. These data reveal a critical role of Polycomb in regulating chromatin architecture during mammalian oocyte growth and early development.