In more detail
A file begins with the bytes MZ (4D 5A), and a pointer near the start leads to the signature PE\0\0 (50 45 00 00). That pointer, called e_lfanew, sits at offset 0x3C. After the signature come the file header (with the target machine type, the number of sections and a build timestamp), the optional header (the entry point and the preferred load address) and a table describing each section. Sections such as .text (code), .data and .rsrc (resources) each have a size and an entropy that analysts inspect. See the Executable Forensics.
Why it matters
The headers explain how Windows will run the file, so they reveal a lot before anything executes: which libraries it needs (see Import table), whether code sits in an odd section, and whether a section looks compressed (see Packer). Fields can be forged, though. The build timestamp is set by the compiler and is easy to alter, so treat it as a clue rather than fact (see Compile timestamp).
Example
In a small 64-bit Windows launcher, cli-64.exe from the Python setuptools package, the layout is:
offset 0x00 4D 5A "MZ"
offset 0x3C E0 00 00 00 e_lfanew = 0xE0
offset 0xE0 50 45 00 00 "PE\0\0"
offset 0xE4 64 86 Machine = 0x8664 (x86-64)
It has four sections, .text, .rdata, .data and .pdata, with entropies of about 6.37, 5.44, 1.86 and 4.90 bits per byte. A 32-bit program shows machine type 0x014C instead. Plain code and data sit well below 8.0; a section near 8.0 suggests packed or encrypted content.
Common confusion
.exe, .dll and .sys are all PE files; the extension only hints at how Windows uses the file. The related Linux format is ELF, which starts 7F 45 4C 46.