In more detail
The closest thing to a standard is RFC 4180: a value containing a comma, a double quote or a line break is wrapped in double quotes, and quotes inside are doubled. The first line usually holds column names, and lines officially end in CR LF, though LF alone is widely accepted. Some regions use semicolons because the comma is their decimal mark, and the file's text encoding is not recorded anywhere.
Why it matters
CSV has no types, so spreadsheet programs guess: leading zeros vanish from ZIP codes and IDs, long numbers turn into scientific notation and text such as 3-4 becomes a date. Opening a UTF-8 file without a byte-order mark can garble accented letters in some spreadsheet versions. Cells starting with =, +, - or @ can be run as formulas, so exported user data should be sanitised (CSV injection).
Example
A row with a comma and quotes inside, written by a standard CSV library:
id,name,quote
007,"Smith, Jane","She said ""hi"""
Common confusion
TSV (tab-separated values) uses tabs instead of commas and rarely needs quoting; "Excel file" (.xlsx) is a different, binary format.