The two ground planes (AGND and PGND) of LM5036 device should be tied together with a short and direct connection to avoid jitter due to relative ground bounce. The connection point could be at the negative terminal of the input power supply.
The VIN, VCC, REF pin capacitors, and CS_NEG resistor should be tied to PGND plane. UVLO, ON_OFF, RT, RON, RD1 and RD2 resistors, RAMP, RES, SS and SSSR capacitors, and the thermal pad should all be tied to AGND plane.
SW and SW_AUX are switching nodes which switch rapidly between VIN and GND every cycle which are sources of high dv/dt noise. Therefore, large SW/SW_AUX node area should be avoided.
The differential current sense signals at CS_POS and CS_NEG pins should be routed in parallel and close to each other to minimize the common-mode noise.
The area of the loop formed by the main feedback control signal traces (COMP and REF) should be minimized in order to reduce the noise pick up. This can be accomplished by placing the COMP and REF signal traces on top of each other in adjacent PCB layers. In addition, the main feedback control signal traces should be routed away from the SW_AUX switching node to avoid high dv/dt noise coupling.
The gate drive outputs (LSG and HSG) should have short and direct paths to the power MOSFETs to minimize parasitic inductance in the gate driving loop.
The VCC and REF decoupling capacitors should be placed close to their respective pins with short trace inductance. Low ESR and ESL ceramic capacitors are recommended for the boot-strap, VCC and the REF capacitors.
A decoupling capacitor should be placed close to the IC, directly across VIN and PGND pins. The connections to these two pins should be direct to minimize the loop area which carries switching currents.
The boot-strap capacitors required for the high-side gate drivers of the half-bridge converter and auxiliary supply should be located close to the IC and connected directly to the BST/BST_AUX and SW/SW_AUX pins.
The area of the switching loop of the power stage consisting of input capacitor, capacitive divider, transformer, and the primary MOSFETs should be minimized.