The output resolution of channel #0 can not be changed by Dashboard or device settings, the output resolution will always be the same as the input resolution.
This restriction is in place due to channel #0 can be set up to 60 fps.
To change the output resolution of channel #0, change the resolution on the input device.
As mentioned above, in custom settings on Dashboard, channel #0 Vertical resolution will be grayed out at “1080px”. However this does not mean the resolution is currently 1080p if the input device is not set to 1080p resolution.