There is some slight distortion on the jack sound output to my monitor. I've used a stereo male-male connector to connect the X86 to the monitor's input. I notice a slight humm, on the monitor. The connector on the X86 side doesn't seem to get completely inserted into the jack, as if one should press it a bit further, but can't.
If you didn't or couldn't insert the jack all the way, then you left a floating audio ground hence the distortion and hum (ground loop). The jack does work I have used it so either you have the wrong cable, you have something stuck in the jack, or you need to RMA the board. Find some headphones, check the jack first then go from there.
The jack details are covered in the manual on page 28 http://download.udoo.org/files/UDOO_X86/Doc/UDOO_X86_MANUAL.pdf The jack supports both normal stereo and stereo+ mono mic. As I call them 3 ring and 4 ring jacks The way that works is, a normal stereo only has 2 tips for left and right, but the sleeve is extra long so it should reach the ground and short the mic input to ground. However, in your case, you are failing for one reason or another to push the jack all the way in, so the ground is not making the connection. Instead it's connected to the mic input and this causes the wiring to pick up AC hum because of a ground loop condition between the UDOO X86 and your speakers/amp.
There is a little bit of the connector sticking out and don't think I can push it anyfurther without unreasonable force. This is also the case with the HDMI connector, USB. So I think this is as far as they go. I suspect it has something to do with grounding of either the monitor, or the Udoo. Though both are connected to a grounded wall socket.
So a couple of things. #1 many of the connectors had inserts in them during manufacture so the pick and place machine can attach them to the board and also prevent stray flux and debris from entering. I do not remember if the audio jack had this or what it looked like. If you failed to remove this and jammed in your audio connector you may have pushed this plug inside the jack and now it's preventing you from full insertion. Bottom line is only you with hands on the board can really figure out why you cannot plug that jack all the way in, but clearly, from both the symptom and the picture, that is the fault.