[SOLVED] High CPU Load

Discussion in 'Kernels' started by aureq, Oct 26, 2013.

  1. aureq

    aureq New Member

    Joined:
    Oct 25, 2013
    Messages:
    17
    Likes Received:
    0
    Hi

    I received my udoo a day ago. Pretty nice little thing.
    I immediately flashed udoo 1.2 image and then booted it.

    I immediately spotted one major issue. Somehow the CPU load average is very high.
    The load is at 4.00 and after a little bit of time, the udoo board becomes quite warn, so tyou can clearly tell the CPUs are running full time.
    Looking at the process list, none of the processes seem to consume that CPU, so it might be within the kernel perhaps.

    Has anyone experienced that issue and has been able to find a solution to that?

    I tried to stop x11 and few other processes, but same thing.

    Cheers
     
  2. lhl

    lhl New Member

    Joined:
    Jul 8, 2013
    Messages:
    24
    Likes Received:
    0
    Re: High CPU Load

    I'm seeing the same thing (high load). The CPU btw is mostly idle. You can double-check this with top/vmstat etc.
    I setup htop, dstat, iostat, sar, etc to try to figure out what's going on. It looks like the CPU is idle, it's not disk/swap that's causing waits, but there is an abnormally high number of system interrupts and context switches.

    I've tried stopping a bunch of services as well, but I can't find the issue (I haven't tried stopping the various network services though, which is what I'm suspecting where the issue lies).

    Besides the load # being off though, it doesn't seem to be doing much harm, although it'd be nice if someone could chime in on what's actually going on.
     
  3. aureq

    aureq New Member

    Joined:
    Oct 25, 2013
    Messages:
    17
    Likes Received:
    0
    Re: High CPU Load

    If by any chance any one is reading me...

    I stopped all the services, except sshd and dhclient.

    /proc/loadavg is like that
    Code:
    4.19 4.21 3.61 1/65 8740
    and top is like this
    Code:
    top - 01:29:30 up 29 min,  1 user,  load average: 4.06, 4.17, 3.64
    Tasks:  65 total,   1 running,  64 sleeping,   0 stopped,   0 zombie
    Cpu(s):  0.4%us,  0.7%sy,  0.0%ni, 97.9%id,  1.0%wa,  0.0%hi,  0.0%si,  0.0%st
    Mem:    636480k total,   147924k used,   488556k free,    10440k buffers
    Swap:        0k total,        0k used,        0k free,    98672k cached
    
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
     8741 root      20   0  2312  976  736 R    2  0.2   0:00.04 top
        1 root      20   0  3060 1416  956 S    0  0.2   0:02.15 init
        2 root      20   0     0    0    0 S    0  0.0   0:00.00 kthreadd
    So clearly:
    • the CPU is hot, something is "running"
    • which is correlated with the load average
    • but doesn't match the CPU line given by top
     
  4. maara

    maara Member

    Joined:
    Jun 11, 2013
    Messages:
    44
    Likes Received:
    4
    Re: High CPU Load

    I guess the cpuscaling is not working and posibly the cpu is throttleded to the max.
    What is your cpu governor?
    You can use cpufrequtils to view what is your cpu doing...
     
  5. Lifeboat_Jim

    Lifeboat_Jim New Member

    Joined:
    Sep 16, 2013
    Messages:
    399
    Likes Received:
    1
    Re: High CPU Load

    IIRC 'top' reads the loadavg. Indeed you can see the first line of your 'top' output shows similar load averages (circa 4).

    Sounds like a bug to me in 'top' not displaying all the tasks and processes? Perhaps it's looking at one core rather than all of them or something? 'top' has command line options... try working through those.

    Can't try it myself as I don't have my UDOO yet.
     
  6. aureq

    aureq New Member

    Joined:
    Oct 25, 2013
    Messages:
    17
    Likes Received:
    0
    Re: High CPU Load

    So, using cpufreq-info, apprently the selected governor is "ondemand" and the most used frequency is the lowest 396 MHz:99.55%.

    Using "ps -eaf", none of the processes seem to really the CPU as the TIME column is mostly around 0:00:00.

    Could it be possible it's a bug in the kernel and in the way the load average is computed ?
    Or, something related to either network and/or interrupt ?

    the cpufreq-info output
    Code:
    cpufrequtils 007: cpufreq-info (C) Dominik Brodowski 2004-2009
    Report errors and bugs to cpufreq@vger.kernel.org, please.
    analyzing CPU 0:
      driver: imx
      CPUs which run at the same hardware frequency: 0 1 2 3
      CPUs which need to have their frequency coordinated by software: 0 1 2 3
      maximum transition latency: 61.0 us.
      hardware limits: 396 MHz - 996 MHz
      available frequency steps: 996 MHz, 792 MHz, 396 MHz
      available cpufreq governors: interactive, conservative, ondemand, userspace, powersave, performance
      current policy: frequency should be within 396 MHz and 996 MHz.
                      The governor "ondemand" may decide which speed to use
                      within this range.
      current CPU frequency is 396 MHz (asserted by call to hardware).
      cpufreq stats: 996 MHz:0.33%, 792 MHz:0.02%, 396 MHz:99.64%  (57)
    analyzing CPU 1:
      driver: imx
      CPUs which run at the same hardware frequency: 0 1 2 3
      CPUs which need to have their frequency coordinated by software: 0 1 2 3
      maximum transition latency: 61.0 us.
      hardware limits: 396 MHz - 996 MHz
      available frequency steps: 996 MHz, 792 MHz, 396 MHz
      available cpufreq governors: interactive, conservative, ondemand, userspace, powersave, performance
      current policy: frequency should be within 396 MHz and 996 MHz.
                      The governor "ondemand" may decide which speed to use
                      within this range.
      current CPU frequency is 396 MHz (asserted by call to hardware).
      cpufreq stats: 996 MHz:0.33%, 792 MHz:0.02%, 396 MHz:99.64%  (57)
    analyzing CPU 2:
      driver: imx
      CPUs which run at the same hardware frequency: 0 1 2 3
      CPUs which need to have their frequency coordinated by software: 0 1 2 3
      maximum transition latency: 61.0 us.
      hardware limits: 396 MHz - 996 MHz
      available frequency steps: 996 MHz, 792 MHz, 396 MHz
      available cpufreq governors: interactive, conservative, ondemand, userspace, powersave, performance
      current policy: frequency should be within 396 MHz and 996 MHz.
                      The governor "ondemand" may decide which speed to use
                      within this range.
      current CPU frequency is 396 MHz (asserted by call to hardware).
      cpufreq stats: 996 MHz:0.33%, 792 MHz:0.02%, 396 MHz:99.64%  (57)
    analyzing CPU 3:
      driver: imx
      CPUs which run at the same hardware frequency: 0 1 2 3
      CPUs which need to have their frequency coordinated by software: 0 1 2 3
      maximum transition latency: 61.0 us.
      hardware limits: 396 MHz - 996 MHz
      available frequency steps: 996 MHz, 792 MHz, 396 MHz
      available cpufreq governors: interactive, conservative, ondemand, userspace, powersave, performance
      current policy: frequency should be within 396 MHz and 996 MHz.
                      The governor "ondemand" may decide which speed to use
                      within this range.
      current CPU frequency is 396 MHz (asserted by call to hardware).
      cpufreq stats: 996 MHz:0.33%, 792 MHz:0.02%, 396 MHz:99.64%  (57)
    Thank you,
     
  7. HixVAC

    HixVAC New Member

    Joined:
    Oct 25, 2013
    Messages:
    13
    Likes Received:
    0
    Re: High CPU Load

    I'm in the same category. She is hot as hell...I just thought it was a really crumby heatsink the I realized there's no way that's possible after I threw on a temporary full PC heatsink.
     
  8. aureq

    aureq New Member

    Joined:
    Oct 25, 2013
    Messages:
    17
    Likes Received:
    0
    Re: High CPU Load

    Mine is warm, but I can't say it's hot at all.
     
  9. aureq

    aureq New Member

    Joined:
    Oct 25, 2013
    Messages:
    17
    Likes Received:
    0
    Re: High CPU Load

    To all,

    I checked for the same behaviour with UDOO Android 1.0 and I get exactly the same.
    The board is warm, not hot, and the load is slightly above 4.00. This is show from /proc/loadavg and when enabling a developer option to show the load in the top right corner.
     
  10. pko

    pko New Member

    Joined:
    Sep 13, 2013
    Messages:
    19
    Likes Received:
    0
    Re: High CPU Load

    If you look at the actual top command output:

    Cpu(s): 0.4%us, 0.7%sy, 0.0%ni, 97.9%id, 1.0%wa, 0.0%hi, 0.0%si, 0.0%st

    It actually say CPU is 97.9% idle. So there is no need to panic.
     
  11. aureq

    aureq New Member

    Joined:
    Oct 25, 2013
    Messages:
    17
    Likes Received:
    0
    Re: High CPU Load

    Yes @pko, I realised that about 2 hours ago when I managed to run xbmc from on the android.

    Though, when you have no references, seeing a load average of 4.00 is somewhat weird.
     
  12. pko

    pko New Member

    Joined:
    Sep 13, 2013
    Messages:
    19
    Likes Received:
    0
    Re: High CPU Load

    Yes, it is strange. But without looking at the actual system. It is hard to know what is happening. CPU load is typically a reflection of how many job is waiting to be executed by the CPU. However, the jobs may be waiting for something else e.g. waiting for I/O. It also depend on how many core is in your system. If you don't mind, i would like to see the output of the command

    cat /proc/cpuinfo
     
  13. DracoLlasa

    DracoLlasa UDOOer

    Joined:
    Oct 15, 2013
    Messages:
    419
    Likes Received:
    3
    Re: High CPU Load

    The I/O Wait could be a contributor especially if you are not using a high end MicroSD card.
    Previous testing we have done here in the forums have shown that the latency of the SD card slows down a lot of things even preventing the WiFi from working at full speed since the SD card can only read/write so fast. If you are not at a Class 10 then it might be a place to start looking.

    Also regarding the thermal output, i have completed my testing and you can find the details in this thread showing how hot the heatsinks will get under various loads
    viewtopic.php?f=2&t=155
     
  14. lhl

    lhl New Member

    Joined:
    Jul 8, 2013
    Messages:
    24
    Likes Received:
    0
    Re: High CPU Load

    @pko here's the /proc/cpuinfo on my quad if you're curious (although it doesn't have much to do w/ anything):

    Code:
    Processor	: ARMv7 Processor rev 10 (v7l)
    processor	: 0
    BogoMIPS	: 790.52
    
    processor	: 1
    BogoMIPS	: 790.52
    
    processor	: 2
    BogoMIPS	: 790.52
    
    processor	: 3
    BogoMIPS	: 790.52
    
    Features	: swp half thumb fastmult vfp edsp neon vfpv3
    CPU implementer	: 0x41
    CPU architecture: 7
    CPU variant	: 0x2
    CPU part	: 0xc09
    CPU revision	: 10
    
    Hardware	: SECO i.Mx6 UDOO Board
    Revision	: 63012
    Serial		: 101981d4d72d78c5
    
    @DracoLlasa it's not I/O - load is at 4 even when completely idle. iostat shouws %steal at 0.00, vmstat shows swap and io at 0/0 and close to 0/0. The only stat that's elevated is the system interrupts and context switches - this is at about ~300 for both, which is about 2-3x a similarly idle Wandboard Quad (another, very similar iMX.6 Quad dev board).

    These devices are both idling at 396MHz (cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq) - the UDOO Quad measures slightly cooler w/ an IR heat gun: 96 vs 103 degrees F (36 vs 40 C) - pretty neglible.

    It *is* a bit annoying though as the baseline 4 load will make it really confusing when checking how hard the system is really working. Hopefully that kink can get fixed soon. Something weird with the custom kernel?
     
  15. panicking

    panicking New Member

    Joined:
    Oct 28, 2013
    Messages:
    1
    Likes Received:
    0
    Re: High CPU Load

    Hi all,

    I was in Lucca two days ago and I have debugged the pre-built image. Well the problem is the support is far away to be completed:

    - the kernel register and support only one cpu
    - frequency scaling is in performance mode
    - supported frequency are 3 and under the 1Ghz

    I think that you need some professional support

    Michael
     
  16. andcmp

    andcmp New Member

    Joined:
    May 8, 2013
    Messages:
    161
    Likes Received:
    0
    Re: High CPU Load

    Could you elaborate this?
    UDOO features two CPUs, but the CPU who manages the operative system is just one, with 2 or 4 cores.

    The cpufreq governor is "ondemand" by default.

    True, 396MHz, 792MHz and 996MHz.

    You can check these info with the following command:
    Code:
    cpufreq-info
    Of curse we know that UDOO's kernel and software in general is something we need to improve, any help is much appreciated!
     
  17. maara

    maara Member

    Joined:
    Jun 11, 2013
    Messages:
    44
    Likes Received:
    4
    Re: High CPU Load

    I am facing similar sitaution - the system is absolutely idle (no user and system busy) no consumers etc. but the load is 4-6 even nothing is running.
    The heatsink is quite hot and the wifi device is hot. According to the cpu info I am running ondemand and the cores are at their lowest level what is strange that on 300mhz they would produce so much heath....
    The situation is quite suspicious...- when there is no activity when there is so much load on the idle system and why does is the temperature so high???
     
  18. trunet

    trunet New Member

    Joined:
    Oct 25, 2013
    Messages:
    17
    Likes Received:
    0
  19. maara

    maara Member

    Joined:
    Jun 11, 2013
    Messages:
    44
    Likes Received:
    4
    Re: High CPU Load

    Great find!
    I checked the source and the commit is not merged to the udoo kernel source.... Is anyone able to merge/add lines and compile a new kernel to try it please? ;)
     
  20. Lifeboat_Jim

    Lifeboat_Jim New Member

    Joined:
    Sep 16, 2013
    Messages:
    399
    Likes Received:
    1
    Re: High CPU Load

    I'll mention it in IRC but it's down to the UDOO staff/mods here to pass it on/action it
     

Share This Page