The "30% CPU at idle because of System + Interrupts" issue

[SteveBorough]

Dunno why you started a new thread for this but as per the original thread I started (http://www.surfaceforums.net/forum/...ation-when-idle-due-faulty-audio-drivers.html) most (all) of us cannot replicate the issue if the SD card is not in... There is clearly a link between the SD card and sound drivers (both Realtek devices) since playing any sound momentarily stops the high-idle as does disabling the Related HD Audio device from Device Manager (or as aforementioned, booting without an SD card)

 

[javispedro]

Dunno why you started a new thread for this but as per the original thread I started (http://www.surfaceforums.net/forum/...ation-when-idle-due-faulty-audio-drivers.html) most (all) of us cannot replicate the issue if the SD card is not in... There is clearly a link between the SD card and sound drivers (both Realtek devices) since playing any sound momentarily stops the high-idle as does disabling the Related HD Audio device from Device Manager (or as aforementioned, booting without an SD card)

Well in this thread at least two of us now can replicate the issue as much as we want without any SD card plugged in.

But if you really suspect the Realtek SD card reader drivers, why don't you just remove them as I mentioned above? I did that this morning, and as expected, the only thing that happened is that I lost the SD card specific icon in "My Computer"... but drive D: is still there, and the files are still there.

Seriously, the SD card is a EL-CHEAPO USB card reader, with the only difference that it is soldered to the motherboard! From the operating system point of view it is exactly as any other USB pendrive! The ONLY thing the Realtek SD card reader driver does is put a nicer icon in explorer!


That said.

I have been bitten by so many weird Windows bugs that I wouldn't discard the chance that even such a simple icon-changing driver would cause problems in a completely unrelated area. (e.g. such as that one USB Tablet driver that caused the icons in the notification area to become black and white back in Windows 2k days).

If you really believe the SD card reader is related, I suggest you remove the Realtek card reader driver using the above instructions.

But in my opinion, it's not the cause, and it shouldn't be the cause.


(BTW, I haven't experienced "The Issue" ever since I removed the driver this morning, but it's been two boots only so far. At this point I'm hoping it fails soon, otherwise I will have to backpedal on my words ).

EDIT: Ok, I got the 30% CPU just now. To recap,
removing the Realtek SD card reader drivers has no effect on the problem here.

 

[SteveBorough]

Well in this thread at least two of us now can replicate the issue as much as we want without any SD card plugged in.

But if you really suspect the Realtek SD card reader drivers, why don't you just remove them as I mentioned above? I did that this morning, and as expected, the only thing that happened is that I lost the SD card specific icon in "My Computer"... but drive D: is still there, and the files are still there.

Seriously, the SD card is a EL-CHEAPO USB card reader, with the only difference that it is soldered to the motherboard! From the operating system point of view it is exactly as any other USB pendrive! The ONLY thing the Realtek SD card reader driver does is put a nicer icon in explorer!


That said.

I have been bitten by so many weird Windows bugs that I wouldn't discard the chance that even such a simple icon-changing driver would cause problems in a completely unrelated area. (e.g. such as that one USB Tablet driver that caused the icons in the notification area to become black and white back in Windows 2k days).

If you really believe the SD card reader is related, I suggest you remove the Realtek card reader driver using the above instructions.

But in my opinion, it's not the cause, and it shouldn't be the cause.


(BTW, I haven't experienced "The Issue" ever since I removed the driver this morning, but it's been two boots only so far. At this point I'm hoping it fails soon, otherwise I will have to backpedal on my words ).

EDIT: Ok, I got the 30% CPU just now. To recap,
removing the Realtek SD card reader drivers has no effect on the problem here.

As the title of the thread I started suggests, the real issue is with the sound driver, since disabling the Realtek HD audio device is the only thing that 100% fixes the issue. I've never stated it was the SD card drivers since I already tried removing/replacing them back when this issue first arose for me. Perhaps the sound driver needlessly/mistakenly queries for an SD card or something in the Realtek USB Card reader chipset during power-on that is causing the interrupt-flood. The SD card device is not present when there isn't a card present and therefore doesn't get queried by the sound drivers during boot. Have you upgraded or changed your Realtek USB drivers where it somehow remains present even though a card is not in?

 

[javispedro]

Note that the interrupt flood is being caused by ACPI SMI interrupts, according to LatencyMon (first post from this thread) and xperf.
And in my opinion, this is caused by the ACPI AML interpreter getting stuck in a method that repeatedly causes a SMI interrupt to be fired.

I still doubt it is audio related. There a bunch of other things that change when playing audio that are ACPI related, e.g. some low power states are disabled. My next step would be to try to see what happens when the audio driver is disabled and if I can reproduce the issue in that situation.

I've tried using the ETW "Acpi Driver Trace Provider" to try and see if I could figure out which method is being run to no avail. There is hardly any documentation available.

If I could reproduce the issue under Gentoo...

As the title of the thread I started suggests, the real issue is with the sound driver, since disabling the Realtek HD audio device is the only thing that 100% fixes the issue. I've never stated it was the SD card drivers since I already tried removing/replacing them back when this issue first arose for me. Perhaps the sound driver needlessly/mistakenly queries for an SD card or something in the Realtek USB Card reader chipset during power-on that is causing the interrupt-flood. The SD card device is not present when there isn't a card present and therefore doesn't get queried by the sound drivers during boot. Have you upgraded or changed your Realtek USB drivers where it somehow remains present even though a card is not in?

The card reader turns off when there is no card present, as do a lot of other card readers, so I don't think that's possible.

Well as said there is nothing special with the SD card reader on the Surface, it is basically like any other USB pendrive from a logical PoV. So if you can only reproduce this when there is a card present, my theory is that you should be also able to reproduce it when there is any other USB pendrive present.

Otherwise we're entering NSA territory where the sound card driver is speaking a proprietary protocol with the card reader (and I really doubt this).

 

[beq]

If you remove the microSD card and restart, does it happen again? I think the moment you put in the card, even if it's ejected after, the system jams somewhere (probably a driver)

Yeah I had rebooted multiple times since then and I remember still noticing the "System" process sustained spike occasionally afterward.

Anyways I realize everyone's saying it's a driver problem, but long shot question -- do you all still see this even after the Windows 8.1 Update 1?

 

[SteveBorough]

Yeah I had rebooted multiple times since then and I remember still noticing the "System" process sustained spike occasionally afterward.

Anyways I realize everyone's saying it's a driver problem, but long shot question -- do you all still see this even after the Windows 8.1 Update 1?

Jes!

 

[be77solo]

I concur. I've reproduced it several times with no SD card whatsoever, and as mentioned, there's even a logical reasoning for why it can't actually be SD card related.

As for audio, have you tried what I mentioned on the above post? Make a mental note for the next time it happens to you

"Delay after boot" seems interesting.

Ok, did some more tinkering when this just happened again, system and system interrupt 20-30%:

Was using SP2 all morning in dock without issue. Went to bank and returned to SP2 asleep. Woke up and removed from dock, removed power cover and used as tablet. About 15 minutes later noticed back getting warm so checked task manager, sure enough.

At this point checked file explorer and my mSD card was not being recognized although I haven't touched it and it has been fine all day. Odd

So, play a song in Music based on info in this thread and sure enough, instantly CPU usage drops to 1%.

Stop song and close Music, instantly back to 30% CPU.

Physically eject mSD card, no change.

Insert same mSD card, now recognized again, but no CPU change.

Finally tap power button, SP2 goes to sleep, tap power button 5 seconds later, wakes right up and CPU usage is now back to normal 0-1%.

Typing this on SP2 and CPU remains at 0% 20 minutes later. Still have the mSD card inserted and recognized.

This is after the latest Update 1 along with all MS updates.

Edit: Update to post, 5 minutes after posting this while casually browsing, CPU usage is back to 30%. mSD is still recognized and functional. Now going to reboot.

 

[macmee]

Ok, did some more tinkering when this just happened again, system and system interrupt 20-30%:

Was using SP2 all morning in dock without issue. Went to bank and returned to SP2 asleep. Woke up and removed from dock, removed power cover and used as tablet. About 15 minutes later noticed back getting warm so checked task manager, sure enough.

At this point checked file explorer and my mSD card was not being recognized although I haven't touched it and it has been fine all day. Odd

So, play a song in Music based on info in this thread and sure enough, instantly CPU usage drops to 1%.

Stop song and close Music, instantly back to 30% CPU.

Physically eject mSD card, no change.

Insert same mSD card, now recognized again, but no CPU change.

Finally tap power button, SP2 goes to sleep, tap power button 5 seconds later, wakes right up and CPU usage is now back to normal 0-1%.

Typing this on SP2 and CPU remains at 0% 20 minutes later. Still have the mSD card inserted and recognized.

This is after the latest Update 1 along with all MS updates.

So based on your observations do you think the problem is that the mSD adapter ends up in this weird state where the card is inserted and not recognized, and when in this state your device is eating away firing interrupts?

 

[HypermagiK]

Traces from the Microsoft-Windows-Kernel-Acpi provider are filled with the following types of events while this issue is in effect:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
	<EventData>
		<Data Name="GpeRegister">0x0</Data>
		<Data Name="UnexpectedEventMap">60</Data>
	</EventData>
	<RenderingInfo Culture="en-US">
		<Level>Information </Level>
		<Task>GpeEventHandling</Task>
		<Message>Unexpected GPE event was fired on GPE bits that should be disabled. </Message>
	</RenderingInfo>
</Event>

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
	<EventData>
		<Data Name="AmlMethodNameLength">10</Data>
		<Data Name="AmlMethodName">\_GPE._L6D</Data>
		<Data Name="AmlMethodState">1</Data>
	</EventData>
	<RenderingInfo Culture="en-US">
		<Level>Information </Level>
		<Task>AmlMethodTrace</Task>
		<Message>ACPI method \_GPE._L6D evaluation has started . </Message>
	</RenderingInfo>
</Event>

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
	<EventData>
		<Data Name="AmlMethodNameLength">10</Data>
		<Data Name="AmlMethodName">\_GPE._L6D</Data>
		<Data Name="AmlMethodState">0</Data>
	</EventData>
	<RenderingInfo Culture="en-US">
		<Level>Information </Level>
		<Task>AmlMethodTrace</Task>
		<Message>ACPI method \_GPE._L6D evaluation has finished . </Message>
	</RenderingInfo>
</Event>

The first event is repeated multiple times, for GPE registers ranging from 0 to 11 (0xB). The rate of events is very high, around 38K events/s. When this issue is not present, the trace is empty and no events are generated. Also, the events stop when a sound is played.

 

[javispedro]

Traces from the Microsoft-Windows-Kernel-Acpi provider are filled with the following types of events while this issue is in effect:

THANK YOU!!!!!!!!!!!!!!!!!!!!!!! My log is completely empty and I've not been able to configure the Acpi provider to be more verbose!

Here is a dissassembly of _GPE._L6D:

        Method (_L6D, 0, NotSerialized)  // _Lxx: Level-Triggered GPE
        {
            If (LEqual (\_SB.RDGP (Zero), One))
            {
                If (PWBE)
                {
                    Notify (\_SB.PWRB, 0x02)
                }
            }
            Else
            {
                If (LEqual (\_SB.RDGP (One), Zero))
                {
                    If (LAnd (\_SB.PCI0.EHC1.PMEE, \_SB.PCI0.EHC1.PMES))
                    {
                        Notify (\_SB.PCI0.EHC1, 0x02)
                    }

                    If (LAnd (\_SB.PCI0.EHC2.PMEE, \_SB.PCI0.EHC2.PMES))
                    {
                        Notify (\_SB.PCI0.EHC2, 0x02)
                    }

                    If (LAnd (\_SB.PCI0.XHC.PMEE, \_SB.PCI0.XHC.PMES))
                    {
                        Notify (\_SB.PCI0.XHC, 0x02)
                    }

                    If (LAnd (\_SB.PCI0.HDEF.PMEE, \_SB.PCI0.HDEF.PMES))
                    {
                        Notify (\_SB.PCI0.HDEF, 0x02)
                    }

                    Notify (\_SB.PCI0.EHC2, 0x02)
                    Notify (\_SB.PWRB, 0x02)
                }
            }
        }

Some of the hardware involved..
* PWRD is the power button
* EHC1 is EHCI controller (aka USB 2.0)
* EHC2 is ... no idea (maybe Surface Pro 1 hardware? It is not present on my SFPro2).
* XHC is XHCI controller (aka USB 3.0)
* HDEF is HD audio

PMEE and PMES are the "PME Enable" and "PME Status" .

Notify (\_SB.PWRB, 0x02) triggers "ACPI_NOTIFY_BUTTON_PRESSED_FOR_WAKEUP"

Notify (\_SB.PCI0.XXXX, 0x02) triggers "ACPI_NOTIFY_DEVICE_WAKE" for that device

No idea about RDGP. Here it is:

        Method (RDGP, 1, Serialized)
        {
            If (LLessEqual (Arg0, 0x5E))
            {
                Store (Add (Add (GPBS, 0x0100), Multiply (Arg0, 0x08)
                    ), Local0)
                OperationRegion (LGPI, SystemIO, Local0, 0x04)
                Field (LGPI, AnyAcc, NoLock, Preserve)
                {
                        ,   31, 
                    TEMP,   1
                }

                Return (TEMP)
            }

            Return (Zero)
        }

GPBS points to 0x1C00, which is firmware reserved. This snippet seems very common in most Windows 8 devices.


Basically _L6D seems to be the routine that gets invoked whenever the embedded controller wants to wake up the device. No idea exactly under what conditions is GPE 0x6D triggered, but it seems to come from EC. Also, no idea what it uses TEMP for.

 

[HypermagiK]

I don't think it's the EC as it appears to be using GPE 0x38.

To reproduce this issue in Linux you will have to enable interrupts for GPE 0x6D by writing 'enable' to /sys/firmware/acpi/interrupts/gpe6D and re-insert the SD card. The following GPEs seem to be valid: 0x09, 0x1E, 0x38 (EC), 0x61, 0x66, 0x69 and 0x6D. In Linux, interrupts are enabled by default only for 0x1E, 0x38, 0x61 and 0x66.

Checking the ACPI PM base IO space (0x1800) with R&W Everything, I see that the following GPEs are enabled in Windows: 0x11, 0x1E, 0x38, 0x61, 0x66 and 0x6D. When a sound is played, GPE 0x6D is disabled temporarily. GPE 0x6D is also masked when the sound card is disabled.

Later edit: I think I might have found a workaround for our issue. To keep GPE 0x6D masked off, one would have to disable the idle timeouts for the audio device. This can be done by following the steps presented here: http://msdn.microsoft.com/en-us/library/ff536193.aspx. You have to look for the right key under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4d36e96c-e325-11ce-bfc1-08002be10318}. Also, when using the Realtek drivers it seems like the idle parameters cannot be properly overridden and snap back to default values, so this only works with the built-in Microsoft drivers.

 

[Philtastic]

Just chiming in since I recently got an mSD card and started noticing this problem (and only after getting the card - never noticed this issue prior to that). Inserted card, seemed to be working fine, brought it out of sleep at some point and noticed it getting hot, and found it chugging away constantly over at least half an hour at 26% CPU usage (I've got the 4300U CPU). Taking out the card initially brings CPU use back to 0% for roughly 20 seconds, at which point, it goes right back up to 26%. Sometimes putting it to sleep then waking it up fixes the issues but often it doesn't. I'm not certain but I think it sometimes happens even when I wake it from sleep even without insert the card but I haven't had enough time with this issue to reliably report that yet.