logo
Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Options
Go to last post Go to first unread
shawnkhall  
#1 Posted : Tuesday, March 27, 2018 4:33:40 PM(UTC)
shawnkhall


Rank: Advanced Member

Medals: Level 1: Random Act of Kindness! Received One Thanks!

Joined: 2/6/2014(UTC)
Posts: 301
Man
United States

Thanks: 7 times
Was thanked: 26 time(s) in 23 post(s)
I'm using 6.4 server. I currently have hundreds of devices and only one device is exhibiting this behavior. The device in question is Win10 x64 with the latest SC 6.4.

One command on one device runs, processes successfully, and returns the output every 35 seconds. It has been doing this for the last 3 days.

The command is pretty simple: it runs a batch file that downloads and updates an app locally (wget). It was enqueued *once* within the Commands feature, but just will not stop running again and again.

Disabling or turning off SC makes it stop, of course. But, the second I let SC run on the device it begins the looping process all over again. The logs are flooded with confirmations that it has run successfully.

This Command loop has survived purging the contents of c:\windows\temp\screenconnect. It has survived upgrading from 1703 to 1709. It has survived more than a dozen reboots.

How do I make it stop AND preserve the existing associated device logs and information in SC.

Is there a way to edit or view the command queue? (and yes, I know that deleting it from the Commands tab doesn't actually remove it from the queue)

I'm afraid my only fix is going to be removing and reinstalling SC, which will assign a new unique ID for the device. That's all fine and dandy, but it means I lose the entire history of the device: all UPs, notes, commands, chat and other logs are disassociated forever. The UPs and notes I can recreate. The commands, chat, and other logs can not.
Scott  
#2 Posted : Monday, April 9, 2018 12:22:45 PM(UTC)
Scott


Rank: Administration

Medals: Level 4: Wise Old Owl! Received 100 Thanks!

Joined: 3/28/2014(UTC)
Posts: 2,806
United States

Thanks: 3 times
Was thanked: 345 time(s) in 299 post(s)
Did you set the timeout/character killcount to something very high?

Also, if the command is queued from the Host page, you should be able to push it out via another Host page command, such as issuing a Reinstall or something similar. If a Reinstall does not correct the issue then I would guess the looping is happening on the machine itself.
ScreenConnect Team
shawnkhall  
#3 Posted : Tuesday, April 17, 2018 4:40:29 PM(UTC)
shawnkhall


Rank: Advanced Member

Medals: Level 1: Random Act of Kindness! Received One Thanks!

Joined: 2/6/2014(UTC)
Posts: 301
Man
United States

Thanks: 7 times
Was thanked: 26 time(s) in 23 post(s)
The timeout is very high (2h) and the character limit is very large as well (5mb). However, the command is quite simple and returns only 150 characters.

It's not the device itself, but something corrupted within the SC server state or code for the specific device.

I actually had a second computer that had not been on in a while boot up and start experiencing this same problem over the weekend with a different command. It is really no different from the hundreds of other machines I am using Access with. The command that was looping was slightly different, but returned only 180 characters and finished successfully each time it looped every 35 seconds. I tried rebooting the device several times which did not help. The rogue command from hell ran HUNDREDS of times before I was able to resolve it by changing the SCID and restarting SC.

To that, I was able to use my newscid script to reset the ScreenConnect ID on the device and it immediately returned to normal behavior. It was now a new device as far as ScreenConnect was concerned, but doing this worked perfectly well to disassociate it from the defective ScreenConnect device profile and allow the device to exit the command loop from hell. I've successfully run many short- and long-running commands through the host page over the last couple days on this device and it is now operating normally again. As expected, the worst part was losing the device CustomProperties, Notes, Timeline, Messages, and Command history. The CPs and Notes I was able to migrate over, but the Timeline, Messages and Command history now exist only in my MySQL backup. :(
shawnkhall  
#4 Posted : Tuesday, April 17, 2018 4:45:29 PM(UTC)
shawnkhall


Rank: Advanced Member

Medals: Level 1: Random Act of Kindness! Received One Thanks!

Joined: 2/6/2014(UTC)
Posts: 301
Man
United States

Thanks: 7 times
Was thanked: 26 time(s) in 23 post(s)
Oh, and the problem with the command loop from hell is that issuing another command had no effect since as far as SC was concerned, the previous command (from the command loop from hell) was still enqueued first. likewise, 'Reinstall' via SC, and reinstalling via MSI directly on the client made no difference. Further, connecting to the device via SC and attempting to resolve it remotely did not work since I only had a fraction of a second between each 35-second loop iteration to attempt to do anything, and after over 20 minutes of trying I was still unable to open even an administrative command prompt (Win+X,A).

Logging into the device physically and changing the SCID was the only thing that helped.
KatyComputer  
#5 Posted : Wednesday, May 2, 2018 11:36:13 AM(UTC)
KatyComputer


Rank: Member

Joined: 5/5/2014(UTC)
Posts: 21
Man
United States
Location: St Louis

Thanks: 23 times
We have had the same issue. It affects approximately 1% of our machines. When we catch it, we uninstall SC, then use eset Remote Administrator (we use ERA & SC as our RMM system) to install SC.

This seems to correct the problem. The only way we have found to catch it is to wait for client to call complaining about a crazy slow computer - not a great way to run things.

Best answer is a fix. Second best: Some way to find these machines so we can fix. Any ideas?
shawnkhall  
#6 Posted : Wednesday, May 2, 2018 3:09:22 PM(UTC)
shawnkhall


Rank: Advanced Member

Medals: Level 1: Random Act of Kindness! Received One Thanks!

Joined: 2/6/2014(UTC)
Posts: 301
Man
United States

Thanks: 7 times
Was thanked: 26 time(s) in 23 post(s)
i'm fortunate that one of these machines was in my presence so i could test my re-assigning the screenconnect ID fix: it worked. it's quite simple, and doesn't require anything more than a reboot or restart of the services afterwards. you lose the information tied to the machine, so if you're relying on notes or customproperties you have to reassign them afterwards, but you'd have to do that if you reinstalled anyway.

as for monitoring - yes, actually. i wrote a trigger that logs everything (devices and events) to MySQL. a simple query on that events table can pull any command that executes more than x times per hour, so you can quickly discern which ones are broken. i've got to sanitize it before i can publish it - i might have time today.
mcmcdtx  
#7 Posted : Wednesday, May 2, 2018 3:58:41 PM(UTC)
mcmcdtx


Rank: Guest

Joined: 7/19/2016(UTC)
Posts: 4
United States
Location: Texas

I am experiencing this as well. I thought that it was tied to the animations in the newest version. It only appeared after upgrade to v. 6.6.17081.6648 and it seemed to be tied to the new animations (which I am also trying to disable). The behavior doesn't seem obvious on the customer side yet but in the screenconnect interface it constantly flashes because it seems to be re-sending the request for the data which populates the first to tabs (bios model number IP etc) and although I can see the information being returned in the command prompt log it never populates.
shawnkhall  
#8 Posted : Friday, June 15, 2018 8:52:20 AM(UTC)
shawnkhall


Rank: Advanced Member

Medals: Level 1: Random Act of Kindness! Received One Thanks!

Joined: 2/6/2014(UTC)
Posts: 301
Man
United States

Thanks: 7 times
Was thanked: 26 time(s) in 23 post(s)
i have another device exhibiting this behavior right now. it had been offline a couple months and when it came back online several of the commands have been in a command loop again. the fix was pretty simple: change the SCID. i shared the newscid script on another post, linked above. it worked a charm. i pipe all my command results to a mysql database for logging and analysis and will be adding a monitor to it to detect this issue.
shawnkhall  
#9 Posted : Wednesday, June 27, 2018 6:18:09 PM(UTC)
shawnkhall


Rank: Advanced Member

Medals: Level 1: Random Act of Kindness! Received One Thanks!

Joined: 2/6/2014(UTC)
Posts: 301
Man
United States

Thanks: 7 times
Was thanked: 26 time(s) in 23 post(s)
I had another one show up about 10 days ago exhibiting the same behavior. The client turned it off until today. When it came back up the loop did not occur. It took me a little while to figure out why, but I think it's because I have my database maintenance routines set to purge various events older than 60d. This device had been off for about 2 months when it came up 10 days ago. I suspect that the database maintenance process purged the 'commands from hell' from the queue, suggesting that if one were to lower the database maintenance schedule to something much lower and manually run it, they might be able to escape this hell without manually changing the SCID.
shawnkhall  
#10 Posted : Wednesday, June 27, 2018 6:42:59 PM(UTC)
shawnkhall


Rank: Advanced Member

Medals: Level 1: Random Act of Kindness! Received One Thanks!

Joined: 2/6/2014(UTC)
Posts: 301
Man
United States

Thanks: 7 times
Was thanked: 26 time(s) in 23 post(s)
For anyone else experiencing this problem, the following SQL might fix it for you (UNTESTED!!!). You'll need a Sqlite database editor and you'll have to know where your session.db is. I leave that to you - if you can't figure it out, you shouldn't be doing this.

Replace "MyBroken-PC" with the name of the PC you're having problems with (or name the device you're having problems with "MyBroken-PC" until you're done fixing it).

Code:

Delete FROM SessionConnectionEvent
Where SessionID = (Select Session.SessionID FROM Session WHERE Session.Name='MyBroken-PC')
AND EventType = 70

Delete FROM SessionEvent
Where SessionID = (Select Session.SessionID FROM Session WHERE Session.Name='MyBroken-PC')
AND EventType = 44
shawnkhall  
#11 Posted : Wednesday, June 27, 2018 7:09:19 PM(UTC)
shawnkhall


Rank: Advanced Member

Medals: Level 1: Random Act of Kindness! Received One Thanks!

Joined: 2/6/2014(UTC)
Posts: 301
Man
United States

Thanks: 7 times
Was thanked: 26 time(s) in 23 post(s)
I was also able to find a couple devices with bad network cards with the following SQL:
Code:

SELECT Session.Name,Count(SessionConnectionEvent.SessionID) as c,SessionConnectionEvent.EventType
FROM Session
INNER JOIN SessionConnectionEvent ON Session.SessionID = SessionConnectionEvent.SessionID
GROUP BY Session.Name,EventType
ORDER BY c desc


If you're purging the database on a regular basis (30 or 60 days, for example), if you've got devices with a few *thousand* events 10 (connect) and 11 (disconnect), then that device has got a problem.

So glad I took the time today to play with this stuff. :)
Jankos  
#12 Posted : Monday, July 2, 2018 1:37:44 PM(UTC)
Jankos


Rank: Guest

Joined: 7/2/2018(UTC)
Posts: 2
United States

I have experienced this as well with around 100 machines on my 16k+ machine instance. A reboot of the server, while not ideal, appears to have fixed the issue for me.
shawnkhall  
#13 Posted : Friday, July 6, 2018 7:55:07 AM(UTC)
shawnkhall


Rank: Advanced Member

Medals: Level 1: Random Act of Kindness! Received One Thanks!

Joined: 2/6/2014(UTC)
Posts: 301
Man
United States

Thanks: 7 times
Was thanked: 26 time(s) in 23 post(s)
we restart the SC service weekly during our backup routine and it hasn't helped. any reason why a server reboot would make a difference?
Jankos  
#14 Posted : Friday, July 6, 2018 12:51:14 PM(UTC)
Jankos


Rank: Guest

Joined: 7/2/2018(UTC)
Posts: 2
United States

No idea. It may not work for you but it appears to have fixed it for us.
shawnkhall  
#15 Posted : Wednesday, July 11, 2018 5:41:49 AM(UTC)
shawnkhall


Rank: Advanced Member

Medals: Level 1: Random Act of Kindness! Received One Thanks!

Joined: 2/6/2014(UTC)
Posts: 301
Man
United States

Thanks: 7 times
Was thanked: 26 time(s) in 23 post(s)
discovered another client device in a command loop from hell today. since it's patch tuesday, i took the opportunity to reboot the server to see if that helped. nope. purging the database did, though. :/
KatyComputer  
#16 Posted : Wednesday, August 15, 2018 10:49:41 PM(UTC)
KatyComputer


Rank: Member

Joined: 5/5/2014(UTC)
Posts: 21
Man
United States
Location: St Louis

Thanks: 23 times
@ShawnKHill: What do you mean by "Purging the database"?

@ScreenConnect: When are we going to get relief from this issue?
shawnkhall  
#17 Posted : Thursday, August 16, 2018 4:23:30 AM(UTC)
shawnkhall


Rank: Advanced Member

Medals: Level 1: Random Act of Kindness! Received One Thanks!

Joined: 2/6/2014(UTC)
Posts: 301
Man
United States

Thanks: 7 times
Was thanked: 26 time(s) in 23 post(s)
Originally Posted by: KatyComputer Go to Quoted Post
@ShawnKHill: What do you mean by "Purging the database"?


Hi, @KatyComputer,

The database used by ScreenConnect/ConnectWise Control is an SQLite DB located at "%ProgramFiles(x86)%\ScreenConnect\App_Data\Session.db". This database holds all your device information, notes, logs, commands, and so on. Using a database editor (DB Browser for SQLite, for example) will allow you to open the Session.db file and run SQL commands against the database.

The structure is a little funky, but you can purge (remove) commands from the command history, which effectively prevents them from being re-enqueued ad nauseum (aka, "command loop from hell"). I have been able to resolve 8 separate instances of "command loop from hell" now using this method.

Caveats:

  1. It's your entire SC database. If you don't know what you're doing, leave it alone. The risk of significant damage (including crippling your entire platform) does exist. Back it up first:
    Code:
    robocopy "%ProgramFiles(x86)%\ScreenConnect\App_Data\Session.*" "c:\backup" /r:1 /w:1 /zb

  2. Using any interactive editor, such as DB Browser for SQLite, locks your database. This means that while you're fiddling with it any commands, logs, notes and so on that would otherwise occur from the ScreenConnect/Control functionality will fail. Be brief! Open, run, save, close. Practice with a copy. Practice with two copies. Don't practice with your live database. :)
  3. The EventType values are magic numbers, not descriptive terms. This means that you won't see "enqueued command" as an EventType directly in the Session*Event table, but will instead see 44. You won't see "ran command", but will instead see 70. Incomplete/dated list here.
  4. Make sure you replace "MyBroken-PC" with the name of your device. It won't work unless you do. You could also flag devices with a CustomProperty or Note that indicates that it is suffering from command-loop-from-hell syndrome and use that field to select the device instead. Samples further down for those.
  5. You may not want/need to remove the EventType=70 (ran command) if there is important data in one or more of the command outputs from previous commands. In my experience, however, where there were 131k copies of the same looped command that ran on one device, you would be well advised to do it anyway. The amount of wasted space could be enormous. And, of course, you got your backup copy first, right?
  6. Your mileage may vary. While this has worked consistently for me to resolve this (and several other problems), it will not necessarily address every conceivable situation.


Here again is the code I use. This deletes the RanCommand events (70) then the QueuedCommand events (44). This effectively removes any trace of the command that's responsible for the loop.

Code:
Delete FROM SessionConnectionEvent
Where SessionID = (Select Session.SessionID FROM Session WHERE Session.Name='MyBroken-PC')
AND EventType = 70;

Delete FROM SessionEvent
Where SessionID = (Select Session.SessionID FROM Session WHERE Session.Name='MyBroken-PC')
AND EventType = 44;



To effect every single device with a CustomProperty8 with a value of "commandLoopFromHell" use:

Code:
Delete FROM SessionConnectionEvent
Where SessionID = (Select Session.SessionID FROM Session WHERE Session.CustomProperty8='commandLoopFromHell')
AND EventType = 70;

Delete FROM SessionEvent
Where SessionID = (Select Session.SessionID FROM Session WHERE Session.CustomProperty8='commandLoopFromHell')
AND EventType = 44;


You could even add the following line at the end to empty CustomProperty8 to ensure that it doesn't get run again against the same machine:
Code:
UPDATE Session SET CustomProperty8='' WHERE Session.CustomProperty8='commandLoopFromHell';



To effect every single device with a Note that has been added with a value of "commandLoopFromHell" use:

Code:
Delete FROM SessionConnectionEvent
Where SessionID = (Select SessionEvent.SessionID FROM SessionEvent WHERE SessionEvent.Data='commandLoopFromHell')
AND EventType = 70;

Delete FROM SessionEvent
Where SessionID = (Select SessionEvent.SessionID FROM SessionEvent WHERE SessionEvent.Data='commandLoopFromHell')
AND EventType = 44;

Delete FROM SessionEvent WHERE SessionEvent.Data='commandLoopFromHell';


Then close the database.

Finally, there are command-line SQLite engines or you could use VBScript, asp.net or PHP to get the same result as using an interactive editor. In this way you could automate the process to have the least possible downtime (less than 1 second per run). Then, simply adding commandLoopFromHell to a note would then trigger a command purge on that device when your process ran every hour/day or so. It could even check for the note first to ensure that it wasn't wasting an expensive (and blocking) DELETE when a SELECT showed it wasn't necessary. :)
thanks 2 users thanked shawnkhall for this useful post.
KatyComputer on 8/16/2018(UTC), asherxtn on 8/16/2018(UTC)
KatyComputer  
#18 Posted : Thursday, August 16, 2018 10:02:07 AM(UTC)
KatyComputer


Rank: Member

Joined: 5/5/2014(UTC)
Posts: 21
Man
United States
Location: St Louis

Thanks: 23 times
@shawnkhall: Beautiful work. Thank you!
shawnkhall  
#19 Posted : Thursday, August 16, 2018 5:17:53 PM(UTC)
shawnkhall


Rank: Advanced Member

Medals: Level 1: Random Act of Kindness! Received One Thanks!

Joined: 2/6/2014(UTC)
Posts: 301
Man
United States

Thanks: 7 times
Was thanked: 26 time(s) in 23 post(s)
My pleasure. If you find that it doesn't resolve one of your command-loop-from-hell situations, please let me know.
Users browsing this topic
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.