Forum - Thread error

Thread error

Vaughan Data Systems

1-4-22

Operating system: Windows

We recently had this error in the log followed by the Hansa server crashing.

Does anyone know what this could mean?

/THREAD(31) out of file-descriptors

Zilvinas Suliauskas

1-4-22

Hello,

What version of Standard ERP your customer is using?

Lee D Papworth

1-6-22

Created byZilvinas Suliauskas16:30 4 Jan 2022

Hello, What version of Standard ERP your customer is using?

Hi Zilvinas,

SERP 8.5 2021-04-18 Build 85360453

Running on Centos Linux release 7.9.2009

Not had this issue in the previous version of SERP.

We have recently upgrade customer to this build

Thanks

Lee

Zilvinas Suliauskas

1-6-22

Hi Lee,

I spoke to one of our programmers. It could be version specific issue. Could you please test if same problem appears in the latest released version as well.

Lee D Papworth

1-6-22

Created byZilvinas Suliauskas14:38 6 Jan 2022

Hi Lee, I spoke to one of our programmers. It could be version specific issue. Could you please test if same problem appears in the latest released version as well.

Hi Zilvinas,

I am afraid that will be quite difficult to prove. You see we tested SERP with the customer before the upgrade and this issue did not arise. I suspect it has now come to light because in the live environment hundreds more transactions are being processed that cannot be replicated in a test setup.

Can any techie please review this article and see if it makes sense?

https://docs.pingidentity.com/bundle/pingdirectory-73/page/ygy1564011416825-1.html

I am wondering if it can be solved by changing some Linux settings

Thanks

Lee

Lee D Papworth

1-14-22

Created byLee D Papworth15:35 6 Jan 2022

Hi Zilvinas, any feedback from the techies?

Thanks

Lee

Nejc R.

1-17-22

Hello,

Try updating to a later version 8.5 2021-04-18 is pretty old and its likely the issue is resolved in newer versions.

what version are you updating from?

are you running any halcust?

Vaughan Data Systems

1-19-22

Hi Nejc,

They were on 85330538 and are now on 85360453

All of our other customers are on this release and are working fine.

The reason we settled on this release is because of issues with later versions of Standard ERP that I believe still haven't been resolved yet.

Are you able to confirm this was a known issue that has been fixed in newer builds now?

The customer who is getting these crashes is a huge company with lots of files in their halcust, so updating them takes hours of testing and taking time out of their schedule.

Do you know what this error actually means? And is there anything we can try before attempting an update? E.g. changing thread settings or server settings etc

Nejc R.

1-20-22

Hello,

So this was just a build upgrade and the version did not actually change?

If the halcust is not closing files correctly the system can run out of file descriptors.

File descriptors are pointers/handles to opened files in Linux, whenever you want to read/write to a file you first need to open it and then close it when you are done. The amount of files you can have open at once is limited (usually a pretty high number). But if you open a file and do not close it, it will just remain open. Do that enough times and you will run out of file descriptors and the program will crash.

Vaughan Data Systems

1-20-22

Hi Nejc,

No, we went from 2020-01-07 to 2021-04-18

Do you have anything we can try/look for in order to resolve this?

Nejc R.

1-21-22

If there are any OpenFile("filename"); that do not CloseFile; after that definitely can cause it.

Has this error occurred again since then. And is there anything specific being done on the system when it crashes?

Lee D Papworth

1-24-22

Created byNejc R.16:24 21 Jan 2022

If there are any OpenFile("filename"); that do not CloseFile; after that definitely can cause it. Has this error occurred again since then. And is there anything specific being done on the system when it crashes?

Hi Nejc,

there are no OpenFile instances.

We use xdata = ParseXMLFile(filename);

So I am assuming ParseXMLFile actually opens the file for reading and should also terminate with a CloseFile?

Can you come back to me on that please?

Nothing out of the ordinary is running on the system when this crashes. Our import routines have been working without any issues in the last two versions of SERP. It is all happening since the upgrade

Thanks

Lee

Aldevinas Katkus

1-25-22

Try to run on Linux command prompt after you files have been imported:
ps -aux | grep StandardERPServer64

then
lsof -p serp_pid

where serp_pid is PID of the StandardERPServer64 (second column of the result of the previous command)

Or try

lsof -p serp_pid +D 'path_to_your_xml_files_directory'

Lee D Papworth

1-25-22

Hi Aldevinas,

Can you please tell me what I am looking for when I run those commands?

Thanks

Lee

Lee D Papworth

1-27-22

It appears that it might be to do with web functions.

When running the command lsof -p serp_pid there are literally thousands of open files

/home/hansa/StandardERPServer/webcust/styles/HHIStyle24.css
/home/hansa/StandardERPServer/webcust/js/HHIMainapp24.js
/home/hansa/StandardERPServer/webcust/sdlogo.png

We have a set of bespoke web functions that we use for bar code scanning and put away in the Warehouse. These functions have been running perfectly for the last couple of years but since this new release we hit the out of file descriptors snag.

Could it be the web engine is not terminating these files when fresh pages are loaded?

Please please get to me asap on this. Happy to provide any more info.

My only action at the moment is to stop and restart SERP every night for the customer which can only be a short time solution

Many thanks in advance

Lee Papworth

Aldevinas Katkus

1-27-22

I have tested webshop and our custom code which refreshes the page every minute with v. 2021-11-24 and didn't notice anything wrong, lsof did not show any increase in open files.

Lee D Papworth

3-4-22

Hi,

I am really beyond desperate with this now. I am having to stop and restart SERP every evening just so my customer does not get this error/crash during the day.

So just to recap. I have basic web functions to assist my customer with picking and dispatch. I have a stlyesheet, a small javascript file and a company logo. They have a webshop licence but I have written these routines using global webpublic hal functions. Not had any issues with these functions in the last 2 years. Only since we upgraded in December 2021. Have similar functions at other customer sites that do not get this problem.

When I run this command "lsof -p serp_pid" after a restart of SERP I immediately can see a growing number of these entries when they start to use the web functions

/home/hansa/StandardERPServer/webcust/styles/HHIStyle24.css
/home/hansa/StandardERPServer/webcust/js/HHIMainapp24.js
/home/hansa/StandardERPServer/webcust/sdlogo.png

This list of open files just continues to grow and grow and I believe this then causes the "out of file-descriptors" error and crashes SERP

Any ideas please?

Thanks

Lee

Aldevinas Katkus

3-10-22

Do you see the same files are listed multiple times in the lsof output?
Some crazy ideas:

Compare system>>settings>>optional features on the crashing site and other sites.
Are your clients using the same web browser or different ones?

Aldevinas Katkus

3-10-22

Created byAldevinas Katkus15:37 10 Mar 2022

Do you see the same files are listed multiple times in the lsof output? Some crazy ideas: Compare system>>settings>>optional features on the crashing site and other sites. Are your clients using the same web browser or different ones?

try to use AddFileToArea then ParseXMLArea instead.

Lee D Papworth

3-10-22

Hi Aldevinas,

yes the three files I listed previously are constantly repeated when I look at the output of lsof. I have other links in my html to google fonts and a jquery and they do not show in lsof. These are external links. It just seems to be the files I am serving from the webcust folder.

I have compared optional features with other customers. All the same except this server that has the issue is a very high spec so we have more report slots and HTTP threads specified

Lee

Jennifer

4-4-22

Created byLee D Papworth15:42 10 Mar 2022

Hi Lee,

Did you manage to get to the bottom of this?

Jen

Lee D Papworth

4-4-22

Created byJennifer11:15 4 Apr 2022

Hi Lee, Did you manage to get to the bottom of this? Jen

Hi Jen,

unfortunately not :(

I had to embed the css and javascript into the pages rather than using an external link.

Not ideal but at least it is now not crashing SERP

Lee

Back to the list

New Message
Open Questions My Questions