There are a lot of factors affecting the server response time like hardware,underlying softwares,network, application etc.
1. Ensure server stability, check if load average is at an optimum level for all the times.
2.Check if the server resources are enough to handle the traffic.
3.Check the running applications are hogging more server resources.
4.Enable any caching system on the server depends on the application you are using.
5.Ensure webservers as well database servers are well optimized.
6.Check any misconfigured software or hardware firewall denying or limiting the connection rate causing the issue.
7.Enable compression for all website contents.
8.Enable any CDN cache mechanism for the websites.
9.Minify the application code by removing redundant and long variables, unwanted characters, and comments.
10.Remove unwanted and unnecessary plugins from WordPress, ensure that any unoptimized WordPress plugins or themes causing the issue.
11. For WordPress websites, you may enable any WordPress cache plugins to optimize improve the website performance.
12. Check and ensure any silent DOS/DDOS/SYNFLOOD making the server a bit sluggish.
13. You can perform ping test and apache benchmark test to check the resoponse time in seconds.
You can investigate on application code using the online optimization tools,
https://developers.google.com/speed/pagespeed/insights/
https://gtmetrix.com/
- Also you can seek assistance of a code and SEO expert for an application optimization.
- If it is for network or system side, then contact your provider to do a check the stability of underlying hardware Node or software at their end.