KumpeApps LLC Status - Incident history https://kumpeapps.instatus.com KumpeApps LLC Sat, 7 Dec 2024 18:44:00 +0000 Mail Server Outage Type: Incident Duration: 1 hour and 29 minutes Affected Components: KumpeApps Email Dec 7, 20:12:44 GMT+0 - Resolved - This incident has been resolved. Dec 7, 18:44:00 GMT+0 - Investigating - We are currently investigating this incident. Dec 7, 18:51:59 GMT+0 - Identified - This appears to be a data center outage. Raised ticket with data center. Dec 7, 20:06:02 GMT+0 - Identified - Data center is working on resolution and server is currently up. Downgrading to degraded service instead of outage. Type: Incident

Duration: 1 hour and 29 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>20:12:44</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>18:44:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>18:51:59</var> GMT+0</small><br><strong>Identified</strong> - This appears to be a data center outage. Raised ticket with data center..</p> <p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>20:06:02</var> GMT+0</small><br><strong>Identified</strong> - Data center is working on resolution and server is currently up. Downgrading to degraded service instead of outage..</p> ]]> Sat, 7 Dec 2024 18:44:00 +0000 https://kumpeapps.instatus.com/incident/cm4ej6m2w00a8ghbaltkuao6c https://kumpeapps.instatus.com/incident/cm4ej6m2w00a8ghbaltkuao6c Mail Server Outage Type: Incident Duration: 1 day, 4 hours and 32 minutes Affected Components: KumpeApps Email Sep 29, 15:12:11 GMT+0 - Investigating - We are currently investigating this incident. Sep 29, 16:12:41 GMT+0 - Identified - We have identified that our server has lost data connectivity at the data center. We have verified the server is still running but it does not have data connectivity. We have engaged our data center to investigate. Sep 29, 20:26:19 GMT+0 - Identified - We are continuing to work on a fix for this incident. Our data center has reported the entire IP block our server is on is down. They have engaged their ISP for resolution. Sep 29, 23:31:50 GMT+0 - Identified - We are continuing to work on a fix for this incident. We have requested an update from our data center and are awaiting a response. Sep 30, 17:29:01 GMT+0 - Monitoring - Data center issues have been corrected and will take 1-2hrs to replicate world-wide. Sep 30, 19:43:55 GMT+0 - Resolved - This outage has been confirmed as resolved. Type: Incident

Duration: 1 day, 4 hours and 32 minutes

Affected Components:

<p><small>Sep <var data-var='date'> 29</var>, <var data-var='time'>15:12:11</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Sep <var data-var='date'> 29</var>, <var data-var='time'>16:12:41</var> GMT+0</small><br><strong>Identified</strong> - We have identified that our server has lost data connectivity at the data center. We have verified the server is still running but it does not have data connectivity. We have engaged our data center to investigate..</p> <p><small>Sep <var data-var='date'> 29</var>, <var data-var='time'>20:26:19</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident. Our data center has reported the entire IP block our server is on is down. They have engaged their ISP for resolution..</p> <p><small>Sep <var data-var='date'> 29</var>, <var data-var='time'>23:31:50</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident. We have requested an update from our data center and are awaiting a response..</p> <p><small>Sep <var data-var='date'> 30</var>, <var data-var='time'>17:29:01</var> GMT+0</small><br><strong>Monitoring</strong> - Data center issues have been corrected and will take 1-2hrs to replicate world-wide..</p> <p><small>Sep <var data-var='date'> 30</var>, <var data-var='time'>19:43:55</var> GMT+0</small><br><strong>Resolved</strong> - This outage has been confirmed as resolved..</p> ]]> Sun, 29 Sep 2024 15:12:11 +0000 https://kumpeapps.instatus.com/incident/cm1npyve900027pwgbh7uaztf https://kumpeapps.instatus.com/incident/cm1npyve900027pwgbh7uaztf Kumpe3D Outage Type: Incident Duration: 7 hours and 52 minutes Affected Components: Kumpe3D, BOTs, KumpeApps Credentials Manager Sep 14, 15:47:22 GMT+0 - Investigating - Kumpe3D APIs are down after system upgrade. We are currently investigating this incident. Sep 14, 18:48:57 GMT+0 - Identified - We have identified issues with our credentials server and are working to resolve it as quickly as possible. Sep 14, 23:39:21 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 7 hours and 52 minutes

Affected Components: , ,

<p><small>Sep <var data-var='date'> 14</var>, <var data-var='time'>15:47:22</var> GMT+0</small><br><strong>Investigating</strong> - Kumpe3D APIs are down after system upgrade. We are currently investigating this incident..</p> <p><small>Sep <var data-var='date'> 14</var>, <var data-var='time'>18:48:57</var> GMT+0</small><br><strong>Identified</strong> - We have identified issues with our credentials server and are working to resolve it as quickly as possible..</p> <p><small>Sep <var data-var='date'> 14</var>, <var data-var='time'>23:39:21</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Sat, 14 Sep 2024 15:47:22 +0000 https://kumpeapps.instatus.com/incident/cm12bmc7u0002zcib5lhkid9i https://kumpeapps.instatus.com/incident/cm12bmc7u0002zcib5lhkid9i Scheduled Maintenance 9/14/2024 Type: Maintenance Duration: 1 hour and 46 minutes Affected Components: KHome Portal, KKid, A Creative Collection, KumpeApps SSO, UVT for IOS, K-Meds, KumpeApps Email, Kumpe3D, Retool, UVT Web, Kumpe3D Kiosk, Primary Data Server, Secondary Data Server, Third Party: Unsplash → Image Service → CDN, Third Party: Unsplash → API, BOTs, APIs, KumpeApps Credentials Manager Sep 14, 14:00:00 GMT+0 - Identified - We are planning for a scheduled maintenance 9/14/2024 from 9am CST until upgrades are complete which we estimate to be 11am CST. During this time we will be upgrading servers to apply security patches. This upgrade will be done on the servers live and should only effect systems for a few min at a time so actual down time should be minimal. Sep 14, 15:46:01 GMT+0 - Completed - maintenance is complete and most systems are now back up. Only Kumpe3D systems are down which will be put into outage mode. Sep 14, 14:00:00 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 1 hour and 46 minutes

Affected Components: , , , , , , , , , , , , , , , , ,

<p><small>Sep <var data-var='date'> 14</var>, <var data-var='time'>14:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance 9/14/2024 from 9am CST until upgrades are complete which we estimate to be 11am CST. During this time we will be upgrading servers to apply security patches. This upgrade will be done on the servers live and should only effect systems for a few min at a time so actual down time should be minimal..</p> <p><small>Sep <var data-var='date'> 14</var>, <var data-var='time'>15:46:01</var> GMT+0</small><br><strong>Completed</strong> - maintenance is complete and most systems are now back up. Only Kumpe3D systems are down which will be put into outage mode..</p> <p><small>Sep <var data-var='date'> 14</var>, <var data-var='time'>14:00:00</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Sat, 14 Sep 2024 14:00:00 +0000 https://kumpeapps.instatus.com/maintenance/cm0yb76a20016ez7tgfvqmnia https://kumpeapps.instatus.com/maintenance/cm0yb76a20016ez7tgfvqmnia Mail Server Outage Type: Incident Duration: 4 days, 10 hours and 26 minutes Affected Components: KumpeApps Email Sep 3, 11:35:25 GMT+0 - Identified - Server is unreachable. We have raised a ticket with data center. Sep 3, 06:34:00 GMT+0 - Investigating - We are currently investigating this incident. Sep 3, 18:43:11 GMT+0 - Identified - We are continuing to work on a fix for this incident. Our hosting company has identified the IP block we are in is not routing traffic properly. They are continuing to investigate with no ETR. Sep 3, 22:35:49 GMT+0 - Identified - We are continuing to work on a fix for this incident. Our hosting provider is still having issues with our block of IP addresses. We have requested they move us to a new IP block to restore service. This is not an ideal solution but it is better than nothing. Sep 4, 11:27:45 GMT+0 - Monitoring - Our hosting company has resolved the issue by changing our IP address. This appears to have resolved the issue but replication may take a couple of hours to replicate to all users. Sep 7, 17:00:00 GMT+0 - Resolved - Resolved Type: Incident

Duration: 4 days, 10 hours and 26 minutes

Affected Components:

<p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>11:35:25</var> GMT+0</small><br><strong>Identified</strong> - Server is unreachable. We have raised a ticket with data center..</p> <p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>06:34:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>18:43:11</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident. Our hosting company has identified the IP block we are in is not routing traffic properly. They are continuing to investigate with no ETR..</p> <p><small>Sep <var data-var='date'> 3</var>, <var data-var='time'>22:35:49</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident. Our hosting provider is still having issues with our block of IP addresses. We have requested they move us to a new IP block to restore service. This is not an ideal solution but it is better than nothing..</p> <p><small>Sep <var data-var='date'> 4</var>, <var data-var='time'>11:27:45</var> GMT+0</small><br><strong>Monitoring</strong> - Our hosting company has resolved the issue by changing our IP address. This appears to have resolved the issue but replication may take a couple of hours to replicate to all users..</p> <p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>17:00:00</var> GMT+0</small><br><strong>Resolved</strong> - Resolved.</p> ]]> Tue, 3 Sep 2024 06:34:00 +0000 https://kumpeapps.instatus.com/incident/cm0mcr7tv008c7dmhn19l956r https://kumpeapps.instatus.com/incident/cm0mcr7tv008c7dmhn19l956r Server Outage Type: Incident Duration: 14 hours and 42 minutes Affected Components: KHome Portal, KKid, A Creative Collection, KumpeApps SSO, UVT for IOS, K-Meds, KumpeApps Email, Kumpe3D, Retool, UVT Web, Kumpe3D Kiosk, Primary Data Server, Secondary Data Server, Third Party: Unsplash → Image Service → CDN, Third Party: Unsplash → API, BOTs, APIs, KumpeApps Credentials Manager Sep 1, 06:40:00 GMT+0 - Investigating - We are currently investigating this incident. Sep 1, 16:01:39 GMT+0 - Monitoring - We identified large log files that were cleared. We implemented a fix and are currently monitoring the result. Sep 1, 21:22:23 GMT+0 - Resolved - This issue has been confirmed as resolved. Type: Incident

Duration: 14 hours and 42 minutes

Affected Components: , , , , , , , , , , , , , , , , ,

<p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>06:40:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>16:01:39</var> GMT+0</small><br><strong>Monitoring</strong> - We identified large log files that were cleared. We implemented a fix and are currently monitoring the result..</p> <p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>21:22:23</var> GMT+0</small><br><strong>Resolved</strong> - This issue has been confirmed as resolved..</p> ]]> Sun, 1 Sep 2024 06:40:00 +0000 https://kumpeapps.instatus.com/incident/cm0jmf7nk003s13ey3avmoz4y https://kumpeapps.instatus.com/incident/cm0jmf7nk003s13ey3avmoz4y Major Server Outage Type: Incident Duration: 12 hours and 22 minutes Affected Components: KHome Portal, KKid, A Creative Collection, KumpeApps SSO, UVT for IOS, K-Meds, KumpeApps Email, Kumpe3D, Retool, UVT Web, Kumpe3D Kiosk, Primary Data Server, Secondary Data Server, Third Party: Unsplash → Image Service → CDN, Third Party: Unsplash → API, BOTs, APIs, KumpeApps Credentials Manager Aug 14, 22:15:38 GMT+0 - Investigating - Our primary server is currently down and our hosting company's website is not responding. We are working to identify the issue and restore service ASAP. Aug 14, 22:38:01 GMT+0 - Identified - Unable to raise ticket with hosting provider due to their site is down too. Tried to fail-over to secondary server but evidently it is down as well. Aug 15, 00:33:58 GMT+0 - Identified - We are continuing to work on a fix for this incident. We have confirmed the issue is with our hosting partner and they have engaged onsite technicians at the data center to investigate and resolve the issue. No ETA has been given at this time. Aug 15, 10:37:21 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 12 hours and 22 minutes

Affected Components: , , , , , , , , , , , , , , , , ,

<p><small>Aug <var data-var='date'> 14</var>, <var data-var='time'>22:15:38</var> GMT+0</small><br><strong>Investigating</strong> - Our primary server is currently down and our hosting company's website is not responding. We are working to identify the issue and restore service ASAP..</p> <p><small>Aug <var data-var='date'> 14</var>, <var data-var='time'>22:38:01</var> GMT+0</small><br><strong>Identified</strong> - Unable to raise ticket with hosting provider due to their site is down too. Tried to fail-over to secondary server but evidently it is down as well..</p> <p><small>Aug <var data-var='date'> 15</var>, <var data-var='time'>00:33:58</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident. We have confirmed the issue is with our hosting partner and they have engaged onsite technicians at the data center to investigate and resolve the issue. No ETA has been given at this time..</p> <p><small>Aug <var data-var='date'> 15</var>, <var data-var='time'>10:37:21</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Wed, 14 Aug 2024 22:15:38 +0000 https://kumpeapps.instatus.com/incident/clzueu8q0519164hboch9rh0k14 https://kumpeapps.instatus.com/incident/clzueu8q0519164hboch9rh0k14 Mail Server Maintenance 8/10 Type: Maintenance Duration: 1 hour and 10 minutes Affected Components: KumpeApps Email Aug 11, 00:38:26 GMT+0 - Completed - Maintenance has completed successfully. Aug 10, 23:27:59 GMT+0 - Identified - Maintenance is now in progress Aug 10, 23:27:58 GMT+0 - Identified - We are planning for a scheduled maintenance during that time. Type: Maintenance

Duration: 1 hour and 10 minutes

Affected Components:

<p><small>Aug <var data-var='date'> 11</var>, <var data-var='time'>00:38:26</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> <p><small>Aug <var data-var='date'> 10</var>, <var data-var='time'>23:27:59</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Aug <var data-var='date'> 10</var>, <var data-var='time'>23:27:58</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during that time..</p> ]]> Sat, 10 Aug 2024 23:27:58 +0000 https://kumpeapps.instatus.com/maintenance/clzorog5j315283hiohr0jj3plm https://kumpeapps.instatus.com/maintenance/clzorog5j315283hiohr0jj3plm Mail Server Upgrade Type: Maintenance Duration: 3 hours Affected Components: KumpeApps Email Aug 10, 04:00:00 GMT+0 - Completed - Maintenance has completed successfully Aug 10, 01:00:00 GMT+0 - Identified - We are planning for a scheduled maintenance during that time. During this time we will be upgrading our server for better security and for future capabilities of better SPAM filtering. Aug 10, 01:00:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 3 hours

Affected Components:

<p><small>Aug <var data-var='date'> 10</var>, <var data-var='time'>04:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Aug <var data-var='date'> 10</var>, <var data-var='time'>01:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during that time. During this time we will be upgrading our server for better security and for future capabilities of better SPAM filtering..</p> <p><small>Aug <var data-var='date'> 10</var>, <var data-var='time'>01:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Sat, 10 Aug 2024 01:00:00 +0000 https://kumpeapps.instatus.com/maintenance/clzkbo4oa605280i8ooujfihtfo https://kumpeapps.instatus.com/maintenance/clzkbo4oa605280i8ooujfihtfo Server Outage Type: Incident Duration: 14 minutes Affected Components: KHome Portal, KKid, A Creative Collection, KumpeApps SSO, UVT for IOS, K-Meds, KumpeApps Email, Kumpe3D, Retool, UVT Web, Kumpe3D Kiosk, Primary Data Server, Secondary Data Server, Third Party: Unsplash → Image Service → CDN, Third Party: Unsplash → API, BOTs, APIs, KumpeApps Credentials Manager Aug 9, 17:49:01 GMT+0 - Resolved - This incident has been resolved. Aug 9, 17:35:00 GMT+0 - Identified - We are continuing to work on a fix for this incident. Type: Incident

Duration: 14 minutes

Affected Components: , , , , , , , , , , , , , , , , ,

<p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>17:49:01</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>17:35:00</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident..</p> ]]> Fri, 9 Aug 2024 17:35:00 +0000 https://kumpeapps.instatus.com/incident/clzn00rfb91175i1n70xhhqqwt https://kumpeapps.instatus.com/incident/clzn00rfb91175i1n70xhhqqwt Secondary Data Server Degraded Performance Type: Incident Duration: 5 hours and 47 minutes Affected Components: Secondary Data Server Aug 3, 20:59:25 GMT+0 - Resolved - This incident has been resolved. Aug 3, 15:12:00 GMT+0 - Investigating - We are currently investigating this incident. All affected services have been automatically rolled over to primary server. Aug 3, 18:16:06 GMT+0 - Identified - Host company states it is a known issue they are working on a fix. Aug 3, 19:13:13 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Type: Incident

Duration: 5 hours and 47 minutes

Affected Components:

<p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>20:59:25</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>15:12:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident. All affected services have been automatically rolled over to primary server..</p> <p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>18:16:06</var> GMT+0</small><br><strong>Identified</strong> - Host company states it is a known issue they are working on a fix..</p> <p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>19:13:13</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> ]]> Sat, 3 Aug 2024 15:12:00 +0000 https://kumpeapps.instatus.com/incident/clzee5k7a138148hmoct95vm5mc https://kumpeapps.instatus.com/incident/clzee5k7a138148hmoct95vm5mc Mail Server Maintenance Type: Maintenance Duration: 1 day, 7 hours and 23 minutes Affected Components: KumpeApps Email Jul 21, 16:00:01 GMT+0 - Identified - Maintenance is now in progress Jul 21, 16:00:00 GMT+0 - Identified - We are planning for a scheduled maintenance Sunday Jul 21\. Downtime should be minimal. Jul 21, 16:31:22 GMT+0 - Completed - Upgrade has completed successfully. If you have issues with mobile devices send email to [reseteas@automation.kumpedns.us](mailto:reseteas@automation.kumpedns.us) and try again in 1min. Type: Maintenance

Duration: 1 day, 7 hours and 23 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 21</var>, <var data-var='time'>16:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jul <var data-var='date'> 21</var>, <var data-var='time'>16:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance Sunday Jul 21\. Downtime should be minimal..</p> <p><small>Jul <var data-var='date'> 21</var>, <var data-var='time'>16:31:22</var> GMT+0</small><br><strong>Completed</strong> - Upgrade has completed successfully. If you have issues with mobile devices send email to [reseteas@automation.kumpedns.us](mailto:reseteas@automation.kumpedns.us) and try again in 1min..</p> ]]> Sun, 21 Jul 2024 16:00:00 +0000 https://kumpeapps.instatus.com/maintenance/clyvoyeab192532e1n1apaqxm9d https://kumpeapps.instatus.com/maintenance/clyvoyeab192532e1n1apaqxm9d All Servers Down Type: Incident Duration: 8 minutes Affected Components: KHome Portal, KKid, A Creative Collection, KumpeApps SSO, UVT for IOS, K-Meds, KumpeApps Email, Kumpe3D, Retool, UVT Web, Kumpe3D Kiosk, Primary Data Server, Secondary Data Server, Third Party: Unsplash → Image Service → CDN, Third Party: Unsplash → API, BOTs, APIs, KumpeApps Credentials Manager Jul 2, 12:46:00 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Jul 2, 12:44:00 GMT+0 - Investigating - We are currently investigating this incident. Jul 2, 12:51:41 GMT+0 - Resolved - Issue has been resolved and servers appear stable Type: Incident

Duration: 8 minutes

Affected Components: , , , , , , , , , , , , , , , , ,

<p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>12:46:00</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>12:44:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>12:51:41</var> GMT+0</small><br><strong>Resolved</strong> - Issue has been resolved and servers appear stable.</p> ]]> Tue, 2 Jul 2024 12:44:00 +0000 https://kumpeapps.instatus.com/incident/cly4eml2z288642ajmvkegcpa0s https://kumpeapps.instatus.com/incident/cly4eml2z288642ajmvkegcpa0s Kumpe3D Maitenance Type: Maintenance Duration: 2 hours Affected Components: Kumpe3D, Retool, Kumpe3D Kiosk Jun 23, 03:00:00 GMT+0 - Completed - Maintenance has completed successfully Jun 23, 01:00:01 GMT+0 - Identified - Maintenance is now in progress Jun 23, 01:00:00 GMT+0 - Identified - We are planning for a scheduled maintenance during Jun 22 from 8pm until 10pm. We only expect our site to be down for a couple of mins during this time but scheduled 2 hrs so we have time for unexpected issues. This will only affect Kumpe3D systems. Type: Maintenance

Duration: 2 hours

Affected Components: , ,

<p><small>Jun <var data-var='date'> 23</var>, <var data-var='time'>03:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Jun <var data-var='date'> 23</var>, <var data-var='time'>01:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jun <var data-var='date'> 23</var>, <var data-var='time'>01:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during Jun 22 from 8pm until 10pm. We only expect our site to be down for a couple of mins during this time but scheduled 2 hrs so we have time for unexpected issues. This will only affect Kumpe3D systems..</p> ]]> Sun, 23 Jun 2024 01:00:00 +0000 https://kumpeapps.instatus.com/maintenance/clxqnn74k97768ojoh82sbn2tw https://kumpeapps.instatus.com/maintenance/clxqnn74k97768ojoh82sbn2tw Data Center Maintenance 6/14 Type: Maintenance Duration: 1 hour Affected Components: KHome Portal, KKid, A Creative Collection, KumpeApps SSO, UVT for IOS, K-Meds, KumpeApps Email, Kumpe3D, Retool, UVT Web, Kumpe3D Kiosk, Primary Data Server, Secondary Data Server, Third Party: Unsplash → Image Service → CDN, Third Party: Unsplash → API, BOTs, APIs, KumpeApps Credentials Manager Jun 14, 06:30:00 GMT+0 - Identified - We are planning for a scheduled maintenance during that time. We have been notified our Data Center will be performing maitenance to our servers during this time. Jun 14, 07:30:00 GMT+0 - Completed - Maintenance has completed successfully Jun 14, 06:30:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 1 hour

Affected Components: , , , , , , , , , , , , , , , , ,

<p><small>Jun <var data-var='date'> 14</var>, <var data-var='time'>06:30:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during that time. We have been notified our Data Center will be performing maitenance to our servers during this time..</p> <p><small>Jun <var data-var='date'> 14</var>, <var data-var='time'>07:30:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Jun <var data-var='date'> 14</var>, <var data-var='time'>06:30:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Fri, 14 Jun 2024 06:30:00 +0000 https://kumpeapps.instatus.com/maintenance/clx9d1p2i1029bhlszlnt89wm https://kumpeapps.instatus.com/maintenance/clx9d1p2i1029bhlszlnt89wm Mail Server Outage Type: Incident Duration: 4 days, 1 hour and 52 minutes Affected Components: KumpeApps Email Jun 4, 00:55:53 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. The fix may take 12hrs to take full effect for all users. Jun 3, 20:17:48 GMT+0 - Investigating - We are currently investigating this incident. Jun 3, 20:19:39 GMT+0 - Investigating - We are still investigating this incident. We have identified server is unreachable. We are attempting a server reboot. Jun 3, 20:26:28 GMT+0 - Identified - We have identified that our hosting provider changed our IP address without notice. We are working to update systems but the fix will take 12-24hrs to replicate world wide. Jun 7, 22:09:44 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 4 days, 1 hour and 52 minutes

Affected Components:

<p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>00:55:53</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result. The fix may take 12hrs to take full effect for all users..</p> <p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>20:17:48</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>20:19:39</var> GMT+0</small><br><strong>Investigating</strong> - We are still investigating this incident. We have identified server is unreachable. We are attempting a server reboot..</p> <p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>20:26:28</var> GMT+0</small><br><strong>Identified</strong> - We have identified that our hosting provider changed our IP address without notice. We are working to update systems but the fix will take 12-24hrs to replicate world wide..</p> <p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>22:09:44</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Mon, 3 Jun 2024 20:17:48 +0000 https://kumpeapps.instatus.com/incident/clwzexdqw15576bon8we1erai3 https://kumpeapps.instatus.com/incident/clwzexdqw15576bon8we1erai3 Data Center Outage- All servers down Type: Incident Duration: 13 hours and 47 minutes Affected Components: KHome Portal, KKid, A Creative Collection, KumpeApps SSO, UVT for IOS, K-Meds, KumpeApps Email, Kumpe3D, Retool, UVT Web, Kumpe3D Kiosk, Primary Data Server, Secondary Data Server, Third Party: Unsplash → Image Service → CDN, Third Party: Unsplash → API, BOTs, APIs, KumpeApps Credentials Manager May 26, 11:17:00 GMT+0 - Investigating - We are currently investigating this incident. May 27, 01:04:06 GMT+0 - Resolved - This incident has been resolved. May 26, 14:36:26 GMT+0 - Identified - Our hosting provider's entire Dallas data center is currently down. We are unable to roll over to backup servers as our home office is also down due to tornado damage and wide spread power outages. We are working with our hosting company to resolve the issue but power and internet outages at our home office is making it difficult. We appoligize for the inconvenience. Type: Incident

Duration: 13 hours and 47 minutes

Affected Components: , , , , , , , , , , , , , , , , ,

<p><small>May <var data-var='date'> 26</var>, <var data-var='time'>11:17:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>May <var data-var='date'> 27</var>, <var data-var='time'>01:04:06</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>May <var data-var='date'> 26</var>, <var data-var='time'>14:36:26</var> GMT+0</small><br><strong>Identified</strong> - Our hosting provider's entire Dallas data center is currently down. We are unable to roll over to backup servers as our home office is also down due to tornado damage and wide spread power outages. We are working with our hosting company to resolve the issue but power and internet outages at our home office is making it difficult. We appoligize for the inconvenience..</p> ]]> Sun, 26 May 2024 11:17:00 +0000 https://kumpeapps.instatus.com/incident/clwnn3uuh72598brouc9vckhq0 https://kumpeapps.instatus.com/incident/clwnn3uuh72598brouc9vckhq0 Primary Server Outage Type: Incident Duration: 12 hours and 11 minutes Affected Components: KHome Portal, KKid, A Creative Collection, KumpeApps SSO, UVT for IOS, K-Meds, KumpeApps Email, Kumpe3D, Retool, UVT Web, Kumpe3D Kiosk, Primary Data Server, Secondary Data Server, Third Party: Unsplash → Image Service → CDN, Third Party: Unsplash → API, BOTs, APIs, KumpeApps Credentials Manager May 3, 15:13:23 GMT+0 - Investigating - We have recieved outage notices from our primary data server. We are currently investigating this incident. May 3, 15:43:03 GMT+0 - Identified - We have identified an issue with our Dallas data center. We are working with our hosting company to resolve the issue. May 3, 21:05:10 GMT+0 - Monitoring - Our servers are back up with degraded performace. We are continuing to monitor the situation. May 4, 03:23:54 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 12 hours and 11 minutes

Affected Components: , , , , , , , , , , , , , , , , ,

<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>15:13:23</var> GMT+0</small><br><strong>Investigating</strong> - We have recieved outage notices from our primary data server. We are currently investigating this incident..</p> <p><small>May <var data-var='date'> 3</var>, <var data-var='time'>15:43:03</var> GMT+0</small><br><strong>Identified</strong> - We have identified an issue with our Dallas data center. We are working with our hosting company to resolve the issue..</p> <p><small>May <var data-var='date'> 3</var>, <var data-var='time'>21:05:10</var> GMT+0</small><br><strong>Monitoring</strong> - Our servers are back up with degraded performace. We are continuing to monitor the situation..</p> <p><small>May <var data-var='date'> 4</var>, <var data-var='time'>03:23:54</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Fri, 3 May 2024 15:13:23 +0000 https://kumpeapps.instatus.com/incident/clvqtehd476149azoj2r7uqn4u https://kumpeapps.instatus.com/incident/clvqtehd476149azoj2r7uqn4u Server Outage Type: Incident Duration: 3 hours and 12 minutes Affected Components: KHome Portal, KKid, A Creative Collection, KumpeApps SSO, UVT for IOS, K-Meds, KumpeApps Email, Kumpe3D, Retool, UVT Web, Kumpe3D Kiosk, Primary Data Server, Secondary Data Server, Third Party: Unsplash → Image Service → CDN, Third Party: Unsplash → API, BOTs, APIs, KumpeApps Credentials Manager Apr 30, 12:00:00 GMT+0 - Investigating - We are currently investigating this incident. Apr 30, 12:33:16 GMT+0 - Identified - Issues have been identified with hosting company causing degraded performance. A ticket has been raised for them to look at the issue. Apr 30, 15:11:47 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 3 hours and 12 minutes

Affected Components: , , , , , , , , , , , , , , , , ,

<p><small>Apr <var data-var='date'> 30</var>, <var data-var='time'>12:00:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Apr <var data-var='date'> 30</var>, <var data-var='time'>12:33:16</var> GMT+0</small><br><strong>Identified</strong> - Issues have been identified with hosting company causing degraded performance. A ticket has been raised for them to look at the issue..</p> <p><small>Apr <var data-var='date'> 30</var>, <var data-var='time'>15:11:47</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Tue, 30 Apr 2024 12:00:00 +0000 https://kumpeapps.instatus.com/incident/clvmd9hwr52667bdob6w28jppr https://kumpeapps.instatus.com/incident/clvmd9hwr52667bdob6w28jppr Emergency Maitnenance Type: Maintenance Duration: 1 hour Affected Components: KHome Portal, KKid, A Creative Collection, KumpeApps SSO, UVT for IOS, K-Meds, KumpeApps Email, Kumpe3D, Retool, UVT Web, Kumpe3D Kiosk, Primary Data Server, Secondary Data Server, Third Party: Unsplash → Image Service → CDN, Third Party: Unsplash → API, BOTs, APIs, KumpeApps Credentials Manager Apr 18, 23:00:00 GMT+0 - Completed - Maintenance has completed successfully Apr 18, 22:00:01 GMT+0 - Identified - Maintenance is now in progress Apr 18, 22:11:23 GMT+0 - Identified - Primary server backup has completed. Primary servers should now remain stable. Secondary server update has started. Apr 18, 22:00:00 GMT+0 - Identified - Our secondary servers are out of sync. We will be performing maitenance to re-sync which will cause the primary server to be down for a couple min. We are scheduling a 1hr maitenance window but actual down time on primary servers should be minimal. Type: Maintenance

Duration: 1 hour

Affected Components: , , , , , , , , , , , , , , , , ,

<p><small>Apr <var data-var='date'> 18</var>, <var data-var='time'>23:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Apr <var data-var='date'> 18</var>, <var data-var='time'>22:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Apr <var data-var='date'> 18</var>, <var data-var='time'>22:11:23</var> GMT+0</small><br><strong>Identified</strong> - Primary server backup has completed. Primary servers should now remain stable. Secondary server update has started..</p> <p><small>Apr <var data-var='date'> 18</var>, <var data-var='time'>22:00:00</var> GMT+0</small><br><strong>Identified</strong> - Our secondary servers are out of sync. We will be performing maitenance to re-sync which will cause the primary server to be down for a couple min. We are scheduling a 1hr maitenance window but actual down time on primary servers should be minimal..</p> ]]> Thu, 18 Apr 2024 22:00:00 +0000 https://kumpeapps.instatus.com/maintenance/clv5s20pt38964ban60zg6v17r https://kumpeapps.instatus.com/maintenance/clv5s20pt38964ban60zg6v17r Primary Server Down Type: Incident Duration: 9 hours and 30 minutes Affected Components: KHome Portal, KKid, A Creative Collection, KumpeApps SSO, UVT for IOS, , K-Meds, KumpeApps Email, Kumpe3D, UVT Web, Kumpe3D Kiosk, Primary Data Server, BOTs, APIs, KumpeApps Credentials Manager, Kumpe3D Admin → Apr 16, 01:22:40 GMT+0 - Identified - We recieved notification that our KumpeApps system is unresponsive. Upon further investigation our Primary Server and Bot Server both are down. A ticket has been opened with our hosting company for resolution. Apr 16, 01:30:01 GMT+0 - Monitoring - The issue appears resolved. Monitoring for a few min to verify stability. Apr 16, 01:32:10 GMT+0 - Resolved - This issue is resolved. All servers are not back up and running stable. Type: Incident

Duration: 9 hours and 30 minutes

Affected Components: , , , , , , , , , , , , , ,

<p><small>Apr <var data-var='date'> 16</var>, <var data-var='time'>01:22:40</var> GMT+0</small><br><strong>Identified</strong> - We recieved notification that our KumpeApps system is unresponsive. Upon further investigation our Primary Server and Bot Server both are down. A ticket has been opened with our hosting company for resolution..</p> <p><small>Apr <var data-var='date'> 16</var>, <var data-var='time'>01:30:01</var> GMT+0</small><br><strong>Monitoring</strong> - The issue appears resolved. Monitoring for a few min to verify stability..</p> <p><small>Apr <var data-var='date'> 16</var>, <var data-var='time'>01:32:10</var> GMT+0</small><br><strong>Resolved</strong> - This issue is resolved. All servers are not back up and running stable..</p> ]]> Tue, 16 Apr 2024 01:22:40 +0000 https://kumpeapps.instatus.com/incident/clv1p8pdl20983b1nem0jtwdy3 https://kumpeapps.instatus.com/incident/clv1p8pdl20983b1nem0jtwdy3 Kumpe3D Maitenance Type: Maintenance Duration: 1 hour and 17 minutes Affected Components: Kumpe3D Mar 15, 23:00:01 GMT+0 - Identified - Maintenance is now in progress Mar 16, 00:16:51 GMT+0 - Completed - Our maintenance has been completed ahead of schedule. Mar 15, 23:00:00 GMT+0 - Identified - We are planning for a scheduled maintenance on the [kumpe3d.com](http://kumpe3d.com) website during that time Friday March 15th @ 8pm until Saturday March 16th @ 8pm. The Kumpe3D Website will be down during this maintenance period. No other websites will be affected. Type: Maintenance

Duration: 1 hour and 17 minutes

Affected Components:

<p><small>Mar <var data-var='date'> 15</var>, <var data-var='time'>23:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Mar <var data-var='date'> 16</var>, <var data-var='time'>00:16:51</var> GMT+0</small><br><strong>Completed</strong> - Our maintenance has been completed ahead of schedule..</p> <p><small>Mar <var data-var='date'> 15</var>, <var data-var='time'>23:00:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance on the [kumpe3d.com](http://kumpe3d.com) website during that time Friday March 15th @ 8pm until Saturday March 16th @ 8pm. The Kumpe3D Website will be down during this maintenance period. No other websites will be affected..</p> ]]> Fri, 15 Mar 2024 23:00:00 +0000 https://kumpeapps.instatus.com/maintenance/cltsnczaq42686b9n275cjir0q https://kumpeapps.instatus.com/maintenance/cltsnczaq42686b9n275cjir0q Kumpe3D Outage Type: Incident Duration: 18 minutes Affected Components: Kumpe3D Mar 15, 18:44:19 GMT+0 - Investigating - Our Kumpe3D website is unable to access API servers causing there to be no products available. We are currently investigating this incident. Mar 15, 19:02:29 GMT+0 - Resolved - This incident has been resolved. Security keys were missing from our authentication server. This has been corrected and everything appears normal now. Type: Incident

Duration: 18 minutes

Affected Components:

<p><small>Mar <var data-var='date'> 15</var>, <var data-var='time'>18:44:19</var> GMT+0</small><br><strong>Investigating</strong> - Our Kumpe3D website is unable to access API servers causing there to be no products available. We are currently investigating this incident..</p> <p><small>Mar <var data-var='date'> 15</var>, <var data-var='time'>19:02:29</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. Security keys were missing from our authentication server. This has been corrected and everything appears normal now..</p> ]]> Fri, 15 Mar 2024 18:44:19 +0000 https://kumpeapps.instatus.com/incident/cltt0d0gi81309bamuinxlgeiq https://kumpeapps.instatus.com/incident/cltt0d0gi81309bamuinxlgeiq BOT Server Down (should not affect most websites) Type: Incident Duration: 1 day, 18 hours and 10 minutes Affected Components: Kumpe3D, Kumpe3D Kiosk, BOTs, KumpeApps Credentials Manager Mar 10, 06:30:00 GMT+0 - Investigating - Our BOT Server is down and under investigation. Mar 10, 16:48:19 GMT+0 - Identified - Our BOT server is currently down due to a failed migration by our host provider. We are working with our host provider for a swift resolution. This will only affect automated background systems and internal applications. Mar 11, 03:20:48 GMT+0 - Identified - Hosting company has reinstalled our server causing 100% data loss. Backups have bene restord but credential data was evidently not in our backup. Credential data will have to be manually restored then service tokens will have to be re-issued to all BOTS. This will take some time to complete. Mar 11, 15:14:25 GMT+0 - Identified - We have restored Kumpe3D systems and are in progress of restoring the other BOTS. Credentials manager is back up and running but we are still working on restoring it's data. Mar 12, 00:40:16 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 day, 18 hours and 10 minutes

Affected Components: , , ,

<p><small>Mar <var data-var='date'> 10</var>, <var data-var='time'>06:30:00</var> GMT+0</small><br><strong>Investigating</strong> - Our BOT Server is down and under investigation..</p> <p><small>Mar <var data-var='date'> 10</var>, <var data-var='time'>16:48:19</var> GMT+0</small><br><strong>Identified</strong> - Our BOT server is currently down due to a failed migration by our host provider. We are working with our host provider for a swift resolution. This will only affect automated background systems and internal applications..</p> <p><small>Mar <var data-var='date'> 11</var>, <var data-var='time'>03:20:48</var> GMT+0</small><br><strong>Identified</strong> - Hosting company has reinstalled our server causing 100% data loss. Backups have bene restord but credential data was evidently not in our backup. Credential data will have to be manually restored then service tokens will have to be re-issued to all BOTS. This will take some time to complete..</p> <p><small>Mar <var data-var='date'> 11</var>, <var data-var='time'>15:14:25</var> GMT+0</small><br><strong>Identified</strong> - We have restored Kumpe3D systems and are in progress of restoring the other BOTS. Credentials manager is back up and running but we are still working on restoring it's data..</p> <p><small>Mar <var data-var='date'> 12</var>, <var data-var='time'>00:40:16</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Sun, 10 Mar 2024 06:30:00 +0000 https://kumpeapps.instatus.com/incident/cltlqxbnh304253wuoa2vxa2tag https://kumpeapps.instatus.com/incident/cltlqxbnh304253wuoa2vxa2tag Network Overload (Partial Outage) Type: Incident Duration: 14 hours and 6 minutes Affected Components: KHome Portal, , KKid, A Creative Collection, KumpeApps SSO, UVT for IOS, , K-Meds, KumpeApps Email, Kumpe3D, Retool, UVT Web, Third Party: Unsplash → Image Service → CDN, Third Party: Unsplash → API, BOTs, APIs, KumpeApps Credentials Manager, Data Servers → Kumpe3D Admin → Feb 24, 01:00:23 GMT+0 - Investigating - We have noticed degredated service on our servers causing our systems to be slow or not load at all. We are currently investigating this incident. Feb 24, 01:10:20 GMT+0 - Investigating - It has been identified that the CPU on our primary server is unloaded for unknown reasons. We are currently investigating this incident. Feb 24, 01:37:21 GMT+0 - Identified - We have identified that multiple servers are having the same issue at the same hosting provider. Furthermore our hosting provider's website appears to be experiencing the same issues. We have added the remaining servers in our network to this outage as well as it appears to be a data center or regional network issue. We have opened a ticket with our hosting provider to resolve the issue. Feb 24, 01:55:31 GMT+0 - Identified - Our BOT server is now offline. Updated affected systems to reflect outage. We are still waiting for a response form our hosting provider on the degraded services. Feb 24, 07:00:00 GMT+0 - Monitoring - The issue appears to be resolved. We are continuing to monitor services. Feb 24, 15:06:03 GMT+0 - Resolved - We have confirmed the outage has been resolved. Type: Incident

Duration: 14 hours and 6 minutes

Affected Components: , , , , , , , , , , , , , , , ,

<p><small>Feb <var data-var='date'> 24</var>, <var data-var='time'>01:00:23</var> GMT+0</small><br><strong>Investigating</strong> - We have noticed degredated service on our servers causing our systems to be slow or not load at all. We are currently investigating this incident..</p> <p><small>Feb <var data-var='date'> 24</var>, <var data-var='time'>01:10:20</var> GMT+0</small><br><strong>Investigating</strong> - It has been identified that the CPU on our primary server is unloaded for unknown reasons. We are currently investigating this incident..</p> <p><small>Feb <var data-var='date'> 24</var>, <var data-var='time'>01:37:21</var> GMT+0</small><br><strong>Identified</strong> - We have identified that multiple servers are having the same issue at the same hosting provider. Furthermore our hosting provider's website appears to be experiencing the same issues. We have added the remaining servers in our network to this outage as well as it appears to be a data center or regional network issue. We have opened a ticket with our hosting provider to resolve the issue..</p> <p><small>Feb <var data-var='date'> 24</var>, <var data-var='time'>01:55:31</var> GMT+0</small><br><strong>Identified</strong> - Our BOT server is now offline. Updated affected systems to reflect outage. We are still waiting for a response form our hosting provider on the degraded services..</p> <p><small>Feb <var data-var='date'> 24</var>, <var data-var='time'>07:00:00</var> GMT+0</small><br><strong>Monitoring</strong> - The issue appears to be resolved. We are continuing to monitor services..</p> <p><small>Feb <var data-var='date'> 24</var>, <var data-var='time'>15:06:03</var> GMT+0</small><br><strong>Resolved</strong> - We have confirmed the outage has been resolved..</p> ]]> Sat, 24 Feb 2024 01:00:23 +0000 https://kumpeapps.instatus.com/incident/clszdjqm1161104arofmpt5y9ss https://kumpeapps.instatus.com/incident/clszdjqm1161104arofmpt5y9ss Primary Data Server Degraded Service Type: Incident Duration: 12 hours and 44 minutes Affected Components: KHome Portal, APIs, BOTs, , KKid, A Creative Collection, KumpeApps SSO, UVT for IOS, K-Meds, KumpeApps Email, Kumpe3D, Retool, UVT Web, Data Servers → Jan 3, 14:10:02 GMT+0 - Identified - We are continuing to work on a fix for this incident. We have identified that the available RAM is shrinking for unknown reasons. We engaged the hostiing company for resolution. Temp added a swap file for temp relief. Jan 3, 06:00:00 GMT+0 - Investigating - Our primary data server is experiencing degraded service. We have opened a ticket with out hosting company to resolve the issue. Jan 3, 17:42:00 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Jan 3, 18:44:01 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 12 hours and 44 minutes

Affected Components: , , , , , , , , , , , ,

<p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>14:10:02</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident. We have identified that the available RAM is shrinking for unknown reasons. We engaged the hostiing company for resolution. Temp added a swap file for temp relief..</p> <p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>06:00:00</var> GMT+0</small><br><strong>Investigating</strong> - Our primary data server is experiencing degraded service. We have opened a ticket with out hosting company to resolve the issue..</p> <p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>17:42:00</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>18:44:01</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Wed, 3 Jan 2024 06:00:00 +0000 https://kumpeapps.instatus.com/incident/clqxulbb252618cemw4drzorbg https://kumpeapps.instatus.com/incident/clqxulbb252618cemw4drzorbg KHome Portal Error Type: Incident Duration: 18 days, 21 hours and 45 minutes Affected Components: KHome Portal Dec 10, 16:00:00 GMT+0 - Identified - KHome portal is missing permissions to config files after migration. Dec 10, 23:33:44 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 18 days, 21 hours and 45 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 10</var>, <var data-var='time'>16:00:00</var> GMT+0</small><br><strong>Identified</strong> - KHome portal is missing permissions to config files after migration..</p> <p><small>Dec <var data-var='date'> 10</var>, <var data-var='time'>23:33:44</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Sun, 10 Dec 2023 16:00:00 +0000 https://kumpeapps.instatus.com/incident/clq04cwv842036b5o6b7pa9wac https://kumpeapps.instatus.com/incident/clq04cwv842036b5o6b7pa9wac Primary Data Server Offline Type: Incident Duration: 1 day, 15 hours and 28 minutes Affected Components: KHome Portal, APIs, BOTs, , KKid, A Creative Collection, KumpeApps SSO, UVT Web, UVT for IOS, Kumpe3D, K-Meds, Retool, KumpeApps Email, Data Servers → Dec 10, 14:59:10 GMT+0 - Identified - We are continuing to work on a fix for this incident. Database Server, Email, and BOTs have been restored. We are still working on other applications Dec 10, 16:27:27 GMT+0 - Resolved - This incident has been resolved. After monitoring the servers for one hour they are very stable and operating as expected. To resolve the issue we migrated our systems to a new server this morning. During this migration, we decided to go ahead and upgrade the memory and processor of our new server. So the new server has much better memory and CPU speeds as well as an upgraded network which should also resolve issues in the past with our servers locking up during high-traffic events like Black Friday sales and new releases. Dec 9, 16:12:00 GMT+0 - Identified - Our primary data server is unstable. It has been identified as a memory issue on our hosted server. We are working with our hosting company to resolve the issue. Dec 10, 15:29:38 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. We ended up migrating our server to a new hosting company which experienced an outage as well. We have successfully migrated our systems to a new server and restored backups. Our backups are from 7pm Central Time on Dec 9th. Any data between then and now may be lost but we do not believe there was any data during that time as our servers were mostly down after that time. Dec 10, 13:54:43 GMT+0 - Identified - We are continuing to work on a fix for this incident. We have started migration to a new hosting provider which will be very labor intensive. We expect this migration to take several hours. Dec 9, 21:30:00 GMT+0 - Identified - We are continuing to work on a fix for this incident. We have requested our hosting provider to migrate our server to a new cluster. We are waiting for them to complete the migration. We are also looking into restoring our backups to a temporary server to get services back up. Dec 10, 03:13:26 GMT+0 - Identified - We are continuing to work on a fix for this incident. Restoring backups to a temporary server has failed as the server we had reserved for this is experiencing more problems than our primary server. We are now looking into a new hosting provider that can handle our servers better than the one we have. We hope to have a resolution within the next 24 hours. Hopefully, in the meantime, our hosting provider will resolve their issues. We apologize for the inconvenience and hope to have a resolution as quickly as possible. We have found that restarting our servers every 30 minutes will keep systems working with degraded service. But restarting every 30 minutes is a manual process so we will still experience degraded service. Dec 10, 03:25:11 GMT+0 - Identified - We are continuing to work on a fix for this incident. We have temporarily scheduled a reboot of our servers for every 45min which appears to be keeping the server working except for a few seconds every 45min. This should allow for service to continue to function with degraded service while we attempt a permanent fix. Type: Incident

Duration: 1 day, 15 hours and 28 minutes

Affected Components: , , , , , , , , , , , ,

<p><small>Dec <var data-var='date'> 10</var>, <var data-var='time'>14:59:10</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident. Database Server, Email, and BOTs have been restored. We are still working on other applications.</p> <p><small>Dec <var data-var='date'> 10</var>, <var data-var='time'>16:27:27</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. After monitoring the servers for one hour they are very stable and operating as expected. To resolve the issue we migrated our systems to a new server this morning. During this migration, we decided to go ahead and upgrade the memory and processor of our new server. So the new server has much better memory and CPU speeds as well as an upgraded network which should also resolve issues in the past with our servers locking up during high-traffic events like Black Friday sales and new releases..</p> <p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>16:12:00</var> GMT+0</small><br><strong>Identified</strong> - Our primary data server is unstable. It has been identified as a memory issue on our hosted server. We are working with our hosting company to resolve the issue..</p> <p><small>Dec <var data-var='date'> 10</var>, <var data-var='time'>15:29:38</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result. We ended up migrating our server to a new hosting company which experienced an outage as well. We have successfully migrated our systems to a new server and restored backups. Our backups are from 7pm Central Time on Dec 9th. Any data between then and now may be lost but we do not believe there was any data during that time as our servers were mostly down after that time..</p> <p><small>Dec <var data-var='date'> 10</var>, <var data-var='time'>13:54:43</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident. We have started migration to a new hosting provider which will be very labor intensive. We expect this migration to take several hours..</p> <p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>21:30:00</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident. We have requested our hosting provider to migrate our server to a new cluster. We are waiting for them to complete the migration. We are also looking into restoring our backups to a temporary server to get services back up..</p> <p><small>Dec <var data-var='date'> 10</var>, <var data-var='time'>03:13:26</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident. Restoring backups to a temporary server has failed as the server we had reserved for this is experiencing more problems than our primary server. We are now looking into a new hosting provider that can handle our servers better than the one we have. We hope to have a resolution within the next 24 hours. Hopefully, in the meantime, our hosting provider will resolve their issues. We apologize for the inconvenience and hope to have a resolution as quickly as possible. We have found that restarting our servers every 30 minutes will keep systems working with degraded service. But restarting every 30 minutes is a manual process so we will still experience degraded service..</p> <p><small>Dec <var data-var='date'> 10</var>, <var data-var='time'>03:25:11</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident. We have temporarily scheduled a reboot of our servers for every 45min which appears to be keeping the server working except for a few seconds every 45min. This should allow for service to continue to function with degraded service while we attempt a permanent fix..</p> ]]> Sat, 9 Dec 2023 16:12:00 +0000 https://kumpeapps.instatus.com/incident/clpykzrrh107539zhob1ij463ve https://kumpeapps.instatus.com/incident/clpykzrrh107539zhob1ij463ve