You must Sign In to post a response.
  • Category:

    ISC Server crashed and now recovered

    Dear all,

    Most of you would have noticed ISC was down for about 3 days. I just wanted to let our members know what happened.

    ISC and the sister sites are hosted on a server in the USA. We have a pretty old physical server sitting in a great data centre, while most other websites are now running on cloud-based systems. Since we are on a physical server, we have limitations on automated backups etc. We use a separate physical hard disk to take automated backups and periodically download such backups to other computers.

    On 25 May 2021, our server went down. We made every efforts to restart the server remotely and with the help of the support team. I wasn't panic until 24 hours later the support team informed me that the server could not be restored after the Windows crashed and the backup drive it was was crashed which is the root cause of the problem. This left me with a big question - if the backup drive itself is crashed, then I have to rebuild a new server with the data I had downloaded many days ago. All valuable posts from our members for several days will be lost.

    Since the server and support team was in the USA, I had to stay awake all night for 3 days motivating and begging them to make all possible efforts to recover the drives. Finally, they agreed to make a fresh installation of the Windows on the primary hard drive and make all efforts to keep the old data in the drive.

    Finally, after 3 days, we could reinstall Windows on the primary drive and I was excited to see the old data and files remaining on the drive. So, even though we lost all backups, the original data files were still present and I could recover everything without any data loss.

    We lost 3 valuable days but didn't lose any past data. Thanks a lot to the almighty God for this wonderful recovery after the serious situation we faced.

    We have learnt a big lesson the hard way. Now the goal is to ensure this will not happen again.

    In the next few days, we are moving to a more powerful server in India where we can get more support, a lot closer to us and in our time zone. Also, we will be going for a cloud-based server with automated backup systems in place.

    Thanks to all members who patiently waited for us.
  • #731437
    Thanks to the Almighty and thanks a lot for your efforts, Tony sir! We are all really glad that the site has been restored without any loss of data. Hope your efforts to move to a more powerful server within our timezone bears fruit at the earliest.

    Sir, it seems our time zone has changed. This post is dated 29th while today is 28th only.

    'Mastering others is strength. Mastering yourself is true power'. -Lao Tzu

  • #731438
    Happy to see that with the continuous efforts of the webmaster and his team we could come out of this difficult situation. What can be more disastrous than crashing of a server where all the activities of a particular site are resting as well as moving onwards on a daily basis. The decision of shifting the data and activities to a server in India makes sense. I hope with that the difficulties of getting the server to be repaired remotely will ease out. I wish our site to come out of this difficult time soon and have a smooth working in future. The members are with the webmaster and his team in whatever way we are required to stand in this difficult time.
    Knowledge is power.

  • #731442
    Thank you Tony sir! Yes, this is true we all missed ISC a lot those three days.
    @ Ganesh Sir, as you mentioned our time zone has changed, this could be a reason that points of yesterday and today are also displaying wrong.

  • #732580
    Glad to know that ISC is back with all the data intact. I appreciate the efforts of Tony Sir in recovering and restoring ISC to its members.

    When I was unable to access the site, I thought the problem is with my browser but then I realised it is a server crash. I felt bad and equally eager to return to ISC.

    I feel ecstatic to know Tony sir has tirelessly worked for the restoration of ISC and also happy to see a more powerful server for ISC in India with more technical support.

  • #731444
    Indeed, what happened was unexpected. And we have an efficient operator who works tirelessly to solve the problems. Server crashing is a serious one. I can imagine that situation. Even when our own desktop or laptop crashes, we lose everything and take it to the computer workshop for repairs. Sitting in India and managing a server in US is a great thing. And when the system crashes, it would be very difficulty to restore it.

    I really appreciate the way our dear WM managing the ISC server located in the United State by sitting in India Kochi. This should not happen again. The idea of shifting the ISC with a cloud based server in India is welcome.

    No life without Sun

  • #731445
    I appreciate the efforts of the Webmaster and his team in putting back the site on track. Eventhough it is a major breakdown they could solve the problem within a short span. We all should be happy and thank the Webmaster and his team for their concentrated efforts.
    The date and time that is being shown on the post are indicating that there is some adjustment to be made for getting back to our IST zone. In the top contributors' list also my yesterday score is being shown as zero. The same is the case with all the members I think. This point is to be taken care of, I feel. This is just feedback only for the attention of the concerned,
    It is very nice to hear that ISC is also shifting to cloud-based server in India. I feel it is very good and working for the technical group and administration also will become easy.

    always confident

  • #731450
    Glad to know all is well, especially that the site's data was not lost. Appreciate the efforts put in to restore everything and best wishes for the move to a more easily accessible location and using updated technology to keep the data intact.
    When you make a commitment, you create hope. When you keep a commitment you create trust! ~ John C. Maxwell

  • #731452
    Today I saw the clear and detailed statement of the webmaster. Although I do not fully understand the technical details of how our activities coming out, I was able to understand that behind this was the need for constant vigilance, workability, and technical skill and that the ISC had enough of them. At the same time, it is clear from that statement how much distress and mental anguish one has to endure when such a problem arises. Although it is hosted through a server in the United States, it is very difficult to manage such things properly from here, but I would like to thank all the members of the ISC Technical Department for being able to restore anything saved in its original form within three days. Beyond that, I thank God for the invisible power that gave me the courage and thoughts to restore the server without any problems. We pray to God to give you the strength to continue to be vigilant so that we do not go through such crucial problems.
    My yesterday's point become ZERO in todys contribution list published in front. Please verify.

    "Cheruthulli Peruvellam"
    small drops make a mighty ocean

  • #731454

    The zero points problem has been conveyed to Tony Sir, as also the date-time stamp issue, this stamp reflecting as per the US time zone.

    When you make a commitment, you create hope. When you keep a commitment you create trust! ~ John C. Maxwell

  • #731462
    Thank you ME.
    "Cheruthulli Peruvellam"
    small drops make a mighty ocean

  • #731463
    Thanks Tony Sir for getting things restored that too without any loss of data. For three days almost everyone was in darkness, not sure how things will turn up. Hope we shift to a cloud based platform at the earliest. Great efforts by you.
    Patience and perseverance pays

  • #731466
    But the date shown by this post raised by the wembaster shows 29th May to which we are one day far. So there seems to be more glitch as I have raised the same in seperate thread. Nevertheless the site has been restored and we are able to make contributions now. But yesterday I did experience the log in issues again and after some time the site was functioning normally. Now the webmaster as to figure out the time line issue. Previously our date used to change at midnight, then it restored to 5.30 am daily and today it is again in confusion state as our past points got reduced to very less.
    K Mohan @ Moga
    'Idhuvum Kadandhu Pogum "
    Even this challenging situation would ease

  • #731472
    Now the date is showing 30th May 2021. There is some serious issue, and I am sure it will be resolved soon by our technical team and webmaster after then all points will be displayed in the correct manner. Till then we continue our contribution to ISC like normal days.

  • #731479
    Thanks Tony for this post.

    Having worked in an atmosphere of relying on computer operations,data store and support, I know the agony people at helm undergo when things go wrong.
    It is from such serious issues and effects that better and better Disater Recovery methods are being followed.

    The greatest relief and silver lining in this is that unlike the clients of other service operators or organisations, no ISC member or user blamed ISC establishment, but were just perplexed and were themselves undergoing agony.

    All is well that ends well. At least the benefit is that some new lessons learnt.
    Thanks again for Tony for this explanatory post.
    You can rest assured that ISCians will stay put and offer full co-operation to ISC team at any time.

  • #731497
    Very glad that the efforts of the webmaster rejuvenated the website without any data loss. I thank you very much for this. It is heartening to know that ISC will use cloud based hosting in the coming days, thereby preventing data loss with the facility of automatic backups in future.

  • #731504
    I think this is the first time that the ISC website not working for three days in a row. It was surely God's blessing that was able to solve the server problem within three days without losing any data. We can understand how much the webmaster and team tried to solve the server problem in the United States from India. I commend the efforts of the webmaster and his team to bring the ISC site back to normal.
    Lighted to Lighten

  • #732862
    Today morning when I was about to post a thread the submit button was concealed with over lapping ads which was same ad appearing as chain six to seven times and our page extended more down and could not submit the post.
    K Mohan @ Moga
    'Idhuvum Kadandhu Pogum "
    Even this challenging situation would ease

  • #732863
    Today being the second day in a row, I am facing the similar problem of disappearing the written contents all of sudden without any prior warning. In order to post two posts relating to forum responses, I had to rewrite the twice. Hope you would look into at the earliest.

  • #732877
    I am glad that we were able to recover the data without any loss. It is the biggest problem with legacy servers. Maintenance of these servers and backup process is most costly and required more effort along with the risk of going down anytime. It is time for ISC to move to cloud-based solutions such as AWS, Azure, or Google Cloud platform and make ISC website highly available and resilient. Cloud-based solutions offer automated backups and point-in-time recovery too.

  • This thread is locked for new responses. Please post your comments and questions as a separate thread.
    If required, refer to the URL of this page in your new post.