If you own a Kyun service in NL or checked status.kyun.sh in the past ~3 weeks, you probably know the situation, and I'm not going to sugarcoat it. TL;DR:
I would like to start off by thanking Ivo from airnode.nl for pressing our previous upstream into giving our server up, personally picking it up and shipping it. They wouldn't take our money for anything so show them some love.
Here's what happened, but with more words and SNCA:
Around February this year, I bought a server, and with my 2005 Audi and a buddy of mine, we drove 3400 km and personally delivered it to a provider near Amsterdam.
Why deliver personally? I don't know, I'm not the most financially responsible person, but it was fun. Especially driving all the way back to Romania in limp mode.
Anyways, about 6 weeks later, they still had not completed the setup, our server was still offline, at which point I contacted U2 (Upstream 2) to ask whether they would consider personally picking the server up from the datacenter and setting it up in theirs. U2 agreed and picked the server up at the agreed date.
I won't bore you with details but, of course, the server only went up at U2 about a month after the deadline both of us agreed to. I'd already promised NL to everyone in the Kyun community months earlier, and we really couldn't afford to lose even more time and money by moving the server yet again.
My thoughts at this point were: "hey, it's finally up now, U2 just needs to keep it powered and connected to the internet and we can pray that the hardware doesn't fail, they don't need to touch it again. all customer data is RAID10, so storage is safe as long as they swap the disks like, within a week of them failing".
For some reason, they also didn't seem to be willing to set up a way to access our server's out-of-band management system, so in case of any hardware failure, we were completely fucked and at their mercy.
In retrospect, this was the point we should've cut our losses and tried another upstream. But, like I said, we really couldn't afford to.
Shortly afterwards, a clear pattern emerged:
We've asked U2 to swap the RAM, CPU and SSDs into another known good server of theirs we would've rented (we would've paid for this service separately of the rented server, of course), they refused.
We've asked them to allow us to send someone else into their datacenter to troubleshoot our server, they refused.
Of course, seeing all this, we were scrambling to put together money for a plan to get out of this mess with as little customer impact as possible.
I feel the need to remind you all that Kyun isn't some huge company with huge capital. Kyun is a small bootstrapped company and a passion project of mine. This is why Kyun can afford to experiment so much and offer an experience no one else in the industry does, with never-before-seen features. This also means that we depend a lot on bigger companies not fucking us over. We can't just up and leave whenever we feel like it.
The plan was to buy a new server, ship it to another upstream (carefully selected this time because we wouldn't be desperate), and allow customers to live migrate over from U2 to U3 (you guessed it, upstream 3!) on their own time, until a certain deadline, where all of the services from U2 would be automatically moved to U3. Afterwards, the (now empty) U2 server would've been powered down and shipped to U3.
Around September 13, we paid for the new server and found a suitable new upstream, waiting for shipment. Everything was going according to plan, until:
Of course, you can't always get what you want. U2 has announced a physical migration of all of their hardware around this period, and they claimed it shouldn't take longer than 30 minutes.
As expected, 30 minutes came and went. I asked U2 what was going on and they claimed, quote, "we didn't touch your server".
Remember, we have no out-of-band access to this server. We can't know what's wrong if it goes down. So we ask them to check.
30 minutes turn into hours which turn into days with no end in sight. They tell us it's the power supply. We ask them if they have any in stock. They say they do. We ask them to swap them and invoice us. They ignore us.
At this point, we've had enough and we ask U2 to ship the server to U3.
After days of ignoring us, U2 claims they won't ship our server until we pay the rest of the contract. I think to myself "what the fuck is this dude talking about? i didn't agree to any contract, it's on a monthly period"
I ask U2 to show me the contract I signed. They ignore me (I miiiight've also threatened to send the payment for the rest of the contract to a lawyer instead). At this point, I had 2 options:
For those of you who don't know, LowEndTalk is the drama center of the hosting industry. I've said it before, I don't like airing dirty laundry out like this, but I reaaaally didn't want to pay that lawyer. So I thought the post would pressure U2 into doing the right thing and giving us our property.
This is where Ivo comes in.
Long story short, he sees my post and offers to communicate, formally, with U2, in Dutch, and attempt to coordinate pickup with them. I agree and, surprisingly, it works. Maybe it was personal.
On September 29, he drives about 300km just to pick our server up (which was missing the top cover and the rails I delivered the server with (lol)), finds me a box to ship the server in, and pays for the shipping to U3 at his own cost. Honestly, he singlehandedly restored my faith in the Netherlands. Y'all should make him president or something. Again: airnode.nl!!!!
September 30, the old server arrives at U3, intact. I didn't want to take it up without the top cover and with potential hardware issues (if U2 is to be believed). Waiting on the new server.
October 5, the new server also arrives, also intact. The drives from the old server are swapped in the new one, and, 2 hours later (!!!!), the new server is up and configured. Granted, there was a BGP configuration issue on their end which I only got around to figuring out at 11 PM, but they fixed it first thing next day. Shoutout to calibour.com, another great company with extremely competent people. Thank you, Alyx.
Again, every single Kyun NL service, regardless of purchase date, got its renewal date extended by 30 days.
We have a new update cooking with lots of stability improvements, bug fixes and new features (including live migration across locations and Danbo encrypted serial recording), which should ship soon.
Regarding the old server, after we find a new top cover for it and some rails (and some new PSUs just in case), we will run an extensive stress test to ensure stability before we sell anything on it, but expect more NL stock soon-ish.
Thank you for reading and thank you for your support.
- naphtha