a tale of two incidents

 yesterday:

i did sleep, but only briefly, waking up an hour or so before our alarms to jump right back into AI cat wrangling. because my brain was on fire imagining all the cat wrangling i wanted to do.

so i began my day putting together a PR for my boss (not my client) which basically said "this is how it's done", and i was relieved to find it merged by this morning, so he obviously wasn't too offended :P

then i rushed off to the 24/7 to pick up mishloach manot for mr smear to take to school, and afterwards gd and i headed to the dentist for the first part of her two-part root canal, which i had to be there for because her dentist doesn't understand a word of english (or french).

[goddamn i love chocolate-covered pecans]

i dropped her off at home and continued on to the office, where i was completely manic and infected a data engineer and my client with my enthusiastic idea for an AI cat-wrangling IDE extension.

and then i ran a test that my cat-wrangling had produced, confident that it was ready and that i'd had good results from the dry runs.

...

i was pretty bored while waiting for missing user permissions to complete the tests we were running in production, so i took care of some other stuff while i waited.

while i was making myself a cup of coffee, an unknown employee began interrogating me as to why he didn't see me at the company purim party that i didn't go to. he obviously thought he was being funny, but at no point did he get the hint that it would be wise to drop it, and in front of HR staff brought it to a point where had i been so inclined, that could easily have earned him a warning for harassment.

then i was brought into a call with my client team for an incident i knew nothing about and had nothing to do with - let's call it incident B - only to be informed, as i was explaining to them that my coffee wasn't working yet, that it wasn't an issue for us.

at that point i realized that i was falling apart, so i finished my coffee and left for home.

on the way home i tried to pick up a frying pan from kikar ha'medina, but i couldn't find what i was looking for, all the while struggling to get hold of mr smear and then threatening him to get his ass home to assist me on a mission to dizengoff.

...

and then i got home, and learned that the test i'd run earlier in the day had led to an incident.

let's call it incident A.

it took me about ten minutes to figure out what was happening, and another five to realize it was my fault. another two minutes to decide how to fix it, at which point i saw that one of my teammates had already pushed a fix and i'd been typing into silence for five minutes.

over the next twenty minutes i was in contact with my teammate but had received nothing from our client team (who were presumed to be dealing with the incident), and nothing seemed to be happening so i rushed out with mr smear and a washing basket that needed returning to catch a bus to the center.

we were halfway there when my phone started buzzing again, and i'd left my computer at home 🤦‍♂️

the bus was packed, with non-compliant riders arguing with the conductors, and a stop too early i jumped off with mr smear and we raced through the crowded sidewalks, scrambled down to the store, where i had trouble finding a frying pan, eventually giving up and taking an expensive induction one.

we rushed through the center to catch another bus home, and when we returned it was to discover, to my absolute horror, that i'd taken our production services down completely by running a test that i had literally forgotten i hadn't reviewed yet.

i cannot believe i did something so utterly irresponsible, i was mortified and humiliated to post the list of things that had been unnecessarily affected, a danger that i would easily have spotted had i checked it beforehand.

while i scrambled to figure out what was happening - i was in a flat panic, and had no idea where to look for monitors or logs* - my bulgarian teammates were quietly working with AI chewing gum and duct-tape and getting things back online.

* in retrospect, nobody does. and there isn't a lot of monitoring or logging going on to begin with...

all i had to offer were my apologies and volunteering to write up the RCA.

i continued freaking out until my mentor called me (i'd messaged him), and he was very reassuring ("if somebody had lost their life, maybe i wouldn't have called you"). still very shaken, i went to bed early knowing that no good was going to come of me getting less sleep.

today:

surprisingly, i did get some rest.

mr smear gave us some trouble this morning - he's been doing the slow morning thing since we moved, and he's been talking back whenever he gets in trouble - but by the time he had to go to school we were all in a better place.

except for gd, who was pissed off at the new frying pan because it's not as non-stick as it says.

(alanis morissette) ironically, the technician for the stove was due to come this morning and the element was heating up fully. fortunately, they called to tell us he'd been delayed and so we rescheduled for monday, which gives us a chance to test if it's really working or not.

the appraiser for the household insurance came back, an elderly man (a biker, i should say) who was impressively professional and efficient.

finally, the technician from the internet company arrived to sort out internet in our in-home bomb-shelter (our bedroom).

it took a long time. it took a lot of assistance from us. at one point he left the apartment, and then returned but forgot to take his shoes off.

he took so long, that i had to stop following him around to join my morning post-incident meeting with my teammates, which was really important because i knew i had to be able to discuss my disastrous fuck-up while looking them in the eyes.

by the time he was done, i was extremely frustrated. firstly, because the wiring isn't what i thought he was going to install, it's simply a single, very long ethernet cable from one end of the apartment to the other, instead of two wall jacks. secondly, because the outlet in the bedroom doesn't fit properly, so it looks terrible and it was squashing the cable he'd just installed, and i had to explain this to him. thirdly, because as relieved as i am that the network appliance in our bedroom is now working well, the speeds we're getting are inconsistent and surprisingly low. finally, because goddamnit when i tell you you're damaging both my screwdriver and the screw by using the wrong bit, fucking stop.

jesus h. christ.

anyway, so gd and i washed the floor, and there's internet in the bedroom, and on that note i left for the office. it was lunch time already, so i had to skip lunch. on the way i caught up with my mother who'd just had an interesting conversation with the new head of the jewish agency in south africa.

i hope something positive comes out of it.

the first half hour at work was fighting with IT network issues.

the next two hours were a complete mess; trying to understand the impact of incident A as a step towards getting started on the RCA, strategizing with a data engineer, and, amongst other stuff, being dragged back into incident B.

i never did get started on that RCA - i spent five hours on incident B, and was somehow assigned to lead a task force and only walked out of the office a little before 9pm.

not only did i bear witness to a member of my client team - the only guy who legitimately knows what he's doing - threatening to quit, and being mollified by the team lead, but at every step of the way i learned more about the system architecture and was stunned by the absurdity of it all. it's shit, deep shit, all the way down.

also, the guy who just left the team last week? he'd convinced us all that the migration i took over was ready. tonight we discovered that it really wasn't.

oh, and i wasn't home to receive my son, who - in spite of a bit of a freak-out because he'd waited at the wrong bus stop on the way home - had managed to take the comics back to the comics library all by himself, by bus ^_^

my "dinner" was two slices of happy hour vegan pizza (it wasn't bad) and a beer, but when i got home i had gd's ramen (with bok choy and a yo egg) and dessert. after assuring gd that if i do lose my client over calling it quits for the weekend (with the rest of the task force backing me) it'll make him look bad, i immediately sat down to get all this down before i jump in the shower and probably climb straight into bed.

tomorrow's a big day, mostly in preparation for taking mr smear to a purim family trance party on saturday. we have no idea what's happening with iran - obviously if things go crazy we'll have to stay home - but whatever happens i'm mostly confident we'll be fine 🤞🙏