LowEndTalk member @jsg has been on LET for a little more than two years. In that time he’s gained a lot of respect in the community for the depth of his technical understanding on many fronts. From chips to code, @jsg brings a ton of experience and often insightful commentary to LET. I talked him into doing an interview. At first, he thought no one would be that interested but I think as you read the interview, you’ll find out a lot about paths the average LET reader doesn’t often tread.
Q: What part of the world you live in?
A: Western Europe
Q: What is your background with computers?
A: Electronics, haha. As a boy I got an electronics “lab” kit as a response to my, uhm, plenty sufficiently voiced interest. About 10 years later I earned my living during uni studies (physics) designing and building diverse electronics, among others for/at a large American corporation. Back then a software engineer I had come to like asked whether I was willing to help them because they were way behind their schedule on some important project. When objecting that I knew plain nothing about software he simply said “don’t worry, you’ll learn it quickly”. Well, as my own project there was all but finished and as I liked (him and) the company and felt like staying he asked the big boss and he agreed to me staying but now for software.
One point worth mentioning is how they “got me started” because it was a very hard school and I hated it but today I’m glad about it: they gave me a copied multi-100 pages C compiler manual and a Compaq PC with a (I think it was an IBM) C compiler. And that was it. Forward skip some months and I had got myself the K&R bible and had managed to create some useful routines for the project. Small stuff but it (mostly) worked. I guess that coming from electronics actually helped me because I usually knew what some line of code would actually do in hardware.
Since then I have worked on many different projects and have gained some experience with some languages and architectures usually working on code relatively close to the “metal”. One field I tried to stay away from is anything to do with graphics and one field whose importance I learned (the hard way) is security, first as in “safety” and “reliability” and later as in “crypto”.
Q: When you say you learned about security “the hard way” I sense a painful education. Care to share?
A: Kind of. There are actually two points to this. The first one was to painfully learn the importance of security e.g. and mainly to clients in sensitive fields being enraged by even very few safety “hiccups” due to imperfect code. The other point was to – properly – learn what “security” means and how it can be achieved, measured, demonstrated/proven etc.
“Security” has many faces and can mean a lot of things including reliability, safety, resilience, etc. and nowadays often crypto – and all of this on different levels like the code itself, the algorithms, the protocols, etc. When I began my journey there wasn’t a lot of literature, let alone tools available so I had to do quite a bit of research by myself in order to be able to even just properly define e.g. safety and to differentiate it from e.g. reliability as well as to knowing about relevant inter-relations. That plus on the other side clients demanding safe code from me boiled down to a hard and sometimes painful way for me.
Funny side note: Even today when everyone is much more concerned about safety and security and when millions of passwords and even identities are stolen each and every weak, a surprisingly large part of developers still have a rather vague concept of safety and security. Not a good omen …
Q: What kind of work do you do?
A: Nowadays (since quite some years now) mostly IT security related stuff like implementing, porting, (implementation) hardening or (speed) optimizing crypto algorithms, hardening some existing software, designing and implementing hardened routines (e.g. protocols), etc. Sometimes although rarely I also (co)lead, (co)design, and(co)implement a full software product.
Q: You refer sometimes to static analysis and that is probably a foreign concept to most people. What exactly does that refer to? What tools/languages are involved?
A: Static Analysis is a form of specialized mathematically rigorous testing software. Two relatively well known tools are “Frama-C” and “Spark” which is related to Ada. The goal is basically to check whether some piece of code, typically a procedure/function really does what it’s supposed/specified to do. In reality though SA is used only rarely because it’s a lot of work, rather exotic and cumbersome, in part due to basically being math and in part because there are very few good tools that are actually usable for most developers.
At the same time though SA is the only known way to create reliable, safe, and error-free software. Another unfortunate fact is that SA isn’t just “use a weird tool” but it’s just one link of a “formal chain” that starts with formal specification (which next to nobody does although tools like TLA+ make it easier).
Q: What kinds of projects use that level of security? Are they apps that the average user might use, embedded software, industry-specific vertical apps, or…?
A: It’s pretty much only used in very sensitive settings like for example air traffic control, railway control, critical financial systems, etc. And no, virtually all software for mere mortal average users – unfortunately! – is not formally verified. And yes, that’s also true for e.g. virtually all online banking software and operating systems in common use and btw. also for most widely used crypto.
Q: Most developers and sysadmins have some notion of security. Is it fair to think that what you work on is “the next level” or is it something completely different? Is there any application to the work you do for the average developer?
A: Neither and both in a way, on “the next level” and something completely different. As for developers, the relevant factor is almost always product management which, as most developers know, is very rarely about safety/security but rather about “more features!” and shipping dates. As for sysadmins, at least some of them really try hard to configure and run safe and reliable systems but the limiting factor is the fact that they virtually always are limited by having to do with existing software. Example: if you are a linux sysadmin you have to live with the screws linux offers for you to turn. Btw. I do not consider linux particularly safe.
Q: What languages do you prefer or recommend for development?
A: Well that obviously depends on who asks for a recommendation. Frankly, to most developers (and “developers”) I have nothing to recommend; they are far away from even a starting point.
So I’ll rather tell you what languages I prefer. Most of all C, albeit exotic variants and/or heavily formally annotated. C still is the “lingua franca” and available for pretty much any and every CPU/MCU, often as the first language available and sometimes even as the only one. Plus C is a preferred target language for quite a few languages.
My second vote goes to Ada, one of the reasons being that Ada while being somewhat cumbersome to work with and not having lots of libraries available still is the “cheapest” way to build a decent software in reasonable time; if one (knows and does) use Spark too one can even achieve a rather high level of quality/reliability/safety, which probably is the reason why a large part of the (very few) high reliability softwares have been done in Ada.
That said, I usually don’t directly use neither C nor Ada for the bulk of my work but a few rather exotic languages for which there either already exists or has been done by myself good formal tool support. Simple reason: Ada is too limited in some modern fields (e.g. high speed network software) and proper C development is too inefficient. But the languages I use do compile to C and I always look at, check, and verify the generated C code.
If pressed to offer a recommendation to other developers it’s this: use a statically typed language.
Q: So when someone says “Ada,” I think of the US Department of Defense initiative to replace its mass of programming languages. I thought Ada had moved into legacy mode…apparently it’s still being widely used?
A: Well, that depends on what one takes “widely used” to mean. Looking at the complete language and development universe Ada is far behind even not exactly mainstream languages and even in the defense sector Ada lost most of its grip as soon as the “Ada only!” rule was more and more relaxed.
In the field of security though Ada still is a strong contender because it has quite a bit to offer; a really nice and good IDE, a really good multitasking abstraction (which virtually always is multi-threading based), a (somewhat deceiving) similarity to Pascal, and of course Spark, one of the best integrated static analysis engines. Short version: if one really and seriously needs reliable safe code Ada probably is the cheapest way to go among the very few available choices. And, well noted, Ada 2012 (the current incarnation) is very different from Ada 95 (which most people mentally associate with “Ada”) and far less cumbersome to use. So, to summarize it, one might say that Ada changed its focus (and capabilities) quite a bit from being the language forced upon contractors by the DOD towards being a heavily safety and reliability oriented “zero errors” language. Accordingly one finds it being mainly used in very sensitive areas like satellites, air travel and train control/management, the (inner core) of the financial sector and similar.
So, “widely used” would be an exaggeration but so would be calling it “almost dead”. Btw. some of the (rather few) newer safety focused languages try to emulate (or copy) some of Ada’s strengths and concepts and generally speaking Ada still is considered as the king of the safety hill. One funny example: I’m working (pro bono) as the guy in charge of safety and static analysis in a modern (well, it’s 10+ years old and beyond v. 1.0) languages core team and knowing Ada relatively well and having quite a bit of experience using it was the trigger for them to ask for my advice and later for my participation in the core team close to the BDFL. And everyone in the team considers Ada to be the benchmark for a safety oriented language. Also offering capabilities similar to Ada’s Spark as well as an interface that feels very similar is a highly coveted goal.
Q: You’ve often posted comments on the Intel vulnerabilities. What is your view on the current state of chips? Is it completely hopeless?
A: Pretty much, yes. Although I should mention for the sake of fairness that intel actually did at least try to get their designs and masks right, incl. using the “gold standard”, formal methods. The reason why in the end their efforts were bound to fail is “historic baggage”. The X86 architecture stems from a time (quite a few decades ago) when we didn’t and couldn’t know what we know today. Add to that the curse of needing to stay backwards compatible. Another relevant reason is the simple fact that no company throws away billions of $ of investment and of customers/users and just starts fresh with a clean slate.
AMD is somewhat better positioned because they had a long “hibernation” period during which they could – and without doubt did – think over a lot of things while intel understandably was caught in their product cycle.
Q: Do you really think AMD is better than Intel, or have they just been lucky?
A: That’s a tough one. I don’t think that AMD is (intellectually and engineering-wise) better than intel. As I just said I think AMD’s big advantage is basically two factors: (a) They had time, lots of it, to think over a lot of things and to try some better approaches while intel – don’t forget that large corporations have shareholders breathing on their neck – was caught in a wheel needing to keep the revenue flowing and understandably wasn’t eager to interrupt their cycle and operations. (b) Sorry, if that sounds political, but I think intel had more than just bad luck against it. The series of “misfortunes” is just too long and too consistent to believe in happenstance. Well noted, I can’t prove it, but in my minds eye someone, some powerful party or group, wanted and made intel to stumble. Yes, their former CEO seeming to having been a rather flawed person can explain some things but not that whole merciless string of misfortunes that seemed to begin with that CEO’s fall and that still hasn’t ended (e.g. they are still stuck at 14 nm) … and at exactly that time intels only significant potential competitor, AMD, just so happens to awake from a very long slumber and to suddenly have the resources (Mrs. Su, J. Keller, lots of money, …) needed to come up with a product that’s not only competitive but pretty much an intel killer? Again, I can’t prove it but I don’t believe in fairy tales.
As for hope, I don’t have a lot of hope wrt the whole X86 architecture; AMD may seem to offer a future but frankly I think X86 won’t be the relevant architecture anymore in, say, 10 years. It just has too much ugly old baggage. Being at that I wouldn’t bet on Arm either. Yes, there still is quite some gain to be squeezed out of it but it’s not the promised land either although I think it will stay for decades as a very relevant MCU architecure.
I personally see Risc-V as the future – plus some yet to come out chinese architecture; they have done an awful lot of research and they have produced some results that can’t – yet – cope with the western high-end CPUs but IMO they will, one of the driving forces being what drove them all along: independence. Nowadays a self respecting nation, and even more so one striving to be a power player, can not possibly tolerate to not have their own architecture (incl. the capability to mass-produce it), and that happens to be one of the areas where China heavily invests and shows quite some progress. Washington terrorizing and snctionizing the large chinese corporations, e.g. Huawei, will turn out to put oil into the fire of the chinese determination.
Q: What operating systems or distributions do you use?
A: For my development workstation and for my servers: some systemd-free Linux Debian variant and BSD. For sensitive servers I tend to do some hardening and to configure and build my own kernel, usually a rather minimal one. Btw, the BSD part is largely based on BSD often being specified by clients, funnily usually not OpenBSD but rather FreeBSD.
Q: Why FreeBSD instead of OpenBSD? Just preference or is there something about OpenBSD you dislike?
Misunderstanding. It’s not me who made/makes that decision but the clients. Their reasons are usually unknown to me and often even to themselves; I guess it boils down to “that’s our company standard (for whatever reason)” or sometimes even simply to FreeBSD being more widely known. But that’s just my guess. I’m not at all opposed to OpenBSD nor do I think that either one of those BSDs is generally better; they both have their stronger and weaker points but are generally good (and better than linux I’d say). Side note: I’m a proponent of the cathedral but I recognize that the bazaar has opened doors and created visibility; but then I’m not about evangelizing FOSS (which boils down to politics) but rather about quality and safety (which boils down to engineering).
Q: What do you use your VPSes for?
A: The usual stuff, name servers, web servers, etc. and of course some testing, although for development I only use systems on an air gapped LAN, both as often required by my clients and as dictated by reason. Btw, I always have some more or less current ARM and MIPS boards in my network too because where and when sensible I like to support not only X86. Once a reasonable (and reasonably priced) Risc-V board becomes available I’ll add it to the collection.
Q: Tell us about your benchmark.
A: Kind of. I’ll tell you about it when v. 2 is ready (which will be soon) but I can tell you why I wrote it in the first place. Multiple reasons, one of them being that I don’t like shell scripts, let alone downloaded ones, on some of my systems, plus the fact that I don’t like bash and do not have it at all on quite a few of my systems. Another one, and a more important one, being that most widely used benchmarks either don’t deliver what they seem to promise and/or do testing in a way that I consider unsatisfactory. Example: You think you can benchmark NVMEs using shell scripts (slow, often unpredictable) which call (actually often abuse) system programs (like e.g. wget) and a ‘time’ call? Really? Think again! If you want to test those blazingly fast devices which were not even imaginable when many system programs were written, you need millisecond resolution at the very least.
I won’t bore you with more examples of problems, let it suffice to say that I wanted something better and more reliable. Plus I wanted to be able to not only simply benchmark something but also to have a deeper look and dig at it. Example: Does that VPS disk seem to be so fast because it is so fast (e.g. NVMe based, low sharing) or because it is so well cached, and if so, what kind of cache, OS or hardware controller and what size?
Another main reason was the fact that on a VPS, unlike on a dedi, you have “neighbors” and your benchmark shouldn’t behave like an ignorant pig. This is btw. not just a question of politeness but also a technical one. Example: If your benchmark behaves like a pig it likely will overload and/or crash well balanced buffering that was designed to serve many VPSs on a node which may lead to rather worthless pseudo results.
Q: What are your thoughts on LowEndTalk/LowEndBox? Why do you come here and how do you view the community?
A: First I should note that I have no need for many VPSs, so my interest is not merely to find the cheapest VPSs. I got most of the (not so many) VPSs I needed based on the advice of a quite experienced friend&colleague plus I tend to rather pay 5$ more per year and to stick with a few providers I’ve had a positive experience with. The only 2 outliers are HostingSolutions, where I have a dirt cheap storage VPS, and NexusBytes whose first VPSs (here) I benchmarked and reviewed; as it was a surprising and surprisingly positive experience and his nodes are super-fast Ryzen based beasts so I couldn’t resist to buy one. As for @cociu I find it interesting because he delivers quite some bang for the buck but of course for ultra-low prices one can’t possibly expect top notch everything (fast nodes, fast network, fast and good support) – yet quite a few people expect exactly that. You get what you pay for and with HostSolutions you actually get quite a bit for very little money. Also since MikePT helps them with support things have gotten better.
My main interest in LET always was based in some curiosity in and love for the whole hosting business and I expected to learn some things here (I did and still do) and to share some of what I know as well as to have some interesting discussions. But frankly, LET began to turn me off with its IMO not that great admin, diverse, uhm, not exactly nice things (often CC related), the attempt of the former admin to bleed it dry and to gain from it, and other issues. In short I was about to leave at the time @jbiloh took it over. At the beginning I very much distrusted him as I saw him only as “the evil guy from ignorant CC”, but when I saw that he seemed to really dedicate resources, time, efforts, and some love it seems to LET/LEB I thought that we shouldn’t be holier than Jesus Christ and that we should give the “sinner” who seemed to have changed his attitude a fair chance. Well, I’m happy to report that my trust was well placed; @jbiloh really earned it. It’s amazing and a joy to see how he turned LET around. And I think we should support his and the teams efforts; that’s also why I accepted the interview request although I’m still convinced that most people – understandably – have next to no interest in me, but if the LET team and/or @jbiloh ask me for something I’ll do it if I’m able to.
So, a big “Thank you!” to @jbiloh, the LET admins, mods, and pandas as well as to all 3 readers of this interview.
Related Posts:
I also enjoy using LEBs! I have an empire of little guys for VPNs, larger guys for apps, and big guys for databases and storage. When I'm not in front a screen I'm into German Shepherd dogs, quality knives, target shooting, theology, tabletop roleplaying games, and forest hiking.
I enjoy writing technical articles here on LowEndBox to help people get more out of their systems. You can find me on LowEndTalk @raindog308
- Should You Change Your Windows RDP Port?Yes. - April 23, 2024
- DediRock: Not Just Dedicated Servers!They Have Cheap VPSes in Buffalo and Los Angeles, Too! - April 23, 2024
- You Don’t Want to Start a Hosting Company: A Parallel From History - April 22, 2024
Thanks for taking part in the LowEndBox interview series, JSG! I really enjoyed reading your interview.
Great interview guys!