Not sure why but this just popped up on my feed today.
Rework is a huge one. We (well, at least me as I'm able to easily catch them) are already seeing an unprecedented number of bugs in corporate web sites. Testing time has not kept pace and absolutely needs to increase. And even if identified by a QA team, this can become a circle of death.
Related perhaps or maybe its own detriment is debugging. AI generated code is much harder to fix when it does break. As a result these bugs often remain "buggy" for longer periods of time. Even more so for internal corporate apps.
Combined not only does that create user frustration but also increases the ITSec risk/effort/mitigation. As that is my passion, I'd be highly concerned about that alone. Budgets will have to increase in ITSec staffing as very few run the hacking scanners on 100% of newly implemented code - its hard. First identify the newly introduced bug "on the bench", verify it as valid or a false positive, and then throw it back to the rework pile. All the while your CIO is pressuring you to show them results of your new, expensive, AI code.
Forgot to mention (re: rework) I'm using BMAD METHOD (a rigorous AGILE enforcement that guard rails AI slop). It is much more familiar to how I used to run my team back in the day. I'm getting very good results.
Hey Kevin, as a former CISO, I don't know if you'd agree with me, but the biggest problem I'm thinking about is hostile prompt injection. Have you heard about embedded instructions designed to break security. For example, white font (with jail break prompt) against a white background. AI-based screen scrapers come by and those instructions can be used to exploit a user/company. It's a complete nightmare. Have you heard about this new problem?
Not that specifically but it is not surprising. It sounds like the next logical step beyond command/SQL injection, hidden fields to facilitate the same, XSS, and buffer overflow- those were (and still are) the big ones from a web app perspective. Although not through direct hands-on those hacks are expanding into AI LLM's now too which on its own is a concerning thought. With the constant evolution these days corporations use smart vulnerability scanners that are keeping pace for the most part. Many are cloud based - i.e. Tripwire, Nessus, Appscan, OWASP, etc. Think like anti-virus - they can scan for every known exploit and can run for hours. But the real work comes from triage (i.e. do I care if session cookies are not encrypted - the risk is minimal, low risk/high cost. Then if a priority do the validation it's real. Or is there mitigating controls, such as an independent Intrusion Prevention system in front of the app.
And then all of the ancillary stuff - routers, firewalls (both inbound and outbound), real time network monitors, DDOS mitigation, software updates, change management, redundancy (both HW and SW - i.e. 2 different IPS systems, etc.), and so on.
While my experience is a few years old now (an eternity in computer terms), it wasn't necessary to get intimate with the threats as the tools did that. It is occasionally though if using Metasploit but I'd guess that will get old soon enough. Logically the scanners become AI's scanning other AI's.
Anyway, I'm rambling now but it is a long-connected chain - your defense relies upon the weakest link. So it all has to be given consideration. I'm not sure honestly how much of this applies to your case though.
Interesting. My last 20 years have been in a Java world (www.opers.org) and did a lot of C++, assembly (my fav - both early in my career), and in the later years some ASP using Visual Studio.. We were just a few years into code generators when I was working and no doubt they are a lot better. But it was often "that's nice but I really need these tweeks" and our generators at the time had trouble with the few lines manually inserted to comply. Sounds like they are a lot better these days. And personally I did not consider myself to be a "coder" per se (more of a hack and occasional debugger)but had smart folks on my team that could do magic. But more often than not, at that time it took a base-AI generated code set that was manually customized forever thereafter - hence my comments on the rework being expensive. Sounds like they are much better now.
Honestly these LLM's are pretty incredible but they also sound like they have their challenges. I drive a Tesla and follow their self-driving development. It's vision only (multiple cameras, no lidar/radar/ultrasonics) and one huge E2E stack. They say its difficult to correct - essentially feeding the LLM millions of examples of good driving and have the LLM mimic that. But what if it consistently goes too fast - just feed it more often the appropriate behavior instead. But they fix the speed thing and now the windshield wipers don't work or it doesn't yield correctly. It's a constant "2 steps forward, 1 back" but we introduced (or more often RE-introduced) bugs unrelated to the fix. I know, that's an entirely different animal than what you are doing. Musk has said annually since 2016 "we will have full autonomous driving this year". I actually think it performed better before the E2E model started a couple of years ago where they coded things like "a stop sign means stop" and so on - those days are over. It gets even more complicated when the NHTSA tells them "no rolling stops" as real people rarely do so that creates challenges on how to better train the model. I do believe they have manually inserted such logic, again going to the rework question. While they have done some incredible things, they still remain about 95% complete - not nearly good enough for autonomy.
Good stuff. Unfortunately I don't have much hands-on with such tools and essentially none since retiring a few years back. Looks like fun though!
Bumping up a level, I use AI for complex questions but carefully vette those answers. It's hard for me to advise others to do the same as it's a slippery slope. Essentially I'm asking people to "stop and think" - not an easy message. Essentially it makes sense it would be similar with these code generators. I dont trust them without validation that some undesirable behavior has slipped in, unbeknownst to me. That's why I like to validate with vulnerability scanners so much. I don't want to inadvertently learn my input fields are not properly looking for escape characters because I didn't specify that when generating the code. Then again, being away from the coding side (which was a very small part of my personal job back in the day) for several years makes me feel like a dinosaur in that area today. Sure, I can still hack with the best of them but could not build an app today if I wanted to. And like me, I'm sure you see very poorly written web sites more often than not because they are AI-generated and budgets for QA work are tiny. I'm confident the ITSec side is just as neglected.
I assume your RaptureKit code is easily ported. Given any more thought to putting up some dark web locations? I can see value there in the near future. I could help there if it furthers the Kingdom. And I do hope your efforts are not bound financially - I'd hate to see something as unimportant like money hold things up.
LOL → I’m going to borrow this: “Essentially I'm asking people to "stop and think" - not an easy message.” HAHAHAHAHA
FWIW I’m working on re-write for RKPi5 to solve media streaming stability issues with nginx. Also, another project that I’d like your thoughts on. If it’s going to be IP transferred to larger ministries, I want it locked down formally. AGILE with BMAD-METHOD framework.
"Ip transferred to larger ministries" - not following you. You want the same DNS to forward to different mirror IP's? Your site is fully replicated and independent at these larger ministries?
Thanks, Scott, for the summary. In my medicine, certain fields are more adaptable to AI then others, and I welcome relief from tedious tasks. But I don't think a machine will ever be able to give their patient a warm touch, an encouraging smile, or a few words of affirmation and hope that help with their healing.
Hi Connie, you nailed it…nothing can substitute for relationship the way the Lord has purposed them to be. AI is a counterfeit. But many are beginning to see more examples of “AI Psychoses”. AI used for productivity is incredibly useful and well justified. Thank you for your comment.
As is common, I see no mention of the creative abilities of the coders. My assumption is that creativity is not possible by the AI software itself. Are creative developers able to produce unique and/or brand new types of prompts? Obviously, I'm talking beyond my knowledge. I'm one who has done very small amounts of coding. My problem is that coding always put me to sleep.
AI creativity is mimicked during training, so it’s there, but from human origin. Programming is hugely creative, within the strict rules and syntax of the language. I’ve been working in full stack architecture, so on the front end mostly next.js and typescript. On the backend is node.js and mongoDB for the database. AI is getting much more competent in programming, its impressive!
Not sure why but this just popped up on my feed today.
Rework is a huge one. We (well, at least me as I'm able to easily catch them) are already seeing an unprecedented number of bugs in corporate web sites. Testing time has not kept pace and absolutely needs to increase. And even if identified by a QA team, this can become a circle of death.
Related perhaps or maybe its own detriment is debugging. AI generated code is much harder to fix when it does break. As a result these bugs often remain "buggy" for longer periods of time. Even more so for internal corporate apps.
Combined not only does that create user frustration but also increases the ITSec risk/effort/mitigation. As that is my passion, I'd be highly concerned about that alone. Budgets will have to increase in ITSec staffing as very few run the hacking scanners on 100% of newly implemented code - its hard. First identify the newly introduced bug "on the bench", verify it as valid or a false positive, and then throw it back to the rework pile. All the while your CIO is pressuring you to show them results of your new, expensive, AI code.
Good article as always.
Forgot to mention (re: rework) I'm using BMAD METHOD (a rigorous AGILE enforcement that guard rails AI slop). It is much more familiar to how I used to run my team back in the day. I'm getting very good results.
https://github.com/bmad-code-org/BMAD-METHOD/tree/V4/bmad-core/agents
Hey Kevin, as a former CISO, I don't know if you'd agree with me, but the biggest problem I'm thinking about is hostile prompt injection. Have you heard about embedded instructions designed to break security. For example, white font (with jail break prompt) against a white background. AI-based screen scrapers come by and those instructions can be used to exploit a user/company. It's a complete nightmare. Have you heard about this new problem?
Glad your procedure went well!
Not that specifically but it is not surprising. It sounds like the next logical step beyond command/SQL injection, hidden fields to facilitate the same, XSS, and buffer overflow- those were (and still are) the big ones from a web app perspective. Although not through direct hands-on those hacks are expanding into AI LLM's now too which on its own is a concerning thought. With the constant evolution these days corporations use smart vulnerability scanners that are keeping pace for the most part. Many are cloud based - i.e. Tripwire, Nessus, Appscan, OWASP, etc. Think like anti-virus - they can scan for every known exploit and can run for hours. But the real work comes from triage (i.e. do I care if session cookies are not encrypted - the risk is minimal, low risk/high cost. Then if a priority do the validation it's real. Or is there mitigating controls, such as an independent Intrusion Prevention system in front of the app.
And then all of the ancillary stuff - routers, firewalls (both inbound and outbound), real time network monitors, DDOS mitigation, software updates, change management, redundancy (both HW and SW - i.e. 2 different IPS systems, etc.), and so on.
While my experience is a few years old now (an eternity in computer terms), it wasn't necessary to get intimate with the threats as the tools did that. It is occasionally though if using Metasploit but I'd guess that will get old soon enough. Logically the scanners become AI's scanning other AI's.
Anyway, I'm rambling now but it is a long-connected chain - your defense relies upon the weakest link. So it all has to be given consideration. I'm not sure honestly how much of this applies to your case though.
Kevin, check this out. I highly respect this guy. Been watching him for over a year now. Watch the occasional language:
https://youtu.be/8rptE4vVWn4?si=fLioeLO0YXToWtFJ
Interesting. My last 20 years have been in a Java world (www.opers.org) and did a lot of C++, assembly (my fav - both early in my career), and in the later years some ASP using Visual Studio.. We were just a few years into code generators when I was working and no doubt they are a lot better. But it was often "that's nice but I really need these tweeks" and our generators at the time had trouble with the few lines manually inserted to comply. Sounds like they are a lot better these days. And personally I did not consider myself to be a "coder" per se (more of a hack and occasional debugger)but had smart folks on my team that could do magic. But more often than not, at that time it took a base-AI generated code set that was manually customized forever thereafter - hence my comments on the rework being expensive. Sounds like they are much better now.
Honestly these LLM's are pretty incredible but they also sound like they have their challenges. I drive a Tesla and follow their self-driving development. It's vision only (multiple cameras, no lidar/radar/ultrasonics) and one huge E2E stack. They say its difficult to correct - essentially feeding the LLM millions of examples of good driving and have the LLM mimic that. But what if it consistently goes too fast - just feed it more often the appropriate behavior instead. But they fix the speed thing and now the windshield wipers don't work or it doesn't yield correctly. It's a constant "2 steps forward, 1 back" but we introduced (or more often RE-introduced) bugs unrelated to the fix. I know, that's an entirely different animal than what you are doing. Musk has said annually since 2016 "we will have full autonomous driving this year". I actually think it performed better before the E2E model started a couple of years ago where they coded things like "a stop sign means stop" and so on - those days are over. It gets even more complicated when the NHTSA tells them "no rolling stops" as real people rarely do so that creates challenges on how to better train the model. I do believe they have manually inserted such logic, again going to the rework question. While they have done some incredible things, they still remain about 95% complete - not nearly good enough for autonomy.
Good stuff. Unfortunately I don't have much hands-on with such tools and essentially none since retiring a few years back. Looks like fun though!
Bumping up a level, I use AI for complex questions but carefully vette those answers. It's hard for me to advise others to do the same as it's a slippery slope. Essentially I'm asking people to "stop and think" - not an easy message. Essentially it makes sense it would be similar with these code generators. I dont trust them without validation that some undesirable behavior has slipped in, unbeknownst to me. That's why I like to validate with vulnerability scanners so much. I don't want to inadvertently learn my input fields are not properly looking for escape characters because I didn't specify that when generating the code. Then again, being away from the coding side (which was a very small part of my personal job back in the day) for several years makes me feel like a dinosaur in that area today. Sure, I can still hack with the best of them but could not build an app today if I wanted to. And like me, I'm sure you see very poorly written web sites more often than not because they are AI-generated and budgets for QA work are tiny. I'm confident the ITSec side is just as neglected.
I assume your RaptureKit code is easily ported. Given any more thought to putting up some dark web locations? I can see value there in the near future. I could help there if it furthers the Kingdom. And I do hope your efforts are not bound financially - I'd hate to see something as unimportant like money hold things up.
God be with you. I appreciate the feedback.
LOL → I’m going to borrow this: “Essentially I'm asking people to "stop and think" - not an easy message.” HAHAHAHAHA
FWIW I’m working on re-write for RKPi5 to solve media streaming stability issues with nginx. Also, another project that I’d like your thoughts on. If it’s going to be IP transferred to larger ministries, I want it locked down formally. AGILE with BMAD-METHOD framework.
Thanks again,
Scott
"Ip transferred to larger ministries" - not following you. You want the same DNS to forward to different mirror IP's? Your site is fully replicated and independent at these larger ministries?
Thx.
Praying for you brother
I’m praying for your joy brother. Here is some worship music to help kickstart you. 🙌
https://youtu.be/o8Gds6lBick?si=_I2pa8mw5bIIDK7g
Beautiful Carol, thank you!
Thanks, Scott, for the summary. In my medicine, certain fields are more adaptable to AI then others, and I welcome relief from tedious tasks. But I don't think a machine will ever be able to give their patient a warm touch, an encouraging smile, or a few words of affirmation and hope that help with their healing.
Hi Connie, you nailed it…nothing can substitute for relationship the way the Lord has purposed them to be. AI is a counterfeit. But many are beginning to see more examples of “AI Psychoses”. AI used for productivity is incredibly useful and well justified. Thank you for your comment.
As is common, I see no mention of the creative abilities of the coders. My assumption is that creativity is not possible by the AI software itself. Are creative developers able to produce unique and/or brand new types of prompts? Obviously, I'm talking beyond my knowledge. I'm one who has done very small amounts of coding. My problem is that coding always put me to sleep.
AI creativity is mimicked during training, so it’s there, but from human origin. Programming is hugely creative, within the strict rules and syntax of the language. I’ve been working in full stack architecture, so on the front end mostly next.js and typescript. On the backend is node.js and mongoDB for the database. AI is getting much more competent in programming, its impressive!