So you want to be a fast engineer: critical careers of the future

With an eye for design, cartoonists know excellently how stories are given shape in a concise manner. Recently, cartoonist extraordinaire Roz Chast appeared in the New Yorker prompt DALL-E images and I was immediately drawn to her clues beyond the actual output of the machine.

The title of the article, “DALL-E, make me another Picasso, please” is a pun like the old one Lenny Bruce joke about a genie in a bottle that gives an old man everything he wants. The old man asks the ghost to “make me a malted” and poof! the ghost turns him into a milkshake.

Like the mind of the mind, AIs are powerful but unwieldy and prone to abuse, making the advocacy of a fast engineer a new and important task in the field of data science. These are people who understand that when drafting a request, they will rely on artful skill and perseverance to get a good (and non-harmful) result from a machine’s mysterious soul. The best AI prompt engineers would be the ones who would actually consider whether there is a need for more derivative Picasso art, or what obligations should be considered before asking a machine to plagiarize a famous painter’s work.

Lately, concerns have arisen about whether DALL-E will change the already perennially muddy definition of artistic genius. But asking who can be called a creative misses the point. What is art, and who may claim the title of artist, are philosophical (and rarely ethical) questions that have been debated for millennia. They do not address the fundamental fusion that is taking place between data science and humanities. Successful fast crafts, whether DALL-E or GPT-3 or a future algorithm-driven image and language model, will require not only an engineer to understand how machines learn, but also an arcane knowledge of art history, literature and library science as well.

Artists and designers who argue that this kind of AI will end their careers have certainly invested in how this integration will unfold. Vox recently published a video titled “What AI art means to human artists‘ who explore their fear in a way that recognizes that a very real evolution is underway, despite the current lack of ‘fast craft’ and wordsmiths. People are just starting to realize that we can get to a point where trademarking a word or phrase wouldn’t protect intellectual property in the same way it does today. What aspect of a prompt could we even copyright? How would derivative works be recognized? Could there be a metadata tag on each image indicating whether it is “suitable or allowed for AI consumption?” No one seems to mention these speed bumps in the rush to get a personal MidJourney account.

Alex Shoop, an engineer at DataRobot and an expert in AI system design, shared some thoughts on this. “I think an important aspect of the ‘engineer’ part of ‘prompt engineer’ is following best practices such as robust testing, reproducible results, and using technologies that safe,” he said. “For example, I can imagine a quick mechanic setting up many different prompt texts that are somewhat varied, such as ‘cat with red balloon in a backyard’ vs. ‘cat with blue balloon in backyard’ to see how small changes would lead to different results, even though DALL-E and generative AI models are unable to create deterministic or even reproducible results.” Despite this inability to create predictable artistic results, Shoop says he feels it at least testing and monitoring the experiment setups should be a skill he would expect to see in an actual “fast engineer” job description.

Before the advent of advanced graphics and user interfaces, most science and engineering students saw little need to study fine arts and product design. They weren’t as utilitarian as code. Now technology has created a symbiosis between these disciplines. The writer who contributed the original descriptions of the reference text, the cataloger who constructed the metadata for the images as they were scraped and then dumped into a repository, the philosopher who evaluated the bias in the dataset all provide the necessary perspectives in this brave new world of image generation.

The result is a fast engineer with a combination of similar skills who understands the consequences of OpenAI using more male than female performers. Or if the art of one country is more represented than that of another. Ask any librarian about the complexities of cataloging and categorizing as it has been done for centuries and they will tell you: it is meticulous. Prompt engineering requires attention to relationships, subgroups, and location, along with the ability to investigate censorship and respect copyright laws. While DALL-E was trained on representative images of the Mona Lisathe people in the loop with an awareness of these details were critical to reducing bias and encouraging fairness in all outcomes.

It’s not just abusive abuses that can be easily imagined. In a fascinating turn of events there are even multi-million dollar art forgeries is reported by artists who use AI as their medium of choice. All huge data sets or large networks of models, buried deep in the data, contain intrinsic biases, gaps in labeling and outright fraud that challenge quick ethical solutions. Natalie Summers of OpenAI, who runs Instagram from OpenAI account and is the “man in the loop” responsible for enforcing the rules supposed to guard against output that could damage reputations or cause outrage, out of similar concerns.

This leads me to the conclusion that to be a fast engineer is to be someone who is not only responsible for making art, but is also willing to serve as a gatekeeper to prevent abuses such as forgeries, hate speech, copyright violations, pornography, deepfakes and the like. to prevent. Sure, it’s fun to create dozens of weird, slightly disturbingly surreal Dada art “products,” but there should be something more captivating buried under the mound of foam that results from a throw-away visual experiment.

I believe DALL-E has brought us to a turning point in AI art, where both artists and engineers need to understand how data science manipulates and enables behavior, while also understanding how machine learning models work. To design the output of these machine learning tools, we need experience beyond engineering and design, in the same way that understanding the physics of light and aperture photographic art goes beyond the mundane.

This diagram is an abbreviation of Professor Neri Oxman’s ‘Creativity Cycle’.” Her work with the Mediated Matter research group at the MIT Media Lab explored the intersection of design, biology, computer science, and materials technology with a view to how all these fields interact optimally. Also for a “fast engineer(a pre-existing job title that has yet to be formally embraced by a discipline), be aware of these intersections as wide as hers. It is a serious job with multiple specialisms.

Future DALL-E artists, whether self-taught or skilled, will always need the ability to communicate and design an original point of view. Like any librarian with image metadata and management skills; like any engineer who can structure and test reproducible results; like historians who can link Picasso’s influences to what was happening in the world as he painted about war and beauty, “prompt engineer” will be an artistic career of the future, requiring a mix of scientific and artistic talents that the algorithm will lead. It will remain humans injecting their ideas into machines in the service of the newer and ever-changing language of creation.

Tori Orr is a member of DataRobot’s AI Ethics Communications team.