Journey from Chaos to Management — Introduction to Automation | by Greg Rog | The Startup | Apr, 2024

At present I slept by way of a catastrophe.

At 4:35 a.m. I obtained the textual content message that considered one of my servers is down. Purpose? Trivial. No funds on a related bank card. Did I miss some notifications beforehand? No. I flip all of them OFF by default. I consciously opt-out of any issues that will distract my work and life. And with over 150 subscription companies that my enterprise handles I’d be notified each millisecond. However hey! Shouldn’t notifications that come earlier than the blackout be AT LEAST those that I opt-in for? No.

I’ve robots to cope with them 🤖

At 4:35 considered one of them obtained woken up. He checked the steadiness on my totally different disposable bank cards, took the primary one with adequate funds and plugged it in through internet hosting service billing API. At 4:36 I obtained one other textual content message. Server Standing: up.

I noticed the messages after I went biking, had a bathe and morning espresso, round 8. Good ol’ bots — I assumed.

The primary time the thought resembled me whereas studying about nomadlist ran by Pieter Ranges, who employed over 1000 bots on his website, together with these calculating beer costs 🍺 I picked up the thought and began studying about cron, automation, AWS Lambda, internet scrapers and writing first small items of software program.

Holy sh*t, that’s superior! — I assumed.

I’ve realized to code early in my profession, however then switched to internet design, UX and biz dev. Though I might code something I didn’t discover it too thrilling. It was taking too lengthy and I sort of missed the moment gratification you get from graphic and movement design.

After I found bots and automation it immediately felt just like the lacking piece of the puzzle in my life simply obtained in place.

The outcomes are fast and highly effective and no or little code is required to make software program, that really works.

As I used to be exploring extra instruments and methods I found that just about each daunting a part of my work could be changed with sensible automation. Ranging from organising my schedule and sending invoices to my accountant, to operating a complete on-line enterprise on autopilot, with bots taking good care of purchases and returns, invoicing, customer support and advertising. Outcome? At present I run a number of on-line initiatives together with the one with 300k shoppers and solely few folks on board, because of bots and automation.

Most people I speak to have come throughout the idea of automation. Nonetheless, their understanding of this subject is totally different from what I wish to current, so I imagine it is extremely vital for you to concentrate on how the evolution of automation appears from my perspective and the way I perceive this idea. For me, automation is using obtainable instruments to reduce human involvement in varied areas of on-line initiatives, similar to gross sales, customer support, accounting, or advertising. I wish to share my method and set of methods that I exploit day by day to make my work easier and never contain me the place it isn’t crucial.

A easy instance is dealing with returns the place we have to challenge a corrected bill, obtain affirmation from the client, course of the return, and e book it. Seemingly a easy operation, because it seems, in bigger corporations, it includes a number of positions. Polish airways LOT lately processed my refund for a canceled flight for 7 months. That’s not all; altering firm knowledge on an bill within the Microsoft Azure service took virtually a yr and required the alternate of 151 emails. In the meantime, whereas the thread was open, one of many customer support representatives managed to get pregnant, give delivery, and go on maternity go away. In Vimeo, the same course of remains to be ongoing, I can’t rely the messages, however in the mean time, 14 folks on the Vimeo facet are concerned, together with 4 managers, authorized division personnel, accountants, and a number of other others.

You would say these are procedures of enormous firms. In the meantime, in smaller corporations, regardless that the method could also be sooner, more often than not nothing is described, and every case is taken into account individually, with the height of goals being template responses that barely velocity up the work.

In the meantime, all repetitive duties could be delegated to robots and automatic, virtually solely, even when the method is advanced and requires affirmation exchanges. On one hand, this can be a big money and time saver for corporations the place hours are spent on boring and repetitive duties, and then again, an enormous alternative to realize a aggressive benefit, and even compete with a lot bigger corporations, for smaller ones. One other big advantage of automating boring, monotonous duties is unleashing the potential of people that might be way more inventive and glad with their work.

For years, there was a perception (and few nonetheless assume in another way) that automation requires specialised data and armies of programmers. Certainly, till now, course of optimization was primarily carried out by bigger corporations that employed RPA (Robotic Course of Automation) specialists, individuals who first spent 3 months recognizing a course of, then produced devoted software program for a number of months, after which maintained it indefinitely. You would say that automation was solely obtainable to the biggest and wealthiest.

Most likely essentially the most revolutionary change on this space is the evolution of instruments that help automation and, consequently, the democratization of those methods. At present, actually everybody is ready to automate, and most significantly, reaching a really superior stage doesn’t require any programming data. Moreover, instruments supporting automation are reasonably priced, which additionally ensures broad entry to know-how.

When the Web turned virtually universally accessible, choices for knowledge synchronization within the cloud and the flexibility to collaborate concurrently on the identical knowledge emerged. Initially, there have been mission administration functions after which doc enhancing instruments (Google Apps). At present, even designers can work concurrently on the identical mission, for instance, utilizing Figma. This clearly exhibits a development, main us from a state of affairs the place functions labored independently to the purpose the place they moved to the Web and browser-based variations. This facilitated collaborative work and entry to our knowledge from any machine.

And since we’re speaking about units, we’ve got gone by way of a revolution associated to cell units, or somewhat, multi-platform help. This has made the method of software program improvement and design way more advanced. Now, builders not solely needed to develop the principle model of the applying but additionally handle their cell variations for smartphones and tablets. This led to the technique of designing utility structure utilizing APIs, which stands for Software Programming Interface.

To grasp what an API is, you will need to know that an “interface” is a option to allow communication. And an API is a option to allow communication inside one or a number of functions.

This technique simplified the method of making functions for a number of platforms as a result of builders not needed to create a number of variations of the applying. As a substitute (utilizing a simplified schema), a single server offering person knowledge was adequate for functions created on particular platforms, with a decrease stage of complexity on this case.

Moreover, not solely the builders of a particular utility have entry to a well-prepared API but additionally anybody who obtains entry keys (e.g., API key).

All of this led to the creation of a software program improvement technique referred to as “API First”. From what I’ve simply written, this method is totally justified. If we are able to write one code to deal with a number of platforms, why wouldn’t we do it?

And what does this imply for non-programmers? Effectively, the API-First method can be utilized as a enterprise technique and for using functions — because of automation.

As we’ve got already talked about, APIs enable for knowledge alternate inside functions and between functions. This opens up solely new potentialities. It seems that we are able to join varied companies we use and alternate knowledge between them. This implies we’re not restricted by the features obtainable in chosen functions however have a full vary of potentialities supplied by all of the instruments we use!

How is that this attainable? Effectively, builders present entry to their API not solely to their very own functions (for instance, the cell and browser utility of your financial institution), but additionally… to different builders, who can construct options based mostly on their software program.

These different builders can be us, utilizing instruments specialised in particular duties — sending emails (e.g., Mailchimp), SMS messages (e.g., Twilio), enriching buyer knowledge (e.g., Clearbit), notifying the gross sales division (e.g., Slack), and so forth., and mixing them by way of Automation. With this method, we are going to by no means need to log into the graphical interface of the functions we use, for instance, to ship advertising emails. Automation will do it for us by “logging in” to the API and sending emails of particular content material to a delegated group of recipients. Which means that neither we nor our workers will ever need to log in (and neglect passwords) to the web companies we use, which have good, accessible APIs.

The advantages of this method embrace:

• No want to make use of GUI (reloading pages, gradual software operation, forgetting passwords, logging in once more — all of those waste time!)
• The flexibility to attach the most effective, specialised companies as an alternative of counting on one supplier who could continually lack sure performance
• Reactive work, not proactive work — we don’t need to examine what must be carried out; automation assigns duties that people ought to full within the course of (extra on that later)

Taking a look at this by way of the lens of my earlier definition of automation, a technical facet could be added to the listing:

1. Detailed course of mapping
2. Try and simplify/enhance the method
3. Selecting the best instruments for the suitable duties within the course of
4. Connecting these instruments by way of APIs to work on the proper time, with out the necessity for human intervention and logging into GUI
5. Supervision, error correction, and experiments

Let’s take the instance of organizing a web based occasion, similar to a webinar. The webinar software permits for participant registration, so we’ve got entry to electronic mail addresses and cellphone numbers. With the assistance of API and automation, we are able to routinely switch this person’s knowledge to the Clearbit service answerable for Knowledge Enrichment, which enriches knowledge. By sending, for instance, a company electronic mail tackle, we obtain further details about the particular person signing up for our occasion. These enriched knowledge can be routinely despatched to our CRM system (e.g., Pipedrive), positively impacting the gross sales course of as a result of when contacting such a possible buyer, we’ve got extra details about them to higher tailor our companies to their wants. Alternatively, we are able to use the cellphone quantity to routinely ship a brief SMS reminder in regards to the occasion. This will increase attendance, which may profit us and our potential prospects.

If we mix this with a system that collects buyer knowledge on our web site, similar to Intercom, our salesperson may also obtain exact notifications with info like:

Adam from the Browser firm logged in 10 minutes in the past, he participated within the “Monetary Markets” Webinar and visited the pages: supply, faq, contact. His cellphone quantity is: 888999000

As you may simply think about, utilizing one software on this course of is unimaginable. There is no such thing as a single software that may deal with webinars, web site knowledge, firm communicators, and so forth. And that is the facility of automation — we use varied instruments specialised in particular duties, juggle their knowledge, and the outcomes go precisely the place our staff feels comfy.

So, as you may see, we used the API right here to switch knowledge between totally different companies and thus optimize a course of that was both beforehand carried out manually or not carried out in any respect. This method streamlines present processes or creates new ones. I’ve personally discovered that within the case of on-line companies, this method actually takes our initiatives not simply to the following stage however usually ten ranges increased.

For instance, the initiatives we’ve got undertaken, contemplating the size of operations and their profitability, can simply be in comparison with companies that require the involvement of even a staff of dozens of individuals!

As we established above, trendy functions are more and more speaking by way of APIs. Let’s take Fb for instance. The enterprise logic is answerable for including your put up to a sure kind of database and your feed. This logic could be triggered from varied locations. You’ll be able to add a put up by way of a cell utility or the Fb web site, and as you realize properly, different functions can join with Fb and put up in your behalf.

To implement this logic, you should:

1. Know the place the back-end expects this knowledge (an endpoint URL)
2. Ship the information in a method comprehensible to the back-end (often in JSON format)
3. Authenticate utilizing sure entry knowledge (login, token, and so forth.)

And voila! The put up is in your feed.

Your complete web works this manner. And because of this mannequin, everybody can add, modify, and execute instructions in internet functions.

However as an alternative of changing into a developer and studying the right way to construction, format, and ship knowledge by way of varied APIs, you need to use instruments that can do it for you. I name them LEGO instruments. Whenever you authorize any utility in such instruments, they will carry out varied actions in your behalf. For instance, create a spreadsheet and add rows to it.

Now, let’s take into consideration this extra broadly. If LEGO instruments can join with totally different functions, they will additionally alternate knowledge between these functions. As an example, these instruments can take rows from a spreadsheet and put up them in your Fb feed. Sequencing, timing, delays, circumstances — every thing could be set.

Now think about you wish to collect knowledge out of your spreadsheet and distribute it to all of your social media profiles — Fb, LinkedIn, Instagram, YouTube, abruptly. LEGO instruments assist you to join with quite a few functions and companies you utilize and alternate info between them or work on adjustments in a single software whereas notifying others.

Listed here are just a few instance eventualities:

• Trying to find occasions in your calendar and sending you reminders about a very powerful ones;
• Accumulating duties you enter into Trello and syncing them together with your Google calendar;
• Sending emails to webinar contributors, posting on social media, and assigning duties to your staff on Slack — all concurrently.

The fantastic thing about LEGO instruments is that they broaden the capabilities of the software program you already use, permitting you to realize virtually something you may think about. Typically, the software you might be utilizing could lack only one function, similar to textual content reminders. With LEGO instruments, you may add a textual content messaging service to your present arsenal of choices in just some minutes.

The preferred LEGO instruments are zapier and Make. Its fundamental benefit is that it helps a whole bunch of obtainable functions. It presents a transparent and intuitive interface for creating automations. Make It has a really fascinating, visible interface that means that you can construct automations by actually connecting just a few dots. Moreover, it presents a wealthy, free plan to experiment with automation.

These are the instruments I exploit most frequently, however there are additionally many different instruments, often designed for extra particular duties.

• Parabola — an excellent software for knowledge transformation. This implies reorganizing, splitting, altering the order, formatting, and way more. When you’ve got knowledge from a number of sources or a variety of knowledge that you simply wish to remodel and ship to different instruments, this can be a nice selection.
• Actiondesk — based mostly on a spreadsheet interface, permitting manipulation, transformation, and creating logic inside your knowledge. Should you like spreadsheets, it’s price trying out this software. It means that you can make varied transformations on knowledge and ship it to a different software.
• Retool — this software fetches knowledge from APIs or databases and means that you can remodel it and even ship instructions again for execution. If you wish to create an admin interface in your database or a report panel, this can be a good possibility.

Moreover, there are different instruments, created for particular actions, that I don’t categorize as LEGO instruments — they assist you to mix one or two companies and create one thing new. A great instance is Glide, which helps you to create a cell app from a spreadsheet. There are numerous fascinating instruments of this sort, however that’s a subject for an additional article.

If you wish to study extra about totally different instruments and methods we use for automation and productiveness, be sure to take a look at On a regular basis on Autopilot — our productiveness and automation bible, at present 50% off for Techsistence readers solely 💥 That is the most effective begin to your automation journey. Right here’s few latest critiques from Product Hunt.

I believe considered one of prime books in it’s class. Reveals fascinating method to automation. I get to know bunch of latest instruments from it.
— Ben

Imho this e book and different info merchandise from this authors are the highest quality and most advanded soruce of knowlege about proces automation and productiveness improvment! Shut up and take my cash once more! 😉
— Jakub

Thoughts-blowing data. There’s no fluff solely apply-at-once options. Very dependable. Price each minute and cash spent. I assumed I knew one thing about automation and was having fun with driving a motorbike as an alternative of strolling… these guys confirmed me Floo Powder!
— Karolina

Get 10x extra productive! Think about the essence of our data about automation, excluding difficult methods, which you’ll implement even in the midst of one night. This bundle serves because the cornerstone of our work, offering a sturdy basis for optimising your duties and automating each day actions.

All of this will appear overwhelming. However when you perceive the idea behind LEGO instruments, it is going to be fairly straightforward to implement and study any new instruments. It’s price realizing not less than just a few of them. Typically the applying you wish to automate is out there in Zapier however not in Make. Typically you wish to work with a bigger quantity of knowledge, and Parabola could also be extra handy. Sadly, there isn’t a single LEGO software that meets all of your expectations, however that’s… not a nasty factor. By studying totally different instruments, you encounter numerous eventualities and thought patterns.

After I totally grasped how Zapier works, I used to be very excited. However then I struggled with implementing this idea into my enterprise. Tutorials had been obtainable however taught every thing from the mistaken angle, saying “join X to Y” as an alternative of “obtain X by doing Y.” This made it very troublesome for me to work with these instruments, and it took me over a yr to give you good automation methods and their utilization.

First automation we are going to put together based mostly on some of the common platforms on the earth — Zapier. This software could be very beginner-friendly and means that you can join varied instruments and switch knowledge between them with out coding. Our first automation will present you the right way to work with Zaps, as Zapier calls eventualities, or particular person “robots”.

What is going to we automate? We’ll create a easy automation that can save your Gmail attachments to a Google Drive folder. Then we are going to barely modify it to have a sensible utility by saving the invoices you obtain in your mailbox in a single place.

Begin by making a free account on Zapier. To do that, go to zapier and register, then create a brand new Zap by clicking this button, after which Open Editor, making a Customized Zap.

Now you should add a set off — one thing that can begin your robotic. Seek for and choose the Gmail utility. There’s an experimental AI search above, however we’ll skip it to higher perceive the method

Within the subsequent step, you should join your Google account with the Zapier utility and specify the suitable mailbox. You solely do that as soon as, and subsequent automations is not going to require reconnection

The set off occasion for us is New E mail Matching Search, which means we wish to set off the robotic each time we obtain a message with particular subject and attachment

Subsequent, we are able to fine-tune our set off, and right here Zapier presents a really fascinating chance. We will carry out extra advanced queries, based on the Gmail search documentation. For instance, I used bill OR invoice AND filename:pdf, which suggests amongst messages with attachments, it’s going to solely seek for these containing the phrase bill or faktura (or comparable as a result of I did not use citation marks). Moreover, I’m solely wanting amongst outcomes containing PDF as an attachment

You’ll be able to check your automation with the Check Set off function, and if the mailbox already accommodates components that match the entered phrase, you need to obtain them as outcomes (if there may be nothing, simulate check knowledge by sending your self an bill to the mailbox)

And you need to have the ability to see outcomes

Choose the suitable outcome and click on Proceed.

Nice! Now we have configured our set off, which means we are able to obtain messages with attachments, moreover filtered. On this case, we solely must configure what’s going to enable us to save lots of the attachment to Google Drive. As a part of the Motion (what occurs on account of the set off’s motion, we seek for Google Drive)

Then, we select what we wish to do, which is to add the file to Google Drive

Join your Google Drive account, or choose it if it has been beforehand related

Should you haven’t created the suitable bill folder on Drive earlier than, now go to drive.google.com and create a folder named invoices. If it’s already created, it is possible for you to to specify it as the situation in Zapier

Now click on on the file and from the drop-down listing, choose the Attachment possibility, merely passing the attachment as a file to Drive. Depart the opposite fields unchanged, apart from the File Title area, the place you may get inventive and alter the title of the saved file, utilizing varied metadata from the e-mail, such because the sender’s tackle, combining it with the file title. I added hyphens between them for readability

Tip: Should you paste the next phrase within the Zapier area: {{zap_meta_human_now}}, Zapier will substitute the present date in that place. By formatting names on this method, it is going to be straightforward to search out recordsdata with the right date. Moreover, sorting by title or date will now return neatly organized outcomes, which suggests you need not create further folders (however if you wish to do it, be happy to take action. Attempt modifying the automation to put invoices in a folder with the right title, for instance, January. You have to to create these folders manually or with the assistance of one other automation).

That’s it! Choose Proceed, after which Check & Proceed to examine the automation’s operation. If every thing went easily, you will note successful display screen

Now you may Publish your state of affairs! Check bill is already on Google Drive

Bear in mind to activate the Zap utilizing the swap on the backside in order that it begins analyzing all incoming messages any more. In any other case, the Zap gained’t work!

We ready our first automation in Zapier for a motive. Make, the applying we can be working with now, will not be nearly as good at integrating with Google instruments. You would say that Zapier is healthier aligned with Google, and this can be a good instance of why it’s price having varied automation options and utilizing those that work finest in a given state of affairs. In Make, to make use of Gmail, you should have a paid Google Workspace account.

Though automating with Zapier is extremely easy and intuitive, there are a lot of different instruments on the earth of automation that you’ll use. Why? There are numerous causes. For instance, Zapier is a bit costly, and you will discover cheaper options. Moreover, whereas it presents many fascinating options, different instruments typically work higher and performance barely in another way. In Zapier, though it’s attainable, splitting eventualities based mostly on circumstances will not be essentially the most intuitive.

For instance, we are able to create a bit extra superior state of affairs in Make, that searches for invoices then renames, catalogs and save them to drive, checking if the folder construction is right. It appears like this:

On the left, we’ve got the set off, which listens to emails on Gmail. Then we rename the file, and seek for the suitable folder on drive. The folder ought to comprise month and yr to catalog our invoices, so if it doesn’t exist, we are going to create it within the subsequent step after which put the bill there. You don’t want to grasp precisely what these components imply proper now, however the idea ought to be a bit clearer now.

What is going to we automate this time? We gained’t repeat the identical activity in Make. As a substitute, we are going to add one other automation that can ship a notification on the tenth day of the month, informing our accountant or our electronic mail tackle that the collected paperwork are prepared.

Create an account on Make (that is our particular hyperlink, that grants you 1 free month on the professional plan 🎉)

Create a brand new state of affairs (equal to a Zap) utilizing the Create State of affairs possibility

Click on on the primary circle and choose the Gmail utility

Select the motion Ship an electronic mail

Click on Add within the Connection possibility, title your connection, and like in Zapier, hook up with the suitable Google account

Now add recipients, topic, and content material of the despatched electronic mail together with the hyperlink to the shared Invoices folder on Drive

Choose OK to substantiate the adjustments within the module

Now set the schedule, for the Gmail module, click on on the clock, after which schedule sending this electronic mail on the tenth day of the month

Save the module by clicking OK

Flip scheduling to ON and also you’re carried out!

It is a tremendous easy state of affairs, however introduces a pleasant idea of scheduling and operating your robots remotely. If you wish to broaden it, strive sending a Slack message to your self with affirmation, after you ship electronic mail. Small tip:

You have already got your first automations in your account — congratulations! As you may see, it’s youngster’s play. After all, these are simply the fundamentals, and in observe, automations could be way more advanced, together with, for instance:

• Many extra modules and routers, i.e., paths
• Conditional directions “carry out solely when the information accommodates X”
• Features, for instance, looking out by way of returned knowledge
• Knowledge completion, for instance, with the present date or knowledge calculation utilizing mathematical operations
• Error dealing with and modules for testing
• Customized scripts, for instance, JavaScript
• Modules permitting knowledge reformatting (parsing)
• Modules able to iterating, for instance, over data in a database or rows in Google Sheets
• And so forth

There are many potentialities. However let’s break down what actually occurs within the eventualities we created. The truth that Zapier or Make permits us to decide on the Gmail module merely implies that the software can combine with it. How? After all, by way of APIs! For instance, Gmail has an API, the place we are able to check with the suitable endpoint and hearken to incoming emails. As soon as we retrieve details about the incoming message, the following endpoint permits us to obtain the attachments. Zapier, which we use, simply has this stuff built-in, however they work on the usual Gmail or Slack API. We work precisely like a programmer who would put together an integration for us, besides we use a visible builder and a set of ready modules. The account connection we made is basically the authentication within the API utilizing one of many strategies mentioned earlier. It’s programming with out programming!

Make, like Zapier, presents the likelihood to make use of a whole bunch of ready-made eventualities or import any blueprints ready by us or others!

As you in all probability guess, through the years we’ve created numerous automation for our wants. You could be stunned, however at present over 3000 robots work for me, performing a whole bunch of 1000’s of operations each day. That is simply one of many Make accounts I exploit, the place I course of practically half 1,000,000 operations month-to-month!

However that is only the start! Slightly below a yr in the past, we realized how helpful NLP and AI might be to assist us work with automations. It turned out that enter knowledge for automations, which wanted to be formatted in a particular method (we ship them the so-called JSON object), might be ready by AI! What does this imply in observe? It means that you would be able to say or write, for instance:

Add shopper John Doe to CRM

After which the language mannequin will handle translating this into the correct format, which means into the “automation language,” after which ship it to a prepared state of affairs on Zapier or Make! It’s actually superb how huge the chances are when utilizing such an method.

Sadly, there wasn’t a software that allowed us to speak AI with automations on this method, so over a yr in the past, we began engaged on our personal resolution to assist us with this.

And that’s how Alice app was born.

Along with having the ability to chat with Alice like with ChatGPT, her largest benefit lies within the means to remotely set off our automations, and consequently — carry out actions! ChatGPT presents this idea additionally with GPTs and their API commonplace, nevertheless, this method proved to be a lot much less environment friendly thus far, not less than for our use-cases.

Way forward for distant actions in Alice could be very promising, and possibly we are going to, in time, begin including native integrations to Alice. Additionally, efficiency of LLMs is quickly growing, and likewise entry to native fashions in Alice is a chance. At present with Groq we’re in a position to carry out operations sooner than ever earlier than, and this may evolve. Check out this command to Spotify and the velocity of its execution.

On this context, I’ve ready some tutorials for you, displaying how one can combine the Alice app with Make or Zapier. Please, see the way it works in observe!